10.4.1 Overview and feature description of QUEUE jobs and submit jobs
This subsection provides an overview of QUEUE jobs and submit jobs, and describes their features.
- Organization of this subsection
(1) What are QUEUE jobs?
QUEUE jobs are executed via a specified queue. Each QUEUE job is registered in a queue, and is then executed by an agent connected to the queue. In a system that uses the standard database, QUEUE jobs are used when JP1/AJS3 is linked with another system (JP1/NQSEXEC or JP1/OJE, for example). QUEUE jobs can be used in JP1/AJS3 to implement the following processing:
-
Create separate queues in which to register jobs according to their purpose.
-
Connect multiple agent hosts to a queue and distribute jobs based on the priorities assigned to the agent hosts.
The same processing can be realized in a standard database configuration with PC jobs, Unix jobs, action jobs, and custom jobs, using an execution agent group rather than queues. We recommend that you use an execution agent group when defining new PC jobs, Unix jobs, action jobs, or custom jobs.
(2) What are submit jobs?
In JP1/AJS3, jobs can be defined and executed in a jobnet, or they can be executed singly by command, via direct registration in a queue in JP1/AJS3 - Manager. The jobs in the queue are sent to an agent connected to the queue for execution. If you specify the agent when directly registering a job for execution, it will be placed in the default queue. The act of directly registering a job in a queue is called submitting a job, and the job itself is referred to as a submit job. To submit a job, you use the jpqjobsub or other command.
Jobs in a jobnet are executed by a scheduler service, which manages the job execution times and execution sequence, and runs each job according to its schedule. Submit jobs, on the other hand, are submitted and executed by a user executing a command. The execution start time can be specified as a command option.
(3) Queue and agent configurations
The following describes how jobs are executed using a queue and agent configuration. The following figure illustrates the queue and agent concept.
|
(a) What is a queue?
A queue is an area in which registered jobs can be placed temporarily so that the number of QUEUE jobs and submit jobs to be executed concurrently in JP1/AJS3 will not become overly large. One or more agents can be connected to the queue. When the user executes a submit job with an agent host name specified in the -ah option of the jpqjobsub command, the job is registered in the internal queue whose name matches the specified agent host name. This matching queue is the default queue. Multiple agents can also be connected to a default queue. Jobs are queued in order of registration and are transferred sequentially to the connected agent hosts for execution.
You can define the maximum number of jobs that can be registered in a queue. You can also execute multiple jobs concurrently. However, if the number of jobs being executed exceeds the system's processing capacity, performance could decline and errors due to resource shortages could occur. For example, if you attempt to execute more jobs than can be handled comfortably with the available system memory, memory swapping increases and execution performance declines significantly. Continuing to execute more jobs in this situation will cause problems such as jobs terminating abnormally due to insufficient memory.
By using a queue, these sorts of problems can be avoided and jobs can be executed more efficiently. When the number of jobs to be executed concurrently reaches a set limit, subsequent jobs are made to wait and are passed in turn to the agent as preceding jobs finish.
- Cautionary note
-
Submit jobs are not necessarily transferred and executed in order of registration in the queue. Submit jobs might switch places in the queue if, for example, the next submit job cannot be transferred because the execution-locked resource specified when it was registered is being used by another submit job. For details about the execution sequence of submit jobs, see 7.1.1(4)(b) Notes on the execution order of submit jobs in the Job Management Partner 1/Automatic Job Management System 3 System Design (Configuration) Guide.
The following figure shows how a queue works.
|
(b) What is an agent?
An agent is a host that executes jobs. The term can denote a component of the system configuration, and it can denote the functionality and control inherent in an agent. QUEUE jobs and submit jobs are transferred sequentially from the queue to an agent connected to the queue, and are executed by the agent.
You can define how many jobs an agent can execute concurrently. Like the maximum number of jobs that can be registered in a queue, you must set the maximum number of concurrently executable jobs in an agent in line with the system processing capacity.
When multiple agents are connected to the queue, you can assign different priorities to them. This means that if an agent already has a large number of Now running jobs compared with the number of jobs it can execute concurrently, the next job will be sent to the agent with the next highest priority. For details, see (6) Distributing jobs among execution hosts (agent hosts).
You can also distribute jobs within one machine. To do so, you define multiple aliases of the same machine (single IP address) as agents, and register different kinds of jobs in each agent. For details, see 7.1.1(3) Running multiple execution hosts on the same computer in the Job Management Partner 1/Automatic Job Management System 3 System Design (Configuration) Guide.
(c) Flow of execution of QUEUE jobs and submit jobs
The following table describes the processing flow, and the manager/agent relationship, when QUEUE jobs and submit jobs are executed in JP1/AJS3. The descriptions here assume that the job ends normally. In fact, a job could end abnormally or some other situation could apply.
Job status |
Description |
|
---|---|---|
Now queuing |
|
The manager registers the job as requested, and places it in the queue. The job status becomes Now queuing. |
Now running |
|
The manager takes the job from the queue and transfers it to an agent, requesting the agent to execute the job. |
|
The agent OS starts running the executable file or script defined in the job. The job status becomes Now running, and the agent waits for the job to end. |
|
|
When the job ends, the agent records its return code and execution result. |
|
Ended normally |
|
The agent sends the job's return code and execution result to the manager. On receipt, the job status changes to Ended. |
(4) Execution environment for QUEUE jobs and submit jobs
To use QUEUE jobs or submit jobs in JP1/AJS3, you must define a queue using the jpqimport command. You can change the queue and agent configuration while JP1/AJS3 services are active, using the jpqqueadd or other command. For details about setting up the execution environment, see 7.1.2 Setup for using QUEUE jobs and submit jobs in the Job Management Partner 1/Automatic Job Management System 3 System Design (Configuration) Guide.
For details about the jpqimport and jpqqueadd commands, see 1.5.9(2) Commands available in an execution environment for queue or submit jobs in the manual Job Management Partner 1/Automatic Job Management System 3 Command Reference 2.
To execute a QUEUE job that links with another system such as JP1/NQSEXEC or JP1/OJE, you must first create a queue in the other system. For details about linkage with another system, see the Job Management Partner 1/Automatic Job Management System 3 Linkage Guide.
For details about changing the job execution environment to suit your system operation, see 7.1.2(1) Setting the execution environment for QUEUE jobs and submit jobs in the Job Management Partner 1/Automatic Job Management System 3 System Design (Configuration) Guide.
(5) Restricting the number of concurrent jobs
You can restrict the number of QUEUE jobs or submit jobs that can be executed concurrently in JP1/AJS3 according to resource availability at the execution host.
(a) Maximum number of concurrently executable jobs
When jobs are managed in a queue, you can restrict the number of jobs executed at the same time by setting a maximum number of concurrent jobs in the definition information in the agent host. The maximum is specified by period, in units of 30 minutes. You can set different limits for different time periods to distribute the processing load across the system.
The limits apply only to QUEUE jobs and submit jobs executed in JP1/AJS3. Set the limits on an agent basis. You cannot set a maximum number of concurrently executable jobs per queue. When the limit is reached, subsequent jobs remain in Now queuing status (waiting to execute) until the active jobs finish.
(b) Methods of setting the maximum number of concurrently executable jobs
There are two ways of setting the maximum number of concurrently executable jobs: You can set a parameter in the agent host definitions in the file jpqsetup.conf (execution environment configuration definition file for QUEUE jobs and submit jobs) and then execute the jpqimport command, or you can set the relevant command option when adding an agent host using the jpqagtadd command.
If you use the jpqimport command method, the default setting for the maximum number of concurrently executable jobs in the execution environment configuration definition file for QUEUE jobs and submit jobs (jpqsetup.conf) is as follows: limit start time 0000, limit end time 0000, and maximum jobs 5 (maximum of five concurrently executable jobs, 24 hours a day). You can change this setting later as needed, using the jpqagtalt command. If you add an agent host using the jpqagtadd command method, but do not specify the maximum number of concurrently executable jobs (-cp option), the default setting of 00:00-00:00=0 (zero concurrent jobs, 24 hours a day) is assumed.
A maximum of zero concurrently executable jobs means that no jobs can be executed. To enable job execution by the new agent host, specify 1 or higher, or change the setting afterwards using the jpqagtalt command. For an example of specifying a maximum number of concurrently executable jobs using this command, see also jpqagtalt in 3. Commands Used for Special Operation in the manual Job Management Partner 1/Automatic Job Management System 3 Command Reference 2.
(c) Load distribution based on the maximum number of concurrently executable jobs
When multiple agent hosts are connected to the queue, you can distribute the processing load among the agents, based on the agent usage rates (ratio of active jobs to the maximum number of concurrent jobs). For details, see (6) Distributing jobs among execution hosts (agent hosts).
(d) Cautions on setting concurrent execution limits
For cautionary notes about concurrent execution limits, see Notes on setting a maximum number of concurrently executable jobs in 7.1.1(4)(a) Maximum number of concurrently executable jobs (QUEUE jobs and submit jobs) in the Job Management Partner 1/Automatic Job Management System 3 System Design (Configuration) Guide.
(6) Distributing jobs among execution hosts (agent hosts)
(a) Distributing jobs among multiple execution hosts (agent hosts) connected to the queue
When you register a QUEUE job or submit job for execution in JP1/AJS3, the job is placed in a queue (specified or default) managed by the job execution control. You can connect multiple agent hosts to the queue, specifying a different priority for each. Jobs are distributed to the agent hosts based on their specified priorities.
Specify the agent host priorities to suit your system environment. If you assign different priorities, each job will be sent for execution to the agent host hat has the highest priority. When that agent's usage rate# reaches 100%, the next job is sent to the agent host that has the next highest priority. For example, if the agent host with the highest priority is already fully occupied or is in failed or stopped state and cannot run the job, the agent host with the next highest priority can run it instead. Setting different priorities is useful for this type of system operation.
If you set the same priority for all agent hosts, each job will be sent to the agent host that has the lowest agent usage rate. If no particular agent host takes precedence, jobs are distributed evenly among the agent hosts. Setting identical priorities is useful for this type of load-balancing system.
- #
-
Agent usage rate is the ratio of active jobs to the maximum number of concurrent jobs at an agent host. It is given by the following equation, rounded to four decimal points:
Agent usage rate = number of active jobs / maximum number of concurrent jobs
- Cautionary note
-
When jobs are to be distributed, there might be multiple execution agents that have the same priority and agent usage rate. In this case, to balance the load equally, the best way is to distribute the jobs by selecting the execution agent whose idle time is longest.
For details on defining priorities, see 7.1.7 Method of determining the agent host to which a job is distributed in the Job Management Partner 1/Automatic Job Management System 3 System Design (Configuration) Guide.
(b) Detecting failures and distributing jobs to execution hosts (agent hosts)
In the following two cases, there may be a delay of up to 15 minutes in distributing QUEUE jobs or submit jobs in JP1/AJS3 to execution hosts (agent hosts) that are operating normally. During this period, the jobs remain in Now queuing status:
-
When a failure occurs simultaneously on three or more execution hosts
-
When three or more execution hosts have stopped
In such cases, restart the affected execution hosts.
(7) Exclusive control when executing submit jobs (execution-locked resource)
When submit jobs access the same resource, you can control their execution by specifying a preset execution-locked resource when you submit each job.
For example, suppose that multiple jobs (programs) access the same resource, but do not have a process (function) to lock that resource. In this case, you can define the shared resource as an execution-locked resource, using any name of your choice. Specify the same execution-locked resource when you submit each job. Submitted jobs for which the same execution-locked resource is specified cannot be executed at the same time, even if the maximum number of concurrent jobs is set to 2 or higher.
Use the jpqimport or jpqresadd command to specify an execution-locked resource.
- Cautionary note
-
An execution-locked resource cannot be specified for QUEUE jobs. The execution-locked resource specification applies only to jobs submitted by the jpqjobsub command.