10.4.1 Overview and feature description of QUEUE jobs and submit jobs

(a) What is a queue?

A queue is an area in which registered jobs can be placed temporarily so that the number of QUEUE jobs and submit jobs to be executed concurrently in JP1/AJS3 will not become overly large. One or more agents can be connected to the queue. When the user executes a submit job with an agent host name specified in the -ah option of the jpqjobsub command, the job is registered in the internal queue whose name matches the specified agent host name. This matching queue is the default queue. Multiple agents can also be connected to a default queue. Jobs are queued in order of registration and are transferred sequentially to the connected agent hosts for execution.

You can define the maximum number of jobs that can be registered in a queue. You can also execute multiple jobs concurrently. However, if the number of jobs being executed exceeds the system's processing capacity, performance could decline and errors due to resource shortages could occur. For example, if you attempt to execute more jobs than can be handled comfortably with the available system memory, memory swapping increases and execution performance declines significantly. Continuing to execute more jobs in this situation will cause problems such as jobs terminating abnormally due to insufficient memory.

By using a queue, these sorts of problems can be avoided and jobs can be executed more efficiently. When the number of jobs to be executed concurrently reaches a set limit, subsequent jobs are made to wait and are passed in turn to the agent as preceding jobs finish.

Cautionary note

Submit jobs are not necessarily transferred and executed in order of registration in the queue. Submit jobs might switch places in the queue if, for example, the next submit job cannot be transferred because the execution-locked resource specified when it was registered is being used by another submit job. For details about the execution sequence of submit jobs, see 7.1.1(4)(b) Notes on the execution order of submit jobs in the JP1/Automatic Job Management System 3 System Design (Configuration) Guide.

The following figure shows how a queue works.

Figure 10‒24: Processing of a queue

(b) What is an agent?

An agent is a host that executes jobs. The term can denote a component of the system configuration, and it can denote the functionality and control inherent in an agent. QUEUE jobs and submit jobs are transferred sequentially from the queue to an agent connected to the queue, and are executed by the agent.

You can define how many jobs an agent can execute concurrently. Like the maximum number of jobs that can be registered in a queue, you must set the maximum number of concurrently executable jobs in an agent in line with the system processing capacity.

When multiple agents are connected to the queue, you can assign different priorities to them. This means that if an agent already has a large number of Now running jobs compared with the number of jobs it can execute concurrently, the next job will be sent to the agent with the next highest priority. For details, see (6) Distributing jobs among execution hosts (agent hosts).

You can also distribute jobs within one machine. To do so, you define multiple aliases of the same machine (single IP address) as agents, and register different kinds of jobs in each agent. For details, see 7.1.1(3) Running multiple execution hosts on the same computer in the JP1/Automatic Job Management System 3 System Design (Configuration) Guide.

(c) Flow of execution of QUEUE jobs and submit jobs

The following table describes the processing flow, and the manager/agent relationship, when QUEUE jobs and submit jobs are executed in JP1/AJS3. The descriptions here assume that the job ends normally. In fact, a job could end abnormally or some other situation could apply.

Table 10‒10: Processing flow when executing QUEUE jobs and submit jobs

Job status		Description
Now queuing		The manager registers the job as requested, and places it in the queue. The job status becomes Now queuing.
Now running		The manager takes the job from the queue and transfers it to an agent, requesting the agent to execute the job.
		The agent OS starts running the executable file or script defined in the job. The job status becomes Now running, and the agent waits for the job to end.
		When the job ends, the agent records its return code and execution result.
Ended normally		The agent sends the job's return code and execution result to the manager. On receipt, the job status changes to Ended.

(a) Maximum number of concurrently executable jobs

When jobs are managed in a queue, you can restrict the number of jobs executed at the same time by setting a maximum number of concurrent jobs in the definition information in the agent host. The maximum is specified by period, in units of 30 minutes. You can set different limits for different time periods to distribute the processing load across the system.

The limits apply only to QUEUE jobs and submit jobs executed in JP1/AJS3. Set the limits on an agent basis. You cannot set a maximum number of concurrently executable jobs per queue. When the limit is reached, subsequent jobs remain in Now queuing status (waiting to execute) until the active jobs finish.

(b) Methods of setting the maximum number of concurrently executable jobs

There are two ways of setting the maximum number of concurrently executable jobs: You can set a parameter in the agent host definitions in the file jpqsetup.conf (execution environment configuration definition file for QUEUE jobs and submit jobs) and then execute the jpqimport command, or you can set the relevant command option when adding an agent host using the jpqagtadd command.

If you use the jpqimport command method, the default setting for the maximum number of concurrently executable jobs in the execution environment configuration definition file for QUEUE jobs and submit jobs (jpqsetup.conf) is as follows: limit start time 0000, limit end time 0000, and maximum jobs 5 (maximum of five concurrently executable jobs, 24 hours a day). You can change this setting later as needed, using the jpqagtalt command. If you add an agent host using the jpqagtadd command method, but do not specify the maximum number of concurrently executable jobs (-cp option), the default setting of 00:00-00:00=0 (zero concurrent jobs, 24 hours a day) is assumed.

A maximum of zero concurrently executable jobs means that no jobs can be executed. To enable job execution by the new agent host, specify 1 or higher, or change the setting afterwards using the jpqagtalt command. For an example of specifying a maximum number of concurrently executable jobs using this command, see also jpqagtalt in 4. Commands Used for Special Operation in the manual JP1/Automatic Job Management System 3 Command Reference.

(c) Load distribution based on the maximum number of concurrently executable jobs

When multiple agent hosts are connected to the queue, you can distribute the processing load among the agents, based on the agent usage rates (ratio of active jobs to the maximum number of concurrent jobs). For details, see (6) Distributing jobs among execution hosts (agent hosts).

(d) Cautions on setting concurrent execution limits

For cautionary notes about concurrent execution limits, see Notes on setting a maximum number of concurrently executable jobs in 7.1.1(4)(a) Maximum number of concurrently executable jobs (QUEUE jobs and submit jobs) in the JP1/Automatic Job Management System 3 System Design (Configuration) Guide.

(a) Distributing jobs among multiple execution hosts (agent hosts) connected to the queue

When you register a QUEUE job or submit job for execution in JP1/AJS3, the job is placed in a queue (specified or default) managed by the job execution control. You can connect multiple agent hosts to the queue, specifying a different priority for each. Jobs are distributed to the agent hosts based on their specified priorities.

Specify the agent host priorities to suit your system environment. If you assign different priorities, each job will be sent for execution to the agent host hat has the highest priority. When that agent's usage rate^# reaches 100%, the next job is sent to the agent host that has the next highest priority. For example, if the agent host with the highest priority is already fully occupied or is in failed or stopped state and cannot run the job, the agent host with the next highest priority can run it instead. Setting different priorities is useful for this type of system operation.

If you set the same priority for all agent hosts, each job will be sent to the agent host that has the lowest agent usage rate. If no particular agent host takes precedence, jobs are distributed evenly among the agent hosts. Setting identical priorities is useful for this type of load-balancing system.

#

Agent usage rate is the ratio of active jobs to the maximum number of concurrent jobs at an agent host. It is given by the following equation, rounded to four decimal points:

Agent usage rate = number of active jobs / maximum number of concurrent jobs

Cautionary note

When jobs are to be distributed, there might be multiple execution agents that have the same priority and agent usage rate. In this case, to balance the load equally, the best way is to distribute the jobs by selecting the execution agent whose idle time is longest.

For details on defining priorities, see 7.1.7 Method of determining the agent host to which a job is distributed in the JP1/Automatic Job Management System 3 System Design (Configuration) Guide.

(b) Detecting failures and distributing jobs to execution hosts (agent hosts)

In the following two cases, there may be a delay of up to 15 minutes in distributing QUEUE jobs or submit jobs in JP1/AJS3 to execution hosts (agent hosts) that are operating normally. During this period, the jobs remain in Now queuing status:

• When a failure occurs simultaneously on three or more execution hosts

• When three or more execution hosts have stopped

In such cases, restart the affected execution hosts.