OpenTP1 provides the Multiserver facility to enable multiple instances of a service to be executed in parallel and in different processes, in response to multiple requests for the service. When a new service request comes to an already executing user server, a new user server process can be executed for the request.

Not all SPPs can use the Multiserver facility. SPPs that use schedule queues (i.e., servers that receive requests from queues) and MHPs can use the Multiserver facility, but servers that receive requests from a socket cannot use the Multiserver facility. In the user service definition (in the parallel_count operand) specify that a server that receives requests from a socket should use one process only.

A process of a UAP that uses the Multiserver facility can be reserved during OpenTP1 operations or can be reserved dynamically. Processes that are always reserved are called resident processes. Processes that are started when necessary are called non-resident processes.

An advantage of non-resident processes is that they enable efficient use of the memory area in the OpenTP1 system. An advantage of resident processes is that the processing for a user server is performed more quickly than for non-resident processes.

When using the Multiserver facility, in a user service definition you can specify the maximum number of processes to be used. If you specify more than one resident process, only that number of processes will be started and executed in parallel. If you specify more than one non-resident process, only that number of processes will be started dynamically.

If there is no spare system memory, a non-resident process is executed after the finish of any currently executing non-resident process.

Set the number of resident and non-resident processes in the parallel_count operand of the user service definition before starting the user server.

The Multiserver facility can increase or decrease the number of non-resident processes according to the number of service requests in the schedule queue.

The startup timing of non-resident processes depends upon the values specified in the balance_count operand in the user service definition. If there are remaining service requests that exceed (the values specified to the balance_count operand x processes being started), OpenTP1 starts the non-resident processes. If the number of service requests remaining in a schedule queue is less than (the values specified to the balance_count operand x processes being started), OpenTP1 will terminate the non-resident processes.

Figure 3-48 gives an overview of the Multiserver facility based on resident and non-resident processes.

Figure 3-48 Overview of Multiserver facility based on resident and non-resident processes

In the user service definition, you can assign a scheduling priority to each user server. Non-resident processes in a user server that has high scheduling priority are preferentially scheduled compared to non-resident processes in a user server that has low scheduling priority.

Figure 3-49 provides an overview of scheduling priority.

Figure 3-49 Overview of scheduling priority

OpenTP1 can process a heavily-used service on multiple nodes. When many processes are required for processing a service requested from an SPP, OpenTP1 can distribute the processing to SPPs with the same service group name on other nodes. To use this facility for balancing loads among nodes, the following conditions must be satisfied:

• A user server providing the same service to multiple nodes in the same LAN must be operating.
• In the all_node operand in the system common definition, each OpenTP1 node must define the other nodes. This definition must allow the OpenTP1 nodes to exchange the information (name information) of the user servers operating on the nodes.

Service requests are passed to the user server on a randomly selected node. OpenTP1 references the server information of the node and avoids selecting it if it is hard to schedule. Therefore, even when a schedulable user server exists on the local node, the request is not always passed to that user server. To select the user server on the local node first, specify scd_this_node_first=Y in the schedule service definition. With Y specified, OpenTP1 selects a user server on another node only when it is hard for the user server on the local node to accept the request.

The internode load-balancing facility requires that the operation conditions of the user servers on nodes are the same. If the following conditions differ greatly from node to node that is selected, such an environment is unsuitable for the internode load-balancing facility. In this case, do not place the service groups with the same name in multiple nodes. The conditions that should not differ greatly are:

• Operation cost such as the connection fees of the public line
• Line speed
• Line quality
• Performance of individual nodes

Each node reports server information to be referenced when allocating a request to the user server. For a system in which SPPs of the same service group are not distributed among multiple nodes, the server information need not be reported. Especially when a public line is used, charges for unnecessary connections will occur. In such a system, specify scd_announce_server_status=N in the schedule service definitions of all the nodes to suppress reporting of server information.

OpenTP1 can distribute loads for SPPs (servers) that receive requests either from a queue or from a socket. When such an SPP is busy, OpenTP1 passes the service requests for the SPP to another user server in another node. The selection of the other node is almost random, except that for servers that receive requests from a queue, OpenTP1 checks the status of the node to be scheduled and controls selection so that it is difficult to select a node that has a low scheduling efficiency. For a server that receives requests from a socket, however, OpenTP1 neither checks the node status nor controls selection.

Figure 3-50 gives an overview of the Internode Load-Balancing facility.

Figure 3-50 Overview of Internode Load-Balancing facility

Service requests are scheduled according to the load level of each node. The following load levels are used:

LEVEL0

A light load. Service requests are usually scheduled to nodes with LEVEL0 or LEVEL1.

LEVEL1

A medium load. At rescheduling due to an error, service requests may not be scheduled to nodes with LEVEL1. However, if there are only nodes with LEVEL1 or LEVEL2 at rescheduling, service requests are scheduled to those nodes.

LEVEL2

A heavy load. Service requests are usually not scheduled to nodes with LEVEL2. However, if there are only LEVEL2 nodes, service requests are scheduled to LEVEL2 nodes.

The load level of each node is checked at each load check interval. At that time, the current load level is determined according to the previous load level, the number of queued service requests, the number of remaining service requests, and the server processing rate.

Table 3-12 shows the conditions that determine the load level.

Table 3-12 Conditions that determine the load level

Previous load level	Number of queued service requests: Q	Number of remaining service requests: q	Server processing rate: X	Current load level
LEVEL0	Q 1	--	X < 50	LEVEL1
LEVEL1	Q 1	--	75 X	LEVEL0
			50 X < 75	LEVEL1
			X < 50	LEVEL2
LEVEL2	--	q = 0	--	LEVEL0
		q 1		LEVEL2

Legend

--: Ignored.

Number of queued service requests

Number of service requests that are queued into the schedule queue during a load check interval

Number of remaining service requests

Number of service requests that are remaining in the schedule queue when the load is checked

Server processing rate

Processing rate calculated from the following formula:

Server processing rate = (Number of processed services / (Number of queued service requests + Number of remaining service requests)) 100

The number of processed services is the number of service requests that are processed during a load check interval.

If the load level is changed, the server information is reported to the name service of each node and the server information is updated. By using the loadlevel_message operand in the user service definition, a message reporting the change of the load level can be output.

This section describes the definitions and processing on the TP1/Server Base and TP1/Client sides and RPC processing when using the internode load-balancing facility.

(a) When the server side determines how to perform load-balancing

The schedule service of TP1/Server Base distributes loads to nodes that can efficiently process the loads according to the schedule status of each node.

Definition on the server (TP1/Server Base) side

In the definition on the TP1/Server Base side, either:

• include the following settings for the operands in the schedule service definition
  set scd_this_node_first = N (default)
  set scd_announce_server_status = Y (default)
  

or

• omit the schedule service definition

Definition on the client (TP1/Client) side

Define dcscddirect=Y (for TP1/Client/P) in the client environment definition so that TP1/Client makes a request to the schedule service of TP1/Server Base for load balancing. In the definition on the TP1/Client side, specify where to issue a request for load-balancing for the OpenTP1 node's schedule service TP1/Client.

In this case, OpenTP1 nodes are selected to determine scheduling in the order specified in the dchost operand. To select OpenTP1 nodes randomly and not in the order specified in the dchost operand, you must add dchostselect=Y (for TP1/Client/P) to the definition.

(b) When the client side determines how to perform load-balancing according to the load information from the server

When the client is TP1/Client

Definition on the server (TP1/Server Base) side

In the definition on the TP1/Server Base side, either:

• include the following settings for the operands in the schedule service definition
  set scd_this_node_first = N (default)
  set scd_announce_server_status = Y (default)
  

or

• omit the schedule service definition

Definition on the client (TP1/Client) side

Define dccltloadbalance=Y (for TP1/Client/P) in the client environment definition. By this definition, TP1/Client first determines the OpenTP1 node to which a service request should be issued based on the load level information of each server acquired from TP1/Server Base. Then it performs an RPC. In this case, the information including the load level of each server is retained temporarily. This load level is retained in the area where size is specified in the dccache operand (which previously was specified in the dccltcachetim operand). That is, the shorter the value in the dccltcachetime operand becomes, the newer is the load level of each server that is used to determine the request destination to which an RPC should be issued. At the same time, it must be taken into account that accesses to the name service of TP1/Server Base occur more frequently to acquire the load level information.

When the server and client are TP1/Server Base

In the definitions on both server and client sides, either:

• include the following settings for the operands in the schedule service definition
  set scd_this_node_first = N (default)
  set scd_announce_server_status = Y (default)
  

or

• omit the schedule service definition
  In this configuration, TP1/Server Base already has the load level set for the server depending on when it is going to issue a request. Therefore it performs an RPC for a node with a lower load level. At the time when this request is received, the schedule service does not transfer the request according to the verification of load level but processes it within the local node if possible. The schedule service transfers the request to another node only if the server is blocked or only if the load level of the server in the local node is LEVEL2 and some other node has a server with a lower load level.
  

(c) Operations when internode load-balancing facility is used with other facilities

Table 3-13 shows the operations when the internode load-balancing facility is used with other facilities.

Table 3-13 Operations of the internode load-balancing facility used with other facilities

When using	Operation
Permanent connection by TP1/Client	The CUP execution process of TP1/Server Base performs an RPC in the node that established the permanent connection. This is the same operation as in the case when the server and client are TP1/Server Base.
Transaction control API by TP1/Client	The transaction delegated execution process of TP1/Server Base performs an RPC. This is the same operation as in the case when the server and client are TP1/Server Base.
Remote API facility	The RAP-processing server of TP1/Server Base performs an RPC. This is the same operation as in the case when the server and client are TP1/Server Base.

The user can specify the following items:

• Rate of schedules made to the LEVEL0 nodes
  By specifying the schedule_rate operand in the schedule service definition, you can specify the rate of schedules (%) made to the LEVEL0 nodes.
  The schedule_rate operand is valid only when you specify Y for the DCSCDDIRECT operand in the client environment definition of TP1/Client to make service requests.
  Note that if Y is specified for scd_this_node_first in the schedule service definition, service requests are scheduled to the local node with priority.
  The following example shows how service requests are scheduled when you specify 80 in the schedule_rate operand. The number of service requests is 10.
  
  1. The scheduler obtains the load information of all the nodes from the name service and then counts the number of LEVEL0 nodes and the number of LEVEL1 nodes.
  2. The scheduler uses the value specified in the schedule_rate operand to assign a weight to each number of nodes calculated in step 1. Then the scheduler determines the rate of schedules to be made to the LEVEL0 nodes.
     LEVEL0:LEVEL1 = 80 3:20 2 85:15
     
  3. The scheduler selects a LEVEL0 node using the rate calculated in step 2 and schedules service requests.
  Figure 3-51 shows how service requests are scheduled to LEVEL0 nodes.
  

  Figure 3-51 Scheduling service requests to LEVEL0 nodes

  

• Load check interval
  By specifying the loadcheck_interval operand in the user service definition and the user service default definition, you can specify the load check interval for each service group. If the load level is changed when the load is checked, the server information is reported to the name service of each node. Therefore, the server information may be sent out to the network at each load check interval. To prevent this, do not specify a short interval unless it is necessary.
  If you do not specify the loadcheck_interval operand, the load check interval will be 30 seconds. Whether to check the load is determined every 10 seconds. In other words, a load check is executed on every third check.
  When you specify the loadcheck_interval operand, the value specified in this operand is the load check interval. This value determines whether the check of the load are executed at the interval that is calculated from the greatest common measure of 10 and the value specified in the loadcheck_interval operand for each user server. For example, when you specify 3 for the loadcheck_interval operand of SPP1 and 5 for the loadcheck_interval operand of SPP2, the interval of the checks is 1 (second) since it is the greatest common measure of 10, 3, and 5. The load check of SPP1 is executed on every third check. The load check of SPP2 is executed on every fifth check.
  Therefore, to keep the influence to the system to the minimum, specify a multiple of 5 as the value to be specified for the loadcheck_interval operand.
  When you specify 0 for the loadcheck_interval operand, you can suppress the load check in each service group.
  
• Thresholds of load levels
  By specifying the levelup_queue_count operand and the leveldown_queue_count operand in the user service definition and the user service default definition, you can use the numbers of remaining service requests to specify the thresholds that determine the load levels for each service group.
  Specify the thresholds as follows:
  

  set levelup_queue_count = U1,U2
  set leveldown_queue_count = D0,D1

  U1: Number of remaining service requests, which determines that the server's load level is upgraded to LEVEL1
  U2: Number of remaining service requests, which determines that the server's load level is upgraded to LEVEL2
  D0: Number of remaining service requests, which determines that the server's load level is downgraded to LEVEL0
  D1: Number of remaining service requests, which determines that the server's load level is downgraded to LEVEL1
  The current load level will be determined by the previous load level and the number of remaining service requests.
  Table 3-14 shows the correspondence between the number of remaining service requests and the load level.
  

  Table 3-14 Numbersof remaining service requests and load levels

  Previous load level	Number of remaining service requests: q	Current load level
  LEVEL0	q < U1	LEVEL0
  	U1 q < U2	LEVEL1
  	U2 q	LEVEL2
  LEVEL1	q D0	LEVEL0
  	D0 < q < U2	LEVEL1
  	U2 q	LEVEL2
  LEVEL2	q D0	LEVEL0
  	D0 < q D1	LEVEL1
  	D1 < q	LEVEL2

• Number of retries to be made if a communication error occurs
  If a communication error occurs while service requests are scheduled, the service requests are not usually rescheduled and an error is returned.
  However, by specifying the scd_retry_of_comm_error operand in the schedule service definition, you can specify the number of schedule retries to be made to nodes other than the failed node.
  Note that if the value specified in the scd_retry_of_comm_error operand exceeds the number of nodes with active service groups which are the target of the service requests, the number of nodes with active service groups which are the target of the service requests is the upper limit of retries.
  When you specify 0, no retry is made.
  To use this function, TP1/Extension 1 must be installed beforehand. If TP1/Extension 1 is not installed, the operation is not guaranteed.
  

In addition to the regular scheduler daemon (called the master scheduler daemon hereafter), you can start multiple daemon processes specialized to receive service requests (called the multi-scheduler daemon hereafter) to receive several service request messages concurrently. This way, you can avoid the scheduling delay due to reception contention. This solution is called the multi-scheduler facility.

To use the multi-scheduler facility, you must specify:

• scdmulti in the schedule service definition and scdmulti in the user service definition on the RPC receiving side.
• multi_schedule in the user service definition on the RPC sending side.

It is also possible to group multi-scheduler daemons by servers that receive requests from a queue. This grouping prevents servers from contending for receiving of service request messages. When multi-scheduler daemons are grouped, you must specify scdmulti in the schedule service definition on the server side.

This facility requires TP1/Extension 1 installed. If TP1/Extension 1 is not installed, the operation is not guaranteed.

Figure 3-52 gives an overview of the multi-scheduler facility.

Figure 3-52 Overview of multi-scheduler facility