In-memory data processing

Applying in-memory data processing can accelerate batch-processing speed. In-memory data processing loads all data in RDAREAs into memory at one time, and updates only the data in memory while batch processing is being executed, without updating the data on the disk. When batch processing is completed, all of the updated data in memory is written onto the disk at once. During batch processing, no disk input/output operations occur. Disk input/output operations occur only during writing of the data in RDAREAs into memory, and during writing of the updated data in memory onto the disk.

The following figure provides an overview of in-memory data processing.

Figure 6-11 Overview of in-memory data processing

[Figure]

Explanation

The operation of writing all data in RDAREAs into memory at a time is called in-memory copying. In-memory copying is performed for each RDAREA. An in-memory copied RDAREA is called an in-memory RDAREA. Furthermore, a data buffer used during in-memory data processing is called an in-memory data buffer.

Hint: To execute in-memory data processing, use an in-memory data buffer instead of a global buffer. Because an in-memory data buffer is dynamically allocated in shared memory by HiRDB, there is no need for the HiRDB administrator to define one.

During batch processing, only the data in the in-memory data buffer is updated; the data on the disk is not updated. After batch processing is completed, you can execute a command (the pdhold command) to write all updated information in the in-memory data buffer onto the disk at once. No writing of data onto the disk occurs at synchronization points.

To apply in-memory data processing, you need HiRDB Accelerator.

For details about how to apply in-memory data processing, see the manual HiRDB Version 9 Batch Job Accelerator.

Organization of this subsection: (1) Advantages over global buffers; (2) Application criteria for in-memory data processing

(1) Advantages over global buffers

When global buffers are used, disk input/output operations occur regularly. By contrast, when in-memory data buffers are used, disk input/output operations occur only during in-memory copying and during writing of data onto the disk. No writing of data onto the disk occurs at synchronization points either. Therefore, by using in-memory data buffers, you can reduce the number of disk input/output operations compared to when global buffers are used.

Additionally, in-memory data buffers provide the following advantages over global buffers:

Little buffer management processing is needed.
No tuning is required.

For details, see the manual HiRDB Version 9 Batch Job Accelerator.

(2) Application criteria for in-memory data processing

Applying in-memory data processing can provide benefits in the following cases:

When batch processing with a long processing time is being executed
Applying in-memory data processing to batch processing that updates a large volume of data and has a long processing time can provide benefits. Applying the no-log mode can shorten the processing time further.
The longer batch processing takes, the greater the difference in the number of disk input/output operations compared to when in-memory data processing is not used; therefore, applying it can be expected to provide benefits.
When multiple batch processes are being executed in succession
Applying in-memory data processing during successive execution of multiple batch processes can be expected to provide benefits.
The greater the number of batch processes executed in succession, the greater the difference in the number of disk input/output operations compared to when in-memory data processing is not used. Therefore, applying it can be expected to provide benefits.
When performance does not improve because of locked global buffers
When performance does not improve because global buffers are locked, applying in-memory data processing can be expected to provide benefits.
When a global buffer is used (when in-memory data processing is not used), a process of checking whether data is cached in the global buffer (cache search processing) is performed whenever a referencing or updating process occurs. During this process, because all pages of the global buffer become locked, other referencing or updating processes must wait for lock-release. By contrast, when in-memory data processing is used, all data is cached in the in-memory data buffer, and therefore no cache search processing occurs. Because locking of all pages of the global buffer is eliminated, improvements in concurrent executability can be expected, along with comparable improvements in performance. For example, when locking is occurring frequently in batch processing that updates multiple RDAREAs to which the same global buffer is allocated, applying in-memory data processing can be expected to improve performance.
You can check the locking frequency in the statistical information related to global buffers of the statistics analysis utility.
When moving data from a mainframe system
When moving a large amount of data from a mainframe system to HiRDB (loading the data into a HiRDB database), applying in-memory data processing can be expected to provide benefits.

6.8.2 In-memory data processing

(1) Advantages over global buffers

(2) Application criteria for in-memory data processing