OpenTP1 Version 7 Description
This section describes DAM (direct access method) files, which are one of three types of user files used in OpenTP1 application processing. The three types of user files are DAM, TAM, and ISAM. DAM files are created as OpenTP1 files.
The OpenTP1 DAM file service enables OpenTP1 files in an OpenTP1 file system to be accessed as direct-organization files, enables OpenTP1 to manage the access as part of a transaction, and enables OpenTP1 to manage the file status. To use the DAM files in OpenTP1, TP1/FS/Direct Access is needed.
You use the DAM service definition when creating a DAM file. In the definition you must specify the name of a physical file and the name of a logical file for the DAM file. The name of the physical file, which is the complete pathname for an OpenTP1 file entity, is made up of the directory names and the filename. Each name of a logical file matches the name of a corresponding physical file; the name of the logical file is for the convenience of the user and is assigned by the user. Using the logical file name allows the user to use the DAM file without knowledge of the structure of the physical file.
Figure 4-14 shows the configuration of the DAM file.
Figure 4-14 DAM file configuration
Table 4-9 lists the special features of DAM files.
Table 4-9 Special features of DAM files
Item | Special feature |
---|---|
File structure | Direct organization file (an OpenTP1 file). |
Record format | Fixed length, non-blocked format. |
File access | File can be referenced and updated by the direct access method using the relative block number. Consecutive blocks can be input or output all together. |
Relation to transaction | The contents of a block that was updated during transaction processing are updated in the file when a commit operation is performed on a transaction or when the maximum number of blocks specified in the DAM service definition is reached. If the transaction is rolled back, the status before the start of transaction is restored. |
File backup and restore | The DAM file management command dambkup can backup a DAM file to another file. Also, the most recent contents of a DAM file can be recovered by using the backup file and the recovery journals after the backup was obtained. Recovery of a DAM file uses the DAM FRC facility. |
File addition | The DAM file management command damadd can add DAM files into an online system during OpenTP1 operation. |
File separation | The DAM file management command damrm can separate the DAM file used in online processing from the operating OpenTP1. |
File deletion | The DAM file management command damdel can delete a DAM file separated from OpenTP1. |
OpenTP1 files allocated to the OpenTP1 file system are called physical files. Offline DAM files are accessed via physical files. Online DAM files are accessed via a logical file that corresponds to a physical file. In the DAM service definition, you specify which physical file corresponds to which logical file in a DAM file.
DAM files should be created after creating an OpenTP1 file system. There are two ways to create DAM files:
With this method, a UAP issues a DAM access function dc_dam_create() or dc_dam_put() (for file allocation, and outputting initial data) in an offline environment to create the DAM file.
OpenTP1 UAPs can use functions to access DAM files. For details on the DAM file service that can be used by UAPs see the OpenTP1 Programming Guide.
A lock on a DAM file takes effect within a transaction branch or within a global transaction. In the DAM service definition, you can change whether the lock applies to a transaction branch or to a global transaction.
When a lock applies to a global transaction, an error is not returned even if multiple transaction branches in the same global transaction access the same block or the same file.
For parallel processing of access to DAM files in each transaction branch, you should specify locks that apply to transaction branches because specifying locks that apply to a global transaction can reduce transaction efficiency. When a lock applies to a global transaction, even if DAM files are accessed for each transaction branch, parallel access is not possible and access is by sequential access.
For details on locking, see the OpenTP1 Programming Guide.
A lock on a DAM file takes effect within a transaction branch or within a global transaction. In the DAM service definition, you can change whether the lock applies to a transaction branch or to a global transaction.
When a lock applies to a global transaction, an error is not returned even if multiple transaction branches in the same global transaction access the same block or the same file.
For parallel processing of access to DAM files in each transaction branch, you should specify locks that apply to transaction branches because specifying locks that apply to a global transaction can reduce transaction efficiency. When a lock applies to a global transaction, even if DAM files are accessed for each transaction branch, parallel access is not possible and access is by sequential access.
For details on locking, see the OpenTP1 Programming Guide.
When an attempt is made to update, or to reference for an update, a block of an opened DAM file and that block is already locked, you can use a function parameter (e.g., in dc_dam_open(), dc_dam_read(), and dc_dam_write()) to specify whether to wait for the release of the lock. In the lock service definition you can specify how long to wait for the release of the lock. If the lock is not released within this specified period, an error is returned.
In an OpenTP1 deferred update, a block of a DAM file that is updated by a transaction is not updated when the synchronization point is reached; the DAM file is updated asynchronously (though note the exception in the next paragraph). A DAM file with a deferred update specification is actually updated from a process used exclusively for output and which is started after some specified period. Specifying a deferred update for a DAM file can reduce the number of I/O operations, which can improve throughput of transactions per unit of time.
Note, however, that when one transaction outputs updates for both a DAM file that has a deferred update specification and a DAM file that receives an ordinary update, both DAM files are updated at a synchronization point.
In the DAM service definition for each DAM file, you can specify whether or not a DAM file is to have a deferred update.
Before the execution of the process used exclusively for output, in the DAM service definition you must specify the buffer area for storing the block and the period for executing the update from the process used exclusively for output. If this execution period is too long or the value for the buffer area is too small, a buffer area overflow might occur or the system might be forced to wait for free area.
When using the DAM service definition to specify a deferred update, you should take care when calculating the values for the buffer area and the time period.
When OpenTP1 abnormally terminates because of a system failure, a DAM file that has a deferred update specified might not contain the results of a transaction that completed before the failure. To guarantee the results of the transaction update of the DAM file, you must always normally terminate OpenTP1 after OpenTP1 is restarted in a complete recovery.
Even for transactions that were completed before the failure, if the processing of the process used exclusively for output was not completed before the failure, a rollback might occur after a complete recovery. If you do not want to rollback a transaction that was completed before the failure, do not specify a deferred update for the DAM file to be accessed.
When a disk error occurs in a DAM file that has a deferred update specification, you should always execute the DAM FRC (file recovery).
DAM files can be backed up during OpenTP1 online operations. To allow online backups of DAM files, execute the dambkup command with the -o option. If you do not specify the -o option, backup is done offline.
To perform an offline backup of a DAM file:
Recovery of a DAM file by a file that has been backed up online requires the use of fewer unloaded-journals files. This results in shorter recovery processing times for DAM files when compared to recovering a DAM file that uses backup files for which the -o options was not specified.
You can extract user data from a DAM file without the management information. You can use the damload command to allocate the extracted user data as the initial data for a DAM file. However, you cannot use the damrstr command to restore the extracted user data. To extract user data, execute the dambkup command with the -d option.
When you restore a file that was backed up using the dambkup command, you can extend the block length. This feature is called the block length extension facility. By using this facility, you can move data between DAM files with different block lengths.
There are two modes for the block length extension facility: a mode that maintains the block configuration of the backup DAM file and a mode that does not maintain the block configuration of the backup DAM file. See below for details.
Figure 4-15 Extending the block length maintaining the original block configuration
Figure 4-16 Extending the block length without maintaining the original block configuration
Note the following when using the block length extension facility.
You can use standard output to the backup destination of a DAM file and standard input to the input file to be restored. Using standard input/output can redirect a command. If you use standard input/output, specify the -s option in the dambkup and damrstr commands, then execute them.
DAM files can be created without guaranteeing their consistency and error recovery in a transaction. These DAM files are called unrecoverable DAM files. Processing not included in transactions can update and output blocks into the unrecoverable DAM files.
Specify the -n option in damfile, which is a definition command of the DAM service definition, for an unrecoverable DAM file. If an error occurs while accessing an unrecoverable DAM file, the file data error cannot be recovered.
The DAM service stores the block data in the DAM file it reads in the DAM service-exclusive shared memory. When there are multiple reference requests for the same block, the DAM service returns the block data in the DAM service-exclusive shared memory to the UAP to reduce the number of file I/O operations.
In the DAM service-exclusive shared memory, several pieces of block data are chained and managed for each DAM file. An area in the DAM service-exclusive shared memory storing block data is called a cache block.
When a UAP accesses a DAM file, the DAM service secures cache blocks according to the following procedure. This processing is called cache block securing.
The procedure for cache block securing is as follows:
To acquire the number of cache blocks connected to a cache block chain and the total use percentage of the DAM service-exclusive shared memory, use the damchinf command.
Figure 4-17 gives an overview of a cache block chain.
Numbers in the figure correspond to the above step numbers.
Figure 4-17 Overview of a cache block chain
If a specific DAM file is heavily accessed, the number of cache blocks managed by the DAM file increases. As a result, searching for and releasing the target cache blocks takes time, resulting in decreased transaction performance.
By specifying the damchlmt definition command in the DAM service definition, you can set a limit for the maximum number of cache blocks that can be managed by a single DAM file. This limit is called a threshold. Cache blocks are secured up to the threshold. When the threshold is reached, unused cache blocks for the corresponding DAM file are reused and new cache blocks are not secured. If you do not specify a threshold, cache blocks are secured until the shared memory resource is used up.
During online operation you can use damchdef command to dynamically change the value to be specified for the damchlmt definition command in the DAM service definition.
You can calculate the size of the area used for cache blocks as follows:
Cache block area size = M - (M/576) 34 (bytes)
You can calculate the size of each cache block as follows:
Cache block size = (Ab + 8) + 64 (bytes)
M: Size of the DAM service-exclusive shared memory
Ab: Length of a block in the DAM file to be accessed
Specifying a threshold is effective when a specific DAM file is heavily accessed. The following conditions must be satisfied to specify a threshold:
set dam_update_block = 10 set dam_tran_process_count = 5
As described above, when you specify 0 as the threshold for the number of cache blocks, the number of cache blocks connected to the cache block chain depends on the type of access made by the UAP and the values set in the DAM service definition. The same phenomenon occurs when you specify a small value as the threshold for the number of cache blocks. A small value refers to a number smaller than the number of blocks that are accessed in a transaction executed by a UAP.
In Example 3, part of the DAM service-exclusive shared memory is not used as in Example 2.
Imagine that the following information is output after you execute the damchinf command:
CleanUP Count:1 Next CleanUP FILE-No:1 Using Rate:80% FileNo FileName BlkLen BlkNum CchBlkNum PreservNum LimitNum ReUse 1 damfile1 504 10000 7900 0 -1 Exist 0 damfile0 504 10000 100 0 -1 Exist 2 damfile2 504 10000 0 0 -1 None
This section explains the suitable threshold for the number of cache blocks for each type of access made by a UAP.
Note the following:
In the dam_cache_reuse_from operand in the DAM service definition, you can specify the first cache block you want to reuse in the cache block chain when a transaction that accesses a DAM file requires a new cache block.
When you specify last, cache blocks are reused from the oldest cache block connected to the cache block chain.
When you specify first, cache blocks are reused from the latest cache block connected to the cache block chain.
The default is last.
In the damcache definition command in the DAM service definition, you can specify the boundary for reusing cache blocks. If you have specified a boundary, when securing new cache blocks, the reuse of cache blocks for another DAM file takes priority over reuse of cache blocks for the accessed DAM file. This applies if the number of cache blocks has not reached the boundary for reusing cache blocks.
If the number of cache blocks exceeds the boundary for reusing cache blocks, the cache blocks for the accessed DAM file are reused. Therefore, when you specify 0 for the boundary for reusing cache blocks, cache blocks for the accessed DAM file are reused as in the usual processing.
In the dam_default_cache_num operand in the DAM service definition, you can specify the boundary for reusing cache blocks in a logical file without the damcache definition command specified.
Specifying a boundary is effective when you want to heavily access many blocks in a DAM file and then frequently access another DAM file, which means that a small number of DAM files occupy cache blocks and only a few cache blocks are allocated to a different DAM file that is accessed frequently.
When you access multiple DAM files at random, which means that cache blocks are allocated to each DAM file uniformly, specifying a small value for the boundary for reusing cache blocks releases cache blocks frequently, increasing memory securing and data read processing. This may degrade the performance.
For the purpose of explanation, two DAM files are used here and the block lengths are all the same. The maximum number of cache blocks that can be secured in the entire system is 100. A UAP references 98 blocks of DAM file A and then starts to access DAM file B.
When the UAP accesses DAM file B, the number of blocks it references increases by one in each access. The UAP will reference up to 50 blocks.
1st access: Blocks 1 and 2 are referenced.
2nd access: Blocks 1, 2, and 3 are referenced.
3rd access: Blocks 1, 2, 3, and 4 are referenced.
:
49th access: Blocks 1, 2, 3, ... and 50 are referenced.
Under these conditions, the difference between specifying and not specifying the damcache definition command is described below.
When you do not specify the damcache definition command:
When you specify the damcache definition command:
You specify 50 as the boundary for reusing cache blocks for DAM file A and specify 90 as the boundary for reusing cache blocks for DAM file B.
The conditions, such as the block lengths in DAM files, are the same as the previous example. DAM files are accessed as follows:
1st access: Blocks 1 to 50 of DAM file A are referenced.
2nd access: Blocks 1 to 50 of DAM file B are referenced.
3rd access: Blocks 51 to 100 of DAM file A are referenced.
4th access: Blocks 1 to 50 of DAM file B are referenced.
:
The UAP references DAM files A and B alternately. The UAP references the next 50 blocks of DAM file A for each access.
Since the maximum number of cache blocks is 100, 50 cache blocks of DAM file A and 50 blocks of DAM file B are connected by the second access. For the third and later accesses, the processing differs depending on whether you specify the damcache definition command.
Note the following:
As described in 4.4.1(11) Using cache blocks, the DAM service stores the block data in the DAM files that are once read in the DAM service-exclusive shared memory to reduce the number of file I/O operations.
However, for the unrecoverable DAM files only, you can directly perform file I/O operations and return the block data in the disk to UAPs. This is called cacheless access. To enable cacheless access for the unrecoverable DAM files, specify the -f option of the damfile definition command in the DAM service definition.
When you specify cacheless access, file I/O operations are performed for each reference and update, and performance may degrade.
When you use DAM files with cacheless access specified, you can reduce the amount of memory required by obtaining the appropriate size of DAM service-exclusive shared memory and then specifying that value in the dam_cache_size_fix operand in the DAM service definition. For details, see the description of the dam_cache_size_fix operand in the DAM service definition in the manual OpenTP1 System Definition.
All Rights Reserved. Copyright (C) 2006, 2010, Hitachi, Ltd.