OpenTP1 Version 7 Description

[Contents][Glossary][Index][Back][Next]

4.4.1 User files: DAM files (TP1/FS/Direct Access)

This section describes DAM (direct access method) files, which are one of three types of user files used in OpenTP1 application processing. The three types of user files are DAM, TAM, and ISAM. DAM files are created as OpenTP1 files.

The OpenTP1 DAM file service enables OpenTP1 files in an OpenTP1 file system to be accessed as direct-organization files, enables OpenTP1 to manage the access as part of a transaction, and enables OpenTP1 to manage the file status. To use the DAM files in OpenTP1, TP1/FS/Direct Access is needed.

You use the DAM service definition when creating a DAM file. In the definition you must specify the name of a physical file and the name of a logical file for the DAM file. The name of the physical file, which is the complete pathname for an OpenTP1 file entity, is made up of the directory names and the filename. Each name of a logical file matches the name of a corresponding physical file; the name of the logical file is for the convenience of the user and is assigned by the user. Using the logical file name allows the user to use the DAM file without knowledge of the structure of the physical file.

Figure 4-14 shows the configuration of the DAM file.

Figure 4-14 DAM file configuration

[Figure]

Organization of this subsection
(1) Features of a DAM file
(2) Creating a DAM file
(3) I/O functions for a DAM file
(4) Locks on a DAM file
(5) Deferred update of a DAM file
(6) Online backup of a DAM file
(7) Extraction of user data
(8) Block length extension facility
(9) Backing up and restoring DAM files using standard input/output
(10) Access to unrecoverable DAM files
(11) Using cache blocks
(12) Specifying the threshold for the number of cache blocks
(13) Specifying the search sequence when reusing cache blocks
(14) Specifying the boundary for reusing cache blocks
(15) Cacheless access for unrecoverable DAM files

(1) Features of a DAM file

Table 4-9 lists the special features of DAM files.

Table 4-9 Special features of DAM files

Item Special feature
File structure Direct organization file (an OpenTP1 file).
Record format Fixed length, non-blocked format.
File access File can be referenced and updated by the direct access method using the relative block number. Consecutive blocks can be input or output all together.
Relation to transaction The contents of a block that was updated during transaction processing are updated in the file when a commit operation is performed on a transaction or when the maximum number of blocks specified in the DAM service definition is reached. If the transaction is rolled back, the status before the start of transaction is restored.
File backup and restore The DAM file management command dambkup can backup a DAM file to another file. Also, the most recent contents of a DAM file can be recovered by using the backup file and the recovery journals after the backup was obtained. Recovery of a DAM file uses the DAM FRC facility.
File addition The DAM file management command damadd can add DAM files into an online system during OpenTP1 operation.
File separation The DAM file management command damrm can separate the DAM file used in online processing from the operating OpenTP1.
File deletion The DAM file management command damdel can delete a DAM file separated from OpenTP1.

(2) Creating a DAM file

OpenTP1 files allocated to the OpenTP1 file system are called physical files. Offline DAM files are accessed via physical files. Online DAM files are accessed via a logical file that corresponds to a physical file. In the DAM service definition, you specify which physical file corresponds to which logical file in a DAM file.

DAM files should be created after creating an OpenTP1 file system. There are two ways to create DAM files:

With this method, a UAP issues a DAM access function dc_dam_create() or dc_dam_put() (for file allocation, and outputting initial data) in an offline environment to create the DAM file.

(3) I/O functions for a DAM file

OpenTP1 UAPs can use functions to access DAM files. For details on the DAM file service that can be used by UAPs see the OpenTP1 Programming Guide.

(4) Locks on a DAM file

(a) Lock granularity

A lock on a DAM file takes effect within a transaction branch or within a global transaction. In the DAM service definition, you can change whether the lock applies to a transaction branch or to a global transaction.

When a lock applies to a global transaction, an error is not returned even if multiple transaction branches in the same global transaction access the same block or the same file.

For parallel processing of access to DAM files in each transaction branch, you should specify locks that apply to transaction branches because specifying locks that apply to a global transaction can reduce transaction efficiency. When a lock applies to a global transaction, even if DAM files are accessed for each transaction branch, parallel access is not possible and access is by sequential access.

For details on locking, see the OpenTP1 Programming Guide.

(b) Transactions and lock granularity

A lock on a DAM file takes effect within a transaction branch or within a global transaction. In the DAM service definition, you can change whether the lock applies to a transaction branch or to a global transaction.

When a lock applies to a global transaction, an error is not returned even if multiple transaction branches in the same global transaction access the same block or the same file.

For parallel processing of access to DAM files in each transaction branch, you should specify locks that apply to transaction branches because specifying locks that apply to a global transaction can reduce transaction efficiency. When a lock applies to a global transaction, even if DAM files are accessed for each transaction branch, parallel access is not possible and access is by sequential access.

For details on locking, see the OpenTP1 Programming Guide.

(c) Setting lock release for resources

When an attempt is made to update, or to reference for an update, a block of an opened DAM file and that block is already locked, you can use a function parameter (e.g., in dc_dam_open(), dc_dam_read(), and dc_dam_write()) to specify whether to wait for the release of the lock. In the lock service definition you can specify how long to wait for the release of the lock. If the lock is not released within this specified period, an error is returned.

(5) Deferred update of a DAM file

In an OpenTP1 deferred update, a block of a DAM file that is updated by a transaction is not updated when the synchronization point is reached; the DAM file is updated asynchronously (though note the exception in the next paragraph). A DAM file with a deferred update specification is actually updated from a process used exclusively for output and which is started after some specified period. Specifying a deferred update for a DAM file can reduce the number of I/O operations, which can improve throughput of transactions per unit of time.

Note, however, that when one transaction outputs updates for both a DAM file that has a deferred update specification and a DAM file that receives an ordinary update, both DAM files are updated at a synchronization point.

(a) How to specify deferred update for a DAM file

In the DAM service definition for each DAM file, you can specify whether or not a DAM file is to have a deferred update.

Before the execution of the process used exclusively for output, in the DAM service definition you must specify the buffer area for storing the block and the period for executing the update from the process used exclusively for output. If this execution period is too long or the value for the buffer area is too small, a buffer area overflow might occur or the system might be forced to wait for free area.

When using the DAM service definition to specify a deferred update, you should take care when calculating the values for the buffer area and the time period.

(b) Caution required when using DAM files that have deferred update specified

When OpenTP1 abnormally terminates because of a system failure, a DAM file that has a deferred update specified might not contain the results of a transaction that completed before the failure. To guarantee the results of the transaction update of the DAM file, you must always normally terminate OpenTP1 after OpenTP1 is restarted in a complete recovery.

Even for transactions that were completed before the failure, if the processing of the process used exclusively for output was not completed before the failure, a rollback might occur after a complete recovery. If you do not want to rollback a transaction that was completed before the failure, do not specify a deferred update for the DAM file to be accessed.

When a disk error occurs in a DAM file that has a deferred update specification, you should always execute the DAM FRC (file recovery).

(6) Online backup of a DAM file

DAM files can be backed up during OpenTP1 online operations. To allow online backups of DAM files, execute the dambkup command with the -o option. If you do not specify the -o option, backup is done offline.

To perform an offline backup of a DAM file:

  1. Execute the damhold command.
    The logical file is logically shut down.
  2. Execute the damrm command.
    The logically shutdown logical file is separated from online processing.
  3. Execute the dambkup command without the -o option.
    The DAM file is backed up.

Recovery of a DAM file by a file that has been backed up online requires the use of fewer unloaded-journals files. This results in shorter recovery processing times for DAM files when compared to recovering a DAM file that uses backup files for which the -o options was not specified.

(7) Extraction of user data

You can extract user data from a DAM file without the management information. You can use the damload command to allocate the extracted user data as the initial data for a DAM file. However, you cannot use the damrstr command to restore the extracted user data. To extract user data, execute the dambkup command with the -d option.

(8) Block length extension facility

When you restore a file that was backed up using the dambkup command, you can extend the block length. This feature is called the block length extension facility. By using this facility, you can move data between DAM files with different block lengths.

There are two modes for the block length extension facility: a mode that maintains the block configuration of the backup DAM file and a mode that does not maintain the block configuration of the backup DAM file. See below for details.

Note the following when using the block length extension facility.

(9) Backing up and restoring DAM files using standard input/output

You can use standard output to the backup destination of a DAM file and standard input to the input file to be restored. Using standard input/output can redirect a command. If you use standard input/output, specify the -s option in the dambkup and damrstr commands, then execute them.

(10) Access to unrecoverable DAM files

DAM files can be created without guaranteeing their consistency and error recovery in a transaction. These DAM files are called unrecoverable DAM files. Processing not included in transactions can update and output blocks into the unrecoverable DAM files.

Specify the -n option in damfile, which is a definition command of the DAM service definition, for an unrecoverable DAM file. If an error occurs while accessing an unrecoverable DAM file, the file data error cannot be recovered.

(11) Using cache blocks

The DAM service stores the block data in the DAM file it reads in the DAM service-exclusive shared memory. When there are multiple reference requests for the same block, the DAM service returns the block data in the DAM service-exclusive shared memory to the UAP to reduce the number of file I/O operations.

In the DAM service-exclusive shared memory, several pieces of block data are chained and managed for each DAM file. An area in the DAM service-exclusive shared memory storing block data is called a cache block.

When a UAP accesses a DAM file, the DAM service secures cache blocks according to the following procedure. This processing is called cache block securing.

The procedure for cache block securing is as follows:

  1. The DAM service checks if the cache block corresponding to the required block is connected to the cache block chain for the accessed DAM file. If it is connected, the DAM service returns the data in that cache block to the UAP.
  2. If the target cache block is not connected to the cache block chain, the DAM service secures a cache block from the cache block area.
  3. If there is no free space in the cache block area, the DAM service reuses a cache block in the cache block chain for the accessed DAM file. In this case, the cache block should not be currently referenced by the transaction being executed. Cache blocks are reused from the oldest cache block in the cache block chain (the last cache block in the cache block chain). The cache blocks to be reused are determined by the value specified in the dam_cache_reuse_from operand in the DAM service definition. By default, cache blocks are used from the last cache block.
  4. If the DAM service cannot secure a cache block in steps 2 and 3, the DAM service releases all the cache blocks, which are not used by the transaction being executed, in the cache block chain for another DAM file. Then the DAM service secures a cache block from the cache block area. This method of releasing cache blocks is called clean-up processing.
  5. The DAM service copies the block data of the DAM file to the secured cache block and connects the cache block to the beginning of the cache block chain.

To acquire the number of cache blocks connected to a cache block chain and the total use percentage of the DAM service-exclusive shared memory, use the damchinf command.

Figure 4-17 gives an overview of a cache block chain.

Numbers in the figure correspond to the above step numbers.

Figure 4-17 Overview of a cache block chain

[Figure]

(12) Specifying the threshold for the number of cache blocks

If a specific DAM file is heavily accessed, the number of cache blocks managed by the DAM file increases. As a result, searching for and releasing the target cache blocks takes time, resulting in decreased transaction performance.

By specifying the damchlmt definition command in the DAM service definition, you can set a limit for the maximum number of cache blocks that can be managed by a single DAM file. This limit is called a threshold. Cache blocks are secured up to the threshold. When the threshold is reached, unused cache blocks for the corresponding DAM file are reused and new cache blocks are not secured. If you do not specify a threshold, cache blocks are secured until the shared memory resource is used up.

During online operation you can use damchdef command to dynamically change the value to be specified for the damchlmt definition command in the DAM service definition.

(a) Calculations for cache blocks

You can calculate the size of the area used for cache blocks as follows:

Cache block area size = M - (M/576) [Figure] 34 (bytes)

You can calculate the size of each cache block as follows:

Cache block size = (Ab + 8) + 64 (bytes)

M: Size of the DAM service-exclusive shared memory

Ab: Length of a block in the DAM file to be accessed

(b) Examples

Specifying a threshold is effective when a specific DAM file is heavily accessed. The following conditions must be satisfied to specify a threshold:

#: Depends on the status of the hardware or the process.
A simple value is used for the purpose of explanation.

As described above, when you specify 0 as the threshold for the number of cache blocks, the number of cache blocks connected to the cache block chain depends on the type of access made by the UAP and the values set in the DAM service definition. The same phenomenon occurs when you specify a small value as the threshold for the number of cache blocks. A small value refers to a number smaller than the number of blocks that are accessed in a transaction executed by a UAP.

In Example 3, part of the DAM service-exclusive shared memory is not used as in Example 2.

(c) Settings according to the type of access made by UAPs

Imagine that the following information is output after you execute the damchinf command:

 
CleanUP Count:1  Next CleanUP FILE-No:1  Using Rate:80%
FileNo FileName BlkLen BlkNum CchBlkNum PreservNum LimitNum ReUse
     1 damfile1    504  10000      7900          0       -1 Exist
     0 damfile0    504  10000       100          0       -1 Exist
     2 damfile2    504  10000         0          0       -1  None
 

This section explains the suitable threshold for the number of cache blocks for each type of access made by a UAP.

(d) Notes

Note the following:

(13) Specifying the search sequence when reusing cache blocks

In the dam_cache_reuse_from operand in the DAM service definition, you can specify the first cache block you want to reuse in the cache block chain when a transaction that accesses a DAM file requires a new cache block.

When you specify last, cache blocks are reused from the oldest cache block connected to the cache block chain.

When you specify first, cache blocks are reused from the latest cache block connected to the cache block chain.

The default is last.

(14) Specifying the boundary for reusing cache blocks

In the damcache definition command in the DAM service definition, you can specify the boundary for reusing cache blocks. If you have specified a boundary, when securing new cache blocks, the reuse of cache blocks for another DAM file takes priority over reuse of cache blocks for the accessed DAM file. This applies if the number of cache blocks has not reached the boundary for reusing cache blocks.

If the number of cache blocks exceeds the boundary for reusing cache blocks, the cache blocks for the accessed DAM file are reused. Therefore, when you specify 0 for the boundary for reusing cache blocks, cache blocks for the accessed DAM file are reused as in the usual processing.

In the dam_default_cache_num operand in the DAM service definition, you can specify the boundary for reusing cache blocks in a logical file without the damcache definition command specified.

Specifying a boundary is effective when you want to heavily access many blocks in a DAM file and then frequently access another DAM file, which means that a small number of DAM files occupy cache blocks and only a few cache blocks are allocated to a different DAM file that is accessed frequently.

When you access multiple DAM files at random, which means that cache blocks are allocated to each DAM file uniformly, specifying a small value for the boundary for reusing cache blocks releases cache blocks frequently, increasing memory securing and data read processing. This may degrade the performance.

(a) Example that improves the performance

For the purpose of explanation, two DAM files are used here and the block lengths are all the same. The maximum number of cache blocks that can be secured in the entire system is 100. A UAP references 98 blocks of DAM file A and then starts to access DAM file B.

When the UAP accesses DAM file B, the number of blocks it references increases by one in each access. The UAP will reference up to 50 blocks.

1st access: Blocks 1 and 2 are referenced.

2nd access: Blocks 1, 2, and 3 are referenced.

3rd access: Blocks 1, 2, 3, and 4 are referenced.

:

49th access: Blocks 1, 2, 3, ... and 50 are referenced.

Under these conditions, the difference between specifying and not specifying the damcache definition command is described below.

When you do not specify the damcache definition command:

  1. When the UAP accesses DAM file B for the first time, the DAM service secures the cache block area from the shared memory. At this time, 98 cache blocks are already secured for DAM file A. Therefore, the DAM service can secure only two cache blocks. The DAM service reads the data of blocks 1 and 2 into the cache blocks and connects the cache blocks to the cache block chain for DAM file B.
  2. When the UAP references blocks 1, 2, and 3, file I/O operations are not performed for blocks 1 and 2 since their data exists in the cache. The UAP accesses block 3 for the first time. Therefore, the DAM service attempts to secure a cache block for block 3 to read data. However, since free space runs out in the shared memory in step 1, cache blocks for DAM file B are reused. For reuse processing, the DAM service disconnects the cache block containing the data of block 1 from the cache block chain, reads the data of block 3 into the cache block, and then connects the cache block to the beginning of the cache block chain.
  3. When the UAP references blocks 1, 2, 3, and 4, cache blocks are reused as in step 2. The DAM service releases the cache block for block 2, reads block 1 into the cache block, and connects the cache block to the beginning of the cache block chain. Then to read the data of block 2, the DAM service reuses the cache block of block 3 and reads block 2. This processing is repeated for blocks 3 and 4.
  4. When step 3 is repeated, the DAM service reuses cache blocks for DAM file B, which means that the DAM service frequently releases cache blocks from the cache block chain and frequently executes file I/O operations to read data. About 1275 release operations are performed and about 1,275 file I/O operations are performed.

When you specify the damcache definition command:

You specify 50 as the boundary for reusing cache blocks for DAM file A and specify 90 as the boundary for reusing cache blocks for DAM file B.

  1. The DAM service secures two unused cache blocks as in step 1 when you do not specify the damcache definition command, reads the data of blocks 1 and 2 into the cache blocks, and connects the cache blocks to the cache block chain.
  2. When the UAP references blocks 1, 2, and 3, file I/O operations are not performed for blocks 1 and 2 since their data exists in the cache. For the cache block for reading the data of block 3, the DAM service releases and secures a cache block for a DAM file whose number of cache blocks exceeded the boundary for reusing cache blocks (DAM file A in this case). This release and securing of a cache block is performed because since the number of cache blocks for DAM file B does not exceed the boundary for reusing cache blocks. The DAM service reads the data of block 3 into the secured cache block and connects the cache block to the cache block chain. Three cache blocks are now connected to the cache block chain for DAM file B.
  3. When the UAP references blocks 1, 2, 3, and 4, file I/O operations are not performed for blocks 1, 2, and 3 since their data exists in the cache. When the DAM service reads the data of block 4, a single file I/O operation is executed to reuse the unused cache block for DAM file A.
  4. Step 3 is repeated until the number of cache blocks for DAM file A exceeds the boundary for reusing cache blocks. The release from the cache block chain is executed approximately 50 times and file I/O is also executed about 50 times, improving the performance.
(b) Example that degrades the performance

The conditions, such as the block lengths in DAM files, are the same as the previous example. DAM files are accessed as follows:

1st access: Blocks 1 to 50 of DAM file A are referenced.

2nd access: Blocks 1 to 50 of DAM file B are referenced.

3rd access: Blocks 51 to 100 of DAM file A are referenced.

4th access: Blocks 1 to 50 of DAM file B are referenced.

:

The UAP references DAM files A and B alternately. The UAP references the next 50 blocks of DAM file A for each access.

Since the maximum number of cache blocks is 100, 50 cache blocks of DAM file A and 50 blocks of DAM file B are connected by the second access. For the third and later accesses, the processing differs depending on whether you specify the damcache definition command.

(c) Notes

Note the following:

(15) Cacheless access for unrecoverable DAM files

As described in 4.4.1(11) Using cache blocks, the DAM service stores the block data in the DAM files that are once read in the DAM service-exclusive shared memory to reduce the number of file I/O operations.

However, for the unrecoverable DAM files only, you can directly perform file I/O operations and return the block data in the disk to UAPs. This is called cacheless access. To enable cacheless access for the unrecoverable DAM files, specify the -f option of the damfile definition command in the DAM service definition.

When you specify cacheless access, file I/O operations are performed for each reference and update, and performance may degrade.

When you use DAM files with cacheless access specified, you can reduce the amount of memory required by obtaining the appropriate size of DAM service-exclusive shared memory and then specifying that value in the dam_cache_size_fix operand in the DAM service definition. For details, see the description of the dam_cache_size_fix operand in the DAM service definition in the manual OpenTP1 System Definition.