HiRDB Datareplicator Version 8 Description, User's Guide and Operator's Guide

[Contents][Glossary][Index][Back][Next]

4.7.7 Designing the target Datareplicator's resources

This section explains how to design the target Datareplicator's disk and memory resources.

Organization of this subsection
(1) Designing the target Datareplicator's disk resources
(2) Designing the target Datareplicator's memory resources

(1) Designing the target Datareplicator's disk resources

The following table lists the target Datareplicator's disk resources.

Table 4-43 Target Datareplicator's disk resources

File File type Required/optional [number of files] See Action when file is full
R C
Definition files Import system definition file Y N Required
[1 per import system]
(a) None
Import environment definition file Y N Required
[1 per data linkage identifier]
Import definition file Y N Optional
[1 per data linkage identifier]
Duplexing definition file Y N Optional
[1 per target system]
Import information queue files#1, #3 Y Y#2 Required
[2-8 per data linkage identifier]
(b) Swapped#1
Import status file#1, #3 Y Y#2 Required
[1 per source system]
(c) Stops import processing that corresponds to the source system
Import master status file Y Y#2 Required
[1 per target system]
(d) Stops import processing
Unimported information files Y N Required
[2 per data linkage identifier]
(e) Swapped
Import error information files Y N Required
[2 per target system]
(f) Swapped
Activity trace files (import trace files) Y N Optional
[2 per target system]
(g) Swapped
Update information definition files Y N Required
[1 per update information input command execution]
(h) None
SAM files Y N Required
[1 per update information input command execution]
(i) None
Unextracted data storage file Y N Required
[1 per data linkage identifier]
(j) None
Command log files Y N Optional
[2]
(k) Swapped

Legend:
R: UNIX regular file or Windows file
C: UNIX character special file
Y: Can be created.
N: Cannot be created.

#1
If import processing is not completed on the file to be swapped, the target Datareplicator stops receiving update information until the next transmission interval. If import processing is completed by then, the target Datareplicator starts receiving update information.

#2
If you use character special files with the AIX edition, add 1,024 bytes to the formula for determining the size of each file. In the queuesize and statssize operands in the import environment definition, specify the size obtained from the formula minus 1,024 bytes.

#3
If you use a Datareplicator file system area, also see 3.5.3(1) Rules for allocating a Datareplicator file system area.
(a) Sizes of definition files

The sizes of the target Datareplicator's definition files (import system definition file, import environment definition file, and import definition file) depend on the definition operands that are specified.

(b) Size of an import information queue file

Datareplicator stores three types of information in an import information queue file; these include connection information, extraction definition information, and update information. Each type of information is explained below.

Information stored in the import information queue file

Formulas for determining the size of an import information queue file

The following table shows the formulas for determining the size of an import information queue file.

Table 4-44 Formulas for determining the size of an import information queue file

Item Formula (bytes) Description of variables
Connection information 512 None
Extraction definition information 512 x [Figure](650 + D + E + F) / 512[Figure] D = 32 + 96 x number of authorization identifiers + 64 x number of extraction definition tables + 128 x number of extraction definition columns
E = 88 x [Figure]D / update information buffer size#[Figure]
F = 32 x [Figure](D + E) / 32,453[Figure]
Update information [Figure] (TRN + [Figure]INS + [Figure]UPD + [Figure]DEL + [Figure]PRG)
TRN = 104
INS = Ir | Ic
Ir = 96 + Row
Ic = 80 + (8 x Cni) + Row
UPD = Ur | Uc
Ur = 96 + Row
Uc = 90 + (8 x Cnu) + Cl
DEL = 80 + (8 x Dni) + Dl
PRG = 80

[Figure]
The first [Figure] indicates the sum of the update information items. The subsequent [Figure]s (with INS, UPD, DEL, and PRG) indicate the sums for each row.

TRN
Transaction information

INS
INSERT rows

UPD
UPDATE rows

DEL
DELETE rows

PRG
PURGE TABLE rows

|
Indicates OR (selection)

Ir
Row interface

Ic
Column interface

Row
Row length

Cni
Number of columns subject to insertion processing

Ur
Row interface

Ucn
Column interface

Cnu
Sum of the number of columns subject to update processing and the number of mapping keys
-- --

Cl
Sum of the lengths of updated columns and the lengths of mapping key columns

Dl
Length of mapping key

Dni
Number of mapping key columns

#: If the source database is a HiRDB, the size of the update information buffer is the size of the update information (smt_editbufsize operand value in 5.2 Extraction system definition, or the editbufsize operand value in 5.4 Transmission environment definition). For details about the size of the update information buffer when the source database is a mainframe database, see the manual VOS3 XDM Data Linkage Facility XDM/DS Description and Definition.

Notes on estimating the size of an import information queue file

Note the following point about estimating the size of an import information queue file:

(c) Size of the import status file

The following are the formulas for determining the size of the import status file:

The import status file is used to store the extraction definition information that is sent from the source database. Estimate the size of the extraction definition information based on the size of the extraction definition preprocessing file. If the size of the estimated extraction definition information is too small, a file shortage error occurs when update information is sent from the source system.

For UNIX
164 x 1,024 + size of extraction definition information (bytes)

For Windows
282 x 1,024 + size of extraction definition information (bytes)

For a character special file for UNIX whose sector size exceeds 1 kilobyte
164 x SCT_SIZE + [Figure]size of extraction definition information / SCT_SIZE[Figure] x SCT_SIZE (bytes)
SCT_SIZE: Sector size (bytes)
(d) Size of the import master status file

This subsection provides the formula for determining the size of the import master status file.

In the case of the import master status file, the required size is allocated when the target Datareplicator is initialized. The file will never become full because the file capacity is increased during the actual replication.

For UNIX or Windows
2 x 1,024 (bytes)

For a character special file for UNIX whose sector size exceeds 1 kilobyte
2 x SCT_SIZE (bytes)
SCT_SIZE: Sector size (bytes)
(e) Size of an unimported information file

The default size is 16KB. If you want to retain information about SQL errors that might occur, increase the file size.

(f) Size of an import error information file

The default size is 16KB. If you want to retain error information covering a long period of time, increase the file size.

(g) Size of an activity trace file

Use the int_trc_filesz operand in the import system definition to specify the size of an activity trace file (import trace file).

(h) Size of the update information definition file

The size of the update information definition file depends on the definition operands that are specified.

(i) Size of a SAM file

The size of a SAM file depends on the size of a file transferred from PDM2 E2 or RDB1 E2.

(j) Size of the unextracted data storage file

The size of the unextracted data storage file depends on the number of data items that are not subject to extraction processing. The size of the unextracted data storage file is increased as much as the space available in the file system.

(k) Size of a command log file

The size is always 128 KB.

(l) Sizes of other files

The Windows edition of Datareplicator creates several work files in the tmp directory under the installation directory. Therefore, provide 4 MB of space for the work files.

(2) Designing the target Datareplicator's memory resources

The following table provides a list of memory resources for the target Datareplicator.

Table 4-46 List of memory resources for the target Datareplicator

Memory resource Subsection
Import master process (a)
Import communication master process (b)
Import definition server process (c)
Import process (d)
Import SQL process (e)
Import UOC process (f)
Update information input process (g)
Activity trace collection process (h)
Datareplicator agent process (i)
Size of shared memory for storing definition information (j)
Size of shared memory for communication between import processes (k)
Size of shared memory for importing BLOB columns (l)
Number of required semaphores (m)

Table 4-47 List of variables used in the formulas for determining the size of memory for target Datareplicator

Variable name Description of variable
ATTR_NUM Total number of abstract data type attributes subject to data linkage
CNST_NUM Total number of name clauses specified in the const clauses of all format statements specified in the import definition
COL_MAX Maximum value of the total actual data length of columns subject to extraction per table subject to extraction (BLOB and BINARY columns are excluded) (bytes)
COL_MNUM Maximum number of columns per table in all tables subject to extraction
COL_NUM Total number of columns in all tables subject to extraction
CUOC_CLM_NUM Total number of imported columns for which a column data editing UOC routine is specified
DUP_UPD Total number of duplicate update information names defined per extraction table
ELEM_LEN Maximum value of the total length of all repetition column elements per table subject to extraction (bytes)
(This is the actual length extracted, not the definition length)
ELEM_NUM Maximum value of the total number of repetition column elements per table subject to extraction
(This is the actual number of elements extracted, not the defined number of elements)
EXCS_R Total number of columns subject to import (including columns subject to import that are assumed when true is specified in the defmerge operand and the load statement is omitted in the import environment definition)
FLD_NUM Total number of field names specified in all load statements defined in the import definition
FOR_NUM Total number of format statements specified in the import definition
GRP_NUM Total number of group statements specified in the import definition
GRPD_NUM Total number of groups specified in the group statement specified in the import definition
GRPHA_NUM Total number of hash-partitioned RDAREAs for tables for which the hash clause is specified in the group statement in the import definition
GRPHC_LEN Total length of hash key column values in tables for which the hash clause is specified in the group statement in the import definition (bytes)
GRPHC_NUM Total number of hash key columns in tables for which the hash clause is specified in the group statement in the import definition
GRPK_NUM Total number of key range partitioning conditions in the group statement specified in the import definition
GRPT_NUM Total number of tables specified in the group statement specified in the import definition
IDX_NUM Maximum number of indexes created per table subject to import
IDXCLM_NUM Maximum number of index component columns for indexes created per table subject to import
KEY_MAX Maximum value of the total length of mapping keys per table subject to extraction (bytes)
KEY_MNUM Maximum value of the total number of mapping keys per table subject to extraction
KEY_NUM Total number of mapping keys in all tables subject to extraction
LOD_NUM Total number of load statements specified in the import definition
NAME_NUM Total number of name clauses in all format statements specified in the import definition
NCHR_MAX Maximum length of a column definition of national character string attributes (NCHAR, NVARCHAR, and LONG NVARCHAR) subject to extraction (bytes)
PRP_FILE Size of extraction definition preprocessing file (bytes)
RDEF_FILE Size of import definition file (bytes)
REP_NUM Total number of repetition columns in all tables subject to extraction
RMST_FILE Size of import master status file (bytes)
RQUE_FILE Size of import information queue file (bytes)
RST_FILE Size of import status file (bytes)
RUOC_ATTR_NUM Total number of data attributes in ADT that will be passed to an import information editing UOC routine
RUOC_COL_MAX Maximum number of import columns that will be passed to an import information editing UOC routine
RUOC_KEY_MAX Maximum number of mapping key columns that will be passed to an import information editing UOC routine
RUOC_TYPE_NUM Total number of data types in ADT that will be passed to an import information editing UOC routine
SKIP_LIST Number of rows in the import suppression list file
SKIPTYPEONLY Total number of SKIP_TYPE_ONLY clauses specified in the import suppression list file
SV_NUM

When the source is HiRDB:
Total number of back-end servers subject to extraction

When the source is not HiRDB:
1
TBL_NUM Total number of tables subject to extraction
TBLT_R Total number of tables subject to import (including tables subject to import that are assumed when true is specified in the defmerge operand and the load statement is omitted in the import environment definition)
TYPE_NUM Total number of abstract data types in all tables subject to extraction
UBUF

When the source database is HiRDB:
  • If nodemst is specified in sendcontrol in the extraction system definition
    editbufsize value specified in the transmission environment definition
  • If sendmst is specified in sendcontrol in the extraction system definition
    smt_editbufsize value specified in the extraction system definition

When the source database is XDM/DS:
REFLECTBUFF value specified in the XDM/DS start definition. For details, see the manual VOS3 XDM Data Linkage Facility XDM/DS Description and Definition (Specify this value in KB)
UPD_NUM Total number of update information names
(a) Import master process

Number of processes
There can be only one import master process per target Datareplicator.

Size of procedure
204,800 bytes

Dynamic memory size (bytes)
1,024
(b) Import communication master process

Number of processes
Number of corresponding data linkage identifiers + 1

Size of procedure
256,000 bytes

Dynamic memory size (bytes)
The dynamic memory size is the sum of the fixed part shown below and the variable part explained below if applicable.
Fixed part:
UBUF x 1,024 + 136
+ UBUF x 1,024
+ [Figure](UBUF x 1,024 + 64) / 1,024[Figure] x 1,024
+ 216
+ 281
+ 1,024
+ 68,228
+ 5,219
+ 10,280 + MAX(1,024, PRP_FILE / SV_NUM)
+ MAX(RMST_FILE, RST_FILE, RQUE_FILE) - 1,024
+ 512,000
Variable part:
  • Formula to be added if the duplexing function
    65,536

Explanation of the variables
See Table 4-47 List of variables used in the formulas for determining the size of memory for target Datareplicator.
(c) Import definition server process

Number of processes
As many import definition server processes are required as there are corresponding data linkage identifiers.

Size of procedure
409,600 bytes

Dynamic memory size (bytes)
The dynamic memory size is the sum of the fixed part shown below and any applicable variable part explained below.
Fixed part:
([Figure](UPD_NUM + LOD_NUM) / 16[Figure] x 16) x 160
+ ([Figure](COL_NUM + FLD_NUM) / 16[Figure] x 16 + TBL_NUM) x 128
+ [Figure](PRP_FILE / SV_NUM + 152) / 1,024[Figure] x 1,024
+ PRP_FILE / SV_NUM
+ [Figure](PRP_FILE / SV_NUM + 64) / 1,024[Figure] x 1,024
+ PRP_FILE / SV_NUM
+ [Figure](PRP_FILE / SV_NUM + 64) / 1,024[Figure] x 1,024
+ ([Figure]COL_NUM / (TBL_NUM x 16)[Figure] + TBL_NUM) x 384
+ [Figure]TBLT_R / 16[Figure] x 704
+ 260
+ 68,228
+ 5,219
+ 10,280 + MAX(1,024, PRP_FILE / SV_NUM)
+ MAX(RMST_FILE, RST_FILE, RQUE_FILE) - 1,024
+ [Figure]UPD_NUM / 16[Figure] x 704
+ 512,000
Variable part:
  • Formula to be added if the import definition is specified
    RDEF_FILE
  • Formula to be added if the format statement with the const clause specified is defined in the import definition
    [Figure]CNST_NUM / 16[Figure] x 832
  • Formula to be added if the format statement is specified in the import definition
    ([Figure]Total number of format statements specified in the import definition / 16[Figure] x 16) x 36
  • Formula to be added if the format statement is defined in the import definition
    [Figure]FOR_NUM / 16[Figure] x 576
    + [Figure]NAME_NUM / 16[Figure] x 768
  • Formula to be added if the load statement is specified in the import definition
    [Figure]LOD_NUM / 16[Figure] x 832
    + [Figure]LOD_NUM / 8[Figure] x 704
    + [Figure]FLD_NUM / 16[Figure] x 384
  • Formula to be added if an asterisk (*) is not specified in the field specification in the load statement
    [Figure]FLD_NUM / 16[Figure] x 896
  • Formula to be added if the target able specified in the load statement contains character set columns
    32
  • Formula to be added if the group statement is specified in the import definition
    [Figure](GRP_NUM + 2) / 16[Figure] x 2,752
    + ([Figure]GRPT_NUM / 16[Figure] + GRP_NUM) x 64
  • Formula to be added if groups are specified in the group statement
    [Figure](GRPD_NUM + 2) / 16[Figure] x 1,024
  • Formula to be added if the following conditions are true:
    not_null_unique or unique is specified in the mapping_key_check clause in the import environment definition.
    not_null_unique or unique is specified in the check clause of the LOAD statement in the import definition
    8 x IDX_NUM + 32 x IDXCLM_NUM
    + 16 + 4 x IDX_NUM
    + 580
  • Formula to be added if key range partitioning conditions are specified in the group statement in the import definition
    ([Figure]GRPK_NUM / (GRPD_NUM x 16)[Figure] + GRPD_NUM) x 4,416
    + 32
  • Formula to be added if columns with character set specification are linked
    32
  • Formula to be added if abstract data-type columns are linked
    64 + 112 x TYPE_NUM + 64 x ATTR_NUM
  • Formula to be added if the duplexing function
    65,536

Explanation of the variables
See Table 4-47 List of variables used in the formulas for determining the size of memory for target Datareplicator.
(d) Import process

Number of processes
When the transaction-based import method is used, only one import process is required. When the table-based import method is used, the number of required import processes equals the maximum number of import groups.

Size of procedure
256,000 bytes

Dynamic memory size (bytes)
The dynamic memory size is the sum of the fixed part shown below and any applicable variable part explained below.
Fixed part:
1,464
+ MAX(48,000, COL_MAX + 4,096)
+ 32
+ 67
+ UBUF x 1,024
+ 60
+ 845 + KEY_MAX
+ 19,632
+ 12 x [Figure]KEY_MNUM / 10[Figure] x 10
+ 12 x [Figure]COL_MNUM / 50[Figure] x 50
+ 68,228
+ 5,219
+ 10,280 + MAX(1,024, PRP_FILE / SV_NUM)
+ MAX(RMST_FILE, RST_FILE, RQUE_FILE) - 1,024
+ 512,000
Variable part:
  • Formula to be added if the hash clause is specified in the group statement in the import definition
    GRPHA_NUM x 4
    + GRPHC_NUM x 4
    + GRPHC_NUM x 4
    + GRPHC_NUM x 256
  • Formula to be added if the hash clause is specified in the group statement in the import definition and the hash key columns include at least one column with character set specification
    GRPHC_NUM x 4
  • Formula to be added if the hash clause is specified in the group statement in the import definition and the hash value is invalid
    GRPHC_LEN
  • Formula to be added if the import suppression function is used
    92
    + 52 x SKIP_LIST
    + [Figure]SKIPTYPEONLY / 10[Figure] x 40
  • Formula to be added if the source database is XDM/DS and data linkage is applied to the NCHAR, NVARCHAR, or LONG NVARCHAR type
    NCHR_MAX + 2
  • Formula to be added if the duplexing function is used
    65,536

Explanation of the variables
See Table 4-47 List of variables used in the formulas for determining the size of memory for target Datareplicator.
(e) Import SQL process

Number of processes
When the transaction-based import method is used, only one import SQL process is required. When the table-based import method is used, the number of required import SQL processes equals the maximum number of import groups.

Size of procedure
307,200 bytes

Dynamic memory size (bytes)
The dynamic memory size is the sum of the fixed part shown below and any applicable variable part explained below.
Fixed part:
1,448
+ 88 + 40 x COL_MNUM + 39 x KEY_MNUM
+ 197 + 40 x COL_MNUM + 39 x KEY_MNUM
+ 4,096
+ 4,096
+ 68 x TBLT_R
+ 32 x TBLT_R + 16 x (EXCS_R - 1) + 2 x EXCS_R
+ 32 x TBLT_R + 16 x (EXCS_R + KEY_NUM - 1) + 2 x (EXCS_R + KEY_NUM)
+ 32 x TBLT_R + 16 x (EXCS_R - 1) + 2 x EXCS_R
+ 2 x EXCS_R
+ 68,228
+ 5,219
+ 10,280 + MAX(1,024, PRP_FILE / SV_NUM) + MAX(RMST_FILE, RST_FILE, RQUE_FILE) - 1,024
+ 512,000
Variable part:
  • Formula to be added if the WITH LOCK clause is specified in the LOAD statement in the import definition
    48 x [Figure]LCKT_NUM / 10[Figure] x 10
  • Formula to be added if columns with character set specification are linked
    96 x TBLT_R + 2 x EXCS_R
    + 96 x TBLT_R + 2 x (EXCS_R + KEY_NUM)
    + 96 x TBLT_R + 2 x KEY_NUM
  • Formula to be added if abstract data-type columns are linked
    88 + 35 x COL_MNUM
  • Formula to be added if repetition columns are linked
    16 x (REP_NUM + 1)
    + (ELEM_LEN + 2 x ELEM_NUM + 8 x TBLT_R)
  • Formula to be added if abstract data-type columns or repetition columns are linked
    84 + 40 x COL_MNUM + 39 x KEY_MNUM
  • Formula to be added if columns with character set specification are linked
    52 x CNST_NUM
  • Formula to be added if a column data UOC routine is sued
    CUOC_CLM_NUM x 48
    + CUOC_CLM_NUM x 32,008
  • Formula to be added if one extraction table contains duplicate update information names
    20 x [Figure]DUP_UPD / 5[Figure]
  • Formula to be added if abstract data-type columns are linked
    80 x TYPE_NUM
  • Formula to be added if the duplexing function is used.
    65,536

Explanation of the variables
See Table 4-47 List of variables used in the formulas for determining the size of memory for target Datareplicator.
Note:
If the extraction column types include the BLOB type, calculate the corresponding column as 20 bytes in the calculation of the maximum update row length.
(f) Import UOC process

Number of processes
Only one import UOC process is required when a UOC routine is used.

Size of procedure
(102,400 + size of UOC routine) bytes

Dynamic memory size (bytes)
The dynamic memory size is the sum of the fixed part shown below plus any applicable variable part explained below.
Fixed part:
408
+ MAX(48,000, COL_MAX + 4,096)
+ 68,228
+ 5,219
+ 10,280 + MAX(1,024, PRP_FILE / SV_NUM)
+ MAX(RMST_FILE, RST_FILE, RQUE_FILE) - 1,024
+ 512,000
+ 48 x RUOC_COL_MAX
+ 48 x RUOC_KEY_MAX
+ 824
Variable part:
  • Formula to be added if update information names that use an import information editing UOC routine include ADT columns
    120 x [Figure]RUOC_TYPE_NUM / 10[Figure]
    + 84 x RUOC_TYPE_NUM + 60 x RUOC_ATTR_NUM
  • Formula to be added if the file duplexing function is used
    65,536

Explanation of the variables
See Table 4-47 List of variables used in the formulas for determining the size of memory for target Datareplicator.
(g) Update information input process

Number of processes

Only one update information import process is required when data linkage uses a SAM file.

Size of procedure

250KB

Dynamic memory size (KB)

300 + DBUF + TBUF

Explanation of the variables

DBUF: 0.5 x total number of fields in the extraction statements in the update information definition

TBUF: Total length of update data per transaction x 2

(h) Activity trace collection process

Number of processes
One process is required for the target Datareplicator.

Size of procedure
60KB

Dynamic memory size
50KB
(i) Datareplicator agent process

Number of processes
Add the number of processes either at the source system or at the target system.

Size of procedure
70KB

Dynamic memory size (KB)
[Figure]300,000# + (HDS_NUM + DC_NUM x 3 + GRP_NUM) x 64 / 1,024[Figure]

Explanation of the variables
HDS_NUM: Total number of target systems
DC_NUM: Total number of data linkages
GRP_NUM: Total number of import groups
#: Use the value for either the source system or the target system.
(j) Determining the size of shared memory for storing definition information

The following is the formula for determining the size of shared memory for storing definition information that is specified with the defshmsize operand in the import environment definition:

[Figure]192 x (GRPT + 2) + 160 x TBLT_E + 152 x EXCS_E + 156 x TBLT_r + 128 x EXCS_r + 88 x TBLT_R + 24 x EXCS_R + 52 x CNST[Figure] (bytes)
GRPT: Number of import group definitions in the import definition file (number of group statements in the import definition)
TBLT_E: Number of extraction tables
EXCS_E: Number of extraction columns
TBLT_r: Number of import table definitions in the import definition file (number of load statements in the import definition)
EXCS_r: Total number of columns in the import table specified in the import table definition
TBLT_R: Number of import tables (including the import tables that are assumed when true is specified in the defmerge operand described in 5.9 Import environment definition and the load statement is omitted)
EXCS_R: Number of import columns (including the import columns that are assumed when true is specified in the defmerge operand described in 5.9 Import environment definition and the load statement is omitted)
CNST: Total number of const clauses specified in the update information field definition in the import definition file
(k) Determining the size of shared memory for communication between import processes

82,292 + 71,556 x UJ (bytes)
UJ: Number of source systems
(l) Determining the size of shared memory for importing BLOB columns

The following shared memory size is required for each import group:

(LOB_SIZE + 4) x 4 +80 (bytes)
LOB_SIZE: Maximum value in the import group (total length of defined BLOB columns that are included in each update information name)
(m) Determining the number of required semaphores

The following is the formula for determining the number of semaphores used by the target Datareplicator:

5 + 10 x UJ
UJ: Number of source systems

Change the maximum number of semaphores in the entire system that is specified in the kernel parameter (SEMMNS) so that the number of semaphores determined by the above formula can be used at the target system. For details about how to update the kernel parameter, see the applicable OS documentation.