5.3.6 Analyzing the cause of an error

This subsection describes the functions for analyzing the causes of errors in the OpenTP1 system.

Note that the functions listed below assume that TP1/Extension 1 has been installed. If TP1/Extension 1 has not been installed, the operation of these functions cannot be guaranteed.

Organization of this subsection
(1) MCF trace
(2) UAP trace
(3) RPC trace
(4) Performance verification trace
(5) XAR performance verification trace
(6) JNL performance verification trace
(7) LCK performance verification trace
(8) MCF performance verification trace
(9) XAR event trace
(10) TRN event trace
(11) NAM event trace
(12) Process service event trace
(13) FIL event trace
(14) Command logs

(1) MCF trace

With the MCF trace facility, MCF obtains information, such as events and data sent or received, for each process. The MCF trace information is output to the trace area (trace buffer) in the shared memory. If the MCF trace disk output facility is specified in the MCF communication configuration definition and the trace area becomes full, the MCF trace information is output to an MCF trace file.

You might temporarily require the MCF trace information while the disk output facility is disabled. In this case, you can output the MCF trace information to an MCF trace file by using the MCF-trace-information acquisition start command (mcftstrtr) and the MCF-trace-information acquisition stop command (mcftstptr). For details about the commands, see the manual OpenTP1 Operation.

The MCF trace facility collects information about the functioning, before an error occurred, of control functions within a process and the control flow for each event.

(2) UAP trace

When an OpenTP1 UAP (SUP, SPP, or MHP) terminates abnormally, trace information is output from the API called by the UAP. This is called a UAP trace, and the acquired information is referred to as UAP trace information. OpenTP1 acquires UAP trace information separately for each UAP process.

UAP trace information is stored as files named server-nameN.uat (where N represents a serial number appended to the core file of the UAP process) under the $DCDIR/spool/save/ directory. These files are known as UAP trace edit/output files.

UAP trace information can be output from a core file generated at abnormal termination of the UAP. To obtain the trace information, execute the uatdump command, specifying the required core file. The UAP trace information can be output to a specified file by redirecting the uatdump command's output destination.

The value specified in the uap_trace_max operand of the user service definition determines the maximum size of UAP trace information that can be stored in a file. When this value is reached, subsequent trace information is wrapped around to the start of the file.

UAP trace information is acquired separately for each abnormally terminated UAP process. When transaction processing called by an RPC on another node terminates abnormally, UAP trace information is acquired for each of the UAP processes executed on the two nodes.

For details about UAP traces, see description in the OpenTP1 Tester and UAP Trace User's Guide.

(3) RPC trace

OpenTP1 has an RPC trace facility to obtain information about RPC service requests; this information is collected into a file. You can use an RPC management command rpcdump to dump the RPC trace information. Some of the uses of the RPC trace facility are:

You can use the rpcdump command to edit the contents of an RPC trace file.

You can obtain an RPC trace file for each system service. To obtain the file, in the system service definition for each service specify that the trace is to be obtained. You can use the rpcmrg command to merge and print, in chronological order, the RPC trace files from several system services.

(4) Performance verification trace

OpenTP1 has a performance verification trace to acquire trace information, such as an OpenTP1 identifier, for major events of services running in OpenTP1.

The advantages of conducting a performance verification trace are:

The following table lists and describes the system definitions that are related to performance verification trace.

Table 5-5 System definitions related to performance verification trace

Definition nameFormatOperandDescription
System common definitionsetprf_traceSpecifies whether performance verification trace information is to be acquired.
settrn_prf_trace_levelSets the trace acquisition level
Performance verification trace definitionsetprf_file_sizeSets the size of trace file

For details about the definitions, see the manual OpenTP1 System Definition.

You can use the prfget command to retrieve a performance verification trace in binary format and then use the prfed command to output it in character format.

You can also use the dc_prf_utrace_put function to collect user-specific trace information in a trace file and the dc_prf_get_trace_num function to obtain the sequential number of the most recent trace acquired in the process.

Notes:
  • When OpenTP1 is restarted in the normal way or by a hot standby method, no trace information is carried over.
  • Trace acquisition performed by the performance verification trace facility is not subject to control by locks so that performance of online processing will not be affected. For this reason, if contention occurs during trace acquisition in a multiprocessor environment, some trace information may be missing or invalid trace information may be acquired. When trace information is edited using the prfed command, invalid information is displayed as error records.

(5) XAR performance verification trace

OpenTP1 collects trace information about all events involving transactional linkage using the XA resource service, such as transaction requests from an application server and OpenTP1 transaction processing. This information is known as an XAR performance verification trace.

XAR performance verification traces are stored as files with the file name _xr_001, _xr_002, _xr_003, and so on, under the $DCDIR/spool/dcxarinf/ directory. These files are known as XAR trace information files for verification. The output directory and file names cannot be changed for the XAR performance verification trace information files. You can change the size and number of trace information files that can be acquired.

For details, see the description of the XAR performance verification trace definition in the manual OpenTP1 System Definition.

To collect XAR performance verification trace information:

  1. Set Y in the prf_trace operand in the system common definition.
  2. Specify the level of XAR performance trace information to acquire in the xar_prf_trace_level operand in the XA resource service definition.

The following table describes the XAR performance verification trace information that is collected depending on the value of the xar_prf_trace_level operand.

Table 5-6 Relationship between the value specified in the xar_prf_trace_level operand and acquired XAR performance verification trace information

Value of xar_prf_trace_level operandType of XAR trace information collectedEvent IDTrace data size (bytes)Collection timing
00000001​Transaction request from an application server0x4a00128Request to start a transaction branchImmediately after call
0x4a01128Immediately before return
0x4a02128RPC execution request from within a transaction branchImmediately after call
0x4a03128Immediately before return
0x4a04128Request to terminate a transaction branchImmediately after call
0x4a05128Immediately before return
0x4a06128Request to prepare a transaction branch for commitImmediately after call
0x4a07128Immediately before return
0x4a08128Request to commit a transaction branchImmediately after call
0x4a09128Immediately before return
0x4a0a128Request to roll back a transaction branchImmediately after call
0x4a0b128Immediately before return
0x4a0c64Request to report a transaction branch in Prepared or Heuristically Completed statusImmediately after call
0x4a0d64Immediately before return
0x4a0e128Request to discard a Heuristically Completed transaction branchImmediately after call
0x4a0f128Immediately before return
00000002​OpenTP1 transaction process0x4b0064Start a transaction branchImmediately before
0x4b0164Immediately after
0x4b0264Execute an RPC from within a transaction branchImmediately before
0x4b0364Immediately after
0x4b0464Terminate a transaction branchImmediately before
0x4b0564Immediately after
0x4b0664Prepare a transaction branch for commitImmediately before
0x4b0764Immediately after
0x4b0864Commit a transaction branchImmediately before
0x4b0964Immediately after
0x4b0a64Roll back a transaction branchImmediately before
0x4b0b64Immediately after
0x4b0c64Report a transaction branch in Prepared or Heuristically Completed statusImmediately before
0x4b0d64Immediately after
0x4b0e64Discard a Heuristically Completed transaction branchImmediately before
0x4b0f64Immediately after

For details about the xar_prf_trace_level operand, see the description of the XA resource service definition in the manual OpenTP1 System Definition.

To collect XAR trace information files and output edited trace information, use the prfget and prfed commands. The acquisition and edit/output procedures are as follows.

Collecting XAR trace information for verification
To collect only the trace information that has not been acquired with the latest run ID, execute the following commands:

$DCDIR/bin/prfget -f _xr | $DCDIR/bin/prfed -d

To collect all trace information, execute the following commands:

$DCDIR/bin/prfget -a -f _xr | $DCDIR/bin/prfed -d

Editing and outputting XAR trace information for verification
To edit and output trace information from XAR trace information files for verification, execute the prfed command. Specify the options as required.
XAR trace information for verification is output in the same format as the performance verification trace. For details about the output format, see the description of the prfed command in the manual OpenTP1 Operation.

(6) JNL performance verification trace

OpenTP1 collects trace information about various events related to journal buffering and journal output performed by the journal service. This information is called JNL performance verification trace.

JNL performance verification traces are stored as files with the file name _jl_001, _jl_002, _jl_003, etc., under the $DCDIR/spool/dcjnlinf/prfinf directory. These files are called JNL performance verification trace information files. The output destination and names of the JNL performance verification trace information files cannot be changed.

To collect JNL performance verification trace information:

  1. Specify Y in the prf_trace operand in the system common definition.
  2. Also specify in the system common definition the level of JNL performance verification trace information to be acquired. Make this specification in the jnl_prf_event_trace_level operand.

The table below shows the trace information that is collected depending on the specification of the jnl_prf_event_trace_level operand. For details about the timing of acquiring each event, see the description of the acquisition of performance verification trace information in the manual OpenTP1 Operation.

Table 5-7 Relationship between the jnl_prf_event_trace_level operand value and the trace information that is collected

jnl_prf_event_trace_level operand valueEvent IDs of trace information
0xc202, 0xc203, 0xc401, 0xc4020xc001 to 0xc201, 0xc204 to 0xc400
00000000​NN
00000001​YN
00000002​YY
OtherYY
Legend:
Y: Trace information is acquired.
N: Trace information is not acquired.

You use the prfget command to collect JNL performance verification trace information files and the prfed command to edit and output the files. The following describes how to collect the files.

How to collect JNL performance verification trace information files:
To collect only the trace information that has not been acquired with the most recent run ID, execute the following commands:

$DCDIR/bin/prfget -f _jl | $DCDIR/bin/prfed -d

To collect all trace information, execute the following commands:

$DCDIR/bin/prfget -a -f _jl | $DCDIR/bin/prfed -d

(7) LCK performance verification trace

OpenTP1 collects trace information about locking involved in transaction processing. This information is called LCK performance verification trace.

LCK performance verification traces are stored as files with the file name _lk_001, _lk_002, _lk_003, etc., under the $DCDIR/spool/dclckinf/prf directory. These files are called LCK performance verification trace information files. The output destination and names of the LCK performance verification trace information files cannot be changed, but you can change the size and number of LCK performance verification trace information files that can be acquired. For details, see the description of the LCK performance verification trace definition in the manual OpenTP1 System Definition.

To collect LCK performance verification trace information:

  1. Specify Y in the prf_trace operand in the system common definition.
  2. In the lck_prf_trace_level operand in the lock service definition, specify the level of LCK performance verification trace information to acquire.

The table below shows the LCK performance verification trace information that is collected depending on the specification of the lck_prf_trace_level operand.

Table 5-8 Relationship between the lck_prf_trace_level operand value and the LCK performance verification trace information that is collected

lck_prf_trace_level operand valueLCK performance verification trace informationEvent IDTrace data size (bytes)Collection timing
00000000​Trace information about locking is not acquired.------
00000001​Trace information about locking is acquired.0x6400128Locking of resourcesImmediately after call
0x6401128Immediately before return
0x6410128Lock release waitImmediately before start
0x6411128Immediately after termination
0x6420128Unlocking of all resourcesImmediately after call
0x6421128Immediately before return
0x6430128Unlocking with a resource name specifiedImmediately after call
0x6431128Immediately before return

For details about the lck_prf_trace_level operand, see the description of the lock service definition in the manual OpenTP1 System Definition.

You use the prfget command to collect LCK performance verification trace information files and the prfed command to edit and output the files. The following describes how to collect and how to edit and output the files.

How to collect LCK performance verification trace information files:
To collect only the trace information that has not been acquired with the most recent run ID, execute the following commands:

$DCDIR/bin/prfget -f _lk | $DCDIR/bin/prfed -d

To collect all trace information, execute the following commands:

$DCDIR/bin/prfget -a -f _lk | $DCDIR/bin/prfed -d

How to edit and output LCK performance verification trace information files:
To edit and output trace information in LCK performance verification trace information files, execute the prfed command with options specified, as necessary.
The output format of LCK performance verification trace information is the same as for the performance verification trace. For details about the output format, see the prfed command in the manual OpenTP1 Operation.

(8) MCF performance verification trace

OpenTP1 collects trace information (such as the MCF identifier) about the main events involving message transmission using TP1/Message Control. This information is called MCF performance verification trace.

The MCF performance verification trace provides the following advantages:

The following table lists and describes the system definitions that are related to the MCF performance verification trace.

Table 5-9 System definitions related to the MCF performance verification trace

Definition nameFormatOperandDescription
User service definitionsetmcf_prf_traceSpecifies whether MCF performance verification trace information is acquired for each user server
MCF performance verification trace definitionsetprf_file_sizeSize of a trace file for the MCF performance verification trace information
setprf_file_countNumber of trace file generations for the MCF performance verification trace information
Definition of system service informationsetmcf_prf_traceSpecifies whether MCF performance verification trace information is acquired for each MCF communication service
System service common information definitionsetmcf_prf_trace_levelAcquisition level of MCF performance verification trace information

For details about the definitions, see the manual OpenTP1 System Definition.

You can use the prfget command to retrieve an MCF performance verification trace in binary format and then use the prfed command to output it in character format.

Notes
  • The default is that the MCF performance verification trace is not collected. To collect it, you must specify Y in the prf_trace operand or omit this operand in the system common definition and also specify 00000001 in the mcf_prf_trace_level operand in the system service common information definition.
  • You can collect the MCF performance verification trace during message transmission (event IDs 0xa000 and 0xa001) only when you use one of the following protocol products:
    [Figure]TP1/NET/TCP/IP
    [Figure]TP1/NET/XMAP3
    [Figure]TP1/NET/OSAS-NIF
    For events other than message transmission (event IDs other than 0xa000 and 0xa001), you can collect the MCF performance verification trace regardless of the type of communication protocol.

(9) XAR event trace

The XAR event trace acquires the type of the transaction request sent from an application server using the XA resource service as event trace information. The acquired information is called the XAR event trace information.

The following table lists the types of transaction requests sent from an application server, request codes, and their meaning.

Table 5-10 Types of transaction requests and request codes

Request type#Request codeMeaning of request code
Start()xar_startStarts a transaction branch.
Call()xar_callExecutes an RPC from within a transaction branch.
End()xar_endEnds a transaction branch.
Prepare()xar_preparePrepares the commit for a transaction branch (first phase of a two-phase commit).
Commit()xar_commitCommits a transaction branch (second phase of a two-phase commit).
Rollback()xar_rollbackRolls back a transaction branch.
Recover()xar_recoverNotifies a transaction branch in the Prepared status or the Heuristically Completed status.
Forget()xar_forgetDiscards a transaction branch in the Heuristically Completed status.
#
Transaction request types are internal functions of OpenTP1.

The XAR event trace information is acquired in the xarevtr1 or xarevtr2 file under the $DCDIR/spool/dcxarinf/trace/ directory. These files are called the XAR event trace information files. You cannot change the output destinations and files names of XAR event trace information files.

If there is an existing XAR event trace information file when you start the XA resource service, a backup file is created for the existing file and another, unused XAR event trace information file is prepared as the output destination. For example, if the xarevtr1 XAR event trace information file exists, the file is renamed to xarevtr1.bk1. Because this renaming causes the file name xarevtr1 to become available, a new XAR event trace information file is then created with the name of xarevtr1. Up to three generations of backup files are created. Backup files are created only when there is already an XAR event trace information file at the start of the XA resource service. During online processing, if the number of records written to one file exceeds the value of the maximum number of records that can be output to an XAR event trace information file, then no backup file is created. This maximum number is specified in the xar_eventtrace_record operand in the XA resource service definition. If one output file becomes full, the next file is used. If all the files have become full, the oldest one is overwritten with new records. Do not delete the XAR event trace information files while the system is online.

You can specify the output level of the XAR event trace information by using the xar_eventtrace_level operand in the XA resource service definition. By default, the output level is specified to acquire the XAR event trace only if an error occurs. You can acquire all the XAR event trace information if you want to. However, in that case, the online performance is affected. We recommend that you use the default output level except for debugging.

You can use the xarevtr command to edit and display the XAR event trace information acquired in the output file.

For details on how to specify the output level of the XAR event trace information and the number of output records, see the manual OpenTP1 System Definition. For details on how to edit and display the XAR event trace information, see the manual OpenTP1 Operation.

(10) TRN event trace

The TRN event trace is historical information about the XA functions issued in transaction branches, and about the events of transaction services (transaction management service, transaction recovery service, and resource manager monitoring service).

The TRN event trace is stored as files named _tr_nnn (nnn: 3-digit number starting at 001) in the $DCDIR/spool/dctrninf/trace/prf directory. These files are called TRN event trace information files. You cannot rename or move these files.

You can change the size of each TRN event trace information file and the maximum number of TRN event trace information files that can be stored. For details, see the TRN event trace definition in the manual OpenTP1 System Definition.

To collect the TRN event trace information:

  1. Set Y in the prf_trace operand in the system common definition.
  2. Use the trn_prf_event_trace_level operand in the transaction service definition to specify the level for collecting the TRN event trace information.
  3. Use the trn_prf_event_trace_condition operand in the transaction service definition to specify the type of TRN event trace collection.

The following table lists the values that can be specified in the trn_prf_event_trace_condition operand and the types of TRN event trace collection specified by the values.

The table below shows the TRN event trace information that is collected depending on specification of the trn_prf_event_trace_condition operand.

Table 5-11 Relationship between the trn_prf_event_trace_condition operand value and the TRN event trace information that is collected

Value of trn_prf_event_trace_conditionTRN event trace information that is collectedEvent IDTrace data size (bytes)Collection timing
xafuncTrace information about XA functions0x4500320 (192 for the xa_open and xa_close functions)During transaction processing#
trnserviceTrace information about the operating status of the transaction service0x4501192When the transaction service starts, ends, or is recovered
#
For a transaction committed in two phases, the trace amount collected per transaction branch is 12 [Figure] number of resource managers. However, the trace amount varies depending on the conditions, such as the XA interface object files linked to the user server and the transaction optimization settings.

For details about this operand, see the manual OpenTP1 System Definition.

To collect TRN event trace information files and output edited trace information, use the prfget and prfed commands. The acquisition and edit/output procedures are as follows.

Collecting TRN event trace information
To collect only the trace information that has not been acquired with the latest run ID, execute the following commands:

$DCDIR/bin/prfget -f _tr | $DCDIR/bin/prfed -d

To collect all the trace information, execute the following commands:

$DCDIR/bin/prfget -a -f _tr | $DCDIR/bin/prfed -d

Outputting TRN event trace information
To output TRN event trace information, execute the prfed command with the -d option specified. Specify other options as required.
TRN event trace information is output in the same format as the performance verification trace. For details about the output format, see the description of the prfed command in the manual OpenTP1 Operation.
The following shows an example of TRN event trace information output when xafunc is specified in the trn_prf_event_trace_condition operand.

PRF: Rec Node: trn1 Run-ID: 0x4046b806 Process: 26264      Trace: 10
Event: 0x4500 Time: 2004/01/01 12:34:56 678.123.000 Server-name: Sup
Rc:   0      Client: **** - ********** Server: **** Root: trn1 - ********
Svc-Grp: *************************** Svc: ****************************
Trn: 78d0trn100000001trn1trn100000001
   xa_commit                       (IN)  OpenTP1_TAM
   axid:010000000000000c00000010
        3738643074726e330000000174726e3374726e330000000100000000
 Internal code1:0x2007       Internal code2:   0
 Internal code3:0

(11) NAM event trace

OpenTP1 collects trace information about the various events generated by the name service, including communication invoked by the name service and the registration and deletion of service information from the cache. This information is known as an NAM event trace.

NAM event traces are stored as files with the file name _nm_001, _nm_002, _nm_003, and so on, under the $DCDIR/spool/dcnaminf/ directory. These files are known as NAM event trace information files. The output directory and file names cannot be changed.

To collect NAM event trace information:

  1. Set Y in the prf_trace operand in the system common definition.
  2. Specify the level of NAM event trace information to acquire in the nam_prf_trace_level operand in the system common definition.

The following table lists the values that can be specified in the nam_prf_trace_level operand and the types of NAM event trace information acquired for each level. For the collection timing for each event, see the description of collecting performance verification trace information in the manual OpenTP1 Operation.

Table 5-12 Relationship between the value specified in the nam_prf_trace_level operand and acquired NAM event trace information

Value of nam_prf_trace_level operandEvent ID of trace information
0xf000 to 0xf0290xf100 to 0xf1090xf200 to 0xf218
00000000​NNN
00000001​NYN
00000002​YNN
00000003​YYN
00000004​NNY
00000005​NYY
00000006​YNY
00000007​YYY
Other valueYYN
Legend:
Y: Trace information is acquired.
N: Trace information is not acquired.

To collect NAM event trace information files and output edited trace information, use the prfget and prfed commands. The acquisition and edit/output procedures are as follows.

To collect NAM event trace information, execute the following commands:

$DCDIR/bin/prfget -a -f _nm | $DCDIR/bin/prfed -d

(12) Process service event trace

OpenTP1 collects information about process events such as process creation, destruction, activation, and termination. This information is known as a process service event trace.

Process service event traces are stored as files with the file name _pr_001, _pr_002, _pr_003, and so on, under the $DCDIR/spool/dcprcinf/ directory. These files are known as process service event trace information files. The output directory and file names cannot be changed.

To collect process service event traces, set Y in the prc_prf_trace operand in the process service definition.

For the collection timing for each event, see the description of collecting performance verification trace information in the manual OpenTP1 Operation.

To collect process service event trace information files and output edited trace information, use the prfget and prfed commands. The acquisition and edit/output procedures are as follows.

To collect process service event trace information, execute the following commands:

$DCDIR/bin/prfget -a -f _pr | $DCDIR/bin/prfed -d

(13) FIL event trace

OpenTP1 collects event information when the amount of time it takes to process an OpenTP1 file access request issued internally from a process under OpenTP1 control exceeds the value specified in the fil_prf_trace_delay_time operand in the system common definition. This is called a FIL event trace.

By analyzing the FIL event trace, you can check the delay status of processing related to OpenTP1 file access.

FIL event traces are stored as files with the file name _fl_001, _fl_002, _fl_003, etc., under the $DCDIR/spool/dcfilinf/ directory. These files are called FIL event trace information files. The output destination and names of the FIL event trace information files cannot be changed.

You use the prfget command to collect FIL event trace information files and the prfed command to edit and output the files. The following describes how to collect the files.

How to collect FIL event trace information files:

$DCDIR/bin/prfget -a -f _fl | $DCDIR/bin/prfed -d

(14) Command logs

Every time an OpenTP1 operation command is executed, information such as the command start time and command termination time is output to cmdlog1 and cmdlog2 in $DCDIR/spool/cmdlog.

You can open cmdlog1 and cmdlog2 with a text editor such as vi. From the logged command start time and command termination time, you can obtain, for example, the time required to execute the command (response time).