7.7.17 Troubleshooting when the command adapter is executed

For the command adapter, information that can be used for verification of the system performance and analysis of failure causes is output to log files and trace files. The following table shows the types of log and trace data that is output from the command adapter.

Table 7‒153: Types of log and trace data (for the command adapter)
Log/trace	Output information	Description
Message log	Information about events that occur in the command adapter is output in the form of messages to this log.	You can view messages that provide information about system activities (such as start, stop, and failure). Because this log is shared by the application server and all components of service platform, you can centrally check the system activity information.
Maintenance log	The following types of information are output as internal maintenance information for the command adapter: Date Time Execution class name Internal method name Exception that occurred Information about the specified arguments	You can check the issuance times of internal methods to determine in what sequence the methods were issued.
Performance analysis trace	The following types of information are output at the collection points of performance analysis traces in the entire service platform system: Date Time Request information Information about the trace collection source At collection points of performance analysis traces for the command adapter, information specific to the command adapter is output.	You can verify the performance of the service platform system based on the performance analysis information that is output for a sequence of processes from reception of a request by the service requester to a return of the execution result.
Exception log	Information about exceptions that occurred in the command adapter (stack trace) is output to this log.	You can view the information about exceptions that occurred in the command adapter. You can use this information for failure cause analysis.

Organization of this subsection

(1) Snapshot log
(2) Message log
(3) Maintenance log
(4) Performance analysis trace (PRF trace)
(5) Exception log
(6) Standard error output log for external command processes
(7) Action taken if an exception occurs after an external command process is started
(8) Releasing threads
(9) Releasing file descriptors

(1) Snapshot log

To include the command adapter in the targets of information collection by the mngsvrutil and snapshotlog commands, you might need to modify the file that defines the targets for collection of snapshot log data.

To collect log data at the same logging level as the service adapters of the service platform, add entries as shown in the following sections. You do not need to add these entries if they have already been included in the relevant files.

(a) snapshotlog.conf

${cosminexus.home}/CC/server/public/ejb/.+/logs/CSCADP/ADPCMD/maintenance/.+/cscadpcmdexp_.*\.log
&{ejb.public.directory}/ejb/.+/logs/CSCADP/ADPCMD/maintenance/.+/cscadpcmdexp_.*\.log
&{ejb.server.log.directory}/CSCADP/ADPCMD/maintenance/.+/cscadpcmdexp_.*\.log
${cosminexus.home}/CSC/custom-adapter/Command/config/.+
${cosminexus.home}/CSC/custom-adapter/Command/config/.+/.+

(b) snapshotlog2.conf

${cosminexus.home}/CC/server/public/ejb/.+/logs/CSCADP/ADPCMD/maintenance/.+/cscadpcmdmnt_.*\.log
&{ejb.public.directory}/ejb/.+/logs/CSCADP/ADPCMD/maintenance/.+/cscadpcmdmnt_.*\.log
&{ejb.server.log.directory}/CSCADP/ADPCMD/maintenance/.+/cscadpcmdmnt_.*\.log

(c) snapshotlog.param.conf

${cosminexus.home}/CSC/custom-adapter/Command/config/.+
${cosminexus.home}/CSC/custom-adapter/Command/config/.+/.+

To Page Top

(2) Message log

The information about the events that occurred in the command adapter is output in the form of messages to the J2EE server activity log, which is a message log.

For details about the output items and output destination of message log data, see 7.4.1 Message log.

(a) Output level settings

You can set the output level for the messages that are output by the command adapter so that specific types of messages are output to the message log. The following table shows what types of information are to be output according to the output level that is set.

Table 7‒154: Output levels and types of messages that will be output
Information type	Output level
Information type	0	10	20
Error messages Related to occurrences of exceptions and failures	A	A	A
Warning messages Prompting the user to make confirmation during system operation	B	A	A
Information messages Reporting that the status of the command adapter has changed (started or stopped) Reporting events (start and stop of external commands)	B	B	A

Legend:

A: All messages are output.

B: Only important messages are output.

You can set the output level in the command-adapter runtime-environment properties file. The following table shows the property key that is used to set the output level.

Table 7‒155: Output level settings for the message log
Item	Key	Default	Specifiable values
Message log output level	adpcmd.log.messagelog.level	10	0 10 20

(b) Output destination

The output destination of message log data is as follows:

Output-destination path of the J2EE server activity log

You can specify the output destination of J2EE server activity log data by using the ejb.server.log.directory key in the option definition file (usrconf.cfg) for the J2EE server on which the HCSC server operates. For details, see the following sections in the manual Application Server Definition Reference Guide: 2.2.2 usrconf.cfg (Option definition file for J2EE servers) and 2.2.3 usrconf.properties (User property file for J2EE servers).

The following table shows the log file names of message log data.

Table 7‒156: Log file names of message log data
Log file output mode	Log file name
Wraparound mode	`cjmessageN.log`^#
Shift mode	`cjmessage.log`^#

#: The log file is shared by the application server and all components of the service platform. The messages providing operation and failure information about all products on the HCSC server are logged.

(c) Number of files

You can specify the number of output files by using the ejbserver.logger.channels.define.channel-name.filenum key in the user property file (usrconf.properties) for the J2EE server on which the HCSC server operates. For details, see the following section: 2.2.3 usrconf.properties (User property file for J2EE servers) in the manual Application Server Definition Reference Guide or 11.1.1 usrconf.properties (User property file for J2EE servers) in the manual uCosminexus Application Server Compatibility Guide.

(d) File size

You can specify the size of each output file by using the ejbserver.logger.channels.define.channel-name.filesize key in the user property file (usrconf.properties) for the J2EE server on which the HCSC server operates. For details, see the following section: 2.2.3 usrconf.properties (User property file for J2EE servers) in the manual Application Server Definition Reference Guide or 11.1.1 usrconf.properties (User property file for J2EE servers) in the manual uCosminexus Application Server Compatibility Guide.

To Page Top

(3) Maintenance log

The maintenance log can be used to check the times at which internal methods of the command adapter were issued and to determine the sequence in which the methods were issued. Maintenance service personnel use this log for failure analysis.

(a) Output level settings

You can set the level of log data to be output to the maintenance log. The following table shows what types of information are to be logged according to the output level that is set.

Table 7‒157: Output levels and types of messages that will be output to the maintenance log
Information type	Output level
Information type	1	2	3	4	5
Error information and causes of exception occurrences	Y	Y	Y	Y	Y
Interface information for major external modules Information related to the custom adapter framework	N	Y	Y	Y	Y
Interface information for external modules Information related to external command processes	N	N	Y	Y	Y
Internal interface information major components	N	N	N	Y	Y
Debug information	N	N	N	N	Y

Legend:

Y: This type of information is output.

N: This type of information is not output.

You can set the output level in the command-adapter runtime-environment properties file. The following table shows the relevant property key.

Table 7‒158: Output level settings for the maintenance log
Item	Key	Default	Specifiable values
Output level for the maintenance log	adpcmd.log.methodtrace.level	3	1 2 3 4 5

(b) Output format

The following figure shows the format of data that is output to the maintenance log.

Figure 7‒127: Format of data output to the maintenance log (for the command adapter)

(c) Output items

The following table shows the items of data that is output to the maintenance log.

Table 7‒159: Items of data output to the maintenance log (for the command adapter)
Item	Contents
Number	The sequence number of each maintenance log record is output.
Date	The date on which the maintenance log record was collected is output in `YYYY/MM/DD` format: `YYYY`: Four-digit year `MM`: Month number `DD`: Day
Time	The time at which the maintenance log record was collected is output in `hh:mm:ss.SSS` format: `hh`: Hour `mm`: Minute `ss`: Second `SSS`: Millisecond
Product ID	The string `ADPCMD`, which indicates the command adapter, is output as a product identifier together with version information. The format of the version information is as follows: Regular version: `VV-RR` Fixed version: `VV-RR-SS`
pid	The identifier of each process is output.
tid	The identifier of each thread is output.
ID and type	These items are not output.
Output type	The information about the point (position) from which the maintenance log record was collected is output: `BGN`: Beginning of a method `END`: End of a method `CAL`: Calling of a method `RET`: Return of a method `THR`: Throwing of an exception `CTH`: Catching of an exception
Class name	The name of the class that contains the method for which maintenance log data was collected is output.
Method name	The name of the method for which maintenance log data was collected is output.
I/O information	The I/O information or stack trace for the method for which maintenance log data was collected is output.
CRLF	The record end code is output.

(d) Output destination

The output destination of maintenance log data is as follows:

log-data-output-directory-on-the-J2EE-server/CSCADP/ADPCMD/maintenance/service-ID

The following table shows the log file name of the maintenance log.

Table 7‒160: Log file name of the maintenance log (for the command adapter)
Log file output mode	Log file name
Wraparound mode	`cscadpcmdmnt_HCSC-server-name_N.log`
Shift mode	`cscadpcmdmnt_HCSC-server-name_.log`

You can specify the number of log files and the log file size by using properties in the command-adapter runtime-environment properties file or the command-adapter runtime-environment common properties file. To specify the number of log files, use the adpcmd.log.methodtrace.filenum property. To specify the log file size, use the adpcmd.log.methodtrace.filesize property.

For details about the command-adapter runtime-environment properties file, see 6.15.3 Command-adapter runtime-environment properties file in the manual Service Platform Reference Guide. For details about the command-adapter runtime-environment common properties file, see 6.15.4 Command-adapter runtime-environment common properties file in the manual Service Platform Reference Guide.

To Page Top

(4) Performance analysis trace (PRF trace)

Performance analysis information is output for a processing sequence from when a request is received from the service requester to when the execution result is returned. This information can be used to verify the performance of the application server and service platform system. From the collection points of performance analysis traces for the command adapter, information specific to the command adapter is output. Maintenance service personnel use this information for analysis of performance bottlenecks.

(a) Output format

Performance analysis traces are output to files that are in the same format as the performance analysis trace files for the J2EE server. For details, see 7.3 Collecting the trace based performance analysis file by using Management Server in the Application Server Maintenance and Migration Guide.

(b) Output destination

The following table shows the output destination and trace file name for performance analysis traces.

Table 7‒161: Output destination and trace file name for performance analysis traces
Trace output destination path	Trace file name
`directory-set-for-the-PRFSPOOL-environment-variable\utt\prf\PRF-identifier\dcopltrc`^#	prf_<n>

Legend:: n: Integer beginning with 001, no more than the number of files
#: The output destination of performance analysis traces is shared by the application server and all components of the service platform. For details, see 7.3 Collecting the trace based performance analysis file by using Management Server in the manual Application Server Maintenance and Migration Guide.

(c) Number of files

The number of output files is set to four by default. You can change the number of output files via the management portal or by using the cprf command. For details about the management portal, see the manual Application Server Management Portal User Guide. For details about the cprf command, see the Application Server Command Reference Guide.

(d) File size

For the management portal, the default file size is 32 MB. For the cprf command, the default file size is 8 MB. You can change the file size via the management portal or by using the cprf command. For details about the management portal, see the manual Application Server Management Portal User Guide. For details about the cprf command, see the manual Application Server Command Reference Guide.

(e) Collection points of performance analysis traces

The following figure shows the collection points of performance analysis traces.

Figure 7‒128: Collection points of performance analysis traces (for the command adapter)

The following table lists event IDs, trace collection points, and collection levels for performance analysis traces. The numbers in the Number in figure column correspond to the circled numbers in the preceding figure.

Table 7‒162: Collection points of performance analysis traces (for the command adapter)
Event ID	Number in figure	Trace collection point	Level^#1
0xab06	1	At the time of receiving a request	Standard
0xab08	2	At the time of starting an external command	Standard
0xab09	3	At the time of completing an external command process^#2	Standard
0xab07	4	At the time of sending a response	Standard

#1: For details about how to change the collection level for performance analysis traces, see Changing trace collection level, the manual Application Server Command Reference Guide, or the manual Application Server Maintenance and Migration Guide. The default collection level is Standard.
#2: This trace collection point also applies even if a timeout occurs during the processing of an external command.

To Page Top

(5) Exception log

The logging of exception log data starts and ends at the same timings as the logging of maintenance log data. Information about exceptions (stack trace) is collected and output to files. The files of the exception log and maintenance log for the command adapter are saved in the same directory. Different directories cannot be allocated to these logs.

(a) Output format

The following shows the format of data that is output to the exception log.

Figure 7‒129: Output format of exception log data (for the command adapter)

(b) Output items

The following table shows the items that are output in exception log data:

Item	Contents
Number	The sequence number of each exception log record is output.
Date	The date on which the exception log record was collected is output in `YYYY/MM/DD` format: `YYYY`: Four-digit year `MM`: Month number `DD`: Day
Time	The time at which the exception log record was collected is output in `hh:mm:ss.SSS` format: `hh`: Hour `mm`: Minute `ss`: Second `SSS`: Millisecond
Product ID	The string `ADPCMD`, which indicates the command adapter, is output as a product identifier.
pid	The identifier of each process is output.
tid	The identifier of each thread is output.
ID	Blank
Stack trace information	Stack trace information is output.
CRLF	The record end code is output.

(c) Output destination

The output destination of exception log data is as follows:

log-data-output-directory-on-the-J2EE-server/CSCADP/ADPCMD/maintenance/service-ID

The following table shows the exception log file name.

Table 7‒163: Exception log file name (for the command adapter)
Log file output mode	Log file name
Wraparound mode	`cscadpcmdexp_HCSC-server-name_N.log`
Shift mode	`cscadpcmdexp_HCSC-server-name_.log`

You can specify the number of log files and the log file size by using properties in the command-adapter runtime-environment properties file or the command-adapter runtime-environment common properties file. To specify the number of log files, use the adpcmd.log.exptrace.filenum property. To specify the log file size, use the adpcmd.log.exptrace.filesize property.

To Page Top

(6) Standard error output log for external command processes

The information that is sent to the standard error output for external command processes can be logged in files.

(a) Settings of whether to enable logging

You can enable or disable the standard error output logging function for each command adapter. The following table shows the relevant property key in the command-adapter runtime-environment properties file.

Table 7‒164: Settings of whether to enable logging of the standard error output
Item	Key	Default	Specifiable values
Whether to enable the logging of the standard error output	adpcmd.log.stderr.enable	true	true false

To enable the logging of the standard error output, set true.

(b) Character encoding of the standard error output

The character encoding that is used when the content of the standard error output is read can be set. The following table shows the relevant property key in the command-adapter runtime-environment properties file.

Table 7‒165: Character encoding settings for the standard error output
Item	Key	Default	Specifiable values
Character encoding of the standard error output	adpcmd.log.stderr.charset	UTF-8	UTF-8 MS932 EUC-JP

(c) Output format

The following figure shows the format of data that is output to the standard error output log.

Figure 7‒130: Format of data output to the standard error output log

(d) Output items

The following table shows the items of data that is output to the standard error output log.

Table 7‒166: Items of data output to the standard error output log (for the command adapter)
Item	Contents
Number	The sequence number of each standard error output log record is output.
Date	The date on which the standard error output log record was collected is output in `YYYY/MM/DD` format: `YYYY`: Four-digit year `MM`: Month number `DD`: Day
Time	The time at which the standard error output log record was collected is output in `hh:mm:ss.SSS` format: `hh`: Hour `mm`: Minute `ss`: Second `SSS`: Millisecond
Product ID	The string `ADPCMD`, which indicates the command adapter, is output as a product identifier together with version information. The format of the version information is as follows: Regular version: `VV-RR` Fixed version: `VV-RR-SS`
pid	The identifier of each process is output.
tid	The identifier^# of each thread is output.
ID	For a log record for the standard error output: Blank Immediately after starting an external command: KDEC81622-I message
Standard error output	For a log record for the standard error output: Each time an LF code (0x0a) is encountered, the bytes before the LF code are converted to characters, which are then output. However, a CR code (0x0d) that appears immediately before an LF code is not output. Regardless of whether an LF code is encountered, each time consecutive 1,024 bytes are read, the read bytes are converted to characters, which are then output. When an external command process ends, the data that has not been output yet is converted to characters and the conversion results are output. For a message that is output immediately after starting an external command: The message and parameters are output.
CRLF	The record end code is output.

#: This thread ID is specific to this log and is not shared by the other logs.

(e) Output destination

The log data output destination of the standard error output log for external command processes is as follows:

log-data-output-directory-on-the-J2EE-server/CSCADP/ADPCMD/maintenance/service-ID

The following table shows the log file name of the standard error output log.

Table 7‒167: Log file name of the standard error output log (for the command adapter)
Log file output mode	Log file name
Wraparound mode	`cscadpcmdstderr_HCSC-server-name_N.log`
Shift mode	`cscadpcmdstderr_HCSC-server-name_.log`

You can specify the number of log files and the log file size by using properties in the command-adapter runtime-environment properties file or the command-adapter runtime-environment common properties file. To specify the number of log files, use the adpcmd.log.exptrace.filenum property. To specify the log file size, use the adpcmd.log.exptrace.filesize property.

To Page Top

(7) Action taken if an exception occurs after an external command process is started

If a Java exception occurs internally while the system is waiting for a started external command to time out, the system attempts to forcibly terminate the process of the external command.

When forced termination is successful, a warning message (KDEC81559-W) is output to the message log.

If forced termination fails, an error message (KDEC81560-E) is output to the message log. In this case, the system operator must check the status of the external command process.

To Page Top

(8) Releasing threads

The command adapter generates a maximum of three threads for one request.

The following table shows the threads that are generated and the number of generated threads.

Table 7‒168: Threads that are generated and the number of generated threads
Purpose of the thread		Number of generated threads
Thread required during execution of an external command		2
Thread for forced termination^#1	When a process times out^#2	1
Thread for forced termination^#1	When an exception occurs after starting an external command process	1

#1: A maximum of one thread is generated per request.
#2: A thread is generated only if the function that forcibly terminates an external command process that timed out is enabled.

Under some conditions, threads remain after the command adapter responds. Therefore, such remaining threads must be appropriately handled to prevent exhaustion of J2EE server resources.

The following table shows messages that can be output and action to be taken.

Table 7‒169: Messages that can be output and action to be taken for the remaining threads
Message and condition	Number of threads that remain	Action
The KDEC81553-E message is output. Also, the function that forcibly terminates an external command process that timed out is disabled.	2	If necessary, stop the process that was started.
The KDEC81561-E message is output.	2	It is prohibited to start a subprocess from a command process in the background. Review and correct the processing of the commands that are specified from the command adapter. Externally terminate the subprocess that is being executed in the background.
The KDEC81557-E message is output.	1	Check the status of the process that was started. If necessary, stop it.
The KDEC81560-E message is output.	1	Check the status of the process that was started. If necessary, stop it.

Important note: It is prohibited to start a subprocess in the background from an external command process specified in the command adapter. If a subprocess is started in the background, threads might remain regardless of whether a message is output.

To Page Top

(9) Releasing file descriptors

The command adapter consumes a maximum of three file descriptors for one request. Even after the command adapter responds, these file descriptors are not released until the started external command process ends. Stop the process as needed.

For details about how to estimate the number of file descriptors, see I.1 Estimating the required number of file descriptors.

To Page Top