7.7.10 Troubleshootingwhen executing Message Queue

In the Message Queue adapter, the information to validate system performance and analyze the cause of failure is output to the log file or the trace file. The following table lists and describes the type of log and trace output to Message Queue:

Table 7‒82: Type of log and trace (For Message Queue adapter)
Log/trace	Output information	Description
Message log	Each type of information in the Message Queue adapter is output as a message.	You can check operation information (start/stop/ error) as a message. You can check operation status of the system in a batch since the output destination of the message log is common for entire Application Server.
Method trace	The following information is output when you start or end an internal method of the Message Queue adapter: Date Time Execution class name Internal method name Information of specified argument	You can check the publication time and sequence of internal method.
Data trace	The following information, sending and receiving messages between the Message Queue adapter and the resource adapter, Detail of fault response information is output: Date Time I/O type of message Contents of message	You can confirm the validity of message by acquiring contents of the message and information of the argument.
Performance analysis trace	The following information is output to the performance analysis trace collection point of the entire Application server: Date Time Request information Trace collection source information The Message Queue adapter specific information is output in the performance analysis trace collection point of the Message Queue adapter.	You can validate the performance of Application Server system on the basis of the performance analysis information output in the series of processes starting from when you receive a request from the service requester till the execution result is returned.
Exception log	Information of an exception which occurred in the method provided by Message Queue is output: Date Time Information of the exception (Stack race)	You can confirm the time when the exception occurred and the method

For details on points to be considered when a log or a trace is output, see "7.3.1(3) Notes on acquisition of log and trace data: General notes".

In the troubleshooting functionality of the Message Queue adapter, a message is output to the console. When failure occurs, investigate the cause on the basis of the message information. When the cause of failure is not clear, collect the material regarding the definition of the Message Queue adapter and the message log, and contact the administrator.

You can acquire the log and the trace of Message Queue using the troubleshooting functionality of Application Server. The methods for acquiring the log and the trace are as follows:

Using the management command of Management Server (mngsvrutil)

For details, see "Application Server Command Reference Guide ".
Using the management portal

For details, see "Application Server Management Portal User Guide ".
Using the snapshotlog command

For details, see "Application Server Command Reference Guide ".

Organization of this subsection

(1) Message log
(2) Method trace
(3) Data trace
(4) Performance analysis trace (PRF trace)
(5) Exception log
(6) Failure information of the Message Queue adapter
(7) Points to be considered before setting Message Queue adapter
(8) Actions to be taken when error occurs

(1) Message log

Each type of information in the Message Queue adapter is output as a message in the activity log of the J2EE server of the message log.

For details on the output contents or the output destination of the message log, see "7.4.1 Message log".

To Page Top

(2) Method trace

You can check the publication time and the sequence of the internal method in method trace. Management service uses the method trace to analyze failure.

(a) Output destination

Output destination of the method trace of Message Queue is as follows:

<Log output directory of J2EE server>\CSCADP\MQADP\maintenance\<Service ID>

The log data output directory on the J2EE server can be specified by using the ejb.server.log.directory key in the option definition file (usrconf.cfg) for the J2EE server on which the HCSC server operates. For details, see "2.2.2 usrconf.cfg (option definition file for J2EE servers)" in "Application Server Definition Reference Guide".

The following table lists the trace file name of method trace:

Table 7‒83: Trace file name of method trace (Message Queue adapter)
Output mode of trace file	Trace file name
For wraparound mode	mtdtrace<Count>.log
For shift mode	mtdtrace.log

(b) Number of files

The default number of files is 4. You can change the number of files with the methodtracefile.num key of the Message Queue adapter execution environment.

For details on the Message Queue adapter runtime-environment property file, see "6.12.1 Message Queue adapter runtime-environment property file" in "Service Platform Reference Guide". For details on how to change, see the description for creating the Message Queue adapter runtime-environment property file in "3.3.9 Defining Message Queue adapters " in "Service Platform Reception and Adapter Definition Guide".

(c) File size

The default size of the output file is 4MB. You can change the file size using the methodtracefile.size key of the Message Queue adapter runtime-environment property file.

For details on the Message Queue adapter runtime-environment property file, see "Message Queue adapter runtime-environment property file " in "6.12.1 Service Platform Reference Guide". For details on how to change, see the description for creating the Message Queue adapter runtime-environment property file in "3.3.9 Defining Message Queue adapters " in "Service Platform Reception and Adapter Definition Guide".

(d) Output format

The following figure shows the output format of the method trace:

Figure 7‒89: Output format of method trace

(e) Output contents

The following table list and describes contents output in method trace:

Table 7‒84: Items output to the method trace (Message Queue adapter)
Item	Contents
Number	The output serial number of the method trace is displayed.
Date	The collection date of the method trace is output in the yyyy/mm/dd format. yyyy: Christian year mm: Month dd: Day
Time	The collection time of the method trace is output in hh:mm:ss.SSS format. hh: hour mm: minute ss: second SSS: milliseconds Time is shown in milliseconds in local time.
Product ID	"ADPMQ" indicating the Message Queue is displayed as the identifier used for identifying the product. Format of version information is as follows: Regular edition product: VV-RR Modified version product: VV-RR-SS
pid	The ID to identify the process is output.
tid	The ID to identify the thread is output.
ID	Blank
Type	Type of trace collection point is output BGN: Start Method END: End Method CAL: Invoke method RET: Return method THR: Throw exception CTH: Catch exception
Class name	The class name to acquire trace is output. Perform left alignment when there are 30 or less characters, and replace with space in case of shortage of characters.
Method name	The method name to acquire trace is output. Perform left alignment when there are 25 or less characters, and replace with space in case of shortage of characters.
RootAP information	Character string wherein the information of RootAP such as IP address, process ID, communication number, which is output in the performance analysis trace is demarcated with a forward slash (`/`) is output.
I/O information	I/O information of method that acquires trace (Argument value or return value) is output.
CRLF	Record end code is output.

To Page Top

(3) Data trace

In data trace, you can check the message contents or I/O time of messages sent and received by the Message Queue adapter. The user recovers user data with the data trace, and Management Service analyzes failures with the data trace to analyze failures.

(a) Output destination

User messages sent and received in data trace are output. Therefore, specify an output destination for which security is ensured in the data trace.

The output destination of data trace is as follows:

<Log output directory of J2EE server>\CSCADP\MQADP\maintenance\<Service ID>

The log data output directory on the J2EE server can be specified by using the ejb.server.log.directory key in the option definition file (usrconf.cfg) for the J2EE server on which the HCSC server operates. For details, see "Application Server Maintenance and Migration Guide" , "Application Server Definition Reference Guide".

The trace file name of data trace is as follows:

Table 7‒85: Trace file name of data trace
Output mode of trace file	Trace file name
For wraparound mode	dattrace<number of files>.log
For shift mode	dattrace.log

(b) Number of files

The default number of files is 4. You can change number of files using the datatracefile.num key of the Message Queue adapter runtime-environment property file.

For details on the Message Queue adapter runtime-environment property file, see "6.12.1 Message Queue adapter runtime-environment property file" in "Service Platform Reference Guide". For details on how to change, see the description for creating the Message Queue adapter runtime-environment property file "3.3.9 Defining Message Queue adapters " in "Service Platform Reception and Adapter Definition Guide".

(c) File size

The default file size of the output file is 1MB. You can change the size of the output file with the datatracefile.size key of the Message Queue adapter runtime-environment property file.

For details on the Message Queue adapter runtime-environment property file, see "6.12.1 Message Queue adapter runtime-environment property file" in "Service Platform Reception and Adapter Definition Guide". For details on how to change, see the description for creating the Message Queue adapter runtime-environment property file in"3.3.9 Defining Message Queue adapters " in "Service Platform Reception and Adapter Definition Guide".

(d) Output format

The following figure shows the format output to the data trace:

Figure 7‒90: Format output to the data trace

(e) Output contents

The following table lists and describes the items output to the data trace:

Table 7‒86: Contents output to data trace (Message Queue adapter)
Item	Contents
Number	The output serial number of the data trace is displayed
Date	The collection date of data trace is output in the yyyy/mm/dd format. yyyy: Christian year mm: Month dd: Day
Time	The collection time of the data trace is output in hh:mm:ss.SSS format. hh: Hour mm: Minute ss: Second SSS: Millisecond Time is shown in milliseconds in local time.
Product ID	"ADPMQ" and the version information indicating the Message Queue adapter is displayed as the identifier used for identifying the product. Format of version information is as follows: Regular version product: VV-RR Modified version of product: VV-RR-SS
pid	The ID to identify the process is output.
tid	The ID to identify the thread is output.
ID	Blank
Type	I/O type of data is output: REQ: Request message RSP: Response message FLT: Fault message INF: JMS header information ERR: Request/response message that has failed in sending and receiving Space: Continuous data
Format	Format of data is output. XML: XML format BINARY: Binary format Perform left alignment when there are 6 or less characters, and replace with space in case of shortage of characters. For "INF" type, space for 6 characters is output.
RootAP information	Character string wherein the information of RootAP such as IP address, process ID, communication number, which is output in the performance analysis trace is demarcated with a forward slash (`/`) is output.
Message information	Sending and receiving of message information is output. For "XML" format XML text is output. For "BINARY" Hex dump format is output. For "FLT" type Contents set in each element of fault information are output in "Element name= setting value" format. Demarcate each element with ", (Comma)". For "INF" type JMS header information is output.
CRLF	Record end code is output.

(f) Collection point of data trace

In the data trace collection point of the Message Queue adapter, I/O message which is sent and received, and the Fault response information is output:

Figure 7‒91: Collection point of data trace (Normal end)

Figure 7‒92: Collection point of data trace (when sending Fault response)

Figure 7‒93: Collection point of data trace (Message sending fails)

Figure 7‒94: Collection point of data trace (Message transformation fails)

To Page Top

(4) Performance analysis trace (PRF trace)

You can validate the performance of Application Server system on the basis of the performance analysis information output in the series of processes starting from when you receive a request from the service requester till the execution result is returned. Message queue specific information is output in the performance analysis trace collection point of the Message Queue adapter. Maintenance service uses the information to analyze performance bottlenecks.

(a) Output destination

The following table describes the output destination and trace file name of performance analysis trace:

Table 7‒87: Output destination and trace file name of performance analysis trace
Path of output destination of trace	Trace file name
<Setting directory of environment variable PRFSPOOL>`\`utt`\`prf`\`<PRF identifier>`\`dcopltrc^#	prf_<n>

Legend:: n: Enter integer from 001 to maximum value of number of files.
Note#: The output destination of performance analysis trace is common for entire Application Server. For details, see "7.3 Collecting the trace based performance analysis file by using Management Server" in "Application Server Maintenance and Migration Guide".

(b) Number of files

The default number of output files is 4. You can change the number of output files with the management portal or the cprfstart command. For details on management portal, see "Application Server Management Portal User Guide ". For details on the cprfstart command, see "Application Server Command Reference Guide ".

(c) File size

The default size of performance analysis trace (PRF trace) file is as follows:

When file size is set in management portal: 32MB
When file size is set using cprfstart command: 8MB

You can change the file size using the management portal or the cprfstart command. For details on management portal, see "Application Server Management Portal User Guide", and for details on the cprfstart command, see "Application Server Command Reference Guide "

(d) Output format

The format which is output to the performance analysis trace is same as the performance analysis trace of Application Server. For details, see "7.3 Collecting the trace based performance analysis file by using Management Server" in "Application Server Maintenance and Migration Guide"

(e) Collection point of performance analysis trace

In the performance analysis trace collection point of the Message Queue adapter, the Message Queue adapter specific information is output.

The following figure shows the performance analysis trace collection point of the Message Queue adapter:

Figure 7‒95: Collection point of performance analysis trace

A point specific event ID is assigned in the collection point of the performance analysis trace. The Trace event ID output to the Message Queue adapter is 0xA200~0xA2FF.

The following table describes the event ID, trace collection point, and performance analysis trace collection point. The numbers in the Number in figure column of the table correspond to the numbers in the above figure.

Table 7‒88: Performance analysis trace collection point(Message Queue adapter)
Event ID	Number in figure	Trace collection point	Level
0xA200	1	At the time of request reception	B
0xA202	2	When sending message	A
0xA203	3	When message sending ends	A
0xA204	4	When receiving message	A
0xA205	5	When message receiving ends	A
0xA201	6	When sending response	B

Legend:

A: Indicates the "Standard" level.

B: Indicates the "Detailed" level.

When you change collection level of PRF trace, specify 2 values in hexadecimal (8 digits) format of 4 bytes in the -PrfChangeLevel option of the cprflevel command. For details on how to specify the cprflevel command, see "cprflevel (display or change the PRF trace collection level)" in "Application Server Command Reference Guide".

Index number (Index 1, Index 2) is assigned in its respective hexadecimal value.

For details on how to change performance analysis trace level, see "Application Server Maintenance and Migration Guide".The default value is "Standard".

To Page Top

(5) Exception log

The exception information is output to the exception log. For details on the overview of the exception log, see "7.4.7(1) Exception log".

(a) Output destination

Output destination of exception log is as follows:

<Log output directory of J2EE server>\CSCADP\MQADP\maintenance\<Service ID>

The log data output directory on the J2EE server can be specified by using the ejb.server.log.directory key in the option definition file (usrconf.cfg) for the J2EE server on which the HCSC server operates. The following table lists the log file name of the Exception log:

Table 7‒89: Log file name of the Exception log (Message Queue adapter)
Output mode of log file	Log file name
For wraparound mode	exception<Count>.log
For shift mode	exception.log

(b) Number of files

The default number of output files is 4. You can change number of files using the exptracefile.num key of Message Queue adapter runtime-environment property file.

For details on the Message Queue adapter runtime-environment property file, see "6.12.1 Message Queue adapter runtime-environment property file" in "Service Platform Reference Guide". For details on how to change, see the description for creating the Message Queue adapter runtime-environment property file in "3.3.9 Defining Message Queue" in "Service Platform Reception and Adapter Definition Guide".

(c) File size

The default size of the output file is 4MB. You can change the file size using the exptracefile.size key of the Message Queue adapter runtime-environment property file.

For details on the Message Queue adapter runtime-environment property file, see "6.12.1 Message Queue adapter runtime-environment property file " in "Service Platform Reference Guide". For details on how to change, see the description for creating the Message Queue adapter runtime-environment property file in "3.3.9 Defining Message Queue" in "Service Platform Reception and Adapter Definition Guide".

(d) Output format

The following figure shows the format that is output in the Exception log:

Figure 7‒96: Format output in the Exception log

(e) Output contents

The following table lists and describes the contents output in the exception log:

Table 7‒90: Items output in the exception log (Message Queue adapter)
Item	Contents
Number	Output serial number of exception log is displayed.
Date	The collection date of the FTP protocol trace is output in the yyyy/mm/dd format. yyyy: Christian year mm: Month dd: Day
Time	The collection time of the FTP protocol trace is output in hh:mm:ss.SSS format. hh: Hour mm: Minute ss: Second SSS: Millisecond Time is shown in milliseconds in local time.
Product ID	"ADPMQ" indicating the Message Queue adapter is output as the identifier used for identifying the product.
Pid	The ID to identify the process is output.
Tid	The ID to identify the thread is output.
ID	Blank
Stack trace information	Stack trace information is output.
CRLF	Record end code is output.

To Page Top

(6) Failure information of the Message Queue adapter

Message Queue adapter notifies failure using error information in the following cases:

When timeout occurs in the message reception reponse monitoring time for the request to receive messages
When a message does not exist in the reception queue for a request to browse and receive a message

The following table lists and describes each setting value of the error information:

Table 7‒91: Error information set by the Message Queue adapter
Element name	Form	Setting value	Description
faultCode	String	Server	Communication errors are errors at server side.
faultString	String	<Message text of message output by Message Queue adapter>	For details, see "2.14 Messages from KDEH20000 to KDEH30999" in "Service Platform Messages".
faultActor	String	<Service adapter name>	The name of the service adapter where error information is notified.
faultDetail	Document(XML)	Code:1 Message:"timeout"	When timeout occurs in monitoring time of received message or when received message does not exist.

To Page Top

(7) Points to be considered before setting Message Queue adapter

The following are the points to be considered before setting Message Queue adapter. Confirm that there is no problem when the system operates. If there is a problem then eliminate the cause of this problem.

Message Queue adapter executes the connection release process for each request from the business process but, actually the connection between the resource adapter and IBM MQ is not disabled using the connection pool functionality of the resource adapter for a fixed period of time (5 minutes by default).

Depends on specification in Connector property file (Not disconnected in default)
When connection release of a Message Queue adapter fails, status of connection changes to "Released", but in some cases release of the connection is not detected in JMS provider system of connection destination. Due to this, there is a possibility that connection status may become inconsistent, and the next communication process may fail. Therefore, once initialize the established status of the connection in the connection destination JMS provider system.
If failure occurs during Message Queue adapter operations and the stop process is not executed normally, in some cases the release of connection in the connection destination JMS provider system is not detected. Therefore, once initialize the established status of connection destination JMS provider system.
Specify the request message with a value of one or more bytes. Error occurs if you specify 0 in the length of the request message.
The user message in XML format which can be sent and received by the Message Queue adapter is as per the specification of XML Processor. For details, see support range of XML Processor in "XML Processor User's Guide".
Specify the transaction support level provided in the resource adapter as "LocalTransaction" or "XATransaction". However, you can specify "LocalTransaction" only when the status in business process is not persistent. You cannot specify "NoTransaction".

JTA user specification is affected since "LocalTransaction" and "XATransaction" specification is related to JTA (Java Transaction API). For example, when the monitoring time of the JTA message reception is less than the monitoring time (receive_timeout) of reception responseof the message of the Message Queue adapter communication-configuration definition file, timeout occurs in the monitoring time of JTA message reception.

To Page Top

(8) Actions to be taken when error occurs

This section explains actions to be taken when message sending, message receiving, and requests to browse, receive a message or send and receive a message fails.

When message sending, message receiving, and requests to browse, receive a message or send and receive a message fail, you can re-execute the business process or continue execution from a point at which any of these processes have failed. Select actions to be taken according to operating format.

Note that, when you re-execute a business process, take action after you return to a status prior to the failed request publication which includes status of transmission queue and reception queue, and the status of the partner system.

(a) Actions to be taken when message sending fails.

When a request to send message fails, check the message log and confirm whether message KDEH30111-E is output. As per result, cause and actions differ as shown in following table:

Validation result	Cause	Action
When KDEH30111-E is output	Message sending has failed	Reference the message output just before KDEH30111-E message, eliminate the cause of problem, and re-execute request for message sending. If you want check user data when message sending fails, see data trace.
When KDEH30111-E is not output	In some cases message sending is successful.	Check that message is stored in transmission queue.

(b) Actions to be taken when message reception or a request to browse and receive a message fails

When reception of message or a request to browse and receive a message fails, check the message log and confirm that KDEH30113-E message is output. As per result, cause and actions differ as shown in following table:

Validation result	Cause	Action
When KDEH30113-E is output	Message is removed from the reception queue	Reference the data trace and acquire the message. Confirm that the message received from the reception queue does not exist. For a request to browse and receive message, you need not check the reception queue.
When KDEH message is not output when KDEH30113-E message is not output and	Message is removed from the reception queue	Reference the data trace and acquire the message. Confirm that the message received from reception queue does not exist. For a request to browse and receive message you need not confirm reception queue. In this case, you need to set data trace level to 20.
When KDEH30113-E is not output.	Message receiving has failed	Confirm that KDEH30111-E is output. Reference the message output just before KDEH30111-E message, eliminate the cause of failure in message receiving, and re-execute the request to receive a message.
When KDEH30103-W is output	Message does not exist in reception queue.	Re-execute the request to receive a message according to operation status.

(c) Actions to be taken when send and receive of message fails

When send and receive of message fails, the causes and actions to be taken for failure in message sending and receiving are as follows:

When message sending fails

When a request to send message fails, check message log and confirm that KDEH30111-E message is output. As per the result, cause and actions differ as shown in following table:

Validity result	Cause	Action
When KDEH30111-E is output and the communication type is send	When message sending has failed	Reference the message that is output just before the KDEH30111-E message, eliminate the cause, and re-execute the request to send the message. If you want to check the user data when message sending fails, see the data trace.

When receiving of message fails

When the request to receive a message fails, check the message log and confirm that KDEH30113-E message is output. As per the result, cause and actions differ as shown in following table:

Validation Result	Cause	Action
When KDEH30113-E is output	Message sending is successful, and the message is removed from reception queue	Reference the data trace and acquire the receive message, and confirm that the message received from the reception queue does not exist.
When KDEH message is not output when KDEH30113-E is not output	Message sending is successful and message is removed from the reception queue	Reference the data trace and acquire the receive message, and confirm that the message received from reception queue does not exist. In this case, you must set data trace level as 20.
When KDEH30113-E is not output	When message sending is successful but message receiving has failed	KDEH30111-E is output, check that communication type is received. Eliminate the cause of failure in message receiving, execute a request to send and receive message, or execute a request to receive a message according to the operating format.
When KDEH30103-W is output	Message does not exist in reception queue. Message sending is successful.	Re-execute request to receive the message according to operation status.

To Page Top