7.3.3 Collecting failure information (for prerequisite products other than a service platform)

(a) Types of failure information and how to collect it

When a failure occurs and it is difficult to specify the cause of failure from the log or the trace output by the Service Platform, the following failure information output by the J2EE server is required:

• Log of J2EE servers, redirectors, and server management commands

• Log of Administration Agent, Management Agent, and Management Server

• DB Connector log

• Log of EJB client application

• User log

• Maintenance log

Use the following commands to acquire the failure information output by the J2EE server in a batch (can be acquired manually). When failure information is acquired in a batch, there may be information that partially overlaps with the log and the trace output by the Service Platform. For details on how to acquire the failure information output by the J2EEserver, see "Application Server Maintenance and Migration Guide".

• snapshotlog command

  For details on the snapshot log command, see "snapshotlog (collect snapshot logs)" in "Application Server Command Reference Guide".

• mngsvrutil command (specify collect snapshot in the sub command)

  For details on the mngsvrutil command, see "mngsvrutil (Management Server management command)" in "Application Server Command Reference Guide".

By executing the above commands, you acquire the information in the following directories:

• <Installation directory of Service Platform>\CC\server\public\ejb\*\logs\csc

• <Installation directory of Service Platform>\CC\server\public\ejb\*\logs\csc\maintenance

• <Installation directory of Service Platform>\CSC\config\manager

• <Installation directory of Service Platform>\CSC\config\msg

• <Installation directory of Service Platform>\CSC\log\manager

• <Installation directory of Service Platform>\CSC\repository

• <Installation directory of Service Platform>\CSC\spool

• <Installation directory of Service Platform>\CSC\system\msg

Specify the information to be acquired in the definition file for collecting a snapshot log. If required, change the definition file used for collecting a snapshot log, and then change the information to be acquired. For details on the definition file for collecting a snapshot log, see "10.2.1 Definition file for snapshot log collection" in "Application Server Definition Reference Guide".

(b) Preparation before start of operation

Define the type (Path) of the failure information to be acquired beforehand when you perform batch acquisition by using commands. Specify the primary delivery material that is sent immediately by mails and the secondary delivery material that is sent separately. For details, see "2.4 Types of Required Data" in "Application Server Maintenance and Migration Guide".

(c) Definition files required for failure investigation

The definitions set up in the J2EE server are required for investigating failure. The following table describes definition files other than the Service Platform:

Table 7‒5:  Definition file other than the Service Platform

Definition file	Description
Setup file of Component Container#1	All the files and directories under <Installation directory of Service Platform>\CC\server\usrconf\ejb\<J2EE server name>\directory
Setup file of Manager#1	All the files and directories under <Installation directory of Service Platform>\manager\config\directory
Property information of DB Connector set in dbcon-xadisplayname #2	Definition file set in "3.1.2(8) Setting up DB Connector(When Reliable Messaging is used)".
Property information of DB Connector set in dbcon-nodisplayname#2

(a) Messages output by Reliable Messaging

The messages output by Reliable Messaging are output to the following log. There are 2 types of logs; start and stop messages log and message log for the J2EE server. The following table lists and describes the messages output by Reliable Messaging:

Table 7‒6: Log that becomes the output destination of messages output by Reliable Messaging

Log type	Output contents	Directory and file name
Start and stop messages log	Information on start and stop, and value of the property specified by the user.	Directory <Log output directory of J2EE server>\connectors\ File name <Display name of Reliable Messaging><File number>.log
Message log for the J2EE server	Information of Reliable Messaging for starting, stopping, and during operations.	Directory <Log output directory of J2EE server > File name Output to the same file in which the Service Platform outputs the message log.

(b) Relationship between the HCSC server and the trace of Reliable Messaging

The relationship between the HCSC server and the trace of Reliable Messaging differs when you use the local queue and the transfer queue, and when you use the shared queue.

• When you use the local queue or the transfer queue

  There are two types of Reliable Messaging trace namely; method trace and line trace. The method trace outputs error information (cause of exception occurrence) and the entry information, exit information, and debug information of method. The line trace outputs the information of messages transmitted between queues. The following figure shows the trace collection point when using local queue or transfer queue:

  Figure 7‒2:  Trace collection point when using local queue or transfer queue

  The following table lists and describes the relationship between trace collection point and file names. "Number in figure" in this table corresponds to the numbers in "Figure 7-2 Trace collection point when using local queue or transfer queue".

  Table 7‒7: Relationship between the trace collection point and file names

  Number in figure	Collection point	Trace
  		Type	Output machine	Output destination	File name
  1	When sending a request from the service requester to transfer queue	Method trace	Service requester machine	<Log output directory of the J2EE server>\RM\maintenance	mtd_<Display name of Reliable Messaging of the service requester>_<numbers>.log
  2	When transferring from the transfer queue of the service requester to the ocal queue of HCSC server	Line trace			lin_<Display name of Reliable Messaging of the service requester>_<numbers>.log
  3	When accepting transfer from the transfer queue of the service requester to the local queue of the HCSC server, and when transferring from the transfer queue of the HCSC server to the local queue of the service component.	Line trace	Machine on which the HCSC server is running	<Log output directory of the J2EE server >\RM\maintenance	lin_<Display name of Reliable Messaging of the HCSC server>_<numbers>.log
  4	When HCSC server fetches messages from the local queue, and when sending requests from the service adapter to the transfer queue	Method trace			mtd_<Display name of Reliable Messaging of the HCSC server>_<numbers>.log
  5	When accepting transfer from the transfer queue at the HCSC server to the local queue of the service component	Line trace	Machine on which the service component is running	<Log output directory of the J2EE server >\RM\maintenance	lin_<Display name of Reliable Messaging of the service component >_<numbers>.log
  6	When the service component fetches messages from the local queue	Method trace			mtd_<Display name of Reliable Messaging of the service component >_<numbers>.log

• When you use the shared queue

  There are two types of Reliable Messaging trace namely, method trace and shared queue event trace. The method trace outputs the error information (cause for occurrence of exception) and entry information, exit information, and debug information of the method. The share queue event trace outputs the information of an event indicating that the messages are stored in share queue. The following figure shows the trace collection point when using the shared queue:

  Figure 7‒3:  Trace collection point when using the shared queue

  The following table lists and describes the relationship between the trace collection point and the file name. "Number in figure" in this table corresponds to the numbers in "Figure 7-3 Trace collection point when using the shared queue".

  Table 7‒8:  Relationship between the trace collection point and the file name

  Number in figure	Collection point	Trace
  		Type	Output machine	Output destination	File name
  1	When messages are saved from the service requester to the shared queue for reception in HCSC server, and when messages are saved from the service adapter of HCSC server to the shared queue for sending	Shared queue event trace	Machine on which the HCSC server is running	<Log output directory of the J2EE server >\RM\maintenance	shq_<Display name of Reliable Messaging of the HCSC server>_<numbers>.log
  2	When HCSC server fetches messages from the local queue, and when sending requests from service adapters to transfer queue	Method trace			mtd_<Display name of Reliable Messaging of the HCSC server >_<numbers>.log

(c) Preparation before start of operation

To change the type of information that is output by changing the output level of method trace or line trace, you must change Reliable Messaging settings before starting the HCSC server. Perform these changes by using the RMMethodTraceLevel and RMLineTraceLevel configuration properties of Reliable Messaging.

(d) Definition files required for failure investigation

The information set up for Reliable Messaging configuration properties is required for investigating failures. The following table describes the definition files of Reliable Messaging required for investigating failures. For details on definition files, see " 6. Configuration Property" in "Reliable Messaging".

Table 7‒9:  Definition files of Reliable Messaging required for investigating failures

Definition file	Description
Setup information of Reliable Messaging properties	Definition files setup in "3.1.2(9) Setting up Reliable Messaging".

(a) Relationship between the HCSC server and SOAP trace or Application log

This sub-section describes the relationship between the collection point and SOAP trace or file names of application logs in the HCSC server configuration. This relationship differs when invoking SOAP adapter from the reception of Web Services and when invoking SOAP adapter from a reception other than Web Services.

• When invoking SOAP adapter from synchronous standard reception (Web Services) or SOAP reception

  The SOAP trace or application log is output with the following names when you invoke the SOAP adapter from synchronous standard reception (Web Services) or from the SOAP reception. The same names are output when it passes through the business process on the way. The following figure shows the collection point of SOAP trace and application log:

  Figure 7‒4:  Collection point of SOAP trace or application logs (when you invoke the SOAP adapter from the reception of Web Services)

  The following table lists and describes the relationship between the trace collection point and file names. The numbers in the Number in figure column of the table correspond to the numbers in the above figure.

  Table 7‒10:  Relationship between trace collection points and file names (when you invoke the SOAP adapter from the reception of Web Services)

  Number in figure	Collection point	File name
  		Trace#1	Application log#1
  1#2	SOAP communication trace when requests (sending) are sent from the service requester, and when there is a response (receiving) from the service requester	When using SOAP Communication Infrastructure "<prefix>-j2ee-<J2EE server name>-<numbers>.log" <prefix>#3 is set in the client definition file of service requester side. When using JAX-WS engine "cjwmessage<numbers>.log"	When using SOAP Communication Infrastructure "<prefix>-j2ee-<J2EE server name>-aplog-<numbers>.log" <prefix>#3 is set in the client definition file of service requester side. When using JAX-WS engine "cjwtransport<numbers>.log"
  2	SOAP Communication trace when requests are received (receiving) from the service requester with the request reception of the HCSC server, and when there is a response (sending) from the HCSC server	For synchronous standard reception (Web Services) "<cluster name>-j2ee-<J2EE server name>-<numbers>.log" For SOAP reception "<Reception ID>-j2ee-<J2EE server name>-<numbers>.log"	For synchronous standard reception (Web Services) "<cluster name>-j2ee-<J2EE server name>-aplog-<numbers>.log" For SOAP reception "<Reception ID>-j2ee-<J2EE server name>-aplog-<numbers>.log"
  3	SOAP Communication trace when requests are sent (sending) from the service adapter of the HCSC server, and when there is a response (receiving) from the service adapter	Output to a trace file same as number 2 in the figure.	Output to the application log file same as number 2 in the figure.
  4#2	Communication trace when requests from the service adapter of the HCSC server are received (receiving) in the service component , and when there is a response (sending) from the service component	When using SOAP Communication Infrastructure "<prefix>-j2ee-<J2EE server name>-<numbers>.log" <prefix>#3 is set in the server definition file of the service component. When using JAX-WS engine "cjwmessage<numbers>.log"	When using SOAP Communication Infrastructure "<prefix>-j2ee-<J2EE server name>-aplog-<numbers>.log" <prefix>#3 is set in the server definition file of the service component. When using JAX-WS engine "cjwmessage<numbers>.log"

  Note#1

  The output destinations of trace and application logs differ as follows according to the Communication Infrastructure in use:

  When using SOAP Communication Infrastructure

  The <Log output directory of J2EE server>\WS on each running machine.

  When using JAX-WS engine

  The < Log output directory of J2EE server>\CJW on each running machine.

  Note#2

  Traces can be collected when the SOAP Communication Infrastructure or JAX-WS engine provided by Service Platform is used on the machine on which the service requester or service component is running.

  Note#3

  The name setup in prefix (c4web.logger.<identifier>.log_file_prefix) of trace files or application logs in the service definition file (c4websv.cfg) or the client definition file (c4webcl.properties).

• When invoking SOAP adapters from a reception other than Web Services

  The SOAP trace or application log is output with the following names when you invoke SOAP adapters from a request reception other than the synchronous standard reception or the SOAP reception. The following figure shows the collection point of SOAP trace and application log:

  Figure 7‒5:  Collection points of SOAP trace or application log (when you invoke a service adapter from a reception other than Web Services)

  The following table lists and describes the relationship between the trace collection point and file names. The numbers in the Number in figure column of the table correspond to the numbers in the above figure.

  Table 7‒11:  Relationship of trace collection point and file names (when you invoke a SOAP adapter from a reception other than Web Services)

  Number in figure	Collection point	File name
  		Trace#1	Application log#1
  1	SOAP Communication trace when requests are sent (sending) from the service adapters of the HCSC server, and when there is a response (receiving) from service adapters	"<service ID>-j2ee-<J2EE server name>-<numbers>.log"#3	"<service ID>-j2ee-<J2EE server name>-aplog-<numbers>.log"#3
  2#2	SOAP Communication trace when requests from the service adapter of the HCSC server are received (receiving) by service components, and when there is a response (sending) from service components	When using SOAP Communication Infrastructure "<prefix>-j2ee-<J2EE server name>-<numbers>.log" <prefix>#4 is set in the server definition file of the service component. When using JAX-WS engine "cjwmessage<numbers>.log"	When using SOAP Communication Infrastructure "<prefix>-j2ee-<J2EE server name>-aplog-<numbers>.log" <prefix>#4 is set in the server definition file of the service component. When using JAX-WS engine "cjwtransport<numbers>.log"

  Note#1

  The output destination of trace and application logs differs as follows according to the Communication Infrastructure in use:

  When using SOAP Communication Infrastructure

  The <Log output directory of J2EE server>\WS on each running machine.

  When using JAX-WS engine

  The <Log output directory of J2EE server>\CJW on each running machine.

  Note#2

  Can be acquired when a machine on which the service component is running uses SOAP Communication Infrastructure or the JAX-WS engine provided by the Service Platform.

  Note#3

  Even if you specify prefix of trace files and application logs in the client definition file setup in the Service adapter definition screen, the specified value is not enabled. The service ID is assigned as a unique prefix.

  Note#4

  The name where the prefix (c4web.logger.<identifier>.log_file_prefix) is setup for trace files or application logs in the server definition file (c4websv.cfg).

(b) Preparation before start of operation

If you acquire the log of SOAP messages, you can identify the cause of failure when there is an error in the sent and received user messages by checking the message format of SOAP Messages when invoking the HCSC server from the service requester and the SOAP messages when invoking the service component from the service adapter of the HCSC server.

The log output is set in the following common definition files on which HCSC server is running:

• For SOAP Communication Infrastructure: c4webcom.cfg

• For JAX-WS engine:cjwconf.properties

The following table describes the settings to the common definition file of the machine on which HCSC server is running:

Table 7‒12:  Settings of a machine on which the HCSC server is running to the common definition file

Item	Key name	Setting (Default)	Description
Importance of application log output	c4web.logger.aplog_level	WARN	Specify INFO while capturing (check the message format of SOAP Messages) the log of SOAP Messages. Must be set in advance before starting the HCSC server.

For details on creating common definition files used by SOAP Communication Infrastructure, see"10.4 Setting up the common definition file" in "Application Server SOAP Application Development Guide".

For details on creating common definition files used by the JAX-WS engine, see"10.1.2 Settings of the common definition file" in "Application Server Web Service Development Guide".

(c) Definition files required for failure investigation

The setup information of SOAP Communication Infrastructure and the JAX-WS engine is required for investigating failures. The following table describes definition files of SOAP Communication Infrastructure:

Table 7‒13:  Definition file of SOAP Communication Infrastructure

Definition file	Description	File save location	File name
Server definition file	Controls the operations of SOAP Communication in the request reception of a machine on which the HCSC server is running	Following folders of machine on which the HCSC server is running: For Windows <Installation directory of Service Platform>\c4web\conf For other OS /opt/Cosminexus/c4web/conf	c4websv.cfg
Common definition file	Controls the common operations of SOAP Communication the in request reception and the service adapter of a machine on which HCSC server is running	Following folders of machine on which the HCSC server is running: For Windows <Installation directory of Service Platform>\c4web\conf For other OS /opt/Cosminexus/c4web/conf	c4webcom.cfg

For details on definition files when using SOAP Communication Infrastructure, see "10.2 Setting up the server definition file" in "Application Server SOAP Application Development Guide". For details on common definition files when using SOAP Communication Infrastructure, see "10.4 Setting the common definition file" in "Application Server SOAP Application Development Guide ".

The following table describes the definition file for the JAX-WS engine:

Table 7‒14:  Definition file for the JAX-WS engine

Definition file	Explanation	File storage location	File name
Common definition file	Controls the SOAP communication operations common to the request reception and the service adapter of a machine on which the HCSC server is running.	Following folders of machine on which the HCSC server is running: For Windows <Installation directory of Service Platform>\jaxws\conf For other OS /opt/Cosminexus/jaxws/conf	cjwconf.properties

For details on definition files when using JAX-WS engine, see "10.1 Action definition file" in "Application Server Web Service Development Guide".

(a) JavaVM output message log

When the JavaVM crashes, the debug information is output to standard output and error report file by JavaVM. For details, see "4.11 JavaVM Output Message Logs (Standard Output or Error Report File)" in "Application Server Maintenance and Migration Guide". The output destination and output file name of the error report file is as follows:

• For Windows

  <Installation directory of Service Platform>\CC\server\usrconf\ejb\<J2EEserver name>\hs_err_pid<Process ID of server process>.log

• For UNIX

  <Installation directory of Service Platform>/CC/server/usrconf/ejb/<J2EE server name>/hs_err_pid<Process ID of server process>.log

(b) Thread dump of JavaVM

JavaVM statistics and stack status of threads is output. For details, see "4.7 JavaVM thread dump" in "Application Server Maintenance and Migration Guide". The output destination is as follows:

• In Windows

  <Installation directory of Service Platform>\CC\server\usrconf\ejb\<J2EE server name>\javacore*.txt

• In UNIX

  <Installation directory of Service Platform>/CC/server/usrconf/ejb/<J2EEserver name>/javacore*.txt

(c) Java VM GC log

The activities of Java VM GC are output to this log. For details, see "4.8 JavaVM GC Log" in "Application Server Maintenance and Migration Guide".

(d) Memory dump

When there is a failure in the system, the system administrator or Maintenance Service use the memory dump to analyze the error. For details, see "4.9 Memory Dump" in "Application Server Maintenance and Migration Guide".