6.5.1 Troubleshooting when a process is down

This subsection describes troubleshooting when a process is down.

Organization of this subsection

(1) Flow of actions when a process is down
(2) Flow of actions when a process is down

(1) Flow of actions when a process is down

The following figure shows the flow of troubleshooting when a process is down.

Figure 6‒5: Flow of actions when a process is down

The details of the processing shown in the figure are described in subsection (2).

To Page Top

(2) Flow of actions when a process is down

The following points describe the operations according to the contents of the process down flow:

Open the error report file (hs_err_pidprocess-ID.log)

Open the error report file that was output around the time the error occurred and compare it with the Manager log.
- Output destination of the error report file and the output file name
  
  C:\Program Files\Hitachi\Cosminexus\CC\server\public\ejb\server-name\hs_err_pidserver-process-ID.log
Check whether OutOfMemoryError occurs

If OutOfMemoryError has occurred, java.lang.OutOfMemoryError occurred. is displayed.

The following are examples of the output of the error report file and the action to be taken when OutOfMemoryError has occurred and when OutOfMemoryError has not occurred.
When OutOfMemoryError has occurred
Output example
```
  :
#
# java.lang.OutOfMemoryError occurred.
# JavaVM aborted because of specified -XX:+HitachiOutOfMemoryAbort options.
# Please check Javacorefile:D:\Cosminexus\CC\server\public\ejb\MyJ2EEServer\javacore2100.120124144835.txt
#
```
The part shown in bold with a colored background is the string that is output when OutOfMemoryError has occurred.

Action

You need to examine the error report file. Go to step 3.
When OutOfMemoryError has not occurred
Output example
```
 :
#
# A fatal error has been detected by the Java Runtime Environment
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000008303b00, pid=1356, tid=2604
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.20-b23-CDK0970-20150127 mixed mode windows-amd64)
# Problematic frame:
# V [C:\Program Files\Hitachi\Cosminexus\jdk\jre\bin\server\jvm.dll+0x303b00]
#
```
Action

The following checks are required. Perform the following steps. Note that the step numbers correspond to Figure 6-5.

3-1 Check the library where the signal occurred

Check the red portion in the example of output, and make a note of the down library name and the down native function.

In the example of output, the library name is NativeCrash.dll and the down native function is (null)+0x77C785BA.

3-2 Check the signal

If the location specified after "An unexpected error has been detected by HotSpot Virtual Machine: " is down due to signal 6 (SIGABRT, SIGIOT), the invocation destination library must be checked because the library is aborted by a higher abort function.

The above example is not aborted.

In this example, request a detailed check to the Oracle support service.

3-3 Send the collected results to the helpdesk

Send the library you have noted and the signal contents to the module developer or to the helpdesk based on the purchase agreement and request a check.
Check the extended verbosegc information and tune the memory size

Check the extended verbosegc information and adjust the Java heap memory size.

Using the set value of the -Xmx option as an indicator, specify 1.5 to 2 times the value. However, note that the hard disk memory size must not be exceeded.
Restart the machine

Restart the machine and check if the operations can be executed without any problems.

Tip

Troubleshooting is complete when you restart the machine and can perform operations without any problems.

If the operations are not performed correctly even after the machine is restarted, go to step 5.
Output the class-wise statistics

Check the JavaVM GC log (javalog[n].log), or operation log (HJVMStats_YYYYMMDDhhmmTZ.csv), for every 10 MB to 20 MB memory increased, execute the jheapprof command to output the class-wise statistics (txt file).

Repeat the process until the memory is increased to 100 MB or more.

If the command execution interval is short, the leaked memory size is also small, so it is difficult to find the class for which the memory has been increased.

By increasing the execution interval and incrementing the increase in memory, the class from which the memory is leaking is made to stand out.

Example of execution

% jheapprof -p 2463

Note

2463 in the example of execution becomes the process ID. When you execute the command, specify the process ID with the problem.
Output the class structure list

List and output the class structure containing the class picked up in the previous step as a member, create the class list to be checked, and then list up the check targets.

Example of execution

% jheapprof -class org.apache.catalina.loader.WebappClassLoder -p 2463

Note

2463 in the example of execution becomes the process ID. When you execute the command, specify the class with the problem.

org.apache.catalina.loader.WebappClassLoder in the example of execution becomes the class name. Use this class to check OutOfMemoryError caused by an insufficient Metaspace area.
Request the developer to perform the check

Send the information collected in the previous step to the developer and request a check.

To Page Top