The following figure shows the flow of troubleshooting when a process is down.

Figure 6-6 Flow of actions when a process is down

The details of the processing shown in the figure are described in subsection (2).

The following points describe the operations according to the contents of the process down flow:

1. Open the error report file (hs_err_pidprocess-ID.log)
   Open the error report file that was output around the time the error occurred and compare it with the Manager log.
   
   • Output destination of the error report file and the output file name
     C:\Program Files\Hitachi\Cosminexus\CC\server\public\ejb\server-name\hs_err_pidserver-process-ID.log
     
2. Check whether OutOfMemoryError occurs
   If java.lang.OutOfMemoryError occurred. is displayed, the message indicates that OutOfMemoryError has occurred.
   The location where OutOfMemoryError has occurred is indicated in bold characters.
   

   Example of output when OutOfMemoryError occurs

   : # # java.lang.OutOfMemoryError occurred. # JavaVM aborted because of specified -XX:+HitachiOutOfMemoryAbort options. # Please check Javacorefile:D:\Cosminexus\CC\server\public\ejb\MyJ2EEServer\javacore2100.120124144835.txt #

   Example of output when OutOfMemoryError does not occur

   : # # An unexpected error has been detected by Java Runtime Environment: # # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000008303b00, pid=1356, tid=2604 # # Java VM: Java HotSpot(TM) 64-Bit Server VM (11.3-b02-CDK0853-20101027 mixed mode windows-amd64) # Problematic frame: # V [C:\Program Files\Hitachi\Cosminexus\jdk\jre\bin\server\jvm.dll+0x303b00] #

   Note

   If OutOfMemoryError occurs, you must check the error report file. Go to step 3.

   If OutOfMemoryError does not occur, see the description for the point Checks when OutOfMemoryError does not occur.

3. Check the extended verbosegc information and tune the memory size
   Check the extended verbosegc information and adjust the Java heap memory size.
   However, note that the hard disk memory size must not be exceeded.
   
   • For Windows x86
     Specify -Xmx1024m.
     If -Xmx1024m is already specified, you must either reduce the number of concurrently executing threads, or start multiple J2EE servers to distribute the load.
     
   • For Windows x64
     Using the set value of the -Xmx option as an indicator, specify 1.5 to 2 times the value.
     
4. Restart the machine
   Restart the machine and check if the operations can be executed without any problems.
   

   Hint

   Troubleshooting is complete when you restart the machine and can perform operations without any problems.

   If the operations are not performed correctly even after the machine is restarted, go to step 5.

5. Output the class-wise statistics
   Check the JavaVM garbage collection log (javalog[n].log), or operation log (HJVMStats_YYYYMMDDhhmmTZ.csv), for every 10 MB to 20 MB memory increased, execute the jheapprof command to output the class-wise statistics (txt file).
   Repeat the process until the memory is increased to 100 MB or more.
   If the command execution interval is short, the leaked memory size is also small, so it is difficult to find the class for which the memory has been increased.
   By increasing the execution interval and incrementing the increase in memory, the class from which the memory is leaking is made to stand out.
   

   Example of execution

   % jheapprof -p 2463

   Note

   2463 in the example of execution becomes the process ID. When you execute the command, specify the process ID with the problem.

6. Output the class structure list
   List and output the class structure containing the class picked up in the previous step as a member, create the class list to be checked, and then list up the check targets.
   

   Example of execution

   % jheapprof -class org.apache.catalina.loader.WebappClassLoder -p 2463

   Note

   2463 in the example of execution becomes the process ID. When you execute the command, specify the class with the problem.

   org.apache.catalina.loader.WebappClassLoder in the example of execution becomes the class name. Use this class to check OutOfMemoryError caused by an insufficient Permanent area.

7. Request the developer to perform the check
   Send the information collected in the previous step to the developer and request a check.
   

   Hint

   The procedure of performing a check when OutOfMemoryError does not occur is as follows. Note that the step numbers correspond to Figure 6-6.

   Checks when OutOfMemoryError does not occur

   3-1 Check the library where the signal occurred

   Check the red portion in the example of output, and make a note of the down library name and the down native function.

   In the example of output, the library name is NativeCrash.dll and the down native function is (null)+0x77C785BA.

   3-2 Check the signal

   If the location specified after "An unexpected error has been detected by HotSpot Virtual Machine: " is down due to signal 6 (SIGABRT, SIGIOT), the invocation destination library must be checked because the library is aborted by a higher abort function.

   The above example is not aborted.

   In this example, request a detailed check to the Oracle support service.

   3-3 Send the collected results to the helpdesk

   Send the library you have noted and the signal contents to the module developer or to the helpdesk based on the purchase agreement and request a check.