If you want to analyze stack traces of a process or if a Java process does not respond to a SIGQUIT(3) signal in an attempt to generate Java thread dumps, you can attach a debugger to the process or investigate core dumps to examine data that is pertinent to the process in question (for example, stack frames and register information).

A core dump is a file that contains the memory snapshot of a running process and its process information. If a process terminates unexpectedly or does not respond in a timely manner, core dumps that are taken at the time of the incident can be very useful. That is, the core dumps enable you to determine the root cause of the incident by examining all registers, stack frames and memory map of the process.

To access the stack frames of a process, you need to collect the following information:

• the process identifier of a running process or the core dump of a process
• the path of an executable program

To analyze your information, perform the steps in the following sections:

• Attaching the Debugger to a Process
• Generating a Core Dump
• Examining a Core Dump

Note: This document also contains Disclaimer of Warranty and Limitation of Liability information.

Attaching the Debugger to a Process

Before you use the debugger, you first need to determine the process identifier of the process in question or obtain a core dump for that process in order to access the stack frames. Then you can attach the debugger, as follows:

1. Run the process status ps command to find an identifier and an executable of the process in question, as shown in this example command.
   
   [sasinstall@host~]$ ps -ef | grep java

   This command displays the following Java processes:

   202 2763 1 1 15:10 ? 00:00:07 /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java -Djava.system.class.loader=com.sas.app.AppClassLoader -Dsas.app.launch.picklist=/apps/sas/SAS9x/GOLD/config/Lev1/Web/Applications/SASWIPSchedulingServices9.4/dip/picklist -Dsas.app.repository.path=/apps/sas/SAS9x/GOLD/SASHome/SASVersionedJarRepository/eclipse -Dsas.cache.locators=host.sas.com[415] -Dsas.gemfire.log.file= -Dsas.gemfire.log-level=severe -Dspring.profiles.active=client-locators -Dsas.app.class.dirs=/apps/sas/SAS9x/GOLD/config/Lev1/Web/Applications/SASWIPSchedulingServices9.4/dip/lib -classpath /apps/sas/SAS9x/GOLD/SASHome/SASVersionedJarRepository/eclipse/plugins/sas.launcher.jar -Xmx128m com.sas.scheduler.api.servers.ip.engine.mq.client.CommandLineJobServerFromMetadata /apps/sas/SAS9x/GOLD/config/Lev1/Web/Applications/SASWIPSchedulingServices9.4/dip/DIPJobRunner.properties
2. Run the debugger gdb command with the process identifier and the path of an executable, as shown in the following example:
   
   [sasinstall@host ~]$ gdb /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java 2763

   This command generates the following output:

   GNU gdb (GDB) Linux Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. Attaching to program: /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java, process 2763 [New LWP 2774] [New LWP 2773] [New LWP 2772] [New LWP 2771] [New LWP 2770] [New LWP 2769] [New LWP 2768] [New LWP 2767] [New LWP 2766] [New LWP 2765] [New LWP 2764] [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 (gdb)
3. You can run other debugger commands to examine process information. For example, the backtrace command displays frame stacks, as shown in the following example:
   
   (gdb) backtrace #0 0x0000003774c080ad in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f32c7af6505 in ContinueInNewThread0 () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/../lib/amd64/jli/libjli.so #2 0x00007f32c7aeb58a in ContinueInNewThread () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/../lib/amd64/jli/libjli.so #3 0x00007f32c7aee0e0 in JLI_Launch () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/../lib/amd64/jli/libjli.so #4 0x0000000000400686 in main () (gdb)

   In the next example, the info threads command displays thread information such as thread numbers, the target system's thread identifier, and the thread name:

   (gdb) info threads 12 Thread 0x7f32c6c7f700 (LWP 2764) 0x0000003774c0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 11 Thread 0x7f32c618e700 (LWP 2765) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 10 Thread 0x7f32c608d700 (LWP 2766) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 9 Thread 0x7f32c4c26700 (LWP 2767) 0x0000003774c0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 8 Thread 0x7f32c4b25700 (LWP 2768) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 7 Thread 0x7f32c4a24700 (LWP 2769) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 6 Thread 0x7f32c4923700 (LWP 2770) 0x0000003774c0d720 in sem_wait () from /lib64/libpthread.so.0 5 Thread 0x7f32c4822700 (LWP 2771) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 4 Thread 0x7f32c4721700 (LWP 2772) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 3 Thread 0x7f32c4620700 (LWP 2773) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 2 Thread 0x7f32c451f700 (LWP 2774) 0x0000003774c0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 * 1 Thread 0x7f32c7ae7700 (LWP 2763) 0x0000003774c080ad in pthread_join () from /lib64/libpthread.so.0 (gdb)

   You can also issue the thread command with the following arguments to display the frame stacks of all threads:

   (gdb) thread apply all backtrace Thread 11 (Thread 0x7f32c618e700 (LWP 2765)): #0 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f32c748f0b3 in os::PlatformEvent::park() () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #2 0x00007f32c7456a1f in Monitor::IWait(Thread*, long) () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #3 0x00007f32c74571ae in Monitor::wait(bool, long, bool) () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #4 0x00007f32c71cf903 in GCTaskManager::get_task(unsigned int) () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #5 0x00007f32c71d0fc8 in GCTaskThread::run() () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #6 0x00007f32c7495538 in java_start(Thread*) () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #7 0x0000003774c07851 in start_thread () from /lib64/libpthread.so.0 #8 0x00000037748e890d in clone () from /lib64/libc.so.6 . . . more thread lines. . . Thread 6 (Thread 0x7f32c4923700 (LWP 2770)): #0 0x0000003774c0d720 in sem_wait () from /lib64/libpthread.so.0 #1 0x00007f32c7493fda in check_pending_signals(bool) () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #2 0x00007f32c748d675 in signal_thread_entry(JavaThread*, Thread*) () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #3 0x00007f32c75cd5ff in JavaThread::thread_main_inner() () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #4 0x00007f32c75cd705 in JavaThread::run() () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #5 0x00007f32c7495538 in java_start(Thread*) () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #6 0x0000003774c07851 in start_thread () from /lib64/libpthread.so.0 #7 0x00000037748e890d in clone () from /lib64/libc.so.6 . . . more thread lines. . . Thread 1 (Thread 0x7f32c7ae7700 (LWP 2763)): #0 0x0000003774c080ad in pthread_join () from /lib64/libpthread.so.0 #1 0x00007f32c7af6505 in ContinueInNewThread0 () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/../lib/amd64/jli/libjli.so #2 0x00007f32c7aeb58a in ContinueInNewThread () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/../lib/amd64/jli/libjli.so #3 0x00007f32c7aee0e0 in JLI_Launch () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/../lib/amd64/jli/libjli.so #4 0x0000000000400686 in main () (gdb)

   If you want to inspect all registers, issue the info command with the all-registers argument:

   (gdb) info all-registers rax 0xfffffffffffffdfc -516 rbx 0x0 0 rcx 0xffffffffffffffff -1 rdx 0x1 1 rsi 0x189 393 rdi 0x7f32c000a654 139855946360404 rbp 0x7f32c6c7e5d0 0x7f32c6c7e5d0 rsp 0x7f32c6c7e510 0x7f32c6c7e510 r8 0x7f32c000a628 139855946360360 r9 0xffffffff 4294967295 r10 0x7f32c6c7e590 139856060081552 r11 0x202 514 r12 0x1 1 r13 0x7f32c6c7e590 139856060081552 r14 0xffffffffffffff92 -110 r15 0x0 0 rip 0x3774c0b7bb 0x3774c0b7bb <pthread_cond_timedwait@@GLIBC_2.3.2+315> eflags 0x202 [ IF ] cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 st0 0 (raw 0x00000000000000000000) st1 0 (raw 0x00000000000000000000) st2 0 (raw 0x00000000000000000000) st3 0 (raw 0x00000000000000000000) st4 -nan(0xc000000000000000) (raw 0xffffc000000000000000) st5 -nan(0xc000000000000000) (raw 0xffffc000000000000000) st6 0 (raw 0x00000000000000000000) st7 0 (raw 0x00000000000000000000) fctrl 0x37f 895 fstat 0x21 33 ftag 0xffff 65535 fiseg 0x7f32 32562 fioff 0xc7526f48 -950898872 foseg 0x7f32 32562 fooff 0xc6c7cab0 -959984976 fop 0x0 0 mxcsr 0x1fa1 [ IE PE IM DM ZM OM UM PM ] ymm0 {v8_float = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_double = {0x0, 0x0, 0x0, 0x0}, v32_int8 = { 0x0 <repeats 32 times>}, v16_int16 = {0x0 <repeats 16 times>}, v8_int32 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, v4_int64 = {0x0, 0x0, 0x0, 0x0}, v2_int128 = {0x00000000000000000000000000000000, 0x00000000000000000000000000000000}} . . .more register lines. . . (gdb)

   To save all of the data into a core-dump file, issue the gcore command:

   (gdb) gcore Saved corefile core.2763

   The detach command detaches the debugger from the process:

   (gdb) detach Detaching from program: /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java, process 2763

   Finally, the quit command exits from the debugger.

   (gdb) quit

Generating a Core Dump

To generate a core dump, perform one of the following options:

1. Run the gcore command from the debugger, as shown below:
   
   [sasinstall@host ~]$ gdb /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java 2763 (gdb) gcore Saved corefile core.2763 (gdb)
2. Run the gcore command with a process identifier, as shown below:
   
   [sasinstall@host ~]$ gcore 2763 Missing separate debuginfo for /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java Try: yum --disablerepo='*' --enablerepo='*-debug*' install /usr/lib/debug/.build-id/b8/2a586842f6da3df8a61093daadf04180e85c16.debug [New LWP 13838] [New LWP 13836] [New LWP 2774] [New LWP 2773] [New LWP 2772] [New LWP 2771] [New LWP 2770] [New LWP 2769] [New LWP 2768] [New LWP 2767] [New LWP 2766] [New LWP 2765] [New LWP 2764] [Thread debugging using libthread_db enabled] 0x0000003774c080ad in pthread_join () from /lib64/libpthread.so.0 Saved corefile core.2763 [sasinstall@host ~]$

Examining a Core Dump

Follow these steps to examine a core dump:

1. Run the debugger gdb command with the path of an executable and a core dump.
   
   [sasinstall@host ~]$ gdb /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java core.2763 GNU gdb (GDB) Linux [New Thread 2764] [New Thread 2765] [New Thread 2766] [New Thread 2767] [New Thread 2768] [New Thread 2769] [New Thread 2770] [New Thread 2771] [New Thread 2772] [New Thread 2773] [New Thread 2774] [New Thread 2763] Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/../lib/amd64/jli/libjli.so......done. Loaded symbols for /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/../lib/amd64/jli/libjli.so . . .more output lines. . . Loaded symbols for /lib64/libresolv.so.2 Core was generated by `/apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java -'. #0 0x0000003774c0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 (gdb)
2. Run debugger commands (for example, info and backtrace) to examine the process in question, as shown in this example:
   
   (gdb) info threads 12 Thread 0x7f32c6c7f700 (LWP 2764) 0x0000003774c0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 11 Thread 0x7f32c618e700 (LWP 2765) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 10 Thread 0x7f32c608d700 (LWP 2766) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 9 Thread 0x7f32c4c26700 (LWP 2767) 0x0000003774c0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 8 Thread 0x7f32c4b25700 (LWP 2768) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 7 Thread 0x7f32c4a24700 (LWP 2769) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 6 Thread 0x7f32c4923700 (LWP 2770) 0x0000003774c0d720 in sem_wait () from /lib64/libpthread.so.0 5 Thread 0x7f32c4822700 (LWP 2771) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 4 Thread 0x7f32c4721700 (LWP 2772) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 3 Thread 0x7f32c4620700 (LWP 2773) 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 2 Thread 0x7f32c451f700 (LWP 2774) 0x0000003774c0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 * 1 Thread 0x7f32c7ae7700 (LWP 2763) 0x0000003774c080ad in pthread_join () from /lib64/libpthread.so.0 (gdb) (gdb) thread apply all backtrace Thread 12 (Thread 0x7f32c6c7f700 (LWP 2764)): #0 0x0000003774c0b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f32c74942f8 in os::PlatformEvent::park(long) () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #2 0x00007f32c7495044 in os::sleep(Thread*, long, bool) () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #3 0x00007f32c72fd612 in JVM_Sleep () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #4 0x00007f32bd012cd8 in ?? () #5 0x00007f32c6c7e778 in ?? () #6 0x00007f32bd00f3eb in ?? () #7 0x00007f32bd005310 in ?? () #8 0x00007f32c6c7e6f8 in ?? () #9 0x0000000000000000 in ?? () Thread 11 (Thread 0x7f32c618e700 (LWP 2765)): #0 0x0000003774c0b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007f32c748f0b3 in os::PlatformEvent::park() () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #2 0x00007f32c7456a1f in Monitor::IWait(Thread*, long) () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #3 0x00007f32c74571ae in Monitor::wait(bool, long, bool) () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #4 0x00007f32c71cf903 in GCTaskManager::get_task(unsigned int) () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #5 0x00007f32c71d0fc8 in GCTaskThread::run() () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #6 0x00007f32c7495538 in java_start(Thread*) () from /apps/sas/SAS9x/GOLD/SASHome/SASPrivateJavaRuntimeEnvironment/9.4/jre/lib/amd64/server/libjvm.so #7 0x0000003774c07851 in start_thread () from /lib64/libpthread.so.0 #8 0x00000037748e890d in clone () from /lib64/libc.so.6 . . .more output lines. . . (gdb)

Disclaimer of Warranty

This instruction is provided to you "as is" without any warranties, express or implied, including but not limited to implied warranties or merchantability and/or fitness for a particular purpose. The Institute and its licensor(s) disclaim any liability connected with the use of the instruction. The Institute offers no technical support for the instruction.

Limitation of Liability

The Institute and its licensor(s) are not liable for (a) incidental, consequential, special, or direct damages of any sort, whether arising in tort, contract or otherwise, even if the Institute has been informed of the possibility of such damages, or (b) any claim by any other party. Some jurisdictions do not allow the exclusion or limitation of liability for incidental or consequential damages, so this limitation and exclusion may not apply to you.

Operating System and Release Information