This page is intended to help users of operating systems derived from the illumos gate to bring problems to the attention of the illumos gate developers.

Note: All commands in this guide is presumed to be run as a non-privileged user. pfexec will be prepended to commands which need additional privileges. On some systems, the sudo command is used for this purpose.

1. Help! My System Crashed or Locked Up!

You might be reading this because your system suddenly printed a panic message on the console, saved a crash dump, and then rebooted (or the popular alternative: suddenly locked up, showed some disk activity, and then rebooted). Or perhaps your system stopped responding and never recovered. Calm down. We want to know about this problem.

When illumos crashes (panics), a crash dump is automatically saved containing information from kernel memory. This can be copied to a file. A developer can use use the crash dump to inspect the state of the system at the time of the crash. You can also generate a crash dump manually to debug a hang.

If you are on a live session where no storage is available, you may have to manually copy down crash or hang information. A serial console is preferred for this.

Gathering Information from a Crash Dump

If you are on an installed system and a crash occurs, a crash dump will normally be saved. If you are on a live system, skip to the next section. Assuming the crash does not recur immediately after the reboot, you can retrieve information from the crash dump by following these instructions.

  1. Log in as a user with access to pfexec or sudo (typically the user you created at install)
  2. Check your crash dump configuration:
    pfexec dumpadm
    This should print something like:
          Dump content: kernel pages
       Dump device: /dev/zvol/dsk/rpool/dump (dedicated)
Savecore directory: /var/crash/kirin
  Savecore enabled: no
   Save compressed: on
  3. To extract the crash dump from the dump volume:
    pfexec mkdir -p /var/crash/`hostname`
pfexec savecore
cd /var/crash/`hostname`
pfexec savecore -vf vmdump.0
  4. With the crash dump extracted, you can now copy the kernel messages and stack information from the crash to a file:
    echo '::panicinfo\n::cpuinfo -v\n::threadlist -v 10\n::msgbuf\n*panic_thread::findstack -v\n::stacks' | mdb 0 > ~/crash.0

    Save the crash.0 file in your home directory for reporting the crash. If possible, keep the crash dump files on hand for future examination. If you have available bandwidth and disk space, making this full crash dump accessible to developers is invaluable. Note that due to the nature of the dump, it may contain information you consider confidential. Use your judgement when making it generally available.

Notes:

  1. Configuring your crash dump.  The commands "dumpadm" and "coreadm" and be used to list and alter the location and/or the actual contents of the resulting crash dump.

  2. Ensuring a successful crash dump. If you do not see a line like "100% done   .... dump succeeded" on the console (if you have a console) or when you later try to analyze your crash dump via "savecore -vd" or "savecore" you see a message like "savecore: bad magic number 0". You are probably are hitting illumos Bug #1110 and Bug #1369 (disable multi-threaded dump). In this case in order to successfully force a crash dump (if you do not see "dump succeeded")  you should modify your system to perform a non-threaded crash dump by adding the following line to your /etc/system file "set dump_plat_mincpu=0" and then performing a rconfiguration reboot.

  3. Forcing a crash dump. In order to force a crash dump form a working system that still has interactive capabilities you can do one of the following:

Gathering Information from a Running System

If you can reproduce the hang or crash, enabling the kernel debugger beforehand is a way to get valuable information about it. The best option, if problem occurs after the system has access to storage, is to use the debugger to save a crash dump. Otherwise, you should record as much diagnostic information as possible from the debugger.. A serial console or some form of lights-out management (LOM) functionality is preferred for interacting with the debugger.

To use a serial console, you need an onboard or USB serial port, a null modem cable, and another computer with a serial port, and to boot with the appropriate -B console=ttya or similar option.
For a local console, you need a monitor and keyboard. On x86, this must be a PS/2 keyboard or a USB keyboard operating in USB legacy (PS/2 emulation) mode.

To enable verbose boot messages and the kernel debugger (kmdb) on x86:

  1. At the GRUB prompt, highlight the boot environment with the arrow keys and press 'e' to edit the commands.
  2. Highlight the splashimage line, if it exists, and press 'd' to delete.
  3. Highlight the kernel line and press 'e' to edit.
  4. Append -kvd -m verbose to the end of the line (and remove console=graphics if it appears after -B)
  5. Press 'Enter' when finished, and then press 'b' to boot.

To enable verbose boot messages and the kernel debugger (kmdb) on SPARC:

  1. Press 'Stop' ('L1'), then 'A' or send a BREAK on the serial line to interrupt the boot process, if necessary.
  2. At the ok prompt, type boot -kvd -m verbose and press 'Enter'.

On both platforms:

  1. Wait for the kmdb> prompt to appear, then type moddebug/W 80000000 and press 'Enter'.
  2. Type snooping/W 1 and press 'Enter'.
  3. Type :c and press 'Enter'.

If the system panics, copy down ::msgbuf, ::status and ::stack.

If the system hangs, copy down the last few lines regarding modules on screen, and do one of:

When you are successful the system should drop you into a kmdb prompt.

  1. Use $<systemdump to save a crash dump. The system should automatically reboot when finished.
  2. Follow "Gathering Information About A Crash", above.
  3. If you cannot save a crash dump, and are on a serial console, enter ::msgbuf, ::panicinfo, ::cpuinfo -v and::threadlist -v 10 and record the output. (Or take photos if you're unlucky enough to be at a VGA console)..

More resources:

Gathering Information from a Running System, Using only NMI (x86)

In order to force a crash dump form a system that seems hung or frozen e.g. no longer has has interactive capabilities if your system supports NMI (hardware button, service processor command) it it might be a good practice to permanently configure the following:

The above might be a a very good practice as it simplifies forcing a "crash dump" (avoids the x86 requirement of  keyboard input e.g. press 'Shift-Pause'; or 'Esc-B'; or 'F1', then 'A'.).  However this simplification will a) not work on all platforms and b) might even cause issues on a few system types and c) works if your system loads/uses the pcplusmp module (not the apix module).  Because of these requirements despite the fact that many admins use this as a default setting it is not the default configuration for fresh installs of the OS.

More resources:

Inspecting An Application Crash

  1. Check your core file configuration:
    coreadm

    This should print something like:
         global core file pattern: 
     global core file content: default
       init core file pattern: core
       init core file content: default
            global core dumps: disabled
       per-process core dumps: enabled
      global setid core dumps: disabled
 per-process setid core dumps: disabled
     global core dump logging: disabled

    If the global core file pattern is empty and per-process core dumps are enabled, an application crash will save a core file named core in the working directory.
  2. To print the stack information from the crash:
    pstack core

2. Creating an Issue Report

You can report a new issue using our issue tracker.
The fact that this currently requires registration is considered a problem (#442).

A good issue report includes: