---
breadcrumbs:
- - /developers
  - For Developers
page_name: crash-reports
title: Crash Reports
---

The crash reporting system for Google Chrome is called
[Breakpad](https://code.google.com/p/google-breakpad/), although it is being
replaced by [Crashpad](https://crashpad.chromium.org/), which is already in use
on Windows and Mac. Both are integrated into the Chromium source in
[//components/crash](https://code.google.com/p/chromium/codesearch#chromium/src/components/crash/).
Breakpad is disabled by default in Chromium, and crash reports are only
available to Google employees, though they are frequently copied into the bug
tracker for tracking and fixing.

[TOC]

# Enabling Breakpad

Breakpad is disabled by default. The cross-platform way to turn on crash
reporting is to perform an official, branded build of Chromium by setting GN
arguments is_chrome_branded=true and is_official_build=true, which will require
a [src-internal](https://goto.google.com/src-internal) checkout.

To enable Breakpad without doing a branded build, each platform has a unique way
of enabling breakpad:

## In Chromium

### Windows

Windows uses crashpad by default. It's compiled in and enabled by default.

### Linux

Breakpad is compiled in, but is not enabled by default. See
IsCrashReportingEnabled() in chrome_browser_main_linux.cc for details on when it
is enabled.

Set the gyp variable (GYP_DEFINES) linux_dump_symbols=1 to dump the debugging
symbols for the chrome binary in the Breakpad symbol format as part of a build.

### Chrome OS

Breakpad is compiled in, similar to Linux above. See the [Chrome OS crash
reporting FAQ](/chromium-os/packages/crash-reporting/faq) for differences.

### Mac

Mac uses Crashpad as its crash reporter instead of Breakpad. Crashpad is always
compiled in and enabled, though reports are only stored locally and not uploaded
by default.

## Generating Crash Dumps Locally

After Breakpad is enabled, you can have crash report minidumps generated
locally.

To get crash reports for content_shell and layout tests, see [this
page](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/testing/using_crashpad_with_content_shell.md).

### On Windows

Run chrome.exe --noerrdialogs to disable the OS-level error reporter. Any
crashes will then cause a minidump file (.dmp) to be generated in
%HOME%\\AppData\\Local\\Chromium\\User Data\\Crashpad\\reports.

### On Linux

To make chromium write a .dmp file, run it like

CHROME_HEADLESS=1 out/Release/chrome

### On Mac

To make Chromium write crash dumps, no additional work is needed. To test it, a
renderer process can be triggered to crash by visiting “about:crash”, or for
real fun, a browser process crash can be triggered by visiting
“about:inducebrowsercrashforrealz”. The crash dump file is stored in
~/Library/Application Support/Chromium/Crashpad/completed.

$ ls -l ~/Library/Application\\ Support/Chromium/Crashpad/completed

total 448

-rw-------@ 1 mark staff 229248 Mar 13 2015
4bfca440-039f-4bc6-bbd4-6933cef5efd4.dmp

# Life of a Crash Report

When a Chrome process crashes, Breakpad or Crashpad springs into action by
gathering information about the exception state, callstacks, stack memory, and
loaded modules. It takes all of this information and puts it into a minidump
file. This minidump is then HTTP POST uploaded to Google, along with some
metadata. The metadata contains information like Chrome version, OS name and
version, and [crash
keys](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/debugging_with_crash_keys.md).

Once the crash report and metadata are received, a server begins processing it.
The first step is to symbolize the instruction pointers to get function names
associated with the callstack. The processing servers have platform-neutral
[Breakpad symbol
files](https://code.google.com/p/google-breakpad/wiki/SymbolFiles) for Chrome
executables and operating system shared libraries, which allow symbolication of
both Chrome functions and OS functions. After symbolizing the crash, the
processor generates signatures to help group/bucket crash reports by their
cause, implicated by shared stack frames.

The tools used by the processor to symbolize the stack are effectively the same
as the minidump_stackwalk and minidump_dump tools available in Breakpad.

Once a crash report is processed, it is made available for view in the [crash
dashboard](https://goto.google.com/crash/root). The dashboard allows developers
to drill down into crash reports, grouped by metadata. More documentation on
this is available internally, linked to from the dashboard. The original
minidump file can be downloaded from this frontend, as well.

## Crash Signatures

There are three types of signatures generated by the crash processor:

### Stack Signature

The stack signature is the topmost symbol name in the crashed thread, suffixed
with a small hash of the callstack. This signature provides a high-level
grouping and easy identification (as it is human-readable), but it can collide
across versions for different complete stack traces. Variance in the crashing
thread's callstack can also produce several signatures for the same root cause.

### Stack Signature 2

This signature is merely a hash of the crashed stack. This makes it accurate for
grouping crashes precisely, but it is not human readable and slight variance in
a crashed stack defeat this signature's bucketing properties.

### Magic Signature

The magic signature is heuristic-based, using application-specific knowledge of
the Chromium codebase to find the most germane symbol to produce a signature.
The magic signature creator has knowledge of CHECK/DCHECK, hangs caught by the
ThreadWatcher watchdog, out-of-memory conditions, memory corruption signs, and
other heuristics. This signature is highly simplified, which gives it good
bucketing properties and protection against callstack variance, but the
signature can often be generated for disjoint crashes from different versions.
More information can be found in the processor's [design
document](https://goto.google.com/xgqgp) and this
[explanation](https://goto.google.com/gvqex).

# Working with Minidumps

The minidump format originated on Windows, and can be natively loaded into a
[debugger on that
platform](/developers/how-tos/debugging-on-windows#TOC-Debugging-with-WinDBG).
Doing this gives you access to local variables, callstacks, and integrates with
the source viewer in Visual Studio.

On POSIX systems, however, a minidump is an opaque binary blob. It is possible
to examine local variable state, but it requires manual work to do so. To work
with minidumps, Breakpad provides two tools ([more information
here](/developers/decoding-crash-dumps)):

**minidump_stackwalk** prints the stack trace for all of the threads in the
minidump. Note that without Breakpad symbol files, placed in a special directory
structure, this will not symbolize the stack. It will merely print the %EIP,
%EBP, and %ESP (or the x64 equivalent) for each frame and the code module in
which the frame resides.
**minidump_dump** outputs the stack memory for each thread as a hexadecimal
string.

## Extracting Local Variables on POSIX

This is one workflow for manually analyzing a minidump:

1.  Run minidump_dump and minidump_stackwalk on the target minidump
            file, piping output to text files.
2.  Find the target thread in the stackwalk (via the threadid).
3.  Find the corresponding target thread in the stackdump.
4.  Take the raw stack data for that thread, and remove the 0x prefix.
5.  Split the stack data on word-size boundaries (8 hexadecimal
            characters on a 32-bit platform).
6.  Using the information from the stackwalk, annotate EIP, EBP, and ESP
            in the stackdump to delineate stack frames. Note that the byte
            order/endianness in the stackdump may not match host order, i.e. the
            pointer badbeef may be represented in the minidump as efbedbba.
7.  Download the build products (executable and symbol file) for the
            version of Chrome you're working with. Disassemble the function
            you're interested in.
8.  Step through the assembly and see how data is being pushed and
            popped off the stack, and try to correlate it to the data in the
            stackdump.

Unfortunately, this process is neither straightforward nor quick. But there are
alternatives for saving local variables to be visible in a crash report. See
below for more information.

However, it may be possible to get most of the same information using
[WinDBG](/developers/how-tos/debugging-on-windows#TOC-Debugging-with-WinDBG) on
Windows. As long as the CPU architecture is supported by Windows, the debugger
will be able to understand the exception record and disassemble the code around
the exception, show call stacks (without symbols though), and let you inspect
the contents of the stack and registers. That may be enough to get you going on
a good percentage of crashes, especially since the symbolized stack may be
available on the crash server.

See also
<http://www.chromium.org/chromium-os/how-tos-and-troubleshooting/crash-reporting/debugging-a-minidump>

## Symbolizing Minidumps

To symbolize a minidump that was not uploaded, you can use the
[crsym](https://goto.google.com/crsym/) tool (internal instance of
<https://github.com/chromium/crsym/>):

1.  Build minidump_stackwalk from inside the Chromium or Breakpad trees.
2.  Run minidump_stackwalk -m /path/to/minidump.dmp &gt; /tmp/stack.txt
3.  Go to [crsym](https://goto.google.com/crsym/) and set the input type
            to be a Minidump Stackwalk.
4.  Paste the contents of /tmp/stack.txt into the input field.
5.  Press the Symbolize button and wait for the output.

To symbolize a minidump for a local build, follow the steps in
<https://code.google.com/p/chromium/issues/detail?id=304846#c14> for Mac, and
see <http://www.chromium.org/developers/decoding-crash-dumps> for linux.

# Production Debugging

If a crash is not reproducible, the only tools at one's disposal for
investigation are what can be gleaned from a crash report. These are some tips
for debugging builds running in production (i.e. on users' machines).

## Aliasing Against Optimizations

Production builds are highly optimized to make the binary executable fast and
small. As part of this, when the compiler is optimizing, it will reuse space on
the stack for different variables. To defeat this optimization, you can use
base::debug::Alias() on the variable. This will attempt to force the compiler to
keep dedicated space around for the variable, so that its value is inspectable
in a minidump stackdump.

## Crash Keys

Another way to make variable values accessible in a crash report is to report it
as metadata, rather than keeping it in the minidump. The metadata is viewable in
the crash dashboard and is not contained in the minidump file. Learn more about
[debugging with crash keys
here](https://chromium.googlesource.com/chromium/src/+/HEAD/docs/debugging_with_crash_keys.md).

## Checking Out a Release Branch

Production builds are generally not cut from the tip of tree/trunk, but are
branched and stabilized via the dev, beta, and stable release channels. [Follow
these
instructions](https://code.google.com/p/chromium/wiki/UsingGit#Working_with_release_branches)
for how to check out a release branch, which will give you the code for the
version of the crash you're investigating.
