docs/debugging_with_crash_keys.md

# Debugging with Crash Keys

Chrome is client-side software, which means that sometimes there are bugs that
can occur only on users' machines ("in production") that cannot be reproduced by
test or software engineering. When this happens, it's often helpful to gather bug-
specific data from production to help pinpoint the cause of the crash. The crash
key logging system is a generic method to help do that.

[TOC]

## High-Level Overview

The core of the crash key logging system is in [//components/crash/core/common/crash_key.h](https://cs.chromium.org/chromium/src/components/crash/core/common/crash_key.h),
which declares a `crash_reporter::CrashKeyString` class. Every crash key has
an associated value maximum length and a string name to identify it. The maximum
length is specified as a template parameter in order to allocate that amount of
space for the value up-front. When a process is crashing, memory corruption can
make it unsafe to call into the system allocator, so pre-allocating space for
the value defends against that.

When a crash key is set, the specified value is copied to its internal storage.
And if the process subsequently crashes, the name-value tuple is uploaded as
POST form-multipart data when the crash report minidump is uploaded to the
Google crash reporting system. (The data therefore are only accessible to those
with access to crash reports internally at Google). For platforms that use
[Crashpad](https://crashpad.chromium.org) as the crash reporting platform, the
crash keys are also stored in the minidump file itself. For platforms that use
Breakpad, the keys are only available at upload.

The crash key system is used to report some common pieces of data, not just
things that happen in exceptional cases: the URL of the webpage, command line
switches, active extension IDs, GPU vendor information, experiment/variations
information, etc.

## Redaction

Beware that certain on certain platforms (e.g. Android Webview) we
[sanitize the stack in the dump](https://cs.chromium.org/chromium/src/third_party/crashpad/crashpad/snapshot/sanitized/memory_snapshot_sanitized.h)
and only crash keys on an
[allowlist](https://cs.chromium.org/chromium/src/android_webview/common/crash_reporter/crash_keys.cc)
will be captured.

## Getting Started with a Single Key-Value Pair

Imagine you are investigating a crash, and you want to know the value of some
variable when the crash occurs; the crash key logging system enables you to do
just that.

#### 1. Declare the Crash Key

A crash key must be allocated using static storage duration, so that there is
space for the value to be set. This can be done as a static variable in the
global or function scope, or in an anonymous namespace:

    static crash_reporter::CrashKeyString<32> crash_key_one("one");

    namespace {
    crash_reporter::CrashKeyString<64> crash_key_two("two");
    }

    void DoSomething(const std::string& arg) {
      static crash_reporter::CrashKeyString<8> three("three");
      crash_key_two.Set(arg);
      three.Set("true");
    }

The template argument specifies the maximum length a value can be, and it
should include space for a trailing NUL byte. Values must be C-strings and
cannot have embedded NULs. The constructor argument is the name of the
crash key, and it is what you will use to identify your data in uploaded
crash reports.

If you need to declare an array of crash keys (e.g., for recording N values
of an array), you can use a constructor tag to avoid warnings about `explicit`:

    static ArrayItemKey = crash_reporter::CrashKeyString<32>;
    static ArrayItemKey crash_keys[] = {
      {"array-item-1", ArrayItemKey::Tag::kArray},
      {"array-item-2", ArrayItemKey::Tag::kArray},
      {"array-item-3", ArrayItemKey::Tag::kArray},
      {"array-item-4", ArrayItemKey::Tag::kArray},
    };

The crash key system will require your target to have a dependency on
`//components/crash/core/common:crash_key`. If you encounter link errors for
unresolved symbols to `crashpad::Annotation::SetSize(unsigned int)`, adding
the dependency will resolve them.

#### 2. Set the Crash Key

After a key has been allocated, its `Set(base::StringPiece)` and
`Clear()` methods can be used to record and clear a value. In addition,
crash_key.h provides a `ScopedCrashKeyString` class to set the value for the
duration of a scope and clear it upon exiting.

#### 3. Seeing the Data

Using <http://go/crash> (internal only), find the crash report signature related
to your bug, and click on the "N of M" reports link to drill down to
report-specific information. From there, select a report and go to the
"Product Data" section to view all the crash key-value pairs.

## Dealing with DEPS

Not all targets in the Chromium source tree are permitted to depend on the
`//components/crash/core/common:crash_key` target due to DEPS file
`include_rules`.

If the crash key being added is only a temporary debugging aid to track down a
crash, consider adding the dependency temporarily and removing it when done.
A specific include rule can be added for crash_key.h:

    # DEPS
    include_rules = [
      '+components/crash/core/common/crash_key.h',
    ]

Then simply remove it (and the BUILD.gn dependency) once the crash is resolved
and the crash key deleted.

If this crash key is more permanent, then there is an alternate API in //base
that can be used. This API is used by the //content module to set its permanent
crash key information. Note however that the base-level API is more limited in
terms of features and flexibility. See the header documentation in
[//base/debug/crash_logging.h](https://cs.chromium.org/chromium/src/base/debug/crash_logging.h)
for usage examples.

## Advanced Topics: Stack Traces

Now imagine a scenario where you have a use-after-free. The crash reports coming
in do not indicate where the object being used was initially freed, however,
just where it is later being dereferenced. To make debugging easier, it would be
nice to have the stack trace of the destructor, and the crash key system works
for that, too.

#### 1. Declare the Crash Key

Declaring the crash key is no different than written above, though special
attention should be paid to the maximum size argument, which will affect the
number of stack frames that are recorded. Typically a value of _1024_ is
recommended.

#### 2. Set the Crash Key

To set a stack trace to a crash key, use the `SetCrashKeyStringToStackTrace()`
function in crash_logging.h:

    Usemeafterfree::~Usemeafterfree() {
      static crash_reporter::CrashKeyString<1024> trace_key("uaf-dtor-trace");
      crash_reporter::SetCrashKeyStringToStackTrace(&trace_key,
                                                    base::debug::StackTrace());
    }

#### 3. Seeing the Data

Unlike with the previous example, a stack trace will just be a string of
hexadecimal addresses. To turn the addresses back into symbols use,
<http://go/crsym> (internal instance of <https://github.com/chromium/crsym/>).
Using the **Crash Key** input type, give it a crash report ID and the name of
your crash key. Crsym will then fetch the symbol data from the internal crash
processing backends and return a formatted, symbolized stack trace.