docs/testing/layout_tests.md

# Layout Tests

Layout tests are used by Blink to test many components, including but not
limited to layout and rendering. In general, layout tests involve loading pages
in a test renderer (`content_shell`) and comparing the rendered output or
JavaScript output against an expected output file.

[TOC]

## Running Layout Tests

### Initial Setup

Before you can run the layout tests, you need to build the `blink_tests` target
to get `content_shell` and all of the other needed binaries.

```bash
ninja -C out/Release blink_tests
```

On **Android** (layout test support
[currently limited to KitKat and earlier](https://crbug.com/567947)) you need to
build and install `content_shell_apk` instead. See also:
[Android Build Instructions](../android_build_instructions.md).

```bash
ninja -C out/Default content_shell_apk
adb install -r out/Default/apks/ContentShell.apk
```

On **Mac**, you probably want to strip the content_shell binary before starting
the tests. If you don't, you'll have 5-10 running concurrently, all stuck being
examined by the OS crash reporter. This may cause other failures like timeouts
where they normally don't occur.

```bash
strip ./xcodebuild/{Debug,Release}/content_shell.app/Contents/MacOS/content_shell
```

### Running the Tests

TODO: mention `testing/xvfb.py`

The test runner script is in
`third_party/WebKit/Tools/Scripts/run-webkit-tests`.

To specify which build directory to use (e.g. out/Default, out/Release,
out/Debug) you should pass the `-t` or `--target` parameter. For example, to
use the build in `out/Default`, use:

```bash
python third_party/WebKit/Tools/Scripts/run-webkit-tests -t Default
```

For Android (if your build directory is `out/android`):

```bash
python third_party/WebKit/Tools/Scripts/run-webkit-tests -t android --android
```

Tests marked as `[ Skip ]` in
[TestExpectations](../../third_party/WebKit/LayoutTests/TestExpectations)
won't be run at all, generally because they cause some intractable tool error.
To force one of them to be run, either rename that file or specify the skipped
test as the only one on the command line (see below).

Note that currently only the tests listed in
[SmokeTests](../../third_party/WebKit/LayoutTests/SmokeTests)
are run on the Android bots, since running all layout tests takes too long on
Android (and may still have some infrastructure issues). Most developers focus
their Blink testing on Linux. We rely on the fact that the Linux and Android
behavior is nearly identical for scenarios outside those covered by the smoke
tests.

To run only some of the tests, specify their directories or filenames as
arguments to `run_webkit_tests.py` relative to the layout test directory
(`src/third_party/WebKit/LayoutTests`). For example, to run the fast form tests,
use:

```bash
Tools/Scripts/run-webkit-tests fast/forms
```

Or you could use the following shorthand:

```bash
Tools/Scripts/run-webkit-tests fast/fo\*
```

*** promo
Example: To run the layout tests with a debug build of `content_shell`, but only
test the SVG tests and run pixel tests, you would run:

```bash
Tools/Scripts/run-webkit-tests -t Default svg
```
***

As a final quick-but-less-robust alternative, you can also just use the
content_shell executable to run specific tests by using (for Windows):

```bash
out/Default/content_shell.exe --run-layout-test --no-sandbox full_test_source_path
```

as in:

```bash
out/Default/content_shell.exe --run-layout-test --no-sandbox \
    c:/chrome/src/third_party/WebKit/LayoutTests/fast/forms/001.html
```

but this requires a manual diff against expected results, because the shell
doesn't do it for you.

To see a complete list of arguments supported, run: `run-webkit-tests --help`

*** note
**Linux Note:** We try to match the Windows render tree output exactly by
matching font metrics and widget metrics. If there's a difference in the render
tree output, we should see if we can avoid rebaselining by improving our font
metrics. For additional information on Linux Layout Tests, please see
[docs/layout_tests_linux.md](../layout_tests_linux.md).
***

*** note
**Mac Note:** While the tests are running, a bunch of Appearance settings are
overridden for you so the right type of scroll bars, colors, etc. are used.
Your main display's "Color Profile" is also changed to make sure color
correction by ColorSync matches what is expected in the pixel tests. The change
is noticeable, how much depends on the normal level of correction for your
display. The tests do their best to restore your setting when done, but if
you're left in the wrong state, you can manually reset it by going to
System Preferences → Displays → Color and selecting the "right" value.
***

### Test Harness Options

This script has a lot of command line flags. You can pass `--help` to the script
to see a full list of options. A few of the most useful options are below:

| Option                      | Meaning |
|:----------------------------|:--------------------------------------------------|
| `--debug`                   | Run the debug build of the test shell (default is release). Equivalent to `-t Debug` |
| `--nocheck-sys-deps`        | Don't check system dependencies; this allows faster iteration. |
| `--verbose`                 |	Produce more verbose output, including a list of tests that pass. |
| `--no-pixel-tests`          | Disable the pixel-to-pixel PNG comparisons and image checksums for tests that don't call `testRunner.dumpAsText()` |
| `--reset-results`           |	Write all generated results directly into the given directory, overwriting what's there. |
| `--new-baseline`            |	Write all generated results into the most specific platform directory, overwriting what's there. Equivalent to `--reset-results --add-platform-expectations` |
| `--renderer-startup-dialog` | Bring up a modal dialog before running the test, useful for attaching a debugger. |
| `--fully-parallel`          | Run tests in parallel using as many child processes as the system has cores. |
| `--driver-logging`          | Print C++ logs (LOG(WARNING), etc).  |

## Success and Failure

A test succeeds when its output matches the pre-defined expected results. If any
tests fail, the test script will place the actual generated results, along with
a diff of the actual and expected results, into
`src/out/Default/layout_test_results/`, and by default launch a browser with a
summary and link to the results/diffs.

The expected results for tests are in the
`src/third_party/WebKit/LayoutTests/platform` or alongside their respective
tests.

*** note
Tests which use [testharness.js](https://github.com/w3c/testharness.js/)
do not have expected result files if all test cases pass.
***

A test that runs but produces the wrong output is marked as "failed", one that
causes the test shell to crash is marked as "crashed", and one that takes longer
than a certain amount of time to complete is aborted and marked as "timed out".
A row of dots in the script's output indicates one or more tests that passed.

## Test expectations

The
[TestExpectations](../../WebKit/LayoutTests/TestExpectations) file (and related
files, including
[skia_test_expectations.txt](../../skia/skia_test_expectations.txt))
contains the list of all known layout test failures. See
[Test Expectations](https://sites.google.com/a/chromium.org/dev/developers/testing/webkit-layout-tests/testexpectations)
for more on this.

## Testing Runtime Flags

There are two ways to run layout tests with additional command-line arguments:

* Using `--additional-driver-flag`:

  ```bash
  run-webkit-tests --additional-driver-flag=--blocking-repaint
  ```

  This tells the test harness to pass `--blocking-repaint` to the
  content_shell binary.

  It will also look for flag-specific expectations in
  `LayoutTests/FlagExpectations/blocking-repaint`, if this file exists. The
  suppressions in this file override the main TestExpectations file.

* Using a *virtual test suite* defined in
  [LayoutTests/VirtualTestSuites](https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/LayoutTests/VirtualTestSuites).
  A virtual test suite runs a subset of layout tests under a specific path with
  additional flags. For example, you could test a (hypothetical) new mode for
  repainting using the following virtual test suite:

  ```json
  {
    "prefix": "blocking_repaint",
    "base": "fast/repaint",
    "args": ["--blocking-repaint"],
  }
  ```

  This will create new "virtual" tests of the form
  `virtual/blocking_repaint/fast/repaint/...`` which correspond to the files
  under `LayoutTests/fast/repaint` and pass `--blocking-repaint` to
  content_shell when they are run.

  These virtual tests exist in addition to the original `fast/repaint/...`
  tests. They can have their own expectations in TestExpectations, and their own
  baselines.  The test harness will use the non-virtual baselines as a fallback.
  However, the non-virtual expectations are not inherited: if
  `fast/repaint/foo.html` is marked `[ Fail ]`, the test harness still expects
  `virtual/blocking_repaint/fast/repaint/foo.html` to pass. If you expect the
  virtual test to also fail, it needs its own suppression.

  The "prefix" value does not have to be unique. This is useful if you want to
  run multiple directories with the same flags (but see the notes below about
  performance). Using the same prefix for different sets of flags is not
  recommended.

For flags whose implementation is still in progress, virtual test suites and
flag-specific expectations represent two alternative strategies for testing.
Consider the following when choosing between them:

* The
  [waterfall builders](https://dev.chromium.org/developers/testing/chromium-build-infrastructure/tour-of-the-chromium-buildbot)
  and [try bots](https://dev.chromium.org/developers/testing/try-server-usage)
  will run all virtual test suites in addition to the non-virtual tests.
  Conversely, a flag-specific expectations file won't automatically cause the
  bots to test your flag - if you want bot coverage without virtual test suites,
  you will need to set up a dedicated bot for your flag.

* Due to the above, virtual test suites incur a performance penalty for the
  commit queue and the continuous build infrastructure. This is exacerbated by
  the need to restart `content_shell` whenever flags change, which limits
  parallelism. Therefore, you should avoid adding large numbers of virtual test
  suites. They are well suited to running a subset of tests that are directly
  related to the feature, but they don't scale to flags that make deep
  architectural changes that potentially impact all of the tests.

## Tracking Test Failures

All bugs, associated with layout test failures must have the
[Test-Layout](https://crbug.com/?q=label:Test-Layout) label. Depending on how
much you know about the bug, assign the status accordingly:

* **Unconfirmed** -- You aren't sure if this is a simple rebaseline, possible
  duplicate of an existing bug, or a real failure
* **Untriaged** -- Confirmed but unsure of priority or root cause.
* **Available** -- You know the root cause of the issue.
* **Assigned** or **Started** -- You will fix this issue.

When creating a new layout test bug, please set the following properties:

* Components: a sub-component of Blink
* OS: **All** (or whichever OS the failure is on)
* Priority: 2 (1 if it's a crash)
* Type: **Bug**
* Labels: **Test-Layout**

You can also use the _Layout Test Failure_ template, which will pre-set these
labels for you.

## Writing Layout Tests

###  Pixel Tests

TODO: Write documentation here.

### Reference Tests

TODO: Write documentation here.

### Script Tests

These tests use a JavaScript test harness and test cases written in script to
exercise features and make assertions about the behavior. Generally, new tests
are written using the [testharness.js](https://github.com/w3c/testharness.js/)
test harness, which is also heavily used in the cross-vendor
[web-platform-tests](https://github.com/w3c/web-platform-tests) project. Tests
written with testharness.js generally look something like the following:

```html
<!DOCTYPE html>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<script>
test(t => {
  var x = true;
  assert_true(x);
}, "Truth is true.");
</script>
```

Many older tests are written using the **js-test**
(`LayoutTests/resources/js-test.js`) test harness. This harness is
**deprecated**, and should not be used for new tests. The tests call
`testRunner.dumpAsText()` to signal that the page content should be dumped and
compared against an \*-expected.txt file, and optionally
`testRunner.waitUntilDone()` and `testRunner.notifyDone()` for asynchronous
tests.

### Tests that use a HTTP Server

By default, tests are loaded as if via `file:` URLs. Some web platform features
require tests served via HTTP or HTTPS, for example relative paths (`src=/foo`)
or features restricted to secure protocols.

HTTP tests are those tests that are under `LayoutTests/http/tests` (or virtual
variants). Use a locally running HTTP server (Apache) to run. Tests are served
off of ports 8000, 8080 for HTTP and 8443 for HTTPS. If you run the tests using
`run-webkit-tests`, the server will be started automatically.To run the server
manually to reproduce or debug a failure:

```bash
cd src/third_party/WebKit/Tools/Scripts
run-blink-httpd start
```

The layout tests will be served from `http://127.0.0.1:8000`. For example, to
run the test `http/tests/serviceworker/chromium/service-worker-allowed.html`,
navigate to
`http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
tests will behave differently if you go to 127.0.0.1 instead of localhost, so
use 127.0.0.1.

To kill the server, run `run-blink-httpd --server stop`, or just use `taskkill`
or the Task Manager on Windows, and `killall` or Activity Monitor on MacOS.

The test server sets up an alias to `LayoutTests/resources` directory. In HTTP
tests, you can access the testing framework at e.g.
`src="/js-test-resources/js-test.js"`.

### Writing tests that need to paint, raster, or draw a frame of intermediate output

A layout test does not actually draw frames of output until the test exits. If
it is required to generate a painted frame, then use
`window.testRunner.displayAsyncThen`, which will run the machinery to put up a
frame, then call the passed callback. There is also a library at
`fast/repaint/resources/text-based-repaint.js` to help with writing paint
invalidation and repaint tests.

#### Layout test support for `testRunner`

Some layout tests rely on the testRunner object to expose configuration for
mocking the platform. This is provided in content_shell, here's a UML diagram of
testRunner bindings configuring platform implementation:

[![UML of testRunner bindings configuring platform implementation](https://docs.google.com/drawings/u/1/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/export/svg?id=1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg&pageid=p)](https://docs.google.com/drawings/d/1KNRNjlxK0Q3Tp8rKxuuM5mpWf4OJQZmvm9_kpwu_Wwg/edit)

[Writing reliable layout tests](https://docs.google.com/document/d/1Yl4SnTLBWmY1O99_BTtQvuoffP8YM9HZx2YPkEsaduQ/edit)

## Debugging Layout Tests

After the layout tests run, you should get a summary of tests that pass or fail.
If something fails unexpectedly (a new regression), you will get a content_shell
window with a summary of the unexpected failures. Or you might have a failing
test in mind to investigate. In any case, here are some steps and tips for
finding the problem.

* Take a look at the result. Sometimes tests just need to be rebaselined (see
  below) to account for changes introduced in your patch.
    * Load the test into a trunk Chrome or content_shell build and look at its
      result. (For tests in the http/ directory, start the http server first.
      See above. Navigate to `http://localhost:8000/` and proceed from there.)
      The best tests describe what they're looking for, but not all do, and
      sometimes things they're not explicitly testing are still broken. Compare
      it to Safari, Firefox, and IE if necessary to see if it's correct. If
      you're still not sure, find the person who knows the most about it and
      ask.
    * Some tests only work properly in content_shell, not Chrome, because they
      rely on extra APIs exposed there.
    * Some tests only work properly when they're run in the layout-test
      framework, not when they're loaded into content_shell directly. The test
      should mention that in its visible text, but not all do. So try that too.
      See "Running the tests", above.
* If you think the test is correct, confirm your suspicion by looking at the
  diffs between the expected result and the actual one.
    * Make sure that the diffs reported aren't important. Small differences in
      spacing or box sizes are often unimportant, especially around fonts and
      form controls. Differences in wording of JS error messages are also
      usually acceptable.
    * `./run_webkit_tests.py path/to/your/test.html --full-results-html` will
      produce a page including links to the expected result, actual result, and
      diff.
    * Add the `--sources` option to `run_webkit_tests.py` to see exactly which
      expected result it's comparing to (a file next to the test, something in
      platform/mac/, something in platform/chromium-win/, etc.)
    * If you're still sure it's correct, rebaseline the test (see below).
      Otherwise...
* If you're lucky, your test is one that runs properly when you navigate to it
  in content_shell normally. In that case, build the Debug content_shell
  project, fire it up in your favorite debugger, and load the test file either
  from a file:// URL.
    * You'll probably be starting and stopping the content_shell a lot. In VS,
      to save navigating to the test every time, you can set the URL to your
      test (file: or http:) as the command argument in the Debugging section of
      the content_shell project Properties.
    * If your test contains a JS call, DOM manipulation, or other distinctive
      piece of code that you think is failing, search for that in the Chrome
      solution. That's a good place to put a starting breakpoint to start
      tracking down the issue.
    * Otherwise, you're running in a standard message loop just like in Chrome.
      If you have no other information, set a breakpoint on page load.
* If your test only works in full layout-test mode, or if you find it simpler to
  debug without all the overhead of an interactive session, start the
  content_shell with the command-line flag `--run-layout-test`, followed by the
  URL (file: or http:) to your test. More information about running layout tests
  in content_shell can be found
  [here](https://www.chromium.org/developers/testing/webkit-layout-tests/content-shell).
    * In VS, you can do this in the Debugging section of the content_shell
      project Properties.
    * Now you're running with exactly the same API, theme, and other setup that
      the layout tests use.
    * Again, if your test contains a JS call, DOM manipulation, or other
      distinctive piece of code that you think is failing, search for that in
      the Chrome solution. That's a good place to put a starting breakpoint to
      start tracking down the issue.
    * If you can't find any better place to set a breakpoint, start at the
      `TestShell::RunFileTest()` call in `content_shell_main.cc`, or at
      `shell->LoadURL() within RunFileTest()` in `content_shell_win.cc`.
* Debug as usual. Once you've gotten this far, the failing layout test is just a
  (hopefully) reduced test case that exposes a problem.

### Debugging HTTP Tests

To run the server manually to reproduce/debug a failure:

```bash
cd src/third_party/WebKit/Tools/Scripts
run-blink-httpd start
```

The layout tests will be served from `http://127.0.0.1:8000`. For example, to
run the test
`LayoutTest/http/tests/serviceworker/chromium/service-worker-allowed.html`,
navigate to
`http://127.0.0.1:8000/serviceworker/chromium/service-worker-allowed.html`. Some
tests will behave differently if you go to 127.0.0.1 vs localhost, so use
127.0.0.1.

To kill the server, run `run-blink-httpd --server stop`, or just use `taskkill`
or the Task Manager on Windows, and `killall` or Activity Monitor on MacOS.

The test server sets up an alias to `LayoutTests/resources` directory. In HTTP
tests, you can access testing framework at e.g.
`src="/js-test-resources/js-test.js"`.

### Tips

Check https://test-results.appspot.com/ to see how a test did in the most recent
~100 builds on each builder (as long as the page is being updated regularly).

A timeout will often also be a text mismatch, since the wrapper script kills the
content_shell before it has a chance to finish. The exception is if the test
finishes loading properly, but somehow hangs before it outputs the bit of text
that tells the wrapper it's done.

Why might a test fail (or crash, or timeout) on buildbot, but pass on your local
machine?
* If the test finishes locally but is slow, more than 10 seconds or so, that
  would be why it's called a timeout on the bot.
* Otherwise, try running it as part of a set of tests; it's possible that a test
  one or two (or ten) before this one is corrupting something that makes this
  one fail.
* If it consistently works locally, make sure your environment looks like the
  one on the bot (look at the top of the stdio for the webkit_tests step to see
  all the environment variables and so on).
* If none of that helps, and you have access to the bot itself, you may have to
  log in there and see if you can reproduce the problem manually.

### Debugging Inspector Tests

* Add `window.debugTest = true;` to your test code as follows:

  ```javascript
  window.debugTest = true;
  function test() {
    /* TEST CODE */
  }
  ```

* Do one of the following:
    * Option A) Run from the chromium/src folder:
      `blink/tools/run_layout_tests.sh
      --additional_driver_flag='--remote-debugging-port=9222'
      --time-out-ms=6000000`
    * Option B) If you need to debug an http/tests/inspector test, start httpd
      as described above. Then, run content_shell:
      `out/Default/content_shell --remote-debugging-port=9222 --run-layout-test
      http://127.0.0.1:8000/path/to/test.html`
* Open `http://localhost:9222` in a stable/beta/canary Chrome, click the single
  link to open the devtools with the test loaded.
* You may need to replace devtools.html with inspector.html in your URL (or you
  can use local chrome inspection of content_shell from chrome://inspect
  instead)
* In the loaded devtools, set any required breakpoints and execute `test()` in
  the console to actually start the test.

## Rebaselining Layout Tests

_To automatically re-baseline tests across all Chromium platforms, using the
buildbot results, see the
[Rebaselining keywords in TestExpectations](https://www.chromium.org/developers/testing/webkit-layout-tests/testexpectations#TOC-Rebaselining)
and
[Rebaselining Tool](https://trac.webkit.org/wiki/Rebaseline).
Alternatively, to manually run and test and rebaseline it on your workstation,
read on._

By default, text-only tests (ones that call `testRunner.dumpAsText()`) produce
only text results. Other tests produce both new text results and new image
results (the image baseline comprises two files, `-expected.png` and
  `-expected.checksum`). So you'll need either one or three `-expected.\*` files
in your new baseline, depending on whether you have a text-only test or not. If
you enable `--no-pixel-tests`, only new text results will be produced, even for
tests that do image comparisons.

```bash
cd src/third_party/WebKit
Tools/Scripts/run-webkit-tests --new-baseline foo/bar/test.html
```

The above command will generate a new baseline for
`LayoutTests/foo/bar/test.html` and put the output files in the right place,
e.g.
`LayoutTests/platform/chromium-win/LayoutTests/foo/bar/test-expected.{txt,png,checksum}`.

When you rebaseline a test, make sure your commit description explains why the
test is being re-baselined. If this is a special case (i.e., something we've
decided to be different with upstream), please put a README file next to the new
expected output explaining the difference.

## W3C Tests

In addition to layout tests developed and run just by the Blink team, there are
also W3C conformance tests. For more info, see
[Importing the W3C Tests](https://www.chromium.org/blink/importing-the-w3c-tests).

## Known Issues

See
[bugs with the component Blink>Infra](https://bugs.chromium.org/p/chromium/issues/list?can=2&q=component%3ABlink%3EInfra)
for issues related to Blink tools, include the layout test runner.

* Windows and Linux: Do not copy and paste while the layout tests are running,
  as it may interfere with the editing/pasteboard and other clipboard-related
  tests. (Mac tests swizzle NSClipboard to avoid any conflicts).
* If QuickTime is not installed, the plugin tests
  `fast/dom/object-embed-plugin-scripting.html` and
  `plugins/embed-attributes-setting.html` are expected to fail.