Expand docs around pixel testing, esp. running on Android.
Tbr: [email protected]
No-Try: True
Bug: 865957
Change-Id: I219fb646bc76265e0a42b7ffbcd47bd99d71d127
Reviewed-on: https://chromium-review.googlesource.com/c/1297984
Commit-Queue: Kenneth Russell <[email protected]>
Reviewed-by: Kenneth Russell <[email protected]>
Cr-Commit-Position: refs/heads/master@{#602456}diff --git a/docs/gpu/gpu_testing.md b/docs/gpu/gpu_testing.md
index c3139047..c071538 100644
--- a/docs/gpu/gpu_testing.md
+++ b/docs/gpu/gpu_testing.md
@@ -196,19 +196,24 @@
* `run_gpu_integration_test.py screenshot_sync --browser=release`
* `run_gpu_integration_test.py trace_test --browser=release`
+If you're testing on Android and have built and deployed
+`ChromePublic.apk` to the device, use `--browser=android-chromium` to
+invoke it.
+
**Note:** If you are on Linux and see this test harness exit immediately with
`**Non zero exit code**`, it's probably because of some incompatible Python
packages being installed. Please uninstall the `python-egenix-mxdatetime` and
-`python-logilab-common` packages in this case; see
-[Issue 716241](http://crbug.com/716241).
+`python-logilab-common` packages in this case; see [Issue
+716241](http://crbug.com/716241). This should not be happening any more since
+the GPU tests were switched to use the infra team's `vpython` harness.
-You can also run a subset of tests with this harness:
+You can run a subset of tests with this harness:
* `run_gpu_integration_test.py webgl_conformance --browser=release
--test-filter=conformance_attribs`
Figuring out the exact command line that was used to invoke the test on the
-bots can be a little tricky. The bots all\* run their tests via Swarming and
+bots can be a little tricky. The bots all run their tests via Swarming and
isolates, meaning that the invocation of a step like `[trigger]
webgl_conformance_tests on NVIDIA GPU...` will look like:
@@ -227,8 +232,8 @@
'--extra-browser-args=--enable-logging=stderr --js-flags=--expose-gc'
'--isolated-script-test-output=${ISOLATED_OUTDIR}/output.json'`
-You can leave off the --isolated-script-test-output argument, so this would
-leave a full command line of:
+You can leave off the --isolated-script-test-output argument, because that's
+used only by wrapper scripts, so this would leave a full command line of:
* `run_gpu_integration_test.py
webgl_conformance --show-stdout '--browser=release' -v
@@ -240,12 +245,45 @@
[Cloud Storage Credentials]: gpu_testing_bot_details.md#Cloud-storage-credentials
-Pixel tests use reference images from cloud storage, bots pass
-`--upload-refimg-to-cloud-storage` argument, but to run locally you need to pass
-`--download-refimg-from-cloud-storage` argument, as well as other arguments bot
-uses, like `--refimg-cloud-storage-bucket` and `--os-type`.
+### Running the pixel tests locally
-Sample command line for Android:
+The pixel tests run in a few different modes:
+
+* The waterfall bots generate reference images into cloud storage, and pass
+ the `--upload-refimg-to-cloud-storage` command line argument.
+* The trybots use the reference images that were generated by the waterfall
+ bots. They pass the `--download-refimg-from-cloud-storage` command line
+ argument, as well as other needed ones like `--refimg-cloud-storage-bucket`
+ and `--os-type`.
+* When run locally, the first time the pixel tests are run, generated
+ *reference* images are placed into
+ `src/content/test/data/gpu/gpu_reference/`. The second and subsequent times,
+ if tests fail, failure images will be placed into
+ `src/content/test/data/gpu/generated`.
+
+It's possible to make your local pixel tests download the reference images from
+cloud storage, if your workstation has the same OS and GPU type as one of the
+bots on the waterfall, and you pass the `--download-refimg-from-cloud-storage`,
+`--refimg-cloud-storage-bucket`, `--os-type` and `--build-revision` command line
+arguments.
+
+Example command line for running the pixel tests locally on a desktop
+platform, where the Chrome build is in out/Release:
+
+* `run_gpu_integration_test.py pixel --browser=release`
+
+Running against a connected Android device where ChromePublic.apk has
+already been deployed:
+
+* `run_gpu_integration_test.py pixel --browser=android-chromium`
+
+You can run a subset of the pixel tests via the --test-filter argument, which
+takes a regex:
+
+* `run_gpu_integration_test.py pixel --browser=release --test-filter=Pixel_WebGL`
+* `run_gpu_integration_test.py pixel --browser=release --test-filter=\(Pixel_WebGL2\|Pixel_GpuRasterization_BlueBox\)`
+
+More complete example command line for Android:
* `run_gpu_integration_test.py pixel --show-stdout --browser=android-chromium
-v --passthrough --extra-browser-args='--enable-logging=stderr
@@ -253,33 +291,6 @@
chromium-gpu-archive/reference-images --os-type android
--download-refimg-from-cloud-storage`
-<!-- XXX: update this section; these isolates don't exist anymore -->
-You can find the isolates for the various tests in
-[src/chrome/](http://src.chromium.org/viewvc/chrome/trunk/src/chrome/):
-
-* [angle_unittests.isolate](https://chromium.googlesource.com/chromium/src/+/master/chrome/angle_unittests.isolate)
-* [content_gl_tests.isolate](https://chromium.googlesource.com/chromium/src/+/master/content/content_gl_tests.isolate)
-* [gl_tests.isolate](https://chromium.googlesource.com/chromium/src/+/master/chrome/gl_tests.isolate)
-* [gles2_conform_test.isolate](https://chromium.googlesource.com/chromium/src/+/master/chrome/gles2_conform_test.isolate)
-* [tab_capture_end2end_tests.isolate](https://chromium.googlesource.com/chromium/src/+/master/chrome/tab_capture_end2end_tests.isolate)
-* [telemetry_gpu_test.isolate](https://chromium.googlesource.com/chromium/src/+/master/chrome/telemetry_gpu_test.isolate)
-
-The isolates contain the full or partial command line for invoking the target.
-The complete command line for any test can be deduced from the contents of the
-isolate plus the stdio output from the test's run on the bot.
-
-Note that for the GN build, the isolates are simply described by build targets,
-and [gn_isolate_map.pyl] describes the mapping between isolate name and build
-target, as well as the command line used to invoke the isolate. Once all
-platforms have switched to GN, the .isolate files will be obsolete and be
-removed.
-
-(\* A few of the one-off GPU configurations on the chromium.gpu.fyi waterfall
-run their tests locally rather than via swarming, in order to decrease the
-number of physical machines needed.)
-
-[gn_isolate_map.pyl]: https://chromium.googlesource.com/chromium/src/+/master/testing/buildbot/gn_isolate_map.pyl
-
## Running Binaries from the Bots Locally
Any binary run remotely on a bot can also be run locally, assuming the local
@@ -479,6 +490,18 @@
`linux_optional_gpu_tests_rel` tryservers to help ensure that they won't
break the FYI bots.
+## Debugging Pixel Test Failures on the GPU Bots
+
+If pixel tests fail on the bots, the stdout will contain text like:
+
+`See http://chromium-browser-gpu-tests.commondatastorage.googleapis.com/view_test_results.html?[HASH]`
+
+This link contains all of the failing tests' generated and reference
+images, and is useful for figuring out exactly what went wrong. [Issue
+898649](http://crbug.com/898649) tracks improving this user interface,
+so that the failures can be surfaced directly in the build logs rather
+than having to dig through stdout.
+
## Updating and Adding New Pixel Tests to the GPU Bots
Adding new pixel tests which require reference images is a slightly more
