blob: 714a22997e20050ce9af7660e53702a3dee6441b [file] [log] [blame] [view]
andybons3322f762015-08-24 21:37:091# Android Test Instructions
2
andybons3322f762015-08-24 21:37:093[TOC]
4
jbudorick25c17382016-08-03 18:53:075## Device Setup
6
7### Physical Device Setup
8
9#### ADB Debugging
andybons3322f762015-08-24 21:37:0910
11In order to allow the ADB to connect to the device, you must enable USB
12debugging:
andybons22afb312015-08-31 02:24:5113
14* Before Android 4.1 (Jelly Bean):
15 * Go to "System Settings"
16 * Go to "Developer options"
17 * Check "USB debugging".
18 * Un-check "Verify apps over USB".
ntfschr09699f442017-01-12 22:20:4619* On Jelly Bean and above, developer options are hidden by default. To unhide
20 them:
andybons22afb312015-08-31 02:24:5121 * Go to "About phone"
22 * Tap 10 times on "Build number"
23 * The "Developer options" menu will now be available.
24 * Check "USB debugging".
25 * Un-check "Verify apps over USB".
andybons3322f762015-08-24 21:37:0926
jbudorick25c17382016-08-03 18:53:0727#### Screen
andybons3322f762015-08-24 21:37:0928
ntfschr09699f442017-01-12 22:20:4629You **must** ensure that the screen stays on while testing: `adb shell svc power
30stayon usb` Or do this manually on the device: Settings -> Developer options ->
31Stay Awake.
andybons3322f762015-08-24 21:37:0932
33If this option is greyed out, stay awake is probably disabled by policy. In that
34case, get another device or log in with a normal, unmanaged account (because the
35tests will break in exciting ways if stay awake is off).
36
jbudorick25c17382016-08-03 18:53:0737#### Enable Asserts
andybons3322f762015-08-24 21:37:0938
ntfschr09699f442017-01-12 22:20:4639```
40adb shell setprop debug.assert 1
41```
andybons3322f762015-08-24 21:37:0942
jbudorick25c17382016-08-03 18:53:0743#### Disable Verify Apps
andybons3322f762015-08-24 21:37:0944
ntfschr09699f442017-01-12 22:20:4645You may see a dialog like [this
46one](http://www.samsungmobileusa.com/simulators/ATT_GalaxyMega/mobile/screens/06-02_12.jpg),
andybons3322f762015-08-24 21:37:0947which states, _Google may regularly check installed apps for potentially harmful
48behavior._ This can interfere with the test runner. To disable this dialog, run:
ntfschr09699f442017-01-12 22:20:4649
50```
51adb shell settings put global package_verifier_enable 0
52```
andybons3322f762015-08-24 21:37:0953
jbudorick25c17382016-08-03 18:53:0754### Emulator Setup
andybons3322f762015-08-24 21:37:0955
jbudorick25c17382016-08-03 18:53:0756#### Option 1
andybons3322f762015-08-24 21:37:0957
58Use an emulator (i.e. Android Virtual Device, AVD): Enabling Intel's
59Virtualizaton support provides the fastest, most reliable emulator configuration
ntfschr09699f442017-01-12 22:20:4660available (i.e. x86 emulator with GPU acceleration and KVM support). Remember to
Nick Harper8a13e822017-05-19 23:45:4361build with `target_cpu = "x86"` for x86. Otherwise installing the APKs will fail
ntfschr09699f442017-01-12 22:20:4662with `INSTALL_FAILED_NO_MATCHING_ABIS`.
andybons3322f762015-08-24 21:37:0963
641. Enable Intel Virtualization support in the BIOS.
65
662. Set up your environment:
67
68 ```shell
69 . build/android/envsetup.sh
70 ```
71
723. Install emulator deps:
73
74 ```shell
thestigf51ad752015-11-11 18:08:4975 build/android/install_emulator_deps.py --api-level=23
andybons3322f762015-08-24 21:37:0976 ```
77
78 This script will download Android SDK and place it a directory called
79 android\_tools in the same parent directory as your chromium checkout. It
80 will also download the system-images for the emulators (i.e. arm and x86).
81 Note that this is a different SDK download than the Android SDK in the
jbudorick25c17382016-08-03 18:53:0782 chromium source checkout (i.e. `src/third_party/android_emulator_sdk`).
andybons3322f762015-08-24 21:37:0983
844. Run the avd.py script. To start up _num_ emulators use -n. For non-x86 use
85 --abi.
86
87 ```shell
thestigf51ad752015-11-11 18:08:4988 build/android/avd.py --api-level=23
andybons3322f762015-08-24 21:37:0989 ```
90
91 This script will attempt to use GPU emulation, so you must be running the
ntfschr09699f442017-01-12 22:20:4692 emulators in an environment with hardware rendering available. See `avd.py
93 --help` for more details.
andybons3322f762015-08-24 21:37:0994
jbudorick25c17382016-08-03 18:53:0795#### Option 2
andybons3322f762015-08-24 21:37:0996
jbudorick25c17382016-08-03 18:53:0797Alternatively, you can create and run your own emulator using the tools provided
andybons3322f762015-08-24 21:37:0998by the Android SDK. When doing so, be sure to enable GPU emulation in hardware
99settings, since Chromium requires it to render.
100
101## Building Tests
102
jbudorick25c17382016-08-03 18:53:07103If you're adding a new test file, you'll need to explicitly add it to a gn
ntfschr09699f442017-01-12 22:20:46104target. If you're adding a test to an existing file, you won't need to make gn
jbudorick25c17382016-08-03 18:53:07105changes, but you may be interested in where your test winds up. In either case,
106here are some guidelines for where a test belongs:
andybons3322f762015-08-24 21:37:09107
jbudorick25c17382016-08-03 18:53:07108### C++
andybons3322f762015-08-24 21:37:09109
jbudorick25c17382016-08-03 18:53:07110C++ test files typically belong in `<top-level directory>_unittests` (e.g.
111`base_unittests` for `//base`). There are a few exceptions -- browser tests are
112typically their own target (e.g. `content_browsertests` for `//content`, or
113`browser_tests` for `//chrome`), and some unit test suites are broken at the
114second directory rather than the top-level one.
115
116### Java
117
118Java test files vary a bit more widely than their C++ counterparts:
119
ntfschr09699f442017-01-12 22:20:46120- Instrumentation test files -- i.e., tests that will run on a device --
121 typically belong in either `<top-level directory>_javatests` or `<top-level
122 directory>_test_java`. Regardless, they'll wind up getting packaged into one
123 of a few test APKs:
ctzsm34f54d62017-04-21 17:04:07124 - `webview_instrumentation_test_apk` for anything in `//android_webview`
ntfschr09699f442017-01-12 22:20:46125 - `content_shell_test_apk` for anything in `//content` or below
126 - `chrome_public_test_apk` for most things in `//chrome`
127 - `chrome_sync_shell_test_apk` in a few exceptional cases
128- JUnit or Robolectric test files -- i.e., tests that will run on the host --
129 typically belong in `<top-level directory>_junit_tests` (e.g.
130 `base_junit_tests` for `//base`), though here again there are cases
131 (particularly in `//components`) where suites are split at the second
132 directory rather than the top-level one.
andybons3322f762015-08-24 21:37:09133
134Once you know what to build, just do it like you normally would build anything
newt17e4d242015-08-27 09:07:26135else, e.g.: `ninja -C out/Release chrome_public_test_apk`
andybons3322f762015-08-24 21:37:09136
137## Running Tests
138
jbudorick25c17382016-08-03 18:53:07139All functional tests should be runnable via the wrapper scripts generated at
140build time:
141
142```sh
143<output directory>/bin/run_<target_name> [options]
144```
145
146Note that tests are sharded across all attached devices unless explicitly told
147to do otherwise by `-d/--device`.
andybons3322f762015-08-24 21:37:09148
149The commands used by the buildbots are printed in the logs. Look at
xiaoyin.l1003c0b2016-12-06 02:51:17150https://build.chromium.org/ to duplicate the same test command as a particular
andybons3322f762015-08-24 21:37:09151builder.
152
jbudorick25c17382016-08-03 18:53:07153### INSTALL\_FAILED\_CONTAINER\_ERROR or INSTALL\_FAILED\_INSUFFICIENT\_STORAGE
andybons3322f762015-08-24 21:37:09154
jbudorick25c17382016-08-03 18:53:07155If you see this error when the test runner is attempting to deploy the test
andybons3322f762015-08-24 21:37:09156binaries to the AVD emulator, you may need to resize your userdata partition
157with the following commands:
158
159```shell
davve7ae32cd2015-09-22 06:54:09160# Resize userdata partition to be 1G
mikecase7c263052017-03-30 23:46:11161resize2fs android_emulator_sdk/sdk/system-images/android-25/x86/userdata.img 1G
andybons3322f762015-08-24 21:37:09162
163# Set filesystem parameter to continue on errors; Android doesn't like some
164# things e2fsprogs does.
mikecase7c263052017-03-30 23:46:11165tune2fs -e continue android_emulator_sdk/sdk/system-images/android-25/x86/userdata.img
andybons3322f762015-08-24 21:37:09166```
167
168## Symbolizing Crashes
169
ntfschr09699f442017-01-12 22:20:46170Crash stacks are logged and can be viewed using `adb logcat`. To symbolize the
jyasskinc1c76ff2015-12-18 21:39:52171traces, define `CHROMIUM_OUTPUT_DIR=$OUTDIR` where `$OUTDIR` is the argument you
172pass to `ninja -C`, and pipe the output through
173`third_party/android_platform/development/scripts/stack`. If
174`$CHROMIUM_OUTPUT_DIR` is unset, the script will search `out/Debug` and
175`out/Release`. For example:
176
177```shell
178# If you build with
179ninja -C out/Debug chrome_public_test_apk
180# You can run:
181adb logcat -d | third_party/android_platform/development/scripts/stack
182
183# If you build with
184ninja -C out/android chrome_public_test_apk
185# You can run:
186adb logcat -d | CHROMIUM_OUTPUT_DIR=out/android third_party/android_platform/development/scripts/stack
187# or
188export CHROMIUM_OUTPUT_DIR=out/android
189adb logcat -d | third_party/android_platform/development/scripts/stack
190```
andybons3322f762015-08-24 21:37:09191
mlamouri0fbd6cd2015-10-26 12:08:11192## JUnit tests
193
194JUnit tests are Java unittests running on the host instead of the target device.
195They are faster to run and therefore are recommended over instrumentation tests
196when possible.
197
198The JUnits tests are usually following the pattern of *target*\_junit\_tests,
199for example, `content_junit_tests` and `chrome_junit_tests`.
200
201When adding a new JUnit test, the associated `BUILD.gn` file must be updated.
202For example, adding a test to `chrome_junit_tests` requires to update
203`chrome/android/BUILD.gn`. If you are a GYP user, you will not need to do that
204step in order to run the test locally but it is still required for GN users to
205run the test.
206
207```shell
208# Build the test suite.
asvitkinea7fdb062017-02-14 20:57:14209ninja -C out/my_build chrome_junit_tests
mlamouri0fbd6cd2015-10-26 12:08:11210
211# Run the test suite.
asvitkinea7fdb062017-02-14 20:57:14212BUILDTYPE=my_build build/android/test_runner.py junit -s chrome_junit_tests -vvv
mlamouri0fbd6cd2015-10-26 12:08:11213
214# Run a subset of tests. You might need to pass the package name for some tests.
asvitkinea7fdb062017-02-14 20:57:14215BUILDTYPE=my_build build/android/test_runner.py junit -s chrome_junit_tests -vvv
mlamouri0fbd6cd2015-10-26 12:08:11216-f "org.chromium.chrome.browser.media.*"
217```
218
andybons3322f762015-08-24 21:37:09219## Gtests
220
221```shell
222# Build a test suite
jbudorick25c17382016-08-03 18:53:07223ninja -C out/Release content_unittests
andybons3322f762015-08-24 21:37:09224
225# Run a test suite
jbudorick9472f112016-01-27 16:21:56226out/Release/bin/run_content_unittests [-vv]
andybons3322f762015-08-24 21:37:09227
228# Run a subset of tests
jbudorick9472f112016-01-27 16:21:56229out/Release/bin/run_content_unittests [-vv] --gtest-filter ByteStreamTest.*
andybons3322f762015-08-24 21:37:09230```
231
232## Instrumentation Tests
233
234In order to run instrumentation tests, you must leave your device screen ON and
235UNLOCKED. Otherwise, the test will timeout trying to launch an intent.
236Optionally you can disable screen lock under Settings -> Security -> Screen Lock
237-> None.
238
jbudorick9472f112016-01-27 16:21:56239Next, you need to build the app, build your tests, and then run your tests
240(which will install the APK under test and the test APK automatically).
andybons3322f762015-08-24 21:37:09241
242Examples:
243
244ContentShell tests:
245
246```shell
247# Build the code under test
248ninja -C out/Release content_shell_apk
249
250# Build the tests themselves
251ninja -C out/Release content_shell_test_apk
252
jbudorick9472f112016-01-27 16:21:56253# Run the test (will automagically install the APK under test and the test APK)
254out/Release/bin/run_content_shell_test_apk [-vv]
andybons3322f762015-08-24 21:37:09255```
256
newt17e4d242015-08-27 09:07:26257ChromePublic tests:
andybons3322f762015-08-24 21:37:09258
259```shell
260# Build the code under test
newt17e4d242015-08-27 09:07:26261ninja -C out/Release chrome_public_apk
andybons3322f762015-08-24 21:37:09262
263# Build the tests themselves
newt17e4d242015-08-27 09:07:26264ninja -C out/Release chrome_public_test_apk
andybons3322f762015-08-24 21:37:09265
jbudorick9472f112016-01-27 16:21:56266# Run the test (will automagically install the APK under test and the test APK)
267out/Release/bin/run_chrome_public_test_apk [-vv]
andybons3322f762015-08-24 21:37:09268```
269
270AndroidWebView tests:
271
272```shell
ctzsm34f54d62017-04-21 17:04:07273ninja -C out/Release webview_instrumentation_apk
274ninja -C out/Release webview_instrumentation_test_apk
275out/Release/bin/run_webview_instrumentation_test_apk [-vv]
andybons3322f762015-08-24 21:37:09276```
277
ntfschr09699f442017-01-12 22:20:46278In order to run a subset of tests, use -f to filter based on test class/method
279or -A/-E to filter using annotations.
andybons3322f762015-08-24 21:37:09280
281Filtering examples:
282
283```shell
284# Run a test suite
jbudorick9472f112016-01-27 16:21:56285out/Debug/bin/run_content_shell_test_apk
andybons3322f762015-08-24 21:37:09286
287# Run a specific test class
jbudorick9472f112016-01-27 16:21:56288out/Debug/bin/run_content_shell_test_apk -f AddressDetectionTest.*
andybons3322f762015-08-24 21:37:09289
290# Run a specific test method
jbudorick9472f112016-01-27 16:21:56291out/Debug/bin/run_content_shell_test_apk -f \
andybons3322f762015-08-24 21:37:09292AddressDetectionTest#testAddressLimits
293
294# Run a subset of tests by size (Smoke, SmallTest, MediumTest, LargeTest,
295# EnormousTest)
jbudorick9472f112016-01-27 16:21:56296out/Debug/bin/run_content_shell_test_apk -A Smoke
andybons3322f762015-08-24 21:37:09297
298# Run a subset of tests by annotation, such as filtering by Feature
jbudorick9472f112016-01-27 16:21:56299out/Debug/bin/run_content_shell_test_apk -A Feature=Navigation
andybons3322f762015-08-24 21:37:09300```
301
302You might want to add stars `*` to each as a regular expression, e.g.
303`*`AddressDetectionTest`*`
304
305## Running Blink Layout Tests
306
qyearsleyac3af532016-11-16 22:07:32307See [Layout Tests](testing/layout_tests.md).
andybons3322f762015-08-24 21:37:09308
309## Running GPU tests
310
311(e.g. the "Android Debug (Nexus 7)" bot on the chromium.gpu waterfall)
312
xiaoyin.l1003c0b2016-12-06 02:51:17313See https://www.chromium.org/developers/testing/gpu-testing for details. Use
jbudorick25c17382016-08-03 18:53:07314`--browser=android-content-shell`. Examine the stdio from the test invocation on
315the bots to see arguments to pass to `src/content/test/gpu/run_gpu_test.py`.