Cuttlefish: Run stable CTS

This page describes how to run the Compatibility Test Suite (CTS) to evaluate your virtual device based on the AOSP source code.

Build Cuttlefish

Before running CTS, build the virtual device that you want to test with CTS. For AOSP Cuttlefish, do the following:

source build/envsetup.sh
lunch aosp_cf_x86_64_phone-userdebug
make -j

Build CTS

CTS is built separately from your device artifacts. To build CTS, follow these instructions.

From the root of the same tree where Cuttlefish was built, run the following commands:

source build/envsetup.sh
m -j cts WITH_DEXPREOPT_BOOT_IMG_AND_SYSTEM_SERVER_ONLY=false TARGET_PRODUCT=aosp_x86_64

Launch multi-tenant Cuttlefish instances

From the root of the same tree where Cuttlefish was built, run the following commands:

source build/envsetup.sh
lunch aosp_cf_x86_64_phone-userdebug
launch_cvd --num_instances=8 --daemon

When the instances start, verify each of the devices is adb accessible.

adb devices

The output should look similar to the example output below:

List of devices attached
127.0.0.1:6520  device
127.0.0.1:6521  device
127.0.0.1:6522  device
127.0.0.1:6523  device
127.0.0.1:6524  device
127.0.0.1:6525  device
127.0.0.1:6526  device
127.0.0.1:6527  device

Enable Wi-Fi on Cuttlefish instances

Some of the CTS tests fail if Wi-Fi isn't enabled. To enable Wi-Fi on all locally hosted Cuttlefish devices, follow these instructions.

From the root of the same tree where Cuttlefish was built, run the following commands:

for SERIAL in $(adb devices | grep -E '127.+device$' | cut -f1); do
adb -s ${SERIAL} install -r -g tools/tradefederation/core/res/apks/wifiutil/WifiUtil.apk
adb -s ${SERIAL} shell su root svc wifi enable
adb -s ${SERIAL} shell am instrument -e method "connectToNetwork" -e scan_ssid "false" -e ssid "VirtWifi" -w com.android.tradefed.utils.wifi/.WifiUtil
done

Run CTS stable subset

Not all of the CTS pass on Cuttlefish, as some of the tests are more tailored towards physical devices. To run the CTS with the cts-virtual-device-stable test plan that tracks the latest of what is stable on the virtual platform, follow these instructions.

From the root of the same tree where Cuttlefish was built, run the following commands:

source build/envsetup.sh
lunch aosp_cf_x86_64_phone-userdebug
cts-tradefed run cts-virtual-device-stable --no-enable-parameterized-modules --max-testcase-run-count 2 --retry-strategy RETRY_ANY_FAILURE --reboot-at-last-retry --shard-count 8

Expect the test to run for 20 to 25 minutes, and print out a summary similar to following:

=============== Summary ===============
Total Run time: 22m 8s
184/184 modules completed
Total Tests       : 92483
PASSED            : 92363
FAILED            : 0
IGNORED           : 71
ASSUMPTION_FAILURE: 49
Shard 0 used: [127.0.0.1:6522]
Shard 1 used: [127.0.0.1:6525]
Shard 2 used: [127.0.0.1:6524]
Shard 3 used: [127.0.0.1:6521]
Shard 4 used: [127.0.0.1:6520]
Shard 5 used: [127.0.0.1:6527]
Shard 6 used: [127.0.0.1:6526]
Shard 7 used: [127.0.0.1:6523]
============== End of Results ==============

Cleanup

When the tests are done, stop the virtual devices.

From the root of the same tree where Cuttlefish was built, run the following commands:

source build/envsetup.sh
lunch aosp_cf_x86_64_phone-userdebug
stop_cvd