let 'em test 'emselves
This project is maintained by benhylau
A fully Python-based tool that helps you automate Android UI testing and captures everything you need from each test session.
You can see yksp in action here. Higher abstractions of UI interactions used to compose test scripts are available through AndroidViewClient by @dtmilano. This project will focus on tools to inspect, correlate, and validate the set of generated test results. Pull requests are, of course, welcome.
This assumes your environment has python, easy_install and git. You may confirm each installation by typing the following commands in your Terminal.
$ which python
$ which easy_install
$ which git
You may skip this step if you already have the Android SDK installed.
Download the Android SDK. You may download the stand-alone package for your platform instead of the IDE bundles. After extracting the package, you need to download the following tools, as they are not included in the SDK package.
You will download these tools by using yet another tool, which is included in the SDK package. Replace android-sdk-root with the local path to your extracted SDK root directory.
$ cd android-sdk-root/tools
$ ./android list sdk --all
$ ./android update sdk --no-ui --all --filter 1,2,3
Install Pillow and AndroidViewClient.
$ cd ~
$ sudo easy_install --upgrade Pillow
$ sudo easy_install --upgrade androidviewclient
Clone the yksp repo, and AndroidViewClient as a submodule.
$ git clone git://github.com/groundupworks/yksp.git
$ cd yksp
$ git submodule init
$ git submodule update
Set the environment variable $YKSP_HOME, and also $ANDROID_HOME if not already set. Then configure your $PATH.
On a Mac
$ sudo nano ~/.bash_profile
On Ubuntu
$ sudo nano /etc/profile.d/yksp.sh
Add the following text, replacing android-sdk-root, yksp-root, and build-tools-version with your local paths. Log out and log back in.
export ANDROID_HOME=android-sdk-root
export YKSP_HOME=yksp-root/yksp
export PATH=$PATH:$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools:$ANDROID_HOME/build-tools/build-tools-version:$YKSP_HOME
Mine looks something like this on the Mac.
export ANDROID_HOME=/Users/benedict/Dev/android-sdk-macosx
export YKSP_HOME=/Users/benedict/Dev/projects/yksp/yksp
export PATH=$PATH:$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools:$ANDROID_HOME/build-tools/20.0.0:$YKSP_HOME
And... you're all set!
The yksp repo has an examples/flying-photo-booth-tests folder, containing the Flying PhotoBooth APK, along with its PyUnit test scripts in a scripts folder. Connect one or more Android devices to your computer. Ensure that the device is made USB debuggable and screen lock is disabled (like, completely off... not even Swipe Unlock). This is how you would run a typical yksp test session.
$ cd $YKSP_HOME/../examples/flying-photo-booth-tests
$ yksp
Oberserve the test session logs in your Terminal, it should look as follow if everything is set up properly. Results are stored in the newly created results folder, with each test session in a folder named with its own timestamp.
$ yksp
Scanning for APK...
APK: flying-photo-booth-release.apk
Inspecting APK...
Package name: com.groundupworks.flyingphotobooth
Version name: 3.4
Version code: 13
Scanning for test scripts...
Script 1: testAbandonLinking.py
Script 2: testCapture.py
Listing devices...
Device 1: EP7309229R
[EP7309229R] Collecting device properties...
[EP7309229R] [ro.product.manufacturer]: [Sony]
[EP7309229R] [ro.product.model]: [C6502]
[EP7309229R] [ro.build.version.sdk]: [16]
[EP7309229R] Installing APK...
[EP7309229R] Preparing test case 1 of 2 [testAbandonLinking.py] on C6502...
[EP7309229R] Wiping app data...
[EP7309229R] Clearing logcat buffer...
[EP7309229R] Start printing logcat output to file...
[EP7309229R] Executing [testAbandonLinking.py] with command:
python results/2014-09-24-09-34-28/C6502-[EP7309229R]/testAbandonLinking/testAbandonLinking.py --package com.groundupworks.flyingphotobooth --serial EP7309229R --root results/2014-09-24-09-34-28/C6502-[EP7309229R]/testAbandonLinking --logs pyunit.txt --screenshots screenshots --screendumps screendumps
[EP7309229R] Logcat output saved
[EP7309229R] Downloading app data from device...
[EP7309229R] Extracting app data...
1947+0 records in
1947+0 records out
1947 bytes transferred in 0.002251 secs (864984 bytes/sec)
x apps/com.groundupworks.flyingphotobooth/_manifest
x apps/com.groundupworks.flyingphotobooth/db/webviewCookiesChromiumPrivate.db
x apps/com.groundupworks.flyingphotobooth/db/wings.db-journal
x apps/com.groundupworks.flyingphotobooth/db/wings.db
x apps/com.groundupworks.flyingphotobooth/db/webviewCookiesChromium.db
x apps/com.groundupworks.flyingphotobooth/sp/com.facebook.SharedPreferencesTokenCachingStrategy.DEFAULT_KEY.xml
x apps/com.groundupworks.flyingphotobooth/sp/com.groundupworks.flyingphotobooth_preferences.xml
[EP7309229R] App data downloaded
[EP7309229R] Wiping app data...
[EP7309229R] Preparing test case 2 of 2 [testCapture.py] on C6502...
[EP7309229R] Wiping app data...
[EP7309229R] Clearing logcat buffer...
[EP7309229R] Start printing logcat output to file...
[EP7309229R] Executing [testCapture.py] with command:
python results/2014-09-24-09-34-28/C6502-[EP7309229R]/testCapture/testCapture.py --package com.groundupworks.flyingphotobooth --serial EP7309229R --root results/2014-09-24-09-34-28/C6502-[EP7309229R]/testCapture --logs pyunit.txt --screenshots screenshots --screendumps screendumps
[EP7309229R] Logcat output saved
[EP7309229R] Downloading app data from device...
[EP7309229R] Extracting app data...
567+0 records in
567+0 records out
567 bytes transferred in 0.000659 secs (860409 bytes/sec)
x apps/com.groundupworks.flyingphotobooth/_manifest
[EP7309229R] App data downloaded
[EP7309229R] Wiping app data...
[EP7309229R] Uninstalling application...
All 2 test scripts executed on 1 device in 93 seconds
The test session logs should give a pretty good idea of what yksp is doing, but here is a better view.
Find APK from your root directory
Find test scripts from the scripts folder
Find connected devices and write serials to serials.txt
For each device, run in parallel:
Collect device properties and write to device.txt
Install APK
For each test script:
Make a local copy of the test script
Wipe app data
Start logcat
Execute local copy of the test script:
Connect to and wake device in YkspTestCase.setUp()
Run each test...() method in your YkspTestCase subclass:
For each YkspTestCase.saveScreen() call:
Save screenshot to the screenshots folder
Save view tree dump to screendumps folder
Kill app in YkspTestCase.tearDown()
Write PyUnit test results to pyunit.txt
Stop logcat and write to logcat.txt
Dump app data to data.ab
Extract data.ab to data folder
Wipe app data
Uninstall application
Report completion of test session
Test results are organized in a directory structure that would allow a script to easily walk the tree and generate reports from a data set involving multiple devices and test sessions.
results
<DATETIME>
serials.txt
SERIAL-[MODEL]
device.txt
SCRIPT1
SCRIPT1.py
pyunit.txt
logcat.txt
data.ab
data
...
screenshots
0-tag0.png
1-tag1.png
2-tag2.png
screendumps
0-tag0.txt
1-tag1.txt
2-tag2.txt
SCRIPT2
SCRIPT2.py
pyunit.txt
logcat.txt
data.ab
data
...
screenshots
0-tag3.png
1-tag4.png
screendumps
0-tag3.txt
1-tag4.txt
Test scripts define subclasses of YkspTestCase based on the PyUnit framework. You will define and implement one or more methods with names starting with test, like such def testSomething(self):. To figure out what goes in the implementation, a good way to start is to look at testCapture.py and testAbandonLinking.py in examples/flying-photo-booth-tests/scripts.
From the examples, you will find that the YkspTestCase class provides the following convenience methods, documented here.
launchApp(package=None)
refreshScreen(sleep=1)
saveScreen(tag=None, sleep=1)It is important that refreshScreen() or saveScreen() be called after each screen transition on your device, in order for the test to pick up the updated view tree.
To send UI events to your device, we rely on AndroidViewClient by @dtmilano. You have already downloaded the documentaion to here $YKSP_HOME/../AndroidViewClient/AndroidViewClient/doc/index.html as part of the submodule.
To pass or fail test cases, aside from visual inspection of screenshots, we mostly rely on exceptions raised by the findView...() methods from AndroidViewClient, as well as the family of assert...() methods available through the PyUnit framework.
When writing test scripts, while you can find certain views by text sometimes, you will encounter UI elements like an image button, which you need to identify by a unique ID in the view tree. This is when the dump tool in AndroidViewClient becomes handy. Just manually navigate to the screen on your connected device, then type that in Terminal and pick out the ID you need from the view tree.
$ dump
android.widget.FrameLayout id/no_id/1
android.widget.LinearLayout id/no_id/2
android.widget.FrameLayout id/no_id/3
android.widget.FrameLayout id/no_id/4
android.widget.FrameLayout id/no_id/5
android.widget.FrameLayout id/no_id/6
android.widget.LinearLayout id/no_id/7
android.widget.RelativeLayout id/no_id/8
android.widget.TextView id/no_id/9 PHOTO 1 OF 2
android.widget.ImageButton id/no_id/10
android.widget.ImageButton id/no_id/11
android.widget.RelativeLayout id/no_id/12
android.view.View id/no_id/13
android.view.View id/no_id/14
android.view.View id/no_id/15
android.view.View id/no_id/16
android.widget.Button id/no_id/17 CAPTURE
Lastly, it is important to remember that while an app data wipe is performed before and after running each test script to remove the persistent data stored, that is not the case in between test methods within the same script. Instead, the app is killed between test methods, which means in-memory states are destroyed, but disk-persisted states are carried across. For each test to have a true 'fresh start', it is recommended that you separate out your tests into different files, with only a single test method in each YkspTestCase subclass.
After all that, here is a template to get you started!
#! /usr/bin/env python
import sys
import os
try:
sys.path.append(os.environ['YKSP_HOME'])
except:
pass
from yksptestcase import YkspTestCase
class MyTestCase(YkspTestCase):
def testSomething(self):
'''
Tests something... like geese, mustard, cabbages and kings.
'''
self.launchApp()
self.saveScreen('my-start-screen', sleep=1)
# Your implementation...
if __name__ == '__main__':
YkspTestCase.main(sys.argv)yksp has run options to support various use cases. You can see those by entering the following in your Terminal.
$ yksp --help
Usage:
-h, --help OPTIONAL print this help and exit
-l, --linear OPTIONAL run tests on one device after another
-a, --noapk <package> OPTIONAL disable APK installation and run tests using the pre-installed application with the specified package name
-d, --nodump OPTIONAL disable app data dump after executing each test script
By default, yksp spawns multiple processes to execute test scripts on all connected devices in parallel. You may force yksp to run the scripts on one device after another using this flag. It is useful when debugging your test scripts, since the test logs in your Terminal will look cleaner.
Note that this does not imply everything is executed in a single process, as yksp uses multiple processes to do everything it needs to do even for a single device.
This option allows launching off a pre-installed application, specified by its package name. No APK needs to be provided, and yksp will ignore any APK in your directory and skip installation (and uninstallation) altogether. A verification step is performed on each connected device to verify that the application is already installed, and the test scripts will be executed only on the subset of devices that are verified.
This option is useful for integrating yksp with your IDE for development. In Android Studio, you can have your Run Configurations deploy the APK but not launch any Activity, and instead launch an external tool with the following configurations, where com.groundupworks.flyingphotobooth is replaced by the package name of your app, and some-directory is any directory containing a scripts folder containing your test scripts.
Program: yksp
Parameters: --noapk com.groundupworks.flyingphotobooth
Working directory: some-directory
After executing each test script, yksp utilizes the adb backup command to generate the backup file data.ab. The file itself can be used to restore the device state, as it contains all the locally persisted app data, including preferences and databases. yksp also extracts the file into the data folder, which you can conveniently use to validate the state changes as a result of the test script.
Note that if the application manifest has android:allowBackup="false", a data.ab file will be dumped out containing nothing. This is the correct behaviour and the absence of files in the extracted folder can be used to validate this.
The data dump is usually fast, but can optionally be disabled with this flag.
A: The short answer is no. yksp tests an APK, or a pre-installed app, as is. It cannot inject classes into your application, or switch up any build-time configurations. It doesn't mock parts of your application; it mocks you. One way around this limitation is to build the mock components into your APK, make the build-time configurations into run-time configurations, then use UI toggles or a special launch Intent to invoke a code path utilizing the mock components.
A: Again, yksp mocks out the user. Within the PyUnit test method, this mock user can generate events as input, and inspect the UI elements to validate the visible part of the output. However, input events can also result in output not immediately visible through the UI, such as a state change:
The first one is irrelevant, unless there is a way to confirm it through the UI. If a human is to validate the second and third cases, it would be via a database dump, check some server state via a server API call, or in some cases, inspection of the logcat output may be sufficient. Although it is possible to do all of the above within the test case itself and do assertion, using regular Python and adb commands, the recommended way is to use the PyUnit assertions to validate only the UI.
Since yksp dumps out the locally persisted app data and the logcat output after running each test script, just make use of those! Look at the results of the many test sessions as your data set, of which the PyUnit results is only one of the many components. You can write scripts to validate state changes, compare view trees and generate image similarity indices, correlate results across different devices and test sessions, and generate reports that are more comprehensive and meaningful.
A: Maybe because it consists entirely of consonants? Although the y is debatable.