DbTradeAlert for Android: Automated UI Tests – Part 3: Testing with WireMock and Firebase Test Lab for Android

Update 2016-11-24: the bug in handling URL encoded parameters in WireMock is fixed. But the current version throws a “IllegalArgumentException: resource assets not found” before even starting a test. For now DbTradeAlert stays with Wiremock version 2.2.1.

Update 2016-11-08: the free Spark plan now includes 5 tests per day on physical devices in Firebase Test Lab for Android – no credit card required. Details see Firebase Dev Summit.


First post in this series: Introduction to DbTradeAlert

Previous post: Automated UI Tests – Part 2: Testing with Espresso

This post jumps right into the middle of adding a scrolling experience test that was identified as being necessary for DbTradeAlert. Read the previous post to get an idea of the test and the one before it to find out how that scrolling experience test came about.


5.5 Mock Internet Access with WireMock

In its current state the test downloads quotes from the Internet when the recorded action to tap the Refresh button is executed. And to avoid starting the swipes before the list is filled with reports from those quotes the test uses a Thread.sleep(). This approach has various problems:

  • There may simply be no Internet access – for example on a build server
  • Waiting slows down the tests – especially a problem on Android as tests have to be repeated on various devices, Android versions, regional settings, …
  • The quote data is unpredictable – quotes downloaded monday morning will be 3 days old, values may be missing, downloads may result in an error code, …

The solution to all this is to avoid accessing the Internet altogether and just provide canned data. As replacing external components during tests is a common requirement it has a common fulfillment: mocking.

This test will use WireMock which serves as a proxy and responds with a canned .csv file to DbTradeAlert’s rerouted request. WireMock can do much more like return arbitrary HTTP response codes or take any time to answer at all. This will come in handy for later tests.

5.5.1 Download Sample Quotes

While WireMock can be used to record responses it’s easier to just download a .csv file directly. The URL can be determined from:

  • the base URL in QuoteRefresherService.onHandleIntent()
  • the format string specifying the data fields and their order in DbHelper.QuoteDownloadFormatParameter
  • the initial symbols and those from ch_securities.csv

The URL will look like this:
http://download.finance.yahoo.com/d/quotes.csv?f=aa2bc4d1ghl1nopp2st1vx&s=ABBN.VX+CFR.VX+NESN.VX+NOVN.VX+ROG.VX+SYNN.VX+UBSG.VX+ZURN.VX

Just enter it into a browser and save the downloaded quotes.csv file to “…\app\src\androidTest\assets”. To be shure check the asset shows up in Android Studio’s Project view as “quotes.csv (androidTest)”. The file is exactly what the app would have downloaded itself.

5.5.2 Integrate WireMock into the Project

The first step to integrate WireMock is to specify the dependencies in “build.gradle (Module: app)”:

apply plugin: 'com.android.application'

android {
    compileSdkVersion 23
    buildToolsVersion "23.0.3"

    defaultConfig {
        applicationId "de.dbremes.dbtradealert"
        minSdkVersion 15
        targetSdkVersion 23
        versionCode 2
        versionName "1.1"
        archivesBaseName = "${parent.name}-${android.defaultConfig.versionName}"
        // Ad unit Id for sample adverts from
        // https://firebase.google.com/docs/admob/android/google-services.json
        buildConfigField "String", "AD_UNIT_ID", "\"ca-app-pub-3940256099942544/6300978111\""
        testInstrumentationRunner 'android.support.test.runner.AndroidJUnitRunner'
    }
    buildTypes {
        debug {
            // Multidex only needed for WireMock
            multiDexEnabled true
        }
        release {
            minifyEnabled false
            // ProGuard is off
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
            // Keep actual ad unit Id secret by loading it from file outside of Git repo
            Properties props = new Properties()
            props.load(new FileInputStream("$project.rootDir/../../DbTradeAlert/project.properties"))
            buildConfigField "String", "AD_UNIT_ID", "\"${props.getProperty("ad_unit_id")}\""
        }
    }
    productFlavors {
        naked
        playStore {
            applicationId = "${android.defaultConfig.applicationId}.playStore"
        }
        withAds {
            applicationId = "${android.defaultConfig.applicationId}.withAds"
        }
    }
    sourceSets {
        playStore.java.srcDirs = ['src/common/java', 'src/playStore/java']
        withAds.java.srcDirs = ['src/common/java', 'src/withAds/java']
    }
}

dependencies {
    compile fileTree(include: ['*.jar'], dir: 'libs')
    testCompile 'junit:junit:4.12'
    compile 'com.android.support:appcompat-v7:23.3.0'
    compile 'com.android.support:design:23.3.0'
    compile 'com.android.support:support-v4:23.3.0'
    compile 'com.android.support:recyclerview-v7:23.3.0'
    playStoreCompile 'com.google.firebase:firebase-core:9.4.0'
    playStoreCompile 'com.google.firebase:firebase-crash:9.4.0'
    playStoreCompile 'com.google.firebase:firebase-config:9.4.0'
    withAdsCompile 'com.google.firebase:firebase-core:9.4.0'
    withAdsCompile 'com.google.firebase:firebase-ads:9.4.0'
    withAdsCompile 'com.google.firebase:firebase-crash:9.4.0'
    withAdsCompile 'com.google.firebase:firebase-config:9.4.0'
    compile 'org.jetbrains:annotations-java5:15.0'
    androidTestCompile 'com.android.support.test.espresso:espresso-core:2.2.2', {
        exclude group: 'com.android.support', module: 'support-annotations'
    }
    androidTestCompile 'com.android.support.test.espresso:espresso-contrib:2.2.2', {
        exclude group: 'com.android.support', module: 'support-annotations'
        exclude group: 'com.android.support', module: 'support-v4'
        exclude group: 'com.android.support', module: 'design'
        exclude group: 'com.android.support', module: 'recyclerview-v7'
    }
    androidTestCompile 'com.github.tomakehurst:wiremock:2.2.1', {
        // Allows us to use the Android version of Apache httpclient instead
        exclude group: 'org.apache.httpcomponents', module: 'httpclient'
        // Resolves the Duplicate Class Exception
        // duplicate entry: org/objectweb/asm/AnnotationVisitor.class
        exclude group: 'org.ow2.asm', module: 'asm'
        // Fixes Warning conflict with Android's version of org.json
        // org.json:json:20090211 is ignored for debugAndroidTest as it may be conflicting
        // with the internal version provided by Android.
        exclude group: 'org.json', module: 'json'
    }
    // Android compatible version of Apache httpclient.
    androidTestCompile 'org.apache.httpcomponents:httpclient-android:4.3.5.1'
    androidTestCompile 'com.android.support:multidex:1.0.0'
}
// Remove apply plugin: 'com.google.gms.google-services' when building naked variants.
// Those have no google-services.json file and applying the plugin would produce an error:
// "File google-services.json is missing. The Google Services Plugin cannot function without it."
// Building the R class would fail too and produce "cannot resolve symbol R" errors.
apply plugin: 'com.google.gms.google-services'
// Non-naked variants on the other hand will throw an exception when started after being build
// without this line: "FirebaseApp with name [DEFAULT] doesn't exist"

Because WireMock will only be used in instrumented tests references to it are noted with “androidTestCompile” – “testCompile” would apply to unit tests and “compile” affects the app itself.

Adding WireMock requires the use of multidexing – see lines 22 and 86. That’s because Android apps have an upper limit of 64K (65,536) methods and WireMock with its dependencies adds more than 65,000 methods alone. The limit actually affects the app’s DEX (Dalvik EXecutable) file and the solution is to use more than one DEX file. Fortunately Gradle will take care of that when requested. Also WireMock will only be added to the test apk and not to the app itself and Gradle is smart enough to use multidexing only when necessary.

Note that deviating from Handstandsam’s (Sam Edwards) reference build.gradle file – see lines 73 to 83 – I had to change an exclude from
exclude group: 'asm', module: 'asm' to
exclude group: 'org.ow2.asm', module: 'asm' to get rid of the error
ZipException: duplicate entry: org/objectweb/asm/AnnotationVisitor.class
Now how does one find the culprit?

The first step is to list all the app’s references, and their references, and their references, … While you can view them in Android Studio I found the command line output easier to digest. In Android Studio’s Terminal window enter:
gradlew app:dependencies --configuration androidTestCompile

Gradle or actually its wrapper gradlew defaults to the current project’s directory and you’ll get an output like this:

C:\Users\Admin\Documents\AndroidStudioProjects\DbTradeAlert>gradlew app:dependencies --configuration mockCompile
:app:dependencies                                                                         

------------------------------------------------------------
Project :app
------------------------------------------------------------

mockCompile - Classpath for compiling the mock sources.
+--- com.android.support.test.espresso:espresso-core:2.2.2
|    +--- com.squareup:javawriter:2.1.1
|    +--- com.android.support.test:rules:0.5
|    |    \--- com.android.support.test:runner:0.5
|    |         +--- junit:junit:4.12
|    |         |    \--- org.hamcrest:hamcrest-core:1.3
|    |         \--- com.android.support.test:exposed-instrumentation-api-publish:0.5
|    +--- com.android.support.test:runner:0.5 (*)
|    +--- javax.inject:javax.inject:1
|    +--- org.hamcrest:hamcrest-library:1.3
|    |    \--- org.hamcrest:hamcrest-core:1.3
|    +--- com.android.support.test.espresso:espresso-idling-resource:2.2.2
|    +--- org.hamcrest:hamcrest-integration:1.3
|    |    \--- org.hamcrest:hamcrest-library:1.3 (*)
|    +--- com.google.code.findbugs:jsr305:2.0.1
|    \--- javax.annotation:javax.annotation-api:1.2
+--- com.android.support.test.espresso:espresso-contrib:2.2.2
|    +--- com.google.android.apps.common.testing.accessibility.framework:accessibility-test-framework:2.0
|    |    \--- org.hamcrest:hamcrest-core:1.3
|    \--- com.android.support.test.espresso:espresso-core:2.2.2 (*)
+--- com.github.tomakehurst:wiremock:2.2.1
|    +--- org.eclipse.jetty:jetty-server:9.2.13.v20150730
|    |    +--- javax.servlet:javax.servlet-api:3.1.0
|    |    +--- org.eclipse.jetty:jetty-http:9.2.13.v20150730
|    |    |    \--- org.eclipse.jetty:jetty-util:9.2.13.v20150730
|    |    \--- org.eclipse.jetty:jetty-io:9.2.13.v20150730
|    |         \--- org.eclipse.jetty:jetty-util:9.2.13.v20150730
|    +--- org.eclipse.jetty:jetty-servlet:9.2.13.v20150730
|    |    \--- org.eclipse.jetty:jetty-security:9.2.13.v20150730
|    |         \--- org.eclipse.jetty:jetty-server:9.2.13.v20150730 (*)
|    +--- org.eclipse.jetty:jetty-servlets:9.2.13.v20150730
|    |    +--- org.eclipse.jetty:jetty-continuation:9.2.13.v20150730
|    |    +--- org.eclipse.jetty:jetty-http:9.2.13.v20150730 (*)
|    |    +--- org.eclipse.jetty:jetty-util:9.2.13.v20150730
|    |    \--- org.eclipse.jetty:jetty-io:9.2.13.v20150730 (*)
|    +--- org.eclipse.jetty:jetty-webapp:9.2.13.v20150730
|    |    +--- org.eclipse.jetty:jetty-xml:9.2.13.v20150730
|    |    |    \--- org.eclipse.jetty:jetty-util:9.2.13.v20150730
|    |    \--- org.eclipse.jetty:jetty-servlet:9.2.13.v20150730 (*)
|    +--- com.google.guava:guava:18.0
|    +--- com.fasterxml.jackson.core:jackson-core:2.6.1
|    +--- com.fasterxml.jackson.core:jackson-annotations:2.6.1
|    +--- com.fasterxml.jackson.core:jackson-databind:2.6.1
|    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.6.0 -> 2.6.1
|    |    \--- com.fasterxml.jackson.core:jackson-core:2.6.1
|    +--- org.xmlunit:xmlunit-core:2.1.1
|    +--- org.xmlunit:xmlunit-legacy:2.1.1
|    |    +--- org.xmlunit:xmlunit-core:2.1.1
|    |    \--- junit:junit:3.8.1 -> 4.12 (*)
|    +--- com.jayway.jsonpath:json-path:2.2.0
|    |    +--- net.minidev:json-smart:2.2.1
|    |    |    \--- net.minidev:accessors-smart:1.1
|    |    |         \--- org.ow2.asm:asm:5.0.3
|    |    \--- org.slf4j:slf4j-api:1.7.16
|    +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.16
|    +--- net.sf.jopt-simple:jopt-simple:4.9
|    +--- junit:junit:4.12 (*)
|    +--- org.apache.commons:commons-lang3:3.4
|    \--- com.flipkart.zjsonpatch:zjsonpatch:0.2.1
|         +--- com.fasterxml.jackson.core:jackson-databind:2.3.2 -> 2.6.1 (*)
|         +--- com.fasterxml.jackson.core:jackson-core:2.3.2 -> 2.6.1
|         +--- com.google.guava:guava:18.0
|         \--- org.apache.commons:commons-collections4:4.0
+--- org.apache.httpcomponents:httpclient-android:4.3.5.1
\--- com.android.support:multidex:1.0.0

(*) - dependencies omitted (listed previously)

BUILD SUCCESSFUL

Of most interest is of course everything under “com.github.tomakehurst:wiremock” as the error wasn’t there before adding it. Neither the class name “AnnotationVisitor” nor the namespace “objectweb” show up but “org.ow2.asm” looks interesting.

The next step is to change “exclude group: ‘asm’, module: ‘asm'” to “exclude group: ‘org.ow2.asm’, module: ‘asm'” and list the dependencies again – “org.ow2.asm:asm:5.0.3” was gone. And so was the ZipException when compiling the app again.

Solving that problem makes way for the next one:

com.android.build.api.transform.TransformException: com.android.builder.packaging.DuplicateFileException: Duplicate files copied in APK META-INF/LICENSE
File1: C:\Users\Admin\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.6.1\892d15011456ea3563319b27bdd612dbc89bb776\jackson-core-2.6.1.jar
File2: C:\Users\Admin\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient-android\4.3.5.1\eecbb0b998e77629862a13d957d552b3be58fc4e\httpclient-android-4.3.5.1.jar
File3: C:\Users\Admin\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-databind\2.6.1\45c37a03be19f3e0db825fd7814d0bbec40b9e0\jackson-databind-2.6.1.jar
File4: C:\Users\Admin\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-annotations\2.6.1\f9661ddd2456d523b9428651c61e34b4ebf79f4e\jackson-annotations-2.6.1.jar

As the Exception details show the problem is that four LICENSE files are being copied to the same place but of course only one can make it. Luckily checking the project’s licenses showed that all four are Apache 2.0 licenses so adding only one of them is legal:

apply plugin: 'com.android.application'

android {
    compileSdkVersion 23
    buildToolsVersion "23.0.3"

    defaultConfig {
        // ...
    }
    buildTypes {
        // ...
    }
    productFlavors {
        // ...
    }
    sourceSets {
        // ...
    }
    packagingOptions {
        // Keep only 1 license from 3* com.fasterxml.jackson.core + 1* org.apache.httpcomponents
        // All 4 are Apache 2.0 licenses
        pickFirst  'META-INF/LICENSE'
    }
}

dependencies {
    // ...
}
// ...

Finally the app builds with WireMock included.

5.5.3 Integrate WireMock into the Test

For the scrolling experience test the app needs to redirect its quotes request to WireMock and preferably without the redirect affecting other tests or the app itself. In other words it needs a specific build variant applying that mock.

But once you define a mock build type and switch to it the ScrollingExperienceTest class won’t compile anymore. That’s because Gradle evaluates the testCompile and androidTestCompile dependencies only for the debug build type. While a ‘testBuildType “mock”‘ fixes that only one build type can be specified which will shift the problem around. That means to have debug and mock builds working you’d have to add a mockCompile for each testCompile and androidTestCompile.

In this case the test only runs on the mock build so all androidTestCompile specific dependencies for Espresso and WireMock related libraries can be replaced with mockCompile dependencies.

Changes to build.gradle:

  • Lines 23 to 32 define the mock build
  • Line 24 shows how to initialize a build type
  • Line 68 replaces the debug with the mock build type as the default build type for tests
  • The new build type is applied for example in line 86
apply plugin: 'com.android.application'

android {
    compileSdkVersion 23
    buildToolsVersion "23.0.3"

    defaultConfig {
        applicationId "de.dbremes.dbtradealert"
        minSdkVersion 15
        targetSdkVersion 23
        versionCode 2
        versionName "1.1"
        archivesBaseName = "${parent.name}-${android.defaultConfig.versionName}"
        // Ad unit Id for sample adverts from
        // https://firebase.google.com/docs/admob/android/google-services.json
        buildConfigField "String", "AD_UNIT_ID", "\"ca-app-pub-3940256099942544/6300978111\""
        // Prepare rerouting of quote download requests to WireMock for tests
        buildConfigField "String", "HOST", "download.finance.yahoo.com"
        buildConfigField "String", "PORT", "\"80\""
        testInstrumentationRunner 'android.support.test.runner.AndroidJUnitRunner'
    }
    buildTypes {
        mock {
            initWith(buildTypes.debug)
            // Without enhanced google-services.json his leads to the error
            // "No matching client found for package name 'de.dbremes.dbtradealert.playStore.mock'":
            //applicationIdSuffix ".mock"
            buildConfigField "String", "HOST", "\"127.0.0.1\""
            buildConfigField "String", "PORT", "\"8080\""
            // Multidex only needed for WireMock
            multiDexEnabled true
        }
        release {
            minifyEnabled false
            // ProGuard is off
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
            // Keep actual ad unit Id secret by loading it from file outside of Git repo
            Properties props = new Properties()
            props.load(new FileInputStream("$project.rootDir/../../DbTradeAlert/project.properties"))
            buildConfigField "String", "AD_UNIT_ID", "\"${props.getProperty("ad_unit_id")}\""
        }
    }
    productFlavors {
        naked
        playStore {
            applicationId = "${android.defaultConfig.applicationId}.playStore"
        }
        withAds {
            applicationId = "${android.defaultConfig.applicationId}.withAds"
        }
    }
    sourceSets {
        playStore.java.srcDirs = ['src/common/java', 'src/playStore/java']
        withAds.java.srcDirs = ['src/common/java', 'src/withAds/java']
    }
    variantFilter { variant ->
        def names = variant.flavors*.name
        // Only ...-playStore-debug.apk needed of the mocks
        if (variant.buildType.name.equals("mock") && names.contains("playStore") == false) {
            variant.ignore = true
        }
    }
    packagingOptions {
        // Keep only 1 license from 3* com.fasterxml.jackson.core + 1* org.apache.httpcomponents
        // All 4 are Apache 2.0 licenses
        pickFirst  'META-INF/LICENSE'
    }
    testBuildType "mock"
}

dependencies {
    compile fileTree(include: ['*.jar'], dir: 'libs')
    testCompile 'junit:junit:4.12'
    compile 'com.android.support:appcompat-v7:23.3.0'
    compile 'com.android.support:design:23.3.0'
    compile 'com.android.support:support-v4:23.3.0'
    compile 'com.android.support:recyclerview-v7:23.3.0'
    playStoreCompile 'com.google.firebase:firebase-core:9.4.0'
    playStoreCompile 'com.google.firebase:firebase-crash:9.4.0'
    playStoreCompile 'com.google.firebase:firebase-config:9.4.0'
    withAdsCompile 'com.google.firebase:firebase-core:9.4.0'
    withAdsCompile 'com.google.firebase:firebase-ads:9.4.0'
    withAdsCompile 'com.google.firebase:firebase-crash:9.4.0'
    withAdsCompile 'com.google.firebase:firebase-config:9.4.0'
    compile 'org.jetbrains:annotations-java5:15.0'
    mockCompile 'com.android.support.test.espresso:espresso-core:2.2.2', {
        exclude group: 'com.android.support', module: 'support-annotations'
    }
    mockCompile 'com.android.support.test.espresso:espresso-contrib:2.2.2', {
        exclude group: 'com.android.support', module: 'support-annotations'
        exclude group: 'com.android.support', module: 'support-v4'
        exclude group: 'com.android.support', module: 'design'
        exclude group: 'com.android.support', module: 'recyclerview-v7'
    }
    mockCompile 'com.github.tomakehurst:wiremock:2.2.1', {
        // Allows us to use the Android version of Apache httpclient instead
        exclude group: 'org.apache.httpcomponents', module: 'httpclient'
        // Resolves the Duplicate Class Exception
        // duplicate entry: org/objectweb/asm/AnnotationVisitor.class
        exclude group: 'org.ow2.asm', module: 'asm'
        // Fixes Warning conflict with Android's version of org.json
        // org.json:json:20090211 is ignored for debugAndroidTest as it may be conflicting
        // with the internal version provided by Android.
        exclude group: 'org.json', module: 'json'
    }
    // Android compatible version of Apache httpclient.
    mockCompile 'org.apache.httpcomponents:httpclient-android:4.3.5.1'
    mockCompile 'com.android.support:multidex:1.0.0'
}
// Remove apply plugin: 'com.google.gms.google-services' when building naked variants.
// Those have no google-services.json file and applying the plugin would produce an error:
// "File google-services.json is missing. The Google Services Plugin cannot function without it."
// Building the R class would fail too and produce "cannot resolve symbol R" errors.
apply plugin: 'com.google.gms.google-services'
// Non-naked variants on the other hand will throw an exception when started after being build
// without this line: "FirebaseApp with name [DEFAULT] doesn't exist"

A side effect of introducing a new build type is the explosion of variants from 6 to 9. As only the new playStore-mock variant is needed a variantFilter eliminates the others – see lines 56 to 62.

When adding a build type or flavor it’s a good idea to check all uses of the BuildConfig class to make shure they still work. All was good in this case and finally it’s time to work on the test itself.

The actual redirect is achieved by assigning a host and port number depending on the build type – see lines 17 to 19 and 28 to 29 in build.gradle above.

Note how all buildConfigField values are explicitly enclosed in quotation marks despite being declared as strings. Without that weird errors like “error: package download.finance does not exist” pop up because the values end up unquoted in BuildConfig.java.

The base URL specified in QuoteRefresherService.onHandleIntent() is then replaced with
String baseUrl = "http://" + BuildConfig.HOST + ":" + BuildConfig.PORT + "/d/quotes.csv";

For mock builds BuildConfig.HOST will be 127.0.0.1 and BuildConfig.PORT will be 8080 while any other build still uses the real world values.

The ScrollingExperienceTest class will look like this after integrating WireMock:

// ...
@LargeTest
@RunWith(AndroidJUnit4.class)
public class ScrollingExperienceTest {
    private static final String CLASS_NAME = "ScrollingExperienceTest";
    private String symbolParameterValue;

    @Rule
    public ActivityTestRule<WatchlistListActivity> mActivityTestRule
            = new ActivityTestRule<>(WatchlistListActivity.class);

    @Rule
    public WireMockRule wireMockRule = new WireMockRule(Integer.valueOf(BuildConfig.PORT));

    private void createTestData() {
        Log.v(CLASS_NAME, "createTestData(): start");
        String testDataString = readFileFromTestAssets("ch_securities.csv");
        Context targetContext = InstrumentationRegistry.getTargetContext();
        DbHelper dbHelper = new DbHelper(targetContext);
        // SQLite Ids start with 1; 1 == CH watchlist
        long watchlistId = 1;
        dbHelper.importTestSecurities(testDataString, watchlistId);
    } // createTestData()

    @After
    public void deleteTestData() {
        Log.v(CLASS_NAME, "@After - deleteTestData(): start");
        Context targetContext = InstrumentationRegistry.getTargetContext();
        DbHelper dbHelper = new DbHelper(targetContext);
        dbHelper.deleteTestSecurities();
        logRequests();
    } // deleteTestData()

    private void logRequests() {
        final String METHOD_NAME = "logRequests";
        // allServeEvents
        List<ServeEvent> allServeEvents = getAllServeEvents();
        Log.v(CLASS_NAME, METHOD_NAME + "(): allServeEvents.size() = " + allServeEvents.size());
        for (int i = 0; i < allServeEvents.size(); i++) {
            Log.v(CLASS_NAME, String.format("%s(): allServeEvents[%d].Url = %s",
                    METHOD_NAME, i, allServeEvents.get(i).getRequest().getUrl()));
        }
        // unmatchedRequests
        List<LoggedRequest> unmatchedRequests = findUnmatchedRequests();
        Log.v(CLASS_NAME, METHOD_NAME + "(): unmatchedRequests.size() = " + unmatchedRequests.size());
        for (int i = 0; i < unmatchedRequests.size(); i++) {
            Log.v(CLASS_NAME, String.format("%s(): unmatchedRequests[%d] = %s",
                    METHOD_NAME, i, unmatchedRequests.get(i).toString()));
        }
        // nearMisses
        // Currently WireMock reports expected requests as received, too
        // See https://github.com/tomakehurst/wiremock/issues/484
        List<NearMiss> nearMisses = findNearMissesForAllUnmatched();
        Log.v(CLASS_NAME, METHOD_NAME + "(): nearMisses.size() = " + nearMisses.size());
        for (int i = 0; i < nearMisses.size(); i++) { Log.v(CLASS_NAME, String.format("%s(): nearMisses[%d].Diff = %s", METHOD_NAME, i, nearMisses.get(i).getDiff())); } } // logRequests() @Before public void prepareTest() { Log.v(CLASS_NAME, "@Before - prepareTest(): start"); try { // Must match imported symbols from ch_securities.csv + sample symbols from install: symbolParameterValue = URLEncoder.encode( "ABBN.VX+BAYN.DE+CFR.VX+NESN.VX+NOVN.VX+ROG.VX+SIE.DE+SYNN.VX+UBSG.VX+ZURN.VX", "utf-8"); } catch (UnsupportedEncodingException e) { PlayStoreHelper.logError(e); } createTestData(); setupWireMock(); } // prepareTest() private String readFileFromTestAssets(String fileName) { String fileContent = ""; Context testContext = InstrumentationRegistry.getContext(); try { InputStream testDataStream = testContext.getResources().getAssets().open(fileName); byte[] testData = new byte[testDataStream.available()]; int bytesRead = testDataStream.read(testData); fileContent = new String(testData); } catch (IOException e) { PlayStoreHelper.logError(e); } return fileContent; } // readFileFromTestAssets() private void setupWireMock() { Log.v(CLASS_NAME, "setupWireMock(): start"); String quotesCsv = readFileFromTestAssets("quotes.csv"); wireMockRule.stubFor(get(urlPathMatching(".*")) .withQueryParam("f", equalTo(DbHelper.QuoteDownloadFormatParameter)) // Currently WireMock fails to match URL encoded parameters // See https://github.com/tomakehurst/wiremock/issues/515 //.withQueryParam("s", equalTo(symbolParameterValue)) .willReturn(aResponse() .withStatus(HttpURLConnection.HTTP_OK) .withBody(quotesCsv))); } // setupWireMock() @Test public void scrollingExperienceTest() throws VerificationException { final String METHOD_NAME = "scrollingExperienceTest"; Log.v(CLASS_NAME, METHOD_NAME + "(): start"); // Make added securities show up in watchlist: Log.v(CLASS_NAME, METHOD_NAME + "(): tapping Refresh"); ViewInteraction actionMenuItemView = onView( allOf(withId(R.id.action_refresh), withContentDescription("Refresh"), isDisplayed())); actionMenuItemView.perform(click()); // Checking request was correct helps to track down reasons for failed tests verify(1, getRequestedFor(urlPathMatching(".*")) .withQueryParam("f", equalTo(DbHelper.QuoteDownloadFormatParameter)) // Currently WireMock fails to match URL encoded parameters // See https://github.com/tomakehurst/wiremock/issues/515 //.withQueryParam("s", equalTo(symbolParameterValue)) ); Log.v(CLASS_NAME, METHOD_NAME + "(): verified correctness of request"); // Scroll items to trigger creation of new items // Visually check if scrolling stutters -> test failed
        ViewInteraction recyclerView = onView(
                allOf(withId(R.id.list),
                        withParent(allOf(withId(R.id.container),
                                withParent(withId(R.id.main_content)))),
                        isDisplayed()));
        for (int i = 0; i < 8; i++) {
            Log.v(CLASS_NAME, METHOD_NAME + "(): swipeUp" + i);
            recyclerView.perform(actionOnItemAtPosition(i, swipeUp()));
        }
        Log.v(CLASS_NAME, METHOD_NAME + "(): the End");
    } // scrollingExperienceTest()
} // class ScrollingExperienceTest

The first step is to create an instance of the WireMockRule class which takes care of starting and stopping the server as well as resetting it between tests. In this case the creation order of the Rule instances doesnt matter but if there was a dependency you’d add a RuleChain.

In addition to load more symbols before starting the test now we have to tell WireMock what result to serve for which request. Therefore a new prepareTest() method gets the @Before annotation. Before calling createTestData() this method creates the symbolParameterValue which WireMock will have to expect. In this case it’s important to URL encode this value as the “+” characters would represent spaces.

Finally prepareTest() calls setupWireMock(). Like createTestData() this method needs to read a file from the test APK’s assets – the previously added quotes.csv. For that reason readFileFromTestAssets() was factored out of createTestData().

The next step is to define a stub – what happens when. In this case we don’t care about the URL but for the two parameters because they define what columns and rows to return. If the parameters match WireMock should return the contents of quotes.csv and a response code of HttpURLConnection.HTTP_OK because QuoteRefresherService.downloadQuotes() will check that response code. Because of a bug WireMock currently fails to verify URL encoded parameters so only the format parameter is checked.

Finally everything is ready to fix the actual test. In scrollingExperienceTest() the call to Thread.sleep() is gone and a verify() has taken its place. That method basically checks if the stub defined in setupWireMock() got a single exact request – again accounting for the current bug with URL encoded parameters.

In addition to that logRequests() gives you an idea what requests WireMock received and how it triaged them. Please note that due to another bug in WireMock currently NearMiss.getDiff() will report the expected request also as the received request.

After running the test the data it created is deleted like before.

A side note for those with multiple tests in a test class. Like rules the tests are executed in no specific order. That’s because both are identified by reflection which doesn’t make any promises about order. For tests one might actually call that a feature because it gets in the way of tests that depend on each other which is usually a bad idea. But if you need it JUnit from version 4.11 on provides a @FixMethodOrder to run tests for example ordered by name.

5.6 Run the Test in Firebase Test Lab for Android

Basically everything up to now has been a preparation for this step. Firebase Test Lab for Android provides those low end devices on which scrolling may stutter. And yes, those are physical devices.

As using Firebase Test Lab for Android isn’t free you’ll have to give up the free Spark plan. I chose the pay-as-you-go Blaze plan.

To upgrade to the Blaze plan:

  1. Navigate to the Firebase console at https://console.firebase.google.com and log in
  2. Select your project with the respective app
  3. In the navigation pane select Test Lab
  4. On the next page click Upgrade
  5. On the new overlay click Select Plan in the Blaze column
  6. Yet another overlay informs you about upgrading the project – click Continue
  7. Accept or change your country which determines the currency
  8. It seems you cannot use the private account created in the beginning but have to set up a business account. Fill in the missing information like the business’ name, phone number, and credit card details and click Confirm Purchase.
  9. On the confirmation overlay click Got It
  10. The page now shows a Run Your First Test button

Google advises to add its Test Lab screenshot library to the app’s test project. That enables you to initiate screenshots from test code. As the scrolling experience test doesn’t benefit from screenshots and I skipped that step.

A very important part of setting up the test is to select the devices and Android versions to test on. Usually you want to identify those most popular with your target audience and some extremes. The scrolling experience test just needs some low end devices and can use whatever Andoid versions they support.

To start the first test:

  1. Navigate to your project in the Test Lab as before
  2. Click the Run Your First Test button
  3. On the Choose Test Type page select Instrumentation Test and click Continue
  4. In the Select App step provide the app and the test APK. These are independent from any APKs you uploaded already. I provided DbTradeAlert-1.1-playStore-mock.apk and DbTradeAlert-1.1-playStore-mock-androidTest.apk from …\DbTradeAlert\app\build\outputs\apk. Click Continue.
  5. In the Select Dimensions step you specify the devices, API levels, orientations, and locales on which to run the tests. The button at the bottom right shows number of permutations this creates – 8 with the default options.
  6. Physical devices: I choose LG Nexus 4 and Motorola Moto G (2nd gen) as they have a low Basemark OS II score
  7. API levels: the selected devices only support API levels 19 and 22 (only Nexus 4)
  8. Orientation: I left both checked but there shouldn’t be much of a difference
  9. Locales: only testing on en_US is OK as the app isn’t localized
  10. Advanced option: timeout of 5 minutes is OK
  11. Finally click Start Tests

All tests on API level 22 (Lollipop) passed while those on API level 19 (Kitkat) failed with “java.lang.NoClassDefFoundError: com.google.firebase.FirebaseOptions”. Stackoverflow has dozends of threads about this starting in May 2016 when Google renamed a lot of stuff to “Firebase”. They said on Stackoverflow it was a bug on their side but that they have fixed it since.

As I don’t own a pre-Lollipop device and don’t think the scrolling experience depends on the API level anyway I didn’t dig any further.

In my case it took about 5 minutes to complete the 6 tests. Surprisingly each test claims that it took about 2.5 minutes while they complete in seconds on my phone. As Google charges USD 5 per hour for physical devices running the tests cost about USD 1.25. Still a bargain compared to maintaining those devices yourself but definitely more than expected.

Each test result links to details which for example include a video. The videos are about 10 to 15 seconds long with about 5 seconds showing the actual test – like on my physical device. As Google claims not to bill for the setup time I have no idea how those 2.5 minutes come about.

Unfortunately the video stuttered itself – no wonder with only 11 frames per second. To be sure I ran the test again on their Nexus 5 – same stutter. As one cannot configure devices to show screen update problems – or anything else from developer options – Firebase Test Lab for Android isn’t suited for this kind of test.

Another interesting outcome was the test failing on API level 23:
android.support.test.espresso.NoActivityResumedException: No activities in stage RESUMED. Did you forget to launch the activity. (test.getActivity() or similar)?

Watching the video showed that the app was waiting for input on the “Ignore battery optimizations?” dialog. Easily fixed by skipping the call to WatchlistListActivity.ensureExemptionFromBatteryOptimizations() if BuildConfig.BUILD_TYPE.equals(“mock”).

That concludes the first automated test deemed interesting. On to the next …

6. Running a Firebase Test Lab for Android Robo Test

The Firebase Test Lab for Android can do more than just run tests – it can come up with its own. Those are called Robo tests and not to be confused with the Monkey tests provided by the SDK. The difference is that Robo tests use a more systematic approach and even create a map of the app.

To start a Robo test:

  1. Navigate to your project in the Test Lab as before
  2. Click the Run a Test button
  3. Select Run a Robo test
  4. Provide an APK, select test dimensions, and start the test as before

After the tests finish you’ll find details about their outcome as with instrumented tests. But now you get additional screenshots and an activity map.

Excerpt from the activity map

Excerpt from the activity map

That activity map shows the paths through the app as taken by a Robo test. And while that’s a neat feature it also shows the limits of Robo tests – they just tap a button but never enter or change text. Robo tests can still be helpful at early stages of an app where you just want to make shure it doesn’t explode when simply showing a screen on some devices or Android versions. Note that the test was listed as taking 8 minutes and another run took 11 minutes – that’s USD 1.60 spent.

7. Test Connection Fault Handling

Until now the app hasn’t really been tested with faulty networks simply because fault simulation wasn’t available. Fortunately WireMock has this capability. It can return HTTP error codes, return HTTP OK but garbage or an empty response, or take forever to answer at all. As the test involves a tap of the Refresh buttton it will be another instrumented test:

// ...

@LargeTest
@RunWith(AndroidJUnit4.class)
public class NetworkFault1Test {
    private static final String CLASS_NAME = "NetworkFault1Test";

    @Rule
    public ActivityTestRule<WatchlistListActivity> mActivityTestRule
            = new ActivityTestRule<>(WatchlistListActivity.class);

    @Rule
    public WireMockRule wireMockRule = new WireMockRule(Integer.valueOf(BuildConfig.PORT));

    @Test
    public void networkFaultTest() throws VerificationException {
        final String METHOD_NAME = "networkFaultTest";
        Log.v(CLASS_NAME, METHOD_NAME + "(): start");
        ViewInteraction actionMenuItemView = onView(
                allOf(withId(R.id.action_refresh), withContentDescription("Refresh"), isDisplayed()));
        Log.v(CLASS_NAME, METHOD_NAME + "(): Refresh -> HttpURLConnection.HTTP_INTERNAL_ERROR");
        actionMenuItemView.perform(click());
        // In theory one can verify user friendly text + error code in toast. In practice it's
        // nearly impossible to get the timing right - toasts show with a delay and fade away soon.
//        String toastText
//                = QuoteRefresherService.QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA
//                + "download failed (response code " + HttpURLConnection.HTTP_INTERNAL_ERROR + ")!";
//        onView(withText(toastText))
//                .inRoot(
//                        withDecorView(
//                                not(mActivityTestRule.getActivity().getWindow().getDecorView())
//                        )
//                )
//                .check(matches(isDisplayed()));

        Log.v(CLASS_NAME, METHOD_NAME + "(): the End");
    } // networkFaultTest()

    @Before
    public void setupWireMock() {
        Log.v(CLASS_NAME, "@Before - setupWireMock(): start");
        wireMockRule.stubFor(get(urlPathMatching(".*"))
                .willReturn(aResponse()
                        .withStatus(HttpURLConnection.HTTP_INTERNAL_ERROR)
                )
        );
    } // setupWireMock()

} // class NetworkFault1Test

The code is very similar to the ScrollingExperienceTest class. But this time WireMock will simply return HttpURLConnection.HTTP_INTERNAL_ERROR – 500. So no need for adding sample data here.

The resulting toast will show “Error: download failed (response code 500)!” and could theoretically be verified with Espresso. But getting the timing right will be a problem as the toast shows with a slight delay and fades away soon.

Therefore I opted for a manual approach. Remember that the point of this test is to ensure the app handles network faults logically. Whether a toast informs the user of those faults is a separate decision.

In this case I put a break point in QuoteRefresherService.downloadQuotes() and one on the last line of NetworkFault1Test.networkFaultTest(). The second breakpoint is important because the app will be terminated as soon as the test finishes and that will be before you had any chance to single step through QuoteRefresherService.downloadQuotes().

The second test is very similar but WireMock will take 20 seconds to answer – longer than the timeouts in QuoteRefresherService.downloadQuotes().

// ...

@LargeTest
@RunWith(AndroidJUnit4.class)
public class NetworkFault2Test {
    // ...

    @Before
    public void setupWireMock() {
        Log.v(CLASS_NAME, "@Before - setupWireMock(): start");
        wireMockRule.stubFor(get(urlPathMatching(".*"))
                .willReturn(aResponse()
                        .withStatus(HttpURLConnection.HTTP_OK)
                        // Must be longer than timeouts in QuoteRefresherService.downloadQuotes()
                        .withFixedDelay(20000)
                )
        );
    } // setupWireMock()

    // ...
}

This will result in another toast with a slightly different message of “Error: connection timed out!”.

And yes, NetworkFault2Test is a copy of NetworkFault1Test. That comes from the necessity to have separate WireMock instances as the non-invasive approach to testing gives no chance to add URL parameters or anything else to control which stub gets used.

A more elegant approach would be to use WireMock’s stateful behavior. While I scrapped that idea because the “second breakpoint” mentioned above got overrun it’s an interesting solution for similar problems:

// ...

@LargeTest
@RunWith(AndroidJUnit4.class)
public class NetworkFaultTest {
    // ...

    @Before
    public void setupWireMock() {
        Log.v(CLASS_NAME, "@Before - setupWireMock(): start");
        // Internal server error scenario
        wireMockRule.stubFor(get(urlPathMatching(".*"))
                .inScenario(CLASS_NAME)
                .whenScenarioStateIs(STARTED)
                .willReturn(aResponse()
                        .withStatus(HttpURLConnection.HTTP_INTERNAL_ERROR)
                )
                .willSetStateTo("Timeout scenario")
        );
        // Timeout scenario
        wireMockRule.stubFor(get(urlPathMatching(".*"))
                .inScenario(CLASS_NAME)
                .whenScenarioStateIs("Timeout scenario")
                .willReturn(aResponse()
                        .withStatus(HttpURLConnection.HTTP_OK)
                        // Must be longer than timeouts in QuoteRefresherService.downloadQuotes()
                        .withFixedDelay(20000)
                )
                .willSetStateTo("Garbage data scenario")
        );
        // Garbage data scenario
        wireMockRule.stubFor(get(urlPathMatching(".*"))
                .inScenario(CLASS_NAME)
                .whenScenarioStateIs("Garbage data scenario")
                .willReturn(aResponse()
                        //.withStatus(HttpURLConnection.HTTP_OK)
                        // Returns HttpURLConnection.HTTP_OK + garbage data
                        .withFault(Fault.MALFORMED_RESPONSE_CHUNK)
                )
        );
    } // setupWireMock()

    // ...
}

WireMock offers stateful behavior which you are free to configure with scenarios and their states. Each scenario has an initial state of STARTED and defines the next state by calling willSetStateTo(). Subsequent states set their name with whenScenarioStateIs(). This way setupWireMock() creates three consecutive states allowing to control which stub gets used without changing the URL. While it didn’t work out here this would be handy for example to test the calculation behind trailing targets (trailing stop loss) as that would require a specific order of quotes.

One could also use an abstract base class but I found documenting how the test works and getting at the stub configuration for meaningful output more cumbersome than living with copied code.

The last test uses one of WireMock’s special faults: return HTTP OK but garbage data.

// ...

@LargeTest
@RunWith(AndroidJUnit4.class)
public class NetworkFault3Test {
    // ...

    @Before
    public void setupWireMock() {
        Log.v(CLASS_NAME, "@Before - setupWireMock(): start");
        wireMockRule.stubFor(get(urlPathMatching(".*"))
                .willReturn(aResponse()
                        // Returns HttpURLConnection.HTTP_OK + garbage data
                        .withFault(Fault.MALFORMED_RESPONSE_CHUNK)
                )
        );
    } // setupWireMock()

    // ...
}

This test actually surfaced a bug and now QuoteRefresherService.downloadQuotes() handles IOExceptions:

// ...

public class QuoteRefresherService extends IntentService {
    // ...

    private String downloadQuotes(String urlString) throws IOException {
        String result = "";
        InputStream inputStream = null;
        try {
            URL url = new URL(urlString);
            HttpURLConnection conn = (HttpURLConnection) url.openConnection();
            conn.setReadTimeout(10000 /* milliseconds */);
            conn.setConnectTimeout(15000 /* milliseconds */);
            conn.setRequestMethod("GET");
            conn.setDoInput(true);
            int responseCode = -1;
            try {
                // Starts the query
                conn.connect();
                responseCode = conn.getResponseCode();
            } catch (SocketTimeoutException e) {
                sendLocalBroadcast(
                        QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA + "connection timed out!");
                Log.d(CLASS_NAME,
                        QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA + "connection timed out!");
            }
            if (responseCode == HttpURLConnection.HTTP_OK) {
                inputStream = conn.getInputStream();
                try {
                    result = getStringFromStream(inputStream);
                } catch (IOException x) {
                    sendLocalBroadcast(QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA
                            + "could not read response!");
                    PlayStoreHelper.logError(CLASS_NAME,
                            QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA
                                    + "could not read response!");
                }
                Log.d(CLASS_NAME, "downloadQuotes(): got " + result.length() + " characters");
            } else {
                sendLocalBroadcast(QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA
                        + "download failed (response code " + responseCode + ")!");
                PlayStoreHelper.logError(CLASS_NAME,
                        QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA
                                + "download failed (response code " + responseCode + ")!");
            }
        } finally {
            if (inputStream != null) {
                inputStream.close();
            }
        }
        return result;
    } // downloadQuotes()

    // ...
}

That was the third planned test and now it’s time for the last …

8. Run a Monkey Test

A Monkey test exercises your app like a monkey – it does things at random. Fortunately Android SDK includes a pretty smart monkey:

  • It can restrict its input to specified APKs
  • It can favor specified actions
  • It logs what it does
  • It will stop when the app crashes or produces an ANR dialog
  • It remembers exactly what it did

Android’s monkey will not only send input to the app but also to the system. For example it can turn off WiFi or change the volume. But while that kind of testing is valuable I don’t want it to happen on my phone but on an emulator.

This line in Android Studio’s Terminal window lets the monkey loose:
C:\Users\<AccountName>\AppData\Local\Android\sdk\platform-tools\adb shell monkey -p de.dbremes.dbtradealert -v 5000 > %TMP%\monkey_test.txt

In this case the monkey is ordered to send 5000 inputs to package de.dbremes.dbtradealert and it’s actually smart enough to start the app. The resulting log will be in C:\Users\<AccountName>\AppData\Local\Temp\monkey_test.txt and looks like this:

:Monkey: seed=1478409080850 count=5000
:AllowPackage: de.dbremes.dbtradealert
:IncludeCategory: android.intent.category.LAUNCHER
:IncludeCategory: android.intent.category.MONKEY
// Event percentages:
//   0: 15.0%
//   1: 10.0%
//   2: 2.0%
//   3: 15.0%
//   4: -0.0%
//   5: -0.0%
//   6: 25.0%
//   7: 15.0%
//   8: 2.0%
//   9: 2.0%
//   10: 1.0%
//   11: 13.0%

:Switch: #Intent;action=android.intent.action.MAIN;category=android.intent.category.LAUNCHER;launchFlags=0x10200000;component=de.dbremes.dbtradealert/.WatchlistListActivity;end

    // Allowing start of Intent { act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] cmp=de.dbremes.dbtradealert/.WatchlistListActivity } in package de.dbremes.dbtradealert

:Sending Trackball (ACTION_MOVE): 0:(-2.0,-2.0)
:Sending Flip keyboardOpen=false
:Sending Touch (ACTION_DOWN): 0:(145.0,560.0)
:Sending Touch (ACTION_UP): 0:(145.06223,548.3796)
:Sending Trackball (ACTION_MOVE): 0:(0.0,-2.0)
:Sending Touch (ACTION_DOWN): 0:(699.0,577.0)
:Sending Touch (ACTION_UP): 0:(701.53815,585.20026)

    // Rejecting start of Intent { act=android.intent.action.MAIN cat=[android.intent.category.HOME] cmp=com.android.launcher3/.Launcher } in package com.android.launcher3

:Sending Trackball (ACTION_MOVE): 0:(-3.0,0.0)

...

Line 1 shows the seed value used. That’s important to reproduce the exact same sequence of inputs – just add “-s <SeedValue>” to the command. Of course the app needs to be in the same starting state too.

Lines 2 to 4 list the packages and categories from which the monkey is allowed to access activities. That’s controlled by the -c and -p command line switches.

Lines 6 to 17 show how the input is composed which is controlled by the –pct – that’s 2 hyphens – command line switches.

Lines 19 to 21 show the successful switch to an activity.

Lines 23 to 29 show various input events that the monkey generated. ACTION_DOWN and ACTION_UP usually happen at diffferent coordinates so that’s actually a swipe.

Line 31 shows a switch to an activity that was rejected because that activity didn’t match the filter created by the -c and -p command line switches.

Surprisingly the monkey has no direct means of typing text. That can still happen when it manages to hit the on screen keyboard.

If you want more control than the monkey provides you can use the monkeyrunner tool. Despite its name it has nothing to do with the test monkey but is an API for Python programs to control devices and emulators as well as the apps on them.

Additional Resources

Advertisements
This entry was posted in Uncategorized and tagged , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s