DbTradeAlert for Android: Schedule Quote Updates – Part 3

First post in this series: Introduction to DbTradeAlert

Previous post: DbTradeAlert for Android: Schedule Quote Updates – Part 2


2. Provide Signals so They Reach a User Even With His Phone in the Pocket

Currently signals only show up in the respective security’s reports. To actually alert the user DDbTradeAlert will use Android’s notification infrastructure like for example mail apps do.

Again this can be broken down to separate steps:

  1. Create the SQL to find triggered signals
  2. Build and send notifications

2.1. Create the SQL to Find Triggered Signals

Finding the triggered signals actually needs two queries: one to find the securities’ max prices and one to find the triggered signals. DbHelper’s updateSecurityMaxPrice() generates this SQL:

SELECT s._id, max_price, max_price_date, s.symbol, days_high, SUBSTR(last_price_date_time, 0, 11)
 FROM security s
	LEFT JOIN quote q ON q.security_id = s._id
 WHERE COALESCE(max_price, 0) < COALESCE(days_high, 0)
	AND COALESCE(max_price_date, '') < SUBSTR(last_price_date_time, 0, 11)

This finds all securities where Quote.DAYS_HIGH is greater than Security.MAX_PRICE. As a sanity check Security.MAX_PRICE_DATE must also be before Quote.LAST_PRICE_DATE_TIME.

SUBSTR(last_price_date_time, 0, 11) returns the date part from last_price_date_time. COALESCE(max_price, 0) returns max_price if that isn’t NULL and otherwise 0 – remember that the DB may contain NULL for any field except LAST_PRICE and _ID.

If any securities to update are found their MAX_PRICE and MAX_PRICE_DATE fields are updated by using Security.ID as a key and of course in a transaction. This has to be done in a separate step because SQLite doesn’t support JOINs in UPDATEs.

The SQL used by DbHelper.readAllTriggeredSignals() to find triggered signals is a bit longer:

/* Lower target signal */ 
 SELECT 'low' AS actual_name, days_low AS actual_value, 'L' AS signal_name, lower_target AS target_value, s.symbol
 FROM security s
	LEFT JOIN quote q ON q.security_id = s._id
 WHERE (last_price_date_time >= date('now','-1 day'))
	AND (
		lower_target IS NOT NULL
		AND days_low IS NOT NULL
		AND days_low <= lower_target ) 
 UNION ALL /* Upper target signal */ 
 SELECT 'high' AS actual_name, days_high AS actual_value, 'U' AS signal_name, upper_target AS target_value, s.symbol 
 FROM security s
 	LEFT JOIN quote q ON q.security_id = s._id 
 WHERE (last_price_date_time >= date('now','-1 day'))
	AND (
		upper_target IS NOT NULL
		AND days_high IS NOT NULL
		AND days_high >= upper_target
	) 
 UNION ALL /* Trailing stop loss signal */
 SELECT 'low' AS actual_name, days_low AS actual_value, 'T' AS signal_name, max_price * (100 - trailing_target) / 100 AS target_value, s.symbol
 FROM security s
	LEFT JOIN quote q ON q.security_id = s._id
 WHERE (last_price_date_time >= date('now','-1 day'))
	AND (
		trailing_target IS NOT NULL
		AND days_low IS NOT NULL
		AND days_low <= max_price * (100 - trailing_target) / 100
	) 
 ORDER BY s.symbol ASC

This consists of a UNION of 3 queries, each finding a type (lower, trailing, upper) of triggered signal (you’d only have to provide field names in the 1st query). Each query first checks if the quotes are up to date, then if the respective values are not NULL, and finally if the target was hit.

2.2. Build and Send Notifications

Notifications require an icon so you’ll need to add one before building them. Android Studio doesn’t provide anything suitable this time but the SDK does at:
C:\Users\\AppData\Local\Android\sdk\platforms\android-23\data\res\

Like the Refresh button’s icon this one has to be usable on various devices so you need to copy “emo_im_money_mouth.png” four times to the respective subfolder in your project. In my case they are below:
C:\Users\\Documents\AndroidStudioProjects\DbTradeAlert\app\src\main\res\

And the source folders are:

  • drawable-hdpi
  • drawable-mdpi
  • drawable-xhdpi
  • drawable-xxhdpi

Android Studio will pick the new icons up immediately – awesome!

After that sendNotificationForTriggeredSignals() can build and send the notifications:

public class QuoteRefresherService extends IntentService {
    // ...

    private String buildNotificationLineFromCursor(Cursor cursor) {
        String result = null;
        String actualName = cursor.getString(0);
        float actualValue = cursor.getFloat(1);
        String signalName = cursor.getString(2);
        float signalValue = cursor.getFloat(3);
        String symbol = cursor.getString(4);
        result = String.format(Locale.getDefault(),
                "%s(): %s = %01.2f; %s = %01.2f", symbol, actualName,
                actualValue, signalName, signalValue);
        return result;
    } // buildNotificationLineFromCursor()

    @Override
    protected void onHandleIntent(Intent intent) {
        String baseUrl = "http://download.finance.yahoo.com/d/quotes.csv";
        String url = baseUrl
                + "?f=" + DbHelper.QuoteDownloadFormatParameter
                + "&s=" + getSymbolParameterValue();
        String quoteCsv = "";
        try {
            boolean isManualRefresh = intent.getBooleanExtra(INTENT_EXTRA_IS_MANUAL_REFRESH, false);
            if (isManualRefresh || areExchangesOpenNow()) {
                if (isConnected()) {
                        quoteCsv = downloadQuotes(url);
                        DbHelper dbHelper = new DbHelper(this);
                        dbHelper.updateOrCreateQuotes(quoteCsv);
                        // Notify user of triggered signals even if app is sleeping
                        dbHelper.updateSecurityMaxPrice();
                        sendNotificationForTriggeredSignals(dbHelper);
                } else {
                    sendLocalBroadcast(BROADCAST_EXTRA_ERROR + "no Internet!");
                    Log.d(CLASS_NAME, BROADCAST_EXTRA_ERROR + "no Internet!");
                }
                // ...
            } else {
                Log.d(CLASS_NAME,
                    "onHandleIntent(): exchanges closed and not a manual reefresh - skipping alarm");
            }
        } catch (IOException e) {
              // ...
            }
            Log.e(CLASS_NAME, exceptionMessage, e);
        }
    } // onHandleIntent()

    // ...

    private void sendNotificationForTriggeredSignals(DbHelper dbHelper) {
        final String methodName = "sendNotificationForTriggeredSignals";
        Cursor cursor = dbHelper.readAllTriggeredSignals();
        if (cursor.getCount() > 0) {
            Context context = getApplicationContext();
            NotificationCompat.Builder builder = new NotificationCompat.Builder(context)
                    .setColor(Color.GREEN)
                    .setDefaults(Notification.DEFAULT_ALL)
                    .setNumber(cursor.getCount())
                    .setSmallIcon(R.drawable.emo_im_money_mouth);
            // Specify which intent to show when user taps notification
            Intent watchlistListIntent = new Intent(this, WatchlistListActivity.class);
            PendingIntent watchlistListPendingIntent
                    = PendingIntent.getActivity(context, 0, watchlistListIntent, 0);
            builder.setContentIntent(watchlistListPendingIntent);
            // Build back stack
            TaskStackBuilder stackBuilder = TaskStackBuilder.create(this);
            stackBuilder.addParentStack(WatchlistListActivity.class);
            stackBuilder.addNextIntent(watchlistListIntent);
            // Create notification
            if (cursor.getCount() == 1) {
                cursor.moveToFirst();
                String s = buildNotificationLineFromCursor(cursor);
                builder.setContentTitle("Target hit").setContentText(s);
                Log.v(CLASS_NAME, String.format("%s(): Notification = %s", methodName, s));
            } else {
                builder.setContentTitle(cursor.getCount() + " Targets hit");
                NotificationCompat.InboxStyle inboxStyle = new NotificationCompat.InboxStyle();
                builder.setStyle(inboxStyle);
                while (cursor.moveToNext()) {
                    String s = buildNotificationLineFromCursor(cursor);
                    inboxStyle.addLine(s);
                    Log.v(CLASS_NAME, String.format("%s(): Notification = %s", methodName, s));
                }
            }
            // Show notification
            NotificationManager notificationManager = (NotificationManager) context
                    .getSystemService(Context.NOTIFICATION_SERVICE);
            // Show new notification or update pending one
            final int notificationId = 1234;
            notificationManager.notify(notificationId, builder.build());
        }
        Log.d(CLASS_NAME,
                String.format("%s(): created %d notifications", methodName, cursor.getCount()));
        DbHelper.closeCursor(cursor);
    } // sendNotificationForTriggeredSignals()
}

After reading triggered signals the code first sets up a NotificationCompat.Builder object specifying the newly added icon with a circle of Win color behind it. Notification.DEFAULT_ALL will cause the device to use default sound, vibration and notification lights. And the amount of notifications will be visible separately.

The code then sets the intent to show when a user taps the notification with setContentIntent(). And it creates the backstack is so the device’s Back button works as expected.

If only one signal was triggered a single line of text created by buildNotificationLineFromCursor() and the title complete the notification.

If multiple signals were triggered the builder’s style will be NotificationCompat.InboxStyle which allows for multiple lines of text. Again buildNotificationLineFromCursor() creates those lines.

Finally a NotificationManager instance fires off the notification. Note that notificationId can be any number but should be unique. It’s used to update a notification with new content.

Everything happens in QuoteRefresherService.onHandleIntent() to inform the user even if his device is asleep.

The screenshots show the initial icon in the top left corner visualizing the notification – the phone will vibrate and sound depending on settings too – and the expanded notification. You see Visual Studio’s emulator because on a real device the USB related notifications would obscure DbTradeAlert.

Notification icon top left

Notification icon top left

Opened notification

Opened notification

The notification shows a trailing stop loss triggered for NOVN.VX. The trailing target is at 92.07 CHF (sample data: 90 % of the stock’s maximum price of 102.30 CHF) and the day’s low was at 79.55 CHF. It’s visible in the watchlist’s report as well (“T” and red background color in signal TextView).

Android Marshmallow added a twist to notifications: users can turn them off. They can do that for each app in Settings | Sound & notification | App notifications and turn off its notifications, prevent notifications from peeking – which DbTradeAlert doesn’t use – or even allow them in “Do not disturb” mode. Each individual app’s Settings screen also got a Notifications item for configuration.

To check if an app is blocked from sending notifications use NotificationManagerCompat.areNotificationsEnabled() and NotificationManagerCompat.getImportance() let’s you know whether your app’s notifications are priorized. Android provides no way to nag the user for enabling notifications or change their priority because the user has already decided to block explicitly this app’s notifications.

If the device uses a pin or pattern lock there is a fourth setting: hide sensitive notification content. In that case you should create different types of notifications by calling NotificationCompat.Builder.setVisibility() with one of the visibility types like NotificationCompat.VISIBILITY_SECRET. DbTradeAlert doesn’t give away any secrets so I didn’t bother with different visibility types.

Next post: Add Security and Watchlist Management

Additional Resources

Advertisements
Posted in Uncategorized | Tagged , | Leave a comment

DbTradeAlert for Android: Schedule Quote Updates – Part 2

First post in this series: Introduction to DbTradeAlert

Previous post: DbTradeAlert for Android: Schedule Quote Updates – Part 1


1.6 Handling Marshmallow’s Doze and App Standby Modes

Now let’s have a look at the addditional requirements Android Marshmallow creates with Doze mode and App Standby mode. They both aim at saving battery power and affect all apps runnning on Marshmallow irrespective of their target SDK.

Because there is enough documentation – and speculation – on the Internet I’ll skip the theory. Fast facts:

  • Both modes defer regular AlarmManager alarms and JobScheduler jobs for inactive devices (Doze) and apps (App Standby)
  • Alarms created by setAndAllowWhileIdle() or setExactAndAllowWhileIdle() – methods added with Marshmallow – continue to go off
  • Both modes take away network access and wake locks from affected apps
  • Users can exempt an app from most of both modes’ restrictions in Settings | Battery | Overflow | Battery optimization
  • Users can manually toggle an app between active and standby in Settings | Developer Options | Inactive Apps

Let’s see how those battery optimizations affect DbTradeAlert. First change the schedue to 1 minute:

alarmManager.setRepeating(AlarmManager.ELAPSED_REALTIME_WAKEUP, 1 * 60 * 1000, 1 * 60 * 1000, pendingIntent);

Then create the new schedule by deleting the app from the device before running it from Android Studio. Its logcat window should then show a message for the schedule creation followed by:

... D/QuoteRefreshAlarmRec.: onReceive(): quote refresh alarm received; starting QuoteRefresherService
... ...
... D/QuoteRefresherService: downloadQuotes(): got 485 characters
... ...
... D/DbHelper: updateOrCreateQuotes(): success!
... D/QuoteRefresherService: onHandleIntent(): quotes updated - initiating screen refresh
... D/BroadcastReceiver: quotesRefreshedBroadcastReceiver triggered UI update

Those entries repeat about every minute. After a while the screen shuts off resulting in:
… D/WatchlistListActivity: onPause(): quoteRefresherMessageReceiver unregistered

From then on quotesRefreshedBroadcastReceiver doesn’t run anymore but the app still receives quote refresh alarms about once a minute, gets data from the internet, and writes to and reads from the database.

Now let’s see how the app behaves in Doze mode. To force the device into Doze mode execute this line in Android Studio’s Terminal window:

C:\Users\<AccountName>\AppData\Local\Android\sdk\platform-tools\adb shell dumpsys deviceidle force-idle

The answer should be “Now forced in to idle mode”. If the answer is “Unable to go idle; not enabled” execute this line:

C:\Users\<AccountName>\AppData\Local\Android\sdk\platform-tools\adb shell dumpsys deviceidle enable

The answer should be “Idle mode enabled”. Now you can force the device into Doze mode.

After that switch to the logcat window. On my device the alarms still reach the app every minute (they shouldn’t, more on this later) but QuoteRefresherService reports an exception like this:

java.net.SocketTimeoutException: failed to connect to download.finance.yahoo.com/66.196.66.213 (port 80) after 15000ms

That means the app has no network access when the device is in Doze mode.

To fix that exempt the app from battery optimization:

  1. On the device go to Settings | Battery | Overflow | Battery optimization
  2. Select All Apps
  3. Find and tap DbTradeAlert
  4. In the dialog select “Don’t optimize” and tap “Finish”

After that go back to the logcat window. The app can load quotes from the internet again.

You can check the device’s state via this command:

C:\Users\<AccountName>\AppData\Local\Android\sdk\platform-tools\adb shell dumpsys deviceidle > %TEMP%\dumpsys_deviceidle1.txt

Open “dumpsys_deviceidle1.txt” (path on Windows 8.1 is “C:\Users\<AccountName>\AppData\Local\Temp\”). Near the bottom will be a line like this: “mState=IDLE”

To get the device back to normal just turn the screen on. If you then create the file again like above the line will read “mState=ACTIVE”.

Of course your users won’t know that they need to exempt DbTradeAlert from battery optimization. Here is how to ask them for it:

public class WatchlistListActivity extends AppCompatActivity
        implements WatchlistFragment.OnListFragmentInteractionListener {
    // ...

    @SuppressWarnings("NewApi")
    private void ensureExemptionFromBatteryOptimizations() {
        if (Utils.isAndroidBeforeMarshmallow() == false) {
            String packageName = getPackageName();
            PowerManager powerManager = getSystemService(PowerManager.class);
            if (powerManager.isIgnoringBatteryOptimizations(packageName) == false) {
                String explanation = "DbTradeAlert needs to download quotes even when in background!";
                Toast.makeText(this, explanation, Toast.LENGTH_LONG).show();
                Intent intent = new Intent(Settings.ACTION_REQUEST_IGNORE_BATTERY_OPTIMIZATIONS)
                        .setData(Uri.parse("package:" + packageName));
                startActivity(intent);
            }
        }
    } // ensureExemptionFromBatteryOptimizations()

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_watchlist_list);

        this.dbHelper = new DbHelper(this);

        Toolbar toolbar = (Toolbar) findViewById(R.id.toolbar);
        setSupportActionBar(toolbar);
        // Set up the ViewPager with the watchlist adapter.
        WatchlistListPagerAdapter watchlistListPagerAdapter
                = new WatchlistListPagerAdapter(getSupportFragmentManager(), dbHelper);
        mViewPager = (ViewPager) findViewById(R.id.container);
        mViewPager.setAdapter(watchlistListPagerAdapter);
        // Request user to whitelist app from Doze and App Standby
        ensureExemptionFromBatteryOptimizations();
        // Create initial quote refresh schedule (just overwrite existing ones)
        Log.d(CLASS_NAME, "onCreate(): creating quote refresh schedule");
        createQuoteRefreshSchedule();
    } // onCreate()

    // ...
}
package de.dbremes.dbtradealert;

import android.database.Cursor;
import android.os.Build;

public class Utils {
    public static  boolean isAndroidBeforeMarshmallow() {
        return Build.VERSION.SDK_INT < Build.VERSION_CODES.M;
    } // isAndroidBeforeMarshmallow()

    // ...
} // class Utils
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android" package="de.dbremes.dbtradealert">

    <!-- other uses-permission elements -->
    <uses-permission android:name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS" />

    <!-- application element -->

</manifest>

WatchlistListActivity.ensureExemptionFromBatteryOptimizations() shows a Toast explaining the app’s needs while asking for exemption from battery optimization. Asking for exemption from battery optimization requires android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS but the Toast is optional.

Without the “@SuppressWarnings(“NewApi”)” annotation Android Studio would complain that the overload of Context.getSystemService() taking a class and PowerManager.isIgnoringBatteryOptimizations() require API level 23.

Be aware that quite a few apps got banned from the Play Store for asking users to exempt them from battery optimization – or maybe even for just asking for android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS in the manifest. Be sure to check Google’s “Acceptable Use Cases for Whitelisting” before you implement this.

Asking for exemption

Asking for exemption

That said first delete the app from the device. Only then run it from Android Studio so onCreate() gets trigggered. When the app starts you’ll see a dialog and an explaining Toast. Tap “Yes” to give DbTradeAlert network access even when the device is in Doze mode or the app is in App Standby mode.

Optionally check if the app is now exempt from battery optimization:
C:\Users\<AccountName>\AppData\Local\Android\sdk\platform-tools\adb shell dumpsys deviceidle > %TEMP%\dumpsys_deviceidle2.txt

Under “Whitelist user apps:” you should see “de.dbremes.dbtradealert”.

Surprisingly the alarms kept going off every minute even when my device was in Doze mode. It turns out that my Moto G 2nd Gen. doesn’t support the new battery optimization even though it runs Marshmallow and the guys from Cyanogen Mod got it working for that device. My experience with emulators – both Google’s and Visual Studio’s – was even worse. So here is what should work even on a device that properly supports Doze and App Standby:

package de.dbremes.dbtradealert;

import android.app.AlarmManager;
import android.app.PendingIntent;
import android.content.BroadcastReceiver;
import android.content.Context;
import android.content.Intent;
import android.os.SystemClock;
import android.util.Log;

public class QuoteRefreshScheduler extends BroadcastReceiver {
    final static String CLASS_NAME = "QuoteRefreshScheduler";

    @Override
    @SuppressWarnings("NewApi")
    public void onReceive(Context context, Intent intent) {
        // Create schedule for quote refresh
        AlarmManager alarmManager = (AlarmManager) context.getSystemService(Context.ALARM_SERVICE);
        int requestCode = 0;
        Intent newIntent = new Intent(context, QuoteRefreshAlarmReceiver.class);
        PendingIntent pendingIntent = PendingIntent.getBroadcast(context, requestCode,
                newIntent, PendingIntent.FLAG_UPDATE_CURRENT);
        // Create schedule for quote refresh (every hour, starting 1 hour from now)
        if (Utils.isAndroidBeforeMarshmallow()) {
                alarmManager.setInexactRepeating(AlarmManager.ELAPSED_REALTIME_WAKEUP,
                        SystemClock.elapsedRealtime() + AlarmManager.INTERVAL_HOUR,
                        AlarmManager.INTERVAL_HOUR, pendingIntent);
            } else {
                alarmManager.setAndAllowWhileIdle(AlarmManager.ELAPSED_REALTIME_WAKEUP,
                        SystemClock.elapsedRealtime() + AlarmManager.INTERVAL_HOUR, pendingIntent);
        }
        // Use only for testing Doze and App Standby modes!
//        alarmManager.setRepeating(AlarmManager.ELAPSED_REALTIME_WAKEUP,
//                1 * 60 * 1000, 1 * 60 * 1000, pendingIntent);
        // Log what was done
        String scheduleCreationType = "";
        if (intent.getAction() != null) {
            if (intent.getAction().equals(Intent.ACTION_BOOT_COMPLETED)) {
                scheduleCreationType = "after reboot";
            } else {
                scheduleCreationType = "by " + intent.getAction();
            }
        }
        Log.d(CLASS_NAME,
                "onReceive(): quote refresh schedule created " + scheduleCreationType);
    } // onReceive()
} // class QuoteRefreshScheduler

AlarmManager.setAndAllowWhileIdle() cannot create repeating alarms. If you want to see how the result of “adb shell dumpsys alarm” reflects this:

  1. Reinstall the app now in Android Marshmallow or a later version and take a quick snapshot of its alarms – repeatInterval will be 0
  2. Wait for the alarm to go off and take another snapshot – the alarm definition is gone and only the statistics part is left
package de.dbremes.dbtradealert;

import android.content.Context;
import android.content.Intent;
import android.support.v4.content.WakefulBroadcastReceiver;
import android.util.Log;

public class QuoteRefreshAlarmReceiver extends WakefulBroadcastReceiver {
    // Logging tag can have at most 23 characters
    final static String CLASS_NAME = "QuoteRefreshAlarmRec.";

    @Override
    public void onReceive(Context context, Intent intent) {
        scheduleNextQuoteRefresh(context);
        Log.d(CLASS_NAME,
                "onReceive(): quote refresh alarm received; starting QuoteRefresherService");
        Intent service = new Intent(context, QuoteRefresherService.class);
        startWakefulService(context, service);
    } // onReceive()

    private void scheduleNextQuoteRefresh(Context context) {
        // Starting with Android Marshmallow only
        // AlarmManager.setAndAllowWhileIdle() works in Doze / App Standby mode
        // and unlike setInexactRepeating() cannot set repeating alarms. So apps
        // need to set the next alarm themselves each time an alarm goes off.
        if (Utils.isAndroidBeforeMarshmallow() == false) {
            Log.d(CLASS_NAME,
                    "scheduleNextQuoteRefresh(): ");
            Intent intent = new Intent(context, QuoteRefreshScheduler.class);
            intent.setAction("QuoteRefreshAlarmReceiver.scheduleNextQuoteRefresh()");
            context.sendBroadcast(intent);
        }
    } // scheduleNextQuoteRefresh()

} // class QuoteRefreshAlarmReceiver

QuoteRefreshAlarmReceiver simply tells QuoteRefreshScheduler to create another alarm. For logging purposes it also sets an action and WatchlistListActivity.createQuoteRefreshSchedule() will do this too.

As I have no device supporting battery optimization properly I didn’t bother to test DbTradeAlert with App Standby. And hey: It works on my machine! 🙂

Next post: DbTradeAlert for Android: Schedule Quote Updates – Part 3

Additional Resources

  • Why No Doze for Moto G 2014: http://forum.xda-developers.com/moto-g-2014/help/insomnia-doze-t3262507
  • Optimizing for Doze and App Standby (contains Google’s “Acceptable Use Cases for Whitelisting”): https://developer.android.com/training/monitoring-device-state/doze-standby.html
  • Behavior Changes with Android N: https://developer.android.com/preview/behavior-changes.html
  • Diving Into Android ‘M’ Doze: https://newcircle.com/s/post/1739/2015/06/12/diving-into-android-m-doze
  • When do you absolutely need WakefulBroadcastReceiver: http://porcupineprogrammer.blogspot.de/2014/02/when-do-you-absolutely-need.html
  • Posted in Uncategorized | Tagged , | Leave a comment

    DbTradeAlert for Android: Schedule Quote Updates – Part 1

    First post in this series: Introduction to DbTradeAlert

    Previous post: Update Quotes


    As its name implies DbTradeAlert wants to alert the user when a target has been reached. For that DbTradeAlert needs to:

    • Constantly monitor targets
    • Provide signals so they reach a user even with his phone in the pocket

    1. Constantly Monitor Targets

    1.1 Some Theory

    To be able to constantly monitor targets the app needs to constantly update quotes which by now only happens when the user taps Refresh. For several reasons “constantly” doesn’t mean real-time though:

    • The app’s target group of I-check-my-portfolio-every-weekend-and-that’s-enough people certainly doesn’t need it
    • Yahoo Finance doesn’t provide real-time data for most markets; for example while Frankfurt Stock Exchange reports real-time all other German stock exchanges’ reports are delayed by 15 minutes and Swiss Exchange’s reports are even delayed by 30 minutes
    • Real-time information is useless without real-time action which DbTradeAlert doesn’t provide
    • A phone doesn’t provide the availability which relying on real-time quotes would require
    • Constantly downloading quotes can get expensive without an unlimited data plan

    For those reasons DbTradeAlert will only update quotes once per hour. Of course this needs to happen even if the app isn’t active or the device is asleep. A short rundown of the classes that can run tasks periodically in regard to these requirements:

    • AlarmManager:
      • Can start an app and even wake up the device
      • Schedule gets lost when device reboots
    • GcmNetworkManager: replaced by Firebase JobDispatcher
    • Firebase JobDispatcher: backport of JobScheduler that may or may not qualify – found it too late and on Android Marshmallow and later it won’t do what DbTradeAlert needs (see Doze and App Standby below)
      • Works with pre-Lollipop Android
      • Requires Google Play Services
      • Schedule survives reboots but not Google Play Services update
    • JobScheduler works like AlarmManager and additionally:
      • Jobs can be restricted to conditions like connectivity
      • Schedule survives reboots but not Google Play Services update
      • Requires API 21 / Lollipop
    • ScheduledExecutorService: only works as long as the app runs
    • TimerTask: only works as long as the app runs

    While it’s possible to use a service to get around the requirement of a running app it would waste memory and battery life – Android pauses apps for a reason. And while my Moto G2 runs Marshmallow most developers will want their apps to be able to run on Pre-Lollipop Android so AlarmManager wins.

    With Doze mode and App Standby mode Android Marshmallow adds another twist to scheduling tasks. I’ll skip dealing with this here both to make things easier to follow and provide readers interested in Marshmallow requirements with a coherent solution later.

    Scheduling quote updates will be a huge change so let’s see what’s ahead:

    1. To create an hourly schedule DbTradeAlert needs:
      1. to register a new class extending BroadcastReceiver as a receiver of the BOOT_COMPLETED action to recreate the schedule after each reboot
      2. to create the schedule for AlarmManager in this new class’ onReceive() handler
      3. to send a broadcast from WatchlistListActivity.onCreate() to the new class to initially create the schedule
    2. The AlarmManager needs to start a quote refresh. But as there is no more user interaction involved the device could fall asleep at any time. A special pattern is advised to prevent that:
      1. AlarmManager sends a broadcast; the process is guaranteed to stay alive until the receiver’s onReceive() method returns
      2. The work can’t be done in onReceive() itself as it would block the UI because it runs on the main thread
      3. To stop the device from falling asleep before the work is finished the app needs to hold a wakelock for that time
      4. To make using the wakelock easier the WakefulBroadcastReceiver exists; this class requires the work to be done by an IntentService which in contrast to normal services runs on a worker thread
    3. It makes no sense to download quotes when the respective exchanges are closed. Therefore the IntentService will check – for now hard-coded – opening hours of exchanges when the refresh wasn’t started manually. For that the app needs:
      1. the IntentService being able to differentiate between both instantiations

    This requires the following changes to DbTradeAlert:

    1. Replace QuoteRefresherAsyncTask with a new class deriving from IntentService
    2. Create a new class extending WakefulBroadcastReceiver that instantiates the new IntentService when receiving the broadcast from AlarmManager and holds a wakelock until the IntentService tells it to release that lock
    3. Replace the existing start of QuoteRefresherAsyncTask by an instantiation of the new IntentService specifying that this is a manual refresh

    Let’s start with the replacement of QuoteRefresherAsyncTask.

    1.2 Create QuoteRefresherService

    As it will be started by a WakefulBroadcastReceiver QuoteRefresher’s super class had to be changed from AsyncTask to IntentService. Accordingly QuoteRefresherAsyncTask and its file were renamed to QuoteRefresherService.

    public class QuoteRefresherService extends IntentService {
        private static final String CLASS_NAME = "QuoteRefresherService";
        public static final String BROADCAST_ACTION_NAME = "QuoteRefresherAction";
        public static final String BROADCAST_EXTRA_ERROR = "Error: ";
        public static final String BROADCAST_EXTRA_NAME = "Message";
        public static final String BROADCAST_EXTRA_REFRESH_COMPLETED = "Refresh completed";
        public static final String INTENT_EXTRA_IS_MANUAL_REFRESH = "isManualRefresh";
        private static final String exceptionMessage = "Exception caught";
        // ...
        
        private boolean areExchangesOpenNow() {
            final String methodName = "areExchangesOpenNow";
            boolean result = false;
            Calendar now = Calendar.getInstance();
            int hourOfDay = now.get(Calendar.HOUR_OF_DAY);
            if (hourOfDay >= 9 && hourOfDay <= 18) {
                int dayOfWeek = now.get(Calendar.DAY_OF_WEEK);
                if (dayOfWeek != Calendar.SATURDAY && dayOfWeek != Calendar.SUNDAY) {
                    result = true;
                } else {
                    Log.d(CLASS_NAME, String.format(
                            "%s(): Exchanges closed on weekends (day = %d)",
                            methodName, dayOfWeek));
                }
            } else {
                Log.d(CLASS_NAME, String.format(
                        "%s(): Exchanges closed after hours (hour = %d)",
                        methodName, hourOfDay));
            }
            if (result) {
                Log.d(CLASS_NAME, String.format(
                        "%s(): Exchanges open", methodName));
            }
            return result;
        }// areExchangesOpenNow()
    
        @Override
        protected void onHandleIntent(Intent intent) {
            String baseUrl = "http://download.finance.yahoo.com/d/quotes.csv";
            String url = baseUrl
                    + "?f=" + DbHelper.QuoteDownloadFormatParameter
                    + "&s=" + getSymbolParameterValue();
            String quoteCsv = "";
            try {
                boolean isManualRefresh = intent.getBooleanExtra(INTENT_EXTRA_IS_MANUAL_REFRESH, false);
                if (isManualRefresh || areExchangesOpenNow()) {
                    if (isConnected()) {
                        quoteCsv = downloadQuotes(url);
                        DbHelper dbHelper = new DbHelper(this);
                        dbHelper.updateOrCreateQuotes(quoteCsv);
                    } else {
                        sendLocalBroadcast(BROADCAST_EXTRA_ERROR + "no Internet!");
                        Log.d(CLASS_NAME, BROADCAST_EXTRA_ERROR + "no Internet!");
                    }
                } else {
                    Log.d(CLASS_NAME,
                            "onHandleIntent(): exchanges closed and not a manual refresh - skipping alarm");
                }
            } catch (IOException e) {
                Log.e(CLASS_NAME, exceptionMessage, e);
                if (e instanceof UnknownHostException) {
                    // java.net.UnknownHostException:
                    // Unable to resolve host "download.finance.yahoo.com":
                    // No address associated with hostname
                    sendLocalBroadcast(BROADCAST_EXTRA_ERROR + "broken Internet connection!");
                    Log.d(CLASS_NAME, BROADCAST_EXTRA_ERROR + "broken Internet connection!");
                }
                // TODO: cannot rethrow in else case as that doesn't match overridden methods signature?
            }
            finally {
                QuoteRefreshAlarmReceiver.completeWakefulIntent(intent);
            }
        } // onHandleIntent()
    
        // ...
    }
    

    All the work is done in onHandleIntent() which keeps the code from doInBackground() and just adds some error checking and reporting. Losing the ability to pass a Context isn’t a problem because IntentService is indirectly derived from Context. But the intent parameter’s extras come in handy to transport isManualRefresh.

    As already noted there is no way to avoid waking up the device without requiring API 21 – the app can only avoid to start the radio now. onHandleIntent() therefore checks for isManualRefresh and areExchangesOpenNow().

    After completing work the super class’ completeWakefulIntent() is called to signal the WakefulBroadcastReceiver to release the wakelock – no onPostExecute() needed.

    1.3 Start QuoteRefresherService

    The manual start of QuoteRefresherAsyncTask is changed to instantiate QuoteRefresherService and set isManualRefresh:

    public class WatchlistListActivity extends AppCompatActivity
            implements WatchlistFragment.OnListFragmentInteractionListener {
        // ...
    
        @Override
        public boolean onOptionsItemSelected(MenuItem item) {
            // Handle action bar item clicks here. The action bar will
            // automatically handle clicks on the Home/Up button, so long
            // as you specify a parent activity in AndroidManifest.xml.
            int id = item.getItemId();
            switch (id) {
                case R.id.action_refresh: {
                    boolean addTimestamp = false;
                    updateTitle(addTimestamp);
                    Context context = getApplicationContext();
                    Intent service = new Intent(context, QuoteRefresherService.class);
                    service.putExtra(QuoteRefresherService.INTENT_EXTRA_IS_MANUAL_REFRESH, true);
                    startService(service);
                    return true;
                }
                case R.id.action_settings: {
                    return true;
                }
                default:
                    return super.onOptionsItemSelected(item);
            }
        }
    
        // ...
    }
    

    As a service QuoteRefresherService has to be declared in AndroidManifest.xml – see line 14 in the excerpt below:

    <?xml version="1.0" encoding="utf-8"?>
    <manifest xmlns:android="http://schemas.android.com/apk/res/android" package="de.dbremes.dbtradealert">
    
        <!-- other uses-permission elements ... -->
        <uses-permission android:name="android.permission.WAKE_LOCK" />
    
        <application android:allowBackup="true" android:icon="@mipmap/ic_launcher" android:label="@string/app_name" android:supportsRtl="true" android:theme="@style/AppTheme">
    
            <!-- activity element ... -->
    
            <receiver android:name=".QuoteRefreshAlarmReceiver" >
            </receiver>
    
            <service android:name=".QuoteRefresherService" />
        </application>
    </manifest>
    

    1.4 Create QuoteRefreshAlarmReceiver

    Now to the new WakefulBroadcastReceiver. QuoteRefreshAlarmReceiver is the new class that receives broadcasts from AlarmManager to start QuoteRefresherService:

    package de.dbremes.dbtradealert;
    
    import android.content.Context;
    import android.content.Intent;
    import android.support.v4.content.WakefulBroadcastReceiver;
    import android.util.Log;
    
    public class QuoteRefreshAlarmReceiver extends WakefulBroadcastReceiver {
        // Logging tag can have at most 23 characters
        final static String CLASS_NAME = "QuoteRefreshAlarmRec.";
    
        @Override
        public void onReceive(Context context, Intent intent) {
            Log.d(CLASS_NAME,
                    "onReceive(): quote refresh alarm received; starting QuoteRefresherService");
            Intent service = new Intent(context, QuoteRefresherService.class);
            startWakefulService(context, service);
        }
    }
    

    It just creates the appropriate intent and passes it to the super class’ startWakefulService(). As a WakefulBroadcastReceiver holds a wakelock the appropriate permission has to be declared in AndroidManifest.xml – see line 5 in the excerpt above.

    Any class eligible for receiving broadcasts needs to be registered with Android. As AlarmManager isn’t part of the app it cannot use local broadcasts. And broadcasts will arrive even if the app isn’t alive so the registration has to be permanent. This is achieved by extending AndroidManifest.xml – see lines 11 to 12 in the excerpt above.

    1.5 Create QuoteRefreshScheduler

    QuoteRefreshScheduler is the new class that creates the schedule for AlarmManager:

    package de.dbremes.dbtradealert;
    e(addTimestamp
    import android.app.AlarmManager;
    import android.app.PendingIntent;
    import android.content.BroadcastReceiver;
    import android.content.Context;
    import android.content.Intent;
    import android.util.Log;
    
    public class QuoteRefreshScheduler extends BroadcastReceiver {
        final static String CLASS_NAME = "QuoteRefreshScheduler";
    
        @Override
        public void onReceive(Context context, Intent intent) {
            AlarmManager alarmManager = (AlarmManager) context.getSystemService(Context.ALARM_SERVICE);
            int requestCode = 0;
            Intent newIntent = new Intent(context, QuoteRefreshAlarmReceiver.class);
            PendingIntent pendingIntent = PendingIntent.getBroadcast(context, requestCode,
                    newIntent, PendingIntent.FLAG_UPDATE_CURRENT);
                    alarmManager.setInexactRepeating(AlarmManager.ELAPSED_REALTIME_WAKEUP,
                            SystemClock.elapsedRealtime() + AlarmManager.INTERVAL_HOUR,
                            AlarmManager.INTERVAL_HOUR, pendingIntent);
            // Log what was done
            String scheduleCreationType;
            if (intent.getAction() != null
                    && intent.getAction().equals("android.intent.action.BOOT_COMPLETED")) {
                scheduleCreationType = "after reboot";
            } else {
                scheduleCreationType = "initially";
            }
            Log.d(CLASS_NAME, "onReceive(): quote refresh schedule created " + scheduleCreationType);
        } // onReceive()
    } // class QuoteRefreshScheduler
    

    It uses an explicit intent. That means because the receiver’s class is specified Android will deliver the broadcast only to instances of this class and ignore any intent filters. The intent is then wrapped into an PendingIntent to make it usable from outside the app – for example by AlarmManager. As FLAG_UPDATE_CURRENT is specified any existing schedule for this intent will be overwritten with the new one.

    The last step is to actually create the schedule: setInexactRepeating() allows Android to save energy by batching alarms. So INTERVAL_HOUR results in intervals between 1 and 2 hours which is good enough for the app’s purpose. Note that since API level 19 even setRepeating() works like this and the only difference left is the ability to specify not only predefined intervals.

    Specifying ELAPSED_REALTIME_WAKEUP does two things:

    • It’s based on the time passed since last boot and not on a wall clock like RTC_WAKEUP; this avoids having to deal with things like daylight saving time and timezone changes
    • It wakes up the device if it’s asleep

    Lastly setInexactRepeating()’s second parameter determines at which time to start and the code sets it to now.

    This is the broadcast receiver’s registration:

    <?xml version="1.0" encoding="utf-8"?>
    <manifest xmlns:android="http://schemas.android.com/apk/res/android" package="de.dbremes.dbtradealert">
    
        <!-- other uses-permission elements ... -->
        <uses-permission android:name="android.permission.RECEIVE_BOOT_COMPLETED" />
        <uses-permission android:name="android.permission.WAKE_LOCK" />
    
        <application android:allowBackup="true" android:icon="@mipmap/ic_launcher" android:label="@string/app_name" android:supportsRtl="true" android:theme="@style/AppTheme">
    
            <!-- activity element ... -->
    
            <receiver android:name=".QuoteRefreshAlarmReceiver" />
            <receiver android:name=".QuoteRefreshScheduler">
                <intent-filter>
                    <action android:name="android.intent.action.BOOT_COMPLETED" />
                </intent-filter>
            </receiver>
    
            <service android:name=".QuoteRefresherService" />
        </application>
    </manifest>
    

    This not only registers QuoteRefreshScheduler as a broadcast receiver but also makes it receive the BOOT_COMPLETED broadcast so the app can recreate its schedule after each reboot. And finally the app asks for permission to receive the ACTION_BOOT_COMPLETED broadcast from Android.

    Before you run the app take a snapshot of the device’s existing alarms. Snapshots like this will be the only way to check if a new alarm is scheduled as planned because there is no way to programmatically get a list of even your own app’s alarms. And seeing the sheer amount of alarms waiting to go off will also make Google’s decision to batch them more understandable.

    To create a snapshot of the device’s alarms go to Android Studio’s Terminal window and enter this line:
    C:\Users\<AccountName>\AppData\Local\Android\sdk\platform-tools\adb shell dumpsys alarm > %TEMP%\dumpsys_alarm1.txt

    This will create the file C:\Users\<AccountName>\AppData\Local\Temp\dumpsys_alarm1.txt (path on Windows 8.1) – more than 3.600 lines for my device. Searching for the package name – “de.dbremes.dbtradealert” in my case – should return no results.

    Now run the app. Android Studio’s logcat window should then contain a line like this:
    “… D/QuoteRefreshScheduler: onReceive(): quote refresh schedule created initially”

    After that take another snapshot of the device’s alarms and name it “dumpsys_alarm2.txt”. Open the file and search for your package name again. You should find the new alarm schedule now which will look like this:

    Batch{78a4fff num=7 start=1527556349 end=1527580188 flgs=0x8}:
    (... more alarms ...)

    ELAPSED_WAKEUP #3: Alarm{63ded1b type 2 when 1526400000 de.dbremes.dbtradealert}
    tag=*walarm*:de.dbremes.dbtradealert/.QuoteRefreshAlarmReceiver
    type=2 whenElapsed=+5m45s877ms when=+5m45s877ms
    window=+45m0s0ms repeatInterval=3600000 count=0 flags=0x0
    operation=PendingIntent{47e5eb8: PendingIntentRecord{15f9c8d de.dbremes.dbtradealert broadcastIntent}}
    (... more alarms ...)

    Explanation:

    • Batch with ID 78a4fff has 7 alarms
    • The new alarm is the 3rd alarm in the batch, has ID 63ded1b, is of type ELAPSED_WAKEUP / 2, and was set by package “de.dbremes.dbtradealert”
    • It will go off in 5 minutes, 45 seconds, 877 milliseconds and send a broadcast to “de.dbremes.dbtradealert/.QuoteRefreshAlarmReceiver”
    • Its repeat interval is 3.600.000 milliseconds or 1 hour
    • count=0 means the alarm was never skipped

    Near the bottom you’ll additionally find alarm statistics for your package:

    u0a146:de.dbremes.dbtradealert +18ms running, 1 wakeups:
    +18ms 1 wakes 1 alarms, last -15s103ms:
    *walarm*:de.dbremes.dbtradealert/.QuoteRefreshAlarmReceiver

    In this case the app received 1 alarm which did wake up the device if necessary. Processing that alarm took 18 milliseconds and the last alarm happened 15 seconds ago.

    There is no official documentation of the output’s format. And note that force-stopping an app destroys all its alarm schedules.

    So in this case after about 6 minutes the reports should update and the app’s title should be expanded by a timestamp like “DbTradeAlert @ 10:46” – unless the exchanges are closed (remember to correct the timezone if you use an emulator – its timezone defaults to UTC).

    Android Studio’s logcat window should also contain lines like these:
    ... D/QuoteRefreshScheduler: onReceive(): quote refresh schedule created initially
    ... D/QuoteRefreshAlarmRec.: onReceive(): quote refresh alarm received; starting QuoteRefresherService
    ... D/QuoteRefresherService: areExchangesOpenNow(): Exchanges open
    ... D/QuoteRefresherService: downloadQuotes(): got 487 characters
    ... ...
    ... D/DbHelper: updateOrCreateQuotes(): success!
    ... D/QuoteRefresherService: onHandleIntent(): quotes updated - initiating screen refresh
    ... D/BroadcastReceiver: quotesRefreshedBroadcastReceiver triggered UI update
    ... D/WatchlistListActivity: refreshAllWatchLists(): changed cursor for recyclerView with tag = 1
    ... D/WatchlistListActivity: refreshAllWatchLists(): changed cursor for recyclerView with tag = 2

    If that didn’t work check which log entry is the first one missing. Then check the correspoding class’ declaration and possibly required permission in AndroidManifest.xml – there is usually a missing or wrong entry.

    Once the initial schedule is created and alarms go off as expected: test if the app recreates the schedule on reboots. The easiest test is to send the BOOT_COMPLETED broadcast directly to your app. Beware that sending this simply to all apps may cause havoc and some Nexus devices seem to actually reboot. That said enter this line into Android Studio’s Terminal window:
    C:\Users\<AccountName>\AppData\Local\Android\sdk\platform-tools\adb shell am broadcast -a android.intent.action.BOOT_COMPLETED -c android.intent.category.HOME -n de.dbremes.dbtradealert/.QuoteRefreshScheduler

    The response should look like this:

    Broadcasting: Intent { act=android.intent.action.BOOT_COMPLETED cat=[android.intent.category.HOME] cmp=de.dbremes.dbtradealert/.QuoteRefreshScheduler (has extras) }
    Broadcast completed: result=0

    And Android Studio’s logcat window should show entries like those:
    ... D/QuoteRefreshScheduler: onReceive(): quote refresh schedule created after reboot
    ... D/QuoteRefreshAlarmRec.: onReceive(): quote refresh alarm received; starting QuoteRefresherService
    ... ...

    If recreating the schedule worked you should of course try an actual reboot. And if you use a VCS it’s a good time to check in all those changes now.

    Next post: DbTradeAlert for Android: Schedule Quote Updates – Part 2

    Additional Resources

    Posted in Uncategorized | Tagged , | Leave a comment

    Wrestling Google API keys

    Google uses API keys to track and bill your usage of their APIs. While creating and using those keys is generally well documented and even partly automated you are in for some surprises.

    This post focuses on the speed bumps you’ll encounter especially when creating an Android app using Google Maps.

    Surprises in the App’s API Key Infrastructure

    When you add a Google Maps activity to an app the AndroidManifest.xml file will contain this element:

    <!-- ... -->
      <meta-data
            android:name="com.google.android.geo.API_KEY"
            android:value="YOUR_API_KEY"/>       
    <!-- ... -->
    

    While the “Get API Key” documentation initially tells you to just replace YOUR_API_KEY with the actual API key that’s not a good idea – surprise #1.

    A few paragraphs later it tells you to paste your key into some google_maps_api.xml file instead where the interesting element looks like this:

    <!-- ... -->
     <string name="google_maps_key" translatable="false" templateMergeStrategy="preserve">
            ADD_API_KEY_HERE
        </string>
    <!-- ... -->
    

    Replacing ADD_API_KEY_HERE with the API key is better but still not good enough – surprise #2.

    The problem is in both cases you’ll come away with the idea of using a single API key. But you’ll need at least two because an app’s debug and release build types use different certificates which makes two API keys mandatory as they are issued for a combination of package name and certificate finger print.

    The good news: your project contains two google_maps_api.xml files:

    • one in …\app\src\debug\res\values
    • one in …\app\src\release\res\values

    Android Studio will automatically supply you with the correct google_maps_api.xml file when switching between debug and release build types. Also the respective file’s value will replace the one in AndroidManifest.xml. So just copy the respective API key into each google_maps_api.xml file and both release and debug variants of your app can use Google Maps.

    An easy way to check this is to open AndroidManifest.xml and use the Merged Manifest tab. That will show the file’s content after merging and considers the currently selected build type. When you select the respective attribute you can even see where its value came from.

    By the way: one reason for failed authentication is using the wrong package name. It’s determined by the applicationId in build.gradle – the package attribute in AndroidManifest.xml only determines the project’s structure.

    Surprises with Caching

    Once an app successfully authenticates with an API key the result is cached. That means even if you for example provide a wrong key afterwards your app will only break after you uninstalled and reinstalled it – surprise #3.

    How to Keep Your API Keys Private and Find More Surprises

    Google’s solution requires you to store their API keys in the code base. That’s not what you want to do for example in an open source project.

    Gradle to the rescue: it can read values from files outside the project and make them available for example in AndroidManifest.xml.

    In the app’s build.gradle file add lines like these:

    ...
        defaultConfig {
            applicationId "de.dbremes...."
            minSdkVersion 15
            targetSdkVersion 23
            versionCode 1
            versionName "1"
            Properties props = new Properties()
            props.load(new FileInputStream("$project.rootDir/../../MyApp/MyApp.properties"))
            manifestPlaceholders = [apiKey: "${props.getProperty("debug_api_key")}"]
        }
        buildTypes {
            release {
                minifyEnabled false
                proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
                Properties props = new Properties()
                props.load(new FileInputStream("$project.rootDir/../../MyApp/MyApp.properties"))
                manifestPlaceholders = [apiKey: "${props.getProperty("release_api_key")}"]
            }
        }
    ...
    

    This example requires a file named MyApp.properties in a MyApp directory relative to your project’s directory. Don’t worry about getting the directory structure wrong: once you sync Gradle it will inform you if it can’t find the file. The file’s content should look like this:

    debug_api_key=AIzaSy...7J0g
    release_api_key=AIzaSy...tCmc
    

    Then change your AndroidManifest.xml file like this:

    <!-- ... -->
            <meta-data
                android:name="com.google.android.geo.API_KEY"
                android:value="${apiKey}" />
    <!-- ... -->
    

    Finally delete both google_maps_api.xml files.

    When building the app Gradle will replace the variable in AndroidManifest.xml with the value for the respective build type. And the Merged Manifest tab of AndroidManifest.xml will be helpful gain.

    In my case I blindly copied code from a previous project that required quotes around the value. The logcat output complaining about the authorization failure showed them but it took me a while to realize they weren’t supposed to be there:

    ... E/Google Maps Android API﹕ Authorization failure. Please see developers.google.com/maps/documentation/android-api/start for how to correctly set up the map.
    ... E/Google Maps Android API: In the Google Developer Console (https://console.developers.google.com)
      Ensure that the "Google Maps Android API v2" is enabled.
      Ensure that the following Android Key exists:                                                                               	
        API Key: "AIzaSy...Q4"                                                                               	
        Android Application (<cert_fingerprint>;<package_name>): 
        CC:6E:...:62:41;de.dbremes....
    

    The logcat output will show the actual values sent to Google’s servers. So an Ctrl+C here followed by a Ctrl+F, Ctrl+V in the Google Developer Console makes it easy to check the values match – unless you fall in a similar trap like me.

    But this post just has to end with another surprise. When you change an API key in the external file Gradle won’t pick up that change and even switching between build types and recompiling the project won’t help – surprise #4. The solution is to close and reopen Android Studio.

    Additional Resources

    Posted in Android | Tagged | Leave a comment

    DbTradeAlert for Android: Automated UI Tests – Part 3: Testing with WireMock and Firebase Test Lab for Android

    Update 2016-11-24: the bug in handling URL encoded parameters in WireMock is fixed. But the current version throws a “IllegalArgumentException: resource assets not found” before even starting a test. For now DbTradeAlert stays with Wiremock version 2.2.1.

    Update 2016-11-08: the free Spark plan now includes 5 tests per day on physical devices in Firebase Test Lab for Android – no credit card required. Details see Firebase Dev Summit.


    First post in this series: Introduction to DbTradeAlert

    Previous post: Automated UI Tests – Part 2: Testing with Espresso

    This post jumps right into the middle of adding a scrolling experience test that was identified as being necessary for DbTradeAlert. Read the previous post to get an idea of the test and the one before it to find out how that scrolling experience test came about.


    5.5 Mock Internet Access with WireMock

    In its current state the test downloads quotes from the Internet when the recorded action to tap the Refresh button is executed. And to avoid starting the swipes before the list is filled with reports from those quotes the test uses a Thread.sleep(). This approach has various problems:

    • There may simply be no Internet access – for example on a build server
    • Waiting slows down the tests – especially a problem on Android as tests have to be repeated on various devices, Android versions, regional settings, …
    • The quote data is unpredictable – quotes downloaded monday morning will be 3 days old, values may be missing, downloads may result in an error code, …

    The solution to all this is to avoid accessing the Internet altogether and just provide canned data. As replacing external components during tests is a common requirement it has a common fulfillment: mocking.

    This test will use WireMock which serves as a proxy and responds with a canned .csv file to DbTradeAlert’s rerouted request. WireMock can do much more like return arbitrary HTTP response codes or take any time to answer at all. This will come in handy for later tests.

    5.5.1 Download Sample Quotes

    While WireMock can be used to record responses it’s easier to just download a .csv file directly. The URL can be determined from:

    • the base URL in QuoteRefresherService.onHandleIntent()
    • the format string specifying the data fields and their order in DbHelper.QuoteDownloadFormatParameter
    • the initial symbols and those from ch_securities.csv

    The URL will look like this:
    http://download.finance.yahoo.com/d/quotes.csv?f=aa2bc4d1ghl1nopp2st1vx&s=ABBN.VX+CFR.VX+NESN.VX+NOVN.VX+ROG.VX+SYNN.VX+UBSG.VX+ZURN.VX

    Just enter it into a browser and save the downloaded quotes.csv file to “…\app\src\androidTest\assets”. To be shure check the asset shows up in Android Studio’s Project view as “quotes.csv (androidTest)”. The file is exactly what the app would have downloaded itself.

    5.5.2 Integrate WireMock into the Project

    The first step to integrate WireMock is to specify the dependencies in “build.gradle (Module: app)”:

    apply plugin: 'com.android.application'
    
    android {
        compileSdkVersion 23
        buildToolsVersion "23.0.3"
    
        defaultConfig {
            applicationId "de.dbremes.dbtradealert"
            minSdkVersion 15
            targetSdkVersion 23
            versionCode 2
            versionName "1.1"
            archivesBaseName = "${parent.name}-${android.defaultConfig.versionName}"
            // Ad unit Id for sample adverts from
            // https://firebase.google.com/docs/admob/android/google-services.json
            buildConfigField "String", "AD_UNIT_ID", "\"ca-app-pub-3940256099942544/6300978111\""
            testInstrumentationRunner 'android.support.test.runner.AndroidJUnitRunner'
        }
        buildTypes {
            debug {
                // Multidex only needed for WireMock
                multiDexEnabled true
            }
            release {
                minifyEnabled false
                // ProGuard is off
                proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
                // Keep actual ad unit Id secret by loading it from file outside of Git repo
                Properties props = new Properties()
                props.load(new FileInputStream("$project.rootDir/../../DbTradeAlert/project.properties"))
                buildConfigField "String", "AD_UNIT_ID", "\"${props.getProperty("ad_unit_id")}\""
            }
        }
        productFlavors {
            naked
            playStore {
                applicationId = "${android.defaultConfig.applicationId}.playStore"
            }
            withAds {
                applicationId = "${android.defaultConfig.applicationId}.withAds"
            }
        }
        sourceSets {
            playStore.java.srcDirs = ['src/common/java', 'src/playStore/java']
            withAds.java.srcDirs = ['src/common/java', 'src/withAds/java']
        }
    }
    
    dependencies {
        compile fileTree(include: ['*.jar'], dir: 'libs')
        testCompile 'junit:junit:4.12'
        compile 'com.android.support:appcompat-v7:23.3.0'
        compile 'com.android.support:design:23.3.0'
        compile 'com.android.support:support-v4:23.3.0'
        compile 'com.android.support:recyclerview-v7:23.3.0'
        playStoreCompile 'com.google.firebase:firebase-core:9.4.0'
        playStoreCompile 'com.google.firebase:firebase-crash:9.4.0'
        playStoreCompile 'com.google.firebase:firebase-config:9.4.0'
        withAdsCompile 'com.google.firebase:firebase-core:9.4.0'
        withAdsCompile 'com.google.firebase:firebase-ads:9.4.0'
        withAdsCompile 'com.google.firebase:firebase-crash:9.4.0'
        withAdsCompile 'com.google.firebase:firebase-config:9.4.0'
        compile 'org.jetbrains:annotations-java5:15.0'
        androidTestCompile 'com.android.support.test.espresso:espresso-core:2.2.2', {
            exclude group: 'com.android.support', module: 'support-annotations'
        }
        androidTestCompile 'com.android.support.test.espresso:espresso-contrib:2.2.2', {
            exclude group: 'com.android.support', module: 'support-annotations'
            exclude group: 'com.android.support', module: 'support-v4'
            exclude group: 'com.android.support', module: 'design'
            exclude group: 'com.android.support', module: 'recyclerview-v7'
        }
        androidTestCompile 'com.github.tomakehurst:wiremock:2.2.1', {
            // Allows us to use the Android version of Apache httpclient instead
            exclude group: 'org.apache.httpcomponents', module: 'httpclient'
            // Resolves the Duplicate Class Exception
            // duplicate entry: org/objectweb/asm/AnnotationVisitor.class
            exclude group: 'org.ow2.asm', module: 'asm'
            // Fixes Warning conflict with Android's version of org.json
            // org.json:json:20090211 is ignored for debugAndroidTest as it may be conflicting
            // with the internal version provided by Android.
            exclude group: 'org.json', module: 'json'
        }
        // Android compatible version of Apache httpclient.
        androidTestCompile 'org.apache.httpcomponents:httpclient-android:4.3.5.1'
        androidTestCompile 'com.android.support:multidex:1.0.0'
    }
    // Remove apply plugin: 'com.google.gms.google-services' when building naked variants.
    // Those have no google-services.json file and applying the plugin would produce an error:
    // "File google-services.json is missing. The Google Services Plugin cannot function without it."
    // Building the R class would fail too and produce "cannot resolve symbol R" errors.
    apply plugin: 'com.google.gms.google-services'
    // Non-naked variants on the other hand will throw an exception when started after being build
    // without this line: "FirebaseApp with name [DEFAULT] doesn't exist"
    

    Because WireMock will only be used in instrumented tests references to it are noted with “androidTestCompile” – “testCompile” would apply to unit tests and “compile” affects the app itself.

    Adding WireMock requires the use of multidexing – see lines 22 and 86. That’s because Android apps have an upper limit of 64K (65,536) methods and WireMock with its dependencies adds more than 65,000 methods alone. The limit actually affects the app’s DEX (Dalvik EXecutable) file and the solution is to use more than one DEX file. Fortunately Gradle will take care of that when requested. Also WireMock will only be added to the test apk and not to the app itself and Gradle is smart enough to use multidexing only when necessary.

    Note that deviating from Handstandsam’s (Sam Edwards) reference build.gradle file – see lines 73 to 83 – I had to change an exclude from
    exclude group: 'asm', module: 'asm' to
    exclude group: 'org.ow2.asm', module: 'asm' to get rid of the error
    ZipException: duplicate entry: org/objectweb/asm/AnnotationVisitor.class
    Now how does one find the culprit?

    The first step is to list all the app’s references, and their references, and their references, … While you can view them in Android Studio I found the command line output easier to digest. In Android Studio’s Terminal window enter:
    gradlew app:dependencies --configuration androidTestCompile

    Gradle or actually its wrapper gradlew defaults to the current project’s directory and you’ll get an output like this:

    C:\Users\Admin\Documents\AndroidStudioProjects\DbTradeAlert>gradlew app:dependencies --configuration mockCompile
    :app:dependencies                                                                         
    
    ------------------------------------------------------------
    Project :app
    ------------------------------------------------------------
    
    mockCompile - Classpath for compiling the mock sources.
    +--- com.android.support.test.espresso:espresso-core:2.2.2
    |    +--- com.squareup:javawriter:2.1.1
    |    +--- com.android.support.test:rules:0.5
    |    |    \--- com.android.support.test:runner:0.5
    |    |         +--- junit:junit:4.12
    |    |         |    \--- org.hamcrest:hamcrest-core:1.3
    |    |         \--- com.android.support.test:exposed-instrumentation-api-publish:0.5
    |    +--- com.android.support.test:runner:0.5 (*)
    |    +--- javax.inject:javax.inject:1
    |    +--- org.hamcrest:hamcrest-library:1.3
    |    |    \--- org.hamcrest:hamcrest-core:1.3
    |    +--- com.android.support.test.espresso:espresso-idling-resource:2.2.2
    |    +--- org.hamcrest:hamcrest-integration:1.3
    |    |    \--- org.hamcrest:hamcrest-library:1.3 (*)
    |    +--- com.google.code.findbugs:jsr305:2.0.1
    |    \--- javax.annotation:javax.annotation-api:1.2
    +--- com.android.support.test.espresso:espresso-contrib:2.2.2
    |    +--- com.google.android.apps.common.testing.accessibility.framework:accessibility-test-framework:2.0
    |    |    \--- org.hamcrest:hamcrest-core:1.3
    |    \--- com.android.support.test.espresso:espresso-core:2.2.2 (*)
    +--- com.github.tomakehurst:wiremock:2.2.1
    |    +--- org.eclipse.jetty:jetty-server:9.2.13.v20150730
    |    |    +--- javax.servlet:javax.servlet-api:3.1.0
    |    |    +--- org.eclipse.jetty:jetty-http:9.2.13.v20150730
    |    |    |    \--- org.eclipse.jetty:jetty-util:9.2.13.v20150730
    |    |    \--- org.eclipse.jetty:jetty-io:9.2.13.v20150730
    |    |         \--- org.eclipse.jetty:jetty-util:9.2.13.v20150730
    |    +--- org.eclipse.jetty:jetty-servlet:9.2.13.v20150730
    |    |    \--- org.eclipse.jetty:jetty-security:9.2.13.v20150730
    |    |         \--- org.eclipse.jetty:jetty-server:9.2.13.v20150730 (*)
    |    +--- org.eclipse.jetty:jetty-servlets:9.2.13.v20150730
    |    |    +--- org.eclipse.jetty:jetty-continuation:9.2.13.v20150730
    |    |    +--- org.eclipse.jetty:jetty-http:9.2.13.v20150730 (*)
    |    |    +--- org.eclipse.jetty:jetty-util:9.2.13.v20150730
    |    |    \--- org.eclipse.jetty:jetty-io:9.2.13.v20150730 (*)
    |    +--- org.eclipse.jetty:jetty-webapp:9.2.13.v20150730
    |    |    +--- org.eclipse.jetty:jetty-xml:9.2.13.v20150730
    |    |    |    \--- org.eclipse.jetty:jetty-util:9.2.13.v20150730
    |    |    \--- org.eclipse.jetty:jetty-servlet:9.2.13.v20150730 (*)
    |    +--- com.google.guava:guava:18.0
    |    +--- com.fasterxml.jackson.core:jackson-core:2.6.1
    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.6.1
    |    +--- com.fasterxml.jackson.core:jackson-databind:2.6.1
    |    |    +--- com.fasterxml.jackson.core:jackson-annotations:2.6.0 -> 2.6.1
    |    |    \--- com.fasterxml.jackson.core:jackson-core:2.6.1
    |    +--- org.xmlunit:xmlunit-core:2.1.1
    |    +--- org.xmlunit:xmlunit-legacy:2.1.1
    |    |    +--- org.xmlunit:xmlunit-core:2.1.1
    |    |    \--- junit:junit:3.8.1 -> 4.12 (*)
    |    +--- com.jayway.jsonpath:json-path:2.2.0
    |    |    +--- net.minidev:json-smart:2.2.1
    |    |    |    \--- net.minidev:accessors-smart:1.1
    |    |    |         \--- org.ow2.asm:asm:5.0.3
    |    |    \--- org.slf4j:slf4j-api:1.7.16
    |    +--- org.slf4j:slf4j-api:1.7.12 -> 1.7.16
    |    +--- net.sf.jopt-simple:jopt-simple:4.9
    |    +--- junit:junit:4.12 (*)
    |    +--- org.apache.commons:commons-lang3:3.4
    |    \--- com.flipkart.zjsonpatch:zjsonpatch:0.2.1
    |         +--- com.fasterxml.jackson.core:jackson-databind:2.3.2 -> 2.6.1 (*)
    |         +--- com.fasterxml.jackson.core:jackson-core:2.3.2 -> 2.6.1
    |         +--- com.google.guava:guava:18.0
    |         \--- org.apache.commons:commons-collections4:4.0
    +--- org.apache.httpcomponents:httpclient-android:4.3.5.1
    \--- com.android.support:multidex:1.0.0
    
    (*) - dependencies omitted (listed previously)
    
    BUILD SUCCESSFUL
    

    Of most interest is of course everything under “com.github.tomakehurst:wiremock” as the error wasn’t there before adding it. Neither the class name “AnnotationVisitor” nor the namespace “objectweb” show up but “org.ow2.asm” looks interesting.

    The next step is to change “exclude group: ‘asm’, module: ‘asm'” to “exclude group: ‘org.ow2.asm’, module: ‘asm'” and list the dependencies again – “org.ow2.asm:asm:5.0.3” was gone. And so was the ZipException when compiling the app again.

    Solving that problem makes way for the next one:

    com.android.build.api.transform.TransformException: com.android.builder.packaging.DuplicateFileException: Duplicate files copied in APK META-INF/LICENSE
    File1: C:\Users\Admin\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-core\2.6.1\892d15011456ea3563319b27bdd612dbc89bb776\jackson-core-2.6.1.jar
    File2: C:\Users\Admin\.gradle\caches\modules-2\files-2.1\org.apache.httpcomponents\httpclient-android\4.3.5.1\eecbb0b998e77629862a13d957d552b3be58fc4e\httpclient-android-4.3.5.1.jar
    File3: C:\Users\Admin\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-databind\2.6.1\45c37a03be19f3e0db825fd7814d0bbec40b9e0\jackson-databind-2.6.1.jar
    File4: C:\Users\Admin\.gradle\caches\modules-2\files-2.1\com.fasterxml.jackson.core\jackson-annotations\2.6.1\f9661ddd2456d523b9428651c61e34b4ebf79f4e\jackson-annotations-2.6.1.jar
    

    As the Exception details show the problem is that four LICENSE files are being copied to the same place but of course only one can make it. Luckily checking the project’s licenses showed that all four are Apache 2.0 licenses so adding only one of them is legal:

    apply plugin: 'com.android.application'
    
    android {
        compileSdkVersion 23
        buildToolsVersion "23.0.3"
    
        defaultConfig {
            // ...
        }
        buildTypes {
            // ...
        }
        productFlavors {
            // ...
        }
        sourceSets {
            // ...
        }
        packagingOptions {
            // Keep only 1 license from 3* com.fasterxml.jackson.core + 1* org.apache.httpcomponents
            // All 4 are Apache 2.0 licenses
            pickFirst  'META-INF/LICENSE'
        }
    }
    
    dependencies {
        // ...
    }
    // ...
    

    Finally the app builds with WireMock included.

    5.5.3 Integrate WireMock into the Test

    For the scrolling experience test the app needs to redirect its quotes request to WireMock and preferably without the redirect affecting other tests or the app itself. In other words it needs a specific build variant applying that mock.

    But once you define a mock build type and switch to it the ScrollingExperienceTest class won’t compile anymore. That’s because Gradle evaluates the testCompile and androidTestCompile dependencies only for the debug build type. While a ‘testBuildType “mock”‘ fixes that only one build type can be specified which will shift the problem around. That means to have debug and mock builds working you’d have to add a mockCompile for each testCompile and androidTestCompile.

    In this case the test only runs on the mock build so all androidTestCompile specific dependencies for Espresso and WireMock related libraries can be replaced with mockCompile dependencies.

    Changes to build.gradle:

    • Lines 23 to 32 define the mock build
    • Line 24 shows how to initialize a build type
    • Line 68 replaces the debug with the mock build type as the default build type for tests
    • The new build type is applied for example in line 86
    apply plugin: 'com.android.application'
    
    android {
        compileSdkVersion 23
        buildToolsVersion "23.0.3"
    
        defaultConfig {
            applicationId "de.dbremes.dbtradealert"
            minSdkVersion 15
            targetSdkVersion 23
            versionCode 2
            versionName "1.1"
            archivesBaseName = "${parent.name}-${android.defaultConfig.versionName}"
            // Ad unit Id for sample adverts from
            // https://firebase.google.com/docs/admob/android/google-services.json
            buildConfigField "String", "AD_UNIT_ID", "\"ca-app-pub-3940256099942544/6300978111\""
            // Prepare rerouting of quote download requests to WireMock for tests
            buildConfigField "String", "HOST", "download.finance.yahoo.com"
            buildConfigField "String", "PORT", "\"80\""
            testInstrumentationRunner 'android.support.test.runner.AndroidJUnitRunner'
        }
        buildTypes {
            mock {
                initWith(buildTypes.debug)
                // Without enhanced google-services.json his leads to the error
                // "No matching client found for package name 'de.dbremes.dbtradealert.playStore.mock'":
                //applicationIdSuffix ".mock"
                buildConfigField "String", "HOST", "\"127.0.0.1\""
                buildConfigField "String", "PORT", "\"8080\""
                // Multidex only needed for WireMock
                multiDexEnabled true
            }
            release {
                minifyEnabled false
                // ProGuard is off
                proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
                // Keep actual ad unit Id secret by loading it from file outside of Git repo
                Properties props = new Properties()
                props.load(new FileInputStream("$project.rootDir/../../DbTradeAlert/project.properties"))
                buildConfigField "String", "AD_UNIT_ID", "\"${props.getProperty("ad_unit_id")}\""
            }
        }
        productFlavors {
            naked
            playStore {
                applicationId = "${android.defaultConfig.applicationId}.playStore"
            }
            withAds {
                applicationId = "${android.defaultConfig.applicationId}.withAds"
            }
        }
        sourceSets {
            playStore.java.srcDirs = ['src/common/java', 'src/playStore/java']
            withAds.java.srcDirs = ['src/common/java', 'src/withAds/java']
        }
        variantFilter { variant ->
            def names = variant.flavors*.name
            // Only ...-playStore-debug.apk needed of the mocks
            if (variant.buildType.name.equals("mock") && names.contains("playStore") == false) {
                variant.ignore = true
            }
        }
        packagingOptions {
            // Keep only 1 license from 3* com.fasterxml.jackson.core + 1* org.apache.httpcomponents
            // All 4 are Apache 2.0 licenses
            pickFirst  'META-INF/LICENSE'
        }
        testBuildType "mock"
    }
    
    dependencies {
        compile fileTree(include: ['*.jar'], dir: 'libs')
        testCompile 'junit:junit:4.12'
        compile 'com.android.support:appcompat-v7:23.3.0'
        compile 'com.android.support:design:23.3.0'
        compile 'com.android.support:support-v4:23.3.0'
        compile 'com.android.support:recyclerview-v7:23.3.0'
        playStoreCompile 'com.google.firebase:firebase-core:9.4.0'
        playStoreCompile 'com.google.firebase:firebase-crash:9.4.0'
        playStoreCompile 'com.google.firebase:firebase-config:9.4.0'
        withAdsCompile 'com.google.firebase:firebase-core:9.4.0'
        withAdsCompile 'com.google.firebase:firebase-ads:9.4.0'
        withAdsCompile 'com.google.firebase:firebase-crash:9.4.0'
        withAdsCompile 'com.google.firebase:firebase-config:9.4.0'
        compile 'org.jetbrains:annotations-java5:15.0'
        mockCompile 'com.android.support.test.espresso:espresso-core:2.2.2', {
            exclude group: 'com.android.support', module: 'support-annotations'
        }
        mockCompile 'com.android.support.test.espresso:espresso-contrib:2.2.2', {
            exclude group: 'com.android.support', module: 'support-annotations'
            exclude group: 'com.android.support', module: 'support-v4'
            exclude group: 'com.android.support', module: 'design'
            exclude group: 'com.android.support', module: 'recyclerview-v7'
        }
        mockCompile 'com.github.tomakehurst:wiremock:2.2.1', {
            // Allows us to use the Android version of Apache httpclient instead
            exclude group: 'org.apache.httpcomponents', module: 'httpclient'
            // Resolves the Duplicate Class Exception
            // duplicate entry: org/objectweb/asm/AnnotationVisitor.class
            exclude group: 'org.ow2.asm', module: 'asm'
            // Fixes Warning conflict with Android's version of org.json
            // org.json:json:20090211 is ignored for debugAndroidTest as it may be conflicting
            // with the internal version provided by Android.
            exclude group: 'org.json', module: 'json'
        }
        // Android compatible version of Apache httpclient.
        mockCompile 'org.apache.httpcomponents:httpclient-android:4.3.5.1'
        mockCompile 'com.android.support:multidex:1.0.0'
    }
    // Remove apply plugin: 'com.google.gms.google-services' when building naked variants.
    // Those have no google-services.json file and applying the plugin would produce an error:
    // "File google-services.json is missing. The Google Services Plugin cannot function without it."
    // Building the R class would fail too and produce "cannot resolve symbol R" errors.
    apply plugin: 'com.google.gms.google-services'
    // Non-naked variants on the other hand will throw an exception when started after being build
    // without this line: "FirebaseApp with name [DEFAULT] doesn't exist"
    

    A side effect of introducing a new build type is the explosion of variants from 6 to 9. As only the new playStore-mock variant is needed a variantFilter eliminates the others – see lines 56 to 62.

    When adding a build type or flavor it’s a good idea to check all uses of the BuildConfig class to make shure they still work. All was good in this case and finally it’s time to work on the test itself.

    The actual redirect is achieved by assigning a host and port number depending on the build type – see lines 17 to 19 and 28 to 29 in build.gradle above.

    Note how all buildConfigField values are explicitly enclosed in quotation marks despite being declared as strings. Without that weird errors like “error: package download.finance does not exist” pop up because the values end up unquoted in BuildConfig.java.

    The base URL specified in QuoteRefresherService.onHandleIntent() is then replaced with
    String baseUrl = "http://" + BuildConfig.HOST + ":" + BuildConfig.PORT + "/d/quotes.csv";

    For mock builds BuildConfig.HOST will be 127.0.0.1 and BuildConfig.PORT will be 8080 while any other build still uses the real world values.

    The ScrollingExperienceTest class will look like this after integrating WireMock:

    // ...
    @LargeTest
    @RunWith(AndroidJUnit4.class)
    public class ScrollingExperienceTest {
        private static final String CLASS_NAME = "ScrollingExperienceTest";
        private String symbolParameterValue;
    
        @Rule
        public ActivityTestRule<WatchlistListActivity> mActivityTestRule
                = new ActivityTestRule<>(WatchlistListActivity.class);
    
        @Rule
        public WireMockRule wireMockRule = new WireMockRule(Integer.valueOf(BuildConfig.PORT));
    
        private void createTestData() {
            Log.v(CLASS_NAME, "createTestData(): start");
            String testDataString = readFileFromTestAssets("ch_securities.csv");
            Context targetContext = InstrumentationRegistry.getTargetContext();
            DbHelper dbHelper = new DbHelper(targetContext);
            // SQLite Ids start with 1; 1 == CH watchlist
            long watchlistId = 1;
            dbHelper.importTestSecurities(testDataString, watchlistId);
        } // createTestData()
    
        @After
        public void deleteTestData() {
            Log.v(CLASS_NAME, "@After - deleteTestData(): start");
            Context targetContext = InstrumentationRegistry.getTargetContext();
            DbHelper dbHelper = new DbHelper(targetContext);
            dbHelper.deleteTestSecurities();
            logRequests();
        } // deleteTestData()
    
        private void logRequests() {
            final String METHOD_NAME = "logRequests";
            // allServeEvents
            List<ServeEvent> allServeEvents = getAllServeEvents();
            Log.v(CLASS_NAME, METHOD_NAME + "(): allServeEvents.size() = " + allServeEvents.size());
            for (int i = 0; i < allServeEvents.size(); i++) {
                Log.v(CLASS_NAME, String.format("%s(): allServeEvents[%d].Url = %s",
                        METHOD_NAME, i, allServeEvents.get(i).getRequest().getUrl()));
            }
            // unmatchedRequests
            List<LoggedRequest> unmatchedRequests = findUnmatchedRequests();
            Log.v(CLASS_NAME, METHOD_NAME + "(): unmatchedRequests.size() = " + unmatchedRequests.size());
            for (int i = 0; i < unmatchedRequests.size(); i++) {
                Log.v(CLASS_NAME, String.format("%s(): unmatchedRequests[%d] = %s",
                        METHOD_NAME, i, unmatchedRequests.get(i).toString()));
            }
            // nearMisses
            // Currently WireMock reports expected requests as received, too
            // See https://github.com/tomakehurst/wiremock/issues/484
            List<NearMiss> nearMisses = findNearMissesForAllUnmatched();
            Log.v(CLASS_NAME, METHOD_NAME + "(): nearMisses.size() = " + nearMisses.size());
            for (int i = 0; i < nearMisses.size(); i++) { Log.v(CLASS_NAME, String.format("%s(): nearMisses[%d].Diff = %s", METHOD_NAME, i, nearMisses.get(i).getDiff())); } } // logRequests() @Before public void prepareTest() { Log.v(CLASS_NAME, "@Before - prepareTest(): start"); try { // Must match imported symbols from ch_securities.csv + sample symbols from install: symbolParameterValue = URLEncoder.encode( "ABBN.VX+BAYN.DE+CFR.VX+NESN.VX+NOVN.VX+ROG.VX+SIE.DE+SYNN.VX+UBSG.VX+ZURN.VX", "utf-8"); } catch (UnsupportedEncodingException e) { PlayStoreHelper.logError(e); } createTestData(); setupWireMock(); } // prepareTest() private String readFileFromTestAssets(String fileName) { String fileContent = ""; Context testContext = InstrumentationRegistry.getContext(); try { InputStream testDataStream = testContext.getResources().getAssets().open(fileName); byte[] testData = new byte[testDataStream.available()]; int bytesRead = testDataStream.read(testData); fileContent = new String(testData); } catch (IOException e) { PlayStoreHelper.logError(e); } return fileContent; } // readFileFromTestAssets() private void setupWireMock() { Log.v(CLASS_NAME, "setupWireMock(): start"); String quotesCsv = readFileFromTestAssets("quotes.csv"); wireMockRule.stubFor(get(urlPathMatching(".*")) .withQueryParam("f", equalTo(DbHelper.QuoteDownloadFormatParameter)) // Currently WireMock fails to match URL encoded parameters // See https://github.com/tomakehurst/wiremock/issues/515 //.withQueryParam("s", equalTo(symbolParameterValue)) .willReturn(aResponse() .withStatus(HttpURLConnection.HTTP_OK) .withBody(quotesCsv))); } // setupWireMock() @Test public void scrollingExperienceTest() throws VerificationException { final String METHOD_NAME = "scrollingExperienceTest"; Log.v(CLASS_NAME, METHOD_NAME + "(): start"); // Make added securities show up in watchlist: Log.v(CLASS_NAME, METHOD_NAME + "(): tapping Refresh"); ViewInteraction actionMenuItemView = onView( allOf(withId(R.id.action_refresh), withContentDescription("Refresh"), isDisplayed())); actionMenuItemView.perform(click()); // Checking request was correct helps to track down reasons for failed tests verify(1, getRequestedFor(urlPathMatching(".*")) .withQueryParam("f", equalTo(DbHelper.QuoteDownloadFormatParameter)) // Currently WireMock fails to match URL encoded parameters // See https://github.com/tomakehurst/wiremock/issues/515 //.withQueryParam("s", equalTo(symbolParameterValue)) ); Log.v(CLASS_NAME, METHOD_NAME + "(): verified correctness of request"); // Scroll items to trigger creation of new items // Visually check if scrolling stutters -> test failed
            ViewInteraction recyclerView = onView(
                    allOf(withId(R.id.list),
                            withParent(allOf(withId(R.id.container),
                                    withParent(withId(R.id.main_content)))),
                            isDisplayed()));
            for (int i = 0; i < 8; i++) {
                Log.v(CLASS_NAME, METHOD_NAME + "(): swipeUp" + i);
                recyclerView.perform(actionOnItemAtPosition(i, swipeUp()));
            }
            Log.v(CLASS_NAME, METHOD_NAME + "(): the End");
        } // scrollingExperienceTest()
    } // class ScrollingExperienceTest
    

    The first step is to create an instance of the WireMockRule class which takes care of starting and stopping the server as well as resetting it between tests. In this case the creation order of the Rule instances doesnt matter but if there was a dependency you’d add a RuleChain.

    In addition to load more symbols before starting the test now we have to tell WireMock what result to serve for which request. Therefore a new prepareTest() method gets the @Before annotation. Before calling createTestData() this method creates the symbolParameterValue which WireMock will have to expect. In this case it’s important to URL encode this value as the “+” characters would represent spaces.

    Finally prepareTest() calls setupWireMock(). Like createTestData() this method needs to read a file from the test APK’s assets – the previously added quotes.csv. For that reason readFileFromTestAssets() was factored out of createTestData().

    The next step is to define a stub – what happens when. In this case we don’t care about the URL but for the two parameters because they define what columns and rows to return. If the parameters match WireMock should return the contents of quotes.csv and a response code of HttpURLConnection.HTTP_OK because QuoteRefresherService.downloadQuotes() will check that response code. Because of a bug WireMock currently fails to verify URL encoded parameters so only the format parameter is checked.

    Finally everything is ready to fix the actual test. In scrollingExperienceTest() the call to Thread.sleep() is gone and a verify() has taken its place. That method basically checks if the stub defined in setupWireMock() got a single exact request – again accounting for the current bug with URL encoded parameters.

    In addition to that logRequests() gives you an idea what requests WireMock received and how it triaged them. Please note that due to another bug in WireMock currently NearMiss.getDiff() will report the expected request also as the received request.

    After running the test the data it created is deleted like before.

    A side note for those with multiple tests in a test class. Like rules the tests are executed in no specific order. That’s because both are identified by reflection which doesn’t make any promises about order. For tests one might actually call that a feature because it gets in the way of tests that depend on each other which is usually a bad idea. But if you need it JUnit from version 4.11 on provides a @FixMethodOrder to run tests for example ordered by name.

    5.6 Run the Test in Firebase Test Lab for Android

    Basically everything up to now has been a preparation for this step. Firebase Test Lab for Android provides those low end devices on which scrolling may stutter. And yes, those are physical devices.

    As using Firebase Test Lab for Android isn’t free you’ll have to give up the free Spark plan. I chose the pay-as-you-go Blaze plan.

    To upgrade to the Blaze plan:

    1. Navigate to the Firebase console at https://console.firebase.google.com and log in
    2. Select your project with the respective app
    3. In the navigation pane select Test Lab
    4. On the next page click Upgrade
    5. On the new overlay click Select Plan in the Blaze column
    6. Yet another overlay informs you about upgrading the project – click Continue
    7. Accept or change your country which determines the currency
    8. It seems you cannot use the private account created in the beginning but have to set up a business account. Fill in the missing information like the business’ name, phone number, and credit card details and click Confirm Purchase.
    9. On the confirmation overlay click Got It
    10. The page now shows a Run Your First Test button

    Google advises to add its Test Lab screenshot library to the app’s test project. That enables you to initiate screenshots from test code. As the scrolling experience test doesn’t benefit from screenshots and I skipped that step.

    A very important part of setting up the test is to select the devices and Android versions to test on. Usually you want to identify those most popular with your target audience and some extremes. The scrolling experience test just needs some low end devices and can use whatever Andoid versions they support.

    To start the first test:

    1. Navigate to your project in the Test Lab as before
    2. Click the Run Your First Test button
    3. On the Choose Test Type page select Instrumentation Test and click Continue
    4. In the Select App step provide the app and the test APK. These are independent from any APKs you uploaded already. I provided DbTradeAlert-1.1-playStore-mock.apk and DbTradeAlert-1.1-playStore-mock-androidTest.apk from …\DbTradeAlert\app\build\outputs\apk. Click Continue.
    5. In the Select Dimensions step you specify the devices, API levels, orientations, and locales on which to run the tests. The button at the bottom right shows number of permutations this creates – 8 with the default options.
    6. Physical devices: I choose LG Nexus 4 and Motorola Moto G (2nd gen) as they have a low Basemark OS II score
    7. API levels: the selected devices only support API levels 19 and 22 (only Nexus 4)
    8. Orientation: I left both checked but there shouldn’t be much of a difference
    9. Locales: only testing on en_US is OK as the app isn’t localized
    10. Advanced option: timeout of 5 minutes is OK
    11. Finally click Start Tests

    All tests on API level 22 (Lollipop) passed while those on API level 19 (Kitkat) failed with “java.lang.NoClassDefFoundError: com.google.firebase.FirebaseOptions”. Stackoverflow has dozends of threads about this starting in May 2016 when Google renamed a lot of stuff to “Firebase”. They said on Stackoverflow it was a bug on their side but that they have fixed it since.

    As I don’t own a pre-Lollipop device and don’t think the scrolling experience depends on the API level anyway I didn’t dig any further.

    In my case it took about 5 minutes to complete the 6 tests. Surprisingly each test claims that it took about 2.5 minutes while they complete in seconds on my phone. As Google charges USD 5 per hour for physical devices running the tests cost about USD 1.25. Still a bargain compared to maintaining those devices yourself but definitely more than expected.

    Each test result links to details which for example include a video. The videos are about 10 to 15 seconds long with about 5 seconds showing the actual test – like on my physical device. As Google claims not to bill for the setup time I have no idea how those 2.5 minutes come about.

    Unfortunately the video stuttered itself – no wonder with only 11 frames per second. To be sure I ran the test again on their Nexus 5 – same stutter. As one cannot configure devices to show screen update problems – or anything else from developer options – Firebase Test Lab for Android isn’t suited for this kind of test.

    Another interesting outcome was the test failing on API level 23:
    android.support.test.espresso.NoActivityResumedException: No activities in stage RESUMED. Did you forget to launch the activity. (test.getActivity() or similar)?

    Watching the video showed that the app was waiting for input on the “Ignore battery optimizations?” dialog. Easily fixed by skipping the call to WatchlistListActivity.ensureExemptionFromBatteryOptimizations() if BuildConfig.BUILD_TYPE.equals(“mock”).

    That concludes the first automated test deemed interesting. On to the next …

    6. Running a Firebase Test Lab for Android Robo Test

    The Firebase Test Lab for Android can do more than just run tests – it can come up with its own. Those are called Robo tests and not to be confused with the Monkey tests provided by the SDK. The difference is that Robo tests use a more systematic approach and even create a map of the app.

    To start a Robo test:

    1. Navigate to your project in the Test Lab as before
    2. Click the Run a Test button
    3. Select Run a Robo test
    4. Provide an APK, select test dimensions, and start the test as before

    After the tests finish you’ll find details about their outcome as with instrumented tests. But now you get additional screenshots and an activity map.

    Excerpt from the activity map

    Excerpt from the activity map

    That activity map shows the paths through the app as taken by a Robo test. And while that’s a neat feature it also shows the limits of Robo tests – they just tap a button but never enter or change text. Robo tests can still be helpful at early stages of an app where you just want to make shure it doesn’t explode when simply showing a screen on some devices or Android versions. Note that the test was listed as taking 8 minutes and another run took 11 minutes – that’s USD 1.60 spent.

    7. Test Connection Fault Handling

    Until now the app hasn’t really been tested with faulty networks simply because fault simulation wasn’t available. Fortunately WireMock has this capability. It can return HTTP error codes, return HTTP OK but garbage or an empty response, or take forever to answer at all. As the test involves a tap of the Refresh buttton it will be another instrumented test:

    // ...
    
    @LargeTest
    @RunWith(AndroidJUnit4.class)
    public class NetworkFault1Test {
        private static final String CLASS_NAME = "NetworkFault1Test";
    
        @Rule
        public ActivityTestRule<WatchlistListActivity> mActivityTestRule
                = new ActivityTestRule<>(WatchlistListActivity.class);
    
        @Rule
        public WireMockRule wireMockRule = new WireMockRule(Integer.valueOf(BuildConfig.PORT));
    
        @Test
        public void networkFaultTest() throws VerificationException {
            final String METHOD_NAME = "networkFaultTest";
            Log.v(CLASS_NAME, METHOD_NAME + "(): start");
            ViewInteraction actionMenuItemView = onView(
                    allOf(withId(R.id.action_refresh), withContentDescription("Refresh"), isDisplayed()));
            Log.v(CLASS_NAME, METHOD_NAME + "(): Refresh -> HttpURLConnection.HTTP_INTERNAL_ERROR");
            actionMenuItemView.perform(click());
            // In theory one can verify user friendly text + error code in toast. In practice it's
            // nearly impossible to get the timing right - toasts show with a delay and fade away soon.
    //        String toastText
    //                = QuoteRefresherService.QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA
    //                + "download failed (response code " + HttpURLConnection.HTTP_INTERNAL_ERROR + ")!";
    //        onView(withText(toastText))
    //                .inRoot(
    //                        withDecorView(
    //                                not(mActivityTestRule.getActivity().getWindow().getDecorView())
    //                        )
    //                )
    //                .check(matches(isDisplayed()));
    
            Log.v(CLASS_NAME, METHOD_NAME + "(): the End");
        } // networkFaultTest()
    
        @Before
        public void setupWireMock() {
            Log.v(CLASS_NAME, "@Before - setupWireMock(): start");
            wireMockRule.stubFor(get(urlPathMatching(".*"))
                    .willReturn(aResponse()
                            .withStatus(HttpURLConnection.HTTP_INTERNAL_ERROR)
                    )
            );
        } // setupWireMock()
    
    } // class NetworkFault1Test
    

    The code is very similar to the ScrollingExperienceTest class. But this time WireMock will simply return HttpURLConnection.HTTP_INTERNAL_ERROR – 500. So no need for adding sample data here.

    The resulting toast will show “Error: download failed (response code 500)!” and could theoretically be verified with Espresso. But getting the timing right will be a problem as the toast shows with a slight delay and fades away soon.

    Therefore I opted for a manual approach. Remember that the point of this test is to ensure the app handles network faults logically. Whether a toast informs the user of those faults is a separate decision.

    In this case I put a break point in QuoteRefresherService.downloadQuotes() and one on the last line of NetworkFault1Test.networkFaultTest(). The second breakpoint is important because the app will be terminated as soon as the test finishes and that will be before you had any chance to single step through QuoteRefresherService.downloadQuotes().

    The second test is very similar but WireMock will take 20 seconds to answer – longer than the timeouts in QuoteRefresherService.downloadQuotes().

    // ...
    
    @LargeTest
    @RunWith(AndroidJUnit4.class)
    public class NetworkFault2Test {
        // ...
    
        @Before
        public void setupWireMock() {
            Log.v(CLASS_NAME, "@Before - setupWireMock(): start");
            wireMockRule.stubFor(get(urlPathMatching(".*"))
                    .willReturn(aResponse()
                            .withStatus(HttpURLConnection.HTTP_OK)
                            // Must be longer than timeouts in QuoteRefresherService.downloadQuotes()
                            .withFixedDelay(20000)
                    )
            );
        } // setupWireMock()
    
        // ...
    }
    

    This will result in another toast with a slightly different message of “Error: connection timed out!”.

    And yes, NetworkFault2Test is a copy of NetworkFault1Test. That comes from the necessity to have separate WireMock instances as the non-invasive approach to testing gives no chance to add URL parameters or anything else to control which stub gets used.

    A more elegant approach would be to use WireMock’s stateful behavior. While I scrapped that idea because the “second breakpoint” mentioned above got overrun it’s an interesting solution for similar problems:

    // ...
    
    @LargeTest
    @RunWith(AndroidJUnit4.class)
    public class NetworkFaultTest {
        // ...
    
        @Before
        public void setupWireMock() {
            Log.v(CLASS_NAME, "@Before - setupWireMock(): start");
            // Internal server error scenario
            wireMockRule.stubFor(get(urlPathMatching(".*"))
                    .inScenario(CLASS_NAME)
                    .whenScenarioStateIs(STARTED)
                    .willReturn(aResponse()
                            .withStatus(HttpURLConnection.HTTP_INTERNAL_ERROR)
                    )
                    .willSetStateTo("Timeout scenario")
            );
            // Timeout scenario
            wireMockRule.stubFor(get(urlPathMatching(".*"))
                    .inScenario(CLASS_NAME)
                    .whenScenarioStateIs("Timeout scenario")
                    .willReturn(aResponse()
                            .withStatus(HttpURLConnection.HTTP_OK)
                            // Must be longer than timeouts in QuoteRefresherService.downloadQuotes()
                            .withFixedDelay(20000)
                    )
                    .willSetStateTo("Garbage data scenario")
            );
            // Garbage data scenario
            wireMockRule.stubFor(get(urlPathMatching(".*"))
                    .inScenario(CLASS_NAME)
                    .whenScenarioStateIs("Garbage data scenario")
                    .willReturn(aResponse()
                            //.withStatus(HttpURLConnection.HTTP_OK)
                            // Returns HttpURLConnection.HTTP_OK + garbage data
                            .withFault(Fault.MALFORMED_RESPONSE_CHUNK)
                    )
            );
        } // setupWireMock()
    
        // ...
    }
    

    WireMock offers stateful behavior which you are free to configure with scenarios and their states. Each scenario has an initial state of STARTED and defines the next state by calling willSetStateTo(). Subsequent states set their name with whenScenarioStateIs(). This way setupWireMock() creates three consecutive states allowing to control which stub gets used without changing the URL. While it didn’t work out here this would be handy for example to test the calculation behind trailing targets (trailing stop loss) as that would require a specific order of quotes.

    One could also use an abstract base class but I found documenting how the test works and getting at the stub configuration for meaningful output more cumbersome than living with copied code.

    The last test uses one of WireMock’s special faults: return HTTP OK but garbage data.

    // ...
    
    @LargeTest
    @RunWith(AndroidJUnit4.class)
    public class NetworkFault3Test {
        // ...
    
        @Before
        public void setupWireMock() {
            Log.v(CLASS_NAME, "@Before - setupWireMock(): start");
            wireMockRule.stubFor(get(urlPathMatching(".*"))
                    .willReturn(aResponse()
                            // Returns HttpURLConnection.HTTP_OK + garbage data
                            .withFault(Fault.MALFORMED_RESPONSE_CHUNK)
                    )
            );
        } // setupWireMock()
    
        // ...
    }
    

    This test actually surfaced a bug and now QuoteRefresherService.downloadQuotes() handles IOExceptions:

    // ...
    
    public class QuoteRefresherService extends IntentService {
        // ...
    
        private String downloadQuotes(String urlString) throws IOException {
            String result = "";
            InputStream inputStream = null;
            try {
                URL url = new URL(urlString);
                HttpURLConnection conn = (HttpURLConnection) url.openConnection();
                conn.setReadTimeout(10000 /* milliseconds */);
                conn.setConnectTimeout(15000 /* milliseconds */);
                conn.setRequestMethod("GET");
                conn.setDoInput(true);
                int responseCode = -1;
                try {
                    // Starts the query
                    conn.connect();
                    responseCode = conn.getResponseCode();
                } catch (SocketTimeoutException e) {
                    sendLocalBroadcast(
                            QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA + "connection timed out!");
                    Log.d(CLASS_NAME,
                            QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA + "connection timed out!");
                }
                if (responseCode == HttpURLConnection.HTTP_OK) {
                    inputStream = conn.getInputStream();
                    try {
                        result = getStringFromStream(inputStream);
                    } catch (IOException x) {
                        sendLocalBroadcast(QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA
                                + "could not read response!");
                        PlayStoreHelper.logError(CLASS_NAME,
                                QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA
                                        + "could not read response!");
                    }
                    Log.d(CLASS_NAME, "downloadQuotes(): got " + result.length() + " characters");
                } else {
                    sendLocalBroadcast(QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA
                            + "download failed (response code " + responseCode + ")!");
                    PlayStoreHelper.logError(CLASS_NAME,
                            QUOTE_REFRESHER_BROADCAST_ERROR_EXTRA
                                    + "download failed (response code " + responseCode + ")!");
                }
            } finally {
                if (inputStream != null) {
                    inputStream.close();
                }
            }
            return result;
        } // downloadQuotes()
    
        // ...
    }
    

    That was the third planned test and now it’s time for the last …

    8. Run a Monkey Test

    A Monkey test exercises your app like a monkey – it does things at random. Fortunately Android SDK includes a pretty smart monkey:

    • It can restrict its input to specified APKs
    • It can favor specified actions
    • It logs what it does
    • It will stop when the app crashes or produces an ANR dialog
    • It remembers exactly what it did

    Android’s monkey will not only send input to the app but also to the system. For example it can turn off WiFi or change the volume. But while that kind of testing is valuable I don’t want it to happen on my phone but on an emulator.

    This line in Android Studio’s Terminal window lets the monkey loose:
    C:\Users\<AccountName>\AppData\Local\Android\sdk\platform-tools\adb shell monkey -p de.dbremes.dbtradealert -v 5000 > %TMP%\monkey_test.txt

    In this case the monkey is ordered to send 5000 inputs to package de.dbremes.dbtradealert and it’s actually smart enough to start the app. The resulting log will be in C:\Users\<AccountName>\AppData\Local\Temp\monkey_test.txt and looks like this:

    :Monkey: seed=1478409080850 count=5000
    :AllowPackage: de.dbremes.dbtradealert
    :IncludeCategory: android.intent.category.LAUNCHER
    :IncludeCategory: android.intent.category.MONKEY
    // Event percentages:
    //   0: 15.0%
    //   1: 10.0%
    //   2: 2.0%
    //   3: 15.0%
    //   4: -0.0%
    //   5: -0.0%
    //   6: 25.0%
    //   7: 15.0%
    //   8: 2.0%
    //   9: 2.0%
    //   10: 1.0%
    //   11: 13.0%
    
    :Switch: #Intent;action=android.intent.action.MAIN;category=android.intent.category.LAUNCHER;launchFlags=0x10200000;component=de.dbremes.dbtradealert/.WatchlistListActivity;end
    
        // Allowing start of Intent { act=android.intent.action.MAIN cat=[android.intent.category.LAUNCHER] cmp=de.dbremes.dbtradealert/.WatchlistListActivity } in package de.dbremes.dbtradealert
    
    :Sending Trackball (ACTION_MOVE): 0:(-2.0,-2.0)
    :Sending Flip keyboardOpen=false
    :Sending Touch (ACTION_DOWN): 0:(145.0,560.0)
    :Sending Touch (ACTION_UP): 0:(145.06223,548.3796)
    :Sending Trackball (ACTION_MOVE): 0:(0.0,-2.0)
    :Sending Touch (ACTION_DOWN): 0:(699.0,577.0)
    :Sending Touch (ACTION_UP): 0:(701.53815,585.20026)
    
        // Rejecting start of Intent { act=android.intent.action.MAIN cat=[android.intent.category.HOME] cmp=com.android.launcher3/.Launcher } in package com.android.launcher3
    
    :Sending Trackball (ACTION_MOVE): 0:(-3.0,0.0)
    
    ...
    

    Line 1 shows the seed value used. That’s important to reproduce the exact same sequence of inputs – just add “-s <SeedValue>” to the command. Of course the app needs to be in the same starting state too.

    Lines 2 to 4 list the packages and categories from which the monkey is allowed to access activities. That’s controlled by the -c and -p command line switches.

    Lines 6 to 17 show how the input is composed which is controlled by the –pct – that’s 2 hyphens – command line switches.

    Lines 19 to 21 show the successful switch to an activity.

    Lines 23 to 29 show various input events that the monkey generated. ACTION_DOWN and ACTION_UP usually happen at diffferent coordinates so that’s actually a swipe.

    Line 31 shows a switch to an activity that was rejected because that activity didn’t match the filter created by the -c and -p command line switches.

    Surprisingly the monkey has no direct means of typing text. That can still happen when it manages to hit the on screen keyboard.

    If you want more control than the monkey provides you can use the monkeyrunner tool. Despite its name it has nothing to do with the test monkey but is an API for Python programs to control devices and emulators as well as the apps on them.

    Additional Resources

    Posted in Uncategorized | Tagged , | Leave a comment

    DbTradeAlert for Android: Automated UI Tests – Part 2: Testing with Espresso

    First post in this series: Introduction to DbTradeAlert

    Previous post: Automated UI Tests – Part 1: Thoughts about Testing

    This post has two additional requirements in regards to the things set-up so far:

    • Android Studio 2.2 or later is needed for the Espresso test recorder
    • Firebase Blaze plan is needed for access to Firebase Test Lab for Android – the post describes the upgrade process

    Also note that this Espresso test will be somewhat special as it requires a human to decide whether the scrolling was smooth. Espresso is capable of recording a fully automated UI test by just clicking on a screen element to add an assert about its contents.


    5. Automated Scrolling Experience Test

    The scrolling experience with larger watchlists on low end devices was identified to be a concern due to the lack of physical devices to test on. Firebase Test Lab for Android provides those to run for example an Espresso test.

    5.1 Record an Espresso Test

    Recording an Espresso test is not as straight forward as one would think because the test recorder is still in beta. I choose to list every error message thrown at me in the process because those problems will be fairly common.

    Let’s record a swipe test first:

    1. In Android Studio select Run | Record Espresso Test
    2. In the Select Deploymet Target window select your device or emulator and click OK
    3. Wait until the Record Your Test window shows up and the app is ready
    4. Swipe up – see comments if that doesn’t get recorded
    5. In the Record Your Test window click Complete Recording
    6. In the following dialog enter a class name for the test – “ScrollingExperienceTest” in my case – and click Save
    7. Accept the offer to add missing dependencies

    That should leave you with an automated UI test ready to replay. But as Espresso test recorder is still beta it leaves some manual work for you.

    First, make the ScrollingExperienceTest class compile:

    • Fix the R class access – replace “de.dbremes.dbtradealert.withAds.R.” with “R.” in my case
    • Let Andoid Studio add a static import for matchers like withId()

    More importantly the swipe wasn’t recorded because while Espresso supports swipes its test recorder doesn’t as of version 2.2.2. To have it at least build the infrastructure I long-tapped a report which got recorded properly. Then I just coded the swipe manually and added the necessary static imports:

    package de.dbremes.dbtradealert;
    
    import android.support.test.espresso.ViewInteraction;
    import android.support.test.rule.ActivityTestRule;
    import android.support.test.runner.AndroidJUnit4;
    import android.test.suitebuilder.annotation.LargeTest;
    
    import org.junit.Rule;
    import org.junit.Test;
    import org.junit.runner.RunWith;
    
    import static android.support.test.espresso.Espresso.onView;
    import static android.support.test.espresso.action.ViewActions.click;
    import static android.support.test.espresso.action.ViewActions.swipeLeft;
    import static android.support.test.espresso.action.ViewActions.swipeUp;
    import static android.support.test.espresso.contrib.RecyclerViewActions.actionOnItemAtPosition;
    import static android.support.test.espresso.matcher.ViewMatchers.isDisplayed;
    import static android.support.test.espresso.matcher.ViewMatchers.withId;
    import static android.support.test.espresso.matcher.ViewMatchers.withParent;
    import static android.support.test.espresso.matcher.ViewMatchers.withText;
    import static org.hamcrest.Matchers.allOf;
    
    @LargeTest
    @RunWith(AndroidJUnit4.class)
    public class ScrollingExperienceTest {
    
        @Rule
        public ActivityTestRule<WatchlistListActivity> mActivityTestRule = new ActivityTestRule<>(WatchlistListActivity.class);
    
        @Test
        public void scrollingExperienceTest() {
            ViewInteraction recyclerView = onView(
                    allOf(withId(R.id.list),
                            withParent(allOf(withId(R.id.container),
                                    withParent(withId(R.id.main_content)))),
                            isDisplayed()));
    
            recyclerView.perform(actionOnItemAtPosition(0, swipeUp()));
    
            // Freezing the app to see the swipe
            try {
                Thread.sleep(1000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }
    

    To run the test:

    1. In the test method’s context menu select Run scrollingExperienceTest
    2. In the Select Deploymet Target window select your device or emulator and click OK

    The test will probably fail with this error message:
    PerformException: Error performing 'fast swipe' on view. Animations or transitions are enabled on the target device.

    That’s because Espresso can only detect whether the app itself is busy but not when animations or transitions are running. That may lead to tests failing for the wrong reason and Espresso prevents that. Disable animations on the device for Espresso to work:

    1. Go to Settings | Developer options
    2. Disable 3 settings:
      1. Window animation scale
      2. Transition animation scale
      3. Animator duration scale

    Running the test again produced this error on the withAds flavor:
    java.lang.RuntimeException: Action will not be performed because the target view does not match one or more of the following constraints:
    at least 90 percent of the view's area is displayed to the user.
    Target view: "RecyclerView ...

    That’s because the advert covers part of the RecyclerView. So let’s ty the playStore flavor.

    Running the test on the playStore flavor produced this error – buried deep in the stack trace:
    android.support.test.espresso.InjectEventSecurityException: java.lang.SecurityException: Injecting to another application requires INJECT_EVENTS permission

    Hmm, head-scratching after this one. But a simple solution after finally checking the originally recorded long-tap – perform the action on an item and not on the RecyclerView itself. Just replace
    recyclerView.perform(swipeUp()) with
    recyclerView.perform(actionOnItemAtPosition(0, swipeUp()))

    Adding a Thread.sleep(1000) permits to see if the swipeUp() actually moves the watchlist.

    5.2 Generate Test Data

    The app starts with two watchlists each containing two securities. As the idea is to force the creation of new reports while scrolling the app needs more securities. Logging will be added to make shure the reports weren’t created in advance.

    To add the required securities to the watchlist the test will read them from a .csv file. The DbHelper class has already code to parse .csv data and the infrastructure will come in handy later when using WireMock to provide canned quotes.

    5.2.1 Add a CSV File to the Test

    The .csv file is only used for testing and therefore shouldn’t be included in the app itself. Instrumented tests like Espresso tests actually get compiled into a second apk that runs in the same process as the app’s APK. You’ll see that in Andoid Studio’s test window:
    $ adb shell pm install -r "/data/local/tmp/de.dbremes.dbtradealert.playStore"
    ...
    $ adb shell pm install -r "/data/local/tmp/de.dbremes.dbtradealert.playStore.test"

    The first step is to create an assets folder under “<ProjectPath>\app\src\androidTest” with Explorer as Android Studio doesn’t support that.

    After that use Android Studio to create the .csv file:

    1. In the “app” context menu select New | File
    2. In the Choose Destination Directory window select “…\app\src\androidTest\assets” and click OK
    3. In the New File window enter “ch_securities.csv” and click OK
    4. In the Register New File Type window accept the default of opening Text files in Android Studio and click OK

    The contents can be added in any text editor:

    "ABBN.VX"
    "CFR.VX"
    "ROG.VX"
    "SYNN.VX"
    "UBSG.VX"
    "ZURN.VX"
    

    5.2.2 Import CSV Data

    Importing CSV data with DbHelper is straight forward:

    // ...
    public class DbHelper extends SQLiteOpenHelper {
        // ...
    
        private String[][] convertCsvToStringArrays(String csvData) {
            String[][] result = null;
            String[] csvRows = csvData.split("\r?\n|\r");
            int rowCount = csvRows.length;
            int fieldCount = csvRows[0].split(",").length;
            result = new String[rowCount][fieldCount];
            String csvRow = null;
            for (int rowIndex = 0; rowIndex < csvRows.length; rowIndex++) {
                csvRow = csvRows[rowIndex];
                result[rowIndex] = csvRow.split(",");
                // Delete any surrounding quotes
                for (int fieldIndex = 0; fieldIndex < result[rowIndex].length; fieldIndex++) {
                    result[rowIndex][fieldIndex] = result[rowIndex][fieldIndex].replace("\"", "");
                }
            }
            return result;
        } // convertCsvToStringArrays()
    
        // ...
    
        public void importTestSecurities(String csvData, long watchlistId) {
            final String METHOD_NAME = "importTestSecurities";
            final String addSecurityToWatchlistSql = "INSERT INTO " + SecuritiesInWatchlists.TABLE
                    + "(" + SecuritiesInWatchlists.SECURITY_ID
                    + "," + SecuritiesInWatchlists.WATCHLIST_ID + ") VALUES(?,?)";
            final String insertSecuritySql = "INSERT INTO " + Security.TABLE
                    + "(" + Security.SYMBOL + ") VALUES(?)";
            String[][] csvArrays = convertCsvToStringArrays(csvData);
            SQLiteDatabase db = this.getWritableDatabase();
            try {
                db.beginTransaction();
                SQLiteStatement addSecurityToWatchlistStatement
                        = db.compileStatement(addSecurityToWatchlistSql);
                SQLiteStatement insertSecurityStatement = db.compileStatement(insertSecuritySql);
                for (int rowIndex = 0; rowIndex < csvArrays.length; rowIndex++) {
                    // 1-based index for bindString()!
                    insertSecurityStatement.bindString(1, csvArrays[rowIndex][0]);
                    long securityId = insertSecurityStatement.executeInsert();
                    addSecurityToWatchlistStatement.bindLong(1, securityId);
                    addSecurityToWatchlistStatement.bindLong(2, watchlistId);
                    addSecurityToWatchlistStatement.executeInsert();
                }
                db.setTransactionSuccessful();
                Log.d(CLASS_NAME, METHOD_NAME + "(): success!");
            } finally {
                db.endTransaction();
            }
        } // importTestSecurities()
    
        // ...
    }
    

    The new importTestSecurities() method has two duties: parse the csv data and insert the resulting values into the securites table which includes adding them to a watchlist.

    Parsing the csv format was already implemented in updateOrCreateQuotes() and got factored out into convertCsvToStringArrays() which returns a two dimensional string array.

    Inserting the values happens a bit more performance-conscious here than in updateOrCreateQuotes(). As before the main booster is to wrap the inserts in a transaction – that cuts 99% off non-transactional execution time easily because it saves SQLite from needing to wrap each in its own transaction.

    What’s new is the call to SQLiteDatabase.compileStatement(). This way preparation steps like analyzing the SQL are only executed once which can cut the remaining execution time in half. An SQLiteStatement absorbs parameter values with type specific methods like SQLiteStatement.bindString() – note the 1-based index and of course you have to clear a parameter value if you don’t provide a new one. The execution method to call – SQLiteStatement.executeInsert() in this case – is specific to the type of SQL statement. Unlike other databases SQLite provides no way to unprepare a statement to free its resources.

    To be honest I wouldn’t use prepared statements for the few records this test inserts but just wanted to try them with SQLite. If on the other hand you have to insert tens of thousands of records it may be worth doing that in chunks to prevent memory issues.

    SQlite actually supports bulk imports – loading data directly from a file – with “.import <FILE> <TABLE>” but only at the command line interface. And finally starting with version 3.7.11 it provides this insert syntax:
    INSERT INTO 'tablename' ('column1', 'column2') VALUES
    ('data1', 'data2'),
    ('data3', 'data4'),
    ('data5', 'data6'),
    ('data7', 'data8');

    Note that this is only syntactical sugar without performance benefits and certainly non-standard SQL.

    5.2.3 Start Data Import Before the Test

    To start the data import ScrollingExperienceTest gets a new method:

    // ...
    @LargeTest
    @RunWith(AndroidJUnit4.class)
    public class ScrollingExperienceTest {
        // ...
    
        @Before
        public void createTestData() {
            String testDataString = "";
            Context testContext = InstrumentationRegistry.getContext();
            try {
                InputStream testDataStream
                        = testContext.getResources().getAssets().open("ch_securities.csv");
                byte[] testData = new byte[testDataStream.available()];
                testDataStream.read(testData);
                testDataString = new String(testData);
            } catch (IOException e) {
                PlayStoreHelper.logError(e);
            }
            Context targetContext = InstrumentationRegistry.getTargetContext();
            DbHelper dbHelper = new DbHelper(targetContext);
            // SQLite Ids start with 1
            long watchlistId = 1;
            Log.d(CLASS_NAME, "createTestData(): calling importTestSecurities()");
            dbHelper.importTestSecurities(testDataString, watchlistId);
        } // createTestData()
    
        // ...
    }
    

    The @Before annotation tells JUnit to execute createTestData() before running any test. The method reads the .csv file from the test package’s resources – large files should be handled in chunks. Also note that createTestData() uses two different contexts:

    • InstrumentationRegistry.getContext() provides the test app context to access the file in the respective package
    • InstrumentationRegistry.getTargetContext() provides the app context to create a DbHelper instance

    Another thing to note is that SQLite’s automatic identifiers are 1-based.

    5.3 Delete Test Data After the Test

    Every test should leave the app in the same state it was before. That’s the only way to ensure a defined starting state when running multiple tests.

    The easiest way is to wrap the whole test beginning with test data creation in a transaction and roll it back after test completion. Of course this can lock surprisingly large parts of the database and should be used with caution.

    SQLite doesn’t support nested transactions out of the box but uses savepoint and release commands. The SQLite wrapper for Android maps that to nested transactions though.

    But that still doesn’t work for DbTradeAlert’s stateless DbHelper class. The problem is that nested transactions in SQLite have to use the same SQLiteDatabase instance. As DbHelper’s methods create a new SQLiteDatabase instance whenever they need one tests cannot use a transaction without changing DbHelper’s API. Trying it with SQLiteDatabase.beginTransaction() will result in a SQLiteDatabaseLockedException and if you use SQLiteDatabase.beginTransactionNonExclusive() you’ll get a IllegalStateException later – “no transaction pending”.

    So DbHelper gets a new method instead:

    // ...
    public class DbHelper extends SQLiteOpenHelper {
        // ...
    
        /**
         * Caution: deleteTestSecurities() deletes any securities with Id > 4!
         * Logic as deleteSecurity()
         */
        public void deleteTestSecurities() {
            final String methodName = "deleteTestSecurities";
            Log.v(CLASS_NAME, String.format("%s(): securityId > 4", methodName));
            String[] whereArgs = new String[]{String.valueOf(4)};
            int deleteResult = 0;
            SQLiteDatabase db = getWritableDatabase();
            try {
                db.beginTransaction();
                // Delete security's quotes
                deleteResult = db.delete(Quote.TABLE, Quote.SECURITY_ID + " > ?", whereArgs);
                Log.v(CLASS_NAME, String.format(DELETE_RESULT_FORMAT, methodName,
                        Quote.TABLE, deleteResult));
                // Delete security's existing connections to watchlists
                deleteResult = db.delete(SecuritiesInWatchlists.TABLE,
                        SecuritiesInWatchlists.SECURITY_ID + " > ?", whereArgs);
                Log.v(CLASS_NAME, String.format(DELETE_RESULT_FORMAT, methodName,
                        SecuritiesInWatchlists.TABLE, deleteResult));
                // Delete security
                deleteResult = db.delete(Security.TABLE, Security.ID + " > ?", whereArgs);
                Log.v(CLASS_NAME, String.format(DELETE_RESULT_FORMAT, methodName,
                        Security.TABLE, deleteResult));
                db.setTransactionSuccessful();
                Log.d(CLASS_NAME, methodName + "(): success!");
            } finally {
                db.endTransaction();
            }
        } // deleteTestSecurities()
     
       // ...
    }
    

    This new method called deleteTestSecurities() implements the same logic as deleteSecurity() but does so for all securities with Id > 4. This way you can repeat the test without causing a UNIQUE constraint violation when adding the same symbols again. As the tables were created with INTEGER PRIMARY KEY columns and not as AUTOINCREMENT the Ids get recycled.

    This method is called from ScrollingExperienceTest:

    // ...
    @LargeTest
    @RunWith(AndroidJUnit4.class)
    public class ScrollingExperienceTest {
        // ...
    
         @After
        public void deleteTestData() {
            Context targetContext = InstrumentationRegistry.getTargetContext();
            DbHelper dbHelper = new DbHelper(targetContext);
            dbHelper.deleteTestSecurities();
        } // deleteTestData()
    
        // ...
    }
    

    As the method is annotated with @After it gets called automatically after the tests have completed.

    5.4 Run the Test

    Actual test methods like scrollingExperienceTest() are marked with the @Test annotation. In this case the generated method needs an overhaul before it can do anything useful:

    // ...
    @LargeTest
    @RunWith(AndroidJUnit4.class)
    public class ScrollingExperienceTest {
        // ...
    
        @Test
        public void scrollingExperienceTest() {
            Log.d(CLASS_NAME, "scrollingExperienceTest(): start");
            // Make added securities show up in watchlist:
            Log.d(CLASS_NAME, "scrollingExperienceTest(): tapping Refresh");
            ViewInteraction actionMenuItemView = onView(
                    allOf(withId(R.id.action_refresh), withContentDescription("Refresh"), isDisplayed()));
            actionMenuItemView.perform(click());
    
            try {
                Thread.sleep(3000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
    
            ViewInteraction recyclerView = onView(
                    allOf(withId(R.id.list),
                            withParent(allOf(withId(R.id.container),
                                    withParent(withId(R.id.main_content)))),
                            isDisplayed()));
            for (int i = 0; i < 8; i++) {
                Log.d(CLASS_NAME, "scrollingExperienceTest(): swipeUp" + i);
                recyclerView.perform(actionOnItemAtPosition(i, swipeUp()));
            }
            Log.d(CLASS_NAME, "scrollingExperienceTest(): the End");
        } // scrollingExperienceTest()
    
        // ...
    }
    

    The first necessary change is to tap the Refresh button because the new securities will show up only after they got their quotes. Currently the quotes are actually loaded from the internet and a Thread.sleep(3000) makes shure they arrived before continuing the test – this will be corrected soon.

    After that the test issues some calls to swipeUp(). Note that actionOnItemAtPosition() will scroll the respective item into view so don’t just always pass 0 like I did initially.

    A temporary addition to ensure the swipes actually force the creation of additional list items is needed:

    // ...
    public class WatchlistRecyclerViewAdapter 
            extends RecyclerView.Adapter<WatchlistRecyclerViewAdapter.ViewHolder> {
        // ...
    
        @Override
        public void onBindViewHolder(final ViewHolder viewHolder, int reportPosition) {
            // ...
            // Symbol
            viewHolder.Symbol = this.cursor.getString(
                    this.cursor.getColumnIndex(SecurityContract.Security.SYMBOL));
            Log.d(CLASS_NAME, "onBindViewHolder(): symbol = " + viewHolder.Symbol);
            // SymbolTextView
            viewHolder.SymbolTextView.setText(this.cursor.getString(
                    this.cursor.getColumnIndex(SecurityContract.Security.SYMBOL)));
            // ...
        }
    
        // ...
    }
    

    That needs to be commented out after verifying the test works as expected – onBindViewHolder() is the last spot you want to slow down.

    Start the test now by clicking Run ‘scrollingExperienceTest()’ in the method’s context menu. You should be able to watch a tap on Refresh followed by the additional reports showing up and swipes to the end of them. Android Studio’s logcat window should show something like this:

    ...
    ... D/QuoteRefresherService: onHandleIntent(): quotes updated - initiating screen refresh
    ... D/WatchlistRec.ViewAd.: onBindViewHolder(): symbol = BAYN.DE
    ... D/WatchlistRec.ViewAd.: onBindViewHolder(): symbol = SIE.DE
    ... D/WatchlistRec.ViewAd.: onBindViewHolder(): symbol = ABBN.VX
    ... D/WatchlistRec.ViewAd.: onBindViewHolder(): symbol = NESN.VX
    ... D/WatchlistRec.ViewAd.: onBindViewHolder(): symbol = NOVN.VX
    ... D/WatchlistRec.ViewAd.: onBindViewHolder(): symbol = CFR.VX
    ... D/ScrollingExperienceTest: scrollingExperienceTest(): swipeUp0
    ... I/ViewInteraction: Performing 'actionOnItemAtPosition performing ViewAction: fast swipe on item at position: 0' action on view (with id: de.dbremes.dbtradealert.playStore:id/list and has parent matching: (with id: de.dbremes.dbtradealert.playStore:id/container and has parent matching: with id: de.dbremes.dbtradealert.playStore:id/main_content) and is displayed on the screen to the user)
    ... D/WatchlistRec.ViewAd.: onBindViewHolder(): symbol = ROG.VX
    ... D/ScrollingExperienceTest: scrollingExperienceTest(): swipeUp1
    ... I/ViewInteraction: Performing 'actionOnItemAtPosition performing ViewAction: fast swipe on item at position: 1' action on view (with id: de.dbremes.dbtradealert.playStore:id/list and has parent matching: (with id: de.dbremes.dbtradealert.playStore:id/container and has parent matching: with id: de.dbremes.dbtradealert.playStore:id/main_content) and is displayed on the screen to the user)
    ... D/WatchlistRec.ViewAd.: onBindViewHolder(): symbol = SYNN.VX
    ... D/ScrollingExperienceTest: scrollingExperienceTest(): swipeUp2
    ... I/ViewInteraction: Performing 'actionOnItemAtPosition performing ViewAction: fast swipe on item at position: 2' action on view (with id: de.dbremes.dbtradealert.playStore:id/list and has parent matching: (with id: de.dbremes.dbtradealert.playStore:id/container and has parent matching: with id: de.dbremes.dbtradealert.playStore:id/main_content) and is displayed on the screen to the user)
    ... D/WatchlistRec.ViewAd.: onBindViewHolder(): symbol = UBSG.VX
    ... D/ScrollingExperienceTest: scrollingExperienceTest(): swipeUp3
    ... I/ViewInteraction: Performing 'actionOnItemAtPosition performing ViewAction: fast swipe on item at position: 3' action on view (with id: de.dbremes.dbtradealert.playStore:id/list and has parent matching: (with id: de.dbremes.dbtradealert.playStore:id/container and has parent matching: with id: de.dbremes.dbtradealert.playStore:id/main_content) and is displayed on the screen to the user)
    ... D/WatchlistRec.ViewAd.: onBindViewHolder(): symbol = ZURN.VX
    ... D/ScrollingExperienceTest: scrollingExperienceTest(): swipeUp4
    ... I/ViewInteraction: Performing 'actionOnItemAtPosition performing ViewAction: fast swipe on item at position: 4' action on view (with id: de.dbremes.dbtradealert.playStore:id/list and has parent matching: (with id: de.dbremes.dbtradealert.playStore:id/container and has parent matching: with id: de.dbremes.dbtradealert.playStore:id/main_content) and is displayed on the screen to the user)
    ... D/ScrollingExperienceTest: scrollingExperienceTest(): swipeUp5
    ... I/ViewInteraction: Performing 'actionOnItemAtPosition performing ViewAction: fast swipe on item at position: 5' action on view (with id: de.dbremes.dbtradealert.playStore:id/list and has parent matching: (with id: de.dbremes.dbtradealert.playStore:id/container and has parent matching: with id: de.dbremes.dbtradealert.playStore:id/main_content) and is displayed on the screen to the user)
    ... D/ScrollingExperienceTest: scrollingExperienceTest(): swipeUp6
    ... I/ViewInteraction: Performing 'actionOnItemAtPosition performing ViewAction: fast swipe on item at position: 6' action on view (with id: de.dbremes.dbtradealert.playStore:id/list and has parent matching: (with id: de.dbremes.dbtradealert.playStore:id/container and has parent matching: with id: de.dbremes.dbtradealert.playStore:id/main_content) and is displayed on the screen to the user)
    ... D/ScrollingExperienceTest: scrollingExperienceTest(): swipeUp7
    ... I/ViewInteraction: Performing 'actionOnItemAtPosition performing ViewAction: fast swipe on item at position: 7' action on view (with id: de.dbremes.dbtradealert.playStore:id/list and has parent matching: (with id: de.dbremes.dbtradealert.playStore:id/container and has parent matching: with id: de.dbremes.dbtradealert.playStore:id/main_content) and is displayed on the screen to the user)
    ... D/ScrollingExperienceTest: scrollingExperienceTest(): the End
    ... D/DbHelper: deleteTestSecurities(): success!
    

    So as required the swipes cause RecyclerView items to be created.

    Next post: Automated UI Tests – Part 3: Testing with WireMock and Firebase Test Lab for Android

    Additional Resources

    Posted in Uncategorized | Tagged , | Leave a comment

    DbTradeAlert for Android: Automated UI Tests – Part 1: Thoughts about Testing

    First post in this series: Introduction to DbTradeAlert

    Previous post: More Finishing Touches


    Ever wondered why everybody thinks testing is a good idea but nobody does it? Then read on. If you are more of a hands-on person you’ll have to wait for the next post though.

    1. Why Test at All?

    If a blog post about testing asks “why test at all?” you’d expect that to be a rhetorical question. But the goal you pursue with testing determines what you test, how you test it, and what you don’t test.

    To answer the question “why test?” one can simply ask “why create that app?” and in most cases the answer is “to earn money”. That means tests are simply an investment that has to pay off. Think of them as investing in an insurance to avoid the cost of errors.

    Obviously the next question is: how much do errors cost your customer / employer? Do bugs in the software kill people? Or can users just postpone its use for a week if it stopped working altogether? Costs may also come from contractual or legal bindings and brand damage. And finally costs come from having to resolve a problem quickly at the most unsuitable time. That said simply developing the wrong product probably is the most epensive error.

    Of course the answer will be just guesswork but the important point is to get everybody in the team on the same page. That means to consider costs and benefits instead of just wanting to use the framework du jour or to meet their peer’s expectations.

    Figuring out the cost of errors gave you an idea about how much you can spend on mitigation. If tests catch all errors before they make it to the user that’s perfect. But also unrealistic especially considering the limited resources you can spend on tests which includes gaining and keeping the expertise to use the respective tools and frameworks.

    So in reality you’ll have to spend some of the mitigation resources on monitoring to deal with errors that made it to the user. And monitoring will not only alert you to those errors sometimes even before users notice them but it also helps to reproduce those errors.

    Another benefit is that monitoring can uncover unexpected usage scenarios and it may be the only way to mitigate costs of errors induced by external systems. If the customer has a monitoring system already in place that will reduce the cost of mitigation on your part. While monitoring doesn’t reduce the Mean Time Between Failure (MTBF) it reduces the Mean Time To Recovery (MTTR) which is often more valuable. That’s why DbTradeAlert uses Firebase Analytics and Firebase Crash Reporting.

    Another way to mitigate costs of errors is automated deployment as it also reduces MTTR. A lot of projects have automated deployment in place as part of continuous integration. That will again reduce the cost on your part because those resources were spent to tighten the feedback loop a.k.a. being agile.

    2. What and How to Test

    The previous section showed that the resources you can spend on tests are very limited so you need to invest them wisely. Let’s have a look at four types of tests:

    • Unit tests
    • Integration tests
    • End-to-end tests
    • Automated UI Tests

    The first three types of tests differ by what they test and make up the popular test triangle. The fourth type is the 3rd’s automated cousin – the first two types are almost always automated.

    Besides representing the test triangle’s types those test types also have counterparts in Android Studio:

    • Local unit tests located at module-name/src/test/java/: no access to Android dependencies
    • Instrumented tests located at module-name/src/androidTest/java/: access to Android environment and the app
    • Automated UI Tests with Espresso: same as instrumented tests but with a test recorder

    There is no rigid separation between unit and integration tests though as the following sections will show.

    2.1 Unit tests

    Unit tests are written by programmers and exercise small units of code like a method with the intent to isolate it from external behaviour. They became popular with Extreme Programming in the late nineties. Unit tests should be run whenever a change has been made to spot errors before they have costly consequences.

    By now each programming language and software platform has specialized frameworks to execute unit tests and to streamline their creation. For Android and Java in general that’s JUnit 4.

    Unit tests can also serve as documentation of how a piece of software is expected to work. And they are invaluable for the dreaded two screen long method of business logic. Without proper unit tests everybody would be afraid to touch that beast – for good reasons. They are also the only sure-fire way to ensure the method handles every case and does it correctly.

    Going a step further leads to test driven design (TDD) where you write unit tests before the actual code. The most important benefit of that is making you look at your code from a client’s point of view. That will lead to a much better API design.

    But while writing a unit test up-front is certainly more fun than debugging an error afterwards those unit tests don’t come for free. Especially the cost of maintaining unit tests can go through the roof if some change requires a substantial rewrite of them.

    That in turn can lead to people foregoing necessary refactorings to avoid reworking the respective unit tests. In some situations time constraints lead to deactivating unit tests instead of adapting them. And in that case the whole investment is often lost as it’s likely that later project stages bring even more time constraints and those unit tests never get reactivated.

    Short form: unit tests are most effective at verifying business logic.

    2.2 Integration Tests

    Programmers write integration tests to ensure a program’s components – especially its external components – work together as expected. So in a sense they are the opposite of a unit test. This is the kind of test that catches Ninja changes to APIs and botched configurations. Integration tests mostly use the platform’s unit test framework – JUnit in this case – and run in integration or nightly builds so execution time isn’t much of a concern.

    Assuming the component’s interfaces are stable integration tests only require the investment to initially write them. But they don’t benefit from natural boundaries as much as unit tests do. Therefore they tend to invite a mocking framework to shield them from disturbances like unstable network conditions or to circumvent laborious setups. Mocking frameworks in turn tend to invite a dependency injection framework so the mock can be switched in and out. The result not only necessitates structural code changes but an additional set of expertise which is not that prevalent. And as the framework of the day tends to change like fashion finding the required expertise will be even harder – read “more expensive” – in a few years.

    2.3 End-to-End Tests

    End-to-end tests are the only ones that actually prove a program works as expected. They are also the oldest and most common type and require no additional investment in infrastructure if done manually and ignoring hardware requirements to parallelize testing.

    Doing them manually also reveals their biggest drawback – they just eat time. This is somewhat alleviated by the fact that designing and executing manual end-to-end tests should be left to professional testers anyway and therefore doesn’t use development resources.

    There are multiple reasons why you want professional testers for this kind of test:

    • Designing useful end-to-end tests requires a broad view of the product – programmers in contrast have a deep understanding on specific parts
    • Designing and manually executing end-to-end tests provides an opportunity to evaluate the user experience (UX) – programmers are notorious for ignoring UX
    • Testing requires professional sceptics – programmers are born optimists (as each of their estimates shows)
    • Testing ones own software will happen in the same mindset that created the software – that’s exactly how unit tests miss bugs

    The other drawback of end-to-end tests is that they often leave you in the dark about the exact error condition if things go wrong. Digging that out can be very costly and therefore you’ll still have to invest in unit or integration tests as well as in monitoring.

    Android apps add an additional burden to manual tests: a lot of them will have to be repeated on various devices. Those devices may differ in screen or memory size, CPU speed, or Android version or they may be specific to a certain error scenario. Of course that will drive up costs for any manual test and not only end-to-end tests.

    Which is a nice transition to the next section …

    3. Automated UI Tests

    Historically automated UI tests had the odor of being too expensive and too brittle. That’s because in their most simple and ancient form they compared the result screen with a screenshot – it can’t get more brittle than that. In addition to that the recordings couldn’t be adapted easily which means they had to been thrown away as soon as the program changed significantly. Add to that the variation of network and timing issues and you know why automated UI tests had a bad name.

    But the industry has evolved and recognizes the value of automated UI tests which results in various automation frameworks. Google’s latest addition is a UI testing framework named Espresso to which Android Studio 2.2 supplies a test recorder which as of now (2016-09-26) is still in beta though.

    Espresso addresses most problems of the past:

    • Android provides hooks for Espresso tests – no architecting of the software for testing requirements is needed which saves a lot of time and money
    • Espresso deals with the asynchronous nature of Android apps’ UI – no flaky tests due to timing issues and shorter test runs because no Thread.sleep() is needed
    • Espresso tests use the unit test infrastructure – no need to learn yet another tool
    • By using a test recorder the testers can create the tests – that frees developer resources
    • The test recorder creates readable Java code – the test script can easily be adapted to changing software

    Espresso isn’t perfect though:

    • It can only access the app under test – UI elements like notifications and toasts are out of reach because Android owns them
    • It cannot test WebViews or ImageViews

    There are ways around those limitations like combining Espresso with Selendroid to test WebViews or using the OpenCV library to test ImageViews. But as usual detours come at a cost – you need to build up the expertise to apply those solutions.

    4. Automated Tests for DbTradeAlert

    The previous sections determined the goal of testing and the types of automated tests Android provides with their pros and cons. So what automated tests do I choose for DbTradeAlert after learning about Android’s options?

    Let’s go through the test types:

    • Unit tests: the app has barely any business logic so no gain from unit tests
    • Integration tests: there are various areas where I would have felt more comfortable with integration tests:
      • Test how the app deals with network errors
      • Test scheduling without having to wait for the actual alarm
      • Test the behavior in Doze and App Stand by mode
    • End-to-end tests: I did those manually like testing CRUD operations for securities. Not shure if automating them would have saved much time.

    That only leaves automated integration tests as an option to save time. As the app doesn’t have an API those tests can only be automated by UI interactions.

    On the other hand creating automated tests for the app’s scheduling and how it deals with Doze and App Stand by mode would be challenging and won’t happen now.

    Now there are two more areas for which I would like more tests when the app faces a broader audience:

    • The scrolling experience with larger watchlists on low end devices
    • The app’s reaction to unintended usage

    So here’s the plan:

    • Scrolling experience test:
      • Add more securities than fit the screen
      • Record some swipe tests with Espresso
      • Run those tests in Firebase Test Lab for Android on physical low end devices
    • Monkey test 1:
      • Submit the app to abuse by the test monkey that comes with the SDK
      • Provide canned quotes for both the initial securities and any symbol the monkey may enter
    • Monkey test 2:
      • Submit the app to abuse by a Firebase Test Lab for Android Robo Test
      • Provide canned quotes for both the initial securities and any symbol the monkey may enter
    • Connection test:
      • Provide a configurable connection that can time out, return HTTP error codes and garbage data

    Next post: Scrolling experience test – Testing with Espresso

    Additional Resources

     

    Posted in Uncategorized | Tagged , | Leave a comment