First post in this series: Introduction to DbTradeAlert
Previous post: More Finishing Touches
Ever wondered why everybody thinks testing is a good idea but nobody does it? Then read on. If you are more of a hands-on person you’ll have to wait for the next post though.
1. Why Test at All?
If a blog post about testing asks “why test at all?” you’d expect that to be a rhetorical question. But the goal you pursue with testing determines what you test, how you test it, and what you don’t test.
To answer the question “why test?” one can simply ask “why create that app?” and in most cases the answer is “to earn money”. That means tests are simply an investment that has to pay off. Think of them as investing in an insurance to avoid the cost of errors.
Obviously the next question is: how much do errors cost your customer / employer? Do bugs in the software kill people? Or can users just postpone its use for a week if it stopped working altogether? Costs may also come from contractual or legal bindings and brand damage. And finally costs come from having to resolve a problem quickly at the most unsuitable time. That said simply developing the wrong product probably is the most epensive error.
Of course the answer will be just guesswork but the important point is to get everybody in the team on the same page. That means to consider costs and benefits instead of just wanting to use the framework du jour or to meet their peer’s expectations.
Figuring out the cost of errors gave you an idea about how much you can spend on mitigation. If tests catch all errors before they make it to the user that’s perfect. But also unrealistic especially considering the limited resources you can spend on tests which includes gaining and keeping the expertise to use the respective tools and frameworks.
So in reality you’ll have to spend some of the mitigation resources on monitoring to deal with errors that made it to the user. And monitoring will not only alert you to those errors sometimes even before users notice them but it also helps to reproduce those errors.
Another benefit is that monitoring can uncover unexpected usage scenarios and it may be the only way to mitigate costs of errors induced by external systems. If the customer has a monitoring system already in place that will reduce the cost of mitigation on your part. While monitoring doesn’t reduce the Mean Time Between Failure (MTBF) it reduces the Mean Time To Recovery (MTTR) which is often more valuable. That’s why DbTradeAlert uses Firebase Analytics and Firebase Crash Reporting.
Another way to mitigate costs of errors is automated deployment as it also reduces MTTR. A lot of projects have automated deployment in place as part of continuous integration. That will again reduce the cost on your part because those resources were spent to tighten the feedback loop a.k.a. being agile.
2. What and How to Test
The previous section showed that the resources you can spend on tests are very limited so you need to invest them wisely. Let’s have a look at four types of tests:
- Unit tests
- Integration tests
- End-to-end tests
- Automated UI Tests
The first three types of tests differ by what they test and make up the popular test triangle. The fourth type is the 3rd’s automated cousin – the first two types are almost always automated.
Besides representing the test triangle’s types those test types also have counterparts in Android Studio:
- Local unit tests located at module-name/src/test/java/: no access to Android dependencies
- Instrumented tests located at module-name/src/androidTest/java/: access to Android environment and the app
- Automated UI Tests with Espresso: same as instrumented tests but with a test recorder
There is no rigid separation between unit and integration tests though as the following sections will show.
2.1 Unit tests
Unit tests are written by programmers and exercise small units of code like a method with the intent to isolate it from external behaviour. They became popular with Extreme Programming in the late nineties. Unit tests should be run whenever a change has been made to spot errors before they have costly consequences.
By now each programming language and software platform has specialized frameworks to execute unit tests and to streamline their creation. For Android and Java in general that’s JUnit 4.
Unit tests can also serve as documentation of how a piece of software is expected to work. And they are invaluable for the dreaded two screen long method of business logic. Without proper unit tests everybody would be afraid to touch that beast – for good reasons. They are also the only sure-fire way to ensure the method handles every case and does it correctly.
Going a step further leads to test driven design (TDD) where you write unit tests before the actual code. The most important benefit of that is making you look at your code from a client’s point of view. That will lead to a much better API design.
But while writing a unit test up-front is certainly more fun than debugging an error afterwards those unit tests don’t come for free. Especially the cost of maintaining unit tests can go through the roof if some change requires a substantial rewrite of them.
That in turn can lead to people foregoing necessary refactorings to avoid reworking the respective unit tests. In some situations time constraints lead to deactivating unit tests instead of adapting them. And in that case the whole investment is often lost as it’s likely that later project stages bring even more time constraints and those unit tests never get reactivated.
Short form: unit tests are most effective at verifying business logic.
2.2 Integration Tests
Programmers write integration tests to ensure a program’s components – especially its external components – work together as expected. So in a sense they are the opposite of a unit test. This is the kind of test that catches Ninja changes to APIs and botched configurations. Integration tests mostly use the platform’s unit test framework – JUnit in this case – and run in integration or nightly builds so execution time isn’t much of a concern.
Assuming the component’s interfaces are stable integration tests only require the investment to initially write them. But they don’t benefit from natural boundaries as much as unit tests do. Therefore they tend to invite a mocking framework to shield them from disturbances like unstable network conditions or to circumvent laborious setups. Mocking frameworks in turn tend to invite a dependency injection framework so the mock can be switched in and out. The result not only necessitates structural code changes but an additional set of expertise which is not that prevalent. And as the framework of the day tends to change like fashion finding the required expertise will be even harder – read “more expensive” – in a few years.
2.3 End-to-End Tests
End-to-end tests are the only ones that actually prove a program works as expected. They are also the oldest and most common type and require no additional investment in infrastructure if done manually and ignoring hardware requirements to parallelize testing.
Doing them manually also reveals their biggest drawback – they just eat time. This is somewhat alleviated by the fact that designing and executing manual end-to-end tests should be left to professional testers anyway and therefore doesn’t use development resources.
There are multiple reasons why you want professional testers for this kind of test:
- Designing useful end-to-end tests requires a broad view of the product – programmers in contrast have a deep understanding on specific parts
- Designing and manually executing end-to-end tests provides an opportunity to evaluate the user experience (UX) – programmers are notorious for ignoring UX
- Testing requires professional sceptics – programmers are born optimists (as each of their estimates shows)
- Testing ones own software will happen in the same mindset that created the software – that’s exactly how unit tests miss bugs
The other drawback of end-to-end tests is that they often leave you in the dark about the exact error condition if things go wrong. Digging that out can be very costly and therefore you’ll still have to invest in unit or integration tests as well as in monitoring.
Android apps add an additional burden to manual tests: a lot of them will have to be repeated on various devices. Those devices may differ in screen or memory size, CPU speed, or Android version or they may be specific to a certain error scenario. Of course that will drive up costs for any manual test and not only end-to-end tests.
Which is a nice transition to the next section …
3. Automated UI Tests
Historically automated UI tests had the odor of being too expensive and too brittle. That’s because in their most simple and ancient form they compared the result screen with a screenshot – it can’t get more brittle than that. In addition to that the recordings couldn’t be adapted easily which means they had to been thrown away as soon as the program changed significantly. Add to that the variation of network and timing issues and you know why automated UI tests had a bad name.
But the industry has evolved and recognizes the value of automated UI tests which results in various automation frameworks. Google’s latest addition is a UI testing framework named Espresso to which Android Studio 2.2 supplies a test recorder which as of now (2016-09-26) is still in beta though.
Espresso addresses most problems of the past:
- Android provides hooks for Espresso tests – no architecting of the software for testing requirements is needed which saves a lot of time and money
- Espresso deals with the asynchronous nature of Android apps’ UI – no flaky tests due to timing issues and shorter test runs because no Thread.sleep() is needed
- Espresso tests use the unit test infrastructure – no need to learn yet another tool
- By using a test recorder the testers can create the tests – that frees developer resources
- The test recorder creates readable Java code – the test script can easily be adapted to changing software
Espresso isn’t perfect though:
- It can only access the app under test – UI elements like notifications and toasts are out of reach because Android owns them
- It cannot test WebViews or ImageViews
There are ways around those limitations like combining Espresso with Selendroid to test WebViews or using the OpenCV library to test ImageViews. But as usual detours come at a cost – you need to build up the expertise to apply those solutions.
4. Automated Tests for DbTradeAlert
The previous sections determined the goal of testing and the types of automated tests Android provides with their pros and cons. So what automated tests do I choose for DbTradeAlert after learning about Android’s options?
Let’s go through the test types:
- Unit tests: the app has barely any business logic so no gain from unit tests
- Integration tests: there are various areas where I would have felt more comfortable with integration tests:
- Test how the app deals with network errors
- Test scheduling without having to wait for the actual alarm
- Test the behavior in Doze and App Stand by mode
- End-to-end tests: I did those manually like testing CRUD operations for securities. Not shure if automating them would have saved much time.
That only leaves automated integration tests as an option to save time. As the app doesn’t have an API those tests can only be automated by UI interactions.
On the other hand creating automated tests for the app’s scheduling and how it deals with Doze and App Stand by mode would be challenging and won’t happen now.
Now there are two more areas for which I would like more tests when the app faces a broader audience:
- The scrolling experience with larger watchlists on low end devices
- The app’s reaction to unintended usage
So here’s the plan:
- Scrolling experience test:
- Add more securities than fit the screen
- Record some swipe tests with Espresso
- Run those tests in Firebase Test Lab for Android on physical low end devices
- Monkey test 1:
- Submit the app to abuse by the test monkey that comes with the SDK
- Provide canned quotes for both the initial securities and any symbol the monkey may enter
- Monkey test 2:
- Submit the app to abuse by a Firebase Test Lab for Android Robo Test
- Provide canned quotes for both the initial securities and any symbol the monkey may enter
- Connection test:
- Provide a configurable connection that can time out, return HTTP error codes and garbage data
- Android Testing Support – Android Testing Patterns #1: https://www.youtube.com/watch?v=W8LJjfkTKik
- Getting Started with Testing: https://developer.android.com/training/testing/start/index.html
- History of unit testing frameworks: http://www.martinfowler.com/bliki/Xunit.html
- What are unit tests and where did they come from: https://www.agilealliance.org/glossary/unit-test/
- Google Test Automation Conference (GTAC): https://developers.google.com/google-test-automation-conference/
- Just Say No to More End-to-End Tests – and an interesting discussion about that proposal: https://testing.googleblog.com/2015/04/just-say-no-to-more-end-to-end-tests.html
- Martin Fowler’s thoughts about the test pyramid: http://martinfowler.com/bliki/TestPyramid.html
- Why the Testing Pyramid is Misleading: https://www.joecolantonio.com/2015/12/09/why-the-testing-pyramid-is-misleading-think-scales/
- Open Lecture by James Bach on Software Testing: https://www.youtube.com/watch?v=ILkT_HV9DVU
- James Bach on testing in an agile software development team – very valuable parting words on test automation, make shure to watch to the end: https://www.youtube.com/watch?v=vqwyMaHcjQE