As we pointed out in Part 1 of this series, the combination of intense competition in the mobile landscape; a vast migration of consumers to mobile tech (not to mention their growing expectation for flawless mobile experiences); and an explosion in the number of devices, operating systems, screen resolutions, etc., the importance of mobile testing cannot be overstated.
But recognizing the need for mobile testing and understanding how best to create and execute against a mobile testing strategy are two different things. In this post we’ll explore the key considerations and steps necessary for a successful mobile testing strategy.
As with any new product or service, it’s of paramount importance to understand and identify the end-user. Is your mobile application for everyday consumers primarily using smartphones for retail experiences? Or is it targeted toward power users at work using tablets? As the diversity of mobile audiences, devices, and application types grow, so too will the importance of identifying those audiences and ensuring your applications meet their unique requirements.
A better understanding of your user also – though not always – helps to narrow down the device types for which your test plan is to be targeted. Meaning your test plan can be refined if you are confident your mobile users are largely confined to one device type. Realistically, however, you are likely going to need to test for many device types.
With your end users and device type(s) more or less identified, you need to consider the types of testing or Quality Assurance (QA) necessary to meet your needs: manual, automated, and cloud testing.
Use automation for your more frequently tested scenarios, where predictable results are likely, where automation is easiest to execute, or for the especially time-consuming and tedious requirements.
Manual testing is ideal for unique use cases where automation is difficult to implement (or where you first need to establish a use case profile before replicating it for automated testing. Example: a manual test might be where the battery is running low, you are receiving incoming calls and heavy SMS traffic, geolocation services are being engaged, you are mobile and experiencing mobile roaming, etc.
Enables you to create a stable, replicable, geographically dispersed, low-cost test environment where mobile applications on a single device can be tested by any number of users, at any time, and anywhere and under any number of conditions.
Once your test types have been selected, it’s time to actually test the application for speed, responsiveness, stability, effectiveness, battery usage, resolution, etc., for all features. Here is where it’s important to replicate – as much as possible – the real-world experiences likely to impact your end users.
A key consideration, for example, is recognizing that the work environment in which you are testing your application (i.e. an office with high-speed network capacities using first class servers) is almost certainly NOT what your mobile user will experience. Which is why the selection of performance testing and real world virtualization tools (which we’ll cover in our next post) are critically important to the testing process. Remember: your mobile application may be amazing; but if it takes 10 seconds to open in a less-than-ideal operating environment, your application – and your brand – may never recover in the eyes of that consumer.
Assuming your performance testing delivers acceptable results, it’s time to test across device types and operating systems, including earlier OS versions.
One additional consideration – particularly for complex mobile applications – is shift-left testing where the testing regimen is started earlier in the development process. This is important for the simple reason that the longer it takes to identify any issues with complex builds, the more time-consuming and expensive the likely remedy.
In our next post we’ll focus on how best to execute against your mobile strategy – specifically, identifying the tools and partners best suited to meet your needs.