Jamie Matthews
Mar 15, 2016
5 mins reading time

Close enough - Setting up a sufficient device lab

Ultimately, you cannot replicate the exact interaction a customer makes with our website or app, but you can build a device lab of real mobile phones and tablets, with real apps and with real connection spikes. These connection spikes are when a device’s connection to a network encounters issues, the signal dips which results in a delay with the response or no response at all. With this in mind, the website or application can function in a strange way or, of course, not function at all.

When I first joined Holiday Extras as a Tester, we already had a variety of mobile phones and tablets, some of which were fairly dated. This did not matter due to the amount of traffic still hitting our website and app using these particular devices; if they use it, we will develop for it, within reason of course.

The main difficulty came to light before I had even become a Tester. I just didn’t understand the role then and I never thought about software issues from a development point of view. I had no background in Software Testing and now that I had become part of a Web Team, the hurdles became apparent. We do love the big companies like Apple and Samsung, but we also find it difficult to keep up with their continuous device rollouts. With these newer models, came new functions and with these new functions, came new methods of interaction. This is the obstacle when developing for the website and app; we are not a small company, but we do not claim to be big either. Thankfully, the business understands the real need for an up to date device lab, so when newer models are released, you can be sure that Holiday Extras will probably have this particular device on order.

Ultimately our device lab is built from our traffic statistics. This is the real driver when maintaining software and updating our lab. We ensure that we develop for the most used devices and browsers to ensure sufficient coverage without being overly broad.

Another issue we also face on a daily basis is making sure that we have enough devices available for everyone who wants to use them. The lab is shared between Web, SEO and PPC, which is not to say that SEO and PPC are not part of the Web team, but merely to highlight the different people across the business requiring the use of the device lab. Currently we have exactly 30 devices for use across the business. This has grown rapidly over the last year.

Now that we have obtained a variety of devices, the next challenge is how we incorporate them into our job. Which ones do we pick? Do we always select the same ones due to traffic being high? The answer is of course, no. Firstly, we do not have enough of the same devices to share around our testing team of 9 let alone the rest of the business. Most importantly though, if we stuck with the same devices this would mean that the devices we neglect would inevitably have bugs going live, causing issues for the small percentage of users. As an example, some devices have around two percent of traffic; compared to the bigger devices, this is minimal. However, if you look at this from a bigger picture, that is anywhere between 10-25,000 users, which is a big chunk of business. So we need to mix it up, we need to check the older devices, the ones which are 5 years old, the ones that we think would be in the bottom of someone’s drawer.

So how do we go about testing a piece of work with several different devices that function differently? Well, manual testing will always have a place in software development, but how can we make manual testing less tedious, less repetitive, more productive? What we really needed was something that allowed us to synchronize the devices, in which we could go to a particular URL and compare them side by side. Granted, we could have our developers create our own program within Holiday Extras that would function in this way, but this would eat away at resource and of course, slow down the deployment of business critical code, so we needed to look for another way of obtaining this.

I stumbled across Ghostlabs, a program that allows the control of numerous devices and browsers. A normal setup of this for myself would be Chrome, Safari, Firefox, IE11/10, iPhone 6, Samsung S5, iPad 3 and sometimes more. This allows me to synchronise testing, allowing me to drag the URL into the Ghostlab app, start the server, open the required browsers and synchronise the devices to start exploring. Ghostlab is not just for testing however, it can also be used for web development. It allows the developer to quickly inspect any connected client, make the required changes, compile and refresh all devices and browsers showing the developer the newly made changes.

This is currently where we are at within Holiday Extras. It’s made a huge difference having easy access to all of these devices, programs and also the ability to obtain newer models. Our next step is to simulate real life connections. Currently we test within the business WiFi but we all know that customers will often have spikes in connection. This needs to be replicated to understand the hurdles customers may face when having connection issues. We are in the process of updating our device lab with devices on a data plan, allowing us to simulate real connection issues. For now though, we can use inbuilt emulators within Chrome.

Tags Testing,, device,, lab