Test Automation:

When test automation started to evolve a decade ago there were challenges in terms of insufficient tools, complexity of applications and getting skilled resources. Many innovations and advancements over the years helped automation to overcome the initial hurdles and successfully compete with application development approaches. We have seen some significant breakthroughs, where the automation maturity progress has improved a lot more than expected. ‘Test automation in cloud’ is one such breakthrough and let’s see what it has in store.

There have been questions on ROI and how fast the automated tests can validate a build or a feature change. This is still an open discussion with contrasting viewpoints. Earlier when the testing was limited to desktops and then a few browsers, the QA team was able to accommodate those configuration requirements and a few change requests in addition during the course. Since these kind of configurations/support updates were of minimal impact on the QA plan and teams, they could accommodate them in the existing schedule itself without much deviation.

New Challenges in Test Automation:

As you know a mobile-first, responsive design and customer-centric approach were widely used for application development while requirements for testing and automation were not the same as before. The diversity of application takes the center stage during automation feasibility nowadays and success of automation in these kinds of engagements might not be achieved only by considering a legacy setup.

As the development framework supports scaling application/products from desktop to browsers to mobile apps, test automation is expected to cover all the above without much additional effort. When we consider browser support for testing, there are multiple browsers each running in various platforms like Windows, MAC and Linux. Similarly, when considering mobile browsers and native apps, the variations come in terms of OS type, version and different browsers itself. This then extends to tablets or different screen sizes.

But cross browser testing enables engineering teams to release their software confidently and thus it should get priority during the test strategy.

Also, a test automation framework should be capable of scripting once, port and support the required platform/browser/device configuration and cover all the combos during test execution. At the same time, the total execution time should not be a multiplication of combos covered.

Test Automation and Cloud:

If the complexity of handling and managing the variety in platforms & browsers can be obviated, then the QA team can concentrate on the testing process and drive projects with ‘quality’ alone in mind.

Cloud infrastructure helps the QA team to run their automated tests in almost all the platforms, browsers with a variety of versions and mobile devices available nowadays. The cloud provider takes the overhead of maintaining these platforms/browsers/devices for updates, security patches and upgrading OS.

Test automation, having requirements to support different browsers & versions across desktop and devices, can leverage the cloud environment. There are many popular cloud providers in the market and each is specialized in supporting commonly used automation frameworks/tools. Decisions on moving to the cloud and choosing the right provider play an important role in successful testing and this should be taken as early as possible during the test strategy phase itself.

Sig Nag, research director at Gartner says,  “While some organizations are still figuring out where cloud actually fits in their overall IT strategy, an effort to cost optimize and bring forth the path to transformation holds strong promise”

According to a recent Gartner report, Worldwide Public Cloud Services Market is projected to grow 18 percent in 2017 to a total $246.8 billion, up from $209.2 billion in 2016. According to Gartner, Inc. the highest growth will come from cloud system infrastructure services (Infrastructure as a Service [IaaS]), which is projected to grow 36.8 percent in 2017 to reach $34.6 billion.

Aspire systems has engineered test automation frameworks across different tools and technologies which support integration with popular cloud environments like SauceLabs, AWS – device farm and BrowserStack. Through this, Aspire has successfully executed many automation projects across supported cloud environments in various domains.

Benefits of Test Automation in the Cloud:

  1. Coverage in platforms, browsers and devices

The main advantage of cloud testing infrastructure is support for all the available platforms, browsers and devices. All these would have versions ranging from latest to mostly-used to older versions. They also keep supporting newer versions as soon as they are available in consumer markets. Few providers have exclusive lab setup for real mobile devices and tablets apart from emulators and simulators. For example, the top cloud automation providers like SauceLabs, CrossBrowserTesting and BrowserStack supports 1500+ combos across all major browsers and platforms.

There is the option to run tests through a network tunnel so that even internal/intranet systems can also be tested using cloud platforms.

  1.  Cost Effective
  • Pay only for the actual time that was used in services. Plans range from unlimited user to minute basis usage.
  • Eliminates the high costs involved in having the devices/environment in-house and the procurement efforts.
  • No maintenance overhead as the service provider will be responsible for build, execution and updates. Also, no special infrastructure team is required for maintaining the servers, browsers and devices which will be required in the case of having our own test lab.
  • Available as free version for open source projects with limited resources.
  1. Parallel execution:

Parallel execution of the same tests across browsers/devices can be achieved in cloud environments. You could determine how many parallel threads based on the required combos and how quickly the whole test execution should be completed. Cloud providers have flexible plans for the number of parallel threads to choose from.

BrowserStack and CrossBrowserTesting allow performing UI testing across different browsers at once and compare the results using screenshots.

  1.  Scalability

In many development cycles, we would need more number of systems for testing in a short timeframe and then scale down later to minimum configurations. Thus most of the time it is not required to keep the entire setup up and running. In these kinds of scenarios, cloud providers give options to increase or decrease the required configurations in & for a short period. Some providers also have options to choose their license month on month basis.

  1.  Supporting and debugging tools

The cloud systems were well equipped with tools for supporting automation and debugging of test execution. Most of the providers have options to take a screenshot, record screencast of execution, detailed reports with results and compare UI across multiple resolutions and browser versions. This setup will help automation to work seamlessly in the cloud and faster as well.

For example, CrossBrowserTesting gives access to IE Developer Tool, Firebug, Chrome DevTools etc., as they are pre-installed in the browsers.

  1.  Integration with other tools

Continuous integration is vital in today’s testing activities to speed up the releases with a faster go to market strategy.  Cloud providers have support for popular CI tools like TFS, BitBucket pipeline, Jenkins, Travis, Bamboo, Circle, TeamCity etc. Using plug-ins of cloud providers, we will be able to run automated tests immediately after any deployment and code changes and validations.

Popular tools like Slack, JIRA, and HipChat is supported in SauceLabs, CrossBrowserTesting etc.

Recommended Read: What are the benefits of testing on the cloud?

Challenges:

  1.   Data Security

When we talk about cloud, security is of big concern to the organization. Though major players like Amazon Device Farm, SauceLabs etc. claim that they maintain a high standard of security and policies to protect our sensitive data, we need to be thoughtful while evaluating these providers.

  1.  Performance

Running tests in simulator or emulator will be slow and cloud systems also have the same issue. The slower tests will accumulate your license minutes and could have a cost impact. Each cloud provider has a set of best practices to follow so that the test execution speed is optimized.

  1.   Limitations
  • Sessions – Maximum cap on each test run sessions are enforced by almost all the service providers. For example, in Amazon Device Farm there is a limit of 60 minutes on automated testing and can be increased to 150 mins on request.
  • Hardware restrictions – Physically rotating the device, network throttling, simulating different battery conditions etc.

Manual Testing and Cloud:

Cloud has significant support for manual testing too. The automation advantages like parallel execution support for various platforms, browsers, devices and zero maintenance applies to manual testing as well.  Almost all the cloud providers support manual test sessions which can be recorded and shared for any issue reproduction and validation. SauceLabs, BrowserStack and CrossBrowser Testing provide unlimited manual test minutes in their licenses.

Decision to Stay or Move:

The key step here is to decide on whether to stay with in-house lab or move to the cloud for test automation. As mentioned earlier, this decision drives the overall QA process, so it should be based on project requirement after detailed analysis and considering the timeline as well for the release/go live phase.

If there are requirements to cover a variety of configurations which would change as per the market/user trend, then cloud would be an ideal solution since there is no upfront investment and no overhead of upgrading when there is a change in the configurations. Having said that, key factors like the characteristics of the application and the type of testing should also be considered before we decide to move on.

Aspire has good experience working with our client product teams to evaluate their requirement and propose suitable solutions considering all the above aspects.

Cloud Providers

In the current market, there are many providers who are catering to the organization’s need of testing the apps in various platforms such as desktops, browsers, mobile devices etc. Today, the question is which providers we need to choose for our needs. Below considerations shall help us to narrow down the list and choose the service provider as per our requirement.

  • Pricing/Licensing Model
  • Resource Coverage – Platform, Real devices, Emulators/Simulators etc.
  • Supported test automation types
  • Integration with tools – CI/CD
  • Customer support

Key Players

Amazon Device Farm:

  • Supports more frameworks when compared – Appium, Calabash, Instrumentation, UI Automator, Explorer, UI Automation, XCTest
  • Flexible Pricing – Pay as You Go
  • 350+ devices
  • Captures performance samples every second

Sauce Labs

  • Only test cloud that lets you automate your native, hybrid and mobile web tests across emulators, simulators and real devices
  • Supports manual testing with comprehensive platform coverage
  • Test on private cloud
  • Integrations – Plug-ins for Jenkins, Bamboo, VSTS, Team City, JIRA, Slack etc.

Xamarin Test Cloud:

  • The only provider that supports Xamarin Automation Framework
  • Tops the list with 2577 real devices (Android – 1269, iOS – 1308)
  • SIM support
  • Mobile monitoring

Click here to know more about Test Automation in the Cloud – A Key to Accelerated Development.

Latest posts by Jayanth krishnan (see all)