Mercari Engineering Blog

We're the software engineers behind Mercari. Check out our blog to see the tech that powers our marketplace.

Super fast and parallelized Android UITest(Appium) environment using AWS, Docker and Android emulator

f:id:tadashi-nemoto0713:20181129162340j:plain


Note: This is a translation from the Japanese article.

tech.mercari.com

Hello. I'm Tadashi, an Automation Engineer from Mercari’s Automation & Quality Assurance Group (AQA).

I mostly work on test automation and improving mobile CI/CD.

I’ll talk about a test automation environment which can run Appium/Android tests with 20~30 emulators.


f:id:tadashi-nemoto0713:20181127193934j:plain
Jenkins pipeline to execute parallel test automation using Android emulator

Current Android test automation environment and its problems

We use Appium for Android UITest automation.

When we first created tests, they were run every time before release as a regression test.

At that time, the test automation environment was the following:

Jenkins (master) -> mac (node) -> Android real device

It might be enough for a test automation environment at first.

Currently, we are increasing both test cases, test automation engineers, and the frequency of running tests.

We faced some problems due to this situation.

  • Increasing execution time
  • Maintenance cost of real devices
  • Test failures because of device condition such as network
  • Maintenance cost of local environments such as Appium, Ruby


We use Swift / XCUITest / CircleCI(iOS Emulator) to run tests stable, so I was thinking of running these Appium/Android tests in the cloud.

f:id:tadashi-nemoto0713:20181225140928p:plain:w700


It has become common to run Android UITest in the Firebase Test Lab.

However, Firebase Test Lab currently does not support Appium, so we tried a tool called Docker-Android.

Docker-Android

One of the tools that I used to create this test automation environment is called Docker-Android, a Docker image.

github.com

This tool was introduced in AppiumConf2018 in London.

www.youtube.com

This Docker image has the following features.

  • You can use the x86 Android emulator, which is better in performance than the ARM Android emulator.
  • You can start the Appium server inside the container using environmental variable APPIUM=true.
  • You can connect this container to Selenium Grid easily.
  • You can see the behavior of the Android emulator inside the container by using noVNC.
  • Can capture videos.

You can create the Appium + Selenium Grid test automation environment by writing docker-compose.yml below, and running the command$ docker-compose up -d

# docker-compose.yml
version: "2.2"

services:
  selenium_hub:
    image: selenium/hub
    ports:
    - 4444:4444

  nexus_5_8.0:
    image: butomo1989/docker-android-x86-8.0
    privileged: true
    depends_on:
    - selenium_hub
    environment:
    - DEVICE=Nexus 5
    - CONNECT_TO_GRID=true
    - APPIUM=true
    - SELENIUM_HOST=selenium_hub

You can also scale emulators by adding options --scale nexus_5_8.0=XX if the same Node.

How to create in the cloud

We wanted to create this test automation environment in the cloud, considering maintenance cost and stability.

I expected to be able to easily create it because it's provided by the Docker image.

However, there are restrictions for using this Docker-Android in the cloud.

Docker-Android needs hardware virtualization because it uses x86 Android emulators.

We cannot use this feature in general cloud instances because they are run under hardware virtualization.

Therefore, we needed special configurations to run Docker-Android.

Enable Nested Virtualization

You can use hardware virtualization features inside the virtual machine.

You can use this feature in Google Cloud and Microsoft Azure.

Enabling Nested Virtualization for VM Instances  |  Compute Engine Documentation  |  Google Cloud

Nested Virtualization in Azure | Blog | Microsoft Azure

Using Bare Metal Instances

Currently, AWS provides Bare Metal Instances called i3.metal

Amazon EC2 Bare Metal Instances with Direct Access to Hardware | AWS News Blog


There are documents or sample codes to use Docker-Android in each cloud services.

docker-android/README_CLOUD.md at master · budtmo/docker-android · GitHub


We used the AWS Bare Metal Server referencing this article.

sourcediving.com

i3.metal is much better in performance compared to other instances so we were able to create a test automation environment with multiple Android emulators.


f:id:tadashi-nemoto0713:20181127191224j:plain
Selenium Grid

Benefits of using Docker-Android

Finally, we created a test automation environment which can run Appium/Android tests with multiple emulators.

We expect to decrease test execution time with this.

Currently, we run these tests before release and every night, but we expect to run more often this environment.

We also expect decreased maintenance costs by using emulators, Docker containers, and the cloud.

Problems to be solved

We got the above benefits, but we still have problems to be solved.

Divide roles between emulators and real devices

As I said, this test automation environment uses Android emulators.

However, it’s difficult to cover all tests with emulators because some tests like camera tests need to be tested with real devices.

So we need to divide roles between emulators and real devices.


f:id:tadashi-nemoto0713:20181129110342j:plain
Server load to QA environment

Currently, this environment is separated from the QA environments.

Therefore, the QA environment will receive high server load when the number of parallel execution increases.

Some actions will be needed such as using mock servers or scaling the QA environment automatically.

Distribution of tests and test reports

Currently, we divide tests manually with Jenkins pipelines for each report.

tech.mercari.com

We'll need to consider how to divide tests because mobile UITests takes time.

It's better to use some tools such as Knapsack to optimize distributed tests.

Knapsack gem README

Also, we need to update test report functionality for distributed tests.

In SeleniumConf Chicago, there was an example using Allure for distributed selenium tests.

I think it can also be used for our tests.

Allure | Test report and framework for writing self-documented tests


f:id:tadashi-nemoto0713:20181129162445j:plain:w700
Test reports using Allure, introduced in SeleniumConf Chicago

Endnote

As I said above, we still have problems to be solved.

In addition, we need to continue working with developers and QA on what to cover in UITest.

However, I believe faster is better in test automation, and I believe decreasing execution time will make this test more valuable.

We’ll continue to strive for a better test automation environment.