FYI. This is a translation from Japanese article.
Hello. I`m Tadashi, an Automation Engineer from Mercari’s Automation & Quality Assurance Group (AQA).
I mostly work on test automation and improving mobile CI /CD.
I’ll talk about a test automation environment which can run Appium/Android tests with 20~30 emulators.
- Current Android test automation environment and its problems
- How to create in the cloud
- Benefits of using Docker-Android
- Problems to be solved
Current Android test automation environment and its problems
We use Appium for Android UITest automation.
When we first created tests, they were run every time before release as a regression test.
At that time, test automation environment was the following:
Jenkins (master) -> mac (node) -> Android real device
It might be enough for a test automation environment at first.
Currently, we increased both test cases, test automation engineers, and frequency of running tests.
We faced some problems due to this situation.
- Increasing execution time
- Maintenance cost of real devices
- Test failures because of device condition such as network
- Maintenance cost of local environments such as Appium, Ruby
We use Swift / XCUITest / CircleCI(iOS Emulator) to run test stable, so I was thinking of running these Appium/Android tests in the cloud.
It has become popular to run Android UITest in Firebase Test Lab.
However, currently Firebase Test Lab does not support Appium, so we tried a tool called Docker-Android.
One of the tools that I created this test automation environment is called Docker-Android, a Docker image.
This tool is introduced in AppiumConf2018 in London.
This Docker image has features as follows.
- You can use x86 Android emulator, which is better in performance than ARM Android emulator.
- You can start Appium server inside the container using environmental variable
- You can connect this container to Selenium Grid easily.
- You can see behaviors of Android emulator inside the container by using noVNC.
- Can capture videos.
You can create Appium + Selenium Grid test automation environment by writing
docker-compose.yml below, and run the command
$ docker-compose up -d
# docker-compose.yml version: "2.2" services: selenium_hub: image: selenium/hub ports: - 4444:4444 nexus_5_8.0: image: butomo1989/docker-android-x86-8.0 privileged: true depends_on: - selenium_hub environment: - DEVICE=Nexus 5 - CONNECT_TO_GRID=true - APPIUM=true - SELENIUM_HOST=selenium_hub
You can also scale emulators by adding options
--scale nexus_5_8.0=XX if the same Node.
How to create in the cloud
We wanted to create this test automation environment in the cloud, considering maintenance cost and stability.
I expected to easily create it because it`s provided by Docker image.
However, there are restrictions for using this Docker-Android in the cloud.
Docker-Android needs hardware virtualization because it uses x86 Android emulators.
We cannot use this feature in general cloud instances because they are run under hardware virtualization.
Therefore, we needed special configurations to run Docker-Android.
Enable Nested Virtualization
You can use hardware virtualization feature inside the virtual machine.
You can use this feature in Google Cloud and Microsoft Azure.
Using Bare Metal Instances
Currently, AWS provides Bare Metal Instances called i3.metal
There are documents or sample codes to use Docker-Android in each cloud services.
We used AWS Bare Metal Server referencing this article.
i3.metal is much better in performance comparing to other instances so we could create a test automation environment with multiple Android emulators.
Benefits of using Docker-Android
Finally, we created a test automation environment which can run Appium/Android tests with multiple emulators.
We expect to decrease test execution time with this.
Currently, we run these tests before release and every night, but we expect to run more often this environment.
We also expect decreased maintenance costs by using emulators, Docker containers, and the cloud.
Problems to be solved
We got the above benefits, but we still have problems to be solved.
Divide roles between emulators and real devices
As I said, this test automation environment uses Android emulators.
However, it’s difficult to cover all tests with emulators because some tests like camera tests need to be tested with real devices.
So we need to divide roles between emulators and real devices.
Server load to QA environment
Currently, this environment is separated from the QA environments.
Therefore, the QA environment will receive high server load when the number of parallel increases.
Some actions will be needed such as using mock server or scaling the QA environment automatically.
Distribution of tests and test report.
Currently, we divide tests manually with Jenkins pipelines for each report.
We'll need to consider how to divide tests because mobile UITests takes time.
It's better to use some tools such as Knapsack to optimize distributed tests.
Also, we need to update test report functionality for distributed tests.
In SeleniumConf Chicago, there was an example to use Allure for distributed selenium tests.
I think it can also be used for our tests.
As I said above, we still have problems to be solved.
In addition, we need to continue working with developers and QA on what to cover in UITest.
However, I believe faster is better in test automation, and I believe decreasing execution time will make this test more valuable.
We’ll continue to strive for a better test automation environment.