Mercari Engineering Blog

We're the software engineers behind Mercari. Check out our blog to see the tech that powers our marketplace.

How HeadSpin will change mobile testing and monitoring?

This is a translation of the original Japanese article.

tech.mercari.com

Hello. I'm Tadashi, an Automation Engineer from Mercari’s Automation & Quality Assurance Group (AQA).

I mostly work on test automation and improving mobile CI /CD.

We tried HeadSpin's service.

I would like to introduce the service and share some impressions.

What is HeadSpin?

HeadSpin is a service that provides device farm, test automation environment, and monitoring as 24/7 environment for monitoring the performance of mobile applications.

Using various networks, locations and devices, we can do performance test in a way that is close to what customers are using.

www.headspin.io

techcrunch.com

Motivation

We introduced UI test automation for both iOS and Android applications.

These tests are not connected to our QA environment; not to our production environment.

Therefore, current tests for production environment are now done manually.

There is a motivation to automate them and I thought that using HeadSpin would make it easy to test them in a way that is close to what customers are using.

In particular, use of:

  • real devices
  • networks such as 4G
  • real locations such as Tokyo

In addition, with HeadSpin, we can also do performance monitoring in a production environment.

So we thought that we could also provide beneficial feedback to developers.

Features of HeadSpin

Device farm

In HeadSpin, real devices of various location and network are prepared as a device farm, and they can be controlled on Web UI.

Android device farm is based on open source tool called Open STF, and we can control it with the same operability.

HeadSpin is also compatible with iOS.

Screen operations, app installation, screen rotation, etc. are all possible from the Web UI.

f:id:tadashi-nemoto0713:20190212132627j:plain
Device control screen of iOS device on HeadSpin (demo environment)

Test automation environment

HeadSpin makes it easy to run automation tests on HeadSpin's devices.

Test frameworks such as Appium / Espresso / XCTest / EarlGray are supported (see here for details)

In this trial, we used Appium for both iOS / Android.

With Appium, you can run tests by specifying the WebDriver URL and Capabilities displayed on HeadSpin UI.

f:id:tadashi-nemoto0713:20190212135930j:plain
Appium configuration screen on HeadSpin (demo environment)

# Sample code using Ruby
def desired_caps
  {
    caps: {
      platformName: 'iOS',
      deviceName: 'iPhone',
      automationName: 'XCUITest',
      udid: ENV['HEADSPIN_UDID'],
      bundleId: ENV['BUNDLE_ID'],
    },
    appium_lib: {
      server_url: ENV['HEADSPIN_URL']
    }
  }
end

Capybara.register_driver(:appium) do |app|
  Appium::Capybara::Driver.new app, desired_caps
end

Since Appium server is provided by the HeadSpin side, there is no need to install Appium on a local environment or a CI environment.

In this case, only Ruby execution environment is required, so we were able to easily create an testing environment using CircleCI.

version: 2.1

orbs:
  ruby-orbs: sue445/ruby-orbs@volatile

jobs:
  test:
    docker:
      - image: circleci/ruby:2.5.3

    steps:
      - checkout
      - ruby-orbs/bundle-install
      - run: bundle exec rspec

workflows:
  version: 2.1
  hourly_test:
    triggers:
      - schedule:
          cron: "0 * * * *"
          filters:
            branches:
              only:
                - master
    jobs:
      - test

Also, with Appium's GUI tool called Appium Desktop, you can easily connect with HeadSpin's devices.

f:id:tadashi-nemoto0713:20190212153424p:plain
Connection screen to headspin using Appium Desktop

Performance monitoring

The performance of devices during automation tests can be recorded as a Session.

In each Session UI, you can see the performance of devices such as CPU rate, frame rate, etc.

You can also see the response time from devices to the API, CDN, etc.

f:id:tadashi-nemoto0713:20190212181110j:plain
Session UI on HeadSpin

In addition, this data can be stored in a DB provided by HeadSpin.

You can continuously monitor it by linking this DB with BI tools such as Grafana or Looker.

f:id:tadashi-nemoto0713:20190212184625j:plain
Dashboard using Grafana

Impression

Here is my impression based on this trial.

There is no need for an additional SDK

When using HeadSpin, additional SDKs are not needed in the app.

We did not have to worry about unexpected troubles and performance deterioration by putting them, which made us easier to introduce HeadSpin.

Uniqueness as iOS device farm, test automation environment, and performance monitoring

As mentioned earlier, OpenSTF supports Android but iOS is not supported.

HeadSpin is not only an iOS device farm, but it also provides continuous test automation environment / performance monitoring tools.

We can rely on it as a test automation environment to some extent

As mentioned earlier, because we can leave Appium and other test automation environments to HeadSpin, I felt that there are quite a few parts that we have to maintain.

In addition, adb · idevice commands can also be called via HeadSpin's API to operate the devices, so it adds a lot of flexibility.

What monitoring and feedback we can give to engineers

Initially I was supposed to give feedback on performance for mobile engineers (iOS / Android).

This creates some complication because HeadSpin measures the performance that includes outside of the app such as network and etc.

It does not provide feedback at the code level of the mobile application.

If you want to provide feedback only for mobile applications, it will be necessary to combine it with another tool such as Nimble (also from HeadSpin).

Conversely, HeadSpin can measure the total performance including the response time to API and CDN in the same environment that customers actually use.

This may provide valuable information and feedback not only to mobile engineers but also to SRE and backend engineers.

Endnote

Mercari JP is growing in scale, so we believe that great user experience by various customers are becoming more important.

Within that, it is important to know what affects customer experience in the real world environment.

Although it will not have an immediate effect, I think we need to take action to improve customer experience based on the information / feedback obtained from HeadSpin's service.