Espresso / UIAutomator SDK

The Evinced Espresso / UIAutomator SDK integrates with new or existing UI tests to automatically detect accessibility issues. With the addition of a few lines of code you can analyze your entire application to understand how it can become more accessible. At the conclusion of the tests actionable HTML and JSON reports are generated to track issues in any reporting tool.

Supported platforms / frameworks

The Espresso / UIAutomator SDK supports Android UI tests on emulators or real devices with a minimum Android API level of 23 (Android 6.0).

Get Started

Setup

  1. In order to use any of the Evinced Mobile SDKs you first need to create an Evinced account. Once your account is created, you will get your apiKey and a matching serviceAccountId. Initialize the SDK with these tokens it will validate access with Evinced servers when you run your tests. If an outbound internet connection is not available in your running environment - contact us at support@evinced.com to get an offline APIKey.

  2. Add the SDK as a dependency using one of the following options:

    1. Add to Gradle
      Add the Evinced package repository URL to your root build.gradle at the end of repositories:
      1allprojects {
      2 repositories {
      3 //...
      4 maven { url 'https://evinced.jfrog.io/artifactory/public-gradle' }
      5 }
      6 }
      Add the following to your module build.gradle file:
      1androidTestImplementation "com.evinced:espresso-sdk:1.0.0"
    2. Add to Maven:
      Add the Evinced package repository URL to your build file:
      1<repositories>
      2 <repository>
      3 <id>evinced-repository</id>
      4 <url>https://evinced.jfrog.io/artifactory/public-gradle</url>
      5 </repository>
      6</repositories>
      Add the dependency:
      1<dependency>
      2 <groupId>com.evinced</groupId>
      3 <artifactId>espresso-sdk</artifactId>
      4 <version>1.0.0</version>
      5 <type>aar</type>
      6</dependency>
    3. Download .aar library from Evinced package repository: Espresso/UIAutomator SDK and add it manually for your project.
  3. Configure the SDK with Evinced authentication credentials
    Customizing the JUnit Runner (Recommended)

    1. Online setup by using AccountId and ApiKey
      1public class CustomJUnitRunner extends AndroidJUnitRunner {
      2 @Override
      3 public void callApplicationOnCreate(Application app) {
      4 super.callApplicationOnCreate(app);
      5 EvincedEngine.setupCredentials(accountId, apiKey);
      6 }
      7 }
      8 ```
    2. Offline setup by using AccountId and AccessTokenjava public class YourCustomJUnitRunner extends AndroidJUnitRunner { @Override public void callApplicationOnCreate(Application app) { super.callApplicationOnCreate(app); EvincedEngine.setupOfflineCredentials(accountId, accessToken) } } Replace the existing AndroidJunitRunner in build.gradle:
    1defaultConfig {
    2 // testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
    3 testInstrumentationRunner 'com.xx.yy.YourCustomJUnitRunner'
    4}

After creating the custom AndroidJUnitRunner you may need to sync Gradle/Maven with the project.

Authenticating within a test class

Add the setupCredentials call to your @BeforeClass method:

1@BeforeClass
2public static void setup() {
3 // Init instance of EvincedEngine in @BeforeClass method
4 evincedEngine = EvincedEngine.getInstance(InstrumentationRegistry.getInstrumentation());
5 evincedEngine.setupCredentials(accountId, apiKey);
6 // Other setup code...
7}

Your first test

Initialize the SDK instance

Init the EvincedEngine instance in each test class inside the @BeforeClass (@BeforeAll equivalent for JUnit5 runner) method:

1// import statement
2import com.evinced.test.EvincedEngine;
3
4@RunWith(AndroidJUnit4.class)
5public class UITests1 {
6 // Declare EvincedEngine instance
7 private static EvincedEngine evincedEngine;
8 // public static method annotated @BeforeClass executed only once before running all tests in the current class
9 @BeforeClass
10 public static void setup() {
11 // Init instance of EvincedEngine in @BeforeClass method
12 evincedEngine = EvincedEngine.getInstance(InstrumentationRegistry.getInstrumentation());
13 // Other setup code...
14 }
15}

Add accessibility scans to your test

Call the EvincedEngine.analyze() method at every point at which an accessibility scan would capture new areas of the application.

1@RunWith(AndroidJUnit4.class)
2public class UITests {
3 //...
4 @Test
5 public void test1() {
6 // test code...
7 // Call EvincedEngine.analyze() every time the information on the screens changes
8 // or when you want to make accessibility check
9 evincedEngine.analyze();
10 // test code...
11 evincedEngine.analyze();
12 }
13 //...
14 @Test
15 public void testN() {
16 // test code...
17 evincedEngine.analyze();
18 // test code...
19 }
20 }

Run validations and generate reports

The @AfterClass method executes only after all tests in the current class have completed. This makes it the perfect place to run our validations on data collected and stored by the EvincedEngine.analyze() calls and generate HTML and JSON reports in the emulator/device file structure.

1@RunWith(AndroidJUnit4.class)
2public class UITests {
3 //...
4 // @AfterClass method executed only once after running all thes tests in the current class are complete
5 @AfterClass
6 public static void tearDown() {
7 // Run accessibility validations for all collected data stored by the EvincedEngine.analyze()
8 // calls and generate HTML and JSON reports in the emulator/device file structure
9 evincedEngine.reportStored();
10 // Other tearDown code...
11 }
12}

Report

EvincedEngine.reportStored() or EvincedEngine.report() will generate HTML and JSON reports stored in a specific folder in the file structure of the emulator or device on which the tests were executed. The file path to the reports can be found by searching the logs using the key evinced_report_path.

Report path example:
D/evinced_report_path:/storage/self/primary/Evinced_A11Y_Test_Results_08.10.2021_11.57

Run tests locally

If the tests are being run locally reports can be downloaded by using AndroidStudio -> Device File Explorer or the following adb command:

adb pull <reports path> <local path>

Run tests on CI

If the tests are being executed using a CI tool the pipeline script below can be used to pass the Evinced reports to the CI tool reporting structure:

1// add this step after the tests run command
2- run:
3 name: get evinced reports
4 when: always
5 command: |
6 adb logcat -d > logfile.txt
7 REPORT_PATH="$(cat logfile.txt | grep evinced_report_path | awk -F "evinced_report_path:" '{ print $2 }' | tail -n 1 | grep -o '\/.*[0-9]')"
8 echo ${REPORT_PATH}
9 adb pull ${REPORT_PATH} evinced_report
10 zip -r evinced_report.zip evinced_report

Note: The script uses a list of logs that are cached within the emulator storage. If the number of logs is large, it is possible to lose the evinced_report_path key. In this case, please increase the buffer size with the following command adb logcat -G <size> M (default size = 2 Mb)

1//... other steps
2- android/start-emulator-and-run-tests:
3 pre-run-tests-steps:
4 - run: adb logcat -G 16M // increase log buffer size to 16 Mb
5 test-command: ./gradlew :app:connectedFreeDebugAndroidTest //tests run command

For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.

Permissions

The following permissions are required for the Espresso SDK:

android.permission.WRITE_EXTERNAL_STORAGE (for storing HTML / JSON reports in the device file structures) and android.permission.INTERNET (if you use online setup by using EvincedEngine.setupCredentials (accountId, apiKey)). Instrumentation tests use the tested app permissions, so they need to be added to the main manifest file.

If you do not have the required permissions, you can use the following instructions to add permissions to debug build and avoid adding them to the production version.

​​Set the permissions in src/debug/AndroidManifest.xml. If the file doesn't exist, create it.

By default, AndroidTest uses debug as a BuildType, so if you define your testing permissions there, then the manifest merging process will add these permissions to your UI test build.

Below is an example of the AndroidManifest.xml file with the added permissions. Note that You don't need to include the package attribute because it will be inherited from lower priority level manifest.

1<?xml version="1.0" encoding="utf-8"?>
2<manifest xmlns:android="http://schemas.android.com/apk/res/android">
3 <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
4 <uses-permission android:name="android.permission.INTERNET"/>
5
6 <!--required requestLegacyExternalStorage if you run your tests on Android 29-->
7 <application
8 android:requestLegacyExternalStorage="true"/>
9</manifest>

API

EvincedEngine.getInstance(InstrumentationRegistry.getInstrumentation())

Init the EvincedEngine instance - should be called in annotated methods @BeforeClass (@BeforeAll for JUnit5).

Example usage:

1@BeforeClass
2public static void setupEvinced() {
3 evincedEngine = EvincedEngine.getInstance(InstrumentationRegistry.getInstrumentation());
4}

void EvincedEngine.analyze()

Should be called each time you want to scan a specific screen state for accessibility violations. This method collects the current view hierarchy tree to be later analyzed when the reportStored() method is called. No return value.

Example usage:

1@Test
2public void testSwitchTheme() {
3 clickPreference(R.string.user_interface_label);
4 evincedEngine.analyze();
5}

List<Report> EvincedEngine.reportStored()

List<Report> EvincedEngine.reportStored(boolean assertIssues)

Run the accessibility validations using the data collected by the EvincedEngine.analyze() method calls. A list of Report objects will be returned and used to automatically generate HTML and JSON reports in the emulator/device file structure. If assertIssues = true an assertion error will be thrown if any accessibility issues are found. For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.

Example usage:

1@AfterClass
2public static void collectA11yData() {
3 evincedEngine.reportStored();
4}

void EvincedEngine.clearStored()

Clear all stored data collected by EvincedEngine.analyze(). Called automatically after EvincedEngine.reportStored() method.

Example usage:

1@Test
2public void testSwitchTheme() {
3 clickPreference(R.string.user_interface_label);
4 evincedEngine.analyze();
5 // This will throw an assertion error if any accessibility issues are found
6 evincedEngine.report(true)
7 evincedEngine.clearStored();
8 // More test code...
9}

Tutorials

Generating a comprehensive accessibility report for your Android application

In this tutorial, we will enhance our existing Espresso UI test with the Evinced Espresso SDK in order to check our application for accessibility issues. In order to get started you will need the following:

  1. All of the prerequisites for Evinced Espresso SDK should be met
  2. Evinced Espresso SDK should be installed in your project

Preface - Existing UI test overview

Let’s consider the following basic UI test as our starting point.

1@RunWith(AndroidJUnit4.class)
2public class PreferencesTest {
3
4 @Test
5 public void testSwitchTheme() {
6 final int theme = UserPreferences.getTheme();
7 int otherTheme;
8 if (theme == core.R.style.Theme_Light) {
9 otherTheme = R.string.pref_theme_title_dark;
10 } else {
11 otherTheme = R.string.pref_theme_title_light;
12 }
13 clickPreference(R.string.user_interface_label);
14 clickPreference(R.string.pref_set_theme_title);
15 onView(withText(otherTheme)).perform(click());
16 Awaitility.await().atMost(1000, MILLISECONDS)
17 .until(() -> UserPreferences.getTheme() != theme);
18 }
19}

The purpose of this test is to check that the application theme can be changed successfully. For now, this test is only concerned with the functional testing of the app. However, with the help of the Espresso / UIAutomator SDK, we can also check it for accessibility issues along the way. Let’s go through this process with the following step-by-step instructions.

Step #1 - Initialize the Evinced Espresso SDK

Before making any assertions against our app, we need to initialize EvincedEngine object. This object is used primarily as an entry point to all of the accessibility scanning features. Since we are going to use it primarily within our test, the best place for its initialization will be our @BeforeClass method which gets executed first.

1@RunWith(AndroidJUnit4.class)
2public class PreferencesTest {
3
4 @BeforeClass
5 public static void setupEvinced() {
6 evincedEngine = EvincedEngine.getInstance(InstrumentationRegistry.getInstrumentation());
7 }
8
9 @Test
10 public void testSwitchTheme() {
11 final int theme = UserPreferences.getTheme();
12 int otherTheme;
13 if (theme == core.R.style.Theme_Light) {
14 otherTheme = R.string.pref_theme_title_dark;
15 } else {
16 otherTheme = R.string.pref_theme_title_light;
17 }
18 clickPreference(R.string.user_interface_label);
19 clickPreference(R.string.pref_set_theme_title);
20 onView(withText(otherTheme)).perform(click());
21 Awaitility.await().atMost(1000, MILLISECONDS)
22 .until(() -> UserPreferences.getTheme() != theme);
23 }
24}

The only setup needed for the EvincedEngine is to pass it the instrumentation.

Step #2 - Identify which application states you want to check for accessibility issues

Once the SDK is set up, let’s try to carefully consider which states of our application are actually worth checking for accessibility issues. Let’s take a look at our UI test once again:

1@Test
2public void testSwitchTheme() {
3 final int theme = UserPreferences.getTheme();
4 int otherTheme;
5 // This step gets the current application theme and determines what the theme will be changed to
6 if (theme == core.R.style.Theme_Light) {
7 otherTheme = R.string.pref_theme_title_dark;
8 } else {
9 otherTheme = R.string.pref_theme_title_light;
10 }
11 // Before we change the theme lets check the current theme for accessibility issues!
12 clickPreference(R.string.user_interface_label);
13 clickPreference(R.string.pref_set_theme_title);
14 onView(withText(otherTheme)).perform(click());
15 Awaitility.await().atMost(1000, MILLISECONDS)
16 .until(() -> UserPreferences.getTheme() != theme);
17 //Now that we have confirmed the theme has changed let's check the new theme for accessibility issues!
18}

We have identified two places in our UI test where we would like to perform accessibility scans. Each place represents some distinct application state and each of these states may contain certain accessibility issues. So, let’s put our checks into place.

Step #3 - Setup accessibility analyzing points

The only thing we need to do now is to simply add evincedEngine.analyze(); calls to the places in the test we identified above. Here is what it should look like:

1@Test
2public void testSwitchTheme() {
3 final int theme = UserPreferences.getTheme();
4 int otherTheme;
5 // This step gets the current application theme and determines what the theme will be changed to
6 if (theme == core.R.style.Theme_Light) {
7 otherTheme = R.string.pref_theme_title_dark;
8 } else {
9 otherTheme = R.string.pref_theme_title_light;
10 }
11 // Before we change the theme lets check the current theme for accessibility issues!
12 evincedEngine.analyze();
13
14 clickPreference(R.string.user_interface_label);
15 clickPreference(R.string.pref_set_theme_title);
16 onView(withText(otherTheme)).perform(click());
17 Awaitility.await().atMost(1000, MILLISECONDS)
18 .until(() -> UserPreferences.getTheme() != theme);
19
20 //Now that we have confirmed the theme has changed let's check the new theme for accessibility issues!
21 evincedEngine.analyze();
22}

The analyze method collects an accessibility snapshot of the current application state and puts it into internal in-memory storage. This snapshot contains a lot of information including an application screenshot, accessibility labels, and dimensions of UI elements.

Now that we have gathered the information we need we are finally ready to actually find our whether our app is accessible or not.

Step #4 - Assert your screens at the end of your test suite

As our test was executed we collected a lot of accessibility snapshots via the evincedEngine.analyze() calls. We can now perform accessibility assertions at the end of our test suite. Referring back again to our initial UI test the best place for this assertion will be the method that gets invoked last - @AfterClass.

1@AfterClass
2public static void collectA11yData() {
3 evincedEngine.reportStored();
4}

To generate the actual object representation of your accessibility report simply call the reportStored method. The Evinced Espresso SDK will check all of the previously collected accessibility snapshots for accessibility issues. From this moment you are free to assert against the given list of Report objects in any way, you would like (feel free to explore the corresponding chapter in the “API“ section).

As an example, let’s simply assume that our application is accessible as long as it has no accessibility issues found. Thus, if we have at least one accessibility issue detected - we want our tests to fail. Let’s add the corresponding assertion:

1@AfterClass
2public static void collectA11yData() {
3 evincedEngine.reportStored(true);
4}

You are now set to run the test and ensure the accessibility of your application!

Step #5 - Read the results

For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.

FAQ

1. What is the minimum Android API version supported?

The minimum supported Android API level is 23 (Android 6.0).

2. Is there any difference in the number of accessibility validations Evinced will find when testing on an emulator vs. a real physical device?

The Evinced Espresso SDK works equally well on real devices and emulators. The number of validations doesn't depend on the device type (emulator or real device).

3. Can the Engine be used when testing on remote or cloud-based devices?

It may depend slightly on the functionality of the cloud-based device provider, but there are no inherent reasons why the Espresso / UIAutomator SDK wouldn’t work well on any device capable of running Espresso / UIAutomator tests.

4. Do I need to add anything (SDK etc.) to my app during the build process to test for accessibility violations?

The Espresso / UIAutomator SDK is integrated only in UI tests, which run as a separate application. No change or addition is needed to the build process of the application under test.

5. Can the reports be exported directly to the project directory as opposed to remaining on the device?

Unfortunately not. Because the tests are performed as a separate application, it only has access only to the emulator or device file structure. However, you can automate this process of extracting the file by using the example scripts above in your CI tool or as a Gradle task.

6. I got the following error while integrating the Espresso SDK.

1Execution failed for task ':app:processDebugAndroidTestManifest'.
2> Manifest merger failed : Attribute application@extractNativeLibs value=(false) from manifestMerger6360702180557287033.xml:36:9-42
3 is also present at [com.evinced:espresso-sdk:<version>]

Why it happened and how to fix it? The error occurred while merging the AndroidManifest of your tests and the Espresso SDK AndroidManifest. To fix it in your androidTest AndroidManifest.xml add:

1<application
2 tools:replace="android:extractNativeLibs"
3 android:extractNativeLibs="true"
4 ...>