Appium Java SDK

The Evinced Appium Java SDK integrates with new or existing Appium tests to automatically detect accessibility issues. With the addition of as few as 5 lines of code to your Appium framework, you can begin to analyze your entire application to understand how it can become more accessible. At the conclusion of the test, a rich and comprehensive HTML or JSON report is generated to track issues in any reporting tool.

Supported versions / frameworks

Evinced Appium SDK supports the following values for Appium’s automationName desired capability:

  • XCUITest
  • UIAutomator2
  • Espresso

Older versions of automation drivers (e.g deprecated Appium iOS driver based on UIAutomation) are not supported.

Prerequisites:

  • Java 8 or higher
  • Appium Server ver. 1.21 or higher
  • Appium Java client v7 or higher (older version may still be compatible)

Get Started

Setup

In order to use any of the Evinced Mobile SDKs you first need to create an Evinced account. Once your account is created, you will get your apiKey and a matching serviceAccountId. Pass these these tokens using the setupCredentials() method and Evinced will validate access upon test execution. If an outbound internet connection is not available in your running environment - contact us at support@evinced.com to get an offline APIKey.

Install Evinced Appium SDK from the remote registry

If you are using Gradle as your build tool follow the steps below:

  1. Add Artifactory plugin to the plugins section of your build.gradle:

    1 plugins {
    2 // ... other plugins
    3 id 'com.jfrog.artifactory' version '4.24.14'
    4 }
  2. To the bottom of your build.gradle file add the next configuration for Artifactory plugin:

    1artifactory {
    2 contextUrl = 'https://evinced.jfrog.io/artifactory/'
    3
    4 resolve {
    5 repository {
    6 repoKey = 'public-gradle'
    7 maven = true
    8 }
    9 }
    10}

If you are using Maven, add the Evinced package repository URL to your build file:

1<repositories>
2 <repository>
3 <id>evinced-repository</id>
4 <url>https://evinced.jfrog.io/artifactory/public-gradle</url>
5 </repository>
Adding Evinced Appium SDK as a project dependency

The SDK can be added to your project as a Maven or Gradle dependency.

When working with Maven add the following dependency to your pom.xml file:

1<dependency>
2 <groupId>com.evinced</groupId>
3 <artifactId>appium-sdk</artifactId>
4 <version>{sdk-version}</version>
5 <scope>test</scope>
6</dependency>

When working with Gradle add the following to your build.gradle dependencies section:

1dependencies {
2 // ... other project dependencies
3 testImplementation(group: 'com.evinced', name: 'appium-sdk', version: '<sdk-version>', classifier: 'all')
4}

Install Evinced Appium SDK from local ZIP archive

You can also perform installation right from the standalone .jar distribution. In this case, you need to do the following:

  1. Unpack the provided evinced-appium-sdk.zip to any desirable location
  2. Add the following dependencies entries pointing to appium-sdk-all.jar
    1. Gradle:
      1implementation files('/Users/<path-to-your-upacked-folder>/appium-sdk-all.jar')
    2. Maven:
      1. First, install the “all“ jar into your local Maven repository by the following command:
        1mvn install:install-file –Dfile=/Users/<path-to-unpacked-folder>/appium-sdk-all.jar -DgroupId=com.evinced -DartifactId=evinced-appium-sdk -Dversion=1.0
      2. Add the corresponding dependency into your pom.xml:
        1 <dependency>
        2 <groupId>com.evinced</groupId>
        3 <artifactId>evinced-appium-sdk</artifactId>
        4 <version>1.0</version>
        5 </dependency>
    3. Trigger your package manager download new dependencies either via IDE or command line

ZIP archive also contains two complimentary .jar files which you can utilize:

  • appium-sdk-javadoc.jar contains rich JavaDoc documentation which you can view in any web browser
  • appium-sdk-sources.jar can be attached to your Evinced Appium SDK installation in order to let you leverage JavaDoc right in your IDE. Please, refer to the corresponding documentation of your developer tools

Your first test

Initialize the SDK object

Add the import statement to the beginning of your test file.

1import com.evinced.appium.sdk.core.EvincedAppiumSdk;

Under the hood, the Evinced SDK utilizes several Appium commands in order to get accessibility data from the device under test. For this reason, we need to initialize EvincedAppiumSdk with the Appium driver instance in the following way:

1public static EvincedAppiumSdk evincedSdk;
2
3@BeforeClass
4public static void setupAppiumDriver() throws MalformedURLException, IOException {
5 capabilities = new DesiredCapabilities();
6 // ...omitting the setting of capabilities for brevity
7
8 driver = new IOSDriver<IOSElement>(url, capabilities);
9
10 evincedSdk = new EvincedAppiumSdk(driver);
11}

Here we used the JUnit BeforeClass annotation to create a method that will be executed at the beginning of the test suite. This is often where the Appium Driver initialization takes place, therefore we can also use it to initialize the EvincedAppiumSdk instance as well.

Provide your license information

The Evinced Appium SDK requires authentication credentials which consist of two string values mentioned above: apiKey and serviceAccountId.

For this purpose use the method called evincedAppiumSdk.setupCredentials(serviceAccountId, apiKey) as shown:

1public static EvincedAppiumSdk evincedSdk;
2
3@BeforeClass
4public static void setupAppiumDriver() throws MalformedURLException, IOException {
5 //... the rest of our setup code
6
7 evincedSdk = new EvincedAppiumSdk(driver);
8
9 evincedSdk.setupCredentials(<your service account ID>, <your API key>);
10}

You can confirm the success of the authentication in the logs.

Add Evinced accessibility checks

1@Test
2public void testMyAppScreen() {
3 IOSElement someTableCell = (IOSElement) new WebDriverWait(driver, 30)
4 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("TableViewCell")));
5
6 firstStationTableCell.click();
7 //... do other interactions with the app
8
9 // Run analysis and get the accessibility report
10 Report report = evincedSdk.report();
11
12 // Assert that there are no accessibility issues
13 assertTrue(report.getIssues().size() == 0);
14}

Regardless of the test result, the report method will always output JSON and HTML files. You should be able to find the files by browsing the local folder called evinced-reports . For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.

API

EvincedAppiumSdk

public EvincedAppiumSdk(AppiumDriver<?> originalDriver)

Default constructor which accepts an AppiumDriver instance for performing accessibility snapshot collection. Can be easily instantiated in the following way:

1IOSDriver driver = new IOSDriver<IOSElement>(url, capabilities);
2EvincedAppiumSdk evincedSdk = new EvincedAppiumSdk(driver);

public EvincedAppiumSdk(AppiumDriver<?> originalDriver, EvincedAppiumSdkOptions options)

Additional constructor which allows you to configure the different aspects of a11y testing via EvincedAppiumSdkOptions object. For instance, one can set a custom path for storing reporting artifacts by using the example below:

1driver = new IOSDriver<IOSElement>(url, capabilities);
2
3EvincedAppiumSdkOptions evincedSdkOptions = new EvincedAppiumSdkOptions(); // create options object
4evincedSdkOptions.setOutputDir("my-output-dir"); // set custom output directory
5
6evincedSdk = new EvincedAppiumSdk(driver, evincedSdkOptions);

public boolean setupCredentials(String serviceAccountId, String apiKey)

A method to add credentials confirming the validity of the user’s license. It accepts the following arguments:

  • serviceAccountId - Evinced service account ID
  • apiKey - Evinced API key

Returns a boolean flag indicating the success of the operation - if it is false there is something wrong with either validity of the license or with the checking infrastructure. See logs for more detailed output.

public Report report(shouldAssert: boolean)

Generates accessibility report for the current state of the application. Returns a Report instance to allow users to perform any advanced assertions on their own. If it is not needed, a simple assertion may be delegated directly to the SDK by supplying shouldAssert argument set to true. In this case, the SDK will automatically throw an AssertionError if there is at least one issue found. For more information regarding the Report object , please see our detailed Mobile Reports page.

This method always writes HTML and JSON files to the configured directory.

Available overloads:

public Report report();

Example usage:

1Report report = evincedSdk.report(true); // assert is performed by EvincedAppiumSdk
2
3Report reportForMyOwnAssertion = evincedSdk.report();
4assertEquals(reportForMyOwnAssertion.hasIssues(), false); // performing assertions on your own

public void analyze()

Collects an accessibility snapshot of the current application state and puts it into internal in-memory storage. This method is supposed to be used in conjunction with EvincedAppiumSdk.assertStored() for performing actual assertions against the content of this storage.

Example usage:

1 IOSElement firstStationTableCell = (IOSElement) new WebDriverWait(driver, 30)
2 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("StationTableViewCell")));
3 // we have achieved the state of the app we would like to test. Let's take a snapshot.
4 evincedAppiumSdk.analyze();

public List<Report> reportStored(shouldAssert: boolean)

Generates a list of accessibility reports, one for each accessibility snapshot collected by EvincedAppiumSdk.analyze(). Just like EvincedAppiumSdk.report() this method may either perform simple assertion on its own (throwing an exception if there is at least one report with issues) or allow users to define their own assertions on the returned Report objects. This behavior can be configured via the shouldAssert flag.

This method always writes HTML and JSON files to the configured directory. Both of the files contain combined information about all of the application states collected. For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.

Available overloads:

public List<Report> reportStored();

Example usage:

1@AfterClass
2public static void tearDown() {
3 // Option 1
4 List<Report> reportsForOwnAssertion = evincedSdk.reportStored(); // making our own assertions
5 // Option 2
6 List<Report> reports = evincedSdk.reportStored(true); // delegating assertion to the SDK
7 // Option 3
8 List<Report> reportsWithIssues = reportsForOwnAssertion.stream() // Filter out only reports which contain some issues
9 .filter(report -> report.hasIssues())
10 .collect(Collectors.toList());
11 assertEquals(reportsWithIssues.size(), 0); // Making the actual assertion
12
13 driver.quit();
14}

public void clearStored()

Removes all of the accessibility snapshots previously collected by EvincedAppiumSdk.analyze() calls.

EvincedAppiumSdkOptions

public EvincedAppiumSdkOptions()

Default public constructor with no arguments.

public void setOutputDir(String outputDir)

Method for defining a custom output directory where a user would like to store reporting artifacts (JSON and HTML files).

Tutorials

Generating a comprehensive a11y report for your application

In this tutorial, we will enhance our existing Appium UI test with the Evinced Appium SDK in order to check our application for accessibility issues. In order to get started you will need the following:

  1. All of the prerequisites for the Evinced Appium SDK should be met
  2. Evinced Appium SDK should be installed in your project

Preface - existing UI test overview

Let’s consider the following basic UI test as our starting point.

1import io.appium.java_client.MobileBy;
2import io.appium.java_client.ios.IOSDriver;
3import io.appium.java_client.ios.IOSElement;
4import io.appium.java_client.remote.MobileCapabilityType;
5
6import java.io.File;
7import java.io.IOException;
8import java.net.MalformedURLException;
9import java.net.URL;
10import org.openqa.selenium.remote.DesiredCapabilities;
11import org.junit.AfterClass;
12import org.junit.BeforeClass;
13import org.junit.Test;
14import org.openqa.selenium.support.ui.ExpectedConditions;
15import org.openqa.selenium.support.ui.WebDriverWait;
16
17public class SwiftRadioTest {
18
19 public static URL url;
20 public static DesiredCapabilities capabilities;
21 public static IOSDriver<IOSElement> driver;
22
23 @BeforeClass
24 public static void setupAppiumDriver() throws MalformedURLException, IOException {
25 final String URL_STRING = "http://127.0.0.1:4723/wd/hub";
26 url = new URL(URL_STRING);
27
28 final Class clazz = EvincedA11yValidatorTest.class;
29 final File file = new File(clazz.getResource("/SwiftRadio.app").getFile());
30
31 capabilities = new DesiredCapabilities();
32 capabilities.setCapability(MobileCapabilityType.AUTOMATION_NAME, "XCUITest");
33 capabilities.setCapability("useNewWDA", false);
34 capabilities.setCapability(MobileCapabilityType.NO_RESET, true);
35 capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, "iPhone Simulator");
36 capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, "iOS");
37 capabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION, "12.0");
38 capabilities
39 .setCapability(MobileCapabilityType.APP,
40 file.getAbsolutePath());
41
42 driver = new IOSDriver<IOSElement>(url, capabilities);
43 }
44
45 @AfterClass
46 public static void tearDown() {
47 driver.quit();
48 }
49
50 @Test public void testStationsScreen() {
51 IOSElement firstStationTableCell = (IOSElement) new WebDriverWait(driver, 30)
52 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("StationTableViewCell")));
53
54 firstStationTableCell.click();
55
56 IOSElement playButton = (IOSElement) new WebDriverWait(driver, 30)
57 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("XCUIElementTypeButton")));
58 }
59}

We wrote this test for an application called Swift Radio. It is a small open-source app for listening to various radio stations and podcasts.

The purpose of these tests is to check the functionality of the main application screen and ensure a user can successfully navigate to their choice of the radio station. For now, this test is only concerned with the functional testing of the app. However, with the help of the Evinced Appium SDK, we can also check it for accessibility issues along the way. Let’s go through this process with the following step-by-step instructions.

Step #1 - Initialize the Evinced Appium SDK

Before making any assertions against our app, we need to initialize EvincedAppiumSdk object. This object is used primarily as an entry point to all of the accessibility scanning features. Since we are going to use it primarily within our tests, the best place for its initialization will be our setupAppiumDriver method which gets executed first.

1import io.appium.java_client.MobileBy;
2import io.appium.java_client.ios.IOSDriver;
3import io.appium.java_client.ios.IOSElement;
4import io.appium.java_client.remote.MobileCapabilityType;
5
6import java.io.File;
7import java.io.IOException;
8import java.net.MalformedURLException;
9import java.net.URL;
10import org.openqa.selenium.remote.DesiredCapabilities;
11import org.junit.AfterClass;
12import org.junit.BeforeClass;
13import org.junit.Test;
14import org.openqa.selenium.support.ui.ExpectedConditions;
15import org.openqa.selenium.support.ui.WebDriverWait;
16
17public class SwiftRadioTest {
18
19 public static URL url;
20 public static DesiredCapabilities capabilities;
21 public static IOSDriver<IOSElement> driver;
22
23 public static EvincedAppiumSdk evincedAppiumSdk;
24
25 @BeforeClass
26 public static void setupAppiumDriver() throws MalformedURLException, IOException {
27 final String URL_STRING = "http://127.0.0.1:4723/wd/hub";
28 url = new URL(URL_STRING);
29
30 final Class clazz = EvincedA11yValidatorTest.class;
31 final File file = new File(clazz.getResource("/SwiftRadio.app").getFile());
32
33 capabilities = new DesiredCapabilities();
34 capabilities.setCapability(MobileCapabilityType.AUTOMATION_NAME, "XCUITest");
35 capabilities.setCapability("useNewWDA", false);
36 capabilities.setCapability(MobileCapabilityType.NO_RESET, true);
37 capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, "iPhone Simulator");
38 capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, "iOS");
39 capabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION, "12.0");
40 capabilities
41 .setCapability(MobileCapabilityType.APP,
42 file.getAbsolutePath());
43
44 driver = new IOSDriver<IOSElement>(url, capabilities);
45 evincedAppiumSdk = new EvincedAppiumSdk(driver);
46 evincedAppiumSdk.setupCredentials(<your service account ID>, <your API key>);
47 }
48
49 @AfterClass
50 public static void tearDown() {
51 driver.quit();
52 }
53
54 @Test
55 public void testStationsScreen() {
56 IOSElement firstStationTableCell = (IOSElement) new WebDriverWait(driver, 30)
57 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("StationTableViewCell")));
58
59 firstStationTableCell.click();
60
61 IOSElement playButton = (IOSElement) new WebDriverWait(driver, 30)
62 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("XCUIElementTypeButton")));
63 }
64}

The EvincedAppiumSdk requires only an instance of the Appium driver. There are additional configuration options that can be included as well. More information can be found in the API section of this documentation.

Step #2 - Identify which application states you want to check for a11y issues

Once the SDK is set up, let’s try to carefully consider which states of our application are actually worth checking for accessibility issues. Let’s take a look at out UI test once again:

1@Test
2public void testStationsScreen() {
3 IOSElement firstStationTableCell = (IOSElement) new WebDriverWait(driver, 30)
4 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("StationTableViewCell")));
5 // We have reached the main application screen with list of stations. It may have some accessibility issues!
6
7 firstStationTableCell.click();
8
9 IOSElement playButton = (IOSElement) new WebDriverWait(driver, 30)
10 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("XCUIElementTypeButton")));
11 // Here we have reached yet another screen - a concrete station view. It is probably worth checking for accessibility issues as well.
12}

So far we have identified two places in our UI test where we may need to perform accessibility assertion. Each place represents some distinct application state and each of these states may contain certain accessibility issues. So, let’s put our checks into place.

Step #3 - Setup analyzing breakpoints

The only thing we need to do now is to simply add evincedAppiumSdk.analyze() calls in places within our test which we previously identified as “interesting” ones in terms of accessibility. Here is how it should look:

1@Test
2public void testStationsScreen() {
3 IOSElement firstStationTableCell = (IOSElement) new WebDriverWait(driver, 30)
4 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("StationTableViewCell")));
5 // We have reached the main application screen with list of stations. It may have some accessibility issues!
6 evincedAppiumSdk.analyze();
7
8 firstStationTableCell.click();
9
10 IOSElement playButton = (IOSElement) new WebDriverWait(driver, 30)
11 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("XCUIElementTypeButton")));
12 // Here we have reached yet another screen - a concrete station view. It is probably worth checking for accessibility issues as well.
13 evincedAppiumSdk.analyze();
14}

The analyze method collects an accessibility snapshot of the current application state and puts it into internal in-memory storage. This snapshot contains a lot of information including application screenshot, accessibility labels and dimensions of UI elements.

Now that we have gathered the information we need we are finally ready to actually find our whether our app is accessible or not.

Step #4 - Assert your screens at the end of your test suite

As our test was executed we collected a lot of accessibility snapshots via the evincedAppiumSdk.analyze() calls. We can now perform accessibility assertions at the end of our test suite. Referring back again to our initial UI test the best place for this assertion will be the method that gets invoked last - tearDown.

1@AfterClass
2public static void tearDown() {
3 List<Report> reports = evincedAppiumSdk.reportStored();
4 driver.quit();
5}

To generate the actual object representation of your accessibility report simply call the reportStored method. This way Evinced Appium SDK will check all of the previously collected accessibility snapshots for having some accessibility issues. From this moment you are free to assert the given list of Report objects in any way you want (feel free to explore the corresponding chapter in the “API“ section).

For the sake of simplicity of this tutorial let’s simply assume that our application is accessible as long as it has no accessibility issues found. Thus, if we have at least one accessibility issue detected - we want our tests to be failed. Let’s add the corresponding assertion:

1@AfterClass
2public static void tearDown() {
3 List<Report> reports = evincedAppiumSdk.reportStored();
4 List<Report> reportsWithIssues = reports.stream() // Filter out only reports which contain some issues
5 .filter(report -> report.hasIssues())
6 .collect(Collectors.toList());
7
8 assertEquals(reportsWithIssues.size(), 0); // Making the actual assertion
9 driver.quit();
10}

You are now set to run the test and ensure the accessibility of your application! So, go ahead and run it via your IDE or any other tooling you use for Java development.

Step #5 - Read the results

At this point, you will likely see that your test has failed, which is totally fine. Swift Radio certainly has some accessibility flaws, and the fact that we have caught them simply means that Evinced Appium SDK is working!

The only thing which is missing here is the actual list of accessibility issues with would like to look at. First of all, let’s pay a bit of attention to the terminal output of our tests. You should see your tests failed due to AssertionError with the following description:

1Number of a11y issues found. Please refer to the html report located at: /Users/johndoe/dev/swift-radio/evinced-reports/evinced-report-a6a47bed-d7b5-467e-a5c2-9e6ac4c1f247.html
2java.lang.AssertionError: Number of a11y issues found. Please refer to the html report located at: /Users/johndoe/dev/swift-radio/evinced-reports/evinced-report-a6a47bed-d7b5-467e-a5c2-9e6ac4c1f247.html
3 at com.evinced.a11y.validator.appium.java.core.EvincedA11yValidator.assertTreesHaveA11yIssues(EvincedA11yValidator.java:72)
4 //... omited for brevity

According to this output, we have HTML and JSON files located right in the root of our project, inside the folder named evinced-reports. For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.

Step #6 - Make your app more accessible

Congratulations! Over the course of this 5-step tutorial we have managed to integrate accessibility tests right into our existing UI tests with the help of Evinced Appium SDK and moreover, we have even found several issues in the app.

This is the perfect time for us to carefully investigate all of the issues detected (with help of the knowledge base) and put our JSON files into static storage for later use. With the history of all of the results, we can achieve a lot of exciting things including tracking our progress fixing all of the accessibility issues over time, detecting regressions, and gathering additional metrics.

Feel free to continue your journey with the Evinced Appium SDK by browsing the API section.

FAQ

1. Do you have support for languages other than Java?

Not at this moment, unfortunately. We are planning to expand our tooling coverage by writing dedicated libraries for other popular UI testing languages such as JavaScript and Python.

2. Can I configure which validations to run?

Currently, all of the available validations are run. We have not seen any drastic impact on performance caused by any particular validation. In the future, we are planning to introduce more sophisticated capabilities for filtering results by different properties (e.g. severity or problem type) or even by using your own custom predicate.