Appium Java SDK
The Evinced Appium Java SDK integrates with new or existing Appium tests to automatically detect accessibility issues. With the addition of as few as 5 lines of code to your Appium framework, you can begin to analyze your entire application to understand how it can become more accessible. At the conclusion of the test, a rich and comprehensive HTML or JSON report is generated to track issues in any reporting tool.
Supported versions / frameworks
Evinced Appium SDK supports the following values for Appium’s automationName desired capability:
XCUI SDK
UIAutomator2
Espresso
Older versions of automation drivers (e.g deprecated Appium iOS driver based on UIAutomation) are not supported.
Prerequisites:
- Java 8 or higher
- Appium Server ver. 1.21 or higher
- Appium Java client v7 or higher (older version may still be compatible)
Get Started
Setup
In order to use any of the Evinced Mobile SDKs you first need to create an Evinced account. Once your account is created, you will get your apiKey
and a matching serviceAccountId
. Pass these these tokens using the setupCredentials()
method and Evinced will validate access upon test execution. If an outbound internet connection is not available in your running environment - contact us at support@evinced.com to get an offline APIKey.
Install Evinced Appium SDK from the remote registry
If you are using Gradle as your build tool follow the steps below:
Add Artifactory plugin to the plugins section of your
build.gradle
:1 plugins {2 // ... other plugins3 id 'com.jfrog.artifactory' version '4.24.14'4 }Or if you use kotlin to configurate
build.gradle.kts
1 plugins {2 // ... other plugins3 id("com.jfrog.artifactory") version "4.24.14"4 }To the bottom of your build.gradle file add the next configuration for Artifactory plugin:
1artifactory {2 contextUrl = 'https://evinced.jfrog.io/artifactory/'34 resolve {5 repository {6 repoKey = 'public-gradle'7 maven = true8 }9 }10}For
build.gradle.kts
:1artifactory {2 setContextUrl("https://evinced.jfrog.io/artifactory")34 resolve {5 repository {6 setRepoKey("public-gradle")7 setMaven(true)8 }9 }10}
If you are using Maven, add the Evinced package repository URL to your build file:
1<repositories>2 <repository>3 <id>evinced-repository</id>4 <url>https://evinced.jfrog.io/artifactory/public-gradle</url>5 </repository>6</repositories>
NOTE: If you see an error com.github.appium:java-client:jar:927db52 was not found please also at the additional Jitpack repository.
1 <repositories>2 <repository>3 <id>jitpack</id>4 <url>https://jitpack.io</url>5 </repository>6</repositories>
Adding Evinced Appium SDK as a project dependency
The SDK can be added to your project as a Maven or Gradle dependency.
When working with Maven add the following dependency to your pom.xml file:
1<dependency>2 <groupId>com.evinced</groupId>3 <artifactId>appium-sdk</artifactId>4 <version>{sdk-version}</version>5 <scope>test</scope>6</dependency>
When working with Gradle add the following to your build.gradle dependencies section:
1dependencies {2 // ... other project dependencies3 testImplementation(group: 'com.evinced', name: 'appium-sdk', version: '{sdk-version}', classifier: 'all')4}
For build.gradle.kts
:
1dependencies {2 // ... other project dependencies3 testImplementation("com.evinced:appium-sdk:{sdk-version}")4}
Important notice: In case of usage of Appium Java Client 8 the {sdk-version} should include the "_java_client_v8" identifier string. The following example will support Appium Java Client 8:
1<dependency>2 <groupId>com.evinced</groupId>3 <artifactId>appium-sdk</artifactId>4 <version>1.10.0_java_client_v8</version>5 <scope>test</scope>6</dependency>
The following example without the "_java_client_v8" identifier string will support Appium Java Client 7 and lower by default:
1<dependency>2 <groupId>com.evinced</groupId>3 <artifactId>appium-sdk</artifactId>4 <version>1.10.0</version>5 <scope>test</scope>6</dependency>
Replace {sdk-version} with the latest SDK version that can be found here: appium sdk jfrog.
Install Evinced Appium SDK from local ZIP archive
You can also perform installation right from the standalone .jar distribution. In this case, you need to do the following:
- Download the .jar file.
- Unpack the provided
evinced-appium-sdk.zip
to any desirable location - Add the following dependencies entries pointing to
appium-sdk-all.jar
- Gradle:1implementation files('/Users/<path-to-your-upacked-folder>/appium-sdk-all.jar')
- Maven:
- First, install the “all“ jar into your local Maven repository by the following command:1mvn install:install-file –Dfile=/Users/<path-to-unpacked-folder>/appium-sdk-all.jar -DgroupId=com.evinced -DartifactId=evinced-appium-sdk -Dversion={sdk-version}
- Add the corresponding dependency into your pom.xml:1 <dependency>2 <groupId>com.evinced</groupId>3 <artifactId>evinced-appium-sdk</artifactId>4 <version>{sdk-version}</version>5 </dependency>
- First, install the “all“ jar into your local Maven repository by the following command:
- Trigger your package manager download new dependencies either via IDE or command line
- Gradle:
ZIP archive also contains two complimentary .jar files which you can utilize:
appium-sdk-javadoc.jar
contains rich JavaDoc documentation which you can view in any web browserappium-sdk-sources.jar
can be attached to your Evinced Appium SDK installation in order to let you leverage JavaDoc right in your IDE. Please, refer to the corresponding documentation of your developer tools
Your first test
Initialize the SDK object
Add the import statement to the beginning of your test file.
1import com.evinced.appium.sdk.core.EvincedAppiumSdk;
Under the hood, the Evinced SDK utilizes several Appium commands in order to get accessibility data from the device under test. For this reason, we need to initialize EvincedAppiumSdk
with the Appium driver instance in the following way:
1public static EvincedAppiumSdk evincedSdk;23@BeforeClass4public static void setupAppiumDriver() throws MalformedURLException, IOException {5 capabilities = new DesiredCapabilities();6 // ...omitting the setting of capabilities for brevity78 driver = new IOSDriver<IOSElement>(url, capabilities);910 evincedSdk = new EvincedAppiumSdk(driver);11}
Here we used the JUnit BeforeClass
annotation to create a method that will be executed at the beginning of the test suite. This is often where the Appium Driver initialization takes place, therefore we can also use it to initialize the EvincedAppiumSdk
instance as well.
Provide your license information
The Evinced Appium SDK requires authentication credentials which consist of two string values mentioned above: apiKey
and serviceAccountId
.
For this purpose use the method called evincedAppiumSdk.setupCredentials(serviceAccountId, apiKey)
as shown:
1public static EvincedAppiumSdk evincedSdk;23@BeforeClass4public static void setupAppiumDriver() throws MalformedURLException, IOException {5 //... the rest of our setup code67 evincedSdk = new EvincedAppiumSdk(driver);89 evincedSdk.setupCredentials(<your service account ID>, <your API key>);10}
You can confirm the success of the authentication in the logs.
Add Evinced accessibility checks
1@Test2public void testMyAppScreen() {3 IOSElement someTableCell = (IOSElement) new WebDriverWait(driver, 30)4 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("TableViewCell")));56 firstStationTableCell.click();7 //... do other interactions with the app89 // Run analysis and get the accessibility report10 Report report = evincedSdk.report();1112 // Assert that there are no accessibility issues13 assertTrue(report.getIssues().size() == 0);14}
Regardless of the test result, the report
method will always output JSON and HTML files. You should be able to find the files by browsing the local folder called evinced-reports
. For more information regarding the HTML and JSON reports as well as the Report
object itself, please see our detailed Mobile Reports page.
If you only need Report
objects during test runtime, and you don't need JSON and HTML report files saved, you can disable creating report files with disableAllReportsOutputFiles
additional option.
1 final InitOptions initOptions = new InitOptions()2 .putAdditionalOption("disableAllReportsOutputFiles", true);3 evincedAppiumSdk.setOptions(initOptions);
Continuous mode
Continuous mode allows continual scanning of the application without the need to insert individual scan calls with in the test code. Simply specify the start and end points using the following methods evincedEngine.startAnalyze()
and evincedEngine.stopAnalyze()
. Then Evinced engines will automatically scan the application after each Appium command. All issues detected during the tests will be added to the HTML and JSON reports automatically generated with the stopAnalyze()
method is called.
To use continuous mode, you need to replace the standard Appium driver with the EvincedAppiumAndroidDriver
for Android or EvincedAppiumIOSDriver
. No changes are needed to your desired capabilities or other driver setup.
1@BeforeClass2public static void setupAppiumDriver() throws IOException {34 // Usual Appium configuration56 driver = new EvincedAppiumAndroidDriver<>(url, capabilities);7 evincedAppiumSdk = new EvincedAppiumSdk(driver);8 evincedAppiumSdk.setupCredentials(TestBuildConfig.EVINCED_SERVICE_ACCOUNT_ID, TestBuildConfig.EVINCED_API_KEY);9}
To initialize the Evinced engines simply use the evincedAppiumSdk.startAnalyze()
method and finish it use evincedAppiumSdk.stopAnalyze()
. You can use it within a single test method or scan an entire class (or more) using before and after hooks.
Here is an example of using continuous mode in a single test case:
1@Test2public void testContinuousMode() {3 evincedAppiumSdk.startAnalyze();45 // appium commands6 // ...78 evincedAppiumSdk.stopAnalyze();9}
Here is an example of using continuous mode in for an entire class:
1@BeforeClass2public static void setup() {3 // ...4 evincedAppiumSdk.startAnalyze();5}67@AfterClass8public static void tearDown() {9 // ...10 evincedAppiumSdk.stopAnalyze();11}
Optional/Advanced
If you need to use your own AppiumDriver class instead of EvincedAppiumAndroidDriver
or EvincedAppiumIOSDriver
as base classes, you can implement CanSetOnExecuteListener
interface. It should have getOnExecuteListener
and setOnExecuteListener
methods and trigger onExecuteListener.executed
on every execution of appium commands.
So it can be implemented the next way:
1// For iOS it can be IOSDriver instead of AndroidDriver2public class YourOwnCustomDriver<T extends WebElement> extends AndroidDriver<T> implements CanSetOnExecuteListener {34 private OnExecuteListener onExecuteListener = null;56 private boolean busy = false;78//9// Your overridden constructors10// ...11//1213 @Override14 public OnExecuteListener getOnExecuteListener() {15 return onExecuteListener;16 }1718 @Override19 public void setOnExecuteListener(OnExecuteListener onExecuteListener) {20 this.onExecuteListener = onExecuteListener;21 }2223 @Override24 public Response execute(String command) {25 return execute(command, ImmutableMap.of());26 }2728 @Override29 public Response execute(String driverCommand, Map<String, ?> parameters) {30 if (onExecuteListener != null && !busy) {31 busy = true;32 onExecuteListener.executed(driverCommand, parameters);33 busy = false;34 }35 return super.execute(driverCommand, parameters);36 }37}
So you can use this driver with Evinced SDK as usual.
1 YourOwnCustomDriver<AndroidElement> driver = new YourOwnCustomDriver<>(url, capabilities);2 EvincedAppiumSdk evincedAppiumSdk = new EvincedAppiumSdk(driver);
Issue filtering
The Evinced Appium SDK supports advanced settings that allow you to apply additional configurations when generating issue reports. Simply create an instance of the EvincedConfig
object with the appropriate options.
Below is an example of using EvincedConfig
and IssueFilter
:
Configure filters:
1final IssueFilter severityFilter = new IssueFilter()2 .severity(Severity.NeedsReview);3final IssueFilter issueFilter = new IssueFilter()4 .issueType(IssueType.TappableArea, IssueType.ColorContrast);56final MobileElement element = driver.findElementById(viewId);7final IssueFilter elementFilter = new IssueFilter().matchByElements(element1);
Global filtering:
1// Create an example filter to exclude all Needs Review issues globally2final IssueFilter globalFilter = new IssueFilter()3 .severity(Severity.NeedsReview);45// Add the IssueFilter to the creation instance of EvincedAppiumSdk6final EvincedConfig globalEvincedConfig = new EvincedConfig()7 .excludeFilters(globalFilter);8final EvincedAppiumSdk evincedAppiumSdk = new EvincedAppiumSdk(driver, new InitOptions(globalEvincedConfig));
Filtering on an individual scan (analyze()
or report()
call):
1// Passing multiple exclude filters for an individual scan2final IssueFilter multipleSeverityFilter = new IssueFilter()3 .severity(Severity.NeedsReview, Severity.BestPractice);45 final IssueFilter complexFilter = new IssueFilter()6 .matchElementById("com.example:id/myElement")7 .issueType(IssueType.CollidingControls)8 .severity(Severity.NeedsReview);910final IssueFilter complexFilter1 = new IssueFilter()11 .matchElementByAccessibilityId("myElementA11yID")12 .issueType(IssueType.DuplicateName, SpecialCharacters)13 .severity(Severity.NeedsReview, BestPractice);14final EvincedConfig evincedConfig = new EvincedConfig()15 .excludeFilters(multipleSeverityFilter, multipleSeverityFilter, complexFilter);1617// Apply the configuration for a specific screen state scan18evincedAppiumSdk.report(false, evincedConfig);19evincedAppiumSdk.analyze(evincedConfig);
Depending on the purpose of filtering, you can use two directions -include or exclude using the appropriate methods:
EvincedConfig().includeFilters(filter)
EvincedConfig().excludeFilters(filters)
List of available issue filtering options
Option | Method | Description |
---|---|---|
viewId | public IssueFilter matchElementById(String viewId) | Android only, native UI element identifier: resource-id |
accessibilityId | public IssueFilter matchElementByAccessibilityId(String a11yId) | iOS only, for XCUITest it is the element's accessibility-id attribute. |
mobileElement | public IssueFilter matchByElements(MobileElement... mobileElement) public IssueFilter matchByElements(List<MobileElement> mobileElements) | Native Appium element |
severity | public IssueFilter severity(Severity... severity) public IssueFilter severity(List<Severity> severities) | Filter by Severity - not apply on UI elements the accessibility validations with specific Severity types. |
issueType | public IssueFilter issueType(IssueType... issueType) public IssueFilter issueType(List<IssueType> issueTypes) | Filter by IssueType - not apply on UI elements the accessibility validations with specific IssueType . |
recursive | public IssueFilter severity(List<Severity> severities) | |
public IssueFilter setRecursive(boolean recursive) | Determines whether or not to include child elements in the current filter. By default - true |
Config file
Along with the configuration creation options within the code, it is also possible to create filters using a JSON file. To import the file use the examples below:
1 // create EvincedConfig instance from file by specify absolute or relative file path2 final EvincedConfig evincedConfigFromFile = EvincedConfig.fromFile("/project_root_evinced_config.json");34 // create EvincedConfig instance from file by specify file name, located in resources directory5 final EvincedConfig evincedConfigFromResources = EvincedConfig.fromResourcesFile("resources_evinced_config.json");
Example File
1{2 "excludeFilters": [3 {4 "viewId": "viewId",5 "accessibilityId": "a11yId",6 "severities": [7 {8 // you can use id or name separately9 "id": "3c68eb41-9227-481d-8eec-9dc8956b1900",10 "name": "Needs Review"11 }12 ],13 "issueTypes": [14 {15 // you can use id or name separately16 "id": "e871b2e7-1b6d-47bc-96db-7f3dff4bfd5a",17 "name": "Duplicate Name"18 }19 ]20 }21 ]22}
Storing screenshots in reports
By default screenshots are provided in HTML reports for each scan, but for JSON-reports they are disabled by default. You can explicitly enable or disable screenshots using the InitOptions
. Screenshots in JSON reports are provided in Base64 format.
Available screenshot options:
Option | Description |
---|---|
InitOptions.ScreenshotOption.Disabled | The screenshot is available only in the HTML report. Disabled by default. |
InitOptions.ScreenshotOption.Base64 | Add a screenshot to the json report and the Report object in Base64. Available using the screenshotBase64 field |
InitOptions.ScreenshotOption.File | Save a screenshot as separately .jpeg file. the name of the screenshot is the same as the id of the corresponding report. |
InitOptions.ScreenshotOption.Both | Save the screenshot as a .jpeg file and also as Base64 in the json report. See options above |
Global screenshot config:
1final InitOptions initOptions = new InitOptions(InitOptions.ScreenshotOption.Base64);2// setup options during the SDK initialization3final EvincedAppiumSdk evincedApiumSdk = new EvincedAppiumSdk(driver, initOptions);45// change the existing options6final InitOptions newOptions = new InitOptions(InitOptions.ScreenshotOption.Disabled);7evincedApiumSdk.setOptions(newOptions);
Rule validation configurations
RulesConfig allows you to change the accessibility check for different rules. It is set globally and used for all subsequent reports.
Disabling the tappable area validation
This example disables validation for tappable area:
1evincedEngine.setOptions(new InitOptions(new RuleConfig(IssueType.TAPPABLE_AREA, false))
Disabling the label capitalization validation
This example disables validation for label capitalization:
1evincedEngine.setOptions(new InitOptions(new RuleConfig(IssueType.LABEL_CAPITALIZATION, false))
Disabling the color contrast validation
This example disables validation for color contrast:
1evincedEngine.setOptions(new InitOptions(new RuleConfig(IssueType.COLOR_CONTRAST, false)))
Customize color contrast options
Fuzziness factor configuration for color contrast. For example, the WCAG norm for this ratio is 4.5:1, but if your corporate colors have a ratio of 4.45:1, then this can be specified using the following options:
1final Map<String, Object> colorContrastOptions = new HashMap<>();2colorContrastOptions.put("contrastRatioNormalText", 4.45f);3colorContrastOptions.put("contrastRatioLargeText", 4.45f);4colorContrastOptions.put("contrastRatioNonText", 4.45f);5final RuleConfig colorContrastRuleConfig = new RuleConfig(IssueType.COLOR_CONTRAST, true, colorContrastOptions);6evincedEngine.setOptions(new InitOptions().setRulesConfig(colorContrastRuleConfig));
Exporting meaningful labels
To export a JSON report of the accessibility label of images and buttons regardless of the scan results simply add the options flag:
1final InitOptions initOptions = new InitOptions();2initOptions.putAdditionalOption("exportMeaningfulLabels", true);3evincedEngine.setOptions(initOptions);
You should see a new JSON file created beside other report files with a default name: Evinced_Meaningful_Labels_Test_Results_11_15_2023_09_49_51.json. The structure of this JSON file is described in our Mobile Reports page.
To change the default folder destination and the file name of the meaningful labels JSON report you can use the second additional option with an absolute folder path:
1final InitOptions initOptions = new InitOptions();2final File meaningfulLabelsFile = new File("evinced-reports", "SpecificMeaningfulLabels.json");3initOptions.putAdditionalOption("exportMeaningfulLabels", true);4initOptions.putAdditionalOption("exportMeaningfulLabelsPath", meaningfulLabelsFile.getAbsolutePath());5evincedEngine.setOptions(initOptions);
Exporting elements with no issues
To export a JSON report of all scanned elements that had no issues found, simply add the additional options flag:
1final InitOptions initOptions = new InitOptions();2initOptions.putAdditionalOption("exportNoIssuesElements", true);3evincedEngine.setOptions(initOptions);
You should see a new JSON file created beside other report files with a default name: Evinced_Accessibility_Test_Results_No_Issues_Elements_07_25_2024_09_48_55.json.
The structure of this JSON file is described in our Mobile Reports page.
To change the default folder destination and the file name of the elements with no issues JSON report you can use another additional option with an absolute folder path:
1final InitOptions initOptions = new InitOptions();2final File noIssuesElementsFile = new File("evinced-reports", "SpecificNoIssuesElements.json");3initOptions.putAdditionalOption("exportNoIssuesElements", true);4initOptions.putAdditionalOption("exportNoIssuesElementsPath", noIssuesElementsFile.getAbsolutePath());5evincedEngine.setOptions(initOptions);
You can also obtain elements with no issues JSON report on test runtime using subreports:
1final InitOptions initOptions = new InitOptions();2initOptions.putAdditionalOption("exportNoIssuesElements", true);3evincedEngine.setOptions(initOptions);4Report report = evincedAppiumSdk.report();5NoIssuesElementsReport noIssueElementsReport = report.getSubReports().getNoIssueElements();6List<NoIssuesElement> noIssuesElements = noIssueElementsReport.getItems();
Tappable Area
Every native mobile application features a wide variety of UI elements which are interactive via tap. Consider elements like buttons, text fields, toggles and others. Each of them should be tapped in order to become focused and to allow any further interaction. The lower the tappable area of the element (an imaginary square where user should tap) the less convenient it is to operate with the element. Platform constraints describe the width and height of minimum tappable area.
tappableAreaDimensions - the minimum size that does not trigger Tappable Area validation. By default for Android this value is 48dp, for iOS it is 44px.
For example
1final Map<String, Object> tapAreaDimension = new HashMap<>();2tapAreaDimension.put("tappableAreaDimensions", 70);3RuleConfig tapArea = new RuleConfig(IssueType.TappableArea, true, tapAreaDimension);4InitOptions initOptions = new InitOptions().setRulesConfig(tapArea);5evincedAppiumSdk.setOptions(initOptions);
Uploading reports to Evinced Platform
[Evinced Platform] allows you to seamlessly collect, organize, and visualize Evinced accessibility reports in one place. In this section, we will guide you through the key functionalities of the upload methods from Evinced Appium Java SDK, which was introduced in version 1.20.0, to the Evinced Platform. This upload method is fully compatible with the previous versions of the Evinced Espresso SDK API, and is disabled by default.
Initialize report uploading to the Evinced Platform
In order to start using [Evinced Platform] feature you need to include the PlatformConfig
in the InitOptions
:
1@BeforeClass2public static void setup() {3 ...4 final InitOptions options = new InitOptions(5 new InitOptions.PlatformConfig(6 InitOptions.UploadOption.ENABLED_BY_DEFAULT7 )8 );9 evincedAppiumSdk.setOptions(options);10}
In this case, all of the reports from a test class will be uploaded to the [Evinced Platform] where you can view the accessibility issues.
Selective uploading
You can configure uploading only a specific set of reports:
1@Test2public void test1() {3 ...4 final Report report1 = evincedAppiumSdk.report();5 ...6 final Report report2 = evincedAppiumSdk.report(PlatformUpload.ENABLED);7}
In this case, only the second report will be uploaded to [Evinced Platform].
The same behavior applies to internally stored reports reportStored()
after the analyze()
calls and to the stopAnalyze()
in the continuous mode:
1@Test2public void test1() {3 ...4 evincedEngine.analyze();5 List<Report> reportList = evincedAppiumSdk.reportStored(PlatformUpload.ENABLED);6}
Or:
1@Test2public void test1() {3 ...4 evincedEngine.startAnalyze();5 // test body6 List<Report> reportList = evincedAppiumSdk.stopAnalyze(PlatformUpload.ENABLED);7}
Notice: that in these examples we didn't use InitOptions.PlatformConfig
. It's not required to use this config argument to enable the selective report uploading.
Please find more information about continuous mode here if needed.
Disable report uploading to Platform
In order to disable the upload method for all reports without changing arguments for each report()
, reportStored()
or stopAnalyze()
methods you can use the following InitOption
:
1@BeforeClass2public static void setup() {3 ...4 final InitOptions options = new InitOptions(5 new InitOptions.PlatformConfig(6 // Setting the forceDisableReportsUpload flag to true, disable all reports uploading.7 true8 )9 );10 evincedAppiumSdk.setOptions(options);11}1213@Test14public void test1() {15 ...16 final Report report1 = evincedAppiumSdk.report();17 ...18 final Report report2 = evincedAppiumSdk.report(PlatformUpload.ENABLED);19}
Labeling uploaded reports
To be able to distinguish between different reports on [Evinced Platform], the following pre-defined labels will be collected and added programmatically.
Labels | Description |
---|---|
appName | The display name of the target application. |
appVersion | For example 1.1.3 . |
appBuildNumber | For example 56 . |
deviceManufacturer | For example - Google . |
deviceModel | For example - sdk_gphone_x86_64 . |
deviceName | For example - generic_x86_64_arm64 . |
testMethodName | The method name of the actual test. |
testCaseName | The class name of the actual test. |
deviceType | Can be emulator or physical . |
isInAppSdkConnected | Should be true in case of accessibility validation using Evinced InApp SDK inside a target application. |
isInAppSdkInstalled | Should be true in case of the presence of Evinced InApp SDK in the target application. |
osName | Should always be Android . |
osVersion | The version of the target Android. |
SDKBuildNumber | Should be equal to 1.20.0 or higher. |
SDKBuildType | Should always be Espresso SDK . |
Customizing labels before uploading reports
To distinguish between reports from different parts of your applications or tests you can also add custom labels to the uploaded reports.
1@Test2public void test1() {3 ...4 Map<String, String> labels1 = new HashMap<>();5 labels1.put("MyCustomLabelKey1", "MyCustomLabelValue1");6 final Report report1 = evincedAppiumSdk.report(PlatformUpload.ENABLED, labels1);7 ...8 Map<String, String> labels2 = new HashMap<>();9 labels2.put("MyCustomLabelKey2", "MyCustomLabelValue2");10 final Report report2 = evincedAppiumSdk.report(PlatformUpload.ENABLED, labels2);11}
The custom key "MyCustomLabelKey1" and the custom value "MyCustomLabelValue1" will be attached to the first report. The custom key "MyCustomLabelKey2" and the custom value "MyCustomLabelValue2" will be attached to the second report.
Customizing labels before uploading reports for the entire test class
In order to label all reports inside your test class with a common key/value, use the addTestCaseMetadata()
method:
1@BeforeClass2public static void setup() {3 ...4 evincedAppiumSdk.addTestCaseMetadata("MyCustomCommonLabelKey", "MyCustomCommonLabelValue");5 ...6}78@Test9public void test1() {10 ...11 Map<String, String> labels1 = new HashMap<>();12 labels1.put("MyCustomLabelKey1", "MyCustomLabelValue1");13 final Report report1 = evincedAppiumSdk.report(PlatformUpload.ENABLED, labels1);14 ...15 Map<String, String> labels2 = new HashMap<>();16 labels2.put("MyCustomLabelKey2", "MyCustomLabelValue2");17 final Report report2 = evincedAppiumSdk.report(PlatformUpload.ENABLED, labels2);18}
In this case, all reports in the test class scope will have "MyCustomCommonLabelKey"/"MyCustomCommonLabelValue". In addition, "MyCustomLabelKey1"/"MyCustomLabelValue1" will be added only to the first report and "MyCustomLabelKey2"/"MyCustomLabelValue2" will be added only to the second report.
Test assertion
If you want to interrupt the test execution in case the report is not uploaded, it's possible to set the failTestOnUploadError
argument of the InitOptions.PlatformConfig
:
1@BeforeClass2public static void setup() {3 ...4 final InitOptions options = new InitOptions(5 new InitOptions.PlatformConfig(6 InitOptions.UploadOption.DISABLED_BY_DEFAULT,7 // Enabled the failTestOnUploadError flag8 true9 )10 );11 evincedAppiumSdk.setOptions(options);12}1314@Test15public void test1() {16 ...17 Map<String, String> labels1 = new HashMap();18 labels1.put("MyCustomLabelKey1", "MyCustomLabelKey1");19 final Report report1 = evincedAppiumSdk.report(PlatformUpload.ENABLED, labels1);20 // The test will fail in case of any uploading error, it can be an internal connection or routing problems21 ...22}
In the case of upload errors you should see an exception with a detailed error message that can be found in the logcat window to help understand the cause of the error and possible solutions.
List of available Platform upload options
forceDisableReportsUpload
forceDisableReportsUpload = true
- the upload method is completely disabled. All report uploading will be disabled regardless of the upload argument:
1@BeforeClass2public static void setup() {3 ...4 final InitOptions options = new InitOptions(5 new InitOptions.PlatformConfig(6 // forceDisableReportsUpload = true7 true8 )9 );10 ...11}1213@Test14public void test1() {15 ...16 // This report won't be uploaded to Evinced Platform, because of forceDisableReportsUpload = true17 Report report1 = evincedAppiumSdk.reportStored(PlatformUpload.ENABLED);18 ...19 // These reports won't be uploaded to Evinced Platform, because of forceDisableReportsUpload = true20 List<Report> reportList2 = evincedAppiumSdk.reportStored(PlatformUpload.ENABLED);21 ...22 // These reports won't be uploaded to Evinced Platform, because of forceDisableReportsUpload = true23 List<Report> reportList3 = evincedAppiumSdk.stopAnalyze(PlatformUpload.ENABLED);24 ...25}
forceDisableReportsUpload = false
is a default value of the upload to Platform configuration;
uploadOption
InitOptions.UploadOption.ENABLED_BY_DEFAULT
- All reports will be uploaded, except for the reports with the upload argument equals to PlatformUpload.DISABLED
:
1@BeforeClass2public static void setup() {3 ...4 final InitOptions options = new InitOptions(5 new InitOptions.PlatformConfig(6 // uploadOption = InitOptions.UploadOption.ENABLED_BY_DEFAULT,7 InitOptions.UploadOption.ENABLED_BY_DEFAULT8 )9 );10 evincedAppiumSdk.setOptions(options);11 ...12}1314@Test15public void test1() {16 ...17 // This report won't be uploaded to Evinced Platform18 Report report1 = evincedAppiumSdk.reportStored(PlatformUpload.DISABLED);19 ...20 // These reports won't be uploaded to Evinced Platform21 List<Report> reportList2 = evincedAppiumSdk.reportStored(PlatformUpload.DISABLED);22 ...23 // These reports won't be uploaded to Evinced Platform24 List<Report> reportList3 = evincedAppiumSdk.stopAnalyze(PlatformUpload.DISABLED);25 ...26}
InitOptions.UploadOption.DISABLED_BY_DEFAULT
- Only reports that are marked with upload argument equal to PlatformUpload.ENABLED
will be uploaded:
1@BeforeClass2public static void setup() {3 ...4 final InitOptions options = new InitOptions(5 new InitOptions.PlatformConfig(6 // uploadOption = InitOptions.UploadOption.DISABLED_BY_DEFAULT,7 InitOptions.UploadOption.DISABLED_BY_DEFAULT8 )9 );10 evincedAppiumSdk.setOptions(options);11 ...12}1314@Test15public void test1() {16 ...17 // This report will be uploaded to Evinced Platform18 Report report1 = evincedAppiumSdk.reportStored(PlatformUpload.ENABLED);19 ...20 // These reports will be uploaded to Evinced Platform21 List<Report> reportList2 = evincedAppiumSdk.reportStored(PlatformUpload.ENABLED);22 ...23 // These reports will be uploaded to Evinced Platform24 List<Report> reportList3 = evincedAppiumSdk.stopAnalyze(PlatformUpload.ENABLED);25 ...26}
failTestOnUploadError
failTestOnUploadError = true
- Stop the test execution in case of an upload error;
failTestOnUploadError = false
- Continues the test execution no matter of any upload errors. Default value.
The state table of the configurations is as following:
forceDisable ReportsUpload | uploadOption | failTestOn UploadError | upload | Description |
---|---|---|---|---|
false | DISABLED_BY_DEFAULT | false | null | Report won't be uploaded. |
false | DISABLED_BY_DEFAULT | false | ENABLED | Report will be uploaded. |
false | DISABLED_BY_DEFAULT | false | DISABLED | Report won't be uploaded. |
false | ENABLED_BY_DEFAULT | false | null | Report will be uploaded. |
false | ENABLED_BY_DEFAULT | false | ENABLED | Report will be uploaded. |
false | ENABLED_BY_DEFAULT | false | DISABLED | Report won't be uploaded. |
true | DISABLED_BY_DEFAULT | false | null | Report won't be uploaded. |
true | DISABLED_BY_DEFAULT | false | ENABLED | Report won't be uploaded. |
true | DISABLED_BY_DEFAULT | false | DISABLE | Report won't be uploaded. |
true | ENABLED_BY_DEFAULT | false | null | Report won't be uploaded. |
true | ENABLED_BY_DEFAULT | false | ENABLED | Report won't be uploaded. |
true | ENABLED_BY_DEFAULT | false | DISABLE | Report won't be uploaded. |
false | DISABLED_BY_DEFAULT | true | null | Report won't be uploaded. |
false | DISABLED_BY_DEFAULT | true | ENABLED | Report will be uploaded. Test execution will be interrupted in case of an upload error. |
false | DISABLED_BY_DEFAULT | true | DISABLE | Report won't be uploaded. |
false | ENABLED_BY_DEFAULT | true | null | Report will be uploaded. Test execution will be interrupted in case of an upload error. |
false | ENABLED_BY_DEFAULT | true | ENABLED | Report will be uploaded. Test execution will be interrupted in case of an upload error. |
false | ENABLED_BY_DEFAULT | true | DISABLE | Report won't be uploaded. |
List of available Platform upload arguments
report(PlatformUpload.ENABLED)
- Upload the report or reports. Will have an effect when the InitOptions.PlatformConfig
is InitOptions.UploadOption.ENABLED_BY_DEFAULT
or InitOptions.UploadOption.DISABLED_BY_DEFAULT
. Can be used as an argument is report(PlatformUpload.ENABLED)
, stopAnalyze(PlatformUpload.ENABLED)
or reportStored(PlatformUpload.ENABLED)
methods;
report(PlatformUpload.DISABLED)
- Don't upload the report or reports. Will have an effect when the InitOptions.PlatformConfig
is InitOptions.UploadOption.ENABLED_BY_DEFAULT
or InitOptions.UploadOption.DISABLED_BY_DEFAULT
Can be used as an argument is report(PlatformUpload.DISABLED)
, stopAnalyze(PlatformUpload.DISABLED)
or reportStored(PlatformUpload.DISABLED)
methods.
Integration Android Mobile Kit feature
Android/iOS Mobile Kit adds new types of validations for your target application.
Add the import statement to the beginning of your test file.
1import com.evinced.appium.sdk.core.models.MobileKitConfig;
Setup Mobile Kit config.
1@BeforeClass2public static void setupAppiumDriver() throws MalformedURLException, IOException {3 capabilities = new DesiredCapabilities();4 // ...omitting the setting of capabilities for brevity56 driver = new EvincedAppiumAndroidDriver<>(url, capabilities);7 InitOptions initOptions = new InitOptions(InitOptions.ScreenshotOption.Both)8 // Setting mobileKitEnabled = true9 .setMobileKitConfig(new MobileKitConfig(true));1011 evincedAppiumSdk = new EvincedAppiumSdk(driver, initOptions);12 evincedAppiumSdk.setupCredentials(<your service account ID>, <your API key>);13}
Note: Make sure that your target application has the MobileKit dependency as described here: Android Mobile Kit Documentation.
In case you need to get a certain validation and your test relies on this validation. You can stop the test execution in case of Mobile Kit connection issues using the failOnMobileKitConnectionError
flag.
1@BeforeClass2public static void setupAppiumDriver() throws MalformedURLException, IOException {3 capabilities = new DesiredCapabilities();4 // ...omitting the setting of capabilities for brevity56 driver = new EvincedAppiumAndroidDriver<>(url, capabilities);7 InitOptions initOptions = new InitOptions(InitOptions.ScreenshotOption.Both)8 // Setting mobileKitEnabled = true9 // And failMobileKitConnectionError = true10 .setMobileKitConfig(new MobileKitConfig(true, true));11 evincedAppiumSdk = new EvincedAppiumSdk(driver, initOptions);12 evincedAppiumSdk.setupCredentials(<your service account ID>, <your API key>);13}
If you can't use the provided EvincedAppiumAndroidDriver
or EvincedAppiumIOSDriver
in your setup. Please, don't forget to obtain a driver back using evincedAppiumSdk.getDriver()
. Due to the internal SDK logic, the driver will be recreated in case of Mobile Kit usage.
1@BeforeClass2public static void setupAppiumDriver() throws MalformedURLException, IOException {3 capabilities = new DesiredCapabilities();4 // ...omitting the setting of capabilities for brevity56 driver = new YourCustomDriver<>(url, capabilities);7 InitOptions initOptions = new InitOptions(InitOptions.ScreenshotOption.Both)8 // Setting mobileKitEnabled = true9 .setMobileKitConfig(new MobileKitConfig(true));1011 evincedAppiumSdk = new EvincedAppiumSdk(driver, initOptions);12 evincedAppiumSdk.setupCredentials(<your service account ID>, <your API key>);1314 driver = (AndroidDriver<AndroidElement>) evincedAppiumSdk.getDriver();15}
Note: Make sure that your target application has the Mobile Kit dependency.
API
EvincedAppiumSdk
public EvincedAppiumSdk(AppiumDriver<?> originalDriver)
Default constructor which accepts an AppiumDriver
instance for performing accessibility snapshot collection. Can be easily instantiated in the following way:
1IOSDriver driver = new IOSDriver<IOSElement>(url, capabilities);2EvincedAppiumSdk evincedSdk = new EvincedAppiumSdk(driver);
public EvincedAppiumSdk(AppiumDriver<?> originalDriver, EvincedAppiumSdkOptions options)
Additional constructor which allows you to configure the different aspects of a11y testing via EvincedAppiumSdkOptions
object. For instance, one can set a custom path for storing reporting artifacts by using the example below:
1driver = new IOSDriver<IOSElement>(url, capabilities);23EvincedAppiumSdkOptions evincedSdkOptions = new EvincedAppiumSdkOptions(); // create options object4evincedSdkOptions.setOutputDir("my-output-dir"); // set custom output directory56evincedSdk = new EvincedAppiumSdk(driver, evincedSdkOptions);
public boolean setupCredentials(String serviceAccountId, String apiKey)
A method to add credentials confirming the validity of the user’s license. It accepts the following arguments:
serviceAccountId
- Evinced service account IDapiKey
- Evinced API key
Returns a boolean flag indicating the success of the operation - if it is false there is something wrong with either validity of the license or with the checking infrastructure. See logs for more detailed output.
public boolean setupOfflineCredentials(String serviceAccountId, String accesstoken)
A method to add credentials confirming the validity of the user’s license with an offline token. It accepts the following arguments:
serviceAccountId
- Evinced service account IDaccesstoken
- Evinced access token
Returns a boolean flag indicating the success of the operation - if it is false there is something wrong with either validity of the license or with the checking infrastructure. See logs for more detailed output.
public void startAnalyze(EvincedConfig evincedConfig)
Watch for screen state changes and record all accessibility issues until stopAnalyze()
is called. It works the same way as EvincedAppiumSdk#analyze()
but collects snapshot on each appium command that can change application state.
Make sure that driver you set in constructor extends EvincedAppiumAndroidDriver or EvincedAppiumIOSDriver, or implements CanSetOnExecuteListener in similar way.
Available overloads:
public void startAnalyze();
public List<Report> stopAnalyze(boolean shouldAssert)
Stop flow analyze and returns a report with all issues found from the last startAnalyze()
call to allow users to perform any advanced assertions on their own.
If it is not needed, simple assertion may be delegated directly to the SDK by supplying shouldAssert argument set to true. In this case, the SDK will automatically throw an AssertionError if there is at least one issue found.
Available overloads:
public void stopAnalyze();
public Report report(shouldAssert: boolean)
Generates accessibility report for the current state of the application. Returns a Report
instance to allow users to perform any advanced assertions on their own. If it is not needed, a simple assertion may be delegated directly to the SDK by supplying shouldAssert
argument set to true
. In this case, the SDK will automatically throw an AssertionError
if there is at least one issue found. For more information regarding the Report
object , please see our detailed Mobile Reports page.
This method always writes HTML and JSON files to the configured directory.
Available overloads:
public Report report();
Example usage:
1Report report = evincedSdk.report(true); // assert is performed by EvincedAppiumSdk23Report reportForMyOwnAssertion = evincedSdk.report();4assertEquals(reportForMyOwnAssertion.hasIssues(), false); // performing assertions on your own
public void analyze()
Collects an accessibility snapshot of the current application state and puts it into internal in-memory storage. This method is supposed to be used in conjunction with EvincedAppiumSdk.assertStored()
for performing actual assertions against the content of this storage.
Example usage:
1 IOSElement firstStationTableCell = (IOSElement) new WebDriverWait(driver, 30)2 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("StationTableViewCell")));3 // we have achieved the state of the app we would like to test. Let's take a snapshot.4 evincedAppiumSdk.analyze();
public List<Report> reportStored(shouldAssert: boolean)
Generates a list of accessibility reports, one for each accessibility snapshot collected by EvincedAppiumSdk.analyze()
. Just like EvincedAppiumSdk.report()
this method may either perform simple assertion on its own (throwing an exception if there is at least one report with issues) or allow users to define their own assertions on the returned Report
objects. This behavior can be configured via the shouldAssert
flag.
This method always writes HTML and JSON files to the configured directory. Both of the files contain combined information about all of the application states collected. For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.
Available overloads:
public List<Report> reportStored();
Example usage:
1@AfterClass2public static void tearDown() {3 // Option 14 List<Report> reportsForOwnAssertion = evincedSdk.reportStored(); // making our own assertions5 // Option 26 List<Report> reports = evincedSdk.reportStored(true); // delegating assertion to the SDK7 // Option 38 List<Report> reportsWithIssues = reportsForOwnAssertion.stream() // Filter out only reports which contain some issues9 .filter(report -> report.hasIssues())10 .collect(Collectors.toList());11 assertEquals(reportsWithIssues.size(), 0); // Making the actual assertion1213 driver.quit();14}
public void clearStored()
Removes all of the accessibility snapshots previously collected by EvincedAppiumSdk.analyze()
calls.
InitOptions
1public InitOptions()
Default public constructor with no arguments.
1public InitOptions(String outputDir)2public InitOptions(String outputDir, EvincedConfig evincedConfig)
Overloaded constructors
1public InitOptions setOutputDir(String outputDir)
Method for defining a custom output directory where a user would like to store reporting artifacts (JSON and HTML files).
1public InitOptions setEvincedConfig(EvincedConfig evincedConfig)
Method for defining a EvincedConfig
object as a global setting
1public InitOptions screenshotOption(ScreenshotOption screenshotOption)
Set option to save a screenshot in a output JSON report or as a separate .jpeg
file.
1public InitOptions setRuleConfigs(List<RuleConfig> ruleConfigs)
Set rule configs
1public InitOptions setReportName(String reportName)
Set the output HTML/JSON report name prefix
1 public InitOptions setMobileKitConfig(MobileKitConfig mobileKitConfig)
Sets Mobile Kit feature configurations
EvincedConfig
An object that defines the options that will be used when performing accessibility validations and generating reports.
Default public constructors for creating EvincedConfig
:
1public EvincedConfig()
Set exclude filters:
1public EvincedConfig excludeFilters(IssueFilter... excludeFilters)2public EvincedConfig excludeFilters(List<IssueFilter> excludeFilters)
Set include filters:
1public EvincedConfig includeFilters(IssueFilter... includeFilters)2public EvincedConfig includeFilters(List<IssueFilter> includeFilters)
Enable storing screenshots in JSON-report:
ScreenshotOption
Enum that allows enable or disable saving screenshots in JSON-reports and can be Base64
, File
, Both
or Disabled
.
IssueFilter
A filter object that allows you to exclude UI elements from availability reports by certain criteria
IssueFilter
public methodes:
1public IssueFilter matchElementById(String... viewIds)2public IssueFilter matchElementByAccessibilityId(String... accessibilityIds)3public IssueFilter matchByElements(MobileElement... mobileElement)4public IssueFilter matchByElements(List<MobileElement> mobileElements)5public IssueFilter severity(List<Severity> severities)6public IssueFilter severity(Severity... severity)7public IssueFilter issueType(List<IssueType> issueTypes)8public IssueFilter issueType(IssueType... issueType)
RuleConfig
Default constructor:
1public RuleConfig(IssueType name, boolean enabled)
IssueType
- the IssueType on which this rule config should be applied
enabled
- allows to enable/disable specified IssueType. By default enabled = true
Tutorials
Generating a comprehensive a11y report for your application
In this tutorial, we will enhance our existing Appium UI test with the Evinced Appium SDK in order to check our application for accessibility issues. In order to get started you will need the following:
- All of the prerequisites for the Evinced Appium SDK should be met
- Evinced Appium SDK should be installed in your project
Preface - existing UI test overview
Let’s consider the following basic UI test as our starting point.
1import io.appium.java_client.MobileBy;2import io.appium.java_client.ios.IOSDriver;3import io.appium.java_client.ios.IOSElement;4import io.appium.java_client.remote.MobileCapabilityType;56import java.io.File;7import java.io.IOException;8import java.net.MalformedURLException;9import java.net.URL;10import org.openqa.selenium.remote.DesiredCapabilities;11import org.junit.AfterClass;12import org.junit.BeforeClass;13import org.junit.Test;14import org.openqa.selenium.support.ui.ExpectedConditions;15import org.openqa.selenium.support.ui.WebDriverWait;1617public class SwiftRadioTest {1819 public static URL url;20 public static DesiredCapabilities capabilities;21 public static IOSDriver<IOSElement> driver;2223 @BeforeClass24 public static void setupAppiumDriver() throws MalformedURLException, IOException {25 final String URL_STRING = "http://127.0.0.1:4723/wd/hub";26 url = new URL(URL_STRING);2728 final Class clazz = EvincedA11yValidatorTest.class;29 final File file = new File(clazz.getResource("/SwiftRadio.app").getFile());3031 capabilities = new DesiredCapabilities();32 capabilities.setCapability(MobileCapabilityType.AUTOMATION_NAME, "XCUITest");33 capabilities.setCapability("useNewWDA", false);34 capabilities.setCapability(MobileCapabilityType.NO_RESET, true);35 capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, "iPhone Simulator");36 capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, "iOS");37 capabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION, "12.0");38 capabilities39 .setCapability(MobileCapabilityType.APP,40 file.getAbsolutePath());4142 driver = new IOSDriver<IOSElement>(url, capabilities);43 }4445 @AfterClass46 public static void tearDown() {47 driver.quit();48 }4950 @Test public void testStationsScreen() {51 IOSElement firstStationTableCell = (IOSElement) new WebDriverWait(driver, 30)52 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("StationTableViewCell")));5354 firstStationTableCell.click();5556 IOSElement playButton = (IOSElement) new WebDriverWait(driver, 30)57 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("XCUIElementTypeButton")));58 }59}
We wrote this test for an application called Swift Radio. It is a small open-source app for listening to various radio stations and podcasts.
The purpose of these tests is to check the functionality of the main application screen and ensure a user can successfully navigate to their choice of the radio station. For now, this test is only concerned with the functional testing of the app. However, with the help of the Evinced Appium SDK, we can also check it for accessibility issues along the way. Let’s go through this process with the following step-by-step instructions.
Step #1 - Initialize the Evinced Appium SDK
Before making any assertions against our app, we need to initialize EvincedAppiumSdk
object. This object is used primarily as an entry point to all of the accessibility scanning features. Since we are going to use it primarily within our tests, the best place for its initialization will be our setupAppiumDriver
method which gets executed first.
1import io.appium.java_client.MobileBy;2import io.appium.java_client.ios.IOSDriver;3import io.appium.java_client.ios.IOSElement;4import io.appium.java_client.remote.MobileCapabilityType;56import java.io.File;7import java.io.IOException;8import java.net.MalformedURLException;9import java.net.URL;10import org.openqa.selenium.remote.DesiredCapabilities;11import org.junit.AfterClass;12import org.junit.BeforeClass;13import org.junit.Test;14import org.openqa.selenium.support.ui.ExpectedConditions;15import org.openqa.selenium.support.ui.WebDriverWait;1617public class SwiftRadioTest {1819 public static URL url;20 public static DesiredCapabilities capabilities;21 public static IOSDriver<IOSElement> driver;2223 public static EvincedAppiumSdk evincedAppiumSdk;2425 @BeforeClass26 public static void setupAppiumDriver() throws MalformedURLException, IOException {27 final String URL_STRING = "http://127.0.0.1:4723/wd/hub";28 url = new URL(URL_STRING);2930 final Class clazz = EvincedA11yValidatorTest.class;31 final File file = new File(clazz.getResource("/SwiftRadio.app").getFile());3233 capabilities = new DesiredCapabilities();34 capabilities.setCapability(MobileCapabilityType.AUTOMATION_NAME, "XCUITest");35 capabilities.setCapability("useNewWDA", false);36 capabilities.setCapability(MobileCapabilityType.NO_RESET, true);37 capabilities.setCapability(MobileCapabilityType.DEVICE_NAME, "iPhone Simulator");38 capabilities.setCapability(MobileCapabilityType.PLATFORM_NAME, "iOS");39 capabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION, "12.0");40 capabilities41 .setCapability(MobileCapabilityType.APP,42 file.getAbsolutePath());4344 driver = new IOSDriver<IOSElement>(url, capabilities);45 evincedAppiumSdk = new EvincedAppiumSdk(driver);46 evincedAppiumSdk.setupCredentials(<your service account ID>, <your API key>);47 }4849 @AfterClass50 public static void tearDown() {51 driver.quit();52 }5354 @Test55 public void testStationsScreen() {56 IOSElement firstStationTableCell = (IOSElement) new WebDriverWait(driver, 30)57 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("StationTableViewCell")));5859 firstStationTableCell.click();6061 IOSElement playButton = (IOSElement) new WebDriverWait(driver, 30)62 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("XCUIElementTypeButton")));63 }64}
The EvincedAppiumSdk
requires only an instance of the Appium driver. There are additional configuration options that can be included as well. More information can be found in the API section of this documentation.
Step #2 - Identify which application states you want to check for a11y issues
Once the SDK is set up, let’s try to carefully consider which states of our application are actually worth checking for accessibility issues. Let’s take a look at out UI test once again:
1@Test2public void testStationsScreen() {3 IOSElement firstStationTableCell = (IOSElement) new WebDriverWait(driver, 30)4 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("StationTableViewCell")));5 // We have reached the main application screen with list of stations. It may have some accessibility issues!67 firstStationTableCell.click();89 IOSElement playButton = (IOSElement) new WebDriverWait(driver, 30)10 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("XCUIElementTypeButton")));11 // Here we have reached yet another screen - a concrete station view. It is probably worth checking for accessibility issues as well.12}
So far we have identified two places in our UI test where we may need to perform accessibility assertion. Each place represents some distinct application state and each of these states may contain certain accessibility issues. So, let’s put our checks into place.
Step #3 - Setup analyzing breakpoints
The only thing we need to do now is to simply add evincedAppiumSdk.analyze()
calls in places within our test which we previously identified as “interesting” ones in terms of accessibility. Here is how it should look:
1@Test2public void testStationsScreen() {3 IOSElement firstStationTableCell = (IOSElement) new WebDriverWait(driver, 30)4 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("StationTableViewCell")));5 // We have reached the main application screen with list of stations. It may have some accessibility issues!6 evincedAppiumSdk.analyze();78 firstStationTableCell.click();910 IOSElement playButton = (IOSElement) new WebDriverWait(driver, 30)11 .until(ExpectedConditions.visibilityOfElementLocated(MobileBy.className("XCUIElementTypeButton")));12 // Here we have reached yet another screen - a concrete station view. It is probably worth checking for accessibility issues as well.13 evincedAppiumSdk.analyze();14}
The analyze
method collects an accessibility snapshot of the current application state and puts it into internal in-memory storage. This snapshot contains a lot of information including application screenshot, accessibility labels and dimensions of UI elements.
Now that we have gathered the information we need we are finally ready to actually find our whether our app is accessible or not.
Step #4 - Assert your screens at the end of your test suite
As our test was executed we collected a lot of accessibility snapshots via the evincedAppiumSdk.analyze()
calls. We can now perform accessibility assertions at the end of our test suite. Referring back again to our initial UI test the best place for this assertion will be the method that gets invoked last - tearDown
.
1@AfterClass2public static void tearDown() {3 List<Report> reports = evincedAppiumSdk.reportStored();4 driver.quit();5}
To generate the actual object representation of your accessibility report simply call the reportStored
method. This way Evinced Appium SDK will check all of the previously collected accessibility snapshots for having some accessibility issues.
From this moment you are free to assert the given list of Report
objects in any way you want (feel free to explore the corresponding chapter in the “API“ section).
For the sake of simplicity of this tutorial let’s simply assume that our application is accessible as long as it has no accessibility issues found. Thus, if we have at least one accessibility issue detected - we want our tests to be failed. Let’s add the corresponding assertion:
1@AfterClass2public static void tearDown() {3 List<Report> reports = evincedAppiumSdk.reportStored();4 List<Report> reportsWithIssues = reports.stream() // Filter out only reports which contain some issues5 .filter(report -> report.hasIssues())6 .collect(Collectors.toList());78 assertEquals(reportsWithIssues.size(), 0); // Making the actual assertion9 driver.quit();10}
You are now set to run the test and ensure the accessibility of your application! So, go ahead and run it via your IDE or any other tooling you use for Java development.
Step #5 - Read the results
At this point, you will likely see that your test has failed, which is totally fine. Swift Radio certainly has some accessibility flaws, and the fact that we have caught them simply means that Evinced Appium SDK is working!
The only thing which is missing here is the actual list of accessibility issues with would like to look at. First of all, let’s pay a bit of attention to the terminal output of our tests. You should see your tests failed due to AssertionError
with the following description:
1Number of a11y issues found. Please refer to the html report located at: /Users/johndoe/dev/swift-radio/evinced-reports/evinced-report-a6a47bed-d7b5-467e-a5c2-9e6ac4c1f247.html2java.lang.AssertionError: Number of a11y issues found. Please refer to the html report located at: /Users/johndoe/dev/swift-radio/evinced-reports/evinced-report-a6a47bed-d7b5-467e-a5c2-9e6ac4c1f247.html3 at com.evinced.a11y.validator.appium.java.core.EvincedA11yValidator.assertTreesHaveA11yIssues(EvincedA11yValidator.java:72)4 //... omited for brevity
According to this output, we have HTML and JSON files located right in the root of our project, inside the folder named evinced-reports
. For more information regarding the HTML and JSON reports as well as the Report
object itself, please see our detailed Mobile Reports page.
Step #6 - Make your app more accessible
Congratulations! Over the course of this 5-step tutorial we have managed to integrate accessibility tests right into our existing UI tests with the help of Evinced Appium SDK and moreover, we have even found several issues in the app.
This is the perfect time for us to carefully investigate all of the issues detected (with help of the knowledge base) and put our JSON files into static storage for later use. With the history of all of the results, we can achieve a lot of exciting things including tracking our progress fixing all of the accessibility issues over time, detecting regressions, and gathering additional metrics.
Feel free to continue your journey with the Evinced Appium SDK by browsing the API section.
FAQ
1. Do you have support for languages other than Java?
Not at this moment, unfortunately. We are planning to expand our tooling coverage by writing dedicated libraries for other popular UI testing languages such as JavaScript and Python.
2. Can I configure which validations to run?
Currently, all of the available validations are run. We have not seen any drastic impact on performance caused by any particular validation. In the future, we are planning to introduce more sophisticated capabilities for filtering results by different properties (e.g. severity or problem type) or even by using your own custom predicate.
3. Where can I find the release changelogs?
If you are using JetBrains IDE (Android Studio or IntelliJ IDEA):
Open ‘Project' tab (⌘1
) → ‘Project’ view → 'External Libraries’ section → Gradle: com.evinced.appium-sdk:<version>
→ CHANGELOG.md