XCUITest SDK

The Evinced XCUI SDK integrates with new or existing UI tests to automatically detect accessibility issues. With the addition of a few lines of code, you can analyze your entire application to understand how it can become more accessible. At the conclusion of the tests, actionable HTML and JSON reports are generated to track issues in any reporting tool.

Supported versions / frameworks

The XCUI SDK supports UI tests on simulators or real devices on iOS version 12 and above. The Swift language is targeted, but the basic functionality is available in Objective C as well.

Get Started

Setup

  1. In order to use any of the Evinced Mobile SDKs you first need to create an Evinced account. Once your account is created, you will get your apiKey and a matching serviceAccountId. Initialize the SDK with these values it will validate access with Evinced servers when you run your tests. If an outbound internet connection is not available in your running environment - contact us at support@evinced.com to get an offline APIKey.
  2. Include UI testing as part of your project using the standard XCUI SDK. Add the EvincedXCUISDK using either CocoaPods or Swift Package Manager to your UI tests target.

CocoaPods

  1. Make sure you have CocoaPods installed and set for your app workspace.
  2. Add the following line to your UI Test target in your Podfile: pod 'EvincedXCUISDK'
  3. Run pod install.

Swift Package Manager

  1. Add https://github.com/GetEvinced/public-ios-xcuisdk repository as a new Swift package in the Xcode GUI.
  2. Select your UI tests target EvincedXCUISDK to link with.

Your first test

Initialize the EvincedXCUISDK

Add the import statement to the beginning of your test file.

1import EvincedXCUISDK

In order to setup the EvincedXCUISDK you will need to add the following code to the setup and teardown methods in your tests:

1override func setUpWithError() throws {
2 EvincedEngine.testCase = self
3 try EvincedEngine.setupCredentials(serviceAccountId: "your_service_account_id",
4 apiKey: "your_api_key")
5 // Other setup code..
6}

For offline access use EvincedEngine.setupOfflineCredentials(serviceAccountId:, accessToken:)

1override func tearDownWithError() throws {
2 // This method will validate the stored accessibility data and generate an HTML report without failing the test.
3 // Use flag `assert: true` if you prefert to fail the test if an accessibility issue is found.
4 try EvincedEngine.reportStored(assert: false)
5 // Other tear down code...
6}

Add accessibility scans to your test

Within your test, call the evAnalyze() method at any point at which an accessibility scan would capture new areas of the application. Examples would be at the beginning of the test to capture the initial state, and then after a tap or swipe interaction opens a menu or opens a new view. Mark your test method with throws for error handling.

1func testMainStationsView() throws {
2 // Use recording to get started writing UI tests.
3 // Use XCTAssert and related functions to verify your tests produce the correct results.
4 try app.evAnalyze()
5
6 assertStationsPresent()
7 try app.evAnalyze()
8
9 hamburgerMenu.tap()
10 assertHamburgerContent()
11 try app.evAnalyze()
12
13 app.buttons["About"].tap()
14 assertAboutContent()
15 try app.evAnalyze()
16
17 // Other test code...
18}

Now we are ready to run our test suite(s)!

Report

After the test finishes, accessibility HTML and JSON reports are generated with EvincedEngine.reportStored(assert:) call in tearDownWithError() method and stored as test attachments. You can save them via the Xcode GUI or simply run the xcrun xcresulttool ... CLI command.

Another convenient tool to extract test attachments is ChargePoint/xcparse: Command line tool & Swift framework for parsing Xcode 11+ xcresult

For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.

API

EvincedEngine

@objc public class EvincedEngine : NSObject

Main EvincedEngine class

Example usage:

1override func setUpWithError() throws {
2 EvincedEngine.testCase = self
3 // Other setup code..
4}

Public Interface

isLoggingEnabled

Enable or disable logging. Default is true.

@objc static var isLoggingEnabled: Bool { get set }

Example usage:

1override func setUpWithError() throws {
2 EvincedEngine.testCase = self
3 EvincedEngine.isLoggingEnabled = false
4 // Other setup code..
5}

testCase

Test case for adding accessibility report attachment.

@objc static weak var testCase: XCTestCase? { get set }

Example usage:

1override func setUpWithError() throws {
2 EvincedEngine.testCase = self
3 EvincedEngine.isLoggingEnabled = false
4 // Other setup code..
5}

setupCredentials()

Setup licensing credentials for online validation. The method requires a network request.

static func setupCredentials(serviceAccountId: String, apiKey: String) throws

Example usage:

1override func setUpWithError() throws {
2 // Put setup code here. This method is called before the invocation of each test method in the class.
3 try EvincedEngine.setupCredentials(serviceAccountId: "your_service_account_id",
4 apiKey: "your_api_key")
5}

Parameters

ParameterWhere you can find it
serviceAccountId:Your service account ID. This can be found by logging into the Evinced web portal.
apiKey:Your API key. This can be found by logging into the Evinced web portal.

setupOfflineCredentials()

Setup licensing credentials. Method requires no network requests.

static func setupOfflineCredentials(serviceAccountId: String, accessToken: String) -> Bool

Example usage:

1override func setUpWithError() throws {
2 // Put setup code here. This method is called before the invocation of each test method in the class.
3
4
5 // In UI tests it is usually best to stop immediately when a failure occurs.
6 continueAfterFailure = false
7
8 // In UI tests it’s important to set the initial state - such as interface orientation - required for your tests before they run. The setUp method is a good place to do this.
9 EvincedEngine.setupOfflineCredentials(serviceAccountId: "your_service_account_id",
10 accessToken: "your_access_token")
11
12 EvincedEngine.testCase = self
13}

Parameters

ParameterWhere you can find it
serviceAccountId:Your service account ID. You could obtain it on the web portal.
accessToken:Your access token. You could obtain it on the web portal.

Return value:
Result of license validation (true if successful).

report()

Generates the accessibility report for the current state and saves it as test attachment.

func report(for application: XCUIApplication?, assert: Bool = false, file: StaticString = #filePath, line: UInt = #line) throws -> Report

Example usage:

1func testMainStationsView() {
2 hamburgerMenu.tap()
3 assertHamburgerContent()
4 try EvincedEngine.report(assert: true)
5}

Parameters

ParameterWhere you can find it
application:Tested application. If nil currently launched application would be tested. Default is nil.
assert:This parameter determines whether the SDK should fail the test if an accessibility issue was found. Default is false.
file:The file in which the failure occurred. Defaults to the file name of the test case in which this function was called. Usually should not be set manually.
line:The line number on which failure occurred. Defaults to the line number on which this function was called. Usually should not be set manually.

Return value:

Evinced accessibility Report object. For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page. If you’ve run your test using the Xcode GUI, you can access the attached reports as described here. If you run your tests in a CI pipeline, you can copy the reports into any directory on your CI machine using xcparse.

analyze()

Stores the accessibility data of an application for further validation. When no application is set, the currently active app is analyzed.

func analyze(_ application: XCUIApplication? = nil) throws

Example usage:

1func testMainStationsView() throws {
2 hamburgerMenu.tap()
3 assertHamburgerContent()
4 try EvincedEngine.analyze()
5}

Parameters

ParameterWhere you can find it
application:Tested application. If nil currently launched application would be tested. Default is nil.

reportStored()

Reports all the stored accessibility data at once. Clears all the stored test data, creates attachment with HTML test report.

static func reportStored(assert: Bool = false, file: StaticString = #filePath, line: UInt = #line) throws

Example usage:

1override func tearDownWithError() throws {
2 // Put teardown code here. This method is called after the invocation of each test method in the class.
3 // By passing true the test will be fail if an accessibility issue is found
4 try EvincedEngine.reportStored()
5 super.tearDown()
6}

Parameters

ParameterWhere you can find it
assert:This parameter tells if the SDK should fail the test if accessibility issue was found. Default is false.
file:The file in which the failure occurred. Defaults to the file name of the test case in which this function was called. Usually should not be set manually.
line:The line number on which failure occurred. Defaults to the line number on which this function was called. Usually should not be set manually.

clearStored()

Clear all the stored accessibility test data.

@objc static func clearStored()

Example usage:

1override func tearDown() {
2 // Put teardown code here. This method is called after the invocation of each test method in the class.
3 EvincedEngine.clearStored()
4 super.tearDown()
5}

Extension for XCUIApplication

evReport()

Generates the accessibility report for the current state and saves it as a test attachment.

func evReport(assert: default false, file: StaticString = #filePath, line: UInt = #line) throws -> Report

Example usage:

1func testMainStationsView() throws {
2 hamburgerMenu.tap()
3 assertHamburgerContent()
4 try app.evReport()
5}

Parameters

ParameterWhere you can find it
assert:This parameter tells if the SDK should fail the test if any accessibility issue was found. Default is false.
file:The file in which failure occurred. Defaults to the file name of the test case in which this function was called. Usually should not be set manually.
line:The line number on which failure occurred. Defaults to the line number on which this function was called. Usually should not be set manually.

Return value:

Evinced accessibility Report object. For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page. If you’ve run your test using the Xcode GUI, you can access the attached reports as described here. If you run your tests in a CI pipeline, you can copy the reports into any directory on your CI machine using xcparse.

evAnalyze()

Stores the accessibility data of an application for further validation.

func evAnalyze() throws

Example usage:

1func testMainStationsView() throws {
2 hamburgerMenu.tap()
3 assertHamburgerContent()
4 try app.evAnalyze()
5}

Tutorials

Generating a comprehensive accessibility report for your iOS application

In this tutorial, we will enhance our existing XCTest UI test with the Evinced XCUI SDK in order to check our application for accessibility issues. In order to get started you will need the following:

  1. All of the prerequisites for XCUI SDK should be met;
  2. UI tests target should exist in your project

Preface - Existing UI test overview

Let’s consider the following basic UI test from the open-source Swift Radio app as our starting point.

1// SwiftRadioUITests.swift
2// SwiftRadioUITests
3// Created by Jonah Stiennon on 12/3/15.
4// Copyright © 2015 matthewfecher.com. All rights reserved.
5
6import XCTest
7
8class SwiftRadioUITests: XCTestCase {
9
10 let app = XCUIApplication()
11 let stations = XCUIApplication().cells
12 let hamburgerMenu = XCUIApplication().navigationBars["Swift Radio"].buttons["icon hamburger"]
13 let pauseButton = XCUIApplication().buttons["btn pause"]
14 let playButton = XCUIApplication().buttons["btn play"]
15 let volume = XCUIApplication().sliders.element(boundBy: 0)
16
17 override func setUp() {
18 super.setUp()
19 continueAfterFailure = false
20 // UI tests must launch the application that they test. Doing this in setup will make sure it happens for each test method.
21 XCUIApplication().launch()
22 // wait for the main view to load
23 self.expectation(
24 for: NSPredicate(format: "self.count > 0"),
25 evaluatedWith: stations,
26 handler: nil)
27 self.waitForExpectations(timeout: 10.0, handler: nil)
28 }
29
30 override func tearDown() {
31 super.tearDown()
32 }
33
34 func assertStationsPresent() {
35 let numStations:UInt = 5
36 XCTAssertEqual(stations.count, Int(numStations))
37 let texts = stations.staticTexts.count
38 XCTAssertEqual(texts, Int(numStations * 2))
39 }
40
41 func assertHamburgerContent() {
42 XCTAssertTrue(app.staticTexts["Created by: Matthew Fecher"].exists)
43 }
44
45 func assertAboutContent() {
46 XCTAssertTrue(app.buttons["email me"].exists)
47 XCTAssertTrue(app.buttons["matthewfecher.com"].exists)
48 }
49
50 func assertPaused() {
51 XCTAssertFalse(pauseButton.isEnabled)
52 XCTAssertTrue(playButton.isEnabled)
53 XCTAssertTrue(app.staticTexts["Station Paused..."].exists);
54 }
55
56 func assertPlaying() {
57 XCTAssertTrue(pauseButton.isEnabled)
58 XCTAssertFalse(playButton.isEnabled)
59 XCTAssertFalse(app.staticTexts["Station Paused..."].exists);
60 }
61
62 func assertStationOnMenu(_ stationName:String) {
63 let button = app.buttons["nowPlaying"];
64 XCTAssertTrue(button.label.contains(stationName))
65 }
66
67 func assertStationInfo() {
68 let textView = app.textViews.element(boundBy: 0)
69 if let value = textView.value {
70 XCTAssertGreaterThan((value as AnyObject).length, 10)
71 } else {
72 XCTAssertTrue(false)
73 }
74 }
75
76 func waitForStationToLoad() {
77 self.expectation(
78 for: NSPredicate(format: "exists == 0"),
79 evaluatedWith: app.staticTexts["Loading Station..."],
80 handler: nil)
81 self.waitForExpectations(timeout: 25.0, handler: nil)
82
83 }
84
85 func testMainStationsView() {
86 // Use recording to get started writing UI tests.
87 // Use XCTAssert and related functions to verify your tests produce the correct results.
88
89 assertStationsPresent()
90
91 hamburgerMenu.tap()
92 assertHamburgerContent()
93 app.buttons["About"].tap()
94 assertAboutContent()
95 app.buttons["Okay"].tap()
96 app.buttons["btn close"].tap()
97 assertStationsPresent()
98
99 let firstStation = stations.element(boundBy: 0)
100 let stationName:String = firstStation.children(matching: .staticText).element(boundBy: 0).label
101 assertStationOnMenu("Choose")
102 firstStation.tap()
103 waitForStationToLoad();
104
105 pauseButton.tap()
106 assertPaused()
107 playButton.tap()
108 assertPlaying()
109 app.navigationBars["Sub Pop Radio"].buttons["Back"].tap()
110 assertStationOnMenu(stationName)
111 app.navigationBars["Swift Radio"].buttons["btn nowPlaying"].tap()
112 waitForStationToLoad()
113 volume.adjust(toNormalizedSliderPosition: 0.2)
114 volume.adjust(toNormalizedSliderPosition: 0.8)
115 volume.adjust(toNormalizedSliderPosition: 0.5)
116 app.buttons["More Info"].tap()
117 assertStationInfo()
118 app.buttons["Okay"].tap()
119 app.buttons["logo"].tap()
120 assertAboutContent()
121 app.buttons["Okay"].tap()
122 }
123}

The purpose of this test is to check the full user flow of the application. For now, this test is only concerned with the functional testing of the app. However, with the help of Evinced XCUI SDK, we can also check it for accessibility issues along the way. Let’s go through this process with the following step-by-step instructions.

Step #1 - Initialize the Evinced XCUI SDK

Please, make sure EvincedXCUISDK module is added to UI tests target via CocoaPods or Swift Package Manager and properly imported:

1// SwiftRadioUITests.swift
2// SwiftRadioUITests
3// Created by Jonah Stiennon on 12/3/15.
4// Copyright © 2015 matthewfecher.com. All rights reserved.
5
6import XCTest
7import EvincedXCUISDK
8
9// Other stuff...

Before making any assertions against our app, we need to set up the EvincedEngine. This class is used primarily as an entry point to all of the accessibility scanning features. Since we are going to use it primarily within our test, the best place for its initialization will be our func setUp() method of the test case. Because of most of methods of EvincedEngine may throw exceptions we need to adopt our func setUp() to func setUpWithError() throws

1override func setUpWithError() throws {
2 // Put setup code here. This method is called before the invocation of each test method in the class.
3 continueAfterFailure = false
4 // UI tests must launch the application that they test. Doing this in setup will make sure it happens for each test method.
5 XCUIApplication().launch()
6
7 // wait for the main view to load
8 self.expectation(
9 for: NSPredicate(format: "self.count > 0"),
10 evaluatedWith: stations,
11 handler: nil)
12 self.waitForExpectations(timeout: 10.0, handler: nil)
13
14 // Use your real keys here
15 try EvincedEngine.setupCredentials(serviceAccountId: "your_service_account_id",
16 apiKey: "your_api_key")
17 EvincedEngine.testCase = self
18}

The only setup needed for the EvincedEngine is to set licensing credentials and the testCase. The latest is needed to save the HTML and JSON reports.

Step #2 - Identify which application states you want to check for accessibility issues

Once the SDK is set up, let’s try to carefully consider which states of our application are actually worth checking for accessibility issues. Same adaptation of catching exceptions is need. Let’s take a look at our UI test once again:

1func testMainStationsView() throws {
2 // Use recording to get started writing UI tests.
3 // Use XCTAssert and related functions to verify your tests produce the correct results.
4
5 assertStationsPresent()
6
7 hamburgerMenu.tap()
8 assertHamburgerContent()
9 app.buttons["About"].tap()
10 assertAboutContent()
11 app.buttons["Okay"].tap()
12 app.buttons["btn close"].tap()
13 assertStationsPresent()
14
15 let firstStation = stations.element(boundBy: 0)
16 let stationName:String = firstStation.children(matching: .staticText).element(boundBy: 0).label
17 assertStationOnMenu("Choose")
18 firstStation.tap()
19 waitForStationToLoad();
20
21 pauseButton.tap()
22 assertPaused()
23 playButton.tap()
24 assertPlaying()
25 app.navigationBars["Sub Pop Radio"].buttons["Back"].tap()
26 assertStationOnMenu(stationName)
27 app.navigationBars["Swift Radio"].buttons["btn nowPlaying"].tap()
28 waitForStationToLoad()
29 volume.adjust(toNormalizedSliderPosition: 0.2)
30 volume.adjust(toNormalizedSliderPosition: 0.8)
31 volume.adjust(toNormalizedSliderPosition: 0.5)
32 app.buttons["More Info"].tap()
33 assertStationInfo()
34 app.buttons["Okay"].tap()
35 app.buttons["logo"].tap()
36 assertAboutContent()
37 app.buttons["Okay"].tap()
38}

We have identified many places in our UI test where we would like to perform accessibility scans. A good place to add a check is after a user interaction like a tap, swipe, etc. Each interaction may reveal more of the application to check for issues. So, let’s put our checks into place.

Step #3 - Setup accessibility analyzing points

The only thing we need to do now is to simply add EvincedEngine.analyze() calls to the places in the test we identified above. Besides the EvincedEngine method, we could also use the analyze() method from the public XCUIApplication extension as they are identical. Because we have reference to the app object, we prefer the second variant in this example. Additionally, we marked our test as a throwing function for convenience. Here is what it should look like:

1func testMainStationsView() throws {
2 // Use recording to get started writing UI tests.
3 // Use XCTAssert and related functions to verify your tests produce the correct results.
4 try app.evAnalyze()
5
6 assertStationsPresent()
7 try app.evAnalyze()
8
9 hamburgerMenu.tap()
10 assertHamburgerContent()
11 try app.evAnalyze()
12
13 app.buttons["About"].tap()
14 assertAboutContent()
15 try app.evAnalyze()
16
17 app.buttons["Okay"].tap()
18 try app.evAnalyze()
19
20 app.buttons["btn close"].tap()
21 assertStationsPresent()
22 try app.evAnalyze()
23
24 let firstStation = stations.element(boundBy: 0)
25 let stationName = firstStation.children(matching: .staticText).element(boundBy: 1).label
26 assertStationOnMenu("Choose")
27 firstStation.tap()
28 waitForStationToLoad()
29 try app.evAnalyze()
30
31 playButton.tap()
32 assertPaused()
33 try app.evAnalyze()
34
35 playButton.tap()
36 assertPlaying()
37 try app.evAnalyze()
38
39 app.navigationBars[stationName].buttons["Back"].tap()
40 assertStationOnMenu(stationName)
41 app.navigationBars["Swift Radio"].buttons.element(boundBy: 1).tap()
42 waitForStationToLoad()
43 try app.evAnalyze()
44
45 volume.adjust(toNormalizedSliderPosition: 0.2)
46 volume.adjust(toNormalizedSliderPosition: 0.8)
47 volume.adjust(toNormalizedSliderPosition: 0.5)
48 try app.evAnalyze()
49
50 app.buttons["More Info"].tap()
51 assertStationInfo()
52 try app.evAnalyze()
53
54 app.buttons["Okay"].tap()
55 try app.evAnalyze()
56
57 app.buttons["logo"].tap()
58 assertAboutContent()
59 try app.evAnalyze()
60
61 app.buttons["Okay"].tap()
62 try app.evAnalyze()
63}

The evAnalyze() method collects an accessibility snapshot of the current application state and puts it into internal in-memory storage. This snapshot contains a lot of information including an application screenshot, accessibility labels, and dimensions of UI elements.

Now that we have gathered the information we need we are ready to actually find out whether our app is accessible or not.

Step #4 - Validate your screens at the end of your test

As our test was executed we collected a lot of accessibility snapshots via the evAnalyze() calls. We can now perform accessibility validations or assertions at the end of our test suite. We can use reportStored method with assert: flag. The difference is that assert: true will fail the test if any accessibility issue is present and generate HTML and JSON report as a test attachment while the assert: false flag will generate the reports but not fail the test. Referring back again to our initial UI test the best place for this assertion will be the method that is invoked last - tearDown(). Let’s adapt it too for exception tearDown() change for tearDownWithError() throws

1override func tearDownWithError() throws {
2 // Put teardown code here. This method is called after the invocation of each test method in the class.
3 try EvincedEngine.reportStored()
4}

The XCUI SDK will check all of the previously collected accessibility snapshots for accessibility issues and generate the attachments if EvincedEngine.testcase is set up properly.

If you prefer to fail the test if there is any issue present, use:

1override func tearDownWithError() throws {
2 // Put teardown code here. This method is called after the invocation of each test method in the class.
3 try EvincedEngine.reportStored(assert: true)
4}

You are now set to run the test and ensure the accessibility of your application!

Step #5 - Run your UI test

Run your UI test from the Xcode GUI or CI pipeline. If anything is set up properly you’ll see a console log message about saving your reports:

1 t = 25.92s Tear Down
2 t = 26.29s Added attachment of type 'public.json'
3 t = 26.29s Added attachment of type 'public.html'

Step #6 - Access the reports

Now it's time to locate a review the reports. If you’ve run your test using the Xcode GUI, you can access the attached reports as described here. If you run your tests in a CI pipeline, you can copy the reports into any directory on your CI machine using xcparse. For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.

FAQ

1. What is the minimum iOS version supported?

Currently, we support iOS starting from iOS 12, which means we support both UIKit and SwiftUI apps.

2. Is there any difference in the number of accessibility validations Evinced will find when testing on a simulator vs. a real physical device?

Our solution is tested both on real devices and simulators and works identically at both.

3. Do I need to add anything (SDK etc.) to my app during the build process to test for accessibility violations?

No! You don’t need to add anything to your app target itself. You could even test apps download from the App Store on real devices.