XCUI SDK

The Evinced XCUI SDK integrates with new or existing UI tests to automatically detect accessibility issues. With the addition of a few lines of code, you can analyze your entire application to understand how it can become more accessible. At the conclusion of the tests, actionable HTML and JSON reports are generated to track issues in any reporting tool.

Supported versions / frameworks

The XCUI SDK supports UI tests on simulators or real devices on iOS version 12 and above. The Swift language is targeted, but the basic functionality is available in Objective C as well.

Get Started

Setup

1. In order to use any of the Evinced Mobile SDKs you first need to create an Evinced account. Once your account is created, you will get your apiKey and a matching serviceAccountId. Initialize the SDK with these values it will validate access with Evinced servers when you run your tests. If an outbound internet connection is not available in your running environment - contact us at support@evinced.com to get an offline APIKey.
2. Include UI testing as part of your project using the standard XCUI SDK. Add the EvincedXCUISDK using either CocoaPods or Swift Package Manager to your UI tests target.

CocoaPods

1. Make sure you have CocoaPods installed and set for your app workspace.
2. Add the following line to your UI Test target in your Podfile: pod 'EvincedXCUISDK'
3. Run pod install.

Swift Package Manager

1. Add https://github.com/GetEvinced/public-ios-xcuisdk repository as a new Swift package in the Xcode GUI.
2. Select your UI tests target EvincedXCUISDK to link with.

Your first test

Initialize the EvincedXCUISDK

Add the import statement to the beginning of your test file.

1import EvincedXCUISDK

In order to setup the EvincedXCUISDK you will need to add the following code to the setup and teardown methods in your tests:

1override func setUpWithError() throws {
2    EvincedEngine.testCase = self
3    try EvincedEngine.setupCredentials(serviceAccountId: "your_service_account_id",
4                                       apiKey: "your_api_key")
5    // Other setup code..
6}

For offline access use EvincedEngine.setupOfflineCredentials(serviceAccountId:, accessToken:)

1override func tearDownWithError() throws {
2    // This method will validate the stored accessibility data and generate an HTML report without failing the test.
3    // Use flag `assert: true` if you prefert to fail the test if an accessibility issue is found.
4    try EvincedEngine.reportStored(assert: false)
5    // Other tear down code...
6}

Add accessibility scans to your test

Within your test, call the evAnalyze() method at any point at which an accessibility scan would capture new areas of the application. Examples would be at the beginning of the test to capture the initial state, and then after a tap or swipe interaction opens a menu or opens a new view. Mark your test method with throws for error handling.

1func testMainStationsView() throws {
2    // Use recording to get started writing UI tests.
3    // Use XCTAssert and related functions to verify your tests produce the correct results.
4    try app.evAnalyze()
5
6    assertStationsPresent()
7    try app.evAnalyze()
8
9    hamburgerMenu.tap()
10    assertHamburgerContent()
11    try app.evAnalyze()
12
13    app.buttons["About"].tap()
14    assertAboutContent()
15    try app.evAnalyze()
16
17    // Other test code...
18}

Now we are ready to run our test suite(s)!

Report

After the test finishes, accessibility HTML and JSON reports are generated with EvincedEngine.reportStored(assert:) call in tearDownWithError() method and stored as test attachments. You can save them via the Xcode GUI or simply run the xcrun xcresulttool ... CLI command.

Another convenient tool to extract test attachments is ChargePoint/xcparse: Command line tool & Swift framework for parsing Xcode 11+ xcresult

For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.

And for more information about mentioned tools (xcparse, xcresulttool) for extract, parse, and analyze the data contained in .xcresult bundles generated by Xcode, please see Xcode Result Parsers section.

Continuous Mode

Continuous mode allows continual scanning of the application without the need to insert individual scan calls with in the test code. Simply specify the start and end points using the following methods EvincedEngine.startAnalyze() and EvincedEngine.stopAnalyze(). Then Evinced engines will automatically scan the application after each XCUI command. All issues detected during the tests will be added to the HTML and JSON reports automatically generated with the stopAnalyze() method is called.

To initialize the Evinced engines simply use the EvincedEngine.startAnalyze() method and finish it use EvincedEngine.stopAnalyze(). You can use it within a single test method or scan an entire class using override methods setUp and tearDown.

func startAnalyze(config: EvincedConfig? = nil)

Parameters

config: The config for setting report parameters. Default is nil.

Example usage within a single test:

Example No. 1 - Continuous mode usage without EvincedConfig

1    func testContinuousModeAnalyze() throws {
2        EvincedEngine.startAnalyze()
3
4        // XCUI commands
5        // ...
6
7        EvincedEngine.stopAnalyze()
8    }

Example No. 2 - Continuous mode usage with EvincedConfig

1    func testContinuousModeAnalyze() throws {
2        let app = XCUIApplication()
3        let filters = [IssueFilter(elements: .query(app.buttons))]
4        let config = EvincedConfig(includeFilters: filters)
5        EvincedEngine.startAnalyze(config: config)
6
7        // XCUI commands
8        // ...
9
10        EvincedEngine.stopAnalyze()
11    }

Example usage in the initial class:

1import XCTest
2import EvincedXCUISDK
3
4class Test_App_ContinuousMode_UITests: XCTestCase {
5
6    override func setUpWithError() throws {
7        EvincedEngine.setupOfflineCredentials(serviceAccountId: "your_service_account_id",
8                                            accessToken: "your_access_token")
9        EvincedEngine.testCase = self
10        EvincedEngine.startAnalyze()
11    }
12
13    override func tearDownWithError() throws {
14        EvincedEngine.stopAnalyze()
15        try EvincedEngine.reportStored()
16    }
17
18    func testContinuousModeAnalyze() throws {
19        let app = XCUIApplication()
20        app.launch()
21        app.buttons.firstMatch.tap()
22    }
23}

Issue filtering

XCUI SDK supports advanced settings that allow you to apply additional configurations when generating issue reports. Simply create an instance of the EvincedConfig object with the appropriate options. Below is an example of using EvincedConfig and IssueFilter:

Configure filters:

1let severityFilter = IssueFilter(severities: .critical)
2let issueFilter = IssueFilter(issueTypes: [IssueType.tappableArea, IssueType.colorContrast])

Global filtering:

1// Create an example filter to exclude all Needs Review issues globally
2let severityFilter = IssueFilter(severities: .critical)
3
4// Add the IssueFilter to the creation instance of EvincedEngine
5let globalEvincedConfig = EvincedConfig(excludeFilters: [severityFilter])
6EvincedEngine.options.config = globalEvincedConfig

Filtering on an individual scan (analyze() or report() call):

1let app = XCUIApplication()
2// Passing multiple include filters for an individual scan
3let multipleSeverityFilter = IssueFilter(severities: .critical, .moderate)
4let complexElementFilter = IssueFilter(elements: .query(app.buttons), issueTypes: .collidingControls, .tappableArea)
5let evincedConfig = EvincedConfig(includeFilters: multipleSeverityFilter, complexElementFilter)
6
7// Apply the configuration for a specific screen state scan
8try EvincedEngine.analyze(config: evincedConfig)
9try EvincedEngine.report(assert: false, config: evincedConfig)

Depending on the purpose of filtering, you can use two directions -include or exclude using the appropriate methods:

• EvincedConfig(includeFilters: IssueFilter..., excludeFilters: IssueFilter...)

List of available issue filtering options

You can create IssueFilter in different variations:

• IssueFilter(issueTypes: IssueType...)
• IssueFilter(issueTypes: IssueTypeFilter...)
• IssueFilter(elements: ElementFilter, issueTypes: IssueType...)
• IssueFilter(elements: ElementFilter, issueTypes: IssueTypeFilter...)
• IssueFilter(severities: SeverityFilter...)
• IssueFilter(severities: SeverityType...)
• IssueFilter(elements: ElementFilter, isRecursive: Bool)
• IssueFilter(elements: ElementFilter)
• IssueFilter(elements: ElementFilter, issueTypes: IssueTypeFilter..., isRecursive: Bool)
• IssueFilter(elements: ElementFilter, issueTypes: IssueType..., isRecursive: Bool)
• IssueFilter(elements: ElementFilter, severities: SeverityFilter...)
• IssueFilter(elements: ElementFilter, severities: SeverityType...)
• IssueFilter(elements: ElementFilter, severities: SeverityType..., isRecursive: Bool)
• IssueFilter(elements: ElementFilter, severities: SeverityFilter..., isRecursive: Bool)

Config file

Along with the configuration creation options within the code, it is also possible to create filters using a JSON file. To import the file use the examples below:

1let config = try EvincedConfig(named: "EvincedConfig.json")

Example File

1{
2  "includeFilters": [{
3    "severities": [{
4      "id": "5BD03118-5883-4F43-8831-1544AF3AFE7C"
5    }]
6  }]
7}

Rule validation configuration

RulesConfig allows you to change the accessibility check for different rules. It is set globally and used for all subsequent reports.

Earl Grey 2 support

Continuous mode is also supported with EarlGrey 2. Actions performed via EarlGrey will be recognized by Continuous mode. EarlGrey 2 support is enabled by default. However, if you need to disable this feature, you can do so by setting the following option:

1EvincedEngine.options.additionalOptions = ["earlGray2SupportEnabled": false]
2// or
3EvincedEngine.options.additionalOptions["earlGray2SupportEnabled"] = false

Uploading reports to Evinced Platform

Evinced Platform allows you to seamlessly collect, organize, and visualize Evinced accessibility reports in one place. In this section, we will guide you through the key functionalities of the upload methods from Evinced XCUI SDK, which was introduced in version 1.14.0, to the Evinced Platform. This upload method is fully compatible with the previous versions of the Evinced XCUI SDK API, and is disabled by default.

Initialize report uploading to the Evinced Platform

In order to start using Evinced Platform feature you need to include the PlatformConfig in the InitOptions:

Initializing with PlatformConfig

1override func setUpWithError() throws {
2    ...
3    let platformConfig = PlatformConfig(uploadOption: .enabledByDefault)
4    EvincedEngine.options.platformConfig = platformConfig
5}

Separate parameter initialization

1override func setUpWithError() throws {
2    ...
3    EvincedEngine.options.platformConfig.uploadOption = .enabledByDefault
4}

In this case, all of the reports from a test class will be uploaded to the Evinced Platform where you can view the accessibility issues.

Selective uploading

You can configure uploading only a specific set of reports:

1func test1() throws {
2    let app = XCUIApplication()
3    ...
4    let report1 = try app.evReport()
5    ...
6    let report2 = try app.evReport(upload: .enabled)
7    // or
8    let report3 = try EvincedEngine.report(upload: .enabled)
9}

In this case, only the second report will be uploaded to Evinced Platform.

The same behavior applies to internally stored reports reportStored() after the analyze() calls and continuous mode:

1func test1() throws {
2    let app = XCUIApplication()
3    ...
4    try app.evAnalyze()
5    try EvincedEngine.reportStored(upload: .enabled)
6}
7

Notice: that in these examples we didn't use InitOptions.PlatformConfig. It's not required to use this config argument to enable the selective report uploading. Please find more information about continuous mode here if needed.

Disable report uploading to Platform

In order to disable the upload method for all reports without changing arguments for report() or reportStored() methods you can use the following InitOption:

1override func setUpWithError() throws {
2    EvincedEngine.options.platformConfig = PlatformConfig(forceDisableReportsUpload: true)
3}
4
5func test1() throws {
6    let report1 = try EvincedEngine.report() // or try app.evReport()
7    ...
8    let report2 = try EvincedEngine.report(upload: .enabled) // or try app.evReport(upload: .enabled)
9}
10

Labeling uploaded reports

To be able to distinguish between different reports on Evinced Platform, the following pre-defined labels will be collected and added programmatically.

Customizing labels before uploading reports

To distinguish between reports from different parts of your applications or tests you can also add custom labels to the uploaded reports.

1func test1() throws {
2    let app = XCUIApplication()
3    ...
4    let metadata1: [String:String] = ["MyCustomLabelKey1": "MyCustomLabelValue1"]
5    let report1 = try EvincedEngine.report(upload: .enabled, customMetadata: metadata1)
6    // or
7    try app.evReport(upload: .enabled, customMetadata: metadata1)
8    
9    let metadata2: [String:String] = ["MyCustomLabelKey2": "MyCustomLabelValue2"]
10    let report2 = try EvincedEngine.report(upload: .enabled, customMetadata: metadata2)
11    // or
12    try app.evReport(upload: .enabled, customMetadata: metadata2)
13}
14

The custom key "MyCustomLabelKey1" and the custom value "MyCustomLabelValue1" will be attached to the first report. The custom key "MyCustomLabelKey2" and the custom value "MyCustomLabelValue2" will be attached to the second report.

Customizing labels before uploading reports for the entire test class

In order to label all reports inside your test class with a common key/value dictionary, use the EvincedEngine.testCaseMetadata property:

1override func setUpWithError() throws {
2    ...
3    EvincedEngine.testCaseMetadata = ["MyCustomCommonLabelKey": "MyCustomCommonLabelValue"]
4    ...
5}
6
7func test1() throws {
8    let app = XCUIApplication()
9    ...
10    let metadata1: [String:String] = ["MyCustomLabelKey1": "MyCustomLabelValue1"]
11    let report1 = try EvincedEngine.report(upload: .enabled, customMetadata: metadata1)
12    // or
13    try app.evReport(upload: .enabled, customMetadata: metadata1)
14    
15    let metadata2: [String:String] = ["MyCustomLabelKey2": "MyCustomLabelValue2"]
16    let report2 = try EvincedEngine.report(upload: .enabled, customMetadata: metadata2)
17    // or
18    try app.evReport(upload: .enabled, customMetadata: metadata2)
19}

In this case, all reports in the test class scope will have "MyCustomCommonLabelKey"/"MyCustomCommonLabelValue". In addition, "MyCustomLabelKey1"/"MyCustomLabelValue1" will be added only to the first report and "MyCustomLabelKey2"/"MyCustomLabelValue2" will be added only to the second report.

Test assertion

If you want to interrupt the test execution in case the report is not uploaded, it's possible to set the failTestOnUploadError argument of the InitOptions.PlatformConfig:

1override func setUpWithError() throws {
2    EvincedEngine.options.platformConfig = PlatformConfig(uploadOption: .disabledByDefault,
3                                                          failTestOnUploadError: true)
4    // or
5    EvincedEngine.options.platformConfig.uploadOption = .disabledByDefault
6    EvincedEngine.options.platformConfig.failTestOnUploadError = true
7}
8
9func test1() throws {
10    let metadata1: [String:String] = ["key1": "value1"]
11    let report1 = try EvincedEngine.report(upload: .enabled,
12                                           customMetadata: metadata1)
13    // The test will fail in case of any uploading error, it can be an internal connection or routing problems
14}

In the case of upload errors you should see an exception with a detailed error message that can be found in the logcat window to help understand the cause of the error and possible solutions.

List of available Platform upload options

forceDisableReportsUpload

forceDisableReportsUpload = true - the upload method is completely disabled. All report uploading will be disabled regardless of the upload argument:

1override func setUpWithError() throws {
2    EvincedEngine.options.platformConfig = PlatformConfig(forceDisableReportsUpload: true, 
3                                                          uploadOption: .disabledByDefault,
4                                                          failTestOnUploadError: true)
5    // or
6    EvincedEngine.options.platformConfig.forceDisableReportsUpload = true
7    EvincedEngine.options.platformConfig.uploadOption = .disabledByDefault
8    EvincedEngine.options.platformConfig.failTestOnUploadError = true
9}
10
11func test1() throws {
12    // This report won't be uploaded to Evinced Platform
13    let report1 = try EvincedEngine.report(upload: .enabled)
14    
15    // These reports won't be uploaded to Evinced Platform
16    let reports = try EvincedEngine.reportStored(upload: .enabled)
17}

forceDisableReportsUpload = false is a default value of the upload to Platform configuration;

uploadOption

EvincedEngine.options.platformConfig.uploadOption = .enabledByDefault - All reports will be uploaded, except for the reports with the upload argument equals to PlatformUpload.disabled:

1override func setUpWithError() throws {
2    EvincedEngine.options.platformConfig = PlatformConfig(uploadOption: .enabledByDefault)
3    // or
4    EvincedEngine.options.platformConfig.uploadOption = .enabledByDefault
5}
6
7func test1() throws {
8    // This report won't be uploaded to Evinced Platform
9    let report1 = try EvincedEngine.report(upload: .disabled)
10    
11    // These reports won't be uploaded to Evinced Platform
12    let reports = try EvincedEngine.reportStored(upload: .disabled)
13}

EvincedEngine.options.platformConfig.uploadOption = .disabledByDefault - Only reports that are marked with upload argument equal to PlatformUpload.enabled will be uploaded:

1override func setUpWithError() throws {
2    EvincedEngine.options.platformConfig = PlatformConfig(uploadOption: .disabledByDefault)
3    // or
4    EvincedEngine.options.platformConfig.uploadOption = .disabledByDefault
5}
6
7func test1() throws {
8    // This report will be uploaded to Evinced Platform
9    let report1 = try EvincedEngine.report(upload: .enabled)
10    
11    // These reports will be uploaded to Evinced Platform
12    let reports = try EvincedEngine.reportStored(upload: .enabled)
13}

failTestOnUploadError

failTestOnUploadError = true - Stop the test execution in case of an upload error;

failTestOnUploadError = false - Continues the test execution no matter of any upload errors. Default value.

The state table of the configurations is as following:

List of available Platform upload arguments

report(upload: .enabled) or evReport(upload: .enabled) - Upload the report or reports. Will have an effect when the EvincedEngine.options.platformConfig is .enabledByDefault or .disabledByDefault. Can be used as an argument is report(upload: .enabled) or reportStored(upload: .enabled) methods;

report(upload: .disabled) or evReport(upload: .disabled) - Don't upload the report or reports. Will have an effect when the EvincedEngine.options.platformConfig is .enabledByDefault or .disabledByDefault Can be used as an argument is report(upload: .disabled),or reportStored(upload: .disabled) methods.

Export accessibility labels of buttons and images for manual review

Accessibility labels of buttons and descriptive images should be meaningful, so users can fully understand their purpose and easily interact with the application. You can review those labels and validate they are meaningful by exporting a JSON report of the accessibility label of images and buttons, separated from the test report. To export this JSON report, simply add the following flag:

1EvincedEngine.options.additionalOptions = ["exportMeaningfulLabels": true]

You should see a new JSON attachment created beside other reports with a default name: Evinced_A11Y_Test_meaningfulLabelReport.json. The structure of this JSON file is described in our Mobile Reports page.

Exporting elements with no issues

To export a JSON report of all scanned elements that had no issues found, simply add the additional options flag:

1EvincedEngine.options.additionalOptions = ["exportNoIssuesElements": true]

You should see a new JSON file created beside other report files with a default name: Evinced_A11Y_Test_noIssuesElementsReport.json.

The structure of this JSON file is described in our Mobile Reports page.

You can also obtain elements with no issues JSON report on test runtime using subreports:

1EvincedEngine.options.additionalOptions = ["exportNoIssuesElements": true]
2let report = try EvincedEngine.report()
3
4let noIssueElementsReport = report.subReports.noIssueElementsReport
5let noIssuesElements = noIssueElementsReport?.noIssuesElements

Skip export of empty accessibility reports

To skip generating empty availability reports (those that have no issues), use the ReportSavingStrategy enum to specify the following flag in the options:

1EvincedEngine.options.reportSavingStrategy = .skipEmpty

This option will either filter out the empty (having 0 issues) scans from the resulting report or will prevent the entire report file from being saved in case all the scans in the report are empty. Other values of the ReportSavingStrategy are:

• saveAll. This option reflects the default behavior when all reports/scans including the empty ones get exported. This option doesn't have to be specified explicitly.
• skipEmpty. This option switches off saving of all the reports files in the file system. The platform uploading will still happen if it is enabled with the corresponding settings.

API

EvincedEngine

@objc public class EvincedEngine : NSObject

Main EvincedEngine class

Example usage:

1override func setUpWithError() throws {
2    EvincedEngine.testCase = self
3    // Other setup code..
4}

InitOptions

1public struct InitOptions {
2    /// Possible options for storing screenshots in reports.
3    public enum ScreenshotOptions: String, Codable {
4        case base64
5        case disabled
6    }
7    /// Enable or disable logging.
8    public var isLoggingEnabled: Bool
9    /// Common Evinced config, applied to al report and analyze calls by default.
10    public var config: EvincedConfig
11    /// Options for storing screenshots in reports.
12    public var screenshotOptions: ScreenshotOptions
13    /// Configuration for accessibility validation for given rule.
14    public var rulesConfig: [RuleConfig]
15    /// Custom prefix name for reports.
16    public var reportName: String?
17    /// Additional options which could be set for specific keys.
18    public var additionalOptions: [String: Any] = [:]
19    /// Configuration for system accessibility audit.
20    public var systemAuditConfig: SystemAuditConfig
21    /// Configuring reports uploading for the dashboard.
22    public var platformConfig: PlatformConfig
23    /// Option that removes duplications in `continuousMode`, default is `true`
24    public var allowDeduplication: Bool
25}

reportName

A custom prefix for reports. The default prefix is "Evinced_A11Y_Test_Results". Example usage:

1override func setUpWithError() throws {
2    EvincedEngine.testCase = self
3    EvincedEngine.options.screenshotOptions = .disabled
4    EvincedEngine.options.reportName = "Test Example Report Name"
5}

rulesConfig

RulesConfig allows to change the accessibility check for different rules. By default, the value is empty

Usage example:

This example disable validation for Color Contrast.

Add using rule name:

1EvincedEngine.options.rulesConfig = [RuleConfig(name: .name("Color Contrast"), isEnabled: false)]

Add using rule UUID:

1EvincedEngine.options.rulesConfig = [RuleConfig(name: .id(UUID(uuidString: "094EE482-D97A-474C-B8EB-1D8E787E1144")), isEnabled: false)]

Add using IssueType:

1EvincedEngine.options.rulesConfig = [RuleConfig(name: .id(IssueType.colorContrast.rawValue), isEnabled: false)]

Tappable Area

Every native mobile application features a wide variety of UI elements which are interactive via tap. Consider elements like buttons, text fields, toggles and others. Each of them should be tapped in order to become focused and to allow any further interaction. The lower the tappable area of the element (an imaginary square where user should tap) the less convenient it is to operate with the element. Platform constraints describe the width and height of minimum tappable area.

tappableAreaDimensions - the minimum size that does not trigger Tappable Area validation. By default for iOS it is 44px.

For example

1    func testTappableAreaDimensions() throws {
2        let app = XCUIApplication()
3        app.launch()
4
5        EvincedEngine.options.rulesConfig = [RuleConfig(name: .name("Tappable Area"),
6                                                        isEnabled: true,
7                                                        options: ["tappableAreaDimensions" : 30.0])]
8
9        let tappableAreaDimensionsReport = try app.evReport()
10        let isTappableAreaDimensions = tappableAreaDimensionsReport.elements.contains {
11            $0.issues.contains {
12                $0.issueType.type == .tappableArea && $0.accessibilityLabel == "Pencil"
13            }
14        }
15
16        XCTAssertFalse(isTappableAreaDimensions, "Report should not have issues with TappableArea for element with id Pencil")
17    }

Continuous mode

func startAnalyze()

Watch for screen state changes and record all accessibility issues until stopAnalyze() is called. It works the same way as analyze() but collects snapshot on each XCUI command that can change application state.

Available method call:
func startAnalyze(config: EvincedConfig? = nil)

Parameters config: The config for setting report parameters. Default is nil.

Usage example: XCUI SDK | Continuous-Mode

func stopAnalyze()

Stop flow analyze and returns a report with all issues found from the last startAnalyze() call. After calling this method, EvincedEngine will not respond to application state changes and collect accessibility error reports. Note: reportStored() must be called after stopAnalyze() to generate report files.

Additional features

To avoid duplicated reports you can use the allowDeduplication flag in InitOptions, it is enabled by default. Example:

1EvincedEngine.options.allowDeduplication = true

systemAuditConfig

Configuration for system accessibility audits. This setting allows the framework to utilize Xcode's system capabilities to identify and address accessibility issues within the iOS application.

Properties

• dynamicValidationMode: Configures the behavior of dynamic validations that change screen content in real-time. Options:
  • .enabled: Dynamic validations are enabled, allowing real-time changes to be validated.
  • .disabled: Dynamic validations are disabled, avoiding the performance overhead associated with real-time validation.
  • .disabledForContinuousMode: Dynamic validations are disabled during continuous mode operations but enabled for single, one-time analyses.

Usage Example

This example sets up SystemAuditConfig with dynamic validations enabled:

1EvincedEngine.options.systemAuditConfig = SystemAuditConfig(dynamicValidationMode: .enabled)

In this configuration, systemAuditConfig is used to improve the detection of availability issues by enabling dynamic validations.

Additional Information

• Compatibility: Requires Xcode 15 or later, and supports iOS 17 or later.
• Purpose: Designed specifically to help detect accessibility issues using system auditing capabilities.
• Default Setting: By default, the dynamicValidationMode is set to .disabled, to ensure that dynamic validations are performed only when explicitly requested, optimizing for performance during standard usage scenarios.

Public Interface

isLoggingEnabled

Enable or disable logging. Default is true.

@objc static var isLoggingEnabled: Bool { get set }

Example usage:

1override func setUpWithError() throws {
2    EvincedEngine.testCase = self
3    EvincedEngine.isLoggingEnabled = false
4    // Other setup code..
5}

testCase

Test case for adding accessibility report attachment.

@objc static weak var testCase: XCTestCase? { get set }

Example usage:

1override func setUpWithError() throws {
2    EvincedEngine.testCase = self
3    EvincedEngine.isLoggingEnabled = false
4    // Other setup code..
5}

setupCredentials()

Setup licensing credentials for online validation. The method requires a network request.

static func setupCredentials(serviceAccountId: String, apiKey: String) throws

Example usage:

1override func setUpWithError() throws {
2    // Put setup code here. This method is called before the invocation of each test method in the class.
3    try EvincedEngine.setupCredentials(serviceAccountId: "your_service_account_id",
4                                       apiKey: "your_api_key")
5}

Parameters

setupOfflineCredentials()

Setup licensing credentials. Method requires no network requests.

static func setupOfflineCredentials(serviceAccountId: String, accessToken: String) -> Bool

Example usage:

1override func setUpWithError() throws {
2    // Put setup code here. This method is called before the invocation of each test method in the class.
3
4
5    // In UI tests it is usually best to stop immediately when a failure occurs.
6    continueAfterFailure = false
7
8    // In UI tests it’s important to set the initial state - such as interface orientation - required for your tests before they run. The setUp method is a good place to do this.
9    EvincedEngine.setupOfflineCredentials(serviceAccountId: "your_service_account_id",
10                                          accessToken: "your_access_token")
11
12    EvincedEngine.testCase = self
13}

Parameters

Return value:
Result of license validation (true if successful).

report()

Generates the accessibility report for the current state and saves it as test attachment.

func report(for application: XCUIApplication?, assert: Bool = false, file: StaticString = #filePath, line: UInt = #line) throws -> Report

Example usage:

1func testMainStationsView() {
2    hamburgerMenu.tap()
3    assertHamburgerContent()
4    try EvincedEngine.report(assert: true)
5}

Parameters

Return value:

Evinced accessibility Report object. For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page. If you’ve run your test using the Xcode GUI, you can access the attached reports as described here. If you run your tests in a CI pipeline, you can copy the reports into any directory on your CI machine using xcparse.

analyze()

Stores the accessibility data of an application for further validation. When no application is set, the currently active app is analyzed.

func analyze(_ application: XCUIApplication? = nil) throws

Example usage:

1func testMainStationsView() throws {
2    hamburgerMenu.tap()
3    assertHamburgerContent()
4    try EvincedEngine.analyze()
5}

Parameters

reportStored()

Reports all the stored accessibility data at once. Clears all the stored test data, creates attachment with HTML test report.

static func reportStored(assert: Bool = false, file: StaticString = #filePath, line: UInt = #line) throws

Example usage:

1override func tearDownWithError() throws {
2    // Put teardown code here. This method is called after the invocation of each test method in the class.
3    // By passing true the test will be fail if an accessibility issue is found
4    try EvincedEngine.reportStored()
5    super.tearDown()
6}

Parameters

clearStored()

Clear all the stored accessibility test data.

@objc static func clearStored()

Example usage:

1override func tearDown() {
2    // Put teardown code here. This method is called after the invocation of each test method in the class.
3    EvincedEngine.clearStored()
4    super.tearDown()
5}

Extension for XCUIApplication

evReport()

Generates the accessibility report for the current state and saves it as a test attachment.

func evReport(assert: default false, file: StaticString = #filePath, line: UInt = #line) throws -> Report

Example usage:

1func testMainStationsView() throws {
2    hamburgerMenu.tap()
3    assertHamburgerContent()
4    try app.evReport()
5}

Parameters

Return value:

Evinced accessibility Report object. For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page. If you’ve run your test using the Xcode GUI, you can access the attached reports as described here. If you run your tests in a CI pipeline, you can copy the reports into any directory on your CI machine using xcparse.

evAnalyze()

Stores the accessibility data of an application for further validation.

func evAnalyze() throws

Example usage:

1func testMainStationsView() throws {
2    hamburgerMenu.tap()
3    assertHamburgerContent()
4    try app.evAnalyze()
5}

Tutorials

Generating a comprehensive accessibility report for your iOS application

In this tutorial, we will enhance our existing XCTest UI test with the Evinced XCUI SDK in order to check our application for accessibility issues. In order to get started you will need the following:

1. All of the prerequisites for XCUI SDK should be met;
2. UI tests target should exist in your project

Preface - Existing UI test overview

Let’s consider the following basic UI test from the open-source Swift Radio app as our starting point.

1//  SwiftRadioUITests.swift
2//  SwiftRadioUITests
3//  Created by Jonah Stiennon on 12/3/15.
4//  Copyright © 2015 matthewfecher.com. All rights reserved.
5
6import XCTest
7
8class SwiftRadioUITests: XCTestCase {
9
10    let app = XCUIApplication()
11    let stations = XCUIApplication().cells
12    let hamburgerMenu = XCUIApplication().navigationBars["Swift Radio"].buttons["icon hamburger"]
13    let pauseButton = XCUIApplication().buttons["btn pause"]
14    let playButton = XCUIApplication().buttons["btn play"]
15    let volume = XCUIApplication().sliders.element(boundBy: 0)
16
17    override func setUp() {
18        super.setUp()
19        continueAfterFailure = false
20        // UI tests must launch the application that they test. Doing this in setup will make sure it happens for each test method.
21        XCUIApplication().launch()
22        // wait for the main view to load
23        self.expectation(
24            for: NSPredicate(format: "self.count > 0"),
25            evaluatedWith: stations,
26            handler: nil)
27        self.waitForExpectations(timeout: 10.0, handler: nil)
28    }
29
30    override func tearDown() {
31        super.tearDown()
32    }
33
34    func assertStationsPresent() {
35        let numStations:UInt = 5
36        XCTAssertEqual(stations.count, Int(numStations))
37        let texts = stations.staticTexts.count
38        XCTAssertEqual(texts, Int(numStations * 2))
39    }
40
41    func assertHamburgerContent() {
42        XCTAssertTrue(app.staticTexts["Created by: Matthew Fecher"].exists)
43    }
44
45    func assertAboutContent() {
46        XCTAssertTrue(app.buttons["email me"].exists)
47        XCTAssertTrue(app.buttons["matthewfecher.com"].exists)
48    }
49
50    func assertPaused() {
51        XCTAssertFalse(pauseButton.isEnabled)
52        XCTAssertTrue(playButton.isEnabled)
53        XCTAssertTrue(app.staticTexts["Station Paused..."].exists);
54    }
55
56    func assertPlaying() {
57        XCTAssertTrue(pauseButton.isEnabled)
58        XCTAssertFalse(playButton.isEnabled)
59        XCTAssertFalse(app.staticTexts["Station Paused..."].exists);
60    }
61
62    func assertStationOnMenu(_ stationName:String) {
63        let button = app.buttons["nowPlaying"];
64        XCTAssertTrue(button.label.contains(stationName))
65    }
66
67    func assertStationInfo() {
68        let textView = app.textViews.element(boundBy: 0)
69        if let value = textView.value {
70            XCTAssertGreaterThan((value as AnyObject).length, 10)
71        } else {
72            XCTAssertTrue(false)
73        }
74    }
75
76    func waitForStationToLoad() {
77        self.expectation(
78            for: NSPredicate(format: "exists == 0"),
79            evaluatedWith: app.staticTexts["Loading Station..."],
80            handler: nil)
81        self.waitForExpectations(timeout: 25.0, handler: nil)
82
83    }
84
85    func testMainStationsView() {
86        // Use recording to get started writing UI tests.
87        // Use XCTAssert and related functions to verify your tests produce the correct results.
88
89        assertStationsPresent()
90
91        hamburgerMenu.tap()
92        assertHamburgerContent()
93        app.buttons["About"].tap()
94        assertAboutContent()
95        app.buttons["Okay"].tap()
96        app.buttons["btn close"].tap()
97        assertStationsPresent()
98
99        let firstStation = stations.element(boundBy: 0)
100        let stationName:String = firstStation.children(matching: .staticText).element(boundBy: 0).label
101        assertStationOnMenu("Choose")
102        firstStation.tap()
103        waitForStationToLoad();
104
105        pauseButton.tap()
106        assertPaused()
107        playButton.tap()
108        assertPlaying()
109        app.navigationBars["Sub Pop Radio"].buttons["Back"].tap()
110        assertStationOnMenu(stationName)
111        app.navigationBars["Swift Radio"].buttons["btn nowPlaying"].tap()
112        waitForStationToLoad()
113        volume.adjust(toNormalizedSliderPosition: 0.2)
114        volume.adjust(toNormalizedSliderPosition: 0.8)
115        volume.adjust(toNormalizedSliderPosition: 0.5)
116        app.buttons["More Info"].tap()
117        assertStationInfo()
118        app.buttons["Okay"].tap()
119        app.buttons["logo"].tap()
120        assertAboutContent()
121        app.buttons["Okay"].tap()
122    }
123}

The purpose of this test is to check the full user flow of the application. For now, this test is only concerned with the functional testing of the app. However, with the help of Evinced XCUI SDK, we can also check it for accessibility issues along the way. Let’s go through this process with the following step-by-step instructions.

Step #1 - Initialize the Evinced XCUI SDK

Please, make sure EvincedXCUISDK module is added to UI tests target via CocoaPods or Swift Package Manager and properly imported:

1//  SwiftRadioUITests.swift
2//  SwiftRadioUITests
3//  Created by Jonah Stiennon on 12/3/15.
4//  Copyright © 2015 matthewfecher.com. All rights reserved.
5
6import XCTest
7import EvincedXCUISDK
8
9// Other stuff...

Before making any assertions against our app, we need to set up the EvincedEngine. This class is used primarily as an entry point to all of the accessibility scanning features. Since we are going to use it primarily within our test, the best place for its initialization will be our func setUp() method of the test case. Because of most of methods of EvincedEngine may throw exceptions we need to adopt our func setUp() to func setUpWithError() throws

1override func setUpWithError() throws {
2    // Put setup code here. This method is called before the invocation of each test method in the class.
3    continueAfterFailure = false
4    // UI tests must launch the application that they test. Doing this in setup will make sure it happens for each test method.
5    XCUIApplication().launch()
6
7    // wait for the main view to load
8    self.expectation(
9        for: NSPredicate(format: "self.count > 0"),
10        evaluatedWith: stations,
11        handler: nil)
12    self.waitForExpectations(timeout: 10.0, handler: nil)
13
14    // Use your real keys here
15    try EvincedEngine.setupCredentials(serviceAccountId: "your_service_account_id",
16                                       apiKey: "your_api_key")
17    EvincedEngine.testCase = self
18}

The only setup needed for the EvincedEngine is to set licensing credentials and the testCase. The latest is needed to save the HTML and JSON reports.

Step #2 - Identify which application states you want to check for accessibility issues

Once the SDK is set up, let’s try to carefully consider which states of our application are actually worth checking for accessibility issues. Same adaptation of catching exceptions is need. Let’s take a look at our UI test once again:

1func testMainStationsView() throws {
2    // Use recording to get started writing UI tests.
3    // Use XCTAssert and related functions to verify your tests produce the correct results.
4
5    assertStationsPresent()
6
7    hamburgerMenu.tap()
8    assertHamburgerContent()
9    app.buttons["About"].tap()
10    assertAboutContent()
11    app.buttons["Okay"].tap()
12    app.buttons["btn close"].tap()
13    assertStationsPresent()
14
15    let firstStation = stations.element(boundBy: 0)
16    let stationName:String = firstStation.children(matching: .staticText).element(boundBy: 0).label
17    assertStationOnMenu("Choose")
18    firstStation.tap()
19    waitForStationToLoad();
20
21    pauseButton.tap()
22    assertPaused()
23    playButton.tap()
24    assertPlaying()
25    app.navigationBars["Sub Pop Radio"].buttons["Back"].tap()
26    assertStationOnMenu(stationName)
27    app.navigationBars["Swift Radio"].buttons["btn nowPlaying"].tap()
28    waitForStationToLoad()
29    volume.adjust(toNormalizedSliderPosition: 0.2)
30    volume.adjust(toNormalizedSliderPosition: 0.8)
31    volume.adjust(toNormalizedSliderPosition: 0.5)
32    app.buttons["More Info"].tap()
33    assertStationInfo()
34    app.buttons["Okay"].tap()
35    app.buttons["logo"].tap()
36    assertAboutContent()
37    app.buttons["Okay"].tap()
38}

We have identified many places in our UI test where we would like to perform accessibility scans. A good place to add a check is after a user interaction like a tap, swipe, etc. Each interaction may reveal more of the application to check for issues. So, let’s put our checks into place.

Step #3 - Setup accessibility analyzing points

The only thing we need to do now is to simply add EvincedEngine.analyze() calls to the places in the test we identified above. Besides the EvincedEngine method, we could also use the analyze() method from the public XCUIApplication extension as they are identical. Because we have reference to the app object, we prefer the second variant in this example. Additionally, we marked our test as a throwing function for convenience. Here is what it should look like:

1func testMainStationsView() throws {
2    // Use recording to get started writing UI tests.
3    // Use XCTAssert and related functions to verify your tests produce the correct results.
4    try app.evAnalyze()
5
6    assertStationsPresent()
7    try app.evAnalyze()
8
9    hamburgerMenu.tap()
10    assertHamburgerContent()
11    try app.evAnalyze()
12
13    app.buttons["About"].tap()
14    assertAboutContent()
15    try app.evAnalyze()
16
17    app.buttons["Okay"].tap()
18    try app.evAnalyze()
19
20    app.buttons["btn close"].tap()
21    assertStationsPresent()
22    try app.evAnalyze()
23
24    let firstStation = stations.element(boundBy: 0)
25    let stationName = firstStation.children(matching: .staticText).element(boundBy: 1).label
26    assertStationOnMenu("Choose")
27    firstStation.tap()
28    waitForStationToLoad()
29    try app.evAnalyze()
30
31    playButton.tap()
32    assertPaused()
33    try app.evAnalyze()
34
35    playButton.tap()
36    assertPlaying()
37    try app.evAnalyze()
38
39    app.navigationBars[stationName].buttons["Back"].tap()
40    assertStationOnMenu(stationName)
41    app.navigationBars["Swift Radio"].buttons.element(boundBy: 1).tap()
42    waitForStationToLoad()
43    try app.evAnalyze()
44
45    volume.adjust(toNormalizedSliderPosition: 0.2)
46    volume.adjust(toNormalizedSliderPosition: 0.8)
47    volume.adjust(toNormalizedSliderPosition: 0.5)
48    try app.evAnalyze()
49
50    app.buttons["More Info"].tap()
51    assertStationInfo()
52    try app.evAnalyze()
53
54    app.buttons["Okay"].tap()
55    try app.evAnalyze()
56
57    app.buttons["logo"].tap()
58    assertAboutContent()
59    try app.evAnalyze()
60
61    app.buttons["Okay"].tap()
62    try app.evAnalyze()
63}

The evAnalyze() method collects an accessibility snapshot of the current application state and puts it into internal in-memory storage. This snapshot contains a lot of information including an application screenshot, accessibility labels, and dimensions of UI elements.

Now that we have gathered the information we need we are ready to actually find out whether our app is accessible or not.

Step #4 - Validate your screens at the end of your test

As our test was executed we collected a lot of accessibility snapshots via the evAnalyze() calls. We can now perform accessibility validations or assertions at the end of our test suite. We can use reportStored method with assert: flag. The difference is that assert: true will fail the test if any accessibility issue is present and generate HTML and JSON report as a test attachment while the assert: false flag will generate the reports but not fail the test. Referring back again to our initial UI test the best place for this assertion will be the method that is invoked last - tearDown(). Let’s adapt it too for exception tearDown() change for tearDownWithError() throws

1override func tearDownWithError() throws {
2    // Put teardown code here. This method is called after the invocation of each test method in the class.
3    try EvincedEngine.reportStored()
4}

The XCUI SDK will check all of the previously collected accessibility snapshots for accessibility issues and generate the attachments if EvincedEngine.testcase is set up properly.

If you prefer to fail the test if there is any issue present, use:

1override func tearDownWithError() throws {
2    // Put teardown code here. This method is called after the invocation of each test method in the class.
3    try EvincedEngine.reportStored(assert: true)
4}

You are now set to run the test and ensure the accessibility of your application!

Step #5 - Run your UI test

Run your UI test from the Xcode GUI or CI pipeline. If anything is set up properly you’ll see a console log message about saving your reports:

1    t =    25.92s Tear Down
2    t =    26.29s     Added attachment of type 'public.json'
3    t =    26.29s     Added attachment of type 'public.html'

Step #6 - Access the reports

Now it's time to locate a review the reports. If you’ve run your test using the Xcode GUI, you can access the attached reports as described here. If you run your tests in a CI pipeline, you can copy the reports into any directory on your CI machine using xcparse. For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.

Xcode Result Parsers

xparse

The xcparse tool is a command-line utility designed to extract and manage data from .xcresult bundles, which are generated by Xcode after running tests or building projects. The tool is particularly useful for CI/CD pipelines, where automated extraction and reporting of test results and screenshots are required. Key features include:

• Extracting Screenshots: Easily extract screenshots from test results for further analysis or reporting.
• Parsing Logs: Retrieve logs and other artifacts from the .xcresult bundles to better understand test outcomes.
• Compatibility: Works seamlessly with various Xcode versions, ensuring consistent results across different development environments.
• Automation Friendly: Designed to be integrated into automated workflows, making it ideal for continuous integration setups.

Installation

There are a few ways to install .xcresult. Please visit an official repository page to get more information about the installation step.

Usage

To extract only HTML attachments:

1xcparse attachments /path/to/test.xcresult /path/to/outputDirectory --uti public.html

To extract only JSON attachments:

1xcparse attachments /path/to/Test.xcresult /path/to/outputDirectory --uti public.json

xcresulttool GitHub Action

The xcresulttool GitHub Action is a GitHub Action that wraps the xcresulttool (command-line utility included with Xcode). This Action is used within GitHub workflows to query and extract specific data from .xcresult bundles directly in your CI/CD pipelines. Key features include:

• Seamless Integration: Easily integrates with GitHub Actions to automate the processing of .xcresult files as part of your CI pipeline.
• Flexible Data Extraction: Uses the xcresulttool utility to extract specific test results, logs, or performance data in JSON format, allowing for customized reporting and analysis.
• Simplified Setup: As a GitHub Action, it simplifies the process of integrating xcresulttool into your workflows, reducing the need for manual script management.

Installation

To use the xcresulttool GitHub Action in your GitHub workflows, you don't need to install anything manually; instead, you define the action directly in your workflow file. Here's how you can set it up:

Step-by-Step Guide

Note: This action only works on macOS builders.

By default xcodebuild will generate the xcresult bundle file to a randomly named directory in DerivedData. To use this action xcodebuild needs to generate xcresult bundle to an accessible location.

This can be done using the -resultBundlePath flag in xcodebuild.

Usage

To use it for your GitHub repository add the next step in the workflow .yml file:

1- uses: kishikawakatsumi/xcresulttool@v1.7.1
2  with:
3    path: myProjectUITests.xcresult
4  if: success() || failure()

Attention: For correct work of the step presented above you should change the next repository settings: Go Specific Repository -> Settings -> Actions -> General. Scroll down to the Workflow permissions section. Ensure that Read and write permissions item is enabled and set a checkmark for Allow GitHub Actions to create and approve pull requests. Now you can click on the Save button.

Conclusion

Both xcparse and xcresulttool GitHub Action are designed to help developers automate the extraction and analysis of test results generated by Xcode, making them necessary in CI/CD pipelines for iOS and macOS development. They each provide different levels of customization and integration, with xcparse being more universal for various environments and xcresulttool GitHub Action being optimized for use within GitHub Actions.

FAQ

1. What is the minimum iOS version supported?

Currently, we support iOS starting from iOS 12, which means we support both UIKit and SwiftUI apps.

2. Is there any difference in the number of accessibility validations Evinced will find when testing on a simulator vs. a real physical device?

Our solution is tested both on real devices and simulators and works identically at both.

3. Do I need to add anything (SDK etc.) to my app during the build process to test for accessibility violations?

No! You don’t need to add anything to your app target itself. You could even test apps download from the App Store on real devices.