Supported versions / frameworks

Evinced WDIO mobile SDK supports the following values for Appium's automationName desired capability:

• XCUITest
• Espresso
• UIAutomator2

Older versions of automation drivers (e.g deprecated Appium iOS driver based on UIAutomation) are not supported.

Prerequisites

• Node.js 12.x or higher
• Appium Server version 1.21 or higher
• Any Webdriver IO version compatible with the versions of Node.js and Appium outlined above

Get Started

Setup

In order to use any of the Evinced Mobile SDKs you first need to create an Evinced account. Once your account is created, you will get your apiK ey and a matching serviceAccountId. Pass these tokens using the setupCredentials() method and Evinced will validate access upon test execution. If an outbound internet connection is not available in your running environment - contact us at support@evinced.com to get an offline API key.

Install Evinced WDIO SDK from the remote registry

The SDK can be added to your project as a regular NPM dependency by executing the following command:

1npm install --save @evinced/wdio-mobile-sdk

or with Yarn:

1yarn add @evinced/wdio-mobile-sdk

Install Evinced WDIO SDK from a local ZIP archive

You can also perform installation right from the standalone .tar.gz distribution file. Simply add the following entry to your package.json and perform installation by executing the npm install command.

1"dependencies": {
2//... your project dependencies
3"@evinced/wdio-mobile-sdk": "path/to/the/archive"
4}

Install with potentially incompatible peer dependencies

The SDK package enforces a check for compatible peer dependencies during the package installation. In order to ensure successful accessibility testing it is recommended to have only those major versions of the packages installed in the test environment which are compatible with the SDK. For details please refer to the peerDependencies section of the SDK's package.json file. If for any reason you have to use particular package versions which are not compatible peer dependencies of the SDK and you want to suppress related NPM errors during the SDK installation you may add the --force flag to the installation command to proceed with the installation:

1npm install --save --force @evinced/wdio-mobile-sdk

The SDK will still be installed and will be usable, however in such scenario unpredictable compatibility issues may occur.

Your first test

Initialize the SDK object

Add the import statement to the beginning of your test file.

1import { EvincedWdioMobileSdk } from '@evinced/wdio-mobile-sdk';

Create a new instance of the SDK by using the new keyword:

1const evincedWdioSdk = new EvincedWdioMobileSdk();

This will create a new instance of the Evinced WDIO mobile SDK with the default configuration.

Provide your license information

The Evinced WDIO SDK requires authentication credentials which consist of two string values mentioned above: apiKey and serviceAccountId. These values can be found by logging in to the Evinced Product Hub.

For this purpose use the method evincedWdioSdk.setupCredentials(serviceAccountId, apiKey) as shown:

1let evincedWdioSdk: EvincedWdioMobileSdk;
2
3before(() =\> {
4    evincedWdioSdk = new EvincedWdioMobileSdk();
5const isLicenseValid = evincedWdioSdk.setupCredentials(
6    process.env.EVINCED_SERVICE_ACCOUNT_ID,
7    process.env.EVINCED_API_KEY
8);
9if (!isLicenseValid) {
10    //... something went wrong with your credentials, check logs for details
11    }
12});

In this example we put Evinced credentials into environment variables called EVINCED_SERVICE_ACCOUNT_ID and EVINCED_API_KEY. If the isLicenseValid variable is set to true, you are ready to proceed adding accessibility checks into your tests.

Add an Evinced accessibility check

Simply add the .report() method to check the current state for accessibility issues.

1it('should open a radio station no. 1', async () =\> {
2    const stationsScreen = await stationsScreen.show();
3    await stationsScreen.selectStation(0);
4    const screenTitle = await nowPlayingScreen.getTitle();
5    expect(screenTitle).toContain('Absolute country hits');
6    // Add an accessibility check
7    await evincedWdioSdk.report();
8});

Regardless of the test result, the report method will always output JSON and HTML report files. By default the files will be located in the local folder called evinced-reports. For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.

Adding multiple accessibility checks

If you would like to scan multiple states within a test simply add the .analyze() method at each appropriate point followed by the .reportStored() at the end of the test. This will consolidate all of the scans into a single set of report files (HTML and JSON).

1it('should open a radio station no. 1', async () =\> {
2    // Add an accessibility check
3    evincedWdioSdk.analyze();
4    const stationsScreen = await stationsScreen.show();
5    await stationsScreen.selectStation(0);
6    // Add an accessibility check
7    evincedWdioSdk.analyze();
8    const screenTitle = await nowPlayingScreen.getTitle();
9    expect(screenTitle).toContain('Absolute country hits');
10    // Add an accessibility check
11    evincedWdioSdk.analyze();
12    //Create the Evinced HTML and JSON reports
13    await evincedWdioSdk.reportStored();
14});

API

EvincedWdioMobileSdk

constructor(initOptions?: Partial<EvincedWdioMobileSdkOptions>)

Creates an EvincedWdioMobileSdk instance and initializes it according to the options given. In case of no options provided, a default configuration will be applied.

Example of setting a custom directory for JSON/HTML files generated by the SDK:

1const options = { outputDir: './my-a11y-reports' };
2const wdioSdk = new EvincedWdioMobileSdk(options);

setOptions(options: Partial<EvincedWdioMobileSdkOptions>): void

Works the same as initOptions in constructor.

setReportNameAsTestName(testContext: any, option?: ReportNameOption): void

This method was created in order to be able to name the final report in the same way as the name of the test in which this report was created.

Example

1describe('DriverIO - report name as test name', () =\> {
2    it('Specific test info', async function () {
3    wdioSdk.setReportNameAsTestName(this);
4    let report = await wdioSdk.report();
5    // any assert you want
6  });
7});

In this method we three have options:

addReportCallInfo(testContext: any): void

You can use this method to add data about your Mocha test in the report. This includes which line the report method was called called, test method name and test name.

Example of test and method usage:

1describe('WebdriverIO - specific test info', () =\> {
2  it('Specific test info', async function () {
3    wdioSdk.addReportCallInfo(this);
4    let report = await wdioSdk.report();
5    // any assert you want
6  });
7});

setupCredentials(serviceAccountId: string, apiKey: string): boolean

A method to add credentials confirming the validity of the user's license. It accepts the following arguments:

• serviceAccountId - Evinced service account ID
• apiKey - Evinced API key

Returns a boolean flag indicating the success of the operation - if it is false there is something wrong with either validity of the license or with the checking infrastructure. See logs for more detailed output.

setupOfflineCredentials(serviceAccountId: string, accessToken: string): boolean

A method to add credentials confirming the validity of the user's license with an offline token. It accepts the following arguments:

• serviceAccountId - Evinced service account ID
• accessToken - Evinced access token

Returns a boolean flag indicating the success of the operation - if it is false there is something wrong with either validity of the license or with the checking infrastructure. See logs for more detailed output.

async report(shouldAssert: boolean = false, evincedConfig?: EvincedConfig): Promise<Report>

Generates accessibility report for the current state of the application. Returns a Report instance to allow users to perform any advanced assertions on their own. If it is not needed, a simple assertion may be delegated directly to the SDK by supplying shouldAssert argument set to true. In this case, the SDK will automatically throw an Error if there is at least one issue found. For more information regarding the Report object, please see our detailed Mobile Reports page.

The second optional parameter is the evincedConfig, which allows you to add filters by issue. More information about EvincedConfig below.

This method always writes HTML and JSON files to the configured directory.

Example usage:

1const a11yReport = await wdioSdk.report();
2if (a11yReport.elements.length \> 0) {
3  throw new Error(`My app is not accessible!`);
4}

async analyze(evincedConfig?: EvincedConfig): Promise<void>

Collects an accessibility snapshot of the current application state and puts it into internal in-memory storage. This method is meant to be used in conjunction with EvincedWdioSdk.reportStored() for performing actual assertions against the content of this storage. Optional parameter is evincedConfig, which allows you to add filters by issue. More information about evincedConfig below.

Example usage:

1await $$(`-ios predicate string:${myElementSelector}`);
2// waiting until certain elements appears on the screen
3await wdioSdk.analyze();

async reportStored(shouldAssert: boolean = false): Promise<Report[]>

Generates a list of accessibility reports, one for each accessibility snapshot collected by EvincedWdioSdk.analyze(). Just like EvincedWdio Sdk.report() this method may either perform simple assertion on its own (throwing an exception if there is at least one report with issues) or allow users to define their own assertions on the returned Report objects. This behavior can be configured via the shouldAssert flag.

This method always writes HTML and JSON files to the configured directory. Both of the files contain combined information about all of the application states collected. For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.

Example usage:

1after(async () =\> {
2  const a11yReports = await wdioSdk.reportStored();
3  const atLeastNonEmptyReportFound = a11yReports.some((report) =\>
4report.elements.length !== 0);
5  if (atLeastNonEmptyReportFound) {
6    throw new Error(
7      `My app is not accessible!`
8  );}
9});

clearStored(): void

Removes all of the accessibility snapshots previously collected by EvincedWdioSdk.analyze() calls.

async startAnalyze(evincedConfig?: EvincedConfig)

This function starts the SDK in continuous mode. It allows you to simply start monitoring the screen at the beginning of the test, after which if you write just clicks through the web driver, the SDK will register all the clicks and create a separate scan for each of them. As if we were running analyze method on every click on the screen. To end this mode, we need to call stopAnalyze - more on that later.

Example:

1describe('WebDriverIO - continuous mode', () =\> {
2  beforeEach(async () =\> {
3    await wdioSdk.startAnalyze();
4  });
5  it('Continuous mode', async () =\> {
6    const buttonList = await $('id=buttonList');
7    await buttonList.click();
8    const buttonAccessible_name = await $('id=buttonAccessible_name');
9    await buttonAccessible_name.click();
10    const reports = await wdioSdk.stopAnalyze();
11  });
12});

The example shown above will return a report with 3 scans. The first one will be from the home screen that the app will load from, and the next two after each click.

There is support for the browser.action('pointer') trigger that you could use instead of element.click():

Example:

1describe('WebDriverIO - continuous mode', () =\> {
2  beforeEach(async () =\> {
3    await wdioSdk.startAnalyze();
4  });
5  it('Continuous mode', async () =\> {
6    const buttonList = await $('id=buttonList');
7    await browser.action('pointer', { pointerType: 'touch' })
8            .move({duration: 0, origin: buttonList })
9            .down()
10            .up()
11            .perform();
12    const buttonAccessible_name = await $('id=buttonAccessible_name');
13    await browser.action('pointer', { pointerType: 'touch' })
14            .move({duration: 0, origin: buttonAccessible_name })
15            .down()
16            .up()
17            .perform();
18    const reports = await wdioSdk.stopAnalyze();
19  });
20});

There is also support for the element.touchAction('tap') and the browser.touchAction({'tap'}) triggers that you could use instead if the test logic requires that.

[IMPORTANT] The .touchAction() API is only supported by the WebdriverIO of version 8 and below. If you are using WDIO of version 9 and above consider using .click() or .action('pointer') APIs for interactions with the elements on the screen.

Example:

1describe('WebDriverIO - continuous mode', () =\> {
2  beforeEach(async () =\> {
3    await wdioSdk.startAnalyze();
4  });
5  it('Continuous mode', async () =\> {
6    const buttonList = await $('id=buttonList');
7    await buttonList.touchAction('tap');
8    const buttonAccessible\_name = await $('id=buttonAccessible\_name');
9    await buttonAccessible\_name.touchAction('tap');
10    const reports = await wdioSdk.stopAnalyze();
11  });
12});

or:

Example:

1describe('WebDriverIO - continuous mode', () =\> {
2  beforeEach(async () =\> {
3    await wdioSdk.startAnalyze();
4  });
5  it('Continuous mode', async () =\> {
6    const buttonList = await $('id=buttonList');
7    await browser.touchAction({
8        action: 'tap',
9        element: buttonList
10    });
11    const buttonAccessible\_name = await $('id=buttonAccessible\_name');
12    await browser.touchAction({
13        action: 'tap',
14        element: buttonAccessible\_name
15    });
16    const reports = await wdioSdk.stopAnalyze();
17  });
18});

async stopAnalyze(shouldAssert: boolean = false)

The second method is responsible for continuous mode. This method stops work in the continuous mode and returns the reports that were created during the work of the SDK in this mode. All scans are recorded ad returned exactly as with reportStored.

EvincedWdioMobileSdkOptions

outputDir: string = 'evinced-reports'

A path to directory where output files from WebdriverIO SDK should be stored. Either absolute or relative path may be used. If the directory does not exist, it will be created automatically.

rulesConfig: RuleConfig[];

RuleConfig allows to change the accessibility check for different rules. By default, the value is empty.

Example:

Disable tappable area issues:

1const wdioSdk = new EvincedWdioMobileSdk();
2wdioSdk.setOptions({
3  rulesConfig: [{
4    name: { id: "69ab72aa-59be-43ad-b622-4bb686aace81"},
5    enabled: false
6  }]
7});

RuleConfig entity

Properties:

Customize color contrast options

Fuzziness factor configuration for color contrast. For example, the WCAG norm for this ratio is 4.5:1, but if your corporate colors have a ratio of 4.45:1, then this can be specified using the following options.

Possible options:

Example:

1const wdioSdk = new EvincedWdioMobileSdk();
2wdioSdk.setOptions({
3  rulesConfig: [{
4    name: { name: "Color Contrast"},
5    enabled: true,
6    options: {
7      contrastRatioNormalText: 4.5,
8      contrastRatioLargeText: 4.5,
9      contrastRatioNonText: 4.5
10    }
11  }]
12});

TypeScript

In Typescript you can easily get IssueTypeId or IssueTypeName by importing.

Example:

1import { IssueTypeId, IssueTypeName } from "@evinced/wdio-mobile-sdk";
2const wdioSdk = new EvincedWdioMobileSdk();
3wdioSdk.setOptions({
4  outputDir: "test-results/rule-configs/disabled-option",
5  rulesConfig: [{
6    name: {
7      id: IssueTypeId.TAPPABLE\_AREA, // or/and
8      name: IssueTypeName.TAPPABLE\_AREA,
9    },
10  enabled: false,
11  }]
12});

screenshotOption: ScreenshotOptions;

Set option to save a screenshot in a output JSON report or as a separate .jpeg file.

By default screenshots are provided in HTML reports for each scan, but for JSON-reports they are disabled by default. You can explicitly enable or disable screenshots using the InitOptions. Screenshots in JSON reports are provided in Base64 format.

Available screenshot options:

Example:

1. Base64 option

1import { ScreenshotOptions } from '@evinced/wdio-mobile-sdk';
2const wdioSdk = new EvincedWdioMobileSdk();
3wdioSdk.setOptions({ screenshotOption: ScreenshotOptions.Base64 });

JS example:

1const wdioSdk = new EvincedWdioMobileSdk();
2wdioSdk.setOptions({ screenshotOption: "base64" });

1. Both option

1import { ScreenshotOptions } from '@evinced/wdio-mobile-sdk';
2const wdioSdk = new EvincedWdioMobileSdk();
3wdioSdk.setOptions({ screenshotOption: ScreenshotOptions.Both });

JS example:

1const wdioSdk = new EvincedWdioMobileSdk();
2wdioSdk.setOptions({ screenshotOption: "both" });

reportName: string;

A custom prefix for reports. The default prefix is Evinced_Accessibility_Test_Results. Part of initOptions.

Example usage:

1const wdioSdk = new EvincedWdioMobileSdk();
2wdioSdk.setOptions({ reportName: "Custom report name" });

evincedConfig: EvincedConfig;

The Evinced WebdriverIO SDK supports advanced settings that allow you to apply additional configurations when generating issue reports.

Example usage:

1. Exclude filters:
2. TypeScript - Multiple filters by issue type

1import { Filter, IssueTypeId, IssueTypeName } from '@evinced/wdio-mobile-sdk';
2const wdioSdk = new EvincedWdioMobileSdk();
3wdioSdk.setOptions({
4  evincedConfig: {
5    excludeFilters: new Filter()
6      .withFilterByIssueType([
7        { name: IssueTypeName.ACCESSIBLE\_NAME },
8        { id: IssueTypeId.TAPPABLE\_AREA },
9    ])
10    .getFilters(),
11  },
12});

b. TypeScript - Single filter by severity

1import { Filter, SeverityName } from '@evinced/wdio-mobile-sdk';
2const wdioSdk = new EvincedWdioMobileSdk();
3wdioSdk.setOptions({
4  evincedConfig: {
5    excludeFilters: new Filter()
6      .withFilterBySeverity({ name: SeverityName.CRITICAL })
7      .getFilters(),
8    },
9});

c. JavaScript - Single filter by severity

1const { Filter } = require('@evinced/wdio-mobile-sdk')
2const wdioSdk = new EvincedWdioMobileSdk();
3wdioSdk.setOptions({
4  evincedConfig: {
5    excludeFilters: new Filter()
6      .withFilterBySeverity({ name: "Critical" })
7      .getFilters(),
8    },
9});

2. Include filters

a. Filter by id

1const { Filter } = require('@evinced/wdio-mobile-sdk')
2const wdioSdk = new EvincedWdioMobileSdk();
3wdioSdk.setOptions({
4  evincedConfig: {
5    includeFilters: new Filter()
6      .withFilterById("com.example.demoapp:id/smallBtn")
7      .getFilters(),
8  },
9});

b. Complex filter

1const { Filter } = require('@evinced/wdio-mobile-sdk')
2const wdioSdk = new EvincedWdioMobileSdk();
3const element = await $('id=pasteButton');
4wdioSdk.setOptions({
5  evincedConfig: {
6    includeFilters: new Filter()
7      .withFilterByUiElements(element)
8      .withFilterByIssueType({ name: "Color Contrast" })
9      .withFilterById("com.example.demoapp:id/smallBtn")
10      .setRecursive(true)
11      .getFilters()
12  }
13});

List of methods in Filter:

withFilterById(viewIds: string | string[]): Filter

Creates filter by id.

withFilterBySeverity(severity: Severity | Severity[]): Filter

Creates filter by severity.

withFilterByIssueType(issueTypes: IssueType | IssueType[]): Filter

Creates filter by issue type.

async withFilterByUiElements(elements: WebdriverIO.Element | WebdriverIO.Element[]): Promiss<Filter>

Creates filter by WebdriverIO Element, or, in a more accessible language - by the UI element. The only function that works asynchronously in this constructor.

setRecursive(isRecursive: boolean): Filter

Describes if filter is includes only mentioned elements or recursively includes all their descendants.

getFilters(): IssueFilter[]

Must be called after using the desired methods, as it returns the filters we created from the previous methods.

Severity and IssueType types represent object with id: string and/or name: string.

Severity object example:

{ id: '3c68eb41-9227-481d-8eec-9dc8956b19c4', name: 'Critical' }

Evinced config from file

We can also take the evinced config described above from a file, for this we have two helpers methods:

async function configFromFile(filePath: string): Promise<EvincedConfig>

When we pass an absolute.json file path to this method, it will be read and added as evinced config.

Example:

1import { configFromFile } from '@evinced/wdio-mobile-sdk';
2const wdioSdk = new EvincedWdioMobileSdk();
3wdioSdk.setOptions({
4  evincedConfig: await configFromFile("C:\absolute\path\to\file\evinced_config.json")
5});

function configFromResourcesFile(fileName: string): Promise<EvincedConfig>

Using the second method, we can create a resources folder in the root directory of your project. We can then place the .json file with the config here, and then use the configFromResourcesFile method to get this file, passing in the filename, like evinced_config.json.

Example:

1import { configFromResourcesFile } from '@evinced/wdio-mobile-sdk';
2const wdioSdk = new EvincedWdioMobileSdk();
3wdioSdk.setOptions({
4  evincedConfig: await configFromResourcesFile("evinced\_config.json")
5});

Example .json file:

This .json generated by code below:

1{
2"excludeFilters": [
3  {
4    "elements": [
5      {
6        "viewIdResourceName": "com.odnovolov.forgetmenot.debug:id/pasteButton",
7        "contentDescription": "Paste",
8        "className": "android.widget.ImageButton",
9        "boundsInScreen": {
10          "left": 351.42857142857144,
11          "top": 32,
12          "right": 399.42857142857144,
13          "bottom": 80
14        }
15      }
16    ] },
17 {
18   "elements": [],
19   "issueTypes": [{ "name": "Color Contrast" }]
20 }, {
21    "elements": [{
22    "viewIdResourceName": "com.odnovolov.forgetmenot.debug:id/addCardsButton"
23    }]
24  }, {
25    "elements": [],
26    "severities": [{ "id": "4637208d-4f70-483d-b589-ff591c7130a0" }]
27    },
28    {
29      "elements": [],
30      "isRecursive": true
31    } ]
32}

1const wdioSdk = new EvincedWdioMobileSdk();
2const element = await $('id=pasteButton');
3wdioSdk.setOptions({
4  evincedConfig: {
5    excludeFilters: new Filter()
6      .withFilterByUiElements(element)
7      .withFilterByIssueType({ name: 'Color Contrast' })
8      .withFilterById('com.odnovolov.forgetmenot.debug:id/addCardsButton')
9      .withFilterBySeverity({ id: '4637208d-4f70-483d-b589-ff591c7130a0' })
10      .setRecursive(true)
11      .getFilters()
12    }
13});

Include filters work the same as exclude filters.

Skip export of empty accessibility reports

In order to skip creation of empty accessibility reports (ones having 0 issues) import the ReportSavingStrategy enum from the SDK package and use it to specify the following flag in the options:

1wdioSdk.setOptions({
2//...other options
3reportSavingStrategy: ReportSavingStrategy.SkipEmpty
4});

This option will either filter out the empty (having 0 issues) scans from the resulting report or will prevent the entire report file from being saved in case all the scans in the report are empty.

Other values of the ReportSavingStrategy are

• SaveAll. This option reflects the default behavior when all reports/scans including the empty ones get exported. This option doesn't have to be specified explicitly.

• SkipFiles. This option switches off saving of all the reports files in the file system. The platform uploading will still happen if it is enabled with the corresponding settings.

Export accessibility labels of buttons and images for manual review

Accessibility labels of buttons and descriptive images should be meaningful, so users can fully understand their purpose and easily interact with the application. You can review those labels and validate they are meaningful by exporting a JSON report of the accessibility label of images and buttons, separated from the test report. To export this JSON report, simply add the following additional options flag:

1wdioSdk.setOptions(
2    {
3        additionalOptions: {
4            exportMeaningfulLabels: true
5        }
6    }
7);

You should see a new JSON file created beside other report files with a default name: Evinced_Accessibility_Test_Results_01_08_2024_13_32_02_meaningful_labels.json. The structure of this JSON file is described in our [Mobile Reports] page.

To change the default folder destination and the file name of the meaningful labels JSON report you can use the second additional option with an absolute folder path:

1wdioSdk.setOptions(
2    {
3        additionalOptions: {
4            exportMeaningfulLabels: true,
5            exportMeaningfulLabelsPath: "/custom-absolute-path"
6        }
7    }
8);

Export the list of all scanned elements that do not have A11y issues

It is possible to export a list of all the elements that were scanned for A11y issues and did not have any issues found on them. To export this JSON report, simply add the following additional options flag:

1wdioSdk.setOptions(
2    {
3        additionalOptions: {
4            exportNoIssuesElements: true
5        }
6    }
7);

You should see a new JSON file created beside other report files with a default name: Evinced_Accessibility_Test_Results_01_08_2024_13_32_02_no_issues_elements.json.

To change the default folder destination and the file name of the issue free elements JSON report you can use the second additional option with an absolute folder path:

1wdioSdk.setOptions(
2    {
3        additionalOptions: {
4            exportNoIssuesElements: true,
5            exportNoIssuesElementsPath: "/custom-absolute-path"
6        }
7    }
8);

You can also obtain elements with no issues JSON report on test runtime using sub-reports:

1wdioSdk.setOptions(
2    {
3        additionalOptions: {
4            exportNoIssuesElements: true
5        }
6    }
7);
8
9let report = await wdioSdk.report();
10const noIssuesElementsReport = report.subReports?.noIssuesElements;

Uploading reports to Evinced Platform

[Evinced Platform] allows you to seamlessly collect, organize, and visualize Evinced accessibility reports in one place. In this section, we will guide you through the key functionalities of the upload methods from Evinced WebdriverIO SDK, which was introduced in version 1.20.0, to the Evinced Platform. This upload method is fully compatible with the previous versions of the Evinced Espresso SDK API, and is disabled by default.

Initialize report uploading to the Evinced Platform

In order to start using [Evinced Platform] feature you need to include the PlatformConfig in the InitOptions:

1import { UploadOption } from '@evinced/wdio-mobile-sdk';
2
3beforeEach(() => {
4 wdioSdk.setOptions({
5  platformConfig: {
6   uploadOption: UploadOption.ENABLED_BY_DEFAULT
7  }
8 });
9});

In this case, all the reports from a test suit will be uploaded to the [Evinced Platform] where you can view the accessibility issues.

Selective uploading

You can configure the SDK for uploading a specific set of reports only:

1import { PlatformUpload } from '@evinced/wdio-mobile-sdk';
2
3it('Should upload the second report to Evinced Platform', async function () {
4    wdioSdk.addReportCallInfo(this);
5 
6    let report1 = await wdioSdk.report(false);
7
8    let report2 = await wdioSdk.report(false, undefined, PlatformUpload.ENABLED);
9})

In this case, only the second report will be uploaded to [Evinced Platform].

Note: in this example we are putting false as the first argument to avoid automatic assertions and putting undefined as the second argument in case we want to omit passing the EvincedConfig to the report() method.

Note: we are adding a call to wdioSdk.addReportCallInfo() and passing this as an argument to get the correct default labels for the uploaded reports. More information on the labels will be covered later in this article.

The same behavior applies to internally stored reports reportStored() after the analyze() calls and to the stopAnalyze() in the continuous mode:

1import { PlatformUpload } from '@evinced/wdio-mobile-sdk';
2
3it('Should upload the reports to Evinced Platform', async function () {
4    wdioSdk.addReportCallInfo(this);
5    
6    let reports = await wdioSdk.reportStored(false, PlatformUpload.ENABLED);
7})

Or:

1import { PlatformUpload } from '@evinced/wdio-mobile-sdk';
2
3it('Should upload the reports to Evinced Platform', async function () {
4 wdioSdk.addReportCallInfo(this);
5 
6 await wdioSDK.startAnalyze();
7 
8 let reports = await wdioSdk.stopAnalyze(false, PlatformUpload.ENABLED);
9})

Note: in these examples we are putting false as the first argument in both calls to reportStored() or stopAnalyze() to disable automatic assertions.

Note: in these examples we didn't use InitOptions.PlatformConfig. It's not required to use this config section to enable the selective report uploading. Please find more information about continuous mode here if needed.

Disable report uploading to Platform

In order to disable the upload method for all reports without changing arguments for each report(), reportStored() or stopAnalyze() methods you can use the following InitOption:

1import { PlatformUpload } from '@evinced/wdio-mobile-sdk';
2
3beforeEach(() => {
4    wdioSdk.setOptions({
5        platformConfig: {
6            forceDisableReportsUpload: true
7        }
8    });
9});
10
11it('Should not upload any reports to Evinced Platform', async function () {
12    wdioSdk.addReportCallInfo(this);
13 
14    let report1 = await wdioSdk.report(false);
15
16    let report2 = await wdioSdk.report(false, undefined, PlatformUpload.ENABLED);
17})

Labeling uploaded reports

To be able to distinguish between different reports on [Evinced Platform], the following pre-defined labels will be collected and added programmatically.

Note: to get the values for the testMethodName and testCaseName labels populated, the method addReportCallInfo() should be called in the test before any reports are generated. Do not use the arrow function as the second argument for the it() method in this case, otherwise a wrong context will be passed to the method as this:

1import { PlatformUpload } from '@evinced/wdio-mobile-sdk';
2
3it('Should upload the report to Evinced Platform with the correct default labels', async function () {
4 wdioSdk.addReportCallInfo(this);
5
6 await wdioSdk.report(false, undefined, PlatformUpload.ENABLED);
7})

Customizing labels before uploading reports

To distinguish between reports from different parts of your applications or tests you can also add custom labels to the uploaded reports.

1import { PlatformUpload } from '@evinced/wdio-mobile-sdk';
2
3it('Should upload reports with custom labels', async function () {
4    wdioSdk.addReportCallInfo(this);
5
6    const myCustomLabelKey1 = 'MyCustomLabelKey1';
7    const myCustomLabelValue1 = 'MyCustomLabelValue1';
8
9    const myCustomLabelKey2 = 'MyCustomLabelKey2';
10    const myCustomLabelValue2 = 'MyCustomLabelValue2';
11
12    const firstReportLabels = {};
13    firstReportLabels[myCustomLabelKey1] = myCustomLabelValue1;
14
15    const secondReportLabels = {};
16    secondReportLabels[myCustomLabelKey2] = myCustomLabelValue2;
17
18    let report1 = await wdioSdk.report(false, undefined, PlatformUpload.ENABLED, firstReportLabels);
19    let report2 = await wdioSdk.report(false, undefined, PlatformUpload.ENABLED, secondReportLabels);
20})

The custom key "MyCustomLabelKey1" and the custom value "MyCustomLabelValue1" will be attached to the first report. The custom key "MyCustomLabelKey2" and the custom value "MyCustomLabelValue2" will be attached to the second report.

Customizing labels before uploading reports for the entire test class

In order to label all reports inside your test class with a common key/value, use the addTestCaseMetadata() method:

1import { PlatformUpload } from '@evinced/wdio-mobile-sdk';
2
3beforeEach(() => {
4 wdioSdk.addTestCaseMetadata("MyCustomCommonLabelKey", "MyCustomCommonLabelValue");
5});
6
7it('Should upload reports with custom labels and common custom labels', async function () {
8 wdioSdk.addReportCallInfo(this);
9
10 const myCustomLabelKey1 = 'MyCustomLabelKey1';
11 const myCustomLabelValue1 = 'MyCustomLabelValue1';
12
13 const myCustomLabelKey2 = 'MyCustomLabelKey2';
14 const myCustomLabelValue2 = 'MyCustomLabelValue2';
15
16 const firstReportLabels = {};
17 firstReportLabels[myCustomLabelKey1] = myCustomLabelValue1;
18
19 const secondReportLabels = {};
20 secondReportLabels[myCustomLabelKey2] = myCustomLabelValue2;
21
22 let report1 = await wdioSdk.report(false, undefined, PlatformUpload.ENABLED, firstReportLabels);
23 let report2 = await wdioSdk.report(false, undefined, PlatformUpload.ENABLED, secondReportLabels);
24})

In this case, all reports in the test class scope will have "MyCustomCommonLabelKey"/"MyCustomCommonLabelValue". In addition, "MyCustomLabelKey1"/"MyCustomLabelValue1" will be added only to the first report and "MyCustomLabelKey2"/"MyCustomLabelValue2" will be added only to the second report.

Test assertion

If you want to interrupt the test execution in case the report is not uploaded, it's possible to set the failTestOnUploadError argument of the InitOptions.PlatformConfig:

1import { PlatformUpload } from '@evinced/wdio-mobile-sdk';
2
3beforeEach(() => {
4    wdioSdk.setOptions({
5        platformConfig: {
6            failTestOnUploadError: true
7        }
8    });
9});
10
11it('Should upload report to the Evinced Platform and fail in case the uploading was not successful', async function () {
12    wdioSdk.addReportCallInfo(this);
13    await wdioSdk.report(false, undefined, PlatformUpload.ENABLED);
14    // The test will fail in case of any uploading error, it can be an internal connection or routing problems
15    ...
16})

In the case of upload errors you should see a Javascript Error thrown and the upload error logs that can be found in the console window with the information on the cause of the error.

List of available Platform upload options

forceDisableReportsUpload

forceDisableReportsUpload = true - the uploading feature is completely disabled. All report uploading will be disabled regardless of the upload argument:

1beforeEach(() => {
2    wdioSdk.setOptions({
3        platformConfig: {
4            forceDisableReportsUpload: true
5        }
6    });
7});

forceDisableReportsUpload = false is a default value of the upload to Platform configuration;

uploadOption

UploadOption.ENABLED_BY_DEFAULT - All reports will be uploaded, except for the reports with the upload argument equals to PlatformUpload.DISABLED:

1import { UploadOption } from '@evinced/wdio-mobile-sdk';
2
3beforeEach(() => {
4    wdioSdk.setOptions({
5        platformConfig: {
6            uploadOption: UploadOption.ENABLED_BY_DEFAULT
7        }
8    });
9});
10
11
12it('Should upload first 2 reports to the Evinced Platform', async function () {
13    wdioSdk.addReportCallInfo(this);
14
15    // will be uploaded
16    await wdioSdk.report();
17
18    // will be uploaded
19    await wdioSdk.report(false, undefined, PlatformUpload.ENABLED);
20
21    // will not be uploaded
22    await wdioSdk.report(false, undefined, PlatformUpload.DISABLED);
23})

UploadOption.DISABLED_BY_DEFAULT - Only reports that are marked with upload argument equal to PlatformUpload.ENABLED will be uploaded:

1import { UploadOption, PlatformUpload } from '@evinced/wdio-mobile-sdk';
2
3beforeEach(() => {
4    wdioSdk.setOptions({
5        platformConfig: {
6            uploadOption: UploadOption.DISABLED_BY_DEFAULT
7        }
8    });
9});
10
11
12it('Should upload first report to the Evinced Platform', async function () {
13 wdioSdk.addReportCallInfo(this);
14
15 // will be uploaded
16 await wdioSdk.report(false, undefined, PlatformUpload.ENABLED);
17
18 // will not be uploaded
19 await wdioSdk.report();
20 
21 // will not be uploaded
22 await wdioSdk.report(false, undefined, PlatformUpload.DISABLED);
23})

failTestOnUploadError

failTestOnUploadError = true - Stop the test execution in case of an upload error;

failTestOnUploadError = false - Continues the test execution regardless of the upload errors. Default value.

The state table of the configurations is as following:

List of available Platform upload arguments

PlatformUpload.ENABLED - Upload the report or reports. Will have an effect when the InitOptions.platformConfig is UploadOption.ENABLED_BY_DEFAULT or UploadOption.DISABLED_BY_DEFAULT. Can be used as an argument in report(), stopAnalyze() or reportStored() methods;

PlatformUpload.DISABLED - Don't upload the report or reports. Will have an effect when the InitOptions.platformConfig is UploadOption.ENABLED_BY_DEFAULT or UploadOption.DISABLED_BY_DEFAULT Can be used as an argument in report(), stopAnalyze() or reportStored() methods.

Tutorials

Generating a comprehensive accessibility report for your application

In this tutorial, we will enhance our existing WebdriverIO UI test with the Evinced WDIO SDK in order to check our application for accessibility issues. In order to get started you will need the following:

1. All of the prerequisites for the Evinced WDIO SDK should be met
2. Evinced WDIO SDK should be installed in your project

Preface - existing UI test overview

Let's consider the following basic UI test as our starting point.

1describe('Swift radio home page', () =\> {
2  it('should properly redirect users to a chosen station', async () =\> {
3    const availableStationLinks = await homeScreen.getStations();
4    const availableStationNames = await homeScreen.getStationNames();
5    for (let i = 0; i \< availableStationLinks.length; i++) {
6      const currentLink = availableStationLinks[i];
7      const expectedTitle = availableStationNames[i];
8      await currentLink.click();
9      const screenTitle = await nowPlayingScreen.getTitle();
10      await expect(screenTitle).toHaveTextContaining(expectedTitle);
11      await driver.back();
12    }
13  });
14});

We wrote this test for an application called Swift Radio. It is a small open-source app for listening to various radio stations and podcasts.

The purpose of these tests is to check the functionality of the main application screen and ensure a user can successfully navigate to their choice of the radio station. For now, this test is only concerned with the functional testing of the app. However, with the help of the Evinced WDIO SDK, we can also check it for accessibility issues along the way. Let's go through this process with the following step-by-step instructions.

Step #1 - Initialize the Evinced WDIO SDK

Before making any assertions against our app, we need to initialize evincedWdioSdk object. This object is used primarily as an entry point to all of the accessibility scanning features. Since we are going to use it primarily within our tests, the best place for its initialization will be our before method which gets executed first.

1import 'dotenv/config';
2import { EvincedWdioMobileSdk } from '@evinced/wdio-mobile-sdk';
3describe('Swift radio home page', () =\> {
4  let evincedWdioSdk: EvincedWdioMobileSdk;
5  before(() =\> {
6    wdioSdk = new EvincedWdioMobileSdk();
7    wdioSdk.setupCredentials(
8      process.env.EVINCED\_SERVICE\_ACCOUNT\_ID,
9      process.env.EVINCED\_API\_KEY
10    );
11  });
12  it('should properly redirect users to a chosen station', async () =\> {
13    const availableStationLinks = await homeScreen.getStations();
14    const availableStationNames = await homeScreen.getStationNames();
15    for (let i = 0; i \< availableStationLinks.length; i++) {
16      const currentLink = availableStationLinks[i];
17      const expectedTitle = availableStationNames[i];
18      await currentLink.click();
19      const screenTitle = await nowPlayingScreen.getTitle();
20      await expect(screenTitle).toHaveTextContaining(expectedTitle);
21      await driver.back();
22    }
23  });
24});

The EvincedWdioSdk can be called without any arguments. In this case, the instance will apply some default configuration. More information about providing custom configuration object can be found in the API section of this documentation.

Here we decided to keep our Evinced credentials in environment variables by using another Node.js package called dotenv. We like this option, but feel free to use whatever method you like for credentials or other sensitive information.

Step #2 - Identify which application states you want to check for accessibility issues

Once the SDK is set up, let's try to carefully consider which states of our application are actually worth checking for accessibility issues. Let's take a look at our UI test once again:

1it('should properly redirect users to a chosen station', async () =\> {
2  const availableStationLinks = await homeScreen.getStations();
3  const availableStationNames = await homeScreen.getStationNames();
4  // We start from the Home page. We need to ensure its a11y for sure.
5    for (let i = 0; i \< availableStationLinks.length; i++) {
6      const currentLink = availableStationLinks[i];
7      const expectedTitle = availableStationNames[i];
8      await currentLink.click();
9      // Here we visit each and every station screen in the app.
10      // Perhaps they are also worth checking for a11y issues.
11      const screenTitle = await nowPlayingScreen.getTitle();
12      await expect(screenTitle).toHaveTextContaining(expectedTitle);
13      await driver.back();
14    }
15});

So far we have identified two places in our UI test where we may need to perform accessibility assertion. Each place represents some distinct application state and each of these states may contain certain accessibility issues. So, let's put our checks into place.

Step #3 - Setup analyzing breakpoints

The only thing we need to do now is to simply add evincedWdioSdk.analyze() calls in places within our test which we previously identified as "interesting" ones in terms of accessibility. Here is how it should look:

1it('should properly redirect users to a chosen station', async () => {
2    const availableStationLinks = await homeScreen.getStations();
3    const availableStationNames = await homeScreen.getStationNames();
4    await evincedWdioSdk.analyze(); // Analyzing the home page
5    for (let i = 0; i < availableStationLinks.length; i++) {
6      const currentLink = availableStationLinks[i];
7      const expectedTitle = availableStationNames[i];
8      await currentLink.click();
9      await evincedWdioSdk.analyze(); // Analyzing specific station screen
10      const screenTitle = await nowPlayingScreen.getTitle();
11      await expect(screenTitle).toHaveTextContaining(expectedTitle);
12      await driver.back();
13    }
14});

The analyze method collects an accessibility snapshot of the current application state and puts it into internal in-memory storage. This snapshot contains a lot of information including application screenshot, accessibility labels and dimensions of UI elements.

Now that we have gathered the information we need we are finally ready to actually find our whether our app is accessible or not.

Step #4 - Assert your screens at the end of your test suite

As our test was executed we collected a lot of accessibility snapshots via the evincedWdioSdk.analyze() calls. We can now perform accessibility assertions at the end of our test suite. Referring back again to our initial UI test the best place for this assertion will be the method that gets invoked last - tearDown.

1after(async () => {
2    const reports = await evincedWdioSdk.reportStored();
3});

To generate the actual object representation of your accessibility report simply call the reportStored method. This way Evinced WDIO SDK will check all of the previously collected accessibility snapshots for having some accessibility issues. From this moment you are free to assert the given list of Report objects in any way you want (feel free to explore the corresponding chapter in the "API" section).

For the sake of simplicity of this tutorial let's simply assume that our application is accessible as long as it has no accessibility issues found. Thus, if we have at least one accessibility issue detected - we want our tests to be failed. Let's add the corresponding assertion:

1after(async () => {
2  const reports = await evincedWdioSdk.reportStored();
3  const atLeastNonEmptyReportFound = reports.some((report) =\> report.
4elements.length !== 0);
5  if (atLeastNonEmptyReportFound) {
6    throw new Error(
7      `A11y issues were found!`
8    );
9  }
10});

You are now set to run the test and ensure the accessibility of your application! So, go ahead and run it via your IDE or any other tooling you use for Node.js development.

Step #5 - Read the results

At this point, you will likely see that your test has failed, which is totally fine. Swift Radio certainly has some accessibility flaws, and the fact that we have caught them simply means that Evinced WDIO SDK is working!

The only thing which is missing here is the actual list of accessibility issues with would like to look at. First of all, let's pay a bit of attention to the terminal output of our tests. You should see your tests failed due to Error with the following description:

1Number of a11y issues found.
2Please refer to the html report located at:
3  /home/johndoe/dev/swift-radio/evinced-report
4  /Evinced_Accessibility_Test_Results_01_01_1970_11_15_00.html

According to this output, we have HTML and JSON files located right in the root of our project, inside the folder named evinced-reports. For more information regarding the HTML and JSON reports as well as the Report object itself, please see our detailed Mobile Reports page.

Step #6 - Make your app more accessible

Congratulations! Over the course of this 5-step tutorial we have managed to integrate accessibility tests right into our existing UI tests with the help of Evinced WDIO SDK and moreover, we have even found several issues in the app.

This is the perfect time for us to carefully investigate all of the issues detected (with help of the knowledge base) and put our JSON files into static storage for later use. With the history of all of the results, we can achieve a lot of exciting things including tracking our progress fixing all of the accessibility issues over time, detecting regressions, and gathering additional metrics.

Feel free to continue your journey with the Evinced WDIO SDK by browsing the API section.