WebdriverIO JS SDK

The Evinced WebdriverIO JS SDK integrates with new or existing WebdriverIO tests to automatically detect accessibility issues. By adding a few lines of code to your WebdriverIO project, you can begin analyzing all the web pages and DOM changes to provide a dynamic view of how your site can become more accessible. As a result of the test, a rich and comprehensive report is generated to easily track issues to resolution.

Interested in seeing this in action? Contact us to get started!

Prerequisites

• WebdriverIO version 8 or higher
• Node version 16+
• injectGlobals is not set to false in the configuration (by default it is set to true)

Note: There are might be errors in the console from the Evinced SDK when using WDIO v9. These are expected and can be ignored.

Supported test runners

• Mocha
• Jasmine
• Cucumber
• Other test runners are not supported officially but can be used still

Get started

Installation

To install WebdriverIO JS SDK you will need either a file provided by Evinced Support or access to a remote repository that provides it. If you have neither, contact us to get started.

Installation with a locally provided file

With a local copy of the WebdriverIO JS SDK gzipped tar package (.tgz extension), install it in your project using NPM or Node package manager of your choice:

1# Using NPM
2npm install -D <path to webdriverio-sdk-<version>.tgz file>

Installation from a remote repository

Evinced Customers have the option of accessing WebdriverIO JS SDK from a remote repository — the Evinced Jfrog Artifactory — to keep their SDK version up-to-date and to share the SDK internally at their organization.

When access is enabled, WebdriverIO JS SDK is available at
https://evinced.jfrog.io/artifactory/restricted-npm/%40evinced/webdriverio-sdk.

Installation using NPM:

1npm install @evinced/webdriverio-sdk

Authentication

To launch WebdriverIO JS SDK, you need to have a Service ID and an API Key.

Please set credentials in environment variables and reference the environment variables in code.

1# Online mode
2export EVINCED_SERVICE_ID=<serviceId>
3export EVINCED_API_KEY=<apiKey>
4
5# Offline mode, when a JWT has been provided by Evinced
6export EVINCED_SERVICE_ID=<serviceId>
7export EVINCED_AUTH_TOKEN=<token>

Setting credentials, an example:

1// Set online creadentials method
2const serviceId = process.env.EVINCED_SERVICE_ID;
3const secret = process.env.EVINCED_API_KEY;
4await Evinced.setCredentials({
5    serviceId,
6    secret
7})
8
9// OR
10
11// If provided a JWT by Evinced
12// Set offline credentials method
13const serviceId = process.env.EVINCED_SERVICE_ID;
14const token = process.env.EVINCED_AUTH_TOKEN;
15Evinced.setOfflineCredentials({
16    serviceId,
17    token
18});

Your First Test

SDK Initialization

To use WebdriverIO JS SDK, you first need to authenticate. Please refer to Authentication for details.

1// wdio.conf.js
2import EvincedService from '@evinced/webdriverio-sdk';
3const Evinced = EvincedService.default;
4
5exports.config = {
6    // your configuration
7    services: [
8        [
9            Evinced.WdioService,
10            {
11                enableScreenshots: true, // Just an example of configuration
12            },
13        ],
14    ],
15}

1// wdio.conf.ts
2import EvincedService from '@evinced/webdriverio-sdk';
3// @ts-ignore
4const Evinced = EvincedService.default;
5
6export const config: Options.Testrunner = {
7    // your configuration
8    services: [
9        [
10            Evinced.WdioService,
11            {
12                enableScreenshots: true, // Just an example of configuration
13            },
14        ],
15    ],
16}

1{
2    "compilerOptions": {
3        "allowSyntheticDefaultImports": true
4    }
5}

1// wdio.conf.js or wdio.conf.mjs
2import Evinced from '@evinced/webdriverio-sdk';
3
4exports.config = {
5    // your configuration
6    services: [
7        [
8            Evinced.WdioService,
9            {
10                enableScreenshots: true, // Just an example of configuration
11            },
12        ],
13    ],
14}

1// .babelrc
2{
3    "presets": ["@babel/preset-env"]
4}

1// wdio.conf.ts
2import EvincedService from '@evinced/webdriverio-sdk';
3// @ts-ignore
4const Evinced = EvincedService.default;
5
6export const config: Options.Testrunner = {
7    // your configuration
8    services: [
9        [
10            Evinced.WdioService,
11            {
12                enableScreenshots: true, // Just an example of configuration
13            },
14        ],
15    ],
16}

1{
2    "compilerOptions": {
3        "allowSyntheticDefaultImports": true
4    }
5}

1const Evinced = require('@evinced/webdriverio-sdk').default;
2
3exports.config = {
4    // your configuration
5    services: [
6        [
7            Evinced.WdioService,
8            {
9                enableScreenshots: true, // Just an example of configuration
10            },
11        ],
12    ],
13}

Then you must authenticate: you may want to run the authentication method in a before hook from your wdio.config file in the following way:

1// wdio.conf.ts or wdio.conf.js or wdio.conf.mjs
2before: async function(capabilities, specs) {
3    // Set offline credentials method
4    Evinced.setOfflineCredentials({
5        serviceId: '<serviceID>',
6        token: '<token>'
7    });
8
9    // Set online creadentials method (through Evinced licence server)
10    await Evinced.setCredentials({
11        serviceId: '<serviceID>',
12        secret: '<apiKey>'
13    });
14}

After that the SDK is ready to be used in your tests.

Add Evinced Accessibility Checks (Single Run Mode)

This is a simple example of how to add an Evinced accessibility scan to a test. Please note the inline comments that give detail on each test step.

1test("evAnalyze", async () => {
2    await browser.url("https://example.com/");
3    // Scan the page for accessibility issues
4    const issues = await browser.evAnalyze();
5    expect(issues).toHaveLength(0);
6});

Add Evinced Accessibility Checks (Continuous Mode)

This is an example of how to add a continuous Evinced accessibility scan to a test. Using the evStart() and evStop() methods, the Evinced engine will continually scan in the background capturing all DOM changes and page navigation as the test is executed. This will capture all accessibility issues as clicking on drop-downs or similar interactions reveals more of the page. The advantage of continuous mode is that no interaction with the actual test code is needed.

1test("evStart/evStop", async () => {
2    await browser.url("https://example.com/");
3    await browser.evStart();
4    await $("#button-1").click();
5    await $("#button-2").click();
6    const issues = await browser.evStop();
7    expect(issues).toHaveLength(0);
8});

API

Evinced.Wdioservice

Prepares the Evinced object for use in the project.

1services: [
2    [
3        Evinced.WdioService,
4        {
5            axeConfig: {
6                rules: {
7                    'link-name': { enabled: false }
8                }
9            },
10            logging: {
11                LOGGING_LEVEL: 'error',
12                ADD_LOGGING_CONTEXT: true
13            },
14            skip_validations: [],
15            toggles: {},
16        },
17    ]
18]

Refer to Configuration to see examples of initializing with options.

evAnalyze(options)

Scans the current page and returns a list of accessibility issues. This is the recommended method for static page analysis.

Note: This method is not supported if evStart() is already running.

1const issues = await browser.evAnalyze({
2    axeConfig: {
3        rules: {
4            "link-name": { enabled: false },
5        },
6    },
7});

Returns Promise<Issue[]>.

The returned report object contains a list of accessibility issues.

For more information regarding reports as well as the report object itself, please refer to our detailed Web Reports page.

evStart(options)

Continually watches for DOM mutations and page navigation, recording accessibility issues until the evStop() method is called. This method is recommended for dynamic page flows.

1await browser.evStart({
2    axeConfig: {
3        rules: {
4            "link-name": { enabled: false },
5        },
6    },
7});
8await $("#button-1").click();
9await $("#button-2").click();
10const issues = await browser.evStop();

Returns Promise<void>.

evStop(options)

Stops the issue-gathering process started by evStart().

1await browser.evStart();
2await $("#button-1").click();
3await $("#button-2").click();
4const issues = await browser.evStop();

Returns Promise<Issue[]>.

The returned report object includes all accessibility issues detected between the evStart() and evStop() method calls.

For more information regarding reports as well as the report object itself, please refer to our detailed Web Reports page.

evSaveFile(issues, format, destination)

Saves issues in a file with the specified format and location. Supported formats are json, html, sarif, and csv. Find detailed information in the Web Reports page.

1await browser.url("https://demo.evinced.com/");
2
3const issues = await browser.evAnalyze();
4
5const filePath = path.resolve(__dirname, "evinced-report.html");
6await browser.evSaveFile(issues, "html", filePath);

Returns Promise<void>.

Aggregated Report

The aggregated report feature allows you to have a general aggregated report for the whole run (not only for one test or suite). This report will contain all the issues found by the tests where evStart() and evStop() commands were called. It is still possible to use the evSaveFile() command in any place of your code along with this Aggregated Report feature.

For using the aggregated report feature you should update your config file wdio.conf.js.

1services: [
2    {
3        reporterOptions: {
4            generateAggregatedReport: true, // Enables the aggregated report feature. Mandatory.
5            deleteTmpFiles: true, // Deletes tmp files after the final report is generated. Optional. Default: true.
6            reportFormat: 'html', // Sets a desired format for the report. Available options are: html, sarif, and json. Optional. Default: html.
7            fileName: 'aggregatedReport.html', // Specifies a name of the final report. Optional. Default: aggregatedReport.html.
8            outputDir: './evincedReports', // Specifies a path to the final aggregated report file. Optional. Default: ./evincedReports.
9            tmpDir: './evincedReports/tmp' // A directory for storing Evinced tmp files. Optional. Default: ./evincedReports/tmp.
10        }
11    }
12]

Configuration

The same configuration object can be used when initializing the Evinced object using Evinced.Wdioservice and when calling the evStart() and evAnalyze() methods but with a bit different consequences. Providing options when initializing defines a global configuration for all calls of evAnalyze() and evStart(), while providing options to either of those methods affect only the test in which they are called.

Options provided in either evAnalyze() or evStart() override those set in Evinced engine initialization.

Engines Configuration

Evinced uses two separate engines when scanning for accessibility issues, one is the aXe engine and the other is the Evinced engine. By default, Evinced disables the aXe Needs Review and Best Practices issues based on customer request and due to the fact they are mostly false positives. Please note this setting when comparing issue counts directly. See an example of how to enable Needs Review and Best practices issues in the Toggles section.

Configuration Object

1export type TInitOptions = {
2    enableScreenshots?: false,
3    rootSelector?: string,
4    enableShadowDom?: false,
5    filterIssues?: IssuesFilter,
6    axeConfig?: axe.RunOptions,
7    logging?: {
8        LOGGING_LEVEL?: 'error'|'debug'|'warn'|'info',
9        ADD_LOGGING_CONTEXT?: boolean
10    }
11    skip_validations?: {
12        selector: string,
13        urlRegex: string,
14        validationTypes: string[]
15    }[],
16};

Root Selector

Sets a CSS selector to limit the Evinced Engine to scan only the selected element and its children. Must be a valid CSS selector. If not set, the Evinced Engine will scan the entire document.

Default: no value

1const issues = await browser.evAnalyze({
2    rootSelector: "#some .selector"
3});

Axe Configuration

Configures Axe open-source accessibility toolkit, which the Evinced engine includes with its own, more extensive accessibility detection. For full Axe config options, see Axe Core API.

1await browser.evStart({
2    axeConfig: {
3        rules: {
4            "link-name": { enabled: false },
5        },
6    },
7});
8const issues = await browser.evStop();

Engine Logging

Set level of messages the Evinced engine will print to the console. Valid levels are "debug", "info", "warn" and "error".

Default: "error"

1const issues = await browser.evAnalyze({
2    logging: {
3        LOGGING_LEVEL: 'debug',
4        ADD_LOGGING_CONTEXT: true
5    }
6});

Reports Screenshots

When true, the Evinced SDK will include screenshots in its reports that highlight elements with accessibility issues.

Default: false.

Note: Enabling screenshots may affect test run performance.

1exports.config = {
2    services: [
3        [Evinced.WdioService, {
4            enableScreenshots: true
5        }]
6    ]
7};

Toggles

Enables experimental features. Feature names and values may vary from release to release.

Example:

1exports.config = {
2    services: [
3        [Evinced.WdioService, {
4            experimentalFlags: {
5                USE_AXE_NEEDS_REVIEW: true,
6                USE_AXE_BEST_PRACTICES: true
7            }
8        }]
9    ]
10};

Skip Validations

Sets validation types to be skipped for specified URL pattern and CSS selector. Issue type IDs can be found by inspecting a JSON report as described in Web Reports.

Default: no validations skipped.

1const skippedValidation1 = {
2    "selector": 'selector1',
3    "urlRegex": "www\\.mysite\\.org",
4    "validationTypes": [
5        "WRONG_SEMANTIC_ROLE",
6        "NO_DESCRIPTIVE_TEXT",
7    ]
8}
9const issues = await browser.evAnalyze({ skip_validations: [skippedValidation1] });

Shadow DOM Support

Shadow DOM is now supported by default. No additional configuration is needed.

If using an earlier release, configure shadow DOM support as follows:

1const issues = await browser.evAnalyze({ enableShadowDom: true });

Global Switch

When false, disables Evinced functionality. Enabled by default, use this setting to disable Evinced accessibility analysis when not needed during test development or when running CI jobs where accessibility testing is not intended.

Default: true.

When switched off:

• evStart() and evSaveFile() will be bypassed.
• evStop() and evAnalyze() will return an empty report.

Switching Evinced Functionality Off in Configuration

1exports.config = {
2    services: [
3        [Evinced.WdioService, {
4            switchOn: false
5        }]
6    ]
7};

Important! Global Switch environment variable overrides the global configuration option.

Switching Evinced Functionality Off in Environment

1export EV_SWITCH_ON=false

Uploading Reports to Evinced Platform

Introduction

Evinced Platform allows you to seamlessly collect, organize, visualize and monitor Evinced accessibility reports in one place. In this section, we will guide you through the key functionalities of the upload methods of the accessibility reports from the Evinced SDK to the Evinced Platform, which was introduced in version to be determined. This upload method is fully compatible with the previous versions of the Evinced SDK API, and is disabled by default.

Tutorials

You can find fully functional example projects on our GitHub.

Generating a comprehensive accessibility report for your application

In this tutorial, we will enhance our existing WebdriverIO UI test with the Evinced WebdriverIO SDK in order to check our application for accessibility issues. In order to get started you will need the following:

1. All of the prerequisites for the Evinced WebdriverIO SDK should be met
2. Evinced WebdriverIO SDK should be added to your project

Preface - existing UI test overview

Let’s consider the following basic UI test as our starting point.

1describe("Evinced Demo Site tests", () => {
2    test("Search Test", async () => {
3        await browser.url("https://demo.evinced.com/");
4        const BASE_FORM_SELECTOR =
5            "#gatsby-focus-wrapper > main > div.wrapper-banner > div.filter-container";
6        const SELECT_HOME_DROPDOWN = `${BASE_FORM_SELECTOR} > div:nth-child(1) > div > div.dropdown.line`;
7        const SELECT_WHERE_DROPDOWN = `${BASE_FORM_SELECTOR} > div:nth-child(2) > div > div.dropdown.line`;
8        const TINY_HOME_OPTION = `${BASE_FORM_SELECTOR} > div:nth-child(1) > div > ul > li:nth-child(2)`;
9        const EAST_COST_OPTION = `${BASE_FORM_SELECTOR} > div:nth-child(2) > div > ul > li:nth-child(3)`;
10        const SUBMIT_BUTTON = `${BASE_FORM_SELECTOR} > .search-btn`;
11        await $(SELECT_HOME_DROPDOWN).click();
12        await $(TINY_HOME_OPTION).click();
13        await $(SELECT_WHERE_DROPDOWN).click();
14        await $(EAST_COST_OPTION).click();
15        await $(SUBMIT_BUTTON).click();
16        expect(browser).toHaveUrlContaining("/results");
17    });
18});

We wrote this test for a demo travel site called TRVL that has a few known accessibility issues.

The purpose of this test is to check the functionality of the main application screen and ensure a user can successfully select their desired trip. For now, this test is only concerned with the functional testing of the app. However, with the help of the Evinced WebdriverIO SDK, we can also check it for accessibility issues along the way. Let’s go through this process with the following step-by-step instructions.

Step #1 - Verify the Evinced WebdriverIO service

Remove optional configuration in wdio.conf.js from Evinced.WdioService service.

Step #2 - Start the Evinced engine

Now that we have everything we need to scan for accessibility issues, let’s start the Evinced engine. Since we are going to use it scan throughout our test, the best place for its initialization will be after we navigate to the desired website.

1describe("Evinced Demo Site tests", () => {
2    test("Search Test", async () => {
3        await browser.url("https://demo.evinced.com/");
4        // Add command to start recording issues
5        await browser.evStart();
6        const BASE_FORM_SELECTOR =
7            "#gatsby-focus-wrapper > main > div.wrapper-banner > div.filter-container";
8        const SELECT_HOME_DROPDOWN = `${BASE_FORM_SELECTOR} > div:nth-child(1) > div > div.dropdown.line`;
9        const SELECT_WHERE_DROPDOWN = `${BASE_FORM_SELECTOR} > div:nth-child(2) > div > div.dropdown.line`;
10        const TINY_HOME_OPTION = `${BASE_FORM_SELECTOR} > div:nth-child(1) > div > ul > li:nth-child(2)`;
11        const EAST_COST_OPTION = `${BASE_FORM_SELECTOR} > div:nth-child(2) > div > ul > li:nth-child(3)`;
12        const SUBMIT_BUTTON = `${BASE_FORM_SELECTOR} > .search-btn`;
13        await $(SELECT_HOME_DROPDOWN).click();
14        await $(TINY_HOME_OPTION).click();
15        await $(SELECT_WHERE_DROPDOWN).click();
16        await $(EAST_COST_OPTION).click();
17        await $(SUBMIT_BUTTON).click();
18        expect(browser).toHaveUrlContaining("/results");
19    });
20});

Step #3 - Stop the Evinced engine and create reports

As our test was executed we collected a lot of accessibility information. We can now perform accessibility assertions at the end of our test suite. Referring back again to our UI test the best place for this assertion will be the end of test actions. To stop the Evinced engine and generate the actual object representation of your accessibility report simply call the evStop() method. We can then output the report files in JSON format.

1describe("Evinced Demo Site tests", () => {
2    test("Search Test", async () => {
3        await browser.url("https://demo.evinced.com/");
4        // Add command to start recording issues
5        await browser.evStart();
6        const BASE_FORM_SELECTOR =
7            "#gatsby-focus-wrapper > main > div.wrapper-banner > div.filter-container";
8        const SELECT_HOME_DROPDOWN = `${BASE_FORM_SELECTOR} > div:nth-child(1) > div > div.dropdown.line`;
9        const SELECT_WHERE_DROPDOWN = `${BASE_FORM_SELECTOR} > div:nth-child(2) > div > div.dropdown.line`;
10        const TINY_HOME_OPTION = `${BASE_FORM_SELECTOR} > div:nth-child(1) > div > ul > li:nth-child(2)`;
11        const EAST_COST_OPTION = `${BASE_FORM_SELECTOR} > div:nth-child(2) > div > ul > li:nth-child(3)`;
12        const SUBMIT_BUTTON = `${BASE_FORM_SELECTOR} > .search-btn`;
13        await $(SELECT_HOME_DROPDOWN).click();
14        await $(TINY_HOME_OPTION).click();
15        await $(SELECT_WHERE_DROPDOWN).click();
16        await $(EAST_COST_OPTION).click();
17        await $(SUBMIT_BUTTON).click();
18        expect(browser).toHaveUrlContaining("/results");
19        // Add a command to stop recording and retrieve issues list
20        const issues = await browser.evStop();
21        // Save issues to a report file
22        await browser.evSaveFile(issues, "json", "./test/issues.json");
23        await browser.evSaveFile(issues, "sarif", "./test/issues.sarif.json");
24        await browser.evSaveFile(issues, "html", "./test/issues.html");
25        await browser.evSaveFile(issues, "csv", "./test/issues.csv");
26        // Assert issues count
27        expect(issues).toHaveLength(0);
28    });
29});

Fail the test if critical issues are found

Here you can see a way of failing your test if critical accessibility issues are found using WebdriverIO JS SDK.

Using evAnalyze:

1const issues = await browser.evAnalyze();
2
3const criticalIssues = issues.filter((issue) => issue.severity.name === 'Critical');
4await assert(criticalIssues.length === 0, 'found critical issues');

Using evStart/evStop:

1await browser.evStart();
2const issues = await browser.evStop();
3
4const criticalIssues = issues.filter((issue) => issue.severity.name === 'Critical');
5await assert(criticalIssues.length === 0, 'found critical issues');

The criticalIssues array will contain all the critical issues found during the scan. If the array is not empty, the test will fail on the assertion.

Complete Test Suite Integration

Support

Please feel free to reach out to support@evinced.com with any questions.

FAQ

1. Can I configure which validations to run?

Yes, see the Configuration section for details on how to configure Axe validations to your needs.

2. Can I run tests with Evinced using cloud-based services like Sauce Labs, Perfecto, or BrowserStack?

Yes, we have tested the Evinced SDK on many of these types of cloud-based services and expect no issues.