Playwright JS SDK

The Evinced Playwright JS SDK integrates with new or existing Playwright tests to automatically detect accessibility issues. By adding a few lines of code to your Playwright project, you can begin analyzing all the web pages and DOM changes to provide a dynamic view of how your site can become more accessible. As a result of the test, a rich and comprehensive report is generated to easily track issues to resolution.

Interested in seeing this in action? Contact us to get started!

Prerequisites

• Playwright version 1.25 or higher

Get started

Installation

To install Playwright JS SDK you will need either a file provided by Evinced Support or access to a remote repository that provides it. If you have neither, contact us to get started.

Installation with a locally provided file

With a local copy of the Playwright JS SDK gzipped tar package (.tgz extension), install it in your project using NPM or Node package manager of your choice:

1# Using NPM
2npm install -D <path to js-playwright-sdk-<version>.tgz file>

Installation from a remote repository

Evinced Customers have the option of accessing Playwright JS SDK from a remote repository — the Evinced Jfrog Artifactory — to keep their SDK version up-to-date and to share the SDK internally at their organization.

When access is enabled, Playwright JS SDK is available at
https://evinced.jfrog.io/artifactory/restricted-npm/%40evinced/js-playwright-sdk.

Installation using NPM:

1npm install @evinced/js-playwright-sdk

Authentication

To launch Playwright JS SDK, you need to have a Service ID and an API Key.

Please set credentials in environment variables and reference the environment variables in code.

1# Online mode
2export EVINCED_SERVICE_ID=<serviceId>
3export EVINCED_API_KEY=<apiKey>
4
5# Offline mode, when a JWT has been provided by Evinced
6export EVINCED_SERVICE_ID=<serviceId>
7export EVINCED_AUTH_TOKEN=<token>

Setting credentials, an example:

Authenticate Playwright JS SDK in Playwright’s globalSetup as shown below for the typical case using CommonJS or TypeScript with CommonJS.

1// global.settings.js
2const { setCredentials } = require("@evinced/js-playwright-sdk");
3
4async function globalSetup(config) {
5    try {
6        await setCredentials({
7            serviceId: process.env.EVINCED_SERVICE_ID,
8            secret: process.env.EVINCED_API_KEY,
9        });
10    } catch (error) {
11        throw new Error("Evinced SDK authorization failure.");
12    }
13}
14module.exports = globalSetup;

Set the full path to the module in the Playwright configuration:

1// playwright.config.js or playwright.config.ts
2globalSetup: require.resolve("./global.setup.js");

1// playwright.config.mjs
2// use pathToFileURL to resolve full path:
3    globalSetup: pathToFileURL('./global.setup.js').pathname,

1// global.settings.mjs
2import evSdk from "@evinced/js-playwright-sdk";
3const { setCredentials } = evSdk;
4
5async function globalSetup(config) {
6    try {
7        await setCredentials({
8            serviceId: process.env.EVINCED_SERVICE_ID,
9            secret: process.env.EVINCED_API_KEY,
10        });
11    } catch (error) {
12        throw new Error("Evinced SDK authorization failure.");
13    }
14}
15export default globalSetup;

1// global.setup.js
2const { setOfflineCredentials } = require("@evinced/js-playwright-sdk");
3
4async function globalSetup(config) {
5    try {
6        await setOfflineCredentials({
7            serviceId: process.env.EVINCED_SERVICE_ID,
8            token: process.env.EVINCED_AUTH_TOKEN,
9        });
10    } catch (error) {
11        throw new Error("Evinced SDK authorization failure.");
12    }
13}

Your First Test

SDK Initialization

To use Playwright JS SDK, you first need to authenticate. Please refer to Authentication for details.

Add Evinced Accessibility Checks (Single Run Mode)

This is a simple example of how to add an Evinced accessibility scan to a test. Please note the inline comments that give detail on each test step.

This test scans https://demo.evinced.com, then generates an Evinced report. The test passes if the report is generated.

1// Use require in CommonJS module (default)
2const { test, expect } = require('@playwright/test')
3const { existsSync } = require('node:fs')
4const { EvincedSDK } = require('@evinced/js-playwright-sdk')
5
6// Use import in ECMAScript module
7// (.mjs extension or type: module in package.json)
8import { test, expect } from '@playwright/test'
9import { existsSync } from 'node:fs'
10import evExport from '@evinced/js-playwright-sdk'
11const { EvincedSDK } = evExport
12
13// Test is the same with either module system
14test.describe('Evinced Demo Page', () => {
15  test('Single Run Test', async ({ page }) => {
16    const evReport = './test-results/single.html'
17    const evincedService = new EvincedSDK(page)
18    await page.goto('https://demo.evinced.com/')
19    const issues = await evincedService.evAnalyze()
20    await evincedService.evSaveFile(issues, 'html', evReport)
21    expect(existsSync(evReport)).toBeTruthy()
22  })
23})

Add Evinced Accessibility Checks (Continuous Mode)

This is an example of how to add a continuous Evinced accessibility scan to a test. Using the evStart() and evStop() methods, the Evinced engine will continually scan in the background capturing all DOM changes and page navigation as the test is executed. This will capture all accessibility issues as clicking on drop-downs or similar interactions reveals more of the page. The advantage of continuous mode is that no interaction with the actual test code is needed.

In this test, an Evinced report is generated for https://demo.evinced.com/ after some scripted interaction. The test passes if the report is generated.

1// Use require in CommonJS module (default)
2const { test, expect } = require('@playwright/test')
3const { existsSync } = require('node:fs')
4const { EvincedSDK } = require('@evinced/js-playwright-sdk')
5
6// Use import in ECMAScript module
7import { test, expect } from '@playwright/test'
8import { existsSync } from 'node:fs'
9import evExport from '@evinced/js-playwright-sdk'
10const { EvincedSDK } = evExport
11
12// Test is the same with either module system
13test.describe('Evinced Demo Page', () => {
14  test('Continuous Test', async ({ page }) => {
15    const evReport = './test-results/continuous.html'
16    const evincedService = new EvincedSDK(page)
17    await evincedService.evStart()
18
19    await page.goto('https://demo.evinced.com/')
20
21    const BASE_FORM_SELECTOR =
22      '#gatsby-focus-wrapper > main > div.wrapper-banner > div.filter-container'
23    const SELECT_HOME_DROPDOWN = `${BASE_FORM_SELECTOR} > div:nth-child(1) > div > div.dropdown.line`
24    const SELECT_WHERE_DROPDOWN = `${BASE_FORM_SELECTOR} > div:nth-child(2) > div > div.dropdown.line`
25    const TINY_HOME_OPTION = `${BASE_FORM_SELECTOR} > div:nth-child(1) > div > ul > li:nth-child(2)`
26    const EAST_COST_OPTION = `${BASE_FORM_SELECTOR} > div:nth-child(2) > div > ul > li:nth-child(3)`
27
28    await page.locator(SELECT_HOME_DROPDOWN).click()
29    await page.locator(TINY_HOME_OPTION).click()
30    await page.locator(SELECT_WHERE_DROPDOWN).click()
31    await page.locator(EAST_COST_OPTION).click()
32
33    const issues = await evincedService.evStop()
34    await evincedService.evSaveFile(issues, 'html', evReport)
35    expect(existsSync(evReport)).toBeTruthy()
36  })
37})

API

EvincedSDK

Prepares the Evinced object for use in the project.

The specific SDK initialization is not needed. Just import EvincedSDK in your test files.

1// in CommonJS module context
2const { EvincedSDK } = require('@evinced/js-playwright-sdk');
3
4// In ECMAScript module context
5import sdk from '@evinced/js-playwright-sdk'
6const { EvincedSDK } = sdk

Refer to Configuration to see examples of initializing with options.

evAnalyze(options)

Scans the current page and returns a list of accessibility issues. This is the recommended method for static page analysis.

Note: This method is not supported if evStart() is already running.

1const evincedService = new EvincedSDK(page);
2await page.goto("https://demo.evinced.com/");
3const issues = await evincedService.evAnalyze();

Returns Promise<Issue[]>.

The returned report object contains a list of accessibility issues.

For more information regarding reports as well as the report object itself, please refer to our detailed Web Reports page.

evStart(options)

Continually watches for DOM mutations and page navigation, recording accessibility issues until the evStop() method is called. This method is recommended for dynamic page flows.

1const evincedService = new EvincedSDK(page);
2await evincedService.evStart();
3await page.goto('https://demo.evinced.com/');
4const issues = await evincedService.evStop();

Returns Promise<void>.

evStop(options)

Stops the issue-gathering process started by evStart().

1const evincedService = new EvincedSDK(page);
2await evincedService.evStart();
3await page.goto('https://demo.evinced.com/');
4const issues = await evincedService.evStop();

Returns Promise<Issue[]>.

The returned report object includes all accessibility issues detected between the evStart() and evStop() method calls.

For more information regarding reports as well as the report object itself, please refer to our detailed Web Reports page.

evSaveFile(issues, format, destination)

Saves issues in a file with the specified format and location. Supported formats are json, html, sarif, and csv. Find detailed information in the Web Reports page.

1const evincedService = new EvincedSDK(page);
2await page.goto("https://demo.evinced.com/");
3const issues = await evincedService.evAnalyze();
4evincedService.evSaveFile(issues, 'html', 'test-results/evinced-report.html');
5evincedService.evSaveFile(issues, 'csv', 'test-results/evinced-report.csv');
6evincedService.evSaveFile(issues, 'json', 'test-results/evinced-report.json');
7evincedService.evSaveFile(issues, 'sarif', 'test-results/evinced-report.sarif.json');

Returns Promise<void>.

Aggregated Report

The aggregated report feature allows you to have a general aggregated report for the whole run (not only for one test or suite). This report will contain all the issues found by the tests where evStart() and evStop() commands were called. It is still possible to use the evSaveFile() command in any place of your code along with this Aggregated Report feature.

For using the aggregated report feature you should update your config files.

First add the special entry reporterOptions to your evConfig.json file:

1{
2    "reporterOptions": {
3        "reportFormat": "html", // Sets a desired format for the report. Available options are: html, sarif, and json. Mandatory.
4        "fileName": "aggregatedReportForRun.html", // Specifies a name of the final report. Mandatory.
5        "outputDir": "./evincedReports", // Specifies a path to the final aggregated report file. Mandatory.
6        "tmpDir": "./evincedReports/tmp" // A directory for storing Evinced tmp files. 'os.tmpdir()' by default. Optional.
7    }
8}

Then add specify a path to the Evinced reporter in your playwright.config file. It might be found in your node-modules directory, for example:

1'./node_modules/@evinced/js-playwright-sdk/dist/reporter/evincedReporter.js'

Important! The Evinced reporter file path might be different in your project, so please check it before specifying the path in the config file.

To do so add Evinced reporter to a list of reporters, it may look like this:

1reporter: process.env.CI ? 'html' : [['line'], ['./node_modules/@evinced/js-playwright-sdk/dist/reporter/evincedReporter.js']]

Important! Setting of the reportFormat, fileName, and outputDir parameters is mandatory, the report will not be generated if some of them are missing. Setting of the tmpDir is optional, the tmp directory of you environment will be used by default.

And that's it! Now you can run your tests and get the aggregated report in the specified directory.

1// Import the Evinced reporter,
2// it may look like this in your project (in case of using commonjs):
3const EvincedReporter = require('@evinced/js-playwright-sdk/reporter');

1BeforeAll(async function () {
2    // your code here
3    new EvincedReporter().onBegin();
4});

1AfterAll(() => {
2    // your code here
3    new EvincedReporter().onEnd();
4});

Configuration

The same configuration object can be used when initializing the Evinced object using evConfig.json and when calling the evStart() and evAnalyze() methods but with a bit different consequences. Providing options when initializing defines a global configuration for all calls of evAnalyze() and evStart(), while providing options to either of those methods affect only the test in which they are called.

Options provided in either evAnalyze() or evStart() override those set in Evinced engine initialization.

Engines Configuration

Evinced uses two separate engines when scanning for accessibility issues, one is the aXe engine and the other is the Evinced engine. By default, Evinced disables the aXe Needs Review and Best Practices issues based on customer request and due to the fact they are mostly false positives. Please note this setting when comparing issue counts directly. See an example of how to enable Needs Review and Best practices issues in the Toggles section.

Configuration Object

To define global configuration for Playwright JS SDK please use file evConfig.json in a root directory of your project.

1export type EvInitOptions = {
2  rootSelector?: string;
3  enableScreenshots?: boolean;
4  axeConfig?: axe.RunOptions;
5  skipValidations?: SkipValidation[];
6  logging?: {
7    LOGGING_LEVEL: string;
8    ADD_LOGGING_CONTEXT?: boolean;
9  };
10  toggles?: { [key: string]: boolean };
11};

Root Selector

Sets a CSS selector to limit the Evinced Engine to scan only the selected element and its children. Must be a valid CSS selector. If not set, the Evinced Engine will scan the entire document.

Default: no value

1const evincedService = new EvincedSDK(page);
2await evincedService.evStart({
3  rootSelector: '.some-selector'
4});
5await page.goto('https://demo.evinced.com/');
6const issues = await evincedService.evStop();

Axe Configuration

Configures Axe open-source accessibility toolkit, which the Evinced engine includes with its own, more extensive accessibility detection. For full Axe config options, see Axe Core API.

1const evincedService = new EvincedSDK(page);
2await page.goto('https://demo.evinced.com/');
3const issues = await evincedService.evAnalyze({
4    "axeConfig": {
5        "rules": {
6            "link-name": { "enabled": false }
7        }
8    },
9});

Engine Logging

Set level of messages the Evinced engine will print to the console. Valid levels are "debug", "info", "warn" and "error".

Default: "error"

1const evincedService = new EvincedSDK(page);
2await page.goto('https://demo.evinced.com/');
3const issues = await evincedService.evAnalyze({
4    logging: {
5        LOGGING_LEVEL: 'debug',
6        ADD_LOGGING_CONTEXT: true,
7    },
8});

Reports Screenshots

When true, the Evinced SDK will include screenshots in its reports that highlight elements with accessibility issues.

Default: false.

Note: Enabling screenshots may affect test run performance.

1const issues = await evincedService.evAnalyze({
2    enableScreenshots: true,
3});

Toggles

Enables experimental features. Feature names and values may vary from release to release.

Example:

1const issues = await evincedService.evAnalyze({
2    toggles: {
3        USE_AXE_NEEDS_REVIEW: true,
4        USE_AXE_BEST_PRACTICES: true
5    }
6});

Skip Validations

Sets validation types to be skipped for specified URL pattern and CSS selector. Issue type IDs can be found by inspecting a JSON report as described in Web Reports.

Default: no validations skipped.

1const skipSelector = {
2  selector: 'testValue.slds-checkbox',
3  urlRegex: '.*',
4  validationTypes: ["WRONG_SEMANTIC_ROLE", "NOT_FOCUSABLE", "NO_DESCRIPTIVE_TEXT"],
5};
6await evincedService.evStart({ skipValidations: [skipSelector] });

Knowledge-Base Link Overrides

Sets customized knowledge-base links in the reports. The links are displayed in the reports “Issue Type” column, as shown in the following screenshot:

The knowledge base link can be overridden for every issue type ID. Issue type IDs can be found by inspecting a JSON report as described in Web Reports. For example, the issue with name Interactable Role has ID WRONG_SEMANTIC_ROLE.

1const issues = await evincedService.evAnalyze({
2    issuesContentPerType: {
3        "WRONG_SEMANTIC_ROLE": "https://yourKnowlegdeBase.com/"
4    }
5});

Shadow DOM Support

Shadow DOM is now supported by default. No additional configuration is needed.

IFrames Support

When true, accessibility analysis includes iframe that exist inside the page.

Default: false.

For enabling iFrames analysis on a global level edit the evConfig.json file: add the following parameter to properties on top level.

1includeIframes: true

So it may look like:

1{
2    "logging": {
3        "LOGGING_LEVEL": "error",
4        "ADD_LOGGING_CONTEXT": true
5    },
6    "includeIframes": true
7}

It is also possible to enable iFrames analysis within one session or one analysis:

1// Enable the feature for one Continuos mode session.
2await evincedService.evStart({
3    includeIframes: true
4});
5
6// OR
7// Enable the feature for one analysis.
8const issues = await evincedService.evAnalyze({
9    includeIframes: true
10});

Important! The current implementation does not support an analysis of Cross Origin iFrames. Only issues from the iFrames with the origin of the parent iFrame will be gathered.

Proxy

Configures proxy server access settings. Needed to enable outbound communication to the Evinced Platform through a proxy server.

You can configure pages to load over the HTTP(S) proxy or SOCKSv5. Proxy can be either set globally for the entire browser, or for each browser context individually. You can optionally specify username and password for HTTP(S) proxy, you can also specify hosts to bypass proxy for. Here is an example of a global proxy:

1use: {
2    proxy: {
3      server: 'http://myproxy.com:3128',
4      username: 'usr',
5      password: 'pwd'
6    }
7  }

Global Switch

When false, disables Evinced functionality. Enabled by default, use this setting to disable Evinced accessibility analysis when not needed during test development or when running CI jobs where accessibility testing is not intended.

Default: true.

When switched off:

• evStart() and evSaveFile() will be bypassed.
• evStop() and evAnalyze() will return an empty report.

Switching Evinced Functionality Off in Configuration

In file evConfig.json add the following parameter to properties on top level.

1switchOn: false

So it may look like:

1{
2    "logging": {
3        "LOGGING_LEVEL": "error",
4        "ADD_LOGGING_CONTEXT": true
5    },
6    "switchOn": false
7}

Important! Global Switch environment variable overrides the global configuration option.

Switching Evinced Functionality Off in Environment

1export EV_SWITCH_ON=false

Uploading Reports to Evinced Platform

Introduction

Evinced Platform allows you to seamlessly collect, organize, visualize and monitor Evinced accessibility reports in one place. In this section, we will guide you through the key functionalities of the upload methods of the accessibility reports from the Evinced SDK to the Evinced Platform, which was introduced in version 2.14.0. This upload method is fully compatible with the previous versions of the Evinced SDK API, and is disabled by default.

Enable Upload Report to Platform

To enable the uploading functionality of accessibility reports to the Evinced Platform you will need to set the enableUploadToPlatform feature flag to true via the global setUploadToPlatformConfig method:

1import { setUploadToPlatformConfig } from "@evinced/js-playwright-sdk";
2setUploadToPlatformConfig({ enableUploadToPlatform: true });

You can also use the external config evConfig.json and add the values to be loaded.

Important! The external config has more precedence if both initialization options are used.

1  "uploadToPlatformOptions": {
2    "enableUploadToPlatform": true,
3    "setUploadToPlatformDefault": false
4  }

Automatic Report Upload

Once the enableUploadToPlatform method is set to true, then by default all generated reports will be uploaded to the Platform upon calling the evStop() command. If you want to change this behavior, set the setUploadToPlatformDefault feature flag to false.

1    setUploadToPlatformConfig({ enableUploadToPlatform: true, setUploadToPlatformDefault: false });

If the setUploadToPlatformDefault is disabled, you can still upload selected reports to the platform. For that, use the following parameter in the evStop() command:

1    await evincedService.evStop({ uploadToPlatform: true });

Test Names

To facilitate report management and be able to distinguish between different reports on the Platform, use the addLabel method to inform the test name and test class. It’s recommended to do that in the “beforeEach” hook.

1let evincedService;
2test.beforeEach(async ({page}, testInfo)=> {
3    evincedService = new EvincedSDK(page);
4    evincedService.testRunInfo.addLabel({
5        testName: testInfo.title,
6    })
7})

Labels and Custom Fields

You can attach labels and custom fields to your report to enhance readability in the platform. The labels will be added to the uploaded reports. See the following code example of how to set that up:

1let evincedService;
2test.beforeEach(async ({page}, testInfo)=> {
3    evincedService = new EvincedSDK(page);
4    evincedService.testRunInfo.customLabel({
5        'anyCustomParameter': 'demo value',
6        productVersion: '0.0.1'
7    })
8})

Use of beforeEach and afterEach Hooks

We recommend using the beforeEach and afterEach hooks to control analysis sessions and upload reports to the platform. This way, each test will be uploaded separately with its own report.

In beforeEach hook use evStart() to start Evinced analysis and set any labels you want for the report to contain when uploading to the platform. In the afterEach hook call evStop() to stop analysis and upload reports to the platform. See this code example:

1  let evincedService;
2  test.beforeEach(async ({page}, testInfo)=>{
3    const evincedService = new EvincedSDK(page);
4    evincedService.testRunInfo.addLabel({
5      testName: testInfo.title,
6      testFile: testInfo.file,
7      environment: 'Development'
8    })
9    await evincedService.evStart();
10  })
11  test.afterEach(async ()=>{
12    await evincedService.evStop();
13  })

Putting All of This Together

Here is a complete code snippet of how to perform uploads to the platform on a per-test basis.

1import { EvincedSDK, setUploadToPlatformConfig } from '@evinced/js-playwright-sdk';
2
3test.describe('Upload demo', () => {
4  let evincedService;
5  test.beforeAll(() => {
6    setUploadToPlatformConfig({ enableUploadToPlatform: true });
7  })
8  test.beforeEach(async ({page}, testInfo)=>{
9    evincedService = new EvincedSDK(page);
10    evincedService.testRunInfo.addLabel({
11      testName: testInfo.title,
12      testFile: testInfo.file,
13      environment: 'Development'
14    })
15    await evincedService.evStart();
16  })
17
18  test.afterEach(async ()=>{
19    await evincedService.evStop();
20  })
21
22  test('Upload example playwright #0', async ({ page }) => {
23    await page.goto("https://demo.evinced.com");
24  });
25})

Tutorials

You can find fully functional example projects on our GitHub.

Fail the test if critical issues are found

Here you can see a way of failing your test if critical accessibility issues are found using the Playwright JS SDK.

Using evAnalyze:

1const issues = await evincedService.evAnalyze();
2
3const criticalIssues = issues.filter((issue) => issue.severity.name === 'Critical');
4await assert(criticalIssues.length === 0, 'found critical issues');

Using evStart/evStop:

1await evincedService.evStart();
2const issues = await evincedService.evStop();
3
4const criticalIssues = issues.filter((issue) => issue.severity.name === 'Critical');
5await assert(criticalIssues.length === 0, 'found critical issues');

The criticalIssues array will contain all the critical issues found during the scan. If the array is not empty, the test will fail on the assertion.

Complete Test Suite Integration

Support

Please feel free to reach out to support@evinced.com with any questions.

FAQ

1. Can I configure which validations to run?

Yes, see the Configuration section for details on how to configure Axe validations to your needs.

2. Can I run tests with Evinced using cloud-based services like Sauce Labs, Perfecto, or BrowserStack?

Yes, we have tested the Evinced SDK on many of these types of cloud-based services and expect no issues.