Appium Python SDK

The Evinced Appium Python SDK integrates with new or existing Appium tests to automatically detect accessibility issues. With the addition of as few as 5 lines of code to your Appium framework, you can begin to analyze your entire application to understand how it can become more accessible. At the conclusion of the test, a rich and comprehensive HTML or JSON report is generated to track issues in any reporting tool.

Supported versions / frameworks

Evinced Appium Python SDK supports the following values for Appium’s automationName desired capability:

• XCUI SDK
• UIAutomator2
• Espresso

Older versions of automation drivers (e.g. deprecated Appium iOS driver based on UIAutomation) are not supported.

Prerequisites:

• Python 3.7 or higher.

Get Started

Setup

In order to use any of the Evinced Mobile SDKs you first need to create an Evinced account. Once your account is created, you will get your API key and a matching service account ID. Pass these these credentials by calling LicenseManager.setup_credentials(service_id, api_key) and Evinced will validate access upon test execution. If an outbound internet connection is unavailable in your running environment - contact us at support@evinced.com to get an offline token.

Install Evinced Appium SDK from the remote registry

The most basic way to install the latest package would be by pulling it from jfrog repository:

1pip install evinced-appium-sdk --extra-index-url https://evinced.jfrog.io/artifactory/api/pypi/public-python/simple/

No preliminary authentication is required.

Examples

Optional: Setting up the credentials

An additional way to provide credentials to the validator without hardcoding them into your tests would be setting up some env variables.

Windows

1set EVINCED_SERVICE_ID=uuid-type-string-here
2set EVINCED_API_KEY=short-string-here
3set EVINCED_TOKEN=longest-string-here <- if you were given offline access

MacOS and Linux

1export EVINCED_SERVICE_ID=uuid-type-string-here
2export EVINCED_API_KEY=short-string-here
3export EVINCED_TOKEN=longest-string-here <- if you were given offline access

Initialize the SDK

1from appium.options.android import UiAutomator2Options
2from appium import webdriver
3from evinced_appium_sdk import * # This import will provide all necessary structures for writing your solution
4
5# Setup Evinced license
6LicenseManager.setup_credentials(service_account_id, api_key)
7
8# Prepare the target options for Android
9options = UiAutomator2Options()
10options.platform_name = 'Android'
11options.device_name = 'API_30'
12options.app = '/your/path/app.apk'
13driver = webdriver.Remote("http://localhost:4723/wd/hub",  options=options)
14
15# Create Evinced Appium default runner
16with EvincedAppiumDefaultRunner(driver, None) as runner:
17    report = runner.report()

report()

Generates an accessibility report

1...
2# Run analysis and get the accessibility report
3report = runner.report()
4# Assert that there are no accessibility issues
5assert report[0].total == 0

Regardless of the test result, the report method will always output JSON and HTML files. You should be able to find the files by browsing the local folder called evinced-reports. It's also possible to configure this path. Use set_output_directory method.

1with EvincedAppiumDefaultRunner(driver, None) as runner:
2    runner.set_output_directory(Path("/Users/username/output"))
3    runner.report()

For more information regarding the HTML and JSON reports as well as the report object itself, please see our detailed Mobile Reports page.

analyze()

Collects an accessibility snapshot of the current application state and puts it into internal in-memory storage. This method is supposed to be used in conjunction with runner.report_stored() for performing actual assertions against the content of this storage.

report_stored()

Generates a list of accessibility reports, one for each accessibility snapshot collected by runner.analyze().

Here is an example of using analyze() and report_stored() in a test case:

1with EvincedAppiumContinuesRunner(driver, None) as runner:
2    runner.driver.find_element(AppiumBy.ID, "some_selector").click()
3    runner.analyze() # first scan
4    runner.driver.find_element(AppiumBy.ID, "some_selector").click()
5    runner.analyze() # first scan
6    report = runner.report_stored()
7
8    assert report[0].total == 0 # expectation for first scan
9    assert report[1].total == 28 # expectation for second scan

Continuous mode

Continuous mode allows continual scanning of the application without the need to insert individual scan calls with in the test code. Simply substitute EvincedAppiumDefaultRunner with EvincedAppiumContinuesRunner and instead of calling report, analyze and report_stored you need to start a analyzation session using runner.start_analyze() and runner.stop_analyze(). Evinced validator will automatically scan the application upon using appium commands send_keys, clear, click, swipe, scroll, tap. All issues detected during the tests will be added to the HTML and JSON reports automatically generated with the runner.stop_analyze() method is called.

Here is an example of using continuous mode in a single test case:

1...
2driver = get_driver()
3
4with EvincedAppiumContinuesRunner(driver, None) as runner:
5    # Run analysis and get the accessibility report
6    runner.start_analyze()
7    # This click method will produce a dedicated report
8    runner.driver.find_element(By.ID, "SomeButton").click()
9    # ...
10    reports = runner.stop_analyze()

Please, notice that we are using the instance of driver from the runner object runner.driver and not the original driver, they are specifically designed to capture reports on screens upon making an actions.

Note: It's important to note that calling stop_analyze() immediately after making an action could result in a race condition where appium might make a scan mid-animation after the click. This could produce inconsistent results. Currently, there's no way to prevent this from happening in the python ecosystem without a significant cost to performance. We would recommend adding a quick block process before the animation has finished. Here's an example:

1with EvincedAppiumContinuesRunner(driver, None) as runner:
2    runner.start_analyze()
3    runner.driver.find_element(By.ID, "SomeButton").click() # <- animation starts
4    time.sleep(0.25) # <- block thread to prevent appium from scanning too early
5    reports = runner.stop_analyze() # <- will produce a final report correctly

Important

EvincedAppiumContinuesRunner creates own copy of driver, which has patched methods to intercept calls to WebElements's click, send_keys, clear and to WebDriver's swipe, scroll and tap.

Configuration

Evinced Appium SDK provides a configuration for end-users to fit the reports to their liking.

Screenshot Options

By default, screenshots are provided in HTML reports for each scan, but for JSON-reports, they are disabled by default. You can explicitly enable or disable screenshots using the InitOptions. Screenshots in JSON reports are provided in Base64 format.

Available screenshot options:

Filtering

It is very simple to filter returned report data. There are two different filtering options: include filters and exclude filters. Include filters will filter out all the other issue types that were not defined by the filter and exclude filter will remove the issues as defined. In combination the include filter will be prioritized, meaning that if we filter out Best Practice issues from the report and include a particular button, then the report will contain only this button, but it will not have Best Practices issues.

To get the most up-to-date information of possible ways to filter out the issue the end user is encouraged to take a look at source code filtering structures.

Filtering based on Severity level

The validator will produce reports containing issues with folowing severity levels:

• Critical
• Serious
• Moderate
• Minor
• Needs Review
• Best Practice

Filtering based on Issue Type

As of now, the validator will produce reports with the following Issue types:

• Accessible Name
• Color Contrast
• Interactable Role
• Accessibility Not Enabled
• Tappable Area
• Type In Label
• Label Capitalization
• Special Characters
• Sentence Like Label
• Duplicate Name
• Colliding Controls
• State In Name
• Conflicting State In Name
• Invalid Labeling Attribute
• Alternative Text
• Instruction In Name
• Focus Trap
• Primary Context Has Title

Examples:

1# In this example we will filter in all needs review
2report_filter_include = ReportFilter([Severity.needs_review])
3ev_config = EvincedConfig(include_filters=[report_filter_include])
4init_options = InitOptions(evinced_config=ev_config)
5
6with EvincedAppiumDefaultRunner(driver, init_options) as runner:
7    runner.report() # <- this will produce reports with isssues that only have NeedsReview severity level

1# In this example we will filter out all critical and serious
2report_filter_exclude = ReportFilter(severity_filters=[Severity.critical, Severity.serious])
3ev_config = EvincedConfig(exclude_filters=[report_filter_exclude])
4init_options = InitOptions(evinced_config=ev_config)
5
6with EvincedAppiumDefaultRunner(driver, init_options) as runner:
7    runner.report() # <- the report will contain only critical serious

1# In this example we will filter out all critical and serious and other issues
2# That is not Color Contrast issue type
3report_filter_exclude = ReportFilter(severity_filters=[Severity.critical, Severity.serious])
4report_filter_include = ReportFilter(issue_type_filters=[IssueType.color_contrast])
5ev_config = EvincedConfig(exclude_filters=[report_filter_exclude])
6init_options = InitOptions(evinced_config=ev_config)
7
8with EvincedAppiumDefaultRunner(driver, init_options) as runner:
9    runner.report()

RulesConfig

RulesConfig is designed to provide configuration for some of the aspects of reporting. Most notably ColorContrast and TappableArea.

TappableArea controls the threshold for the raising of TappableArea violation. More information can be found here tappable area. For even more information please consult with WCAG guidelines.

1rule_options = TappableArea(conformance_level="AAA")
2rules_config = [RulesConfig(IssueType.tappable_area, False, rule_options.to_dict())]
3options = InitOptions(rules_config=rules_config)

ColorContrast also controls the threshold for raising color contrast issues, for different text sizes and also controls if OCR is enabled. More information is found here

1rule_options = ColorContrast(disable_ocr=True, ratio_large_text=35.0)
2rules_config = [RulesConfig(IssueType.color_contrast, False, rule_options.to_dict())]
3options = InitOptions(rules_config=rules_config)

AdditionalOptions

Additional Options provide additional configuration information for the validator

Here's the list what you can pass and their default values: exportNoIssuesElements: bool = False exportNoIssuesElementsPath: str = None exportMeaningfulLabels: bool = None exportMeaningfulLabelsPath: str = None isAnalyticsEnabled: bool = True disableAnimations: bool = False

example:

1# check this module for other const values
2from evinced_appium_sdk.core.models.additional_options import (
3    AdditionalOptions,
4    IS_ANALYTICS_ENABLED,
5)
6
7add_options = AdditionalOptions()
8add_options.add_additional_option(IS_ANALYTICS_ENABLED, True)

Skip export of empty accessibility reports

In order to skip creation of empty accessibility reports (ones having 0 issues) import the ReportSavingStrategy enum from the SDK package and use it to specify the following flag in the options:

1from evinced_appium_sdk.core.models.enums import ReportSavingStrategy
2
3init_options = InitOptions(report_saving_strategy: ReportSavingStrategy.SKIP_EMPTY)
4
5with EvincedAppiumDefaultRunner(driver, init_options) as runner:
6    runner.report()

This option will either filter out the empty (having 0 issues) scans from the resulting report or will prevent the entire report file from being saved in case all the scans in the report are empty.

Other values of the ReportSavingStrategy are

• SAVE_ALL. This option reflects the default behavior when all reports/scans including the empty ones get exported. This option doesn't have to be specified explicitly.

• SKIP_FILES. This option switches off saving of all the reports files in the file system. The platform uploading will still happen if it is enabled with the corresponding settings.

Exporting elements with no issues

To export a JSON report of all scanned elements that had no issues found, simply add the additional options flag:

1additional_options = AdditionalOptions()
2additional_options.add_additional_option("exportNoIssuesElements", True)
3init_options = InitOptions(additional_options=additional_options)

You should see a new JSON file created beside other report files with a default name.

The structure of this JSON file is described in our Mobile Reports page.

To change the default folder destination and the file name of the elements with no issues JSON report you can use another additional option with an absolute folder path:

1additional_options = AdditionalOptions()
2additional_options.add_additional_option("exportNoIssuesElementsPath", "absolute_path/evinced_reports/SpecificNoIssuesElements.json")
3init_options = InitOptions(additional_options=additional_options)

You can also obtain elements with no issues JSON report on test runtime using sub_report:

1additional_options = AdditionalOptions()
2additional_options.add_additional_option("exportNoIssuesElements", True)
3init_options = InitOptions(additional_options=additional_options)
4
5with EvincedAppiumDefaultRunner(driver, init_options) as runner:
6    report = runner.report()
7    no_issue_elements = report.sub_report.no_issue_elements.items

However, providing additional option exportNoIssuesElements with False value and exportNoIssuesElementsPath option with a path will result in not exporting the NoIssuesElements:

1additional_options = AdditionalOptions()
2additional_options.add_additional_option("exportNoIssuesElements", False)
3additional_options.add_additional_option("exportNoIssuesElementsPath", "absolute_path/evinced_reports/SpecificNoIssuesElements.json")
4init_options = InitOptions(additional_options=additional_options)
5
6with EvincedAppiumDefaultRunner(driver, init_options) as runner:
7    report = runner.report()
8    assert report.sub_report is None

Uploading reports to Evinced Platform

[Evinced Platform] allows you to seamlessly collect, organize, and visualize Evinced accessibility reports in one place. In this section, we will guide you through the key functionalities of the upload methods from Evinced Python Appium SDK, which was introduced in version 1.21.0, to the Evinced Platform. This upload method is fully compatible with the previous versions of the Evinced Python SDK API, and is disabled by default.

Initialize report uploading to the Evinced Platform

In order to start using [Evinced Platform] feature you need to include the platform_config in the InitOptions:

1from evinced_appium_sdk.core.models.enums import UploadOption
2from evinced_appium_sdk.core.models.platform_config import PlatformConfig
3
4platform_config = PlatformConfig(upload_option=UploadOption.ENABLED_BY_DEFAULT)
5init_options = InitOptions(platform_config=platform_config)
6
7with EvincedAppiumDefaultRunner(driver, init_options) as runner:
8    report = runner.report()

In this case, all of the reports from a test class will be uploaded to the [Evinced Platform] where you can view the accessibility issues.

Selective uploading

You can configure uploading only a specific set of reports:

1from evinced_appium_sdk.core.models.enums import PlatformUpload
2
3with EvincedAppiumDefaultRunner(driver, init_options) as runner:
4    ...
5    report1 = runner.report()
6    ...
7    report2 = runner.report(platform_upload=PlatformUpload.ENABLED)

In this case, only the second report will be uploaded to [Evinced Platform].

The same behavior applies to internally stored reports report_stored() after the analyze() calls and to the stop_analyze() in the continuous mode:

1with EvincedAppiumDefaultRunner(driver, init_options) as runner:
2    runner.analyze()
3    report = runner.report_stored(platform_upload=PlatformUpload.ENABLED)

Or:

1with EvincedAppiumDefaultRunner(driver, init_options) as runner:
2    runner.start_analyze()
3    report = runner.stop_analyze(platform_upload=PlatformUpload.ENABLED)

Notice: that in these examples we didn't use InitOptions.platform_config. It's not required to use this config argument to enable the selective report uploading. Please find more information about continuous mode here if needed.

Disable report uploading to Platform

In order to disable the upload method for all reports without changing arguments for each report(), report_stored() or stop_analyze() methods you can use the following InitOption:

1platform_config = PlatformConfig(force_disable_reports_upload=True)
2init_options = InitOptions(platform_config=platform_config)
3
4with EvincedAppiumDefaultRunner(driver, init_options) as runner:
5    report1 = runner.report()
6
7    report2 = runner.report(platform_upload=PlatformUpload.ENABLED)

Labeling uploaded reports

To be able to distinguish between different reports on [Evinced Platform], the following pre-defined labels will be collected and added programmatically.

Customizing labels before uploading reports

To distinguish between reports from different parts of your applications or tests you can also add custom labels to the uploaded reports.

1with EvincedAppiumDefaultRunner(driver, init_options) as run:
2    labels1 = {"MyCustomLabelKey1": "MyCustomLabelValue1"}
3    report1 = run.report(platform_upload=PlatformUpload.ENABLED, custom_metadata=labels1)
4
5    labels2 = {"MyCustomLabelKey2": "MyCustomLabelValue2"}
6    report2 = run.report(platform_upload=PlatformUpload.ENABLED, custom_metadata=labels2)

The custom key "MyCustomLabelKey1" and the custom value "MyCustomLabelValue1" will be attached to the first report. The custom key "MyCustomLabelKey2" and the custom value "MyCustomLabelValue2" will be attached to the second report.

Customizing labels before uploading reports for the entire test class

In order to label all reports inside your test class with a common key/value, use the add_test_case_metadata() method:

1class TestPlatformConfig:
2    EvincedAppiumDefaultRunner.add_test_case_metadata("MyCustomCommonLabelKey", "MyCustomCommonLabelValue")
3
4    def test_platform_config(self, driver):
5        with EvincedAppiumDefaultRunner(driver, init_options) as run:
6            labels1 = {"MyCustomLabelKey1": "MyCustomLabelValue1"}
7            report1 = run.report(platform_upload=PlatformUpload.ENABLED, custom_metadata=labels1)
8
9            labels2 = {"MyCustomLabelKey2": "MyCustomLabelValue2"}
10            report2 = run.report(platform_upload=PlatformUpload.ENABLED, custom_metadata=labels2)

In this case, all reports in the test class scope will have "MyCustomCommonLabelKey"/"MyCustomCommonLabelValue". In addition, "MyCustomLabelKey1"/"MyCustomLabelValue1" will be added only to the first report and "MyCustomLabelKey2"/"MyCustomLabelValue2" will be added only to the second report.

Test assertion

If you want to interrupt the test execution in case the report is not uploaded, it's possible to set the fail_test_on_upload_error argument of the InitOptions.platform_config:

1platform_config = PlatformConfig(fail_test_on_upload_error=True)
2init_options = InitOptions(platform_config=platform_config)
3
4with EvincedAppiumDefaultRunner(driver, init_options) as run:
5    labels1 = {"MyCustomLabelKey1": "MyCustomLabelValue1"}
6    report1 = run.report(platform_upload=PlatformUpload.ENABLED, custom_metadata=labels1)
7    # The test will fail in case of any uploading error, it can be an internal connection or routing problems

In the case of upload errors you should see an exception with a detailed error message to help understand the cause of the error and possible solutions.

List of available Platform Config options

force_disable_reports_upload

force_disable_reports_upload = True - the upload method is completely disabled. All report uploading will be disabled regardless of the upload argument:

1platform_config = PlatformConfig(force_disable_reports_upload=True)
2init_options = InitOptions(platform_config=platform_config)
3
4with EvincedAppiumDefaultRunner(driver, init_options) as runner:
5    # This report won't be uploaded to Evinced Platform, because of force_disable_reports_upload = True
6    report = runner.report(platform_upload=PlatformUpload.ENABLED)

force_disable_reports_upload = False is a default value of the upload to Platform configuration;

upload_option

UploadOption.ENABLED_BY_DEFAULT - All reports will be uploaded, except for the reports with the upload argument equals to PlatformUpload.DISABLED:

1platform_config = PlatformConfig(upload_option=UploadOption.ENABLED_BY_DEFAULT)
2init_options = InitOptions(platform_config=platform_config)
3
4with EvincedAppiumDefaultRunner(driver, init_options) as runner:
5    # This report will be uploaded to Evinced Platform
6    report1 = runner.report()
7    # This report won't be uploaded to Evinced Platform, because of PlatformUpload.DISABLED
8    report2 = runner.report(platform_upload=PlatformUpload.DISABLED)

UploadOption.DISABLED_BY_DEFAULT - Only reports that are marked with upload argument equal to PlatformUpload.ENABLED will be uploaded:

1platform_config = PlatformConfig(upload_option=UploadOption.DISABLED_BY_DEFAULT)
2init_options = InitOptions(platform_config=platform_config)
3
4with EvincedAppiumDefaultRunner(driver, init_options) as runner:
5    # This report won't be uploaded to Evinced Platform
6    report1 = runner.report()
7    # This report will be uploaded to Evinced Platform, because of PlatformUpload.ENABLED
8    report2 = runner.report(platform_upload=PlatformUpload.ENABLED)

fail_test_on_upload_error

fail_test_on_upload_error = True - Stop the test execution in case of an upload error;

fail_test_on_upload_error = False - Continues the test execution no matter of any upload errors. Default value.

The state table of the configurations is as following:

List of available Platform upload arguments

report(PlatformUpload.ENABLED) - Upload the report or reports. Will have an effect when the InitOptions.platform_config.upload_option is either UploadOption.ENABLED_BY_DEFAULT or UploadOption.DISABLED_BY_DEFAULT. Can be used as an argument is report(PlatformUpload.ENABLED), stop_analyze(PlatformUpload.ENABLED) or report_stored(PlatformUpload.ENABLED) methods;

report(PlatformUpload.DISABLED) - Don't upload the report or reports. Will have an effect when the InitOptions.platform_config.upload_option is either UploadOption.ENABLED_BY_DEFAULT or UploadOption.DISABLED_BY_DEFAULT Can be used as an argument is report(PlatformUpload.DISABLED), stop_analyze(PlatformUpload.DISABLED) or report_stored(PlatformUpload.DISABLED) methods.

Test example

Minimum workable test for the method analyze()

1from appium.options.android import UiAutomator2Options
2from appium.webdriver.common.appiumby import AppiumBy
3from evinced_appium_sdk import LicenseManager
4from evinced_appium_sdk.core.runners import (
5    EvincedAppiumDefaultRunner,
6)
7from appium import webdriver
8
9caps = UiAutomator2Options()
10caps.platform_name = 'Android'
11caps.device_name = 'API_30'
12caps.app = '/apk/your_target_app'
13
14SERVICE_ID = "your_service_id"
15API_KEY = "your_api_key_token"
16
17LicenseManager().setup_credentials(service_id=SERVICE_ID, api_key=API_KEY)
18driver = webdriver.Remote("http://localhost:4723/wd/hub", options=caps)
19
20
21def test_example():
22    with EvincedAppiumDefaultRunner(driver, init_options=None) as runner:
23        runner.analyze()  # first scan
24        driver.find_element(AppiumBy.ID, "your_view_id").click()
25        # also can be called through the created class "runner"
26        # runner.driver.find_element(AppiumBy.ID, "your_view_id").click()
27        runner.analyze()  # second scan
28        report = runner.report_stored()
29        assert report[0].total == 5 # number of issues detected in the first scan
30        assert report[1].total == 0 # number of issues detected in the second scan