Developer Documentation

Mobile Reports

Automatically Generated HTML / JSON Reports

The report and reportStored methods will automatically generate actionable HTML and JSON reports regardless of whether an assertion is implemented. The reports can be found in the specified output directory (Appium) or in the device file system.

HTML Report Example

An example of a report. Shows a screenshot of the scanned view and a table with the accessibility issues that were found on the scan.

It may look a little bit overwhelming at first glance, so let’s just try to break it down into meaningful pieces.

The HTML file encompasses several accessibility reports (one for each application state captured by EvincedEngine.analyze() call) where each report is represented as a table. The table contains all of the information which comes in handy for both debugging an issue or figuring out what the issue actually includes:

  • Type of the accessibility issue
  • Severity
  • Summary
  • Several means of the programmatic location of the element subjected to violation (e.g. accessibility ID for iOS or content description for Android)

On top of that, you can always use visual means of element location just by referring to the screenshot attached where all of the issues are highlighted.

Not all of the issues can be described fully in the scope of such a concise report. This is the reason why every accessibility issue found comes with a corresponding link to the Evinced Knowledge Base which has full-blown descriptions for every rule, along with their WCAG tags, possible remedies, and groups of users most impacted.

Since we have the HTML report mostly covered, it is a good time to think about a report format that is convenient to use with programmatic analysis. While HTML is great for human perception, it is too tedious and error-prone to parse and put into an analytical pipeline.

JSON Report Example

1[
2    {
3        "id": "a94cf289-989e-4eeb-8b8a-f2e21aba712f",
4        "elements": [
5            {
6                "id": "278ca419-e03b-4c0c-8229-f679b0fef955",
7                "index": 1,
8                "issues": [
9                    {
10                        "description": "This element has no accessible label, making it impossible to perceive and interact with when using some assistive technologies.",
11                        "summary": "This element has no accessible label, making it impossible to perceive and interact with when using some assistive technologies.",
12                        "coordinates": {
13                            "x": 8,
14                            "y": 20,
15                            "width": 41,
16                            "height": 44
17                        },
18                        "id": "9cf6ed0f-504a-4bbc-957e-ff8ea1928aad",
19                        "elementId": "278ca419-e03b-4c0c-8229-f679b0fef955",
20                        "issueType": {
21                            "id": "f6af18bf-0a5b-472b-a8da-fe3bb0096f05",
22                            "name": "Accessible Name"
23                        },
24                        "severity": {
25                            "id": "3c68eb41-9227-481d-8eec-9dc8956b19c4",
26                            "name": "Critical"
27                        },
28                        "metadata": null,
29                        "accessibilityIdentifier": null,
30                        "accessibilityLabel": null,
31                        "elementType": "Button",
32                        "viewIdResourceName": null,
33                        "contentDescription": null,
34                        "className": null
35                    },
36                    {
37                        "description": "Tappable area of this element is too small",
38                        "summary": "Tappable area of this element is too small",
39                        "coordinates": {
40                            "x": 8,
41                            "y": 20,
42                            "width": 41,
43                            "height": 44
44                        },
45                        "id": "b453f9ce-cd78-4c45-956e-758dccbf1021",
46                        "elementId": "278ca419-e03b-4c0c-8229-f679b0fef955",
47                        "issueType": {
48                            "id": "69ab72aa-59be-43ad-b622-4bb686aace81",
49                            "name": "Tappable Area"
50                        },
51                        "severity": {
52                            "id": "3c68eb41-9227-481d-8eec-9dc8956b1900",
53                            "name": "Needs Review"
54                        },
55                        "metadata": {
56                            "expectedWidth": 44.0,
57                            "actualHeight": 44.0,
58                            "actualWidth": 41.0,
59                            "expectedHeight": 44.0
60                        },
61                        "accessibilityIdentifier": null,
62                        "accessibilityLabel": null,
63                        "elementType": "Button",
64                        "viewIdResourceName": null,
65                        "contentDescription": null,
66                        "className": null
67                    }
68                ]
69            }
70            //... omitting the rest of the report for brevity
71        ]
72    }
73]

That’s a lot of data to analyze. Again, if you pay a close attention to this file you will quickly notice that this is a simple JSON object representation of the List<Report> instance generated by EvincedEngine.reportStored() method. It brings the same amount of information as HTML report does, but in a shape more suitable for automated analytics.

The Report Object

Most of the EvincedEngine methods return a special object called Report which represents accessibility report containing all of the issues found for a given application screen. EvincedEngine is capable of performing assertions against this object on its own, however, if you want to have some custom processing (e.g. filter out some kind of issues) feel free to operate directly on it.

The Report object has the following fields:

  • id - unique ID for this report
  • createdAt - creation date of this report
  • total - total number of issues found
  • elements - list of ReportElement objects
  • hasExecutionException - indicates the presence of any errors that occur during runtime before creating output files

public boolean hasIssues();

A helper method for quickly assessing whether this report contains any accessibility issues or not.

Element

Element represents one element, where Evinced SDK managed to find one or more accessibility violations. This object holds the following data:

  • id - unique ID for the entry
  • index - running index
  • issues - list of Issue objects

Issue

An Issue object represents a particular violation of some accessibility rule found on the element.

This object holds the following data:

  • id - unique ID for the problem
  • coordinates - a Coordinates object which describes physical location of the element with this problem
  • description - text description of the issue found
  • summary - summary of the issue
  • nodeId - ID of the node with the issue
  • metadata - a dictionary of type Map<String, Object> which contains some meta information relevant to aparticular violation (e.g. expected sizes for TAPPABLE_AREA_IS_TOO_SMALL accessibility rule)
  • issueType - an IssueType object which represents its formal ID along with its human-readable name
  • severity - a common Evinced Severity object which shows which level of severity the issue has

Severity

An enum for holding the programmatic identifier of different kinds of issues severity. Its members include all of the currently available severity levels:

  • SeverityType.Critical
  • SeverityType.Serious
  • SeverityType.Moderate
  • SeverityType.Minor
  • SeverityType.BestPractice
  • SeverityType.NeedsReview

Members of this enum contain the following fields:

  • id - a numeric ID of the severity
  • name - human-readable name of the severity

IssueType

An enum for holding the programmatic identifier of different kinds of accessibility violations. Its members include all of the currently available accessibility rules:

  • A11yRule.AccessibleName
  • A11yRule.ColorContrast
  • A11yRule.OffTheAccessibilityTree
  • A11yRule.InteractableRole
  • A11yRule.SmallTappableArea
  • A11yRule.SentenceLikeLabel
  • A11yRule.RedundantTypeDeclaration
  • A11yRule.SpecialCharacters
  • A11yRule.IndistinctiveNames
  • A11yRule.CollidingControls
  • A11yRule.LabelsCapitalization

Members of this enum contain the following fields:

  • id - a numeric ID of the issue
  • name - human-readable name of the issue

Meaningful labels report

1{
2  "createdAt": "2023-11-15T08:50:09.852270654Z",
3  "id": "edce9021-f0cd-4724-8a1f-0b62f94d5a6c",
4  "type": "meaningful_labels",
5  "payload": {
6    "scans": [
7      {
8        "id": "60129c66-7a98-450b-b61e-5b774fcf9039",
9        "createdAt": "2023-11-15T08:50:09.848718113Z",
10        "meaningfulLabels": {
11          "buttons": {
12            "severity": "Critical",
13            "description": "Alternative text for a button should describe it so that a user can understand exactly what it does before clicking on it.",
14            "criteria": [
15              {
16                "type": "wcag",
17                "title": "Non-text Content",
18                "link": "https://www.w3.org/TR/WCAG22/#non-text-content",
19                "version": "2.0",
20                "criteria": "1.1.1",
21                "level": "A"
22              },
23              {
24                "type": "wcag",
25                "title": "Name, Role, Value",
26                "link": "https://www.w3.org/TR/WCAG22/#name-role-value",
27                "version": "2.0",
28                "criteria": "4.1.2",
29                "level": "A"
30              }
31            ],
32            "items": [
33              {
34                "id": "053fed8d-eea5-4e1c-8114-d4bd36f14392",
35                "coordinates": {
36                  "height": 29.0,
37                  "width": 172.0,
38                  "x": 110.0,
39                  "y": 80.0
40                },
41                "elementName": "android.widget.Button",
42                "accessibleName": "",
43                "viewId": "com.example.demoapp:id/btnColorContrast",
44                "snapshotIndex": 1
45              },
46              {
47                "id": "032dba7a-2121-48de-bce3-ba91d568eaee",
48                "coordinates": {
49                  "height": 45.0,
50                  "width": 116.0,
51                  "x": 16.0,
52                  "y": 109.0
53                },
54                "elementName": "android.widget.Button",
55                "accessibleName": "",
56                "viewId": "com.example.demoapp:id/btnGoneText",
57                "snapshotIndex": 2
58              }
59            ]
60          },
61          "images": {
62            "severity": "Serious",
63            "description": "Alternative text for an image should describe the image and how it fits into the context of the screen or page.",
64            "criteria": [
65              {
66                "type": "wcag",
67                "title": "Non-text Content",
68                "link": "https://www.w3.org/TR/WCAG22/#non-text-content",
69                "version": "2.0",
70                "criteria": "1.1.1",
71                "level": "A"
72              }
73            ],
74            "items": [
75              {
76                "id:": "9779d429-de2d-4a72-8d6a-3aa330712541",
77                "coordinates": {
78                  "left": 0,
79                  "top": 0,
80                  "right": 392,
81                  "bottom": 829
82                },
83                "elementName": "android.widget.ImageButton1",
84                "alternativeText": "Navigate1",
85                "viewId": "com.example.demoapp:id/navigationImg1",
86                "snapshotIndex": 3
87              },
88              {
89                "id:": "9779d429-de2d-4a72-8d6a-3aa330712541",
90                "coordinates": {
91                  "left": 0,
92                  "top": 0,
93                  "right": 392,
94                  "bottom": 829
95                },
96                "elementName": "android.widget.ImageButton2",
97                "alternativeText": "Navigate2",
98                "viewId": "com.example.demoapp:id/navigationImg2",
99                "snapshotIndex": 2
100              }
101            ]
102          }
103        }
104      }
105    ]
106  }
107}

Feel free to explore the knowledge base to find out more about these rules.