| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 1 | # Accessibility Overview |
| 2 | |
| 3 | Accessibility means ensuring that all users, including users with disabilities, |
| 4 | have equal access to software. One piece of this involves basic design |
| 5 | principles such as using appropriate font sizes and color contrast, |
| 6 | avoiding using color to convey important information, and providing keyboard |
| 7 | alternatives for anything that is normally accomplished with a pointing device. |
| 8 | However, when you see the word "accessibility" in a directory name in Chromium, |
| 9 | that code's purpose is to provide full access to Chromium's UI via external |
| 10 | accessibility APIs that are utilized by assistive technology. |
| 11 | |
| 12 | **Assistive technology** here refers to software or hardware which |
| 13 | makes use of these APIs to create an alternative interface for the user to |
| 14 | accommodate some specific needs, for example: |
| 15 | |
| 16 | Assistive technology includes: |
| 17 | |
| 18 | * Screen readers for blind users that describe the screen using |
| 19 | synthesized speech or braille |
| 20 | * Voice control applications that let you speak to the computer, |
| Anastasia Helfinstein | 4cc18b2d | 2019-11-12 18:22:00 | [diff] [blame] | 21 | * Switch Access that lets you control the computer with a small number |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 22 | of physical switches, |
| 23 | * Magnifiers that magnify a portion of the screen, and often highlight the |
| 24 | cursor and caret for easier viewing, and |
| 25 | * Assistive learning and literacy software that helps users who have a hard |
| 26 | time reading print, by highlighting and/or speaking selected text |
| 27 | |
| 28 | In addition, because accessibility APIs provide a convenient and universal |
| 29 | way to explore and control applications, they're often used for automated |
| 30 | testing scripts, and UI automation software like password managers. |
| 31 | |
| 32 | Web browsers play an important role in this ecosystem because they need |
| 33 | to not only provide access to their own UI, but also provide access to |
| 34 | all of the content of the web. |
| 35 | |
| 36 | Each operating system has its own native accessibility API. While the |
| 37 | core APIs tend to be well-documented, it's unfortunately common for |
| 38 | screen readers in particular to depend on additional undocumented or |
| 39 | vendor-specific APIs in order to fully function, especially with web |
| 40 | browsers, because the standard APIs are insufficient to handle the |
| 41 | complexity of the web. |
| 42 | |
| 43 | Chromium needs to support all of these operating system and |
| 44 | vendor-specific accessibility APIs in order to be usable with the full |
| 45 | ecosystem of assistive technology on all platforms. Just like Chromium |
| 46 | sometimes mimics the quirks and bugs of older browsers, Chromium often |
| 47 | needs to mimic the quirks and bugs of other browsers' implementation |
| 48 | of accessibility APIs, too. |
| 49 | |
| 50 | ## Concepts |
| 51 | |
| 52 | While each operating system and vendor accessibility API is different, |
| 53 | there are some concepts all of them share. |
| 54 | |
| 55 | 1. The *tree*, which models the entire interface as a tree of objects, exposed |
| 56 | to assistive technology via accessibility APIs; |
| 57 | 2. *Events*, which let assistive technology know that a part of the tree has |
| 58 | changed somehow; |
| 59 | 3. *Actions*, which come from assistive technology and ask the interface to |
| 60 | change. |
| 61 | |
| 62 | Consider the following small HTML file: |
| 63 | |
| 64 | ``` |
| 65 | <html> |
| 66 | <head> |
| 67 | <title>How old are you?</title> |
| 68 | </head> |
| 69 | <body> |
| 70 | <label for="age">Age</label> |
| 71 | <input id="age" type="number" name="age" value="42"> |
| 72 | <div> |
| 73 | <button>Back</button> |
| 74 | <button>Next</button> |
| 75 | </div> |
| 76 | </body> |
| 77 | </html> |
| 78 | ``` |
| 79 | |
| 80 | ### The Accessibility Tree and Accessibility Attributes |
| 81 | |
| 82 | Internally, Chromium represents the accessibility tree for that web page |
| 83 | using a data structure something like this: |
| 84 | |
| 85 | ``` |
| 86 | id=1 role=WebArea name="How old are you?" |
| 87 | id=2 role=Label name="Age" |
| 88 | id=3 role=TextField labelledByIds=[2] value="42" |
| 89 | id=4 role=Group |
| 90 | id=5 role=Button name="Back" |
| 91 | id=6 role=Button name="Next" |
| 92 | ``` |
| 93 | |
| 94 | Note that the tree structure closely resembles the structure of the |
| 95 | HTML elements, but slightly simplified. Each node in the accessibility |
| 96 | tree has an ID and a role. Many have a name. The text field has a value, |
| 97 | and instead of a name it has labelledByIds, which indicates that its |
| 98 | accessible name comes from another node in the tree, the label node |
| 99 | with id=2. |
| 100 | |
| 101 | On a particular platform, each node in the accessibility tree is implemented |
| 102 | by an object that conforms to a particular protocol. |
| 103 | |
| 104 | On Windows, the root node implements the IAccessible protocol and |
| 105 | if you call IAccessible::get_accRole, it returns ROLE_SYSTEM_DOCUMENT, |
| 106 | and if you call IAccessible::get_accName, it returns "How old are you?". |
| 107 | Other methods let you walk the tree. |
| 108 | |
| 109 | On macOS, the root node implements the NSAccessibility protocol and |
| 110 | if you call [NSAccessibility accessibilityRole], it returns @"AXWebArea", |
| 111 | and if you call [NSAccessibility accessibilityLabel], it returns |
| 112 | "How old are you?". |
| 113 | |
| 114 | The Linux accessibility API, ATK, is more similar to the Windows APIs; |
| 115 | they were developed together. (Chrome's support for desktop Linux |
| 116 | accessibility is unfinished.) |
| 117 | |
| 118 | The Android accessibility API is of course based on Java. The main |
| 119 | data structure is AccessibilityNodeInfo. It doesn't have a role, but |
| 120 | if you call AccessibilityNodeInfo.getClassName() on the root node |
| 121 | it returns "android.webkit.WebView", and if you call |
| 122 | AccessibilityNodeInfo.getContentDescription() it returns "How old are you?". |
| 123 | |
| 124 | On Chrome OS, we use our own accessibility API that closely maps to |
| 125 | Chrome's internal accessibility API. |
| 126 | |
| 127 | So while the details of the interface vary, the underlying concepts are |
| 128 | similar. Both IAccessible and NSAccessibility have a concept of a role, |
| 129 | but IAccessible uses a role of "document" for a web page, while NSAccessibility |
| 130 | uses a role of "web area". Both IAccessible and NSAccessibility have a |
| 131 | concept of the primary accessible text for a node, but IAccessible calls |
| 132 | it the "name" while NSAccessibility calls it the "label", and Android |
| 133 | calls it a "content description". |
| 134 | |
| 135 | **Historical note:** The internal names of roles and attributes in |
| 136 | Chrome often tend to most closely match the macOS accessibility API |
| 137 | because Chromium was originally based on WebKit, where most of the |
| 138 | accessibility code was written by Apple. Over time we're slowly |
| 139 | migrating internal names to match what those roles and attributes are |
| 140 | called in web accessibility standards, like ARIA. |
| 141 | |
| 142 | ### Accessibility Events |
| 143 | |
| 144 | In Chromium's internal terminology, an Accessibility Event always represents |
| 145 | communication from the app to the assistive technology, indicating that the |
| 146 | accessibility tree changed in some way. |
| 147 | |
| 148 | As an example, if the user were to press the Tab key and the text |
| 149 | field from the example above became focused, Chromium would fire a |
| 150 | "focus" accessibility event that assistive technology could listen |
| 151 | to. A screen reader might then announce the name and current value of |
| 152 | the text field. A magnifier might zoom the screen to its bounding |
| 153 | box. If the user types some text into the text field, Chromium would |
| 154 | fire a "value changed" accessibility event. |
| 155 | |
| 156 | As with nodes in the accessibility tree, each platform has a slightly different |
| 157 | API for accessibility events. On Windows we'd fire EVENT_OBJECT_FOCUS for |
| 158 | a focus change, and on Mac we'd fire @"AXFocusedUIElementChanged". |
| 159 | Those are pretty similar. Sometimes they're quite different - to support |
| 160 | live regions (notifications that certain key parts of a web page have changed), |
| 161 | on Mac we simply fire @"AXLiveRegionChanged", but on Windows we need to |
| 162 | fire IA2_EVENT_TEXT_INSERTED and IA2_EVENT_TEXT_REMOVED events individually |
| 163 | on each affected node within the changed region, with additional attributes |
| 164 | like "container-live:polite" to indicate that the affected node was part of |
| 165 | a live region. This discussion is not meant to explain all of the technical |
| 166 | details but just to illustrate that the concepts are similar, |
| 167 | but the details of notifying software on each platform about changes can |
| 168 | vary quite a bit. |
| 169 | |
| 170 | ### Accessibility Actions |
| 171 | |
| 172 | Each native object that implements a platform's native accessibility API |
| 173 | supports a number of actions, which are requests from the assistive |
| 174 | technology to control or change the UI. This is the opposite of events, |
| 175 | which are messages from Chromium to the assistive technology. |
| 176 | |
| 177 | For example, if the user had a voice control application running, such as |
| 178 | Voice Access on Android, the user could just speak the name of one of the |
| 179 | buttons on the page, like "Next". Upon recognizing that text and finding |
| 180 | that it matches one of the UI elements on the page, the voice control |
| 181 | app executes the action to click the button id=6 in Chromium's accessibility |
| 182 | tree. Internally we call that action "do default" rather than click, since |
| 183 | it represents the default action for any type of control. |
| 184 | |
| 185 | Other examples of actions include setting focus, changing the value of |
| 186 | a control, and scrolling the page. |
| 187 | |
| 188 | ### Parameterized attributes |
| 189 | |
| 190 | In addition to accessibility attributes, events, and actions, native |
| 191 | accessibility APIs often have so-called "parameterized attributes". |
| 192 | The most common example of this is for text - for example there may be |
| 193 | a function to retrieve the bounding box for a range of text, or a |
| 194 | function to retrieve the text properties (font family, font size, |
| 195 | weight, etc.) at a specific character position. |
| 196 | |
| 197 | Parameterized attributes are particularly tricky to implement because |
| Katie Dektar | 4a165e1b | 2017-09-27 16:15:13 | [diff] [blame] | 198 | of Chromium's multi-process architecture. More on this below. |
| 199 | |
| 200 | ### Tools for inspecting the Accessibility tree |
| 201 | |
| 202 | Developers can inspect the accessibility tree in several ways: |
| 203 | |
| 204 | * By navigating to [chrome://accessibility/](chrome://accessibility) |
| 205 | and inspecting a tree directly. Note that you may want to enable the |
| 206 | 'Internal' option. Click 'show accessibility tree' for a particular tab, |
| 207 | then click again to refresh that tree. |
| 208 | * Using the [https://developer.chrome.com/extensions/automation]( |
| 209 | Automation API). |
| 210 | * Installing the [https://github.com/google/automation-inspector]( |
| 211 | Automation Inspector Chrome extension). |
| 212 | * Or by using native tools: |
| 213 | |
| 214 | - Android: UIAutomatorViewer |
| 215 | - macOS: Accessibility Inspector |
| 216 | - Windows: Inspect, AViewer, accProbe (and many others) |
| 217 | |
| Kevin Babbitt | 805b781 | 2021-06-14 18:18:16 | [diff] [blame] | 218 | ### Supported Platforms and APIs |
| 219 | |
| 220 | * Windows: IAccessible (also known as Microsoft Active Accessibility or MSAA), |
| 221 | IAccessible2, [UI Automation](uiautomation.md). Chromium also supports |
| 222 | [mapping between IAccessible2 and UI Automation nodes](ia2_to_uia.md). |
| 223 | * Mac: NSAccessibility |
| 224 | * Linux: ATK |
| 225 | * Android: [AccessibilityNodeInfo and AccessibilityNodeProvider](android.md) |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 226 | |
| 227 | ## Chromium's multi-process architecture |
| 228 | |
| 229 | Native accessibility APIs tend to have a *functional* interface, where |
| 230 | Chromium implements an interface for a canonical accessible object that |
| 231 | includes methods to return various attributes, walk the tree, or perform |
| 232 | an action like click(), focus(), or setValue(...). |
| 233 | |
| 234 | In contrast, the web has a largely *declarative* interface. The shape |
| 235 | of the accessibility tree is determined by the DOM tree (occasionally |
| 236 | influenced by CSS), and the accessible semantics of a DOM element can |
| 237 | be modified by adding ARIA attributes. |
| 238 | |
| 239 | One important complication is that all of these native accessibility APIs |
| 240 | are *synchronous*, while Chromium is multi-process, with the contents of |
| 241 | each web page living in a different process than the process that |
| 242 | implements Chromium's UI and the native accessibility APIs. Furthermore, |
| 243 | the renderer processes are *sandboxed*, so they can't implement |
| 244 | operating system APIs directly. |
| 245 | |
| 246 | If you're unfamiliar with Chrome's multi-process architecture, see |
| 247 | [this blog post introducing the concept]( |
| 248 | https://blog.chromium.org/2008/09/multi-process-architecture.html) or |
| 249 | [the design doc on chromium.org]( |
| 250 | https://www.chromium.org/developers/design-documents/multi-process-architecture) |
| 251 | for an intro. |
| 252 | |
| 253 | Chromium's multi-process architecture means that we can't implement |
| 254 | accessibility APIs the same way that a single-process browser can - |
| 255 | namely, by calling directly into the DOM to compute the result of each |
| 256 | API call. For example, on some operating systems there might be an API |
| 257 | to get the bounding box for a particular range of characters on the |
| 258 | page. In other browsers, this might be implemented by creating a DOM |
| 259 | selection object and asking for its bounding box. |
| 260 | |
| 261 | That implementation would be impossible in Chromium because it'd require |
| 262 | blocking the main thread while waiting for a response from the renderer |
| 263 | process that implements that web page's DOM. (Not only is blocking the |
| 264 | main thread strictly disallowed, but the latency of doing this for every |
| 265 | API call makes it prohibitively slow anyway.) Instead, Chromium takes an |
| 266 | approach where a representation of the entire accessibility tree is |
| 267 | cached in the main process. Great care needs to be taken to ensure that |
| 268 | this representation is as concise as possible. |
| 269 | |
| 270 | In Chromium, we build a data structure representing all of the |
| 271 | information for a web page's accessibility tree, send the data |
| 272 | structure from the renderer process to the main browser process, cache |
| 273 | it in the main browser process, and implement native accessibility |
| 274 | APIs using solely the information in that cache. |
| 275 | |
| 276 | As the accessibility tree changes, tree updates and accessibility events |
| 277 | get sent from the renderer process to the browser process. The browser |
| 278 | cache is updated atomically in the main thread, so whenever an external |
| 279 | client (like assistive technology) calls an accessibility API function, |
| 280 | we're always returning something from a complete and consistent snapshot |
| 281 | of the accessibility tree. From time to time, the cache may lag what's |
| 282 | in the renderer process by a fraction of a second. |
| 283 | |
| 284 | Here are some of the specific challenges faced by this approach and |
| 285 | how we've addressed them. |
| 286 | |
| 287 | ### Sparse data |
| 288 | |
| 289 | There are a *lot* of possible accessibility attributes for any given |
| 290 | node in an accessibility tree. For example, there are more than 150 |
| 291 | unique accessibility API methods that Chrome implements on the Windows |
| 292 | platform alone. We need to implement all of those APIs, many of which |
| 293 | request rather rare or obscure attributes, but storing all possible |
| 294 | attribute values in a single struct would be quite wasteful. |
| 295 | |
| 296 | To avoid each accessible node object containing hundreds of fields the |
| 297 | data for each accessibility node is stored in a relatively compact |
| 298 | data structure, ui::AXNodeData. Every AXNodeData has an integer ID, a |
| 299 | role enum, and a couple of other mandatory fields, but everything else |
| 300 | is stored in attribute arrays, one for each major data type. |
| 301 | |
| 302 | ``` |
| 303 | struct AXNodeData { |
| 304 | int32_t id; |
| Dominic Mazzoni | dcef1b73 | 2018-01-26 17:57:04 | [diff] [blame] | 305 | ax::mojom::Role role; |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 306 | ... |
| Dominic Mazzoni | dcef1b73 | 2018-01-26 17:57:04 | [diff] [blame] | 307 | std::vector<std::pair<ax::mojom::StringAttribute, std::string>> string_attributes; |
| 308 | std::vector<std::pair<ax::mojom::IntAttribute, int32_t>> int_attributes; |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 309 | ... |
| 310 | } |
| 311 | ``` |
| 312 | |
| 313 | So if a text field has a placeholder attribute, we can store |
| 314 | that by adding an entry to `string_attributes` with an attribute |
| Dominic Mazzoni | dcef1b73 | 2018-01-26 17:57:04 | [diff] [blame] | 315 | of ax::mojom::StringAttribute::kPlaceholder and the placeholder string as the value. |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 316 | |
| 317 | ### Incremental tree updates |
| 318 | |
| 319 | Web pages change frequently. It'd be terribly inefficient to send a |
| 320 | new copy of the accessibility tree every time any part of it changes. |
| 321 | However, the accessibility tree can change shape in complicated ways - |
| 322 | for example, whole subtrees can be reparented dynamically. |
| 323 | |
| 324 | Rather than writing code to deal with every possible way the |
| 325 | accessibility tree could be modified, Chromium has a general-purpose |
| 326 | tree serializer class that's designed to send small incremental |
| 327 | updates of a tree from one process to another. The tree serializer has |
| 328 | just a few requirements: |
| 329 | |
| 330 | * Every node in the tree must have a unique integer ID. |
| 331 | * The tree must be acyclic. |
| 332 | * The tree serializer must be notified when a node's data changes. |
| 333 | * The tree serializer must be notified when the list of child IDs of a |
| 334 | node changes. |
| 335 | |
| 336 | The tree serializer doesn't know anything about accessibility attributes. |
| 337 | It keeps track of the previous state of the tree, and every time the tree |
| 338 | structure changes (based on notifications of a node changing or a node's |
| 339 | children changing), it walks the tree and builds up an incremental tree |
| 340 | update that serializes as few nodes as possible. |
| 341 | |
| 342 | In the other process, the Unserialization code applies the incremental |
| 343 | tree update atomically. |
| 344 | |
| 345 | ### Text bounding boxes |
| 346 | |
| 347 | One challenge faced by Chromium is that accessibility clients want to be |
| 348 | able to query the bounding box of an arbitrary range of text - not necessarily |
| 349 | just the current cursor position or selection. As discussed above, it's |
| 350 | not possible to block Chromium's main browser process while waiting for this |
| 351 | information from Blink, so instead we cache enough information to satisfy these |
| 352 | queries in the accessibility tree. |
| 353 | |
| 354 | To compactly store the bounding box of every character on the page, we |
| 355 | split the text into *inline text boxes*, sometimes called *text runs*. |
| 356 | For example, in a typical paragraph, each line of text would be its own |
| 357 | inline text box. In general, an inline text box or text run contians a |
| 358 | sequence of text characters that are all oriented in the same direction, |
| 359 | in a line, with the same font, size, and style. |
| 360 | |
| 361 | Each inline text box stores its own bounding box, and then the relative |
| 362 | x-coordinate of each character in its text (assuming left-to-right). |
| 363 | From that it's possible to compute the bounding box |
| 364 | of any individual character. |
| 365 | |
| 366 | The inline text boxes are part of Chromium's internal accessibility tree. |
| 367 | They're used purely internally and aren't ever exposed directly via any |
| 368 | native accessibility APIs. |
| 369 | |
| 370 | For example, suppose that a document contains a text field with the text |
| 371 | "Hello world", but the field is narrow, so "Hello" is on the first line and |
| 372 | "World" is on the second line. Internally Chromium's accessibility tree |
| 373 | might look like this: |
| 374 | |
| 375 | ``` |
| 376 | staticText location=(8, 8) size=(38, 36) name='Hello world' |
| 377 | inlineTextBox location=(0, 0) size=(36, 18) name='Hello ' characterOffsets=12,19,23,28,36 |
| 378 | inlineTextBox location=(0, 18) size=(38, 18) name='world' characterOffsets=12,20,25,29,37 |
| 379 | ``` |
| 380 | |
| 381 | ### Scrolling, transformations, and animation |
| 382 | |
| 383 | Native accessibility APIs typically want the bounding box of every element in the |
| 384 | tree, either in window coordinates or global screen coordinates. If we |
| 385 | stored the global screen coordinates for every node, we'd be constantly |
| 386 | re-serializing the whole tree every time the user scrolls or drags the |
| 387 | window. |
| 388 | |
| 389 | Instead, we store the bounding box of each node in the accessibility tree |
| 390 | relative to its *offset container*, which can be any ancestor. If no offset |
| 391 | container is specified, it's assumed to be the root of the tree. |
| 392 | |
| 393 | In addition, any offset container can contain scroll offsets, which can be |
| 394 | used to scroll the bounding boxes of anything in that subtree. |
| 395 | |
| 396 | Finally, any offset container can also include an arbitrary 4x4 transformation |
| 397 | matrix, which can be used to represent arbitrary 3-D rotations, translations, and |
| 398 | scaling, and more. The transformation matrix applies to the whole subtree. |
| 399 | |
| 400 | Storing coordinates this way means that any time an object scrolls, moves, or |
| 401 | animates its position and scale, only the root of the scrolling or animation |
| 402 | needs to post updates to the accessibility tree. Everything in the subtree |
| 403 | remains valid relative to that offset container. |
| 404 | |
| 405 | Computing the global screen coordinates for an object in the accessibility |
| 406 | tree just means walking up its ancestor chain and applying offsets and |
| 407 | occasionally multiplying by a 4x4 matrix. |
| 408 | |
| 409 | ### Site isolation / out-of-process iframes |
| 410 | |
| 411 | At one point in time, all of the content of a single Tab or other web view |
| 412 | was contained in the same Blink process, and it was possible to serialize |
| 413 | the accessibility tree for a whole frame tree in a single pass. |
| 414 | |
| 415 | Today the situation is a bit more complicated, as Chromium supports |
| 416 | out-of-process iframes. (It also supports "browser plugins" such as |
| 417 | the `<webview>` tag in Chrome packaged apps, which embeds a whole |
| 418 | browser inside a browser, but for the purposes of accessibility this |
| 419 | is handled the same as frames.) |
| 420 | |
| 421 | Rather than a mix of in-process and out-of-process frames that are handled |
| 422 | differently, Chromium builds a separate independent accessibility tree |
| 423 | for each frame. Each frame gets its own tree ID, and it keeps track of |
| 424 | the tree ID of its parent frame (if any) and any child frames. |
| 425 | |
| 426 | In Chrome's main browser process, the accessibility trees for each frame |
| 427 | are cached separately, and when an accessibility client (assistive |
| 428 | technology) walks the accessibility tree, Chromium dynamically composes |
| 429 | all of the frames into a single virtual accessibility tree on the fly, |
| 430 | using those aforementioned tree IDs. |
| 431 | |
| 432 | The node IDs for accessibility trees only need to be unique within a |
| 433 | single frame. Where necessary, separate unique IDs are used within |
| 434 | Chrome's main browser process. In Chromium accessibility, a "node ID" |
| 435 | always means that ID that's only unique within a frame, and a "unique ID" |
| 436 | means an ID that's globally unique. |
| 437 | |
| 438 | ## Blink |
| 439 | |
| 440 | Blink constructs an accessibility tree (a hierarchy of [WebAXObject]s) from the |
| sashab | b498325a | 2017-05-17 01:18:46 | [diff] [blame] | 441 | page it is rendering. WebAXObject is the public API wrapper around [AXObject], |
| 442 | which is the core class of Blink's accessibility tree. AXObject is an abstract |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 443 | class; the most commonly used concrete subclass of it is [AXNodeObject], which |
| 444 | wraps a [Node]. In turn, most AXNodeObjects are actually [AXLayoutObject]s, |
| 445 | which wrap both a [Node] and a [LayoutObject]. Access to the LayoutObject is |
| sashab | b498325a | 2017-05-17 01:18:46 | [diff] [blame] | 446 | important because some elements are only in the AXObject tree depending on their |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 447 | visibility, geometry, linewrapping, and so on. There are some subclasses of |
| 448 | AXLayoutObject that implement special-case logic for specific types of Node. |
| sashab | b498325a | 2017-05-17 01:18:46 | [diff] [blame] | 449 | There are also other subclasses of AXObject, which are mostly used for testing. |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 450 | |
| 451 | Note that not all AXLayoutObjects correspond to actual Nodes; some are synthetic |
| 452 | layout objects which group related inline elements or similar. |
| 453 | |
| 454 | The central class responsible for dealing with accessibility events in Blink is |
| 455 | [AXObjectCacheImpl], which is responsible for caching the corresponding |
| 456 | AXObjects for Nodes or LayoutObjects. This class has many methods named |
| 457 | `handleFoo`, which are called throughout Blink to notify the AXObjectCacheImpl |
| 458 | that it may need to update its tree. Since this class is already aware of all |
| 459 | accessibility events in Blink, it is also responsible for relaying accessibility |
| 460 | events from Blink to the embedding content layer. |
| 461 | |
| 462 | ## The content layer |
| 463 | |
| 464 | The content layer lives on both sides of the renderer/browser split. The content |
| Dan Clark | 2256222 | 2021-08-24 22:39:50 | [diff] [blame] | 465 | layer translates WebAXObjects into [ui::AXNodeData]. The ui::AXNodeData class |
| 466 | and related classes are Chromium's cross-platform accessibility tree. The |
| 467 | translation is implemented in [BlinkAXTreeSource]. This translation happens on |
| 468 | the renderer side, so the ui::AXNodeData tree now needs to be sent to the |
| 469 | browser, which is done by calling the remote method |
| 470 | [ax.mojom.RenderAccessibilityHost::HandleAXEvents()] with the payload being |
| 471 | serialized delta-updates to the tree, so that changes that happen on the |
| 472 | renderer side can be reflected on the browser side. |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 473 | |
| 474 | On the browser side, these IPCs are received by [RenderFrameHostImpl], and then |
| 475 | usually forwarded to [BrowserAccessibilityManager] which is responsible for: |
| 476 | |
| 477 | 1. Merging AXNodeData trees into one tree of [BrowserAccessibility] objects, |
| 478 | by linking to other BrowserAccessibilityManagers. This is important because |
| 479 | each page has its own accessibility tree, but each Chromium *window* must |
| 480 | have only one accessibility tree, so trees from multiple pages need to be |
| 481 | combined (possibly also with trees from Views UI). |
| 482 | 2. Dispatching outgoing accessibility events to the platform's accessibility |
| 483 | APIs. This is done in the platform-specific subclasses of |
| 484 | BrowserAccessibilityManager, in a method named `NotifyAccessibilityEvent`. |
| 485 | 3. Dispatching incoming accessibility actions to the appropriate recipient, via |
| 486 | [BrowserAccessibilityDelegate]. For messages destined for a renderer, |
| 487 | [RenderFrameHostImpl], which is a BrowserAccessibilityDelegate, is |
| Mario Sanchez Prada | 52f04e7 | 2020-04-22 15:34:21 | [diff] [blame] | 488 | responsible for calling the remote method |
| 489 | [ax.mojom.RenderAccessibility.PerformAction()], implemented by the renderer, |
| 490 | with the appropriate payload (of type [ax.mojom.AXActionData]). This IPC call |
| 491 | will be received by [RenderAccessibilityManager], which will then relay on |
| 492 | the [RenderAccessibilityImpl] where the actual logic is implemented. |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 493 | |
| 494 | On Chrome OS, RenderFrameHostImpl does not route events to |
| 495 | BrowserAccessibilityManager at all, since there is no platform screenreader |
| 496 | outside Chromium to integrate with. |
| 497 | |
| 498 | ## Views |
| 499 | |
| David Tseng | 0fa4a63 | 2019-05-23 23:57:46 | [diff] [blame] | 500 | Views generates a [ViewAccessibility] for each View, which is used as the |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 501 | delegate for an [AXPlatformNode] representing that View. This part is relatively |
| 502 | straightforward, but then the generated tree must be combined with the web |
| 503 | accessibility tree, which is handled by BrowserAccessibilityManager. |
| 504 | |
| 505 | ## WebUI |
| 506 | |
| 507 | Since WebUI surfaces have renderer processes as normal, WebUI accessibility goes |
| 508 | through the blink-to-content-to-platform pipeline described above. Accessibility |
| 509 | for WebUI is largely implemented in JavaScript in [webui-js]; these classes take |
| 510 | care of adding ARIA attributes and so on to DOM nodes as needed. |
| 511 | |
| 512 | ## The Chrome OS layer |
| 513 | |
| 514 | The accessibility tree is also exposed via the [chrome.automation API], which |
| 515 | gives extension JavaScript access to the accessibility tree, events, and |
| 516 | actions. This API is implemented in C++ by [AutomationInternalCustomBindings], |
| 517 | which is renderer-side code, and in JavaScript by the [automation API]. The API |
| 518 | is defined by [automation.idl], which must be kept synchronized with |
| Chris Hall | dd6f87c | 2019-10-14 05:20:47 | [diff] [blame] | 519 | [ax_enums.mojom]. |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 520 | |
| Dominic Mazzoni | 7821334d | 2021-09-01 20:41:40 | [diff] [blame] | 521 | ## Further reading |
| 522 | |
| 523 | For more detail, read [How Chrome Accessibility Works](how_a11y_works.md). |
| 524 | |
| John Palmer | 046f987 | 2021-05-24 01:24:56 | [diff] [blame] | 525 | [ax.mojom.AXActionData]: https://source.chromium.org/chromium/chromium/src/+/main:ui/accessibility/mojom/ax_action_data.mojom;l=13 |
| 526 | [ax.mojom.RenderAccessibilityHost::HandleAXEvents()]: https://source.chromium.org/chromium/chromium/src/+/main:content/common/render_accessibility.mojom;l=47 |
| 527 | [ax.mojom.RenderAccessibility.PerformAction()]: https://source.chromium.org/chromium/chromium/src/+/main:content/common/render_accessibility.mojom;l=86 |
| Katie D | e5c2c70 | 2019-06-26 17:54:50 | [diff] [blame] | 528 | [AutomationInternalCustomBindings]: https://cs.chromium.org/chromium/src/extensions/renderer/api/automation/automation_internal_custom_bindings.h |
| Kent Tamura | 6943cf79 | 2018-04-09 05:24:54 | [diff] [blame] | 529 | [AXLayoutObject]: https://cs.chromium.org/chromium/src/third_party/blink/renderer/modules/accessibility/ax_layout_object.h |
| 530 | [AXNodeObject]: https://cs.chromium.org/chromium/src/third_party/blink/renderer/modules/accessibility/ax_node_object.h |
| Sharon Yang | 8c9500e | 2019-06-14 17:41:40 | [diff] [blame] | 531 | [AXObject]: https://cs.chromium.org/chromium/src/third_party/blink/renderer/modules/accessibility/ax_object.h |
| Kent Tamura | 6943cf79 | 2018-04-09 05:24:54 | [diff] [blame] | 532 | [AXObjectImpl]: https://cs.chromium.org/chromium/src/third_party/blink/renderer/modules/accessibility/ax_object_impl.h |
| 533 | [AXObjectCacheImpl]: https://cs.chromium.org/chromium/src/third_party/blink/renderer/modules/accessibility/ax_object_cache_impl.h |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 534 | [AXPlatformNode]: https://cs.chromium.org/chromium/src/ui/accessibility/platform/ax_platform_node.h |
| 535 | [AXTreeSerializer]: https://cs.chromium.org/chromium/src/ui/accessibility/ax_tree_serializer.h |
| 536 | [BlinkAXTreeSource]: https://cs.chromium.org/chromium/src/content/renderer/accessibility/blink_ax_tree_source.h |
| 537 | [BrowserAccessibility]: https://cs.chromium.org/chromium/src/content/browser/accessibility/browser_accessibility.h |
| 538 | [BrowserAccessibilityDelegate]: https://cs.chromium.org/chromium/src/content/browser/accessibility/browser_accessibility_manager.h?sq=package:chromium&l=64 |
| 539 | [BrowserAccessibilityManager]: https://cs.chromium.org/chromium/src/content/browser/accessibility/browser_accessibility_manager.h |
| Kent Tamura | 6943cf79 | 2018-04-09 05:24:54 | [diff] [blame] | 540 | [LayoutObject]: https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/layout/layout_object.h |
| David Tseng | 0fa4a63 | 2019-05-23 23:57:46 | [diff] [blame] | 541 | [ViewAccessibility]: https://cs.chromium.org/chromium/src/ui/views/accessibility/view_accessibility.h |
| Sharon Yang | 8c9500e | 2019-06-14 17:41:40 | [diff] [blame] | 542 | [Node]: https://cs.chromium.org/chromium/src/third_party/blink/renderer/core/dom/node.h |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 543 | [RenderAccessibilityImpl]: https://cs.chromium.org/chromium/src/content/renderer/accessibility/render_accessibility_impl.h |
| John Palmer | 046f987 | 2021-05-24 01:24:56 | [diff] [blame] | 544 | [RenderAccessibilityManager]: https://source.chromium.org/chromium/chromium/src/+/main:content/renderer/accessibility/render_accessibility_manager.h |
| danakj | d3baa13 | 2020-09-10 15:33:56 | [diff] [blame] | 545 | [RenderFrameHostImpl]: https://cs.chromium.org/chromium/src/content/browser/renderer_host/render_frame_host_impl.h |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 546 | [ui::AXNodeData]: https://cs.chromium.org/chromium/src/ui/accessibility/ax_node_data.h |
| Kent Tamura | 6943cf79 | 2018-04-09 05:24:54 | [diff] [blame] | 547 | [WebAXObject]: https://cs.chromium.org/chromium/src/third_party/blink/public/web/web_ax_object.h |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 548 | [automation API]: https://cs.chromium.org/chromium/src/chrome/renderer/resources/extensions/automation |
| Katie D | e5c2c70 | 2019-06-26 17:54:50 | [diff] [blame] | 549 | [automation.idl]: https://cs.chromium.org/chromium/src/extensions/common/api/automation.idl |
| Chris Hall | dd6f87c | 2019-10-14 05:20:47 | [diff] [blame] | 550 | [ax_enums.mojom]: https://cs.chromium.org/chromium/src/ui/accessibility/ax_enums.mojom |
| dmazzoni | 2f48975 | 2017-02-16 03:39:16 | [diff] [blame] | 551 | [chrome.automation API]: https://developer.chrome.com/extensions/automation |
| 552 | [webui-js]: https://cs.chromium.org/chromium/src/ui/webui/resources/js/cr/ui/ |
