Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates for CR #322

Closed
wants to merge 6 commits into from
Closed

Updates for CR #322

wants to merge 6 commits into from

Conversation

WilcoFiers
Copy link
Collaborator

@WilcoFiers WilcoFiers commented Jan 15, 2019

<p>Composite rule: Header cells in HTML tables (<a href="https://www.w3.org/WAI/WCAG21/quickref/#info-and-relationships" target="_blank">WCAG 2 Success Criterion 1.3.1</a>).</p>
<p>This rule uses outcomes from the following rules:</p>
<p>Composite rule: video elements have an audio description or media alternative (<a href="https://www.w3.org/WAI/WCAG21/quickref/#audio-description-or-media-alternative-prerecorded" target="_blank">WCAG 2.1, success criterion 1.2.3 Audio Description or Media Alternative</a>).</p>
<p>Each HTML `video` element meets expectations from at least one of the following rules:</p>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested edit:

Each HTML video element meets the expectations defined by at least one of the following rules:

Rationale: "meets expectations" is vague - it could mean meeting only parts of the defined expectation.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"meets all expectations"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going with "meets the expectations" as suggested. "All expectations from one rule" is a bit confusing IMO. I'm not using "as defined in", as I don't think it helps readability. This is an example, making this editorial. We can revisit it if we have to in CR.

@@ -156,7 +150,7 @@ For each accessibility requirement in the mapping, an ACT Rule MUST indicate wha
</ul></blockquote>
</div>

ACT Rules can be used to test accessibility requirements that are not part of a W3C accessibility standard, such as [[WAI-ARIA]] requirements, or a testing methodology like [RGAA 3](https://disic.github.io/rgaa_referentiel_en/criteria.html). An ACT Rule MUST indictate whether or not the [=accessibility requirement=] it maps to is required for conformance in its [=accessibility requirements document=]. Examples of accessibility requirements that are not required for conformance are WCAG sufficient techniques, or a company style guide which includes both requirements and optional "best practices". That distinction between what is required and what is optional has to be made clear.
ACT Rules can be used to test accessibility requirements that are not part of a W3C accessibility standard, such as [Hypertext Markup Language](https://www.w3.org/TR/html/) [[HTML]] requirements, or a testing methodology like [RGAA 3 2016](https://disic.github.io/rgaa_referentiel_en/criteria.html). An ACT Rule MUST indictate whether or not the [=accessibility requirement=] it maps to is required for conformance in its [=accessibility requirements document=]. Examples of accessibility requirements that are not required for conformance are WCAG sufficient techniques, or a company style guide which includes both requirements and optional "best practices". The distinction between what is required and what is optional has to be made clear.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"The distinction ... has to be made clear" - should that be a MUST requirement?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

editorial

@@ -188,7 +182,7 @@ ACT Rules can be used to test accessibility requirements that are not part of a

If the `failed` outcome can not be mapped to *not satisfied* for an accessibility requirement, that requirement MUST NOT be included in the accessibility requirements mapping. The optional [Background section](#background) could be used to list accessibility requirements and standards that may be thematically related, but for which the rule is not a failure test.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Misuse of "may" keyword. Here some suggestions:

  • could be used to list accessibility requirements and standards that are thematically related (remove "may")
  • could be used to list accessibility requirements and standards when they are thematically related (replace "may")
  • MAY be used to list accessibility requirements and standards that are thematically related (move "may" to make it a requirement)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Going with option 2. Didn't pick the third one as the background section already has a "MAY" in it.

@@ -198,20 +192,27 @@ If the rule does not map to any [=accessibility requirement=], the accessibility
</div>

<div class=note>
**Note**: While rules are often designed for one, or possibly a small collection of [=accessibility requirements document=], it is likely that other accessibility requirements documents also map to those rules. For example, rules may be written for WCAG 2.1, but many of those may map to a company's internal accessibility policy. In such a scenario, an external accessibility requirements mapping could be created. An external accessibility requirements mapping ammend the accessibility requirements mapping of a rule by adding mapping to a different accessibility requirements document. An external accessibility requirements mapping avoids duplication of a rule, for the sole purpose of changing the mapping.
**Note**: While rules are often designed for one, or possibly a small collection of [=accessibility requirements document=], it is likely that other accessibility requirements documents also map to those rules. For example, rules may be written for WCAG 2.1, but many of those may map to a company's internal accessibility policy. In such a scenario, an <dfn>external accessibility requirements mapping</dfn> could be created. An external accessibility requirements mapping ammend the accessibility requirements mapping of a rule by adding mapping to a different accessibility requirements document. An external accessibility requirements mapping avoids duplication of a rule, for the sole purpose of changing the mapping.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Replace "may" with "can" in the example.

@@ -198,20 +192,27 @@ If the rule does not map to any [=accessibility requirement=], the accessibility
</div>

<div class=note>
**Note**: While rules are often designed for one, or possibly a small collection of [=accessibility requirements document=], it is likely that other accessibility requirements documents also map to those rules. For example, rules may be written for WCAG 2.1, but many of those may map to a company's internal accessibility policy. In such a scenario, an external accessibility requirements mapping could be created. An external accessibility requirements mapping ammend the accessibility requirements mapping of a rule by adding mapping to a different accessibility requirements document. An external accessibility requirements mapping avoids duplication of a rule, for the sole purpose of changing the mapping.
**Note**: While rules are often designed for one, or possibly a small collection of [=accessibility requirements document=], it is likely that other accessibility requirements documents also map to those rules. For example, rules may be written for WCAG 2.1, but many of those may map to a company's internal accessibility policy. In such a scenario, an <dfn>external accessibility requirements mapping</dfn> could be created. An external accessibility requirements mapping ammend the accessibility requirements mapping of a rule by adding mapping to a different accessibility requirements document. An external accessibility requirements mapping avoids duplication of a rule, for the sole purpose of changing the mapping.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider making "could be created" to "MAY be created" (ie. changing it to a requirement).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't pick up this suggestion, as an external mapping is not part of the rules format. Therefore we can't make suggestions for it.

Copy link

@annethyme annethyme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of my comments:

  • Biggest concern: Having composite rules only apply to a whole test subject at a time (line 248 + 299 + 325)
  • Question: With the "Findings" definition it seems that the page-level aggregation is suddenly in again as part of the spec. Is this correct? (line 299)
  • A few things for the new "Rules input" section:
    • Using "... list" here in the headings as the only place in the spec seems a bit weird (line 67)
    • Weird that the subheadings but not the overall heading is mentioned in the rules structure (line 67+199)
    • Wishes for tightening up definitions of "Rules input " + the "Input Aspects list", see comments (line 202 + 211 + 215)
  • Right now everything refers to WCAG 2.1. Do we need to make it clearer that the same applies to WCAG 2.0, that still is the standard used in some laws, e.g. Section 508, that isn't up for an update any time soon? (see e.g. line 43)
  • plus a bunch of editorials, whenever I saw something


- <dfn>Atomic rules</dfn> describe how to test a specific type of solution. It contains a precise definition of what elements, nodes or other parts of a [=test subject=] are to be tested, and when those elements are considered to fail the rule. These rules are to be kept small and *atomic*. Meaning that atomic rules test a single "failure condition", and do so without using the [=outcome=] of other rules.
- <dfn>Atomic rules</dfn> describe how to test a specific type of solution. It contains a precise definition of what elements, nodes or other "parts" of a [=test subject=] are to be tested, and when those elements are considered to fail the rule. These rules are to be kept small and *atomic*. Meaning that atomic rules test a single "failure condition", and do so without using the [=findings=] from other rules.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"These rules are to be kept small and atomic. Meaning that atomic rules test..." --> "These rules are to be kept small and atomic, meaning that atomic rules test..."


- <dfn>Composite rules</dfn> describe how the [=outcomes=] of multiple [=atomic rules=] or composite rules can be combined into a single outcome. A composite rule can have multiple "satisfying conditions", each of these provided in separate atomic or composite rules. The logic in the composite rule describes how any one of these satisfying conditions, or some combination of them is used to determine a single outcome.
- <dfn>Composite rules</dfn> describe how the [=findings=] of multiple [=atomic rules=] or composite rules can be combined into a single [=outcome=]. A composite rule can have multiple "satisfying conditions", each of these tested in separate rules. The logic in the composite rule describes how any one of these satisfying conditions, or some combination of them is used to determine a single outcome.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing a comma?
"how any one of these satisfying conditions, or some combination of them is used to determine a single outcome" --> "how any one of these satisfying conditions, or some combination of them, is used to determine a single outcome"


<div class="example">
<p>Composite rule: Header cells in HTML tables (<a href="https://www.w3.org/WAI/WCAG21/quickref/#info-and-relationships" target="_blank">WCAG 2 Success Criterion 1.3.1</a>).</p>
<p>This rule uses outcomes from the following rules:</p>
<p>Composite rule: video elements have an audio description or media alternative (<a href="https://www.w3.org/WAI/WCAG21/quickref/#audio-description-or-media-alternative-prerecorded" target="_blank">WCAG 2.1, success criterion 1.2.3 Audio Description or Media Alternative</a>).</p>

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is both a WCAG 2.0 and 2.1 requirement. How do we expect people to communicate this? And should we show this in the examples?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should video be uppercase? Composite rule: Video elements have an audio...

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decided in the ACT TF meeting that this is not relevant for the examples.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is both a WCAG 2.0 and 2.1 requirement. How do we expect people to communicate this? And should we show this in the examples?

I don't think rules should have to quote every single version that a requirement appears in or you'll wind up with a very long list. Maybe simply have the standard without the version and a link pointer would be best.

@@ -66,7 +64,7 @@ An ACT Rule MUST consist of at least the following items:
* [Rule Description](#rule-description)
* [Rule type](#rule-types)
* [Accessibility Requirements Mapping](#accessibility-requirements-mapping)
* [Aspects Under Test](#input-aspects) (for atomic rules) OR [Input Rules List](#input-rules-list) (for composite rules)
* [Input Aspects List](#input-aspects-list) (for atomic rules) OR [Input Rules List](#input-rules-list) (for composite rules)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we use "List" here? Other items could be lists as well, e.g. the accessibility requirements, but we don't add "List" to these sections.

@@ -105,15 +103,11 @@ An ACT Rule MUST have a description that is in plain language, and provides a br
**Example**: This rule checks that the HTML page has a title

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggesting to make it a bit more specific: "This rule checks that each HTML document has a title"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree. If the test subject is a single page, you wouldn't talk about "each HTML document". Similarly, we don't say "Each img element in each HTML document". You'd use that language if you're talking about a set of web pages. So for example: "The rule checks that the title of an HTML page is unique within a set of web pages".


Each expectation MUST be distinct, unambiguous, and be written in plain language. Unlike in applicability, a certain level of subjectivity is allowed in expectations. Meaning that the expectation has only one possible meaning, but that meaning MAY not fully quantifiable.

Commonly, an expectation comes in the form of an "Each test target has ..." statement. In such an expectation, all test targets need to <dfn>meet the expectations</dfn> for the rule to pass. It is also possible to have an expectation where not all test targets meet the expectations. For example: "85% of all test targets have an accessible name" is an expectation that is true even when 15% of the test targets do not meet the expectations.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"meet the expectations" --> "meet all expectations"

act-rules-format.bs Show resolved Hide resolved

When all expectations are true for a test target, the test target `passed` the rule. If one or more expectations is false, the test target `failed` the rule. This works the same way for atomic rules. A composite rule expectation MUST NOT use information from [test aspects](#input-aspects).
All expectations of a [=composite rule=] MUST describe the logic that is used to determine a single `passed` or `failed` [=outcome=] for a [=test subject=], based on the [=findings=] of rules in its [input rules list](#input-rules-list). A composite rule expectation MUST NOT use information from [input aspects](#input-aspects-list).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"determine a single passed or failed [=outcome=] for a [=test subject=]" implies that composite rules will always apply to e.g. a whole page at a time, without possibility to pinpoint which element passed/failed.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct. We've defined outcome in relationship to test subject, not to test targets.

<li>Video elements have a media alternative (video-text-alt)</li>
</ul>
</blockquote>
<p>Composite rule: video elements have an audio description or media alternative (<a href="https://www.w3.org/WAI/WCAG21/quickref/#audio-description-or-media-alternative-prerecorded" target="_blank">WCAG 2.1, success criterion 1.2.3 Audio Description or Media Alternative</a>).</p>

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's nice that we list the SC here. Can we do that consistently in other examples too, where relevant?

</ul>
</blockquote>
<p>Composite rule: video elements have an audio description or media alternative (<a href="https://www.w3.org/WAI/WCAG21/quickref/#audio-description-or-media-alternative-prerecorded" target="_blank">WCAG 2.1, success criterion 1.2.3 Audio Description or Media Alternative</a>).</p>
<p>Each HTML `video` element meets expectations from at least one of the following rules:</p>

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be about the outcomes of the input rules here (passed/failied/inapplicable), rather than about meeting the expectations? This could open up for (potentially) picking and choosing between different expectations in the same input rule...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As above.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still say that it has to meet all expectations of at least one input rule. It can't meet half an input rule and still pass.

@moekraft
Copy link
Collaborator

Under Rule Structure, I think we should have a section for Outcome rather than a link to ACT Rules Format outcome definition which is static. This differs from Romain's suggestion in Issue #284

@annethyme
Copy link

As discussed in our last ACT TF teleconference meeting, Siteimprove was not aware that we were looking at changing the scope of outcome reporting for individual rules from reporting at the test target level to reporting at the test subject level. This change means that the rules should be reporting a “passed”/”failed” outcome for a whole page at a time instead of reporting back which specific elements/attributes/things are passing or failing the rule.

The result of this change, as we see it, are less granular results that will not give the same level of comparability across tools and methodologies that we hoped to get from the ACT Rules Format.

Our suggestion is to have an outcome mapping where either the result is “inapplicable” – no test targets were found within the test subject, or a “passed”/”failed” outcome for each test target. We don’t see a problem with this “mix”, that seems quite logical to us.

We thought that this was what we were getting, since closely reading the rules format said “test target” everywhere, and no issues or agenda items have mentioned that we were looking at switching “test target” out with “test subject” level reporting. This is the only reason we have not argued for this previously.

It seemed on the call last week that we might be the only ones with this concern though, so if we cannot revert to the previous wording, we would like to suggest that as a compromise, we mention in the ACT Rules Format, that reporting a passed/failed outcome per test subject is recommended to ensure transparency and comparability of results.

Copy link

@annethyme annethyme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please save comments prefixed "Editorial" for a rainy day after CR.

act-rules-format.bs Show resolved Hide resolved

- <dfn>Atomic rules</dfn> describe how to test a specific type of solution. It contains a precise definition of what elements, nodes or other parts of a [=test subject=] are to be tested, and when those elements are considered to fail the rule. These rules are to be kept small and *atomic*. Meaning that atomic rules test a single "failure condition", and do so without using the [=outcome=] of other rules.
- <dfn>Atomic rules</dfn> describe how to test a specific type of solution. It contains a precise definition of what elements, nodes or other "parts" of a [=test subject=] are to be tested, and when the test subject is considered to fail the rule. These rules are to be kept small and *atomic*, this means that atomic rules test a single "failure condition", and do so without using the [=findings=] from other rules.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Editorial: "These rules are to be kept small and atomic. This means that atomic rules..."


<div class=example>
<p><strong>Example:</strong> An ACT Rule that tests a best practice not included in any accessibility standard:</p>
<p>**Example:** An ACT Rule for WCAG 2.1 that tests if landmarks used to satsify [WCAG 2.1, success criterion 2.4.1 bypass blocks](https://www.w3.org/TR/WCAG21/#bypass-blocks):</p>

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Editorial:
"if landmarks used to satsify [WCAG 2.1, success criterion 2.4.1 bypass blocks]"
-->
"if landmarks are used to satisfy [WCAG 2.1, success criterion 2.4.1 Bypass Blocks]"


A [=composite rule=] uses [=outcomes=] from [=atomic rules=] or other composite rules and applies a logic to them so that a single outcome can be determined for each [=test target=]. The identifiers of all rules used in the [expectations](#expectations-composite) MUST be listed in the composite rule. The rules list describes the input for composite rules, similar to how [aspects under test](#input-aspects) describe the input for atomic rules.
A [=composite rule=] uses [=findings=] from [=atomic rules=] or other composite rules and applies a logic to them so that a single [=outcome=] can be determined for the [=test subject=]. The [identifier](#rule-identifier) and [description](#rule-description) of all rules used in the [expectations](#expectations-composite) MUST be listed in the composite rule. The input rules describes the input for composite rules, similar to how [input aspects](#input-aspects) describe the input for atomic rules.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of "description", shouldn't it be the "descriptive title" that should be included here? (see ACT Rule Structure)


The applicability describes what (parts of) the [=test subject=] are tested.
The applicability describes what (parts of the) [=test subject=] are tested.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Editorial: This sentence doesn't work without the parenthesis, suggesting it maybe shouldn't be there.


Applicability MUST be described objectively, unambiguously and in plain language.

An objective description is one that can be resolved without uncertainty, in a given technology. Examples of objective properties in HTML are element names, their computed role, the spacing between two elements, etc. Subjective properties on the other hand, are concepts like decorative, navigation mechanism and pre-recorded. Even concepts like headings and images can be misunderstood. For example, describing that the rule examines the tag name, the accessibility role, or the element's purpose on the web page. The latter of which is almost impossible to define objectively. When used in applicability, potentially ambiguous concepts MUST be defined objectively. This definition can be part of a larger glossary shared between rules.
An objective description is one that can be resolved without uncertainty, in a given technology. Examples of objective properties in HTML are tag names, their computed role, the distance between two elements, etc. Subjective properties on the other hand, are concepts like decorative, navigation mechanism and pre-recorded. Even concepts like headings and images can be misunderstood. These terms could refer to the tag name, the semantic role, or the element's purpose on the web page. The latter of which is almost impossible to define objectively. When used in applicability, potentially ambiguous concepts MUST be defined objectively. These definitions MUST be put in the rule's [glossary](#glossary).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest that definitions are only have to be put into the glossary if it's not possible to define it unambiguously in the applicability or expectation itself (which should be the goal).

<blockquote>
<p>Any <code>video</code> or <code>audio</code> element(s) with the <code>autoplay</code> attribute, as well as any <code>object</code> element(s) that is used for automatically playing video or audio when the web page loads.</p>
<p>Each `video` or `audio` element with the `autoplay` attribute, as well as each `object` element that is used for automatically playing video or audio when the web page loads.</p>
<p>**Note: A web page is considerd "loaded" when the `document.readySate` is set to `complete`.</p>

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This note should be in the Glossary instead according to latest edits.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not if we allow inline definitions like you suggested before. That's a good addition.


The applicability of a composite rule is defined as the union of all the applicability sections of its [=atomic rules=] as well as atomic rules that make up any composite rules in the [input rules list](#input-rules-list). Rule authors MAY describe the applicability for composite rules. This can be useful if it is difficult to express the combined applicability in plain language. If the composite rule includes applicability, it MUST be the union of all the applicability sections of the atomic rules.
The applicability of a composite rule is defined as the union of all the applicability sections of rules in the [input rules](#input-rules). This includes [=atomic rules=], and [=composite rules=] which themselves have a union applicability. Rule authors MAY ommit a description of the applicability for composite rules. This can be useful if it is difficult to express the combined applicability in plain language. If the composite rule includes applicability, it MUST be the union of all the applicability in the [input rules](#input-rules).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Editorials needed for "it MUST be the union of all the applicability in the [input rules]". Maybe add "sections" to "applicability"

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disagree. We're not merging the sections, we're creating a new section that describes what the union.

Copy link
Contributor

@nitedog nitedog Jan 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the issue is with the "the" in the sentence. How about "all applicability definitions" instead of "sections"? Later on (in the note) you simply use "applicability" - maybe this could work too. In any case, the grammar of this sentence does seem wrong to me too.


Note that atomic rules in a composite rule MAY have different applicability. Because of this, not every element applicable within the composite rule is tested by every atomic rule.
Note that input rules in a composite rule MAY have different applicability. Because of this, not every element applicable within the composite rule is tested by every input rule.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

element --> test target?

</ul>
</blockquote>
<p>Composite rule: video elements have an audio description or media alternative (<a href="https://www.w3.org/WAI/WCAG21/quickref/#audio-description-or-media-alternative-prerecorded" target="_blank">WCAG 2.1, success criterion 1.2.3 Audio Description or Media Alternative</a>).</p>
<p>Each HTML `video` element meets expectations from at least one of the following rules:</p>

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still say that it has to meet all expectations of at least one input rule. It can't meet half an input rule and still pass.

@annethyme
Copy link

If the change of outcome reporting from the test target to the test subject level came out of the discussion at TPAC on how to accommodate possible future accessibility requirement that has the form of e.g. "85% of images have a text alternative", I would say that this could also be done in a format where the outcome is related to the test target.

Example:

Applicability: This rule applies to the full set of img elements in a HTML document.
Expectation: 85% of the elements in the applicable set have an accessible name.

Example of an Auto-WCAG draft rule that is treating a set of elements: https://github.com/auto-wcag/auto-wcag/pull/220/files.

<ul>
<li>An HTML page, including all embedded scripts, style and images</li>
<li>An EPUB document</li>
<li>A Vue component file</li>
</ul>
</div>
<div class=note>
<p><strong>Note</strong>: Implementers using the [[EARL10-Schema]] can express the test subject with the [subject property](https://www.w3.org/TR/EARL10-Schema/#subject)</p>
<p>**Note:**>: Implementers using the [[EARL10-Schema]] can express the test subject with the [subject property](https://www.w3.org/TR/EARL10-Schema/#subject)</p>
</div>
</dd>

<dt><dfn>Test Target</dfn></dt>
<dd>
<p>A distinct part of the [=test subject=].</p>

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should it say here that the test targets are the things that are applicable to the rule?
I know that it becomes a bit circular when following the link from the Applicability, but out of context, the definition seems very thin.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is a good idea. Maybe simply:

A distinct part of the [=test subject=], as defined by Applicability.

<p>Composite rule: Header cells in HTML tables (<a href="https://www.w3.org/WAI/WCAG21/quickref/#info-and-relationships" target="_blank">WCAG 2 Success Criterion 1.3.1</a>).</p>
<p>This rule uses outcomes from the following rules:</p>
<p>Composite rule: video elements have an audio description or media alternative (<a href="https://www.w3.org/WAI/WCAG21/quickref/#audio-description-or-media-alternative-prerecorded" target="_blank">WCAG 2.1, success criterion 1.2.3 Audio Description or Media Alternative</a>).</p>
<p>Each HTML `video` element meets the expectations from at least one of the following rules:</p>
Copy link
Contributor

@nitedog nitedog Jan 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(minor/editorial) I suggest changing "meets the expectations" to "meets all expectations", to emphasize the meaning and for better readability ("the" implies a definite set, which makes me pause to think which set you actually mean. On the other hand, "all" implies the entire set and does not require further processing cylces in my view).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following the ACT TF discussion on 31 January 2019, I suggest we also change "Each HTML 'video' element" to "All HTML 'video' elements", to better emphasize the scope.


When an ACT Rule is designed to test the conformance to one or more [=Accessibility requirements documents=], the rule MUST list all [=accessibility requirements=] from those documents that are not satisfied when the outcome of the rule is `failed`. For example, when designing a rule for WCAG 2.1 that tests if image buttons have an alternative text, the rule maps to success criteria 1.1.1 Non-text content, and 4.1.2 Name, Role, Value.
When an ACT Rule is designed to test one or more [=Accessibility requirements documents=], the rule MUST list all [=accessibility requirements=] from those documents that are not satisfied when the [=outcome=] of the rule is `failed`. For example, when designing a rule for WCAG 2.1 that tests if image buttons have an alternative text, the rule maps to success criteria 1.1.1 Non-text content, and 4.1.2 Name, Role, Value. That ACT Rule will list both success criteria in its accessibility requirements mapping.

Each [=accessibility requirement=] in the mapping MUST include the following:

Copy link
Contributor

@nitedog nitedog Jan 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(medium) In the first list item below, add "identifier" to "name, title, or summary"

@@ -126,30 +122,33 @@ Each [=accessibility requirement=] in the mapping MUST include the following:
3. a link to the [=accessibility requirements document=] if one exists, and
Copy link
Contributor

@nitedog nitedog Jan 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(medium) Change "link" to "a link or reference" (eg. if the document is not publicly available or directly linkable for any other reason)


Commonly, an expectation comes in the form of an "Each test target has ..." statement. In such an expectation, all test targets need to [=meet expectations|meet all expectations=] for the rule to pass. It is also possible to have an expectation where not all test targets meet all expectations. For example: "85% of all test targets have an accessible name" is an expectation that is true even when 15% of the test targets do not meet the expectations.
Copy link
Contributor

@nitedog nitedog Jan 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(important) I think this paragraph needs to be rewritten, to better explain the different types of definition forms for expectations. It seems that we have two types:

  1. individual test targets - the expectation applies to individual test targets, like "every test target has an accessible name"
  2. sets of test targets - the expectation applies to a set of test targets, like "85% of all test targets have an accessible name"

I think in these cases, also the applicability definitions need to be formulated accordingly. For the first type of expectation, the applicability is "every X" (eg. every element), while for the the section the applicability is "all X) (eg. all elements). I think it would be illogical to mix these types of forms, for example to say "every element ... 85% of all elements have an accessible name".

See more below on definition for "meets all expectations".

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

... building on what Shadi said, I think that the "85%..." type of expectation rules out saying anything about whether a particular test target meets the expectation. These types of rules can only say something about the set they apply to.

(Even though an implementation could choose to report which elements count toward/from meeting the expectation. And it would probably be nice to report some calculations too, e.g. how many % had an accessible name, and how far is it from the 85% threshold).

I also don't really see why it's not always "meet all expectations". I agree that it's not always "all test targets", like in the 85% case, but I cannot imagine an example where we would allow for only meeting e.g. 2 out of 3 expectations in a rule. If the expectation is "85%...", this is still an expectation that needs to be met.

Copy link
Contributor

@nitedog nitedog Feb 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following the ACT TF discussion on 31 January 2019, I suggest we change this paragraph as follows:

Expectation can be statements in the form of "Each test target has ..."; for example, "Each test target has an accessible name". For such expectations, all test targets need to [=meet expectations|meet the expectation=] for the rule to pass. Expectations can also be statements in the form of "The set of test targets has ..."; for example, "85% of all test targets have an accessible name". For such expectations, the entire set of test targets needs to [=meet expectations|meet the expectation=] for the rule to pass.

<blockquote>No test target has the same ID</blockquote>
</div>
<div class=example>
<p>**Example:** An HTML `video` element without captions does not meet the following expectation:</p>
Copy link

@annethyme annethyme Jan 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do we know if this video is in the 80% or 20%?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. In my view this example does not make sense. We are trying to match an individual test target on an expectation about an entire set of test targets (see suggestions for expectations suggestion above).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following the ACT TF discussion on 31 January 2019, I suggest we change this example to:

A set of 3 HTML 'video' elements of which 2 do not have captions does not meet the following expectation:

<dt><dfn>Outcome</dfn></dt>
<dd>
<p>One of three types of conclusions that come from evaluating an ACT Rule on a [=test subject=]. An outcome can be:<p>
<ul>
<li>**Inapplicable:** No part of the test subject matches the applicability</li>
<li>**Passed:** The test subject meets all expectations</li>
<li>**Failed:** The test subject does not meet all expectations</li>
<li>**Passed:** All expectations are true for the test subject</li>
Copy link

@annethyme annethyme Jan 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we use the "meets all expectations" definition here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering why we are distinguishing "meets" and "is true".


The ACT Rules Format can be used to describe ACT Rules dedicated to testing the [=accessibility requirements=] defined in Web Content Accessibility Guidelines [[WCAG]], which are specifically designed for web content. Other accessibility requirements applicable to web technologies can also be testable with ACT Rules. For example, ACT Rules could be developed to test the conformance of web-based user agents to the [User Agent Accessibility Guidelines](https://www.w3.org/WAI/standards-guidelines/uaag/) [[UAAG20]]. The ACT Rules Format might not always be suitable to describe tests for other types of accessibility requirements.


ACT Rule Types {#rule-types}
============================

In accessibility, there are often different technical solutions to make the same type of content accessible. Multiple solutions could be tested in a single rule; however, such a rule tends to be quite complex, making it difficult to understand and maintain. The ACT Rules Format solves this by providing two types of rules:
In accessibility, there are often different technical solutions to make the same type of content accessible. For example, there are multiple methods for giving an `img` element in HTML an accessible name. Multiple solutions could be tested in a single rule; however, such a rule tends to be quite complex, making it difficult to understand and maintain. The ACT Rules Format solves this by providing two types of rules:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Editorial: Suggest added sentence read as follows...
For example, there are multiple methods to give an accessible name to an HTML 'img' element.


- <dfn>Atomic rules</dfn> describe how to test a specific type of solution. It contains a precise definition of what elements, nodes or other parts of a [=test subject=] are to be tested, and when those elements are considered to fail the rule. These rules are to be kept small and *atomic*. Meaning that atomic rules test a single "failure condition", and do so without using the [=outcome=] of other rules.
- <dfn>Atomic rules</dfn> describe how to test a specific type of solution. It contains a precise definition of what elements, nodes or other "parts" of a [=test subject=] are to be tested, and when the test subject is considered to fail the rule. These rules are to be kept small and *atomic*. This means that atomic rules test a single "failure condition", and do so without using the [=findings=] from other rules.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I understand the reason for changing "outcome" to "findings". The "outcome" is the finding of an ACT rule, whether it be atomic or composite. I don't think we need two different terms for that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See issue #327 on removing "failure", and simply say "condition" instead.

</div>

The separation between atomic rules and composite rules can be seen as a division of responsibility. Atomic rules typically test if web content is correctly implemented in a particular solution. Composite rules can test if a combination of outcomes from other rules satisfy the accessibility requirement, in part or as a whole.
Not all atomic rules have to be part of a composite rule. Composite rules are used when the [=findings=] of multiple rules need to be combined in order to determine if a [=test subject=] does not satisfy an [=accessibility requirement=].
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Editorial: Remove "in order"


ACT Rules MUST be written in a format that conforms to the Web Content Accessibility Guidelines [[WCAG]] or a comparable accessibility standard. ACT Rule [test cases](#test-cases) are allowed to contain inaccessible content. If any test case contains accessibility issues listed in [WCAG 2.1 Section 5.2.5 Non-Interference](https://www.w3.org/TR/WCAG21/#cc5), users MUST be warned of this in advance. Using an accessible format supports people with disabilities. It also makes internationalization of ACT Rules easier.
The ACT Rules format does not prescribe what format ACT Rules can be written in (e.g. HTML, DOCX, PDF, etc.). However, ACT Rules MUST be written in a document that conforms to the Web Content Accessibility Guidelines [[WCAG]] or a comparable accessibility standard. This ensures that ACT Rules are accessible to people with disabilities. ACT Rule [test cases](#test-cases) are allowed to contain inaccessible content. If any test case contains accessibility issues listed in [WCAG 2.1 Section 5.2.5 Non-Interference](https://www.w3.org/TR/WCAG21/#cc5), users MUST be warned of this in advance. In addition to supporting people with disabilities, using an accessible format also makes internationalization of ACT Rules easier.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where should accessibility issues in test cases be documented to meet this "warned in advance" requirement? In the test case itself, in the rule description? Is that flexible?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, there is the use of normative language - "MUST" and this is a note, right? Is this requirement in the ACT Rules Format somewhere?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To your latter comment @maryjom, AFAIK this is not a note but a paragraph in a normative section of the document. It actually sets two normative requirements (accessible content and advance warning).


- <dfn>Composite rules</dfn> describe how the [=outcomes=] of multiple [=atomic rules=] or composite rules can be combined into a single outcome. A composite rule can have multiple "satisfying conditions", each of these provided in separate atomic or composite rules. The logic in the composite rule describes how any one of these satisfying conditions, or some combination of them is used to determine a single outcome.
- <dfn>Composite rules</dfn> describe how the [=findings=] of multiple [=atomic rules=] or composite rules can be combined into a single [=outcome=]. A composite rule can have multiple "satisfying conditions", each of these tested in separate rules. The logic in the composite rule describes how any one of these satisfying conditions, or some combination of them, is used to determine a single outcome.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe remove "satisfying" from here (just "conditions", as suggested in issue #327)?

<blockquote>
<p>Any <code>video</code> or <code>audio</code> element(s) with the <code>autoplay</code> attribute, as well as any <code>object</code> element(s) that is used for automatically playing video or audio when the web page loads.</p>
<p>Each `video` or `audio` element with the `autoplay` attribute, as well as each `object` element that is used for automatically playing video or audio when the web page loads.</p>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following the ACT TF discussion on 31 January 2019, Please change "Each 'video' or 'audio' element ... as well as each 'object' element" to "All 'video' and 'audio' elements ... as well as all 'object' elements"

==============================
----------------------------

An ACT Rule MUST contain one or more expectations. The expectations describe what the requirements are for [=test targets=] derived from the [applicability](#applicability). An expectation is an assertion about all [=test targets=] that is true if the [=test subject=] satisfies the [=accessibility requirement=]. When all expectations are true for a test subject, the test subject `passed` the rule. If one or more expectations are false, the test subject `failed` the rule. If there are no test targets, the [=outcome=] for the test subject is `inapplicable`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we certain about "that is true if the test subject satisfies the accessibility requirement"? In think that individual expectations in individual atomic rules may be not met in a composite rule, while the test subject overall satisfies the accessibility requirement.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following the ACT TF discussion on 31 January 2019, I think we should rather say:

An expectation is an assertion about all test targets or about the collective set of test targets.

==============================
----------------------------

An ACT Rule MUST contain one or more expectations. The expectations describe what the requirements are for [=test targets=] derived from the [applicability](#applicability). An expectation is an assertion about all [=test targets=] that is true if the [=test subject=] satisfies the [=accessibility requirement=]. When all expectations are true for a test subject, the test subject `passed` the rule. If one or more expectations are false, the test subject `failed` the rule. If there are no test targets, the [=outcome=] for the test subject is `inapplicable`.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Proposal:

An expectation is an assertion about a [=test target=]. This assertion is true when the [=test subject=] satisfies the [=accessibility requirement=]. When a test target meets all expectations, the test target passed the rule. If the test target does not meet one or more expectations, the test target failed the rule. If there are no test targets, the [=outcome=] for the rule is inapplicable.

</div>
</dd>

<dt><dfn>Outcome</dfn></dt>
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: A rule has one passed of failed outcome for every test target. When there are no test targets the rule has one inappliable outcome. This means that each test subject will have one or more outcomes.

@WilcoFiers
Copy link
Collaborator Author

PR has been superseded.

@WilcoFiers WilcoFiers closed this Feb 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants