Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CR Proposal #283

Merged
merged 12 commits into from Oct 14, 2018
Merged

CR Proposal #283

merged 12 commits into from Oct 14, 2018

Conversation

WilcoFiers
Copy link
Collaborator

@WilcoFiers WilcoFiers commented Oct 8, 2018

Issues resolved by this PR:

Outstanding discussions before CR:

Outstanding editorials before CR:


Preview | Diff

* [Evaluation Instructions](#evaluation-instructions)
* [Test Cases](#test-cases)

ACT Rules MUST be written in an format that conforms to the Web Content Accessibility Guidelines [[WCAG]] or a comperable accessibility guidelines recommendation. ACT Rule [test cases](#test-cases) are allowed to contain inaccessible content. If any test case contains accessibility issues listed in [WCAG 2.1 Section 5.2.5 Non-Interference](https://www.w3.org/TR/WCAG21/#cc5), users MUST be warned of this in advance. Using an accessibility format supports people with disabilities. It also makes internationalisation of ACT rules easier.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Editorial:

  • First sentence: ...written in "a" format, not "an" format and "comparable" instead of "comperable"
  • First sentence: Suggest you say "comparable accessibility standard" since not all are called "guidelines" or are a "recommendation", like the EN 301 549 or other non-W3C standard.
  • You should probably provide an explanation of why there MUST be a warning if the (do you really mean test case or do you mean rule?) contains accessibility issues listed in the Non-interference section of WCAG.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't address the last bullet. This is not a big important point, so I don't want to spend too much text on it. The point is that if someone has a test case that can be a keyboard trap, they are allowed to do that, but they need to warn the user for it.

The ACT Rules Format can be used to describe ACT Rules dedicated to testing the [accessibility requirements](#structure-accessibility-requirements) defined in Web Content Accessibility Guidelines [[WCAG]], which are specifically designed for web content. Other accessibility requirements applicable to web technologies can also be testable with ACT Rules. For example, ACT Rules could be developed to test the conformance of web-based user agents to the [User Agent Accessibility Guidelines](https://www.w3.org/WAI/standards-guidelines/uaag/) [[UAAG20]]. However, the ACT Rules Format may not always be suitable to describe tests for other types of accessibility requirements.

Because ACT Rules rarely test the entirety of an accessibility requirement, passing ACT rules does not necessarily mean that an accessibility requirement is met. It is important to understand that **ACT Rules test non-conformance** to accessibility requirements. In some cases conformance can be inferred from the absence of failures. Unlike WCAG sufficient techniques, ACT Rules can not be used for conformance claims unless the rule explicitly states it can be used that way. See [Rule Aggregation](#rule-aggregation) for details.
The ACT Rules Format can be used to describe ACT Rules dedicated to testing the [accessibility requirements](#accessibility-requirements) defined in Web Content Accessibility Guidelines [[WCAG]], which are specifically designed for web content. Other accessibility requirements applicable to web technologies can also be testable with ACT Rules. For example, ACT Rules could be developed to test the conformance of web-based user agents to the [User Agent Accessibility Guidelines](https://www.w3.org/WAI/standards-guidelines/uaag/) [[UAAG20]]. The ACT Rules Format may not always be suitable to describe tests for other types of accessibility requirements.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"may" -> "might"


The separation between atomic rules and composite rules creates a division of responsibility. Atomic rules test if web content is correctly implemented in a particular solution. Composite rules test if a combination of `pass` outcomes from atomic rules is sufficient for an accessibility requirement not to `fail`. Not all atomic rules have to be part of a composite rule. Atomic rules only have to be part of a composite rule if failing that atomic rule does not directly fail [Accessibility Requirements](#structure-accessibility-requirements). An atomic rule MAY be part of multiple composite rules.
- <dfn>Composite rules</dfn> describe how the [=outcome=] of [=atomic rules=] should be used to decide if an accessibility requirement is (non-)conformant. If there are multiple `failed` outcomes necessary for an accessibility requirement to be (non-)conformant, each "type" of `failed` outcome would come from a separate atomic rules, and the logic for determining the conformance or non-conformance for the accessibility requirement, across the atomic rules, should be handled by a composite rule.
Copy link
Contributor

@nitedog nitedog Oct 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The second sentence is long and complex. It also uses the keyword "should", I think incorrectly. Consider this edit:

"If this involves multiple "failure conditions", each of these is provided in a separate atomic rule. The logic for determining (non-)conformance to the accessibility requirement according to outcome of the individual the atomic rules is specified in the composite rule."


- <dfn>Composite rules</dfn> describe how results from atomic rules should be used to decide if an accessibility requirement was failed. If there are multiple failures necessary for an accessibility requirement to fail, each "type" of failure should be written as separate atomic rules, and the logic for determining the ´pass´ or ´fail´ for the accessibility requirement, across the atomic rules, should be handled by a composite rule.
- <dfn>Atomic rules</dfn> describe how to test a specific type of solution. It contains a precise definition of what elements, nodes or other parts of a [=test subject=] are to be tested, and when those elements are considered to fail the rule. These rules should be kept small and *atomic*. Meaning that atomic rules test a single "failure condition", and do so without using the [=outcome=] of other rules.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sentence "These rules should be kept small and atomic" uses the "should" keyword, and I think incorrectly. Consider "These rules are to be kept small and atomic" or just "These rules are kept small and atomic" or such.

</div>

The separation between atomic rules and composite rules creates a division of responsibility. Atomic rules test if web content is correctly implemented in a particular solution. Composite rules test if a combination of outcomes from atomic rules is sufficient for an accessibility requirement to be conformant or not.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Atomic rules test if web content is correctly implemented in a particular solution" -> "Atomic rules test if test subjects correctly implement particular solutions" or "Atomic rules test for particular solutions within test subjects"

"Composite rules test if a combination of outcomes from atomic rules is sufficient for an accessibility requirement to be conformant or not." -> "Composite rules test if a combination of outcomes from atomic rules satisfy/meet the conformance for an accessibility requirement."

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changing the first part. The group has gone over that sentence multiple times already.

</div>

The separation between atomic rules and composite rules creates a division of responsibility. Atomic rules test if web content is correctly implemented in a particular solution. Composite rules test if a combination of outcomes from atomic rules is sufficient for an accessibility requirement to be conformant or not.

Not all atomic rules have to be part of a composite rule. Only if the outcome of multiple atomic rules has to be combined in order to determine (non-)conformance of the [Accessibility Requirements](#accessibility-requirements) can composite rules be used. An atomic rule MAY be part of multiple composite rules.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Only if the outcome of multiple atomic rules has to be combined in order to determine (non-)conformance of the Accessibility Requirements can composite rules be used" -> "Composite rules are used when the outcome of multiple atomic rules needs to be combined in order to determine (non-)conformance [of a test subject] to the Accessibility Requirements"


[=Atomic rules=] and [=composite rules=] SHOULD identify the accessibility requirements that fail when the outcome of a rule is Fail. An ACT Rule is a complete or partial test for one or more accessibility requirements. This means that most ACT rules test only part of an accessibility requirement, but it MUST NOT test more than the accessibility requirement it lists.
Accessibility requirements are just that: A requirement that a particular web page must conform to for it to be considered accessible. A common example of accessibility requirements are the WCAG 2.1 success criteria. Often organizations have additional requirements which may come from different sources, such as laws, internal standards, etc. These too are considered accessibility requirements which can be tested using the ACT Rules Format.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"web page" -> "test subject"
"must conform to" -> "conforms to" (drop the "must")
"Often organizations" -> "Some organizations"
"which may come from" -> "that come from" (drop the "may")

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Test subjects aren't required to conform to WCAG, web pages are. The way I see it, web pages are a type of test subject. JSX or Vue files are examples of test subjects that are not web pages. Instead, they are used to build a web page. WCAG does not apply to those files, nevertheless those can be tested by ACT Rules. I think this sentence is correct.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why you want to limit "accessibility requirements" to web page and WCAG only. As I understand it, accessibility requirements is more generic. It can be used for any requirement, for any test subject. We happen to only currently use it for WCAG and web pages in Auto-WCAG. Anyway, this sentence does not set a normative requirement and could be edited later, if we decide to.

</ul>
</div>

One reason why an [=atomic rule=] might not list accessibility requirements is because it is part of a [=composite rule=], and only a `failed` [=outcome=] in the composite rule would mean not satisfying the success criteria. Another reason might be that an ACT Rule is designed to evaluate a best practice, which may not be part of any accessibility standard.
Copy link
Contributor

@nitedog nitedog Oct 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"not satisfying the success criteria" -> "not satisfying the accessibility requirement" (we are not WCAG-specific here)
"may" -> "is" (drop the "may")



Aspects Under Test (Atomic rules only) {#input-aspects}
=======================================================

An aspect is a distinct part of the [test subject](#output-test-subject) or its underlying implementation. For example, rendering a particular piece of content to an end user involves multiple different technologies, some or all of which may be of interest to an ACT Rule. Some rules need to operate directly on the [Hypertext Transfer Protocol](https://tools.ietf.org/html/rfc7230) [[http11]] messages exchanged between a server and a client, while others need to operate on the [Document Object Model](https://dom.spec.whatwg.org) [[DOM]] tree exposed by a web browser. Rules may need to operate on several aspects simultaneously, such as both the HTTP messages and the DOM tree.
An aspect is a distinct part of the [=test subject=]. For example, rendering a particular piece of content to an end user involves multiple different technologies, some or all of which may be of use in an [atomic rule]. Some rules need to operate directly on the [Hypertext Transfer Protocol](https://tools.ietf.org/html/rfc7230) [[http11]] messages exchanged between a server and a client, while others need to operate on the [Document Object Model](https://dom.spec.whatwg.org) [[DOM]] tree exposed by a web browser. Rules may need to operate on several aspects simultaneously, such as both the HTTP messages and the DOM tree.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"may be of use" -> "can be of use" (unless you really mean "MAY")
"may need to operate" -> "can operate"



Expectations for Atomic Rules {#expectations-atomic}
----------------------------------------------------

An [=atomic rule=] MUST contain one or more expectations. An expectation is an assertion that must be true about each [test target](#output-test-target) (see [Applicability](#applicability)). Each expectation MUST be distinct, unambiguous, and be written in plain language. Unlike in applicability, a certain level of subjectivity is allowed in expectations. Meaning that the expectation has only one possible meaning, but that meaning isn't fully quantifiable.
An [=atomic rule=] MUST contain one or more expectations. An expectation is an assertion that must be true about each [=test target=] (see [Applicability](#applicability)). Each expectation MUST be distinct, unambiguous, and be written in plain language. Unlike in applicability, a certain level of subjectivity is allowed in expectations. Meaning that the expectation has only one possible meaning, but that meaning MAY not fully quantifiable.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"that must be true" -> "that is true" (drop the "must")


When all expectations are true for a test target, the test target `passed` the rule. If one or more expectations are false, the test target `failed` the rule. If the atomic rule is used in a [=composed rule=], the composed rule may be `passed` when the atomic rule `failed`, depending on the [applicability](#applicability-composed) and [exxpectations-composed] of the composed rule.
When all expectations are true for a test target, the test target `passed` the rule. If one or more expectations are false, the test target `failed` the rule. If the atomic rule is used in a composed rule, the outcome of the composed rule may be `passed` when the atomic rule `failed`, depending on the [expectations](#expectations-composed) of the composed rule.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"composed" -> "composite" (twice)
"may be" -> "can be" (drop the "may)

Suggestion: "If the atomic rule is used in a composed rule, the outcome of the composed rule may be passed when the atomic rule failed" -> "When atomic rules are used in composite rules, the outcome of the composite rules can be passed when the outcome of individual atomic rule is failed"

====================================

ACT Rules are designed to test the conformance of content using web technologies to accessibility requirements. However, not every feature of a web technology is implemented in all assistive technologies or user agents that a website may need to support. The concept of [accessibility supported](https://www.w3.org/TR/WCAG20/#accessibility-supporteddef) use of a Web technology is defined in WCAG [[WCAG]]. Because of this, ACT Rules are not necessarily applicable in all test scenarios. For instance, a web page that has to work in assistive technologies that have no WAI-ARIA [[WAI-ARIA]] support, wouldn’t be tested with an ACT Rule that relies on WAI-ARIA support, since this rule could give results that are inconsistent with the accessibility experience of users of those assistive technologies.
ACT Rules are designed to evaluate the conformance of content using web technologies to accessibility requirements. However, not every feature of a web technology is implemented in all assistive technologies or user agents that a website may need to support. The concept of [accessibility supported](https://www.w3.org/TR/WCAG20/#accessibility-supporteddef) use of a Web technology is defined in WCAG [[WCAG]]. Because of this, ACT Rules are not necessarily applicable in all test scenarios. For instance, a web page that has to work in assistive technologies that have no WAI-ARIA [[WAI-ARIA]] support, wouldn’t be tested with an ACT Rule that relies on WAI-ARIA support, since this rule could give results that are inconsistent with the accessibility experience of users of those assistive technologies.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"that a website" -> "that content" (to be less website-centric)

====================================

ACT Rules are designed to test the conformance of content using web technologies to accessibility requirements. However, not every feature of a web technology is implemented in all assistive technologies or user agents that a website may need to support. The concept of [accessibility supported](https://www.w3.org/TR/WCAG20/#accessibility-supporteddef) use of a Web technology is defined in WCAG [[WCAG]]. Because of this, ACT Rules are not necessarily applicable in all test scenarios. For instance, a web page that has to work in assistive technologies that have no WAI-ARIA [[WAI-ARIA]] support, wouldn’t be tested with an ACT Rule that relies on WAI-ARIA support, since this rule could give results that are inconsistent with the accessibility experience of users of those assistive technologies.
ACT Rules are designed to evaluate the conformance of content using web technologies to accessibility requirements. However, not every feature of a web technology is implemented in all assistive technologies or user agents that a website may need to support. The concept of [accessibility supported](https://www.w3.org/TR/WCAG20/#accessibility-supporteddef) use of a Web technology is defined in WCAG [[WCAG]]. Because of this, ACT Rules are not necessarily applicable in all test scenarios. For instance, a web page that has to work in assistive technologies that have no WAI-ARIA [[WAI-ARIA]] support, wouldn’t be tested with an ACT Rule that relies on WAI-ARIA support, since this rule could give results that are inconsistent with the accessibility experience of users of those assistive technologies.

Even within a [=composite rule=], some [=atomic rules=] may not always be applicable. This leaves one fewer solution for passing that particular composite rule. To support users of ACT Rules in properly defining the [accessibility support baseline](https://www.w3.org/TR/WCAG-EM/#step1c) in their test scenarios, an ACT Rule SHOULD include a warning if there are significant accessibility support concerns known about a rule.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"may not" -> "might not" (drop the "may")
"include a warning if there are" -> "indicate" (more concise; also not really a "warning", it is information like all other sections)

Rules are tested in two steps. Firstly, the applicability is used to find a list of [Test Targets](#output-test-target) (elements, tags or other "components") within the web page or other [test subject](#output-test-subject). Then each test target is tested to see if all of the [expectations](#expectations) are true. This will give the outcome for each test target. For contextual information, the output data must also include [test subject](#output-test-subject) and the [rule identifier](#rule-identifier).

This will mean that every time a rule is executed on a page, it will return a set with zero or more results, each of which MUST have at least the following properties:
Evaluation Instructions {#evaluation-instructions}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The name is a bit odd. I suggest "Accessibility Requirements Mapping" (see more below on why this name may be more apt)

- [Test Subject](#output-test-subject) (Web page)
- [Test Target](#output-test-target) (pointer)
- [Outcome](#output-outcome) (`Passed`, `Failed`, or `Inapplicable`)
An ACT Rule MUST include evaluation instructions. Evaluation instructions describes how applicability and expecations can be used to determine (non-)conformance to the [listed accessibility requirements](#accessibility-requirements). Evaluation instructions MUST indicate how `passed`, `failed` and `inapplicable` [=outcomes=] can be determined for each [=test subject=], when evaluating the rule.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first sentence is already covered by section 4 "Rule Structure"; the second sentence is already covered by section 11 "Expectations"; and the third sentence is already covered by the definition of "outcomes". I think we can lose this entire paragraph. It re-defines existing normative requirements and definitions, which is very confusing.

</div>

If any [accessibility requirements](#accessibility-requirements) are listed, the ACT Rule MUST indicate how each outcome impacts the conformance to the listed accessibility requirements. For WCAG 2.1 Success Criteria, this means indicating whether or not the success criterion can be satisfied, or if further testing is needed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"impacts" -> "relates to"

Replace the second sentence with the following:
"Typically failing a rule means the associated accessibility requirement was not met. At the same time, passing rule does not usually mean that the associated accessibility requirement was met, unless the rule was designed to address all fail and pass conditions of that accessibility requirement."

<ol>
<li>If no parts of the test subject match the applicability, the outcome is `inapplicable`</li>
<li>Otherwise, if all expectations of the rule are true for each test target, the outcome is `passed`.</li>
<li>Otherwise the outcome is `failed`.<li>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example should also be dropped. This is rewriting the expectation, and seems unnecessary to me.

<div class="note">
While *inapplicable* is a valid result for ACT Rules, it may not be a valid result for all [accessibility requirements](#structure-accessibility-requirements). Notably the success criteria of WCAG 2.0 and WCAG 2.1 can only be evaluated to true (passed) or false (failed). This translation of results is part of [rule aggregation](#rule-aggregation)
<div class=note>
In WCAG 2.1, success criteria do not evaluate to `passed`, `failed` or `inapplicable`. Rather they can be *satisfied* (or not). (See the [WCAG 2.1 definition: satisfies a success criterion](https://www.w3.org/TR/WCAG21/#dfn-satisfies).) If a success criterion is *not satisfied* a web page can only conform if there is a conforming alternative version, as described in [WCAG 2.1 Conformance Requirement 1](https://www.w3.org/TR/WCAG21/#cc1).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is already described in the definition for outcome and can be removed from here.

<div class="note">
In addition to `Passed` `Failed` and `Inapplicable`, [[EARL10-Schema]] also defined an `Incomplete` outcome. While this should never be the outcome of a rule when applied in its entirety, it often happens that rules are only partially executed. For example, when applicability was automated, but the expectations have to be evaluated manually. Such "interim" results can be expressed with the "Incomplete" outcome.
<div class=note>
In WCAG 2.1, all content must satisfy the success criteria. Because of this, all ACT Rules for WCAG 2.1 will require a `passed` outcome for every test target, in order for the test subject to be `passed`. While this logic of "every element must pass for the page to pass" will be common, it is not required by the ACT Rules Format.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what this means but I think it is already sufficiently addressed by my proposed rewrite, and by the example above.


Test cases are (snippets of) content that can be used to validate the implementation of an [=atomic rule=]. They consist of two pieces of data, a snippet of each [test aspect](#input-aspects) for a rule, and the [expected result](#output) that should come from that rule. Test cases serve two functions, firstly as example scenarios for readers to understand when a rule passes, when it fails, and when it is inapplicable. But also for developers and users of automated accessibility test tools to validate that a rule is correctly implemented.
When evaluating a rule, the test aspect(s), for instance an HTML code snippet, is executed by applying the rule's [evaluation instructions](#evaluation instructions). This results in an outcome (`passed`, `failed`, `inapplicable`) for the [=test subject=], and a list of zero or more `passed` and `failed` test targets. This is then compared to the expected outcome of the test case.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should remove this paragraph. This is not evaluating a rule, but actually an implementation of the rule. It is also not the evaluation instructions that are applied but the applicability and expectations. These provide the outcomes.

@WilcoFiers
Copy link
Collaborator Author

@nitedog @maryjom I've updated the PR. I think this is ready for survey.

</div>

The separation between atomic rules and composite rules creates a division of responsibility. Atomic rules test if web content is correctly implemented in a particular solution. Composite rules test if a combination of outcomes from atomic rules satisfy the conformance for an accessibility requirement.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may be confusing as this separation of responsibility is not always the case, as stated in the following paragraph: sometimes an atomic rule itself would test for conformance to an entire accessibility requirement, sometimes it requires a composed rule. My suggestion is simply to drop this paragraph, or make it a "note" block with a slight reword like

The separation (...) creates can be seen as a division of responsibility. Atomic rules test typically test if web content is correctly implemented in a particular solution, while composite rules test would test if a combination of outcomes from atomic rules satisfy the conformance for an accessibility requirement . as a whole.”

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agree with the reworded suggestion.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed on the call. We'll leave it in place, not as a note but taking over the editorial changes


The applicability of a composite rule is defined as the union of all the applicability sections of its [=atomic rules=]. Because of this, applicability of a composite rule can be inferred from the atomic rules it uses results from. Since the applicability can be inferred, rule authors MAY omit applicability for a composite rule. This can be useful if it is difficult to express the combined applicability in plain language.
The applicability of a composite rule MUST be the union of all the applicability sections of the rules in its [atomic rules list](#atomic-rules-list). Because of this, applicability of a composite rule can be inferred from the atomic rules it uses outcomes from. Since the applicability can be inferred, rule authors MAY omit applicability for a composite rule. This can be useful if it is difficult to express the combined applicability in plain language.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I preferred the previous version, saying “The applicability is defined as” rather than “The applicability MUST be”: if we say it's defined as such, then there is no room for interpretation, and implementations don't have to guess what to do if the author didn't follow the spec and defined the applicability otherwise. If we use a MUST, then authors may (will) break this statement and implementations are left with an unspecified situation.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Decision from the call is to leave the "MUST" in here, it is a requirement for the rule author. We can work on rewording this to address the issue separately.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: "The applicability of a composite rule is defined as the union of all the applicability sections of its [=atomic rules=]. This can be inferred from the list of atomic rules. However, rule authors MAY provide a description of the applicability for composite rules. This can be useful if it is difficult to express the combined applicability in plain language. In that case, the applicability of the composite rule provided by the author MUST also be the union of all the applicability sections of its."


The separation between atomic rules and composite rules creates a division of responsibility. Atomic rules test if web content is correctly implemented in a particular solution. Composite rules test if a combination of `pass` outcomes from atomic rules is sufficient for an accessibility requirement not to `fail`. Not all atomic rules have to be part of a composite rule. Atomic rules only have to be part of a composite rule if failing that atomic rule does not directly fail [Accessibility Requirements](#structure-accessibility-requirements). An atomic rule MAY be part of multiple composite rules.
- <dfn>Composite rules</dfn> describe how the [=outcome=] of [=atomic rules=] is used to decide if an accessibility requirement is (non-)conformant. If this involves multiple "failure conditions", each of these is provided in a separate atomic rule. The logic for determining (non-)conformance to the accessibility requirement according to outcome of the individual the atomic rules is specified in the composite rule.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • "decide if an accessibility requirement is (non-)conformant" does not make sense. Maybe "decide if a test subject conforms to an accessibility requirement"?
  • Same with "logic for determining (non-)conformance to the accessibility requirement" -- needs "test subject" -- "logic for determining conformance of the test subject to the accessibility requirement"
  • BUT if you define composite rules that way, then each composite rule MUST be related to an accessibility requirement. However, the requirement later on in section 7 is only a SHOULD requirement. You need to either change that requirement or this definition of composite rules.


<div class=note>
**Editor's note:** The ACT Task Force is looking for feedback about the use of the term "pass" in relation to rules. While rules can "pass", their corresponding accessibility requirement can fail. A section has been added to make this explicit in the rule, but we would like to know if this is sufficient.
An ACT Rule SHOULD list an identifier and reference for each accessibility requirement that it maps to. Exactly how conformance to accessibility requirements is determined is described in the [Evaluation Instructions](#evaluation-instructions).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • According to the definition of composite rules in section 3, they actually MUST identify the accessibility requirements they relate to (see corresponding comment above)
  • There is no "Evaluation Instructions" section (anymore). I don't think that sentence is needed anyway.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need to remove "evaluation instructions".


[=Atomic rules=] and [=composite rules=] SHOULD identify the accessibility requirements that fail when the outcome of a rule is Fail. An ACT Rule is a complete or partial test for one or more accessibility requirements. This means that most ACT rules test only part of an accessibility requirement, but it MUST NOT test more than the accessibility requirement it lists.
If any accessibility requirements are listed, the ACT Rule MUST indicate how each outcome impacts the conformance to the listed accessibility requirements. For WCAG 2.1 Success Criteria, this means indicating whether or not the success criterion can be satisfied, or if further testing is needed.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest making that a sub-section 7.1 -- maybe call it "Rule Mapping" or such

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll break it up, add 2 headings, one for acc requirements list, second for rule mapping.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!


The separation between atomic rules and composite rules creates a division of responsibility. Atomic rules test if web content is correctly implemented in a particular solution. Composite rules test if a combination of `pass` outcomes from atomic rules is sufficient for an accessibility requirement not to `fail`. Not all atomic rules have to be part of a composite rule. Atomic rules only have to be part of a composite rule if failing that atomic rule does not directly fail [Accessibility Requirements](#structure-accessibility-requirements). An atomic rule MAY be part of multiple composite rules.
- <dfn>Composite rules</dfn> describe how the [=outcome=] of [=atomic rules=] is used to decide if an accessibility requirement is (non-)conformant. If this involves multiple "failure conditions", each of these is provided in a separate atomic rule. The logic for determining (non-)conformance to the accessibility requirement according to outcome of the individual the atomic rules is specified in the composite rule.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove "the" between individual and atomic

@WilcoFiers WilcoFiers changed the title No implementation CR Proposal Oct 11, 2018
@@ -37,7 +37,7 @@ In accessibility, there are often different technical solutions to make the same

- <dfn>Atomic rules</dfn> describe how to test a specific type of solution. It contains a precise definition of what elements, nodes or other parts of a [=test subject=] are to be tested, and when those elements are considered to fail the rule. These rules are to be kept small and *atomic*. Meaning that atomic rules test a single "failure condition", and do so without using the [=outcome=] of other rules.

- <dfn>Composite rules</dfn> describe how the [=outcome=] of [=atomic rules=] is used to decide if an accessibility requirement is (non-)conformant. If this involves multiple "failure conditions", each of these is provided in a separate atomic rule. The logic for determining (non-)conformance to the accessibility requirement according to outcome of the individual the atomic rules is specified in the composite rule.
- <dfn>Composite rules</dfn> describe how the [=outcome=] of multiple [=atomic rules=] can be combined into a single outcome. A composite rule can have multiple "satisfying conditions", each of these provided in a separate atomic rule. The logic in the composite rule describes how any one of these satisfying conditions, or some combination of them is used to determine a single outcome.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • I really like the idea of saying "satisfying conditions" because it is not limited to "failure" or "success" but could be any. I think we need to reflect that phrasing in the atomic rules definition too. That is, "satisfying condition" instead of "failure condition".
  • The last sentence seems to repeat the first two. Actually it contradicts them somehow -- is it "multiple" or "any one or more"? I'm not sure what the last sentence adds that is not already in the first two, and think it should be removed.
  • Following the previous point, I think the word "multiple" is wrong in the first two sentences, regardless if you keep the third sentence or drop it. It should read "any one or more" instead of "multiple".

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest the last sentence reads, "The logic in the composite rule describes one or more of these satisfying conditions is used to determine a single outcome."

@@ -51,7 +51,7 @@ In accessibility, there are often different technical solutions to make the same
<p>If any one of these rules passes, the outcome of the composite rule for the table cell is "passed".</p>
</div>

The separation between atomic rules and composite rules creates a division of responsibility. Atomic rules test if web content is correctly implemented in a particular solution. Composite rules test if a combination of outcomes from atomic rules satisfy the conformance for an accessibility requirement.
The separation between atomic rules and composite rules can be seen as a division of responsibility. Atomic rules typically test if web content is correctly implemented in a particular solution. Composite rules would test if a combination of outcomes from atomic rules satisfy the accessibility requirement, in part or as a whole.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not the grammar expert but "would" jumps out at me. I think it should be "can".

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would simply delete the word 'would'.

@@ -210,12 +219,12 @@ Applicability for Composite Rules {#applicability-composite}

A [=composite rule=] defines how the [=outcomes=] from rules in its [atomic rules list](#atomic-rules-list) are used to determine a single outcome.

The applicability of a composite rule MUST be the union of all the applicability sections of the rules in its [atomic rules list](#atomic-rules-list). Because of this, applicability of a composite rule can be inferred from the atomic rules it uses outcomes from. Since the applicability can be inferred, rule authors MAY omit applicability for a composite rule. This can be useful if it is difficult to express the combined applicability in plain language.
The applicability of a composite rule is defined as the union of all the applicability sections of its [=atomic rules=]. This can be inferred from the [list of atomic rules](#atomic-rules-list). Rule authors MAY describe the applicability for composite rules. This can be useful if it is difficult to express the combined applicability in plain language. If the composite rule includes applicbility, it MUST be the union of all the applicability sections of the atomic rules.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

applicbility -> applicability

Copy link
Contributor

@nitedog nitedog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With some suggested comments

@@ -83,7 +83,7 @@ Rule Identifier {#rule-identifier}
An ACT Rule MUST have a unique identifier that can be any unique text, such as plain text, URL or a database identifier.

<div class=example>
<p>ACT Rules in the Auto-WCAG community groups use file names as identifiers. They include a number for a WCAG 2.1 success criterion, and a handle that includes an element name or attribute:</p>
<p><strong>Example</strong>: ACT Rules in the Auto-WCAG community groups use file names as identifiers. They include a number for a WCAG 2.1 success criterion, and a handle that includes an element name or attribute:</p>
Copy link
Collaborator

@maryjom maryjom Oct 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested edit: 'ACT Rules developed by the Auto-WCAG community groups use..." What happens if a rule can be used for testing multiple WCAG criteria? Does that happen? If so, what should you list as the criteria number?

Also, to clarify understanding of the 2nd sentence...Do you mean, "They include the number corresponding to a WCAG 2.1 success criterion, combined with a handle that includes an element name or name being tested:"
Does that change characterize it more clearly without changing your meaning?


<div class=example>
<p>The outcome of this rule effects conformance to the following WCAG 2.1 success criteria:</p>
<p><strong>Example</strong>: The outcome of this rule effects conformance to the following WCAG 2.1 success criteria:</p>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think "affects" instead of "effects" is a better word.

If any accessibility requirements are listed, the ACT Rule MUST indicate how each outcome impacts the conformance to the listed accessibility requirements. For WCAG 2.1 Success Criteria, this means indicating whether or not the success criterion can be satisfied, or if further testing is needed.

<div class=example>
<p>Outcome of this rule map to the accessibility requirements in the following manner:</p>
<p><strong>Example</strong>: Outcome of this rule map to the accessibility requirements in the following manner:</p>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nitpicking here, but an editorial change. "The outcome of this rule maps..."

@@ -350,7 +361,7 @@ While the ACT Rules Format is designed to stimulate harmonization, there are no
Harmonization occurs when a group of rule implementors collectively accept the validity of an ACT Rule. For example, a community group of accessibility testing tool vendors could declare they've harmonized on a particular set of ACT Rules. Such a group can set acceptance criteria for rules, and have quality requirements that go beyond the ACT Rules Format. For example:

<div class=example>
<p>The ACT EPUB group might have the following acceptance criteria:</p>
<p><strong>Example</strong>: The ACT EPUB group might have the following acceptance criteria:</p>
<ul>
<li>An ACT EPUB Rule is harmonized when it was approved members of 3 organisations, AND</li>
<li>An ACT EPUB Rule is harmonized when it has 2 independent implementation</li>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last bullet should say, "implementations".

@@ -403,10 +416,10 @@ Definitions {#definitions}
<li>**Failed**: The test subject does not meets all expectations</li>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use "meet" instead of "meets"

<ul>
<li>An HTML page, including all embedded scripts, style and images<li>
<li>An EPUB document</li>
<li>A Vue component file</li>
</ul>
</div>
<div class=note>
<p>Implementers using the [[EARL10-Schema]] can express the test subject on the [subject property](https://www.w3.org/TR/EARL10-Schema/#subject)</p>
<p><strong>Note</strong>: Implementers using the [[EARL10-Schema]] can express the test subject on the [subject property](https://www.w3.org/TR/EARL10-Schema/#subject)</p>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking it should say "using the [subject property]" rather than "on..."

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Used "with" instead.

<ul>
<li>Nodes within an HTML page</li>
<li>A period of time within a video</li>
<li>A range of characters within a text document</li>
</ul>
</div>
<div class=note>
<p>Implementers using the [[EARL10-Schema]] can express the test target on the [pointer property](https://www.w3.org/TR/EARL10-Schema/#pointer)</p>
<p><strong>Note</strong>: Implementers using the [[EARL10-Schema]] can express the test target on the [pointer property](https://www.w3.org/TR/EARL10-Schema/#pointer)</p>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as previous comment, "using" instead of "on".

</div>
<div class="note">
<p>Implementers using the [[EARL10-Schema]] can express the outcome on the [outcome property](https://www.w3.org/TR/EARL10-Schema/#outcome). In addition to `passed` `failed` and `inapplicable`, EARL 1.0 also defined an `incomplete` outcome. While this can not be the outcome of an ACT Rule when applied in its entirety, it often happens that rules are only partially evaluated. For example, when applicability was automated, but the expectations have to be evaluated manually. Such "interim" results can be expressed with the `incomplete` outcome.
<p><strong>Note</strong>: Implementers using the [[EARL10-Schema]] can express the outcome on the [outcome property](https://www.w3.org/TR/EARL10-Schema/#outcome). In addition to `passed` `failed` and `inapplicable`, EARL 1.0 also defined an `incomplete` outcome. While this can not be the outcome of an ACT Rule when applied in its entirety, it often happens that rules are only partially evaluated. For example, when applicability was automated, but the expectations have to be evaluated manually. Such "interim" results can be expressed with the `incomplete` outcome.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment, "using the [outcome property]" instead of "on".

Do we have 'incomplete' as a valid result defined in our list of outcomes?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, incomplete is not an outcome recognised in the ACT Rules Format.

@WilcoFiers WilcoFiers merged commit e4084af into master Oct 14, 2018
@WilcoFiers WilcoFiers deleted the no-implementation branch October 14, 2018 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment