1. Introduction
There
are
currently
many
test
procedures
and
tools
available
which
aid
their
users
in
testing
web
content
for
conformance
to
accessibility
standards
such
as
the
Web
Content
Accessibility
Guidelines
[WCAG]
[WCAG22]
.
As
the
Web
develops
in
both
size
and
complexity,
these
procedures
and
tools
are
essential
for
managing
the
accessibility
of
resources
available
on
the
Web.
This format is intended to enable a consistent interpretation of how to test conformance to WCAG and other accessibility requirements documents and promote consistent results in accessibility testing. The rules format is designed to describe both manual accessibility tests, as well as automated tests as performed by accessibility testing tools.
Documenting how to test accessibility requirements will result in accessibility tests that are transparent, with test results that are reproducible. The Accessibility Conformance Testing (ACT) Rules Format defines the requirements for these test descriptions, known as Accessibility Conformance Testing Rules (ACT Rules).
An ACT Rule is a plain language description of how to test a specific type of content for a specific aspect of an accessibility requirement. An ACT Rule describes what kind of content it applies to and which conditions are true about the applicable elements for them to meet all expectations of the rule. In the context of WCAG, ACT Rules test for failures in satisfying Success Criteria. Such failures are often described in common failures documented for WCAG. ACT Rules are written for the testing process and are usually more specific than common failures.
The ACT Rules Format defines the requirements and rule structure for the types of information each rule needs to include to be called an ACT Rule. The structure of the ACT rule is defined in the ACT Rule Structure section. Each ACT Rule also contains a plain language description of the type of content under test, the test to perform, and the expected result. Where the test result affects conformance, the rule documents the particular requirement being tested. The resulting outcomes from the test can be used to help determine conformance or non-conformance to the requirement. Test cases are also written as part of the ACT rule to provide a way to verify that implementations of the rule can successfully determine the expected outcomes.
2. Scope
The ACT Rules Format defined in this specification is designed for rules that can be used in testing content created using web technologies, such as Hypertext Markup Language [HTML] , Cascading Style Sheets [css-2018] , Accessible Rich Internet Applications [WAI-ARIA] , Scalable Vector Graphics [SVG2] , EPUB 3 , and more. The ACT Rules Format is designed to be technology agnostic, meaning that it can conceivably be used to describe test rules for other technologies.
The
ACT
Rules
Format
can
be
used
to
describe
ACT
Rules
dedicated
to
testing
the
accessibility
requirements
defined
in
the
Web
Content
Accessibility
Guidelines
[WCAG]
[WCAG22]
,
which
are
specifically
designed
for
web
content.
Other
accessibility
requirements
applicable
to
web
technologies
can
also
be
testable
with
ACT
Rules.
For
example,
ACT
Rules
could
be
developed
to
test
the
conformance
of
web-based
user
agents
to
the
User
Agent
Accessibility
Guidelines
[UAAG20]
.
The
ACT
Rules
Format
might
not
always
be
suitable
to
describe
tests
for
other
types
of
accessibility
requirements.
3. ACT Rule Types
In
accessibility,
there
are
often
different
technical
solutions
to
make
the
same
type
of
content
accessible.
For
example,
there
are
multiple
methods
for
giving
an
img
element
in
HTML
an
accessible
name.
Multiple
solutions
could
be
tested
in
a
single
rule;
however,
such
a
rule
tends
to
be
quite
complex,
making
it
difficult
to
understand
and
maintain.
The
ACT
Rules
Format
solves
this
by
providing
two
types
of
rules:
-
Atomic rules describe how to test a specific type of solution. It contains a precise definition of what elements, nodes or other "parts" of a test subject are to be tested, and when those test targets are considered to pass or fail the rule. These rules are to be kept small and atomic . This means that atomic rules test a single "condition" and do so without using the outcomes from other rules.
-
Composite rules describe how the outcomes of multiple atomic rules can be combined into a single outcome for each test target . A composite rule can have multiple "conditions", each of these tested in a separate atomic rule. The logic in the composite rule describes how any one of these satisfying conditions, or some combination of them, is used to determine a single outcome.
Composite rules cannot contain other composite rules. Any time a nested composite rule would be needed, all of the relevant atomic rules can be combined directly into the new composite rule.
Not all atomic rules have to be part of a composite rule. Composite rules are used when the outcomes of multiple atomic rules need to be combined to determine if a test subject does not satisfy an accessibility requirement .
The separation between atomic rules and composite rules creates a division of responsibilities. Atomic rules test if web content is correctly implemented in a particular solution. Composite rules can test if a combination of outcomes from other atomic rules satisfy the accessibility requirement, in part or as a whole.
4. ACT Rule Structure
An ACT Rule must consist of at least the following items:
-
Descriptive Title
-
Rule Input , which is one of the following:
-
Input Aspects (for atomic rules) OR
-
Input Rules (for composite rules)
-
-
Change LogRule Versions
An ACT Rule MAY also contain:
-
Related Rules (optional) under Background
Other Resources (optional) under Background
Issues List (optional)
-
BackgroundImplementations (optional) -
AcknowledgementsAcknowledgments (optional)
The
ACT
Rules
format
does
not
prescribe
what
format
ACT
Rules
can
be
written
in
(e.g.
HTML,
DOCX,
PDF,
etc.).
However,
ACT
Rules
must
be
written
in
a
document
that
conforms
to
the
Web
Content
Accessibility
Guidelines
[WCAG]
[WCAG22]
or
a
comparable
accessibility
standard.
This
ensures
that
ACT
Rules
are
accessible
to
people
with
disabilities.
ACT
Rule
test
cases
are
allowed
to
contain
inaccessible
content.
If
any
test
case
contains
accessibility
issues
listed
in
WCAG
2.1
2.2
Section
5.2.5
Non-Interference
,
users
must
be
warned
of
this
in
advance.
In
addition
to
supporting
people
with
disabilities,
using
an
accessible
format
also
makes
internationalization
of
ACT
Rules
easier.
4.1. Rule Identifier
An ACT Rule must have an identifier. This identifier must be unique when the rule is part of a ruleset. The identifier can be any text, such as plain text, URL, or a database identifier.
In addition to the identifier, each new release of an ACT Rule must be versioned with either a date or a number. A reference to the previous version of that rule must be available. The identifier must not be changed when the rule is updated. For substantial changes, a new rule should be created with a new identifier, and the old rule should be deprecated.
4.2. Rule Description
An ACT Rule must have a description that is in plain language which provides a brief explanation of what the rule does.
4.3. Rule Type
An ACT Rule must have a rule type which is one of the following:
- Atomic rule
- Composite rule
Refer to the Rule Type section for detailed definitions of the rule types.
4.4. Accessibility Requirements Mapping
When
an
ACT
Rule
is
designed
to
test
conformance
to
one
or
more
Accessibility
requirements
documents
,
the
rule
must
list
all
accessibility
requirements
from
those
documents
that
are
not
satisfied
when
one
or
more
of
the
outcomes
of
the
rule
is
failed
.
For
example,
when
designing
a
The
rule
for
WCAG
may
list
accessibility
requirements
that
tests
if
image
buttons
have
alternative
text,
could
be
not
satisfied
when
the
rule
maps
to
success
criteria
1.1.1
Non-text
content,
and
4.1.2
Name,
Role,
Value.
That
ACT
Rule
will
list
both
success
criteria
in
its
outcome
is
failed.
There
are
two
types
of
accessibility
requirements
mapping.
requirements:
Each accessibility requirement in the mapping must include the following:
-
either the name, title, identifier or summary of the accessibility requirement, and
-
the name of the accessibility requirements document , and
-
a link or reference to the accessibility requirements document if one exists, and
-
the conformance level associated with the accessibility requirement, if one
exists.exists, and whether the requirement is a conformance requirement or a secondary requirement.
4.4.1. Outcome Mapping
For each4.4.1.1. Conformance Requirements
When a rule is designed to test a specific accessibility requirement, the accessibility requirement is mapped as a Conformance Requirement when both of the following conditions are true:
Failed Outcomes: When one or more of the outcomes for a test
targetsubject isfailed
, the accessibilityrequirements arerequirement is not satisfied for the testsubject.subject, andPassed or Inapplicable Outcomes: When all of the outcomes are
passed
orinapplicable
for a test subject, the accessibility,requirementsrequirement could be satisfied,or further testing is needed.for the test subject.
Rules that can be used to determine if an accessibility requirement is satisfied are called satisfying tests .
-
Passed: When all of the outcomes are
passed
, the accessibility requirement is satisfied for the test subject.
Note:
In
the
Web
Content
Accessibility
Guidelines
[WCAG]
[WCAG22]
,
success
criteria
do
not
evaluate
to
passed
,
failed
or
inapplicable
.
Rather
they
can
be
satisfied
(or
not).
(See
the
WCAG
2.1
2.2
definition:
satisfies
a
success
criterion
.)
If
a
success
criterion
is
not
satisfied
,
a
web
page
can
only
conform
if
there
is
a
conforming
alternative
version,
as
described
in
WCAG
2.1
2.2
Conformance
Requirement
1
.
4.4.1.2. Secondary Requirements
A secondary accessibility requirement is a requirement that is correlated with the rule, but for which the rule is not designed to test. The outcome of the rule impacts the result of the accessibility requirement, but the rule is not intended to test the conformance of that requirement. This correlation often results in some of the rule’s test cases not satisfying the secondary accessibility requirement.
When the rule is not designed to test the accessibility requirement, or failed outcomes of the rule still require further testing for the accessibility requirement, the rule may map the accessibility requirement as Secondary. When an ACT rule maps to a Secondary requirement, it must include an explanation of why that requirement is Secondary in the Background section of the rule.
When the rule is designed to test an accessibility requirement, differences in accessibility support must not be a reason for that requirement to be a secondary accessibility requirement.
Some scenarios when a rule will have Secondary requirements include:
Scenario 1 : the rule is not as strict as a requirement
A rule was designed to test a minimum accessibility requirement and there exists a stricter requirement. The rule’s failed outcomes can determine that the stricter accessibility requirement is not satisfied, and the rule’s passed outcomes may not determine that the stricter requirement is satisfied. It is possible that the accessibility requirement may be not satisfied when the rule’s outcomes are passed. The stricter requirement is a Secondary requirement.
Scenario 2 : the rule is stricter than a requirement
A rule was designed to test a specific solution for an accessibility requirement, but the requirement can be satisfied by other solutions that are not included in the rule. In this scenario, the rule’s failed outcomes cannot determine that an accessibility requirement is not satisfied because further testing is needed. The rule’s passed outcomes can determine that an accessibility requirement is satisfied. The accessibility requirement is a secondary requirement.
A rule was designed to test an accessibility requirement and there exists a less strict accessibility requirement. In this scenario, the rule’s passed outcomes can determine that the less strict requirement is satisfied, and the rule’s failed outcomes may not determine that the less strict requirement is not satisfied. It is possible that the accessibility requirement may be satisfied when the rule’s outcome is failed. The less strict accessibility requirement is a secondary requirement.
Scenario 3 : the rule may impact the result of the accessibility requirement, but the rule is not intended to test the conformance of that requirement
A rule was designed to test an accessibility requirement and under certain conditions, other accessibility requirements apply. In this scenario, the other accessibility requirements are sometimes, but not always, satisfied or not satisfied because they are not always applicable in the rule. These other accessibility requirements are secondary requirements.
4.4.2. Mapping Outside WCAG
ACT Rules can be used to test accessibility requirements that are not part of a W3C accessibility standard, such as accessibility requirements in Hypertext Markup Language [HTML] , or tests in a methodology like RGAA 3 2016 . An ACT Rule must indicate whether or not the accessibility requirement it maps to is required for conformance in its accessibility requirements document . Examples of accessibility requirements that are not required for conformance are WCAG sufficient techniques, or a company style guide that includes both requirements and optional "best practices". The distinction between what is required and what is optional has to be clear.
4.4.3. Rules Without Accessibility Requirements
If the rule does not map to any accessibility requirement , the accessibility requirement mapping will only contain the explainer that it is not required for conformance to the accessibility requirements document . This is common with atomic rules used in composite rules.
If
the
failed
outcome
cannot
be
mapped
to
an
accessibility
requirement
,
there
must
not
be
an
accessibility
requirement
in
the
accessibility
requirements
mapping.
The
optional
Background
section
could
be
used
to
list
accessibility
requirements
documents
when
they
are
thematically
related,
but
for
which
the
rule
is
not
a
failure
test.
4.4.4. External Accessibility Requirements Mapping
This section is non-normative .
While
rules
are
often
designed
for
one,
or
possibly
a
small
collection
of
accessibility
requirements
documents
,
it
is
likely
that
other
accessibility
requirements
documents
also
map
to
those
ACT
Rules.
For
example,
rules
can
be
written
for
the
Web
Content
Accessibility
Guidelines
[WCAG]
[WCAG22]
,
but
many
of
those
could
also
map
to
a
company’s
internal
accessibility
policy.
In
such
a
scenario,
an
external
accessibility
requirements
mapping
could
be
created.
An
external
accessibility
requirements
mapping
amends
the
accessibility
requirements
mapping
of
an
ACT
rule
by
adding
mapping
to
a
different
accessibility
requirements
document.
An
external
accessibility
requirements
mapping
avoids
duplication
of
a
rule
for
the
sole
purpose
of
changing
the
mapping.
4.5. Rule Input
To evaluate content using an ACT Rule, the rule requires some information from the test subject . This is the input for the rule. What input is required is made explicit, to help testers understand what capabilities are required to use a rule. Atomic rules and composite rules have different input.
-
Atomic rules have Input Aspects
-
Composite rules have Input Rules
4.5.1. Input Aspects (Atomic rules only)
An input aspect is a distinct part of the test subject . Rendering a particular piece of content to an end user commonly involves different technologies, some or all of which are required as input for an atomic rule . For example, some rules need to operate directly on the Hypertext Transfer Protocol [http11] messages exchanged between a server and a client, while others need to operate on the Document Object Model [DOM] tree exposed by a web browser.
Atomic rules must list the aspects used as input for the applicability and expectations of the atomic rule. Rules can operate on several aspects simultaneously, such as both the HTTP messages and the DOM tree.
Some input aspects are well defined in a formal specification, such as HTTP messages, the DOM tree, and CSS styling [css-2018] . For these, a reference to the corresponding section in the Common Input Aspects note is sufficient as a description of the aspect. For input aspects that are not well defined, an ACT Rule must include either a detailed description of the aspect in question, or a reference to a well defined description.
The method through which an input aspect is served is not relevant. For example a DOM tree can be served through HTTP as HTML, it can be bundled as several pages in an EPUB publication , or it can be inferred from a JSX source file . All rules that have only DOM tree as an input aspect can be applied to those technologies.
4.5.2. Input Rules (Composite rules only)
A composite rule uses outcomes from atomic rules and applies logic to them so that a single outcome can be determined for each test target . The identifier and descriptive title of all the atomic rules used in the expectations must be listed in the composite rule. The input rules describe the input for composite rules, similar to how input aspects describe the input for atomic rules.
4.6. Applicability
The applicability describes what parts of the test subject are tested.
4.6.1. Applicability for Atomic Rules
The applicability section is a required part of an atomic rule . It must contain a precise description of the parts of the test subject to which the rule applies. For example, specific nodes in the DOM [DOM] tree, or tags that are incorrectly closed in an HTML [HTML] document. These are known as the test targets . The applicability must only use information made available through the listed input aspects in the rule. No other information can be used in the applicability.
Applicability must be unambiguous so that the applicability can only be understood in one way. For example, concepts like headings and images are ambiguous since they could refer to the tag name, the semantic role, or the element’s purpose on the web page. Conversely, a rule that specifically only applies to elements with a heading tag would satisfy the unambiguous requirement.
Additionally,
the
applicability
should
be
described
objectively,
unambiguously
objectively
and
in
plain
language.
An
objective
description
is
one
that
can
be
resolved
without
uncertainty,
in
a
given
technology.
Examples
of
objective
properties
in
HTML
are
tag
names,
their
computed
role,
and
the
distance
between
two
elements,
etc.
elements.
Subjective
properties
on
the
other
hand,
are
concepts
like
decorative,
navigation
mechanism
and
pre-recorded.
Even
concepts
like
headings
mechanism,
and
images
pre-recorded
that
may
be
defined
in
a
variety
of
ways,
often
relying
on
human
judgement.
When
possible,
avoid
subjectivity
since
it
can
easily
be
misunderstood.
These
terms
could
refer
In
cases
where
it
is
impossible
to
objectively
define
the
tag
name,
the
semantic
role,
or
the
element’s
purpose
on
applicability,
the
web
page
because
they
are
ambiguous.
The
latter
use
of
which
a
subjective
description
in
the
applicability
is
almost
impossible
to
define
objectively.
acceptable.
When
used
in
crafting
a
subjective
applicability,
potentially
ambiguous
concepts
must
if
possible,
split
the
objective
and
subjective
parts
of
the
applicability
into
separate
rules.
This
will
ensure
the
designed
rule
meets
the
atomic
rule
requirement
of
testing
a
single
condition.
An
example
would
be
defined
objectively.
testing
headings
with
correct
semantic
markup
in
a
different
rule
from
those
without
semantic
markup.
Definitions can be put in the rule glossary , or they can be defined in the section where they are used.
4.6.1.1. Applicability Type Designation (optional)
Rules can optionally include an applicability type identifier signifying whether the rule contains an objective or a subjective applicability. This identifier is intended to benefit rule readers and implementers by clearly stating the rule author’s intention of the applicability and reducing confusion due to different reader and implementer interpretations.
4.6.2. Applicability for Composite Rules
The applicability of a composite rule is defined as the union of all applicability definitions from the rules listed in the input rules . Rule authors may omit a description of the applicability for composite rules. This can be useful if it is difficult to express the combined applicability in plain language. If the composite rule includes applicability, it must be the union of all the applicability in the input rules .
Note that input rules in a composite rule may have different applicability. Because of this, not every test target applicable within the composite rule is tested by every input rule.
4.6.2.1. Applicability Type Designation (optional)
Composite rules can optionally include an applicability type identifier signifying whether the rule contains an objective or a subjective applicability. The applicability type of a composite rule is calculated as subjective if any of the input rules contain a subjective applicability or objective otherwise.
4.7. Expectations
An
ACT
Rule
must
contain
one
or
more
expectations.
The
expectations
describe
what
the
requirements
are
for
the
test
targets
derived
from
the
applicability
.
An
expectation
is
an
assertion
about
a
test
target
.
When
a
test
target
meets
all
expectations,
the
test
target
passed
the
rule.
If
the
test
target
does
not
meet
all
expectations,
the
test
target
failed
the
rule.
If
there
are
no
test
targets,
the
outcome
for
the
rule
is
inapplicable
.
Each expectation must be distinct, unambiguous, and be written in plain language.
4.7.1. Expectations for Atomic Rules
All
expectations
of
an
atomic
rule
must
describe
the
logic
that
is
used
to
determine
a
single
passed
or
failed
outcome
for
a
test
target
.
The
expectation
must
only
use
information
available
in
the
input
aspects
,
from
the
applicability,
and
other
expectations
of
the
same
rule.
No
other
information
can
be
used
in
the
expectation.
So
for
instance,
an
expectation
could
be
"Expectation
1
is
true
and
...",
but
it
can’t
be
"Rule
XYZ
passed
and
...".
This
ensures
that
atomic
rules
are
encapsulated.
Note: Sometimes there is the need for rules with more complex aggregation, for example that X% of all images on a web page are expected to have text alternatives. In this case, the page itself needs to become the test target. The expectation would then be "The test target (the page) has a text alternative for X% of all img elements". The logic for calculating the expectations in such rules is left to the implementations, to avoid over-complexity of this rules format.
4.7.2. Expectations for Composite Rules
All
expectations
of
a
composite
rule
must
describe
the
logic
that
is
used
to
determine
a
single
passed
or
failed
outcome
for
a
test
target
,
based
on
the
outcomes
of
rules
in
its
input
rules
.
A
composite
rule
expectation
must
not
use
information
from
input
aspects
.
The
outcome
for
a
composite
rule
is
inapplicable
when
all
input
rules
are
inapplicable.
4.8. Background
An ACT Rule’s Background must contain the Assumptions and Accessibility Support sections. Additional information may be included about the development of the rule, or references to relevant reading in Other Resources, or Related Rules. Whenever a reference is included in the rule, the relationship to the relevant reading can be specified. Examples of relevant background references for a rule for a Web Content Accessibility Guidelines [WCAG22] success criterion could be WCAG 2.2 Understanding documents , WCAG 2.2 Techniques , or WAI-ARIA 1.2 , CSS [css-2018] , or HTML [HTML] specifications.
4.8.1. Assumptions
An
ACT
Rule
must
list
any
known
assumptions,
limitations
or
any
exceptions
for
the
evaluation,
the
test
environment,
technologies
being
used
or
the
subject
being
tested.
For
example,
a
rule
that
would
partially
test
WCAG
2.1
2.2
success
criterion
1.4.3
Contrast
(Minimum)
based
on
the
inspection
of
CSS
properties
could
state
that
it
is
only
applicable
to
HTML
text
content
styleable
with
CSS,
and
that
the
rule
does
not
support
images
of
text.
Sometimes
there
are
multiple
plausible
ways
that
an
accessibility
requirement
can
be
interpreted.
For
instance,
it
is
not
immediately
obvious
if
emoji
characters
are
"text"
or
"non-text
content"
in
the
Web
Content
Accessibility
Guidelines
[WCAG]
[WCAG22]
.
Whatever
the
interpretation
is,
this
must
be
documented
in
the
rule.
While the assumptions must be included in the ACT Rule, it may be empty when there are no known assumptions, limitations or exceptions.
4.9.
4.8.2.
Accessibility
Support
Content
can
be
designed
to
rely
on
the
support
for
particular
accessibility
features
by
different
assistive
technologies
and
user
agents.
For
example,
content
using
a
particular
WAI-ARIA
1.1
1.2
feature
relies
on
that
feature
to
be
supported
by
assistive
technologies
and
user
agents.
This
content
would
not
work
for
assistive
technologies
and
user
agents
that
do
not
support
WAI-ARIA.
See
the
WCAG
definition
for
accessibility
supported
use
of
a
web
technology.
An ACT Rule must include known limitations on accessibility support.
While an accessibility support section must be included in the ACT Rule, it may be empty when there are no known accessibility support issues.
Note: The Website Accessibility Conformance Evaluation Methodology (WCAG-EM) provides guidance on defining an accessibility support baseline for test scenarios, which can help users of ACT Rules to select the appropriate rules for their own circumstance.
4.10.
4.8.3.
Related
Rules
(optional)
Related rules are other rules that test the same accessibility requirement. For example, two related rules of a composite rule can be the two atomic rules that contribute to its outcome. Similarly, each atomic rule in this example can list the other atomic rule and the composite rule as its related rules.
4.8.4. Other Resources (optional)
Whenever a resource is included in the rule, the relationship to the relevant reading can be specified. Examples of relevant background references for a rule for a Web Content Accessibility Guidelines [WCAG22] success criterion could be WCAG 2.2 Understanding documents , WCAG 2.2 Techniques , or WAI-ARIA 1.2 , CSS [css-2018] , or HTML [HTML] specifications.
4.9. Test Cases
Test
cases
are
(snippets
of)
content
that
can
be
used
to
validate
the
implementation
of
ACT
Rules.
Each
rule
must
have
one
or
more
test
cases
for
passed
,
failed
,
and
inapplicable
outcomes
.
A
test
case
consists
of
two
pieces
of
data,
a
snippet
of
each
input
aspect
for
a
rule,
and
the
expected
outcome
for
that
rule.
Test
cases
serve
two
functions,
firstly
as
example
scenarios
for
readers
to
understand
when
the
outcome
of
a
rule
is
passed
,
failed
,
or
inapplicable
.
It
also
serves
developers
and
users
of
accessibility
testing
tools
and
methodologies
to
validate
that
a
rule
is
correctly
implemented.
Each
passed
and
inapplicable
test
case
of
an
ACT
Rule
must
satisfy
all
the
rule’s
conformance
requirements
.
For
each
failed
test
case,
all
conformance
requirements
must
be
not
satisfied
.
4.11.
4.10.
Change
Log
Rule
Versions
It
is
important
to
keep
track
of
changes
to
the
versions
of
ACT
Rules
so
that
users
of
the
rules
can
understand
if
changes
in
test
results
are
due
to
changes
in
the
rules
used
when
performing
the
tests,
or
from
changes
in
the
content
itself.
All
changes
to
previous
versions
of
an
ACT
Rule
that
can
change
the
outcomes
of
an
evaluation
must
be
recorded
in
a
change
log.
The
change
log
can
either
be
part
of
in
the
rule
document
itself
or
be
referenced
from
it.
This section was previously named "Change Log". Either section name is acceptable.
4.11. ACT Rules Format Version
An ACT Rule is written for a specific version of the ACT Rules Format. Each ACT Rule must indicate the version of the ACT Rules Format for which it is written.
4.12. Glossary
ACT
Rules
must
have
a
glossary
section.
The
glossary
must
contain
the
outcome
definition,
as
well
as
any
definitions
used
in
applicability
and
expectations
sections
in
the
rule.
Since
changes
to
the
definition
change
the
rule,
those
definitions
cannot
be
maintained
independently
of
the
rule.
If
a
shared
glossary
is
used
for
rules,
any
definition
changes
must
be
included
result
in
the
change
log
a
new
rule
version
of
all
rules
that
use
that
definition.
4.13. Issues List (optional)
An
ACT
Rule
may
include
a
list
or
a
reference
to
a
list
of
any
known
issues.
The
issues
list
would
be
used
to
record
cases
where
an
outcome
of
an
ACT
Rule
was
failed
where
it
ought
to
have
been
passed
or
inapplicable
,
or
vice
versa.
There
are
several
reasons
why
this
might
occur.
See
rule
accuracy
for
more
information.
The issues list serves two purposes. For users of ACT Rules, the issues list gives insight into why an inaccurate result occurred, as well as provide confidence in the result of that rule. For the designer of the rule, the issues list is also useful to plan future updates to the rule. In a new version of the rule, resolved issues would be moved to the change log.
4.14.
Background
Implementations
(optional)
An ACT Rule may contain a list of implementations , where each implementation has one or more checks that are consistent or partially consistent with that rule. An implementation is an accessibility test methodology or test tool that uses checks to compute outcomes indicating whether an accessibility requirement could be satisfied. These checks can be fully automated, completely manual, or some combination of the two.
Checks are not required to have a one-to-one mapping to an ACT Rule. A single check can be consistent with multiple ACT Rules or a set of checks can together be consistent with a single ACT Rule.
For
each
implementation,
information
about
such
as
the
background
for
following
could
be
included:
Name, title or identifier of any consistent or partially consistent checks
test modes of the
developmentchecks (e.g. automatic, manual, semiAuto)Name of implementation
Version used in testing consistency and coverage
Name of the
rule, or references to relevant reading. The background informationvendor
4.14.1. Consistency
Consistency
describes
how
well
a
check
is
optional,
but
whenever
it
aligned
with
the
intent
of
an
ACT
Rule.
If
consistency
is
included
in
an
ACT
Rule
it
must
be
determined
as
defined
in
this
section.
A
consistent
check
of
an
ACT
Rule
is
one
that
can
be
used
to
identify
some
or
all
non-conformance
issues
as
described
by
the
rule,
rule.
A
check
is
consistent
when
all
the
relationship
to
following
are
true:
True positives : There are no inconsistencies with the
relevant reading can be specified. Examplestest cases, meaning:The
passed
andinapplicable
test cases are not reported asfailed
; andThe
failed
test cases are not reported aspassed
orinapplicable
; and
Completeness : There are no gaps in coverage, meaning:
None of
relevant background referencesthe check 's outcomes areuntested
; andAt least one of the
failed
test cases is reported asfailed
; and
Rule Mapping : The accessibility requirements are reported consistently, meaning:
The check is reported as testing for all the rule’s conformance requirements , except requirements of a level or standard not supported by the implementation; and
All accessibility requirements the rule
forreports to be aWeb Content Accessibility Guidelines [WCAG]part of are either conformance requirementssuccess criterion could be WCAG 2.1 Understanding documents , WCAG 2.1 Techniques ,orWAI-ARIA 1.1 , CSS [css-2018]secondary requirements , except for requirements of accessibility requirement documents not mentioned in the rule.
When a check reports more than one outcomes for a test case, the first outcome that appears on the following ordered list is considered for determining consistency:
failed
untested
cantTell
passed
inapplicable
4.14.2. Partial Consistency
A
check
that
is
not
consistent
is
considered
partially
consistent
if
the
true
positive
condition
is
true
and
not
all
outcomes
it
reports
for
the
rule’s
test
cases
are
cantTell
or
HTML
[HTML]
untested
.
4.14.3.
Consistency
with
multiple
checks
specifications.
An implementation can include checks that test only parts of a single ACT Rule. The combination of those checks can be consistent if all parts of the ACT Rule are covered. The consistency of a set of checks can be determined by treating all partially consistent checks as though they were a single check.
A set of checks is consistent with an ACT Rule if it meets all requirements for a consistent check . The outcomes for this set of checks is a list of all outcomes of checks that are partially consistent with the ACT Rule. The accessibility requirements of this set of checks is a list of all accessibility requirements of checks that are partially consistent with the ACT rule.
4.15.
Acknowledgements
Acknowledgments
(optional)
An
ACT
Rule
may
contain
acknowledgements.
acknowledgments.
This
can
include,
but
is
not
limited
to:
-
List of rule authors
-
List of rule reviewers/contributors
-
Funding or other support
5. Rule Accuracy
This section is non-normative .
While test cases can be used to determine if an ACT Rule was correctly implemented, they do not guarantee that implementations will never produce incorrect results. When writing ACT Rules, it is almost inevitable that edge cases will be overlooked. Technologies are always evolving, and content authors are constantly coming up with new and unexpected ways to use them. Some examples of causes for inaccuracy are:
-
Assumptions were made about the test subject that turned out to be untrue
-
Technologies were used in an unusual and difficult to predict manner
-
Technologies have changes, or aspects of the technologies were overlooked
-
The accessibility requirement was not correctly interpreted
There are two types of inaccuracies that can produce incorrect results. Inaccuracies in the implementation can be addressed with test cases, but inaccuracies in the ACT Rule itself cannot. After all, rule inaccuracies come from the rule author being unaware of a particular edge case.
When a test result incorrectly indicates non-conformance to an accessibility requirement, this is known as a false positive. Opposite, when a rule incorrectly indicates conformance, this is a false negative. A percentage of false positives and false negatives can be calculated by comparing it to results from an accessibility audit:
-
False positives: This is the percentage of test targets , that were
failed
by the rule, but satisfy the accessibility requirements . -
False negatives: This is the percentage of test targets , that were
passed
by the rule, but do not satisfy the accessibility requirements .
The ever present possibility of false positives and false negatives with ACT Rules means they will likely require ongoing maintenance. Designing a process for maintaining ACT Rules is outside the scope of the ACT Rules Format, which is limited to the rules themselves. Nevertheless, it is suggested that rule authors work out a process for maintaining their rules.
6. Harmonization
This section is non-normative .
While the ACT Rules Format is designed to stimulate harmonization, there are no direct requirement in the ACT Rules Format that prevent a rule author from writing rules inconsistent with already established ACT Rules. Neither are there requirements for ACT Rules to have a certain number of implementations, or to have a certain level of accuracy. This allows quality requirements to be different for different rulesets, and allows them to develop over time.
Harmonization occurs when a group of rule implementors collectively accept the validity of an ACT Rule. For example, a community group of accessibility testing tool vendors could declare they have harmonized on a particular set of ACT Rules. Such a group can set acceptance criteria for rules, and have quality requirements that go beyond the ACT Rules Format.
An example of such a process is the WCAG ACT Review Process .
7. Definitions
- Accessibility requirement
-
AAn accessibility requirement is acondition thatrequirement aimed at improving access for people with disabilities to an ICT product. In the context of ACT rules mapping, a requirement can be compulsory or advisory. When compulsory, it has to be satisfied in order to conform to a standard, or to comply with a contract, policy or regulation.An accessibility requirementWhen advisory, it isa requirement aimed at improving access for people with disabilitiesrecommended, but not satisfying it does not lead toan ICT product.non-conformance or non-compliance.A common example of accessibility requirements are the WCAG success criteria. There are other standards, including W3C standards, that have recommendations for accessibility, such as WAI-ARIA and HTML. Accessibility requirements are also often found in company policies, regional standards or in legislation.
- Accessibility requirements document
-
A document, such as a standard, contract, policy or regulation, that includes accessibility requirements . For example, WCAG
2.1,2.2, WAI-ARIA1.1,1.2, HTML 5.2, EPUB Accessibility 1.0, BBC HTML Accessibility Standards v2.0 - Check
A procedure resulting in one or more outcomes when used to evaluate the accessibility of a web page or other test subject. The procedure can be a step by step description of how to manually perform a test, a fully automated test, or some combination of manual and automated testing.
- Implementation
An accessibility test methodology or test tool, containing a set of checks which can be used to determine (non-)conformance of accessibility requirements .
- Outcome
-
A conclusion that comes from evaluating an ACT Rule on a test subject or one of its constituent test target . An outcome can be one of the
threefive following types:-
Inapplicable:inapplicable: No part of the test subject matches the applicability -
Passed:passed: A test target meets all expectations -
Failed:failed: A test target does not meet all expectations - cantTell: Whether the rule is applicable, or not all expectations were met could not be fully determined by the tester.
- Untested: The tester has not attempted to evaluate the test subject.
Note: A rule has one
passed
orfailed
outcome for every test target . When a tester evaluates a test target it can also be reported ascantTell
if the rule cannot be tested in its entirety. For example, when applicability was automated, but the expectations have to be evaluated manually.When there are no test targets the rule has one
inapplicable
outcome. If the tester is unable to determine whether there are test targets there will be onecantTell
outcome. And when no evaluation has occurred the test target has oneuntested
outcome. This means that each test subjectwill havealways has one or more outcomes.Note: Implementers using the [EARL10-Schema]Outcomes used in ACT Rules canexpress the outcome withbe expressed using the outcome property. In addition to passed , failed and inapplicable , EARL 1.0 also defined an incomplete outcome. While this cannot be the outcomeofan ACT Rule when applied in its entirety, it often happens that rules are only partially evaluated. For example, when applicability was automated, but the expectations have to be evaluated manually. Such "interim" results can be expressed withtheincomplete outcome.[EARL10-Schema] . -
- Test Subject
-
A resource or collection of resources that can be evaluated by an ACT Rule.
Note: Implementers using the [EARL10-Schema] can express the test subject with the subject property
- Test Target
-
A distinct part of the test subject , as defined by the applicability .
Note: Implementers using the [EARL10-Schema] can express the test target with the pointer property
Appendix 1: Expressing ACT Rule results with JSON-LD and EARL
This section is non-normative .
This
section
provides
examples
of
expressing
results
from
carrying
out
ACT
Rules
using
EARL
and
JSON-LD
(See
Evaluation
and
Report
Language
and
A
JSON-based
Serialization
for
Linked
Data
(JSON-LD)
).
More
examples
and
background
is
proivided
provided
on
the
JSON-LD
serialization
of
EARL
GitHub
repository.
Examples in this section include:
-
Example 1: Minimal outcome from one
Assertionassertion -
Example 2: Results from more than one
Assertionassertion -
Example 3: Aggregating based on
Requirementrequirement -
Example 4: Aggregating based on 'Test Subject'
-
Example 5: Assumed
Contextcontext for this section
Example
1:
Minimal
outcome
for
one
Assertion
assertion
{ "@context" : "context.json" , "@type" : "Assertion" , "assertedBy" : "https://example.org/MyTool" , "subject" : "https://example.org/page1.html" , "test" : "ACT-CG:rules/23a2a8" , "result" : { "outcome" : "earl:failed" , "pointer" : "html > body > h1:first-child" } }
Example
2:
Results
for
more
than
one
Assertion
assertion
{ "@context" : "context.json" , "@graph" : [{ "@type" : "Assertion" , "assertedBy" : "https://example.org/MyTool" , "subject" : "https://example.org/page1.html" , "test" : "ACT-CG:rules/23a2a8" , "result" : { "outcome" : "earl:failed" , "pointer" : "html > body > h1:first-child" } }, { "@type" : "Assertion" , "assertedBy" : "https://example.org/AnotherTool" , "subject" : "https://example.org/page1.html" , "test" : "ACT-CG:rules/23a2a8" , "result" : { "outcome" : "earl:passed" , "pointer" : "html > body > h1:nth-child(2)" } }] }
Example
3:
Aggregating
based
on
Requirement
requirement
(eg.
WCAG
Success
Criteria)
{ "@context" : "context.json" , "@type" : "Assertion" , "assertedBy" : "https://example.org/MyTool" , "subject" : "https://example.org/page1.html" , "test" : { "@type" : "earl:TestRequirement" , "@id" : "WCAG22:non-text-content" }, "result" : { "outcome" : "earl:failed" , "source" : [{ "test" : "ACT-CG:rules/23a2a8" , "result" : { "outcome" : "earl:failed" , "pointer" : "html > body > h1:first-child" } }, { "test" : "ACT-RULES-CG:rules/23a2a8" , "result" : { "outcome" : "earl:passed" , "pointer" : "html > body > h1:nth-child(2)" } }] } }
Example 4: Aggregating based on 'Test Subject' (eg. for a website)
{ "@context" : "context.json" , "@type" : "Assertion" , "assertedBy" : { "@type" : "Organization" , "@id" : "_:myOrg" , "title" : "My Organization" , "description" : "Accessibility testing service" , "homepage" : "http://example.org/myOrg/" }, "subject" : { "@type" : [ "WebSite" , "TestSubject" ], "@id" : "https://example.org/" }, "test" : { "@type" : "earl:TestRequirement" , "@id" : "http://www.w3.org/WAI/WCAG2A-Conformance" }, "result" : { "outcome" : "earl:failed" , "source" : [{ "test" : { "@type" : "earl:TestRequirement" , "@id" : "WCAG22:non-text-content" }, "result" : { "outcome" : "earl:failed" , "source" : [ …] } }, { "test" : { "@type" : "earl:TestRequirement" , "@id" : "WCAG22:audio-only-and-video-only-prerecorded" }, "result" : { "outcome" : "earl:passed" , "source" : [ …] } }, { "test" : { "@type" : "earl:TestRequirement" , "@id" : "WCAG22:captions-prerecorded" }, "result" : { "outcome" : "earl:passed" , "source" : [ …] } }, …] } }
Example
5:
Assumed
Context
context
for
this
section
{ "@vocab" : "http://www.w3.org/ns/earl#" , "cnt" : "http://www.w3.org/2011/content#" , "dct" : "http://purl.org/dc/terms/" , "earl" : "http://www.w3.org/ns/earl#" , "foaf" : "http://xmlns.com/foaf/0.1/" , "htp" : "http://www.w3.org/2011/http#" , "ptr" : "https://www.w3.org/2009/pointers#" , "schema" : "http://schema.org/" , "xsd" : "https://www.w3.org/2001/XMLSchema#" , "WCAG20" : "https://www.w3.org/TR/WCAG20#" , "WCAG22" : "https://www.w3.org/TR/WCAG22#" , "ACT-RULES-CG" : "https://act-rules.github.io/" , "WebSite" : "schema:WebSite" , "WebPage" : "schema:WebPage" , "title" : "dct:title" , "description" : "dct:description" , "date" : "dct:date" , "hasVersion" : "dct:hasVesion" , "isPartOf" : "dct:isPartOf" , "hasPart" : "dct:hasPart" , "source" : "dct:source" , "Agent" : "foaf:Agent" , "Person" : "foaf:Person" , "Organization" : "foaf:Organization" , "Group" : "foaf:Group" , "Document" : "foaf:Document" , "name" : "foaf:name" , "firstName" : "foaf:firstName" , "surname" : "foaf:surname" , "mbox" : "foaf:mbox" , "mbox_sha1sum" : "foaf:mbox_sha1sum" , "member" : "foaf:member" , "homepage" : "foaf:homepage" , "assertedBy" : { "@type" : "@id" }, "subject" : { "@type" : "@id" }, "test" : { "@type" : "@id" }, "mode" : { "@type" : "@id" }, "outcome" : { "@type" : "@id" }, "pointer" : { "@id" : "earl:pointer" , "@type" : "ptr:CSSSelectorPointer" } }
Appendix 2: Acknowledgments
This section is non-normative .
Participants of the AG WG active in the development of this document
Shadi Abou-Zahra, Trevor Bostic, Romain Deltour, Kathy Eng, Wilco Fiers, Alistair Garrison, Kasper Isager, Maureen Kraft, Mary Jo Mueller, Jey Nandakumar, Charu Pandhi, Stein Erik Skotkjerra, Anne Thyme Nørregaard, Kathleen Wahlbin
Enabling funders
This publication has been developed with support from the WAI-Tools Project , co-funded by the European Commission (EC) under the Horizon 2020 Program (Grant Agreement 780057). The content of this publication does not necessarily reflect the views or policies of the European Commission (EC) or any of the European Union (EU) Member States.
Appendix 3: Change History
This section is non-normative .
- 4. Rule Structure Accessibility Support and Assumptions section are now subsections of Background.
- 4.4. Accessibility Requirements Mapping Accessibility requirements are now categorized as either Conformance requirements or Secondary requirements.
-
4.6
Applicability
Subjective
applicability
statements
are
now
allowed.
Objective
and
plain
language
requirements
have
been
made sincereduced to should instead of must . - 4.8. Background New optional Related Rules and Other Resources subsections have been added
- 4.10. Change Log The Change Log has been renamed to Rule Versions
- 4.11 ACT Rules Format Version New requirement to identify ACT Rules Format version compatibility
- 4.14. Implementations New section has been added, including a method for determining consistency with ACT Rules
- 7. Definitions The Outcome definition is updated to include cantTell and untested
- Overall Links to W3C specifications are updated to their latest recommendation
All
changes
from
the
previous
Proposed
Recommendation
published
version
can
be
viewed
using
the
Editor’s
draft
of
30
July
2019
.
to
Rules
Format
1.0
Recommendation
diff
link