 |
Convened by the American Association for the
Advancement of Science
Main | Program | Participants
| Synopses
The research issues described here are based on the findings and recommendations
in NIST Special Publication 500-256 "Improving the Usability and Accessibility
of Voting Systems and Products" which was written to address the mandate
in the Help America Vote Act (HAVA) of 2002, Public Law 107-252 for the Election
Assistance Commission in consultation with the National Institute of Standards
and Technology (NIST) to submit a human factors report to Congress. This report
"assesses the areas of human factor research, including usability engineering
and human-computer and human-machine interaction, which feasibly could be applied
to voting products and systems design to ensure the usability and accuracy of
voting products and systems, including methods to improve access for individuals
with disabilities (including blindness) and individuals with limited proficiency
in the English language and to reduce voter error and the number of spoiled
ballots in elections." The report was delivered to Congress in May 2004
and can be found at http://vote.nist.gov and http://www.eac.gov .
The report examines the key needs for developing a set of voting system standards
that will ensure a high level of usability and accessibility. Because voting
equipment will be tested for conformance against these standards and qualified/certified
to the standard, the standards must be testable and the test protocols must
assure repeatability of the results. There are ten recommendations in the report
covering not just voting equipment standards, but also recommending the development
of guidelines for ballot design, layout, usability of documentation and training,
and vendor user centered design. I also refer readers to Chapter 5 of that report
which summarizes current research in human factors, user interaction, usability,
and accessibility and discusses what types of research in these areas can be
applied to voting systems. In this short synopsis, we focus on research issues
for usability testing.
In general, the single most critical need identified in the report is a set
of usability standards for voting systems that are performance-based and that
support objective measures and associated conformance test procedures that can
be used for the certification and qualification of voting products and systems.
These measures and conformance test procedures primarily depend upon having
objective usability testing processes. (Note that when we are referring to usability
we include all users, with and without disabilities, at different levels of
reading proficiency and from different cultural and economic backgrounds.) A
set of design requirements cannot properly address the issues of usability for
voting system products. Also, no document can contain a sufficient set of design
requirements to ensure voting product usability unless the document completely
specifies a design already shown to be usable. Further, formative or diagnostic
tests are valuable tools in the design process; they do not guarantee that the
final product is usable as measured by the standard usability metrics (efficiency,
effectiveness, and satisfaction) since they are used during the design process,
not on a final product. Even tests that are conducted on a final product design
are generally not conducted in a way that would allow the results to be generalized
to the intended populations (i.e., the participants of the study may or may
not be appropriately extrapolated to a majority of all actual users). This is
particularly true for voting system products since the range of users required
for such a test would make this type of testing cost prohibitive to most vendors.
In addition, there are currently no defined standards for usability metrics
that vendors could use as benchmarks for their testing.
To ensure usability of a voting product, it is imperative that the product
be tested with actual users performing realistic tasks in a realistic environment,
in sufficient numbers, and using a broad enough cross-section of users to be
truly representative of the voting population. Further, to ensure good usability
of the system, we must test not only the interaction of voter with the product
but also the interaction of the voters, election administrators, and poll workers
with the entire voting system.
We recommend the development of a valid, reliable, repeatable, and reproducible
process for usability testing of voting products against agreed-upon usability
pass/fail requirements. In particular, there must be a careful definition of
the metrics, such as what counts as an error, how to measure error rate, time
on task, etc., by which systems are to be measured. The pass/fail criteria should
be restricted to usability problems leading to partial failure, and usability
problems leading to total failure. Since we are dealing with outcomes, usability
problems prior to success need not be specifically included, but would be represented
in the time on task measure from testing. Note that while excessive time required
does not lead to failure, it is still unacceptable.
Since human users are involved in the process, it is unlikely that the error
rates will be zero for any criteria established, so a specific acceptable error
rate and margin of error will likely be required. For example, it may be possible
to enforce a requirement that no user be allowed to consciously cast a ballot
with an over vote for one or more contests since this error represents the action
of the voter. However, a voter still might inadvertently cast a vote for an
unintended candidate in any product but this error cannot be detected without
knowing the intent of the voter. Yet, both of these conditions must be tested.
This test process must be defined at a high enough level of generality that
the same procedure could be applied to any product (i.e., we do not want to
define product-specific tests). Otherwise, the results for various products
would not be comparable. Fortunately, the task requirements for voting are specific
enough that this should not be difficult to do. It might be necessary, however,
to have technology-specific variants of the test procedure and protocol (e.g.
DRE vs. paper-based), although we believe the differences can and should be
kept minimal.
Research needs to be conducted to determine: (1) the nature of errors possible
during a voting process (this includes voter errors and poll worker errors),
and (2) the level (rate) of these errors (both the current levels for existing
products and recommendations for "acceptable" levels of each error
type). Once this information is available, we recommend that a set of repeatable
and reproducible processes be defined and that each voting product be tested
using these test processes and usability test pass/fail criteria. This would
include the definition of all test procedures, the data collection required,
the data analysis approach, participant screening and selection procedures,
and reporting requirements.
|
|
