HCI 3
Summarize What were the major trends of the tests? What were the major problem areas? How did users in general feel about the interface? Prioritize Which are the most important fixes? Note: the problem, the suggested fix, estimated time for the fix Rank the fixes in terms of importance This is important, since not all fixes will be made
2 impacts of your Usability Tests
-FAE- http://fae20.cita.illinois.edu/ -WAVE- http://wave.webaim.org/ -A-Checker- http://achecker.ca/checker/index.php
3 Free accessibility testing tools
Task/time performance Problematic areas Subjective comments (especially on things such as layout and color, which are hard to quantify)
3 tasks for when you have finished UT...
-Odellus ComplyFirst http://www.odellus.com/ -Deque Worldspace: http://www.deque.com/ -Compliance Sheriff (used to be from HiSoftware) https://www.cryptzone.com/products/compliance-sheriff -SSB AMP (used to be InFocus) http://www.ssbbartgroup.com/amp/
4 Commonly used accessibility tools for Section 508 compliance (for $)
You will usually use only one or two of these, not all of them. -Heuristic review (most common) -Consistency inspection (most common) -Guidelines review (somewhat common) -Cognitive walkthrough (less common) -Formal usability inspection (least common) -You could consider the AMAI (my method for accessibility inspections) an expert review
6 Types of expert reviews
-They can't determine "common sense" -They often miss actual problems, and point out problems where there are none
A lot of previous research has found that automated tools are problematic:
Software tools examine an interface, comparing the interface with a large series of guidelines (such as WCAG 2.0)
Automated Usability Testing
web
Automated tools tend to focus on _____ accessibility
Family increases, friends decreases Less communication weakens strength
Changes relationships
-Experts simulate being users, walking through tasks, especially high frequency and high error areas (used infrequently) -This may be used if there are specific areas of concern that require deep inspection
Cognitive walkthrough
Multiple screens are examined for consistency in layout, color, font, positioning, terminology, look and feel, etc...
Consistency inspections
Video/audio? Task performance data? Time performance data? Error rates? (are any of these important for your interface?) Thinking aloud protocol (qualitative data) Satisfaction surveys?
Decide what type of data you are going to collect:
The difference is that an interface expert, NOT a representative user, examines an interface using a structured methodology Tends to be INSPECTION-focused, not task-focused
Expert Review vs UT
An expert review is AKA an expert inspection or a usability inspection
Expert review?
Designers justify their choices to a panel of usability experts (VERY uncommon)
Formal usability inspection
-Like a heuristic evaluation, but uses a much longer set of guidelines (20-200). -They can be very intensive and take a long time, however, when done well, can be very helpful and thorough
Guidelines review
-WCAG -Section 508 ADA -EU mandate 376 -BSI 8878 Web access code of practice: Global Public Inclusive Infrastructure -UN convention on rights of persons with disabilities (Art 9 and 21)
HCI informing public policy
non-partisan
HCI policies generally
-When an expert compares an interface to a short (7-10) set of guidelines, known as heuristics -These heuristics focus the major problems in interfaces -It's big "bang for the buck" -It's the most common type of expert review
Heuristic review
You could have multiple quantitative metrics: Task performance Time performance User satisfaction (use the QUIS!) Error Rate Time to Learn In a fixed laboratory (with video/audio taping) 100 users taking part Re-do the interface multiple times, until it reaches a certain threshold (say, 90% success on task performance)
How formal is your test?
Some say that 5 users will find 80% of usability flaws Others say that 7 users for a small project, 15 for a larger project? Another: maximum cost/benefit is 3.2 users This is a raging debate in HCI. Really! The tasks you choose have more of an influence on flaws found than # of users!
How many users do you need for UT?
Successfully completed Completed but incorrect Attempted, but did not complete (this includes starting a task and moving on to another task before completing) Did not attempt a task
How to describe task performance
Automated Usability Testing
If usability testing is the first method, and expert reviews are the second method, what's the third method?
More communication more loneliness Communication with friends and family Decline depression Communication with stranger Increase depression
Influence well-being
-User: Knows their tasks and domain knowledge Doesn't know interfaces -Interface Expert: Knows interfaces Doesn't know aspects of the interface that specifically relate to tasks or domain knowledge Expert reviews should ALWAYS come before user reviews
Interface expert vs user
Usability testing is very different from a controlled experiment, even if some of the methods have been adopted from experimental research UT: the goal is to find and fix flaws in a certain interface...no more, no less Experimental design: Have two or more treatment groups and attempt to show statistically significant results than can be generalized UT: interfaces changes are sometimes made after each user!
Is UT an experiment?
In a usability test, the user is NOT being tested, the software application is being tested (users can never fail!) The user helps us uncover some of the problems with an interface We can then prioritize the biggest problems in the interface and fix them!
Key thing to remember about usability testing
Usability testing can take a number of forms: Very structured, in a videotaped setting, 100 users, quantitative measurements All the way to: Very unstructured, sitting next to 5 users and asking them what they didn't like about an interface (called quick-and-dirty usability)
Many ways to do usability testing
-Building HCI rep in science and technology -Get people more involved USACM, EUACM -Greaater presence in general immunity -HCI content edu -Workshops, annual events
Org actions for HCI involvement
Designers will try their best to design an appropriate interface, but users need to test the interfaces to determine if it meets their needs! Designers become invested in a certain design, and it's hard to change their designs or imagine other possibilities It can also be hard to be objective about your own design Usability testing attempts to address this!
Problems with designers that UT addresses
HCI limited by data collection laws, public/private record gov
Public policy in HCI is limited by
-National survey - describe particular communication Session that happened yesterday In person/telephone better than digital Digital sessions are shorter Less content Care less about Less substantial topics
Quality communication
Confirm with users that they are coming and when Confirm that the location you scheduled is still available Test any equipment that you will be using both usability lab equipment and the user computer) Especially the camera and microphones Make sure that task list and satisfaction surveys are ready to go!
Right before a UT
Recruit users Choose a location Plan testing tasks
Steps WAY before a UT
More responses faster longer content more Satisfied
Support for diseases
-Tools that evaluate the usability/accessibility of screens -Tools that simulate the presence of a disability in terms of interface design (often used for testing pages for color-blindness)
There are two different types of tools:
Educational background Computer training Age Job responsibility (test medical applications with only Doctors or Nurses, as appropriate!)
These users must represent the targeted user for this interface and application in terms of:
Formative testing Earlier stage of development More focused on general design choices More qualitative Summative testing Later stage of development Focused on specific interface widgets More quantitative No dividing line between them
Two general approaches usability testing
User-based tests are the most common! There are also: Expert-based tests (AKA inspections or walkthroughs) Automated tests (software)
Types of usability tests
Fully functional interfaces Partially-working prototypes Ever paper prototypes!
Usability Testing can occur with types of prototypes:
Focus on high-level organizational views of whether accessibility features are present, not if a specific page is accessible
Way to approach automated tools
The process of testing and improving the ease of use of a specific interface
What is usability testing?
In a workplace setting You can actually bring a portable usability lab to a workplace setting (it's surprisingly non-intrusive, and captures both the user's face and screenshots) Equipment can take time to setup In general, a workplace setting is more naturalistic and users feel more comfortable You can always just sit next to the user with pencil and paper
Where can a test take place 2?
Over the Internet You can record their screenshots using data logging software (like UserZoom) Or you can just ask them to e-mail/call comments Advantages: Access to more users, fewer logistical concerns Clickstream data can be collected, heatmaps, data automatically analyzed Disadvantages: Can't see user's face, can't hear their voice Can't provide instructions when things "go wrong" Can't ask any probing questions, often miss the context of what was happening
Where can a test take place 3?
In a fixed, formal lab Two separate rooms, one for user, one for evaluators Soundproofing and/or a one-way mirror between the two rooms Video/audio recording User doesn't feel as if they are being watched! You record the user's screen, user's face and their reactions to the interface Major companies and government organizations have multiple usability lab setups This can cost >$100k to formally install...or you can rent! Or build cheaper versions using webcams
Where can a test take place?
-Expert reviews can find the major interface flaws, the things that are "the low-hanging fruit" -Then, you make immediate improvements to the interface before users get to evaluate it -Then, the user testing can find the more fine-grained problems, the issues that are more tied into the tasks and domain knowledge, which won't be obvious to the expert reviewers -The combination of tests can uncover more interface flaws than just one method
Why expert reviews first?
Ask a few follow-up questions after the usability test Make sure to be specific to the site/app/os tested Ask about suggested improvements, surprises, color scheme, terminology, etc. Make sure to ask if the user had ever used this site/app/os before
satisfaction questions in UT
