Finals HCI : Evaluation Techniques
Model-based evaluation
Calculations or simulations (type of approach)
True
Can be performed on a design specification so it is useful for evaluating early design. But it can also be used on prototypes, storyboards and fully functioning systems. It is therefore a flexible, relatively cheap approach. Hence it is often considered a discount usability technique. (T/F)
True
Cognitive walkthrough was originally proposed and later revised by Polson and colleagues, as an attempt to introduce psychological theory into the informal and subjective walkthrough technique. (T/F)
-cognitive walkthrough -heuristic evaluation -the use of models -use of previous work.
Consider 4 approaches to expert analysis:
Number of fixations
The more fixations the less efficient the search strategy. (Eye tracking for usability evaluation)
True
The origin of the cognitive walkthrough approach to evaluation is the code walkthrough familiar in software engineering. Walkthroughs require a detailed review of a sequence of actions. (T/F)
Match between system and the real world
The system should speak the user's language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. (heuristics)
Automatic protocol analysis tools
These offer a means of editing and annotating video, audio and system logs and synchronizing these for detailed analysis. (Observational technique)
True
Think aloud is a form of observation where the user is asked to talk through what he is doing as he is being observed; for example, describing what he believes is happening, why he takes an action, what he is trying to do. (T/F) (Observational technique)
True
(Using Previous Studies In Evaluation) Some of this is specific to a particular domain, but much deals with more generic issues and applies in a variety of situations. Examples of such issues are the usability of different menu types, the recall of command names, and the choice of icons. (T/F)
-Laboratory studies -Field studies
2 Styles of evaluation
True
A heuristic is a guideline or general principle or rule of thumb that can guide a design decision or be used to critique a decision that has already been made. (T/F)
True
A hypothesis is a prediction of the outcome of an experiment. (T/F)
True
A third expert-based approach is the use of models. Certain cognitive and design models provide a means of combining design specification and evaluation into the same framework. (T/F)
Flexibility and efficiency of use
Allow users to tailor frequent actions. (heuristics)
Visibility of system status
Always keep users informed about what is going on, through appropriate feedback within reasonable time. (heuristics)
Dependent variables
Are the variables that can be measured in the experiment, their value is ________ on the changes made to the independent variable. (type of variable)
Independent variables
Are those elements of the experiment that are manipulated to produce different conditions for comparison. (type of variable)
Jakob Nielsen and Rolf Molich
Developed by _______________ and ________________, is a method for structuring the critique of a system using a set of relatively simple and general heuristics.
Aesthetic and minimalist design
Dialogs should not contain information that is irrelevant or rarely needed. (heuristics)
Experimental Video Annotator
EVA(meaning) is a system that runs on a multimedia workstation with a direct link to a video recorder. The evaluator can devise a set of buttons indicating different events.
True
EVALUATION THROUGH USER PARTICIPATION Include empirical or experimental methods, observational methods, query techniques, and methods that use physiological monitoring, such as eye tracking and measures of heart rate and skin conductance. (T/F)
-Participants -Variables -Hypotheses
Empirical methods: experimental evaluation 3 terms:
Help users recognize, diagnose and recover from errors
Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution. (heuristics)
3
Evaluation has _ main goals.
-Eye tracking for usability evaluation (eye track) -Physiological measurements (sensors)
Evaluation through monitoring physiological responses: (what are the two)
Using Previous Studies
Experimental psychology and human-computer interaction between them possess a wealth of experimental results and empirical evidence (type of approach)
Variables
Experiments manipulate and measure variables under controlled conditions, in order to test the hypothesis. (empirical method: experimental)
Help and documentation
Few systems can be used with no instructions so it may be necessary to provide ___ and _________________.
1. A specification or prototype of the system. It doesn't have to be complete, but it should be fairly detailed. Details such as the location and wording for a menu can make a big difference. 2. A description of the task the user is to perform on the system. This should be a representative task that most users will want to do. 3. A complete, written list of the actions needed to complete the task with the proposed system. 4. An indication of who the users are and what kind of experience and knowledge the evaluators can assume about them.
Four things in cognitive walkthrough:
-Goals -Operators -Methods -Selection
GOMS meaning
True
If participants are not actual users, they should be chosen to be of a similar age and level of education as the intended user group. (T/F)
Evaluation through user participation
In this section we will look at a number of different approaches to ____________ through _______ _____________.
Heart activity
Indicated by blood pressure, volume and pulse. These may respond to stress or anger. (Physiological measurements)
Activity of the sweat glands
Indicated by skin resistance or galvanic skin response (GSR). These are thought to indicate levels of arousal and mental effort. (Physiological measurements)
Scan path
Indicating areas of interest, search strategy and cognitive load. Moving straight to a target with a short fixation at the target is the optimal scan path but plotting scan paths and fixations can indicate what people look at, how often and for how long. (Eye tracking for usability evaluation)
Participants
Is vital to the success of any experiment. (empirical method: experimental)
Hypotheses
It is framed in terms of the independent and dependent variables, stating that a variation in the independent variable will cause a difference in the dependent variable. (empirical method: experimental)
Fixation duration
Longer fixations may indicate difficulty with a display. (Eye tracking for usability evaluation)
Error prevention
Make it difficult to make errors. Even better than good error messages is a careful design that prevents a problem from occurring in the first place. (heuristics)
Recognition rather than recall
Make objects, actions and options visible. (heuristics)
Electrical activity in the brain
Measured by the electroencephalogram (EEG). These are associated with decision making, attention and motivation. (Physiological measurements)
Electrical activity in muscle
Measured by the electromyogram (EMG). These appear to reflect involvement in a task. (Physiological measurements)
10
Nielsen's have __ heuristics.
-Think aloud and cooperative evaluation -Protocol analysis -Automatic protocol analysis tools -Post-task walkthroughs
Observational techniques:
True
Participants should be chosen to match the expected user population as closely as possible. (T/F)
-Paper and pencil (primitive) -Audio recording (thinking aloud) -Video recording (camera) -Computer logging (system) -User notebooks (logs of activity)
Protocol analysis five methods.
-General (background, age sex, prev experience) -Open-ended (unprompted opinion) -Scalar (numeric scale) -Multi-choice (multiple choices) -Ranked (ordering)
Questionnaires There are a number of styles of question that can be included in the questionnaire. These include the following:
-Independent variables -Dependent variables
Two main types of variables:
True
User participation in evaluation tends to occur in the later stages of development when there is at least a working prototype of the system in place. (T/F)
User control and freedom
Users often choose system functions by mistake and need a clearly marked 'emergency exit' to leave the unwanted state without having to go through an extended dialog. (heuristics)
Consistency and standards
Users should not have to wonder whether words, situations or actions mean the same thing in different contexts. (heuristics)
-to assess the extent and accessibility of the system's functionality -to assess users' experience of the interaction -to identify any specific problems with the system.
What are the 3 main goals: ...
GOMS
low-level modelling techniques such as keystroke-level model, design methodologies, design rationale, dialog models
Field studies
takes the designer or evaluator out into the user's work environment in order to observe the system in action. Again this approach has its pros and cons. (style of evaluation)
Laboratory studies
users are taken out of their normal work environment to take part in controlled tests, often in a specialist usability laboratory (although the 'lab' may simply be a quiet room). This approach has a number of benefits and disadvantages. (style of evaluation)