Finals HCI : Evaluation Techniques

Model-based evaluation

Calculations or simulations (type of approach)

True

Can be performed on a design specification so it is useful for evaluating early design. But it can also be used on prototypes, storyboards and fully functioning systems. It is therefore a flexible, relatively cheap approach. Hence it is often considered a discount usability technique. (T/F)

True

Cognitive walkthrough was originally proposed and later revised by Polson and colleagues, as an attempt to introduce psychological theory into the informal and subjective walkthrough technique. (T/F)

-cognitive walkthrough -heuristic evaluation -the use of models -use of previous work.

Consider 4 approaches to expert analysis:

Number of fixations

The more fixations the less efficient the search strategy. (Eye tracking for usability evaluation)

True

The origin of the cognitive walkthrough approach to evaluation is the code walkthrough familiar in software engineering. Walkthroughs require a detailed review of a sequence of actions. (T/F)

Match between system and the real world

The system should speak the user's language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. (heuristics)

Automatic protocol analysis tools

These offer a means of editing and annotating video, audio and system logs and synchronizing these for detailed analysis. (Observational technique)

True

Think aloud is a form of observation where the user is asked to talk through what he is doing as he is being observed; for example, describing what he believes is happening, why he takes an action, what he is trying to do. (T/F) (Observational technique)

True

(Using Previous Studies In Evaluation) Some of this is specific to a particular domain, but much deals with more generic issues and applies in a variety of situations. Examples of such issues are the usability of different menu types, the recall of command names, and the choice of icons. (T/F)

-Laboratory studies -Field studies

2 Styles of evaluation

True

A heuristic is a guideline or general principle or rule of thumb that can guide a design decision or be used to critique a decision that has already been made. (T/F)

True

A hypothesis is a prediction of the outcome of an experiment. (T/F)

True

A third expert-based approach is the use of models. Certain cognitive and design models provide a means of combining design specification and evaluation into the same framework. (T/F)

Flexibility and efficiency of use

Allow users to tailor frequent actions. (heuristics)

Visibility of system status

Always keep users informed about what is going on, through appropriate feedback within reasonable time. (heuristics)

Dependent variables

Are the variables that can be measured in the experiment, their value is ________ on the changes made to the independent variable. (type of variable)

Independent variables

Are those elements of the experiment that are manipulated to produce different conditions for comparison. (type of variable)

Jakob Nielsen and Rolf Molich

Developed by _______________ and ________________, is a method for structuring the critique of a system using a set of relatively simple and general heuristics.

Aesthetic and minimalist design

Dialogs should not contain information that is irrelevant or rarely needed. (heuristics)

Experimental Video Annotator

EVA(meaning) is a system that runs on a multimedia workstation with a direct link to a video recorder. The evaluator can devise a set of buttons indicating different events.

True

EVALUATION THROUGH USER PARTICIPATION Include empirical or experimental methods, observational methods, query techniques, and methods that use physiological monitoring, such as eye tracking and measures of heart rate and skin conductance. (T/F)

-Participants -Variables -Hypotheses

Empirical methods: experimental evaluation 3 terms:

Help users recognize, diagnose and recover from errors

Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution. (heuristics)

3

Evaluation has _ main goals.

-Eye tracking for usability evaluation (eye track) -Physiological measurements (sensors)

Evaluation through monitoring physiological responses: (what are the two)

Using Previous Studies

Experimental psychology and human-computer interaction between them possess a wealth of experimental results and empirical evidence (type of approach)

Variables

Experiments manipulate and measure variables under controlled conditions, in order to test the hypothesis. (empirical method: experimental)

Help and documentation

Few systems can be used with no instructions so it may be necessary to provide ___ and _________________.

1. A specification or prototype of the system. It doesn't have to be complete, but it should be fairly detailed. Details such as the location and wording for a menu can make a big difference. 2. A description of the task the user is to perform on the system. This should be a representative task that most users will want to do. 3. A complete, written list of the actions needed to complete the task with the proposed system. 4. An indication of who the users are and what kind of experience and knowledge the evaluators can assume about them.

Four things in cognitive walkthrough:

-Goals -Operators -Methods -Selection

GOMS meaning

True

If participants are not actual users, they should be chosen to be of a similar age and level of education as the intended user group. (T/F)

Evaluation through user participation

In this section we will look at a number of different approaches to ____________ through _______ _____________.

Heart activity

Indicated by blood pressure, volume and pulse. These may respond to stress or anger. (Physiological measurements)

Activity of the sweat glands

Indicated by skin resistance or galvanic skin response (GSR). These are thought to indicate levels of arousal and mental effort. (Physiological measurements)

Scan path

Indicating areas of interest, search strategy and cognitive load. Moving straight to a target with a short fixation at the target is the optimal scan path but plotting scan paths and fixations can indicate what people look at, how often and for how long. (Eye tracking for usability evaluation)

Participants

Is vital to the success of any experiment. (empirical method: experimental)

Hypotheses

It is framed in terms of the independent and dependent variables, stating that a variation in the independent variable will cause a difference in the dependent variable. (empirical method: experimental)

Fixation duration

Longer fixations may indicate difficulty with a display. (Eye tracking for usability evaluation)

Error prevention

Make it difficult to make errors. Even better than good error messages is a careful design that prevents a problem from occurring in the first place. (heuristics)

Recognition rather than recall

Make objects, actions and options visible. (heuristics)

Electrical activity in the brain

Measured by the electroencephalogram (EEG). These are associated with decision making, attention and motivation. (Physiological measurements)

Electrical activity in muscle

Measured by the electromyogram (EMG). These appear to reflect involvement in a task. (Physiological measurements)

10

Nielsen's have __ heuristics.

-Think aloud and cooperative evaluation -Protocol analysis -Automatic protocol analysis tools -Post-task walkthroughs

Observational techniques:

True

Participants should be chosen to match the expected user population as closely as possible. (T/F)

-Paper and pencil (primitive) -Audio recording (thinking aloud) -Video recording (camera) -Computer logging (system) -User notebooks (logs of activity)

Protocol analysis five methods.

-General (background, age sex, prev experience) -Open-ended (unprompted opinion) -Scalar (numeric scale) -Multi-choice (multiple choices) -Ranked (ordering)

Questionnaires There are a number of styles of question that can be included in the questionnaire. These include the following:

-Independent variables -Dependent variables

Two main types of variables:

True

User participation in evaluation tends to occur in the later stages of development when there is at least a working prototype of the system in place. (T/F)

User control and freedom

Users often choose system functions by mistake and need a clearly marked 'emergency exit' to leave the unwanted state without having to go through an extended dialog. (heuristics)

Consistency and standards

Users should not have to wonder whether words, situations or actions mean the same thing in different contexts. (heuristics)

-to assess the extent and accessibility of the system's functionality -to assess users' experience of the interaction -to identify any specific problems with the system.

What are the 3 main goals: ...

GOMS

low-level modelling techniques such as keystroke-level model, design methodologies, design rationale, dialog models

Field studies

takes the designer or evaluator out into the user's work environment in order to observe the system in action. Again this approach has its pros and cons. (style of evaluation)

Laboratory studies

users are taken out of their normal work environment to take part in controlled tests, often in a specialist usability laboratory (although the 'lab' may simply be a quiet room). This approach has a number of benefits and disadvantages. (style of evaluation)