Human Factors - Chapter 3

What are the limitations of subjective measures?

Subjective measures (people experiences) may not always best support system performance. Also not a great indicator in responses to surprising events or different conditions.

What are the benefits of subjective measures?

Subjective measures are peoples responses on what they experienced. They are often easier and less expensive to obtain with large sample sizes.

Describe four essential aspects of protecting participants in research.

-Protection of participants from mental or physical harm -Right of participants to privacy -Assurance that participation is completely voluntary -Right of participants to be informed beforehand about the experimental procedures so they can give informed consent

Describe the steps of heuristic evaluation

1. Select human factors principles that are applicable 2. Carefully inspect design and identify where it violates design principles 3. Each examiner should then communicate and aggregate findings 4. Convey results to design team

How many participants do you need for a usability study?

5 participants with a minimum of two, although ideally 5+ tests.

How many analysts should be used to assess a system with a heuristic evaluation?

At least three evaluators, preferably five

What is the relationship between a construct and a measure?

Constructs describe abstract entities that cannot be directly measured. Because there is not a direct bridge to measure constructs, you can take subjective and objective data that is able to be measured and apply that to the construct.

What is the main difference between cognitive walkthroughs and heuristic evaluations?

Cognitive Walkthroughs have a more structured approach and give a more formal and detailed assessment of the design. Cognitive walkthroughs consider each task associated with system interaction and poses a series of questions to highlight potential problems.

Before a large system is deployed that is being developed using a Vee development cycle, what evaluation technique would you be expected to use?

Cognitive walkthroughs since they are thorough and formal evaluations. It goes through each task to highlight potential problems. Comprehensive system evaluations can also provide an inclusive and summative assessment, if controlled experiments are applicable to the process.

In using Scrum in a high-risk domain, what evaluation technique might be difficult to complete even though it might be the right thing to do?

Comprehensive system evaluations are very time consuming, which would not fit entirely well with the Scrum cycle. They also require large amounts of control. However they provide lots of great insight.

Show why it is useful to consider confidence intervals and not rely on p-values.

Confidence intervals can show the difference where the mean values of two conditions were large, but so was variability or mean values were small but so was variability.

How do controlled experiments, quasi experiments, and descriptive studies differ?

Controlled experiments have careful manipulation of independent variables and all other variables are held constant. Quasi experiments manipulate the independent variable without random assignments or ordering conditions. Descriptive studies simply describe relationships that could not be manipulated or controlled.

Describe a descriptive statistic used to assess the distraction potential of a vehicle entertainment system.

Correlation Coefficient, r, gives a measure of strength of relationship.

What is the role of descriptive statistics and how does it differ from inferential statistics?

Descriptive statistics are a way to summarize the dependent variable for different treatment conditions while inferential statistics tell us the likelihood that any differences across groups are real or fluctuations due to chance.

How is evaluation related to understanding in the human factors design cycle?

Evaluation identifies opportunities to improve a design so that it serves the needs of people more effectively. Evaluation is both the final step in assessing a design and the first step of the next iteration of the design where it provides a deeper understanding of what people need and want.

Describe what is meant by experimental control and its role in designing an experiment, quasi-experiment, and a descriptive study.

Experimental control is when the independent variable is controlled and all other variables are held constant or controlled. Since this is not always possible, quasi-experiments allow for partial control. Descriptive studies have no experimental control and are simply observing and describing a relationship without any manipulation. Control can make some experiments hard to generalize.

What inferential statistical approach is most commonly used for multi-factor experiments?

F-test gives a p value that is a collective measure of comparison between all conditions in an experiment.. Each condition will have its own standard error and confidence interval.

Identify a method suited to formative evaluation and another more suited to summative evaluation.

Formative - Open ended survey, Heuristic evaluation Summative - Field test, Task analysis

Describe the role of formative and summative evaluations in design.

Formative evaluations help understand how people use a system and how the system can be improved Summative evaluations asses whether system performance meets design requirements and benchmarks

Describe an example of confounding in a field test of a vehicle entertainment system.

If all of the subjects were NASCAR drivers or all from one population. If all the subjects were interacting with a similar entertainment system before the study.

Identify the evaluation method best suited to designs that are in service.

In service evaluations are conducted once a design has been released. They use descriptive studies since control is impossible. A-B testing - compare two versions of a system

Describe two evaluation methods that do not require human collection

Literature review, heuristic evaluation, cognitive walkthrough

Identify the evaluation method best suited to early design concepts.

Literature reviews, heuristic evaluations, and cognitive walkthroughs because they are fast to apply (do not involve collecting data from people using the system). Analyst may suffer from learned intuition or more knowledge of system than user.

Give examples of quantitative data in evaluating a vehicle entertainment system.

Measure response time, use frequency, subjective workload assessments

What is the difference between practical and statistical significance?

Statistical significance - proves that there is an actual difference between groups Practical significance - take on system perspective to estimate the overall safety or economic consequences of the difference observed

Give examples of qualitative data in evaluating a vehicle entertainment system.

Observations, Interviews, Answers to general questions on what features like or main problems

How many usability tests would you recommend as part of an iterative design process

Probably at least 10? So that you can come up with new ideas as well as mix ideas from different variations?

What is an important limit of both cognitive walkthroughs and heuristic evaluation?

Problems that frustrate a person less familiar with the system can be overlooked. Evaluators can have learned intuition and a more extensive knowledge on how the system works.

Would qualitative or quantitative data be more useful in diagnosing why a design is not performing as expected?

Qualitative

What is the role of quantitative and qualitative data in system design?

Qualitative data is important for evaluation what needs improvement in formative evaluations Quantitative data gives numeric indicators of performance and is used in summative evaluations

Would qualitative or quantitative data be more useful in assessing whether a design meets safety and performance requirements?

Quantitative

Describe how the driving performance data in Figure 3.9 represents a two way interaction and what the graph would look like without the interaction.

The interaction shows varying conditions in both the x and y axis (varying lane deviation and varying traffic levels) across two different groups (with phone and without phone). Because the line between phone and nonphone users is nonparallel, you know that the two independent variables interact. If they did not interact, there would be parallel lines.

Why are evaluation methods that do not require human subjects data collection useful in design?

They can be done quickly.

For an evaluation of a vehicle entertainment system, identify possible dependent variables.

Things that can be measured: - time it takes to adjust radio - average volume things kept at - awareness of road while using features (# swerves) - safety measures (# almost accidents) -user rating of system

What is the purpose of representative sampling in selecting people, tasks, and situations in designing a study?

To eliminate confounding variables and to get an accurate, generalizable result.

Describe Type I and Type II errors and the implications for evaluation.

Type I error (alpha) - concluding independent variable had an effect when it was really just chance Type I errors can lead to false theories and misplaced expectations of benefits of design change. Type II error (beta) - concluding there was no effect when there was actually an effect Type II errors can lead to improvements being rejected

What are the three general purposes of evaluation?

Understand how to improve (formative evaluation) Diagnose problems with prototypes (formative evaluation) Verify (summative evaluation)

Identify the evaluation method best suited to pre production designs.

Usability testing is iterative and can adjust designs, and comprehensive system evaluations which include controlled experiments

What evaluation techniques would be most useful in a Scrum process?

Usability testing would be useful in the Scrum process because it is also iterative. It can also be done fairly quickly. It follows the five variables: learnability, efficiency, memorability, errors, and satisfaction

Identify the evaluation method best suited for prototypes.

Usability testing, controlled experiments

Why is qualitative data an important part of usability testing?

Useful for diagnosing problems and identifying opportunities for improvement, especially in iterative design processes

When would you use a between subjects design and when would you use a within subjects design?

Within subjects design has the same person doing multiple tasks. This can be used to avoid confounding differences between groups, but order of variables must be controlled (can get fatigued or more experience with systems). Between subjects design has different groups of people testing different conditions. This is useful if there is a large enough sample and random assignment can be used to distribute differences across groups.

Describe an inferential statistic in assessing the distraction potential of a vehicle entertainment system.

t-test gives p value indicates if result is due to chance. confidence interval that gives range of mean values to be expected in repeated experiment. (usually at 95%). effect size is how much the independent variable had in effect compared to variability of sample.