UX Evaluation: User Testing
If there are 2 user profiles how many participants are needed
- 12 participants - needs 5 participants to cover the hypothesis and 2 extra to account for the no shows
If there is only one user profile how many participants are needed?
- 7 participants as there needs to be at least 6 to test the hypothesis and another one is to account for no shows
How many participants are needed for correlational studies and how many usability issues
- 8 to 10 participants identify about 80 percent of usability issues
How to calculate the NPS
- Calculate the difference between the percentage of promoters and detractors - the promoters are consumers who give a 9 or 10 - the detractors are those who gave score of 6 to 0 - the undecided gave the company a score of 7 to 8 - the NPS is presented in absolute number between -100 to 100
Correlational studies
- Carries out a homogenous sample of participants who must perform a certain number of task with the evaluation object - can't claim causation
Performance
- NASA TLX - good to poor - how successful were you in performing the task - how satisfied were you with the performance
effort
- NASA TLX - low to high - how hard did you have to work (mentally or physically) to accomplish your level of performance
frustration
- NASA TLX - low to high - how irritated, stress of annoyed were you with the content
Mental demand
- NASA TLX - low to high - how much mental and perceptual activity was required - was the task easy or demanding, simple or complex
Physical demand
- NASA TLX - low to high - how much physical activity was required - was the task easy or demanding, slack or strenuous
Temporal demand
- NASA TLX - low to high - how much pressure did you feel due to the pace at which the task occurred - was the pace slow or rapid
Moderator
- Neutral person who conducts the user test sessions. They lead the sessions and give instruction to the participant to complete the user test tasks - observes the reactions and actions of the participant - this allow to gather contextual information - at the end they conduct an interview with the participant in which they are asked about their overall impressions and their experiences. A question guide is used to make sure each is consistent
Is there one magic structure that you should use when writing the mandate?
- No, it's to you and your experiencd to structure tne sections of the report and if necesssry add aditonal sections
context of use
- Refers to the users, tasks, equipment, physical, and social environment in which a product is used. - the very different content of who is and where they are additional usability constraints - the usage contexts will have to be anticipated to ensure that the interface is adequate effectiveness according to the anticipated by users
What are the advantages of the SAM scale?
- SAM scald is highly correlated with implicit measures abs captures part of the user automatic and unconscious reaction - it useful to quickly measure the perception of emotions
What measure are used to measure someone cognitive when using a interface
- The NASA TLX - The customer effort score (CES)
NASA TLX
- The scale was developed in 1988 by Hart and Staveland for NASA to measure the mental load perceived by users after interaction with a technology. - it was used at end of each task or at the end of the test - there are six dimension which are mental demand, physical demand, temporal demand, performance, effort, and frustration level
Are there any rules governing for how many words should be used in a stand alone report
- There isn't and the the focus should be more on making sure that the report makes sense on if own without seeing the presentation. - this includes taking advantage of boxes and footnotes to add details essential to understanding your analysis
Guerrilla testing
- To conduct fast and inexpensive user test - use it in the formative mode to get low cost feedback as often as possible - could lead to wrong conclusions and be more costly in the long run
What are the different parts of the mandate
- a description of the object of the evaluation - the goals of the evaluation - the research questions - the user groups targeted by the evaluation - the experience boundaries - the format of the deliverables
inductive reasoning for classifying info
- appropriate where prior knowledge concerning the topic is limited or when the technological field is not mature enough - categories or themes emerge during the analysis of the data by the UX researcher - this is the approach that is most often used
inductive reasoning coding
- begins without a predetermined category - used for exploratory research - read all the observation collected through once fully. To know the vocabulary the person uses - the second time the researcher will gradually will see different themes emerge - the researcher will create as many themes as they think are necessary - review the themes to review the different themes and merge any if necessary
What are the four main categories of measure
- behavioral - cognitive and emotional state measures - perceptual and attitudinal measures - measure of intention
What are the three main types of experimental design
- between subject design - within-subjects design - correlational design
What is measure of success used to analysis
- binary values with only two modalities (0 = fail or 1= success) - sometimes it could be useful to compare partial success thus you can code for 0 = failure, 1= partial success 2= success - Preference: If you want to compare interface A vs B there has to be binary variable. If the participation preferences A put one and 0 for B
How do you report the findings of the tests?
- by the severity of the need or problem of what was found - there is no object test to establish what is the most critical or severe. You have to use your opinion to determine this. It's important to use example from the data to show the severity
What consequences does not testing a product have?
- can have serious economic consequences for an organization - research has shown that discovering and correcting problems very early on in the product development costs 100 times less than fixing the problem while the product/service is already in production
What can happen if the evaluation mandate doesn't clearly define target groups
- can lead to very heterogeneous results - will be hard to make concrete recommendations to the developers - lead to wrong conclusions and lead the team in the wrong directions
What should the participant be informed with before they start?
- clear instruction of the task at hand - If the task has a time limit inform the participant - inform situations that may arise and give moderators instruction in the protocol
How to clearly establish the object of the evaluation
- clearly identify the artifact to be evaluated by saying the formal name and version - specify on which software platforms the evaluation will be carried out - specify the functions or function enabled by the artifact
What are the steps of the scientific method
- coming up with research questions - lit review -formulate hypothesis - test hypothesis through experiment - data is analyzed to see if it will confirm hypothesis - leading to answers to research questions
How should headlines be written
- could be used for a title newspaper with this they are descriptive in which they give you an idea what the content is going to be. They also should be interesting. - it's not interesting to just write results as that is boring and doesn't want to make people read forward
What type of variable is measurement of scales and where are they found
- discrete and ordinal between two limits
What is important to mention during the intro of user testing?
- duration of the study - how much they will be paid - that they are allowed to ask questions
within-subjects design
- each subject is consecutively and randomly exposed to different experimental manipulations.
What are the 12 dimension of WebQual
- ease of understanding - intuitive operations - response time - trust in website - tailored communications - informational fit to track - emotional appeal - visual appeal - innovativeness - consistent image - online completeness - relative advantage
The participant selection should be in line with the what?
- evaluation objective - the objective is not always to test all user groups as it is important to choose only the user group that will provide answers to the research questions
What should the test plan include?
- experimental design - profiles of participants and recruitment strategy - the interfaces being evaluated - scenarios and tasks - the test environment - tools and measures - the detailed protocol
What are the two main ways to conduct evaluation
- expert evaluation which is based on the judgement of one or more of the evaluators who asses a system to identify potential usability issues a system to identify potential usability issues including deviations from established performance criteria. - user based: participation of representatives users performing tasks to identify usability issues or address other dimensions - conformity assessment: used to demonstrate that specified requirements for a process or system are met
What are the three types of measurements
- explicit - implicit - think aloud
When and how should content transcription
- ideally it should happen in real time that two people take notes and compare them afterwards - it is important to do this as close to actual session as possible to not forget context
When to do a two-tailed test
- if we hypothesize that there is a difference between two phenomena without assuming the direction of the difference
When is one tailed test needed
- if we hypothesized there is not only an difference but the difference will be smaller or larger
Eye tracking
- implicit - behavioral - measures the user attention during the task
The feature usage rate
- in some cases it may be useful to measure participants actions - only rely on video recording of the sessions and encode the results afterwards
The preference test
- in the context of an evaluation choices are often about alternative design - different version are presented and at the end they have to tell which one they prefer - this often linked to the success rate, completion time -represented in a ratio - behavioral measure of preference
Self-Assessment Manikin
- is a discrete non-verbal photographic scale developed by Bradley and Lang in the 1980s - it measures three construct which are valance, intensity of the emotion, and level of dominance - valance is defined as directionality of the emotion ( from unhappy to happy) - arousal as the intensity of the emotion (non-excited to excited) - the participant's level of dominance during the interaction (from controlled to in control)
Customer satisfaction score
- is a tool to measure satisfaction through a single question - clear and quick to answer and is rarely skipped by participants - rarely skipped by participants - are satisfied with your product/ the answers provided by our employees and service rendered - discrete 5 point scale that goes from very satisfied to not satisfied at all
Net Promoter Score
- is one of the common measure of customer experience used by businesses around the world. - it used to measure user loyalty - used before or after the task or test
What is the role of interview guide
- is to collect additional info during the experiment
Does there has to be compensation?
- it depends on what is being required of the participants - if the participants is short it's possible to get people to participant - but for an one hour it is best to offer an compensation it doesn't have to be monetary
When a formal report and oral presentation always be required
- it will typically depend what is required by the client and organization. Also, how the results will be used. - when you're doing an evaluation part of a developmental project a report in form of presentation will be requested - the report can be informal as well
Why choose a correlational design
- it's a matter of time and resources - the test budget might be enough to recruit different groups of participants - can be used as an alternative to experiments designs especially for formative tests when the project is still in development
How many grouping should a affinity diagram lead to?
- less than 10
How to enter time measurements
- limited to upper bound by maximum time allowed for a task - converted to seconds
Research questions
- linked to the tasks that the user performs on the interface and should be drafted with an action verb - clearly operationalized
Recruiting very specific individuals for an evaluations can be what ( 3 words)
- logistically complex - time consuming - costly
What does the correlational design make it possible to do?
- make some correlations within the sample to outline a avenues for further reflection
What is a requirement for a experiment
- manipulation therefore an evaluation where the characteristics of the participants and the interface are not manipulated it's not a experiment - in short without manipulation is not possible to have an formal hypothesis or to measure rigorously the effect on a dependent variable
The customer effort score
- measures users effort to perform an interaction with the object being evaluated - how much effort did you have to put into using our digital service/application - discrete five anchor scale very weak, weak, neural, strong, and very strong - divide the sum of the CES of all of the participants by the total number of participants
Where should user testing be conducted
- must take place in a controlled space so the external stimuli will not disturb the user - this won't replicate the exact experience of being in the environment but it's possible to conclude that the issues would be worse in real context
What measures is the CES usually used with?
- net promoter score - customer satisfaction score
What are the two score that measure measure of intention
- net promoter score or NPS - customer satisfaction score CSAT
Is it necessary to write the significance, the name of the test, and other statistical details
- no at least not in the main body of the report instead asterisk can be used so in the footnote people can see the level of significance - it's also common to not include the test name or other statistical details as it can clutter the report
Is 7 to 10 enough to test a hypothesis
- no there isn't there needs to be at least 12
Should you always use all 12 dimension of the WebQual
- no, as it depends on the context of the study
Is it common to make a complete textual analysis
- no, it is not instead it's important to pick the important insights from the interview and capture every world of the user
In a within-subject design are all the participants exposed to all of the conditions
- no, there is going to be cases where there are too many conditions that each participants would have to go to - this could impossible just in general but especially since usability test should never go on for over an hour to avoid fatigue
Need
- perquisite requires as necessary for a user or a set of user to achieve a desired result implicit or declares in a specific context l
What are ways to reduce systematic bias?
- pre-registration process via an online form that you can advertise on different platforms - put a short list of questions to qualify participants and thus generate diversity among your groups of participants
How should you end the report
- provide the client food for thought on future research questions that would deepen or develop understanding of the user experiment
What type of data is quantitative analysis used for
- psychometric-type data such as standardized measures scales and behavioral measures like time and success
What reminders containing instruments should be included
- related to the measurement tools - if you are using a camera it is important to indicate when the recording will start in the detailed log - if you are using oculometric of neuropsychological measurements you will provide instruction for mounting and calibration of these instruments
Why is good to not have too many targeted groups
- requires more participants which may require more financial and other resources
The user testing process
- research aim and associated questions - through stakeholder interviews you will understand what they want to test - forms a hypothesis - tests the hypothesis using usability study - analysis results and sees if data supports hypothesis - the observation will be converted into recommendations listed according to severity
What biases is your perception subjected to
- retrospective bias: it can be difficult for the user to accurately recall the interaction
What is a contextual interview
- seek ti understand the why behind specific actions or reactions from the participants
What should interviews be
- short - focused
deductive reasoning categorization
- start with known theories and preconceived codes/ categories - the goal is to build upon a known typology
What might a participant do before they begin the task?
- take a pre-test survey - link to the survey should be included in the protocol - may be useful to have a PowerPoint slide with the instruction and link - if there are multiple surveys there should be identifiers to connect the surveys with the person
How to calculate the WebQual results
- take the average of the three questions corresponding to each of the dimension
What could happen if the target group sample size is small
- the company has to be able to put the necessary resources into recruiting and know that this can be the most costly part
What happens when the experience boundaries are hard to define
- the experience may not start or end with the participants using the product/service as they may have to research, open the box, and put it together for example - there could be various different end points - with both of these it should be established with the stakeholders what they want the start and end point to be
What does the experimental design identifies
- the manipulated factors or independent variable ( for example, version A and B of an interface) - as well as the dependent variable on which the independent is likely to have an effect ( for example the success or failures of a task by the user)
Where should the important findings and recommission be put in the report
- the most important findings should be presented early on in the executive summary or highlight sections
Why deductive approach easy for coding
- the researchers uses an already existing coding scheme with different categories - a revision will take place to adapt to fit the objectives - will measure have times a theme appears and will compare them - typically to verify existing hypothesis
What are measure are used to measure someone emotions when using a interface
- the self-assessment manikin (SAM) Scale - The Affective Slider
What do participant tell in the interview
- their thoughts - their opinions - what they believe - their value - what motivates them - why they did what they did
If there are three user profiles how many participants are there?
- there needs to be 4 participants of each user profiles - since there is overlap there doesn't have to be as many participants - there has to be 2 extra participants - 14 in total
How to identify tasks that the users will perform during the evaluation
- they should be directly related to objective and the research questions - keep in mind the expected duration of the evaluation and this tell you many task you can do
What are the inclusion and exclusion criteria
- to identify what are the characteristics that will determine whether or not user is eligible for the usability tests
What are the steps to making a affinity diagram
- transcribe unitary unitary ideas that is each of the category of ideas on posit notes. They come ideas that were recorded verbatim. Note how many times that each occurred to keep in mind how importance is each - group these based on higher level concepts to give a name to larger categories
What is the challenge carry out user tests? What is the solution
- typically used testing had a very small simple size but in stats you typically need to large sample size to make assumptions about it. This is why a parametric statistics is used because it doesn't make assumptions
What does testing allow us to do
- uncover usability issues - provide ideas on how to improve the interface and functionalities of the product/service - will ensure that the users will want to use it again
Is usability a binary measure and explain?
- usability is not a binary measure and it's a gradient
What can result from systematic bias?
- usability issues can missed because the test wasn't done with the right users
How to reflect the voice of the user in the report?
- use excerpts from your participants' verbatim to support the results or give an example of their reaction - use the exact words of the participants even if the vocabulary or syntax aren't great - put what number of participant or their first name to identify nsnd
What are the six dimensions of usability
- utility - effectiveness - efficiency - satisfaction - learnability - accessibility
Affective slider
- was developed in 2016 by Betella and Vershue with the objective of replacing the SAM scale in online surveys - two slider controls are used to allow the users to indicate their level of pleasure and arousal by dragging the cursor on the response bar
What are the three core elements?
- what are the typical tasks engaged in by your target users - what is the typical duration or desired duration to complete those tasks - what are the success criteria associated with each of those task
Sources of info for interview guide
- what does the mandate want the test to clarify - improve the questions with info from the unofficial heuristic review that was conducted for the study
When can systemic bias result from?
- when you cut corners and choose convenience it's possible that the users can be too similar that could skew the results - the participants may not be representative of the population
It is useful to have an interface image? If so, how should it be used?
- yes, the report should arrows to point to different elements of interest for which you have recommendations
System Usability Scale (SUS)
-developed by Brooke - used to evaluate the perception of an application after it use - it has 10 items with 5 anchor points - normative nature
What are the two different ways that content analysis can be used? Depending on what?
-inductive and deductive - context and the maturity of what is being studied
How many more participants should recruit to account for the no shows?
10-20 percent or 14 participant
What is the general rule for one hour long test
3-5 tasks
How long should interviews be
5 to 15 minutes long
_______ to _______ will discover what percent of usability issues
7 to 10 80 percent
User Interface element
A basic component of a user interface that is presented to the user by the interactive system
How many questions does the interview have
A few questions
Usability Defect
Product attributes that leads to a mismatch between user intention and/or user actions and the system attributes and behavior
content analysis
Qualitative research method that allows for subjective interpretation of written or oral documents through a systematic classifucation process The goal is to group them into different themes typically have a lot of detail under each theme but it depends on the objectives of the study
User
Refers to the person who interacts with a system, product, or service.
Perception
Refers to the user perceived experience it is conscious experience that the user process cognitively and is able to verbalize and explain
quantitative analysis
Represents the second type of analysis necessary to carry out a rigorous evaluation
Modal window
Requires users to interact with it in someway before they can return to the system
What comes after the objective of evaluation
Research questions
A UX evaluation is first the study of an interface using _________________
Scientific approach
Measure of intention
Seek to understand the future actions in relation to the experience and what is being evaluated Measure before and after a task
Message box
Small window that provides information to users and requires them to take action before going forward
Usability is measured for a population of user with
Specific characteristics
What is the terms of reference
Specifies how you and your term members will work together to accomplish common goals. To create shares expectations but also accountabilities among the team members
How to calculate and analyze: the SUS
Subtract 1 from odd items Subtract 5 from even items Add the recalculated items Multiply the sum by 2.5
What is important to avoid no matter what the recruitment method
Systematic bias
What is the next step after the evaluation purpose
Term of reference
satisfaction
The absence of discomfort and a positive attitude towards the use of the product/service
Usability
The degree to which a digital product or service can be used by users in order to achieve specific goals with effectiveness, efficiency, and satisfaction for a product/service in a specific context
Effectiveness
The precision and level of completion in which a users can achieve a specific goal
Objective of evaluation
The purpose is not to explain the specific problem while using the object. But a specific objective which will make it possible to identify important issues. To justify the investment that needs to be made. It should include one or several of the usability attributes
Efficiency
The resources devoted to the accuracy and level of completion with which users reach the objectives
After the introduced what should as soon as possible
The test environment in order to foster as much trust as possible. Depending on the environment and the tools used it could intimidate the participants. If there are cameras be sure to point all the cameras out. If there is a one-way mirror be sure to explain who is behind the mirror.
Note taker
This is also recommended to have and they note observation during the test
Counterbalancing
This is where all participants are all exposed to all the conditions. In different orders with equal number of participants being exposed to the different conditions.
Why is it important to have a standard speech?
To ensure consistency in the information communicated to the participants before
What is the aim of the evaluation
To highlight usability issues abs ways to improve the user experience to ensure uninterrupted use of the interface - the usability of the system may lead to usability errors which will be judged on their severity
What is purpose of the interview
To know why the participant did what they did
Why do participant complete a post task survey
To measure perception, attitude, and state of the individual in the relation to the task
What can content analysis be used to analyze
Virtually any type of qualitative UX data methods
SUS Score less than 25
Worst imaginable
Are quantitative and qualitative data necessary to have complete perspective
Yes, an assessments that does not use these two types of data would be incomplete and might not be able to properly meet the needs of the Mac date
Should you put limits and if so. where?
Yes, as all studies have limits the clients should what your were. In the conclusion there should be a reminder of the methodological choices and the limits
Behavioral measures
aim to capture the user actions and choices during a task
SUS score 40 to 52
Acceptable
Evaluation mandate
Aims to define the objective and the scope of the technological interface to be evaluated. This is crucial step in the preparation of an evaluation project. A poorly written mandate can have serious consequences on the process
User Interface
All components of an interactive system that provides information and controls for the user to accomplish specific tasks with the interactive system
Checkboxes
Allow the user to select one or more options from a set.
What does choosing compare different experiences for the same participant
Allows to answer questions relating to intra-related designs. This type of design leads to data organized in pairs ie each participant is associated with at least two measures. For example, compare the completion time of two different interfaces with the same group of participants
Text fields
Allows user to enter text. It can allow enter a single line or multiple lines of text
Dropdown list
Allows user to select one item at a time but are more compact allowing you to save more space.
Breadcrumbs
Allows users to identify their current location within the system by providing a clickable trail of proceeding pages to navigate by
Tooltip
Allows users to see hints when they hover over an item indicating the name or purpose of the item
Date picker
Allows users to select a date and/or time. By using a picker the information is consistently formatted and input into the system
List boxes
Allows users to select multiple items at a time but are more compact and can support a longer list of options if needed
What does choosing compare the user experience between two groups of participants
Allows you to answer for questions related to a inter-participant designs. This type of design leads to obtaining independent measures (each participant is associated with only one measure). For example, comparing the completion time of the same interface using two groups of participant
What does the ux calculator allow you to do?
Allows you to run statistical tests tailored to the main analyses that a UX professional will have to perform as part of assessment
Radio buttons
Are used to allow users to select individual item at a time.
What does the international organization for standardization defines user experience as
As a person perception and responses resulting from the use or intended use of a product
Perceptual and attitudinal measures
Asses the participant subjective appreciation of the experience and objective being evaluated This can be assess before and after a task
Between subject design
Assigns participants to a single experimental conditions and compare the results between groups. In other words, participants only see one experimental condition. The average of the results obtained by all participants in one condition is compared with the average of the results obtained in other conditions
SUS score 26 to 39
Bad
What type of measure is success rate of task
Behavioral measure
SUS score 86 to 100
Best imaginable
Explicit measure
Best known and most frequently used aims to measure the phenomena while the users are consciously aware of it Before the test, after the task, and after the test
Toggle
Button allows the user to change a setting between two states. They are most effective when the on/off states are visually distinct
Which of these evaluation objectives is not well defined and why? A. User satisfaction after a first experience B. The ability of users to quickly learn features C. The experience of a new application D. The accessibility of a website to a website with visual impairment
C. It's too vague what do they want to know about the experience? What do we want to measure this doesn't tell us anything. There is not one of the 6 dimension of usability mention here.
implicit measures
Captured automatically and often unconsciously during these tests
Psychophysiological measures
Captures the participant cognitive and emotional reactions using sensors
Type one error How can this happen
Concluded that there is no issues when there is Selection of users didn't really match the population and expertise of the users were too high relative to targeted users
Type 2 error Examples
Concludes that there are issues when there aren't Users selected are too novice and don't understand how to use the interface
Requirements
Conditions must be met by a system, system component, product or service to satisfy a contract, standard, or specification
The dropdown boxes
Consists of a button that when clicked displays a list of mutually exclusive items
What type of interview is used
Contextual interview
Time measurements what kind of variable
Continuous
What are majority of evaluation in UX research
Correlational studies
The cognitive and emotional state measure
Deal with the experience lived and perceived by the participants Assessed during and after the test
iterative approach
Development teams interacts with the actual user of the digital produt or service during the developmental phase to get their feedback and adapt the design accordingly.
What type of variable is measure of success
Discrete
pagination
Divides content up between pages and allows users to skip between pages or go in order through the content
The ISO CIF ISO/ IEC 25066: 2016 format is often required when
During the conformity assessments or by institutional clients
The scientific method is an___________ approach to the developmental of knowledge based on _______________ and observation
Empirical Experimentation and observation
SUS score 74 to 85
Excellent
What is the first step of the test plan
Experimental design
Accessibility
Extent to which product, service, systems, services, and environment can be used by people in a population with the broadest range of characteristics and abilities to achieve a goal in a specific context of use
Learnability
Extent to which products/service allow users to quickly become familiar with them and make good use out of all of their feature and capabilities
Testing the experience of new digital product and services before making them available to users allows you to discover and correct a problem early on in their development. This costs 10 times less than correcting the same problem when the product or service is already available. True or false and why?
False as it is 100 times and not 10 times
A evaluation can either be _____________ or ______________
Formative or summative
SUS Score 53 to 73
Good
In what format should the number of participant who liked a UI or succeeded at the task and why?
In a ratio as the number of participants/ total number of users Percentage should be avoided as you don't want to make sure reference about the total population as it's often not random or big enough to do so
Button
Indicates an action upon touch and is usually labeled using text, icon, or both
Dialogue
Interaction between the user and an interactive system as a sequence of user actions and system responses in order to achieve a goal
What should the evaluation typically end with?
Interview to collect impressions and perceptions
Prototype
Is a representation of all or part of the interactive which is limited in some way but can be used for analysis, design, and evaluation
test plan
Is a tool that the UX researcher uses to organize and structure the evaluation. It purpose is to document all activities needed for the test preparation, execution, and analysis. It must include all the asserts and instruments needed to make the evaluation possible. The test plan should also be defined and environment in which the test will be run. It is also a mean of communicating the test process to internal and external stakeholders. Defining the evaluation is also a way of getting feedback to improve the overall process
Think Aloud method
Is an alternative to the implicit measure it involves asking users to describe our loud their emotional and cognitive actions while performing a task This method helps to understand the reasoning and motivation behind each user action Used exploratory studies when seeking to understand the user mental models It tends to slow down the participants as a result task completion times are not accurate
System
Is defined as a combination of interacting elements organized to achieve one or more states purposes. It's very large and integrates all the underlying software and applications.
summative evaluation
Is done when the design is near completion or already complete. It aims is to evaluate the quality of a developed system and check all the requirements are key
Utility
Is the extent to which a person believes that the use of technology improves their ability to perform a tasks
What is the goal of the scientific approach
Is to provide rigorous data to make design decisions.
What is the objective of the experimental design is to what?
Is to structure the evaluation in such a way to answer the research questions.
Affinity Diagram
Is useful tool to help classify data of a qualitative nature. In a content analysis this can help gather common ideas to help reduce number of codes
Why is counterbalancing used?
It can help with learning effect, which if you have the same amount of participants starting at different conditions and going to next one in different orders. It allows for researchers to eliminate the chance of people like version B because they got familiar with it when they used version A
Success rate
It is a binary variable that represents the completion or not of a task by specific success criteria However, the users may want to look for partial successes in this cases it would not be a binary variable It's generally reported in a ratio as it doesn't make generalization about the population
What happens if the mandate is too narrow
It might miss the real issues altogether of an interface
What happens if the evaluation mandate is too vague or broad
It will lead to deviation from the recipient expectations and potentially disappoint them
Objects of the task
Key units of information or data with which users interact to perform the task
Formative evaluation
Made during the development of a system and aims to provide recommendations for the systems improvements
Observers
May be present during the test sessions. Observes the participants who are talking or doing related to the system
WebQual
Measure the quality of an eCommerce website Developed by Loiacano, Watson, and Goodhue To what extent do you agree with the following statements anchor points are generally used 1 = strongly disagree to 7 = strongly agree
UX Evaluation is based on a ________ approach
Multi-method approach
It is important to develop a ________ framework in the report
Narrative
The design of a new system and its interface requires a good understanding of user __________ in order to establish the __________
Needs Requirements
Is it necessary to include all the data you have found in the report?
No, as it's often too much and the goal is to answer the research questions and only using the relevant data to do so. If necessary the additional information can be put in the appendix
Should the instructions always be detailed and directive
No, as you occasionally you want to understand how the user will navigate by themselves on the interface. But they should still be detailed and give them a scenario
The ISO CIF ISO/ IEC 25066: 2016 format is often useful to
Often useful to meet the requirements of a tender and to assess compliance.
What is the most difficult tasks in UX evaluation is
Participant recruitment strategy
Latin square design
Participants are randomly assigned to some of the experimental conditions. All conditions are going to be seen by equal amount of participants
The purpose of the report is not only to ______________ but to _____________
Present the results of the test To answer the research
