Psyc 350 Final
a pilot test is
A scientific evaluation of the test's performance, followed by revisions to determine the final form that the test will take
Norms ->_______-> _______ ____________
norms -> 60 -> normal and cut scores
Test fairness is
not statistical-it's SOCIAL not statistical
Types of test formats
objective and subjective
survey questions can be:
open ended or closed ended, fill in the blank, column format, row format, implied no choice, single-item choice, multiple choice, ranking, rating, likert-type format, semantic differential.
After writing the initial test questions, the test developer conducts a
pilot test
Whare the the 2 types of sampling
probability and nonprobability sampling
cumulative model is
produces interval data
categorical model is
produces nominal data
Examples of subjective formats
projective tests, essay (open-ended), interview
Survey questions are
purposeful and straightforward, unambiguous, correct syntax (complete sentences with an orderly arrangement of words), Appropriate rating scale, appropriate categorical alternatives, no double barreled questions (a question that is actually asking two or more questions in one), and it has to have a comfortable reading level (prefer low readability level).
Piloting the test provides an opportunity to gather
quantitative and qualitative data about the test.
What things do you consider for a target audience?
reading level, disabilities, cognitive motor or sensory impairments, are they motivated to fake their answer?
complex test formats-performance assessments
require test takers to directly demonstrate their skills and abilities to perform a group of complex behaviors and tasks.
Random response
responding to items in a random fashion by marking answers without reading or considering them.
What is a pilot test?
scientific investigation of evidence for reliability and validity.
For essay and interview questions you
score them anonymously and should have multiple scorers.
criterion approach
score to qualify as passing or failing
empirically based tests
sort individuals into 2 or more categories based on their scores on the criterion measure.
Composing the test items. Test items can be
statements, pictures, incomplete sentences.
Way to gather evidence of validity of a survey:
survey assesses the concept that a researcher attempts to measure.
Example of a survey research firm
survey monkey and qualtrics
stimulus is a
test question
Ways to gather evidence of reliability/precision of a survey:
test-retest, alternate forms, split-half reliability. split-half is more practical and most common.
Test plan for educational settings
the curricula (assigned readings, handouts, lectures) provide the basis for developing a test plan.
Why do we use internal consistency for pilot testing?
the developer consults the interitem correlation matrix to see which items should be dropped or revised to increase the test's overall internal consistency. Reliability/precision and Gonbach Alpha
Defining the target audience
the developer makes a list of the characteristics of the persons who will take the test.
homogeneity of the population
the more similar the members of the population, the smaller the sample that is necessary. The more dissimilar the members of the population, the larger the sample that is necessary to have this variation represented in the sample.
The front page of a survey must include
title, seal, logo, appeal, and instructions
What is pilot testing used for
to determine how a new test performs.
What are you looking for in a pilot test?
to see if test scorers are accurate/meaningful, if the test questions are easy or difficult, if high scores reflect a high level of having that quality, if it's biased, it can be used for any kind of group, and if instructions are clear/easy to follow and the length of time.
On a survey there has to be
transitions, bolded, white space, print->3 pages or double sided or spiral and booklet(for long surveys).
For MC and T/F items, what should you do and shouldn't do?
try not to use the word "NOT". Male sure all responses are similar in detail and length. Have 1 correct answer and try using "sometimes" or "often". Try not to use inclusive distractors like ALL OF THE ABOVE or NONE OF THE ABOVE.
What is nonprobability sampling?
type of sampling in which not everyone has an equal change of being selected from the population. Often used because they are convenient and less expensive than probability sampling.
What is probability sampling?
type of sampling that uses statistics to ensure that a sample is representative of a population. Simple random sampling, stratified random sampling, and cluster sampling are examples of probability sampling methods.
D=+?
upper group answered correctly
What happens after piloting the test?
validating the test. Now you collect evidence for validity. Test + another test measuring same construct + criteria
After calculating D value, the test developers look for items that have high positive numbers. Negative numbers indicate? D=-?
who scored low on the test responded to this item correctly
Test plan for organizational tests
will be based on job analysis that defines the knowledge, skills, abilities, and other characteristics required to perform a job successfully.
Is the scientific method that's used for survey designs the same for psychological tests?
yes
Example of Free Choice:
if the psychology department started a psychology club, would you attend the meetings? ____ Yes ___Probably ____ Uncertain
Surveys
instruments used to collect important information from individuals.
What is particularly important for subjective tests?
interrater reliability (two raters)
What is the main purpose for pretesting surveys
looking for nonsampling (associated with the design and administration) measurement errors
Self Administered surveys include
mail and individually administered
How many research firms were there in 2015?
more than 2000
normative approach
higher score will receive the job
You want interitem correlations to be
highly positively correlated
Objective items (stimulus) True/False
1 out of 2 (50%)
Presenting the findings include:
1. outline a report 2. order content 3. using slides and handouts.
response bias-acquiescence
50% negative 50% positive. for instance, someone who labels each statement on a true/false test as true would be demonstrating a response set of acquiescence.
interitem correlation matrix
A matrix that displays the correlation of each item with every other item
When would forced item be used?
Appropriate for when the potential test takers are likely to answer dishonestly. It has low face validity.
example of Likert-type format:
Being able to approach your professors is a major advantage of attending a small college. Please indicate the extend to which you agree or disagree with the following statements about your professors.
Survey Research Method
Both a science and an art. It's science because it's a process you have to follow.
Objective items (stimulus) Forced Item-
Do you go more by facts or principal? Do you prefer the planned events or unplanned events?
Purpose of the test
Does it have a normative approach or a criterion approach
example of column format:
For each of the courses listed below, identify how many hours you study per week and how many pages you read for the course per week. Course Hours per week you study Read Psychology _________ ______
How do you set up the pilot test?
For example: to diagnose emotional disabilities in adolescent's it has to be in school setting. Part should be adolescents who have been determined to have emotional disabilities and the others should be adolescents who have been determined not to have emotional disabilities. Also has to be male AND female from various economic and ethnic backgrounds.
Quantitative data of a pilot test
How you decide if item is difficult or not. You divide # of persons who answered correctly by # of person who responded to the question (P). .4 to .6 is range 0-.2 is too difficult .9-1 is too easy
Subjective items Sentence completion-
I feel happiest when I am ______________.
Preparing for survey:
Identify the objectives (lit review will help you or contact experts), define your objectives operationally (operational definitions), construct a plan (estimate the cost, administer, timeline for completing each phase of the survey).
example of an open ended question:
Last year you were a member of student government. please comment on your experience.
Examples of objective formats
Multiple choice, true/false, fill in the blank, matching
New theory of why we develop a new test.
New theory is fresh definition of constructs.
example of semantic differential:
Please circle the number of representing the demeanor of your professor Happy 1234567Grumpy
example of row format:
Please indicate how many hours per week you study for each of the following courses. Course Hour Pychology _______ Biology _______
What are the 5 steps for survey research?
Pre, construct, administer, analyze, communicate findings.
Final Phase of survey development:
Presenting the findings
Which is the most popular? Survey monkey or qualtrics?
Qualtrics
qualitative analysis
Questionnaires, group discussions, panel of experts to discuss items.
Types of surveys
Self Administered and personal interviews
What are people who deal with scientific method of survey design called?
Survey researchers
Conducting the pilot test
Test + Criterion + Questionnaire, Interviews, Group Discussions, and expert panel.
What type of instructions should be given for the administrators?
Test is in a group or individually, requirements for administration location, privacy, quiet, and comfortable chairs, tables, or desks. No.2 pencils, a computer with a dvd, time limits, script for administrator to read including answers to questions that takers are likely to answer, credentials or training required for the test administrator.
Who do you have to write administration instructions for?
Test taker, administrator, and test scorer.
faking
The inclination of some test takers to try to answer items in a way that will cause a desired outcome or diagnosis
Example of Single-Item Choice:
There are two methods that can be used to evaluate your understanding of course material. One method is to give you an in class exam, and the other method is to have you write a term paper. The question is, which method do you prefer? ________ In-class exam ________Term Paper
What do survey researchers do?
They design, conduct, and analyze surveys
Why do we need to know about surveys?
To know psychology, we need to know about surveys. Psychological tests are used for individual outcomes which are overall scores. Surveys are also used for group outcomes which are the question level.
Example of Implied No Choice:
Why didn't you pass your psychology exam? _____ I did not study. _____I did not feel well. ____ I don't know.
General rule of thumb for developers of tests.
Write twice as many items as you expect to use in final test.
a sample means
a representative subset of the population
Sampling error
a statistic that reflects how much error can be attributed to the lack of representation of the target population by the sample of respondents chosen. Fewer people will leave to increased errors.
population means
all members of the target audience
Why develop a new test?
an achievement test may be needed for a special population of individuals with a disability that affects how they perceive or answer the test questions.
Subjective items Essay Questions-
analysis, synthesis and evaluation many students prefer essay questions because this format allows them to focus on demonstrating what they have learned rather than limiting them to answering specific questions.
complex test formats-portfolio
architects
Response sets
are patterns of responding that result in false or misleading information.
What considers them a survey research firm?
company's server is accessed by user
Discrimination index
compares the performance of those who obtained very high test scores (the upper group [U]) with the performance of those who obtained very low test scores (the lower group [L]) on easy item. U= # of people in U group who were correct over total # in upper group and then X 100. L= # of people in L group who were correct over total # in lower group and then x 100.
complex test formats-simulation
consider the job of a bomb disposal technician
Objective items (stimulus) Multiple choice-
consists of questions or partial sentences, called a stem followed by a number of responses. The incorrect responses are called distractions. Guessing correct answer is 1 out of 4.
When choosing the final items, test developers do what?
construct a matrix. Lists each item, followed by its performance in terms of internal consistency, validity, difficulty, discrimination, and bias.
Administering and scoring the tests first step:
deciding if it's cumulative or categorical model
What is the first step in test development?
define the testing universe, the target audience, and the purpose of the test. Define construct operationally (that means in terms of behavior)
example of an closed ended question:
did you attend last week's biology study group? yes___ no___
To decode subjective formats, is it easy or hard?
easy
To score objective formats, is it easy or hard?
easy
Subjective items Projective Techniques-
evaluating childrens play
Test plan for clinical tests
example: developmental and psychological problems It's Based on carefully researched constructs.
personal interview surveys include:
face to face and telephone interviews
Response bias-social desirability
favorable impression
Subjective items Interview questions (organizational settings)-
general in scope. Interviewer decides what is a good or poor answer.
To decode objective formats, is it easy or hard?
hard
To score subjective formats, is it easy or hard?
hard
What are the 5 steps for regular research?
have a plan, design the study, conduct your study, analyze, communicate findings.
