EXAM 1 - test and measurements
External Validity
Extent to which the results can be generalized to other populations and settings
Amount of time in test retest reliability
The amount of time allowed between measures is critical. The shorter the time gap, the higher the correlation; the longer the time gap, the lower the correlation. This is because the two observations are related over time. Optimum time betweem administrations is 2 to 4 weeks.
measurement error
Any fluctuation in test scores that results from factors related to the measurement process that are irrelevant to what is being measured. The difference between the observed score and the true score is called the error score. S true = S observed - S error Developing better tests with less random measurement error is better than simply documenting the amount of error.
internal consistency
Measures the reliability of a test solely on the number of items on the test and the intercorrelation among the items. Therefore, it compares each item to every other item. If a scale is measuring a construct, then overall the items on that scale should be highly correlated with one another. A common way of measuring internal consistency ... Cronbach's Alpha: .80 to .95 (Excellent) .70 to .80 (Very Good) .60 to .70 (Satisfactory) <.60 (Suspect)
Describe and give example of how a test score is the combination of a true score and an error score
Never know what a score is, it changes moment to moment, always includes some error.
Explain some pro's and con's about different aspects of the subject (p.13): precision, good tools (validity and reliability), take on many forms, interpreted only within context measured, test misuse, achievement tests (p.13-15)
Precision: can measure ones ability, but HOW one solves is difficult. Tools: sometimes use the best, sometimes what's convenient. Best= more time and money - more accurate and reliable. Take on many forms: paper pencil, computer, observation - select form which best fits what Q you're asking. Interpreted only within the context measured: keep test scores in perspective & understand them within the initial purpose for the testing. misuse: know purpose of test, who to give to, quality, how to interpret. Achievement tests: those who know material, those who don't. some are bad test takers.
Define Quartile; Q1, Q2, Q3
Q1 = the 25th percentile, also called first quartile Q2 = the median, also the 50th percentile Q3 = the 75th percentile, third quartile. anyone with a score of 1-25 goes in first quartile. etc.
nominal scales
Separated into different categories NAMES All categories are equal - Cats, dogs, rats NOT: 1st, 2nd, 3rd Republicans, Democrats, Independents There is no magnitude within a category One dog is not more dog than another No intermittent categories No dog/cat or cat/fish categories Membership in only one category, not both Mutually exclusive properties
closed ended should have categories that are?
Should have response categories that are Exhaustive Can use category: "Other" Mutually Exclusive Bad Example: "How many times have you hit your children in the last year?" 0 to 5 5 to 10 10 to 20 20 or more
Advantages of split half and odd-even
Simplest method - easy to perform Time and Cost Effective
Four General Categories of Variables
Situational variables Response variables Participant or subject variables Mediating variables
Test retest reliability
Test-retest reliability is usually measured by computing the correlation coefficient between scores of two administrations. If a scale is measuring a construct consistently, then there should not be radical changes on the scores between administrations --- unless something significant happened. The rationale behind this method is that the difference between the scores of the test and the retest should be due to measurement solely.
open ended questions positive points:
Give us a rich source of information Often used to construct a closed-ended survey for later use Usually included at the end of every survey as a 'catch all' question Example: "Is there anything else you want to share with use about your parents?"
reliability is synonymous with?
consistency.... It is the degree to which test scores for a an individual test taker or group of test takers are consistent over repeated applications. No psychological test is completely consistent, however, a measurement that is unreliable is worthless. The consistency of test scores is critically important in determining whether a test can provide good measurement.
Measurement:
The assignment of labels to a variable or an outcome SAT (variable): 740 (measurement) Age : 22 Weight : 160 Gender : male
IV is on what axis?
X axis
Is this open or closed? "About how many times did you get a spanking as a child?" Never 1 to 20 21 or more
closed.
T/F? The higher the level of measurement the more precise the value.
true
ordinal scales
what is measured by ranks, 1st 2nd 3rd, Although there is a ranking difference between the groups, the actual difference between the group may vary. Marathon runners classified by finish order The times for each group will be different Top ten 4- to 5-hour times Bottom ten 4- to 5-week times
Because no unit of measurement is exact, any time you measure something (observed score), you are really measuring WHAT two things
. True Score - the amount of observed score that truly represents what you are intending to measure. Error Component - the amount of other variables that can impact the observed score Observed Test Score = True Score + Errors of Measurement
Variable
Anything that can take on more than one value SAT score Age Weight Gender
disadvantages of parallel/alternate forms method
Are the two forms of the test actually measuring the same thing? More Expensive Requires additional work to develop two measurement tools. Are the two forms of the test actually measuring the same thing? More Expensive Requires additional work to develop two measurement tools.
Conclusion Validity
Draws reasonable conclusions based upon an analysis of the data
closed ended positive points:
Gives us uniform objective responses Can be easily input into a data file for analysis
Describe how personality tests first got started
In WW1 to place people into different areas in the military . Maturity, leadership, etc.
IV
Independent Variables The variables that are considered to be the "cause" Usually manipulated by the researcher
how can we determine reliability?
Internal Consistency - Test-retest Reliability - Interrater Reliability - Split-half Methods - Odd-even Reliability - Alternate Forms Methods
Survey Instruments can also be developed to assess individual attributes. EXAMPLES:
Personality Anxiety Depression Happiness
administrator factors
Poor or unclear directions given during administration or inaccurate scoring can affect reliability.
types of questionnaires
Self-administration Face-to-face
interval scales
Someone or thing is measured on a scale in which interpretations can be made by knowing the resulting measure. The difference between units of measure is consistent. Height Speed Four miles really is twice as far as two miles
Describe what a professional must do when they notice a particular behavior or set of behaviors
TAKE AN ACTION - they reach conclusion, then take action
causality - Inferences of Cause and Effect Require Three Elements:
Temporal precedence Covariation between the two variables Need to eliminate plausible alternative explanations
open ended difficulties
Hard to summarize across many surveys Must be 'coded' so that statistics can be generated Coding can be too subjective
Min. corelation in test - retest reliability is?
.50 The higher the correlation (in a positive direction) the higher the test-retest reliability
What is the likert scale?
1. Strongly Agree 2. Agree 3. Disagree 4. Strongly Disagree Sometimes insert a "Neutral" response
How high should reliability be?
A highly reliable test is always preferable to a test with lower reliability. .80 > greater (Excellent) .70 to .80 (Very Good) .60 to .70 (Satisfactory) <.60 (Suspect) A reliability coefficient of .80 indicates that 20% of the variability in test scores is due to measurement error.
FACTORS that affect reliability
Administrator Factors Number of Items on the instrument The Instrument Taker Length of Time between Test and Retest
disadvantages of split-half and odd even
Many was of splitting Each split yields a somewhat different reliability estimate Which is the real reliability of the test?
closed ended difficulties:
May fail to include proper response categories Example: "How many sexual partners have you had in the last year?" 1 to 5 6 or more
NONEXPERIMENTAL VERSUS EXPERIMENTAL METHODS
Nonexperimental Method Direction of Cause and Effect The Third-Variable or Confounding Variable Problem Experimental Method Experimental Control Randomization
Give example of predictive validity and how the GRE is an example of a test with problems
The GRE doesn't necessarily predict how well one will do in graduate school.
Internal consistency estimates are a function of:
The Number of Items - if we think that each test item is an observation of behaviour, high internal consistency strengthens the relationship --- i.e., There is more of it to observe. Average Intercorrelation - the extent to which each item represents the observation of the same thing observed. The more you observe a construct, with greater consistency RELIABILITY
types of variables
discrete (belongs to unique and separate categories, i.e.: dogs, cats, rats) If there are only two categories, then it is a dichotomous variable i.e. open or closed, male or female. Continuous variables: What is measured varies along a line scale and can have small or large units of measure Length Temperature Age Distance Time
The casual possibilities in a non-experimental study
exercise -> anxiety anxiety -> exercise income -> exercise and anxiety
List some common tests and acronyms
APGAR: appearance, pulse, grimise, acuity, respiration (new born test) SAT, GRE, FCAT, MCAT
Internal Validity
Ability to draw conclusions about causal relationships from our data
Perceptions and attitudes in questionnaires
in general Perceptions and attitudes Public views on a variety of subjects Sentencing policies Gun control Police performance Drug abuse Example: Campus Security: fear > victimization
Give some examples of independent and dependent variables
IV - whats manipulated, how much alcohol DV - whats being measured, how drunk are you
ratio scale
Just like an interval scale, and there is a definable and reasonable zero point. Time, weight, length Seldom used in social sciences All ratio scales are also interval scales, but not all interval scales are ratio scales
Give examples for the different types of validity (p.64)
predictive validity - how well a test outcome is consistent with a criterion that occurs in the future - do high scores in high school predict one doing well in college. content validity - where the test items sample the universe for items for which the test is designed - achievement tests, certification, licensing. examine closely to be sure its accurate. Criterion validity - when you want to know if test scores are systematically related to other criteria that indicate the test taker is competent in a certain area (correlate scores with another measure which is valid and assess same abilities) construct validity - if a test measures some underlying psychological construct -how well a test score reflects an underlying construct.
In-Person Interviews?
Completion rates are very high (85% +) Reduces 'don't knows' Can explain questions to eliminate confusion Can observe the respondent and make valuable notes on their behavior Can 'probe' for deeper information But: people are more likely to give 'socially-acceptable' answers
Construct Validity
Adequacy of the operational definition of variables
question ordering can have what affect?
Administer the most sensitive questions near the end of the survey Example: Health; Income; Sexuality Be aware that questions that appear earlier can influence answers later Example: Questions about neighborhood crimes before questions about gun control laws
open/closed ended should be?
Clear and unambiguous "I feel fear when I must speak in public." Measured with Likert Scale Short items are best Don't use Negative Items "I don't think people should be afraid to speak." Rather use "I like to speak to large groups." Wording should be examined for potential bias: "assistance to poor" rather than "welfare" Results: 63% of respondents said too little was given vs. 23% too little for welfare "people who are homeless" rather than "bums"
Construct Validity
Evaluate the adequacy of the operational definition. Is the operational definition sufficiently measuring the construct it claims to measure?
the test retaker
If you took an instrument in August when you had a terrible flu and then in December when you were feeling quite good, we might see a difference in your response consistency. If you were under considerable stress of some sort or if you were interrupted while answering the instrument questions, you might give different responses.
types of reliability
Internal Consistency (Consistency of the items) - Test-retest Reliability (Consistency over time) - Interrater Reliability (Consistency between raters) - Split-half Methods - Alternate Forms Methods
The book's authors define a test as:
everything - exams, test, procedure. any way of measuring things
What are some problems with Stanine scores
lumps people together, no individual differences, cannot tell differences in performance, limit to nine categories, assumption that placing in one of nine stanines makes conceptual sense.
Problem in test-retest
memory effect. Which means that a respondent may recall the answers from the original test, therefore inflating the reliability.
Mediating Variables
number of bystanders -> diffusion of responsibility -> helping behaviors
Define Stanine
one of nine equal segments in a normal distribution. Half of one standard deviation normally.
number of items
The larger the number of items, the greater the chance for high reliability. : Use longer tests or accumulate scores from short tests.
DV is on what axis?
Y axis
How did Alfred Binet contribute to tests and measurement
investigated why children didn't do well which led to intelligence testing
types of questions
open ended: "Tell us about your relationship with your parents." closed ended: Single worded answer - "did you parents hit you?
what are levels of measurement?
ordinal, nominal, interval, ratio
Define a percentile or percentile rank
percentage of people who scored lower than you. The point in a distribution below which s percentage of scores fall.
Explain and give examples why as psychologists we act as if ordinal level data is really interval level data
tradition
What is controversial in Kaplan's approach to tutoring
he offered tutoring for standardized testing test so people could do better. built up tutoring school, which still exist today. still assumption we can teach people, how to do better on tests. CONTROVERSY = no data to show that tutoring actually helps.
DV
The variables that are considered to be the "effect" Usually measured by the researcher
Participant or Subject Variables
These extraneous variables are related to individual characteristics of each participant that may impact how he or she responds.
why do Psychologists typically treat their measurements as if they are Interval Data
This is done mostly out of tradition This is done despite the fact that IQ or Personality Scores are actually Ordinal Data And - despite the fact that the statistical tests we choose demand Interval or Ratio level data We justify this because our tests are 'robust'
OPERATIONAL DEFINITIONS OF VARIABLES
Variable is an abstract concept that must be translated into concrete forms of observation or manipulation Studied empirically Help communicate ideas to others
type of variable : continuous variables
What is measured varies along a line scale and can have small or large units of measure Length Temperature Age Distance Time
Example of true experiment: in class
Women in Drug Treatment (Tx) I.V.'s... Assessed for pre- post-Tx drug use (D.V.) using surveys Other D.V.'s measured
Measurement Error is Reduced By?
Writing items clearly - Making instructions easily understood - Adhering to proper test administration - Providing consistent scoring
Describe and give examples of how we might assess: Achievement, Personality, Aptitude, Ability (intelligence and vocational)
achievement: history exam, or exam Personality: criteria test Aptitude: how good is someone ins sales ability: skills and competence (intelligence - learn and apply)
The chapter title is "Welcome to Lake Woebegone..." This comes from: a fictional town in Minnesota from a character who reports on the radio: "Well, that's the news from Lake Wobegon, where all the women are strong, all the men are good looking, and all the children are above average." Why is this relevant to our discussion of percentiles
because ignore fact that there is variability and that there are other places for distribution.
reliability
consistency of the instrument
SPLIT HALF reliability
refers to determining a correlation between the first half of the measurement and the second half of the measurement (i.e., we would expect answers to the first half to be similar to the second half).
Describe the concept of validity
the assessment tool does that it says it does. Wants to know whats being tested and reliability how consistently. ADD more items to make it more valid.
Parallel/Alternate Forms Method
refers to the administration of two alternate forms of the same measurement device and then comparing the scores. Both forms are administered to the same person and the scores are correlated. If the two produce the same results, then the instrument is considered reliable. A correlation between these two forms is computed just as the test-retest method.
Describe and give examples for the five main reasons we do tests: Selection, Placement, Diagnosis, Hypothesis Testing, Classification for career
selection: training, who to train, who to accept placement: giving a math test to see what course to put them in diagnoses: mental disorders hypothesis: did the program work? classification for career: what will suit you the best?
ADVANTAGES OF MULTIPLE METHODS
Artificiality of Experiments Ethical and Practical Considerations Participant Variables Description of Behavior Successful Predictions of Future Behavior
advantages of parallel/alternate forms method
Eliminates the problem of memory effect. Reactivity effects (i.e., experience of taking the test) are also partially controlled. Can address a wider array of issues than the test-retest method.
two types of self reports
Frequency of occurrence How many drugs used over time Prevalence of occurrence How many women use drugs
Response Variables
All potential responses to the manipulation of an independent variable or observed in reaction to the environment.
If you can... (types of data) why type do you have?
Assign just names Then you have Nominal Data Put things in order Then you have Ordinal Data If the distance between things is consistent Then you Interval Data If the scale has an absolute zero Then you have Ratio Data
How did Darwin's work influence (directly and indirectly) tests and measurement
Darwin saw individual differences which led to his cousin francis goltman took idea and applied to humans in anthroprometric method= metrics for differences in humans ig weight, height,
Questionnaire construction
Don't squeeze too much on a page Leave it open and 'breezy' looking Reduces errors Doesn't demoralize the respondent as they move through the pages quickly Don't use abbreviations like "abbrev." as they can confuse everyone Use 'contingency' or 'skip-questions' so only certain people get certain questions Example: Pregnancy questions for women - otherwise "If male - skip to question 60" Use matrix for questions with the same potential responses Example: Agree Disagree Spanking helps [ ] [ ] Hit with hand only [ ] [ ] Using belt is okay [ ] [ ] Useful for Likert responses: SA A D SD
Internal Validity
Evaluate the extent that it was the independent variable that caused the changes or differences in the dependent variable. Are there alternative explanations (confounds)?
External Validity
Evaluate the extent that the results can generalize to other populations and settings. Can the results be replicated with other participants? Can the results be replicated in other settings?
Situational Variable
Extraneous features of the environment that are present and may influence dependent variable responses.
Explain what you would do if your test validity is low in: content validity and construct validity
content - rewrite, consult someone who is in this field and they will rewrite it. construct - take a better look at the theoretical rationale that underlies the test you developed and the items you created to reflect that rationale. maybe the definition and theoretical model are underdeveloped.
What is the purpose of textbook
expose us to assign and use the assessment instruments in sociological, legal, political testing. See legal implications.
interrater reliability
Interrater reliability means that if two different raters scored the scale using the scoring rules, they should attain the same result. Interrater reliability is usually measured by computing the correlation coefficient between the scores of two raters. Here the criterion of acceptability is pretty high (e.g., a correlation of at least .9), but what is considered acceptable will vary from situation to situation
ODD EVEN reliability
refers to the correlation between even items and odd items of a measurement tool. In this sense, we are using a single test to create two tests, eliminating the need for additional items and multiple administrations. Since in both of these types only 1 administration is needed and the groups are determined by the internal components of the test, it is referred to as an internal consistency measure.
Why do test scores vary - Possible Sources of Variability of Scores
General Ability to comprehend instructions - Stable response sets (e.g., answering "C" option more frequently) - The element of chance of getting a question right - Conditions of testing - Unreliability or bias in grading or rating performance - Motivation - Emotional Strain
