PSYCH 309 Midterm #1
Know the four scales of measurement
(NOIR) nominal, ordinal, interval, ratio
magnitude
(property of scale) property of moreness
equal intervals
(property of scales) Difference between 2 points on a scale has the same meaning as the difference between 2 other points that differ by the same number of units
absolute zero
(property of scales) nothing of the measured property exists
Define and explain how the extreme group and point biserial methods differ.
-Extreme Group Method: compares people who have done well with those who have done poorly on the test -The Point Biserial Method: find the correlation between performance on the item and performance on the total test
What factors should be considered when choosing a reliability coefficient?
-Homogeneity v heterogeneity of items: is the test measuring a multi-faceted or uni-faceted construct? -Dynamic v static characteristics: is the true score fluctuating or relatively stable over time? Does it change frequently or from situation to situation?
When shown an item characteristic curve, be able to determine good or poor discrimination
-high positive slope - good discrimination -no or negative slope- bad discrimination
How do the different aspects of internal consistency differ?
-internal consistency: examine how people perform similar subsets of items selected from the same form of the measure consistency of items within the same test. evaluate the extent in which the different items on a test measure the same ability or trait -Split half: corrected correlation between two halves of the test -KR20: requires you to find the proportion of people who got each item "correct" -Alpha: designed to use KR20 with tests where there is not a right or wrong answer (such as a likert scale test) more general reliability estimate
Polytomous format
-resembles dichotomous format but there are more than two alternatives, multiple choice tests, easy to score, takes little time for the test taker, incorrect choices are called distractors, Well-chosen distractors are an essential ingredient for good items -ADVANTAGE: good to tell if someone really know the information -DISADVANTAGE: poorly written distractors can adversely affect the quality of a test
how do you measure split half reliability (internal consistency)
-this can cause problems when one half is more difficult than the other, if this is the case its better to use the odd-even system where one subscore is obtained from odd numbered items and vice versa -to estimate the reliability you need to find the correlation between the two halves -spearman brown formula: corrects for the half length: corrected r= 2r/1+r where r is the estimated correlation between the two halves of the test if each test had the total # of items in the test
Dichotomous Format
-two alternatives for each item Most common is T/F -ADVANTAGES: easy construction and easy scoring, T/F require absolute judgment -DISADVANTAGES: encourage students to memorize material, "truth" comes in shades of gray, do not allow test takers to show they understand complexity, tends to be less reliable and less precise
skewness
A measure of the shape of a data distribution
criterion validity
A property exhibited by a test that accurately measures performance of the test taker against a specific learning goal.
interval scale
Adjacent values on the scale represent equal intervals in magnitude of the attribute being measured. What properties does it have? -data can thus be: Classified, Counted, Proportioned, Rank-ordered, added (creating a total-scale score), Subtracted, divided to form averages (calculating a scale mean) -data does not have a "0" point and cannot be: Divided to form ratios -temperature
advantages and disadvantages of mode
Advantages: easy to obtain, only measure that can be used for nominal scale Disadvantages: not stables sample to sample, there may be more than one mode for a particular set of scores
advantages and disadvantages of median
Advantages: less sensitive to extreme scores, distributions that are skewed the median is the best measure Disadvantages: it responds to how many scores are above or below but not how far away the scores are
advantages and disadvantages of mean
Advantages: the best choice when we need a measure of central tendency to reflect the total scores, stable from sample to sample, most resistant to chance sample variation Disadvantages: reactive to exact position of each score, it gives undue weightage to extreme values
measurable phenomenon
All phenomena the construct generates (gives rise) (ex. Panic attacks and operational def if minutes spent worrying, # of anxious thoughts)
ordinal scale
Assignment of ranks according to the degree to which the measured attribute is present/absent. What properties does it have? -data can be: Classified, Counted, Proportioned, Rank-ordered -data cannot be: Added/Subtracted or Divided to form averages/ratios -ex. how do you feel on a scale of 1-10
Know and be able to identify examples of a double-barreled item.
Avoid "double-barreled" items that convey two or more constructs at the same time
Define Content Validity: How is it measured?
Based on the correspondence between the item content and the domain the items represent. Measured through asking whether the items are a fair sample of the total potential, consider the wording of test items (Does my test get at the whole domain?)
What are the primary differences between the Likert and Category formats?
Category Format: Like the Likert format but has an even greater number of choices,10 point rating scale, people will change ratings depending on the context, problems can be avoided if the endpoints of the scale are clearly defined and the subjects are frequently reminded of the definition of the endpoints
What are the two types of evidence in construct validity?
Convergent and discriminant
What is the Correlation Coefficient? With what concept should correlation not be confused?
Describes how much two measures or items covary. How similar is the variance between the variables. Not to be confused with causation
What types of questions are answered by psychologists through assessment?
Diagnosis and Treatment Planning, Monitor Treatment Progress, Help clients make more effective life choices/changes, Program evaluation, Helping third parties make informed decisions
IQR
Discards the distribution's upper and lower 25% and taking what remains IQR = Q3 - Q1 (middle 50% or 75th to 25th%)
To avoid bias, how should error be distributed in a psychological test?
Double blind, random sampling, want error to be unsystematic and random!
Which two types of validity are logistical and not statistical? Why?
Face validity and content validity Require good logic, intuitive skills, and perseverance
What are the five characteristics of a good theory?
Has explanatory power, broad scope, systematic, fruitful, Parsimonious
Why types of irregularities might make reliability coefficients biased or invalid?
How could you introduce bias basically. environment, personal, evaluator bias, not the same scoring material, tired participants, personal effects
How can one address/improve low reliability?
Increase the number of items, Factor and Item analysis, the reliability of a test depends on the extent to which all of the items measure one common characteristic, Correction for attention: estimating what the correlation between tests would have been if there had been no measurement error, Estimate the true correlation if the test did not have measurement error
Kurtosis
Index of the "peakedness" vs. "flatness" of a distribution
In what settings do psychologists assess and what is their primary responsibility in each?
Inpatient, Schools, Forensic (legal) settings, Employment settings, such as corporations and law firms, Career counseling, Pre-marital counseling
What is the Pearson product moment correlation? What meaning do the values -1.0 to 1.0 have?
It is a ratio used to determine the degree of variation in one variable that can be estimated knowledge about variation in the other variable. The closer to +1, the stronger the positive correlation is. The closer r is to -1, the stronger the negative correlation is. If |r| = 1 the variables are perfectly correlated! (continuous variables)
ratio scale
Measured on a scale with a true "0" point. Allows all mathematical operations. It can be meaningfully: Classified, Counted, Proportioned, Rank-ordered, added (creating a total-scale score), Subtracted, divided to form averages (calculating a scale mean), divided for form ratios -ex. weight scale
median
Middle score in the distribution (50% ↑ 50% ↓) Rank the scores (include repeating scores) from lowest to highest If an odd number of scores, select the middle score. If an even number of scores, take the average of the middle scores. measure of central tendency
Playtikurtic
Negative kurtosis = flatter distribution (Plate)
percentiles
Percentage of test-takers whose scores fall below a given raw score
Leptokurtic
Positive kurtosis = more peaked distribution (Leaping)
hypothetical construct
Processes that are not directly measurable, but which are inferred to have real existence and to give rise to measurable phenomena
What is psychometry? What are the two major properties of psychometry?
Psychometry: the branch of psychology dealing with the properties of psychological tests -Reliability: dependability, consistency, or repeatability of the test results (measuring tool) -Validity: Does a test measure what it purports to measure? Accuracy
What is the relationship between reliability and validity?
Reliability is not necessary but sufficient for validity
What is factor analysis?
Studies interrelationships among items within a test. Data-reduction technique. Can be used as measure of internal consistency
How are T scores different from Z scores?
T scores (Unlike Z Scores): Mean = 50 and standard deviation = 10. Are all positive, Values > 70 are often considered "clinically significant" (2 SD's above)
What are the stages of test development?
Test conceptualization, Test construction, Test Tryout, Item Analysis, Test revision
What should be asked when generating a pool of candidate test items?
Test is covering universe of construct. What content domain should the test items cover? How many items? What are the demographics of population? How should I word my items?
Norm (testing)
Testing in which scores are compared with the average performance of others. Test norms are created during the standardization process and must be periodically updated.
Ecological validity
The extent to which a study is realistic or representative of real life.
What is the co-efficient of determination? What is the purpose of the co-efficient of determination?
The proportion of the total variation in scores on Y that we explain through X (r^2)
standardization. why is it important to obtain a standardization samle?
The uniform procedures used in the administration and scoring of a test. This is important because without it the meaning of scores would be almost impossible to evaluate
What is restricted range? To what does it lead?
Using a sample of people who won't fit the test or test is too easy or hard. It reduces range and variability (leads to flooring and ceiling effect)
nominal scale
Variables can be named - put into categories. What properties does it have? -Values symbolize category membership, and can be: Classified, Counted, Proportioned -data cannot be meaningfully: Ranked, Added/Subtracted, divided to form averages/ratios -ex. labels
Define item analysis. What two methods are closely associated with item analysis?
a general term for a set of methods used to evaluate test items, one of the most important aspects of test construction. The basic methods involve assessment of item difficulty and item discriminability
test
a measurement device or technique used to quantify behavior or aid in the understanding and predictions of behavior
What is a scatterplot (scatter diagram)? How does it work?
a picture of the relationship between 2 variables. each point on the diagram shows where a particular individual scored on both X and Y
item
a specific stimulus to which a person responds overtly (questions on a test)
Define split half reliability
a test is given and divided into halves that are scored separately. the results from each half are compared to one another
If a test is reliable its results are what?
accurate, dependable, consistent, or repeatable
What is a construct?
an indicator variable that measures a characteristics, or trait. For example, college admission scores are constructs that measure how well a student is likely to do in their first year. Construct validity measures how well the observed construct predicts the outcome expected.
What is incremental validity?
are you offering something new with your test
What components make up Classical Test Score Theory?
assumes that each person has a true score that would be obtained is there were no errors in measurement. This is used to understand and improve reliability of test. X=T+E (observed score=true score+error)
What is systematic error variance called? Is it good or bad and why?
bias, bad
Know the different types of correlations and when they are used.
biserial correlation, point biserial correlation, phi coefficient, spearmans rho
norm referenced tests
compares a test takers performance to others
psychological assessment
comprised of tests, interviews, case studies, behavioral observations, apparatus, etc. it is comprised of psychological tests
What are the three main types of validity evidence?
construct related, criterion related, content related
concurrent validity
criterion is at the same time as the measure. you work at the factory and they give you a test to make sure you know what your doing while you are there
Define item difficulty. What does the proportion of people getting the item correct indicate?
defined by the number of people who get a particular item correct.1st things a test constructor needed to determine is the probability that an item could be answered correctly by chance alone
operational definition
defining a way to measure a hypothetical construct
Reliability
dependability, consistency, or repeatability of the test results (measuring tool) -The proportion of true to total observed score variance
Define item discriminability. What is good discrimination? What are two ways to test item discriminability?
determines whether the people who have done well on particular items have also done well on the whole test - 2 ways to test item discrimination - Extreme group method - top 3rd and bottom 3rd - Point biserial - all people tested
standardization
develop specific (standardized) procedures for the administration, scoring, and interpretation of a test
What is a z score? How is it calculated?
difference between a score and the mean, divided by the SD
Define parallel/alternate forms reliability. What are its advantages and disadvantages?
different forms of the same test (ACT/SAT). They are hard to make the same in difficulty but are better tests to administer.
content validity
does the test represent the whole content. does it get to the whole breathe and depth of the content ex: if you are being tested on the first 6 chapters of a book, then content related validity is provided by the correspondence between the items on the test and the information in the chapters.
construct validity
does your test measure what it should measure. ex: trying to find out if an educational program increases emotional maturity in elementary school age children. Construct validity would measure if your research is actually measuring emotional maturity.
What is the regression formula? Understand the different components of the formula and how they are applied.
equation of the form Y = a + bX, where X is the explanatory variable and Y is the dependent variable. The slope of the line is b, and a is the intercept (the value of y when x = 0)
In what ways can error impact the observed score?
error pulls from the true score
test-retest method
evaluates the error associated with taking a test at two different time (rorshark ink blot tests are not appropriate for this evaluation) there is a possibility of a carryover effect: when the first testing session influences scores from the second session (some remember their answers from their first test). or practice effects, when some skills improve with practice
Internal consistency
examine how people perform on similar subjects of items selected from the same form of the measure
What is the content validity ratio and how is it calculated?
experts look at it and the rate if it is related or not and give it a CVR content validity ratio. The formula of content validity ratio is CVR=(Ne - N/2)/(N/2), in which the Ne is the number of panelists indicating "essential" and N is the total number of panelists (good for educational testing)
biserial correlation
expresses the relationship between a continuous variable and an artificial dichotomous variable.
What is Criterion-related Validity?
how well the test relates to the criterion we are using.
postdictive validity
if the test is a valid measure of something that happened before. For example, does a test for adult memories of childhood events work?
Be able to define and recognize the Likert Format. What scales most frequently use the Likert format?
indicate the degree of agreement with a particular question, used with attitude and personality, "Strongly agree... strongly disagree"
Central tendency
indices of the central value or location of a frequency distribution with respect to the X Axis.
The Normal distribution
is the most common continuous probability distribution. The function gives the probability that an event will fall between any two real number limits as the curve approaches 0 on either side of the mean. Area underneath the normal curve is always equal to 1
Mesokurtic
kurtosis at zero. normal distribution (Medium)
Define item characteristic curve. Know what information the X and Y axes give as well as slope
learn about items by graphing their characteristics. X is ability level Y is probability of correct responses
What are the three properties of scales that make scales different from one another?
magnitude, equal intervals, absolute zero
Three types of central tendency
mean, median, mode
mode
measure of central tendency, the most frequently occurring score in the distribution
What is the Kappa statistic and how does it relate to reliability?
measures Inter rater reliability (observations of the samples with more types of judgment) indicates the actual agreement as a proportion of the potential agreement following correction for chance agreement
Criterion referenced tests
measures performance against an established criterion
spearman's rho
method of correlation for finding the association between two sets of ranks tetrachoric correlation: if both dichotomous variables are artificial
Which type of validity has been referred to as "the mother of all validities", or "the big daddy" and why?
mother: construct validity. it measures what it should. you want your test to measure the construct
multiple regression
multivariate analysis that considers the relationship among combinations of three or more variables; the goal is to find the linear combination of the three variables that provides the best prediction
Name and define the three subtypes of criterion related validity. Be able to give examples.
predictive, concurrent, postdictive
covariance
relationships between variables (How much both variables change together)
What example was given in class regarding reliability
rubber yardstick
Define representative sample. Know when and why representative samples are collected.
sample comprised of individuals similar to those for whom the test is to be used When: when used for the general population, a rep. Sample must reflect all segments of the population Why: it can be used as a standardized sample and be representative of an entire population which raised the validity
predictive validity
sat and freshman GPA, the test is trying to predict something in the future
simple linear regression
seeks to find the linear explanation for the relationship between 2 variables
convergent evidence of validity
shows how similar your test is to other test that are measuring the same thing
discriminant evidence of validity
shows that your test is different from other tests of different constructs
variance
standard deviation squared
What is the principle of least squares? How does it relate to the regression line?
statistical procedure to find the best fit for a set of data points by minimizing the sum of the offsets or residuals of points from the plotted curve
negative skew
tail points to the left (towards - end)
positive skew
tail points towards to the right (towards + end)
What contributes to measurement error?
test construction, test administration, test scoring and interpretation
What prerequisites exist for validity?
test needs to be RELIABLE. you can't have validity without reliability (you can have reliability without validity)
Test reliability is usually estimated in one of what three ways?
test retest method, parallel forms method, internal consistency
intelligence testing
testing a person's general potential to solve problems
aptitude testing
testing potential for learning or acquiring a specific skill
achievement testing
testing previous learning
incremental validity
tests add something to new science
skrinkage
the amount of decrease observed when a regression equation is created for one population and applied to another
mean
the arithmetic average of a distribution, obtained by adding the scores and then dividing by the number of scores. measure of central tendeny
What is Construct-related Validity?
the degree to which a test measures what it claims, or purports, to be measuring
residual
the difference between the predicted and the observed values
Define Face Validity. How does it differ from other aspects of validity?
the mere appearance of that measure has validity Not technically validity but still important. When its obvious what you are measuring.
percentile rank
the percentage of scores below a specific score in a distribution of scores. equation is (BN/)X100. B is # of cases below individual score and N is total # of scores -Ex. Runner finishes 62nd out of 63. 1/63 = .016 = 1.6
What is the meaning of a squared validity coefficient?
the percentage of variation in the criterion that we can expect to know in advance bc of our knowledge of the test scores. ex: from the previous example we will know .40 squared, or 16% of the variation in college performance bc of info we have from the SAT test. the remaining 84% variance of why they preform differently is still unexplained
standard deviation
the positive square root of the variance
psychological testing
the process of measuring psychology-related variables by obtaining information
What is the validity coefficient?
the relationship between a test and a criterion is usually expressed as a correlation called validity coefficient. ex: the SAT has a validity coefficient of .40 for predicting GPA at a particular university. bc the coefficient is significant we can say that it tells us more about how well people will do in college than we would know by chance. (r)
How do construct underrepresentation and construct-irrelevant variance relate to content validity?
the score that you get on a test should represent your comprehension of the content you are expected to know. construct underrepresentation is the failure to capture important components of a construct. construct irrelevant variance occurs when scores are influences by factors irrelevant to a construct.
What is the standard error of estimate? What is its relationship to the residuals?
the standard deviation of the residuals
Define and be able to apply the broad definition of validity.
the usefulness and meaning of the results. can be defined as an agreement between a test score or measure and the quality it is believed to measure. is can also be defined as the answer to a question. does the test measure what is it supposed to measure?
Parallel forms method
this compares two equivalent forms of a test that measures the same attribute Advantages: Reduces memory bias, One of the most rigorous assessment of reliability Disadvantages: Hard to construct
Understand the major components of inter-rater reliability.
three different ways to do this: most common method is to record the percentage of times that two or more observers agree. Kappa statistic is the best method for assessing the level of agreement among several observers
What is the purpose of factor and item analysis?
to see if a certain item is bringing the reliability down. see how many factors there are in the test
observed score
true score plus error
what are test batteries?
two or more tests used in conjunction
point biserial correlation
used when the dichotomous variable is true, meaning that the variable naturally forms two categories
What does the standard error of measurement do?
uses standard deviation of errors as the basic measure of error. allows us to estimate the degree to which a test provides inaccurate readings. the larger the standard error of measurement, the less certain we can be about the accuracy with which an attribute measured
phi coefficient
when both variables are dichotomous and at least one of the dichotomies is true
