Testing and Measurement - Test 1 - Dr. Alvarez
criterion-referenced interpretation
interpret test performance in relation to some well-defined external criterion rather than in relation to norms e.g. classroom exams: >90% = A, <60% = F; EPPP (Nat'l Board Exam for Psych) is a criterion reference exam/interpretation Applicability: requires a well-defined body of content
Non-normal shapes of distribution
kurtosis - the "peakedness" of the distribution leptokurtic - more peaked than normal distribution platykurtic - flatter than normal distribution modes: bimodal (2 modes), multimodal, etc.
Major Force: Practical Applications
most developments came from a need to satisfy practical problems (e.g., Binet tried to solve a practical problem for Parisian schools)
frequency distribution
organizes the data into groups of adjacent scores
frequency histogram
plots score ranges along the x-axis and score frequencies along the y-axis
Educational Uses of Tests
primarily group-administered tests of ability or achievement, or predicted future success in academic work E.G., SAT, ACT, LSAT, OLSAT, GRE
Personnel/Employment Uses of Tests
primarily used in business and the military for 2 purposes 1) select individuals most qualified for a position 2) assign individuals to different tasks to optimize the organizations overall efficiency
Tests Info: Book on Single Tests
Entire book devoted to just one test E.g., WAIS-IV (Wechsler Adult Intelligence Scale), MMPI-2/MMPI-2-RF (Minnesota Multiphasic Personality Inventory) Strengths: outstanding, in-depth coverage Weaknesses: available for only a few tests, typically written by person who are positive about the test, not impartial
Tests Info: Other Users
Fellow professional in the field may be a helpful source of info Strengths: good source for what is typically used and for peculiarities of a test Weaknesses: not all are up-to-date
Descriptive Statistics
Help to summarize (describe) raw data to aid our understanding of the data (e.g. mean, median, mode, SD)
Wilhelm Wundt
His psychophysics lab (1879) established experimental psych and emphasized standardized conditions and precision of measurement. Trained many key figures Theory: Structuralism - mind elements and how they work together
Test development question about tests
How do we develop a good test?
Norms question about tests
How do we interpret the scores?
Validity question about tests
How do we know what the test actually is measuring?
Speed vs. Power Tests
How fast the examinee completes easy items. How accurately the examinee completes questions testing the depth of their knowledge
Practical issues questions about tests
How much does it cost? (can be v expensive) How long does it take? (we want a much info as possible) Is it easily obtained? Is it available in languages other than English?
Reliability question about tests
How stable are the scores on a test?
Francis Galton
In "The Roots" Era Darwin's cousin interested in hereditary genius - believed in fixed IQ Invented the bi-variate distribution chart - his friend Pearson made "r"
Alfred Binet
In "The Roots" Era Parisian schools Binet-Simon Scale in 1905
Charles Spearman
In "The Roots" Era created 1st modern, empirically-based theory of intelligence; developed by an early form of factor analysis
Horace Mann
In the "Setting the Stage" Period Through the Boston School Committee he advocated for improvement in the way schools evaluate their students
4 Major Uses and Users of Tests
1. Clinical 2. Educational 3. Personnel/Employment 4. Research
6 Common Elements of a Test
A tests is a: 1) Standardized 2) process or device 3) that yields info 4) about a sample of 5) behavior or cognitive processes 6) in a quantified manner
Mental Ability
Category of Psychological Testing includes cognitive functions such as vocab and memory Historcially centered on intelligence 3 subtypes
2 Major textbooks during the Consolidation Period
Chronbach's Essentials of Psychological Testing (1949) Anastasia's Psychological Testing (1954)
Percentiles
Concept: percentage of cases in the norm group falling below a score Range: 1-99, Median: 50
Defining a variable from most general to most specific
Construct --> Measure --> Raw Data
Tests Info: Textbooks on Testing
Lists of tests in major categories but not intended to be the primary source about particular tests Strengths: good overview Weaknesses: examples/illustrative, may be outdated
Standard Scores to Know: T-Scores
M = 50, SD = 10 Widely used in personality tests (e.g. MMPI-2)
Mode (Mo)
Measure of Central Tendency the most frequent score
Median (Md)
Measure of Central Tendency the score that falls in the middle of the distribution of scores
Mean (M)
Measure of Central Tendency the statistical average of the scores of all participants in a sample: add up all the scores and divide by the # of participants Disadvantage: affected by outliers
Major Force: Rise of Clinical Psych
Testing is closely tied to the emergence of this field As new tests developed, clinicians used them or developed tests themselves
Inferential Statistics
drawing conclusions (inferences) from the sample to the population as a whole (e.g. t-test, Chi Square, ANOVA) All of these statistics yield a p-value
Major Force: Statistical Methodology
interactive relationship with test dvpt. (a number of stat methods were invented specifically in response to dvpts. in testing)
Setting the Stage Period
1840-1880 Concern for mental illness increased and diagnostic methods began Formal written exams - Horace Mann Darwin's theory of evolution got people thinking about differences between species and individuals Experimental Psych established 1879 by Wundt
The Roots Period
1880-1915 Important People: Francis "psych testing" Galton, Alfred "intelligence testing" Binet, Charles "modern intelligence theory" Spearman Concern for reliability emerging - created test that could be scored objectively - hope to make education scientific and increase scorer reliability
The Flowering Period
1915-1940 Many new tests during this era: Terman revised Binet-Simon Scale --> Stanford-Binet (1916) Otis revised Stanford-Binet --> group-administered forms called Army Alpha & Beta (1918/WWI) Rorschach ink blots (1921) Wechsler-Bellevue Intelligence Scale (1939)
The Consolidation Period
1940-1965 Development and revision of tests Application of testing in WWII, clinical pracitce, schools, and industry
Just Yesterday Period
1965-2000 Item response theory - using item characteristic curves, replacing classical test theory which was big in Consolidation Era Legislative and judicial activism - accountability in education, civil rights movement, appropriate use of tests with disabled persons) Public criticism - against standardized tests of ability & achievement Influence of computers on recent testing (1990s-on)
And Now Period
2000-present huge increase in number and diversity of tests influence of managed care: desire for more focused testing on diagnosis --> treatment --> outcomes emergence of evidence-based practice Online administration/reporting of test and dvpt of computer programs to simulate human judgement in the analysis of responses
Tests Info: Electronic Listings
4 Major Electronic Sources: 1) Educational Testing Service 2) Mental Measurements Yearbook 3) Health and Psychosocial Instruments (HaPI) 4) PsycTESTS (http://library.tulane.edu) Strengths: Quick access, convenient Weaknesses: some provide only basic info, some entries are outdated
Interval
Type of scale Equal intervals on scale (e.g. Fahrenheit thermometer, z-scores)
Nominal
Type of scale classifies, assigns numerals (e.g. Student ID #)
Ordinal
Type of scale places in order of magnitude (e.g. College football rankings, percentiles, Likert Scale)
Ratio
Type of scale true zero point (e.g. True zero point on the Kelvin scale, height, weight)
Remote Background Period
Up to 1840 Concern for what was common Oral examinations much more common than written exams Chinese civil service (gov't workers) exams (beginning 200 B.C.) helped with job placement
9 Major Sources of Information about Tests
1) Comprehensive Listings 2) Systematic Reviews 3) Electronic Listings 4) Special Purpose Collections 5) Book on Single Tests 6) Textbooks on Testing 7) Professional Journals 8) Test Publishers 9) Other Users
4 Assumptions of Testing
1) Humans have traits and differences in these traits (e.g. extroversion, verbal ability) are potentially important 2) We can quantify these traits on a continuum 3) traits are reasonably stable 4) measures of the traits relate to actual behavior in real-life situations
3 Research Uses for Tests
1) May serve as the operational definition of a dependent variable in a study 2) May be used to describe samples 3) Research on the tests themselves
Standard Score Characteristics
1) Mean is set at a standard value 2) SD is set at a standard value 3) Since SS's follow the Normal distribution, value of any score expresses its relative location in the distribution Note: if raw scores are not normally distributed, need an area transformation to have a normalized SS
5 Fundamental questions about tests
1) Norms - numerical summary of standardized scores 2) Reliability - consistency of the measurement 3) Validity - accuracy 4) Test Development 5) Practical Issues
3 Major types of norms
1) Percentiles 2) Standard Scores 3) Developmental Norms These 3 are systematically related to one another and conceptualized in the context of the normal curve
3 Significant Publications during the Flowering Period
1) Psychometrika 2) Educational & Psychological Measurement 3) Mental Measurements Yearbook - reviews of tests
7 Historical Periods
1) Remote Background (Up to 1840) 2) Setting the Stage (1840-1880) 3) The Roots (1880-1915) 4) The Flowering (1915-1940) 5) The Consolidation (1940-1965) 6) Just Yesterday (1965-2000) 7) And Now (2000-present)
"Standardized" Test
1) Tests has norms 2) Uniform Procedures 3) group administered, machine scored, multiple choice
6 Major Forces
1) The Scientific Impulse 2) Concern for the Individual 3) Practical Applications 4) Statistical Methodology 5) Rise of Clinical Psychology 6) Computers
5 Major Categories of Tests
1. Mental Ability 2. Achievement 3. Personality 4. Interests and Attitudes 5. Neuropsychological
5 Additional Test Categories
1. Paper & Pencil vs. Performance Tests 2. Speed vs. Power Tests 3. Individual vs. Group Tests 4. Maximum vs. Typical Performance Tests 5. Norm-referenced vs. Criterion-referenced Tests
Tests Info: Test Publishers
All test publishers have a catalog of their products (e.g., Multidimensional Self-Esteem Inventory MSEI) Most have professional personnel available for consultation Strengths: most important for up-to-date info (notices about upcoming new editions) Weaknesses: obviously not unbiased - want you to spend $
Primary Types (5) of Achievement Tests
Batteries (e.g., Stanford Achievement Test) Single Subject (e.g., Graduate Record Examinations GRE, Subject Test in Psychology) Certification, Licensing (e.g., Examination for Professional Practice in Psychology EPPP) Gov't-Sponsored Programs (e.g., National Assessment of Educational Progress NAEP) - Grades 4, 8, 12 Individual Achievement Tests (e.g., Woodcock-Johnson Tests of Achievement)
Interests and Attitudes
Category for Psychological Testing Aimed at assessing one's interests and attitudes 2 types
Neuropsychological
Category for Psychological Testing designed to yield info about the functioning of the CNS (especially the brain) this type of comprehensive assessment overlaps with other categories of psychological testing but takes a unique perspective.
Personality
Category for Psychological Testing designed to yield information about the human personality 3 categories of tests
Achievement
Category of Psychological Testing assesses a person's level of knowledge or skill in a particular domain 5 primary types
Standard Scores (SS)
Conversion of z-scores into a new system with an arbitrarily chosen M and SD Several versions of SS exist (e.g. T-scores, SATs, IQs) Usually a convenient M & SD KNOW THE FORMULA
Major Force: Concern for the Individual
Differential perspective Nearly always an interest in individuals
Clinical Uses of Tests
Includes counseling, school, and neuropsychology Testing may help to do 3 things: 1) identify the nature and severity of a problem 2) provide suggestions about how to deal with the problem 3) measure progress in dealing with the problem
Subtypes (3) of Mental Ability Tests
Individually administered (e.g., Wechsler Adult Intelligence Scale WAIS) group administered (e.g., Otis-Lennon School Ability Test OLSAT) other abilities (e.g., memory (WMS-IV)
Tests Info: Special Purpose Collections
Info about tests in a narrow range of topics E.g., Handbook of Personality Assessment, Positive Psychological Assessment Strengths: very concentrated coverage Weaknesses: only availalbe in a few areas, often not updated
Construct
Most general level of defining a variable Trait or characteristic we want to measure (e.g. egalitarianism)
Standard Scores Linear vs. Nonlinear
Most will be linear transformations of raw scores Some are derived from nonlinear transformations when the raw scores are not normally distributed so we do an area transformation making normalized standard scores
Paper & Pencil vs. Performance Tests
Multiple choice or true/false questions answered using pencil and paper examinee completes an action such as assembling a product, delivering a speech, conduction an experiment, or leading a group
Raw Data
Numbers resulting from application of the measures - most specific level of a variable. Do statistics to this to find out info about the construct
Categories (3) of Personality Tests
Objective - objectively scored (e.g., Minnesota Multiphasic Personality Inventory MMPI) Projective - the examiner interprets the examinee's responsses to ambiuous, unstructured stimuli/tasts as potential indicators of personality (e.g., Rorschach Inkblot Test) Other Approaches - all other techniques devised to assess personality (e.g., clinical interviews)
Measure
Operational definition, often a test
Major Force: Computers
Originally: just statistical processing Phase 2: computers prepared reports of test scores Phase 3: Test administration in 3 ways 1) Computer-based test admin: puts test questions in a text file 2) computer-adaptive testing: next test item is based on previous responses 3) automated scoring: program simulates human judgement in scoring essays - NOT YET
Tests Info: Professional Journals
Some journals are devoted exclusively to testing and measurement topics: Psychometrika; Educational & Psychological Measurement - Flowering Era Journal of Applied Psych often features testing topics Strengths: provide the latest technical research developments in testing methodology Weaknesses: limited for considering the ordinary use of a test - not practical info on a test
Percentiles Strengths and Weaknesses
Strengths: simple concept, easy to explain, easy to calculate Weaknesses: confusion with %-correct score they represent an ordinal scale of measurement so no absolute differences between examinees can be calculated inequality of units throughout the scale - it's a non-linear transformation because percentiles are distributed flat / bunched up in the middle and spread to extremes
Tests Info: Comprehensive Listings
Summary: Breadth but not depth 1) Tests in Print (TIP) - all English commercially published tests 2) Tests - comprehensive reference for tests 3) Directory of Unpublished Experimental Mental Measures - excludes tests from commercial publishers Strengths: very thorough Weaknesses: just basic info about each tests but would have to order and buy them to get use of one (just a catalog)
Tests Info: Systematic Reviews
Summary: Depth but not breadth Mental Measurement Yearbook (MMY) - new copy every 3-4 years and covers each test in depth. Started in Flowering Period Test Critiques - similar but not as many tests as MMY Strengths: critical evaluations of tests Weaknesses: not available for all tests (or most recent editions), only opinion
Major Force: Scientific Impulse
The need to measure scientifically
z-score
The score resulting from subtracting the mean (M) from a raw score (X), then dividing by the standard deviation (SD) It has M = 0 and SD = 1 They are used to map out the normal curve in terms of areas under the curve
Standardized IQ Scores
These scores fall on a normal curve with M=100 and SD=15
Types (2) of Interests and Attitudes Tests
Vocational Interests - help individuals explore careers relevant tot heir interests (e.g., Strong Interest Inventory) Attitude Scales - measure attitudes toward topics, groups, and practices (e.g. attitudes toward older adults)
Central Tendency (3 kinds)
What? the center around which the raw data tend to cluster Mean (M) Median (Md) Mode (Mo)
Variable
a construct or dimension along which individuals differ/vary
Differential perspective
a general disposition to view human behavior in terms of differences between people rather than in terms of general laws that apply to everyone
Normal ("bell") curve
a unimodal, symmetrical around its central axis shape of distribution. Most naturally occurring phenomena are normally distributed (e.g. intelligence)
Individual vs. Group Tests
administered to one person at a time test administered to many people at the same time
categorical variable
can assume only a finite # of values (e.g., had heart attack: yes or no)
Variability
how much participants' scores differ from one another
Maximum vs. Typical Performance Tests
how well the examinees perform when at their best (achievement & ability tests) someone's typical or normal personality, attitude, interest, etc.
Norm-referenced vs. Criterion-referenced Tests
if a score is interpreted based on how groups of people actually perform on the test (e.g. percentile score) if a score is interpreted in relation to some well-defined external criterion (e.g. 90% correct, 70% correct)
Raw score
immediate result of an individuals responses to a test difficult to interpret itself so we usually norm/standardize the scores
Range
simplest measure of variability: it displays the difference between the highest and lowest value observed on the variable Negative: can be biased due to outliers
Departures from normality
skewness: the degree of symmetry for the right and left sides of the curve Negative skew: long tail on the left Positive skew: long tail on the right
Standard Deviation (SD)
the amount the average participant deviates from the mean of the sample. NEED TO KNOW THE FORMULA
Normed Score
the examinee's raw score is compared with the scores of individuals in the normative (standardization) group
continuous variable
theoretically can take on an infinite # of values (e.g. body weight)
