Final

Three steps to conducting a pilot test *Chapter 12*

*1.* Setting ip the pilot test - sample should be representative of target audience - pilot test setting should mirror planned test setting - participants should understand they are participating in a research study - developers should take steps to maintain confidentiality *2.* Conducting the plot study - depth and breadth should depend on size and complexity of target audience - should gather data on criterion measure and length of time needed to compete the test - may use questionnaires/interviews to gather information about the test *3.* Analyzing the results - may gather quantitative and qualitative data

Steps to gather evidence of validity for structures interview *Chapter 15*

*1.* conduct a formal job analysis or a competency model *2.* correlate a job description *3.* develop questions using the competency model or job analysis for guidance

What are the first *four* steps to the test development process? *Chapter 11*

*1.* defining the testing universe, audience, and purpose *2.* defining a test plan *3.* composing the test items *4.* writing the administration instructions

What are the primary reasons to develop a new test? *Chapter 11*

*1.* meet needs of a special group of test takers *2.* sample behaviors rom a newly defined test domain *3.* improve accuracy of test scores for intended purpose

Phases of developing a survey *Chapter 10*

*PHASE 1: Preparing the survey* - identifying the objectives (e.g., can come from literature reviews, experts, etc) - defining the objectives operationally - constructing a plan *PHASE 2: Constructing the survey* - selecting the type of survey - writing the survey questions - preparing the survey instrument - pre-testing the survey *PHASE 3: Administering the survey* - selecting the appropriate respondents - determining the sample size - distributing the survey *PHASE 4: Coding, entering, and analyzing survey data* - coding the survey questions - entering and verifying data - conducting statistical analysis - interpreting the results *PHASE 5: Presenting the findings* - outlining the report - ordering and determining the contents of a presentation - using slides and handouts

Comparison of *Objective* and *Subjective* formats *Chapter 11*

*SAMPLING* *Objective--* provide opportunity to sample universe *Subjective--* limited to # of questions/ topics to which test taker can respond in one session; validity based on content may suffer *CONSTRUCTION* *Objective--* require much thought and development time *Subjective--* easier to construct and revise; better suited for testing higher order skills *SCORING* *Objective--* simple and can be done by computer *Subjective--* time consuming and relies on judgement *RESPONSE SET* *Objective--* can guess correct response *Subjective--* may bluff or pad answers with superfluous or excessive information; scorers may be influences by irrelevant factors

How are *surveys* different from *psychological tests*? *Chapter 10*

*Surveys* - focus on group outcomes - results are reported at the question level *Psychological tests* - focus on individual outcomes - results reported as overall derived score or scaled scores

what are the 3 types of cultural bias that arise in tests? *Chapter 12*

*construct bias:* when items do not have the same meaning from culture to culture *method bias:* when the mechanics of the test work differently for various cultural groups *differential item functioning:* when test takers from different cultures have the same ability level on the test construct, but the item or tests yields very different scores for the two cultures

Defining the testing universe (Step 1 to test development) *Chapter 11*

*defining the testing universe* - prepare working definition of construct - if abstract constructs, review psychological literature to locate studies that explain construct and find tests that measure construct *defining the target audience* - list characteristics of the persons who will take the test - consider appropriate reading level, possible disabilities, likelihood of answering honestly *defining the test purpose* - identify exactly what the test will measure - how test scores will be used (e.g., normative or criterion approach)

Conducting the validation study *Chapter 12*

*defining unfair discrimination:* explore whether test unfairly discriminates among groups *testing for validity equivalence:* explores whether two forms of a test are comparable - the tests must yield identical scores - the tests should yield the same distributions of scores - the individual ink-ordering of scores should be identical - the score from each test must correlate equally well with external criteria

How is constructing a good survey like a science and an art? *Chapter 10*

*like a science:* - follows a scientific process *like an art:* - knowing how much rigor to put into the design and writing good questions take years of practice constructing reliable and valid surveys takes time

Types of integrity testing *Chapter 15*

*polygraph:* best known physiological measure associated with evaluating how truthfully an individual responds to questioning *paper-and-pencil:* alternative to physiological tests for screening applicants *overt tests:* ask test takers to provide information about their past behavior or to respond to hypothetical situations *personality-oriented tests:* purport to measure characteristics that are predictive of honest behavior and positive organizational citizenship using items that relate to the big Five personality factors

What are the types of surveys? *Chapter 10*

*self-administered* - mail - individually administered *personal interviews* - face to face - telephone *structured record reviews*

Different types of rating scales *Chapter 10*

*simple qualitative* - how would you rattle person in this area (e.g., from 'poor' to 'excellent') *extent* - to what extent does the statement describe the person? (e.g., from 'very little' extent to 'very great') *frequency* - how often does the person demonstrate effective use of the behavior or skill (e.g., from 'almost never' to 'almost always') *developmental (strength/development needed)* - how would you rate the person's capabilities in this area? (e.g., from 'significant development' need to 'exemplary-- a role model') *comparison (with others)* - compared with others with whom i have worked, this person is _____ in this area (e.g., from 'significantly below average' to 'the best I've ever worked with') *performance* - how would you rate his or her performance in this area? (e.g., from 'does not meet expectations' to 'far exceeds expectations') - this theme would make sense for performance appraisal purposes, but probably not for development purposes where peers and subordinates also provide ratings

Helmstadter's *six methods for acquiring knowledge* *Chapter 10*

*tenacity:* we acquire information based on superstition or habit, leading us to continue believing something we have always believed - ex: we come to believe that walking under a ladder will bring us back luck *intuition:* we acquire information without any resining or inferring - ex: we come to believe that one brand of aspirin is more effective than another brand just because we believe it is so *authority:* we acquire information from a highly respected source - ex: we come to believe that one brand of aspirin is better than another brand because a pharmacists tells us so *rationalism:* we acquire information through reasoning - ex: we come to believe that one brand of aspirin reduces headaches more effectively than another brand because the first brand includes ingredient that are similar to another medicine we use that really works for us *empiricism:* we acquire information through personal experience - ex: we come to believe that one brand of aspirin reduces headaches more effectively because we have used it and it works for us *scientific method:* we acquire information by testing ideas and beliefs according to a specific testing procedure that can be observed objectively. - this method is *WITHOUT* personal beliefs, perceptions, biases, values, attitudes, and emotions - ex: we come to believe that one brand of aspirin is more effective at reducing headaches because systematic research has accumulated evidence of its effectiveness

What are things to consider when preparing the survey instrument? *Chapter 10*

*title and seal* - title (large font, centered @ top) - title indicates what questionnaire is about - identify target audience - seal of company logo (for credibility) *Appeal and instructions* - state the purpose of the survey - explain who is sponsoring or conducting the survey - explain why it is important for respondents to complete the survey to the best of their ability - explain how to complete the form - assure respondents that their answers are confidential and will be reported in group format only - thank respondents for their cooperation *Headings and subheadings* - helps guide respondents through the survey *Transitions* - introduced to each new section with a topic heading that informs them about the next section of the survey *Response instructions* - tell respondents how to answer appropriately or that lead them to another part of the questionnaire (short directions) *Bold typeface* - to emphasize key points in directions or questions *Justification of response spaces* - a line, a checkbox to the left of response choices and columns to the right *Shading* - shade where you DO NOT want respondents to write anything (leave blank) - can also separate rows of text on a horizontal layout togged respondents across a page of text *White space* - leave margins of 1 inch - ample space to respond to the questions *Printing* - surveys fewer than three pages are usually printed on one sheet of paper (both sides) - if longer than 2 pages, staple in top left corner - really long surveys may be spiral bound of in booklet format *Font* - attractive, business-like, and readable

Types of *probability sampling* *Chapter 10*

- Simple Random Sampling (SRS) - Systematic sampling - Stratified sampling - Cluster sampling

Questions to consider when specifying administration formats *Chapter 11*

- Will the test be administered in writing, orally, or by computer?- How much time will the test takers have to complete the test? - Will the test be administered to groups or individuals? - Will the test be scores by a test publisher, the test administrator, or the test takers - What type of data is the test expected to yield?

What *two* elements do test formats provide? *Chapter 11*

- a *stimulus* to which the test taker responds - a *mechanism* for response

What instructions need to come with a test? *Chapter 11*

- administrator instructions - test taker instructions - scoring instructions

Legal constraints for preemployment testing *Chapter 15*

- any process that is used to make hiring decisions is defined as a "test" *Uniform Guidelines on Employee Selection Procedures (1978)* - procedures organizations should follow to enhance fairness and legal defensibility of employment practices/tests - apply to all employment screening devices-- including application blanks (forms), reference checks, and letters of reference - all tests should be job related and based on job analysis - employers should use only tests for which there is evidence of validity - organizations should maintain records regarding the race, sex, and ethnic group membership of the applicant pool and the final group of persons hired for the job - when a test results in adverse impact-- exclusion of a disproportionate number of persons in a group protected by federal law, employer may need to find an alternative method for assessing job candidates

What are the best practices for test taker instructions? *Chapter 11*

- are simple, concise, and written at a low reading level - are delivered by test administrator from a script - appear in writing at beginning of test - include encouragement for answering accurately and honestly - include "context" for answering (e.g., "when responding, think about...")

Guidelines for writing effective *multiple choice* items *Chapter 11*

- avoid negative stems and responses - make all responses similar in length and detail - make sure the item has only one best answer - avoid words such as "alway" and "never" - avoid overlapping responses - avoid using inclusive distractors - use random assignment to position correct response

methods for demonstrating evidence of internal validity *Chapter 10*

- content - criterion-related - construct - face validity

Types of *nonprobability sampling* *Chapter 10*

- convenience sampling

Benefits of interviews *Chapter 15*

- do serve useful purposes other than prediction of job performance - provides candidates with useful information regarding the organization - can set expectations about what the job will entail - when candidates have realistic expectations, they are likely to remain on the job longer

What are the types of *Subjective test items*? *Chapter 11*

- essay questions - interview questions - projective techniques - sentence completion

What are the methods for collecting information? *Chapter 10*

- experimental research techniques - descriptive or correlation-based research techniques

What are the general characteristics of a "good" survey? *Chapter 10*

- have specific and measurable objectives - contain straightforward questions that can be understood similarly by most people - have been pretested to ensure that there are no unclear questions or incorrect skip patterns - have been administered to an adequate sample of respondents so that the results are reflective of the population of interest - include the appropriate analysis to obtain the objectives - include an accurate reporting of results (both verbal and written) - have evidence of reliability and validity *using a scientific approach to survey research increases the chances that a survey will have these features*

Guidelines for writing effective test items *Chapter 11*

- identify item topics by consulting the test plan - ensure each item is based on an important objective - write items that assess information or skills drawn only from the testing universe - write each item in a clear and direct manner - use vocabulary and language appropriate for target audience - avoid slang or colloquial language - make all items independent - ask someone else to review items to reduce ambiguity and inaccuracies

Different components of quantitative item analysis *Chapter 12*

- item difficulty - item discrimination - inter-item correlations - item-total correlations - item-criterion correlations - item characteristic curve - item bias

What should be included in test taker instructions? *Chapter 11*

- know where to respond - know how to respond - have specific directions for each type of item

what are the types of rating errors made in performance appraisals? *Chapter 15*

- leniency errors - severity errors - central tendency errors - helo effect

What are the popular survey item types? *Chapter 10*

- open-ended - closed-ended - yes/no questions - fill in the blank - implied no choice - single-item choice - enfolded - free choice - multiple choice - ranking - rating - Guttman format - likert and other intensity scale formats - semantic differential - paired comparisons and constant-referent comparisons

When conducting statistical analysis and interpreting the results, it is important to... *Chapter 10*

- plan analysis at time of survey construction *important statistics* - response rate - reliability - sampling error - univariate analyses' - bivariate and multivariate analyses

What are some potential sources of measurement error? *Chapter 10*

- poorly worded questions - ambiguous terms - inappropriate reading level - unclear directions, incorrect skip patterns

Pre-employment testing *Chapter 15*

- psychological assessment often provided basis for hiring employees - most popular assessment method is the employment interview - sometimes supplemented by psychological tests that measure performance, skills abilities, or personality characteristics - drug and integrity testing have become acceptable methods for screening candidates *see photo for myths*

what should be included in the test manual? *Chapter 12*

- rationale for constructing the test - history of the development process - results of reliability and validation studies - appropriate target audience - instructions for administering and scoring test - norms and information on interpreting individual scores

Guidelines for writing effective "essay and interview" questions *Chapter 11*

- use essay items appropriately - consider the time necessary for response - prepare an answer key - score essays anonymously - use multiple independent scores

What should be included in administrator instructions? *Chapter 11*

- whether the test should be administered in a group and/or individually - specific requirements for the test administration location, such as privacy, quiet, and comfortable chairs, tables, or desks - required equipment such as No. 2 pencils, a computer with a DVD or CD-ROM die, and/or internet access - time limitations or the approximate time for completion the test when there are no time limits - a script for the administrator to read to test takers, including answers to questions that test takers are likely to ask - credentials or training required for the test administrator

What are the best practices for scoring instructions? *Chapter 11*

- written to ensure all persons follow same scoring process - explain how scores relate to construct measured (what does high/low score mean)

A short history of employment testing *Chapter 15*

1915-- *"The Scientific Selection of Salesman" by Walter Dill Scott* - proposed that employers use group tests for personnel selection - inspired interest and debate among academic psychologists - influence in employment testing has had lasting effect WWI-- US Army alpha/beta tests - lead to development of tests for employment - advantages of testing investigated - methods for measuring job performance studies - methods proposed for placing workers in industrial jobs based on skills and qualifications - interest inventories and tests of mental ability developed WWII-- widespread use of psychological tests for employment - Army General Classification Test developed to place US Army recruits - Office of Strategic explored assessment-center method - Tests developed to measure morale and attitudes - large companies and federal government began using psychological tests for selection, placement, attitude assessment, and performance appraisal - civil rights movement in the US drew attention to issues of test validity and fairness - passage of Title VII of the Civil Rights Act (1964) stimulated interest in fair employment practices - Federal government's Uniform Guidelines on Employee Selection Procedures of 1978 developed - Today, tests used for many different reasons by organizations

anchor points *Chapter 10*

5 -- exceptional strength (anchor point) 4 -- 3 -- competent (anchor point) 2 -- 1 -- weak (anchor point)

intercept bias *Chapter 12*

A type of measurement bias in which the intercept of the regression line that is used to predict a criterion of interest is not the same for all demographic groups used in the regression (e.g., men and women) when a common regression line is used for prediction when intercept bias is present, the performance of one group will be over predicted and those of the other will be under predicted

true/false *Chapter 11*

Item on a test that asks: "Is this statement true or false?" - 50% chance that if you guess, you'll guess correctly

cross-validation *Chapter 12*

The final analysis that is conducted on a test to a sample of test takers that are representative of the target audience - for tests that rely on criterion-related evidence of validity to make predictions - this second administration can be expected to yield lower correlations with criterion measures

portfolio *Chapter 11*

a collection of work products that a person gathers over time to demonstrate his or her skills and abilities in a particular area

protected class *Chapter 15*

a group protected by federal law

Database *Chapter 10*

a matrix in the form of a spreadsheet that shows the responses given by each participant (row) and for each question (column) in the survey

forced ranking *Chapter 15*

a method of performance appraisal - to rank employees, the supervisor must decide who is the "best" employee, the "next best" employee, and so on based on a predetermined dimensions or criteria

Stratified random sampling *Chapter 10*

a population is divided into subgroups or strata (e.g., gender, age, socioeconomic status) - a random sample is selected from each stratum

Scientific method *Chapter 10*

a process for generating a body o knowledge that allows us to obtain more accurate information through systematic and objective observations STEPS: *1.* identifying a problem or an issue and forming a hypothesis *2.* Designing a study to explore the problem or issue and test the hypothesis *3.* Conducting the study *4.* Analyzing and interpreting the data *5.* Communicating the research results

Double-barreled question *Chapter 10*

a question that is actually asking two or more questions in one - these are not good for one survey item - survey items should only ever ask one single question

pilot test *Chapter 11*

a scientific evaluation of the test's performance

pilot test *Chapter 12*

a scientific investigation of evidence that suggests that the test scores are reliable and valid for their specified purpose - involves administering the test to a sample of the test's target audience and analyzing the data obtained from the pilot test

Sampling error *Chapter 10*

a statistic that reflects how much error can be attributed to the lack of representation of the target population by the sample of respondents chosen

meta-analysis *Chapter 15*

a statistical technique that accumulates the results of multiple studies on comparable or similar constructs into a single result

Decennial research survey *Chapter 10*

a survey that is administered by the US Census Bureau every 10 years, primarily to determine the population of the United States

job analysis *Chapter 15*

a systematic assessment method for answering the question "What do you do?" in organizations - provide a systematic and detailed procedure for documenting the activities of the job *typical job analyses include:* - functions: accomplish objectives of the job - tasks: accomplish job functions - knowledge: a body of related information that the worker needs to know to perform job tasks - skills: a group of observable and measurable behaviors acquired through training that the worker needs to perform a vary of job tasks - abilities: a physical or mental competency based on innate characteristics (generally not trained) that the worker needs to perform job tasks - other characteristics: interests or personality traits that the worker needs to perform in order to cope with the job environment

Nonprobability sampling *Chapter 10*

a type of sampling in which not everyone has an equal chance of being selected from the population

Probability sampling *Chapter 10*

a type of sampling that uses statistics to ensure that a sample is representative of the population

Individually administered surveys *Chapter 10*

administered by a facilitator in person for respondents to complete in the presence of the facilitator

Population *Chapter 10*

all members of the target audience

Pretesting *Chapter 10*

allows you to: *1.* identify sources of non sampling measurement errors *2.* examine the effectiveness of revisions to a question or an entire survey *3.* examine the effect of alternative versions of a question or survey *4.* assess the final version of a survey for respondent understanding, time to complete, and ease of completion *5.* obtain data from the survey and make changes to the format that might make data entry or analysis more efficient

Field test *Chapter 10*

an administration of the survey to a large representative group of individuals to determine any problems with administration, item interpretation, and so on

Face-to-face surveys *Chapter 10*

an interviewer asks a series of questions in respondents' homes, a public place, or the researchers' office

Telephone surveys *Chapter 10*

an interviewer calls respondents and asks them questions over the phone

behaviorally anchored rating scales (BARSs) *Chapter 15*

another rating method that uses on-the-job behaviors as anchors for rating scales that represent job dimensions

differential item functioning (DIF) *Chapter 12*

arises when test takers from different cultures have the same ability level on the test construct, but the item or test yields very different scores for the two cultures - provides an indication of unexpected behavior by item on a test - an item does not display DIF if people from different groups have a different probability to give a certain response; it displays DIF if people from different groups of same underlying true ability have a different probability to give a certain response - more precisely, an item displays DIF when the difficult level (b), the discrimination (a) or the lower asymptotes (c)-- estimated by item response theory (IRT) -- of an item differs across groups

behavioral interview *Chapter 15*

ask respondents to describe a situation and give detailed information about how they performed in that situation

cognitive tests *Chapter 15*

assessments that measure the test taker's mental capabilities, such as general mental ability tests, intelligence tests, and academic tests - used in preemployment testing when cognitive skills critical to success on the job - high validity (.42) with job performance criteria and even higher validity for predicting training criteria - most accurate for "thinking" jobs (r=0.53) such as manager and salesperson

testing universe *Chapter 11*

body of knowledge or behaviors that test represents

Focus group *Chapter 10*

bring together people who are similar to the target respondents in order to discuss issues related to the survey

item-response theory (IRT) *Chapter 12*

can provide estimates of the ability of test takers that is independent of the difficulty of the items presented as well as estimates of item difficulty and discrimination that are independent of the ability of the test takers - uses item characteristic curves We can look at the the number of questions each person got right/wrong on a test - "Item response theory takes into account the number of questions answered correctly and the difficulty of the question" --> this is used to further develop the test

Survey research firms *Chapter 10*

companies that specialize in the construction and administration of surveys and analysis of survey data

discrimination index *Chapter 12*

compares the performance of those who obtained very high test scores (the upper group [U]) with the performance of those who obtained very low test scores (the lower group [L]) on each item high positive numbers = good item discrimination

Univariate analysis *Chapter 10*

computation of statistics that summarize individual question responses - includes frequency counts, percentages, means, modes, and medians

multiple choice *Chapter 11*

consists of a question or partial sentence, called a stem, followed by a number of responses (usually 4 or 5), of which only one is correct - most common item format

item-criterion correlations *Chapter 12*

correlate item responses with a criterion measure such as a measure of job performance for pre-employment tests or a diagnostic measure for clinical tests

cut scores *Chapter 12*

decision points for dividing test scores into pass/fail groupings *identifying cute scores:* - panel of experts provide opinion/rating about number of test items a minimally qualifies person is likely to answer correctly - correlate the test and an outside criterion to predict test score a person who performs a minimum level is likely to gain

Are most surveys experimental or descriptive? *Chapter 10*

descriptive

choosing final items *Chapter 12*

developers write mire items than are needed - use quantitative and qualitative analyses of the test to choose those items that together provide the most information about the construct being measured

item-total correlations *Chapter 12*

displays the correlation of each item to all test items as a whole - measures overall test consistency and item's ability to discriminate high from low scoring individuals - each item is ideally correlated with all items as a whole (measuring same construct) *what to look for:* - positive item-total correlations

interitem correlation matrix *Chapter 12*

displays the correlation of each item with every other item - ideally, *each item should be correlated with every other item measuring the same construct* and should not be correlated with items measuring a different construct - measures internal consistency

subjective test format *Chapter 11*

do not have a single response that is designated as "correct" - interpretation of the response as correct or providing evidence of a specific construct is left to the judgement of the person who scores or interprets the test taker's response - ex: essay question

360 feedback *Chapter 15*

employees receive ratings from those they work with on a day-to-day basis, such as their ,managers, peers, direct reports, and customers - employees also rate themselves - ratings are aggregated provides rich feedback for the person being rated and may be perceived by some as fairer because misperceptions or rating errors by one person can be balanced by the opinions of others

personality traits *Chapter 15*

enduring traits that predispose persons to behave in certain ways - ex: contentiousness, extraversion, agreeableness

Nonsampling measurement errors *Chapter 10*

errors associated with the design and administration of the survey

Random errors *Chapter 10*

errors in survey research that are *unpredictable*

Measurement errors *Chapter 10*

errors in survey research that are associated with how a survey performs in a particular population

behavioral checklist *Chapter 15*

evaluation of performance by rating how frequently the employee performs important behaviors on the job

item analysis *Chapter 12*

evaluation of the performance of each test item

Simple random sampling *Chapter 10*

every member of a population has an equal chance of being chosen as a member of the sample *does not guarantee the sample will be representative of the population*

Systematic sampling *Chapter 10*

every n'th (e.g., every fifth) person in a population is chosen as a member of the sample

synthetic validity *Chapter 15*

existing evidence of validity is generalized from one group of jobs to another if it can be shown that the same job components are present in both - AKA job component validity

external validity *Chapter 10*

extent to which the survey results obtained from a sample are generalizable to a larger population

structured behavioral interview *Chapter 15*

focuses on past behaviors rather than attitudes or opinions

performance appraisal *Chapter 15*

formal evaluations of employees job performance - usually each employees supervisor or manager completes a performance appraisal form that requires assigning numerical values to employee's performance *ranking employees:* compare the performance of one employee with the performance of other employees *rating employees:* rate employees performance using a scale that specifies job dimensions or job behaviors

Structured observations *Chapter 10*

forms that guide an observer in collecting behavioral information (e.g., using a form to document the play behaviors of children on the play ground)

Structured record reviews *Chapter 10*

forms that guide data collection from existing records (e.g., using a form to collect information from personnel files)

interview questions *Chapter 11*

general in scope, and the interviewer decides what is a "good" or "poor" answer - the interviewee has a wide range of responses - introduces many opportunities for causing inaccurate judgement

Characteristics of good survey questions *Chapter 10*

good survey questions are: - purposeful and straightforward - are unambiguous - are written in correct syntax - include appropriate rating scale and response options - ask one and only one question - are written at comfortable reading level (some recommend 6th grade level)

norms *Chapter 12*

group of scores that indicate the average performance of a group and the distribution of scores above and below this average *developing norms:* - administer test to a representative sample of target audience - may use some of the validation data

objective test format *Chapter 11*

has one response that is designated as "correct" or that provides evidence of a specific construct - ex: multiple choice, true/false

Descriptive research techniques *Chapter 10*

help us describe a situation or a phenomenon - most surveys are descriptive

Experimental research techniques *Chapter 10*

help us determine cause and effect - generally, surveys are not commonly used to determine cause and effect

item nonresponse rate *Chapter 10*

how often an item or question was not answered

Homogeneity of the population *Chapter 10*

how similar the people in your population are to one another

distracters *Chapter 11*

incorrect responses found in multiple choice questions - designed to appear correct to someone who does not know the correct answer

Experts *Chapter 10*

individuals who are knowledgeable about the survey topic or who will be affected by the survey's outcomes

Surveys *Chapter 10*

instruments used to collect important information from individuals enables us to describe and compare people's: - *attitudes* (how they feel) - *knowledge* (what they know) - *behaviors* (what they do)

assessment centers *Chapter 15*

large-scale simulations of the job that require candidates to solve typical job problems by role-playing or by demonstrating proficiency at job functions such as making presentations and fulfilling administrative duties

polygraph *Chapter 15*

lie detector best-known physiological measure associated with evaluating how truthfully an individual responds to questioning *theory:* - when an individual gives an untruthful response, he or she exhibits increases in skin resistance, pulse or heart rate, and respiration *problems with this* - physiological responses may increase for other reasons...discomfort, nervousness, etc. - high rate of false positives (i.e., type I errors)

Americans with Disabilities (ADA) *Chapter 15*

limits an employer's ability to administer medical tests that might unfairly screen out workers with disabilities. Employers can require a medical examination only after a job offer has been made, for example - the ADA cloaks the results of a medical examination with certain privacy protections. Data gathered in medical examinations examinations must be kept in a separate file available only to those with a demonstrable "need to know" - once an employee is already on the job, an employer's right to conduct a medical examination is usually limited to so-called "fitness for duty" situations. When an employee has exhibited objective indications that he or she is physically or mentally unfit to perform the essential functions of the job, an employer may ask about the employee's condition or request that the employee take a medical examination

Mail surveys *Chapter 10*

mailed to respondents with instructions for completing and returning them

Personality inventories *Chapter 15*

measure cognitive constructs referred to as personality traits

forced distribution *Chapter 15*

method of performance appraisal which requires the supervisor to assign a certain number of employee's to each performance category, such as "poor," "below average," "average," "above average, " and "outstanding," so that appraisals are distributed in a way that resembles a normal curve

false positives *Chapter 15*

mistakenly classifying innocent test takers as guilty

anchors *Chapter 15*

numbers or words that allows people to be ranked in categories

transportability *Chapter 15*

occurs when a case is made that because a test has been shown to be valid for a particular use, it can be presumed to also be valid for another, similar use

halo effect *Chapter 15*

occurs when raters let their judgement on one dimension influence judgements on other dimensions

predictive bias *Chapter 12*

occurs when the predictions made about a criterion score based on a test score are different for subsets of test takers (e.g., minority vs. majority, men vs. women) - indicated when a test has different validity coefficients for different groups

What are the methods for pre-testing? *Chapter 10*

one-on-one interviews - *concurrent think-aloud:* respondents describe their thoughts as they think about and answer the survey questions - *retrospective think-aloud:* typically, these are one-n-one interviews during which the interviewer asks respondents how they went about their answers after they complete the survey - *paraphrasing:* the interviewer asks respondents to repeat questions in their own words - *confidence ratings:* after answering survey questions, respondents rate how confident they are that their responses are correct Respondent focus groups behavior coding of responding-interviewer interactions interviewer and respondent debriefings split-sample tests item nonresponse

stem *Chapter 11*

partial sentence often found in multiple choice questions - usually complete the stem with the correct response)

response sets *Chapter 11*

patterns of responding that result in false or misleading information (AKA response style)

Survey researchers *Chapter 10*

people who design and conduct surveys and analyze their results

categorical model of scoring *Chapter 11*

places test takers in a particular group or class - test taker displays a pattern of responses (such as those associated with a specific trait)

sentence completion *Chapter 11*

presents an incomplete sentence such as "I feel happiest when I am ________"

situational judgement test *Chapter 15*

presents job candidates with written or video-based scenarios that pose work-related dilemmas along with four or fine possible courses of action that the candidates might take in response - task is to identify the most effective action in each situation the closer the correspondence between the test takers' responses and the subject matter experts' opinions, the higher the score

five-factor model *Chapter 15*

proposes that there are five central personality dimensions: - extroversion (AKA surgency), emotional stability, agreeableness, conscientiousness and openness to experience

Multivariate analysis *Chapter 10*

provide information on three or more variables or groups - includes calculating correlation coefficients, cross-tabulations, chi-square comparisons, t-tests, or analyses of variance

Bivariate analysis *Chapter 10*

provide information on two variables or groups

graphic rating scales *Chapter 15*

provide visual indicators to help the manager rate employees

subtle questions *Chapter 12*

questions that have no apparent relation to the criterion

how do you investigate item bias? *Chapter 12*

range from comparing score for passing rates of various groups to comparing group performance in terms of an external criterion - compare scores or pass rates of groups - compute item characteristics by group using item characteristics curve--> by plotting the curves on a graph, differences in difficulty and discriminating can be detected readily

test format *Chapter 11*

refers o the type of questions that the test will contain

faking *Chapter 11*

refers to the inclination of some test takers to try to answer items in a way that will cause a desired outcome or diagnosis

high-fidelity tests *Chapter 15*

replicate job settings as realistically as possible - test takers use the same equipment that is used on the job and complete actual job tasks

Sample *Chapter 10*

representative subset of the population

performance assessments *Chapter 11*

requires test takers to directly demonstrate their skills and abilities to perform a group of complex behaviors and tasks

forced-choice *Chapter 11*

requires the test taker to choose one of two more words or phrases that appear unrelated but are equally acceptable - sometimes respondents find this aggravating because they want to respond "it depends"

cognitive aspects of answering questions *Chapter 10*

research suggests that when people answer survey questions, they go through at least four stages: *Comprehension* - must understand each word in the question/ what the entire question is asking - must have long enough attention spans to pay attention to the question - must have a certain level of language ability so that they can understand the vocabulary of the entire question - must have the appropriate general knowledge so that they can understand certain concepts contained in the question *Retrieval* - must search memory for the appropriate answer - must recognize that most appropriate answer - must search memory thoroughly for the correct answer - When the appropriate cues or hints are available, respondents will probably find the answer. When inappropriate cues are available, respondents will take bite and pieces of the information and reconstruct an answer to fit what is most likely to be the answer *Judgement* - must judge or decide whether the information meets the objectives of the question *Response communication* - must communicate the answers to the question - must match their answers to the available choices -must construct understandable responses - respondents will evaluate whether the answers to the question meet their own personal motives and objectives

qualitative analysis *Chapter 12*

researchers ask test takers how they viewed the test itself and how they answered the test questions - also how changes in test content or administration instructions would improve the accuracy of the test results *questionnaire for test takers:* provide information on how test takers received the test and how changes in the text content or administration instructions might improve accuracy of results *expert panels:* those knowledgeable about the test's content or about testing in general provide their opinions on possible sources of error or bias

random responding *Chapter 11*

responding to items in a random fashion by marking answers without reading or considering them

leniency errors *Chapter 15*

result when raters give all employees better ratings than they deserve

severity errors *Chapter 15*

result when raters give all employees worse ratings than they deserve

central tendency errors *Chapter 15*

results when raters use only the middle of the rating scale and ignore the highest and lowest scale categories

simulation *Chapter 11*

similar to a performance assessment in that it requires test takers to demonstrate their skills and abilities to perform

low-fidelity tests *Chapter 15*

simulate the task using a written, verbal, or visual description - test taker may respond by answering an open-ended or multiple-choice question

work samples *Chapter 15*

smaller scale assessments in which candidates complete a job-related task such as building a sawhorse or designing a doghouse

Operational definitions *Chapter 10*

specific behaviors that represent the purpose - *measurable* definition of the construct

test plan *Chapter 11*

specifies the characteristics of the test, including an operational definition of the construct and the content to be measured (the testing universe), the format for the questions, and the administration and scoring of the test

structured interviews *Chapter 15*

standardized, and the same questions are asked to each job candidate

quantitative item analysis *Chapter 12*

statistical analyses of the responses test takers gave to individual items to examine test item performance and identify which to keep, modify, or delete

subgroup norms *Chapter 12*

statistics that describe subgroups of the target audience - for demographic characteristics such as race, sex, and age, or for locations, such as regions and states - large sample needed for each subgroup

test items *Chapter 11*

stimuli that are presented on tests in the form of statements, pictures, pr incomplete sentences - questions on the tests)

Personal interviews *Chapter 10*

surveys that involve direct contact between survey researcher and the respondents in person or by phone

Literature reviews *Chapter 10*

systematic examinations of published and unpublished reports on a topic

projective tests *Chapter 11*

test takers are said to project their thoughts onto ambiguous stimuli - ex: The Thematic Apperception Test - subjective test format because the stimuli for these tests are ambiguous pictures

empirically based tests *Chapter 12*

tests that are designed so that test scores can be used to sort individuals into two or more categories based on their scores on the criterion measure - these are empirically based tests because the decision to place an individual in a category is based solely on the quantitative relationship between the predictor (test score) and the criterion (possible categories)

What does simple random sampling not guarantee? *Chapter 10*

that the sample will be representative of the population

Survey software and survey researchers choice in deciding software's *Chapter 10*

the choice of software depends on a person or company's specific needs, desires, and constraints - while one individual may have an immediate need to design and implement only one, relatively simple survey, another may need to design and implement more complex surveys on an ongoing basis *popular survey software's* - Surveymonkey - Qualtrics

testing environment *Chapter 11*

the circumstances under which the test is administered

acculturation *Chapter 12*

the degree to which an immigrant or a minority member has adapted to a country's mainstream culture

survey validity *Chapter 10*

the degree to which the survey reflects or assesses the concept that a researcher is attempting to measure

traditional interview *Chapter 15*

the interviewer pursues different areas of inquiry with each job candidate

item characteristic curves (ICC's) *Chapter 12*

the line that results when we graph the probability of answering an item correctly with the level of ability on the construct being measured - provides a picture of the item's difficulty and how well it discriminates high performers from low performers "If the item has different ICCs for different groups, it is measuring different characteristics for the two groups. In such cases, the item is not working properly for at least one group. On the other hand, if the ICCs look the same, the item might be working appropriately for all groups."

cumulative model of scoring *Chapter 11*

the more the test taker responds in a particular fashion (with either "correct" answers or ones that are consistent with a particular attribute), the more the test taker exhibits the attribute being measured - most common method for determining an individuals final test score - receive points for correct answers (such as a multiple choice test)

Response rate *Chapter 10*

the number of individuals who responded to the survey divided by the total number of individuals to whom the survey was sent

Sample size *Chapter 10*

the number of people needed to represent the target population accurately

item difficulty *Chapter 12*

the percentage of test takers who respond correctly formula: - calculates how difficult an item is - need variability in test scores to make comparisons want to avoid items everyone answers right or wrong norm-referenced tests involve comparing these scores of individuals

Survey objectives *Chapter 10*

the purpose of the survey and what the survey will measure

phi coefficient *Chapter 12*

the result of correlating two dichotomous (having only two values) variables - interpreted exactly like Pearson product--moment correlation coefficients

Convenience sampling *Chapter 10*

the survey researcher uses any available group of participants to represent the population

social desirability *Chapter 11*

the tendency for some test takers to provide or choose answers that are socially acceptable or that present themselves in a favorable light

acquiescence *Chapter 11*

the tendency ti agree with any ideas or behaviors presented

generalizable *Chapter 12*

the test can be expected to produce similar results even though it has been administered in different locations

ipsative model of scoring *Chapter 11*

the test taker is usually presented with two to four statements in a forced-choice format - test taker is required to indicate which of the items is "most like me" and which of the items is "least like me"

Self-administered surveys *Chapter 10*

those that individuals complete themselves

who should rate people using performance appraisals? *Chapter 15*

traditionally evaluated by manager/supervisor more companies using 360 feedback for appraisal and development - rated b those you work with on a day to day basis - includes self, manager, peers, direct reports, customers

Cluster sampling *Chapter 10*

used when it is not feasible to list all of the individuals who belong to a particular population and is a method often used with surveys that have large target populations - clusters (e.g., regions of the country, states, high schools) are selected and participants are selected from each cluster

projective techniques *Chapter 11*

uses highly ambiguous stimulus to elicit an unstructured response from the test taker - the test taker "projects" his or her perspective and perceptions onto a neutral stimulus

essay questions *Chapter 11*

usually general in scope and require lengthy written responses by testing takers - provide a freedom of response that facilitates assessing higher cognitive behaviors such as analysis, synthesis, and evaluation - introduces many opportunities for causing inaccurate judgement

single-group validity *Chapter 12*

when a test is valid for one group but not for another group (e.g., valid for whites by not for blacks)

differential validity *Chapter 12*

when a test yields significantly different validity coefficients for subgroups

item bias *Chapter 12*

when an item is easier for one group than for another group

measurement bias *Chapter 12*

when the scores on a test taken by different subgroups in the population (e.g., men, women) need to be interpreted differently because some characteristic of the test not related to the construct being measured

slope bias *Chapter 12*

when the slopes of the separate regression lines that relate the predictor to the criterion are not the same for one group as another results in differential validity for the groups being predicted - *differential validity:* "differential validity is the difference between the validity coefficients that exists between two groups" - "an aptitude exam taken by White Americans may result in a correlation of coefficient of 0.59, whereas the same exam taken by Hispanics may results in a coefficient of 0.40. This difference in validity coefficients shows that the test was more biased towards the White Americans" type of predictive bias