Psych testing ch 6

Although it is not always necessary to gather all evidences of validity for a single test, researchers try to gather more than one type of evidence.

While it's acceptable to rely on one strategy at first, the more evidence you can gather to support the validity of a test, the better.

AERA and Standards view validity as a

unitary or single concept.

4. Establishing a test format:

will it be a written test, a paper-and-pencil test in which a test taker must answer a series of questions; a computerized test; or a practical test, which requires a test taker to actively demonstrate skills in specific situations? If it is a written test, the test developer must also decide what types of questions (multiple choice, true/false, matching, and so on) to use.

In recent years an increasingly common approach for gathering content-based evidence of validity for employee selection tools has been a less intensive process called

competency modeling

Construct validity

Evidence based on internal structure - It focuses on whether the conceptual framework used in test development could be demonstrated using appropriate analytical techniques. also used to determine whether certain items on the test are more difficult for some groups of people than for others,

Criterion-related validity

Evidence based on relations with other variables - also another part of construct validity, this source of validity evidence typically involves correlating test scores with other measures to determine whether those scores are related to other measures to which we would expect them to relate. Likewise, we would want to know that the test scores are not related to other measures to which we would not expect them to relate

content validity

Evidence based on test content - this source of validity evidence involves logically examining and evaluating the content of a test (including the test questions, format, wording, and tasks required of test takers) to determine the extent to which the content is representative of the concepts that the test is designed to measure without either underrepresenting those concepts or including elements that are irrelevant to their measurement. .

Fundamental Interpersonal Relations Orientation-Behavior (FIRO-B)

To help people improve their interactions with others. The test is designed to measure needs in three areas—inclusion, control, and affection—believed to relate to three fundamental dimensions of interpersonal relationships. Test takers are asked to respond to behavioral statements that are expected to be related to each of these needs.

Concrete attributes

are attributes that can be clearly described in terms of observable and measurable behaviors. such as knowledge of psychological testing and the ability to play the piano.

Abstract attributes

are those that are more difficult to describe in terms of behaviors because people may disagree on what these behaviors represent. - Other tests measure abstract attributes such as personality, intelligence, creativity, and aggressiveness.

criterion-related validity is more appropriate for what type of tests

employment tests, college admissions tests, and diagnostic clinical tests.

Evidence of validity based on test content involves

examining the questions or behaviors required on the test and making a judgment regarding the degree to which the test provides an adequate sample of the construct being measured. Paper and pencil, employment, classroom tests, etc

Gathering evidence of validity based on relationships with other vairbales is ________ when the attribute being measured, such as personality or intelligence, is abstract because such attributes need to be carefully defined and linked to observable behaviors.

more difficult (but not necessarily less appropriate)

Demonstrating evidence of validity based on test content

1. during test development 2. defining the test universe 3. developing the test specifications 4. establishing a test format 5. constructing test questions

5 sources of validity

1. evidence based on test content 2. evidence based on response processes 3. evidence based on internal structure 4. evidence based on relations with out variables 5. evidence based on the consequences of testing

Evidence of validity based on test content is demonstrated in:

1. involves performing a series of systematic steps as a test is being developed—steps to ensure that the construct being measured is clearly defined and that the items used to measure the construct are representative of the construct's domain. - This method does not result in any final number (quantitative value) that represents the evidence of validity of the test; rather, it provides the test developer and user with confidence that the questions on the test are representative of a clearly defined domain. 2. involves evaluating the content of a test after the test has been developed. This method may be done by the test developer as part of the validation process or by others using the test. - This method may result in a number that can be used to quantify the content validity of the test.

Evidence based on the consequences of testing

Any time we make a psychological measurement, both intended and unintended consequences may occur. if the test is biased, an unintended consequence might be that test scores appear to favor one group over another. it is also important to understand that just because different groups score differently on a test does not automatically mean that the test is biased. test users need to be aware that it is important to distinguish between consequences of testing associated with the validity of the test itself (i.e., whether correct inferences are being made from the interpretation of the test scores) versus other outcomes not related to the purpose for which the scores will be used.

Demonstrating evidence of validity based on test content after test development

One technique involves examining the extent to which experts agree on the relevance of the content of the test items With this technique, experts review and rate how essential test items are to the attribute the test measures. Then a content validity ratio is calculated, providing a measure of agreement among the judges. content validity ratios can range between -1.00 and 1.00, where 0.00 means that 50% of the experts believe that an item is essential. Based on minimum values, questions that experts agree are essential are considered as evidence of validity, and items that experts do not agree are essential are not considered evidence of validity.

3. Developing the test specifications:

Test specifications are very similar to the blueprints that are prepared prior to building a home. Although the contents of test plans vary depending on the type of test being developed, many test specifications, especially those for knowledge tests, include not only a clearly defined testing universe but also the content areas—the subject matter that the test will measure—and the number of questions that will be included to assess each content area. The Standards for Educational and Psychological Testing - more broadly define the definition of test specifications to also include the documentation of the purpose and intended use of the test; the format; the length of the test; psychometric characteristics of the items; how the test is delivered, administered, and scored; as well as details of the content.

Sources of evidence of validity

Test validity is a function of how the scores are used—the inferences that will be made about the meaning of a person's test scores (or the conclusions that will be drawn from the test scores). The standards definition - validity refers to whether there is evidence supporting the interpretation of the resulting test scores for their proposed use. we should not refer to a test as valid; rather we should discuss whether there is evidence that supports the proposed use of the resulting test scores.

does the title of the test tell us something

The title of a test actually tells us very little

Evidence based on response processes

This source of validity evidence involves observing test takers as they respond to the test and/or interviewing them when they complete the test. We use these observations and interviews to understand the mental processes that test takers use to respond.

test specifications

a documented plan containing details about a test's content.

job analysis

a process that identifies the knowledge, skills, abilities, and other characteristics required to perform a job. -some types of job analysis also provide details of the actual tasks and activities that are performed on the job. - Often these data are gathered through interviews and direct observations by trained observers of the job being analyzed

Content validity ratios have been used to provide the evidence of validity based on

a test's content of employment tests, measures of the work behavior of psychiatric aides mathematics achievement tests

test can measure exactly what it is intended to measure and yet not be valid to use for a particular purpose. Ex:

a valid personality trait does not mean it can determine successful salespeople

Face validity

answers the question, "Does it appear to the test taker that the questions on the test are related to the purpose for which the test is being given? face validity is concerned only with how test takers perceive the attractiveness and appropriateness of a test. the concept of face validity does not have anything to do with the evaluation of test validity, it sometimes can be an important consideration in choosing a test that might be used in an applied setting. face validity is never an acceptable means of demonstrating evidence of the validity of a test.

measures of validity can provide us with

confidence that the interpretations or inferences we make about test-takers from their test scores are likely to be correct. More than just does the test measure what it's supposed to measure

Evidence of validity based on relationships with other variables, or _________, is appropriate when a test measures an abstract construct such as marital satisfaction.

construct validity

Evidence of validity based on relationships with other variables (criterion or construct validity) involves

correlating the test scores to a measure of performance or another test.

Evidence of validity based on relationships with other variables, __________________, is most appropriate for tests that claim to predict outcomes such as success on the job.

criterion-related validity,

Evidence of validity using test content is ______ to develop for tests such as achievement tests that measure concrete attributes.

easiest

current view of validity centers on the correct interpretations that will be made from a test's scores, we do not mean to

imply that investigating whether a test measures what it is supposed to measure is not important

Use of various sources of evidence of validity

in order for the inferences made from a test score to be appropriate, it is important that the test measure the attribute, trait, or characteristic it was designed to measure or predict the outcome it claims to predict. Developing different types of evidence of validity helps determine whether a test measures what it says it measures or predicts what it says it predicts.

1. During test development:

involves performing a series of systematic steps as a test is being developed.

2. Defining the test universe:

involves reviewing other instruments that measure the same construct, interviewing experts who are familiar with the construct, and researching the construct by locating theoretical or empirical research on the construct. The purpose is to ensure that you clearly understand and can clearly define the construct you will be measuring. Evidence of validity based on test content requires that the test cover all major aspects of the testing universe (of the construct) in the correct proportion

competency modeling

is a procedure that identifies the knowledge, skills, abilities, and other characteristics most critical for success on some or all of the jobs in an organization It also defines the behaviors that, when performed well, indicate that a person possesses each of those competencies. These competencies could be applicable to a wide range of leadership roles. Also, once the critical competencies have been identified, you can construct interview questions to measure them. Instead, many competency models concentrate on what are often termed "soft skills," such as communication, decision making, planning and organizing, time management, and the like. Because the process used in the development of competency models is usually less scientifically rigorous than that used for a formal job analysis competency modeling may not always be the appropriate method to use to develop content-based evidence of validity for tests used in personnel selection

construct

is an attribute, trait, or characteristic that is not directly observable but can be inferred only by looking at observable behaviors believed to indicate the presence of that construct. - We can't directly measure constructs, but we can directly measure behaviors.

if a test is designed to measure a specific attribute, it demonstrates evidence of validity based on

its content when the items in the test are a representative sample of the universe of items that represent the attribute's entire domain.

testing universe

the body of knowledge or behaviors that a test represents.

The appropriate strategy for gathering validity evidence often depends on

the purpose of the test.

5. Constructing test questions:

write the test questions or items, being careful that each question represents the content area and objective it is intended to measure. guidelines provide a uniform set of principles that employers are expected to follow when using any instrument (not just written tests) to make employment decisions. This would include interviews, performance appraisals, application forms, physical requirements, or anything else that an employer uses as part of the decision process. If job applicants (or employees) demonstrate to the court that they have good reason to believe that the employer discriminated against them, even unintentionally, because of race, color, sex, religion, or national origin, the employer would be required to produce evidence that the tools that were used as part of the selection system were valid. This type of evidence for validity is frequently gathered during the time that the selection tools are being developed.

would it be incorrect to assume that one type of validity is better or more scientific than another type of validity

yes Ultimately, it is the combined evidence from multiple sources and research approaches that will determine whether any particular inference made about the meaning of a test score is appropriate and defensible.