chapter 6

Heterotrait-Monomethod Triangles

correlations among measures that share the same method of measurement • Note: these correlations share a method, not trait or method • If these correlations are high, its because measuring different things with the same method results in correlated measures; got a strong "methods" factor

Validity Diagonal (monotrait-heteromethod)

correlations between measures of the same trait measured using different methods • Looks at convergent validity • There is one validity diagonal in each method block • This is the correlation between 2 measures of the same trait measure with 2 different measures. • Bc the 2 measures are of the same trait or concept, we would expect them to be strongly correlated.

Measurement validity

measures the validity of a measure; how validly does it measure the intended construct

Face Validity

occurs when a measure appears to be a reasonable measure of some trait -in order to have valid measure of a social construct, never stop at achieving only face validity - this is not sufficient -never skip establishing face validity, because the other components of validity can't be achieved without it

Criterion Validity

psychological measure is able to predict some future behavior or is meaningfully related to some other measure (a more rigorous test than face and content validity) a. Predictive validity: can your measure predict future behavior or attitude? b. Concurrent validity: does your measure give scores that agree with other related measures?

disadvantages of MTMM

• It requires a lot of subjective judgment to interpret • Can be limiting in use since for the most accurate measurement, it requires a fully crossed measurement design • Many researchers want to test for construct validity → use reliability coefficient instead

Adding more constructs and measures will enhance ability to assess construct validity using MTMM

...

Convergent Validity

Construct validity measures of theoretically similar constructs should be highly correlated • degree to which similar concepts that should be related theoretically are interrelated in reality • in MTMM and leaving out methods factor → o Expect correlation to be high correlation • strong intercorrelation • Ex: if each item in measure actually measures construct, should be high

Reliability Diagonal (monotrait-monomethod)

estimates reliability of each measure in the matrix. • There are as many correlations as there are measures - in example, 9 measures and 9 reliabilities. • Shows the highest correlations/sets a cap on how high other correlations should be - cap on how high reliability can be

Predictive validity:

Criterion Validity can your measure predict future behavior or attitude?

Concurrent validity

Criterion Validity does your measure give scores that agree with other related measures?

Factors influencing validity

Reliability, Heterogeneity of group with respect to test and criterion • Reliability: the greater the error (lower the reliability), the lower the validity coefficient o the most similar thing to a test is itself (reliability). If that relationship is weak, the test will not be able to relate to something other than itself (validity). This relationship can be expressed mathematically o reliability coefficient: ratio of variance = true score variance/observed score variance o validity coefficient: Pearson correlation coefficient → correlate your observed test scores with your chosen criterion measure. o Reliability coefficient = (validity coefficient)^2 • If test-retest reliability of a test is .64, then the maximum validity coefficient you can achieve is .8, which when squared is .64 • Reliability coefficient is already squared, and if given validity coefficient, needs to be squared to get into those same terms • Heterogeneity of group with respect to test and criterion: o If you use the test data as a selection device, thereby limiting the range of scores on the test, the test's ability to predict any criterion is impaired. • Ex: gre predicting success in grad school o If the criterion is so common or so rare, prediction is very hard since there is no variance associated with the measure • Ex: school shooting

Discriminant Validity

measures of theoretically different constructs should not correlate highly with each other • degree to which concepts that should not be related theoretically are not interrelated in reality • in MTMM and leaving out methods factor → o because measure are of different constructs - low correlation o Low correlation - evidence for validity • To claim that measures have construct validity, must demonstrate both convergence and discrimination.

Research validity

measures the validity of research. Measures how accurately a research study measures its intended research questions/theoretical concept. It encompasses internal validity, external validity, construct validity, and statistical validity.

Content Validity

occurs when a measure appears to be a reasonable measure of some trait as determined by experts in that field ο As opposed to face validity, experts are aware of nuances in the construct that may be rare or elusive of which the layperson may not be aware ο Many studies proceed following content validity achievement, but this doesn't mean the measures used are entirely valid

Construct Validity

the measure accurately assesses some construct; refers to when the construct itself is valid; refers to whether the operational definitions used for independent and dependent variables are valid • Need construct validity and discriminant validity to prove construct validity • the most rigorous validity • can be used as part of the hypothetico-deductive scientific method o the data we collect is dictated by the conceptual definition of the construct and the data then helps refine the nature of the construct • ex: if an attitude survey has construct validity, lower attitude scores, indicating negative attitude, should correlate negatively with life satisfaction survey scores, and positively with life stress scores • these other constructs do not necessarily have to be predictive or concurrent a. Convergent Validity: measures of theoretically similar constructs should be highly correlated • degree to which similar concepts that should be related theoretically are interrelated in reality • in MTMM and leaving out methods factor → o Expect correlation to be high correlation • strong intercorrelation • Ex: if each item in measure actually measures construct, should be high b. Discriminant Validity: measures of theoretically different constructs should not correlate highly with each other • degree to which concepts that should not be related theoretically are not interrelated in reality • in MTMM and leaving out methods factor → o because measure are of different constructs - low correlation o Low correlation - evidence for validity • To claim that measures have construct validity, must demonstrate both convergence and discrimination.

Heterotrait-Heteromethod Triangles

these are correlations that differ in both trait and method. • Looks at discriminant validity, low correlation • Because these correlations share neither trait nor method, we expect them to be the lowest in the matrix • Low correlation in homotrait-heteromethod also proves discriminant validity

Heteromethod Blocks

these consist of all correlations that do not share the same methods

Monomethod Blocks

these consist of all the correlations that share the same method of measurement - there are as many blocks as there are methods of measurement

Basic Rules for MTMM

• 1) Coefficients in reliability diagonal = consistently highest in matrix • 2) Coefficients in validity diagonals = sig diff from 0, high enough to warrant further investigation • 3) Validity coefficient = higher than values in its column and row in the same heteromethod block • 4) Validity coefficient = higher than all coefficients in the heterotrait-monomethod triangles o emphasizes that trait factors should be stronger than methods factors • when this is not the case, shows evidence that there might be a methods factor. • 5) Same pattern of trait interrelationship should be seen in all triangles o Some are more clear, others - triangles might be a multiple of the diagonal

4 Main aspects of research validity

• Internal validity: measures whether the changes in the DV can be attributed to the IV or a confound variable (DV affects IV) • External validity: measures whether the research findings are applicable to the general population (generalizability) • Construct validity: measures whether the instrument/measurement used are validly measuring the DV • Statistical validity: measures whether the appropriate statistical methods were used (appropriately sampling methods, appropriate way to analyze data)

Multitrait-Multimethod Matrix (MTMM)

• MTMM is a matrix arranged to facilitate the interpretation of the assessment of construct validity • Can assess both convergent and discriminant validity using the MTMM • To claim that your measures have construct validity, you have to demonstrate both convergence and discrimination • MTMM assumes that you measure each of several concepts (called traits) by each of several methods (e.g., direct observation, paper and pencil test, performance measure) and you measure each concept by each method

MTMM using NO METHODS

• Matrix can show convergent & discriminant validity, but does NOT say if these three measures actually measure self esteem or locus of control • However, it does show that the 3 measures seem to reflect the same construct and that the two sets of measures seem to be reflecting 2 different constructs • Idea of this matrix: only looks at intra and interrelationships - no methods factor o looks simultaneously at pattern of convergence and discrimination o emphasis on methods as a potential confounding factor • methods issue is more an issue of generalizability rather than one of construct validity • We move away from multitrait multimethod to multitrait matrix that enables us to examine convergent and discriminant validity → construct validity • Correlation coefficient: used to estimate the degree to which any two measures are related to eachother

advantages of MTMM

• provides methodology for assessing construct validity • visual of validity correlations - importance to look for the effects of how we measure and what we measure • All research must prove convergent and discriminant validity to prove construct validity