RM Unit 9: VALIDITY OF RESEARCH RESULTS

Internal validity

"approximate validity with which we infer that a relationship between two variables is causal" (p. 37).

list of prominent threats to validity in single-group designs:

-History -Maturation -Testing -Instrumentation -Regression Artifacts

Statistical conclusion validity refers to statistical inferences:

1)The first statistical inference is whether a relationship exists between the independent and dependent variables. 2)The second statistical inference is an estimate of the magnitude of the relationship between the independent and dependent variables.

Three types of evidence are needed to reach a conclusion of causation:

Condition 1: the relationship condition This requires evidence that the independent and dependent variables are associated, correlated, or related. Do changes in the independent variable correspond to changes in the dependent variable? Condition 2: the temporal antecedence condition The second type of evidence needed to infer causation is the correct temporal ordering of the variables being investigated, because a cause must precede an effect. If you cannot establish the correct temporal order (i.e., you don't know if X causes Y or if Y causes X), the problem is known as ambiguous temporal precedence and is a threat to your design and ability to draw a causal conclusion that X affects Y. Condition 3: the lack of alternative explanation condition The third type of evidence needed is that the variables being investigated are the ones that are causally related rather than being caused by some confounding extraneous variable. In other words, we must look for variables other than the independent variable that might explain the change observed in the dependent variable, and these competing explanations must be ruled out. The key point is that the third-variable problem means that two variables of interest might be correlated not because they are causally related but because they are both caused by or related to some third variable

Three types of validity are especially relevant to qualitative research:

Descriptive validity Interpretive validity Theoretical validity

effect size indicator;

Effect size estimates are obtained by computing a statistical index, this gives you an estimate of the strength of the relationship between the independent and dependent variables.

multiple operationalism

The most accurate representation of a construct involves measuring it in several different ways. The use of multiple operationalism is the recommended approach to use in research studies.

Researcher bias is a common threat to the trustworthiness of a study. Which of the following is a reason why this may occur in a qualitative research study?

The nature of qualitative research tends to be exploratory, opened-ended, and less structured than quantitative research.

one-group pretest-posttest design.

This is a research design in which one group of participants is pre-tested on some dependent variable. They are then administered a treatment condition. After this treatment is administered, they are post-tested on the dependent variable.

Differential Selection

This is a type of additive and interactive effect. It is a threat to the internal validity of a multi-group study when a difference exists, at the outset of the research study, between the characteristics of the participants forming the various comparison groups (e.g., treatment and control groups).

negative-case sampling.

This means that researchers attempt carefully and purposively to search for examples that go against their expectations and explanations about what they are studying. If you use this approach, you will find it more difficult to ignore important information, and you will come up with more credible and defensible results.

researcher bias

This occurs when researchers obtain results consistent with what they want to find. Researcher bias tends to result from selective observation and selective recording of information and also from allowing one's personal views and perspectives to affect how data are interpreted and how the research is conducted. Researcher bias is frequently an issue in qualitative research because qualitative research tends to be exploratory, open-ended, and less structured than quantitative research. However, researcher bias is a threat to validity in all research methods.

Instrumentation

This refers to any change that occurs in the measuring instrument. An instrumentation threat occurs in two primary ways: It can occur when the measurement instrument that is used during pre-testing is different from that used during post-testing. If the tests used during pre-testing and post-testing are not equivalent, a difference between the two performance measures might be strictly due to the difference in the way the two tests are assessing performance. An instrumentation effect can also occur when data are collected through observation. Many educational researchers use human observers to collect data. Human observers are, unfortunately, subject to such influences as fatigue, boredom, and learning processes. This is why studies that use human observers to collect data typically use more than one observer and have each observer go through a training program. In this way, some of the biases that are inherent in making observations can be minimized, and the various observers can serve as checks on one another to ensure that accurate data are being collected.

History

This refers to any event, other than a planned treatment event, that occurs between the first and second measurements of the dependent variable. The history threat is especially worrisome when: something in addition to the treatment occurs between the pre-test and post-test measurements of the dependent variable. the time interval between pre-test and post-test measurement is lengthy.

Testing

This refers to changes that may occur in participants' scores obtained on the second administration of a test as a result of previously having taken the test. The experience of having taken a pre-test may alter the results obtained on the post-test, independent of any treatment effect or experimental manipulation intervening between the pre-test and the post-test.

Maturation

This refers to physical or mental changes that may occur within individuals over time, such as aging, learning, boredom, hunger, and fatigue. Such changes can affect an individual's performance on the dependent variable. Because such changes might alter performance on the dependent variable, they represent threats to the internal validity of a study.

Regression Artifacts

This refers to the fact that extreme (high or low) scores will tend to regress or move toward the mean of a distribution on a second testing or assessment. In other words, when an individual's scores are extremely or unusually high or low, there is a strong chance that they will perform more typically upon subsequent measurements.

Additive and Interactive Effects

This refers to the fact that the threats to internal validity can combine to produce complex biases in multi-group designs. Keep in mind that in a multi-group design, the goal is for the comparison groups (e.g., treatment group and control group) to be similar on all variables except for the independent variable that delineates the different conditions for comparison.

validity

When we discuss whether findings and conclusions from research can be trusted

Population validity

ability to generalize from the sample of individuals on which a study was conducted to the larger target population and across different subpopulations.

two types of causal relationships

causal description and causal explanation.

Causal description

describing the consequences of manipulating an independent variable. This involves describing how changes made to an independent variable (X) causes changes to a dependent variable (Y); X → Y.

causal validity

establishing trustworthy evidence of cause and effect.

Causal explanation

explaining the mechanisms through which and the conditions under which a causal relationship holds. A full causal explanation of why the causal relationship exists "show[s] how the causally efficacious parts of the treatment influence the causally affected parts of the outcome through identified mediating processes"

Population validity, therefore, has two components of generalizing. Generalizing from a sample of individuals to the larger target population is a two-step process.

https://courses.acrobatiq.com/courseware/uvm_erm_1/media/content/validity_research_results/webcontent/Quality_Text/RDA_Module06_LO1_Two-step%20Process.png

It is important to think about the issue of validity in qualitative research and to examine some strategies that have been developed to maximize validity. The table below describes these strategies.

https://courses.acrobatiq.com/courseware/uvm_erm_1/validity_research_results/quality_three_approaches_and_action/Quality_Text_3

The focus here is on Onwuegbuzie and Johnson's nine types of mixed method research validity, also called types of legitimation. The words validity and legitimation are used interchangeably.

https://courses.acrobatiq.com/courseware/uvm_erm_1/validity_research_results/quality_three_approaches_and_action/Quality_Text_4

There are four major types of validity in quantitative research

internal, external, construct, and statistical conclusion.

ontological beliefs

involve assumptions of what is real or true in the world at large.

Operationalism

means that terms or constructs are represented by a specific set of steps or operations, indicators, or measures. The researcher should select the best measure available for the construct of interest. Use of a particular measure should be justified in the writing of the research report. Specification of a set of operations is required for accuracy in communication. Operationalizations tell you in a concrete and specific way how a construct was represented or measured in a particular study.

Interpretive validity

portraying accurately the meanings attached by participants to what is being studied by the researcher. More specifically, it refers to the degree to which the qualitative researcher accurately understands research participants' viewpoints, thoughts, feelings, intentions, and experiences and portrays them in the research report.

research reliability

present when the same results would be obtained if the study were conducted again (i.e., replicated)

External validity

refer to the extent to which the results of a study can be generalized to and across populations of persons, settings, times, outcomes, and treatment variations. It might help you to think of external validity as also being called generalizing validity.

trustworthiness

refer to this quality difference. When qualitative researchers speak of research validity, they are usually referring to qualitative research that is plausible, credible, trustworthy, and therefore defensible.

Treatment variation validity

refers to the ability to generalize the results across variations of the treatment.

Reactivity

refers to the alteration in performance that can occur as a result of being aware that one is participating in a research study.

Theoretical validity

refers to the degree to which a theoretical explanation developed from a research study fits the data and is therefore credible and defensible. Theory usually refers to discussions of how a phenomenon operates and why it operates as it does. Theory is usually more abstract and less concrete than description and interpretation. Theory development moves beyond "just the facts" and provides an explanation of the phenomenon.

Construct validity

refers to the extent to which a higher-order construct is accurately represented in the particular study.

Descriptive validity

refers to the factual accuracy of the account as reported by the researchers. The key questions that are addressed in descriptive validity are these: Did what was reported as taking place in the group being studied actually happen? Did the researchers accurately report what they saw and heard? Descriptive validity refers to accuracy in reporting descriptive information such as events, objects, behaviors, people, and settings. This form of validity is important because description is a major objective in nearly all qualitative research.

Statistical conclusion validity

refers to the validity with which you can infer that two variables are related and the estimated strength of that relationship is accurate.

The key strategy that is used to understand researcher bias

reflexivity negative-case sampling.

Internal validity

s to the ability to infer that a causal relationship exists between two variables. To infer that one variable caused an effect observed in another variable, we must control for all possible causes.

Outcome validity

the ability to generalize the results across different but related dependent variables. Many studies investigate the effect of some independent variable on one or more dependent variables. Outcome validity refers to the extent to which the independent variable influences a number of related outcome measures.

Ecological validity

the ability to generalize the results of a study across settings. Ecological validity therefore exists to the extent that the study results are independent of the setting in which the study was conducted.

research validity

the correctness or truthfulness of the inferences that are made from the results of the study.

Temporal validity

the extent to which the results of a study can be generalized across time.

accessible population

the group of research participants who are available to the researcher for participation in research.

target population

the larger population, such as all children with a learning disability, to whom the research study results are to be generalized. Within this larger target population, there are many subpopulations, such as male and female children with a learning disability.

reflexivity

the researcher actively engages in critical self-reflection about his or her potential biases and predispositions. Through reflexivity, researchers become more self-aware, and they monitor and attempt to control their biases.