Research week 7

Random error

due to fluctuations in participants, measurement, or conditions

Threats to External Validity

External validity determines to what extent the results of a study represent or apply to other situations (i.e., generalize beyond the current study) • Threats include: a. Reactivity arrangements 1. Placebo effect 2. Hawthorne effect 3. Demand characteristics 4. Pygmalion effect (experimenter expectancy) 5. Evaluation apprehension b. Order effects c. Treatment interaction effects • Ways to control for threats to external validity: d. Sampling

Convenience sampling

Involves gathering participants who are available and willing to participate. The degree of generalizability of a convenience sample to the population under study is often questionable, so reporting demographic characteristics of the sample is critical to determining similarities to a given population of interest.

Maturation

Maturation refers to the natural process of aging and development over time, or short-term physiological or psychological effects. Maturation can play a major role in longitudinal studies and when studying children. • Control of maturation can be addressed through subject matching or randomization.

Snowball Sampling

Snowball sampling: developed to obtain a sample by locating one hard-to-find participant and then developing a referral network to locate other likely participants. Mostly used when the researcher has difficulty locating a certain kind of participant pool. In this way, the single participant snowballs into a larger and larger sample

Controlling for Specific Threats to Internal Validity

There are four ways to minimize threats to validity: 1. Standardize the conditions in which the study occurs 2. Know the characteristics of the participants in the study 3. Know the details of the study (i.e., where, when, and the extraneous events that may occur) 4. Plan and choose a suitable design Other ways to minimize or control threats to internal validity: • Randomization or random sampling is the process of drawing a research sample in which each member of a population has an equal chance of being selected (random selection) • Random selection • Random assignment • Matching • Analysis of covariance

3 main types of evidence to determine internal validity of a study

Three main types of evidence that researchers may use to determine the internal validity of a study: 1. Content-related evidence 2. Criterion-related evidence 3. Construct-related evidence

Random selection

reduces sampling biases and increases generalizability of the results to the larger population (external validity)

Types of Sampling

1. Convenience sampling 2. Simple random sampling 3. Stratified random sampling 4. Cluster sampling 5. Purposeful sampling 6. Snowball sampling 7. Multistage sampling 8. Sample size and bias

Experimental control considerations when doing quantitative studies:

1. Maximizing the variability due to the independent variable 2. Control of confounding (extraneous) variables 3. Minimizing random error 4. Deciding whether to use control groups

Reactivity Arrangements

1.Placebo effect 2.Hawthorne effect 3.Demand characteristics 4.Pygmalion effect (experimenter expectancy) 5.Evaluation apprehension

Methods of Control in Experimental Research

Any validity threat that may affect the dependent variable can create problems or discrepancies in the results and necessitate a greater need for control. • Experimental research answers two primary questions: 1. Does a relationship exist between the independent variable and a dependent variable? 2. Is this relationship causal in nature? • Accurate answers to these two questions rely on the researcher's ability to maximize the experimental variance caused by the independent variable while minimizing: • Systematic error: caused by extraneous or confounding variables • Random error: due to fluctuations in participants, measurement, or conditions

Deciding Whether to Use Control Groups

Commonly used in experimental research because they provide reference groups for comparison with experimental treatment groups • Differences between participants in control or treatment groups can be attributed to the independent variable under study • Researchers ordinarily use either: • No-treatment control group: participants receive no treatment at all, whereas treatment group participants receive the experimental treatment (e.g., intervention, drug). • Placebo control group: participants are exposed to a "treatment" unrelated to the experimental treatment and not expected to alter the participants' performance on the dependent variable (e.g., a sugar pill as a substitute for another drug). • TAU (treatment as usual) control group: what the clients would ordinarily get if they presented for treatment. • Exercising more control over the variables in the research tends to decrease the generalizability or external validity of the results.

Construct-Related Evidence

Construct-related evidence includes a variety of different types of evidence supporting the characteristic being measured. • Three common ways to measure construct-related validity includes use of: 1. A clearly defined variable 2. The hypotheses based on theory explaining the variable 3. Logical and empirically tested hypotheses

Content-related evidence

Content-related evidence asks the question of whether the instrument(s) or sample accurately represents the variable under study. • To ensure content-related validity, researchers should look over the content (Do the questions or items accurately reflect the definition of the variables and the sample of participants?) and format (e.g., clarity, readability, font type and size, language) of the instrument to be used. • Content experts are used to determine how well an instrument represents a given domain of information or behavior.

Criterion-Related Evidence

Criterion-related evidence (i.e., empirical validity) is used to determine validity by comparing the instrument used in the study to another instrument or form of assessment presumed to measure the same variable. A correlation coefficient is often used to determine whether a relationship exists between the scores from the two instruments. • Two forms of criterion-related validity: 1. Predictive validity is obtained by administering the instrument and then allowing an elapsed time interval to pass for later comparison with the criterion scores (e.g., aptitude test and end-of-semester GPA). 2. Concurrent validity requires administration of the instrument and criterion data at the same point in time (e.g., attitude of students compared to teacher's observations).

Experimental Validity

Establishing experimental validity is the process of verifying that the research instruments and the results accurately reflect the research question in order to make correct conclusions and inferences (i.e., generalizability)

External Validity

External validity: also known as generalizability and refers to whether the results of the sample in a particular study can be generalized or applied to a population, group, condition, setting, or other participants.

Hawthorne Effect

Hawthorne effect: changes in performance by the mere presence of others Having a control group controls for this

Validity

I. Experimental validity II. Internal validity III.External validity IV.Methods of control in experimental research V. Controlling for specific threats to internal validity VI.Threats to internal validity VII.Threats to external validity

Threats to external validity

If a study's internal validity is limited. • When the observed effects of an independent variable on a dependent variable cannot be guaranteed outside of the experimentally controlled circumstance. • To control for threats to internal and external validity, researchers use various methods of experimental control.

Implementation

Implementation refers to variations in the way that the intervention is introduced or conducted, which may affect the results of the study. Single-subject researchers place a high degree of emphasis on implementation. • Single-subject researchers must ensure that procedures are implemented with a high degree of integrity. Interrater reliability data are required of manuscript writers when publishing a journal. Researchers are also required to report step-by-step procedures of their study for this reason.

Maximizing the Variability Due to the Independent Variable

Independent variables are selected because of their perceived potency in influencing the dependent variable, thus creating or controlling experimental variance. To enhance the effectiveness of the independent variable, researchers often try to make various levels of the independent variable as different as possible. Such differentiation allows the effect of the independent variable to be measured.

Placebo Effect

Placebo effect: occasions when participants in a study act according to expectations derived from inadvertent cues to the anticipated results of the study. It is not unusual for about 20% of the participants who are taking the placebo to report a substantial reduction in symptoms. It is assumed that these participants are responding in accord with what they believe the experimenter wants them to report. Conducting a blind study reduces the possibility of this threat by ensuring that participants are not aware of anticipated outcomes.

Purposeful Sampling

Purposeful sampling: commonly used in ethnographic or other qualitative methodologies when the researcher is interested more in diverse data sources than generalizability of results. Subtypes include: • Comprehensive sampling: selecting numerous participants with diverse characteristics and perspectives • Extreme-case sampling: selecting interesting, extreme, or outlier cases with interesting perspectives • Typical-case sampling: selecting normal or typical cases to study

Pygmalion Effect

Pygmalion effect (experimenter expectancy): occurs when the experimenter acts in ways that bias the study without necessarily affecting the participants directly, or when the experimenter unintentionally provides cues (demand characteristics) to let participants know what is expected of them

Internal Validity

Refers to a researcher's ability to determine whether a causal relationship exists between the independent variable (i.e., treatment variable) and the dependent variable (i.e., criterion or outcome variable), rather than due to some extraneous variable (i.e., one not related to the experiment). • Describes the level of confidence in which the results of a study are supported by the design or methodology. • When it is possible to conclude that extraneous variables caused changes in the dependent variable, internal validity and causal inference are threatened.

Sample Size and Bias

Sample sizes are often determined by the research design and nature of the analyses selected. • A general rule is that the larger the sample size, the greater the statistical power. • Professional counselors can follow the general rule that the larger the sample and more representative the sample is of the population under study, the greater the power and the better the study as long as high-quality randomized sampling procedures were used. • When determining the sample size: • Consider the magnitude of difference (effect size) necessary to determine statistical significance. • Select the desired statistical power (minimum accepted standard is typically .80, which means that researchers are willing to accept a 20% probability that they will fail to detect a true effect through their research procedures); G Power • Use a statistical table to determine the appropriate sample size for the methodology

Simple random sampling

Simple random sampling: randomly selecting a portion of the population to comprise the sample. In simple random sampling, every participant from the population has an equal chance of being selected into the sample. Also, selection of one participant must not affect the selection of any other participant. This is different from random assignment, which involves randomly assigning selected participants to the various treatment conditions (if any). In true experimental research designs, researchers strive to use simple random sampling to obtain a representative sample from the population and then assign these sample participants to various treatment conditions using some randomizing procedure.

Control of Confounding (Extraneous) Variables

Variables that contribute unwanted variance are called extraneous variables or confounding variables. • Researchers typically use four procedures to control for extraneous variables: 1. Random assignment: participants are randomly assigned to various treatment and control conditions, equalizing the potential error variance contributed by the extraneous variable across all of the groups. 2. Homogeneous participants: if the researcher is aware of the potential effects of an extraneous variable, she or he can select only participants who are homogeneous on the variable, thus controlling the effects by holding them constant. 3. Matching: if the researcher is aware of an extraneous variable, she or he can match participants who are equivalent on the extraneous variable before assigning them to various treatment conditions, thus ensuring equivalence. 4. Blocking variables: an identifiable extraneous variable can be built into the design of the study in order to control for its effects.

Random assignment

a type of randomization, this is the process of assigning individuals or groups to different treatment conditions to maximize the probability that subject characteristics are matched between the experimental groups, thus controlling for confounding variables (e.g., internal validity)

Systematic error

caused by extraneous or confounding variables

Multistage sampling

frequently used in large-scale surveys in which initial stratified random samples of larger entities (e.g., states, counties) are followed by simple random samples of smaller entities (e.g., residences, communities) and sometimes then even further by simple random sampling of individual participants within the smaller entity. Uses both randomization and stratification to minimize sampling bias.

Matching

refers to the technique of equating groups on one or more variables with the result of each member of one group having a direct counterpart in another group. Random assignment can be accomplished through the use of a table of random numbers

Analysis of covariance

tests the main and interaction effects of categorical variables on a continuous dependent variable, controlling for the effects of selected other continuous variables that covary with the dependent variable ACOVA

External validity: population generalizability

the degree to which the sample is representative of the population of interest on the particular characteristic or variable being studied.

Threats to Internal Validity

• Selection • History • Maturation • Mortality • Instrumentation • Testing • Location • Implementation • Experimental bias • Statistical regression

Demand characteristics

• Demand characteristics: environmental cues or statements that let participants know about the purpose of the study or what actions, behaviors, or outcomes are expected of participants • As a result of these insights, participants may react in accordance with those expectancies

Evaluation apprehension

• Evaluation apprehension: occurs when participants act in ways they believe will help them avoid the negative evaluations of the researcher, observers, or other participants • Evaluation apprehension results from an eagerness to please (reactivity) and some facets of social desirability

Experimental Bias

• Experimental bias refers to the researchers' possible bias toward the expected or hypothesized results—a violation of the objectivity required in experimental research. • Attitudinal effect refers to the possible perception by either group that they may be receiving special attention, which may affect the results of the study. • Can be addressed by using an experimenter who is unaware of the anticipated results (e.g., double-blind study) to administer the intervention(s) or treatment. Double-blind studies address both researcher bias and attitudinal effect because only the researchers would know if the treatments were actual medications or placebos (not the assistant administering them) and participants would not know which they were assigned. Thus, neither the researcher's bias nor the attitudinal effect of participants could influence the outcome of the study.

History

• History refers to confounding events outside of the research study that can alter or affect participants' performance, such as experiential environmental events. • Using randomization procedures can often minimize this risk, ensuring that outside events that occur in one group are also likely to occur in the other. A time-series design can also help detect the influence of a historical event, although it will not control for the effects of an event.

Instrumentation

• Instrumentation is the change in the measurement device(s) used during the course of the study. Changes in scores may be related to the instrument differences rather than the independent variable. • Reliability refers to the consistency of the scores obtained for each individual from one administration of a test or instrument to another. Consistency of scores is affected by any changes in data collector characteristics, data collector bias (e.g., unintentional bias of data collector, changes in the accuracy of the scorer over time), or instrument decay (e.g., changes in instruments over time). • To control for instrumentation, it is recommended that pretests and posttests be identical and that measurement procedures and measurement devices not change over time. Researchers are strongly encouraged to use instruments that yield highly reliable scores.

Location

• Location may affect the accuracy of scores if differences exist for the place of testing and the data-collection process (e.g., testing site). • Factors such as lighting, noise, or comfortable seating can affect a person's ability to concentrate. • Thus, it is important to ensure that the environment is the same across conditions.

Mortality

• Mortality (i.e., attrition) refers to the possibility of a differential effect due to those participants who drop out during the study versus if the participants had stayed in the study. This is problematic for longitudinal studies, as the sample that the researcher ends up with may differ substantially from the sample selected at the study's onset. • According to standard review board policies, all research participants must have the right to withdraw from the study at any point without penalty. Thus, it is difficult for researchers to control mortality. • Pretesting of participants often allows researchers to determine whether those who later drop out of a study differ in important ways from those who remain.

Order Effects

• Order effects (or carryover effects) refer to treatment effects derived from the order in which treatment is administered rather than from the treatment itself. • Counterbalancing the order of treatments sometimes helps to counteract order effects, but the primary problem is that generalizability of results from the sample to the general population is often not possible unless the population is also exposed to the multiple treatments (and in the most effective order).

Minimizing Random Error

• Random error includes unpredictable fluctuations that occur in participants (e.g., distractions, fatigue, illness), instruments (e.g., scoring errors, lack of appropriate calibration), or experimental or ecological conditions (e.g., the regular room is not available; the room temperature is uncomfortably cold). It is difficult to predict and control. • Preparation and standardization can minimize the influence of random error. Researchers should strive to keep the environment free of distractions, periodically calibrate instruments or check for scorer accuracy, and take extraordinary steps to be sure participants arrive physically and mentally prepared.

Reactivity Arrangements

• Reactivity involves participants behaving (reacting) in certain ways because they have knowledge that they are being observed or experimented on. Some consider reactivity to threaten both internal and external validity because experimenter bias can affect both the validity of the study (internal validity) and generalizability of findings to the general population (external validity). • Reactivity is generally best controlled by using: • A single-blind or double-blind methodology • Unobtrusive measures so participants do not know they are being observed • Deception (when all else fails)

Cluster sampling

• Requires the researcher to (randomly or conveniently) select units (clusters) of participants (e.g., several classes of counseling students) and then either randomly select individual participants or select all participants from the cluster. • Using random cluster sampling followed by randomized individual participant selection will lead to better representation and generalizability. • Larger numbers of clusters enhance demographic strata diversity.

Selection

• Selection is the manner in which participants are chosen to participate in a study and the manner in which they are assigned to groups. • Subject characteristics are those individual factors that may account for an observed effect, such as age, gender, critical thinking, or ability. • Differences that may be present between the groups prior to the study will continue throughout the study and may result in a treatment effect when no true experimental effect is present. • Randomization (i.e., random sampling and random assignment) can reduce the possibility of effects due to subject characteristics, as can participant matching at the outset of the study through use of a pretest to determine preexisting similarities on the dependent variable.

Statistical Regression

• Statistical regression, or regression to the mean, refers to the tendency for participants with extreme scores (very high or very low) to score more toward the mean on subsequent testing. This is simply a statistical phenomenon readily observed when working with participants who were selected for their extreme performance or characteristics (e.g., gifted students, depressed patients, mentally retarded adults, hyperactive children). • Use of control groups or the collection of multiple baselines can help minimize the effects of statistical regression

Testing

• Testing refers to the practice effect of participants who are exposed to the intervention or test on multiple occasions (i.e., test-retest). Chances are that participants will perform better the second time merely due to practice. • Alternate forms may control for this effect. In lieu of alternate forms of an instrument, control group studies are recommended. In addition, some researchers administer the test only once, whereas others design the test to minimize practice effects.

Treatment Interaction Effects

• Treatment interaction effects refer to the potential for a treatment protocol to have an effect based on the characteristics of the participant rather than on the group as a whole. This is known as an interaction effect of treatment and selection and is confounded because the characteristics of the selected participants may make them respond to treatment in a way that nonparticipants may not. • Administration of a pretest may sensitize the participant to the study's purpose, thus changing the participant's reactions to the treatment condition. Consequently, unless individuals in the population are also exposed to and sensitized by the pretest, the results from the sample may not generalize to the population.

Stratified random sampling

• Used when the researcher wants to ensure that certain characteristics of participants are reflected in the final sample in the same proportion that they occur in the population. These characteristics are called strata. • Commonly used strata are sex, age, socioeconomics, race, ethnicity, educational level, and area of residence.