EDF6481_CMB-21Fall 00391

Regression

(also known as Regression to the Mean) occurs when participants are selected from their extreme scores, but have less extreme scores on other measures, appearing as if an effect occurred

Fishing

(also known as p-hacking or error rate problem) occurs when multiple statistical tests are conducted on the same data, increasing the likelihood of finding a significant effect by chance alone.

Parametric statistics

(t-tests, Analyses of variance, and correlations) assume that the data that are analyzed have certain characteristics (i.e. continuous variables are normally distributed, variances are similar across categories, and observations are independent of each other).

Compute the sum of squares for the following values: 6, 4, 3, 10, 7

30

Statistical Regression

A gifted and talented program was set up in a high school. Candidates for the program were selected from the school wide comprehensive tests given at the end of the year. Unfortunately, these tests were not reliability measures of achievement. After a year in the special "gifted and talented" program, students appeared to have learned no more than the others on the next annual comprehensive test. What is most likely to account for this?

unreliability of treatment implementation

Aaliyah is a family and marital therapist who wants to compare different role-playing approaches: the empty chair and role reversal techniques. Since both of these interventions require several, individual sessions with couples, she assigns 10 counselors to work with 40 couples. Five counselors will ask 20 couples to employ the empty chair approach as part of their therapy sessions, and another five counselors will ask the other 20 couples to practice role reversal as part of their sessions. After 10 weeks of therapy she gives the couples a marital satisfaction test and finds that those who used the empty chair technique were more satisfied than those who used the role reversal technique. After completing the study, she learns that three of the counselors in the role reversal group were allowing couples to revert back to their original roles after only 10 minutes of the exercise. What threat to validity is most likely affecting this result?

How do researchers typically analyze data from a single subject design?

By looking at a graph that measures behavior over time

low statistical power

Cat and Renee are looking to see if RAMP schools have better academic performance outcomes than schools without the RAMP designation. They ran a t-test and found that there was no difference between the two types of schools (RAMP & non-RAMP), t(8) = 2.15, ns. If they only had 10 schools, they should be concerned with what threat to validity?

Dr. Xu wants to know if providing a study guide to students improves their test scores. To determine this, she gives students the midterm exam without a study guide. Before the final exam, she distributes a study guide to students who were randomly assigned to receive it. She then gives all students the final exam and finds that those who had the study guide had higher scores on the final exam. What type of study is this?

Causal

interaction of the causal relationship with units

Chantel works for Beefcake Gym and wants to know if her clients are more likely to loose weight with high-intensity interval training (HIIT) or low-intensity steady-state (LISS) training. She recruits 100 participants and discovers that 35 of them are overweight and 60 are obese. She randomly assigns her participants to one of the two training conditions and verifies that participants in each group are similar in weight and proportions of males and females. After eight weeks of training (four times a week for 30 minutes), she finds that all participants have lost weight. However, there is not a significant difference in the amount of weight lost between the HIIT and LISS groups. This surprises her, as other exercise scientists have found that HIIT is more effective for weight loss than LISS. What threat to validity might explain the difference between her results and those found in previous studies?

Which of the following is an example of a directional hypothesis?

Children who are fully included learn more than children who are not fully included.

Dr. Douglass wants to know if there is a relationship between college major (psychology vs. history) and extracurricular involvement (number of hours spent in non-academic programs). He obtains a sample and uses questionnaires to assess the major of each participant and how many hours he or she spends in student and community organizations. He finds that history majors spent more time in extracurricular activities than psychology majors. What type of study is this?

Correlational

Dr. Weston wants to know if college students who work part-time have lower GPAs. To determine this, she asks students how many hours they work at a job per week and obtains college GPAs from students' academic records. She finds that those who work part time actually have higher GPAs than those who do not work at all. What type of study is this?

Correlational

Dr. Rodriguez wants to know what percentage of pets in animal shelters are cats. He mails surveys to a random sample of 200 animal shelters in the United States asking how many of each type of animal they house in their shelters. He receives 188 of those surveys back and determines that he has an equal representation from each state. He finds that 36% of the pets housed in his sample shelters are cats. What type of study is this?

Descriptive

interaction of the causal relationship with settings

Due to excessive rain this summer, the Manatee's Little League baseball team has been practicing for their games by using indoor batting cages. In the batting cage, the players have a mean batting average of .25; however, during the games, their batting average drops to .20. What is the most likely threat to validity for the change in their batting performance?

experimenter bias

Edgar is given a class of students with exceptional FCAT scores from the previous year. However, he has been told that his students' Florida Standards Assessment (FSA) scores ranged from poor to very good. The students are told to dress in certain fashions to mimic certain social groups (i.e. some are wearing trendy, name-brand clothing, some are dressed shabbily, and some are dressed conservatively). By the middle of the semester, the conservative students have considerably higher grades than the trendy or shabby students. What threat to validity is most likely affecting the students' performance?

What is the method in which a researcher observes behavior from first-hand experience by immersing himself/herself into the population he/she is studying?

Ethnography

The use of systematic procedures to provide useful feedback and to investigate the effectiveness or value of social programs or objects is called:

Evaluation

inductive hypothesis

Generalization from a number of observations based on trends, regularities, patterns, or relationships that are observed in clinical practice

Statistical conclusion validity

How well researchers can conclude that cause and effect variables covary

External validity

How well researchers can generalize causal relationships across units, treatments, observations, and settings

Construct validity

How well researchers can infer that the research operations sufficiently represent constructs of interest

Which is not a type of sampling strategy?

Hypothetical sampling

What is the event, behavior or characteristic that is manipulated by the researcher and expected to influence another variable?

Independent variable

What type of research validity is concerned with how well one can make inferences about causal relationships between variables?

Internal validity

convience sampling (non-random)

Involves selecting participants who are readily available without any attempt to make the sample representative of a population.

Selection

Kelly wants to examine the effectiveness of i-Ready on reading comprehension with as little disruption to current class procedures as possible. Therefore, she has Ms. Paul use i-Ready in her class and asks Ms. Spencer to teach reading to her class using a traditional approach. After two months, she finds that those in Ms. Spencer's class scored higher on a reading comprehension exam than Ms. Paul's students. Later, Kelly wonders whether the difference between the groups on the reading comprehension exam could be due to the different initial make-up of the two classes. Kelly is concerned with which threat to validity?

Sampling in Qualitative Research

Qualitative researchers are as concerned as quantitative researchers with the quality of their samples, but they use different considerations in selecting study participants. Selection of sample members guided by desire for information-rich data sources "Representativeness" not a key issue Random selection not considered productive

A qualitative research method in which language is analyzed to detect the meaning of signs and symbols is

Semiotics

Non-directional hypothesis

Simply states that there is a difference but not what the difference will be.

directional hypothesis

States the direction of the difference or relationship

Qualitative studies

Studies that involve a small number of cases and do not allow the researcher to use statistical techniques to analyze the data describe phenomena of interest; include participant observation, in-depth interviews, and focus groups

accessible population (available)

The portion of the target population that is accessible to the researcher, from which a sample is drawn

Inadequate explication of constructs

When a construct is not clearly defined making it difficult to sufficiently operationalize it

Maturation

When a natural progression or change that occurs between pretest and posttest at the same time as the treatment, program or intervention and affects the outcome or dependent variable

History

When an event that affects the outcome or dependent variable occurs between pretest and posttest at the same time as the treatment, program or intervention

Extraneous variance in the experimental setting

When distractors or events that are not part of the treatment, program or intervention occur during an experiment that affect the outcome or dependent variable

Ambiguous temporal precedence

When it is not clear whether or not the cause preceded the effect

Mono-operation bias

When only one operation or measure is used to assess a construct

Selection

When participants are not randomly assigned to treatment groups, resulting in systematic differences between groups prior to a treatment, program or intervention

Hypothesis guessing

When participants attempt to guess the researchers' hypotheses and alter their responses to support or refute those hypotheses

Treatment sensitive factorial structure

When participants respond differently to an operation because they were exposed to a treatment, program or intervention

Reactivity to the experimental situation

When participants respond differently under experimental conditions because they are aware that they are being evaluated

Testing

When repeatedly exposing units to a test may permit them to learn the test, appearing as if a treatment effect occurred

Low statistical power

When sample size, effect size or alpha level is not large enough to detect a significant effect for a treatment

Treatment diffusion

When the control group learns about the program or intervention from the treatment group and also begins to use it

Interaction of the causal relationship with settings

When the results of a study cannot be generalized to another setting

Interaction of the causal relationship with outcomes

When the same results of a study cannot be found if the researchers use a different operation or measure of the outcome or dependent variable

Heterogeneity of units

When the variablity of scores on the outcome or dependent variable is so large within groups that it is difficult to detect an effect between groups

Attrition (Mortality)

When units drop out of the experiment after being assigned to treatment conditions

When must a researcher provide participants with a debriefing form?

Whenever deception is used in an experiment

reactive self-report changes

Zhiqiang is an Internship Coordinator for Google. This year he plans to select 1,500 interns to work at Google. However, over 40,000 people apply each year. Given that these internships are very desirable and often lead to permanent positions at Google or other tech companies, many applicants "pad" their resumes. In his experience, he has found that about 20% of the them exaggerate or falsify their educational or prior work experiences. If you consider the internship as an employment training program (i.e., intervention), Zhiqiang is concerned with which threat to validity?

quota sampling

a nonprobability sampling method in which elements are selected to ensure that the sample represents certain characteristics in proportion to their prevalence in the population

operational definition

a statement of the procedures used to define research variables clarify important terms in a study so that all readers understand the precise meaning the research intends examples: "high anxiety" may be a score on the Acme Anxiety Inventory in the upper 30% of student scores, and "low anxiety" may be a score in the lowest 30% of students. "POOR" performance on a math test may be operationally defined in terms of certain math subtest scores on the California Achievement Test a statement of the procedures (operations) used to define research variables. For example, human intelligence may be operationally defined as what an intelligence test measures.

Reactive Self-Report Changes

changes occur when participants are motivated to alter their responses on a measure. This is most likely to occur when the participants believe that they will be rewarded (or avoid punishment) by having especially high or low scores on a measure.

Descriptive studies are used to

describe a population from a sample

Qualitative Sampling approaches

intensity, homogeneous, criterion, snowball, and random purposive

The Counseling Center has recently developed a questionnaire to measure marital satisfaction. Clients' responses are scored on a five-point rating scale, then summed for a total score. To ensure that the questions are all measuring the same construct, presumably marital satisfaction, Simone needs to know that the responses given on the set of items are similar. What psychometric property does Simone hope to establish?

internal consistency

purposive sampling (non-random)

involves using researchers' knowledge about the population to handpick sample members

Reliability of Treatment Implementation

is a concern when a treatment, program, or intervention is implemented differently for different participants in a study.

Reliability of Measurement

is a concern when the instrument used to measure the construct (thinking or behavior) of interest does not provide consistent scores or assessments.

Restriction of Range

is a problem when the scores obtained from a sample on a particular variable do not span the total possible range of values. Very often when this threat is plausible, so is confounding constructs with levels of constructs (see threats to construct validity).

systematic sampling

is sampling in which every Kth individual is selected from a list of all members in the population. K is a variable determined by dividing the number of individuals on the list by the number of subjects desired for the sample

cluster sampling

is sampling in which groups, not individuals, are randomly selected. Clusters can be communities, states, school districts, and so on.

Sampling

is the process of selecting a number of individuals for a study so that the individuals represent the larger group from which they were selected.

simple random sampling

is the process of selecting a sample so that all individuals in the defined population have an equal and independent chance of selection for the sample. It is the best way to obtain a representative sample, although representation of specific subgroups is not guaranteed

stratified sampling

is the process of strategically selecting a sample so that it guarantees desired representation of relevant subgroups within the sample

Confounding Constructs with Levels of Constructs

is when an inference is made about a construct based on a restricted range of levels of that construct (i.e. only high or low values of performance). Very often when this threat is plausible, so is a restriction of range (see threats to statistical conclusion validity).

Evaluation apprehension

is when participants try to provide more favorable responses because they are anxious about being evaluated

Inaccurate Effect Size Estimation

is when the statistic used to analyze the data tends to over- or underestimate treatment effects. This most often occurs when the sample size is extremely high or low, as many statistics are sample size dependent - that is they are easily influenced by the sample size. It can also occur when an inappropriate statistic is used, such as using a related samples (within-subjects) statistic for a between-subjects design.

Experimenter Expectancies

occur when experimenters have certain expectations about participants' thinking or behaviors, which may bias how they interpret participants' responses. This can be avoided by not telling the experimenter which condition participants are in.

Additive and Interactive Threats to Internal Validity

occur when the impact of a threat can be compounded by (or may depend on the level of) another threat.

Instrumentation

occurs when changes in the instrument used to measure responses provide different results for the outcome, appearing as if an effect occurred. This often occurs when observations are taken over an extended period of time and a measure is revised.

Violation of Statistical Tests

occurs when data violate assumptions of statistical tests that are used to assess the relationship between variables

Extraneous Variance in the Experimental Situation

occurs when distracters or events that are not part of the treatment, program or intervention occur during an experiment.

A researcher wants to know if alcohol consumption increases levels of aggression. She obtains a sample, of which, half are randomly assigned to the treatment condition and given alcohol and half are not. She then observes participants and gives them a questionnaire to measure levels of aggression. Aggression levels of the treatment group are then compared to the control group, and she finds that those who drank alcohol were indeed more aggressive than those who had not consumed alcohol. Which type of design was used in this study?

posttest only

guiding hypothesis

qualitative researchers that generate new hypotheses as a result of their studies rather than solidifying the intention of the research beforehand

Qualitative vs. Quantitative Research

qualitative: derives data from observation, interviews or verbal interactions and focuses on meaning and interpretation of participants. quantitative is based on collected objective data that can be subjected to statistical analysis

deductive reasoning

reasoning in which a conclusion is reached by stating a general principle and then applying that principle to a specific case (The sun rises every morning; therefore, the sun will rise on Tuesday morning.)

Quantitative studies

studies that involve a large number of cases, allowing the researcher to use statistical techniques to analyze the data

Problem Statement

the first item in the introduction to a research plan and final research report

Mono-Method Bias

when all operations or measures use the same method to assess a construct

Context Dependent Mediation

when an explanatory mediator of one causal relationship would not mediate a similar relationship in another context

Construct Confounding

when an operation or measure involves more than one construct. This frequently occurs when constructs are not clearly defined or tend to be strongly related to each other.

Novelty and Disruption Effects

when participants respond especially well or poorly to a new intervention, simply because it is different from their usual routine

Resentful Demoralization

when the control group resents not receiving a desirable treatment, program, or intervention and responds more negatively to outcomes or dependent variables.

Interaction of the Causal Relationship over Treatment Variations

when the results of a study cannot be generalized to a different type or part of a treatment, program or intervention.

Interaction of the Causal Relationship with Units

when the results of a study cannot be generalized to another group of participants.

Compensatory Rivalry

when the treatment, program, or intervention provides desirable services and the control attempts to find a similar treatment or services

Compensatory Equalization

when the treatment, program, or intervention provides desirable services, and an administrator or staff member provides similar services to compensate the control group.

Random sampling

◦Anyone in population has an equal chance of being in your sample ◦Theoretically ideal, but not practical

convience sampling

◦Anyone who is easily accessible to you University center Place where you work Classmates

Purposive

◦Specific group of interest African-American women studying political science Orphans between the ages of 3 and 12 ◦Screening for inclusion/exclusion criteria