EDF6481_CMB-21Fall 00391
Regression
(also known as Regression to the Mean) occurs when participants are selected from their extreme scores, but have less extreme scores on other measures, appearing as if an effect occurred
Fishing
(also known as p-hacking or error rate problem) occurs when multiple statistical tests are conducted on the same data, increasing the likelihood of finding a significant effect by chance alone.
Parametric statistics
(t-tests, Analyses of variance, and correlations) assume that the data that are analyzed have certain characteristics (i.e. continuous variables are normally distributed, variances are similar across categories, and observations are independent of each other).
Compute the sum of squares for the following values: 6, 4, 3, 10, 7
30
Statistical Regression
A gifted and talented program was set up in a high school. Candidates for the program were selected from the school wide comprehensive tests given at the end of the year. Unfortunately, these tests were not reliability measures of achievement. After a year in the special "gifted and talented" program, students appeared to have learned no more than the others on the next annual comprehensive test. What is most likely to account for this?
unreliability of treatment implementation
Aaliyah is a family and marital therapist who wants to compare different role-playing approaches: the empty chair and role reversal techniques. Since both of these interventions require several, individual sessions with couples, she assigns 10 counselors to work with 40 couples. Five counselors will ask 20 couples to employ the empty chair approach as part of their therapy sessions, and another five counselors will ask the other 20 couples to practice role reversal as part of their sessions. After 10 weeks of therapy she gives the couples a marital satisfaction test and finds that those who used the empty chair technique were more satisfied than those who used the role reversal technique. After completing the study, she learns that three of the counselors in the role reversal group were allowing couples to revert back to their original roles after only 10 minutes of the exercise. What threat to validity is most likely affecting this result?
How do researchers typically analyze data from a single subject design?
By looking at a graph that measures behavior over time
low statistical power
Cat and Renee are looking to see if RAMP schools have better academic performance outcomes than schools without the RAMP designation. They ran a t-test and found that there was no difference between the two types of schools (RAMP & non-RAMP), t(8) = 2.15, ns. If they only had 10 schools, they should be concerned with what threat to validity?
Dr. Xu wants to know if providing a study guide to students improves their test scores. To determine this, she gives students the midterm exam without a study guide. Before the final exam, she distributes a study guide to students who were randomly assigned to receive it. She then gives all students the final exam and finds that those who had the study guide had higher scores on the final exam. What type of study is this?
Causal
interaction of the causal relationship with units
Chantel works for Beefcake Gym and wants to know if her clients are more likely to loose weight with high-intensity interval training (HIIT) or low-intensity steady-state (LISS) training. She recruits 100 participants and discovers that 35 of them are overweight and 60 are obese. She randomly assigns her participants to one of the two training conditions and verifies that participants in each group are similar in weight and proportions of males and females. After eight weeks of training (four times a week for 30 minutes), she finds that all participants have lost weight. However, there is not a significant difference in the amount of weight lost between the HIIT and LISS groups. This surprises her, as other exercise scientists have found that HIIT is more effective for weight loss than LISS. What threat to validity might explain the difference between her results and those found in previous studies?
Which of the following is an example of a directional hypothesis?
Children who are fully included learn more than children who are not fully included.
Dr. Douglass wants to know if there is a relationship between college major (psychology vs. history) and extracurricular involvement (number of hours spent in non-academic programs). He obtains a sample and uses questionnaires to assess the major of each participant and how many hours he or she spends in student and community organizations. He finds that history majors spent more time in extracurricular activities than psychology majors. What type of study is this?
Correlational
Dr. Weston wants to know if college students who work part-time have lower GPAs. To determine this, she asks students how many hours they work at a job per week and obtains college GPAs from students' academic records. She finds that those who work part time actually have higher GPAs than those who do not work at all. What type of study is this?
Correlational
Dr. Rodriguez wants to know what percentage of pets in animal shelters are cats. He mails surveys to a random sample of 200 animal shelters in the United States asking how many of each type of animal they house in their shelters. He receives 188 of those surveys back and determines that he has an equal representation from each state. He finds that 36% of the pets housed in his sample shelters are cats. What type of study is this?
Descriptive
interaction of the causal relationship with settings
Due to excessive rain this summer, the Manatee's Little League baseball team has been practicing for their games by using indoor batting cages. In the batting cage, the players have a mean batting average of .25; however, during the games, their batting average drops to .20. What is the most likely threat to validity for the change in their batting performance?
experimenter bias
Edgar is given a class of students with exceptional FCAT scores from the previous year. However, he has been told that his students' Florida Standards Assessment (FSA) scores ranged from poor to very good. The students are told to dress in certain fashions to mimic certain social groups (i.e. some are wearing trendy, name-brand clothing, some are dressed shabbily, and some are dressed conservatively). By the middle of the semester, the conservative students have considerably higher grades than the trendy or shabby students. What threat to validity is most likely affecting the students' performance?
What is the method in which a researcher observes behavior from first-hand experience by immersing himself/herself into the population he/she is studying?
Ethnography
The use of systematic procedures to provide useful feedback and to investigate the effectiveness or value of social programs or objects is called:
Evaluation
inductive hypothesis
Generalization from a number of observations based on trends, regularities, patterns, or relationships that are observed in clinical practice
Statistical conclusion validity
How well researchers can conclude that cause and effect variables covary
External validity
How well researchers can generalize causal relationships across units, treatments, observations, and settings
Construct validity
How well researchers can infer that the research operations sufficiently represent constructs of interest
Which is not a type of sampling strategy?
Hypothetical sampling
What is the event, behavior or characteristic that is manipulated by the researcher and expected to influence another variable?
Independent variable
What type of research validity is concerned with how well one can make inferences about causal relationships between variables?
Internal validity
convience sampling (non-random)
Involves selecting participants who are readily available without any attempt to make the sample representative of a population.
Selection
Kelly wants to examine the effectiveness of i-Ready on reading comprehension with as little disruption to current class procedures as possible. Therefore, she has Ms. Paul use i-Ready in her class and asks Ms. Spencer to teach reading to her class using a traditional approach. After two months, she finds that those in Ms. Spencer's class scored higher on a reading comprehension exam than Ms. Paul's students. Later, Kelly wonders whether the difference between the groups on the reading comprehension exam could be due to the different initial make-up of the two classes. Kelly is concerned with which threat to validity?
Sampling in Qualitative Research
Qualitative researchers are as concerned as quantitative researchers with the quality of their samples, but they use different considerations in selecting study participants. Selection of sample members guided by desire for information-rich data sources "Representativeness" not a key issue Random selection not considered productive
A qualitative research method in which language is analyzed to detect the meaning of signs and symbols is
Semiotics
Non-directional hypothesis
Simply states that there is a difference but not what the difference will be.
directional hypothesis
States the direction of the difference or relationship
Qualitative studies
Studies that involve a small number of cases and do not allow the researcher to use statistical techniques to analyze the data describe phenomena of interest; include participant observation, in-depth interviews, and focus groups
accessible population (available)
The portion of the target population that is accessible to the researcher, from which a sample is drawn
Inadequate explication of constructs
When a construct is not clearly defined making it difficult to sufficiently operationalize it
Maturation
When a natural progression or change that occurs between pretest and posttest at the same time as the treatment, program or intervention and affects the outcome or dependent variable
History
When an event that affects the outcome or dependent variable occurs between pretest and posttest at the same time as the treatment, program or intervention
Extraneous variance in the experimental setting
When distractors or events that are not part of the treatment, program or intervention occur during an experiment that affect the outcome or dependent variable
Ambiguous temporal precedence
When it is not clear whether or not the cause preceded the effect
Mono-operation bias
When only one operation or measure is used to assess a construct
Selection
When participants are not randomly assigned to treatment groups, resulting in systematic differences between groups prior to a treatment, program or intervention
Hypothesis guessing
When participants attempt to guess the researchers' hypotheses and alter their responses to support or refute those hypotheses
Treatment sensitive factorial structure
When participants respond differently to an operation because they were exposed to a treatment, program or intervention
Reactivity to the experimental situation
When participants respond differently under experimental conditions because they are aware that they are being evaluated
Testing
When repeatedly exposing units to a test may permit them to learn the test, appearing as if a treatment effect occurred
Low statistical power
When sample size, effect size or alpha level is not large enough to detect a significant effect for a treatment
Treatment diffusion
When the control group learns about the program or intervention from the treatment group and also begins to use it
Interaction of the causal relationship with settings
When the results of a study cannot be generalized to another setting
Interaction of the causal relationship with outcomes
When the same results of a study cannot be found if the researchers use a different operation or measure of the outcome or dependent variable
Heterogeneity of units
When the variablity of scores on the outcome or dependent variable is so large within groups that it is difficult to detect an effect between groups
Attrition (Mortality)
When units drop out of the experiment after being assigned to treatment conditions
When must a researcher provide participants with a debriefing form?
Whenever deception is used in an experiment
reactive self-report changes
Zhiqiang is an Internship Coordinator for Google. This year he plans to select 1,500 interns to work at Google. However, over 40,000 people apply each year. Given that these internships are very desirable and often lead to permanent positions at Google or other tech companies, many applicants "pad" their resumes. In his experience, he has found that about 20% of the them exaggerate or falsify their educational or prior work experiences. If you consider the internship as an employment training program (i.e., intervention), Zhiqiang is concerned with which threat to validity?
quota sampling
a nonprobability sampling method in which elements are selected to ensure that the sample represents certain characteristics in proportion to their prevalence in the population
operational definition
a statement of the procedures used to define research variables clarify important terms in a study so that all readers understand the precise meaning the research intends examples: "high anxiety" may be a score on the Acme Anxiety Inventory in the upper 30% of student scores, and "low anxiety" may be a score in the lowest 30% of students. "POOR" performance on a math test may be operationally defined in terms of certain math subtest scores on the California Achievement Test a statement of the procedures (operations) used to define research variables. For example, human intelligence may be operationally defined as what an intelligence test measures.
Reactive Self-Report Changes
changes occur when participants are motivated to alter their responses on a measure. This is most likely to occur when the participants believe that they will be rewarded (or avoid punishment) by having especially high or low scores on a measure.
Descriptive studies are used to
describe a population from a sample
Qualitative Sampling approaches
intensity, homogeneous, criterion, snowball, and random purposive
The Counseling Center has recently developed a questionnaire to measure marital satisfaction. Clients' responses are scored on a five-point rating scale, then summed for a total score. To ensure that the questions are all measuring the same construct, presumably marital satisfaction, Simone needs to know that the responses given on the set of items are similar. What psychometric property does Simone hope to establish?
internal consistency
purposive sampling (non-random)
involves using researchers' knowledge about the population to handpick sample members
Reliability of Treatment Implementation
is a concern when a treatment, program, or intervention is implemented differently for different participants in a study.
Reliability of Measurement
is a concern when the instrument used to measure the construct (thinking or behavior) of interest does not provide consistent scores or assessments.
Restriction of Range
is a problem when the scores obtained from a sample on a particular variable do not span the total possible range of values. Very often when this threat is plausible, so is confounding constructs with levels of constructs (see threats to construct validity).
systematic sampling
is sampling in which every Kth individual is selected from a list of all members in the population. K is a variable determined by dividing the number of individuals on the list by the number of subjects desired for the sample
cluster sampling
is sampling in which groups, not individuals, are randomly selected. Clusters can be communities, states, school districts, and so on.
Sampling
is the process of selecting a number of individuals for a study so that the individuals represent the larger group from which they were selected.
simple random sampling
is the process of selecting a sample so that all individuals in the defined population have an equal and independent chance of selection for the sample. It is the best way to obtain a representative sample, although representation of specific subgroups is not guaranteed
stratified sampling
is the process of strategically selecting a sample so that it guarantees desired representation of relevant subgroups within the sample
Confounding Constructs with Levels of Constructs
is when an inference is made about a construct based on a restricted range of levels of that construct (i.e. only high or low values of performance). Very often when this threat is plausible, so is a restriction of range (see threats to statistical conclusion validity).
Evaluation apprehension
is when participants try to provide more favorable responses because they are anxious about being evaluated
Inaccurate Effect Size Estimation
is when the statistic used to analyze the data tends to over- or underestimate treatment effects. This most often occurs when the sample size is extremely high or low, as many statistics are sample size dependent - that is they are easily influenced by the sample size. It can also occur when an inappropriate statistic is used, such as using a related samples (within-subjects) statistic for a between-subjects design.
Experimenter Expectancies
occur when experimenters have certain expectations about participants' thinking or behaviors, which may bias how they interpret participants' responses. This can be avoided by not telling the experimenter which condition participants are in.
Additive and Interactive Threats to Internal Validity
occur when the impact of a threat can be compounded by (or may depend on the level of) another threat.
Instrumentation
occurs when changes in the instrument used to measure responses provide different results for the outcome, appearing as if an effect occurred. This often occurs when observations are taken over an extended period of time and a measure is revised.
Violation of Statistical Tests
occurs when data violate assumptions of statistical tests that are used to assess the relationship between variables
Extraneous Variance in the Experimental Situation
occurs when distracters or events that are not part of the treatment, program or intervention occur during an experiment.
A researcher wants to know if alcohol consumption increases levels of aggression. She obtains a sample, of which, half are randomly assigned to the treatment condition and given alcohol and half are not. She then observes participants and gives them a questionnaire to measure levels of aggression. Aggression levels of the treatment group are then compared to the control group, and she finds that those who drank alcohol were indeed more aggressive than those who had not consumed alcohol. Which type of design was used in this study?
posttest only
guiding hypothesis
qualitative researchers that generate new hypotheses as a result of their studies rather than solidifying the intention of the research beforehand
Qualitative vs. Quantitative Research
qualitative: derives data from observation, interviews or verbal interactions and focuses on meaning and interpretation of participants. quantitative is based on collected objective data that can be subjected to statistical analysis
deductive reasoning
reasoning in which a conclusion is reached by stating a general principle and then applying that principle to a specific case (The sun rises every morning; therefore, the sun will rise on Tuesday morning.)
Quantitative studies
studies that involve a large number of cases, allowing the researcher to use statistical techniques to analyze the data
Problem Statement
the first item in the introduction to a research plan and final research report
Mono-Method Bias
when all operations or measures use the same method to assess a construct
Context Dependent Mediation
when an explanatory mediator of one causal relationship would not mediate a similar relationship in another context
Construct Confounding
when an operation or measure involves more than one construct. This frequently occurs when constructs are not clearly defined or tend to be strongly related to each other.
Novelty and Disruption Effects
when participants respond especially well or poorly to a new intervention, simply because it is different from their usual routine
Resentful Demoralization
when the control group resents not receiving a desirable treatment, program, or intervention and responds more negatively to outcomes or dependent variables.
Interaction of the Causal Relationship over Treatment Variations
when the results of a study cannot be generalized to a different type or part of a treatment, program or intervention.
Interaction of the Causal Relationship with Units
when the results of a study cannot be generalized to another group of participants.
Compensatory Rivalry
when the treatment, program, or intervention provides desirable services and the control attempts to find a similar treatment or services
Compensatory Equalization
when the treatment, program, or intervention provides desirable services, and an administrator or staff member provides similar services to compensate the control group.
Random sampling
◦Anyone in population has an equal chance of being in your sample ◦Theoretically ideal, but not practical
convience sampling
◦Anyone who is easily accessible to you University center Place where you work Classmates
Purposive
◦Specific group of interest African-American women studying political science Orphans between the ages of 3 and 12 ◦Screening for inclusion/exclusion criteria
