psyc 2140 Dr. Freida final CH 10 & 11
what is a latin square design?
-a formal system to ensure that every condition appears in each position at least once -another technique for partial counterbalancing
HOW is effect size measured?
-for an association: the correlation coefficient 'r' is used to evaluate the effect size of -for an experiment: the standardized effect size 'd' is used to measure the difference or overlap between means and scores within each group
how do you prevent a history, regression, and attrition threat
-have a comparison group
how do you prevent a maturation threat
-have a comparison group -see if the treatment group improved more than the control group, if they are the same then it is maturation threat
what impact can systematic variability have on an experiment
-it will destroy the internal validity of the experiment ***ONLY if a potential confound is correlated with the independent variable -can often make a manipulation appear to have an effect where there is actually none
what kind of problem is this an example of and how could you fix it: a researcher conducting a study on if money leads to happiness gives one participant $0.00, one $1.00 and one $2.00. The next day, there is no change in anyone's level of happiness
weak manipulation -try giving higher amounts of money; one gets $0.00, one gets $30.00, and one gets $150.00
what is a placebo group
when a control group is exposed to an inert treatment, such as a sugar pill
what is an instrumentation threat
when a measuring instrument changes over time
what is error variability
variability in DV scores that is due to factors other than the IV -ex) individual differences, measurement error, extraneous variation
how can you control for selection effects
use random assignment or matched groups
how do you prevent an instrumentation threat
-switch to a posttest-only design -collect data from each instrument to be sure that they all are calibrated the same -take steps to ensure that the pretest and posttest measures are equivalent -retrain coders throughout the experiment to establish reliability and validity at both pretest and posttest -use clear coding manuals
what are the three criteria for casual claims?
-temporal precedence -covariance -internal validity
describe a two group design
-the most basic experimental design -compare two means and use a t-test to see if the two means differ significantly -use a bar graph when drawing figures comparing two means
what can lead to ceiling and floor effects
-the result of a problematic IV -poorly designed DV
why do researchers use pretest/posttest design
-to demonstrate how random assignment made groups equal -to ensure that there are no selection effects -enables researchers to track people's change in performance over time
if an experiment is testing the effectiveness of a new medication, the researchers might assign some participants to take the medication and some participants to take a sugar pill --which is the treatment group and which is the placebo control group?
-treatment group: the medication -placebo group: sugar pill
how do you prevent a testing threat
-use a comparison group -abandon a pretest and just use a posttest-only design -use alternative forms of the test for the two measurements
when can selection effects happen
-when experimenters let participants choose which group they want to be in -when the kinds of participants in one level of the independent variable are systematically different from those in the other
what is partial counterbalancing? when is it used?
-when only some of the possible condition orders are represented to each participant -researchers present the conditions in a randomized order for every subject -used when the number of conditions increases, and therefore the number of possible orders increases too
what are the two types of two group designs? -give an example of each
1) one control group and one experimental group -IV = drug to treat depression -G1 receives a placebo or does NOT receive IV = control group -G2 receives 10 mg of IV = experimental group 2) both groups are experimental groups -IV = drug to treat anxiety -G1 receives 10 mg of IV -G2 receives 50 mg of IV
why does the posttest-only design satisfy all three criteria for causation
1- allows researchers to test for covariance by detecting difference in the DV 2- establishes temporal precedence and internal validity since the IV comes first in time 3- no design confounds or selection effects
what are the twelve threats to internal validity
1- design confounds 2- selection effects 3- order effects 4- maturation 5- history 6- regression 7- attrition 8- testing 9- instrumentation 10- observer bias 11- demand characteristics 12- placebo effects
what are the three main disadvantages of a within groups design
1- potential order effects than can threaten internal validity 2-might not be the most possible or practical 3- participants may change the way the normally act when exposed to all levels of the IV
what are selection effects
An influence on the probability that certain phenomena will be detected or selected, which can alter the outcome of a survey and makes it impossible to determine the true results
what is a double blind study
neither the participants nor the researchers/observers know who the treatment group is or who the comparison group is
what is weak manipulation
It is important to ask how the researcher operationalized the IV -ask about construct validity
TRUE OR FALSE: Independent groups design, between subjects design, random assignment design are all synonyms
TRUE
TRUE OR FALSE: a design confound has poor internal validity and can NOT support a casual claim
TRUE
TRUE OR FALSE: all experiments need a comparison group so the researchers can compare one condition to another
TRUE
TRUE OR FALSE: experiments are more powerful than non experimental designs
TRUE
TRUE OR FALSE: experiments control temporal precedence and internal validity
TRUE
TRUE OR FALSE: not every experiment has, or needs, a control group. often, a clear control group does not even exist
TRUE
TRUE OR FALSE: the more unsystematic variability there is within each group, the more the scores in the two groups overlap with each other
TRUE
TRUE OR FALSE: when data shows less variability with groups, the effect size will be larger and the mean differences are more likely to be statistically significant
TRUE
TRUE OR FALSE; within subjects design, repeated measures and correlated groups design are synonyms
TRUE
TRUE OR FALSE: the less within-group variability, the less likely it is to obscure a true group difference
TRUE ex) easier to detect four shakes of hot sauce in just tomatoes rather than salsa
what is a maturation threat
a change in the DV due to an internal change in the participant's behavior that emerges spontaneously over time -happens mostly in developmental studies
what are demand characteristics
a cue that can lead participants to guess an experiment's hypothesis, which might make them change their behavior and then creates an alternative explanation for a study's results -also known as an experimental demand
what is measurement error
a human or instrument factor that can inflate or deflate a person's true score on the DV -can be a reason for high within-group variability
what is a control group
a level of an independent variable that is intended to represent "no treatment" or a neutral condition
what is a history threat
a result from an external factor, beyond the researcher's control, that happens between the pre and posttest that systematically affects most members of the treatment group -therefore, it is unclear whether the change is caused by the treatment received r the external factor
what is a design confound
a second variable that happens to vary systematically along the intended independent variable and therefore is an alternative explanation for the results -an experimenter's mistake in designing the independent variable
what is a testing threat
a specific kind of order effect that refers to a change in the participants as a result of taking a test more than once -also includes practice effects
what is external validity
ability for an experiment to generalize to other people, places, and times -use random sampling to figure this out
define random assignment and why it is used
all participants have an equal chance of being assigned to any group -this distributes "differences" between individual subjects in both groups -helps control extraneous variables and biases so the experiment is unbiased -does NOT ALWAYS work, but usually does -ALWAYS use this for a true experiment
what is full counterbalancing ? when is it used?
all possible conditions are represented -used when within-groups experiment has only two or three levels of an IV
what is a floor effect
all the scores are clustered at the low end
what is a ceiling effect
all the scores are squeezed together at the high end
what is systematic variability
an anomaly or inaccuracy in observations which are the result of factors which are not under statistical control
what is a control variable?
any variable that an experimenter holds constant on purpose -important for establishing internal validity and eliminating confounds
what kind of threat is this an example of: the three healthiest participants withdraw from a study on metabolic syndrome OR when the rowdiest camper withdraws from a study on the whole cabin's behavior
attrition threat
why is a pretest-posttest design NOT a repeated measure design?
bc in a pretest-posttest design, participants see only ONE level of the IV while in a repeated measures design, participants see ALL levels of the IV
why do extraneous variables or confounds need to be controlled or gotten rid of?
bc they will make it impossible to determine the effect of the IV
how does adding more participants help
because it increases the chances of having a full representation of all the possible errors
why does too much within group variability make it harder to detect group differences
because too much variability can lead to noise
how do you avoid order effects and practice effects in a within-groups design
by using counterbalancing -half of participants get IV and the other half get placebo, then switch also measure participants on and off the IV, so they are their own controls
how do you avoid observer bias and demand characteristics
conduct a double-blind study
What is an independent groups design?
different groups of participants are placed into different levels of the independent variable -use random assignment to create independent groups -there are NO connections between subjects -also known as between subjects group design
what is unsystematic variability
essentially random variability within individuals and/or groups of individuals -NOT a confound
what kind of threat is this an example of: large clearance sale happens during a study on shopping behavior
history threat
what do you do if a large mount of variability exists due to individual differences
increase the sample size -the more people you measure, the less impact any single person will have on the group's average
what is the main difference between independent groups designs and within groups designs
independent group designs have DIFFERENT participants at EACH level of IV while within groups designs have SAME participants at ALL levels of IV
what kind of problem is this an example of and how could you fix it: an online reading game is supposed to improve reading scores by 2 points, but after participants take a simple pass/fail reading test, there is no change in their scores
insensitive measures -use a test that measures in between scores rather than just passing or failing -need to use detailed, quantitative increments when measuring DV
what kind of threat is this an example of: the way a researcher measures a variable changes over the course of an experiment
instrumentation threat
what two validities can observer bias threaten
internal validity construct validity
what impact can unsystematic variability have on an experiment
it can obscure the experiment's effects or true differences, but is not considered a confound
why are matched groups used
it has the advantage of randomness, so it prevents selection effects -also ensures that the groups are equal on some important variable before the manipulation of the IV
What does it mean if something is statistically significant?
it is unlikely to be obtained by chance from a population in which nothing is happening -can be confidence the difference between groups is NOT a fluke or drawn by chance
what does adding more participants to a study do?
it reduces the influence of individual differences WITHIN groups, thereby enhancing the study's ability to detect differences BETWEEN groups
when would independent be simpler and when would correlated be simpler
large # of participants = independent (more than 20 ppl) small # of participants = correlated (at least 5 needed)
define temporal precedence
maintains that the variable assumed to have the causal effect must precede the effect it is supposed to cause -the cause must come before the effect ex) depression (the effect) is a result of decreased/increased amount of sleep (the cause) -cause: sleeping less -effect: become depressed
what is the difference between random assignment and matching groups
matching groups require more time and resources than random assignment -matching groups are based on a variable and then randomly assigned while random assignment is completely random and is not based on anything
what kind of threat is this an example of: students in a study on grade performance naturally improve during the study
maturation threat
what is a pretest/posttest design
participants are randomly assigned to at least two different groups and are tested on the DV twice -once before exposure to IV and once after exposure to IV
how are matched groups created
participants are sorted from lowest to highest, or vice versa, on some variable and grouped into sets of two. Then, individuals within each set are assigned at random to the two experimental groups
what are the two basic forms of independent groups design
posttest-only design pretest/posttest design
what are confounds
potential threats to internal validity
what is a regression threat
refers to a statistical concept called regression to the mean -when a group average (mean) is unusually extreme at pretest and then less extreme/closer to its typical or average performance at posttest
what kind of threat is this an example of: Germany scores 7 goals in their first soccer game, and then score 1 in their second soccer game OR participants showed high stress scores at the pretest and then average stress scores at the posttest
regression threat
What are Cohen's guidelines for interpreting d scores which measures effect size?
small d (.20 - .50) = scores of participants in two experimental groups overlap more medium d (.50 - .80) large d (.80 - >) = IV caused the DV to change for more of the participants in the study
why are comparison groups so important
so researchers can compare one condition to another and see how the levels of the independent variables differ -all experiments need a comparison group ex) must have at least two different sizes of bowls to see which one makes you eat more
WHAT is effect size
statistical measure that conveys info concerning the magnitude of the effect produced by the IV -shows how much of an impact the IV has
what kind of threat is this an example of: students perform better on the posttest than the pretest OR participants change their answers on a racism survey after taking it a second time
testing threat
what is the difference between an instrumentation threat and a testing threat
testing: participants have changed from pre to post instrumentation: measuring instrument has changed from pre to post
describe a casual claim
the boldest kind of claim a scientist/researcher can make and can only be made or based on experiments -uses verbs such as: makes, influences, affects -when someone makes a statement about interventions and random assignment to treatment, random selection, and has an IV with at least two levels
what is counterbalancing
the levels of the IV are presented to participants in different sequences -any order effects should cancel each other out
what is a treatment group
the other level or levels of the independent variable when a study has a control group
what is a masked design/blind design
the participants know which group they are in, but the researchers/observers do NOT
what is a posttest-only design
the simplest independent-groups experimental designs -participants are randomly assigned to independent variable groups -one group receives IV and the comparison group receives placebo -then the dependent variable is tested
describe non experimental designs and give an example
they are descriptive, there is no control by researcher and no manipulation of independent variable -ex) surveys/questionnaires and observational studies that "measure" behavior
WHY do we measure effect size
to help evaluate the strength of the covariance (the difference) -the larger the effect size = the more important and stronger the casual effect is
define internal validity
when a researcher controls all extraneous variables and the only variable influencing the results of a study is the one being manipulated by the researcher -this rules out any confounds
what is a one group pretest posttest only design a bad research design -why is it bad?
when a researcher recruits ONE group of participants, measures them on a pretest, exposes them to a treatment, intervention or change, and then measures them on a posttest -bad bc it only has ONE group instead of two (so no comparison group)
what is observer bias
when a researcher's expectations influence their interpretation of the results -comparison threats can NOT control for observer bias
what are insensitive measures
when a study finds a null result because the researchers have not used an operationalization of the DV with enough sensitivity
TRUE OR FALSE:
when a study's result is statistically significant, it is not always the same as having a large effect size
define random selection
when all members of the population are equally likely to be chosen -impacts generalizability or external validity
what is a double-blind placebo control study
when neither the participants nor the researchers/observers know who is in the real group and who is in the placebo group
what is a placebo effect
when participants receive a fake treatment, but they still believe they improve or worsen because they believe it is a real, effective treatment
what is a null effect
when the IV does NOT make a difference in the DV; therefore there is NO significant covariance between the two -this is surprisingly common
what is an attrition threat
when there is a reduction in participant numbers because people drop out before the end -becomes a problem for internal validity when a certain participant that had a big influence on the results drops out
what is a within groups design
when there is only one group of participants and each participant is presented with ALL levels of the independent variable -usually, tested with no level of IV, then tested with a different level of the IV until they have been tested with all levels -be careful of practice effects -very powerful design
define covariance
when two factors have a relationship to each other and one changes, there should be a change seen in the other factor also, either positive or negative ex) distinct levels of the independent variable are associated with different levels of the dependent variable