psyc 2140 Dr. Freida final CH 10 & 11

what is a latin square design?

-a formal system to ensure that every condition appears in each position at least once -another technique for partial counterbalancing

HOW is effect size measured?

-for an association: the correlation coefficient 'r' is used to evaluate the effect size of -for an experiment: the standardized effect size 'd' is used to measure the difference or overlap between means and scores within each group

how do you prevent a history, regression, and attrition threat

-have a comparison group

how do you prevent a maturation threat

-have a comparison group -see if the treatment group improved more than the control group, if they are the same then it is maturation threat

what impact can systematic variability have on an experiment

-it will destroy the internal validity of the experiment ***ONLY if a potential confound is correlated with the independent variable -can often make a manipulation appear to have an effect where there is actually none

what kind of problem is this an example of and how could you fix it: a researcher conducting a study on if money leads to happiness gives one participant $0.00, one $1.00 and one $2.00. The next day, there is no change in anyone's level of happiness

weak manipulation -try giving higher amounts of money; one gets $0.00, one gets $30.00, and one gets $150.00

what is a placebo group

when a control group is exposed to an inert treatment, such as a sugar pill

what is an instrumentation threat

when a measuring instrument changes over time

what is error variability

variability in DV scores that is due to factors other than the IV -ex) individual differences, measurement error, extraneous variation

how can you control for selection effects

use random assignment or matched groups

how do you prevent an instrumentation threat

-switch to a posttest-only design -collect data from each instrument to be sure that they all are calibrated the same -take steps to ensure that the pretest and posttest measures are equivalent -retrain coders throughout the experiment to establish reliability and validity at both pretest and posttest -use clear coding manuals

what are the three criteria for casual claims?

-temporal precedence -covariance -internal validity

describe a two group design

-the most basic experimental design -compare two means and use a t-test to see if the two means differ significantly -use a bar graph when drawing figures comparing two means

what can lead to ceiling and floor effects

-the result of a problematic IV -poorly designed DV

why do researchers use pretest/posttest design

-to demonstrate how random assignment made groups equal -to ensure that there are no selection effects -enables researchers to track people's change in performance over time

if an experiment is testing the effectiveness of a new medication, the researchers might assign some participants to take the medication and some participants to take a sugar pill --which is the treatment group and which is the placebo control group?

-treatment group: the medication -placebo group: sugar pill

how do you prevent a testing threat

-use a comparison group -abandon a pretest and just use a posttest-only design -use alternative forms of the test for the two measurements

when can selection effects happen

-when experimenters let participants choose which group they want to be in -when the kinds of participants in one level of the independent variable are systematically different from those in the other

what is partial counterbalancing? when is it used?

-when only some of the possible condition orders are represented to each participant -researchers present the conditions in a randomized order for every subject -used when the number of conditions increases, and therefore the number of possible orders increases too

what are the two types of two group designs? -give an example of each

1) one control group and one experimental group -IV = drug to treat depression -G1 receives a placebo or does NOT receive IV = control group -G2 receives 10 mg of IV = experimental group 2) both groups are experimental groups -IV = drug to treat anxiety -G1 receives 10 mg of IV -G2 receives 50 mg of IV

why does the posttest-only design satisfy all three criteria for causation

1- allows researchers to test for covariance by detecting difference in the DV 2- establishes temporal precedence and internal validity since the IV comes first in time 3- no design confounds or selection effects

what are the twelve threats to internal validity

1- design confounds 2- selection effects 3- order effects 4- maturation 5- history 6- regression 7- attrition 8- testing 9- instrumentation 10- observer bias 11- demand characteristics 12- placebo effects

what are the three main disadvantages of a within groups design

1- potential order effects than can threaten internal validity 2-might not be the most possible or practical 3- participants may change the way the normally act when exposed to all levels of the IV

what are selection effects

An influence on the probability that certain phenomena will be detected or selected, which can alter the outcome of a survey and makes it impossible to determine the true results

what is a double blind study

neither the participants nor the researchers/observers know who the treatment group is or who the comparison group is

what is weak manipulation

It is important to ask how the researcher operationalized the IV -ask about construct validity

TRUE OR FALSE: Independent groups design, between subjects design, random assignment design are all synonyms

TRUE

TRUE OR FALSE: a design confound has poor internal validity and can NOT support a casual claim

TRUE

TRUE OR FALSE: all experiments need a comparison group so the researchers can compare one condition to another

TRUE

TRUE OR FALSE: experiments are more powerful than non experimental designs

TRUE

TRUE OR FALSE: experiments control temporal precedence and internal validity

TRUE

TRUE OR FALSE: not every experiment has, or needs, a control group. often, a clear control group does not even exist

TRUE

TRUE OR FALSE: the more unsystematic variability there is within each group, the more the scores in the two groups overlap with each other

TRUE

TRUE OR FALSE: when data shows less variability with groups, the effect size will be larger and the mean differences are more likely to be statistically significant

TRUE

TRUE OR FALSE; within subjects design, repeated measures and correlated groups design are synonyms

TRUE

TRUE OR FALSE: the less within-group variability, the less likely it is to obscure a true group difference

TRUE ex) easier to detect four shakes of hot sauce in just tomatoes rather than salsa

what is a maturation threat

a change in the DV due to an internal change in the participant's behavior that emerges spontaneously over time -happens mostly in developmental studies

what are demand characteristics

a cue that can lead participants to guess an experiment's hypothesis, which might make them change their behavior and then creates an alternative explanation for a study's results -also known as an experimental demand

what is measurement error

a human or instrument factor that can inflate or deflate a person's true score on the DV -can be a reason for high within-group variability

what is a control group

a level of an independent variable that is intended to represent "no treatment" or a neutral condition

what is a history threat

a result from an external factor, beyond the researcher's control, that happens between the pre and posttest that systematically affects most members of the treatment group -therefore, it is unclear whether the change is caused by the treatment received r the external factor

what is a design confound

a second variable that happens to vary systematically along the intended independent variable and therefore is an alternative explanation for the results -an experimenter's mistake in designing the independent variable

what is a testing threat

a specific kind of order effect that refers to a change in the participants as a result of taking a test more than once -also includes practice effects

what is external validity

ability for an experiment to generalize to other people, places, and times -use random sampling to figure this out

define random assignment and why it is used

all participants have an equal chance of being assigned to any group -this distributes "differences" between individual subjects in both groups -helps control extraneous variables and biases so the experiment is unbiased -does NOT ALWAYS work, but usually does -ALWAYS use this for a true experiment

what is full counterbalancing ? when is it used?

all possible conditions are represented -used when within-groups experiment has only two or three levels of an IV

what is a floor effect

all the scores are clustered at the low end

what is a ceiling effect

all the scores are squeezed together at the high end

what is systematic variability

an anomaly or inaccuracy in observations which are the result of factors which are not under statistical control

what is a control variable?

any variable that an experimenter holds constant on purpose -important for establishing internal validity and eliminating confounds

what kind of threat is this an example of: the three healthiest participants withdraw from a study on metabolic syndrome OR when the rowdiest camper withdraws from a study on the whole cabin's behavior

attrition threat

why is a pretest-posttest design NOT a repeated measure design?

bc in a pretest-posttest design, participants see only ONE level of the IV while in a repeated measures design, participants see ALL levels of the IV

why do extraneous variables or confounds need to be controlled or gotten rid of?

bc they will make it impossible to determine the effect of the IV

how does adding more participants help

because it increases the chances of having a full representation of all the possible errors

why does too much within group variability make it harder to detect group differences

because too much variability can lead to noise

how do you avoid order effects and practice effects in a within-groups design

by using counterbalancing -half of participants get IV and the other half get placebo, then switch also measure participants on and off the IV, so they are their own controls

how do you avoid observer bias and demand characteristics

conduct a double-blind study

What is an independent groups design?

different groups of participants are placed into different levels of the independent variable -use random assignment to create independent groups -there are NO connections between subjects -also known as between subjects group design

what is unsystematic variability

essentially random variability within individuals and/or groups of individuals -NOT a confound

what kind of threat is this an example of: large clearance sale happens during a study on shopping behavior

history threat

what do you do if a large mount of variability exists due to individual differences

increase the sample size -the more people you measure, the less impact any single person will have on the group's average

what is the main difference between independent groups designs and within groups designs

independent group designs have DIFFERENT participants at EACH level of IV while within groups designs have SAME participants at ALL levels of IV

what kind of problem is this an example of and how could you fix it: an online reading game is supposed to improve reading scores by 2 points, but after participants take a simple pass/fail reading test, there is no change in their scores

insensitive measures -use a test that measures in between scores rather than just passing or failing -need to use detailed, quantitative increments when measuring DV

what kind of threat is this an example of: the way a researcher measures a variable changes over the course of an experiment

instrumentation threat

what two validities can observer bias threaten

internal validity construct validity

what impact can unsystematic variability have on an experiment

it can obscure the experiment's effects or true differences, but is not considered a confound

why are matched groups used

it has the advantage of randomness, so it prevents selection effects -also ensures that the groups are equal on some important variable before the manipulation of the IV

What does it mean if something is statistically significant?

it is unlikely to be obtained by chance from a population in which nothing is happening -can be confidence the difference between groups is NOT a fluke or drawn by chance

what does adding more participants to a study do?

it reduces the influence of individual differences WITHIN groups, thereby enhancing the study's ability to detect differences BETWEEN groups

when would independent be simpler and when would correlated be simpler

large # of participants = independent (more than 20 ppl) small # of participants = correlated (at least 5 needed)

define temporal precedence

maintains that the variable assumed to have the causal effect must precede the effect it is supposed to cause -the cause must come before the effect ex) depression (the effect) is a result of decreased/increased amount of sleep (the cause) -cause: sleeping less -effect: become depressed

what is the difference between random assignment and matching groups

matching groups require more time and resources than random assignment -matching groups are based on a variable and then randomly assigned while random assignment is completely random and is not based on anything

what kind of threat is this an example of: students in a study on grade performance naturally improve during the study

maturation threat

what is a pretest/posttest design

participants are randomly assigned to at least two different groups and are tested on the DV twice -once before exposure to IV and once after exposure to IV

how are matched groups created

participants are sorted from lowest to highest, or vice versa, on some variable and grouped into sets of two. Then, individuals within each set are assigned at random to the two experimental groups

what are the two basic forms of independent groups design

posttest-only design pretest/posttest design

what are confounds

potential threats to internal validity

what is a regression threat

refers to a statistical concept called regression to the mean -when a group average (mean) is unusually extreme at pretest and then less extreme/closer to its typical or average performance at posttest

what kind of threat is this an example of: Germany scores 7 goals in their first soccer game, and then score 1 in their second soccer game OR participants showed high stress scores at the pretest and then average stress scores at the posttest

regression threat

What are Cohen's guidelines for interpreting d scores which measures effect size?

small d (.20 - .50) = scores of participants in two experimental groups overlap more medium d (.50 - .80) large d (.80 - >) = IV caused the DV to change for more of the participants in the study

why are comparison groups so important

so researchers can compare one condition to another and see how the levels of the independent variables differ -all experiments need a comparison group ex) must have at least two different sizes of bowls to see which one makes you eat more

WHAT is effect size

statistical measure that conveys info concerning the magnitude of the effect produced by the IV -shows how much of an impact the IV has

what kind of threat is this an example of: students perform better on the posttest than the pretest OR participants change their answers on a racism survey after taking it a second time

testing threat

what is the difference between an instrumentation threat and a testing threat

testing: participants have changed from pre to post instrumentation: measuring instrument has changed from pre to post

describe a casual claim

the boldest kind of claim a scientist/researcher can make and can only be made or based on experiments -uses verbs such as: makes, influences, affects -when someone makes a statement about interventions and random assignment to treatment, random selection, and has an IV with at least two levels

what is counterbalancing

the levels of the IV are presented to participants in different sequences -any order effects should cancel each other out

what is a treatment group

the other level or levels of the independent variable when a study has a control group

what is a masked design/blind design

the participants know which group they are in, but the researchers/observers do NOT

what is a posttest-only design

the simplest independent-groups experimental designs -participants are randomly assigned to independent variable groups -one group receives IV and the comparison group receives placebo -then the dependent variable is tested

describe non experimental designs and give an example

they are descriptive, there is no control by researcher and no manipulation of independent variable -ex) surveys/questionnaires and observational studies that "measure" behavior

WHY do we measure effect size

to help evaluate the strength of the covariance (the difference) -the larger the effect size = the more important and stronger the casual effect is

define internal validity

when a researcher controls all extraneous variables and the only variable influencing the results of a study is the one being manipulated by the researcher -this rules out any confounds

what is a one group pretest posttest only design a bad research design -why is it bad?

when a researcher recruits ONE group of participants, measures them on a pretest, exposes them to a treatment, intervention or change, and then measures them on a posttest -bad bc it only has ONE group instead of two (so no comparison group)

what is observer bias

when a researcher's expectations influence their interpretation of the results -comparison threats can NOT control for observer bias

what are insensitive measures

when a study finds a null result because the researchers have not used an operationalization of the DV with enough sensitivity

TRUE OR FALSE:

when a study's result is statistically significant, it is not always the same as having a large effect size

define random selection

when all members of the population are equally likely to be chosen -impacts generalizability or external validity

what is a double-blind placebo control study

when neither the participants nor the researchers/observers know who is in the real group and who is in the placebo group

what is a placebo effect

when participants receive a fake treatment, but they still believe they improve or worsen because they believe it is a real, effective treatment

what is a null effect

when the IV does NOT make a difference in the DV; therefore there is NO significant covariance between the two -this is surprisingly common

what is an attrition threat

when there is a reduction in participant numbers because people drop out before the end -becomes a problem for internal validity when a certain participant that had a big influence on the results drops out

what is a within groups design

when there is only one group of participants and each participant is presented with ALL levels of the independent variable -usually, tested with no level of IV, then tested with a different level of the IV until they have been tested with all levels -be careful of practice effects -very powerful design

define covariance

when two factors have a relationship to each other and one changes, there should be a change seen in the other factor also, either positive or negative ex) distinct levels of the independent variable are associated with different levels of the dependent variable