Research Methods 3

Partial counterbalancing

only some of the possible condition orders of a within groups experiment are represented. One way to do this is to present the conditions in a randomized order for each subject.

cross-leg correlations

our primary interest. What researchers are most interested in. They show whether the earlier measure of one variable is associated with the later measure of the other variable. Helps investigate how people change overtime, therefore establish temporal precedence.

An effect size in which d

.2= small or week .5=medium or moderate .8= larger or strong

one group, pretest/posttest designs threats to internal validity MATURATION

-A change that emerges more or less spontaneously over time. Changes not happen from outside intervention, they just happen. -EX: Adapting to an unfamiliar environment -prevention: inclue comparison group.

one group, pretest/posttest designs threats to internal validity REGRESSION OR REGRESSION TO THE MEAN

-Extreme scores when re-measured will often "regress" to the average of performance -Random factors that combine to produce unusual results -prevention: including a comparison group and analysis of extreme scores

Construct Validity and Theory Testing

-For construct validity, the operationalizations of both the independent and dependent variables are assessed -The standard is typically the theory that the researcher is testing, the standard for evaluating these operational variables is provided by the theory the study is testing. -construct validity allows researchers to say that the results of the study support their theory.

one group, pretest/posttest designs threats to internal validity TESTING

-Order effect where participants get better or worse just by being exposed to the measurement several times -Solution: comparison group or post-test only design

Demand Characteristics-internal validity threat that can apply to any experiment

-Participants change their behavior because they guess what the study is about -Solution - Use double-blind studies: both participant and researcher are blind to who is the treatment and comparison group.

one group, pretest/posttest designs threats to internal validity HISTORY

-When a historical event occurs to everyone in the treatment at the same time. -EX: the weather could have changed and impacted kilowatt use Prevention: include comparison group

observer bias- internal validity threat that can apply to any experiment

-When a researchers expectations influence their interpretation of the results -Solution - Use blind studies- or masked design: keeping observer unknown of who is the treatment and comparison group

Placebo Effects- internal validity threat that can apply to any experiment

-When people who receive a treatment really improve but only because they think that they are receiving a valid treatment -use a double blind placebo control study-neither observer nor participant know who is receiving the real treatment

Why conduct experiments

-Would like to understand as much as we can about the nature of a relationship between two variables so that we can possible influence some outcome. -We can design interventions and treatments

Why you can't do experiments for longitudinal designs

-ethics- unethical to assign people to certain conditions-i.e. smoking. -practicality- can't randomly assign preferences, such as a preference for violent TV. people either like what they like or don't.

Statistical validity of experiments

-how well do the data support your causal conclusion -are the results statistically significant:does covariance exist (as A changes B changes) between the variables in the population from which the sample was drawn?--shows result was not done by chance -what is the effect size: like r coefficient, but experiments use d. When d is larger it usually means the independent variable caused the dependent variable to change for more of the participants in the study. When d is smaller it usually means the scores of participants in the two experimental groups overlap more.

Construct Validity of experiments

-need to ask about the construct validity of both the independent and dependent variables. -dependent: is the measure a good representation of the construct of interest -Independent: how well the researchers manipulated them. -manipulation checks: researchers use this to collect empirical data on the construct validity of their independent variables.an extra dependent variable that a researcher can insert into an experiment to help them quantify how well an experimental manipulation worked -pilot study: A simple study using a separate group of participants, that is completed before (or sometimes after) conducting the study of primary interest. Usually before a full blown study to test a manipulation.

selection effects

-occurs in an experiment when the kinds of participants in one level of the independent variable are systematically different from those in the other. -can occur when the experimenters let participants choose which group they want to be in, or if the experimenters assign one type of person to one condition (all women) and another type of person to another condition (all men). -study on autism where parents could choose to treat their autistic child with their standard treatment, or treat them with the new 40 hour per week treatment. showed clear selection effect-families willing to do the 40 hour per week treatment are probably systematically different than the treatment as usual group.

Pretest/posttest design

-participants are randomly assigned to at least two groups and are tested on the key dependent variable twice- once before and once after exposure to the independent variable. EX: giving the group the anagram test and seeing their performance, then exposure to red, then seeing their anagram performance. -might use to evaluate whether random assignment made groups equal. -also works well to track how participants in the experimental groups have changed over time in response to some manipulation

one group, pretest/posttest designs threats to internal validity ATTRITION

-people drop out of the study before it ends -problem when one type of participant systematically leave- the most dramatic scores. EX: the most rambunctious camper. -Prevention- eliminate them completely from the experiment.

Which design is better? posttest only design or pretest/posttest design?

-posstest design may be the most basic type of independent groups experiment. But random assignment & a manipulated variable can lead to powerful causal conclusions. -the pretest/posttest design adds a pretesting step, might use this if they want to be extra sure that the two groups were equivalent at pretesting-- this is possible only if the pretest does not make the participants change their more spontaneous behavior.

Null Results Reasons: not enough variation between levels

-weak manipulation: the level of the IV was not enough to change the DV -insensitive measures: measurement level not sensitive enough to pick up the difference. EX: asking someone who hates spicy to food to choose the better spicy salsa- he would say they are both horrible. -Ceiling and floor Effects: either all results high or all results low-- are effects of weak manipulations and insensitive measures. -manipulation checks: (adding separate dependent variable to make sure the manipulation worked) help detect weak manipulations and ceilings and floors. -confounds acting in reverse: confound counteracts some true effect of an independent variable--instead of providing an untrue result, it shows no result at all

Phrases used to show regression in popular press

1. "controlled for": most common sign of regression analysis. The things the study controlled for are the predictor variables. 2. "Taking into account": even when they controlled for multiple variables they still found a relationship between the original dependent variable and predictor. 3. "Correcting for": ex. After correcting for factors that may affect scores, etc.

Multivariate Designs and the Four Validities

1. Construct: How well each variable is measured. 2. External: to whom or what can we generalize? 3.Internal: what other third variables could you control for? 4.Statistical: How strong and significant are the effects.

3 rules for causation in longitudinal studies

1. Covariance: significant relationships help establish covariance. When 2 variables are significantly correlated there is covariance. 2. Temporal precedence: Because each variable is measured in at least two different points in time, they know which one came first. comparing the strength of the cross-lag correlations-- the stronger path helps determine which came first. 3. Internal Validity: Can measure a third variable to rule it out, for example gender. To rule out the potential variables as being a cause, measure it with the study. In the TV example they measure if boys reacted the same as girls.

Adding more predictors to a regression helps answer 2 questions

1. Helps control for several third variables at the same time 2. Examining all of the beta weights helps us get a better understanding of which factors most strongly affect the variable we are trying to explain

Within groups designs disadvantages

1. order effects, which can threaten internal validity-- fixed with counterbalancing. 2. Within groups design might not be possible- two methods of training someone to ride a bike, cannot un-teach someone how to ride a bike, therefore can not expose them to all levels of the independent variable. 3. People can change their behavior depending on exposure to previous conditions. When people see all levels of the independent variable and then change the way they act. **demand characteristic: When an experiment contains cues that lead participants to guess its hypotheses, the experiment has demand characteristics. If demand characteristics are high they may create an alternative explanation for a studys results.- participants try to act like "good participants"

internal validity in experiments

Are there alternative explanations for the outcome? -This is the most important validity for experiments! -Three fundamental internal validity questions: 1. Did the design ensure that there were no design confounds? 2. If an independent-groups design was used, were selection effects controlled for by random assignment or matching? 3. If a within-groups design was used were order effects controlled for by counterbalancing?

TV violence and aggression study

Both a longitudinal and multivariate study. Collected data on 875 children in 1960 measuring aggression and interest in violent TV. Found 427 of them 10 years later and measured the same variables. It is longitudinal because the researchers measured the same variables in the same group of people across time- 10 years apart. It is a Multivariate correlational study because four variables were measured-- both aggression and interest in violent tv at time 1 and time 2

getting at causality with patter and parsimony

Cigarette and cancer example. A pattern of results is best explained by a parsimonious causal explanation. The best explanation that requires the fewest exceptions or qualifications.

How to design independent variables to show covariance.

Control group: level of independent variable that represents the neutral or no treatment group. not every experiment needs or has a control group. Treatment group: level of IV that receive the treatment. Comparison group: the levels of independent variables differ in some intended and meaningful way. All experiments need a comparison group, to compare one condition to another, but the comparison group does not have to be a control group. Placebo group: the control group is exposed to an inert treatment such as a sugar pill.

Three causal criteria

Covariance: A goes with B Temporal precedence: A comes before B Internal Validity: A is the only thing that could have caused B

interaction effect: two independent variables

Does the effect of the original independent variable (cell use) depend on another independent variable ( age) -mathematically: is there a difference in the difference?

Steps in studying 3 or more variables with multiple regression designs

First choose the criterion variable (dependent variable)- the one we are interested in understanding or prediction ex: behavior problems The rest of the variables are the predictor variables (independent variables)- ex: amount of recess time, amount of students eligible for free lunch, amount of boys in the classroom, etc.-- All of the possibilities that could cause behavior problems.

Covariance, temporal precedence and internal validity in within-groups designs.

Good for covariance and temporal precedence. Do have potential for a particular threat to internal validity- order effects.

experiments and covariance

Is the causal variable related to the effect variable? Are the levels of the independent variables associated with distinct levels of the dependent variables. Covariance was explained by the difference in means of the two groups in the Elliot study

Of the three causal criteria which do bivariate correlations show and not show

It shows covariance. It does not show temporal precedence or internal validity- no control for third variable

Mediators Vs. Moderators

Mediators: Why are the two variables linked? Why? Moderators: Are these two variables linked the same way for everyone or in every situation? For whom or when?

Internal Validity in Experiments

Most important. Allows us to rule out other explanations for the change in the dependent variable.

Is the Pretest/Posttest a within-groups design?

No, true within groups designs expose each participant to every level of the independent variable. Pretest/postest -DV->IV level A-> DV Within groups designs IV level A->DV->IV level B->DV

Does regression establish causation?

No. It helps with internal validity, but it can not establish temporal precedence and cannot control for unmeasured variables.

Temporal precedence in experiments

The researcher is able to control which variable comes first. By manipulating the independent variable, the experimenter virtually ensures that the cause comes before the effect. Better than correlational study where the variables are measured at the same time.

Mediators Vs. Third Variables

Third variable seen more of a nuisance- might not be of central interest to researchers. The proposed third variable is external to the two variables in the original bivariate correlation. --> ex. if family income really were a third variable that is responsible for the recess/behavior relationship, recess and behavior are correlated with each other only because each one is correlated separately with family income. Mediator: When a mediator is proposed they are interested in which aspect of the causal variable is responsible for that relationship. A mediator variable is internal to the causal variable. ex. physical activity is an important aspect of recess-- the one responsible for reducing behavior problems.

one group, pretest/posttest designs threats to internal validity INSTRUMENTATION

a measuring instrument changes overtime. Ex: people observing the behavior become more strict. Also, different forms for the pretest and posttests that are not sufficiently equivalent. Prevention: post test only; calibration of the instruments.

Design confounds

an experimenters mistake in designing the independent variable; it is a second variable that happens to vary systematically along with the independent variable, therefore is an alternative explanation for the results. Ex: if in the elliot study he had given the red ink group a more difficult test than the other groups, the difficulty of the test would have been a design confound.

multiple regression (multivariate regression)

can help rule out some third variables. can help address questions of internal validity. By conducting a multivariate design, researchers can evaluate whether a relationship between two key variables still holds when they control for another variable.

ruling out third variables with multiple regression designs

correlational studies have a third variable problem-- something else in the environment could be causing the relationship between the two variables. To prevent this, measuring other variables controls for their impact.

Experiments meet three causal rules

covariance, temporal precedence, internal validity

measured variable

dependent variable. take the form of records of behavior or attitudes, such as self reports, behavioral observations, or physiological measures.

Independent Group designs

different groups of participants are assigned to different levels of the independent variables. Also called between subjects and between groups. EX: Different bowl size or different color ink.

situation noise: too much variability between levels

external distractions of any kind- is a third factor that can cause variability within groups and obscure true group differences. External factor that causes the dependent variable to change.

Measurement error: too much variability between levels

factors that inflate or deflate a true score on a dependent measure. prevention: use reliable measures, and test more than once

Control variables

help constant on purpose so they cannot explain the change in the dependent variable. Allow researchers to separate one potential cause from another, eliminating alternative explanations for the results. They are important for establishing internal validity.

full counterbalancing

in a within groups experiment when it only has two or three levels of an independent variable this is used. It is when all possible condition orders are represented.

Simple experiments pros and cons

independent groups designs: require more people, no contamination across independent variable levels Within groups designs: require fewer people, individuals serve as their own control, potential order effects, change of experimental demand. **power: the ability of a study to show a statistically significant result when an independent variable truly has an effect in the population. "like being at a noisy party"

multivariate designs

involving more than two measured variables.

longitudinal design

measures the same variables in the same people at different points in time.-- this provides temporal precedence. Often used in developmental psychology to study changes in a trait or an ability as a person grows older. This is also adapted to test causal claims.

Beta with Multiple- Regression Designs

one beta value for each predictor--similar to r no quick guideline for beta to indicate effect sizes that are weak macerate or strong. Betas change depending on what other predictor variables are being used-- being controlled for-- in the regression.

Comparison group

part of covariance. It asks the question, "Compared to what?" It separates an experiment from just using our own judgement. The independent variable has to have different levels in order to establish covariance. If there is nothing to compare it to it does not tell you anything. -Outcome matters: if there is no difference in the outcomes of the levels of the independent variables, there would be no covariance.

Posttest only design

participants are randomly assigned to independent variable groups and are tested on the dependent variable once

Confounds

potential threats to internal validity.

Cross-sectional correlations

test to see whether two variables, at the same point in time, are correlated. With the TV violence example it would be measuring the correlation between TV violence and aggression in 3rd grade and then again measuring that same correlation in 13th grade. The arrow in the model points both ways because there is no way to tell which variable came first.

autocorrelations

the association of each variable with itself overtime. They determine the correlation of one variable with itself, measured at two different occasions. In the model example, the arrow points one way because you know which variable becomes first in time.

parsimony

the degree to which a good scientific theory provides the simplest explanation of some phenomenon.

manipulated variable

the independent variable. the variable that is being controlled.

experiment

the researchers manipulated at least one variable and measured another.

Within- group designs

there is only one group of participants and each person is presented with all levels of the independent variable. Also called within subjects design. EX: if all the participants had to take the anagram with each ink color.

Matched groups

to prevent selection effects and the faults of random assignment use a matched group. Ex: measure everyones IQ, then place the three top people in three different group, and so on and so forth. disadvantage is that it requires an extra step.

External validity of experiments

to whom or to what can the causal claim generalize? -generalizing to other people: random sampling -generalizing to other situations: takes more than 1 study to generalize

counterbalancing

trying to avoid order effects. This is when researchers present the levels of the independent variables to participants in different orders. This should cancel out any order effects.

Practice effects

type of order effect. also known as fatigue effect. a long sequence might lead participants to get better at the task, or get tired or bored towards the end.

carryover effects

type of order effect. some form of contamination carries over from one condition to the next. EX: brushing your teeth then taking a sip of orange juice- the first fast effects the taste of the second.

concurrent measures

type of within group designs. participants are exposed to all the levels of an independent variable at roughly the same time, and a single attitudinal or behavioral preference is the dependent variable. EX: babies shown both female and male faces, measured which one they stared at the longest.

Repeated measures

type of within groups design. participants are measured on the dependent variable more than one time- after exposure to each level of the independent variable. EX: having a mother interact with her own toddler, then measuring her oxytocin level, then having her interact with a new toddler, then measuring her oxytocin again.

noise: too much variability between levels

unsystematic variability within each group, also called error variance. --can cause null result

Parsimony in the popular press

usually only reports the results of the latest study- only showing a part of the scientific process. do not describe the context of the research -make readers believe that scientists conduct single, unconnected studies at a whim. not a good scientific report.

Unsystematic Variability

variability that is random across conditions. Does not harm internal validity but can cause other issues. Ex: everyone serving the pasta in all the groups had different attitudes.

Mediator

variables that explain the relationship between two other variables.

p value for beta

when p is less than .05 the beta is considered statistically significant. If it is greater than .05 it is not significant (meaning we can not conclude beta is different from zero)

systematic variability

when the variability is uneven across groups. Ex: If students in the red ink group all happened to be bad at anagrams, but the students in the green and black groups were really good at them, it would vary systematically with the ink color. harms internal validity

Order effects

within groups potential threat to internal validity. Sometimes being exposed to one condition changes how participants react to the other condition. They happen when exposure to one level of the independent variable influences responses to the next level of the independent variable. This is a confound in a within groups design-participant performance at later levels of the independent variable might be caused not by the experiment manipulation but rather by the sequence in which the conditions were experienced.