Principles of Research Design Final Exam
When the p value (alpha) is less than ______, it is said to be significant
.05
problems with behavioral observation
1. Natural behaviors must not be disrupted or influenced by the presence of the observer (can cause demand characteristics and reactivity) 2. Requires a level of objectivity by the observer that may not be possible. This type of measurement is based on an individual observer's judgment, which is subjective to a certain extent. (issues of reliability)
Survey research designs must solve four problems to get accurate and meaningful results
1. What questions to use 2. How to construct the survey 3. Recruit participants 4. How to administer the survey
Frequency method
1. count the instances of each specific behavior that occurs during an established period of time. Ex: how many aggressive behaviors in the 30-minute period
Interval Method
1. divide the established period of time into a series of intervals and record it if a specific behavior happens in each interval. Ex: 30-minute period is 30 1-minute intervals
The strength of the relationship in correlational data
1. measured and described by the numerical value obtained from the correlation coefficient. Range from -1.00 to +1.00. -1.00 and +1.00 indicate a perfectly consistent relationship, but this essentially never happens in behavioral sciences data.
Duration Method
1. record how much time an individual spends doing a specific behavior during an established period of time. Ex: 18 minutes of playing alone during the 30-minute period
2 x 3 x 2 factorial design
3 factors: Factor A has 2 levels Factor B has 3 levels Factor C has 2 levels Treatment Conditions (2x3x2) 12
Case History
: a case study when no treatment is administered to the individual
Case study design
: an in-depth study and detailed description of a single individual. Detailed description of observations and experiences during diagnosis and treatment
monotonic relationship in correlational data
: consistently one-directional, positive or negative, but the amount of increase or decrease varies Use a Spearman correlation to analyze monotonic relationships with ordinal variables.
linear relationship in correlational data
: data points cluster around a straight line Use a Pearson correlation to analyze linear relationships when the variables are interval/ratio
case study research
: describe a single individual in great detail
behavioral observation
: direct observation and systematic recording of behaviors Usually as the behaviors occur in a natural situation
rating scale questions
: participant selects a numerical value on a predetermined scale IE strongly agree to strongly disagree
Restricted Questions
: presents the participant with a limited number of responses · Multiple choice questions · Can include a free response blank (incorporating open-ended element) · Ordered set of responses o Range from "not at all," "once," "twice," "three or more"
correlational coefficient
: the numerical value that measures and describes the relationship between two variables The sign (+ or -) indicates the direction of the relationship The numerical value (0.0 to 1.0) indicates the strength of the relationship The type of correlation (Pearson or Spearman) indicates the form (shape)
bar graph is used when
: used when the measurement scale is nominal or ordinal The height of the bar indicates the frequency of each score Leave space between the bars
positive correlation
A correlation where as one variable increases, the other also increases, or as one decreases so does the other. Both variables move in the same direction.
combined strategy
A factorial study that combines two different research strategies, such as experimental and nonexperimental or quasi-experimental, in the same factorial design.
between subjects is best when
A lot of participants are available Individual differences are small or unlikely Order effects are likely to occur
Situations in which a factorial design is particularly useful:
Adding a factor to an existing study Using a participant variable to control variance in a between-subjects design Using the order of treatments as a second factor in a within-subjects design
Adding a factor to previous research creates a factorial design
Builds on previous research findings Can replicate results by showing the same main effect of treatment/no treatment Can expand on previous research by seeing if the effect exists in new situations or with different populations
Statistical analyses for factorial designs
Compute a mean for each treatment condition Use ANOVA to see if there is a difference between means If a two-factor design, use a two-factor ANOVA This will produce three F-ratios (what the ANOVA produces) Main effect for Factor A Main effect for Factor B Interaction effect
Advantages of survey design
Efficient way to gather large amounts of information Attitudes, opinions, personal characteristics, behaviors Instead of observing what people do, surveys just ask people what they do No waiting for the behavior to occur
Factorial designs: experimental vs nonexperimental
Experimental: all factors (IVs) can be manipulated variables Nonexperimental: one or more factors may be impossible or unethical to manipulate, so they are quasi-independent variables.
3 x 5 factorial design
Factor A: 3 levels Factor B: 5 levels # of treatment conditions (3 x 5) 15
Case study design approaches ideographic nomothetic
Idiographic approach: study of individuals Nomothetic approach: study of groups
Correlational significance
In a small sample, correlations can be very strong, but probably don't represent a real relationship in the population. In a large sample, correlations found are more likely to represent a real relationship. Note: Significant does not mean that the correlation is necessarily large!
Statistical analysis for factorial designs if both factors are between subjects: If both factors are within subjects: If mixed design (both between and within):
Independent measures two factor ANOVA repeated-measures two-factor ANOVA mixed-design two-factor ANOVA
Using a participant variable to control variance in a between-subjects design
Individual differences can become confounding variables in between-subjects designs Large variance can obscure significant differences between conditions The individual difference can become a new variable Ex: looking at two different learning methods, results may be affected by age so make age a second variable so it doesn't become a confounding variable This reduces the individual difference (age) in each group, but keeps the full age range Age differences still exists but they are differences between groups (becoming levels of a quasi-independent variable) rather than affecting the variance Variance is reduced without threatening external validity
Interval method balances
Interval: balances the disadvantage of the frequency and duration methods
Magnitude of Cohen's d
Magnitude of d Evaluation of Effect Size d = 0.2 Small effect d = 0.5 Medium effect d = 0.8 Large effect
Factorial design advantages
More realistic than looking at only one IV Behavior is influenced by a lot of things because life is messy! Seeing how variables in combination can influence behaviors Two variables acting together can create unique conditions different from either variable acting alone
Three Descriptive research designs
Observational research, survey research, case study research
Quanitifying observations
Observations must be turned into numerical scores to be useful. How? frequency method, duration method, interval method
Factorial designs can combine the elements of between subjects, within subjects, experimental and nonexperimental designs. What does this mean?
One factor may be between-subjects and another factor may be within-subjects One factor may be manipulated and the other factor may be naturally occurring groups
Purpose of statistical methods
Organize and summarize data
Applications of the Correlational Strategy
Prediction Measuring reliability and validity of measurement Evaluating theories
Type 1 error
Rejecting null hypothesis when it is true when the researcher finds evidence for a significant result when, in fact, there is no effect in the population Example: vaccine causes autism
quasi-experimental design
Research method similar to an experimental design except that it makes use of naturally occurring groups rather than randomly assigning subjects to groups. independent variable cannot be manipulated IE gender
statistics parameters
Statistic: a summary value that describes a sample Ex: an average score for a sample Describes or summarizes the entire set of scores for a sample Which, in turn, can tell us about the population the sample represents Parameters: a summary value that describes a population Ex: the average for the entire population
Strengths and Weaknesses of the Correlational Research Strategy
Strengths · Often used for preliminary work in a new area of research o Identify important variables o Describe variables o Describe relationships between variables · Discovering a relationship between two variables can lead to important experimental research to determine the cause-and-effect relationships between the variables · Used to study variables that are impossible or unethical to manipulate · Measures what exists naturally (high external validity) Weaknesses · Cannot determine causality, an unambiguous explanation for the relationship (low internal validity) · Third-variable problem: It is always possible that a third variable that has not been identified is controlling the two variables of interest and is responsible for producing the observed relationship. Ex: a positive correlation between ice cream sales and crime. · Directionality problem: a correlational study cannot determine which variable is the cause and which is the effect. Ex: a positive correlation between testosterone and aggression
Strength and weakness of case study design
Strengths of Case Study Design § Intense detail: wide variety of variables, events, responses o Things that would be overlooked or eliminated (controlled for) § A way to study special situations or unique variables § Convincing, personal, vivid Weaknesses of Case Study Design § Only describes (cannot account for relationships or causality) o No internal validity (no unambiguous explanations) § Weak external validity: reports on only a single individual § Bias from many sources is possible o Researcher: § Inclined to report successful or dramatic cases, not boring cases § Subjective interpretation: observations of the individual § Decides what information to include in the report o Client: § Exaggerate, minimize, lie, imagine events § Weaknesses are reduced by replication
pretest-posttest design
Test participants before and after receiving treatment
Inferential Statistics
The goal: to use the limited information gathered from a sample to form general conclusions about the population the sample represents General conclusion = inference (therefore, inferential statistics!)
Using graphs to compare groups of scores
Using graphs to compare groups of scores, the y-axis is the central tendency (mean, median, mode) · Line graph: use when the x-axis values are interval/ratio Bar graph: use when the x-axis values are nominal or ordinal
Likert Scale
a 5-point response scale presented with equal spacing between the different responses.
A between subjects factorial design means that each condition requires 2 x 4 x 3
a different group of participants 2 X 4 X 3 factorial design (three factors) Factor A: 2 levels Factor B: 4 levels Factor C: 3 levels (Rule of thumb: 30 participants per condition is a good number to find meaningful differences) How many conditions for a 2 X 4 X 3 between-subjects factorial design? 2 x 4 x 3 = 24 conditions 24 conditions with 30 people each = 720 people!
between-subjects design
a different group of subjects is tested under each condition each treatment condition is a different group of people
scatter plot
a graph that represents each individual with a single point. The horizontal coordinate is the individual's score on variable X The vertical coordinate is the individual's score on variable Y
Cohen's d
a measure of effect size that assesses the difference between two means in terms of standard deviation, not standard error measures the mean difference in terms of the standard deviation d = Sample mean difference Sample standard deviation
coefficient of determination
a measure of the amount of variation in the dependent variable about its mean that is explained by the regression equation r squared
effect size
a measure of the strength of the relationship between two variables or the extent of an experimental effect provides information about the absolute size of the treatment effect that is not influenced by outside factors
correlation coefficient
a statistical index of the relationship between two things (from -1 to +1) a numerical value calculated from the data gathered about two variables. Identified with the letter r. (just called a correlation
Central tendency
a statistical measure that identifies a single score that defines the center of a distribution. The goal: to identify the value that is most typical or representative of the whole group mean, median, mode
factorial design
a study in which there are two or more independent variables, or factors. IE: two factors, three factors, etc
response set
a tendency to respond to questions in a particular way that is unrelated to the content of the questions · when participants use the same response on all or most questions o May be due to not having strong feelings, wanting to get it over with, only disagree if it's serious disagreement
Descriptive research strategy
a type of research that is used to describe the characteristics of a population involves measuring a variable or set of variables. Purpose is to describe individual variables (no concern with relationships or causality) Useful for preliminary research: understanding a variable as it naturally exists forms a basis of knowledge for further research
histogram polygon
a) Histogram: the height of the bar indicates the frequency of each score Polygon: the height of the point indicates the frequency of each score
within-subjects design
an experimental design in which the same subjects are tested under each condition participants are exposed to all levels of the independent variable each treatment condition is experienced by one group of people in some sort of sequence/simultaneously
Mixed design
an experimental design that combines within-subjects and between-subjects methods of data collection
What does 2 x 3 indicate in factorial designs?
an experimental design that has two factors, one with levels, one with 3 levels
quasi independent variable
an independent variable that has levels that cannot be manipulated, as it is naturally occurring. IE age, gender, grade in school
Negative correlation
as one variable increases, the other decreases
Factorial designs can be _______-subjects or ________-subjects
between within
nonresponse bias
bias introduced to a sample when a large fraction of those sampled fails to respond : the people who complete surveys are a self-selected sample that may not be representative of the population
Use a chi-square hypothesis test when
both variables are nominal
Within-subjects factorial design: each condition is experienced
by the same group of individuals 2 X 4 X 3 factorial design requires 24 conditions (see above) How many participants for this study? 30! BIG disadvantage: each participant must be measured in 24 different treatment conditions! Extremely time-consuming, high likelihood of participant attrition, testing effects like fatigue and practice likely, counterbalancing extremely difficult
When to use a t-test
comparing difference of means in 2 groups. behavioral science
When to use Spearman's rho
correlation - ordinal data
habituation
decreasing responsiveness with repeated stimulation. get used to a smell get used to the presence of researchers
survey research
describe people's responses to questions about attitudes/behaviors
nominal
existing in name only
Higher-order factorial designs
factorial designs with three or more factors
Type 2 error
false negative when the sample data do not show evidence of a significant effect when, in fact, a real effect does exist in the population
In a study in which the independent variable has two levels and the dependent variable has two levels, how many treatment conditions are there?
four
observational research
gathering primary data by observing relevant people, actions, and situations describe observations as they occur in natural settings the researcher observes and systematically records the behavior of individuals for the purpose of describing behavior Uses behavioral observation simply for descriptive purposes
Duration method is good for Disadvantages of
good for specific behaviors Duration disadvantage: if each behavior is only 2 seconds long and occurs 30 times, the duration score would be 60 seconds (also not a great representation of the behavior)
Frequency method is good for Disadvantages of
good for specific behaviors Frequency disadvantage: if the individual does the behavior continuously for 30 minutes, that makes their frequency score 1 (not a great representation of the behavior) Ex: a bird singing for the whole 30 minutes
When to use Pearson
in correlational statistic
factor
independent variable
Open ended Questions Advantages and Disadvantages
introduces a topic and allows participants to respond in their own words. (What do you think about 8-week courses?) · Advantages: o Participant has the option for a lot of flexibility in their answer, which may lead to true thoughts and opinions · Disadvantages: o Different participants may understand the question differently, making it difficult to group answers, compare answers, or summarize answer o Difficult to analyze using statistical methods § Researchers may group by "generally positive" and "generally negative" o Limited by the participant's willingness or ability to express their thoughts May not capture the true breadth of their thoughts or opinions
Archival research
looking at historical records (archives) to measure behaviors or events that occurred in the past Ex: looking at public records like birth certificates, marriage records, telephone listings
Statistical significance
means that the result is unlikely to have been produced by chance or random variation
inter-rater reliability
measure of agreement among observers on how they record and classify a particular event
Percentage of variance (r2 and h2):
measure of effect size measures the percentage of variance for one variable that can be accounted for by knowing another variable.
Determine the number of treatment conditions by
multiplying the the levels for each factor, for instance an experiment with a factorial design of 2 x 3, has two variables, one with 2 levels, one with 3 levels. This means it has (2 x 3) 6 treatment conditions
Frequency distributions
o A table with two columns of information § The first column: the scale of measurement or the set of categories § The second column: the number of individuals (frequency) in each category o A graph § The x-axis: the scale of measurement or set of categories The y-axis: the frequencies
Advantages of rating scale questions
o Produce numerical values: use statistical procedures to compute means, compare scores, interpret results o Easy to understand and answer (not forced into yes or no) o Efficient and fast
Disadvantages of rating scale questions
o When participants produce a response set
When responding to rating scale questions participants tend to
o avoid the extreme scores at either end of the scale, which makes those two scores obsolete and reduces the scale by two § Opposite extremes are called anchors and generally labeled, midpoint is also labeled (especially if midpoint is neutral)
Primary advantage of factorial designs
observe how two variables effect an outcome and if they depend on each other
Increasing the sample size increases the likelihood of
of a significant result
descriptive statistics
statistical methods that help researchers organize, summarize, and simplify the results of a research study They describe the data Can be: graphs, tables, average scores
inferential statistics
statistical methods that use the results obtained from samples to help make generalizations about the population Research questions ask about populations, research studies try to find answers using a sample from the population. Inferential statistics help researcher know when it is appropriate to generalize from the sample to a population.
significant result means that
that it is extremely unlikely that the research result was obtained simply by chance.
What does a p value of less than .05 mean?
that there is less than a 5% probability that the results are caused by chance.
Predictor variable
the dependent variable in a correlational study that is used to predict the score on another variable
null hypothesis
the hypothesis that there is no significant difference between specified populations, any observed difference being due to sampling or experimental error.
interaction
the interplay that occurs when the effect of one factor (such as environment) depends on another factor (such as heredity) occurs whenever two factors, acting together, produce overall mean differences that are not explained by the main effects of the two factors. When the effects of one factor depend on the different levels of a second factor
Sampling error:
the naturally occurring difference between a sample statistic and the corresponding population parameter Basically: the sample is not a perfect representation of the population There will be some discrepancies between what we know about the sample and the true situation of the population. These discrepancies are called error. Each sample drawn from the population will also be somewhat different from all other samples drawn from the same population.
main effect
the overall effect of one independent variable on the dependent variable, averaging over the levels of the other independent variable
criterion variable
the variable whose value is being predicted
non parallel or intersecting lines of graphed data means
there is an interaction
Parallel lines of graphed data means
there is no interaction effect
Correlational research
two or more variables are measured to obtain a set of scores for each individual. The measurements are then examined to see if there is a relationship between the variables and to measure the strength of the relationship. Goal: to examine and describe the relationship between variables Data collected: each individual is measured for all the variables Variables are only measured (no manipulation or control like in an experiment) Correlational studies can demonstrate the existence of a relationship between two variables and the strength of that relationship. Cannot demonstrate cause and effect. Data in a correlational study Two or more scores for each individual Two variables are studied at a time (even if there are more variables measured in the study) Variable X and variable Y
Content analysis
using the techniques of behavioral observation to measure the occurrence of specific events in literature, movies, television shows, or other media that presents replicas of behaviors Ex: recording acts of aggression in television shows
Interviewer bias
§ : when the interviewer influences the natural responses in some way (tone of voice or rephrasing questions)
Applications of case study design
§ Studying individuals can complement and expand on theory § Rare phenomena and unusual clinical cases o Rare diagnoses like dissociative disorders o Brain injuries § New therapy methods or applications o Presenting the successful application of a new therapy Presenting an old therapy in a new area
alpha level
· ): the maximum probability that the research result was obtained simply by chance. o The goal of a hypothesis test is to rule out chance as the best explanation o Typical alpha level is p < .05, meaning that the test demands that there is less than a 5% probability that the results are caused by chance. o Researchers set an alpha level to determine if the results are significant.the probability level used by researchers to indicate the cutoff probability level (highest value) that allows them to reject the null hypothesis
how to address the observers subject judgement problem in behavior observation
· Behavior categories: before the observation begins specific behaviors are identified and categorized o Includes a list of categories of behaviors Ex: group play, play alone, aggression o Create a list of specific behaviors within each category Ex: aggression behaviors- hit, kick, push, fight, name calling, tease, etc. · Use well-trained observers o To establish reliability two or more observes are required · Use multiple observers to assess inter-rater reliability o The degree of agreement is computed § Compute a correlation between scores for the two observers § Compute a proportion of agreement ranging from 1 (perfect) to 0 (none)
how to address the presence of observer problems with behavioral observation
· Conceal the observer so the people don't know they are being watched o This is ethical for public behaviors in public places · Habituation: the participants become habituated to the observer's presence. o Repeated exposure until the observer's presence is no longer a novel stimulus
Disadvantages of Observational research
· Ethical concerns about spying on people Potentially violating a person's privacy and right to informed consent · Disadvantage of all descriptive research strategies: only describes behavior Cannot determine relationships between variables or causality
Advantages of observational research
· Major strength: observing actual behavior (survey research relies on participants' reports of their behavior) · Often have high external validity because most take place in field setting · Can observe antecedents (what may prompt a behavior), the specific behavior, and consequences of the behaviors
Constructing a survey guidelines
· Put demographic questions at the end (because they're boring) · Sensitive questions should be in the middle · Group questions together that deal with the same topic · Group questions together that have the same format · Format should be simple and uncluttered · Vocabulary and language should be easy to understand
Reliability and Validity:
· Reliability evaluates the consistency or stability of the measurements · Validity evaluates the extent to which the measurement procedure actually measures what it claims to be measuring · Both reliability and validity are often defined using correlations o Test-retest reliability: computing a correlation between two sets of measurements. A consistent relationship indicates that the measurement procedure is reliable. o Concurrent validity: computing a correlation between the scores on a new test and the scores on an accepted, old test to show that they are measuring the same thing. A strong positive relationship would indicate high concurrent validity.
When the scores are numerical values and you've computed a mean
· Standard deviation: describes how the scores are scattered around the mean o Measures the average distance from the mean of the scores § When the scores are close to the mean: small standard deviation § When the scores scattered widely: large standard deviation o General rule about SD: approximately 70% of the scores in a distribution are within a distance of one standard deviation of the mean, 95% within two · Variance: measures the variability of the scores by computing the average squared distance from the mean o The average squared difference from the mean o Part of this calculation is making an adjustment called the degrees of freedom. This helps the sample variance better represent the population variance. To describe nominal and ordinal data report the proportion or percentage in each category. Ex: 20% of the UM-Flint students are psychology majors. Mode is also useful (identifies the most commonly occurring category)
Interpreting the Strength of a Relationship
· Value The closer to -1.00 or +1.00, the stronger the relationship. 0.0 indicates no relationship. · Coefficient of determination: the value of the correlation squared (r2) · Significance of the correlation
standard error
· the average distance between a sample statistic and the corresponding population parameter o Important because it tells you how much difference is reasonable to expect between a statistic and a parameter No sample can perfectly represent its population, so the standard error helps you know what to expect
