Psychology 2911 Final
If you fail to find evidence supporting the rejection of the null hypothesis, what statement would you make? What could you NOT say?
"Fail to reject the null hypothesis" is the best answer. Should not say "accept the null hypothesis" or " retain the null hypothesis" because it implies that you believe the null hypothesis is true when, in fact, you usually have little evidence for this.
What are some examples of nonparametric statistical tests?
1) Chi-square 2) Sign test
What are the four outcomes in regards to the null hypothesis?
1) Don't reject the null hypothesis when, in reality, the null hypothesis is true. This is the correct decision (1-α). 2) Reject the null hypothesis when, in reality, the null hypothesis is true. This is a type one error (α). 3) Reject the null hypothesis when, in reality, the null hypothesis is false. This is the correct decision (1-β). 4) Don't reject the null hypothesis when, in reality, the null hypothesis is false. This is a type two error (β).
What are the steps of program evaluation?
1) Evaluation of need: Are there problems that need to be addressed in a target population? 2) Program theory: Determine how the problem will be addressed and how this approach will benefit the target population. 3) Program monitoring: Monitor program to determine whether is is reaching the target population, whether it is attracting enough clients, and whether the staff is providing the planned services. 4) Outcome evaluation (impact assessment): Are the intended outcomes of the program being realized? 5) Efficiency assessment: Was it worth it? Weigh the costs and benefits (including could resources be better used elsewhere).
After conducting an a priori power analysis, what can you do to increase the power if it is predicted to be too low?
1) Increase the strength of manipulation of independent variable. You will end up increasing the strength of the effect size. 2) Increase the number of participants. This is the most common approach. 3) Make changes to reduce variability (more trials per participant, more accurate measurement, more careful procedures, or a more homogeneous sample, if possible). This reduces the error.
What are some possible problems with repeated measures factorial ANOVA?
1) Order effects, which can be minimized through counterbalancing and randomized order. 2) It is not always possible. For example, somebody cannot be in groups for both old and young people.
What should the results of a distribution of sample means show?
1) Sample means should pile up near the population mean 2) pile of sample means should be similar to a normal distribution 3) The larger the sample size, the closer the sample means should be to μ.
What assumptions are made for confidence intervals?
1) Samples are representative of the populations from which they come. However, in reality, samples are not perfectly representative and there is some sampling error.
What assumptions are made for a chi-squared test?
1) Scores should be independent (category of one individual does not affect the category of another). 2) There are a minimum of five subject/events per cell. 3) The dependent variable is a frequency or count.
What are some examples of parametric statistical tests?
1) t-tests 2) F tests
What are the three main factors affecting an experiment's power?
1) the alpha level 2) The error in estimating the population parameter, including the reliability of the sample and the size of the sample. 3) the effect size
What is Sequential Design?
A combination of the longitudinal and cross-sectional methods. It take less time to complete than a longitudinal study, but allows you to examine for participant differences.
What is a sign test?
A conservative, nonparametric test that asks how many rather than how much. It can sometimes lack power because it ignore the magnitude of the difference.
What is a Reversal design (Withdrawal Design)?
A design that partially addresses the issue of whether treatment was the reason for the change in behaviour. You would test before the experiment for a baseline, measure during the experiment, and measure again after the experiment for a control. (ABA treatment)
What is a factorial design?
A design where subjects are randomly assigned to one of the conditions (between subjects design).
What is a one-group Pre-test Post-test design?
A one-group Post-test design with an added baseline measure to provide a basis for comparison. However, this also has a major problem: Results could be due to outside factors. There could also be a selection bias, as people are choosing to join the experiment.
What is the standard error of M?
A value that describes the distribution of sample means. It measures how well an individual sample mean represents the entire distribution, as well as how well it represents μ, the mean of the sampled population.
What is an issue with ABA design that does not exist for ABAB design?
ABA design may be unethical in some circumstances, in that is ends with the withdrawal of treatment. However, ABAB design allows you to end with treatment.
What is patching?
Adding control groups to a quasi-experimental design to rule out alternate explanations.
When would you conduct a post-hoc test on main effect in a Factorial ANOVA?
All three of these MUST be true: 1) Significant main effect 2) The independent variable has more than three levels 3) The interaction is not significant
What is a sensitivity analysis?
An analysis of the smallest effect size that can be detected for a certain power and sample size.
What is a Single Case Experimental Design (Single Subject/Single Participant Design)?
An experiment on only one subject. You would measure baseline, measure during experiment, and measure after experiment. The pre- and post-experimental measurements are controls. However, any change may not be strictly due to manipulation. Three types: 1) Reversal/Withdrawal ABA 2) Reversal/Withdrawal ABAB 3) Multiple Baseline
What is a Control Series Design?
An improved version of an interrupted time series design, but with an appropriate control group. This involves finding a similar population that did not receive a particular manipulation.
What is history?
Any event that occurs between the first and second measurements but is not part of the manipulation.
What is the relationship between n and σM?
As sample size (n) increases, standard error (σM) systematically decreases.
What is a standardized residual?
Basically post-hoc analysis in ANOVA designs, but for Chi-squared tests. They examine each cell and determines whether the cell makes a substantial contribution to the overall significance.
Why is a chi-squared test often called a "goodness of fit" measure?
Because it measures how well the observed frequencies fit the expected frequencies.
What is an issue with ABAB and ABA design? Which type of design solves this issue?
Both ABA and ABAB remove treatment, which may be unethical in some circumstances (alcoholism, for example). A solution is Multiple Baseline Design.
What is instrument decay?
Characteristics of the measuring instrument, or the way participants use it, change over time.
How are effect sizes related between independent t-tests and one-way ANOVAs?
Cohen's f is used as a measure of effect size for F, Cohen's d is used as a measure of effect size for t. For a small effect size, d = 0.2, f = 0.10. For a medium effect size, d = 0.5, f = 0.25. For a large effect size, d = 0.8 and f = 0.40. Therefore, f is always half of d for the same effect size.
What are the limitations of Single Case Experimental Designs? What are the benefits?
Con: Cannot be analyzed with inferential statistics (similar limitations as descriptive case studies). Pros: 1) Can be replicated 2) Can be informative
What is the difference in power for an independent t-test and a dependent t-test?
Dependent t-tests require far fewer people to obtain the same power as an independent t-test. Therefore, dependent samples t-tests have greater power than independent samples t-tests.
What is a Repeated Measures Factorial ANOVA?
Design where each participant serves in every level of every independent variable.
What is a Split-Plot/Mixed-Factorial ANOVA?
Design where one variable is between-subjects while another is within subjects.
What is a One-Group Post-test Only Design?
Design where you compare the responses of a number of individuals exposed to the same event. However, you do not have a control group, which is a major problem.
What is non-equivalent control group design?
Design with a separate control group. participants in the two conditions are not equivalent because they were not randomly assigned (because it was impossible due to a constraining factor). The differences become a confounding variable that provides an alternative explanation for the results.
What are developmental Designs?
Designs that study the ways that individuals change as a function of age. Three types: 1) Longitudinal 2) Cross-sectional 3) Sequential method
What is the most common example of random effects?
Drug dosages
What is an 'a priori' power analysis?
Estimating the likely power of an experiment before conducting the study. This allows you to revise the experiment to increase the power, if the predicted estimate is too low.
What is program evaluation?
Evaluating programs designed to achieve some positive effect, such as a new school curriculum, an anti-smoking campaign, or a healthy eating campaign. True experiments are usually not possible, therefore quasi-experimental research techniques are used.
What is Cross-sectional design?
Examining a group of subjects simultaneously in various stages of development (different ages). This is much cheaper than longitudinal design. However, it is possible that participants differ in ways other than age.
What is an Interrupted Time Series Design?
Experiment examining the dependent variable over an extended period of time, both before and after the independent variable is implemented. You need to worry about the regression to the mean in this design.
What is Multiple Baseline Design?
Experimental design where multiple measures are made before and after the manipulation. If the manipulation is effective, a change in behaviour will be immediately observed, and the change will continue to be reflected in further measures of the behaviour. This rules out the possibility that other events (chance, historical events, etc) were responsible. Can also be done across behaviours or across situations.
Under what circumstance is there a relationship between t and F?
F = t^2, when you do a t-test for independent groups and an one-way ANOVA (with only two groups). You get the same p value, as well.
Which of the assumptions about factorial ANOVA is the most critical?
Factorial ANOVA is robust against violations of normality and homogeneity of variance assumptions- generally, you will get the same decision if they have been violated. However, the assumption that the scores are independent is critical and should not be violated.
What is the Central Limit Theorem?
For any population with mean μ and standard distribution σ, the distribution of sample means for sample size n will have a mean of μ, a standard deviation of σ/(sqrt n), and will approach a normal distribution as n approaches infinity.
When is Split-Plot/Mixed-Factorial ANOVA uses?
Frequently used when multiple measurements are taken over time.
Under which circumstances will the distribution of sample means be nearly perfect (normal)?
If one of the following is true: 1) The population from which the samples are selected is a normal distribution 2) The number of scores (n) in each sample is relatively large (approximately 30 or more).
How should you phrase your significance findings?
If significant: Never say it is highly or very significant, or that it is marginally significant. It is a binary decision, no degrees of significance. If not significant: Never say the result is "nonsignificant". The phrase to use is "not significant".
How can error bars tell you if two means are significantly different?
If the error bars have no overlap, the means are significantly different.
What affect does the null hypothesis have on the effect size?
If the null hypothesis is true, the effect size is zero. If the null hypothesis is false, the effect size is greater than zero.
What is a benefit of using repeated measures factorial ANOVA?
It is efficient, as it requires fewer participants.
What is an exception to the normality assumption?
Measurements of time, including reaction time.
What has more power: parametric or nonparametric statistical tests?
Parametrical
Between point and interval estimates, which is the most precise? Which would you have more confidence in?
Point estimates are more precise, but you generally have more confidence in interval estimates. There is a trade-off between precision and confidence.
What does it mean when chi-squared is significant, but none of the standardized residuals are significant?
Post-hocs, in general, are conservative by nature.
What are testing effects?
Pretest can change behaviour.
How could you obtain estimates of effect size and standard deviation?
Review literature to find a similar study. Make sure to check for non-normal data (unequal n values, violation of assumptions, etc).
Why do we not set β = 0.05, when α = 0.05?
Setting β = α suggests that a type II error is just as bad as a type I error, when in actuality, a type I error is worse than a type II error. Therefore, we accept the increase risk of a Type II error relative to Type I.
What are nonparametric statistics?
Statistics used when data are nominal or when underlying distributions violate the assumptions of parametric tests.
What are parametric statistics?
Statistics where you assume that underlying distributions are assumed to be normal and can be described by means, standard deviations, etc.
What is Null Hypothesis Testing (NHT)?
Test that is used to determine the number of times that a confidence interval contains the unknown quantity.
What is the difference between Mixed-Factorial and other designs?
The between-subjects factor and within-subjects factor have different error terms.
What is the distribution of sample means?
The collection of sample means for all the possible random samples of a particular size (n) that can be obtained from a population.
What is internal validity?
The degree to which the independent variable causes changes to the dependent variable. Can be affected by history or maturation.
What is effect size?
The degree to which the phenomenon is present in the population. Or, the degree to which the null hypothesis is false.
Normality Assumption
The dependent variable should come from a population that is normally distributed.
What is the power of a repeated measures ANOVA?
The higher the correlation between the measures, the more power, and the lower the sample size.
What is the relationship between the value of the effect size and power?
The larger the effect size, the greater the degree to which the phenomenon is manifested, and the more power the study has (easier to find bigger things). The smaller the effect size, the lesser the degree to which the phenomenon is manifested, and the less power the study has (harder to find smaller things).
What is the relationship between confidence and sample size? Why?
The larger the sample size, the narrower the confidence interval. The smaller the sample size, the wider the interval. This is because a larger sample size has a smaller standard error, as well as a larger df and smaller t.
What is the differences with two-factor repeated measures factorial ANOVA?
The main effects and interaction each have a unique error term.
What is the Expected Value of M?
The mean of the distribution of sample means is equal to the population mean, μ. Or, in other words, the mean of the sampling distribution equals the mean of the sampled population.
What is power?
The probability of rejecting the null hypothesis when it is false (1-β). The probability of finding the result that we wish to find, when there is actually a result to find.
What assumptions are made for factorial ANOVA?
The same assumptions as one-way between subjects ANOVA: 1) The dependent variable should come from a population that is normally distributed (normality assumption) 2) Variances of the levels of the IVs should come from populations whose variances do not differ (homogeneity of Variance assumption) 3) Scores should be independent through all levels
What is longitudinal design?
The same group of people are observed at different points in time as they grow older. Issues: 1) Expensive and time consuming 2) Large drop-out (attrition) rate 3) Stuck with questions designed years ago 4) Possible testing effect
When conducting a sensitivity analysis, what happens to your effect size when you half the sample size, n?
The smallest effect size that can be detected increases. As in, the effect size must be greater in order for us to detect it.
What is regression toward the mean?
The tendency for people who receive particularly high or low scores on a measure to score closer to the mean on a subsequent testing. This happens because observed scores are often different from 'True Scores' due to error. This applies to both individuals and groups. If you retest any group of people, some people's scores will go up an some will go down. The issue with regression to the mean is that it can cause what looks like an effect, but is actually just a statistical phenomenon.
What is the relationship between confidence and precision?
The wider a confidence interval, the less precise it is. The narrower a confidence interval, the more precise it is.
What is non-equivalent control group pre-test/post-test design?
Two groups, one control and one experimental. Both groups have a pre-test and a post-test. There is no random assignment.
What is a 3x2 factorial design?
Two independent variables, where the first has three levels and the second has two levels. There are six total conditions.
How could you approximate the sign test?
Use a z score.
What is a power curve?
Used to determine sample size which maximizes power given any constraints, such as time or money.
What are quasi-experimental research techniques?
Used when control (eg, random assignment) of true experiments cannot be achieved. It has lower internal validity than true experiments and is only used when true experimentation is not possible. You cannot use causal language!! Six types: 1) One group post-test only 2) One-group pre-test post-test 3) Non-equivalent control group 4) Non-equivalent control group pre-test post-test 5) Interrupted Time Series 6) Control Series
What is an interval estimate?
Using a range of values as an estimate of an unknown quantity.
What is a point estimate?
Using a single number as an estimate of an unknown quantity.
When is a chi-squared test used?
Usually used to analyze frequency data. Analyzes whether an observed number differs from chance or differs from what was expected.
What is a random effect?
When levels of the independent variable are randomly chosen from all possible levels.
When would you not conduct a post-hoc test on a main effect in Factorial ANOVA?
When one or more of the following is true: 1) When main effect is not significant 2) When the independent variable has only two levels 3) When the interaction is significant
How does the value of Standard Error of M relate to the location of sample means?
When the Standard Error of M is small, all sample means are close together.
What is an interaction in factorial design?
When the effect of one variable is different across the levels of the other variable.
What is a fixed effect?
When the experimenter is interested in only those levels of the independent variable being tested and is not interested in generalizing to other possible levels.
When would you not conduct a post-hoc on interaction in a Factorial ANOVA?
When the interaction is not significant.
When would you conduct a post-hoc on interaction in a Factorial ANOVA?
When the interaction is significant.
What is an example of a time when would you want to predict power?
When you are planning a replication of an experiment.
What is a 2x2 design Chi-squared test?
When you are testing two factors, and asking if they are independent. IF they are independent, knowledge of a person's score on one factor does not let you predict score on the other factor. IF they are not independent, knowledge of a person's score on one factor allows you to predict score on the other factor.
What is an ABAB design?
Withdrawal design with an additional experimental period. This is an improved version of ABA design, as a single reversal is not the best evidence for the effectiveness of the treatment. The change in behaviour could be due to some random event that coincided with the experiment. Therefore, rule out alternative hypotheses by demonstrating treatment has the same effect a second time.
What sort of post-hoc or additional tests would you do after a Mixed-Factorial ANOVA?
You could do post-hoc analysis of main effect/interaction, or could do additional comparisons in ANOVA.
What should β be set to?
β = 0.20, where a power of 0.8 (1-β) reflects this.
