PSYC 217: Chapter 11/12

Outcome Evaluation

Are the intended outcomes of the program being realised?

Efficiency Assessment

Is the cost of the program worth the outcomes?

process evaluation / program monitoring

Is the program addressing the needs appropriately? Is it being implemented appropriately?

multiple correlation

a number of different variables may be related to a given behaviour. Used to combine a number of predictor variables to increase the accuracy of prediction of a given criterion or outcome variable. Is the correlation between a combined set of predicator variables and a single criterion variable.

mean

a set of scores obtained by adding all the scores and dividing by the number of scores.

quasi-experimental designs

address the need to study the effect of an independent variable in settings in which the control of true experimental designs cannot be achieved. Allows us to examine the impact of an independent variable on a dependent variable, but causal inference is much more difficult because they lack important features of true experiments such as control conditions and random assignments. Lower internal validity than do true experiment, only used when true experiments cannot be conducted.

overcoming threats to internal validity

addressed by the use of an appropriate control groups. A group that does not receive treatment provides adequate control for the effects of history, statistical regression etc. If the treatment group differs from the control group on the dependent measure administered after manipulation, the differences can be attributed to effect of the experimental manipulation.

Regression towards the mean

aka statistical regression, likely to occur whenever participants are selected to participate because they score extremely high or low on some variable. When they are tested again, their scores tend to change in the direction of the mean. Statistically speaking, extreme scores are likely to become less extreme over time, simply because they started out so extreme. Actually rooted in the reliability of the measure.

History

any event that occurs between the first and second measurements but is not part of the manipulation. Any such event is confounded with the manipulations. Can be caused by virtually any confounding events that occurs during or after the experimental manipulation, but before the posttest.

needs assessment

are there problems that needs to be addressed in a target population?

reversal designs

basic issue of how to determine that the manipulation has been done. Demonstrate the method can be undone.

comparing group percentages

calculate the percentage.

regression equations

calculations used to predict a person's score on one variable when that person's score on another variable is already known. Essentially prediction equations that are based on known information about the relationship.

3 ways of describing results

comparing group percentages, correlating individual scores and comparing group means

Sequential method

compromise between the longitudinal and cross sectional method. First phrase begins with the cross sectional method, study groups of * ages. These individuals are then studied using the longitudinal method with each individual tested at least one more. Takes fewer years to complete and the researchers reaps immediate rewards because data on the different age groups are available in the first year of study.

single case experiments

designs in this chapter offer systematic ways to examine hypotheses when limited by one or a few participants, yet because of this limitation, complex statistical analyses are often not possible. Suffer from the same limitations as descriptive case studies. Especially valuable for someone who is applying some treatment to help someone in particular improve their behaviour

cohort effects

differences among cohorts reflect different educational systems, different child rearing [ractices, music, political conditions etc. Difference among groups of different ages may reflect developmental age changes, alternative explanation.

non-equivalent control group design

employs a separate control group, but the participants in the two conditions- experimental group and the control group - are not equivalent because the participants were not randomly assigned. The differences become a confounding variable that provides an alternative explanation of the results.

ABAB design

experimental treatment is introduced a second time. Single reversal is not extremely powerful evidence. Second problem: does not seem right to end the design with the withdrawal of a treatment that may be beneficial.

Cohen's d

expresses effect size in terms of standard deviation units. D value of 1 tells you that the means are 1 standard deviation apart. Commonly used: .2 indicates a small effect, .5 moderate, .8 large effect.

Effect size

general term the refers to the strength of association between variables. Measured in many different ways: correlation coefficients and Cohen's d

threats to internal validity

history, maturation, testing, instrument decay, and regression to mean

program theory assessment

how will the problems be addressed? Will the proposed program actually address the needs appropriately?

Testing

if simply taking the pretest changes the participant's behaviour. Simply keeping track of smoking might be sufficient to cause a reduction in the number of cigs. Taking a pretest may sensitise people to the purpose of the experiment or make them more adept at a skill being tested.

restriction of range

important that the researcher sample from the full range of possible values. If range is restricted, the magnitude of the correlation coefficient is reduced. Will not get an accurate picture. With restricted range, there is restricted variability in the scores and thus less variability can be explained. Occurs when the individuals in sample are very similar or homogenous on the variable you are studying.

Control Series Design

improve the interrupted time series design is to find some kind of control group.

Effect size in two group experiments

in an experiment with two or more treatment conditions, common to calculate a value called cohen's d to describe the magnitude of the effect of the independent variable on the dependent variable.

frequency distributions

indicates the number of participants who receive or select each possible score on a variable.

Correlation coefficients

indicates the strength of the linear association between two variables. The values of this is from 0 to 1. When using the correlation: don't need to worry about the direction of relationship so plus/minus are not used. Provides us with a scale of values that is consistent across all types of studies. Sometimes preferred to see a squared value. This calculation changes the obtained r to proportion, which can be converted to a percentage.

Comparing group means

interested in comparing the mean number of aggressive acts by children in the two condition...

Single case experimental designs

known as single participant designs. Often seen in clinical, counselling, educational and other applied setting. Developed from a need to determine whether an experimental manipulation had an effect on a single research participant.

one group posttest only design

lacks a crucial element of a true experiment: a control or comparison group. Must be some sort of comparison condition and ideally, random assignment to conditions. Lacks internal validity. Not an experiment that will allow us to draw causal inferences about the effect of an independent variable on a dependent variable because results are open to many potential alternative interpretations. We do not know if the score on the dependent variable would have been equal, lower or even higher without the program. People conducted may have implicit idea of how a control group would perform, but fails to realise that the idea is insufficient evidence

standard deviation

measure of spread. Indicates the average deviation of scores from the mean. Derived by first calculating the variance. Set of scores is small when most people have scores close to the mean, becomes larger as more people have scores that lie further.

mode

most frequent score. Only measure of central tendency that is appropriate if a nominal scale was used. Indicates the most frequently occuring variable.

Interrupted Time Series Design

multiple pretests and multiple posttests instead of just one. Commonly used to examine effects of natural manipulations in society like the passing of laws. This allows for examination of over a period. Really a one-group pretest posttest design with all that design's problems of internal validity.

calculate partial correlation

need to have scores on the two primary variable of interest and the third variable that you want to examine. When calculated, you can compare the partial correlation with the original correlation to see if the third variable did have an effect.

5 types of questions that program evaluation seeks to answer?

needs assessment, program theory assessment, process evaluation, outcome evaluation, efficiency assessment

one group pretest posttest design

obtain a comparison is to measure participants before manipulation and after. Index of change could then be computed. Design has failed to take into account several alternative explanations, including, history, maturation, testing, instrument decay and gregression towards the mean. These alternative explanations are threats to the internal validity using this design

non-equivalent control group pretest-post test design

one of the most useful (internally valid) quasi-experimental designs, can never be as internally valid as a true experiment. Assignment to groups is not random. We can see whether the groups were the same on the pretest, we can look at the changes in scores from the pretest to the posttest. If IV had affect, the experimental group should show a greater change than the control group.

pie charts

particularly useful when representing nominal scale information. Frequently used in applied research reports and in newspaper...

Curvilinear relationship

pearson product correlation coefficient designed to only detect linear relationship.

cross sectional method

people at different ages are studied at only one point in time. More common because it is less expensive and immediately yields useful results. The researcher must infer that differences among age groups are due to the developmental variable of age. Developmental change is not observed directly among the same group of people, rather is based on comparisons among different cohorts of individuals.

maturation

people change over time. Brief period, they become bored, fatigued, wiser and hungrier. Longer periods: children become more coordinated and analytical. Any changes that occur systematically over time are called maturation effects. Could be a problem in the smoking reduction as people become more concerned about health as they get older. Any such time related factor might result in a change from the pretest to the posttest.

Graphing frequency distributions

pie charts, bar graphs, frequency polygons, histograms

partial correlation

provides a way of statistically controlling third variables in non-experiments. A correlation between the two variables of interest, with the influence of the third variable removed or partialed out of the original correlation. Provides an indication of what the correlation between the primary variables would be if the third variable were held constant. Not the same as actually keeping the variable constant, but it is a useful approximation.

ABA designs

requires that behaviour be observed and measured during baseline control, then treatment, then period after the experimental treatment has been removed. Eg. effect of reinforcement in child's academic performance.

program evaluation

research on programs that are proposed and implemented to achieve some positive effect on a group of individuals. Culture of evaluation where all such programs are honestly evaluated to determine whether they are effective.

partial correlation and the third variable problem

researchers face the third variable problem in non experimental research when some uncontrolled third variable may be responsible for the relationship between the two variable of interest.

problems of pearson product correlation coefficient

restriction of range, curvilinear relationship

across situations

same behaviour is measured in different settings, such as at home or at work. Manipulation is introduced at a different time in each setting, with the expectation that a change in the behaviour in each situation will occur only after the manipulation

Median

scores that divides the group in half. Abbreviated as Mdn. Appropriate when scores are on an ordinal scale because it takes into account only the rak order of the scores.

bar graphs

separate and distinct bar for each piece of information. Commonly used for comparing group means, also for group percentages.

across behaviour

several different behaviours of a single subject are measured over time. At different times, the same manipulation is applied to each of the behaviours.

range

simply the difference between the highest score and lowest.

squared correlation coefficient

sometimes referred as the percent of shared variance between the two variables.

Instrument Decay

sometimes, the basic characteristics of the measuring instrument, changes over time. Over time an observer may gain skill, become fatigued, or change the standards on which observations are based.

baseline

subject's behaviour is measured over time during the baseline. Manipulation is then introduced during treatment period, and the subject's behaviour continues to be observed.

central tendency

tells us what the sample as a whole or on the average is like. Three measures of central tendency - mean, median and mode.

across subjects

the behaviour of several subjects is measured over time, for each subject, though, the manipulation is introduced at a different point in time

selection differences

the differences become a confounding variable that provides an alternative explanation for the results. Usually occurs when participants who form the two groups in the experiment are chosen from existing natural groups. Problem can arise in this design even when the researcher apparently has successfully manipulated the independent variable using two similar groups.

Multiple baseline degisns

the effectiveness of the treatment is demonstrated when a behaviour changes only after the manipulation is introduced. TO demonstrate the effectiveness of the treatment, such a change must be observed under multiple circumstances to rule out the possibility that other events were responsible.

Longitudinal method

the same group of people is observed at different points in time as they grow older. ONly way to conclusively study changes that occur as people grow up. Best way to study how scores on a variable at one age are related to another variable at another age. Over the course, people may move, die or lose interest. Greater risk of high participant mortality rate

frequency polygons

use a line to represent frequencies. Most useful when the data represent interval or ratio scales as in the modelling and aggression data shown.

histograms

use bars to display a frequency distribution for a quantitave variable. Bars are drawn next to each other, reflects the fact that the variable on the x is measured in continous values.

multiple regression

used frequently to study basic research topics.

Varability

we can determine how much variability exists in a set of scores. Measure of variability is a number that characterises the amount of spread in a distribution of scores.

pearson product moment correlation coefficient

when both variables have interval or ratio scale properties. Provides information about the strength of the relationship and the direction of the relationship. To calculate: need to obtain pairs of observations from each subject. Each individual has 2 scores. Data can be visualised on a scatterplot.

correlating individual scores

when you do not have distinct groups of subjects. Instead, each participant is measured on two variables, and each variable has a range of numerical values. Purpose is to describe whether and how the variables relate to each other in non-experimental designs.