NCE Research and Program Evaluation A

The study that would best rule out chance factors would have a significance level of P =

.001 (as compared to .01, .08, .05). the smaller the value for P the more stringent the level of significance. Here, the .001 level is the most stringent level listed, indicating that there is only one chance in 1000 results are due to chance. It is easier to get significant results using .08, .05, or .01, that it is using .001.

Which level of significance would best rule out chance factors?

.001. Some researchers refer to the level of significance as to where "one draws the line" or the "cut off point" between findings that should or should not be ascribed to chance factors. The significance level must be set low. If, for example, a researcher foolishly set the level at .50, then the odds would be 50-50 that the results were due to pure chance.

in the social sciences the accepted probability level is usually:

.05 or less. .05 and .01 are the two most popular levels of significance.

a causal comparative design

A true experiment--such as biofeedback research where the groups are randomly assigned. Data gleaned from the causal comparative ex-post facto or after-the-fact design can be analyzed with a test of significance (e.g., a "t" test or ANOVA) just like any true experiment.

basic research vs applied research

Basic research is conducted to advance our understanding of theory, while applied research (also called action research or experience-near research) is conducted to advance our knowledge of how theories, skills, and techniques can be used in terms of practical application.

Experimenters should always abide by a code of ethics. The variable you manipulate/control in an experiment is:

IV or independent variable. ["I am the researcher so I manipulate or experiment with the IV."] Now, in any experiment the counselor researcher is guided by ethics: This suggests 1st, that subjects are informed of any risks; 2nd, that negative after effects are removed; 3rd, that you will allow subjects to withdraw at any time; 4th, that confidentiality of subjects will be protected; 5th, that research report results will be presented in an accurate format that is not misleading; and 6th, that you will use only techniques that you are trained in. Research is considered a necessary factor for professionalism in counseling.

an analysis of variance or ANOVA

If there are more than 2 groups, then the analysis of variance is utilized. The results of an ANOVA yield an F-statistic. The researcher then consults an "F" table for a critical value of "F." If "F" obtained (i.e., computed) exceeds the critical "F" value in the table, then the null hypothesis is rejected.

inferential statistics

In order to compare 2 groups, "inferential statistics," which infer something about the population, are necessary.

quasi-experiment

In this experiment the researcher uses pre-existing groups, and hence the IV (independent variable) cannot be altered (e.g., gender or ethnicity). In a quasi-experiment you cannot state with any degree of statistical confidence that the IV caused the DV (dependent variable). One popular type of quasi-experiment is known as the "ex post facto study."

Hypothesis testing is closely related to the work of:

R. A. Fisher. A hypothesis is a hunch or an educated guess which can be tested utilizing the experimental model. A hypothesis might be that biofeedback raises board exam score; or that reality therapy reduces dysfunctional classroom behavior and high school students; or perhaps that cognitive therapy relieves depression in males in the midst of a divorce. A hypothesis is a statement which can be tested regarding the relationship of the IV and the DV.

Assume the biofeedback experiment is conducted. the results indicate that the biofeedback helped raise written board exam scores but in reality this is not the case. The researcher has made a:

Type I error. First, write down the null hypothesis regarding the experiment in question. In this null would mean that biofeedback did not raise board exam scores. This question tells you that the experimental results revealed that biofeedback did raise board scores, so you will reject the null hypothesis. The question then goes on to say that in reality biofeedback did not really cause the results. Therefore, you have rejected null when it is true/applicable. This is the definition of a Type I or alpha error. Since the experimenter sets the alpha level, s/he is always cognizant of the probability of making a Type I error.

internal and external validity

When conducting or perusing a research study a counselor is very concerned with "internal and external validity." Threats to internal validity include maturation of subjects (psychological and physical changes including fatigue due to the time involved), mortality (i.e., subjects withdrawing), instruments used to measure the behavior or trait, and statistical regression (i.e., the notion that extremely high or low scores would move toward the mean if the measure is utilized again). Internal validity refers to whether the DVs were truly influenced by the experimental IVs or whether other factors had an impact. External validity, on the other hand, refers to whether the experimental research results can be generalized to larger populations (i.e., other people, settings, or conditions). Thus, if the results of the study only apply to the population in the study itself then the external validity is said to be low.

percentile rank

a descriptive statistic that tells the counselor what percentage of the cases fell below a certain level. Hence, if Joe's score puts him at the 50th percentile, then 50% of the people had raw scores lower than his particular score. Do not confuse percentiles with percentage scores. A percentage score is just another way of stating a raw score. The percentage score of 50 could be a very high, a very low, or an average score on the test. It merely says that the examinee got half the answers correct. Graphically speaking, the distribution of percentile scores will always appear rectangular and flat.

chi-square

a nonparametric statistical measure that tests whether a distribution differ significantly from an expected theoretical distribution.

meta-analysis

a study that analyzes the findings of numerous studies. Hence, a study of reality therapy that looked at the results of 20 reality therapy studies would be a meta-analysis.

From a purely statistical standpoint, in order to compare a control group (which does not receive the IV or experimental manipulation) to the experimental group the researcher will need:

a test of significance.

If a researcher changes the significance level from .05 to .001, then:

alpha errors decrease; however, beta errors increase.

Type I and type II errors are called ___ and ___ respectively.

alpha; beta. A Type I (alpha error) occurs when a researcher rejects the null hypothesis when it is true; and a Type II error (beta error) occurs when you accept null when it is false. The probability of committing a Type I error equals the level of significance mentioned earlier. Therefore, the level of significance is often referred to as the "alpha level." 1 minus beta is called "the power of the statistical test." In this respect, power connotes a statistical test's ability to reject correctly a false null hypothesis.

an alternative hypothesis (or affirmative hypothesis):

asserts that the IV has indeed caused a change

A professor of counselor education hypothesized that biofeedback training could reduce anxiety and improve the average score on written board exams. If this professor decides to conduct a formal experiment the IV will be the ___, and the DV will be the ___.

biofeedback; board exam score. [Hint: "I manipulate... or I experiment with....The "I" statement gives you your "IV;" remembered DV begins with a "d" like "data."]

To complete a "t" tasks you would consult a table value of "t." In order to see if significant differences exist in an ANOVA you would:

consult a table for "F" values. More elaborate tests (e.g., Tukey's, Duncan's multiple range, and Scheffe's test) can determine whether a significant difference exists between specific groups. Group comparison test such as these are called "post hoc" or "a posteriori" tests for ANOVA calculations.

In order for the professor of counselor education to conduct an experiment regarding his hypothesis he will need a ___and a ___.

control group; experimental group. the control group and the experimental group both have the same characteristics except that members of the control group will not have the experimental treatment applied to them. The control group does not receive the IV. The experimental group receives the IV. The basic presupposition is that the averages (or means) of the groups do not differ significantly at the beginning of the experiment. Remember that if you cannot randomly assign the subjects to the 2 groups then your exam will consider the research a quasi-experiment. Most experts suggest that you need at least 30 people to conduct a true experiment. Correlational research requires 30 subjects per variable while a survey should include at least 100 people.

a correlation coefficient (or correlational research):

does not make use of the paradigm in which an IV is experimentally introduced

the Chi-Square nonparametric test

examines whether a change frequencies differ significantly from expected frequencies. (Note: Statisticians have created nonparametric tests that parallel the popular parametric measures.)

parametric tests

have more power than nonparametric statistical tests. Parametric tests are used only with interval and ratio data.

In experimental terminology IV stands for___and DV stands for___.

independent variable; dependent variable. Variables in an experiment are categorized as independent variables (IVs) or dependent variables (DVs). A variable is merely a behavior or a circumstance that can exist on at least 2 levels or conditions; a factor that "varies" or is capable of change. In an experiment the IV is the variable that the researcher manipulates, controls, alters, or wishes to experiment with. The DV expresses the data regarding factors you wish to measure. IVs and DVs--the variables of the experimental trade--can be discrete (e.g., a brand of counseling or occupation) or continuous (e.g., height or weight).

zero correlation

indicates an absence of a relationship between the variables in question.

biserial correlation

indicates that one variable is continuous (i.e., measured using an interval scale) while the other is dichotomous. An example would be evident if you decided to correlate state licensing exam scores to NCC status (here the dichotomy is licensed/unlicensed). If both variables are dichotomous (i.e., two-valued) then a phicoefficient correlation is necessary. Imagine a researcher who wants to correlate NCC status with CCMHC status or perhaps gender with certification status (has certification/does not have certification).

Occam's Razor suggests that experimenters:

interpret the results in the simplest manner. Exams often refer to parsimony as Occam's Razor, the principle of economy, or Lloyd Morgan's 1894 Canon (canon in this sense means "law"). Conway Lloyd Morgan was an English psychologist/physiologist, while William of Occam was a 14th century philosopher and theologian. The early behaviorists (e.g., Watson) adhered closely to this principle. NOTE: Have you ever placed a sticker on your car and tried to smooth it out? No matter how many times you attempt to do this the sticker usually retains a few trapped air bubbles. This analogy has often been used in conjunction with research, in the sense that flaws in research are often called bubbles.

Experiments emphasize parsimony, which means:

interpreting the results in the simplest way. Parsimonious literally means a tendency to be miserly and to not overspend. A parsimonious individual is said to be overly economical and stingy. In research, we strive for parsimony in the sense that the easiest and less complex explanation is said to be the best; an economical description if you will. Simply put, the simplest explanation of the findings is always preferred. The factor analysis is parsimonious in the sense that 10 tests which measure the dimensions of an effective counselor can be explained via a short measure which describes 3 underlying variables. Factor analysis then, is concerned with data reduction.

ethnographic research

involves research that is collected via interviews, observations, and inspection of documents

A counselor educator is running an experiment to test a new form of counseling. Unbeknownst to the experimenter one of the clients the study is secretly seeing a Gestalt therapist. This experiment:

is confounded/flawed. The experiment is said to be invalid due to an extraneous independent variable (IV) (e.g., the gestalt therapy). Variables which are undesirable confound or "flaw" the experiment. The only experimental variable should be the independent variable - in this case the new form of counseling. The IV must have the effect on the dependent variable (here the DV would be some measure of the client's mental health). In this experiment any change could not be attributed with any degree of certainty to the new form of counseling since dependent variable changes could be due to the gestalt intervention (an extraneous confounding variable). All correlational research is said to be confounded.

the Spearman correlation or Kendalls' tau

is used in place of the Pearson "r" when parametric assumptions cannot be utilized

the Kruskal-Wallis

is used instead of the one-way ANOVA when the data are nonparametric

A Type II error:

it is also called a beta error; it means you accept null when it is false. Although lowering the significance level (e.g., .01 to .001) lowers Type I errors, it "raises" the risk of committing a Type II or beta error. simply think of the Type I/Type II relationship as a seasaw in the sense that when one goes up the other goes down. Hence, in determining an alpha level, the researcher needs to decide which error results in the most serious consequences. The safest bet is to set alpha at a very stringent level and then use a large sample size. If this can be accomplished, it is possible to make the correct decision (i.e., accept or reject null) the majority of the time.

ex post facto

literally means "after the fact," connoting a correlational study or research in which intact, pre-existing groups are utilized. In the case of the ex post facto study, the IV was administered before the research began.

descriptive statistics

merely describes data (e.g., the mean, the median, or the mode).

an organismic variable

one the researcher cannot control yet exists such as height, weight, or gender. To determine whether an organismic IV exists you simply ask yourself if there is an experimental variable being examined which you cannot manipulate. in most cases, when you are confronted with IV/DV identification questions, that IV will be of the "manipulated variety."

Nondirective is to person-centered as:

parsimony is to Occam's Razor (both roughly synonymous). Important exam reminder: Most counselors see themselves as practitioners rather than researchers. Research, nevertheless, helps the entire field of counseling advance. It has been pointed out that we know about the work of many famous counselors and career counselors because of their published research not because of what transpired in their sessions.

If data indicate that students who study a lot get very high scores on state counselor licensing exams, then the correlation between study time and LPC exam scores would be:

positive. A positive correlation is evident when both variables change in the same direction. A negative correlation is evident when the variables are inversely associated; one goes up and the other goes down. In this case the relationship is positive since as study time increases, LPC exam scores also increase.

When you see the letter "P" in relation to a test of significance it means:

probability or the level of significance. Traditionally, the probability in social science research (often indicated by a "P") has been set at .05 or lower (i.e., 01 or .001). The .05 level indicates that differences would occur via chance only 5 times in 100. The significance level must be set before the experiment begins.

The American Psychological Association's "Journal of Counseling Psychology:"

publishes more counseling research articles than any other periodical in our field.

A counselor educator decides to increase the sample size of her experiment. This will:

reduce Type I and Type II errors. Raising the size of the sample helps to lower the risk of chance/error factors. Differences revealed via large samples are more likely to be genuine than differences revealed using a small sample size.

factor analysis

refers to statistical procedures that use the important or underlying "factors" in an attempt to summarize a lot of variables. Hence, a test which measures a counselor's ability may try to describe the 3 most important variables (factors) that make an effective helper, although literally hundreds of factors may exist. Using factor analysis procedures, a brief test that measures the 3 major factors may be able to predict who will be an effective counselor as accurately as 10 other tests that examine hundreds of variables or so-called factors.

a test of significance

such statistical tests are used to determine whether a difference in the group scores is "significant" or just due to chance factors. In this case a "t" test would be used to determine if a significant difference between 2 means exists. This has been called the "two-groups" or "two-randomized-groups" research design. In this study, the 2 groups were independent of each other in the sense that the change (or lack of it) in one group did not influence the other group. Thus, this is known as an "independent group comparison to sign." If the researcher had measured the same group of subjects without the IV and with the IV, then the study would be a "repeated-measures comparison design."

the experimental or alternative hypothesis (the hunch):

suggests that a difference will be evident between the control group and the experimental group (i.e., the group receiving the IV).

the null hypothesis

suggests that there will not be a significant difference between the experimental group which received the IV and the control group which did not. The null hypothesis asserts that the samples will not change (i.e., they will still be the same) even after the experimental variable is applied. The null hypothesis is simply that the IV does not affect the DV.

parameter

technically a value obtained from a population, while a statistic as a value drawn from a sample. A parameter summarizes a characteristic of a population (e.g., the average male's height is 5'11").

the analysis of covariance or ANCOVA

tests 2 or more groups while controlling for extraneous variables that are often called "covariates"

The assertiveness researcher now attempts a more complex experiment. One group receives no assertiveness training, a 2nd group receives 4 assertiveness training sessions, and a 3rd receives 6 sessions. The statistic of choice would be:

the ANOVA, since the simple ANOVA or one-way analysis of variance is used when there is more than one level of the single IV, which in this case is the assertiveness training.. A one-way analysis of variance is used for testing one independent variable, while a two-way analysis of variance is used to test 2 independent variables. When a study has more than one DV the term multivariate analysis of variance (MANOVA) is utilized.

The most valuable type of research is:

the experiment, used to discover cause-and-effect relationships. Experimental research is the process of gathering data in order to make evaluative comparisons regarding different situations. An experiment must have the conditions of treatment controlled via the experimenter and random assignments (also called randomization) used in the groups. An experiment attempts to eliminate all extraneous variables.

In order for the professor of counselor education to conduct the experiment the experimental group would need to receive:

the manipulated IV and the biofeedback training. the experimental group receives the IV, which in this case is the biofeedback training.

When a researcher uses correlation, then there is no direct manipulation of the IV. A researcher might ask, for example, how IQ correlates with the incidence of panic disorder. Again, nothing is manipulated; just measured. In cases such as this a correlation coefficient will reveal:

the relationship between IQ and panic disorder. A statistic that indicates the degree or magnitude of relationship between two variables is known as a "correlation coefficient" and is often abbreviated using a "r." A coefficient of correlation makes a statement regarding the association of two variables and how a change in one is related to the change in another. Correlations range from 0.00, no relationship, to 1.0 or -1.0 which signify perfect relationships. Important: A positive correlation is not a stronger relationship than a negative one of the same numerical value. A correlation of -.70 is still indicative of a stronger relationship than a positive correlation of .60. The - merely describes the fact that as one variable goes up the other goes down.

A counselor believes that clients who receive assertiveness training will ask more questions in counseling classes. An experimental group receives assertiveness training while the control group does not. In order to test for significant differences between the groups the counselor should utilize:

the students "t" test. When comparing 2 sample groups the "t" test, which is a simplistic form of the analysis of variance, is utilized. The "t" test is used to ascertain whether to sample means are significantly different. The researcher sets the level of significance and then runs the experiment. The "t" test is computed and this yields a "t" value. The researcher then goes to a "t" table found in the index of most statistics' texts. If the "t" value obtained statistically is lower than the "t" value (sometimes called "critical t") in the table, then you accept the null hypothesis. Your computation must exceed the number cited in the table in order to reject null. If there are more than 2 groups, then the analysis of variance is utilized.

If the assertiveness researcher utilized two IVs then the statistic of choice would be:

the two-way ANOVA or MANOVA. Two IVs requires a two-way ANOVA, three IVs, a three-way ANOVA, etc.

P = .05 really means that:

there is only a 5% chance that the difference between the control group and the experimental groups is due to chance factors. Many experts in the field feel it is misleading when many exams refer to this as the "95% confidence interval," meaning that the results would be due to chance only 5 times out of 100. When P = .05, differences in the experimental group and the control group are evident at the end of the experiment, and the odds are only one in 20 that this could be explained by chance. So, your exam could refer to the "level of significance" as the level of confidence or simply the confidence level. The meaning is intended to be the same.

the Mann-Whitney U-test

to determine whether two uncorrelated means differ significantly when data are nonparametric (the "u" can remind you of "uncorrelated")

An experiment is said to be confounded when:

undesirable variables are not kept out of the experiment. Confounding is said to occur when an undesirable variable (contaminating variable) which is not controlled by the researcher is introduced to the experiment.

the Wilcoxon signed rank test

used in place of the "t" test when the data are nonparametric and you wish to test whether two correlated means differ significantly (use the "co" to remind you of "correlated")

between-subjects design

when a research study uses different subjects for each condition. If the same subjects are employed (e.g., such as in repeated measures), your exam could refer to it as a "within-subjects design." In a between-subjects design, each subject receives only one value of the ID. In a within-subjects design, 2 or more values or levels of the IV are administered to each subject.

negative correlation

would be expected when correlating an association like the number of dental cavities and time spent brushing one's teeth; as brushing time goes that dental cavities probably go down.

A Type I error occurs when:

you reject null when it is true. Since all statistical tests rely on probability there is always the chance that the results were merely chance occurrences. Researchers call these chance factors "errors." [Hint: "RA" or "reject when applicable/true."]