Research- Chapter 4-13
You are conducting a single system research study to determine if your client's weight changes after starting psychotropic medication. You ask the client to give you their weight before beginning the medication, then ask for them to give you their weight measurement on the 1st and 15th of each month, for the next three months as they continue to take the medication. You will then compare their weight before and after taking the medication. This is an example of what type of design?
ABA
When the response rate is low, you must be careful to avoid which of the following?
B and C
In a single system research design, you are tracking the behaviors of a 7-year-old child diagnosed with ADHD. You ask the parents to count the number of daily temper tantrums for two weeks. You then implement a behavior modification program and ask the parents to count the number of daily temper tantrums for two more weeks. At the end of this period, you find that the temper tantrums have increased. The parents decide to enroll the child in play therapy. You ask them to again track the number of daily temper tantrums for two weeks. The play therapy phase would be coded as what?
C
Inferential statistics should be used to analyze data collected in single subject research designs.
FALSE? pg. 181
Which of the following is/are true about quasi-experimental designs?
a and c only
When conducting an evaluation of child welfare supervision practices in one state, the researchers surveyed those involved in an intervention designed to improve supervision. The supervisors in the sample represented what type of sample?
a purposeful sample
When developing your study, which of the following timelines should you employ?
a. develop a baseline of data, collect data at regular intervals using the same instrument
Measuring client satisfaction is important because
all of the above
Quota sampling:
all of the above
When treatment fidelity is not maintained, what is/are the risk(s) for the evaluator?
all of the above
Which of the following is a strategy to utilize in single system research designs to ensure that the repeated administration of a measure is credible?
all of the above
Which of the following is a type of group research design?
all of the above
Which of the following is an example of convenience sampling? d.
all of the above
Which of the following is/are a form of data collection in qualitative research?
all of the above
Why is the use of inferential statistics not recommended for use with single system research design (SSRD) data?
all of the above
Which of the following is/are a strategy(ies) for conducting culturally sensitive evaluations?
all of the above are appropriate strategies
Which of the following is NOT criterion used for selecting evaluation instruments?
all of the above are criteria
Determining if the survey is the right length and not too complex is to evaluate its
appropriateness
Clients who have been involved with a human services program for a long period of time:
b. may rate services high to help convince themselves the treatment was worthwhile
Consistently high client satisfaction survey results can be attributed to
b. provider's pre-established reputation for reliability
Which of the following is/are true regarding instrument selection for client satisfaction studies?
b. the researcher should first consult the literature to find a scale that has good reliability and validity
Some customer satisfaction studies (e.g. Chen et al., 2006) have found that as customer satisfaction was high, so also was what?
b. the unmet needs of the client
A method of collecting client satisfaction surveys where every client coming into the agency during a specified period is asked to complete a short questionnaire and can be assisted by the non-clinical staff is known as which of the following?
ballot box
When describing your single system research study using the schematic code, A indicates the ______________ phase and B indicates the _______________ phase.
baseline; intervention
Researchers Ingram and Chung (1997) recommend not focusing only on the problems of dissatisfied clients, but recommend working toward a goal of 90% being maximally satisfied. Why did they suggest this?
because customer relation studies they found indicated that clients who are extremely displeased will never completely give up their bad feelings about an agency
In group research design, an X symbolizes the _____________ and an O symbolizes the ___
beginning of an intervention; an assessment period
Aldana, et al. (2001) found the most powerful predictor of client satisfaction was:
c. politeness of the health provider
What is important to include in a qualitative report, because it is often the only section closely read by those to whom the report is disseminated?
conclusions and reccomendations
Administering a scale known to be a valid measure of a certain concept, along with a new scale intending to measure the same concept and finding that the scores correlate is a way to establish __________________ validity.
concurrent
______________ validity is concerned with the theoretical relationship of the scale to other variables and involves testing the presumed relationships and hypotheses.
construct
When experts are asked to review an instrument to determine if the entire range of the concept is represented in the items, this is done to establish ___________________.
content validity
Which of the following is NOT a dimension of treatment fidelity?
cost effectiveness
Examining the number of tickets received while also looking at a score on an instrument rating driving skill is an example of working to establish what kind of validity?
criterion
When planning a qualitative research study, what areas typically require the most work, and therefore must be well-planned in order to meet your budgets and timeline?
data collection and analysis
The type of sampling utilized when needing to explore people who have had unusual successes or failures as a result of participation in a program is called _________________________.
deviant case sampling
When evaluating the implementation of a medication algorithm program in mental health, you notice that the average times a client changes medication dosage is 2. However, you find that for a few clients, they have had medications changed 8 or more times. You would like to interview them regarding their experiences in the program. Looking for individuals to interview such as these who are extreme on a range of factors is known as what?
deviant case sampling
One of the requirements of single system research design is that the results
displayed in graph form to see the measures over time.
When a program gets modified and changes over time, but also preserves some of the features of the original design, this is referred to as program __________________.
drift
Deciding the procedures to following including the sample and what type of data will be collected, as well as the timeframe for the data collection is known as the
evaluation plan
A program evaluation study in which some clients were on a waiting list for a treatment, but compared to those who began treatment immediately would be what type of design?
experimental design
When more than one group is observed in a single system research design (SSRD), this is known as what?
experimental design
A study that does not have a pretest measure, but has a post-test only with random assignment to a control or intervention group is an example of a
experimental design.
Single system research designs are often criticized for their lack of generalizability, or application beyond the participants in the study. This is known as __________________________________.
external validity
A concern with client satisfaction studies is that the majority of participants typically rate satisfaction with services received as low.
false
A convenience sample is one that selected randomly using a table of random numbers.
false
A sample of convenience is an example of probability sampling.
false
An instrument is reliable when it measures what it was designed to measure.
false
Classical experiments are necessary in the social sciences in order to prove any findings
false
Client satisfaction studies measure if clients liked a program or not and cannot include other variables.
false
Complex treatments increase the chances that treatment fidelity will be honored.
false
Group research designs cannot be used in program evaluation to determine if outcomes have been met.
false
Historically speaking, evaluation research has been primarily qualitative in nature.
false
If a measure is found to be valid by empirical evidence, then it can be assumed that it is also reliable.
false
In single system research design schematic coding, C stands for the baseline phase.
false
One goal of process evaluation is to describe the problem.
false
One of the main goals of summative evaluation is program description.
false
Program monitoring usually requires an experimental research design.
false
Single system research designs are considered qualitative research.
false
Single system research designs can only be applied to evaluate practice with individual clients.
false
Studies without random assignment and pre-tests are among the stronger of the group research designs.
false
The emphasis on client satisfaction began in the 1980s-1990s with the advent of managed care and authorization for services.
false
The purpose of formative evaluation is to show that participants of the program are improving as a result of the intervention in which they are taking part.
false
Treatment differentiation refers to the extent to which the service provider follows the specified procedures of a treatment.
false
Typically, when items are added to a scale, it decreases the reliability.
false
What type of evaluation focuses on gathering information to aid in program refinement and improvement?
formative or process
When deciding on the measurement instrument to use, what is the first consideration?
how the program's success can best be demonstrated
In qualitative analysis, what is usually the unit of analysis?
individuals
Data collected for program evaluation that involves observations, interviews or making judgments and involves more than one person assigning the ratings independently must build in measures of which of the following?
inter-rater reliability
When each item within a scale is examined to see how well they correlate with the other items as a whole, this is a test for _____________________.
internal consistency
Which of the following is a tool used for diagramming theoretically how a program will work to meet its outcomes?
logic model
When a problem resolves itself over time and the change is not just the result of the intervention, this is which alternative explanation or threat to internal validity?
maturation
A ________________________ explains the work of the agency, providing clarity to those within the organization and is used to gain the support of those outside, and is very important to know in program monitoring.
mission statement
What is the order for listing program components in a logic model?
model, inputs, activities, outputs, client outcomes
High fidelity is likely to occur when there is conformity to prescribed program elements and a lack of __________________.
non-prescribed elements
_____________________________ indicators are data items usually collected from participants of a program after their completion to determine if it works, while _____________________________ indicators are collected by the program routinely to determine if it is being implemented as planned.
outcome, process
_________________________ are the countable products resulting from a program, while ________________________ are the changes in clients that are achieved as a result of program participation.
outputs, outcomes
Developing an alternative version of the scale and administering it to a group similar to those in the group completing the original instrument, then determining if the versions correlate with each other, is known as which technique for measuring internal consistency?
parrallel
A study design that focuses on empowering the least powerful stakeholders in the evaluation, and valuing their input from the beginning to the end of the study is known as _____________________.
participatory evaluation
Though they can be applied across settings, single system research designs are primarily employed to study which of the following?
practice with individuals
A one-group pretest-posttest or posttest only design would fall under which category?
pre-experimental design
A study to determine the impact of a new policy or law in which the evaluator examines data points before and after passage, without a comparison group, is which type of design?
pre-experimental design
A study designed to test the knowledge of the participants on job skills training after they complete the course is an example of a(n)
pre-experimental design.
What are the three overarching purposes of process or formative evaluation?
program description, program monitoring, quality assurance
What type of process evaluation is mainly focused on checking the progress of a program in terms of meeting their objectives; particularly exploring the inputs, process, and outputs?
program monitoring
Qualitative methods:
provide vivid descriptions and an understanding of how results came to pass
A focus on naturalistic inquiry, use of the researcher as an instrument in data collection and reports that focus on narrative are common ingredients of what type of studies?
qualitative
_____________ methods are typically used to study process evaluation, while ________________methods are typically used to determine outcomes.
qualitative, quantitative
What type of evaluation is mainly concerned with determining if an agency or program is in compliance with a certain set of standards, such as the Joint Commission of Accreditation of Health Organizations?
quality assurance
What is one of the differences between quality assurance and "program evaluation?"
quality assurance relies more on peer review and sampling records to see that expected standards were met
When a questionnaire is ____________________________, that means it consistently and dependably measures the concept accurately.
reliable
When collecting data in a qualitative study, and you begin to notice repetition in the data and confirmation of existing findings, not anything new revealed, this point is termed what?
saturation
To be used in single system research, outcomes measures must be reliable, valid, and __________________________.
sensitive to change
You are conducting a study on illegal immigrants and employment history. You cannot post ads designed to elicit a sample due to the fear of the population regarding deportation if discovered. You know an illegal immigrant and have gained their trust. They agree to talk to you anonymously. After the interview, you ask the person to refer another immigrant to you to interview, also anonymously. After that interview, you ask for another referral from that participant, and so on. This method of obtaining a sample is known as
snowball sampling.
Asking questions such as if an instrument will be appropriate to your populations' reading level and vocabulary, as well as their emotional state, is to evaluate which criteria for selecting an evaluation instrument?
suitability
When a scale is found to be consistent after it is administered to the same group of participants repeated times, this is a demonstration of what type of reliability?
test-retest
In a study in which a data point is gathered more than once and the same instrument or survey is used each time, results of the study may be questioned due to which threat of internal validity?
testing
An experimental single subject research design is useful because it allows the researcher to claim what?
that the treatment was perhaps responsible for observed change
Treatment fidelity is important to maintain so that when a program evaluation reveals significant results, it can confidently be attributed to _______, not to other factors or activities.
the intervention itself
Treatment adherence is the extent to which
the therapist uses the specified procedures to deliver an intervention.
A provider's skill level and knowledge about an intervention can affect which dimension of fidelity?
therapist competence
Which of the following statements is/are true about experimental designs?
they are the most rigorous of the designs
When the employees of an agency being studied voice concern about negative findings resulting from the evaluation, and the ramifications that could occur after the findings are made public, this is an example of which of the following?
threat of evaluation
The accuracy with which a program or intervention is implemented as intended is known as which of the following?
treatment fidelity
Using more than one type of data collection to corroborate findings and verify the interpretations from each is known as ________________________________.
triangulation of data
A breakdown in intervention fidelity creates threats to the validity of a study.
true
A major consideration in choosing a sample size is the confidence level chosen by the researcher.
true
Agency administrators may be motivated to present their program(s) in the best possible manner and this can affect treatment fidelity.
true
Answering questions on a survey more positively to please a therapist or evaluator is known as social desirability.
true
As a beginning program evaluator, it is recommended that you choose a single program with which to began, rather than a multi-program study.
true
Before deciding on an instrument, you should pilot test it with a small group of representative clients or individuals.
true
Behavioral data can be obtained through client self-monitoring.
true
Even if an instrument is objective, an evaluator cannot determine the magnitude of a client's problems with any amount of precision.
true
Highly skilled therapists are more likely to deviate from treatment protocols than less skills therapists.
true
In probability sampling, the goal is to choose a sample that is reflective of the larger population.
true
It is generally considered best practice to use qualitative methods to evaluate community level practice, as opposed to single-system designs.
true
It is important to consider the evaluation audience when designing the evaluation plan.
true
Like quantitative methods, qualitative analysis is empirical and systematic, with analysis grounded in the data
true
Logic models can be conceptual, as a program is in development, or actual after it has been operating for a period of time.
true
Negotiating a contract before beginning the evaluation is one strategy for managing for the political nature of evaluation.
true
Single system designs can be used as part of formative evaluations to help enhance programs.
true
When conducting an evaluation, it is not good practice to provide on-site feedback as you uncover information that could benefit the agencies' work during the study.
true
When conducting client satisfaction surveys, you can expect there to be a low response rate.
true
When items are well chosen, internal consistency can be obtained with only a few items.
true
A researcher is conducting a program evaluation of a clinic that treats people with depression. They ask each clinician to think about two or three clients that represents an "average" client in terms of symptoms, length of treatment, etc. What type of sampling is this? d.
typical case sampling
When an instrument measures the concept for which it was designed, it is said to be __________________________.
valid
