MIDTERM EXAM PRACTICE EVALUATION
Which of the following is/are a form of data collection in qualitative research?
taking notes when observing services provided, transcripts from interviews conducted, documents and other existing data acquired
Which of the following is/are true about quasi-experimental designs?
there is no random sampling, there is sometimes a control group
How is the cost-effectiveness ratio for a program computed?
Divide the total cost of the program by the effectiveness outcome indicator.
Which of the following is NOT one of the four A's to guide planning?
Assessment
When describing your single system research study using the schematic code, A indicates the ______________ phase and B indicates the _______________ phase.
Baseline; Intervention
In group research design, an X symbolizes the _____________ and an O symbolizes the ____________________.
Beginning of an intervention; an assessment period.
What is important to include in a qualitative report, because it is often the only section closely read by those to whom the report is disseminated?
Conclusions and recommendations
In cost-benefit analysis, the evaluator must assign a monetary value to the benefits of a social service program. True or False?
True
One of the requirements of single system research design is that the results...
Displayed in graph form to see the measures over time.
Using more than one type of data collection to corroborate findings and verify the interpretations from each is known as ________________________________.
Triangulation of data
A cost-benefit analysis can help an administrator determine if the money spent on a program is related to its outcomes. True or False?
True
Common issues that can affect your needs assessment approach include:
the importance of the effort, budget and available resources, and time to finish the project.
Which of the following statements is/are true about experimental designs?
They are the most rigorous of the designs.
___________________ involve(s) a small group of individuals gathered to be asked questions by a facilitator related to a certain topic, with the goal being to gather relevant information about a problem or issue about which all the members are aware or have direct knowledge.
Focus groups
In qualitative analysis, what is usually the unit of analysis?
Individuals
A researcher utilizes a survey as well as focus groups to determine the outcomes of a particular intervention. When analyzing the data, the researcher is careful to analyze how the different focus groups as well as the survey yield similar findings (or not). This is called what type of analysis?
Convergent
A concern with client satisfaction studies is that the majority of participants typically rate satisfaction with services received as low. True or False?
False
The program that provides the best results for the most money is the most efficient. True or False?
False
Which of the following is/are true regarding instrument selection for client satisfaction studies?
The researcher should first consult the literature to find a scale that has good reliability and validity
Some customer satisfaction studies (e.g. Chen et al., 2006) have found that as customer satisfaction was high, so also was what?
The unmet needs of the client
The societal perspective of a cost-effectiveness analysis takes into account all costs and effects of a program, not just the perspective of only one group. True or False?
True
When conducting client satisfaction surveys, you can expect there to be a low response rate. True or False?
True
When disseminating the results of the needs assessment, you should present preliminary results to the stakeholders before finalizing and presenting the information to the community. True or False?
True
You are conducting a single system research study to determine if your client's weight changes after starting psychotropic medication. You ask the client to give you their weight before beginning the medication, then ask for them to give you their weight measurement on the 1st and 15th of each month, for the next three months as they continue to take the medication. You will then compare their weight before and after taking the medication. This is an example of what type of design?
A-B
In a single system research design, you are tracking the behaviors of a 7-year-old child diagnosed with ADHD. You ask the parents to count the number of daily temper tantrums for two weeks. You then implement a behavior modification program and ask the parents to count the number of daily temper tantrums for two more weeks. At the end of this period, you find that the temper tantrums have increased. The parents decide to enroll the child in play therapy. You ask them to again track the number of daily temper tantrums for two weeks. The play therapy phase would be coded as what?
C
In a cost-benefit analysis, effort is made to measure which of the following?
Costs and benefits in monetary units
When planning a qualitative research study, what areas typically require the most work, and therefore must be well-planned in order to meet your budgets and timeline?
Data collection and analysis
According to Bradshaw (1977), a(n) _____________________ need is one that is expressed by actual clients and those directly affected by the issue.
Felt
Cost-effectiveness studies can help administrators make decisions about which of the following?
allocation of scarce resources in programs, use of staff time in programs, program successes compared to program costs.
Which of the following is a type of group research design?
pre-experimental, quasi-experimental, experimental.
Researchers Ingram and Chung (1997) recommend not focusing only on the problems of dissatisfied clients, but recommend working toward a goal of 90% being maximally satisfied. Why did they suggest this?
Because customer relation studies they found indicated that clients who are extremely displeased will never completely give up their bad feelings about an agency
The difficulty of assigning monetary value to some of the benefits derived from social services programs is illustrated by which of the following?
Determining the value of reduced stress on a family when a child is returned from out-of-home care.
When developing your study, which of the following timelines should you employ?
Develop a baseline of data, collect data at regular intervals using the same instrument.
An expressed need is when a need of one group is determined to be the need of a group with similar characteristics, but you are unable to talk to the other group directly. True or False?
False
Classical experiments are necessary in the social sciences in order to prove any findings. True or False?
False
Client satisfaction studies measure if clients liked a program or not and cannot include other variables. True or False?
False
Group research designs cannot be used in program evaluation to determine if outcomes have been met. True or False?
False
Historically speaking, evaluation research has been primarily qualitative in nature. True or False?
False
People who have knowledge of the problem you are assessing, through work or educational experience with a program or agency, or with a certain population with which you should consult before conducting your needs assessment are known as which of the following terms?
Key informants
Clients who have been involved with a human services program for a long period of time:
May rate services high to help convince themselves the treatment was worthwhile.
When more than one group is observed in a single system research design (SSRD), this is known as what?
Multiple baseline-design
Before conducting the needs assessment, it may be helpful to consult census data available in geographical units to learn more about the population being studied. This is an example of use of __________________________________.
Secondary data
To be used in single system research, outcomes measures must be reliable, valid, and __________________________.
Sensitive to change
Answering questions on a survey more positively to please a therapist or evaluator is known as social desirability. True or False?
True
As a beginning program evaluator, it is recommended that you choose a single program with which to begin, rather than a multi-program study. True or False?
True
It is generally considered best practice to use qualitative methods to evaluate community level practice, as opposed to single-system designs.
True
It is important to consider the evaluation audience when designing the evaluation plan. True or False?
True
Like quantitative methods, qualitative analysis is empirical and systematic, with analysis grounded in the data. True or False?
True
Single system designs can be used as part of formative evaluations to help enhance programs. True or False?
True
The __________________ technique is utilized when a group of experts is asked to comment on a questionnaire designed for a study, provide input, and assist the researcher in refining the measure.
Delphi
A method of collecting client satisfaction surveys where every client coming into the agency during a specified period is asked to complete a short questionnaire and can be assisted by the non-clinical staff is known as which of the following?
Delphi technique
Which of the following steps comes first when planning a needs assessment?
Determine the information needed and what you already have.
Measuring the efficiency of a program refers to all of the following EXCEPT:
Determining if the program is based on empirical research
The type of sampling utilized when needing to explore people who have had unusual successes or failures as a result of participation in a program is called _________________________.
Deviant case sampling
A study that does not have a pretest measure, but has a post-test only with random assignment to a control or intervention group is an example of a...
Experimental design.
What type of survey is utilized to determine the extent of problems in a certain area and then express the findings as rates affected within the population?
Epidemiologic
Needs assessments serve primarily which purpose?
Estimating what is not available in the service array
Deciding the procedures to following including the sample and what type of data will be collected, as well as the timeframe for the data collection is known as the ________________________________.
Evaluation plan
Single system research designs are often criticized for their lack of generalizability, or application beyond the participants in the study. This is known as __________________________________.
External validity
In cost-effectiveness studies, when the results show that a program costs the least to run, it should be considered the most effective for clients. True or False?
False
In single system research design schematic coding, C stands for the baseline phase. True or False?
False
Inferential statistics should be used to analyze data collected in single subject research designs. True or False?
False
Needs assessments are a final step to take when developing a new program. True or False?
False
Previously collected data for another purpose cannot be used by researchers for program evaluation studies. True or False?
False
Public hearings and community forums are not considered methods of needs assessments. True or False?
False
Single system research designs are considered qualitative research. True or False?
False
Single system research designs can only be applied to evaluate practice with individual clients. True or False?
False
Studies without random assignment and pre-tests are among the stronger of the group research designs. True or False?
False
The emphasis on client satisfaction began in the 1980s-1990s with the advent of managed care and authorization for services. True or False?
False
Which of the following is the first step in conducting a cost-effectiveness study?
Gaining an understanding of the agency mission and outcome indicators
When a problem resolves itself over time and the change is not just the result of the intervention, this is which alternative explanation or threat to internal validity?
Maturation
Aldana, et al. (2001) found the most powerful predictor of client satisfaction was:
Politeness of the health provider
Though they can be applied across settings, single system research designs are primarily employed to study which of the following?
Practice with individuals
A one-group pretest-posttest or posttest only design would fall under which category?
Pre-experimental design
A study designed to test the knowledge of the participants on job skills training after they complete the course is an example of a(n)
Pre-experimental design.
Qualitative methods:
Provide vivid descriptions and an understanding of how results came to pass.
When determining which treatment should be offered to a client, the choice should be the treatment that will yield the most benefit while also ________________________________.
Requiring the lowest cost to provide
When collecting data in a qualitative study, and you begin to notice repetition in the data and confirmation of existing findings, not anything new revealed, this point is termed what?
Saturation
In a study in which a data point is gathered more than once and the same instrument or survey is used each time, results of the study may be questioned due to which threat of internal validity?
Testing
In a cost-effectiveness study, what is the sensitivity analysis?
Testing the recommendations based on the cost-effectiveness data.
Which is true about the literature regarding cost-benefit analysis of human services programs?
There is a lack of consensus on how to define benefits and unit costs in social service programs.
Measuring client satisfaction is important because...
dissatisfied clients are likely to leave the agency, managers need to detect problems before they get out of control, it is required by many regulatory agencies.
Which of the following is a strategy to utilize in single system research designs to ensure that the repeated administration of a measure is credible?
measure repeatedly without any intervention introduced, measure repeatedly while simultaneously introducing the intervention, measure repeatedly both before and after the intervention is introduced.
When the response rate is low, you must be careful to avoid which of the following?
over generalizing the results from a small sample, determining that your program is achieving its results.
When computing costs as part of a cost-effectiveness study, which of the following is a category of costs to consider?
personnel including salary and fringe benefits, facilities costs including rent and maintenance, equipment required by the program.
Why is the use of inferential statistics not recommended for use with single system research design (SSRD) data?
practitioners not skilled in analysis may be reluctant to conduct studies, the data in SSRDs typically violate assumptions needed for use of parametric statistics, the data are not independent.
A program evaluation study in which some clients were on a waiting list for a treatment, but compared to those who began treatment immediately would be what type of design?
quasi-experimental design
A study to determine the impact of a new policy or law in which the evaluator examines data points before and after passage, without a comparison group, is which type of design?
quasi-experimental design
When collecting data by using a survey, asking people who live close to your agency and/or seek services from there will yield results that adequately represent the community. True or False?
False
When conducting an evaluation, it is not good practice to provide on-site feedback as you uncover information that could benefit the agencies' work during the study.
False
A consultant is hired by your agency to conduct a gaps analysis to determine groups of people not being served by the current services provided to the homeless. The consultant concludes that adolescents ages 17-18 who have formerly been in foster care comprise a large percentage of the homeless and you do not have any programs that target this group. This is an example of what type of need?
Normative
A study design that focuses on empowering the least powerful stakeholders in the evaluation, and valuing their input from the beginning to the end of the study is known as _____________________.
Participatory evaluation
Consistently high client satisfaction survey results can be attributed to:
Provider's pre-established reputation for reliability
A focus on naturalistic inquiry, use of the researcher as an instrument in data collection, and reports that focus on narrative are common ingredients of what type of studies?
Qualitative
_____________ methods are typically used to study process evaluation, while ________________methods are typically used to determine outcomes.
Qualitative; Quantitative
An experimental single subject research design is useful because it allows the researcher to claim what?
That the treatment was perhaps responsible for observed change.
