CHAPTER 14: EVALUATION

Process-recording

- Narrative, step-by-step descriptions of a group's development - Can help a worker analyze member interactions - Useful in training and supervision of beginning group workers - Time-consuming and, therefore, rarely used by experienced group workers

Decide how a concept will be measured

-Based on behavior -Based on cognition -Based on affect

Decide on the form in which data will be collected:

-By interviewing members -By written response to a questionnaire -By audio or video recording

Strength of case studies:

-Can provide a clear, detailed, vivid description of the processes and procedures of a group in action -Are often more feasible to apply in practice settings WEAKNESS: lack of generalizability

Evaluation Measures

-Choosing Measures -Types of Measures -Observational Measures

Efficiency Evaluations

-Compare the benefits of a group program with its cost -Attempt to place a monetary value on the outcomes of the group and compare with cost of conducting the group

Choosing Measures

-Decide on the objectives of the evaluation -The worker should be familiar with the properties of measures that govern the quality of data to be collected -Decide on the form in which data will be collected -Decide how a concept will be measured

Single-System Methods aka Single-Subject Designs

-Developed to evaluate data collected over time from a single system/group -May include information about a single group member or the group as a whole -Data collection uses quantitative research methods -Baseline data are collected prior to an intervention and then compared to the data collected some time after the intervention takes place

Obtaining Program Information

-Examine records from previous groups that focused on similar concerns -Attend workshops and conferences where group workers share recent developments in the field -Review relevant journals and books -Obtain information about how similar objectives and goals were accomplished by other groups -Attend meetings of groups working on similar concerns

To determine the significance of an intervention, effectiveness and efficiency evaluations rely on:

-Experimental and quasi-experimental designs (exper: random assignments (can't always ethical) quasi-exp: don't randomly assign, we never know for sure -Reliable (consistency) and valid (accuracy) measures -Statistical procedures

Effectiveness Evaluations

-Focus on the extent to which a group accomplishes its objectives -Gain objective feedback about the helpfulness of the methods being used and the outcomes received

Benefits of Conducting Evaluations

-Help workers improve their leadership skills -Satisfy workers' professional concerns about the effects of specific interventions -Demonstrate the usefulness of a specific group or method to an agency, funding source, or society -Assess the progress of group members and purposes -Members can express satisfactions/dissatisfactions -Gather knowledge to be shared with others who are using group methods for similar purposes

Steps in a Developmental Evaluation

-Identify a need or problem -Gather and analyze relevant data -Develop a new group program or method -Evaluate the new program or method -Modify the program or method on the basis of the data obtained

Compared with other types of evaluations, effectiveness and efficiency evaluations are:

-Less flexible -More technically complex -More difficult to conduct

Summary-recording

-Less time-consuming, more selective, and more focused than process recording -Focuses on critical incidents that occur in a group -Questions are most often used for monitoring a group's progress after each group session -Use of clear, open-ended questions so recordings are consistent

Evaluations for Monitoring a Group

-Monitoring methods have received more attention in group work literature than any other eval method -Monitoring is the least demanding and the most flexible of the evaluation procedures -Steps in the Monitoring Process: --Decide what information to collect --Decide how the needed information will be collected -Monitoring by the group worker -Monitoring by the Group Members

Evaluations for Planning a Group

-Obtaining Program Information -Needs Assessment

Needs Assessment

-Potential members' willingness to attend the group -Their motivations for attending -Their capabilities for helping the group achieve its purposes

Type of Data-Colleciton Instruments

-Progress notes -Self-reports or personal interview data -Questionnaires -Review of audiotapes/videotapes of group meetings -Role-play or in vivo performance tests -Reliable and valid scales

Types of Measures

-Self-Report Measures -Observational Measures

Evaluations for Developing a Group

-Steps in a Developmental Evaluation -Single-System Methods (aka Single-Subject Designs) -Case Study Methods

Self-Report Measures

-The most widely used evaluation methods -Can be written or oral -Members are asked to respond to questions about a particular issue or concept -Particularly useful in measuring intrapersonal phenomenon such as beliefs or attitudes -BDI is an example

Observational Measures

-Use independent, objective observers to collect data -Less susceptible to biases and distortions than are self-report measures -Used less frequently than self-report measures because they require the availability or one or more trained observers to collect the data -The most well-known observational measure for groups is Bales' Interaction Process Analysis

Bales' Interaction Process Analysis

-Useful tool for the evaluation of group interactions -Consists of 12 categories -Interactions are coded by assigning each person a number -Interactions are then marked in the appropriate category

Decide on the objective of the evaluation: (Choosing Measures cont'd)

-What information is needed? -For what will the information be used? -By whom will the information be used?

The worker should be familiar with the properties of measures that govern the quality of data to be collected: (Choosing Measures cont'd)

-What is the reliability of the measure? -What is the validity of the measure?

Time Considerations

-Workers should consider how much time they have available to conduct an evaluation -Estimate the amount of time it will take them to collect, process, and analyze the information -Compare the time needed for the evaluation with the time they have available -Decide whether the evaluation is feasible

Case Study Methods

Rely on precise descriptions, accurate observations, and detailed analyses of a single example or case Developed by researchers interested in qualitative, rather than quantitative, research methods

Steps in the self-monitoring procedure:

What data will be collected? When will the data be collected? How much data should be collected? When should the information collected by members be analyzed by the group?

Evaluation

the process of obtaining information about the effects of a single intervention or the effect of the total group experience