RESEARCH METHODS

SNOWBALL SAMPLING

field researchers learn from participants about who represents different subgroups in a setting.

Fisher Exact Test

for 2X2 tables - results in the exact probability of the occurrence of the observed frequencies given the marginal totals

WHEN SHOULD THE "N" BE INDICATED IN FREQUENCY DISTRIBUTIONS?

if table is based on percentages the total number of cases in the distribution (the base number N) should be indicated.

Gamma

similar to kendall's but ignores ties

Left-Skewed

values are more spread out on the left side - mean is on the left (Outliers-low values, Mean, Median, Mode)

Right-Skewed

values are more spread out on the right side - mean is on the right (Mode, Median, Mean, Outliers-high values)

Numerical Summary

a number that summaries the data: -Mean, Median, Standard Deviation -Proportion (Fraction or percent)

MEASURES OF ASSOCIATION

a type of descriptive statistic used to summarize the strength of an association.

Cramer's V coefficient

alternative to contingency coefficient and is used when the tables are asymmetrical

ratio

an absolute zero, magnitude ex. heart rate, percentage

variable

anything that can take on more than one value ex. age, SAT score, gender, color of hair

BAR CHART

contains solid bars separated by spaces. -It is a good tool for displaying the distribution of variables measured at the NOMINAL LEVEL and other discrete categorical variables because there is, in effect a gap between each of the categories.

BAR CHART

contains solid bars separated by spaces. -It is a good tool for displaying the distribution of variables measured at the NOMINAL LEVEL and other discrete categorical variables because there is, in effect a gap between each of the categories. **solid bars separated by spaces - good for displaying distribution in discrete categories

Qualitative Data (Categorical)

describe qualities or categories: -Eye color -Handedness -Class Rank -Bar/Pie Chart -Proportion (fraction or percent)

Distribution of a Variable (or data set)

describes the values taken on by the variable and the frequency (or relative frequency) of these values Important characteristics: -Shape -Center -Spread

level of measurement

how much information is being provided by the outcome measure

Variation

how widely data values are spread out about the center of a distribution (skinny-low, middle-moderate, fat-high)

Symmetric Distribution

if its left half is a mirror image of its right half - same on both sides

WRITING QUESTIONS:

**The centerpiece of survey research is asking people questions. - Questions can be a straightforward and efficient means to measure demographic characteristics, levels of knowledge, opinions an attitude, feelings and symptoms and behaviors. - But survey questions may result in misleading or inappropriate answers** **All questions proposed for a survey must be screened carefully for their adherence to basic guidelines and then tested and revised until the researcher feels some confidence that they will be clear to the intended respondents. **

DESIGN CONSIDERATIONS

**once we have decided on the focus of the evaluation, there are still important decisions to be made about how to design the specific evaluation project. These include the following questions: •Black box or program theory: do we care how the program gets results? •Researcher or stakeholder orientation: whose goals matter most? •Quantitative or qualitative methods: which methods provide the best answers? •Simple or complex outcomes: how complicated should the findings be?

DESCRIBE FENCE SITTERS & FLOATERS.

*Two related problems in question writing stem from people's desire to choose an acceptable answer, even if they do not know or have no opinion* •FENCE SITTERS: oSee themselves as being neutral and whose responses may skew the results if you force the m to choose between opposites. oAdding an explicit neutral response option is appropriate when you want to find out who is a fence-sitter. oBut adding a neutral response may provide an easy escape for respondents who do not want to reveal their true feelings. •FLOATERS oAre respondents who choose a substantive answer when they really do not know? oBecause there are so many floaters in the typical survey sample, the typical survey sample, the decision to include an explicit "Don't Know" option for a question is important. oDon't Know" responses are offered more often by those with less education - except for questions that are really impossible to decipher, to which more educated persons are likely to say they do not know oUnfortunately, the inclusion of the "Don't know" response choice leads some people who do have a preference to take he easy way out and choose "Don' t know"

WHAT IS CROSSTABULATION?

-(crosstab) is one of the simplest methods for looking at the association among two or more variables. -A crosstabulation displays the distribution of one variable for each category of another variable; it can also be turned a bivariate distribution.

WHAT IS STATISTICAL SIGNIFICANCE (POWERPOINT)

-Avoid concluding that an association exists in the population from which the sample was drawn unless the probability that the association was due to chance is less than 5 percent -Statistical Significance: when an analyst can demonstrate at least 95% confidence that an association was not due to chance, reported as p<.05 -weak associations can be significant in large populations; just because something is significant does not mean it is important.

VERSATILITY

-Can be used to investigate almost every field of social work, such as child welfare, gerontology, health, mental health, income maintenance, community building and community development. -Surveys are used in agencies to assess the impact of policy changes, assess community needs, track changes in community characteristics, monitor and evaluate program effectiveness, and assess client satisfaction with programs.

WHAT IS A MONOTONIC RELATIONSHIP?

-In a MONOTONIC relationship, the value of cases consistently increases (or decreases) on one variable as the value of cases increases (or decreases) on the other variable. -Monotonic is often defined a bit less strictly, with the idea being that as the value of cases on one variable increases parentheses (or decreases), the value of cases on the other variable to increase (or decrease) and at least does not change direction.

WHAT IS A GAMMA?

-One popular measure of association is crosstabular analyses with variables measured at the ordinal level is GAMMA -As with many measures of association the possible values of gamma very from -1, meaning the variables are perfectly associated in an inverse direction; to 0, meeting there is no association of the type that gamma measures; to +1, meaning there is a perfect positive association of the type that gamma measures -(powerpoint) a measure for ordinal level variables that ranges from -1 to +1 with zero indicating no relationship

WHAT ARE SOME SYSTEMATIC APPROACHES TO SAMPLING IN PARTICIPANT OBSERVATION?

-PURPOSIVE SAMPLING -SNOWBALL SAMPLING -QUOTA SAMPLING -THEORETICAL SAMPLING

interval

-assign a value to an outcome -no natural zero -continuum of equal interval ex. 1-2, temperature, time

questions about measurement

-can you assign a name? -canyou assign order?(moreness) -can you assign underlying continuum -can you assign an absolute zero

nominal

-difference in quality rather than quantity -categorical variables ex. names

WHAT IS THE STANDARD DEVIATION?

-the square root of the variance. By taking the square root, the sample standard deviation is expressed in the original units of the measure. •It is the square root of the average squared deviation of each case from the mean SEE PAGE 306

WHAT IS STATISTICAL SIGNIFICANCE?

-when an analyst can demonstrate at least with 95% confidence that an association was not due to chance, reported as <.05 -Weak associations can be significant in large populations; just because something is significant does not mean it is important. ______________________Book: -means the relationship is not due to chance; Statistical Significance means that an association is not liekly to be due to chance, according to some criterion set by the analyst. - the criterion is referred to as te alpha level (a) or p value - the probability level that will be used to evaluate statistical significance.

.

.

QUANTITATIVE

.

WHAT ARE THE 3 IMPORTANT FEATURES OF THE DISTRIBUTION'S SHAPE?

1. Central Tendency, 2. Variability, 3. Skewness

WHAT ARE THE GOALS OF PHOTOVOICE?

1. Enable people to record and reflect on their community's strength and concerns 2.Promote critical dialogue and knowledge about personal and community issues through large and small group discussions of photographs 3.Reach policy makers

WHAT ARE THE 4 POPULAR MEASURES OF VARIATION?

1. RANGE 2. INTERQUARTILE RANGE 3. VARIANCE 4. STANDARD DEVIATION

WHAT ARE THE DIFFERENT SURVEY METHODS?

1. TELEPHONE 2.FACE -TO-FACE 3. WEB 4. MAIL 5. GROUP (NOTES) ( SEE BOOK NOTES AS WELL)

WHAT ARE THE 3 DIFFERENT MODES OF READING TEXT IN QUALITATIVE ANALYSIS?

1. When the researcher reads the text, LITERALLY, the focus is on its literal content and form, so the text "leads" the dance. 2. When the researcher reads the text REFLEXIVELY, the researcher focuses on how his or her own orientation shapes interpretations and focus. Now the researcher leads the dance. 3. When the researcher reads the text INTERPRETIVELY, the researcher tries to construct his or her own interpretation of what the text means. • In this way, analyzing text involves both inductive and deductive processes: the researcher generates concepts and linkages between them based on reading the text and also checks the text to see whether the concepts and interpretations are reflected in it.

WHAT ARE THE STEPS TO PREPARING DATA FOR ANALYSIS?

1.ASSIGN A UNIQUE IDENTIFYING NUMBER 2.REVIEW THE FORMS 3. CODE OPEN ENDED QUESTIONS 4. CREATE A CODEBOOK 5.ENTER THE DATA 6. CLEANING THE DATA -data cleaning -check coding

WHAT BASIC PRINCIPLES SHOULD BE USED TOWARD DEVELOPING CLEAR AND MEANINGFUL QUESTIONS:

1.Avoid Confusing Phrasing 2.Avoid Vagueness 3.Provide a Frame of Reference 4.Avoid Vague Words 5.Avoid Negative Words and Double Negatives 6.Avoid Double Barreled Questions 7.Avoid Jargon 8.Reduce The Risk Of Bias 9.Memory Questions

WHAT PHASES ARE SHARED BY MOST APPROACHES TO QUALITATIVE DATA ANALYSIS?

1.Documentation of the data and the process of data collection 2.Organization/categorization of the data into concepts 3.Connection of the data to show how one concept may influence another 4.Corroboration/legitimization by evaluating alternative explanations and disconfirming evidence and searching for negative cases. 5.Representing the account (reporting the findings)

WHAT ARE THE TWO TYPES OF OUPUTS PROGRAMS PRODUCE?

1.INTERMEDIATE OUTPUTS •Reflecting units of service 2.FINAL OUTPUTS •Called service completions. •Units of service describe activities undertaken by the program staff; such units of service are often described in terms of frequency or number, time or duration or tangible good. •Service units are often found on staff reporting forms. •For example, the output of an individual counseling session might be described as the number of individual counseling sessions, the amount of time spent in providing counseling or the number of counseling sessions lasting a certain amount of time. • Units of service reflect the aggregation of activities completed with clients during a specific time period.

WHAT THREE COMPONENTS SHOULD BE CONSIDERED WHEN DESIGNING A QUESTIONNAIRE

1.Maintain Consistent Focus 2.Build on Existing Instruments 3.Order the Questions.

IN WHAT WAYS DO QUALITATIVE METHODS INFORM PRACTICE

1.Qualitative methods can be used to assess the design and delivery of services. 2.Qualitative methods are useful in uncovering the process and nature of service delivery 3.Qualitative methods can clarify clients' perceptions of interventions 4.These methods can help social workers understand why people in need of services do not seek help. EXAMPLES PAGE 232-233

WHAT ARE THE COMPONENTS OF THE LOGIC MODEL?

1.The first component of a logic model is to identify the social problem •Programs are designed to address a social condition deemed undesirable or to enhance desirable condition. - the actual program may not address the entire social problem but an aspect of the problem. 2.The next step is to identify the TARGET POPULATION - the "for whom" column •There are a wide range of possible criteria to describe the target populatin, such as age-group, geographic residence or status. •Because programs are designed to impact social conditions, the program design is based on both assumptions about the causes of the social condition and an appropriate intervention method. •This suggests that two agencies may address the same social condition but each may design different programs with different expected outcomes. (SEE EXHIBIT 10.2 &EXAMPLE PG 268) 3.The next component of the logic model is to detail the inputs required to run the program. •The logic model description of inputs is specific, identifying, for example the type of staff, their number and their expertise. •Inputs also include material resources such as equipment or space. •The elevator identifies the specific activities (the program processes) that comprise the program. •Activities include actions taken directly with the client, such as assessment, counseling or family meetings and activities that may not directly involve the client but are necessary for the program such a treatment planning or team meetings. 4.The last category of the logic model identifies the program's expected outcomes •These are the kinds of changes in a client that are expected to occur as the result of the program. •There are INTERMEDIATE OUTCOMES and FINAL or ULTIMATE OUTCOMES •INTERMEDIATE OUTCOMES - are the changes in the client tat are necessary before the FINAL or ULTIMATE OUTCOME might occur, such as increased anger management skill, fewer behavioral problems, or improved problem-solving skills. •If the program assumptions are correct about why the child cannot function independently in the community and if the intermediate outcomes are achieved, the ultimate outcome of the child functioning independently in school, home, and the community should be more likely. (You might notice that there is a relationship between the assumptions About the factors related to the social problem and the intermediate and Final outcomes. The activities in the program are intended to improve upon Each of the areas identified in the assumptions. So the intermediate Outcomes reflect the desired changes in each of the se areas.) (This symmetry also occurs with activities and intermediate outputs as each Activity has a measure of service.)

WHAT IS A THICK DESCRIPTION?

A Thick Description provides a sense of what it is like to experience that setting from the standpoint of he natural actors in that setting.

CREATE A CODE BOOK

A codebook contains the setoff instructions used to link a number to a category for a particular variable. This is a record for you to know the values assigned to the response categories for each variable. You may define each variable as you build a data set in a statistical program, or you may create a paper version of your codebook. You should also use the codebook to keep track of any new variables you create as you analyze the data.

FREQUENCY POLYGON

A continuous line connects the points representing the number or percentage of cases with each value. -The frequency polygon is an alternative to the histogram when the distribution of a quantitative continuous variable must be displayed; this alternative is particularly useful when the variable has a wide range of values

FREQUENCY POLYGON

A continuous line connects the points representing the number or percentage of cases with each value. -The frequency polygon is an alternative to the histogram when the distribution of a quantitative continuous variable must be displayed; this alternative is particularly useful when the variable has a wide range of values **a continuous line connects the points representing the number or percentage of cases with each value.

DICHOTOMOUS

A dichotomous variable is one that takes on one of only two possible values when observed or measured. The value is most often a representation for a measured variable (e.g., age: under 65/65 and over) or an attribute (e.g., gender: male/female). If the dichotomous variable represents another variable, that underlying variable may be either observed or unobserved. For example, a dichotomous variable may be used to indicate whether a piece of legislation passed. The dichotomous variable (pass/fail) is a representation of the actual, and observable, vote on the legislation. In contrast, each legislator's vote is the result of an individual, and unobserved, probability distribution between voting either... Two mutually exclusive or contradictory categories. (like when you flip a coin)

WHAT IS PARTICIPANT OBSERVATION?

A qualitative method for gathering data that involves developing a sustained relationship with people while they go about their normal activities.

WHAT IS INTENSIVE INTERVIEWING

A qualitative method that involves open-ended relatively unstructured questioning in which the interviewer seeks in-depth information on the interviewees' feelings, experiences an perceptions.

WHAT ARE FOCUS GROUPS?

A qualitative method that involves unstructured group interviews in which the focus group leader actively encourages discussion among participants on the topics of interest.

WHAT IS A MEASURE OF ASSOCIATION?

A type of descriptive statistics used to summarize the strength of and dissociation. There are many measures of Association, some of which are appropriate for variables measured at particular levels. -(powerpoint) a type of descriptive statistic used to summarize the strength of an association.

WHEN DO YOU AVOID CONCLUDING THAT AN ASSOCIATION EXISTS FROM YOUR POPULATION SAMPLE?

Avoid concluding that an association exists in the population from which teh sample was drawn unless the probability that the association was due to chance is less than 5 percent.

Contingency coefficient

C - measure of association that is used with tables larger than 2x2 but the number of rows columns must be equal

WHAT IS THE GOAL OF THE COMPLETE OBSERVER?

COMPLETE OBSERVATION - researchers try to see things as they happen without actively participating.

THE STRENGTH OF A RELATIONSHIP FOR RATIO VARIABLES CAN BE DETERMINED USING A WHAT?

CORRELATION COEFFICIENT such as Pearson's "r"

COST BENEFIT ANALYSIS

COST BENEFIT ANALYSIS: Involves comparing the specific costs and benefits to the program. oThe analyst must identify the particular costs and benefits to be evaluated; this in turn requires the analyst to identify whose perspective will be used in order to determine what can be considered a benefit and cost. oProgram clients will have a different perspective on these issues than do program staff. oOnce potential costs and benefits have been identified, they must be measured. Costs and benefits must be translated into a common monetary unit in order to make a comparison. The challenge is estimating the dollar value of how clients benefit form the program.

COST EFFECTIVENESS

COST-EFFECTIVENESS: It is often difficult to assign a dollar value to outcomes produced in social work programs. oCOST-EFFECTIVENESS ANALYSIS compares the costs of different programs (or interventions) with the actual program outcomes in lieu of assigning a dollar value to the outcomes. In these comparisons the program costs are calculated, whereas the benefits are listed and not assigned a cash value.

WHAT ARE THE GUIDELINES FOR GROUPING DATA?

Categories should be logically defensible and preserve the shape of the distribution. Categories should be mutually exclusive and exhaustive (rules of ordinal data)

WHAT IS CENTRAL TENDENCY. VARIABILITY AND SKEWNESS?

Central Tendency - common or typical response Variability - the spread or variability of the responses Skewness - the shape of the responses.

COMPUTER-ASSISTED QUALITATIVE DATA ANALYSIS: EXPLAIN COMPUTER ASSISTED QUALITATIVE DATA ANALYSIS?

EXPLAIN COMPUTER ASSISTED QUALITATIVE DATA ANALYSIS •The analysis process can be enhanced in various ways by using a computer •Programs designed fro qualitative data can speed up the analysis process, make it easier for researchers to experiment with different codes, test different hypotheses about relationships, and facilitate diagrams f emerging theories and preparation of research reports. •The steps involved in COMPUTER-ASSISTED QUALITATIVE DATA ANALYSIS parallel those used traditionally to analyze such text as notes, documents, or interview transcripts: preparation, coding, analysis and reporting •Two of the most popular programs to illustrate these steps are HYPER-RESEARCH and QSR NVIVO •SEE PAGES 254-257

WHAT ARE TWO TYPES OF ERRORS IN SURVEYS?

For a survey to succeed it must minimize the risk of two types of error: poor measurement of cases that are surveyed -ERRORS OF OBSERVATION --- and omission of cases that should be surveyed -ERRORS OF NONOBSERVATION.

WHAT DO FREQUENCY DISTRIBUTIONS DO?

Frequency distributions and graphs of frequency distributions are the two most popular approaches for displaying variation; both allow the analyst to display the distribution of cases across the value categories of a variable.

DEDUCTION

I

INDUCTION

I

.

I.

WHAT IS THE DIFFERENCE BETWEEN INDUCTION & DEDUCTION?

INDUCTION is creation of theory from empirical data -Empirical Observation = every crow ever seen is black -Induced Theory - all crows are black -There is no certainty that a white crow won't be found tomorrow, although past experience would make such an occurrence seem unlikely. DEDUCTION is a logical process in which conclusions are drawn from a set of general principles (i.e. theory) -Theory = Alcoholism is a genetically based disease. -Hypothesis = Successful treatment of alcoholism will focus on biologically based interventions -Testing/Observation = setting up a pre-experimental, quasi-experimental or experimental design to test the success of biologically based interventions.

IF THE MEDIAN FALLS BETWEEN CASES - WHAT DO YOU DO?

If the median point falls between two cases, which happens if the distribution has an even number of cases, the median is defined as the average of the two middle values and is computed by adding the values of the two middle cases and dividing by 2.

HOW IS THE MEAN COMPUTED

It is computed by adding up the value of all the cases and dividing by the total number of cases, thereby taking into account the value of each case in the distribution - Mean = sum of value of cases/number of cases -Example: to calculate the mean value of 8 cases we add the value of all cases and divide by the number of cases.

WHAT TWO ACTIVITIES ENABLE AGENCIES TO INTEGRATE EVIDENCE BASED PRACTICE?

Logic Modeling and in-depth evaluations are two activities that enable agencies to integrate evidence based practice.

WHAT TYPE OF ERROR ARE MATRIX QUESTIONS SUSCEPTIBLE TO?

MATRIX QUESTIONS - are susceptible to another form of error: -RESPONSE SET - when scales are used with the same set of response categories, there is a possibility that, rather than reading and answering each question, the respondent simply circles the same response down the entire set of questions. -To avoid this problem, researchers often phrase some questions in the opposite direction; if the questions are worded using positive language, they will ask some questions in negative language. - The assumption that if the respondent then answers all the questions with the same response, it is clear he or she was just circling them without thinking.

WHAT ARE THE REASONS TO GROUP DATA?

More than 15-20 potential values of variable or changing interval/ratio data into ordinal data. (Sometimes collapsing the distribution of potential values of a variable makes finings clearer)

AUTHENTICATING CONCLUSIONS

No standards exist for evaluating the validity or "authenticity" of conclusions in a qualitative study, but the need to consider carefully the evidence and methods on which conclusions are based is just as great as with other types of research.

DESCRIBE THE NOTE TAKING PROCESS IN PARTICIPANT OBSERVATION AND WHY THEY ARE IMPORTANT

Notes are the primary means of recording participant observation data. •Researchers jot down partial notes while observing and then retreat to computers to write up more complete notes on a daily basis. •Some researchers will maintain a daily log in which each days activities are recorded. •Field notes must be as complete, detailed and true to what was observed and heard as possible. -Direct quotes should b distinguished clearly from paraphrased quotes, and both should e set off from the researcher's observations and reflections. -The surrounding context should receive, as much attention as possible and a map of the setting always should be included, of indications of where individuals were at different times. •Careful note taking yields a big payoff. oField notes will suggest new concepts, causal connections and theoretical propositions. -Notes also should include both descriptions of the situational context and a record of the researcher's feelings and thoughts while observing. Notes like these provide a foundation for later review of the likelihood of bias or of inattention to some salient features of the situation.

ONGOING EVALUATION FACILITATES WHAT?

Ongoing program evaluation facilitates the generation of evidence about a program's impacts, as well as understanding what contributes to a program's success.

WHAT DO OUTCOME EVALUATIONS DO?

Outcome evaluations provide some level of evidence about the effectiveness of the program for the target population; process evaluation can identify mechanisms of a program's implementation tat contribute to the program's success (or lack of success)

What are Logical Pitfalls to Drawing Conclusions Based on Qualitative RESEARCH?

PITFALLS: 1. Provincialism 2. Going Native 3. Emotional Reactions 4. Hasty Conclusions 5. Questionable Cause 6. Suppressed Evidence

WHAT ARE THE PROS OF CLOSE ENDED QUESTIONS?

PROS -Most surveys of a large number of people contain primarily closed-ended questions, because they are easier to process with computers and analyze with statistics. -Including the response choices reduces ambiguity, and respondents are more likely to answer eh question that the researcher really wants them to answer.

WHAT IS THE BENEFIT OF PHOTOVOICE

Photovoice can be a way to see and hear the voice of people. It is consistent with the empowerment tradition in social work because it brings groups together and enables them to move to social action through their participation in the process.

WHAT ARE TEH WAYS PROGRAMS MAY FAIL TO ACHIEVE THEIR DESIRED PROGRAM OUTCOMES?

Programs may fail to achieve their desired outcomes for different reasons. •The inputs may not be sufficient to carry out the program activities (INPUT FAILURE) •The inputs may be sufficient, but the actual activtieis may be incomplete, insufficient or poorly designed so that they do not set into motion the causal process (PROGRAM FAILURE) •The inputs may be sufficient, and the program activities appropriate, but the causal process may not produce the desired outcomes (THEORY FAILURE OR WRONG THEORY)

WHAT IS THE KEY COMPONENT OF PROCESS EVALUATION STUDIES?

Qualitative methods are often a key component of process evaluation studies because they can be used to elucidate and understand internal program dynamics - even those that were not anticipated. Qualitative researchers may develop detailed descriptions of how program participants engage with each other, how the program experience varies for different people and how the program changes and evolves over time.

DESCRIPTIVE STATISTICS

Research in which social phenomena are defined and described. Describe the distribution of variables. -Quantitatively describing the main features of a collection of data. -Are distinguished from INFERENTIAL STATISTICS in that it aims to summarize a sample, rather than use data to learn about the population that the sample of data is thought to represent. -DESCRIPTIVE STATISTICS unlike INFERENTIAL are not developed on the basis of probability theory -Are a set of vrief descriptive coefficients that summarize a given data set - the measures used to describe a data set are MEASURES OF CENTRAL TENDENCY & MEASURES OF VARIABILITY -DESCRIPTIVE STATISTICS privide simple summaries about the sample and about the observations that have been made. They can be either: 1. Qualitative (summary statistics) 2. Visual Graphs

WHAT IS SURVEY RESEARCH?

SURVEY RESEARCH involves the collection of information from a sample of individuals through their responses to questions.

HOW CAN SAMPLING IN QUALITATIVE STUDIES BE BIASED?

Sampling in qualitative studies can lead to bias. •Example: a nonrandom sample of elderly persons in institutional settings can be biased by staff motivation to suggest residents who will present the institution in a particular light or simply by staff lack of reliable knowledge about residents' diagnoses, ensure or other characteristics that are important in the researcher's sampling strategy.

RESEARCHER OR STAKEHOLDER ORIENTATION

Should evaluation researchers insist on designing the evaluation project and specifying its goals or should they accept the suggestions and adopt the goals of the funding agency? What role should the preferences of program staff or clients play? The different answers that various evaluation researchers have given to these questions are reflected in different approaches to evaluation

Shapes of Distributions

Single Peaked - unimodal Double Peaked - bimodal

WHAT ARE THE 6 PATTERNS THAT MIGHT BE IN YOUR QUALITATIVE DATA?

Six Patterns that might be in your qualitative data 1. Frequencies - How often does a behavior, phenomenon or interaction occur? 2. Magnitudes - What is the level of a behavior, phenomenon or interaction? 3. Structures - What are the different structures or categories that make up a behavior, phenomenon or interaction? 4. Processes - How do the structures or categories identified as part of the observed behavior, phenomenon or interaction work together? 5. Causes - Why does a behavior, phenomenon or interaction occur? 6. Consequences - What are the impacts or outcomes of a behavior, phenomenon or interaction?

WHAT ARE MEMORY QUESTIONS?

Some questions require respondents to try to remember an event. - Remembering an event is affected by the length of time since the event occurred and how important the event was to the respondent.

WHY IS REFLEXIVITY IMPORTANT?

Such an account is important because of the evolving and variable nature of field research: to an important extent the researcher "makes up" the method in the context of the particular investigation, rather than applying standard procedures that are specified before the investigation begins.

WHY IS SURVEY RESEARCH SO POPULAR? WHAT ARE THE PROS?

Survey research is popular because of its versatility, efficiency and generalizability.

WHAT IS TYPE II ERROR

TYPE I ERROR is not the only wrong conclusion that we can make. Let us return to the test of the relationship between the umber of hours studied and examination scores. In our sample, we find that there is not a statistically significant relationship and conclude that the number of hours studied is unrelated to the examination scores. But we have to consider the possibility that we have concluded that there is no relationship based on our one sample, but in fact there is a relationship between to variables in the population we sampled. (SEE EXHIBIT 11.21). this is referred to as TYPE II ERROR and is summarized by beta (B).

THE MEAN IS BEST SUITED FOR WHAT LEVEL OF MEASUREMENT?

Technically, the mean is only suited to measure central tendency for variables measured at the interval and ratio levels, but a s we have already noted, some researchers treat ordinal variables as interval-level.

HISTOGRAMS

The bars are adjacent, are used to display the distribution of quantitative variables that vary along a continuum that has no necessary gaps.

HISTOGRAMS

The bars are adjacent, are used to display the distribution of quantitative variables that vary along a continuum that has no necessary gaps. **contains adjacent bars - good for displaying the distribution of quantitative variables that vary along a continuum that has no necessary gaps

THE CONTEXT CREATED BY THE QUESTIONNAIRE

The context created by the questionnaire has a major impact on how individual questions are interpreted and answered. As a result, survey researchers must carefully design the questionnaire, as well as individual questions. There is no precise formula for a well-designed questionnaire. Nonetheless, some key principles should guide the design of any questionnaire, and some systematic procedures should be considered for refining it.

WHAT DOES THE CORRESPONDENCE OF THE STANDARD DEVIATION TO THE NORMAL DISTRIBUTION ENABLE US TO INFER?

The correspondence of the standard deviation to the normal distribution enables us to infer how confident we can be that the mean (or some other statistic) of a population sampled randomly is within a certain range of the sample mean

WHAT IS THE MEAN?

The mean is the arithmetic average of all scores in the distribution. -the mean and the median in general the mean is the most commonly used measure of central tendency for quantitative values because it takes into account the values of all cases in the distribution and it is the foundation for many other more advanced statistics.

THE MEDIAN IS NOT APPROPRIATE FOR WHAT LEVEL & WHY?

The median is inappropriate for variables measured at the nominal level because their values cannot be put in order and so there is no meaningful middle position.

THE MEDIAN IS BEST SUITED FOR WHAT LEVEL OF MEASUREMENT?

The median is most suited to measure the central tendency of variables measured at the ordinal level, and it can also be used to measure the central tendency of variables measured at the interval and ratio levels.

WHAT IS THE MEDIAN

The median is the value that divides the variable's distribution in half (the 50th percentile).

ALTERNATIVES IN QUALITATIVE DATA ANALYSIS

The qualitative data analyst can choose from many interesting alternative approaches.

WHAT ARE THE REWARDS OF EVALUATION RESEARCH?

The rewards of evaluation research can provide social scientists with rare opportunities to study complex social processes, with real consequences and to contribute to the public good.

WHAT SHOULD BE THE PURPOSE OF STATISTICAL SUMMARY?

The single most important influence on the median and mean should be the purpose of statistical summary. ( If the purpose is to report the middle position in one or more distributions in the median is the appropriate statistic regardless of whether the distribution is skewed.)

WHAT UNITS CAN BE SURVEYED IN CONTENT ANALYSIS?

The units that are surveyed in content analyses can include newspapers, journal articles, court decisions, books, videotapes, themes expressed in agency documents or propositions made in different statements. •Words or other features of these units are then coded to measure the variables involved in research question.

WHAT IS THE DOWNSIDE TO HAVING A VARIETY OF APPROACHES TO QUALITATIVE ANALYSIS?

The variety of approaches to qualitative data analysis makes it difficult to provide a consistent set of criteria for interpreting their quality.

TO CALCULATE THE MEASURES OF VARIANCE THE VARIABLES MUST BE AT WHAT LEVEL OF MEASUREMENT?

To calculate these measures, the variable must be at the interval or ratio level. (Statistical measures of variation are used infrequently with qualitative variables, so these measures are not presented here. )

ONE-SAMPLE T-TEST

Used when comparing pre and post test results of teh same sample -independent is NOMINAL and DICHOTOMOUS - pre and post -dependent variable is INTERVAL/RATIO

ANOVA

Used when dependent variable is INTERVAL/RATIO and independent variable is NOMINAL and has more than 2 categories.

INDEPENDENT SAMPLE T-TEST

Used when the dependent variable is INTERVAL/RATIO and independent is DICHOTOMOUS

WEB SURVEYS

WHAT ARE THE ADVANTAGES OF WEB SURVEYS? •When they are appropriate for a particular population, web surveys have some unique advantages: oSurveys completed on the web can select more honest reports of illicit behavior and of victimization as compared with phone interviews. oThey are easy to complete as respondents simply check on response boxes, and the survey can be programmed to move respondents easily through sets of questions, not presenting questions that do no apply to the respondent. oPictures, sounds and animations can be used as a focus of particular questions and graphic variation can be used to enhance visual survey appeal. o Definitions of terms can "pop up" when respondents scroll over them.

BLACK BOX OR PROGRAM THEORY: WHAT IS A BLACK BOX?

WHAT IS A BLACK BOX? •The "meat and potatoes" of most evaluation research involves determining whether a program has the intended effect. If the effect occurred, the program "worked"; if the effect did not occur, then some would say the program should be abandoned or redesigned. •In this approach, the process by which a program has an effect on outcomes is often treated as a BLACK BOX - that is, the focus of the evaluation research is on whether cases seem to have changed as a result of their exposure to the program between the time they entered the program as inputs and when they exited the program as outputs. •The assumption is that evaluation research requires only the test of a simple input-output model. (SEE EXHIBIT 10.1). There may be no attempt to open the black box of the program process. •SEE EXAMPLE PAGE 275 •If an investigation of program process is conducted a program theory may be developed. A program theory describes what has been learned about how the program has its effect. When a researcher has sufficient knowledge before the investigation begins, outlining a program theory can help guide the investigation of program process in the most productive directions. This is termed THEORY-DRIVEN EVALUATION.

WHEN ARE THE MEAN AND MEDIAN THE SAME?

When a distribution is perfectly symmetric, so that the distribution of values below the median is a mirror image of the distribution of values above the median, the mean and median are the same.

WHAT IS A UNIMODAL DISTRIBUTION?

When a variable distribution has one case or interval that occurs more often than others

WHAT IS HAPPENING WHEN THE DISTRIBUTION IS SKEWED IN A POSITIVE DIRECTION

When the value of the mean is larger than the median, the distribution is skewed in a positive direction, with proportionately more cases with lower than higher values;

WHAT IS TYPE I ERROR

Yet, note: there still remains a 5% possibility that we have reached the wrong conclusion. We have to consider the possibility that we have concluded that there is a relationship based on our one sample, but if that there is no relationship between the two variables in the population we sample. This type of error, called TYPE I ERROR, threatens our ability to conclude that there is an association. TYPE I ERROR is easy to calculate, as it is equal to the alpha level you chose as a criterion for statistical significance p value produced as part of an analysis computed by statistical software. TYPE I ERROR is influenced by the effect of the intervention or the strength of the relationship between an independent variable and a dependent variable. The greater the effect or impact of the intervention, the more likely the effect will be significant. Smaller effects or weaker relationships are less likely to provide statistically significant results. TYPE I ERROR is also influenced by sample size. A small sample is less likely to produce a statistically significant results for a relationship of any given strength. However, larger sample sizes are likely to find statistically significant relationships even when the strength of the relationship is weak.

Histogram

a bar graph for quantitative data categories -Horizontal axis: shows the values of the quantitative data in numerical order -Bars height: represent the frequency of each range or bin of data

Bar Chart

bars for frequency or relative frequency of each category

ordinal

can be ordered along a continuum, rank order ex.faster and slower, stronger and weaker

Phi Coefficient(I)

can be used to express the degree of association between two normal variables in a 2x2 table

PURPOSIVE SAMPLING

can be used to identify opinion leaders and representatives of different roles.

Graphical Summary

chart or graph that visually summaries the data collected -Histogram -Bar/Pie Chart

STAKEHOLDERS

individuals and groups who have some basis of concern with the program.

FEEDBACK

information about service delivery system outputs, outcomes or operations that is available to any program inputs

THEORETICAL SAMPLING

is a systematic approach used when field researchers focus on particular processes that seem to be important and select new settings or individuals that permit comparisons to check their perceptions.

measurement

labeling an outcome or variable ex. how fast someone run

INFERENTIAL STATISTICS

mathematical tools for estimating how likely it is that a statistical result based on data from a random sample is representative of the population from which the sample is assumed to have been selected. Make inferences from data to more general conditions -estimate the degree of confidence that can be placed in generalizations from a sample to the population from which the sample was drawn. (generalize your sample to a larger population) -judgments of the probability that an observed difference between groups is a dependable one or that might have happened by chance in this study. (the difference you see in the DV are not by chance or natural error - or is it he thing of interest (independent variable) With inferential statistics, you are trying to reach conclusions that extend beyond the immediate data alone. For instance, we use inferential statistics to try to infer from the sample data what the population might think. Or, we use inferential statistics to make judgments of the probability that an observed difference between groups is a dependable one or one that might have happened by chance in this study. Thus, we use inferential statistics to make inferences from our data to more general conditions; we use descriptive statistics simply to describe what's going on in our data. ALSO SEE EXTRA SHEET

QUOTA SAMPLING

may be employed to ensure the representation of particular categories of participants.

Chi-Squared

nonparametric statistic used to determine if a distribution of observed frequencies differs from theoretical expected frequencies

INTEGRATIVE APPROACHES

oAttempt to cover issues of concern to both stakeholders and evaluators and to include stakeholders in the group from which guidance is routinely sought. oThe emphasis given to either stakeholder or social concerns is expected to vary with the specific project circumstances. oIntegrative approaches seek to balance the goal of carrying out a project that is responsive to stakeholder concerns with the goal of objective, scientifically trustworthy, and generalizable results. When the research is planned, evaluators are expected to communicate and negotiate regularly with key stakeholders and to take stakeholder concerns into account. Findings from preliminary inquiries are reported back to program decision makers so they can make improvements in the program before it is formally evaluated. When the actual evaluation is conducted, the evaluation research team is expected to operate more autonomously, minimizing intrusions from program stakeholders.

HOW DOES SKEWNESS AFFECT THE MEAN AND MEDIAN

oBut the values of the mean and median are affected differently by skewness, the presence of cases with extreme values on one side of the distribution but not on the other side. o Because the median takes into account only the number of cases above and below the median point, not the value of these cases, it is not affected in any way by extreme values. oBecause the mean is based on adding the value of all the cases, it will be pulled in the direction of exceptionally high (or low) values.

WHAT CRITERIA IS USED TO EVALUATE STATISTICAL SIGNIFICANCE? (BOOK)

oThe criterion is referred to as the alpha level (a) or p value - the probability level that will be used to evaluate statistical significance. oThe alpha level is usually set by the researcher prior to the analysis, while the p value is the probability level often computed by statistical software packages. oIt is conventional in statistics use an alpha level of .05 - that is, to avoid concluding that an association exists in the population from which the sample was drawn unless the probability that the association was due to chance is less than 5%. oIn other words, a statistician normally will not conclude that an association exist between two variables unless he or she can be at least 95% confident that the association was not due to chance.

Quantitative Data

represent numbers: -Height -Credits completed -Brother Cameron's age -Histogram -Mean, Median, Standard Deviation

INPUTS

resources, raw materials, clients and staff that go into a program

WHEN DOES SAMPLING ERROR DECREASE?

sampling error decreases as sample size increases. For this same reason, an association is less likely to appear on the basis of chance in a larger sample than in a smaller sample. Even rather weak associations can be statistically significant with such a large random sample, which means that the analyst must be careful not to assume that just because a statistically significant association may be statistically significant, but still be too weak to be substantively significant.

PROGRAM PROCESS

the complete treatment or service delivered by the program

WHAT IS VARIATION?

the extent to which cases are spread out through the distribution or clustered in just one location. - the spread or variability of responses.

OUTCOMES

the impact of the program process on the cases processed

WHAT IS THE MODE?

the most frequent value in a distribution. -It is also termed the probability average because, being the most frequent value, it is the most probably.

OUTPUTS

the services delivered or new products produced by the program process

WHAT IS THE PRIMARY CONCERN OF THE ANALYST WHEN DOING A FREQUENCY DISTRIBUTION?

to accurately display the distribution's shape - that is, to show how cases are distributed across the values of the variable.

Pie Chart

use wedges of a pie to represent the relative frequency of each category

CHI SQUARE TEST OF ASSOCIATION

used when both Independent and dependent variables are NOMINAL

WE CAN NOT TALK ABOUT THE SKEWNESS (LACK OF SYMMETRY) OF A VARIABLE MEASURES AT WHAT LEVEL? AND WHY'?

we cannon talk about the skewness (lack of symmetry) of a variable measured at the nominal level (a qualitative variable) -If the values of a variable cannot be ordered from lowest to highest, if the ordering of the variables is arbitrary, we cannot say whether the distribution is symmetric because we could just reorder the values to make the distribution more or less symmetric. -Some measures of central tendency and variability are also inappropriate for nominal-level variables.

WHAT IS HAPPENING WHEN THE DISTRIBUTION IS SKEWED IN A NEGATIVE DIRECTION

when the mean is smaller than the median, the distribution is skewed in a negative direction.

WHAT THREE THINGS MUST YOU CONSIDER WHEN CHOOSING AN APPROPRIATE MEASURE OF CENTRAL TENDENCY?

you must consider a variable's level of measurement, the scenes of a quantitative variable's distribution, and the purpose for which the statistic is used.

HOW DO YOU AVOID USING VAGUE WORDS IN A QUESTIONNAIRE?

• It is important to avoid vague language; there are words whose meaning may differ from respondent to respondent. • For Example: "do you usually or occasionally attend program at the community center?" o This will not provide useful information... for the meaning of usually or occasionally can differ for each respondent. o A better way of defining usually (2 or 3 times a week) and occasionally ( 2 or 3 times a month) o OR: as the respondent "How many times they attended programs at the community center in the last month" - the researcher can then classify the responses into categories.

WHAT IS AN IMPORTANT DIFFERENCE BETWEEN QUALITATIVE AND QUANTITATIVE ANALYSIS?

• One of the important differences between qualitative and quantitative data analysis is in the priority given to the views of the researcher and to those of the subjects of the researcher and to those of the subjects of the research. • Qualitative analysis seek to describe their textual data in ways that capture the setting measures and hypotheses what this means is that qualitative data analysis tends to be inductive - drawing from their data, analysts identify important categories in the data, as well as patterns and relationships, through a process of discovery. • There are often no predefined measures or hypotheses. • Good qualitative data analyses are also distinguished by their focus on the interrelated aspects of the setting or group or person under investigation; the focus is on the case as a whole rather than breaking the whole into separate parts. And so the social context of events thought and actions becomes essential for interpretation. o Within this framework it does not really make sense to focus on two variables out of an interacting set of influences and test the relationship between just those two.

WHAT CAN THE QUALITATIVE DATA ANALYST LEAR FROM A TEXT?

• Qualitative analysts may have two different goals: 1. Some view analysis of a text as a way to understand what participants really thought, felt or did in some situation or at some point in time. . The text becomes a way to get "behind the numbers" that are recorded in a quantitative analysis to see the richness of real social experience. 2. Other qualitative researchers have adopted a hermeneutic perspective on texts - that is, a perspective that views text as an interpretation that can never be judged true or false. The text is only one possible interpretation among many.

What is a QUESTIONNAIRE? INTERVIEW SCHEDULE?

•Questionnaire - The survey instrument containing the questions in a self-administered survey. •Interview Schedule - the survey instrument containing the questions asked by the interviewer in an in-person or phone survey

DESCRIBE THE PROCESS OF PROGRESSIVE FOCUSING.

• Qualitative data analysis is an iterative and reflexive process that begins as data are being collected rather than after data collection has ceased. • Next to the field notes or interview transcripts, the qualitative analyst jots down ideas about the meaning of the text and how it might relate to other issues. • This process of reading through the data and interpreting them continues throughout the project. • The analyst adjusts the data-collection process when it begins to appear that additional concepts need to be investigated or new relationships explored - this process is termed PROGRESSIVE FOCUSING.

WHAT ARE THE FEATURES OF QUALITATIVE DATA?

• The focus on text, on qualitative data rather than on numbers, is the most important feature of qualitative analysis. • The "texts" are most often transcripts of interviews or notes from participant observation sessions, but text can also refer to pictures or images that the researcher examines.

WHAT ARE TWO TYPES OF EFFECIENCY ANALYSIS?

• Two types of efficiency analysis are COST-BENEFIT ANALYSIS and COST-EFFECITIVENESS ANALYSIS

WHEN IS AUTHENTICITY ACHIEVED IN QUALITATIVE ANALYSIS?

• When an analysis of qualitative data is judged as successful in terms of these criteria we can conclude that the goal of authenticity has been achieved.

WHAT ARE OTHER FEATURES OF QUALITATIVE RESEARCH?

•A COLLECTION OF QUALITATIVE RATHER THAN QUANTITIATIVE DATA -Qualitative designs methods emphasize observations about natural behavior and artifacts that capture social life as it is experienced by the participants rather than in categories predetermined by the researcher. •EXPLORATORY RESEARCH QUSTIONS, WITH A COMMITMENT TO INDUCTIVE REASONING -Qualitative researchers typically begin their projects seeking not to test hypotheses but to discover what people think and how and why they act in some social setting. Only after many observations do qualitative researchers try to develop general principles to account for their observations. •A FOCUS ON PREVIOUSLY UNSTUDIED PROCESSES AND UNANTICIPATED PHENOMENA -Previously unstudied attitudes and actions cannot adequately be understood with a structured set of questions or within highly controlled experiment. Therefore, qualitative methods have their greatest appeal when we need to explore new issues, investigate hard-to-study groups, or determine the meaning people give to their lives and actions. •AN ORIENTAION TO SOCIAL CONTEXT, TO THE INTERCONNECTIONS BETWEEN SOCIAL PHENOMENA RATHER THAN TO THEIR DISCRETE FEATURES. -The context of concern may be a program or organization, a case, or a broader social context. •A FOCUS ON HUMAN SUBJECTIVITYON THE MEANINGS THAT PARTICPANTS ATTACH TO EVENTS AND THAT PEOPLE GIVE TO THEIR LIVES. -Through life stories people account for their lives - the themes people create are the means by which they interpret and evaluate their life experiences and attempt to integrate these experiences to form a self-concept •A FOCUS ON THE EVENTS LEADING UP TO A PARTICULAR EVEN OR OUTCOME INSTEAD OF GENERAL CAUSAL EXPLANATIONS. -With its focus on particular sectors and situation and the processes that connect them, qualitative research tends to identify causes as particular events embedded within an unfolding, interconnected action sequence. The language of variables and hypotheses appears only rarely in the qualitative literature. •REFLEXIVE RESEARCH DESIGN -The design develops as the research progresses. -Each component of the design may need to be reconsidered or modified in response to new developments or to changes in some other component. •SENSITIVITY TO THE SUBJECTIVE ROLE OF THE RESEARCHER -Little pretense is made of achieving an objective perspective on social phenomena.

WHAT ARE THE 4 ASPECTS OF CROSSTABULATION?

•A cross tabulation table reveals four aspects of the association between two variables: 1.EXISTENCE - the percentage distributions very at all between categories of the independent variable? 2.STRENGTH - how much do the percentage distributions buried between categories of the independent variable? 3.DIRECTION - for quantitative variables, two values on the D pending variable tend to increase or decrease with an increase in value on the independent variable? 4.PATTERN - for quantitative variables are changes in the percentage distribution of the dependent variable fairly regular (simply increasing or decreasing), or do they vary (perhaps increasing, then decreasing, or perhaps gradually increasing and then rapidly increasing)?

HOW DOES QUALITATIVE ANALYSIS DIFFER FROM QUANTITATIVE ANALYSIS?

•A focus on meanings, rather than on quantifiable phenomena •Collection of many data on a few cases, rather than few data on many cases. •Study in depth and detail without predetermined categories or directions, rather than emphasis on analyses and categories determined in advance •Conception of the researcher as an "instrument" rather than as the designer of the objective instruments to measure particular variables •Sensitivity to context, rather than seeking universal generalizations •Attention to the impact of the researcher's and others' values on the course of the analysis, rather than presuming the possibility of value-free inquiry •A goal of rich descriptions of the world, rather than measurement of specific variables.

EXPLAIN A GRAPH OF "NORMAL DISTRIBUTION"

•A graph of a NORMAL DISTRIBTION looks like a bell, with one "hump" in the middle, centered on the population mean, and the number of cases tapering off on both sides of the mean (SEE EXHIBIT 11.15). •A normal distribution is symmetric: if you fold it in half at its center (at the population mean), the two halves would match perfectly.

HOW SHOULD A QUALITATIVE RESEARCHER'S CONCLUSIONS BE ASSESSED?

•A qualitative researcher's conclusions should be assessed by his or her ability to provide a credible explanation for some aspect of social life. •That explanation should capture group member's TACIT KNOWLEDGE of the social processes that were observed, not just their viral statements about these processes. •TACIT KNOWLEDGE - "the largely unarticulated, contextual understanding that is often manifested in nods, silences, humor and naughty nuances" - is reflected in participants' actions as well as their words and in what they fail to state, but nonetheless feel deeply and even take for granted. **Comparing conclusions from a qualitative research project to those obtained by other researchers conducting similar projects can also increase confidence in their authenticity.

WHAT IS THE LOGIC MODEL

•A schematic representation of the various components that make up a social service program. •A LOGIC MODEL is simply a chart of the different components that go into a program. There is no single logic model design; the categories you choose to include often depend upon the purpose for the logic model. •LOGIC MODELS may describe: oTheory and its link to change (THEORY APPROACH MODEL), where attention is on "how and why" a program works oOutcomes (OUTCOME APPROACH MODEL), where the focus of the logic model is to connect resources and activities to expected changes oActivities (ACTIVITIES APPROACH MODEL), or describing what the program actually does.

HOW CAN YOU AVOID CONFUSING PHRASING?

•A simple direct approach to asking a question minimizes confusion. •Use shorter rather than longer words: (brave rather than courages) •Use shorter sentences when you can •A lengthy question often forces the respondent to have to "work hard" that is, to have to read and reread the entire question. - Lengthy questions can go unanswered or can be given only a cursory reading without much thought.

CLEAN THE DATA

•After the data are entered, check carefully for errors - a process called DATA CLEANING •When using CHECK CODING, a second person recodes a sample of the forms and then the percentage of agreement on all the items on the forms is computed; if the percentage falls below a pre-established criterion for accuracy, then all forms should be recoded and reevaluated a second time. •You should also examine the frequency distribution for every variable to see if there are cases with values that fall outside the range of allowable values for a given variable. •When contingency questions are used, you should compute a crosstabulation whereby you compare responses to two variables. . This helps you identify instances in which a respondent should have skipped a question but a response was actually coded. •Any mistakes you find can be corrected by going back to the original questionnaire with the corresponding identification number.

WHY SHOULD RESEARCHERS WRITE CLEAR QUESTIONS?

•All hope for achieving MEASUREMENT VALIDITY is lost unless the questions in a survey are clear and convey the intended meaning to respondents. •The following must be considered: -Survey questions must be asked of many people, not just one person. -The same survey question must be used with each person, not tailored to the specifics of a given conversation. -Survey questions must be understood in the same way by people who differ in many ways. -You will not be able to rephrase a survey question if someone does not understand it, because that would result in a different question for that person. -Survey respondents do not know you and so cannot be expected to share the nuances of expression that help you and your friends communicate. *Every question considered must be reviewed carefully for clarity and ability to convey the intended meaning. - Question clear to a population may not be clear to another - you cannot assume questions used in a previous study were carefully evaluated.

WHAT ARE SOME PROBLEMS WITH OPEN ENDED QUESTIONS?

•Although OPEN ENDED QUESTIONS provide a wealth of information, they also require careful consideration •Administering, analyzing and summarizing open-ended questions can be time consuming and difficult. •People do not like writing a log and may find open-ended question taxing. • The amount of information provided by a person may depend on the person's personality - some people may provide short answers whereas others may provide extensive answers with a great deal of relevant and irrelevant information.

WHAT ARE OPEN ENDED QUESTIONS:

•Are questions without explicit response choices so that the respondents provide their own answers in their own words? •This is usually used when there is little knowledge about a particular topic, and you want to learn as much as possible without limiting the responses. •The Reponses in these types of questions can be used for response categories for future close-ended questions. •Open Ended Questions may also be used to explain what a particular concept means to a respondent. -EXAMPLE: a questionnaire dealing with psychiatric conditions, respondents were asked "yes-No" question "in the last 2 weeks, have you had thoughts that you would be better off dead or hurting yourself in some way?" they were then asked: "can you tell me about it" the purpose of the second question was to expand on the first question and help the analyst to determine whether there was a threat of suicide.

WHY SHOULD YOU AVOID JARGON?

•Avoid using jargon or technical language related to a profession or academic discipline. Words like social justice, empowering and strengths may appear in social work literature but they do not necessarily have a shared meaning in other professions, or the broader community. •Using initials to abbreviate phrases is also a form of professional jargon and should be avoided.

WHAT LEVEL OF MEASUREMENT WORKS BEST WITH THE MEAN?

•Because computing the mean requires adding up the values of the cases, it makes sense to compute a mean only if the values of the cases can be treated as actual quantities, that is if they reflect an interval or ratio level of measurement, or if they are ordinal and we assume that ordinal measures can be treated as interval. •It would make no sense to calculate the mean for a nominal variable such as religion or race.

WHY IS THE STANDARD DEVIATION USED?

•Because the variance provides a measure of the square of the deviations, it does not express the spread and the original units of the measure, so it is hard to interpret the variance. •To correct this, variation is often expressed by the STANDARD DEVIATION •The standard deviation has mathematical properties that make it the preferred measure of variability in many cases, particularly when a variable is normally distributed

WHAT IS THE DIFFERENCE BETWEEN A COMPLETE OBSERVER AND COVERT PARTICIPANT?

•COMPLETE OBSERVER - who does not participate in group activities and is publicly defined as a researcher •COVERT PARTICIPANT - who acts just like other group members and does not disclose his or her research role. Many field researchers develop a role between those extremes, publicly acknowledging being a researcher, but nonetheless participating in group activities.

CONTENT ANALYSIS: WHAT IS CONTENT ANALYSIS?

•CONTENT ANYLYSIS - is the systematic objective, quantitative analysis of message characteristics. •The goal of CONTENT ANALYSIS is to develop inferences from text. -You can think of a content analysis as a survey of some documents or other records of prior communication. In fact, a content analysis is a survey designed with fixed-choice responses so that it produces quantitative data that can be analyzed statistically. •As a form of textual analysis, content analysis is like qualitative data analysis. It involves coding and categorizing text and identifying relationships among constructs identified in the text. •However, as a quantitative procedure, content analysis overlaps with qualitative data analysis only at the margins - the points where qualitative analysis takes on quantitative features or where content analysis focuses on qualitative features of the text.

REFLEXIVITY: WHAT IS REFLEXIVITY?

•Confidence in the conclusions from a field research study is also strengthened by an honest and informative account about how the researcher interacted with subjects in the field, what problems he or she encountered and how these problems were or were not resolved. •Such a "natural history" of the development of the evidence, termed REFLEXIVITY, enables others to evaluate the findings.

WHAT DISTINGUISHES INTENSIVE INTERVIEWING FROM LESS STRUCTURED FORMS OF QUESTIONING?

•Consistency & thoroughness •The goal is to develop a comprehensive picture of the interviewee's background, attitudes and actions in his or her own terms; listen to people as they describe how they understand the worlds in which they live and work •Intensive interview studies do not reveal as directly as does participant observation the social context in which action is taken and opinions are formed. •Like participants observation studies, intensive interviewing engages researchers actively with subjects. •The researchers must listen to lengthy explanations, ask follow-up questions tailored to the preceding answers and seek to learn about interrelated belief systems or personal approaches to things, rather than measure a limited set of variables. •As a result interviews are much longer than standardized interviews - the intensive interview is more like a conversation between partners than between a researcher and a subject. •Intensive interviewers actively try to probe understandings and engage interviewees in a dialogue about what they mean by their comments.

WHAT ARE THE 3 SOURCES OF ERRORS OF NONOBSERVATION?

•Coverage of the population can be inadequate due to poor sampling frame. •The process of random sampling can result in sampling error - differences between the characteristics of the sample members and the population that arise due to chance. •Nonresponse can distort the sample when individuals refuse to respond or cannot be contacted. Nonresponse to specific questions can distort the generalizability of the responses to those questions. ** A well-designed survey will maximize the social rewards, minimize the costs for participating in the survey, and establish trust that the rewards will outweigh the costs. **

HOW IS DOCUMENTATION CRITICAL TO QUALITATIVE RESEARCH?

•Documentation is critical to qualitative research for several reasons: 1.It is essential for keeping track of what will be rapidly growing volume of notes, tapes and documents 2.It provides a way of developing an outline for the analytic process; 3.It encourages ongoing conceptualizing and strategizing about the text.

WHY SHOULD YOU AVOID DOUBLE BARRELED QUESTIONS?

•Double-barreled questions produce uninterpretable results because they actually ask two questions but allow for only one answer. •A respondent may support one part of the questions and not the other.

ETHNOGRAPHY: WHAT IS ETHNOGRAPHY?

•ETHNOGRAPHY- is the study of a culture or cultures that some group of people share. •As a method, it usually is meat to refer to the process of participant observation by a single investigator who immerses him or herself in the group for a long period of time. (often 1 or more years) •There are no particular methodological techniques associated with ethnography, other than just "being there." The analytic process relies on the thoroughness and insight of the researcher to "tell us like it is" in the sitting as he or she experienced it. •EXAMPLE PAGE 250

DESCRIBE SOCIAL SCIENCE APPROACHES

•Emphasize the importance of researcher expertise and maintenance of some autonomy in order to develop the most trustworthy, unbiased program evaluation. •It is assumed that "evaluators cannot passively accept the values and views of the other stakeholders" •Evaluators who adapt this approach derive a program theory from information they obtain on how the program operates and extant social science theory and knowledge, not fro the views of stakeholders.

DESCRIBE STAKEHOLDER APPROACHES

•Encourage researchers to be responsive to program stakeholders. •Issues for study are to be based on the views of people involved with the program and reports are to be made to program participants. •The program theory is developed by the researcher to clarify and develop the key stakeholders' theory of the program. •In STAKEHOLDER PARTICIPATORY RESEARCH - stakeholders are engaged with the researchers as co-researchers and help design, conduct and report the research. •SEE EXAMPLE PG 276

WHAT ARE STAKEHOLDERS?

•Evaluation research also broadens this loop to include connections to parties outside of the program. The evaluation process as a whole and feedback in particular can be understood only in relation to the interests and perspectives of program stakeholders. •STAKEHOLDERS - are those individuals and groups who have some interest in the program. •They might be clients, staff, managers, funders or the public and each has an interest in the outcome of any program evaluation.

WHAT IS EVALUATION RESEARCH?

•Evaluation research is not a method of data collection, like survey research or experiments. •Evaluation research (or program evaluation) utilizes the tools and processes of research for distinctive purpose: to investigate social programs such as substance abuse treatments programs, welfare programs, mental health programs or employment and training programs. •For each evaluation project, the researcher (or evaluator) must select a research design, sample, and method of data collection that are useful for answering the particular questions posed and appropriate for the particular program investigated.

QUANTITATIVE OR QUALITATIVE METHODS

•Evaluation research that attempts to identify the effects of a social program typically is quantitative: Did housing retention improve? Did substance abuse decline? •It is fair to say that when there is an interest in comparing outcomes between an experimental and control group, or tracking change over time in a systematic manner, quantitative methods are favored. •But qualitative methods can add much to quantitative evaluation research studies, including more depth, detail, nuance and exemplary case studies. • The greatest contribution qualitative methods can make in many evaluation studies is investigating program process, that is, finding out what is "inside the black box" • Although it is possible to track service delivery with such quantitative measures as contact hours of staff and frequency of complaints, finding out what is happening to clients and how clients experience the program can often best be accomplished by observing program activities and interviewing staff and clients intensively. • Another good reason for using qualitative methods in evaluation research is the importance of learning how different individuals react to the treatment. • For example, a quantitative evaluation of participant reactions to an adult basic skills program for new immigrants relied heavily on participant's initial statements of their goals. However, qualitative interviews revealed that most new immigrants lacked sufficient experience in America to set meaningful goals; their initial goal statements simply reflected their eagerness to agree with their counselors' suggestions. • Qualitative methods can also help in understanding how social programs actually operate. Complex social programs have many different features, and it is not always clear whether, it is the combination of those features or some particular features that are responsible for the program's effect - or for the absence of an effect.

WHEN DO YOU USE INFERENTIAL STATISTICS?

•Even if it appears that there is a relationship between two variables that is consistent with the researchers hypotheses, the association may have been just due to chance - such are the vagaries of sampling on a random basis (of course, the problem is even worse if the sample is not random). •Therefore, researchers perform test of their hypotheses using inferential statistics to determine whether it is likely that an association exists in the larger population from which the sample was drawn. •INFERENTIAL STATISTICS include many different bivariate statistical test (tests of the relationship of two variables) in multivariate statistical test (test of the relationship of three or more variables). •It is these tests that enable practitioners to conclude that the effectiveness of an intervention is not just a function of chance.

EXAMINING RELATIONSHIPS AND DISPLAYING DATA

•Examining relationships is the centerpiece of the analytic process because it allows the researcher to move form simple description of the people and settings to explanations of why things happened as they did with hose people in that setting. •The process of examining relationships can be captured in a MATRIX that shows how different concepts are connected or perhaps what causes are linked with what effects. (SEE EXHIBIT 9.2 246)

WHY SHOULD YOU AVOID NEGATIVE WORDS OR DOUBLE NEGATIVES IN A SURVEY?

•Example: "Do you disagree that juveniles should not be tried as adults if they commit murder? -Respondents of this question will have a hard time trying to figure out which response matches their sentiments because the statement is written as a double negative. •Such errors can be easily avoided with minor word changes.

WHAT ARE FILTER QUESTIONS/SKIP PATTERNS/CONINGENCY QUESTIONS?

•FILTER QUESTIONS: are important to ensure that questions are asked only of relevant respondents. •If you ask questions about work - you first need to know who is working and who is not working. -These filter questions create --->SKIP PATTERNS: •Based on the response to a filter question, respondents will be asked to either skip one or more questions or to answer those questions. oThe questions asked of the more limited group are referred to as ---→ CONTINGENCY QUESTIONS •Skip patterns should be indicated clearly with arrows or other direction in the questionnaire as demonstrated in (EXHIBIT 7.2 PAGE 188)

WHAT ARE THE BENEFITS OF FOCUS GROUPS?

•FOCUS GROUP methods share with other field research an emphasis on discovering unanticipated findings and exploring hidden meanings. •Although they do no provide a means for developing reliable, generalizable results (traditional strong suits of survey research) focus groups can be an indispensable aid for research studies, such as to develop hypotheses and test survey questions, or for social work practice, such as for needs assessment or to quickly assess the range of opinion about an issue.

WHAT IS FORMATIVE EVALUATION? WHEN IS IT USED?

•FORMATIVE EVALUATION may be used instead of process evaluation when the evaluation procedures that are incorporated into the initial development of the service program can specify the treatment process and lead to changes in recruitment procedures, program delivery, or measurement tools. •Process Evaluation can employ a wide range of indicators. Program coverage can be monitored through program records, participant surveys, community surveys, or comparisons of people using the program, people who drop out of the program, and people who are ineligible for the program. •Service delivery can be monitored through service records completed by program staff, a management of information system maintained by program administrators or reports by program recipients. •Service delivery can also be reviewed through the use of a flowchart to describe program activities. The flowchart typically includes program activities; decision points were, based on the decision a client may receive a different set of activities; and point at which documentation is necessary. •Flowcharts are used to answer questions such as these: •Are all necessary activities present in the program or are there missing activities? •Is there duplication of service effort? •How long does it take for a client to go from initial intake to assessment to intervention?

WHAT ARE FOCUS GROUPS

•Focus groups are groups (usually 7-10 people) of unrelated (do not know each other) individuals that are formed by a researcher and then led in-group discussion of a topic for 1-2 hours. •Researchers ask specific questions and guides the discussion to ensure that group members address these questions, but the resulting information is qualitative and relatively unstructured. •Focus groups do not involve representative samples; instead a few individuals who have the time to participate, have some knowledge pertinent to the focus group topic, and share key characteristics with the target population are recruited for the group. •The characteristics of individuals that determine their inclusion are based on the researcher's conception of the target population for the study.

WHAT IS PHOTOVOICE?

•For years people have been creating a record f the social world with photographs. This creates the possibility of observing the social world through photographs and interpreting the resulting images as "text". •Photovoice builds on this history, but like other qualitative methods, it is designed to preserve the voice of respondents. •Photovoice allows he participants to construct their social reality. •Photovoice is a method to enable people to identify, represent and enhance their community through a specific photographic technique. •Photovoice is a form of participatory research that can be used for community needs assessment.

HOW CAN SOMEONE PROVIDE A FRAME OF REFERENCE IN A QUESTIONAIRRE?

•Frame of reference provides specificity about how respondents should answer the question. •For Example: "overall, the performance of this caseworker is - excellent; good; average; or poor?" -This lack a "Frame of Reference" - the researcher does not know the basis of comparison the respondent is using. -Some may compare the case worker to other caseworkers; some may use a personal "absolute scale" -Therefore the basis of comparison should be specifically stated: "compared with other caseworkers you have had, the performance of this caseworker is..."

WHAT IS A SCATTERPLOT?

•Graphs provide an efficient tool for summarizing relationships among variables. •A SCATTERPLOT - is used to display the relationship between two continuous variables.

FACE-TO-FACE SURVEYS

•IN-PERSON interviews range from highly structured including both closed-ended and open-ended questions, to the relatively unstructured in which the interviewer covers the same topics but varies questions according to the respondent's answers to previous questions. WHAT ARE THE ADVANTAGES OF IN-PERSON INTERVIEWING. •The in-person interviewing has several advantages: oResponse rates are higher than with any Other design oQuestionnaires can be much longer than with mailed or phone surveys; oThe survey can be complex, with both open ended and close ended questions and controlled by the interviewer oThe physical and social circumstances of the interview can be monitored; oThe respondents' interpretations of questions can be probed and clarified.

CONCEPTUALIZATION, CODING, AND CATEGORIZING:

•Identifying and refining important concepts is a key part of the iterative process of qualitative research. •Sometimes conceptualization beings with a simple observation that is interpreted directly, "pulled apart" and then put back together more meaningfully. (EXAMPLE PG 245) •More often analytic insights are tested against new observations, the initial statement of problems and concepts is refined, the researcher then collects more data and interacts with it again, and the process continues. (EXAMPLE PG 236)

WHAT OCCURS WHEN A VARIABLE IS NORMALLY DISTRIBUTED?

•If a variable is normally distributed, 68% of the cases will live between plus and minus one standard deviation from the distributions me, and 95% of the cases will live between 1.96 standard deviations above and below the mean

BUILD ON EXISTING INSTRUMENTS

•If another researcher already has designed a set of questions to measure a key concept and this measure is reliable and valid, then use that instrument. •Although: questions used previously may not concern quite the right concept or may not be appropriate in some ways to your population. •Scales developed much earlier may no longer be appropriate for your population as times change

WHAT ARE THE DISADVANTAGES OF STAKEHOLDER AND SOCIAL SCIENCE APPROACHES?

•If stakeholders are ignore, researchers may find that participants are uncooperative, that their reports are unused and that the next project remains unfunded. •However, if social science procedures are neglected, standards of evidence will be compromised, conclusions about program effects will likely be invalid and results are unlikely to be generalizable to other settings. •These equally undesirable possibilities have led to several attempts to develop more integrated approaches to evaluation research

TELEPHONE SURVEYS

•In a phone survey, interviewers question respondents over the phone and then record their answers. (are generally used with short surveys) WHAT TO MATTERS UNDERMINE THE VALIDITY OF A PHONE SURVEY? 1.Not reaching the proper sampling units 2.Not getting enough complete response to make the results generalizable. RANDOM DIGIT DIALING •A machine calls random phone numbers within the designated exchanges regardless of whether the numbers are published. •When the machine reaches an inappropriate household. (Such as business) the phone number is simply replaced with another. •When the households are contacted, the interviewers must ask a series of questions at the start of the survey to ensure that they are speaking to the appropriate member of the household. ISSUES: 1.Since people often are not home, multiple callbacks are needed for many sample members. Survey research organizations may call back households as many as 20 times. 2.Interviewers must be prepared to deal with distractions as the respondent is interrupted by other household members. 3.Sprinkling interesting questions throughout the questionnaire may help to maintain respondent interest. Rapport between the interviewer and respondent is likely to be lower with phone surveys than with in-person surveys than with in-person interviews and so respondents may tire and refuse to answer all the questions.

DESCRIBE A SUCCESSFUL PROGRAM

•In a successful program, the inputs (resources) are sufficient and appropriate to carry out the specific activities of the program, which set into motion a causal process (based on some practice theory) resulting in the desired outcome.

WHAT 3 CRITERIA SHOULD BE ASSESSED FOR AUTHENTICITY?

•Individual items of information can be assessed in terms of at least three criteria 1.How credible was the information •Were statements made by someone who the researcher had a relationship of trust or someone the researcher just met? Did the informant have a reason to lie? If the statements do not seem to be trustworthy as indicators of actual events can they be used at least to help understand the informant's perspective? 2.Were statements made in response to the researcher's questions or were they spontaneous •Spontaneous statements are more likely to indicate what would have been said had the researcher not been present. 3.How does the presence or absence of the researcher or the researcher's informant influence the actions and statements of other group members? •Reactivity to being observed can never be ruled out as a possible explanation for some directly observed social phenomenon. However, if the research carefully compares what the informant says goes on when the researcher is not present, what the researcher observes directly, and what other group members say about their normal practices, the extent of reactivity can be assessed to some extent.

WHAT IS INTENSIVE INTERVIEWING?

•Intensive interviewing relies on open ended questions (unlike the more structured interviewing used in survey research) •Qualitative researchers do not presume to know the range of answers that respondents might give and seek to hear these answers in the respondents' own words. •Rather than asking standard questions in a fixed order, intensive interviewers allow the specific content and order of questions to vary from one interviewee to another.

WHAT IS PARTICIPANT OBSERVATION

•PARTICIPANT OBSERVATION - is a method in which natural social processes are studied as they happen in their natural setting and are left relatively undisturbed. •It is a means for seeing the social world as the research subjects see it. - in its totality - and for understanding subject's interpretations of that world. •By observing people in interacting with them in the course of their normal activities, participant observers seek to avoid the artificiality of experimental designs and the unnatural structured questioning of survey research. •This method encourages consideration of the context in which social interaction occurs of the complex and interconnected nature of social relations and of the sequencing of events. •Participant Observer represents a continuum of roles ranging from being a COMPLETE OBSERVER to a COVERT PARTICIPANT

NEEDS ASSESSMENT: WHAT IS THE IMPORTANCE OF A NEEDS ASSESSMENT FOR EVALUATION RESEARCH?

•Is the new program needed or is an old one still required? •A NEEDS ASSESSMENT attempts to answer this question with systematic, credible evidence. •Need may be identified and enumerated by social indicators, such as the poverty rate or school drop out rate; by interviews with local experts, such as mental health providers; by surveys of community residents or service providers; by using structured groups, such as focus groups with community residents; or by taking a resource inventory of available services and service capacity, such as is often done by a local United Way. •The evaluation will enumerate need, whereas the assessment of need and subsequent priorities will depend ultimately on the final judgment of key stakeholders. •Evaluators must determine whose definitions or perceptions should be used to describe the level of need. -The decision from whom to collect data is crucial, as different stakeholders will have different perceptions. -A student will describe their needs differently than if professors were asked to comment on student's needs -A good evaluator will do his or her best to capture different perspectives on need and then to help others make sense of the results.

WHAT ARE THE QUESTIONS EVALUATION PROJECTS FOCUS ON RELATED TO THE OPERATION OF OSCIAL PROGRAMS AND THE IMPACT THEY HAVE?

•Is the program needed? •How does the program operate? •What is the program's impact? •How efficient is the program? •(You can see how the logic model is helpful because it provides detail for these questions. If you wanted to measure program activities the outputs column provides measures. If you are interested in programs' impact then the intermediate and final outcomes can serve as a guide. The specific methods used in an evaluation research project depend in part on the particular question of interest.

WHAT IS SOCIAL DESIRABILITY?

•Is the tendency for individuals to respond in ways that make them appear in the best light to the interviewer. •The error is that respondents are not providing their true opinions or answers. •Social Desirability effects are likely to occur when people are discussing issues that are controversial or expressing views that are not popular.

AT THE CONCLUSION OF ANALYSIS QUALITATIVE DATA ANALYST SHOULD ASK WHAT QUESTIONS?

•It is suggested that at the conclusion of their analysis qualitative data analysts ask the following questions: 1.Do they illuminate the phenomenon as lived experience? In other words do the materials bring the setting alive in terms of the people in that setting? 2.Are they based on thickly contextualized materials? We should expect thick descriptions that encompass the social setting studied 3.Are they historically and relationally grounded? There must be a sense of the passage of time between events and the presence of relationships between social actors. 4.Are they processual and interactional? The researcher must have described the research process and his or her interactions within the setting 5.Do they engulf what is known about the phenomenon? This includes situating the analysis in the context of prior research and also acknowledging the researcher's own orientation on starting the investigation.

SIMPLE OR COMPLEX OUTCOMES?

•It is unlikely for a program to only have one outcome. There may be some unanticipated outcomes •Most evaluation researchers attempt to measure multiple outcomes. The result usually is a much more realistic and richer understanding of a program impact SEE EXHIBIT 10.4 PAGE 279

WHAT ARE LIKERT-TYPE RESPONSES?

•LIKERT-TYPE RESPONSES generally ask respondents to indicate the extent to which they agree or disagree with statements. •The response categories list choices for respondents to select their level of agreement with a statement from strongly agree to strongly disagree. •EXHIBIT 7.3 PAGE 188

WHAT ARE MATRIX QUESTIONS?

•MATRIX QUESTIONS - are actually a series of questions that concern a common theme and that have the same response choices. •The questions are written so that a common initial phrase applies to each one (EXHIBIT 7.3) •This format shortens the questionnaire by reducing the number of words that must be used for each question. •It also emphasizes the common theme among the questions and so invites answering each question in relation to other questions in the matrix. •It is important to provide explicit instructions to "circle one response on each line" in a matrix question, since some respondents will think they have completed the entire matrix after they have responded to just a few of the specific questions.

MAIL SURVEYS

•Mail survey is conducted by mailing a questionnaire to respondents, who then administer the survey. •The central concern in a mail surrey is maximizing the response rate. oEven an attractive questionnaire full of clear questions requires additional efforts to maximize the response rate. oA response rate of 70% or higher is desirable. oLower response rates call into question the representativeness of the sample. WHAT IS THE SINGLE MOST IMORTANT RESUIREMENT FOR OBTAINING AN ADEQUATE RESPONSE RATE FOR A MAIL SURVEY? •Sending follow up mailings to nonrespondents is the single most important requirement for obtaining an adequate response rate to a mail survey. •Follow-up mailings explicitly encourage initial nonrespondents to return a completed questionnaire and convey the importance of the efforts.

NARRATIVE ANALYSIS: WHAT IS NARRATIVE ANALYSIS?

•Narrative "displays the goals and intentions of human actors; it makes individuals, cultures, societies, and historical epochs comprehensible as wholes" •NARRATIVE ANALYSIS focuses on "the story itself" and seeks to preserve the integrity of personal biographies or series of events that cannot adequately be understood in terms of their discrete elements. •The coding for a narrative analysis is typical of the narratives as a whole, rather than of the different elements within them. The coding strategy revolves around reading the stories and classifying them into general patterns.

WHAT CAN EFFECTS CAN HAPPEN WITH COMPLETE OBSERVATION?

•Of course, the researchers very presence as an observer alters the social situation being observed - This is called: REACTIVE EFFECTS •It is not "natural" in most social situations for someone to record observations for research, so individuals may alter their behavior. •In social settings involving many people, the complete observer is unlikely to have much effect on social processes. •When the social setting involves few people and observing is unlike the usual activities in he setting, or when the observer differs in obvious respects from eh participants, the COMPELTE OBSERVER is more likely to have an impact. •Many field researchers adopt a role that involves some active participation in the setting. Usually they inform at least some group members of their research interests, but then participate in enough group activities to develop rapport with members and to gain a direct sense of what group members experience.

WHAT IS THE PROCESS OF PHOTOVOICE?

•Photovoice includes the visual images provided by photographs and is accompanied by narratives provided by the participants. •Cameras are distributed to participants who share something in common, such as a social condition like homelessness or living in a particular community. •There is a group facilitator who trains the group members in using the cameras, serves as a resource for information and enables the group to work together. •Their participants engage in the analysis of the photographs, choosing representative pictures, discussing their meaning and developing themes and patterns.

WHAT CAN LEAD TO ERRORS IN NONOBSERVATION?

•Potential problems that can lead to ERRORS OF OBSERVATION: 1.Stem from the ways the questions are written 2.The characteristics of the respondents who answer the questions 3.The way questions are presented in questionnaires 4.The interviewers used to ask the questions SEE EXHIBIT 7.1 PAGE 179

WHAT ARE QUALITATIVE METHODS

•QUALITATIVE METHODS refer to several distinctive research designs including: -PARTICIPANT OBSERVATION -INTENSIVE INTERVIEWING -FOCUS GROUPS.

QUALITATIVE RESEARCH

•QUALITATIVE METHODS refer to several distinctive research designs including: oPARTICIPANT OBSERVATION oINTENSIVE INTERVIEWING oFOCUS GROUPS. •A COLLECTION OF QUALITATIVE RATHER THAN QUANTITIATIVE DATA oQualitative designs methods emphasize observations about natural behavior and artifacts that capture social life as it is experienced by the participants rather than in categories predetermined by the researcher. •EXPLORATORY RESEARCH QUSTIONS, WITH A COMMITMENT TO INDUCTIVE REASONING oQualitative researchers typically begin their projects seeking not to test hypotheses but to discover what people think and how and why they act in some social setting. Only after many observations do qualitative researchers try to develop general principles to account for their observations. •A FOCUS ON PREVIOUSLY UNSTUDIED PROCESSES AND UNANTICIPATED PHENOMENA oPreviously unstudied attitudes and actions cannot adequately be understood with a structured set of questions or within highly controlled experiment. Therefore, qualitative methods have their greatest appeal when we need to explore new issues, investigate hard-to-study groups, or determine the meaning people give to their lives and actions. •AN ORIENTAION TO SOCIAL CONTEXT, TO THE INTERCONNECTIONS BETWEEN SOCIAL PHENOMENA RATHER THAN TO THEIR DISCRETE FEATURES. oThe context of concern may be a program or organization, a case, or a broader social context. •A FOCUS ON HUMAN SUBJECTIVITYON THE MEANINGS THAT PARTICPANTS ATTACH TO EVENTS AND THAT PEOPLE GIVE TO THEIR LIVES. oThrough life stories people account for their lives - the themes people create are the means by which they interpret and evaluate their life experiences and attempt to integrate these experiences to form a self-concept •A FOCUS ON THE EVENTS LEADING UP TO A PARTICULAR EVEN OR OUTCOME INSTEAD OF GENERAL CAUSAL EXPLANATIONS. oWith its focus on particular sectors and situation and the processes that connect them, qualitative research tends to identify causes as particular events embedded within an unfolding, interconnected action sequence. The language of variables and hypotheses appears only rarely in the qualitative literature. •REFLEXIVE RESEARCH DESIGN oThe design develops as the research progresses. oEach component of the design may need to be reconsidered or modified in response to new developments or to changes in some other component. •SENSITIVITY TO THE SUBJECTIVE ROLE OF THE RESEARCHER oLittle pretense is made of achieving an objective perspective on social phenomena.

WHY SHOULD RESEARCHERS REPORT ON WHY INDIVIDUALS DID WHAT THEY DID IN AN OBSERVATION?

•Qualitative data analysts, more often than quantitative researchers, display real sensitivity to how a social situation or process is interpreted from a particular background and set of values and not simply from the situation. •By reporting how and why they think they did what they did, they can help others determine whether or how the researchers' perspectives influenced their conclusions. •"There should be clear 'tracks' indicating the attempt [to show the hand of the ethnographer] has been made"

WHAT ARE THE BENEFITS OF QUALITATIVE STUDIES

•Qualitative research allows researchers to obtain a richer and more intimate view of the social world than more structured methods. •Qualitative methods provide a greater, voice to the people being studied than do the rigid structures imposed by quantitative methods. •Qualitative methods are therefore useful for research with diverse population groups as they focus on the meaning and interpretations ascribed to behavior and not just the behavior alone. •Qualitative methods are useful in shedding light on what is the most appropriate practice for a particular client, given a particular situation and given the context in which it is used. •Qualitative research allows the careful investigator to obtain a richer and more intimate view of the social world than would be possible with more structured methods. •The emphases in qualitative research on inductive reasoning and incremental understanding help to stimulate and inform other research approaches. Exploratory researches of the subjective meanings that motivate individual action are particularly well served by the techniques of participant observation, intensive interviewing and focus groups.

WHAT ARE THE BENEFITS OF QUALITATIVE STUDIES

•Qualitative research allows researchers to obtain a richer and more intimate view of the social world than more structured methods. •Qualitative methods provide a greater, voice to the people being studied than do the rigid structures imposed by quantitative methods. •Qualitative methods are therefore useful for research with diverse population groups as they focus on the meaning and interpretations ascribed to behavior and not just the behavior alone. •Qualitative methods are useful in shedding light on what is the most appropriate practice for a particular client, given a particular situation and given the context in which it is used. •Qualitative research allows the careful investigator to obtain a richer and more intimate view of the social world than would be possible with more structured methods. •The emphases in qualitative research on inductive reasoning and incremental understanding help to stimulate and inform other research approaches. Exploratory researches of the subjective meanings that motivate individual action are particularly well served by the techniques of participant observation, intensive interviewing and focus groups.

WHAT ARE KEY POINTS OF QUALITATIVE DATA

•Qualitative research involves a mode of thinking and investigating different from that used in experimental and survey research both approaches can help social scientists learn about eh social world; the researcher must be ready to use either. •Qualitative data are often supplemented with counts of characteristics or activities. •Quantitative data are often enriched with written comments and observations, and focus groups have become a common tool of survey researchers seeking to develop their questionnaires. Thus the distinction between qualitative and quantitative research techniques is to always clear-cut and combining methods is often a good idea.

WHAT IS STATISTICAL SIGNIFICANCE (BOOK)

•Quantitative findings that a practice method is effective rely on the analyst's ability to demonstrate that a relationship between two variables - an intervention and it's effect or a comparison between two interventions in their effects on an outcome is statistically significant. •STATISTICAL SIGNIFICANCE means that an association is not likely to be due to chance, according to some criterion set by the analyst.

WHAT IS A CLOSE ENDED QUESTION?

•Questions can be designed with or without explicit response choices. When explicit response categories are offered, the type of question is a CLOSED ENDED QUESTION. •SEE EXAMPLE PAGE 185 •CLOSE ENDED QUESTIONS response choices should be mutually exclusive and exhaustive so that every respondent can find one and only one choice that applies to him or her. (Unless the question is one that is "check all that apply") -To make the choices exhaustive, researchers may need to offer at least one option with room for ambiguity. -SEE EXAMPLE PAGE 185

HOW CAN YOU AVOID VAGUENESS?

•Questions should avoid vagueness or abbreviation in a way that results in confusion. •For Example: Residential Location__________? -Does not provide sufficient focus; it is a general question when a specific kind of answer is desired. -Someone could answer squirrel hill (their neighborhood) or their city or just their street.

WHAT TWO THINGS CAN OCCUR WHEN A REPONDENT TRIES TO REMEMBER AN EVENT?

•RECALL LOSS: occurs when a respondent does not remember an event or behavior or can remember only some aspects of the event •Events important to the respondent are likely to be remembered even if they happened long ago. - Whereas events unimportant, even if they happened recently are likely to e forgotten. •A second issue - TELESCOPING EFFECT - in which an event is thought to have happened during a particular time period, although it actually happened before that time period. -Sometimes we remember an event "just like it happened yesterday" because it was so meaningful or important. Unfortunately the event can be reported that way too. •Researchers may choose to use a life history calendar with which respondents sequence the timing of personal events by using standardized visual cues including years (or months) and other vents like births, job changes or moves.

DESCRIBE THE SELECTION PROCESS OF INTENSIVE INTERVIEWS

•Random selection is rarely used to select respondents for intensive interviews, but the selection method still must be carefully considered. •Researchers try to select interviewees who are knowledgeable about the subject of the interview, who are open to talking and who represent the range of perspectives. •Selection of the interviewees should continue if possible, at least until the SATURATION POINT is reached, the point when new interviews seem to yield little additional information.

WHAT IS SYSTEMATIC OBSERVATION?

•SYSTEMATIC OBSERVATION - develops a standard form on which to record variation within the observed setting in terms of the variable of interest. •Such variables might include the frequency of some behavior, the particular people observed and environmental conditions. •SEE EXAMPLE PAGE 226 (Shapiro & Mangelsdorf 1994)

DESCRIBE THE SAMPLING METHOD OF PARTICIPANT OBSERVATION

•Sampling decisions are guided by the need to intensively study people, places or phenomena of interest •Most qualitative researchers limit their focus to just one or a few sites, programs, or specific types of people so they can focus all their attention on the social dynamics of those settings or the activities or attitudes of these people. •The sample must be appropriate and adequate for the study, even if it is not representative. •The qualitative researcher may select one or more "critical cases" precisely because they are judged to be typical or "deviant cases" that provide a useful contrast. Within a research site, plans may be made to sample different settings, people, events and artifacts. (EXHIBIT 8.2 PG 222). •Studying one case for setting almost always strengthens the causal conclusions and makes the findings more generalizable.

WHAT ARE SERVICE COMPLETIONS?

•Service Completions refer to the agency's definition of a client who has received the "full package of services" as opposed to someone who began the program but dropped out prematurely. Service completions are often defined as encompassing a mix of services oFor example: clients in a partial hospitalization program have completed the full range of services if they attended 80% of their assigned groups, participated in 80% of their scheduled individual counseling sessions and attended al meetings with their families. oAnother way of defining a service completion is by using some measure of time. In the hospitalization program, client reviews are done every 30 days; if it made sense to the agency administrator a service completion might have been defined as this unit of time. oDefining a service completion is particularly important for evaluations that examine program impact on the clients. A client should have the "full does" of activities if improvements are to occur, or else why provide the various activities? This is similar to the warning your doctor gives you when prescribing an antibiotic for an ear infection, namely that if you do not take the antibiotic for the full 12 days. U will not get better. When evaluators do studies of client change, they want to include only those persons who have received the full mix of program activities. (The logic model succinctly summarizes a great deal of information about the Program. After completing the logic model, the evaluator and often Stakeholders, should have a better understanding about he LOGIC Underlying the program)

WHY IS IT IMPORTANT TO REDUCE THE RISK OF BIAS?

•Specific words in a survey should not trigger biases unless that is the researcher's intent. -These types of questions are referred to as LEADING QUESTIONS - because they lead the respondent to a particular answer. •Biases or Loaded Phrases tend to produce misleading answers as certain responses become less attractive. •Responses can also be bias when response categories do not reflect the full range of possible alternatives. -SEE EXAMPLE PAGE 184 -A similar bias occurs when some but not all possible responses are included in the question. •Example: "what do you like about your community such as he parks and schools? - Focuses the respondent on those categories, and other answers may be ignored. •It is best left to the respondent to answer the question without such response cues.

GENERALIZABILITY

•Survey methods lend themselves to probability sampling from large populations. Thus it is appealing when sample generalizability is a central research goal. •Survey research is often the only means available for developing a representative picture of the attitudes and characteristics of a large population. •They are also the method of choice when cross-population generalizability is a key concern because they allow a range of social contexts and subgroups to be sampled. - The consistency of the findings can be examined across various sampled subgroups

EFFICIENCY

•Surveys are efficient in that many variables can be measured without substantially increasing the time or cost. •It can be collected from many people at relatively low cost and depending on the survey design, relatively quickly

WHAT ARE THE STAGES OF CONTENT ANALYSIS?

•The CONTENT ANALYSIS proceeds through several stages: •Identify a population of documents or other textual sources for stud. •Determine the unites of analysis. These could be items such as a newspaper article court decisions, research articles or case studies. •Select a sample of units from the population. The simplest strategy might be a simple random sample of documents. However a stratified sample might be needed to ensure adequate representation of community newspapers in large and small cities. •Design coding procedures for variables to be measured. This requires deciding what unit of text to code such as words, sentences, themes or paragraphs. Then the categories into which the text units are to be coded must be defined. These categories may be broad, such as client goal, or narrow such as client improves behavior. •Test and refine the coding procedures. Clear instructions and careful training of coders are essential. •Base statistical analyses on counting occurrences of particular words, themes or phrases and test relations between different variables. •After coding procedures are developed, their reliability should be assessed by comparing different coders' codes for the same variables. The criteria for judging qualitative content analyses of text require the same standards of reliability and validity used for quantitative data.

WHAT IS THE RANGE?

•The RANGE - is a simple measure of variation, calculated as the highest value in a distribution minus the lowest value: •Range = Highest value - Lowest Value •It often is important to report the whole range of actual values that might be encountered. However, because the range can be drastically altered by just one exceptionally high or low value termed an outlier, it is not a good summary measure of a variable's distribution for most purposes.

MAINTAIN CONSISTENT FOCUS

•The contents of the survey should be guided by a clear conception of the research problem. Under investigation and the population to be sampled. •Throughout the process of the questionnaire design - the research objective should be the primary basis for making decisions about what to include and exclude and what to emphasize or treat in a cursory fashion. •The questionnaire should be viewed as an integrated whole in which each section and every question serves a clear purpose related to the study's objective and each section complements other sections.

DOCUMENTATION: DESCRIBE THE PROCESS OF DOCUMENTATION.

•The data for a qualitative study most often are notes jotted down in the field or during an interview - from which the original comments, observation and feelings are reconstructed - or text transcribed from audiotapes. •The first formal analytical step is DOCUMENTATION -The various contacts, interviews, written documents, and whatever it was that preserves a record of what happened all need to be saved and listed.

ORDER THE QUESTIONS

•The order the questions are presented will influence how respondents will react to the questionnaire as a whole and how they may answer some questions. •The 1st step: questions should be sorted into broad thematic categories, which then become separate sections in eh questionnaire. -Both the sections and the questions within the sections must then be organized in logical order that would make sense in a conversation. -The first question deserves special attention, particularly if the questionnaire is to be self-administered. - This question signals to the respondent what the survey is all about, if it will be interesting and how easy it will be to complete. •It is not the time to ask sensitive questions as this might turn off a potential respondent. •The first question should also apply to everyone in the sample. •Question order can lead to CONTEXT EFFECTS: when one or more questions influence how subsequent questions are interpreted. •The potential for context effects is greatest when two ore more questions concern the same issue or closely related issues.

WHAT IS THE PROCES OF EVALUATION RESEARCH?

•The process of evaluation research as a simple systems model: •1st - clients, customers, students or some other persons or units enter the program as INPUTS. oIn addition to clients, social programs require other inputs such as staff and resources such as money, supplies and equipment. oClients or consumers then participate in a set of activities that constitute the PROGRAM PROCESS. It is these activities that are designed to have some impact on the client. •The direct product of the program's service delivery process is its OUTPUT. oProgram outputs may include clients served, case managers trained, food parcels delivered, or child abuse reports investigated. oThe program outputs primarily serve to indicate that the program is operating. oProgram outcomes indicate the impact of the program on the recipients of the activities. Any social program is likely to have multiple outcomes, some intended and some unintended. •A distinctive feature of evaluation research is that variation in both outputs and outcomes influence the inputs to the program through a FEEDBACK process. (EXHIBIT 10.1) oThis feedback can identify factors that lead to a successful program.

PROCESS EVALUATION: WHAT IS THE PURPOSE OF A PROCESS EVALUATION?

•The purpose of PROCESS EVALUATION is to investigate how the program is operating. •Process Evaluations are completed to answer a variety of different questions related to the operation of a particular program. -One question concerns program coverage: is the program serving it Target Population? -Other questions focus on service delivery: has the program been implemented as designed? What are the outputs of various program activities? -Process evaluations are conducted to identify specific activities that produced program outcomes as well as to determine client satisfaction with the program activities.

WHAT IS AGREEMENT BIAS?

•The tendency for some respondents to agree with a statement just to avoid seeming disagreeable. -To reduce the likelihood of agreement bias, you should present both sides of attitude scales in the question. -In general, do you belief that individuals or social condition are more to blame for poverty in the United States? Should be phrased to make each one seem socially acceptable. -You should also consider replacing a range of response alternatives with other alternatives that focus on the word agree. •EXAMPLE: the question "to what extent do you support or oppose the new health care plan"? (Response choices range from strongly support to strongly oppose) is probably a better approach than the question "to what extent do you agree or disagree with the statement: the new health car plan is worth of support"? (Response choices range from strongly agree to strongly disagree.

GROUNDED THEORY: WHAT IS GROUNDED THEORY?

•Theory development occurs continually in qualitative data analysis. •The goal of many qualitative researchers is to create GROUNDED THEORY - that is, to inductively build a systematic theory that is "grounded" in, or based on, the observations. •The observations are summarized into conceptual categories, which are tested directly in the research setting with more observations. Over time as the conceptual categories are refined and linked a theory evolves. •As observation, interviewing and reflection continue, researchers refine their definitions of problems and concepts and select indicators. They can check the frequency and distribution of phenomena: how many people made a particular type of comment? How often did social interaction lead to arguments? Social systems models may then be developed, which specify the relationships among different phenomena. •For the final analysis, the researchers check their models carefully against their notes and make a concerted attempt to discover negative evidence that might suggest that the model is correct

ENTER THE DATA

•There are several methods of data entry •Research using computer-assisted telephone interviews are designed so that as responses are given, the data are immediately entered into a computer program. •Another method is to use optical scan sheets. •A third method is to directly enter the data by hand into a spreadsheet such as Excel or Lotus or into a statistical package such as SPSS or SAS.

CODE OPEN-ENDED QUESTIONS

•There are two types of open ended questions: a.When the entire question is left open ended b.Where you have other (specify) as a potential response in a list or responses. •You will have to develop categories for each type; although the process is similar it is easier to develop response categories for "other" because the responses are likely to be fewer in number and most common responses or at least most anticipated responses already appear in the questionnaire.

REVIEW THE FORMS

•There may be mistakes or unanticipated problems. You need to establish rules that you will follow when you encounter such mistakes. •Here are some problems you may encounter: a.Responses that are not clearly indicated. For example, circles crossing more than one category or an X or check mark falling between responses b.Respondents misreading instructions. Sometimes respondents do not follow instructions about how to respond to a question. They might check responses when they are asked to rank different responses. Or may circle multiple answers when asked to choose the best answer. c.Incomplete questionnaires. Some respondents may decide not to complete the entire instrument or may end the interview before it is completed. You have to decide whether to include the Reponses you have obtained and treat the rest as missing or consider the entire instrument as missing d.Unexpected responses. You may get responses that you had not anticipated, for example, you might ask age and get a response like 30 ½ decide how you will treat such responses.

ASSIGN A UNIQUE IDENTIFYING NUMBER

•This number should be assigned to each form, questionnaire, survey or transcript •It should appear on the form •It should include the identifier as a variable in the data •Having an identifier enables you to go back to the original form if you find data entry errors or decide to enter additional information •If you collect data from the same people at different pints in time, the identifier helps you link their responses

QUALITATIVE COMPARATIVE ANALYSIS: WHAT IS QUALITATIVE COMPARITIVE ANALYSIS?

•This procedure identifies the combination of factors that had to be present across multiple cases to produce a particular outcome. •EXAMPLE 251-252

HOW DO YOU DETERMINE THE MEDIAN?

•To determine the median, we simply list the variable's values in numerical order and find the value of the case that has an equal number of cases above and below it. •The median in a frequency distribution is determined by identifying the value corresponding to a cumulative percentage of 50

WHAT ARE THE GOALS OF COVERT PARTICIPATION?

•To lessen the potential for reactive effects and to gain entry to otherwise inaccessible settings, some field researchers have adopted the role of COVERT PARTICIPATION - keeping their research secret and trying their best to act like other participants in a social setting or group. •Some problems include: - Covert participants cannot ask questions that will arouse suspicion, so they often have trouble clarifying the meaning of other participants attitudes or actions. -The role is difficult to play and researchers' spontaneous reactions to every event are unlikely to be consistent with those of the regular participants and raising suspicion that the researcher is not "one of us"

EFFICIENCY ANALYSIS: WHAT IS EFFICIENCY ANALYSIS?

•Whatever the program's benefits, are they sufficient to offset the program's cost? Are the funders getting their money's worth? •These efficiency questions can be the primary reason that funders require evaluation of the programs they fund. •As a result, EFFICIENCY ANALYSIS, which compares program effects with costs, is often a necessary component of an evaluation research project.

LOGIC MODEL

•When an agency completes a LOGIC MODEL of one or ore of its programs, it is deconstructing its program. •The logic model provides the basis for an agency asking a variety of evaluation questions: is the program needed? Are the program assumptions correct? Is the treatment model appropriate? Are the activities implemented correctly with the right kinds of resources? Does the program produce the desired outcomes?

WHAT GUIDELINES SHOULD BE CONSIDERED WHEN WRIING RESPONSE CATEGORIES FOR CLOSE ENDED QUESTIONS?

•When writing response categories for closed ended questions there are several guidelines that might help improve the questions. •It is important the questions are mutually exclusive and that the list is exhaustive. Others include: 1.AVOID MAKING AGREEMENT AGREEABLE (Agreement Bias) 2.MINIMIZE FENCE-SITTING AND FLOATING 3.FILTER QUSTIONS 4.UTILIZE LIKERT-TYPE CATEGORIES