EST255 Research Methods Exam 2

Qualitative Data Analysis Strategies:

-A priori: determined prior to data analysis -Posteriori: determined during data analysis -Strategy is determined by: --Level of knowledge about the subject of inquiry --Theory used to inform analysis

NVivo

-Available in 7 different languages -Data can be formatted as Word documents, PDFs, audio files, database tables, spreadsheets, videos and pictures -Nodes serve as virtual filing boxes for sorting data -You can interchange information between NVivo and other applications like Microsoft Word and Excel, IBM SPSS Statistics and EndNote -Allows storage of a project database and research materials as a single file -Can merge two or more projects

Method Drawbacks

-Categories are not in themselves the analysis -Loss of meaning and depth when transitioning qualitative data into quantifiable terms -Does not attempt to make sense of what is communicated

Technological Objectivity

-Categories must be clear and operationally defined -Rules are created, determining when and when not to code

Method Benefits

-Categorizes data for purposes of analysis -Systemic -Objective -Quantifiable Used to examine trends, anomalies, comparisons -Conducive to hypothesis testing

Monetizing Costs

-Cost of capital -Sunk costs -Indirect costs

Mitigation Strategies for Reliability

-Create a coding guide -Set a unit of observation -Have more than one coder and test for intercoder reliability

Qualitative Research Characteristics

-Data non-numerical -Researcher can dive deeper into subject, but difficult to get representative samples of a population -usually uses inductive approach- not conducive to hypothesis testing -Greater risks of subjectivity

Regardless of whether it is a CBA or a CEA, there are a:

-Host of assumptions -Complicated calculations -Careful judgement required

When to Use a CBA

-Most useful when you are analyzing a single program or policy to determine whether the program's total benefits to society exceed the costs. -Also useful when you are comparing alternative programs to see which one achieves the greatest benefit to society. -Major difficulty: it is often difficult to place dollar values on all (or most) costs and benefits.

When to Use a CEA

-Most useful when you know the outcome you desire and you are determining which of a set of alternative programs or projects achieves the greatest outcome for the costs. -Also useful in cases where major outcomes are either intangible or otherwise difficult to monetize. -Major difficulty: it provides no value for the output, leaving that to the subjective judgment of the policymaker.

Rigorousness of the Method

-Need to be careful of following to ensure results -Obtaining a representative sample of the content of interest -Determining measurable units -Obtaining reliability in coding -Defining terms categories for operation -Ensuring validity and utility of finding(s)

Qualitative Data Analysis: Reliability

-Operations of study can be repeated -Operations of study are consistent --Across researchers --Across projects

Characteristics of Actor-Network Theory (ANT)

-Originally developed to think about science and technology innovation -Methodology treats systems as networks of connections -Treats object within networks as actors interacting through connections (not all actors are human) -Networks are volatile, fluid, and emergent like real life! -Useful for thinking thru complexities involving social and biophysical system -note: different factors in the network can influence each other reciprocally -Not conducive to hypothesis testing exploratory- cannot be made to make predictions -Can take advantage of multiple types of qualitative data

"Snowball Sample"

-Recruit a participant, ask them who else we should talk to -Asking them who thinks differently than you?

General Process for Qualitative Research

-Well-defined research objective -Careful collection of evidence -Systematic processing and analysis of evidence -Interpreting the results of analysis based on well-defined criteria -Drawing conclusions

What is a text?

-Written words -Sounds -Images -Art -The body -Anything that can serve as a sign, relay a message or be interpreted

Content Analysis

-a systematic analysis of the content rather than the structure of a communication, such as a written work, speech, or film -used to measure human behavior through how we communicate -Allows for the quantification of qualitative data

Goals of ANT

-deconstructs complexity -Models scientific, social (including economic an political and cultural factors, their interactions and influences

Steps of a Content Analysis

1. Determine research Q 2. Examine past and current knowledge of the subject (lit rev) 3. Define terms/categories operationally (i.e. create a codebook) 4. Determine measurable units 5. Determine coding process 6. Code relevant content 7. Analyze codes based on research Q (s) 8. Interpret results

Steps in CBA and CEA

1. Set the framework for the analysis 2. Decide whose costs and benefits should be recognized 3. Identify and categorize costs and benefits 4. Project costs and benefits over the life of the program, if applicable 5. Monetize (place a dollar value on) costs 6. Quantify benefits in terms of units of effectiveness (for CEA), or monetize benefits (CBA) 7. Discount costs and benefits to obtain present values 8. Compute a cost-effectiveness ratio (for CEA) or a net present value (for CBA) 9. Perform sensitivity analysis 10. Make a recommendation where appropriate

Steps 1

1. Set the framework for the analysis: -Will you undertake a cost-benefit analysis or a cost-effectiveness analysis? -Are you evaluating one program or comparing two or more? -Does the program have multiple objectives or just one major focus?

Steps 2

2. Decide whose costs and benefits should be recognized: How do we decide?

Steps 8

Compute a cost-effectiveness ratio (for CEA) or a net present value (for CBA): -Use CEA & CBA formulas

Tips for CBAs and CEAs (1)

Consider each activity in turn, asking: What is the real cost of doing this? What is at risk in this transaction? Is there an opportunity cost? Who loses? Who gains or benefits? How does this contribute to our success? What opportunities does this activity present?

Steps 7

Discount costs and benefits to obtain present values: -$100 today is worth more to a person or organization than the same $100 promised to that person or organization one year from now, and much more than the same $100 promised for ten years from now. -The social discount rate is meant to reflect society' s impatience or preference for consumption today over consumption in the future

Qualitative Data Analysis Software

Does: Helps to manage and organize data -Code -Sort -Search -Count Does not: -data analysis for you- that's all you

Computer-Assisted Qualitative Data Analyses Software (CAQDAS)

Examples of software: -ATLAS.ti (available for PC &Mac) -QSR NVivo (available for PC&Mac) -Dedoose (cloud-based software) -Today's: NVivo

Qualitative Research Characteristics Best Used For:

Exploring process versus outcomes (i.e., better at answering "why" and "how" questions) Formative studies where not much is known about a subject - relates to its use in mixed methods studies Topics where numerical data collection just is not possible/appropriate (e.g., researching conflict situations)

Steps 3

Identify and categorize costs and benefits: -How do you categorize costs and benefits? -What is the difference between financial and social costs and benefits?

Mitigation Strategies for Validity Concerns

Internal validity: -Check findings against own biases -Check interpretations against researchers Construst validity -think through variables & scenarios External validity: -Sample until saturation point on a subject is reached -Seek out alternative points of view to the study participants already represented

Before the Analysis..

Know the status quo: the state of the world in the absence of the program or policy

Tips for CBAs and CEAs (2)

Look for costs/risks or improvements in: Financial costs Performance Knowledge loss or gain Learning curves and competence Allocation of resources (including human) Demands on internal structures and systems Physical environment

Qualitative Data Analysis

Looking to address validity and reliability to get middle bulls eye

Steps 10

Make a recommendation where appropriate: -How decision will be taken based on the CBA or CEA results?

Steps 5

Monetize (place a dollar value on) costs: -How do you monetize costs?

Monetizing Benefits

Nonmarket goods and services --Willingness to pay --Shadow prices (think ecosystem services) -Cost avoidance/savings -Time saved -Increased productivity -Increased property values -Taxes (produce revenue) -Value of the environment

Steps 9

Perform sensitivity analysis: -Types of sensitivity analyses: partial (one assumption) and extreme case (all the uncertain parameters).

Types of Qualitative Data

Primary data: collected by the researcher --Interview transcripts --Focus group transcripts Secondary: produced by someone else, but used by the researcher --Texts

Steps 4

Project costs and benefits over the life of the program, if applicable: -How do the costs and benefits change over time?

Steps 6

Quantify benefits in terms of units of effectiveness (for CEA), or monetize benefits (for CBA): -How do we quantify benefits?

Social CBAs and CEAs:

Takes into account the costs and benefits - whether monetary or non-monetary - that accrue to everyone in society

Luhmann's Theory of Social Function Systems

The big 6 -Economy -Law -Religion ..

Before determining software think:

Will it meet your needs in all aspects? -Compatibility -Cost -Shortcuts -Is data set large enough that data management software will be useful? -Intention with data

Inductive

a type of reasoning that presents cases or evidence that lead to a logical conclusion

Qualitative Data Analysis External Validity

ability to generalize Threats to external validity Participant selection bias Setting threats

Transfers:

alter the distribution of resources within society

Financial:

are cash outlays of the organization considering the program or project

Indirect:

by-products, multipliers, spillovers, or investment effects of the project or program

Cost-benefit analysis (CBA):

compares the costs with the dollar value of all (or most) of a program's many benefits

Qualitative Data Analysis Validity

credibility and generalizability of study conclusions Internal validity: ensuring that a research is interpreting findings correctly Construct validity: ability to give oneself a reasonable chance of being able to actually find the causal relationship

Conducive

making a certain situation or outcome likely or possible

Units of effectiveness:

measure of any quantifiable outcome central to the program's objective (e.g. dollars per life saved)

Real:

net gains or losses to society

Qualitative Data Analysis Coding:

process of organizing data into categories or themes -cognitive -behavioral -motivational

Social:

represents real costs to society

Intangible:

such things as the value of wilderness or an increased sense of community

Cost-effectiveness analysis (CEA):

technique that relates the costs of a program to its key outcomes or benefits

Direct:

those that are closely related to the primary objective of the project

Tangible:

those that the analyst can readily identify in unit terms for CEA and can convert to dollars for CBA.

Net Benefits =

total benefits - total cost

Cost-Effectiveness Ratio =

total cost / units of effectiveness