EST255 Research Methods Exam 2
Qualitative Data Analysis Strategies:
-A priori: determined prior to data analysis -Posteriori: determined during data analysis -Strategy is determined by: --Level of knowledge about the subject of inquiry --Theory used to inform analysis
NVivo
-Available in 7 different languages -Data can be formatted as Word documents, PDFs, audio files, database tables, spreadsheets, videos and pictures -Nodes serve as virtual filing boxes for sorting data -You can interchange information between NVivo and other applications like Microsoft Word and Excel, IBM SPSS Statistics and EndNote -Allows storage of a project database and research materials as a single file -Can merge two or more projects
Method Drawbacks
-Categories are not in themselves the analysis -Loss of meaning and depth when transitioning qualitative data into quantifiable terms -Does not attempt to make sense of what is communicated
Technological Objectivity
-Categories must be clear and operationally defined -Rules are created, determining when and when not to code
Method Benefits
-Categorizes data for purposes of analysis -Systemic -Objective -Quantifiable Used to examine trends, anomalies, comparisons -Conducive to hypothesis testing
Monetizing Costs
-Cost of capital -Sunk costs -Indirect costs
Mitigation Strategies for Reliability
-Create a coding guide -Set a unit of observation -Have more than one coder and test for intercoder reliability
Qualitative Research Characteristics
-Data non-numerical -Researcher can dive deeper into subject, but difficult to get representative samples of a population -usually uses inductive approach- not conducive to hypothesis testing -Greater risks of subjectivity
Regardless of whether it is a CBA or a CEA, there are a:
-Host of assumptions -Complicated calculations -Careful judgement required
When to Use a CBA
-Most useful when you are analyzing a single program or policy to determine whether the program's total benefits to society exceed the costs. -Also useful when you are comparing alternative programs to see which one achieves the greatest benefit to society. -Major difficulty: it is often difficult to place dollar values on all (or most) costs and benefits.
When to Use a CEA
-Most useful when you know the outcome you desire and you are determining which of a set of alternative programs or projects achieves the greatest outcome for the costs. -Also useful in cases where major outcomes are either intangible or otherwise difficult to monetize. -Major difficulty: it provides no value for the output, leaving that to the subjective judgment of the policymaker.
Rigorousness of the Method
-Need to be careful of following to ensure results -Obtaining a representative sample of the content of interest -Determining measurable units -Obtaining reliability in coding -Defining terms categories for operation -Ensuring validity and utility of finding(s)
Qualitative Data Analysis: Reliability
-Operations of study can be repeated -Operations of study are consistent --Across researchers --Across projects
Characteristics of Actor-Network Theory (ANT)
-Originally developed to think about science and technology innovation -Methodology treats systems as networks of connections -Treats object within networks as actors interacting through connections (not all actors are human) -Networks are volatile, fluid, and emergent like real life! -Useful for thinking thru complexities involving social and biophysical system -note: different factors in the network can influence each other reciprocally -Not conducive to hypothesis testing exploratory- cannot be made to make predictions -Can take advantage of multiple types of qualitative data
"Snowball Sample"
-Recruit a participant, ask them who else we should talk to -Asking them who thinks differently than you?
General Process for Qualitative Research
-Well-defined research objective -Careful collection of evidence -Systematic processing and analysis of evidence -Interpreting the results of analysis based on well-defined criteria -Drawing conclusions
What is a text?
-Written words -Sounds -Images -Art -The body -Anything that can serve as a sign, relay a message or be interpreted
Content Analysis
-a systematic analysis of the content rather than the structure of a communication, such as a written work, speech, or film -used to measure human behavior through how we communicate -Allows for the quantification of qualitative data
Goals of ANT
-deconstructs complexity -Models scientific, social (including economic an political and cultural factors, their interactions and influences
Steps of a Content Analysis
1. Determine research Q 2. Examine past and current knowledge of the subject (lit rev) 3. Define terms/categories operationally (i.e. create a codebook) 4. Determine measurable units 5. Determine coding process 6. Code relevant content 7. Analyze codes based on research Q (s) 8. Interpret results
Steps in CBA and CEA
1. Set the framework for the analysis 2. Decide whose costs and benefits should be recognized 3. Identify and categorize costs and benefits 4. Project costs and benefits over the life of the program, if applicable 5. Monetize (place a dollar value on) costs 6. Quantify benefits in terms of units of effectiveness (for CEA), or monetize benefits (CBA) 7. Discount costs and benefits to obtain present values 8. Compute a cost-effectiveness ratio (for CEA) or a net present value (for CBA) 9. Perform sensitivity analysis 10. Make a recommendation where appropriate
Steps 1
1. Set the framework for the analysis: -Will you undertake a cost-benefit analysis or a cost-effectiveness analysis? -Are you evaluating one program or comparing two or more? -Does the program have multiple objectives or just one major focus?
Steps 2
2. Decide whose costs and benefits should be recognized: How do we decide?
Steps 8
Compute a cost-effectiveness ratio (for CEA) or a net present value (for CBA): -Use CEA & CBA formulas
Tips for CBAs and CEAs (1)
Consider each activity in turn, asking: What is the real cost of doing this? What is at risk in this transaction? Is there an opportunity cost? Who loses? Who gains or benefits? How does this contribute to our success? What opportunities does this activity present?
Steps 7
Discount costs and benefits to obtain present values: -$100 today is worth more to a person or organization than the same $100 promised to that person or organization one year from now, and much more than the same $100 promised for ten years from now. -The social discount rate is meant to reflect society' s impatience or preference for consumption today over consumption in the future
Qualitative Data Analysis Software
Does: Helps to manage and organize data -Code -Sort -Search -Count Does not: -data analysis for you- that's all you
Computer-Assisted Qualitative Data Analyses Software (CAQDAS)
Examples of software: -ATLAS.ti (available for PC &Mac) -QSR NVivo (available for PC&Mac) -Dedoose (cloud-based software) -Today's: NVivo
Qualitative Research Characteristics Best Used For:
Exploring process versus outcomes (i.e., better at answering "why" and "how" questions) Formative studies where not much is known about a subject - relates to its use in mixed methods studies Topics where numerical data collection just is not possible/appropriate (e.g., researching conflict situations)
Steps 3
Identify and categorize costs and benefits: -How do you categorize costs and benefits? -What is the difference between financial and social costs and benefits?
Mitigation Strategies for Validity Concerns
Internal validity: -Check findings against own biases -Check interpretations against researchers Construst validity -think through variables & scenarios External validity: -Sample until saturation point on a subject is reached -Seek out alternative points of view to the study participants already represented
Before the Analysis..
Know the status quo: the state of the world in the absence of the program or policy
Tips for CBAs and CEAs (2)
Look for costs/risks or improvements in: Financial costs Performance Knowledge loss or gain Learning curves and competence Allocation of resources (including human) Demands on internal structures and systems Physical environment
Qualitative Data Analysis
Looking to address validity and reliability to get middle bulls eye
Steps 10
Make a recommendation where appropriate: -How decision will be taken based on the CBA or CEA results?
Steps 5
Monetize (place a dollar value on) costs: -How do you monetize costs?
Monetizing Benefits
Nonmarket goods and services --Willingness to pay --Shadow prices (think ecosystem services) -Cost avoidance/savings -Time saved -Increased productivity -Increased property values -Taxes (produce revenue) -Value of the environment
Steps 9
Perform sensitivity analysis: -Types of sensitivity analyses: partial (one assumption) and extreme case (all the uncertain parameters).
Types of Qualitative Data
Primary data: collected by the researcher --Interview transcripts --Focus group transcripts Secondary: produced by someone else, but used by the researcher --Texts
Steps 4
Project costs and benefits over the life of the program, if applicable: -How do the costs and benefits change over time?
Steps 6
Quantify benefits in terms of units of effectiveness (for CEA), or monetize benefits (for CBA): -How do we quantify benefits?
Social CBAs and CEAs:
Takes into account the costs and benefits - whether monetary or non-monetary - that accrue to everyone in society
Luhmann's Theory of Social Function Systems
The big 6 -Economy -Law -Religion ..
Before determining software think:
Will it meet your needs in all aspects? -Compatibility -Cost -Shortcuts -Is data set large enough that data management software will be useful? -Intention with data
Inductive
a type of reasoning that presents cases or evidence that lead to a logical conclusion
Qualitative Data Analysis External Validity
ability to generalize Threats to external validity Participant selection bias Setting threats
Transfers:
alter the distribution of resources within society
Financial:
are cash outlays of the organization considering the program or project
Indirect:
by-products, multipliers, spillovers, or investment effects of the project or program
Cost-benefit analysis (CBA):
compares the costs with the dollar value of all (or most) of a program's many benefits
Qualitative Data Analysis Validity
credibility and generalizability of study conclusions Internal validity: ensuring that a research is interpreting findings correctly Construct validity: ability to give oneself a reasonable chance of being able to actually find the causal relationship
Conducive
making a certain situation or outcome likely or possible
Units of effectiveness:
measure of any quantifiable outcome central to the program's objective (e.g. dollars per life saved)
Real:
net gains or losses to society
Qualitative Data Analysis Coding:
process of organizing data into categories or themes -cognitive -behavioral -motivational
Social:
represents real costs to society
Intangible:
such things as the value of wilderness or an increased sense of community
Cost-effectiveness analysis (CEA):
technique that relates the costs of a program to its key outcomes or benefits
Direct:
those that are closely related to the primary objective of the project
Tangible:
those that the analyst can readily identify in unit terms for CEA and can convert to dollars for CBA.
Net Benefits =
total benefits - total cost
Cost-Effectiveness Ratio =
total cost / units of effectiveness