EDUC 35 Quiz 1
Research Methods
(Questions, Data collection, Data Analysis, Interpretation, Validation)
The Peer Review Process
1. Idea, theory, hypothesis 2. Research and write 3. Manuscript (proposed article) 4. Send to journal editor (ex. our Professor) 5. Reviewed by 3-5 experts in the field 6. Decision is made to accept, revise, or reject
What are some ethical violations in research (Important!)
1. Lack of informed consent 2. Putting subjects at risk 3. Coercion/undue pressure on volunteers (or on a parent to volunteer their child) 4. Exploitation of a vulnerable population 5. Withholding available treatment 6. Withholding information about risks 7. Deception 8. Fabricating data/results 9. Failing to protect anonymity in small studies 10. Inequitable access to benefits 11. Failing to protect sensitive information (e.g., grades, income, health history, etc.)
3 research approaches
1. Qualitative research 2. Quantitative research 3. Mixed methods Research approach
Many think tanks produce excellent research, but what are 3 reasons why you can't always trust them?
1. They might Publish bogus "research" 2. Research often does Not undergo peer review 3. Some think tanks have an explicit Political agenda More on this in discussion
Philosophical worldview meaning
A general philosophical orientation about the world and the nature of research that a researcher brings to a study.
Peer-reviewed journal
A journal in which work published undergoes review by relevant disciplinary experts (who are not part of the research team or the editorial board) who are charged with evaluating the quality of the research Ex. American Educational Research Association (AERA), American Psychological Association (APA),
What are some questions to ask when we find research from a source that we're not familiar with?
a. What are the Credentials and affiliations of the researchers? b. Do respected scholars in the field cite research from this source? c. Does the source have a commercial interest or political affiliation? d. Are the findings consistent with the peer reviewed literature or are they wildly different?
Think tank
an institute, corporation, or group organized to study a particular subject (such as a policy issue or a scientific problem) and provide information, ideas, and advice
What is reflexivity?
analyzing and critically considering our own role in, and effect on, our research (being transparent about past experiences that could affect the interpretation of data)
qualitative coding
analyzing collected interview and identify patterns and information
American Educational Research Association (AERA) Journals
American Educational Research Journal Educational Researcher Review of Educational Research Educational Evaluation and Policy Analysis Journal of Education and Behavioral Stats Review of Research in Education Also private journal publishers like SAGE, Taylor & Francis, etc.
Qualitative research approach
An approach for exploring and understanding the meaning individuals or groups ascribe to a social or human problems, The researcher makes interpretations of the meaning of the data, Emphasizes the reporting of the complexity/richness of a situation, Inductive style (Specific -> general)
Quantitative research approach
An approach for testing objective Theories by examining the Relationships among Variables, the variables are measured on instruments to get numbered data that can be analyzed statistically, Deductive style, Focuses on protecting against bias, controlling for alternative/counterfactual explanations, & being able to generalize/replicate the findings
Mixed methods research approach
An approach to inquiry involving collecting both quantitative and qualitative data, integrating the two forms of data, & using distinct designs that may involve philosophical assumptions and theoretical frameworks. yields additional insight beyond that of the qualitative/quantitatibe data alone
What should we look at to evaluate credibility
Author's expertise and credentials Source "quality" Point of view Publication date
Postpositivist Worldview
Believes that absolute truth can never be found, and evidence found in research is always imperfect (this is why researchers never say they prove a hypothesis) Data and evidence shape knowledge Focuses on empirical observation & numeric measurement Focuses on how variables influence other (relationships among variables) Starts with a theory, deductively More geared toward quantitative research Is objective Think, experimentation
Transformative Worldview
Believes that research should intertwine with politics and a political agenda to confront social inequalities & oppression where it occurs Collaborative; Provides a voice for marginalized participants, raising awareness and advancing an agenda for change to improve their lives Also geared toward qualitative research
How did Clotfelter define "effective teachers"?
By years of experience the teacher had before they started teaching (experience)
Research approaches as worldview, designs, & methods: Qualitative example 1
Constructivist worldview, ethnographic design, and observation of behavior
Mixed Methods Designs Hint: 3 main designs
Convergent mixed methods Explanatory sequential mixed methods Exploratory sequential mixed methods
Common Features of Quantitative Research (11)
Deductive Reasoning, Well-defined factors, Focus variables identified in advance, Testing hypothesis and theories, Large number of participants, Highly constrained participant responses, Numerical data (primary/secondary), Statistical analysis and modeling, Frequent use of secondary data, Data collected, then analyzed, Generalization and replication of findings
Descriptive vs Interpretive Fieldnotes
Descr: Physical/Social Env, Participants/their roles, Dialogue, researcher impact Interp: Ideas/impressions, insights, open questions, clarification on other fieldnotes, future observations
Research Approaches & their corresponding designs
Designs: Quantitative (e.g. Experiments), Qualitative (e.g. Ethnographies), Mixed methods (Explanatory sequential)
Anatomy of the research process flowchart
Identify a research topic -> Review of the literature -> Determine research gaps -> Formulate researchable questions -> Choose appropriate method, data, and means of analysis -> Present results and conclusions based upon findings
Common Features of Qualitative Research
Inductive reasoning, Unknown or little-known factors, Patterns or themes emerge during analysis, Developing/refining hypotheses and theories, Small number of participants, Minimally constrained participant responses, Data are (text, audio, image, video, [fieldnotes]), Noticing and coding emerging themes, Researchers typically use primary data, Data are simultaneously gathered and analyzed, Focus on reflecting context/complexity
Ethics: Who monitors the ethical conduct of research at UCLA?
Institutional Review Board (IRB) Office of the Human Research Protection Program (OHRPP)
Common types of research work include?
LSCT Literature review Synthesis or Meta-analysis Commentary Theoretical
Qualitative Designs Hint: NPGEC
NPGEC Narrative research, Phenomenological research, Grounded theory, Ethnography, Case studies
What are fieldnotes?
Notes created by the researcher during a field study to record/remember behaviors, activities, events, and other features of an observation
4 Philosophical Worldviews (Important!)
PCTP 1. Post positivism 2. Constructivism 3. Transformative 4. Pragmatism
In planning a study, researchers need to think through the _________ world______ assumptions that they bring to the study, the research ________ that is related to this worldview, and the specific research _________ or procedures that translate the approach into practice.
PHILOSOPHICAL, WORLDVIEW, DESIGN, METHODS
Where do we find Education research?
Peer-reviewed journals (*) Think tanks (*) Open access sites Databases University/researcher pages News/media reports Twitter/social media (*) = emphasis
example of correlation
People eat more ice cream get sunburnt
What are Research Approaches?
Plans and the procedures for research that span the Steps from Broad assumptions to Detailed methods of data collection, analysis, and interpretation
Research approaches as worldview, designs, & methods: Quantitative example
Postpositivist worldview, experimental design, and pretest and posttest measures of attitudes
Research approaches as worldview, designs, & methods: Mixed methods example
Pragmatic worldview, collection of both quantitative and qualitative data sequentially in the design
Pragmatic Worldview
Problem-centered Finding realistic solutions to problems Focuses on applications and what works best in a certain situations Looks at consequences of actions Pluralistic Relates to mixed methods research
Survey design
Provides a quantitative/numeric description of trends, attitudes, or opinions of a population by studying a sample of that population Uses questionnaires or structured interviews for data collection, aiming to generalize from a sample to a population
Research Methods involve the forms of data collection, analysis, and interpretation. Explain Quantitative, Mixed, and Qualitative Methods characteristics
Quantitative Methods: are pre-determined, use instrument based questions, have performance/attitude/observational/census data, use statistical analysis, interprets stats Mixed Methods: involve pre determined and emerging methods, use open & close ended questions, use multiple forms of data, use statistical & text analysis, interprets across databases Qualitative Methods: involve emerging methods, use open-ended questions, use interview/observation/document/audiovisual data, use text & image analysis, interprets themes & patterns
inductive reasoning
Reasoning from specific, detailed facts to general principles/themes
What informs the decision of which research approach to use?
Research designs, Research methods of data collection, analysis, and interpretation, nature of the research problem/issue being addressed, researchers' personal experiences, philosophical assumptions, audiences for this study
Convergent mixed methods design
Researcher converges/merges quantitative and qualitative data to provide a comprehensive analysis of the research problem
Grounded theory design
Researcher derives a general & abstract theory of a process, action, or interaction grounded in the Views of participants. Uses multiple stages of data collection & the refinement & interrelationship of information categories
Phenomenological research
Researcher describes the lived experiences of individuals about a phenomenon as described by participants Focuses on the experiences of many individuals that have all experienced the phenomenon; typically involves conducting interviews
Case study design
Researcher develops an in-depth analysis of a case (often a program, event, activity, process, or one or more individuals) over a sustained period of time Collects detailed info using a variety of data collection procedures Cases are bound by time and activity
Explanatory sequential mixed methods design
Researcher first conducts quantitative research, analyzes the results and then builds on the results to explain them in more detail with qualitative research
Narrative research design
Researcher studies the lives of individuals and asks them to provide stories about their lives Often retold by the researcher in a narrative chronology that may combine views from the participant and the researcher's lives collaboratively
Ethnography design
Researcher studies the shared patterns of behaviors, language, and actions of an intact cultural group in a natural setting over a prolonged period of time. Data collection by observations and interviews
Researchers write for audiences that will _______ their research. These audiences may be journal _______ and readers, faculty committees, conference attendees, or _________ in the field.
Researchers write for audiences that will ACCEPT their research. These audiences may be journal EDITORS and readers, faculty committees, conference attendees, or COLLEAGUES in the field.
Office of the Human Research Protection Program (OHRPP)
Responsible for ensuring the safety and welfare of participants in Human Research Projects conducted at UCLA. The OHRPP provides the campus and five UCLA Institutional Review Boards (IRBs) with professional guidance and administrative support
Experimental design
Seek to determine if a specific treatment influences an outcome Researcher provides a specific treatment to one group and withholds the treatment from another group, and then determines how both groups scored on an outcome Ex. true experiments (use random assignment), quasi-experiments and single-subject designs (non-randomized assignments)
example of causation
Sun/Sunny weather causes sunburns and people to eat more ice cream
Quantitative Designs Hint: 2 main designs
Surveys (non-experimental) Experiments (experimental)
Identify one peer reviewed journal that publishes research around that topic and could be a good fit your study. Why do you believe this is a credible publication?
The Intensive Partnerships for Effective Teaching: A Comparison of School Outcomes published by RAND Corporation would be a great peer reviewed journal to help with our study. Because of their research integrity and objective approach, we believe Rand Corporation is a credible publication. Their mission is to use research to improve policy and decision-making. They aim to represent diversity, equity, and inclusion. RAND is a nonpartisan and nonprofit organization dedicated to serving public interest.
Institutional Review Board (IRB)
The committee at a college, university, or research center responsible for evaluating the ethics of proposed research involving human subjects
Empirical education research (Start of Lecture Slides)
The deliberate, systematic process of collecting and analyzing data to understand a particular question or phenomenon related to education
Exploratory sequential mixed methods design
The researcher first begins with a qualitative research phase. and explores participants' views, then, the data is analyzed and the information is used to build into a second, quantitative phase. The initial qualitative phase can be used to specify variables that will go into the follow-up quantitative phase
Constructivist Worldview
The researcher's intent is to make sense of/interpret the subjective meanings that others have about the world People/participants can have various meanings of the world around them Develops a theory inductively More geared toward qualitative research
Research approaches as worldview, designs, & methods: Qualitative example 2
Transformative worldview, narrative design, and open-ended interviewing
Research Designs
Types of inquiry that provide specific direction for procedures in a research study
Deductive reasoning
Uses a general statement as the basis for reaching a conclusion about specific examples
nominal variables + examples
Variables are simply names or labels, with no specific order Example: Gender, ethnicity, first language, disability status
We should always take note of a think tank's _________ leaning & _________.
We should always take note of a think tank's POLITICAL leaning & CREDIBILITY. Ex. A think tank could lean liberally or conservatively, this can be discovered by reading their mission statements Look for words like "nonpartisan (which means not biased), but that word can still be used in combination with other words like "progressive" in the mission statement
What is one way you could operationalize and empirically investigate the question: Do all students get effective teachers?
We would start with an empirical question for investigation such as "What does it mean to be an effective teacher?" We would collect data by conducting a qualitative interview, in which we would focus on interviewing all the teachers from three elementary schools, with each school being from different districts: a low-income class, a middle class, and an upper-class school district. Interviewing three different schools from different districts can allow us to compare teaching styles and how that can be affected by socioeconomic class.
Choosing a research approach: When should we choose a Quantitative approach? (important!)
When the problem calls for the identification of factors that influence an outcome, the utility of an intervention, or understanding the best predictors of outcomes When you need to test a theory/explanation
Choosing a research approach: When should we choose a Mixed methods approach? (important!)
When the quantitative or qualitative approach, each by itself, is inadequate to best understand a research problem, and the strengths of both quantitative and qualitative research (and its data) can provide the best understanding.
Choosing a research approach: When should we choose a qualitative approach? (important!)
When the researcher does not know the. important variables to examine When the topic is new, the subject hasn't been addressed with a certain sample of ppl, and existing theories don't apply to the sample When you want to focus on the richness and complexity of participant experience
Is there bias/Can bias exist in peer-reviewed literature?
Yes, this is why journals get peer reviewed by multiple people to prevent bias
Variable
a characteristic or condition that changes or has different values for different individuals/organizations being studies
Winnowing the data
a process of focusing in on some of the data and disregarding other parts of it (text/image/audio especially are too dense)
What 3 methods allow us to empirically investigate questions like our big 2 questions (Do all students get effective teachers? & Are standardized tests fair?)
a) Qualitative b) Quantitative c) Multiple/mixed methods
Interval/Ratio (Continuous) Variables + examples
reflect placement along a continuum, numbers/quantities, can directly compute distance between values Example: Income, GPA, SAT score, Score in PHQ depression scale
ordinal variables (categorical) + examples
take a range of ordered values, know which are larger/smaller, exact quantitative difference between them is unknown Example: Socioeconomic status ("low income" to "wealthy"); Agreement ("extremely" to "not at all")
