Research Methods Exam
Given an unfamiliar study in communication sciences and disorders, accurately state the type qualitative study design used, the goal of the study, the type of data collected and the end product or theory that was developed from it.
!!!!!
Given an example of a research study, identify all threats to validity.
!!!!!!
Develop a literature search using Boolean terms (based on #6 above) (be able to state the actual steps you would take using your computer)
!!!!!!!
BE ABLE TO DEVELOP A PICO QUESTION
!!!!!!!!!!!!
regression
- "Participants score very high or low on a measure, are then retested and they score close to the mean when retested" • Trochim (2016) states this is because extreme scores have nowhere else to go (based on statistical averages) o Important because this might indicate that someone has "improved" on a measure with our without treatment---need to include a control or alternate treatment group in study to control for this o Was it the treatment or did he just naturally get better
selection
- Occurs when researchers work with "intact" groups such as a classroom or community groups. Maturation difference might occur naturally, outside of the control of the researcher. Will account for differences (e.g. faster learning in one over the other) • One serves as the control, the other as the experiment, but these are not truly experimental groups because changes might be related to group differences unrelated to studies • This is not experimental design.
Provide an example of a threat to validity.
--Defined as "the source of changes in the dependent measure" --In studies, we use control groups to minimize these and these control groups should be comparable to our experimental group, with the exception of the variable we are examining• --Pre-test you used a really good microphone that picked up everything the examinee said, but post-test you ignored the fact that your microphone needed new batteries. Therefore, it was not an effective tool the second time around.
Define the steps involved in the scientific method
1) Identify a problem (embark on background research on problem) 2) Develop a question 3) Develop a hypothesis 4) Plan a set of procedures for testing the hypothesis/answer the questions 5) Collect data 6) Analyze the data 7) Make a decision about the viability of the problem
Belmont Principles
1) Respect for Persons •Participation is voluntary and not coerced •Researchers should safeguard participants from psychological and physical harm •Participants should have the appropriate amount of information about the nature and purpose of the research before they take part in the study •How do we ensure informed consent with Participants with language disorders? 2) Beneficence • -protect the well-being of a person involved in a research study • "Direct benefit to the participant or obtaining information that contributes to the good of society" • Researcher weighs the benefits & risks-decided whether they are justified • Do no harm & maximize benefits 3) Justice • Who will benefit from research and who will bear it's burdens? • The need to make decisions regarding who participates in research • Decisions should be equitable, sound and based on science • Are some populations being systematically selected because they are easily accessible, in a • compromised position or because they are the target population? • Benefits from research should be equitable
Study types (from most to least rigorous)—Bahr (2006):
1. Systematic Reviews and Meta-Analysis 2. Randomized Controlled Trials (RCT) with Definitive Results 3. Randomized Controlled Trials (RCT) with Non-Definitive Results 4. Cohort Studies 5. Case-Control Studies 6. Cross Sectional Studies 7. Case Reports
Define each study type Bahr (2006):
1. Systematic Reviews and meta-analyses- • A systematic review answers a defined research question by collecting and summarizing all empirical evidence that fits pre-specified eligibility criteria. • A meta-analysis is the use of statistical methods to summarize the results of these studies.-combined analysis of several randomized clinical trials 2. Randomized Controlled trials with definitive results- This type of study involves random assignment of participants to either a treatment or control group. If a large number of individuals are tested in this way, then one can assume that individual differences between participants have been minimized, so that any effect that is noted can be related to the treatment. 3. Randomized Controlled trials with non-definitive results 4. Cohort studies-the responses of one or more groups of participants over time. Most typically, these studies are conducted with a group with a particular disorder and one that does not carry the diagnosis of interest. 5. Case-control studies-control study. For this type of study, the investigator picks two groups of participants that differ in terms of the diagnosis of interest and then seeks to determine what factors contributed to the development of that disorder. 6. Cross sectional studies-surveys or methodologies that focus on the responses of a large group of people at a particular point in time. 7. Case reports-This research format carefully studies the behaviors and responses of a particular person (or perhaps two or three persons) to note their response to various stimuli and/or treatments.
Discuss why the use of a control group might be controversial in the study of communication sciences and disorders.
A control group could be an ethical issue. It could be seen as withholding therapy.
Name 3 Bolean terms and create and example
And, Or, Not
Draw the evidence cycle
Assess→Ask→Acquire→Appraise→Apply and repeat around Patient
Illustrate the circle of measure development and testing
Define constructs, concepts, domains--> Reuse/create items--> Pretest and validate content, reduce items--> evaluate reliability--> Evaluate validity--> Evaluate responsiveness--> Define Objective of Measure
Name 3 reasons why literature searching is relevant to SLPs
EVP Avoid information overload Answering clinical questions
Illustrate a measure that is precise and valid vs. a measure that is not precise or valid (e.g. the bull's eye)
Image: https://onlinecourses.science.psu.edu/stat507/node/34
What is the root of qualitative inquiry?
Is an inquiry process of understanding based on distict methodological traditions of inquiry that explore a social or human problem. Philosophical Underpinnings
Pre-test-Post-Test Randomized Control Group Design
Most common true experimental design Behavior is measured twice: At beginning of study At end of study, typically after treatment R O X O R O O
Illustrate quasi experimental research designs discussed in class.
One treatment/no-treatment group N O X O N O O One treatment/alternate treatment/no treatment groups N O X1 O N O X2 O N O
mortality
Participants drop out before the end of the study i.e. "attrition" o If the attrition is random, not as much of a threat as when it is the low-performers, the individuals not benefitting from treatment-à this makes the treatment appear more beneficial than it really is o Use of pre-test scores might be a way to control or a intent-to-treat analysis o It is a threat when people drop out because they are not doing well o Intent to treat analysis-proving that the people leaving that did not negatively effect things-what do these people have in common-age? From the same educational background?
Mixed Model Design
Repeated measures/within subjects factor (relative to each other) Between subjects factor (treatment/no treatment) (relative to themselves R [ O X [O] R O [O]
Factorial Design
Researchers manipulate 2 or more independent variables Each IV has two or more levels e.g. adults with ASD/adolescents with ASD high-functioning/low functioning ASD Helps determine how more than one IV affects the DV and how the two IV's influence one another [Sex] [Intervention Approach] [Men] [Tranditional] [New] [Women] [Tranditional] [New]
Identify one factor that is related to decisions regarding sample size.
Single subject or qualitative—one person might be enough Larger sampleà may still be biased if you use a convenience sample Factors which affect decisions about sample size: • The more variable a population isà the larger a sample you need • How large the group is • Too small of a sampleà unable to detect differences • It all comes down to numbers of what you need statistically speaking
Switching Replication Design
The participants are randomly assigned to treatment and control groups to start the study. After the first posttest, the participants switch roles: HT treatment group becomes the control group and the control group becomes the treatment group for the second phase of the study. ROXO__O RO__OXO
Explain the value of experimental control.
True experimental design--> increases the validity of conclusions regarding cause and effect
5. Discuss what the purpose of a research design is and how a research design is chosen by a researcher.
a. Chose based on your research question b. Use for developing your plan to collect data c. Experimental or non-experimental d. Do you need to establish cause and effect
Distinguish control vs. experimental group
a. Control à does not receive treatment by researchers b. Experimental à received the treatment/variable being tested c. Control group allows us to see the effect of the d. Experimental manipulation
*****Compare and contrast between collecting data in a focus group vs. individual interviews?----Need more research
a. Focus groups • People are more likely not to share • One person my over power the group • Group think b. Individual interviews i. The way you ask questions influences the answers ii. More intimate situation
6. Define and give an example of an dependent vs. independent variable and identify them given a sample study in communication sciences and disorders
a. Independent Variable—the variable that is manipulated by researcher • E.g. Age, race, sex • TBI or no TBI • ASD or no ASD • Speeded vs. Unspeeded • Treatment 1 vs. Treatment 2 vs No Tmt b. Dependent Variable---the outcome measures that measure the effects of the experimental manipulation c. E.g. • Test Score • Level of aspiration • Mean length utterance (MLU) • Syllables per minute • Maximum phonation time • Speech Intelligibility
State one example of a sampling technique used in qualitative research
a. Intensity sampling-appropriate if you plan to explore different components of a case, phenomenon, situation and or behavior with varied intensity b. Homogeneous sampling-focusing on participants who have similar experiences, beliefs, and or background c. Criterion sampling-selecting participants who meet specified criteria d. Snowball sampling-Recruit participants based on the recommendation of initial participnats sampled e. Random purposive sampling-randomly sampling participants who have purposively sampled
How do qualitative researchers make sure their studies have scientific rigor?
o "Trustworthy" "Plausible" or "Credible" vs "Valid" o Issue of bias o Reflexivity, negative case sampling and researcher triangulation o Prolonged engagement, methods triangulation, data triangulation o Interpretive adequacy—involves the research participants e.g. having them read an account of the research
Identify 3 sources of bias in sampling.
o 1) failing to identify all members of a group e.g. parents of preschoolers without email access, not available via phone during the recruitment calls, without literacy skills o 2) Using a convenience sample- a sample that is easy to access. E.g. psychology introductory courses using WEIRD (Western, Industrialized, Rich and Democratic) population. 80% of research studies/12% world population o http://www.apa.org/monitor/2010/05/weird.aspx o 3) Volunteerism- cannot be avoided but people have different motivations for participating in research (e.g. family history, more time, etc.) Use of incentives helps to reduce this bias
Define types of qualitative research designs (use Dr. Delgado slides here as well)
o Ethnography: "in field" research, based on anthropological methods, field notes, interviews. Could be remote cultures but could be a classroom, and organization, etc. o Grounded Theory: Goal is to develop a model or theory for observes behaviors, events, situations, etc. to explain a phenomenon. Good approach for a beginner. Assigns codes to themes that emerge from data. o Case Study: Case= person, event, family or institution. Goal is to understand how the case functions within society, what are some motivations to their behavior, etc. o Phenomenology: To study a phenomenon from the viewpoint of a participant. How they perceive & interpret a situation. Narrative answers to a general question: "Please describe for me a situation....." Arranged in parts. Transcripts generated and psychological interpretation (transformation) o Conversation Analysis: Examine conversations (usually recorded) between providers/patients, parents/children, counselors/clients. Data could be #of turns, # of repairs. Use codes
Distinguish between an intended population and a an accessible population
o Intended: what your research question attempts to answer a question about, wants the results to apply to o Accessible: the group from which participants are recruited (because of study limitations i.e. geography, bureaucracy, etc.). Multi-site research has become increasingly important because of this. o Illiterate people almost always left out
Compare and contrast between a sample and a population:
o Population—persons of interest for a particular study, defined at the beginning stages of the study, all members have one or more predetermined characteristics o All members have at least one characteristic in
Define a study sample
o Sample—people who actually participate in a study (from larger population) o APA guidelines states that the use of "subjects" should be avoided o Refer to participants in who they are e.g. "high-schoolers" "1st graders" "bilingual pre-schoolers" or by what they do e.g. "listeners", "participants" *but also use person-first language (e.g. avoid "aphasics" "stutterers"
o instrumentation
• " A change in the instrument between pre-test and post-test" • "Instrument"—standardized test or the human instrument who observes and records behavior • How to eliminate this threat? o Everything needs to systematic o Write things down o Machines need to be calibrated o Microphone • Es muy complicado • Envioronment • Sometimes MRI machines aren't even the same • Change is due to the instrument changing rather than the treatment effect • Calibration, use of interobsever reliability (training research staff) • Use of control group—will affect both groups equally
State and describe the sampling methods discussed in class and give a detailed example for each:
• (1) Simple Random- every member has an equal chance of being selected - uses random numbers via table. o Test scores for 200 students; n=5 o Students have been assigned a number from 1-200 o Point to a spot on the table between 1-200 o Continue down column until you find 5 numbers between 1-200 • (2) Systematic Sampling- start with a list of potential participants, establish a sampling interval and select a every so many participants according to your interval. o List of 2500 participants o Goal is to recruit 125, 2500/125=20 → every 20th person will be selected to be in the study.
Given a reliability coefficient (i.e. a #), be able to state whether reliability is at an acceptable level for research in rehabilitation.
• 0.80+ high • 0.60 - 0.79 moderately high • 0.40 - 0.59 moderate • 0.20 - 0.39 moderately low • 0.00 - 0.19 low • Acceptable levels of reliability • Individuals: >0.90 • Groups: >0.70 • P values are not important. Interested in magnitude of reliability coefficient
How does research satisfy basic scientific inquiry?
• Approach is systematic not haphazard • Enables us to answer singular questions in research • Satisfies scientific curiosity by: i. Identifying questions ii. Designing information-gathering procedures to answer those questions
5 steps for acquiring information:
• Assess - review the situation • Ask - define clinical problem as a question • Acquire - select most likely resource to answer question, design strategy, search for the answer • Appraise - summarize the evidence yield • Apply - apply the evidence
Given an example article, define what kind of study and level of evidence it provides (see Table 1, Bahr 2006):
• Bahr (2005): o Case Control - level 2 evidence
Be familiar with the 3 sources of Evidence-based practice:
• Best available evidence from systematic research • Clinician's own professional experience and expertise • Client and family considerations • (American Speech-Hearing Association, 2014)
Define strata
• Characteristics that characterize population of interests (% of men and women, family income level, education, race, etc.)
"Rules" for identifying search terms (Dennis & Abbot):
• Clearly define the question • Separate out the concepts and key terms • Think of the words that researchers have used in different contexts over time, between countries, and in different clinical or research disciplines. • Draw up a plan to combine those search terms to maximize both sensitivity and precision. • Consider using limits or filters where available or appropriate.
Define Criterion-Related and Construct-Related Validity and be able to apply this definition given an example:
• Criterion-Related- Degree to which scores of an instrument are an adequate reflection of a "gold standard". • Construct-Related- Relates to the theory underlying the measure and its meaning. o Most abstract type of validity
Define evidence-based practice:
• EBP is about deciding which among the many acceptable options is likely to work best for a particular individual. • An approach in which clinicians use the best available research to guide their decisions about how to evaluate and treat persons with communication disorders. • EBP is not, however , about identifying the one best approach-it is about deciding which among the many acceptable options is likely to work best for particular individual.
Define types of research
• Empirical → collection of new data or information through observation and measurement of behavior and physical properties. • Qualitative → data often includes verbal information, behavior, statements. • Quantitative → numerical information, test scores, frequency counts.
Define types of qualitative research designs:
• Ethnography: Anthropology describes and interprets cultural constructs. • *Grounded Theory: Generate or discover a theory. o Most popular • Case Study: Goal is to understand how the person functions within society and what motivates them. • Phenomenology: A lived experience/phenomenon of meaning • Conversation Analysis: Examine conversations between providers/patients, parents/children, and counselors/clients. • Descriptive: Explore a phenomenon • Narrative: Stories told by individuals
Distinguish between experimental and non-experimental research:
• Experimental → involves manipulation by experimenter o Ex: giving someone a test and asking them to take it - you are manipulating them to take the test. • Non-Experimental → no manipulation, observation/descriptive o Ex: walk into a center and record 10 people with Aphasia, take data, and describe how they spoke, word finding skills, etc.
Distinguish between face and content validity.
• Face- Instrument appears to test what it is supposed to • The degree to which the items of an instrument 'look' as if they are an adequate reflection of the construct to be measured (face) • • Content- Items that make up the measure adequately sample the content defining the variable. Should cover all the content that you are interested in • The degree to which the content of a measure is an adequate reflection of the construct to be measured
What makes a sample "representative"?
• If characteristics of a sample are a good match to characteristics of the population. "? A sample is representative if characteristics of a sample are a good match to characteristics of the population e.g. education level, mental health status, etc.
State two sources of Unreliability:
• Inconsistency in the characteristics of subjects being assessed • Inherent flaws in the scale or instrument being used • Errors made by the assessor
How does a researcher draw conclusions about his or her research?
• Inferences → indirect conclusion about a population based on the data gathered from the sample. The accuracy is based on how well the sample represents the population. • Parameters → based on an entire population • Statistics → based on a sample
Give two examples of digitally-available sources of research evidence:
• MEDLINE • CINAHL • SCOPUS • TRIP • ASHA Evidence Maps • Cochrane Database of Systematic Reviews and other EBM resources
Define Quasi-Experimental Design:
• Manipulation of a variable • Random assignment to different experimental groups
Define Experimental Design:
• Manipulation of a variable • Random assignment to different experimental groups • Yields stronger evidence • Increases the validity of conclusions regarding cause & effect
Define experimental design
• Manipulation of a variable • Random assignment to different experimental groups • Yields stronger evidence • Increases the validity of conclusions regarding cause & effect
Draw Hierarchy of evidence and define the reasoning behind the hierarchy
• Meta analysis • Systematic Review • Randomized control trial • Cohort studies • Case control studies • Case series/case reports • Animal research • UP-Control for comparison, Less Bias, Stronger methodology→ • All information is not equal, the hierarchy helps determine the methodology level (stronger methodology, less bias, controls for comparison).
Discuss how reliability affects patient care and research.
• Misclassification may occur • Implications for sample size • Alters magnitude or direction of association • *** Has a direct impact on responsiveness (reliability estimate included in formula for Standard Error of Measurement. Empirically demonstrated using ICC (Girardeau, 2003)
Name three ways in which researchers can establish content validation.
• Obtain ideas from patients, significant others, clinicians and colleagues. • Review literature. Incorporate information from related measures. • Establish broad domains; verify. • Generate items for each domain. • Collect data on items; analyze data • Select items by pre-set criteria • Verify content • Cognitively debrief appropriate individuals • Formally pre-test measure
Define and give an example of a PICO question:
• Population: stroke patients with aphasia • Intervention: early initiation of "X" treatment • Comparison: Treatment initiated early (within 3 months post-onset) versus later (after 6 months post-onset) • Outcome: Functional communication abilities "Are patients with aphasia who receive speech-language treatment shortly after their stroke more or less likely to achieve improved functional communication abilities than those who receive such treatments later?"
Define reliability and give 2 examples of external or internal reliability
• Proportion of true variance to total variance OR proportion of observed variation in scores that reflect that variation in the characteristic being measured. • Internal Reliability- internal consistency • External Reliability- o Test - retest o Inter - rater o Intra - rater
State two ways to improve reliability
• Provide clear instructions or a well defined protocol • Make scoring as objective as possible • Train raters and test periodically • Write items unambiguously • Increase the number of items • Increase number of responses • Replicate measurements and take the mean
Compare and contrast qualitative vs. quantitative research (e.g. type of data collected, role of theory, environment, sample size):
• Qualitative: o Type of Data: verbal statements, direct quotes, behavior description o Role of Theory: no hypothesis → identifies trends/themes → theory o Environment: natural (living room, bedroom, home) o Sample Size: small • Quantitative: o Type of Data: surveys, MLU, maximum phonation time, test scores o Role of Theory: hypothesis based on theory o Environment: same each time (lab room, clinic room) o Sample Size: large number
Given an example of a study design, be able to distinguish identify a specific qualitative designs (i.e. is it narrative, grounded theory, phenomenology?)
• Qualitativeà often a small number of participants, spend a lot of time reviewing data from these participants • Quantitative--> large number of participants, spend a short time in the lab setting, analyze in batches of large group data • Is quantitative scientific enough • Ethnography: "in field" research, based on anthropological methods, field notes, interviews. Could be remote cultures but could be a classroom, and organization, etc. • Grounded Theory: Goal is to develop a model or theory for observes behaviors, events, situations, etc. to explain a phenomenon. Good approach for a beginner. Assigns codes to themes that emerge from data. • Case Study: Case= person, event, family or institution. Goal is to understand how the case functions within society, what are some motivations to their behavior, etc. -an introspective person with • Phenomenology: To study a phenomenon from the viewpoint of a participant. How they perceive & interpret a situation. Narrative answers to a general question: "Please describe for me a situation....." Arranged in parts. Transcripts generated and psychological interpretation (transformation) -an introspective person with is highly perfered • Conversation Analysis: Examine conversations (usually recorded) between providers/patients, parents/children, counselors/clients. Data could be #of turns, # of repairs. Use codes -just looking at it
Distinguish between Random error and Systematic error
• Random Error- unpredictable error, most commonly due to sampling error. • Systematic Error- systematic error (error always goes in one direction).
How is data presented in a single-subject/single-case research design?
• Results for each participant are reported individually. • Does not mean the study only had one participant. • Participants experience both treatment and control conditions.
Define psychometrics
• Science of using standardized tests or scales to measure attributes of an individual. • A set of assumptions, models, and procedures used to develop tests and interpret test scores in 20th century.
4. Match your question with the suggested best type of study design
• Therapy: RTC> Prospective cohort • Diagnosis: prospective, blind comparison to a gold standard • Etiology/Harm: RCT> cohort> case control> case series • Prognosis: cohort study > case control> case series • Prevention: RCT> cohort study> case control> case series • Clinical Exam: prospective, bling comparison to gold standard • Cost: economic analysis
State and define threat of history, maturation, regression, instrumentation, selection and mortality:
• Threat of History- an outside influence occurred during your study and this outside influence can account for changes in the outcome at the end of the study. • Threat of Maturation- increases in performance due to participants' growth and development over time - very important when studying children. • Threat of Instrumentation- a change in the instrument between pre-test and post-test. • Threat of Selection- occurs when researchers work with "intact" groups such as classroom or community groups. • Threat of Mortality- participants drop out before the end of the study.
Why is literature searching relevant for the practice of speech language pathology?
• To answer everyday clinical questions • To find background information for research • To stay current in your area of expertise • To build personal confidence and • To ensure professional competence • As a result, you will gain a LIFELONG LEARNING SKILL FOR EVIDENCE BASED PRACTICE AND RESEARCH.
Define quasi-experimental research and justify its use.
• Used to estimate the causal impact of an intervention on its target population without random assignment. • Valuable for investigating cause and effect relationships but more susceptible to threats.
Vulnerable populations
•Homeless population•Incarcerated population•Low SES•Low IQ•Individuals with communication disorders•Orphans•Lack of cognitive capacity
