Systematic Review and Meta Analysis

4

Case series (and low-quality cohort and case-control study)

Systematic review

Comprehensive analysis of full range of literature on a particular topic - intervention, diagnostic test, prognostic factor

Cochrane Collaboration

Archie Cochrane, physician, wrote a letter suggesting that a critical summary of randomized clinical trials was needed in medical specialties to provide a reliable source of evidence for medical care (1979).

Inclusion criteria

Are defined and eligibility criteria are set. Construct definitions to distinguish relevant from irrelevant studies. •Types of participants - dx, age, gender, time since occurence, severity level •Types of interventions •Types of outcome measures

Research

-A tool -Scientific and scholarly investigation

Meta synthesis

Examples •The impact of childhood chronic illness on the psycho-social wellbeing of fathers: A meta-synthesis •Meta-synthesis of clients' experience of using support dogs: Practical implications •A qualitative meta-synthesis on the effects of using sign language interpreters in therapy

EBP

Examples of this in healthcare •Using it for clinical problem solving •Using it to determine course of treatment •Using it to guide assessment procedures •Using it in in-service trainings •Using it in client and family education •Using it to develop interns and students •Using it to guide research

5

Expert opinion without explicit critical appraisal

Systematic review

Format 1.Title (this semester) 2.Abstract 3.Background 4.Methods (this semester) 5.Results 6.Discussion, Limitations, Conclusions

Meta analysis

How to use it? •Vote counting; combining r-values; frequency analysis; combining means and standard deviations of the experimental and control groups •Need consistent data, similar assessments

Flow Diagram

How you set up your systematic review

Nonsignificant

In forest plots, outcomes are _______________ when it crosses the red line

Significant

In forest plots, outcomes are ________________ when they do not cross the red line

Evidence Table

Includes -Level of evidence, participants inclusion criteria, study setting, intervention and control groups, outcome measures, results

2B

Individual prospective cohort study, low-quality RCT (e.g., <80% follow-up or low number or participants; pilot and feasibility studies); ecological studies; and two-group, non-randomized studies

3B

Individual retrospective case-control study; one-group, non-randomized pre-posttest study; cohort studies

Quantitative evidence and data analysis

Meta analysis Primary and Secondary analysis fall into what meta analysis step?

Data evaluation and coding

Meta analysis •Exclusion/inclusion criteria Randomized Clinical Trials (RCT's) only (?) •The design maybe associated with achieving positive or negative results, a poor design is associated with the likelihood of positive outcomes

Reporting results

Meta analysis •Includes the findings of the effect size used, how it was computed, the basic unit of analysis and the actual statistical outcomes •Reviewers also summarize the findings, compare them to previous narrative reviews and primary research studies and suggest areas of further investigation

Data collection

Meta analysis •Major study features are coded according to the objectives of the review •Study outcomes are transformed to a common metric called effect size (ES), so that they can be compared (Uncovering systematic variation was pioneered by Cohen 1988)

Analysis and interpretation

Meta analysis •Significance testing is strongly influenced by the size of the sample and is less informative as evidence •The question turns from whether a treatment exists to how much of an effect exists •Combining multiple studies may be misleading because: ••Significant p levels are more likely to be published (Grey Literature & Publication bias)

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Systematic review

Purpose •To gather together, evaluate and summarize the published research addressing all that we know about a specific topic •Need to be very specific about how the articles are selected (How you searched and chose them)

Meta synthesis

Review of qualitative studies or qualitative reviews, still called systematic reviews sometimes

Topic

Selected followed by a formulation of a research question relevant to the topic of interest. It includes identification of the problem and formation of the research questions.

Abstract

Short Summary of the major aspects of the entire paper in a prescribed sequence that could include: •Importance •Objective •Data sources •Study selection and data collection •Findings •Conclusion & relevance •What this article adds •In 150-200 words

Forest plot

Shows the results of the meta analysis review and each individual study. -CI -Shows whether or not there was a significant effect or not

Moderate

Strength of Evidence -At least one Level 1A/B study or multiple Level 2A/B or 3A/B -The available evidence is sufficient to determine the effects on health outcomes, but confidence in the estimate is constrained by such factors as --The number, size, or quality of individual studies --Inconsistency of findings across individual studies -As more info becomes available, the magnitude or direction of the observed effect could change, and this change may be large enough to alter the conclusion related to the usefulness of the intervention

Low

Strength of Evidence -Small number of low-level studies, flaws in the studies, etc.

Strong

Strength of Evidence -Two or more Level 1A/B studies -The available evidence usually includes consistent results from well-designed, well-conducted studies. The findings are unlikely to be strongly called into question by the results of future studies

Conclusion

Summary of what your found in a nutshell

3A

Systematic review of case control studies

2A

Systematic review of cohort studies

1A

Systematic review of homogeneous RCTs (similar population, intervention, etc.) with or without meta-analysis

Primary

The original analysis of data in a research study

Secondary

The re-analysis of data for the purpose of answering an original question with statistical techniques

Scoping review

This can •Examine the extent, range, and nature of research activity • Address questions beyond those related to intervention effectiveness •Generate findings that can complement the findings of clinical trials •Determine the value of undertaking a full systematic review •Summarize and disseminate research findings •Identify gaps in the existing literature

1B

Well-designed individual RCT (Not a pilot or feasibility study with a small sample size)

Results of meta analysis, compares their outcomes

What does a forest plot show you?

Publication bias

What does a funnel plot show you?

Meta analysis

What is it? •Summarizing & synthesizing information (data) •"Like a systematic review, but extra"

Scoping: not enough research/literature to do a full review (missing levels of evidence), systematic: usually quantitative data, has levels of evidence Meta-analysis: looks further into statistics in comparison

What is the difference between a scoping review and a systematic review?

Helps us see a wide range of research being done on topics so we can use EBP and see the big picture in one place

What is the value of scoping reviews, meta-syntheses, systematic reviews and meta-analyses to OT Practice?

They must have homogenous data to compare

When can you do a meta-analysis?

Meta analysis

When could you use it? •Homogeneous data (sample characteristics, measures), getting enough data to statistically compare it together is the most difficult part

Qualitative

When would you want to do a meta-synthesis?

Meta analysis

Why use it? •Increase power •Generate evidence for treatment of individual cases •Studies must have consistent data to be included

Scoping review

•'Mapping,' a process of summarizing a range of evidence in order to convey the breadth and depth of a field •Authors do not typically assess the quality of included studies •Process requires analytical reinterpretation of the literature •May be particularly relevant to disciplines with emerging evidence, such as rehabilitation science, in which the paucity of randomized controlled trials makes it difficult for researchers to undertake systematic reviews

Meta analysis

•A summary of previous research that uses uses quantitative methods to COMPARE outcomes across a range of studies •It is designed to test relationships between variables •Obtains a statistical consensus across study-highest evidence

Literature search strategy

•Exhaustive, sensitive, and extensive search strategy to collect studies addressing your topic. Probably don't include grey literature (literature that hasn't been published). •Key words and variables determine sources of potentially relevant studies.

Meta synthesis

•Have a research question which is compatible with an interpretative approach (e.g. 'how' or 'why' questions (more open ended)) •Are explicit about how the synthesis was conducted •Justify methodological choices and analytical procedures within a qualitative or interpretative paradigm •Involve producing novel and integrative interpretations of a body of qualitative literature (or even re-interpretation of the themes presented in selected studies) than is available in any single study.

Meta synthesis

•Intentional and coherent approach to analyzing data across qualitative studies •Process that enables researchers to identify a specific research question and then search for, select, appraise, summarize, and combine qualitative evidence to address the research question •Meta ethnography •Meta phenomenology

Discussion

•Overall level of evidence •Themes •Strengths & weaknesses •Gaps & future research needs •Implications for occupational therapy

Meta analysis

•Refers to a set of statistical procedures that are used to quantitatively integrate and aggregate the results of multiple primary studies, to arrive at an overall conclusion or summary •Systematic review does not need to contain statistical synthesis of the results

Funnel plot

•Scatterplot of treatment effect against a measure of precision or variability for each study •Shows you if the studies are a biased sample of the literature: identifies publication bias •Should be symmetrical (shows no bias)

Study bias

•Selection Bias •Performance Bias •Detection Bias •Attrition Bias •Reporting Bias

Scoping review

•Summarize a range of evidence in order to convey the breadth and depth of a field •May be particularly relevant to disciplines with emerging evidence, such as rehabilitation science, in which the paucity of randomized controlled trials makes it difficult for researchers to undertake systematic reviews •Clarify a complex concept and refine subsequent research inquiries •Good for identifying gaps in the existing literature •Do not typically assess the quality of included studies

Evidence

•Tracking down of research support •Helps in rendering better client care •Includes client preference & clinical expertise

Literature Review

•Why systematic review topic is important •What we already know and how the systematic review fills a gap in our knowledge