Systematic Reviews and Meta-analysis

Other Techniques for Meta-Analysis Presentation of Data

Graphic demonstration of the data can show other relevant trends, including: Effect size; Study publication date; Population differences

Why systematic reviews are necessary

The volume of published material makes it impractical for an individual clinician to remain up to date on a variety of common conditions. This is further complicated when individual studies report conflicting conclusions, a problem that is prevalent when small patient samples and retrospective designs are used.

Systematic Review

A review that is conducted according to clearly stated, scientific research methods, and is designed to minimize biases and errors inherent to traditional, narrative reviews; An attempt to limit bias in interpreting results from studies; The methodology used for the systematic review may still result in bias.

Funnel Plot

A scatter plot of treatment effects against study size; The funnel - inverted - is based on the relationship between precision and sample size; Small studies scattered at bottom; larger studies with greater precision at top; used as a way to assess publication bias in meta-analysis

Publication Bias

selective publication of articles that show positive treatment of effects and statistical significance

Database Bias

No single database is likely to contain all published studies on a given subject.

Methods: Heterogeneity

Quantitative assessment: Cochran Q - Chi Square - test for heterogeneity: Low p-value <0.1 suggests differences between studies. I2 test for variability between studies: High I2 >50% suggests variability. If there is heterogeneity, pooling the summary estimates may be inappropriate or misleading!

Reporting Meta-Analysis Effect Sizes

Total mortality from trials of β blockers in secondary prevention after MI Black square= OR for each RCT: Size ≈ "weight" of each RCT Horizontal line=95% CI for each RCT Diamond= combined OR + 95% CI; 22% reduction in odds of death

narrative review characterisitics

research question is often broad; search strategy is not defined or systematic; article selection is not systematic; appraisal of study quality may not be performed; qualitative summary of findings

Meta-analysis: The Research Question

Common questions addressed in meta-analysis are whether one treatment is more effective than another or if exposure to a certain agent will result in disease

Heterogeneity, Fixed or Random Effects

Assessing these parameters involves judgment; Interpretation of reported results is not simple; Clinicians should consider numerous factors in interpreting results from meta-analysis

English-language bias

occurs when reviewers exclude papers published in languages other than English

Citation bias

occurs when studies with significant or positive results are referenced in other publications, compared with studies with inconclusive or negative findings

examples of bias

Database Bias; Publication Bias; English-language bias; Citation bias

Putting it all together: Meta-analysis and Systematic Review

Does this SR/MA apply to our specific question? Are we satisfied of the inclusion/ exclusion criteria of studies chosen, and are we satisfied of the completeness of their search for literature? Looking at the included studies, were results similar across different studies? Why not? For SR: Were the studies "judged" according to certain criteria? For MA: Were there formal tests of heterogeneity, and was the correct statistical method used for combining results? Was there a statistically significant difference favoring a treatment/ intervention/ screening? Were these results clinically significant and applicable to my patient and my problem?

qualities of a good Systematic Review

Formulates a question; Conducts a literature search; Refines the search by applying predetermined inclusion and exclusion criteria; Extracts the appropriate data and assess their quality and validity; Synthesizes, interprets, and reports data

Funnel Plot structure

Funnel Plot

Funnel plot asymmetry

Funnel plot asymmetry because of differences in study populations (so there was heterogeneity between these studies)

The Cochrane Collaboration

International non-profit organization that prepares, maintains, and disseminates systematic up-to-date reviews of health care interventions; The Cochrane Collaboration was established in October 1992; The Collaboration relies on funding from a number of different sources to complete its work; I like to think of a review as starting out as a jigsaw puzzle - especially if there is heterogeneity - ie. differences in interventions, settings, study designs, outcomes.

In this study: does the SR discuss the reasons for any variations / heterogeneity between individual RCTs

Look at the overall Forest plot for all antispasmodics. You can see visually how the results vary by individual agent and RCT. It should be remembered that some of these individual analyses are based on small numbers, often from only one RCT. You can also easily see how the amount of available evidence varies widely between different agents

Fixed vs Random Effects Models

Meta-analyses often weight effects based on precision of the results; In a Fixed effects model, it is assumed that there is one true effect size for all included studies; In a Random effects model, it is assumed that the true effect size can vary between studies based on various factors

Meta-Analysis

Meta-analysis is a statistical technique for combining the results of independent, but similar, studies to obtain an overall estimate of treatment effect. While all meta-analyses are based on systematic review of literature, not all systematic reviews necessarily include meta-analysis.

Limitations of systematic reviews

Methodology can be flawed; Results may still be inconclusive; There may be no trials/evidence; The trials may be of poor quality; The intervention may be too complex to be tested by a trial; Practice does not change just because you have the evidence of effect/effectiveness

Very good funnel plot

More sophisicated; uses log scale on bottom....uses fixed effects model for MA which weights larger studies greater, and diagonal lines are 95% CI for which 95% of studies should fit within in the absence of heteroge or selection bias (they don't in this study about whether mag is helpful for ACS mortality

Meta-analysis: Is there heterogeneity?

Pooling data increases the power of an analysis, and may demonstrate a significant effect with an intervention that was not apparent when looking at the results of smaller individual studies. However, a key question is when is it reasonable to combine data from different studies and when is it not. There may be clinical heterogeneity between studies. This could be in the intervention used, the population studied, or the outcomes measured, amongst others. For example, it may be reasonable to combine the results of different studies using different beta-blockers to lower blood pressure, where one would not combine these data with studies using relaxation therapy to lower blood pressure, even though the aim was the same (to lower blood pressure). It can be a difficult decision whether it is acceptable to combine trial data. For example, in the case of combining data on multidisciplinary trials where the actual components of multidisciplinary care may vary widely between different RCTs. You need to consider this issue. It may require a judgement on your part. It should also be clear what data from the RCTs have been used in the analysis. A systematic review may, say, combine data on an intention-to-treat basis, whereas the original study may have reported a per-protocol analysis

Statistical Analysis for Meta-analysis

Select measure of effect (rate difference, odds ratio, standardized mean difference etc.): RCT: rate difference (EER v. CER) or rate ratio Case Control: odds or rate ratios Standardized mean difference between E and C For binary outcome variable, use OR or RR: e.g., disease vs no disease; remission, yes or no For continuous outcome variable e.g., changes in blood pressure

Evaluating Systematic Reviews and Meta-analysis

Systematic reviews (well conducted) represent the highest level of evidence; Must consider BOTH the quality of the review itself as well as the quality of the studies included; The methodological quality of the included studies may affect what weight you may wish to put on the results. A review should discuss this issue, and may present a sensitivity analysis (for example, an analysis just including high-quality studies to see if this differs from the analysis of all available studies)

defining the question

Very defined study question; Determines eligibility criteria for article selection: Population; Type of intervention or exposure; Outcomes of interest; Study design; PICO

Tests for heterogeneity

Whenever you combine data from different RCTs there is going to be some heterogeneity. The question is what degree of heterogeneity is acceptable. When results are numerically combined, a statistical test of heterogeneity should be reported. If there is a high degree of heterogeneity among RCTs, this suggests there may be something different among the RCTs, and that their results should not be combined. If there is statistical heterogeneity among RCTs in an analysis, you should expect the review to comment on the reasons for this. Often the review will exclude trials that account for the heterogeneity and recalculate the analysis, and may also report other sensitivity analyses. If a review does exclude trials, it should have a good underlying reason for doing so, other than its results are different.

how to detect or manage publication bieas

unpublished studies through a manual search of conference proceedings, correspondence with experts, and a search of clinical trials registries; Funnel Plots

systematic review characteristics

well focused clinical question; answerable question; explicit search strategy outlining study inclusion/exclusion criteria; article selection is specific to inclusion/exclusion criteria; articles are critically appraised and strengths and weaknesses documented; qualitative or quantitative analysis of findings