Systematic Reviews and Meta-Analyses

Combining results in a meta-analysis when to used fixed-effects vs. random effects

A fixed-effects model is used when heterogeneity is not significant, and a random-effects model is used when statistical heterogeneity is present

Narrative review (or traditional review) - 6: often includes, tends to be, does not, often, can (2)

Often includes a discussion of one or more aspects of disease etiology, diagnosis, or treatment Tends to be mainly descriptive Does not involve a systematic search of the literature Often focuses on a subset of studies in an area chosen based on availability or author selection Can often include elements of selection bias Can be confusing at times if similar studies have diverging results and conclusions

Publication bias

When an entire study remains unreported, the standard term is publication bias The reason for publication bias is that studies without statistically significant results (negative studies) are less likely to be published than studies that reveal apparent differences (positive studies) A study of FDA reports found that they often included numerous unpublished studies and the findings of these studies can appreciably alter the estimates of effect Publication bias is a specific type of reporting bias

How to appraise a systematic review and meta-analysis? Two systems to rate quality of evidence

- GRADE - Cochrane Collaboration risk of bias assessment methods

Statistical tests of heterogeneity (3)

- test for heterogeneity - cochrane Q - Higgin's I2 statistic

How to appraise a systematic review and meta-analysis? 1. credibility 2. confidence

1. Credibility refers to the extent to which the review's design and conduct are likely to have protected against misleading results Credibility may be undermined by inappropriate eligibility criteria, inadequate literature search, or failure to optimally summarize results 2. A review with credible methods, however, may leave clinicians with low confidence in effect estimates Common reasons for lower confidence include high risk of bias of the individual studies; inconsistent results; and small sample size of the body of evidence, leading to imprecise estimates

1. Advantages of a meta-analysis 2. Limitations of meta-analysis 3. Types of meta-analyses

1. Increased power (strength) from combining results from several studies More conclusive results even if individual studies were not conclusive Greater generalizability due to larger and possibly more diverse population 2. Publication bias: ─ Positive results are more likely to be published ─ Meta-analysis based on published trail could lead to false positive ─ Should include unpublished data Issues related to combining: ─ Combining dissimilar study could lead to misleading results 3. Aggregate data meta-analysis (most common) Individual participant data meta-analysis (or individual patient data meta-analysis) Network meta-analysis

Reporting bias/publication bias Reporting bias

1. The most difficult types of bias for meta-analysis authors to address stem from the inclination of authors of original studies to publish material based on the magnitude, direction, or statistical significance of the results We call the systematic error in the body of evidence that results from this inclination reporting bias

Meta-analysis Heterogeneity 1. differences between study results can arise from

1. differences in the population enrolled (eg, large effects in the more ill, smaller in the less ill) differences in the interventions (eg, if large doses are more effective than small doses) differences in the comparators (eg, smaller effects when standard care is optimal than when it is not) study methods (eg, larger effect in studies with high risk of bias vs those with low risk of bias)

Comparison of fixed-effects and random-effects models Random effects models 1. conceptual considerations 2. statistical considerations 3. practical considerations

1. - Estimates effect in a population of studies from which the available studies are a random sample - Assumes effects differ across studies and the pooled estimate is the mean effect 2. Variance is derived from both within-study and between-study variances 3. - Wider confidence interval - Large studies have more weight that small studies, but the gradient is smaller than in fixed-effects models

Comparison of fixed-effects and random-effects models Fixed effects model 1. conceptual considerations 2. statistical considerations 3. practical considerations

1. - Estimates effect in this sample of studies - Assumes effects are the same in all studies 2. Variance is only derived from within-study variance 3. - Narrower confidence interval - Larger studies have much more weight than small studies

Meta-analysis 1. what is it 2. When the treatment effect (or effect size) is consistent 3. When the effect varies 4. A meta-analysis can be combined with other types of reviews:

1. A statistical pooling or aggregation of results from different studies to provide a single best estimate of effect 2. meta-analysis can be used to identify this common effect 3. meta-analysis may be used to identify the reason for the variation 4. ─Systematic review and meta-analysis ─ Meta-analysis but not a systematic review ─ Systematic review but not a meta-analysis

Systematic review 1. what is it 2. typically involves 3. can be found

1. A summary of research that attempts to address a focused clinical question in a systematic, reproducible manner 2. a detailed and comprehensive plan and search strategy derived a priori, with the goal of reducing bias by identifying, appraising, and synthesizing all relevant studies on a particular topic 3. in journals as well as organizations and databases that promote and disseminate them (Cochrane Collaboration)

Differences between a systemic and narrative review Systemic review 1. searching 2. appraisal of included articles 3. synthesis 4. inferences 5. use for

1. Extensive, intended to locate all primary studies on a particular research question 2. Critical appraisal is meticulous 3. A qualitative summary is provided, quantitative when the data can be pooled 4. Usually evidence-based 5. Clinical evidence

Differences between a systemic and narrative review Systemic review 1. topic 2. data sources and search strategy 3. authorship 4. article selection criteria

1. Focused research question 2. Explicitly described and comprehensive with a list of all databases utilized is listed 3. A team of experts having methodological and clinical expertise 4. Consistently applied inclusion and exclusion criteria

Differences between a systemic and narrative review Narrative review 1. searching 2. appraisal of included articles 3. synthesis 4. inferences 5. use for

1. May be extensive, intended to locate literature on the topic area in question 2. Depend on the reviewer, may be variable 3. A qualitative summary is usually provided 4. Sometimes evidence-based 5. Background information

Funnel plots Because visual determination of symmetry can be subjective, meta-analysts sometimes apply statistical tests for the symmetry of the funnel (Egger's test) 1. what is it used for 2. if P >0.05

1. The Egger's test is used to statistically examine the funnel plot for asymmetry. The nullhypothesis for Egger's test is that symmetry exists in the funnel plot, with the alternative indicatingthat asymmetry is present 2. If P > 0.05, then the null hypothesis cannot be rejected and it can be concluded that symmetry exists in the funnel plot

Higgin's I2 statistic 1. what is it 2. focuses on 3. When the I2 is 0%,

1. The I2 statistic is a preferred alternative approach for evaluating heterogeneity 2. magnitude of variability rather than the statistical significance of variability 3. chance provides a satisfactory explanation for the variability in the individual study point estimates, and clinicians can be comfortable with a single summary estimate of treatment effect

Test for heterogeneity 1. null (H0) 2. alternative (H1) 3. Goal

1. There is no difference between studies (all studies are similar; homogeneous) 2. there are differences between the studies (there is variability/heterogeneity) 3. p-value >0.05 DO NOT want to reject H0

Differences between a systemic and narrative review Narrative review 1. topic 2. data sources and search strategy 3. authorship 4. article selection criteria

1. Typically broad-scoped 2. Not explicitly described 3. Recognized expert(s) on the topic 4. Typically not specified

Meta-analysis 1. provides 2. The weighting process depends on 3. Studies that are more precise have 4. 2 types of outcomes:

1. a best estimate of effect (often called a summary or pooled estimate) from the weighted averages of the results of the individual studies 2. sample size or number of events or, more specifically, study precision 3. narrower confidence intervals (CIs) and larger weight in meta-analysis 4. - Dichotomous outcomes - Continuous outcomes

Cochrane Q -most common test used 1. generates 2. threshold for significance 3. low p value 4. high p-value 5. When a meta-analysis includes studies with small sample sizes and a correspondingly small number of events, the test of heterogeneity may 6. is

1. a probability based on a χ2 distribution that between-study differences in results equal to or greater than those observed are likely to occur simply by chance 2. a conventional threshold of P < 0.05 or a more conservative threshold of P < 0.10 3. A low P value of the test for heterogeneity means that random error is an unlikely explanation for the differences in results from study to study 4. A high P value of the test of heterogeneity, on the other hand, increases our confidence that the assumption underlying combining studies holds true 5. not have sufficient power to detect existing heterogeneity 6. not very accurate and requires an additional test

Visual assessment 1. left example 2. middle 4. right

1. all four are statistical significant -combined together and shows it favors treatment -shows huge heterogeneity -no overlap with the 1+2 and 3+4 2. none of statistical significant -but once combined statistical significance -minimal heterogeneity -this is homogenous meta-analysis 3. these disagree -good amount of overlap however - some heterogeneity but not too much -on

Visual assessment of variability 1. we are comfortable with a single summary effect when 2. A better approach to assessing heterogeneity focuses on the magnitude of the differences in the point estimates of the studies a. large differences b. small differences 3. Lastly, if CIs overlap widely 4. When CIs do not overlap,

1. all studies suggest benefit or all studies suggest harm 2. a. Large differences in point estimates make clinicians less confident in the pooled estimate b. Small differences in the magnitude of point estimates support the underlying assumption that, across the range of study patients, interventions, and outcomes included in the meta-analysis, the effect of interest is more or less the same 3. random error, or chance, remains a plausible explanation for the differences in the point estimates. 4. random error becomes an unlikely explanation for differences in apparent treatment effect across studies

Reporting bias/publication bias 1. The consequences of publication and reporting bias 2. Meta-analyses that fail to identify and include unpublished studies face a risk of 3. The risk of publication bias is probably higher for systematic reviews and meta- analyses that are based on

1. can corrupt the body of evidence, usually exaggerating estimates of magnitude of treatment effect 2. presenting overly confident estimates of treatment effectiveness 3. small studies Small studies are more likely to produce nonsignificant results due to lack of statistical power and are easier to hide Larger studies are not, however, immune. Sponsors and authors who are not pleased with the results of a study may delay publication or choose to publish their study in a journal with limited readership or a lower impact factor

Forest plots 1. If a study's 95% CI touches or crosses over (includes) the vertical line of no effect, it is 2. The combined summary effect is usually presented as a 3. As the CI widens, uncertainty about the magnitude of effect 4. when the CI crosses no effect (RR or OR of 1.0), there is

1. not statistically significant 2. diamond, with its width representing the CI for the combined effect 3. increases; 4. uncertainty about whether the intervention has any effect at all

Funnel plots 1. what is it 2. should be 3. A gap or empty area in the funnel suggests that

1. relates the precision of studies included in a meta-analysis to the magnitude of treatment effect, the resulting display should resemble an inverted funnel 2. symmetric, around the point estimate (dominated by the largest trials) or the results of the largest trials themselves 3. studies have been conducted and not published

Forest plots 1. show 2. point estimate 3. solid line at 1.0

1. the effect (ie, the result) from every study 2. presented as a square with a size that is proportional to the weight of the study, and the CI is presented as a horizontal line 3. (1.0 if the effect is a ratio, 0 if the effect is a difference) indicates no effect, and sometimes a dashed line is centered on the meta-analysis combined summary effect

Meta-analysis Dichotomous outcomes (yes/no) 1. In a meta-analysis of a therapeutic question looking at dichotomous outcomes (yes/no) for estimates of the magnitude of the benefits or risks, you should look for 2. When the outcome is analyzed using time-to-event methods (eg, survival analysis), the results could be presented as

1. the relative risk (RR) or theodds ratio (OR) 2. a hazard ratio (HR)

Meta-analysis Continuous outcomes In the setting of continuous variables rather than dichotomous outcomes, meta-analysts typically use 1 of 2 options to aggregate data across studies: 1. If the outcome is measured the same way in each study (eg, duration of hospitalization), 2. Sometimes the outcome measures used in the primary studies are

1. the results from each study are combined, taking into account each study's precision to calculatewhat is called a weighted mean difference 2. similar but not identical.

Superiority RCT 1. null (H0) 2. alternative (H1) 3. Goal 4. Outcome

1. there is not difference between the groups 2. there is a difference between the groups 3. p-value <0.05 to reject the null 4. is statistically significant if p-value less than alpha or CI does not cross 1 or 0 - 1 if looking at ratio (<1 means intervention is superior) -0 if looking at difference (<0 means intervention is superior)

Why seek systematic reviews 1. single studies are liable to 2. Collecting and appraising a number of studies takes 3. A systematic review is often accompanied by 4. If the systematic review is performed well, it will 5. Systematic reviews include

1. to be unrepresentative of the total body of evidence, and their results may therefore be misleading 2. time you probably do not have 3. a meta-analysis to provide the best estimate of effect that increases precision and facilitates clinical decision making 4. likely provide all of the relevant evidence with an assessment of the best estimates of effect and the confidence they warrant 5. a greater range of patients than any single study, potentially enhancing your confidence in applying the results to the patient before you

Meta-analysis Heterogeneity 1. what is it 2. solves the dilemma of

1. variability 2. combining the results of diverse studies may violate the starting assumption of the analysis (that across the range of study patients, interventions, and outcomes included in the analysis, the effect of interest is more or less the same) and lead to false conclusions - evaluate the extent to which results differ from study to study

Higgin's I2 statistic 1. As the I2 increases, 2. It has been suggested that the adjectives low, moderate, and high (heterogeneity) be assigned to I2 values of 25%, 50%, and 75% What is significant?

1. we become progressively less comfortable with a single summary estimate, and the need to look for explanations of variability other than chance becomes more compelling 2. An I2 ≤ 25% indicates homogeneity, whereas I2 ≥ 50% indicates significant heterogeneity

Types of bias 1. measurement bias 2. detection bias 3. observer bias 4. recall bias 5. selection bias 6. confounding bias 7. funding bias 8. reporting bias 9. publication bias

1. when methods of measurement are dissimilar among groups of patients 2. when a phenomenon is more likely to be observed for a particular set of study subjects 3. when the researcher subconsciously influences the experiment due to judgement 4. due to differences in the accuracy or completeness of participant recollections of past events 5. when comparisons are made between groups of patients that differ in determinants of outcome other than the one under study 6. when two factors are associated and the effect on one is confused with or distorted by the effect of the other 7. when selection of outcomes, test samples, or test procedures are to favor a study's financial sponsor 8. a skew in the availability of data, such that observations of a certain kind are more likely to be reported 9. tendency for published studies to be systematically different from all completed studies of aquestion; published studies are more likely to be "positive"