Module 8

Meta-analysis

A statistical analysis of the results (averaging/pooling) from independent studies, which generally aims to produce a single, more precise estimate of a treatment effect (therapy), sensitivity and specificity (diagnosis), estimates of risk/prognosis, etc

Weighted mean difference in meta-analysis

All studies use same scale (ex: numerical scale from 1-10) use OR, RR, mean

Systematic Review

Application of scientific/systematic strategies to limit bias in the gathering, critical appraisal, and synthesis of relevant (current) studies on a specific topic

Why do we like quality scales?

Avoids thinking but assumes someone else got it right To explain heterogeneity Exclude or decrease the weight assigned to studies of low quality

What are good things to do to reduce bias in quality assessment?

Blind the assessor to the author, institution, and journal as these may be well-known, or renowned Create pilot forms Duplicate, independent ratings

Risk of bias does not mean ________

bias occurred therefore should not exclude just bc risk of bias, should exclude if bias occured

Standard mean difference in meta-analysis

if studies use different measures, must convert each study's results onto a normal distribution where mean = 0 and each study is analyzed based on relative standard deviation units from the mean

Reporting of outcomes in protocol in the manuscript is relevant to _________

internal validity

Methodological heterogeneity - ______ validity

internal validity; methodological ways that trials differ (blinding, allocation concealment, proportion lost to follow up)

It is important to include the ______ and ________ of hypotheses about sources of heterogeneity

magnitude and direction ex. studies that blinded the outcome assessor will show smaller treatment effects in favour of the experimental treatment compared to studies that do not blind the outcome assessor

Risk of bias increases if the positive studies are _____ or _____

small or sponsored

Steps to evaluating heterogeneity

1. Create hypotheses that can be tested later 2. Conduct analysis + produce forest plot 3. Test hypotheses 4. Re-run analysis and produce new forest plots - is the heterogeneity gone? 5. Present the results from each group. Conclusions/recommendations should be separate for each group.

What are the steps of systematic review?

1. Formulate the question/questions 2. List possible sources of heterogeneity 3. Conduct the search of the literature 4. Define eligibility criteria and select relevant studies 5. Assess the quality of included studies 6. Extract the data 7. Conduct the meta-analysis (if appropriate) 8. Evaluate sources of heterogeneity (if necessary/if were found)

Steps for study selection with full texts?

1. Identify eligibility criteria 2. Create + pilot (and explain) a reviewer data form 3. Pair reviewers 4. Determine report agreement - classify as exclude or include or uncertain 5. Report agreement as weighted Kappa 6. Reach consensus on disagreements (reviewer may have misread article, or being in experienced 3rd reviewer) 7. Keep a record of excluded studies and reason why (table as online supplement)

Steps for study selection with titles and abstracts?

1. Identify eligibility criteria 2. Create + pilot (and explain) a reviewer data form 3. Pair reviewers 4. Determine report agreement - classify as exclude, or 'review in full text' 5. Determine and report agreement as weighted Kappa 5. Retrieve full text articles for articles marked by both reviewers to review in full text and ones where 1 reviewer said to review full text (disagreements)

For precision evaluate _____ of effect estimate not ______

CIs not p-values

Unpublished and non-peer reviewed

Clinicaltrials.gov or other protocol registries; grey lit, theses; personal communication with experts; conference proceedings

Extracting Data practices

Create pilot forms/potentially create manual Duplicate, independent reviews Consider writing to authors for more/missing info Include a 3rd reviewer Ensure there is a good method to detect errors in copying

Why do we need to decide whether to use fixed or random effects to conduct meta-analysis?

Decision is tied up in which sources of variability or noise should be counted in the denominator of the statistic

I Squared test - what is it?

Estimates the percentage of variation that is due to heterogeneity Low (25%), moderate (50%), high (75%) heterogeneity

What could indicate random sampling error in meta-analyses?

Even if CIs are narrow, if SS or # events is small, or the study stopped early for efficacy --> RSE is probable not miracle

What can we use to illustrate meta-analyses?

Forest plots provide a visual display by showing whether study favoured Tx or Ct with confidence intervals of RR, OR, etc Larger square means larger study

How do we explain heterogeneity?

Generate hypotheses Test hypotheses and if explained = factor like mild/mod patients vs severe --> similar results in two groups--> explained heterogeneity

What happens if b/w variability is small? B/w variability is large?

If same, then fixed and random effects will gave similar estimate If bigger, then random will give wider CIs

What is the Fragility Index?

Indicator of precision - tells you how many events would be required to change the results from favouring one group --> opposing group If low --> fragile --> low precision and less confidence

When uncertain, what do you do if outcome is proportion and event rate is low (<10%!) in control group?

Interpret both the relative (RR, OR) and absolute perspective (risk difference) where one of the perspective may be uncertain, the other may offer certainty

What is the first step to narrowing down the search yield for study selection?

Looking at Titles and Abstracts

What is the problem with quality scales?

Many quality assessments include external validity and reporting issues when we really want to rate internal validity Not all methodological features are necessary for every research question. Ex: if it is objective measure, blind to Tx group does not matter as much but is v important with subjective

Why are positive studies more likely to be included in systematic review?

More likely to be published, more likely to be published rapidly (time lag bias), more likely to be published in English (language bias), more likely to be cited by others (citation bias)

Peer reviewed literature search exaples

More than one electronic database (MEDLINE, PUBMED, EMBASE, etc); reference list of relevant articles; hand search of relevant journals

Why is it good to look at unpublished work?

Not just pubs bc publication bias

Publication bias in systematic review - usually ______, _______ studies

Omission of studies that should be included (usually the small, negative studies thus tend to overestimate the effect)

If OIS is not met --> rate ______

Rate down for imprecision unless sample size is very large (>200 events and 4000 patients) + evaluate CI's

What are the problems with funnel plots and the trim and fill method?

Relies on large studies - sometimes large studies can be biased any correct of overall tool is still incorrect; prone to issues of power (removing studies, decreases sample size)

What do you do if for a continuous outcome, studies use different outcome measures for the same construct (SMD situation)?

Select an outcome that has been shown to have good measurement properties and include precise estimates of MCID - note: will likely have to change to a b/w group MCID

Is it always desirable to a systematic review? A meta-analysis?

Sys review - always Meta-analysis - not always, may be misleading to statistically pool results from separate studies

Cochrane Chi Squared test - what is it? what does p < 0.05 imply? what is its main drawback?

Tests whether the estimates of effects between studies are similar (homogeneous), where p < 0.05 implies significant heterogeneity (bad) Unfortunately test is affected by # of studies so rejecting heterogeneity could be because the review is underpowered

Optimal Information Size is ______ as an individual study

The same calculation as for an individual study

What are fixed effects? Why would someone prefer using them?

W/in study variability (it is both) - assume all studies are using subjects from the same population and variability b/w studies is not a source of error

What are random effects? Why would someone prefer using them?

W/in study variability (variability if same study with same subjects reported); b/w study variability (variability if same study is repeated in different population) - do not believe claim of subject being from the same population Involves more terms in denominator/noise --> more conservative --> wider confidence intervals

Why shouldn't we pool/average heterogeneic studies?

Will not account for subgroups of patients who respond differently. May be better to give averages of each group rather than averages being inaccurate/misleading.

Consider ______ of treatment

consequences (side effects, costs, inconveniences)

If OIS is met and CI overlaps no effect --> rate ____

down if CI fails to exclude important benefit or important harm (equality)

Clinical heterogeneity - ______ validity

external validity; Refers to difference in clinically important features like patient selection, baseline features, severity, admin of intervention)

Funnel plot

most common graphical method to detect publication bias

Trim and Fill

most common statistical method to adjust for bias. 1. Trim = Remove small positive studies that do not have negative counterpart and calculate pooled estimate. 2. Fill = Replace with hypothetical negative studies that mirror the positive studies --> CI of estimate

If OIS is met and 95% CI excludes no effect (RR = 1) --> rate ______

precision as adequate