Meta-analysis
Have to read the article-authors do not label forest plot well but treatment was on left and standard care on right so yes, it was effective
Was the culturally tailored approach effective?
No, I2 was 0% The studies were consistent in their results
Was there heterogeneity?
No, I2 was 1.4% The studies were consistent in their results
Was there heterogeneity?
YES Based on size of samples (Column 8 labeled "Weight")
Were the studies weighted?
- The CI indicates a *range* of values that is likely to include the true value (which is unknown and unknowable but our best estimate) - The p value is our best *point* estimate - The CI contains two numbers, an upper and lower number - The two numbers that make up the lower and upper ends of the CIs are the lower and upper confidence limits (CLs)
Confidence Intervals (CI)
- circle formed by the 'C' of Cochrane and the mirror image 'C' of Collaboration reflects international collaboration - The inner part of the logo uses a FOREST PLOT to illustrate a systematic review of data from a famous review of 7 randomized controlled trials (RCTs), comparing one health care treatment - steroids in pregnant women - with placebo - Each horizontal line represents the results of one RCT (the shorter the line, the more certain the result). - The bolded diamond (lower left) represents their combined results of ALL the studies in the M-A
Describe the aspects of the Conchrane Logo - What do the C's represent? - What is the inner part of the logo? - The length of the horizontal lines? - The diamond?
Yes; p < 0.00001 (lower left corner, test for overall effect) Do you think it is clinically significant? I think so- this is an inexpensive approach to helping patients. Would be interesting to see if American HD patients benefited.
Did listening to music significantly decrease anxiety? - Is it clinically significant?
NO! If Intervention A increases length of life in gravely ill people 2.5 weeks compared to standard care, is it worth it if it also causes severe pain and costs $110,00 per person? - You need to make a judgment about that - In general, we look for about a 20% difference between groups when thinking about clinical/practical significance
Does statistical significance mean clinical or practical significance?
This plot indicates possibility of publication bias --> asymmetrical and indicates possible publication bias
Does this study show publication bias?
but they are not ready to use in practice! -For changing practice, you need a clinical guideline from our old friends Joanna Briggs and Archie Cochrane (both residing at the BU Library) and guidelines.gov and tripdatabase.com (both residing on the internet) -Think of systematic reviews and meta-analyses as groceries and guidelines as dinner. -The systematic reviews and meta-analyses have all the ingredients but they need to be transformed into a guideline before consumed!
Systematic reviews and meta-analyses are at the top of the evidence pyramid, can info from these be used in clinical practice?
- The vertical line (Line down the middle) indicates the position around which the horizontal lines would cluster if the two treatments compared in the trials had similar effects -The vertical line is also called the line of no effect -If a horizontal line touches the vertical line, it means that that particular trial found no clear difference between the treatments The position of the diamond ◊ to the left of the vertical line indicates that the treatment studied is beneficial. Horizontal lines or a diamond to the right of the line would show that the treatment did more harm than good.
- What does the vertical line represent and what is it called? - What does it mean if the horizontal line touches the middle? - What does it mean that the diamond is on the left side of the line?
- You need enough previously completed high quality studies to do the work-this is a common problem ex. Youngmee- she was only able to locate 5 music and hemodialysis studies - The research can be difficult related to accessing materials?? - M-A is usually a big job and requires many people-Youngmee worked by herself (on the STUDY- wrote article with others) but I think this was a dissertation. Sometimes have 5-10 people working to review thousands of articles (or at least abstracts) - Because it takes time for sufficient studies to accumulate and to do the work, the M-A results could be obsolete by the time they are published - If one study is heavily weighted, it could distort the results
Disadvantages of Meta Analysis
no publication bias
Does this study show publication bias?
- They are graphical representations of the meta-analysis - The term originated from the idea that graph had a "forest of lines" - The plot originated in the early 80s although the term forest plot was coined only in 1996 - They are sometimes called blobbograms
Forest Plot
Also-Meta-analyses help clinicians manage the huge volume of data they are confronted with every week from journals, list servs, FDA & CDC Bulletins, employers, drug reps, etc.
How do Meta-analyses help clinicians?
- The CI varies inversely with the square root of the sample size (as one goes up, the other goes down) - This means that if you quadruple the sample size you will cut the width of the CI in half - You want a narrow CI - A narrow CI is more precise (and more desirable) - A wide CI is less precise (and therefore less desirable) - Therefore, it is important to have a good sample size
How do sample size and confidence intervals relate? --> do you want a narrow or wide confidence interval?
(Studies with large sample sizes and strong effect sizes are often given more weight than studies with smaller samples and smaller effect sizes) - Studies are (usually) weighted, so that studies contribute to the total based on their own sample size - Smaller studies have less influence than larger studies with more subjects
How does a study in a meta- analysis get more weight?
- 2007 federal requirement that if a researcher is receiving funding from a major source (e.g. NIH) or wishes to publish in a major journal, the study must be registered at clinicaltrials.gov - However, the law is not always being followed
How is publication bias being combated?
SEVEN
How many studies were included in this meta-analysis?
175 in the music (treatment) groups and 176 in the standard care hemodialysis control groups, total N = 351 (N = total sample size)
How many subjects were in the combined studies?
- THIS LEFT/RIGHT DISTINCTION IS NOT STANDARD! - YOU MUST READ THE LABELS on the value axis at the bottom of the graph! - Look at the Youngmee study on slide 35!!! - The diamond INDICATING THE INTERVENTION WAS BETTER THAN CONTROL IS NOW ON the RIGHT - In comparison, the diamond indicating treatment was better than control in the M-A depicted on the Cochrane logo is ON THE LEFT - Must read the labels!
If the diamond is on the left side of the line of no effect will it always indicate the treatment is beneficial?
- Higher Confidence Levels have wider confidence intervals - For example, 95% CI = 97-114 BUT 99% CI = 88-123 - This means we are 95% confident the TRUE value is between 97 and 114 - But, if we wanted do be 99% confident of finding the true value, we would have to widen the interval, for example to this: - 99% CI = 88-123 - It comes down to how worried you are about being wrong - Most of the time people are OK with a 5% risk of being wrong and use 95% Confidence Levels
If you had a high confidence level, how would that affect your confidence interval?
Tertiary
Is a M-A a primary, secondary or tertiary data source?
- d: the standardized difference between 2 groups (e.g., Treatment vs. Control) on an outcome for which a mean can be calculated (e.g., BMI) - Odds Ratio (OR): Relative odds for two groups on a dichotomous outcome (e.g., smoke/not smoke) - r: correlation between 2 continuous variables (e.g., age and depression)
Major effect size indexes
-Combines sample sizes from mult. studies to increase the total sample size and therefore stronger results -(Remember, N is total sample size and n is sub-sample size)
Meta analysis and sample size
just talk through it
Review Questions What is the difference between a systematic review and a meta-analysis? Between a meta-analysis and a metasynthesis? Between a meta-analysis and a guideline? What does it mean in a forest plot if the diamond crosses the line of no effect?
1. Identify the clinical problem 2. The researcher looks for published and unpublished literature on the subject 3. The research question or hypothesis should be essentially identical across studies --> Must be a sufficient knowledge base, must be enough studies of acceptable quality 4. NOW M-A WORK STARTS - Phase 1. Extract the data. Pull out data on the sample and outcomes from each study - Phase 2. Researcher decides whether it is reasonable to calculate the pooled average result (effect size) of the studies under review 5. An effect size is calculated for each study and then a single effect size for the entire group of studies
Steps in a Meta analysis
- Summarize and assess research studies - Systematic reviews and meta-analyses are sometimes used interchangeably but *ALL* meta-analyses use stats to evaluate data but only *some* systematic reviews do - Systematic reviews are more rigorous than traditional narrative reviews - SR require careful collection, review, and evaluation or articles - SR are not mere summaries like narrative review AKA traditional stand alone literature reviews - BUT, to be fair, narrative reviews have a place- they can be very helpful when you want to learn about a new or complex topic- e.g., you are changing jobs and need to get proficient about HF real fast
Systematic Reviews
- The probability that the confidence interval encompasses the true value is called the confidence level of the Confidence Interval - The most commonly used value of confidence level is 95% --> Although you can use any confidence level you wish (e.g., 99%, 90%, etc.) - When reporting a confidence interval, you must also report the confidence level, for example, 95% Confidence Interval = 97-114 - It comes down to how worried you are about being wrong - Most of the time people are OK with a 5% risk of being wrong and use 95% Confidence Levels
What are Confidence Levels?
- Each study is represent by a single dot - If the data are symmetrical ("well-behaved") it indicates there probably was not bias in publication of studies - If the data are NOT symmetrical it indicates there may be publication bias - Funnel plots are generally considered to be flawed in concept
What are Funnel Plots?
- These are the broadest category of reviews - Can include quantitative or qualitative research and/or theory literature - They DO NOT include statistics to summarize studies ands generate conclusions (If they did they would be a META-ANALYSIS!) - They do include suggestions for future research
What are Integrative Reviews
- Metasynthesis (theoretical integration and interpretation of *qualitative* findings) - Bring together and break down findings, examine them, discover essential features, and combine phenomena into a transformed whole - Integrations that are more than the sum of the parts—develop novel interpretations of findings - Nevertheless, BC these are QUALitative studies, the metasynthesis will be at the bottom of the EBP pyramid while the meta-analysis is always at the top
What are Metasynthesis?
- Traditional review of literature - Typically lack a method section - More idiosyncratic, biased-avoid doing one of these- aim higher! - If you are going to read or do a review-make it a systematic review
What are Narrative Reviews?
- Studies with negative results (showing no difference between groups) or weak differences between groups are less likely to be submitted at all for publication, or to be slow to be submitted, and are more likely to be published in less prestigious journals - Drug companies own their research data and have been known to not publish studies with negative results! - So...excluding them could result in overestimating beneficial effects. We would mistakenly be over-optimistic about the direction and magnitude of effects of a mediocre drug on lowering cholesterol!
What are some problems with publication bias?
- Objectivity: Use of statistically integrated studies eliminates or at least decreases bias in drawing conclusions when results in different studies are at odds - Increased power: Reduces risk of Type II error compared to single study because your total sample size has increased-This is discussed in detail on the next slide -Increased precision: Large sample size results in smaller confidence intervals than single studies
What are the advantages to an M-A?
- Omit low-quality studies (e.g., in intervention studies, eliminate non-RCTs (e.g., no poorly done RCTs, no quasi-experimental designs, no observational studies, etc.) - Give more weight to high-quality studies - Analyze low and high-quality studies to see if effects differ (sensitivity analyses) and make some judgments about what to keep/toss
What are the approaches Meta- Analysts use when handling study quality?
- I2 will range from 0% to 100% - I2 measures heterogeneity - Heterogeneity is the variability between studies; it gives an indication how comparable studies in the M-A are or are not - If the horizontal lines in the confidence intervals overlap, the studies are homogenous-this is good (Look down the rows) - We kind of hope that the studies are consistent-it make it easier to make recommendations - If studies are inconsistent- back to the drawing board (They do overlap in both previous examples)
What does I2 measure?
- A 95% Confidence Level (CL) contains the true value 95% of the time - A 95% CL does not contain the true value 5% of the time - Most of the time people are OK with a 5% risk of being wrong and use 95% Confidence Levels
What does a 95% Confidence Level mean?
- If the studies are heterogeneous, it means there are inconsistent results between studies - If I2 is less than 25%, studies are homogenous, - If I2 is over 75%, there is high heterogeneity - This matters because I2 will determine whether you will use a fixed or random effects model (beyond the scope of this discussion )
What does it mean if there is heterogeneous results?
The weight of the study - The larger the box the heavier it is weighted
What does the box size indicate
- Gray literature is unpublished literature not available on commercial databases such as CINAHL and Medline - This could include material such as conference proceedings, dissertations, internal organizational documents (e.g., Mercy Hospital morbidity and mortality data for 2015) - These are termed "gray" or "fugitive" literature because they can be difficult or impossible to track down - For example, a graduate student in Wyoming completes an MSN thesis on music and hemodialysis and a copy is kept in the U of Wyoming University Library. The thesis is in the University library catalog but not on any other database (Maybe BC the student is too tired to write an article & submit for publication) - How would you access this thesis??? - I think the only way would be word of mouth, for example you went to conference and heard the author speak. If that were the case you could then request a copy/actual document through interlibrary loan
What is Gray Literature?
- It's a systematic summary of research studies-almost always of RCTs but sometimes high-quality quasi-experimental designs or even observational studies - M-A are done to establish a precise estimate of the effect (impact) of an intervention (IV) on an outcome (DV)
What is a Meta-analysis? (M-A) - M-A are done to ___
(usually a difference in scores between/among groups) is a result that is not due to chance
What is a statistically significant result?
- Type II error means you report an intervention did not make a difference when it did - This is a lost opportunity-People could be benefitting from something if only you had not told them they would not!
What is a type 2 error?
the observed differences between group scores was just the result of chance or happenstance
What is sampling error?
- probability value - It is one of the outcomes of a statistical test when making a comparison between scores of two (or more) groups - The p-value is the probability of obtaining the difference you got from your sample if there really is not a difference for all subjects
What is the P value?
- A Meta-analysis is a statistical assessment of previous research studies - A systematic review is a critical assessment of previous research (meaning the researcher rigorously evaluates studies but does so without using statistics) - ONLY the meta-analysis creates NEW knowledge
What is the difference between a systemic review and a meta-analysis?
The key piece of data (The effect size) is how large are the differences between the treatment group and the control groups in each study (How much difference did the IV make on the treatment group compared to the control group?) --> A meta-analysis effect size is an estimate of how large the differences are between the treatment and control groups for the summarized studies
What is the effect size?
0.05, meaning that you are taking a chance of being wrong 1 time in 20 This is usually fine unless working with very delicate material such as the ebola vaccine
What is the minimal acceptable P value?
Let's us see immediately whether or not the intervention was better than the comparison/standard care
What is the significance of the diamond?
The line of no effect-if the diamond touches this line the results of the M-A are not significant and if it doesn't the results are significant
What is the solid line down the middle of the forest plot?
The Cochrane Collaboration
What is the world's largest collection of systematic reviews and meta-analyses
- This diagram shows the results of a systematic review of RCTs of a short, inexpensive course of a corticosteroid given to women about to give birth prematurely - The first of these RCTs was reported in 1972. The diagram summarizes the evidence that would have been revealed had the available RCTs been reviewed systematically. - A decade later it indicates strongly that corticosteroids reduce the risk of babies dying from the complications of immaturity. - By 1991, seven more trials had been reported, and the picture had become still stronger. Corticosteroid treatment reduces the odds of babies dying from the complications of immaturity by 30% to 50%. - Because no systematic review of these trials had been published until 1989, most obstetricians had not realized that the treatment was so effective - As a result, tens of thousands of premature babies have probably suffered and died unnecessarily (and needed more expensive treatment than was necessary). - This is just one of many examples of the human costs resulting from failure to perform systematic, up-to-date reviews of RCTs of health care!
What study is the Cochrane Logo depicting?
0.55-0.98. - 0.76 is the best estimate and 0.55-0.98 is the confidence interval- we are 95% sure the true score lies between 0.55 and 0.98 and 0.76 is our best estimate
What was the confidence interval?
Lower left corner: 95% CI
What was the confidence level?
Authors are showing us all the studies overlap and are therefore not heterogenous. (Could have just calculated I2)
What's the broken line down the middle?
O'Hare, 2004. Anderson's box looks about the same size but look at the weights- 14.33 vs. 12.89
Which study was weighed the most?
- Because previous studies may have been too small to detect an effect of an IV on a DV (they were underpowered-remember Jeff?), or results may have been inconsistent and we want to convincingly establish an answer. Also, we cannot change practice based on individual studies- we need a coherent package of evidence
Why are M-A needed?
