Meta-analysis
What is the objectives within formulation of a research question
Identify the central question/objective that will guide the review formula: Intervention for population with condition Define the context from which the review arises
What is the overall treatment effect in a meta-analysis
It is the weighted average of the individual OR's
In the absence of publication bias, what would a funnel plot look like
It should be symmetrical
What does the validity of a review article depend on
Its methodological quality
How do meta-analyses and systematic reviews relate to each-other
Often, systematic reviews include a meta-analysis component which involves using statistical techniques to synthesise the data from several studies into a single quantitative estimate or summary effect size (Petticrew & Roberts, 2006).
Define point estimate
The best guess of the true effect in the population
Regarding the diamond at the bottom of a forest plot, what does it show
The middle of the diamond is the pooled meta-analysis result for the treatment effect, and the width of the diamond is the certainty of the result, again usually presented as a 95% confidence interval. If the confidence interval crosses the vertical line, then there is no significant difference in the effectiveness of the corticosteroid treatment group and the placebo group.
When are OR and RR used
When independent and dependent variables are dichotomous, often coming from chi squared analysis or logistic regression analysis
What is the content of data extraction forms
- Study design and research methodology - Participant characteristics - Exposure measures - Outcome measures - Statistical measures needed to calculate an Effect Size (ES) 1. Continuous measures (scales numeric data)- Means/SD's- Difference between groups-(t-test, F-statistic, p-value) 2. Categorical data - Number in treatment and control groups along with the total N's for each group- Statistical tests for differences (chi-square, df, p-value)
What are the most common indices of effect size
Standardized mean difference (cohen's d), odds-ratio, correlation coefficient.
What are the 3 main issues to consider when appraising a systematic review
1 - The validity of the trial methodology 2 - The magnitude and the precision of the treatment effect 3- The applicability of the results to your patient or population
What are the 6 keys steps for a systematic review
1 - Topic formulation (define study objectives and central questions/hypothesis) 2- Study design (Develop protocol, specify conditions/populations/settings/exposure/outcomes, define exclusion/inclusion criteria) 3 - Sampling (sampling plan where the sampling unit is each study, consider all relevant studies and develop a search strategy) 4 - Data collection (data abstraction from studies onto standardised forms) 5 - Data analysis (descriptive data and analytical) 6- Reporting (Description of results in narratives/tables/graphs, interpretation and discussion, implications for policy, practice and further research)
List the methodology for a systematic review of RCT's
1- State objectiveness of the review and outline eligibility criteria 2 - Comprehensively search for trials that seem to meet that criteria 3- Tabulate characteristics of each trial identified and assess its methodological quality 4- Apply eligibility criteria and justify any exclusions 5- Assemble the most comprehensive dataset feasible 6- Analyse the results of eligible RCT's using statistical synthesis of data (meta-analysis if appropriate and possible) 7- Compare alternative analyses if appropriate and possible 8 - Prepare a critical summary of the review, stating aims, describing materials and reporting results
What are the two main stages within a meta-analysis
1- The calculation of a measure of treatment effect with its 95% confidence intervals (CI) for each individual study. The summary statistics that are usually used to measure treatment effect include odds ratios (OR), relative risks (RR), and risk differences. 2- an overall treatment effect is calculated as a weighted average of the individual summary statistics.
What is the minimum number of databases you should check when conducting a systematic review
3
Provide the definition of a clinical trial
A Systematic investigation in human subjects for evaluating the safety and efficacy of a new drug/treatment.
Define systematic review
A form of research that provides a summary of medical reports on a specific clinical question, using explicit methods to search, critically appraise and synthesise the world literature systematically
In a forest plot, what is the pooled estimate represented by
A symbol below the studies - the middle of the symbol shows the pooled estimate and the size of the symbol shows the 95% CI
Discuss 'differences' within the context of forest plots
All differences usually used as effect measures in meta-analyses are absolute measures. with 0 indicating no effect. f 0 is included in the 95% confidence intervals, it indicates that there is no statistical significance at 5% significance levels. If 0 is not included in the 95% confidence intervals, the results are statistically significant at 5% significance levels. This is applicable for effect estimates for the individual study level and for the overall estimate. Whether the intervention is beneficial or harmful depends upon the context.
What is the Cochrane Collaboration?
An international organisation that prepares and maintains a body of systematic reviews that focus on health care interventions.
Give an interpretation of OR and RR relative to 1
An odds ratio or relative risk >1 indicates increased likelihood of the stated outcome being achieved in the treatment group. If the odds ratio or relative risk is <1, there is a decreased likelihood in the treatment group. A ratio of 1 indicates no difference—that is, the outcome is just as likely to occur in the treatment group as it is in the control group.
How many reviewers should independently asses the quality of included studies within a review and why
At least 2 - to minimise the risk of selection bias
How can publication bias be minimised
At the selection stage explicit efforts should be made to identify all studies and not just those published.
Why is meta-analyses good in terms of power
By combining the samples of individual studies the overall sample size is increased thereby improving power
Describe data synthesis for quantitative data
Combining the results from 2 or more studies (i.e. effect size results) of the same research question to provide a new statistical result (i.e. a combined effect size)
Provide a summary of meta-analysis as a method
Describes statistical procedures of combining quantitative data from different studies Results from individual studies are converted to one or more common metrics, called effect sizes Effect sizes are combined across studies, which have used different measures of the same construct (for e.g. Different measures for ADHD) or report results in different ways.
Describe data synthesis for qualitative data
Data from primary studies are qualitatively reported and summarised i.e. discussed in the text.
When an overall treatment effect is calculated within a meta-analysis what is important to remember when factoring in all studies into this statistic
Data from the individual studies are not simply combined as if they were from a single study. Greater weights are given to the results from studies that provide more information, because they are likely to be closer to the ''true effect'' we are trying to estimate. The weights are often the inverse of the variance (the square of the standard error) of the treatment effect, which relates closely to sample size.
What are the features of a forest plot
Each study is represented by a line, midpoint shows the point effect estimate of each study, area is the weight given to each study. The diamond represents the overall effect estimate. The width of the line shows the CI of the overall effect estimate for the study, the width of the diamond = CI for overall effect estimate
What is a crossover design study
Each subject receives multiple treatments in a randomised order with suitable gaps between treatments (washout) and outcome/response is compared within subjects
Name the tests used to detect publication bias
Egger test, Rosenthal's Fail-safe N
What makes a good research question
Feasible Interesting Novel Ethical Relevant (FINER)
How is a systematic review different from a meta-analysis
Following a systematic review, data from individual studies may be pooled quantitatively and reanalysed using established statistical methods. This technique is called meta-analysis.
What is the name of the graph usually used to represent the results of a meta-analysis
Forest plot
Name graphical methods used to detect publication bias
Funnel plot asymmetry
What are forest plots
Graphical representation of the meta-analysis - Visual presentation of the amount of variation between the results of studies - Show data from individual studies that went into the meta-analysis
Discuss pooling results and heterogeneity
If the results of the individual studies were pooled in a meta- analysis, it is important to determine whether it was reasonable to do so. The statistical validity of combining the results of the various trials should be assessed by looking for homogeneity of the outcomes from the various trials. In other words, there should be some consistency in the results of the included trials. (look at forest plot for similarities) Some articles may also report a statistical test for heterogeneity, but it should be noted that the statistical power of many meta-analyses is usually too low to allow the detection of heterogeneity based on statistical tests.
Describe Heterogeneity - Problem of combining information
If the results of the individual studies were pooled in a meta-analysis, it is important to determine whether it was reasonable to do so. The statistical validity of combining the results of the various trials should be assessed by looking for homogeneity of the outcomes from the various trials. In other words, there should be some consistency in the results of the included trials. One way of doing this is to inspect the graphical display of results of the individual studies (i.e. forest plot) looking for similarities in the direction of the results.
What is a non-inferiority design
Interventions are compared to each other, but the design is powered to conclude that there is little evidence of a difference between them- rather than one being superior to the other
What is the vertical line within a forest plot
It corresponds to the value of 1 (sometimes!), this is the line of no effect and shows which direction favours experimental or control conditions
What is a factorial design
Multiple interventions are all compared together
Name the 4 main types of review articles
Narrative reviews, systematic reviews, meta-analysis, pooled analysis
What are the differences between the different types of review articles
Non-systematic reviews (e.g. narrative) are mainly descriptive and do not involve a systematic search. They focus on a subset of studies and there is an element of selection bias, usually without a methods section. Systematic reviews (of which MA is a type), involve detailed and comprehensive plan and search strategy and the methods section is very important In a meta-analysis there is the use of statistical techniques to synthesise the data from several studies into a single quantitative estimate or summary effect size.
What are the two names for guidelines that can be used for writing up a systematic review
Prisma, Moose
Would should the protocol for a systematic review include (prior to its commencement)
Question, primary and secondary outcomes, Inclusion and exclusion criteria, Appraisal tool, method of analysis
Why is there a need for research synthesis
Reviews give a more complete picture (single study may provide conflicting information to the wider field and helps to see a large quantity of information in a distilled form) Helps clinical practice (e.g. if findings vary by subgroups or not) Helps guide policy making e.g. regarding legislation about treatments
Discuss risk and relative risk
Risk, as opposed to odds, is calculated as the number of patients in the group who achieve the stated end point divided by the total number of patients in the group. Risk ratio or relative risk is a ratio of two ''risks''.
What conditions would need to be met for a clinical trial to be conducted
Satisfactory information has been gathered on the quality of the nonclinical safety, the health authority/ethics committee approval is granted in the country where approval of the drug is sought
What type of research are systematic reviews and meta-analyses
Secondary research
What three things are focused on in the process of study quality appraisal
Selection bias, attrition bias, interviewer bias
What is a parallel design study
Subjects receive only one treatment and change in outcome response in one group of subjects receiving the treatment of interest is compared with that in another group receiving a different or control treatment
What is attrition bias
Systematic differences between groups in dropouts and withdrawals that alter the group composition
What is interviewer bias
Systematic differences in outcome assessment for e.g. expectancy effects due to unblinded assessment
What is selection bias
Systematic differences in the compositions of groups
What is the highest levl of evidence in the hierarchy of research designs
Systematic reviews of RCT's
Discuss how CIs are used to indicate the precision of the treatment effect
The 95% CI of an estimate (for example, of odds ratios or relative risks) will be the range within which we are 95% certain that the true population treatment effect will lie. The width of a confidence interval indicates the precision of the estimate. The wider the interval, the less the precision.
Discuss what happens if CI's overlap in a forest plot
The 95% confidence intervals of all the studies overlap 1. The 95% confidence intervals of the overall effect estimate also overlaps 1. So, there is no statistical significance either at the study level or the meta-analysis level.
In a meta-analysis what does the 95% CI represent
The 95% confidence intervals would contain the true underlying effect in 95% of the occasions if the study was repeated again and again
Discuss odds and OR
The odds for a group is defined as the number of patients in the group who achieve the stated end point divided by the number of patients who do not. The odds ratio, as the name implies, is a ratio of two odds. It is simply defined as the ratio of the odds of the treatment group to the odds of the control group.
Provide the definition for a meta-analysis
The use of statistical techniques in a systematic review to integrate the results of included studies
Define 95% confidence intervals
There is 95% chance that the true effect in the population will lie within the range. If the trial is repeated, 95% chance that the point estimate from the trial lies within the range
What is true if the 95% CI of all the studies overlap with 1 but the 95% CI of the overall effect estimate does not overlap 1
There is no statistical significance at the study level but there is at the meta-analysis level
Discuss ratios in relation to forest plots
They are relative measures where 1 indicates no effect. If one is included in the 95% CI then there is no significant difference at 5% significance levels. If one is not included, opposite is true (indicating benefit or harm depending on the context).
Provide some further advantages of systematic reviews according to Akobeng
They can be used to establish whether scientific findings are consis- tent and generalisable across populations, set- tings, and treatment variations, or whether findings vary significantly by particular sub- groups. Moreover, the explicit methods used in systematic reviews limit bias and, hopefully, will improve reliability and accuracy of conclusions.
Outline both systematic reviews and meta-analysis
They provide a summary of research with distinct but compatible approaches to research synthesis. Both methods involve the identification, analysis and synthesis of quantitative evidence from previous studies. They are used to inform researchers, clinicians and policy makers.
What is the PICO framework
Used to define eligibility criteria for studies included within the systematic review (Participants Interventions Comparisons Outcomes)
When is the correlation coefficient (r) used
Usually when most of the studies have continuous independent or dependent variables
Generally what is the rule regarding weight of a study and sample size within a forest plot
Usually, the larger the sample size within a study, the larger the percentage of weight given to that study
What are the components of a search strategy?
What are the components of a search strategy? • Run electronic searches in multiple bibliographic databases (e.g. Medline, Embase); check reference list of key articles • Use of wildcards such as * or & to find versions of the same word- (adolescen* will match to adolescence, adolescent, adolescents...) • Manual searching content of relevant journals • Contacting experts in the field • Previous reviews of the literature - follow the references listed • Dissertations Document each search - In order to replicate the search and for later updates
Provide the definition of a systematic review
a review of a clearly formulated question that uses systematic explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review
Describe the 3 components within the process of selecting studies to include
a) Breadth • Is the research question a narrow one or a more generic one? b) Quality of study • What quality standards are required from studies? c) Representativeness • Are the studies representative of the target population? i.e. have all studies been identified?
When is the Mean difference/Standardised Mean Difference (Cohen's d) used
d is usually used when most studies have an independent variable that is dichotomous (e.g. interventions vs control) and a dependent variable is continuous (e.g. mean 1 - mean 2)
Should heterogeneity tests for studies in a meta-analysis be significant
it is preferable to have non-significant findings for heterogeneity (so the studies are similar enough to compare). It is still possible to pool studies when significant heterogeneity exists but results should be treated with caution
Name the domains for different study questions and why are they important to consider
• Etiology [cohort, case-control] • Therapy [RCT] • Prognosis [cohort] • Harm [cohort, case-control] • Diagnosis [cross-sectional, case-control] • Economic [cost-effectiveness analysis, etc.] Important to consider as domains are usually addressed by different study design
Describe effect size
• Is a measure of the magnitude and direction of a relationship between variables • It standardizes findings across studies such that they are directly comparable even if they didn't use the same methodology or rating scales • There are many different effect size measures, each suited to different research questions
Provide a summary of systematic reviews as a method
• Location and synthesis of research • Organised, transparent, and replicable procedures. • Minimise error and bias • Follow a protocol: - Study objectives - Methods - Analysis - Discussion • Careful documentation of steps and procedures
List the advantages of systematic reviews
• Objective/transparent view of evidence • Enables resource allocation by health economists and policy makers • Collation of research across different settings enables the examination of the range of impact of the intervention • Power of the study enhanced, i.e. detection of small effects which are clinically significant • Indication for further research • Replicable • Resolves controversy between conflicting findings
What aspects of a study are assessed to determine its quality
• Overall study design • Attrition • Duration of follow-up • Types and measures of outcome (multiple informants) • Intervention quality control (random allocation, attrition bias, blinding)
List the disadvantages of systematic reviews
• Publication bias (Negative studies are less likely to be published, and hence less likely to be reviewed, than positive ones) • Magnification of study bias (If lots of studies suffer from the same bias, reviewing those studies will just compound the bias - effects may be overestimated) • Heterogeneity (Studies may have different characteristics - effects of participants, design and treatment effects maybe obscured) Subjectivity (Subjective decisions as to what to include, how to rate)
