Topic 5.1: Systematic reviews and meta-analysis
How to assess the quality of a systematic review?
Flawed systematic review -> Flawed results Using checklist like AMSTAR to rate the overall confidence in the results of the systematic review. JBI critical appraisl checklist
Deatils of steps in developing a systematic review
#1 Develop the question A systematic review critically evaluates primary research on a specific topic, it does not simply summarise the primary research on a specific topic. It is often challenging for researchers to formulate the 'right' question for their systematic review. This is because systematic review questions need to be FOCUSED. Relates to what you really need to know about your topic: Is answerable Is specific and focused Is relevant to your field Has not (recently) been answered by anyone else Your question will guide your search strategy and data analysis, so it's very important to figure it out early in the process. *using PICO framework to form a research question #2 Plan/Method Depending on the purpose of the systematic review, there are many things that need to be considered early in the process. Before starting, it's really important that you ensure the question has not already been effectively answered in a recent review, and that no reviews on the topic are planned or already in progress. A prospective systematic review may be referred to as a PROTOCOL. A protocol clearly documents what the reviewers intend to do in their systematic review. The registration of a protocol provides a mean of ascertaining that the screening, selection and reporting in a completed review are based on the pre-defined eligibility criteria and the methodological approach outlined at the outset, rather than influenced by study findings. Making protocols PUBLICLY AVAILABLE increases TRANSPARENCY and can REDUCE DUPLICATION of effort. This is usually done in a REGISTRY (such as Prospero) or a dedicated JOURNAL (such as Systematic Reviews) during the planning stages of the review. #3 Develop inclusion/Exclusion criteria An important characteristic of a systematic review is that the SEARCH is REPLICABLE: someone else should be able to FOLLOW the strategy and END UP WITH THE SAME (or very similar) articles. It's very easy to look at an article and decide that you want to include it in your study because the results look good or you've heard of the authors or any other factors. To eliminate these PERSONAL BIAS (as far as possible), it's crucial to develop clear inclusion and exclusion criteria BEFORE BEGINNING YOUR FINAL SEARCH. i) Language of publication (reviewer or co-reviewer can read/ use translator) ii) Year of publication (fast-paced, limit your search to relatively recent publications, if looking for topic that is under-researched or looking for changes over time, you need to include much older articles) iii) Humans or animals iv) Age group (decide to include/exclude a study using wide age group) iv) Context (pre-hospital, emergency room, surgical wards, schools, clinics, client's home, if relevant to exposure or outcome, you need to decide to includ eor exclude) v) Data type/Study type (RCT, single-case studies/grey literature such as perspectives from NGO) vi) Study design (may exclude study with high attrition rates (subject left the study half way), or there are questions around ethics) #4 Design search Once you have developed a focused question and determined your inclusion and exclusion criteria you need to create an effective and efficient search process to retrieve all relevant evidence from a range of sources. Searching is an iterative (adaptable/refinable) process. You will need to develop a search and continually review and refine it for constant improvement. The search strategy for a systematic review needs to be comprehensive in order to capture all the relevant studies. Your search terms need to be made up of subject headings, plus a wide range of keywords/phrases for each concept. #5 Test and Run Search It is important to carefully check your search strategy for errors and to test it to determine the effectiveness of the search terms you have chosen. A number of factors influence the recall or precision of a search strategy, including the use of relevant subject headings. #6 Document It is important to keep good records of your final searches in each database so that your search process is transparent and reproducible. There are various ways of doing this. One method is to save copies of your search histories within each database. To do this you generally need to set up a free personal account in each relevant database - look for a link at the top of the main search page. Alternatively, you may choose to create a spreadsheet or search diary to record the details of your searches. EndNote can also help you keep track of what you have done. #7 Screen/Apply Criteria When reviewing the final search results from your chosen databases (and other sources if relevant), multiple reviewers (you and your supervisors or co-reviewers) will make decisions on which articles to include and exclude based on the criteria specified in your protocol. The first stage of this is usually based on titles and abstracts, then a full text analysis follows before data extraction. i) Pre-screening: Record the numbers of results from each database or source recorded before screening commences ii) Title/abstract screening: Each reviewer will need to scan titles and abstracts to see if they match the criteria or have some value to the systematic review. This is done separately by each reviewer to ensure minimal bias. Results are then compared. iii) Full-text screening: Reviewers look through the full-text of included articles to fine-tune the final collection of articles that will contribute to the review. A PRISMA (Preferred Reporting Items for Systematic Reviews and Meta Analyses) diagram is used to show the steps involved in the review process, as well as recording the number of articles excluded at each stage. As each reviewer assesses the articles returned by the searches, they must adhere to the inclusion and exclusion criteria that were defined in the protocol. This step is usually performed independently by multiple reviewers to reduce bias. Reviewers then compare their results until agreement is reached. Sometimes an additional reviewer is needed at this stage if inclusion of any articles is particularly contentious有争议的. The articles remaining are the ones which will be evaluated and analysed. #8 Critical Appraisal of Included Studies Included studies need to be critically appraised to determine if the results are valid, reliable and generalisable. Tools such as the Critical Appraisal Programme (CASP) from Discovery 1.1 and 3.2 provide checklists to complete this appraisal. Review Identifying Bias and Critical Appraisal from Discovery 3.2 for a refresher about how systematic errors in study design can impact results. #9 Extract/Manage Data Once the final list of articles for inclusion is determined, it is time to extract any relevant data. This requires careful planning to ensure it is managed appropriately (and in line with any ethics requirements). Some considerations might include: -Only extract data you need to answer the review question -Use a consistent approach to summarising the data and apply this each study -Make clear any abbreviations -Carefully convert all data to the SAME UNITS -Name columns in a way that is meaningful to others -Keep content brief #10 Synthesis Synthesis is a stage in the systematic review process where extracted data are pooled together and evaluated. It is one of the most important stages of the systematic review process, as it determines the OUTCOMES of the review. There are two commonly accepted methods of synthesis in systematic reviews: -Quantitative data synthesis (usually with meta-analysis) -Qualitative data synthesis This division is NOT ABSOLUTE, and even if you are conducting a quantitative systematic review, aspects of qualitative data synthesis are relevant in explaining why studies have been excluded differ from the findings of the meta-analysis. #11 Writing In general, the writing process for a systematic review is similar to the process of writing any other kind of literature review. The writing process includes summarising the main points or steps followed in the review and as analysis of the findings, where the author makes connections and expresses their point of view (which can still have BIASES). However, biases in the results should be minimised by following the systematic process described in the steps above (click here for full details) Content: brief description of the process/document, Identify what points are important and how concepts are related Questions to ask: What concepts are important, How are they related, What roles to different variables displayed, What contributions, strengths or weaknesses are apparent? Point of view: report what is there, express your evidence-based evaluation of the material Language: simple and succinct, critical and insightful, clear and concise
Where to find systematic review?
-usually in peer-reviewed journals -can locate in MEDLINE, PubMed and Embase by limiting the results to 'systematic reviews' -leading resource for systematic review: Cochrane Reviews and protocol for Cochrane Reviews -database is updated regularly as Cochrane Reviews are published 'when ready' -reviews are also available as short (less than 5 minute) podcasts read by the author of the review -> stay up-to-date with new evidence -results of systematic reviews in a point-of-care format for health professionals is published in "Cochrane Clinical Answers" -> summarise the results of systematic reviews in a short question and answer format. -Podcast: The recommended dose (discusses big questions in health and explores the insights, evidence and ideas of extraordinary researchers, thinkers, writers and health professionals from around the globe.)
Steps involved in developing a systematic review
1. Develop question 2. Plan/Method 3. Develop inclusion and exclusion criteria 4. Design search 5. Test and run search 6. Document 7. Screen/ Apply criteria 8. Critical appraisal of included studies 9. Extract/ manage data 10. Synthesis 11. Writing
2 types of study design of high priority in the EBM hierachy
1. Systematic review 2. Meta analysis
Forest plot
Imagine a circle, everything inside it concerned a question, everything outside outside does not. In this circle, relevant data will be included. A search for relevant data will begin. Data can come from many sources, including data from a clinical trials. Imagine the shapes are data from different dataset. Dataset must be of the right shape to fit in the circle, i.e. fulfilling certain inclusion and exclusion criteria. E.g. research that has good quality answer to the defined question. If the research meets the criteria, more detailed information about the data could be collected or extracted. Information extracted can include how the research is done (method), who participates in the research including how many of them (subject), how they was paid for (funding sources), what happened (the outcomes). These information is judge against criteria in order to assess the quality of the research. The information collected are then combined using statistical method (data visualisation) to give an overall result from all of the data (e.g. forest plot/blobbogram) The area of inquiry defined by the question can be divided into YES/positive or NO/negative halves. The shorter the line, the more confident with the data it is telling us Longer line represents less focused & more scattered data. Shorter line represents more focused and bunched data. Imagine knowledge as light and ignorance as darkness. The more spread the focus of the light, the weaker it is and the less clear things are. If the light is focused and data is grouped more clearly, we can be more confident of what we see. The diamonds represents the combined results of all the data included. And bcaue these combined results uses data from more sources than one data set, it is considered more reliable and better evidence. The more data there is, the more confident we can be.
Meta analysis
Many systematic reviews pool the data from individual studies to increase the overall sample size. This data can then be analysed statistically in a PROCESS known as meta-analysis. During meta-analysis, data from different studies are WEIGHTED depending on their SAMPLE SIZE and relevant CRITERIA, and evaluated to determine the cumulative outcome. The results of a meta-analysis are usually presented in a forest plot as described in the video above. -workshop: may have multiple outcomes (from different studies)
Primary Sources
Report ORIGINAL research, ideas or scientific discoveries for the first time, or the results/findings/data from experiments or studies that the authors conducted themselves Frequently include Methods, Results, and Discussion sections, and should explain the research methodology used Are commonly found in academic journals and may be referred to as primary research, primary articles, journal articles, or research studies They could also be papers from conferences, dissertations, meetings or laboratory notebook data, technical reports, statistical data, interviews, surveys, fieldwork, correspondence, or patents.
Narrative review
Review articles, usually termed narrative reviews, summarise the evidence on a clinical topic. Narrative reviews are usually prepared by experts in the field and aim to provide a SUMMARY of the evidence on a topic. Narrative reviews can be useful to CONDENSE the available evidence on a topic if conducted with appropriate methodological rigor. However, it is common for authors to use INFORMAL, SUBJECTIVE METHODS to collect and interpret studies, and they may select studies that reinforce their personal views. This bias in narrative reviews means that the recommendations made may be inappropriate.
Secondary sources
Seek to ADD VALUE to the information reported in the primary literature Describe, interpret, analyse, evaluate or appraise the primary sources, the evidence provided or the significance of the research Literature reviews, systematic reviews (e.g. Cochrane reviews) and meta-analyses are examples of secondary sources Other examples of secondary sources can include letters-to-the-editor of scientific journals, editorials, commentaries, biographies传记, book chapters, treatises论文, or any index or bibliography参考书目 used to locate primary sources.
Systematic review
Systematic reviews are designed to overcome this risk of bias and deliver reliable findings. Systematic review is a type of SECONDARY literature source (Cochrane) A systematic review is a literature review which follows a transparent, clearly defined and reproducible process. A systematic review aims to remove the reviewer's biases as far as possible by outlining an INCLUSION/EXCLUSION criteria for the review. A high quality systematic review follows a number of STEPS to ensure transparency and the absence of bias.
Tertiary sources
These consist of information which is a compilation of primary and secondary sources in a specific subject area Materials in which the information from secondary sources has been condensed or put into a convenient, easy-to-read form, often to inform clinical decision making at the point of care Examples of tertiary sources include clinical practice guidelines, point-of-care guidelines (e.g. Therapeutic Guidelines), handbooks (e.g. Australian Medicines Handbook), manuals, almanacs年历, indexes指数, fact books, chronologies, directories, encyclopaedias, patient care sheets or point-of-care tools (e.g. UpToDate).
New direction for systematic reviews
1. Network meta analysis Systematic reviews usually focus on a single intervention that was investigated in randomised controlled trials (ie drug A versus placebo). There is a new meta-analysis method called network meta-analysis, which allows for both direct and indirect comparisons of interventions and enable researchers to simultaneously assess the effects of more than two interventions for the same condition. For example, you are interested in how drug A compares to drug B for treatment of diabetes, but there are no studies comparing these drugs directly, only trials of drug A versus placebo and drug B versus placebo. A network meta-analysis allows you to compare the effects of drug A and drug B. For PHR2012, we will not explore this type of meta-analysis but you may come across these in the medical literature and your future career. 2. Living systematic reviews Many systematic reviews are completed as a one-off publication, with the literature review conducted at a single point in time. Cochrane reviews are often REPEATED PERIODICALLY (eg every 5 years). In the AMSTAR2 checklist, a quality review should be completed within 24 months of completing the literature search. You can see how this lag time could MISS KEY NEW research that has been published since the literature search was conducted. To overcome this, some systematic reviews are now CONTINUALLY UPDATED to incorporate relevant new evidence as it becomes available. These are known as living systematic reviews. Cochrane has 5 living systematic reviews available, out of their database of thousands of systematic reviews. You will see these become more common as TECHNOLOGY to automate the update process is developed.