Abstract

Objectives Overviews search for reviews rather than for primary studies. They might have the potential to support decision making within a shorter time frame by reducing production time. We aimed to summarize available instructions for authors intending to conduct overviews as well as the currently applied methodology of overviews in international Health Technology Assessment (HTA) agencies. Methods We identified 127 HTA agencies and scanned their websites for methodological handbooks as well as published overviews as HTA reports. Additionally, we contacted HTA agencies by e-mail to retrieve possible unidentified handbooks or other related sources. Results In total, eight HTA agencies providing methodological support were found. Thirteen HTA agencies were found to have produced overviews since 2007, but only six of them published more than four overviews. Overviews were mostly employed in HTA products related to rapid assessment. Additional searches for primary studies published after the last review are often mentioned in order to update results. Conclusions Although the interest in overviews is rising, little methodological guidance for the conduct of overviews is provided by HTA agencies. Overviews are of special interest in the context of rapid assessments to support policy-making within a short time frame. Therefore, empirical work on overviews needs to be extended. National strategies and experience should be disclosed and discussed. Copyright © 2013 John Wiley & Sons, Ltd.
Hide

Abstract

In the past two decades, a new form of scholarship has appeared in which researchers present an overview of previously conducted research syntheses on the same topic. In these efforts, research syntheses are the principal units of evidence. Overviews of reviews introduce unique problems that require unique solutions. This article describes what methods overviewers have developed or have adopted from other forms of scholarship. These methods concern how to (a) define the broader problem space of an overview, (b) conduct literature searches that specifically look for research syntheses, (c) address the overlap in evidence in related reviews, (d) evaluate the quality of both primary research and research syntheses, (e) integrate the outcomes of research syntheses, especially when they produce discordant results, (f) conduct a second-order meta-analysis, and (g) present findings. The limitations of overviews are also discussed, especially with regard to the age of the included evidence. (PsycINFO Database Record (c) 2012 APA, all rights reserved).
Hide

Abstract

CONTEXT AND OBJECTIVE: Healthcare decision-making is complex and should involve healthcare professionals, patients and the best level of evidence. The speed of information production creates barriers against keeping up to date. In this light, methodologists have proposed a new type of study: overviews of systematic reviews (OoRs). The aim here was to introduce and demonstrate the role of OoRs in information synthesis for healthcare professionals, managers, researchers and patients. DESIGN AND SETTING: Time-series study conducted at the Brazilian Cochrane Center, jointly with the Postgraduate Program on Internal Medicine and Therapeutics, Discipline of Emergency Medicine and Evidence-Based Medicine, Department of Medicine, Federal University of São Paulo. METHODS: To show the growth in the numbers of published papers that provide high-level evidence and thus demonstrate the importance of OoRs for synthesis and integration of information, three filters for study designs were applied to two databases. An equation for predicting the expected number of published papers was developed and applied. RESULTS: Over the present decade, the number of randomized controlled trials in Medline might reach 2,863,203 and the number of systematic reviews might reach 174,262. Nine OoRs and 15 OoRs protocols have been published in the Cochrane Library. CONCLUSIONS: With the exponential growth of published papers, as shown in this study, a new type of study directed especially towards healthcare decision-makers was proposed, named "overview of systematic reviews". This could reduce the uncertainties in decision-making and generate a new hierarchy in the pyramid of evidence.
Hide

Abstract

BACKGROUND: Systematic reviews are promoted as being important to inform decision-making. However, when presented with a set of reviews in a complex area, how easy is it to understand how and why they may differ from one another? METHODS: An analysis of eight reviews reporting evidence on effectiveness of community interventions to promote physical activity. We assessed review quality and investigated overlap of included studies, citation of relevant reviews, consistency in reporting, and reasons why specific studies may be excluded. RESULTS: There were 28 included studies. The majority (n = 22; 79%) were included only in one review. There was little cross-citation between reviews (n = 4/28 possible citations; 14%). Where studies appeared in multiple reviews, results were consistently reported except for complex studies with multiple publications. Review conclusions were similar. For most reviews (n = 6/8; 75%), we could explain why primary data were not included; this was usually due to the scope of the reviews. Most reviews tended to be narrow in focus, making it difficult to gain an understanding of the field as a whole. CONCLUSIONS: In areas where evaluating impact is known to be difficult, review findings often relate to uncertainty of data and methodologies, rather than providing substantive findings for policy and practice. Systematic 'maps' of research can help identify where existing research is robust enough for multiple in-depth syntheses and also show where new reviews are needed. To ensure quality and fidelity, review authors should systematically search for all publications from complex studies. Other relevant reviews should be searched for and cited to facilitate knowledge-building.
Hide

Abstract

BACKGROUND: Despite favourable results from past meta-analyses, some recent large trials have not found heart failure (HF) disease management programs to be beneficial. To explore reasons for this, we evaluated evidence from existing meta-analyses. METHODS: Systematic review incorporating meta-review was used. We selected meta-analyses of randomized controlled trials published after 1995 in English that examined the effects of HF disease management programs on key outcomes. Databases searched: MEDLINE, EMBASE, Cochrane Database of Systematic Reviews (CDSR), DARE, NHS EED, NHS HTA, Ageline, AMED, Scopus, Web of Science and CINAHL; cited references, experts and existing reviews were also searched. RESULTS: 15 meta-analyses were identified containing a mean of 18.5 randomized trials of HF interventions +/- 10.1 (range: 6 to 36). Overall quality of the meta-analyses was very mixed (Mean AMSTAR Score = 6.4 +/- 1.9; range 2-9). Reporting inadequacies were widespread around populations, intervention components, settings and characteristics, comparison, and comparator groups. Heterogeneity (statistical, clinical, and methodological) was not taken into account sufficiently when drawing conclusions from pooled analyses. CONCLUSIONS: Meta-analyses of heart failure disease management programs have promising findings but often fail to report key characteristics of populations, interventions, and comparisons. Existing reviews are of mixed quality and do not adequately take account of program complexity and heterogeneity.
Hide

Abstract

BACKGROUND: Hundreds of studies of maternity care interventions have been published, too many for most people involved in providing maternity care to identify and consider when making decisions. It became apparent that systematic reviews of individual studies were required to appraise, summarise and bring together existing studies in a single place. However, decision makers are increasingly faced by a plethora of such reviews and these are likely to be of variable quality and scope, with more than one review of important topics. Systematic reviews (or overviews) of reviews are a logical and appropriate next step, allowing the findings of separate reviews to be compared and contrasted, providing clinical decision makers with the evidence they need. METHODS: The methods used to identify and appraise published and unpublished reviews systematically, drawing on our experiences and good practice in the conduct and reporting of systematic reviews are described. The process of identifying and appraising all published reviews allows researchers to describe the quality of this evidence base, summarise and compare the review's conclusions and discuss the strength of these conclusions. RESULTS: Methodological challenges and possible solutions are described within the context of (i) sources, (ii) study selection, (iii) quality assessment (i.e. the extent of searching undertaken for the reviews, description of study selection and inclusion criteria, comparability of included studies, assessment of publication bias and assessment of heterogeneity), (iv) presentation of results, and (v) implications for practice and research. CONCLUSION: Conducting a systematic review of reviews highlights the usefulness of bringing together a summary of reviews in one place, where there is more than one review on an important topic. The methods described here should help clinicians to review and appraise published reviews systematically, and aid evidence-based clinical decision-making.
Hide

Abstract

Background: To date, the Cochrane Child Health Field has published 18 overviews of reviews in our journal, Evidence-based Child Health: A Cochrane Review Journal. In this article, we highlight some of the logistical and methodological challenges of producing such syntheses. As this is a new and evolving publication type, we hope that our experience will benefit others who engage in this process. Current Methods: We discuss the process we have developed to produce overviews of reviews relevant to our mandate, including identification of the research question, establishment of the author team, selection of outcomes and included SRs, and presentation of findings. Ongoing Development: We discuss the lessons we have learned, outstanding challenges for overview authors, and the limitations of overviews. Conclusions: Overviews of reviews are only as good as the SRs and primary studies on which they are based; gaps or lack of currency in this evidence will weaken the overview of reviews. Future directions in this work must address questions of bias and loss of information. Methods for overviews of reviews targeted for specific groups, such as children, need more elaboration. Copyright © 2011 John Wiley & Sons, Ltd.
Hide

Abstract

Background: An overview of reviews summarizes multiple intervention reviews addressing the effect of two or more interventions for a single health problem. We will conduct several overviews concerning the evidence for the effect of physiotherapy interventions on different chronic diseases commissioned by a Dutch policy maker. We were confronted with some practical and methodological issues when conducting our first overview addressing the effect of physiotherapy on fibromyalgia syndrome. Objectives: To describe the lessons learned from the challenges that we encountered while conducting overviews of reviews and to provide suggestions for improvement. Methods: The Cochrane Database of Systematic Reviews, MEDLINE, EMBASE and PEDRO were searched to find Cochrane and non-Cochrane reviews. We used the methods for conducting an overview of reviews as described in Chapter 22 of the Cochrane Handbook. The AMSTAR checklist was used to assess the quality of the included reviews, the algorithm of Jadad to choose between reviews with the same topic and GRADE to assess quality of evidence. Results: We included one Cochrane review and seven non-Cochrane reviews. Extracting data took a lot of time and essential data were missing in non-Cochrane reviews. Therefore GRADE could only be applied to one systematic review, the Cochrane review. Conclusion: When including non-Cochrane reviews it is important to only select systematic reviews that encompass essential and detailed information such as quality of included randomized controlled trials, outcome measures and point estimates. GRADE is an essential tool for policy makers to base their decisions on but can only be applied to systematic reviews that provide sufficient information such as Cochrane reviews. Additional issues and solutions for conducting overviews will be presented at the Colloquium.
Hide

Abstract

Bayesian networks (BNs) are tools for representing expert knowledge or evidence. They are especially useful for synthesising evidence or belief concerning a complex intervention, assessing the sensitivity of outcomes to different situations or contextual frameworks and framing decision problems that involve alternative types of intervention. Bayesian networks are useful extensions to logic maps when initiating a review or to facilitate synthesis and bridge the gap between evidence acquisition and decision-making. Formal elicitation techniques allow development of BNs on the basis of expert opinion. Such applications are useful alternatives to ?empty? reviews, which identify knowledge gaps but fail to support decision-making. Where review evidence exists, it can inform the development of a BN. We illustrate the construction of a BN using a motivating example that demonstrates how BNs can ensure coherence, transparently structure the problem addressed by a complex intervention and assess sensitivity to context, all of which are critical components of robust reviews of complex interventions. We suggest that BNs should be utilised to routinely synthesise reviews of complex interventions or empty reviews where decisions must be made despite poor evidence. Copyright © 2013 John Wiley & Sons, Ltd.
Hide

Abstract

Multiple meta-analyses may use similar search criteria and focus on the same topic of interest, but they may yield different or sometimes discordant results. The lack of statistical methods for synthesizing these findings makes it challenging to properly interpret the results from multiple meta-analyses, especially when their results are conflicting. In this paper, we first introduce a method to synthesize the meta-analytic results when multiple meta-analyses use the same type of summary effect estimates. When meta-analyses use different types of effect sizes, the meta-analysis results cannot be directly combined. We propose a two-step frequentist procedure to first convert the effect size estimates to the same metric and then summarize them with a weighted mean estimate. Our proposed method offers several advantages over existing methods by Hemming et al. (2012). First, different types of summary effect sizes are considered. Second, our method provides the same overall effect size as conducting a meta-analysis on all individual studies from multiple meta-analyses. We illustrate the application of the proposed methods in two examples and discuss their implications for the field of meta-analysis.
Hide

Abstract

Systematic reviews and meta-analyses usually synthesise evidence from studies reporting outcomes from particular interventions in specific diseases. For example, a meta-analysis of prophylactic antibiotics (intervention) in elective arterial reconstruction (disease) for rates of wound infection (outcome). However, because systematic reviews and meta-analyses are so widespread, a body of evidence often exists around specific intervention effects on particular outcomes over a range of diseases. So for example, a multitude of independent meta-analyses have evaluated rates of wound infection with and without the use of prophylactic antibiotics over multiple surgery types. A systematic review of systematic reviews is a means of synthesising evidence for the same intervention over multiple disease types. We propose a panoramic meta-analysis as a means of pooling effect estimates over systematic reviews of systematic reviews. We explore several methods ranging from a simple two-step approach, to a meta-regression or mixed effects approach, where variation between diseases are modelled as fixed covariate effects and between-study variation by random effects, and to a three-level hierarchical model in which exchangeability is assumed, which allows both a between-disease component of variance and a between-study (within disease) component of variance. In the surgery example, we pool 18 meta-analyses (each including between 4 and 26 studies) of prophylactic antibiotics reporting rates of wound infection from 18 different surgery sites to obtain a single pooled estimate of effect and estimates of between-disease, within-disease and within-study variability. Copyright (c) 2011 John Wiley & Sons, Ltd.
Hide

Abstract

BACKGROUND: Overviews of systematic reviews compile data from multiple systematic reviews (SRs) and are a new method of evidence synthesis. OBJECTIVES: To describe the methodological approaches in overviews of interventions. DESIGN: Descriptive study. METHODS: We searched 4 databases from 2000 to July 2011; we handsearched Evidence-based Child Health: A Cochrane Review Journal. We defined an overview as a study that: stated a clear objective; examined an intervention; used explicit methods to identify SRs; collected and synthesized outcome data from the SRs; and intended to include only SRs. We did not restrict inclusion by population characteristics (e.g., adult or children only). Two researchers independently screened studies and applied eligibility criteria. One researcher extracted data with verification by a second. We conducted a descriptive analysis. RESULTS: From 2,245 citations, 75 overviews were included. The number of overviews increased from 1 in 2000 to 14 in 2010. The interventions were pharmacological (n?=?20, 26.7%), non-pharmacological (n?=?26, 34.7%), or both (n?=?29, 38.7%). Inclusion criteria were clearly stated in 65 overviews. Thirty-three (44%) overviews searched at least 2 databases. The majority reported the years and databases searched (n?=?46, 61%), and provided key words (n?=?58, 77%). Thirty-nine (52%) overviews included Cochrane SRs only. Two reviewers independently screened and completed full text review in 29 overviews (39%). Methods of data extraction were reported in 45 (60%). Information on quality of individual studies was extracted from the original SRs in 27 (36%) overviews. Quality assessment of the SRs was performed in 28 (37%) overviews; at least 9 different tools were used. Quality of the body of evidence was assessed in 13 (17%) overviews. Most overviews provided a narrative or descriptive analysis of the included SRs. One overview conducted indirect analyses and the other conducted mixed treatment comparisons. Publication bias was discussed in 18 (24%) overviews. CONCLUSIONS: This study shows considerable variation in the methods used for overviews. There is a need for methodological rigor and consistency in overviews, as well as empirical evidence to support the methods employed.
Hide

Abstract

Objective: To evaluate the reporting and methodological quality of overviews of systematic reviews. Method: We developed an 18-item assessment tool for overviews of systematic reviews. We then performed a systematic search for such overviews using the terms ('overview' AND ('meta analys*' OR 'systematic review*')) OR 'umbrella review' in the title. We only included those overviews that were limited to systematic reviews or meta-analyses. Their methodological and reporting quality were assessed by two independent reviewers using the checklist, and differences were resolved by a third reviewer. Data analyses was conducted by SPSS version 15.0 for Windows. Results: We identified 41 overviews of systematic reviews whose mean total reporting score was 10.78 (SD 3.84) and methodological score 3.05 (SD 2.09). Some important items were not adequately reported: only 69% reported defined eligibility criteria, 76% reported search strategy, 49% reported the process of review selection, 44% reported the data collection process, 5% reported evaluating the reporting quality, 46% reported evaluating methodological quality, and 20% reported assessing the evidence level for each outcome. Conclusion: The reporting and methodological quality of overviews of systematic reviews was very poor, and there is still much room for improvement. A checklist for overviews of systematic reviews should be developed and used.
Hide

Abstract

OBJECTIVE: To examine published overviews of systematic reviews in terms of descriptive and methodological characteristics. STUDY DESIGN AND SETTING: MEDLINE, Database of Abstracts of Reviews of Effects, the Cochrane Database of Systematic Reviews, and several Health Technology Assessment databases were searched for overviews of reviews up to February 2012. We extracted data from the methods and results sections of the included overviews. These data were analyzed descriptively as frequencies or medians and interquartile ranges. RESULTS: We included 126 overviews of reviews. According to our sample, publication rates for overviews have risen in the last decade. The quality of the included reviews was systematically appraised in 64% of the overviews. The most commonly used assessment tools were the Overview Quality Assessment Questionnaire (26%), Assessment of Multiple Systematic Reviews (11%), and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (5%). Only three of 18 overviews restricted to Cochrane reviews in our sample performed a quality assessment. Strategies to deal with discordant reviews were reported in 5% of the overviews. Searches for additional primary studies were conducted in 5% of the overviews. CONCLUSION: Overviews of reviews often lack methodological rigor. Methodological standards and reporting guidelines for overviews are needed to improve the quality of this new publication type. Copyright © 2012 Elsevier Inc. All rights reserved.
Hide

Abstract

Background: There is now often more than one systematic review on the same or a similar question, which has led to the development of systematic reviews of systematic reviews (overviews). Overviews offer the opportunity of providing decision makers with a broader summary of evidence and contrasting the results of different reviews on the same question. Little is known about their characteristics so far. Objectives: We set out to examine a sample of overviews in terms of different descriptive and methodological characteristics, including which databases were searched and how the quality of the included systematic reviews was assessed. Methods: We searched MEDLINE, DARE, the CDSR and different HTA databases for overviews of systematic reviews. Two reviewers independently screened titles and abstracts of the retrieved citations for eligible publications. Data extraction was conducted by one reviewer and checked by a second. Results: A total of 99 overviews were eligible. The median publication date was 2008 (interquartile range [IQR]: 2005-2010). The overviews included a median of 15 reviews (IQR: 7 to 35). Typically, four electronic databases were searched for systematic reviews (IQR: 2 to 6). The quality of the included reviews was systematically appraised in a good half of the overviews (56/99 [57%]). To assess methodological quality, most overviews used the Oxman&Guyatt Index (24/99 [24%]), followed by AMSTAR (8/99 [8%]) and PRISMA/QUORUM (5/99 [5%]). Primary studies published after the search date of the most recent included review were included in 5 overviews [5%]. Conclusions: Overviews should use rigorous methods, if they ought to be a useful tool in health care decision making. The use of a quality assessment tool can help to determine if eligible reviews meet minimum quality requirements. In our sample only a small proportion of overviews used a validated quality assessment tool.
Hide

Abstract

Background: Overviews of (systematic) reviews, or umbrella reviews, are designed to compile evidence from multiple systematic reviews of interventions into one document and provide the reader with a quick overview relevant to a specific decision. Objectives: To evaluate reporting and methodological qualities of overviews of reviews. Methods: Searches were conducted using ('overview' AND ('meta analys*' OR 'systematic review*')) OR 'umbrella review' in title/abstract in PubMed, the Cochrane library, EMBASE, ISI Web of Knowledge. All searches were conducted in February 2010, and updated in September 2010. We included those overviews that only included systematic reviews or meta-analyses. We developed an 18-item assessment tool to assess the methodological and reporting qualities of overviews of systematic reviews. Two independent reviewers assessed the qualities with this checklist, and resolved differences with a third reviewer. Results: We found 41 overviews of systematic reviews, whose mean total reporting and methodological score was 10.78 (SD 3.84) of 18 items and 3.05 (SD 2.09) of 8 items. Some necessary items for the reporting and methodological qualities in the overview were not reported, for example 69% mentioned inclusion and exclusion criteria, 76% mentioned information sources, 49% mentioned review selection, 44% mentioned data collection, 7% mentioned reporting quality assessment, 46% mentioned methodological quality assessment, and 20% mentioned quality of evidence assessment. Conclusions: The reporting and methodological qualities of overviews of systematic reviews were very poor, and there is still much room for improvement.
Hide