Guideline Standardization
Appraisal of Guidelines, Research and Evaluation in Europe
keywords Main objective
clinical practice guidelines, quality, Health policy, evaluation, research

AGREE is an international collaboration of researchers and policy makers who seek to improve the quality and effectiveness of clinical practice guidelines by establishing a shared framework for their development, reporting and assessment.
Objectives are listed as follows:
  1. The creation of an appraisal instrument (AGREE) to assess the quality of clinical guidelines
  2. The development of standard recommendations for guideline developers
  3. A comparison of guideline development programmes.
  4. A content analysis of guidelines on asthma, diabetes and breast cancer
  5. An appraisal of individual recommendations.

Main results include:

  1. The AGREE Instrument (published September 2001): a framework for assessing the quality of clinical practice guidelines. Translations of the document are available in around a dozen languages. A training manual is also available.
  2. The AGREE electronic library for guideline developers: a "web-based resource bank ... structured around the AGREE appraisal criteria, and is intended to support dissemination of the AGREE instrument to guideline developers. The library includes information on the different ways in which guidelines are developed and documented throughout Europe; and provides examples, tools, and templates from members of the AGREE Collaboration which may be adapted by new guideline developers."

AGREE Collaboration [Cluzeau FA, Burgers JS, Brouwers M et al]. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003 Feb;12(1):18-23.

[PubMed]   [BMJ]

" Background: International interest in clinical practice guidelines has never been greater but many published guidelines do not meet the basic quality requirements. There have been renewed calls for validated criteria to assess the quality of guidelines. Objective: To develop and validate an international instrument for assessing the quality of the process and reporting of clinical practice guideline development. Methods: The instrument was developed through a multi-staged process of item generation, selection and scaling, field testing, and refinement procedures. 100 guidelines selected from 11 participating countries were evaluated independently by 194 appraisers with the instrument. Following refinement the instrument was further field tested on three guidelines per country by a new set of 70 appraisers. Results: The final version of the instrument contained 23 items grouped into six quality domains with a 4 point Likert scale to score each item (scope and purpose, stakeholder involvement, rigour of development, clarity and presentation, applicability, editorial independence). 95% of appraisers found the instrument useful for assessing guidelines. Reliability was acceptable for most domains (Cronbach’s alpha 0.64–0.88). Guidelines produced as part of an established guideline programme had significantly higher scores on editorial independence and, after the publication of a national policy, had significantly higher quality scores on rigour of development... Guidelines with technical documentation had higher scores on that domain ... Conclusions: This is the first time an appraisal instrument for clinical practice guidelines has been developed and tested internationally. The instrument is sensitive to differences in important aspects of guidelines and can be used consistently and easily by a wide range of professionals from different backgrounds. The adoption of common standards should improve the consistency and quality of the reporting of guideline development worldwide and provide a framework to encourage international comparison of clinical practice guidelines. "

Burgers JS, Grol R, Klazinga NS, Makela M, Zaat J; AGREE Collaboration. Towards evidence-based clinical practice: an international survey of 18 clinical guideline programs. Int J Qual Health Care. 2003 Feb;15(1):31-45.

[PubMed]   []

" OBJECTIVE: To describe systematically the structures and working methods of guideline programs. DESIGN: Descriptive survey using a questionnaire with 32 items based on a framework derived from the literature. Answers were tabulated and checked by participants. STUDY PARTICIPANTS: Key informants of 18 prominent guideline organizations in the United States, Canada, Australia, New Zealand, and nine European countries. MAIN OUTCOME MEASURES: History, aims, methodology, products and deliveries, implementation, evaluation, procedure for updating guidelines, and future plans. RESULTS: Most guideline programs were established to improve the quality and effectiveness of health care. Most use electronic databases to collect evidence and systematic reviews to analyze the evidence. Consensus procedures are used when evidence is lacking. All guidelines are reviewed before publication. Authorization is commonly used to endorse guidelines. All guidelines are furnished with tools for application and the Internet is widely used for dissemination. Implementation strategies vary among different organizations, with larger organizations leaving this to local organizations. Almost all have a quality assurance system for their programs. Half of the programs do not have formal update procedures. CONCLUSIONS: Principles of evidence-based medicine dominate current guideline programs. Recent programs are benefiting from the methodology created by long-standing programs. Differences are found in the emphasis on dissemination and implementation, probably due to differences in health care systems and political and cultural factors. International collaboration should be encouraged to improve guideline methodology and to globalize the collection and analysis of evidence needed for guideline development. "
Burgers JS, Cluzeau FA, Hanna SE, Hunt C, Grol R. Characteristics of high-quality guidelines: evaluation of 86 clinical guidelines developed in ten European countries and Canada. Int J Technol Assess Health Care. 2003 Winter;19(1):148-57.

[PubMed]   []

" OBJECTIVES: To identify predictors of high-quality clinical practice guidelines. METHODS: A total of 86 guidelines from 11 countries were assessed by four independent appraisers per guideline using the AGREE instrument (23 items). Six aspects of guideline development were considered to explain the variation in quality scores: care level (primary/secondary care), scope (diagnosis/treatment), type of guideline (new/update), year of publication, type of agency (governmental/professional), and whether the guideline was produced within a structured and coordinated program. RESULTS: Guidelines produced within a guideline program and by governmental agencies had higher scores than their counterparts. Differences in the applicability of the guidelines could not be explained by the variables studied. CONCLUSION: To ensure high quality, clinical guidelines should be produced within a structured and coordinated program. Professional organizations or specialist societies that aim to develop guidelines may adopt quality criteria from leading guideline agencies. "

Burgers JS, Fervers B, Haugh M et al. International assessment of the quality of clinical practice guidelines in oncology using the Appraisal of Guidelines and Research and Evaluation Instrument. J Clin Oncol. 2004 May 15;22(10):2000-7.

[PubMed]   [J Clin Oncol.]

" PURPOSE: To describe the quality of oncology guidelines developed in different countries. METHODS: The Appraisal of Guidelines and Research and Evaluation (AGREE) Instrument was used to assess the quality of 100 guidelines (including 32 oncology guidelines) from 13 countries. The criteria of the instrument are grouped into six quality domains: scope and purpose, stakeholder involvement, rigor of development, clarity and presentation, applicability, and editorial independence. RESULTS: Oncology guidelines had significantly higher scores on rigor of development than nononcology guidelines (42.2% v 29.4%; P = .02). In particular, systematic methods to search for evidence were more often used (P = .01); the methods for formulating the recommendations were more clearly described (P = .02); and health benefits, risks, and side effects were more often considered in formulating the recommendations (P = .03). Although the standardized scores for the other domains were not significantly different, the oncology guidelines had significantly higher scores for items measuring inclusion of all relevant professional groups (P = .05), consideration of patient views (P = .04), and presentation of different options (P = .05). Only three organizations producing oncology guidelines had standardized scores more than 60% for more than three domains. CONCLUSION: The quality of clinical practice guidelines (CPGs) is modest in general, but for certain domains, oncology guidelines seem to be of better quality than others. The experience of the organization may explain higher scores for some items. Research projects and training aimed at improving the quality of guidelines should be developed. The AGREE instrument could provide a basis for defining steps in a shared development approach to produce high-quality CPGs. "

start date end date location support
1998   EU (covering Denmark, England and Wales, Finland, France, Germany, Italy, the Netherlands, Scotland, Spain and Switzerland), Canada, New Zealand and the USA BIOMED-2 Programme of the European Union
contact Links
The AGREE collaboration is co-ordinated by the Health Care Evaluation Unit at St George's Hospital Medical School in London.

Health Care Evaluation Unit
St George’s Hospital Medical School
Cranmer Terrace
London SW17 ORE

Contact: Françoise Cluzeau
 bullet   AGREE collaboration  bullet   AGREE Instrument (framework for assessing the quality of clinical practice guidelines)  bullet   AGREE electronic library for guideline developers
Entry in directory: April 04 2004
Last main update: April 04 2004

Search this site


Privacy policy User agreement Copyright Feedback

Last modified:
© Copyright OpenClinical 2002-2011