OpenClinical logo

Reviews of DSS Evaluation Studies

Reviews of evaluation studies of Clinical Knowledge Management technologies and applications

Reference Short summary of study

Garg AX, Adhikari NK, McDonald H et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005 Mar 9;293(10):1223-38.

[PubMed]   [JAMA]

" Context Developers of health care software have attributed improvements in patient care to these applications. As with any health care intervention, such claims require confirmation in clinical trials. Objectives To review controlled trials assessing the effects of computerized clinical decision support systems (CDSSs) and to identify study characteristics predicting benefit. Data Sources We updated our earlier reviews by searching the MEDLINE, EMBASE, Cochrane Library, Inspec, and ISI databases and consulting reference lists through September 2004. Authors of 64 primary studies confirmed data or provided additional information. Study Selection We included randomized and nonrandomized controlled trials that evaluated the effect of a CDSS compared with care provided without a CDSS on practitioner performance or patient outcomes. Data Extraction Teams of 2 reviewers independently abstracted data on methods, setting, CDSS and patient characteristics, and outcomes. Data Synthesis One hundred studies met our inclusion criteria. The number and methodologic quality of studies improved over time. [...] Conclusions Many CDSSs improve practitioner performance. To date, the effects on patient outcomes remain understudied and, when studied, inconsistent. "

Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280:1339-46.

[PubMed]    []

Review of 68 controlled trials of CDSS (meeting specified criteria) on physician performance and patient outcomes.
Conclusion: "Published studies of CDSSs are increasing rapidly, and their quality is improving. The CDSSs can enhance clinical performance for drug dosing, preventive care, and other aspects of medical care, but not convincingly for diagnosis. The effects of CDSSs on patient outcomes have been insufficiently studied."
Johnston ME, Langton KB, Haynes RB, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. Ann Intern Med 1994; 120: 135-142

[PubMed]    []

Review of 28 controlled trials meeting predefined criteria (from 793 citations covering the period 1974-1994) on the effects of CDSSs on clinician performance and patient outcomes.
Conclusion: "Strong evidence suggests that some CDSSs can improve physician performance. Additional well-designed studies are needed to assess their effects and cost-effectiveness, especially on patient outcomes."
Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc. 1999 Mar-Apr;6(2):104-14.



Review of functionality and effectiveness of CDSS described in 25 studies published between 1992-1998. All systems provided patient-specific recommendations.
Conclusion (part): "Guideline adherence improved in 14 of 18 systems." ... "In the studies described here, different types of guidelines, different settings, and different system implementations make conclusions about the relationship between information management services and outcomes difficult."
Chuang JH, Hripcsak G, Jenders RA. Considering clustering: a methodological review of clinical decision support system studies. Proc AMIA Symp. 2000;:146-50.

[PubMed]    [AMIA]

Review of 24 CDSS evaluation studies (meeting inclusion criteria) to determine whether the design and analysis methods of cluster-based studies were adequately addressed.
Conclusion: "Investigators should publish estimates of the intracluster correlation coefficients and variance components in their reports to guide the planning of the future studies".

Estabrooks C, Goel V, Thiel E st al. Decision aids: are they worth it? A systematic review. J Health Serv Res Policy. 2001 Jul;6(3):170-82.

[PubMed]   []

" OBJECTIVES: The field of consumer decision support interventions has grown rapidly in the last 20 years, particularly since the increased formalization of evidence-based decision-making in health care. However, little is known as yet about the effect these decision aids have on relevant outcomes. To identify outcomes influenced by consumer decision aids (CDAs) and the particular effects of CDAs on those outcomes. METHOD: Published reports of randomized controlled trials and other designs in English identified from searches of Medline, Cancerlit, CINAHL, PsycINFO, Social Science Abstracts, the Cochrane Library and Current Contents, and from ancestry searches, hand searches and consultation with key informants. Studies were included if consumers were the decision-makers, decisions were real and not hypothetical, the intervention met a strict definition of a CDA and decisions involved treatment or screening. Data on setting, subjects, decision types, decision aid features and outcomes were extracted. The validity of each study included was evaluated by two team members. Summative ratings were calculated and categorized as high, medium and low. RESULTS: Over 500 titles were initially assessed; 96 study reports were obtained and screened for inclusion. Twenty-two reports representing 20 studies met the inclusion criteria. Eight studies with a low rating were excluded. Results were synthesized from the remaining 12 studies. Published studies of CDAs are increasing although still few in number. Outcomes such as knowledge and decision-making processes are influenced by CDAs but effects on treatment preferences, the actual decision or other outcomes have yet to be clearly established. CONCLUSIONS: Improved outcomes result from the use of CDA in some categories of outcome but insufficient evidence exists yet to support improved outcomes in all categories or to evaluate whether, given the considerable costs of many CDAs, funding for extensive ongoing development and testing can be sustained. "

Balas EA, Austin SM, Mitchell JA, Ewigman BG, Bopp KD, Brown GD. The clinical value of computerized information services. A review of 98 randomized clinical trials. Arch Fam Med. 1996 May;5(5):271-8.

[PubMed]    []

Objective: "To review all randomized clinical trials addressing the efficacy of clinical information systems and to determine the clinical settings, types of interventions, and effects studied."
Conclusion: "Randomized clinical trials confirm that four generic information interventions are active ingredients of computer systems and can make a significant difference in family medicine (physician and patient reminders, treatment planner, and patient education). To manage care and improve quality, primary care computer systems should incorporate these effective information services."
Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc. 1996 Nov-Dec;3(6):399-409.

[PubMed]    []

Meta-analysis of all 16 randomized, controlled trials of computer-based reminder systems in ambulatory settings found in the literature in the period 1966-1994.
Conclusion: "Evidence from randomized controlled studies supports the effectiveness of data-driven computer-based reminder systems to improve prevention services in the ambulatory care setting."
Walton R, Dovey S, Harvey E, Freemantle N. Computer support for determining drug dose: systematic review and meta-analysis. BMJ 1999 Apr 10;318(7189):984-90

[PubMed]    [BMJ]    [PubMedCentral]

Systematic review of 18 studies in hospital settings meeting inclusion criteria, period 1966-1996, where computers gave advice to clinicians on the most appropriate drug dose.
Conclusion: "This review suggests that using computers to determine the correct dose of certain drugs in acute hospital settings is beneficial. Computers may give doctors the confidence to use higher doses when necessary, adjusting the drug dose more accurately to individual patients. Further research is necessary to evaluate the benefits in general use."

Reviews of Neural Networks in healthcare

A review of evidence of health benefit from artificial neural networks in medical intervention. Lisboa PJ. Neural Netw 2002 Jan;15(1):11-39.

[PubMed]    []

" The purpose of this review is to assess the evidence of healthcare benefits involving the application of artificial neural networks to the clinical functions of diagnosis, prognosis and survival analysis, in the medical domains of oncology, critical care and cardiovascular medicine. The primary source of publications is PubMed listings under Randomised Controlled Trials and Clinical Trials. The rjle of neural networks is introduced within the context of advances in medical decision support arising from parallel developments in statistics and artificial intelligence. This is followed by a survey of published Randomised Controlled Trials and Clinical Trials, leading to recommendations for good practice in the design and evaluation of neural networks for use in medical intervention. "

Evaluating the literature on trials of CDSS

Kaplan B. Evaluating informatics applications--clinical decision support systems literature review. Int J Med Inf. 2001 Nov;64(1):15-37.

[PubMed]    [Int J Med Inf]

Review of the literature covering the evaluation of CDSS.
Discussion of the strengths and weaknesses of methods for evaluation, outcome measures, barriers to system use, issues and anomalies arising from a number of studies etc.
Conclusion: "A plurality of methodological approaches and research questions in evaluation is needed so as to broaden our understanding of clinical acceptance and use of informatics applications."
A.G. Randolph, R.B. Haynes, J.C. Wyatt et al. Users' guides to the medical literature: XVIII. How to use an article evaluating the clinical impact of a computer-based clinical decision support system. J. Am. Med. Assoc. 282 1 (1999), pp. 67-74.

[PubMed]    []



Meta-analyses of effectivenes of guidelines

Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations. Lancet. 1993 Nov 27;342(8883):1317-22.

[PubMed]    []

Meta-analysis of 59 published evaluations of clinical guidelines that met defined criteria.
Conclusion: "All but 4 of these studies detected significant improvements in the process of care after the introduction of guidelines and all but 2 of the 11 studies that assessed the outcome of care reported significant improvements. We conclude that explicit guidelines do improve clinical practice, when introduced in the context of rigorous evaluations. However, the size of the improvements in performance varied considerably."

Promoting the adoption of guidelines and DSS in clinical care

Sim I, Gorman P, Greenes RA et al. Clinical decision support systems for the practice of evidence-based medicine. J Am Med Inform Assoc. 2001 Nov-Dec;8(6):527-34.

[PubMed]    []

Objective: using the proceedings of the 2000 AMIA Spring Symposium, to investigate "the research and policy challenges for capturing research and practice-based evidence in machine-interpretable repositories, and to present recommendations for accelerating the development and adoption of clinical decision support systems for evidence-based medicine." Conclusion: "Although the promise of clinical decision support system-facilitated evidence-based medicine is strong, substantial work remains to be done to realize the potential benefits."
Bates DW. Using information systems to improve practice. Schweiz Med Wochenschr. 1999 Dec 11;129(49):1913-9.

[PubMed]    []

(See title of paper)
Davis DA, Taylor-Vaisey A. Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines. CMAJ. 1997 Aug 15;157(4):408-16.

[PubMed]    []

Review of strategies for the implementation of guidelines in practice.
Conclusion: "The evidence shows serious deficiencies in the adoption of CPGs in practice. Future implementation strategies must overcome this failure through an understanding of the forces and variables influencing practice and through the use of methods that are practice- and community-based rather than didactic."
Cabana MD, Rand CS, Powe NR, Wu AW, Wilson MH, Abboud PA, Rubin HR. Why don't physicians follow clinical practice guidelines? A framework for improvement. JAMA. 1999 Oct 20;282(15):1458-65.

[PubMed]    []

Review of 76 articles (period 1966-1998) describing barriers to adherence to clinical practice guidelines. 293 potential barriers were identified.
Conclusion: "Studies on improving physician guideline adherence may not be generalizable, since barriers in one setting may not be present in another. Our review offers a differential diagnosis for why physicians do not follow practice guidelines, as well as a rational approach toward improving guideline adherence and a framework for future research."
Langley C, Faulkner A, Watkins C, Gray S, Harvey I. Use of guidelines in primary care--practitioners' perspectives. Fam Pract 1998 Apr;15(2):105-11

[PubMed]    [Fam Pract]

"Concern about the inadequate take-up of guidelines in general practice has concentrated on problems arising from the process of their development and implementation. However, these perspectives fail to take account of the needs, attitudes and problems of GPs themselves. In this study we aimed to identify barriers to the use of guidelines and opportunities for tackling them, from the point of view of the GP, so that future guideline development and policy could be more sensitive to the needs of GPs in the environment in which they work."

Evaluation methods

Friedman, C. and Wyatt J. (1997) Evaluation methods in medical informatics. New York: Springer-Verlag.

[PubMed]    []

J. Wyatt and D. Spiegelhalter, Evaluating medical expert systems: what to test and how. Med. Inform. 15 (1990), pp. 205-217

[PubMed]    []

Clarke K, O'Moore R, Smeets R, Talmon J, Brender J, McNair P, Nykanen P, Grimson J, Barber B. A methodology for evaluation of knowledge-based systems in medicine. Artif Intell Med. 1994 Apr;6(2):107-21.

[PubMed]    []

"In this paper we describe an iterative, four-phased development evaluation cycle covering the following areas: (i) early prototype development, (ii) validity of the system, (iii) functionality of the system, and (iv) impact of the system."
Kaplan B. Evaluating informatics applications--some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inf 2001 Nov;64(1):39-56

[PubMed]    [Int J Med Inf]

"This paper critiques RCT and experimental evaluation approaches" which don't answer questions concerning whether or how systems will be used. The paper presents alternative approaches to evaluation "that investigate social, cultural, organizational, cognitive, and other contextual concerns".

Sailors RM, East TD, Wallace CJ et al. Testing and validation of computerized decision support systems. Proc AMIA Annu Fall Symp. 1996;:234-8.

[PubMed]    []

CDSS software design and testing method.
O'Connor AM, Stacey D, Rovner D et al. Decision aids for people facing health treatment or screening decisions (Cochrane Review). In: The Cochrane Library, Issue 2, 2002. Oxford: Update Software.



This paper provides: 1. An inventory of existing decision aids. 2. A systematic review of randomized controlled trials of decision aids designed to improve clinical decision making and outcomes. (24 randomised trials were selected for review.) A conclusion: "Our review demonstrated that decision aids can improve decision making by increasing the likelihood that people understand probable benefits and harms of options and participate actively in decision making."

Search this site

Privacy policy User agreement Copyright Feedback

Last modified:
© Copyright OpenClinical 2002-2011