Evolution of a Knowledge Base for a Clinical Decision Support System Encoded in the Arden Syntax

Transcription

1 Evolution of a Knowledge Base for a Clinical Decision Support System Encoded in the Arden Syntax Robert A. Jenders, MD, MS; Hao Huang, MPhil; George Hripcsak, MD; Paul D. Clayton, PhD Department of Medical Informatics Columbia University, New York, New York Clinical decision support systems (CDSS) are being used increasingly in medical practice. Thus, longterm maintenance of the knowledge bases (KB) of such systems becomes important. To quantify changes that occur as a KB evolves, we studied the KB at the Columbia-Presbyterian Medical Center. This KB has a total of 229 Medical Logic Modules (MLMs) encoded in the Arden Syntax. Eliminating those never used in practice, we retrospectively analyzed 156 MLMs developed over 78 months. We noted 2020 distinct versions of these MLMs that included 5528 changed statements over time. These changes occurred primarily in the logic slot (38.7% of all changes), the action slot (17.8%), in queries (15.0%) and in the data slot exclusive of queries (12.4%). We conclude that long-term maintenance of a KB for a CDSS requires significant changes over time. We discuss the implications ofthese results for the design ofkb editorsfor the Arden Syntax. INTRODUCTION Knowledge Representation Clinical decision support systems (CDSS) are being used increasingly in medical practice. CDSSs have been employed effectively for a wide variety of purposes, including preventive health, quality assurance and computer-aided diagnosis [1]. Representational standards in medical computing, covering areas such as controlled terminologies and messaging, have arisen in part to accommodate communication and knowledge sharing among different organizations and among providers and others within an organization. Knowledge representation for CDSSs is no exception in this trend. In particular, the Arden Syntax for Medical Logic Modules (MLMs), developed in part at the Columbia-Presbyterian Medical Center (CPMC), has been promoted as an open standard for the procedural representation and sharing of medical knowledge [2]. MLMs serve as the representational unit of a knowledge base (KB) for a CDSS encoded in the Arden Syntax. Implemented using a clinical event monitor at CPMC, MLMs provide over 1000 clinical alerts and many research messages each month [3]. However, as KBs grow in size and complexity, longterm maintenance of such knowledge becomes paramount. The area of KB research known as knowledge refinement typically focuses on correction of KB errors after initial release of an expert system when it fails to perform as expected. Much of this work has been confined to nonmedical domains [4]. However, this emphasis on error detection is present even in some work on medical KBs, such as GARVAN-ES1, an expert system used in the care of persons with diabetes [5]. Despite this emphasis, long-term maintenance of a KB also requires creation of new portions of a KB as well as modification of extant portions [4]. Indeed, a previous experiment involving the update of 15 disease profiles in the QMRTm (First Databank, Portland, Oregon) KB demonstrated that a total of 16% of all KB entries were modified, including up to 95% of the entries in a disease profile [4]. Thus, as medical science changes, KB maintenance is required, and this in turn may consume significant time and effort. The CPMC Environment In order to promote expansion of the CPMC KB, we have created a knowledge acquisition tool, now called EzMLM, that permits clinical domain experts with no knowledge of the Arden Syntax or the CPMC clinical repository to add MLMs encoded in the Syntax to the KB. Testing of the first version of this tool, intended to allow the creation of MLMs for clinical research, was successful [6]. Other workers, including those at Link6ping University [7] and LDS Hospital [8] have created knowledge editors for Arden, and change in a KB over time has significant implication for these tools as well /91$ AMIA, Inc. 558

2 Previous experimental work using the Arden Syntax informed the development of EzMLM. In a study of a transfer of 7 MLMs from CPMC to LDS Hospital, workers noted that all of the output statements and over 40% of the logic statements had to be changed in order for the MLMs to function in the receiving institution [9]. Accordingly, we understood that we needed to design carefully the parts of the tool that treated these areas in order to accommodate the future possibility of use of the tool at other sites. In addition, in work done at our institution in 1993, changes in the initial 20 MLMs were examined over a time period of 2.5 years [10]. Now, as the CPMC KB grows, it is clear that more than just siteindependent modularity in the creation of MLMs must be provided. We also must support long-term maintenance of the knowledge base in an attempt to control the time and effort associated with such a task. Goals of the Analysis Therefore, in order to guide the development of this knowledge acquisition tool by demonstrating which modules within the tool required special emphasis, we undertook a retrospective analysis of the MLMs in the CPMC KB. In doing so, we sought to study the evolution of a CDSS KB by quantitatively characterizing changes that occur over time in the natural evolution of a KB. We perfomed the current study in part to expand the earlier analysis by including many more MLMs over a longer period of time. METHODS We extracted copies of all versions of all MLMs stored in the CPMC KB dating from the oldest MLM present (written in October, 1991) through March, All versions of MLMs are retained in the KB for archival purposes even if they are no longer used actively. For each MLM, we recorded the date of the first and last version. After the initial one, each version of a MLM represented at least one change in a statement from its immediately preceding version. We eliminated from the analysis those MLMs that had never been activated in the KB for use for clinical, research or administrative purposes. We then sought to characterize the types of changes that occurred over the lifetime of each MLM. To do this, we created PERL scripts that, for each MLM, used the Unix diff program to isolate changes between chronologically adjacent pairs of MLMs. We accomplished this by taking into account the standardized structure of a MLM. A MLM, equivalent to a single rule in a rule-based expert system, contains enough medical knowledge and data to make a single clinical decision [2]. It consists of three categories: maintenance, library and knowledge. Each category consists of a set of slots, each of which has one or more values. The first two categories address the history and purpose of a particular MLM and serve as documentation of the knowledge. The knowledge category contains the slots that enable triggering of the MLM (evoke slots and event definitions); querying the data base (data slot); calculating other data based on these queries (data slot); logically processing these data (logic slot); and directing an output message (action slot) to a particular destination (defined in the data slot) if warranted by the logical processing [2]. Taking advantage of this slot-value structure, we augmented the PERL script to identify where and how many changes occurred when each new version of a MLM was created. We defined a change as either a new statement or one or more alterations in an extant statement from one version of a MLM to the succeeding version. A statement may be a single slot value in the maintenance and library categories; a single database query; a single trigger; a single destination; a single output string; a single calculation; or a single if-then-else clause. These automatically detected changes were confirmed by manual inspection. We then classified each changed statement as belonging to one of these functional parts of a MLM: documentation; trigger; database query; other data statement; logic; urgency; and action. Because the version number slot changes by definition with each new version and because the date changes in nearly all of our MLM versions, we excluded these slots from the analysis. RESULTS We extracted a total of 229 MLMs from the KB, developed over the 78 months from October, 1991 through March, Of these, 75 are currently active, while another 62 were previously activated but are no longer used. An additional 19 were never activated for clinical, research or administrative purposes but served as precursors of MLMs that were activated. Thus, a total of 156 MLMs were included in the analysis. The other 73 MLMs were used to test a variety of ideas or database queries but either never 559

3 were or have not yet been promoted to active use in the CDSS. These 156 MLMs accounted for 2020 distinct versions over the study period, an average of 12.9 versions per MLM. The number of versions per MLM ranged from 2 to 127. Using the previously described process, we then analyzed these versions in order to quantify the various kinds of changes in MLMs as the KB evolved. These data are detailed in Table 1. A total of 5528 statements were changed in the 2020 MLM versions in the selected set of MLMs over the study period. This is an average of 2.7 changed statements per version and 35.4 changed statements per MLM over its lifetime. Table 1. Changes in 156 MLMs over 78 months. "Functional Unit" signifies a part of a MLM and is defined in the Methods. Functional Unit Changed Average % Total Statements Changes Changed per Statements MLM Logic Action Queries Non-query Data Destination Trigger Documentation Urgency TOTAL 5528 _ 100 The most common changes occurred in the logic slot of the MLMs. An average of 13.7 logic statements were changed in the typical MLM over its lifetime. This accounted for 38.7% of all changed statements in the KB. The next most common change involved the action slot, in which an average of 6.3 statements were changed in the typical MLM, accounting for 17.8% of all changed statements. Beyond this, the most frequently changed functional units of the MLM were the database queries that constituted 15.0% of all changes; the data slot exclusive of queries with 12.4% of all changes; the destination of the MLM output with 6.9% of all changes; the trigger slots with 4.6% of all changes; and the documentation categories with 4.5% of all changes. Urgency statements rarely changed. DISCUSSION The current analysis suggests that long-term maintenance of a KB for a CDSS encoded in the Arden Syntax involves a considerable number of changes as the KB evolves. In the CPMC database this included over 2000 distinct versions with over 5000 enumerated changed statements. This is consistent with work elsewhere in analyzing changes to the QMR KB, which also demonstrated that a significant number of changes were required in order to update the KB. It is interesting to note the relatively large number of MLMs, 31.8% of the total KB, that were effectively "dead ends" that did not lead to modules that were actively used in the CDSS. In consideration of development of knowledge acquisition and management tools such as EzMLM, it is informative to note which sections of the selected MLMs changed most frequently. In order of importance, these were the logic slot, action slot, database queries and the data slot exclusive of queries. Other components such as the destination of output messages changed less frequently. Interestingly, background documentation that illustrates its software history and its purpose did not change as much, nor did the definition of the triggers that cause execution of the MLMs. Thus, any KB editor, in order to serve as an effective tool for longterm management, must provide a facile mechanism for updating the logic, action and data slots of MLMs but may not need to be as well-developed in delineating the triggers, maintenance category or library category. These functional units in which change occurred most frequently also have implication for using the Arden Syntax for knowledge sharing among institutions. This was one of the key design goals of the Syntax [2]. However, previous work demonstrated that significant numbers of changes must occur in the data, logic and action slots in order to use MLMs written at one institution in the KB of another institution whose CDSS uses the Arden Syntax [9]. This challenge is not specific to the Arden Syntax and would attend any standard form of knowledge representation intended to permit knowledge sharing [6]. Thus, incorporating features in KB editors that facilitate these changes also has implications for knowledge transfer. Previous work at our institution looked at our initial 20 MLMs over approximately 2.5 years [10]. This showed that the MLM data slot had significantly 560

4 more changes than other slots (183 versus 99 in the logic slot and 97 in the action slot). Results of the current analysis, by contrast, show relatively more changes in the logic slot than in the data slot (which is a combination of queries and non-query data). One possible explanation for this change over time may be a change in our environment. During the earlier study, a variety of database systems, including relational, VSAM and IMS systems, were used. Now, nearly all CPMC data reside in a relational database. This concentration may have reduced the need for changes to database references within MLMs. We used the current analysis to inform development of EzMLM, our knowledge acquisition tool. For example, by noting the most common values and changes involved in database queries, we altered the tool so that these common elements could be inserted quickly and easily into new MLMs instead of constructing queries with these elements from scratch. Similarly, we modified the logic part of the tool in order to make creation of the most common logical constructs and changes very easy. Now, this study does have some limitations. The key limitation is the relatively small sample size of MLMs in the KB. This is partly the result of limitations at our campus in the number of writers of MLMs, a problem that EzMLM was designed to overcome. Another explanation is the absence of a provider order entry tool at our institution, and this limits the kinds of MLMs that we may implement. However, the KB has been used actively for over 6 years and contains MLMs addressing a wide variety of topics. Thus, though small, the CPMC KB likely is similar to other CDSS KBs. Another limitation may be the relatively high percentage of MLMs in the KB that we excluded from the analysis. However, these never were used actively for clinical, research or administrative purposes and did not lead directly to MLMs that were used for such purposes. As a result, excluding them allowed us to focus the analysis on the core of the KB that is actively used and updated. FUTURE WORK With these results in mind, we will continue development of EzMLM as a knowledge acquisition and maintenance tool for KBs encoded in the Arden Syntax. Our current work centers on converting the initial version to a World-Wide-Web-compliant model that uses HTML forms and common gateway interfaces (CGIs) to capture clinical knowledge and create MLMs. We will use the results of the current analysis to direct our efforts in implementing EzMLM in a modular way that facilitates creation of MLMs for multiple institutions. SUMMARY As CDSSs are increasingly used in medical practice, long-term maintenance of the KB for such systems becomes important. In order to quantify the changes that occur in the evolution of such a KB, we retrospectively studied the changes that occurred in the CPMC KB during the 78 months since its inception. This KB is encoded as MLMs written in the Arden Syntax. Selecting 156 of a total of 229 MLMs after eliminating those never used in actual CDSS operation, we extracted 2020 versions of these MLMs from the KB. In comparing successive versions of each MLM, we discovered a total of 5528 changed statements. The largest share of changes occurred in the logic slot, and these accounted for 38.7% of the changes. Changes in the action slot (17.8%), data queries (15.0%) and non-query data slot (12.4%) also were important. These findings have implications for the design of knowledge editors that use the Arden Syntax, and we are using these data to guide an implementation of one such editor at CPMC, EzMLM. Acknowledgments We are indebted to Stephen B. Johnson, James J. Cimino, Eric H. Sherman and others whose efforts were instrumental in the creation of the CPMC decision support system. This work was supported in part by continuing generous assistance from The New York and Presbyterian Hospital. References 1. Johnston ME, Langton KB, Haynes BR, Mathieu A. Effects of computer-based clinical decision support systems on clinician performance and patient outcome. Ann Intern Med 1994;120: Hripcsak G, Clayton PD, Pryor TA, Haug P, Wigertz OB, Van der lei J. The Arden Syntax for medical logic modules. In Miller RA, editor. Proc of the Fourteenth Annual Symposium on Computer Applications in Medical Care. New York: IEEE Computer Press, 1990;

5 3. Jenders RA, Hripcsak G, Sideli RV, DuMouchel W. Zhang H, Cimino JJ, Johnson SB, Sherman EH, Clayton PD. Medical decision support: experience with implementing the Arden Syntax at the Columbia-Presbyterian Medical Center. In Gardner RM, ed. Proc of the Nineteenth Annual Symposium on Computer Applications in Medical Care. Philadelphia: Hanley & Belfus, 1995; Giuse DA, Giuse NB, Miller RA. Evaluation of long-term maintenance of a large medical knowledge base. JAMIA 1995;2: Compton P, Jansen R. Diabetes expert systems: planning for long term use. Horm Metab Res Suppl 1990;24: Jenders RA, Dasgupta B. Assessment of a knowledge-acquisition tool for writing medical logic modules in the Arden Syntax. Proc AMIA Annu Fall Symp 1996; Gao X, Shahsavar N, Arkad K, Ahlfeldt H. Hripcsak G, Wigertz 0. Design and functions of medical knowledge editors for the Arden Syntax. In Lun KC et al. Eds. Proc of the Seventh World Congress on Medical Informatics (MEDINF092). North-Holland: Elsevier, 1992; Sailors RM. MLM Builder: an integrated suite for development and maintenance of Arden Syntax medical logic modules. Proc AMIA Annu Fall Symp 1997; Pryor TA, Hripcsak G. Sharing MLM's: An experiment between Columbia-Presbyterian and LDS Hospital. In Safran C, ed. Proc of the Seventeenth Annual Symposium on Computer Applications in Medical Care. New York: McGraw-Hill, 1993; Hripcsak G, Johnson SB, Clayton PD. Desperately seeking data: knowledge base-database links. In: Safran C, editor. Proc of the Seventeenth Annual Symposium on Computer Applications in Medical Care. New York: McGraw-Hill, Inc., 1994;