Descriptor and Citation Retrieval in the Medical Behavioral Sciences Literature: Retrieval Overlaps and Novelty Distribution*

Transcription

1 Descriptor and Citation Retrieval in the Medical Behavioral Sciences Literature: Retrieval Overlaps and Novelty Distribution* Katherine W. McCain College of Information Studies, Drexel University, Philadelphia, PA Search results for nine topics in the Medical Behavioral Sciences are reanalyzed to compare the overall performance of descriptor and citation search strategies in identifying relevant and novel documents. Overlap percentages between an aggregate descriptor-based database (MEDLINE, EXCERPTA MEDICA, PSYCINFO) and an aggregate citation-based database (SCI- SEARCH, SOCIAL SCISEARCH) ranged from 1% to 26%, with a median overlap of 6% relevant retrievals found using both search strategies. For seven topics in which both descriptor and citation strategies produced reasonably substantial retrievals, two patterns of search performance and novelty distribution were observed: 1) Where descriptor and citation retrieval showed little overlap, novelty retrieval percentages differed by 17-23% between the two strategies; 2) Topics with a relatively high percentage retrieval overlap showed little difference (l-4%) in descriptor and citation novelty retrieval percentages. These results reflect the varying partial congruence of two literature networks and represent two different types of subject relevance. Introduction Cited references and thesaurus terms are generally considered to be complementary approaches to the identification and retrieval of relevant documents in a bibliographic search. Experienced online searchers include references to key older works in their search strategies with the expectation that citation searching will add unique relevant documents to the retrieval. The assumption is that the choice of references to include (by the author) and the choice of controlled vocabulary terms representing subject content (by the indexer) tap two different kinds of subject relevance relationships [ 11. In the case of new interdisciplinary areas, citation retrieval may be much more effective than thesaurus terms [2]. There are, however, only a few published studies specifically comparing the results of descriptor retrieval and citation retrieval for the same set of queries (as *This research was supported in part by NIH Contract NOl-LM from the National Library of Medicine by John Wiley & Sons, Inc. opposed to general coverage or overlap studies with undifferentiated search strategies). Pao and Fu[3] report that 12% of a 276 document set on sulfur dioxide air pollution were retrievable both from MEDLINE (the use of MeSH terms for MEDLINE retrieval is implied) and by citations. Pao [4] examined the results of 10 searches (on the topic of enteral hyperalimentation) using both descriptors and citations as retrieval tools in a proprietary database. In all 10 queries, citation searching produced fewer relevant retrievals than descriptor searching, recall and precision varied widely across the searches, and the retrieval overlap between the two techniques was reported as low. In a very different context, White et al. [5] report a low overlap between 1) source articles linked to cocited document clusters (in Social Science Citation Index) and 2) articles retrieved from MEDLINE, BIOSIS PREVIEWS, EXCERPTA MEDICA or PSYCINFO. In this set of experiments, subject retrieval used frequently occurring descriptors assigned to those citing source articles which were included in the latter four databases. In a set of articles from the circulation literature which were indexed in Science Citation Index and in the MEDLARS system, Rapp [6] found little overlap between document sets clustered by cocitation and by MeSH terms. These studies focus on single comparisons of citation access with a specific controlled vocabulary. It may also be of interest to compare citation and descriptor retrievals in a more generalized situation, to remove at least some of the system-specific effects of vocabulary choice, indexing practices, and document coverage policies in the various descriptor-oriented databases. This should provide a more stringent test for the ability of citation retrieval to provide unique relevant documents. In addition, it may suggest aspects of the subject field, search topic and key citations which potentially influence retrieval effectiveness. Retrieval studies generally focus on the standard measures of system performance: recall (the percentage of known Received October 14, 1986; revised December 19, 1986; accepted January 12, JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE. 40(2): , 1989 CCC /89/ $04.00

2 relevant documents retrieved) and precision (the percentage of the retrieval set judged relevant to the query). It has been argued that the provision of relevant documents previously unknown to the user could be considered a higher goal, since a retrieval consisting entirely of relevant documents known to the user would not (in most cases) meet his or her information needs. Novelty, the percentage of relevant retrievals judged novel (necessarily by the user posing the query) has been proposed by Lancaster [7] as an additional useful measure of system performance but has not been used in these nor (to the author s knowledge) in other studies comparing retrieval strategies. Methods Test retrievals on topics in the Medical Behavioral Sciences (MBS) literature were conducted as part of a broad study of collection and indexing practices at the National Library of Medicine [8]. Eleven search topics (Table 1) were provided by experienced biomedical researchers. These experts also suggested relevant older contributions likely to be cited in more recent work. Table 2 gives the number of key citations provided by each subject expert and the range of publication dates. The inclusion of articles known to be by the author or co-workers is also indicated. Each topic was searched in MEDLINE, EXCERPTA MEDICA, PSYCINFO, SCISEARCH and SOCIAL SCISEARCH using three separate search strategies: MED- LINE, EXCERPTA MEDICA, PSYCINFO were searched using parallel natural language and descriptor strategies; natural language and citation searches (retrieving all source articles citing older relevant contributions) were conducted in SCISEARCH and SOCIAL SCISEARCH. All searches were performed on DIALOG. Search results were evaluated for relevance and novelty by the same researchers who posed the original queries. Detailed results of these test retrievals are reported elsewhere [9,10]. For this article, retrieval data were reanalyzed to focus directly on the generalized use of descriptors and key documents to produce relevant and novel retrievals, avoiding considerations of specific vocabularies such as MeSH, PSYCINFO Thesaurus terms or EXCERPTA MEDICA descriptors assigned descriptor codes. (All search strategies are included in Ref. 9, Appendix A.) For each topic, system-specific relevant retrievals were identified as 1) retrieved only by descriptor search in MEDLINE, PSYCINFO, and/or EXCERPTA MEDICA, 2) retrieved only by citation search in SCI- SEARCH and/or SOCIAL SCISEARCH, or 3) retrieved by both strategies.* The novelty judgment provided by the expert for each of these retrievals was also recorded. The aggregation of MEDLINE, PSYCINFO and EX- CERPTA MEDICA retrievals creates, in effect, a very large, multi-lingual database with a very broad range of document sources. The combination of SCISEARCH and SOCIAL SCISEARCH retrievals broadens the coverage and thus the database of source documents searchable on DIALOG but adds no new language capabilities nor any new document rypes.+ *For each topic, additional relevant records were retrieved only by a natural language search strategy. These retrievals are not considered in this report. Ideally, comparisons should be done 1) on a single database providing both descriptor and citation access [4] or 2) across a matched set of docu- ments indexed in both databases [6]. In this reanalysis of search results, the identification of document records occurring in both the SCIiSSCI and MEDLINElEMiPSYCINFO joint files was essentially impossible. These joint files should provide a more closely matched source coverage than any one descriptor-based database would have with either SC1 or SSCI. TABLE 1. Search topics. 1. WEIGHT LOSS and other health-related programs in pre-existing groups. 2. Rehabilitation and therapy of APHASIA following stroke. 3. The neuropsychiatric aspects of systemic LUPUS erythematosus. 4. NERVIOS and other (stress-related) culture-bound syndromes. 5. Intervention to prevent parental neglect in the case of HIGH-RISK INFANTS. 6. INTROSPECTION (self-monitoring) and the reporting of physical symptoms. 7. The classical CONDITIONING of drug effects. 8. LEARNED HELPLESSNESS [the reactions of animals and humans to unavoidable stress]. 9. Interpersonal PROBLEM-SOLVING. 10. TIMING AND RHYTHM in normal speech and normal speech development. 11. Language or cognitive dysfunction associated with ALZHEIMER S Disease. TABLE 2. Characteristics of key references for citation retrieval. Topic # Key Refs. Pub. Dates Includes Own Work? APHASIA LUPUS NERVIOS HIGH-RISK INFANTS CONDITIONING LEARNED HELPLESSNESS PROBLEM-SOLVING TIMING AND RHYTHM ALZHEIMER S No Yes Yes No Yes Yes Yes Yes No JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-March

3 Results Descriptor retrieval and citation retrieval for nine topics are compared in Table 3. (Less than 25 relevant documents were retrieved for two topics. These search results are not considered further in this paper.) For each of the nine topics, unique and overlap percentages are calculated based on the total system-specific relevant retrievals by both approaches (far right column). For the topic HIGH-RISK INFANTS, 45 relevant document records were retrieved using either descriptor or citation search strategies, or both. 34 (76%) were retrieved only by a descriptor search in one of the three databases indexing with descriptors (MEDLJNE, PSYCINFO, EXCERPTA MEDICA), 10 (22%) retrieved only by citation searching of older references provided by the researcher, and one record (2%) was found by both strategies. The percentage of novel relevant documents retrieved by each search approach is shown in Table 4 for the seven topics having a reasonably substantial citation retrieval. The topics are listed in the same order as in Table 3. The denominator in each novelty ratio is the sum of the unique retrievals and the overlap retrievals for that strategy (in Table 3). In HIGH-RISK lnfants, for instance, descriptor retrieval provided 34 unique relevant documents and 1 overlapping document. Nineteen of these were judged as novel by the subject expert posing the query, for a descriptor-retrieval novelty percentage of 54%. Across both strategies, 51% of the relevant retrievals (23 of the 45 system-specific relevant documents) were judged as novel. Discussion Overlap It is evident from the data in Table 3 that both the descriptor and the citation approach to retrieval are capable of producing a reasonably substantial set of relevant documents and that there is likely to be relatively little overlap between the two sets. This confirms both the observations by White and Pao and the experience of online searchers. Across the nine-topic set of system-specific retrievals TABLE 3. Overlaps between descriptor retrieval and citation retrieval of relevant documents across nine topics. Descriptor-only Citation-only Topic Overlapping Total NERVIOS 28 (88%) 1 (03%) 3 (09%) 32 ALZHEIMER S 88 (92%) 1 (01%) 7 (07%) 96 HIGH-RISK INFANTS 34 (76%) 1 (02%) 10 (22%) 45 TIMING & RHYTHM 60 (69%) 2 (02%) 25 (29%) 87 APHASIA 42 (48%) I (08%) 39 (44%) 88 PROBLEM-SOLVING 41 (69%) 6 (10%) 12 (20%) 59 LEARNED HELPLESSNESS 47 (29%) 23 (14%) 94 (57%) 164 CONDITIONING 40 (51%) 18 (23%) 20 (26%) 78 LUPUS 20 (34%) 15 (26%) 23 (40%) 58 OVERALL 400 (57%) 74 (10%) 233 (33%) 707 MEDIAN PERCENTAGE OVERLAP = (08%) TABLE 4. Percentage of novel relevant retrievals in each search category across seven topics. Topic Novel/All Descriptor Novel/All Citation Total Novel Percentage of Total (Table 3) HIGH-RISK INFANTS 19/35 (54%) 4/11 (36%) 23 51% TIMING & RHYTHM 32/62 (52%) 19/27 (70%) 50 57% APHASIA 35/49 (71%) 22/46 (48%) 55 62% PROBLEM-SOLVING 32/47 (68%) 4/18 (22%) 35 59% LEARNED HELPLESSNESS 43/70 (61%) 67/117 (57%) % CONDITIONING 39/58 (67%) 24/38 (63%) 50 64% LUPUS 29/35 (83%) 31/38 (82%) 47 81% OVERALL 229/356 (64%) 171/295 (58%) % 112 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-March 1989

4 (descriptor plus citation), citation searching yielded about one-third of the documents, with an average overlap of 10% and a median overlap of 8%. Ceteris paribus, the use of judiciously chosen older key references in citation searching is likely to improve the recall of a search. Citation retrieval was not equally successful for all topics, however, and there is a large variation in performance within the nine topics. The results in Table 3 are arranged in order of increasing percentage overlap between descriptor and citation retrievals. The first two topics, ALZHEIMER S and NERVIOS illustrate the worst case outcome for citation retrieval. In these two cases, citation searching produced only 4 and 8 documents, respectively-approximately 10% of the entire set of systemspecific relevant retrievals. The NERVIOS search demonstrates the effect of the lag between publication and citation. Searches in the study were conducted over the years (or early 1983). Since this is essentially the same time span as that of the key references provided (Table 2), most had little opportunity to receive citations in the literature. In both cases, the citation networks linking related research may not be well formed-medical anthropology ( NERVIOS ) being a relatively new specialty area and the subject matter of the ALZHEIMER S search being markedly broad. In the remaining seven topics, from approximately l/4 to 2/3 of the retrievals were provided via citation searching. LEARNED HELPLESSNESS may be an illustration of a best case performance of citation retrieval. This research topic has been well established for some years, the researcher is a prominent and knowledgeable contributor to this literature, and he is highly cited for his research. Citation retrieval netted over 70% of the system-specific relevant citations, with 14% of these also retrieved via controlled vocabulary terms in one of the three descriptorbased databases. The lower proportion of documents retrieved only by descriptor may also reflect the relatively recent introduction of the MeSH term helplessness, learned and the small number of documents indexed with this term over the search period ( ). CONDITIONING and APHASIA show the highest percentage overlap- approximately l/4 of the documents were retrieved by both citation searching and by a descriptor search in at least one of the three subject-oriented databases. They differ in the relative unique contribution of each retrieval strategy. In particular, descriptor retrieval in CONDITIONING was enhanced by the availability of specific drug names in the various thesauri, as well as relatively useful terms representing classical conditioning (see Ref. 9, Appendix A). Novelty Both approaches to retrieval are capable of identifying substantial numbers of relevant documents with which the user was not previously familiar. Across the seven topics listed in Table 4, about 63% of the relevant retrievals were judged novel. This may be a more satisfactory level of performance than a similar precision or recall percentage. As a measure of system performance, novelty should be as much an indicator of the user s knowledge state as of breadth of coverage, indexing quality, or appropriateness of search strategy. That is, given good recall and precision, a search of a topic of long standing and intense interest to the researcher is likely to produce a lower percentage of novel relevant documents than is a search of a topic peripheral to his or her current interests. (Examples of the latter might include searching in support of a grant proposal in a new area or in support of teaching an infrequently offered course.) Indeed, a 100% novel retrieval is probably not desirable, since the lack of known relevant documents in the retrieval set would give both the searcher and the user little confidence in the search results. The topics in Table 4 are arranged in order of increasing percentage retrieval overlap (Table 3). Only seven sets of judgments are available- a small number upon which to speculate concerning performance patterns. It is intriguing, however, to note that topics in which descriptor and citation retrieval showed little overlap (less than lo%, Table 3) also display a marked difference in novelty percentage (17-23%) between these two retrieval approaches. Conversely, topics with a relatively high percentage overlap show little difference (l-4%) between the percentage of novel retrievals by descriptor and by citation. Literature Networks A higher overlap between descriptor and citation retrievals is likely to be achieved in searching those research topics which are both reasonably easy to define using existing controlled vocabularies and in which key works are broadly recognized and cited. The first condition requires that indexed documents are about well established research topics with readily recognized concepts and relatively standardized terminology. Thesauri are likely to be up-to-date and indexers better able to perform consistently in recognizing and assigning vocabulary terms. The second condition requires that at least a few documents be key contributions to the research topics (or generally descriptive of it, in the case of review articles), that the importance of these articles be generally recognized by researchers, that the norms of scholarship require citation of these key contributions, and lastly that sufficient time pass after publication of the key articles to generate a body of citing work. The two literatures, one linked by recognized subject relationships and the other by intellectual indebtedness, are then fairly congruent. Under these conditions, the likelihood of a novel document being linked to either literature network may be approximately the same and the degree of novelty achieved in the search strongly representative of the knowledge state of the user. A low overlap, coupled with high unique contributions on both sides, may represent two essentially noncongruent but equally relevant literatures-one well-defined by assignment of descriptors but in which authors do not cite a common background literature and the other linked JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-March

5 by citations (possibly to basic theoretical or methodological works) but which do not share an easily recognized terminology or subject matter. The preliminary results in Table 4 suggest that this noncongruence may be reflected in markedly differing novelty retrieval performance in the two literature networks. The source of this discrepancy is not clear-it may derive from the user s longterm focus on one portion of the total relevant literature (i.e. human vs. animal research) or recent extension of research interest to more peripheral areas only partially linked by citation. Retrieval sets exhibiting one of these two outcomes, good performance by both descriptor indexing/retrieval and citation creation/retrieval with either high overlap or low overlap, should be useful test beds for analysis of the complementary roles of terminology and citation in representing and defining subject literatures. How much of a descriptor-defined subject literature could one reasonably expect to capture, using a wellselected set of key papers, or key authors? Do cited works serve different functions or occur in different contexts in overlapping and nonoverlapping source papers? Do nonoverlapping descriptor-linked and citationlinked document sets represent separate coherent lines of research (only one of which is familiar to the citationprovider), or does one (or both) consist of scattered works brought together by broad general subject terms or widely-used research methods? What is the effect of the user s knowledge state and scholarly style on the ability of both descriptor and citation-retrieval to identify novel documents? Do novel or known descriptor-retrieved and citationretrieved documents meet different information needs of the user posing the query? Summary Both controlled vocabulary terms and key references will identify sets of documents subsequently judged relevant by the user posing the query. The overlap between these two retrieval sets is likely to be low-the maximum overlap observed in this study was 26%-and the two approaches to online bibliographic retrieval appear to be complementary. The assignment of thesaurus terms by indexers and the selection of references by authors create different, partially congruent literatures. Subsets of both literatures may be relevant to a given researcher s information needs, serving related rather than identical functions. Both approaches to retrieval can yield relevant documents previously unknown to the user, and, in a broad multidatabase search, well over half the relevant retrievals may be novel. As a retrieval performance measure, the best novelty percentage is likely to fall below loo%, and depend as much on the prior knowledge state and scholarly style of the user as on database scope, domain, quality of indexing, and available search approaches. References Pao, M. L. Semantic and Pragmatic Retrieval, Proceedings of the 47th ASIS Annual Meeting, : ; Goffman, W.; Pao, M. L. Retrieval of Biomedical Information for Emerging Interdisciplinary Problems, Proceedings 4th International Congress on Medical Librarianship, 39-50; Pao, M. L.; Fu, T. T. W. Titles Retrieved from Medline and from Citation Relationships, Proceedings 48th ASIS Annual Meeting, : ; Pao, M. L. Comparing by Keywords and Citations, Proceedings National Online Meeting, ; White, H. D. et al. Quality of Indexing: the Development of a Methodology Comparing Databases Covering the Medical Behavioral Sciences (MBS) Literature, Final Report, NLM-Drexel MBS Project, Study 3. Philadelphia, PA: Drexel University, College of Information Studies, NOI-LM Rapp, Barbara. A Comparison of Document Clusters Derived from Co-cited References and Co-assigned Index Terms. Ph.D. dissertation, Drexel University, Lancaster, F. W. Information Retrieval Systems: Characteristics, Testing and Evaluation 2nd. ed. New York: John Wiley & Sons; Griffith, B. C. et al. Tests of Methods for Evaluating Bibliographic Databases: An Analysis of the National Library of Medicine s Handling of Literatures in the Medical Behavioral Sciences, Journal of the American Society for information Sciences 37: ; McCain, K. W. et al. Test in Five Major Databases Covering the Medical Behavioral Sciences (MBS) Literature, Final Report, NLM-Drexel MBS Project, Study 2. Philadelphia, PA: Drexel University, College of Information Studies, NOI-LM McCain, K. W.; White, H. D.; Griffith, B. C. Comparing Retrieval Performance in Online Databases, in submission. 114 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE-March 1989