Scott Hamilton 1, Helen Thurlow 2 1. Queensland University of Technology, Brisbane, Australia 2. State Library of Queensland, Brisbane, Australia As Queensland's major public reference and research library, the State Library of Queensland provides a diverse range of services to the State s 3.6 million people. Each year, some 5.5 million visits are made to the State Library in person or via the Internet. Staff answer 80,000 information enquiries and lend approximately 500,000 items to the State's public libraries and other institutions. The State Library uses the Voyager integrated library system (www.endinfosys.com) and the catalogue of over 2.1 million items attracts the highest number of hits on the Library s website. The OPAC (online public access catalogue) has not been altered since its introduction in 2002 and a review is being undertaken to identify design and content improvements which could be incorporated when software is upgraded in 2006. In 2004, an industry project undertaken by a graduate Library Studies student from the Queensland University of Technology (QUT), one of Australia s largest universities with some 40,000 students, analysed the OPAC transaction logs and developed a user friendly toolset to enable the analysis. The project s objectives were to review the logs for the following: most common search terms used by patrons most common search strategies used by patrons relative success or failure rates relating to searches undertaken. The project also provided an opportunity for the State Library to obtain significant research analysis and results on which to potentially support decision making in a number of areas: collection development most/least popular subject areas in the collection; collection gaps identified by unsuccessful searches cataloguing practices that may improve search success rates; relationship between keywords and subject headings client services assess current client training and help screen content staff training assess staff understanding of OPAC search options and develop appropriate training OPAC design search options presented on search screens and terminology used. Opinions on OPAC design abound. Therefore, using an evidence-based approach to the OPAC redevelopment will ensure decisions incorporate the characteristics of evidence-based practice, including the collection, interpretation, and integration of valid, important and applicable user-reported, librarian-observed, and research-derived evidence (Booth, 2004).
Research Project Overview The project evaluated four months of data from 2004 and some earlier data relating to search terms. The methodology was based on Interrogative Theory of Information and Knowledge as described by Quigley and Debons (1999); ie the who, when, what, where, how and why analysis of system usage. The final report includes a comprehensive literature review of TLA (Transaction Log Analysis) and extensive details of the project s results, including charts, graphs and tables to illustrate the investigation process. Research Project Methodology: Analysis using interrogative theory makes distinctions between data, information and knowledge as follows: Knowledge text that answers how/why in the problem space Information text that answers when/where/who/what in the problem space Data text that answers no question in the problem space. Transaction logs provided for this project contained overwhelming loads of data. An interrogation tool called QlikView, by Swedish firm QlikTech International, was used to convert the log data into useful information. QlikView is an innovative, low cost, application which simulates the complex architecture of a Business Intelligence (BI) implementation within a standalone application. It has a simple, but powerful, user interface and proved to be the perfect tool for analysing the transaction log files. Literature Review The literature review briefly followed the history of the OPAC, noting the predominance of research prior to the mid 1990 s. It showed that developments from the first to second generation of OPACs provided some limited help to a mainly expert user community, but that, on the whole, they failed to successfully translate into third generation systems and the overall experience for non-expert users remained poor. In the past ten years, the trend has been away from structured analytical searching strategies towards a simple search/browse strategy. This has been influenced by the Internet generation of searchers who prefer the simple interface of a Google search, and the resulting vast, if not always appropriate, array of webaccessible resources. Transaction log analysis has remained a prominent analysis technique during this time, although a stronger focus on web transaction analysis can been seen in the literature beyond 1995. The underlying problem with TLA remains that, despite advances in technology and user interfaces, it does not assist the understanding of the motivation behind user s actions. Despite this, valuable user trends can be identified and, with the appropriate toolset, data on user search strategies can be evaluated and used to improve the overall performance of OPAC systems. 2
Project Outcomes The literature review revealed a number of definitions of a successful and failed search. Measures for determining search success or failure include lexical, syntactic, semantic and conceptual issues, but this information is not generally captured in log files, and thus remained outside the scope of this project. Consequently, a more simplistic approach for defining search success or failure was used. The issue of what constitutes a successful search is very grey. How can you know what the user was thinking when they entered a particular search string? What is a successful result from their viewpoint? It is difficult for us to know. We can only follow like a bush tracker, trying to interpret the tracks left behind and, from that, make educated guesses about what the target of our tracking required. The literature review suggested that the following rules may be appropriate: assuming a user is undertaking a known item search, a result of exactly one record may indicate a successful search - the user knew what they wanted, and most likely found it if a user is undertaking an unknown item search, a result of between one and ten items may represent a good outcome - the range of options may suit their needs, but they have not been overwhelmed with results. Using this hypothesis, a one item result may be considered very successful, while a two to ten results range may be considered adequately successful. Result hits between 11-50 are also considered successful, though slightly less so. Retrieving more than 50 hits starts to become overwhelming for most users and could be considered the least successful search result, effectively equivalent to a failure. A result of zero is considered to be a probable search failure, either because the search failed due to user error (eg wrong search terms, spelling, syntactic or conceptual errors), or because the user confirmed that what they wanted did not exist, still leaving them with an unresolved need. The final option in this research is a -1 hits result which, in Voyager, indicates the user has performed a browse search. In this case, the logs did not indicate success or failure, just the type of search was attempted. Search success rating is summarised in the following table: Number of Hits Success Rating -1 n/a Subject search 0 Search failure 1 Highly successful 2-10 Very successful 11-50 Moderately successful > 50 Minimally successful 3
Using this rating system, it was possible to analyse the difference search options selected by users to determine probable search success for each type of search. Figure 1: Outcomes of search success or failure rates based on search types used. Green suggests success, red suggests failure. (4 Months data from 2004.) A significant percentage of search attempts are initiated using the default search option Any words, which uses a boolean or search. The research suggests that this option may not be the most successful strategy. The alternate Guided Search offers a choice of or, and, or phrase searching and may retrieve more relevant, succinct results. Therefore, although it has more search boxes, and could be perceived as a more complicated screen, Guided Search may be a more successful default search option. Users could still select the simple interface, but this would be a conscious decision rather than a default option. Implementing the change for a period, followed by a similar analysis of search logs, would indicate if search results were more successful according to the above success ratings. The question to be considered is that with the current search options focused on providing a simple interface, is the State Library actually setting up their users to fail? Project Recommendations: The Project presented a number of recommendations to the State Library: 1. Review and analyse the full project report 2. Review suitability of QlikView for undertaking this and other analysis tasks 4
3. Investigate specific initial business requirements, and apply this tool and approach to that problem domain 4. Consider changing the default OPAC search tab Guided Search 5. Undertake ongoing analysis of log files to observe impact of changes Where to from here? The report has been considered by the State Library, with some findings supporting staff opinions, and others being flagged for further investigation. As stated in the Introduction, the State Library hoped the project would provide research results to assist decision making in a range of areas, and some findings have been immediately relevant: Collection development o Transaction logs indicated frequently searched terms and their retrieval results - some search terms did not retrieve as many results as expected; further research is required to determine if this is due to a weakness in the collection, or a user s poor search strategy which prevented material from being retrieved o the State Library s Collection Development Policy is being revised and will reflect conspectus subject areas; it is expected that TLA will then be used as a tool to monitor collection usage Cataloguing practices o The project offered staff the opportunity to review search terms and are considering the role of authorities in search behaviour. Further research is required. Staff training o staff OPAC training sessions will incorporate examples of user search strategies to assist staff to understand how users approach searching Help pages content o OPAC help pages are being revised to reduce text and include commonly-used search terms as examples o Novotny (2004) reported that implementing changes resulting from TLA on library help screens lead to changed searching behaviours and improved user success. OPAC design o As the State Library is reviewing its OPAC design, the research and hypothesis testing contributes evidence to assist the decision-making process. Other sources of evidence for OPAC redesign TLA as undertaken in this project will not provide sufficient evidence to support OPAC redesign. A range of other sources must also be considered. 5
Usability testing As a government agency, the State Library adheres to website standards developed by the State of Queensland and complies with World Wide Web Consortium Web Content Accessibility Guidelines (V1.0). Usability testing can determine user satisfaction, and identify design, labelling and terminology issues. The State Library has recently commissioned such testing and it is hoped the results will provide further evidence to support OPAC redesign decisions. Best Practice Guidelines and standards developed and adopted by the library sector provide documentation to support decision making. After seven years of development, the International Federation of Library Associations and Institutions (IFLA) (2005) has released its final report Guidelines for OPAC displays which makes recommendations for best practice in the design of OPACs. FRBR (Functional Requirements for Bibliographic Records) is an IFLA-supported entity-relationship model of metadata for information objects, in place of the single flat record concept underlying current cataloguing standards. The model has four levels of representation: work, expression, manifestation, and item, and brings resources together as "works" to help users sift through the myriad information resources. Enhanced logging Transaction logging provides quantitative data about the choice of search options and search terms, but provides no qualitative data about the client s requirements or whether the client was satisfied with the search outcome. Capture of quantitative OPAC data could track a user s post-search activity: o did the search result in a requests for item (applicable in closed access collections)? o was the search saved in the user s own patron record to create a current awareness alert, or for future re-execution? o was the record saved to the user s bookbag? o did the user email the record? Tracking such activity could increase evidence of client satisfaction. Current research The project s literature review indicated that much OPAC research predated the advent of online searching and web search engines. Recent research focuses on the need for the OPAC to satisfy user requirements in the face of competition from web search engines. Further, much research emanates from the academic library sector and results are not necessarily transferable to the state/public library setting where the mission, service environment and clients are different. The challenge is to understand how our clients interact with the wide range of physical and virtual resources currently available. 6
Dempsey (2005) suggests three ways of combining current user interfaces, search engines and data: 1. monolithic search systems different interface for each collection of indexed data 2. metasearch extra layer added to monolithic systems to hide boundaries (eg via Z39.50) and reduce search costs, but high cost in terms of configuration and maintenance 3. It s the data stupid user determines interface and search options; data is exposed for gathering by search engines. Many systems still tend to be monolithic, with some offering elements of metasearch. OCLC s release of bibliographic records to Google and Yahoo embraces the third option, providing a new means of enabling access to library resources and expanding the size of the potential audience. Novotny (2004) found that catalogue users are heavily influenced by trends in Web searching and reports a number of OPAC services which users considered would improve library catalogues: o more like this to find related information o extensive use of hyperlinks for navigation o summaries and tables of contents o relevance ranking and exact title marching o context-sensitive and interactive help. Such services support a move to Dempsey s third option by offering interactive access to information outside the catalogue. Conclusion This industry sponsored project between the State Library and QUT offered an opportunity to follow the evidence-based process, providing a practical application of evidence-based theory. In association with other means of evaluating information search and retrieval systems, the State Library will use the TLA research and results to consider changes to OPAC design and content. In our dynamic working environment, the application of evidence-based theory and practice provides a framework in which to make professional, defensible decisions. Such practical application of the theory demonstrates a successful, beneficial collaboration between researchers and practitioners. References Booth, Andrew. (2004). E is for Evidence Based Effectiveness. [File Format: Microsoft Powerpoint 97]. Retrieved 18 August, 2004 from: http://www.ucrg.org.uk/events/conference_reports/booth_181104.ppt Booth, Andrew & Brice, Anne (Ed.). (2004) Evidence-based practice for information professionals: a handbook. London: Facet. Dempsey, Lorcan 2005. The three stages of library search. Retrieved 24 August 2005 from: http://www.cilip.org.uk/publications/updatemagazine/archive/archive2004/november/lorcan.htm 7
IFLA (2005). Guidelines for Online Public Access Catalogue (OPAC) Display. Saur. iprospect (2004) Search engine marking firm iprospect survey confirms importance of visibility on the first three pages of search results. Retrieved November 9 th, 2004 from http://www.emediawire.com/releases/2004/4/prweb120172.htm Novotny, Eric. (2004). I don t think, I click College & Research Libraries, 65(6) 533. OCLC Research Activities and IFLA's Functional Requirements for Bibliographic Records. Retrieved 18 August 2005 from: http://www.oclc.org/research/projects/frbr/default.htm Quigley, E. J., & Debons, A. (1999). Interrogative theory of information and knowledge. In J. Prasad (Ed.), Proceedings of the 1999 ACM SIGCPR Conference on Computer Personnel Research (New Orleans, Louisiana, United States, April 08-10, 1999) (pp. 4-10). New York : ACM Press. Retrieved 24 August, 2004 from http://infosci.tripod.com/paper.htm. Slone, D.J. (2000). Encounters with the OPAC: On-line searching in public libraries. Journal of the American Society for Information Scienc, 51(8), 757. Retrieved 2 August, 2004 from ProQuest database. Social and Behavioural Research Institute (2001) Phase II Information Competence Final Report. Retrieved November 9 th, 2004 from http://www.csupomona.edu/~kkdunn/icassess/currentsite/phase2report.pdf 8