SEARCHING THE STATE LIBRARY CATALOGUE

Transcription

1 SEARCHING THE STATE LIBRARY CATALOGUE Searchers Working Paper 5 Vivienne Waller December 2008 The Searchers Institute for Social Research Swinburne University of Technology

2 Acknowledgements This Working Paper is part of the Australian Research Council project The Searchers. We would like to thank the many staff at the State Library of Victoria for their assistance and suggestions, in particular Sue Hamilton, Ian Patterson, Janice van de Velde, Anne Beaumont, Kirstie McRobert, Leneve Jamieson and Patrick Gregory. This research was funded by Australian Research Council grant LP077215, conducted under the guidance of Chief Investigators Professor Julian Thomas and Professor Denise Meredyth and with the assistance of Dr Ian McShane. The Institute for Social Research takes all responsibility for the content. The views of the authors are not necessarily those of the State Library of Victoria.

3 Table of Contents Executive Summary...2 Background to the analysis...4 Scope of the analysis...4 Overview of the analysis...5 Who uses the catalogue?...5 z39.50 searches...6 How do the public use the catalogue?...8 Types of search undertaken...8 Limits placed on searches...14 Number of searches conducted within each session...15 Duration of searches...18 Search results...19 Failure rates...20 Abandonment rates...25 What do people search for...27 Conclusions

4 Executive Summary This report details an analysis undertaken in September-November 2008, on logs of searches on the catalogues of the State Library of Victoria (SLV or State Library). It forms one of the outputs of the Searchers project. This three year project is a collaborative research venture supported by the Australian Research Council, being undertaken by the Institute for Social Research at Swinburne University of Technology in partnership with the State Library of Victoria. The focus of this analysis is on explicit use of the Main and Pictures catalogue by members of the public, either accessing the catalogue from inside the State Library or remotely. Within the limitations of log data, the analysis examines who uses the catalogue, and how they use it, including how successfully they use it. It also provides a preliminary analysis of what people are searching for. The following points summarise the report s findings and discussion. Conduct of searches The distribution of search types was stable across the years for the Main catalogue. In the Pictures catalogue, the proportion of author browse searches decreased from 6.5% in 2005 to 1.9% in 2006 and 2007, increasing slightly to 2.8% in Almost one in five (18.0%) of users of the Main catalogue commenced with a Title search; one in seven (13.6%) commenced with an Author Browse search. While just under half of the searchers of the Main catalogue used Keyword Relevance search for their first search, 80% of searchers of the Pictures catalogue used Keyword Relevance search for their first search. In the main catalogue, the limit facility was used for only around 6% of those searches for which limits could be set. Date was the most commonly used limit, used for 3.7% of searches. Date is the only limit that can be set in the Pictures catalogue. This was used to limit less than 2% of searches. Two fifths (39.8%) of searchers undertook one search in the Main catalogue and just 5.1% undertook more than 10 searches. Half of all users who conducted more than one search spent less than 2.2 minutes searching the Main catalogue and less than 2.4 minutes searching the Pictures catalogue. The median session duration for the Main catalogue has decreased by 24 seconds between 2005 and

5 Those using their own wireless connection tended to spend longer on the catalogue than those using a public terminal within the library. Search results - Failure rates and Abandonment rates More than half of all Boolean Searches, Left anchored title searches and journal title searches received zero hits. The outside public were most likely to get no hits for their first search (17% compared to 11% of those using public terminals and 12% of those using a wireless connection in the library). The outside public were also more likely to get zero hits for their second search (in addition to zero hits for their first search). If a user gets no hits for any of their searches and terminates the session before getting any hits, they can be considered to have abandoned the search. The abandonment rates were much higher for those using the catalogue outside the State Library than those using the public terminals inside the library. One quarter (26%) of users outside the State Library abandoned their search after receiving zero hits for the first search. One fifth of those outside the State Library who conducted a second search gave up after receiving zero hits for both the first and second search. One tenth (11%) of users at public terminals in the State Library abandoned their search after receiving zero hits for the first search. Given the increasing speed of computers and search engines over the last few years, it could be expected that people expect to find things faster and are getting less patient with using a catalogue. The data provides some evidence for this as abandonment rates are significantly higher in 2008 than in What do people search for More than half (57.6%) of searches in the Main catalogue were categorised as general searches and 42.4% of searches were for specific items. Most (83.1%) of searches in the Pictures catalogue were categorised as general searches and 16.9% were for specific items. Conclusion The analysis suggested that many catalogue users would benefit from a Did You Mean? (spell check) as well as an online chat live help facility to assist in formulating successful queries. 3

6 Background to the analysis The Searchers project examines the strategic challenges for major public libraries presented by the online information environment. The project has three components: research on current information-seeking and the role of public libraries in online information provision, analysis of wider developments in online environments and search models, and evaluation of the policy and strategic implications for public libraries highlighted by the research. The project will provide guidance for the State Library of Victoria in developing appropriate models of information provision and contribute new thinking on the role and position of major libraries in the digital age. This report contributes to the research on current information-seeking, analysing how people use the Main and Pictures catalogue of the State Library of Victoria. A subsequent report will include an analysis of the types of search terms that people type into the catalogue. In late November 2008, the State Library of Victoria upgraded the catalogue interface, and incorporated the Pictures and Manuscripts catalogue into the Main catalogue. From mid 2009, the State Library will supplement their existing catalogue with a New Generation catalogue, designed to be particularly easy to use. As well as providing insights into how the catalogue is used, this analysis will: provide useful baseline data in evaluating the success of the new catalogue interface and the New Generation catalogue. inform the design of the log for the New Generation catalogue. provide a model for future analyses of catalogue data. The particular way in which the data is presented here is the result of a range of complex decisions about how the data should be conceptualised and interpreted in order to build up a picture of how the catalogue is being used. This baseline exercise of designing an approach to the analysis of SLV catalogue data will greatly facilitate future catalogue analysis. Scope of the analysis In September-November 2008, analysis was undertaken on logs of searches on the catalogues of the State Library of Victoria (SLV or State Library). The focus of the analysis is on explicit use of the Main and Pictures catalogue by members of the public. This includes members of the public accessing the catalogue from inside the State Library and those accessing it remotely. The analysis is mainly presented in terms of searchers rather than searches. Within the limitations of log data, the analysis examines who uses the catalogue, and how they use it, including how successfully they use it. It also provides a preliminary analysis of what people are searching for. A more complete analysis of what people search for, including an analysis of search terms, will be the subject of a separate report. The quantitative 4

7 analysis in this report is supplemented by some illustrative individual examples. Overview of the analysis The State Library uses a Voyager (ExLibris) catalogue. Data extracts from three of the catalogues were analysed: the main catalogue, the pictures catalogue and the Victorian Government publications catalogue. The main catalogue contains records for books, magazines, newspapers, electronic books and journals, websites, video recordings, music, maps and oral history. The pictures catalogue contains more than 200,000 digital images and records of photographs, prints, and drawings in the Library s picture collection. Most of these pictures relate specifically to Victoria. The Victorian Government publications catalogue contains records for Victorian State and Local government publications received after The extracts contained logs of searches undertaken in August 2005, August 2006, May 2007 and May (The data for May 2007 was incomplete as it could not be obtained for the first week of May. This should not affect the results of the analysis.) Table 1 shows the number of records in each extract. Table 1: Number of records in catalogue analysis Date of Extract Catalogue Main Pictures Victorian Govt Publications Aug Aug May May Total Only summary statistics were analysed for the Victorian Government Publications catalogue as it is to be discontinued in Most of the records are already in the Main catalogue and other records will be transferred. Who uses the catalogue? Although no demographic data is available, the IP address of the computer used to conduct the search enabled identification of whether the user was an SLV staff member or a member of the public. Distinctions also could be made between members of the public who were using a public computer in the State Library (slv public terminal), those using their own computer in the State Library (slv public wireless), and those using a computer outside the State Library, for example from home or work or a public library (outside public). 5

8 z39.50 searches z39.50 searches are searches undertaken via other library databases, such as Libraries Australia, and Google Book search; the latter since an agreement in May 2008 between the Library of Congress and Google. These z39.50 searches are indicated separately in Figures 1-3 but are excluded from further analysis as there is limited data on these searches and the users may not even be aware that they have searched an slv catalogue. Figures 1-3 shows how the numbers of searches by each type of user varied across the years under study for each type of catalogue Note: May 2007 only contains three weeks of data staff z39.50 slv wireless (public) slv terminal (public) outside public Aug 2005 Aug 2006 May 2007 May 2008 Figure 1: Main catalogue type of searcher by year While use of the Main catalogue appeared to increase over time (Figure 1), use of the pictures catalogue dipped in

9 Note: May 2007 only contains three weeks of data staff z39.50 slv wireless (public) slv terminal (public) outside public Figure 2: Pictures catalogue type of searcher by year As Figures 2 and 3 show, both the Pictures catalogue and the Victorian Government Publications catalogue are much more likely to be used by people outside the library than the Main catalogue. Most of the people who access the catalogue in the State Library building are using a public terminal although wireless access was introduced at the end of July Note: May 2007 only contains three weeks of data staff slv wireless (public) slv terminal (public) outside public Figure 3: Victorian Government Publications catalogue type of searcher by year 7

10 The remainder of the analysis is limited to the Pictures and Main catalogue. It also excludes staff searches and z39.50 searches. In other words, it relates to explicit use of the Main and Pictures catalogue by members of the public. How do the public use the catalogue? Types of search undertaken There are two distinctly different ways of searching the Main and Pictures catalogue. One can type in a term, or click on a hyperlink in a catalogue record. The option of clicking in a catalogue record is only available subsequent to an initial typed-in search. When typing a search term into the main catalogue, the user can choose from ten user options (the default is Keyword Anywhere). Table 2 lists these options and indicates how they are recorded in the Search log. Table 2: User options in Main catalogue User options in Main catalogue Keyword Anywhere (Relevance Ranked) Title List (Omit initial article the, a, an) Author's Name (Last name first) Subject List Journal/Magazine/Newspaper Title List By or About a Person (Last name first) By or About an Organisation (Words in exact order) Call Number List Keyword Boolean (Link words with and, or, not) Construct a search (second search tab) Recorded in Search log Keyword Relevance Search Left Anchored Title Author Browse Subject Browse Journal Title Search By or About a Person Simple Search Call Number Browse Command Search Construct a search When typing a search term into the Pictures catalogue, the user is also presented with ten different ways of searching. Table 3 lists these options and indicates how they are recorded in the Search log. It can be seen that in the Pictures catalogue, Simple Search is used to refer to any one of four different types of searches: a search By or About a Person, a search By or About an Organisation, a search of the Collection/Series Name List or a search of the Accession Number List. 8

11 Table 3: User options in Pictures catalogue User options in Pictures catalogue Keyword Anywhere (Relevance Ranked) Artist/Creator s Name By or About a Person (Last name first) By or About an Organisation (Words in exact order) Subject List Title List (Omit initial article the, a, an) Collection/Series Name List Accession Number List Boolean Search (Link words with and, or, not ) Construct a search (second search tab) Recorded in Search log Keyword Relevance Search Author Browse Simple Search Simple Search Subject Browse Left Anchored Title Simple Search Simple Search Command Search Construct a search The catalogue search results are all clickable as illustrated in Figure 4. This figure is a snapshot of part of the record for a book called 24/7 retrieved using Author Browse, for Hassan, R. This snapshot is annotated to illustrate how the search log will record clicks by a user on various parts of this record. For example: Clicks on Stanford business books or Twenty-four seven, will be logged as a Left Anchored Title search. Clicking on any of the listed Subjects, will be logged as a Subject Browse search. Clicking on Other authors, will be logged as an Author search Clicking on the Call Number List will be logged as a Call Number Browse search. 9

12 Figure 4: Sample record from main catalogue While obviously the first search undertaken in a session is always typed in by the user, the log does not allow precise identification of which searches are the results of requests typed in by the user and which are the result of clicking in the resulting record. Some types of search are always typed and Author in the Main catalogue and Artist/Creator in the Pictures catalogue are always the result of clicking in the results. Other types of searches can be either typed in or clicked. Table 4 summarises which search types are always the result of typing and which could be due to typing or clicking. Table 4: User action according to search type Search type Author Author Browse Boolean Search By and/or about a Person Call Number Browse Construct a Search Keyword Relevance Search Left Anchored Title (inc Journal title) Simple Search Subject Browse User action Clicking Typing Typing Typing Typing or Clicking Typing Typing Typing or Clicking Typing Typing or Clicking 10

13 In order to isolate those searches that were definitely typed in, a program was run which excluded Author searches and included search types that were definitely typed in. Those types of search which could have been the results of typing or clicking (call number browse, left anchored title, journal title and subject browse) were only included if they were the first search of a session (24% of all searches of these types). As Figure 5 shows, in the Main catalogue, 29.6% of search logs were searches typed in as the first search of a session. Two fifths (39.2%) of searches in the Pictures catalogue were searches typed in as the first search of a session. Figure 5: Conduct of searches by the public In both catalogues, more than 80% of searches were typed in by the user. The first search undertaken in a session (hereafter referred to as first search) is always typed in. This is reported on separately to give a better indication of the types of search undertaken by members of the public using the State Library catalogue. This helps avoid double counting of what is essentially part of the one search. Table 5 shows the proportion of each type of search recorded in the search log extracts 1 for the Main catalogue and the Pictures catalogue. It distinguishes between all searches and first searches in the session (first search). It can be seen that reporting on all searches gives a different picture; in particular, it underestimates the proportion of keyword relevance searches and overestimates the proportion of subject browse searches. These data suggest that most subject browse searches are the result of clicking on headings in a search result rather than being typed in. 1 The search log recorded a negligible percentage of the following search types: A, Builder, Command, and Keyword. These categories were excluded from the analysis as their contribution was negligible and it was not clear how they were derived. 11

14 Table 5: Main catalogue and Pictures catalogue - search types MAIN PICTURES Type of search first search of all searches first search all searches session of session Author/Artist/Creator na 1.2% na 1.9% Author Browse 13.6% 12.9% 3.0% 2.7% By and/or about a Person.8% 1.2% na na Call Number Browse 1.3% 3.0% na na Boolean Search 0.7% 1.2% 0.4% 1.5% Construct a search (tab 2) 5.3% 7.9% 2.1% 5.2% Journal Title Search 6.5% 6.4% na na Keyword Relevance Search 46.6% 37.2% 79.7% 65.7% Left Anchored Title 18.0% 19.1% 1.6% 4.2% Simple Search 3.1% 1.6% 8.6% 8.1% Subject Browse 4.1% 8.3% 4.0% 10.7% Total 100.0% 100.0% 100.0% 100.0% The distribution of search types was stable across the years for the Main catalogue. In the Pictures catalogue, the proportion of author browse searches decreased from 6.5% in 2005 to 1.9% in 2006 and 2007 and 2.8% in The proportion of simple searches increased from 4% in 2005 to more than 9% in subsequent years. Figure 6 compares the first searches of the session on the Main catalogue with first searches on the Pictures catalogue. It can be seen that the range of search types were more important for users of the Main catalogue than users of the Pictures catalogue. Almost one in five (18.0%) of users of the Main catalogue commenced with a Title search. One in seven (13.6%) users of the Main catalogue commenced with an Author Browse search. While just under half of the searchers of the Main catalogue used Keyword Relevance search for their first search, 80% of searchers of the Pictures catalogue used Keyword Relevance search for their first search. Those searching the Pictures catalogue were more likely to use Simple search for their first search than users of the Main catalogue. They were equally as likely as Main catalogue users to use Subject Browse for their first search. 12

15 By and/or about a Person 1% Boolean Search Subject Browse 1% 4% Construct a search 5% Left Anchored Title 18% Boolean Search 0% Subject Browse 4% Simple Search 9% Author Browse 3% Construct a search 2% Left Anchored Title 2% Journal Title Search 6% Call Number Browse 1% Author Browse 14% Keyword Relevance Simple Search Search 3% 47% Main catalogue Pictures catalogue Keyword Relevance Search 80% Figure 6: Types of first search (Main and Pictures catalogue) Table 6 shows that the type of first search varied among the different types of users. Users within the State Library were much more likely to start their session with a Keyword Relevance search (53.7% of users within the library compared to 42.6% of those outside the library). Those using a public terminal were much less likely to start their search with a Left Anchored Title search (13.0% of those using a public terminal compared to 20.4% of those using the wireless connection and 20.6% of those outside the library). Table 6: Type of first search by type of user Main catalogue Type of search outside public Type of user slv public slv wireless Total Author Browse 14.5% 11.8% 14.0% 13.6% By and/or about a Person.8%.9%.2%.8% Call Number Browse.9% 1.9% 1.5% 1.3% Boolean Search.7%.5%.2%.7% Construct a search 5.5% 5.2% 3.0% 5.3% Journal Title Search 6.8% 6.1% 3.6% 6.5% Keyword Relevance Search 42.6% 53.7% 53.6% 46.6% Left Anchored Title 20.6% 13.0% 20.4% 18.0% Simple Search 4.3% 1.2%.2% 3.1% Subject Browse 3.4% 5.6% 3.2% 4.1% Total 100.0% 100.0% 100.0% 100.0% 13

16 Limits placed on searches When searching the Catalogue, it is possible to place limits on the types of results returned, except when doing a browse type search (author browse, call number browse, or subject browse). The results can be limited by language (hundreds to choose from), location of collection (for example, browsing collection, children s collection), date, place of publication (hundreds of countries to choose from) and type of item (book, serial, manuscript, etc). Main Catalogue The limit facility was used for only around 6% of those searches for which limits could be set. The following table shows how limits in the Main catalogue set varied according to the type of search. Table 7: Limits set for each search type Search type Limit Limit Total set not set By and/or about a Person 7.9% 92.1% 100.0% Boolean Search 4.2% 95.8% 100.0% Construct a search 5.8% 94.2% 100.0% Keyword Relevance 6.7% 93.3% 100.0% Search Left Anchored Title 3.8% 96.2% 100.0% Simple Search 7.1% 92.9% 100.0% Total 5.8% 94.2% 100.0% Date was the most commonly used limit, used for 3.7% of searches. This is the easiest limit to use as it appears on the first page of the catalogue. Users need to go to the second search page for the other limits. Although retrievals can be limited to one of more than 100 different languages, more than 90% of the language limits set were for English language items. Those using the catalogue at a public terminal in the State Library were slightly more likely to set limits (6.7% of those using a public terminal compared to 5.2% of those using a wireless connection or 5.2% of those outside the library). Pictures Catalogue Date is the only limit that can be set in the Pictures catalogue. This was used to limit less than 2% of searches (for which limits can be set). 14

17 Table 8: Types of limits set - Main catalogue Main Pictures Limit type frequency % of all searches frequency % of all (for which limits can be set) searches (for which limits can be set) Date % % Location of % na na collection Item type* % na na Language % na na Place of % na na publication Any limit % % *Excludes journal title searches as these include an automatic limit on item type. Limits were most likely to be set after the first search (limits were set for 2.8% of first searches, compared to 7.2% of subsequent searches). Searchers were more likely to get more than 1000 hits in their first search (45.5% of first searches got more than 1000 hits compared to 38.7% of subsequent searches) and much more likely to get zero results after the first search (16.2% of first searches received zero hits compared to 28.5% of subsequent searches). This could indicate that there are a set of searchers who receive too many hits in the first search, set limits that are too restrictive and then receive zero hits. The following data suggest that this is likely to be the case. While those who set limits in their first search were just as likely as those who didn t set any limits to get zero hits, in the second search, those who set limits were more likely than those who didn t set limits to get zero hits (24.6% of those who set limits received zero hits compared to 19.4% of those who didn t set a limit). Number of searches conducted within each session. The number of searches conducted within each session was calculated and is presented in Figure 7. 15

18 percent Main Pictures more than 10 no of searches per session Figure 7: Number of searches conducted within each session (Main and Pictures catalogue) Main catalogue The average number of searches per session was 3.4 and the maximum number of searches undertaken within a session was 207. Two fifths (39.8%) of searchers undertook one search. Just under one third (31.5%) undertook two or three searches, just under one quarter (23.6%) undertook 4-10 searches and 5.1% undertook more than 10 searches. Those outside the State Library were most likely to conduct only one search (42.4% compared to 35.1% of users inside the State Library) Pictures catalogue The average number of searches per session was 2.6 and the maximum number of searches undertaken within a session was 205. Three fifths (59.0%) of searchers undertook one search. Just under one quarter (23.0%) undertook two or three searches, just under one sixth (14.7) undertook 4-10 searches and 3.3% undertook more than 10 searches. Those outside the State Library were most likely to conduct only one search (59.9% compared to 45.6% of users inside the State Library) Table 9 shows how the total number of searches per session varied according to the type of search first used in the session. It can be seen that people who undertook a simple search were much more likely to undertake just one search (77.5%) than the average searcher (39.8%). This could mean that those undertaking simple searches were most likely to find what they wanted straight away although one in five (19.8%) searchers who started their catalogue session with a simple search got no hits. Of those that got no hits 46.8% gave up, that is, only did one search. 16

19 People whose first search was subject browse were least likely to undertake just one search (31.7%) and most likely to undertake more than 10 searches (7.0%). This is to be expected, as undertaking a subject browse takes the searcher to a list of clickable headings. Table 9: Total number of searches by first search type Main Catalogue total no of searches Type of first search more than Total 10 Author Browse 45.3% 29.6% 20.4% 4.7% 100.0% By and/or about a Person 33.3% 34.3% 27.1% 5.3% 100.0% Call Number Browse 38.0% 31.7% 26.2% 4.2% 100.0% Boolean Search 42.2% 28.0% 23.9% 5.9% 100.0% Construct a search 46.6% 25.8% 21.8% 5.8% 100.0% Journal Title Search 40.8% 31.7% 22.9% 4.7% 100.0% Keyword Relevance 35.7% 32.9% 26.0% 5.4% 100.0% Search Left Anchored Title 39.3% 33.4% 22.2% 5.1% 100.0% Simple Search 77.5% 14.0% 7.1% 1.4% 100.0% Subject Browse 31.7% 33.4% 27.9% 7.0% 100.0% Total 39.8% 31.5% 23.6% 5.1% 100.0% It can be seen from Table 10 that people who started with a Boolean search or a Construct a search were much more likely to conduct more than one search (74.3% of those who started with a Boolean search and 61.5% of those who started with Construct a search) than the average searcher (41.0% conducted more than one search). This could be because it took several attempts for those using Boolean or Construct a search to successfully construct their search. Table 10: Total number of searches by first search type - Pictures catalogue total no of searches grouped more than 10 Total Author Browse 52.2% 27.3% 16.9% 3.6% 100.0% Boolean 25.7% 28.4% 39.2% 6.8% 100.0% Construct a search 38.5% 29.7% 25.4% 6.4% 100.0% Keyword Relevance Search 59.6% 22.8% 14.4% 3.1% 100.0% Left Anchored Title 59.8% 23.5% 13.5% 3.2% 100.0% Simple Search 68.3% 17.4% 11.7% 2.6% 100.0% Subject Browse 45.5% 30.4% 18.7% 5.4% 100.0% Total 59.0% 23.0% 14.7% 3.3% 100.0% 17

20 Duration of searches When a user first conducts a catalogue search, they commence a session which will time out if the user does not conduct another search within 9 minutes. This makes it possible to calculate the duration of each session, for those users who undertook more than one search. This was done by subtracting the time stamp of the first search in the session from the time stamp of the final search in that session 2. Main Catalogue The maximum length of a search in the Main catalogue was 189 minutes. This searcher was outside the slv and conducted 127 searches, 110 of which were title searches and 25 of which got zero hits. For those users who conducted more than one search (60% of users), the average amount of time spent on the Main catalogue was 4.7 minutes. Because the mean can be affected by a few extremely high values, it is also instructive to look at the median, the value above and below which half of the cases fall. The median was 2.2 minutes. In other words, half of all users who conducted more than one search spent less than 2.2 minutes searching the Main catalogue. Pictures Catalogue The maximum length of a search in the Pictures catalogue was 178 minutes. Excluding those users who conducted only one search, the average amount of time spent on the Pictures catalogue was 6.1 minutes. The median was 2.4 minutes. It is possible that people are getting faster at using the catalogue. The data provides some evidence that this is the case. Table 11 shows that the average session duration has slightly decreased and that the median session duration has decreased by 24 seconds between 2005 and Table 11: Average and median session duration by year - Main and pictures catalogue* MAIN PICTURE year of catalogue Average duration Median duration N Average duration Median duration extract (minutes) (minutes) (minutes) (minutes) N Total *Note: This is calculated for those sessions that had more than one search. 2 Of course, the one searcher may let a session time out and then commence another session. The extent to which this occurs cannot be gauged from the data. 18

21 The average and median duration of searches on the Pictures catalogue were both less in 2008 than 2005, although the decrease was not steady over these years. The distribution of time spent on the Pictures catalogue was very similar to that for the Main Catalogue. The time spent on the Main catalogue was compared for different types of user and was found to be significantly different. Figure 8 shows that the public using the Main catalogue inside the library tended to spend longer on the catalogue than the public using the catalogue outside the library. For example, 33.6% of users outside the library spent less than one minute on the catalogue compared to 24.4% of those using a public terminal within the library and 21.5% of accessing the slv wireless connection. Figure 8: Session duration by user type - Main catalogue Almost one in five (19.3%) of those using their own computer within the library spent more than 10 minutes using the Main catalogue, compared to one in 10 users outside the library (10.3%). There is often competition to use the public terminals within the library, including the catalogue-only terminals. This may explain why those using their own wireless connection tended to spend longer on the catalogue than those using a public terminal within the library. Search results A browse search (Subject Browse, Author Browse or Call Number Browse) locates the search term within an ordered list. Other types of search return a certain number of hits. 19

22 Main catalogue In the Main catalogue the maximum number of hits was 193,620 and the average was 5,234 hits. The number of hits varied according to the type of search undertaken. As would be expected, keyword relevance searches were the most likely to receive more than 1,000 hits (73.5% of all searches). Pictures catalogue In the picture catalogue the maximum number of hits was 115,085 and the average was 2,809 hits. As with the Main catalogue, Keyword relevance searches were the most likely to receive more than 1,000 hits (38.1% of all searches). Main catalogue Figure 9 shows the total number of searches undertaken according to the number of hits in the first search. It can be seen that those users who received only one hit in the first search were the most likely to undertake just one search. This possibly indicates that they found what they wanted to know. Conversely, it could indicate that they gave up. (This possibility is explored in the section on abandonment rates.) Those most likely to stop at one search in the session were those who received either zero hits or more than 45 hits. Those who received no hits in the first search were also slightly more likely to undertake more than 10 searches. 100% 5% 7% 4% 4% 4% 6% 90% 80% 70% 21% 30% 16% 24% 17% 25% 21% 26% number of searches in session percent 60% 50% 40% 29% 41% 28% 33% more than % 20% 45% 57% 54% 47% 36% 10% 22% 0% browse more than 45 number of hits Figure 9: Number of hits by number of searches - Main catalogue Failure rates Receiving no hits is defined here as a failure. Hence, "failure rates" are the 20

23 proportion of searches (excluding browse searches) which got no hits. Browse searches are excluded from the calculation of failure rates as the system records all browse searches as receiving "-1" hits. However, although failure rates cannot be calculated for browse searches, this type of search could be considered as quite prone to failure; as the typing has to be exact and the catalogue doesn t recognize stop words. For example, typing "The Kite Runner" as a Title Search will not return the book "The Kite Runner" even if it is in the State Library collection; one would need to type in "Kite Runner." Similarly, "Kiterunner" would not return the book 3. Obviously, it is not possible from the data to identify which browse searches fail. However, it seems that many users do not really have confidence in the searches they are undertaking and repeat searches they have previously undertaken. This is illustrated in the example of a search for a book/books written by Adrian Shaughnessy (see Table 12). Table 12: Example of lack of confidence in searches undertaken Search Search type Search string number 1 Author Browse Adrian Shaughnessy 2 Author Browse Shaughnessy Adrian 3 Author Browse Adrian Shaughnessy 4 Author Browse Shaughnessy Adrian The first search would not have retrieved any books by Adrian Shaughnessy as the surname needs to be specified first. The second search would have led to three books for which Adrian Shaughnessy is an author. Possibly the searcher is looking for a different book by Adrian Shaughnessy as they have repeated the incorrectly formatted search with the first name first and then also repeated their second search. It is difficult to determine the extent of these repeat searches from the data, but visual scanning suggests that it may not be infrequent. Figure 10 shows the failure rates for different types of searches. 3 This is still the case with the new catalogue interface. 21

24 60% 50% 40% 30% main pictures 20% 10% 0% By and/or about a Person Boolean Search Construct a search Journal Title Search Keyword Relevance Search Left Anchored Title Simple Search Figure 10: Failure rates: proportion of each search type resulting in zero hits Main catalogue The Boolean Search was most likely to get zero hits (55.6% of all searches of this type) and more than half of title searches received zero hits (51.2% of journal title searches and 52.7% of Left Anchored Title searches). As would be expected, Keyword relevance searches were by far the least likely to receive zero hits (4.7% of all searches). Pictures catalogue As with the Main catalogue, Boolean Search was the search type most likely to get zero hits (54.3%). Keyword relevance searches were by far the least likely to receive zero hits (6.7% of all searches). Some searches fail because they are incorrectly constructed; for example incorrect order of terms or spelling mistakes. Others fail because there are no matching items in the SLV collection. It is searches which are incorrectly constructed that are of most concern. Half of all Boolean searches received zero hits. Boolean searches require the user to link words with and, or or not. Without these linking words, an error is registered and the users returns zero hits. A visual inspection of four hundred search strings typed in as Boolean searches, showed that about half of these strings were incorrectly 22

25 constructed. The following example of a user at a public terminal in the SLV suggests how frustrating this may be. This example illustrates zero hits in the context of a sequence of searches in a session. Example of a possibly unsuccessful search session Table 13 lists the 35 searches conducted by this user. They all concern the one topic of packaging waste. Most of these searches received zero hits. Although this user tried a variety of different types of search and tried to use more advanced features, such as limits and Boolean searches, as the following shows, it seems doubtful that they found what they were after. It can be seen that this user started by doing a keyword relevance search on the word 'packaging'. This search returned 408 hits which presumably were too many results for the user to sort through as their next search was another keyword relevance search on the word packaging but this time with the following limits Date 2004, Language English, Location -Browsing collections, Place of publication - Australia. Because the keyword relevance search had failed to return useful results, the user then undertook a range of Journal Title searches and Boolean searches, some with a range of limits set. Most of these returned zero hits, either because of spelling mistakes or because there were no journals beginning with the specific title typed in. Most of the Boolean searches had the requisite Boolean operator ("and"), but several of them towards the end of the session failed because they did not have a Boolean operator. It can also be seen that the user repeated several searches. It is easy to do this as one has to return to a previous screen to conduct a new search; there is no indication in this new screen of what searches have already been undertaken. One has to move to a separate screen (Search History) to see searches already undertaken 4. This user spent 24 minutes on this session and may not have found any items of relevance to them; their last three searches returned zero hits. Table 13: Example of an unsuccessful user's search session Search number Search type Search string Limits set No of hits 1 Keyword Relevance packaging Keyword Relevance Packaging DATE=2004 LANG=ENG 0 LOCA=Browsing collections PLAC=at 3 Journal Title Search PACKAGING DATE=2004 LANG=ENG 0 LOCA=Browsing collections PLAC=at 4 Journal Title Search PASTIC BAGS DATE=2004 LANG=ENG 0 LOCA=Browsing collections PLAC=at 5 Boolean Search pastic bags, enviromental impact 0 4 The updated version of the catalogue also requires the user to go to a separate screen to see their search history. In addition, they have the option of logging on with their barcode and last name to retain a record of their searches. 23

26 Search number Search type Search string Limits set No of hits 6 Boolean Search pastic and enviroment DATE=2000- LANG=ENG 0 LOCA=Browsing collections PLAC=at 7 Boolean Search plastic and enviroment DATE=2000- LANG=ENG 0 LOCA=Browsing collections PLAC=at 8 Boolean Search plastic and enviroment DATE=2000- LANG=ENG 0 LOCA=Browsing collections PLAC=at 9 Boolean Search plastic and enviromental impact 0 10 Boolean Search packaging and waste DATE=2000- LANG=ENG 0 LOCA=Browsing collections PLAC=at 11 Boolean Search plastics and waste DATE=2000- LANG=ENG 0 LOCA=Browsing collections PLAC=at 12 Journal Title Search PLASTICS AND WASTE DATE=2000- LANG=ENG 0 LOCA=Browsing collections PLAC=at 13 Journal Title Search PACKAGING AND WASTE DATE=2000- LANG=ENG 0 LOCA=Browsing collections PLAC=at 14 Boolean Search packaging and waste Journal Title Search PACKAGING AND 0 ENVIROMENTAL IMPACT 16 Journal Title Search PACKAGING AND BAGS 0 17 Journal Title Search PACKAGING AND USAGE 0 18 Journal Title Search PACKAGING AND WASRE 0 19 Journal Title Search PACKAGING AND WASTE 0 20 Boolean Search packaging and waste Boolean Search packaging and waste Boolean Search packaging and waste Journal Title Search PACKAGING AND 0 ENVIRONMENTAL IMPACT 24 Journal Title Search PACKAGING AND 0 ENVIRONMENTAL AND IMAPCT 25 Journal Title Search PACKAGING AND 0 ENVIRONMENTAL AND IMAPACT 26 Journal Title Search PACKAGING Journal Title Search PACKAGING Journal Title Search PLASTIC BAGS 0 29 Boolean Search plastic bags 0 30 Boolean Search enviromental damage 0 31 Boolean Search environmental damage 0 32 Boolean Search environmental and damage Boolean Search packaging and enviromental and 0 damage 34 Boolean Search packaging, enviromental and 0 damage 35 Boolean Search packaging and environmental, impact 0 Figure 11 depicts the failure rates for different types of user. It can be seen that the outside public were most likely to get no hits for their first search (17% compared to 11% of those using public terminals and 12% of those using a wireless connection in 24

27 the library). The outside public were also more likely to get zero hits for their second search (in addition to zero hits for their first search). 25% 21% 20% 17% 15% 15% 15% Percent 11% 12% no hits first search no hits second search 10% 5% 0% outside public slv public slv wireless Figure 11: Failure rates by user - Main catalogue Abandonment rates If a user gets no hits for any of their searches and terminates the session before getting any hits, they can be considered to have abandoned the search. (Of course, there will also be those who abandon their search because they do not find the particular results to be useful, or are overwhelmed by too many hits, but these cannot be identified from the data). Abandonment rates for the main catalogue are analysed as follows: Main Catalogue Just over one fifth (22.3%) of those who got no hits in the first search gave up, that is, did no more searches in that session. Of those who conducted another search, just under one fifth (18.0%) got no hits in their second search either. As shown in Figure 12, the abandonment rates were much higher for those using the catalogue outside the State Library than those using the public terminals inside the library. This difference is not due to the different search types. 25

28 30% 25% 26% abandon after one null search 20% 21% abandon after two null searches Abandonment rate 15% 11% 11% 12% 18% 10% 5% 0% outside public slv public slv wireless Type of catalogue user Figure 12: : Abandonment rates by user - Main catalogue One quarter (26%) of users outside the State Library abandoned their search after receiving zero hits for the first search. One fifth of those outside the State Library who conducted a second search gave up after receiving zero hits for both the first and second search. One tenth (11%) of users at public terminals in the State Library abandoned their search after receiving zero hits for the first search. Given the increasing speed of computers and search engines over the last few years, it could be expected that people expect to find things faster and are getting less patient with using a catalogue. The data in Table 14 provide mixed evidence for this. Table 14: Abandonment rates by year - Main catalogue Year of catalogue extract Abandon after one search Abandon after two searches % 15.2% % 16.7% % 19.5% % 20.5% Total 22.3% 18.0% Abandonment rates are significantly higher in 2008 than in

29 In 2008, one in four users (25.0%) abandoned after one null search compared to just over one in five (21.9%) in However, the abandonment rate went down slightly in In 2008, 20.5% abandoned after one two null searches compared to 15.2% in What do people search for An attempt was made to categorise search types on the basis of whether the user is looking in the catalogue for a specific item or is researching a general topic and to compare the number of searches and the session duration for each type of search. Table 15: Classification of types of search as specific or general Search type Author/Author Browse Boolean Search By and/or about a Person Call Number Browse Construct a Search (search tab 2) Keyword Relevance Search Left Anchored Title (inc Journal title) Simple Search (By or about an organization) Subject Browse Type of search Specific item General topic General topic General topic General topic General topic Specific item General topic General topic Using this categorisation, 57.6% of searches in the Main catalogue were categorised as general searches and 42.4% of searches were for specific items. Most (83.1%) of searches in the Pictures catalogue were categorised as general searches and 16.9% were for specific items. Table 16: Number of searches by type of first search - Main catalogue MAIN catalogue Type of first search PICTURES catalogue Type of first search total no of searches general specific Total general specific Total grouped % 44.2% 39.8% 58.3% 63.8% 59.0% % 30.5% 31.5% 23.4% 20.6% 23.0% % 20.7% 23.6% 15.0% 12.8% 14.7% more than % 4.6% 5.1% 3.3% 2.8% 3.3% Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% Main Catalogue Those who started with a specific search were more likely to just do the one search (44.2% of those who started with a specific search compared to 36.6% of those who started with a general search). Presumably, these searchers were satisfied that they had established from the first search whether or not the item they were looking for was in the collection. 27

30 Pictures Catalogue As with the Main catalogue, those who started with a specific search were more likely to just do the one search (63.8% of those who started with a specific search compared to 58.3% of those who started with a general search). Main Catalogue As might be expected, the session time tended to be shorter for those who were looking for a specific item. As Table 16 shows, excluding those who did just the one search in the Main catalogue, just over one third (35.4%) of those who started with a specific search searched for less than one minute, compared to just over one quarter (26.5%) of those who started with a general search. A similar pattern was observed in the Pictures catalogue. Table 17: Duration of session by type of first search MAIN catalogue PICTURES catalogue Type of first search Type of first search general specific Total general specific Total less than 1 minute 26.5% 35.4% 29.9% 28.3% 36.5% 29.3% 1-2 minutes 23.8% 22.3% 23.2% 21.8% 21.5% 21.7% 3-5 minutes 19.0% 17.1% 18.2% 16.8% 15.4% 16.6% 5-10 minutes 17.6% 15.3% 16.7% 16.1% 12.6% 15.6% more than 10 minutes 13.2% 10.0% 11.9% 16.7% 12.8% 16.2% Total 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% It could be hypothesized that the proportion of general searches would show a decrease as people increasingly use an Internet search engine rather than the catalogue. The data did not support this hypothesis for the Main catalogue. The proportion of general searches showed no definite pattern across the years. As table 17, shows, this proportion increased slightly between 2005 and 2006 then decreased, remaining constant between 2007 and However, the proportion of general searches in the Pictures catalogue showed a very slight, but statistically significant (p<0.001) decrease between 2005 and Table 18: General and specific searches by year MAIN catalogue Year of extract PICTURES catalogue year of extract Total Total general 59.7% 61.3% 55.6% 55.5% 57.9% 87.6% 86.8% 85.3% 85.1% 86.2 % specific 40.3% 38.7% 44.4% 44.5% 42.1% 12.4% 13.2% 14.7% 14.9% 13.8 % Total 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% Table 18 shows that those accessing the Main catalogue from outside the State Library were most likely to start with a search for a specific item than those within the State Library. Someone who is after a specific item may wish to see whether it is in the State Library collection before making a physical visit to the State Library. Those 28

31 using the Main catalogue at a SLV computer were most likely to undertake a general search. Table 19: General and specific first searches according to type of user User of MAIN catalogue User of PICTURES catalogue outside slv slv Total outside slv slv Total public public wireless public public wireless general 53.0% 66.9% 61.5% 57.9% 86.3% 84.0% 83.3% 86.2% specific 47.0% 33.1% 38.5% 42.1% 13.7% 16.0% 16.7% 13.8% Total 100% 100% 100% 100% 100% 100% 100% 100% The pattern was different for those accessing the Pictures catalogue. Those outside the library were slightly more likely to start with a general search than search for a specific item. This is understandable as the Pictures catalogue gives immediate access to many digitised items. Hence, those outside the library who just want a picture, say of early Melbourne can access one without physically visiting the library. Conclusions At present, the catalogue has online help screens that briefly outline the mechanics of the catalogue. However, if someone requires help in formulating their search, their options are limited. If they are within the library, they need to physically queue up at the Information desk or hail a passing librarian. If they are accessing the catalogue remotely, they can phone the Library. The analysis has indicated several ways in which searchers could be assisted in their use of the catalogue. Although no quantitative analysis was done on the actual number of spelling mistakes, visual inspection of samples of the data indicate that many users would be greatly aided by a Did You Mean? (spell check) that picks up spelling mistakes. The analysis shows that abandonment rates are high and have been increasing slightly since This finding indicates the need to provide assistance to users to help them successfully formulate their search. Such assistance provided at or before the point of failure can reduce the likelihood of the user abandoning their search. For example, an online chat, live help facility could be equally effective in helping both those using the catalogue within the library and those outside the library. 29