An Economic Model for Comparing Search Services

Transcription

1 An Economic Model for Comparing Search Services Stephen K. Kwan MIS, San José State University Shailaja Venkatsubramanyan MIS, San José State University Abstract Search services are now ubiquitously employed in searching for documents on the Internet and on enterprise intranets. This research develops an economic model for comparing search services based on a user s information requirement in a decision making scenario. The model considers the noise effects of querying, search and filtering of results. Different search engines might return different results for the same query based on the characteristics of the search engine s algorithms and the extent of captured data. Users are thus faced with the selection of a search service in order to minimize cost, reduce uncertainty, and maximize the benefits derived for their efforts. A methodology for comparing search services based on the model is presented. This comparison can also be used by search service providers to enhance alignment between their objectives and that of users as well as for pricing of their products and services. A preliminary experiment comparing three popular search services is used to illustrate the model. 1. Introduction With the growth of the World Wide Web, one technology that has become ubiquitous and indispensable is that of Web search. Search services are now widely employed in searching for documents on the Internet and on enterprise intranets. There are many commercial search services available to users. Users are thus faced with the task of comparing search services in order to minimize costs, reduce uncertainty, and maximize the benefits derived for their efforts. In this paper, the term search engine is used to refer to the software system with its search and ranking algorithms that operate on indexed data while search service is the service or portal provider. Traditionally, precision and recall have been used to evaluate search. Other measures used include the stability of a search service over time, recall or precision over a subset of retrieved documents, and correlation between human and engine ranking [2,8,21]. However, these measures do not differentiate between noise or garbling introduced during various stages of the search process. First, user characteristics that may play a role in the search outcome are not taken into account. Noise may be introduced into the search process by the inability on the part of the user to provide good query terms. When users translate information needs to keyword queries, the quality of keyword and phrase construction could influence the results returned by the search service. Second, certain search services may be better at handling certain types of information needs than others. It has been pointed out in previous research that the traditional measures of precision and recall may not be appropriate for evaluating tasks that need high accuracy [19]. For instance, when a user is looking for a very specific answer to a narrow question such as what was the name of Abraham Lincoln s wife?, precision and recall may be less indicative of performance of a search service as compared to a measure that takes into account the rank of the first document in the results set that answers the user s question. Lastly, users have to use their filtering skills to find useful documents. The filtering skills could depend on several factors such as the user s domain knowledge and skill with searching, as well as the user interface of the search service. Users are thus faced with the choice of a search service to get the best possible results for their information needs while factoring in user ability as well as effectiveness of the search engine. Currently, most Internet-based search services are free to the general public. However, there is a possibility they may have to pay a price for using that technology in the future [10]. Besides, whether use of search service is free or not, users are faced with making a decision about which search service to use. The decision is bound to depend on user characteristics that affect the search process, the type of information need, and search /06/$20.00 (C) 2006 IEEE 1

2 engine characteristics. There is no standard or recommended manner in which search services can be compared in order to pick one that would be most appropriate for specific users with their individual information needs. In the case of intranets, more often than not, search services have to be purchased for a price. In such a situation, a decision has to be made about which search service would be most useful for the needs of that organization. This paper describes a methodology for comparing search services within the context of certain types of user needs. The process used to search for information is composed of multiple steps. An example is Kuhlthau s information search process or ISP [12]. Kuhlthau s ISP is composed of the tasks of initiation of information need, selection of topic to be investigated, exploration of feelings of confusion, formulation of a sense of clarity, collection of information, and presentation or use of findings of search. For the purpose of our research, we assume that the search process begins when a user faced with a decision problem. The user then formulates a query and submits it to a search engine to obtain search results. The user filters the search results to look for information relevant to the decision problem. Based on the filtering, the user may reformulate or refine the query and submit it to the same or a different search engine. Finally, the user either makes a decision based on the search results or decides to abandon the search. (See Figure 1) Decision Problem Query Reiterating Search Service Search Environment Filtering Figure 1. Decision making model Deciding Keeping this process in mind, the goal of the study was to develop a methodology that could help evaluate the following: (a) effect of user characteristics (such as domain knowledge and experience with search engines) on query formulation; (b) effect of type of information need on query formulation; (c) effect of formulated queries on quality of search results; (d) role of search engine itself on quality of search results; (e) effect of user s domain knowledge and their experience with a particular search engine on the ease of filtering; (f) effect of quality of search results on ease of filtering; and (g) effect of information need type on ease of filtering. We conducted a pre-test using three popular search engines Google, Yahoo and MSN, and collected some preliminary feedback. The rest of the paper is organized as follows: Section 2 presents a formalization of the web search scenario so as to provide a theoretical framework for our experimental results. Section 3 presents the experimental design and the results from the pre-test. Finally, conclusions and future work possibilities are presented in Section Web Search and Information Needs A comparison of search engines requires a deeper understanding of how users translate their information need into a web search. In this section, we first describe what we mean by the term information need. We then describe the model of web search to be used in our analysis and illustrate how the user s search process is an integral part of this model. 2.1 Information Need Information need refers to the type of information sought by the user. Belkin and Croft define an information need as a problematic situation where a person cannot attain some goals due to inadequacy of resources or knowledge [1]. Kuhlthau defines an information need as the gap between the user s problem or topic and what the user needs to know to solve a problem [12]. Information needs have been classified in various manners by different researchers. Tague- Sutcliffe [20] classified information needs into categories such as quick reference questions, howto-do questions, questions that involve collecting and synthesizing information about a topic, and doing a literature search for a project. These were based on the kind of information required for the user task or question for which information is sought, as well as whether there would be variation among users about expected results. Glover et. al [6] suggested categories based on the kind of information sought. Categories include research papers, home pages of research organizations, topical current events, and introductory articles. Kelly et al. [11] categorized user needs into task oriented questions and fact oriented questions. Our classification system consists of the following types of information needs - one answer (atomic), one page (jewel), some of all pages, all of the pages, and any related pages (meta search). The following table shows the various information needs with examples. 2

3 Information Need Type One answer (atomic) One page Table 1. Information Needs Information Need Description A very short answer to a question A single document A selection of documents Some of the pages All of the pages Every document matching a criterion Related pages (meta search) Exploratory research 2.2 Web Search Model Example What is/are the telephone area codes for Tucson, AZ? Where is the webpage for WWW conference 2005? Documents about US Policy on North Korea All documents authored by Richard Feynman "I want to learn about RFID. What are the subtopics?" This section describes the model of a web search scenario that takes into account the factors that affect the efficiency of a search by a particular search service. The model is represented in settheoretic terms to illustrate the differences between what a user wants, asks for, and what the user actually gets. The set-theoretic approach was used in describing a similar situation of uncertainty in information retrieval from a scientific database [14]. A discussion of the relevancy of accessed information in the context of an efficient internet search market can be found in [4]. The principal sets in this model are shown in Figures 2 and 3. (1) Let R be defined as the Real World Phenomena that is represented by the current contents of the World Wide Web. The contents of the WWW are continually changing. (2) Let Z be the subset of contents in the WWW that is of interest to the user in this decision problem. Z R. (3) Let Y be the set of contents that has been indexed by the search engine 1. Y R. Since the contents of the WWW change over time, some of the indexed contents are gradually becoming obsolete. Furthermore, the indexed content is an incomplete representation of R because there are new contents added to R since the last indexing and it is unlikely that the search engine will ever have complete coverage of the WWW. The set Y Z represents the set of contents that is 1 It is reported that Yahoo had indexed over 20 billion objects (19.2 billion documents and 1.6 billion images. Google had indexed 11.3 billion objects (8.2 billion web pages and 2.1 billion images). Source: TechTicker, Business Section, San Jose Mercury News, August 9, relevant to the user s decision problem and is available for query retrieval. (4) Let W be the set of contents that is retrieved by the search engine for the user s query. The set Z W represents the relevant contents retrieved. The set ( Z Y ) W represents the set of relevant contents that were not retrieved due to the inadequacy of the query or the search process. This could well be considered opportunities lost to the decision maker. (5) The set W Z represents contents retrieved by the query that are not relevant to the decision problem. These contents can be categorized into two types. First, the set ( R Z) W represents contents retrieved that are current but not relevant. The second set W R represents contents retrieved that are not relevant due to obsolescence and data corruption. Incomplete Indexing R WWW Z Contents of Interest W Retrieved Contents Y Indexed Contents Figure 2. Set diagram showing WWW contents in a search environment Relevant Contents Retrieved Incomplete Retrieval Opportunities Lost R Z W Irrelevant Content (but current) Y Obsolete and Corrupt Content Retrieved Figure 3. Set diagram showing search contents and results 3

4 The ideal search scenario for the decision maker is the one where W Z. Here, the decision maker receives perfect information about the real world phenomena that is relevant to his decision problem. In a realistic situation, the decision maker will receive less than perfect information as a result of the quasi-garbling 2 effects of Querying, Searching and Filtering steps. 2.3 The Decision Making Process The decision making process that is the foundation of our analysis is based on the statistical decision model from Marschak [16,17] and later applied to a computing environment in [13]. In the model, the decision making process is divided into inquiring, communicating and deciding subprocesses with costs associated with each. Marschak also developed the concept of informative-ness based on quasi-garbling to be used in comparison of information structures. In our analysis, information structures is equivalent to modern search services which transform events of the environment into search results. We are applying the concepts here and present the detailed formulation in Appendix A. The decision making model (see figure 1) that incorporates the search environment has the following steps: Decision Problem: The user faces a decision problem and needs information to help with the decision. Query: The user formulates a query for the search service to find information he needs. This may be a simple query consisting of one or more keywords, or an advanced query consisting of keywords as well as operators such as +, -, or quotation marks. The complexity (number of words in the user query, as well as the number of complex operators used) of the query being formulated is influenced by the characteristics of the user such as prior knowledge of the decision domain, and experience using search engines [15]. It is expected that given the same decision making scenario, different users will formulate queries with varying degree of complexity that produce different results of varying quantity and quality under the influence of the factors listed above. In our model, this step is one of the sources of quasigarbling. In other words, the quality of the query 2 The effect is called quasi-garbling because the result is not garbled (not readable) per se. But the result is not in its original form and is difficult to process because noises of various forms have been introduced. This noisy information could detract the decision maker from making the right decision. could potentially reduce the quality of the output in terms of informative-ness to the decision maker. It is also assumed that there are costs associated with this sub-process such as cost of training, formulation and debugging. The term debugging is used here to describe what the user does to formulate a query that he considers correct to retrieve the desired results. This might require more than one attempt. This activity is different from reiterating where the user refines his query in multiple passes to get better results. The correctness and fineness of the query have direct effects on the quality of returned results from the search engine. For example, a query might not be directing the search engine correctly and result in retrieving not relevant results ( W Z ). Another example is that due to the fineness (or not so fineness) of the query, the results miss relevant contents( Z Y ) W ). The effect of the query on the quality of the results in the form of quasigarbling is represented in the economics model as in section A.4. Q The goal of our experimental design is to measure and analyze the influence of the various factors on query complexity. In particular, we focus on the four factors: search engine, information need, experience with using search engines and domain knowledge. We include these factors in our experimental hypotheses in order to test their impact. Search Service: Once the user s query is submitted to the search service, the search engine is deployed to processes the query, execute the underlying algorithms to return results to the user. Each search service can be characterized by how it spiders the Web contents, how often it performs the spidering, its indexing algorithm, its internal organization, its ranking method, and user interface (format of presenting results). A review of past literature reveals that for the same query different search engines could produce different results of varying quantity and quality [8]. The search engine with its characteristics is thus another source of quasi-garbling. This is represented as S in section A.4. Search engine quality is experimentally evaluated as a source of noise in a user s search experience. Filtering: When the user is presented with the results, the user filters the results in order to evaluate the quality of the results as related to the decision problem. In other words, the user tries to find results relevant to the decision domain. Depending on user characteristics such as those described in the Query step, the filtering is a source 4

5 of quasi-garbling which produces results that could vary from user to user. The results of the query being returned to the user contain both organic or natural results as well as paid placements in the form of sponsored links and advertisements [18]. The user will have to spend considerable time and effort in filtering out the relevant information from the irrelevant using experiential knowledge as well as the specific information need. Our assumption is that ease of filtering is impacted by user characteristics such as domain knowledge and experience with search as well as factors such as quality of search results, proportion of organic results and paid placements, and the design of the user interface. Our experimental design focuses on testing the impact of search quality, domain knowledge and search engine on the ease of filtering. The effect of filtering in detracting the user from relevant results in the form of quasi-garbling is represented in the economics model as F in section A.4. Reiterating: If the user is not satisfied with the results, he can refine his query and seek better results by going back to the Query step. Deciding: Based on the information filtered from the search results, the user makes the decision to solve the problem based on the relative benefits of the alternatives (see A.26 in Appendix A). The formulation of the economics of decision in sections A.2 and A.3 leading to (A.26) depended on only a single source of information. That is, an information structure with quasi-garbling effects from Query, Search and Filtering formulated (repeated here) as: (A.32) zw Q In the modern Web search scenario, a decision maker is faced with the easy availability of multiple search services that could be employed as his information structure. The formulation for comparing the informative-ness of information structures is given in section A.3. From (A.31), the decision maker s choice of services is thus an economic decision based on: (1) B(, a ) max B( a ) In this research project, the set of information structures representing different search services is (2) G Y M {,, } where GGoogle YYahoo MMSN S F 3. Experimental Design A set of hypotheses was developed to test our assumptions about the relationship between user experiential factors, search results, and the effect of quasi-garbling introduced in the search process. An online instrument was created to collect data about users and their search experience. A preliminary experiment using 20 undergraduate students (juniors and seniors with age ranging from 18 to 22) majoring in Management Information Systems as subjects was conducted. Data collected from this pre-test was used to validate the instrument and the experimental design. In the experiment the subjects were asked to compare their experience with Google, Yahoo and MSN since these are the leaders in terms of number of web pages indexed (see Footnote 1 and [7]). Three scenarios representing different information needs were presented to the subjects and they were asked to formulate one query for each scenario for each search engine. The data collected from this limited sample were not significant enough to test the hypotheses but the conduct of the experiment provided us with very valuable feedback about the experimental design and the construction of the scenarios. We intend to conduct larger scale experiments in the fall of 2005 to collect significant data to test our hypotheses. 3.1 Research Hypotheses In section 2 we have developed a model depicting how noise could be introduced in each step of a user s search experience: the query formulation (Q), the characteristics of the search services (S), and the filtering of the results returned by search (F). In the first stage, we hypothesize that user experiential factors of search service experience and domain knowledge, compounded with the information need of the decision problem would influence the complexity of the query being formulated. Conceivably Q is composed of both the number of and the actual terms and operators used in the query. We have adopted query complexity (number of terms and operators) as a surrogate measure of Q because it is impossible to compare the actual terms used among the subjects. We also wanted to find out whether these influences would be different across the search services being investigated. In the second stage, we can only treat the search services as black boxes and test whether the same query would produce results with different quality as measured by First 5

6 Relevant Document Rank, Jewel Measure and from different search services (these measures are explained in more detail in the next section). Search result quality is used as a surrogate measure of S. In the third stage, we hypothesize that the same factors as in the first stage (search engine experience, domain knowledge and information need) would influence the ease of filtering of the results. Ease of filtering is used as a surrogate measure of F. We also wanted to ascertain whether the ease of filtering is affected by the user interface presented by each of the search services. The following null hypotheses are formulated for the three stages: 1. Hypotheses related to query formulation: There will be no variance in query complexity among subjects across the scenarios. There will be no variance in query complexity among subjects across the scenarios for the same search service. People use the same query complexity regardless of information need when using a particular search service. There will be no variance in query complexity among subjects across search services for a given scenario. People use the same query complexity for the same scenario regardless of search service. There will be no variance in query complexity among subjects for the same scenario based on level of search service experience. There will be no variance in query complexity among subjects for the same scenario based on level of domain knowledge about the scenario. 2. Hypotheses related to quality of search service performance There will be no variance in search result quality among search services for a given scenario. 3. Hypotheses related to filtering of search results Given a particular scenario, there will be no relationship between the user s domain knowledge and their ability to separate out relevant content ( Z W ) from irrelevant content (W Z). Given a particular scenario, there will be no difference between the ease of filtering across search services. The user s ability to filter results will not be related to their experience with a particular search service. 3.2 Experiment Details On the first screen of the survey instrument, subjects were asked to enter experiential factors such as their major or discipline, their year in school, and a self appraisal of their experience with each of the search engines (Google, Yahoo and MSN) Figure 4. Survey instrument (screens 1-3) For the purpose of this study, we took into account the first three information need types described in Section 2 atomic, one page, and some of the pages. We did not take into account the all of the pages information need because it is not possible to compute recall on the Web. The reason we did not consider the related search or the meta-search information need type is that it is difficult to come up with an objective measure to evaluate the results. One scenario was constructed for each information need: Atomic: Which are the three most populous cities in California? One Page: You plan on visiting India and want to find the official web page that describes the US Government s recommendations for visiting India. Some of the pages: You recently adopted a Labrador Retriever. You want to find titles of books that you could purchase to learn how to train your dog. On the second and third screens, subjects were presented with these search scenarios. They were then asked to rate their prior domain knowledge about the scenarios, and asked to construct queries for each of the three search engines. Although the expectation was that the users would submit the same query to all three engines, they were still asked to enter the query once for each engine. These queries would be analyzed later to determine 6

7 whether different queries were used for different engines for the same scenario, demonstrating some prior knowledge about the behavior of specific engines. On the fourth screen, the subjects were presented with results returned by the search engine for a canned standard query. The subjects were asked to rate their experience with the search, how much time was spent, where (which page and which rank) they found the answers, etc. The reason for using a canned query instead of the user s query was that in order to compare users filtering ability, the results had to be held constant for all users. Figure 5: Survey instrument (screen 4 for each engine) On completion of the experiment, the appropriate measures were computed in order to evaluate the performance of each engine for each user query for each scenario. Some of the measures selected for comparison are described below: First Relevant Document Rank (FRDR): Rank of the first document containing the piece of information sought by the user. In other words, once the user reads this document, they can terminate their search. This measure was used to evaluate the performance of the one answer or atomic information need. Jewel measure: Rank of the document considered most appropriate for the user s query. In other words, it is the rank of search result pointing to the document sought after by the user. This measure was used to evaluate the performance in case of the one page information need. 10: Since research has shown that users tend to view only one or two pages of results returned by a search engine, we have considered only the first 10 results for computing precision. This measure was used to evaluate the performance in case of the some of the pages information need. 3.3 Preliminary Test Results Although we had a small sample size in the pretest (20), we were able to perform some analysis of variance tests on the results based on the hypotheses. The statistical tests although not for reporting significance, helped us verify that the data collected from the instruments support the testing of the hypotheses. We were able to see some patterns indicating that, with a proper sample size, some interesting conclusions could be drawn. For example, there were differences in the query complexity measure among scenarios but not among search services. The subjects demonstrated their unfamiliarity with the new MSN search service by rating it low in ease of filtering. In certain cases, the subjects wrote simpler queries when they were not familiar with the domain of the scenario. The pre-test also provided valuable feedback to us about the instrument and the conduct of the experiment. For example, the subjects indicated that they did not understand clearly the meaning of the term page rank. Although the subjects were asked to evaluate the ease of filtering the results, it was difficult to record which documents retrieved were deemed relevant by the subjects. A surprising incident showed the unpredictability of the web. Some web sites high-jacked the functionality of the browser (e.g., erasing the browsing history) so the subjects were unable to return to the evaluation page of the instrument. We also realized that we would need to conduct the experiment with groups of subjects with different search experience for comparison purposes. 4. Conclusions and Future Work This paper presents an economic model and a methodology to allow users to compare search services. Different search services (with their search engines) usually return different results with the same query based on the characteristics of the search engine s algorithms and the extent of captured data. The methodology presented here can be useful to search service providers to enhance alignment between their objectives and that of the users as well as for competitive pricing of their products and services. A pre-test was conducted in order to perform a realistic comparison among three popular search 7

8 services. The results of the pre-test helped validate the survey instrument and obtain preliminary feedback. An enhancement of the instrument is being undertaken now to automatically capture the amount of time the user took in formulating the query and in filtering in order to capture a surrogate measure of search cost. Tools were also developed to speed up evaluation of search results for individual user queries. Preliminary tools were created, and are now being refined. Future work includes refining the instrument using the lessons learned, administering the instrument to a larger sample to get significant results, and using measures of higher accuracy to evaluate the performance of search services for specific information needs. Examples of additional measures that we are investigating include a measure that takes into account the rank of the last relevant document in the results list, the number of page flips required in order to get to the answer or the required page, usefulness of the summary of the document presented in the documents results list, and the impact of paid placements on search quality. 5. References [1] Belkin, N. et. al. Information filtering and information retrieval: two sides of the same coin?, Communications of the ACM, 35(12), 29 38, December [2] Buckley, C., and Voorhees, E. M. Evaluating evaluation measure stability, Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July [3] Chau, M. Posters: Applying web analysis in web page filtering, Proceedings of the 4th ACM/IEEE-CS joint conference on Digital libraries, June [4] Choi, S., Stahl, D., and Whinston, A.B. The Economics of Electronic Commerce, MacMillan Technical Publishing, [5] Eastman, C. M., and Jansen, B. J. Coverage, relevance, and ranking: The impact of query operators on Web search engine results, ACM Transactions on Information Systems (TOIS), 21(4), October [6] Glover, E. J., Lawrence S., Birmingham, W. P., and Giles, L. C. "Architecture of a metasearch engine that supports user information needs", Proceedings of the Eighth International Conference on Information and Knowledge Management, November [7] Gulli, A., and Signorini, A. Posters: The indexable web is more than 11.5 billion pages, Special interest tracks and posters of the 14th International Conference on World Wide Web, May [8] Hawking, D., Craswell, N., Bailey, P., and Griffiths, K. Measuring search engine quality. Information Retrieval, Journal of Information Retrieval, 4 (1), 33-59, [9] Jansen, B. J., Spink A., Bateman J., and Saracevic T. Real life information retrieval: a study of user queries on the Web, ACM SIGIR Forum, 32(1), [10] Kauffman, R. J. and Walden, E. A. Economics and Electronic Commerce: Survey and Directions for Research, International Journal of Electronic Commerce, 5 (4), 5-116, Summer [11] Kelly D., Yuan X., Belkin N. J., Murdock V., and Croft B. W. "Poster session: Features of documents relevant to task- and fact- oriented questions", Proceedings of the Eleventh International Conference on Information and Knowledge Management, November [12] Kuhlthau, C. C. Inside the Search Process: Information Seeking from the User's Perspective. Journal of the American Society for Information Science, 42 (5), , [13] Kwan, S. K. A Job Contractor Function in Computing System Scheduling, PhD in Management dissertation, Graduate School of Management, UCLA, [14] Kwan, S. K., Olken, F. and Rotem, D. "Uncertain, Incomplete and Inconsistent Data in Scientific and Statistical Databases", in Motro, A. and Smets, P. (ed.) Uncertainty Management in Information Systems: From Needs to Solutions, Kluwer Academic Publishers, Norwell, [15] Lucas, W. and Topi, H. Form and function: The impact of query term and operator usage on Web search results, Journal of the American Society for Information Science and Technology, 53 (2), , November [16] Marschak, J. Economics of Inquiry, Communicating, Deciding, American Economics Review, 58, 1-18, May [17] Marschak, J. and Radner, R. Economic Theory of Teams, Cowles Foundation Monograph 22, Yale University Press, [18] Moran, M and Hunt, B. Search Engine Marketing Inc. Driving Search Traffic to Your Company s Web Site, IBM Press, Upper Saddle River, New Jersey, July

9 [19] Shah, C., and Croft, B. W. Opening session: Evaluating high accuracy retrieval techniques, Proceedings of the 27th Annual International Conference on Research and Development in Information Retrieval, July [20] Tague-Sutcliffe, J. "Measuring the informativeness of a retrieval process ", Proceedings of the 15th annual International ACM SIGIR Conference on Research and development in information retrieval, June [21] Vaughan, L. New measurements for search engine evaluation proposed and tested, Information Processing & Management, 40 (4), , May Appendix A - Statistical Decision Model (from [9] and [12]) A.1 Noiseless Information (A.1) x,..., x ) (A.2) W ( w 1,..., w n ) X Set of states of the ( 1 m (A.3) a,..., a ) (A.4) (x) ( 1 n environment Set of messages received by the decision maker A Set of possible actions to be taken by the decision maker π Prior probability of state x (A.5) α ( W ) a Decision maker s strategy upon receipt of signals W ( a, x) b (A.6) β Decision maker s benefits in utility units, is the benefit function ( x ) w (A.7) Noiseless information structure (A.8) B B( α; π, β ) π ( x) β ( α( ( x)), (A.9) x V ( ) Max B α ( α; π, β ) Expected benefits; Where and are controlled by the decision maker, and characterize the decision maker Value of information (A.10) K (x) Costs of inquiry in utility units (A.11) (x) K Costs of decision in α utility units (A.12) K( α) π ( x)( K ( x) + K x α ( x)) Total expected costs, K is the cost function (A.13) U ( α) B( α) K( α) Net expected benefits, and are dropped for brevity A.2 Random States, Events and Information with Noise Partition the set X of the states of nature into W consisting of subsets of X called messages and Z consisting of subsets of X, called events (see set diagrams in Figures 1 and 2). Then redefine as depending on actions and events z (but not on messages w): (A.14) b β ( a, z) (A.15) Let, p ( z, w) Joint probability associated with intersection of z and w (A.16) Prior event probability π ( z ) w p ( z, w) (A.17) q ( w) z p ( z, w) (A.18) p( w z) p( z, w) / π ( z) (A.19) p ( w z) 1. or 0. (A.20) p ( z w) p( z, w) / q( w) (A.21) B B( α; π, β ) z w π ( z) z w zw Message probability Conditional probability of w given z If w is noiseless with respect to z, see section A.1 above Posterior event probability π ( z) p( w z) β ( α( w), z)) β ( α( w), z) Thus, of (A.7) is treated as an m x n Markov matrix with properties of: (A.22) p ( w z ) 0. ij j i for all z w zw (A.23) 1. Given any fixed zw, the information structure, then the decision maker s optimal strategy is defined as: (A.24) B ( α ) Maxα B( α ) V ( ) A procedure to find is to find, for each message w the appropriate response 9

10 (w) a w, the action that maximizes the conditional expectations of (a,z), given the message w: (A.25) Maxa p( z w) β ( a, z) z p( z w) β ( a z) z w Then the resulting expected benefit is obtained by summing (A.25) over all w in W, (A.24) becomes; (A.26) From (A.20) B( α ) q ( w ) w) z p ( z w ) β ( a z w z w, ( p( z, w) / q( w) β ( a w z w, z p (, w) β ( a w z w, z) q ) From (A.19) (A.22) π ( z ) β ( a, z) w z zw w A.3 Comparison of Information Structures An information structure is more informative than another if it tells more about the environment to the decision maker through its signals, thus resulting in better benefits to the decision maker. In comparing information structures, the concept of informativeness is usually used. The following theorem is due to David Blackwell. Theorem A.1: [ zw ] than [ z w ] Markov matrix [ ] is more informative if an only if there exists a such that Q Q Q ww ). The transformation performed by Q is called quasi-garbling. Proof: Let a w and a w denote the optimal actions in response to message w and w respectively (see (A.25)). Then for each we have: (A.27) (, α ) B π ( z ) β ( a, z) w z z w w π ( z) β ( a, z) w z zw B (A.28) ( α ) π ( z ) β ( a, z) w z zw w π ( z) β ( a, z ) w z zw all a in A all a in A Since a w and a w are both in A, from the two equations above: (A.29) B (, α ) π ( z) w z zwβ ( aw, z) Q ; that is: Now, since (A.30) zw w zwqww We then have: (A.31) B(, α ) π ( z) zwqww w π z) w zw 1. π ( ) β ( a w z zw w, β ( a w w z w w w z w B( α ), z) Q ( β ( a, z) z z) by ( A.29) The inequality holds for any and ; it thus proves that is more informative than. A.4 Application of the Statistical Decision Model to Modern Web Search A model of modern web search was described in section 1 of this paper. An information structure with noise as defined in A.22 and A.23 is in effect the depiction of a web search service in the model. A web search does not produce perfect information, it returns a mix of relevant and irrelevant information to the user. It was shown in section 2 that the query, search and filtering sub-processes contribute to the quasigarbling of information retrieved from a search service. That is, they contribute to the make-up of the information structure described in A.22 and A.23, In effect, (A.32) zw Q where QQuery, SSearch and FFiltering. Each of the three steps moves the decision maker further away from receiving perfect information. Given A.32, the cost of performing a web search is then made up of the cost of each of the steps (cf. A.10). Theorem A.1 defined how information structures can be compared based on their informativeness. In the modern web search model, a comparison can be made among search services using Theorem A.1 where each search service is defined uniquely by its own zw. A decision maker should choose the search service that will provide the highest net benefits (A.26). S F 10