Beyond Ten Blue Links Seven Challenges

Transcription

1 Beyond Ten Blue Links Seven Challenges Ricardo Baeza-Yates VP of Yahoo! Research for EMEA & LatAm Barcelona, Spain Thanks to Andrei Broder, Yoelle Maarek & Prabhakar Raghavan Agenda Past and Present Wisdom of Crowds Current Challenges Query Assistance Universal Search Post-search Experience Application Integration Future Contextualization Web of Objects Implicit Search 1

2 History of Web Search Generation Technology Wisdom First: Classical IR Writers Second: Third: Link Analysis +Anchor text +Click-through voting +Usage data mining +Webmasters +Readers Everyone Fourth: 2008-?? +Query intent detection +Learning to rank Everyone Today: Internet and the Web Between 1 and 2.5 billion people connected 5 billion estimated for billion mobile phones At least 500 million had mobile broadband in 2010 Internet traffic has increased 20x in the last 5 years More than 500 million Web servers The Web is in practice unbounded Dynamic pages are unbounded Static pages: over 50 billion? Boom of Social Media and UGC 4 2

3 Today: Search Rectangle Very little differences between major search engines A rectangle text box for your queries Other forms of rectangles? Embedded in a portal Always here in a toolbar Ultimate rectangle: omnibox Today: Web Content Quantity Usergenerated Traditional publishing Quality 3

4 Today: Trends User Generated Content Massive (quality vs. quantity) Social Networks Real time (people + physical sensors) Impact Fragmentation of ownership Fragmentation of access (longer tail) Fragmentation of right to access Viability Business model based in advertising Search is Evolving Already, more than a list of docs Moving towards identifying a user s task Enabling means for task completion New experiences based on the Web 2.0 Permanent challenges: on-line, scalability 4

5 The Wisdom of Crowds James Surowiecki, a New Yorker columnist, published this book in 2004 Under the right circumstances, groups are remarkably intelligent Importance of diversity, independence and decentralization Aggregating data large groups of people are smarter than an elite few, no matter how brilliant they are better at solving problems, fostering innovation, coming to wise decisions, even predicting the future. 9 Geo-tagged Photos in Flickr 5

6 Yahoo! Clues The Wisdom of Crowds Popularity Diversity Quality Coverage Long tail 6

7 The Head of the Wisdom Heavy Tail of User Interests Many queries, each asked very few times, make up a large fraction of all queries Movies watched, blogs read, words used One explanation Interests People Normal people 14 Weirdos 7

8 Personal distribution has a heavy tail Many queries, each asked very few times, make up a large fraction of all queries Applies to word usage, web page access, We are all partially eclectic The reality Interests People 15 Broder, Gabrilovich, Goel, Pang; WSDM Query Assistance Related queries Spelling correction Query suggestions Instant previews What the user would like to see? 8

9 11/28/11 Y! Search Assist Contextualization Context: Local: geography, language, Person: do we know the user? enough data? Social Task Personalization: Data volume vs. privacy Contextualization: Small crowds What is the right interface? 9

10 Zipf: The Principle of Least Effort Data per user is a power law 10

11 Contextualization Challenges We are far from being done with innovation in search engines Large scale usage data is key BUT Usage data at a very large scale over larger and larger populations over longer and longer periods of time Personalization More data via larger communities, makes data less personalized wisdom of crowds does not work well on small corpora Privacy Over personalization endangers privacy Long-term logs endanger privacy Universal Search What sources and media to show? How many results from each source? How to rank mixed media? How to display the results? Aggregated Search 11

12 More Information in One Search Shortcuts Deep Links Enhanced Results 4. Web of Objects We move from a Web of Pages to a Web of Objects Objects are people, places, businesses, restaurants (named entities) Objects have attributes Missing, noisy, etc. Intents are satisfied by presenting objects and attributes Attributes define faceted search 24 12

13 Research Challenges Crawling objects Object extraction (entities) Object disambiguation Object consolidation Object normalization Object indexing Object ranking Object visualization 25 Time Explorer Finding Relations among Entities in News Past, present or future! Baeza-Yates, Searching the Future, The clue is the interface Part of the Living Knowledge EU project Winner of the HCIR 2010 Challenge New York Times collection ( ) Found many interesting examples Generates new NLP research problems 26 13

14 Time Explorer ((c) Timelijne with entity trends Time Explorer 14

15 5. Post-search Experience User feedback (like, +1, ) Enhanced results Faceted search Sharing Translating How to manipulate and enhance results? Application Integration Integrate third-party applications in the user experience Trigger applications based on query intent Example: Yahoo! QuickApps When and how to trigger? Which application to trigger? App Market 30 15

16 7. Implicit Search Solve the task searching for the user Recommendations Enable related things Search as a back end process that is triggered depending on the context Writing Browsing news How to predict well? How to do it well? 31 Conclusions Web search is no longer about document retrieval Means for web-mediated goals New breed of search experiences Demands search ecosystem combining content with intent Exploiting the Wisdom of Crowds behind the Web 2.0 Contextualization versus personalization Optimize common tasks Move away from privacy issues 16

17 The New Frontiers 33 Front-end and user experience The most probable reason for users to switch between quasi-equivalent engines is a better user experience Depart from the rectangle/ranked list paradigm Get rid of queries? Implicit search Content delivery is one flavor But in general, why should we even have to formulate a query? What s next? Fourth generation From Information Retrieval to Information Supply Explicit demand for information driven by a user query Increase use of context Active information supply driven by user activity and context 34 17

18 Second edition appeared in 2011 Questions?