Recent advances in computational advertising: design and analysis of ad retrieval systems Evgeniy Gabrilovich

Transcription

1 Recent advances in computational advertising: design and analysis of ad retrieval systems Evgeniy Gabrilovich 1

2 What is Computational Advertising? A new scientific sub-discipline that provides the foundation for building online ad retrieval platforms To wit: given a certain user in a certain context, find the most suitable ad At the intersection of Large scale text analysis Information retrieval Statistical modeling and machine learning Optimization Microeconomics Technologies described might or might not be in actual use at Yahoo! 2

3 Computational Advertising at Yahoo! Research 3

4 Online advertising spending 4

5 Textual advertising 1. Ads driven by search keywords Sponsored Search (a.k.a. keyword driven ads, paid search, etc.) 2. Ads directly driven by the content of a web page Content Match (a.k.a. a context driven ads, contextual ads, etc.) Textual advertising on the Web is strongly related to NLP and information retrieval 5

6 Sponsored search Text-based ads driven by a keyword search 6

7 Content match ads Text-based ads driven by the page content Content t match ads 7

8 Anatomy of an ad Title Creative Display URL Bid phrases: {SIGIR 2010, computational advertising, Evgeniy Gabrilovich,...} Bid: $0.10 Landing URL: utorials/sigir10_compadv Landing page 8

9 So when do advertising dollars actually change hands? CPM = cost per thousand impressionsi Typically used for graphical/banner ads (brand advertising) CPC = cost per click Typically used for textual ads CPT/CPA = cost per transaction/action a.k.a. referral fees or affiliate fees 9

10 Beyond keyword matching Matching ads is relatively simple for explicitly bid keywords What about queries on which there are no bids? Advertisers should be able to bid on broad queries and/or concept queries Advertisers need volume the total amount of searches on bid phrases is not enough! Suppose your ad is Good prices on Seattle hotels Naïve approach: bid on any query that contains the word Seattle Problems Seattle's Best Coffee Chicago Alaska cruises start point Ideally: bid on any query related to Seattle as a travel destination 10

11 The old school: heuristic ad matching Sponsored search Exact match between the query and the bid phrase of the ad (modulo simple normalization, e.g., stemming) Advertisers cannot possibly bid on all relevant queries (especially rare ones) Use advanced match (e.g., through query-to-query rewrites) Content match Extract bid phrases from pages, thus reducing the problem to exact match Both essentially perform record lookup 11

12 The old school (cont d) Query Front end Abbey Road lyrics Simplistic query expansion Query Query rewriting module Query rewrites Ignoring (or underusing) the multitude of information available Exact match Candidate ads Revenue reordering Ad slate 12

13 The new approach: knowledge-based ad retrieval Ad indexing and scoring based on all the information available (bid terms, title, creative, URL, landing page,...) Similar to document indexing in IR Use standard IR tools (text preprocessing tokenization, stemming, entity extraction; inverted indexes etc.) Use multiple features of the query and the ad Elaborate query expansion 2 nd pass relevance reordering (re-ranking) Using features not available to the 1 st pass model (e.g., set-level features, click history) 13

14 The new approach (cont d) Query Miele Rich query Front end Ad query generation Ad query <Miele, appliances, kitchen, appliances repair, appliance parts, Business/Shopping/Home/Appliances> The hidden parts of ads (bid phrases + landing pages) allow us to augment the ads (cf. query expansion) Ad search engine First pass retrieval Relevance reordering Candidate ads Revenue reordering Ad slate 14

15 Research questions Can we generate bid phrases (or even entire ad campaigns) automatically? How to index the ad corpus? How to select relevant ads? Should we show ads at all? What is the interplay between the organic and sponsored results? Should we use the landing for indexing? Can we optimally choose the landing page? 15

16 How to select relevant ads? Feature generation for improved ad retrieval (SIGIR 2007, w. Broder et al.; ACM TWEB 2009, Gabrilovich et al.) 16

17 Query classification using Web search results Humans often find it hard to readily see what the query is about But they can easily make sense of it once they look at the search results Let computers do the same thing Infer the query intent from the top algorithmic search results ( pseudo relevance feedback ) Classify search results (either summaries or full pages) Let these results vote to determine the query class(es) in a large taxonomy of commercial topics Our goal: Construct additional features to retrieve better ads 17

18 Example: ex560lku CATEGORIES 1. Computing/Computer/ Hardware/Computer/Peripherals/Computer Modems 18

19 If we know it is about actiontec usb modem then we have plenty of ads 19

20 Our approach Traditional approach: Query Classifier Insufficient data Our approach: Query Search engine Very large scale Search results Classifier Using Web as external knowledge Pre-classify all pages just once! 20

21 Research questions Snippets or full pages? Number of search results to obtain Number of classes per search result Aggregation: bundling or voting? 21

22 The effect of using Web search results 22

23 Beyond the bag of words: matching textual ads in the enriched feature space (SIGIR 2007, Broder et al.; CIKM 2008, w. Broder et al.) 23

24 What can we do about non-english queries? CIKM 2008, w. Wang et al.; WSDM 2009, w. Wang et al.) Developing a taxonomy and building a query classifier for every language is prohibitively expensive Solution: apply off-the-shelf MT to the search results in the source language g Very short text Machine Translation Sufficiently long text 24

25 The effect of query expansion prior to applying MT. The gap for infrequent queries is wider Baseline = translate the query (using MT), then classify the result as an English query (Head) more frequent (Tail) less frequent 25

26 How to index the ad corpus? The Anatomy of an ad: Structured indexing and retrieval for sponsored search (WWW 2010, w. Bendersky et al.) 26

27 Structure of online ad campaigns: the ad schema Advertiser Buy appliances on Black Friday New Year deals on lawn & garden tools Account 1 Account 2 Kitchen appliances Campaign 1 Campaign 2 Ad group 1 Ad group 2 Creatives Brand name appliances Compare prices and save money Ad Bid phrases { Miele, KitchenAid, Cuisinart, } Can be just a single bid phrase, or thousands of bid phrases (which are not necessarily topically coherent) 27

28 Implications of the campaign structure What is the appropriate indexing unit? Cartesian product of creatives and bid phrases? Ad group? Leveraging information from higher levels to address data sparsity at children nodes What is the right approach to document length normalization? Large variability of document lengths Probability of shorter documents (smaller ad groups) to be retrieved is higher than their probability of being relevant How to index and score templated ads? Prior work mostly considered ads as independent atomic units and ignored hierarchical campaign structure 28

29 Possible approaches 1. Term index (Cartesian product of all creatives and bid terms) Huge index, small focused documents 2. Creative index (a creative is coupled with all the bid terms in the ad group) Two-stage retrieval (first choose the creative, then pick the term) Bid terms are duplicated across creatives 3. Ad group index Indexing units are entire ad groups Three-stage retrieval (first choose the ad group, then the creative, and finally pick the term) Most compact index 29

30 Retrieval speed vs. relevance Arewetrading effectiveness for efficiency? Term index yields most relevant ads, yet is least efficient (20x slower than the ad group index) Ad group index is most efficient (2x faster than creative index), yet least effective 30

31 Using learning to rank techniques: structured re-ranking ranking Step 1: Retrieve an initial set of candidates using the ad group index Step 2: Re-rank the candidate set using structural features (instead of ignoring the structure and scoring creatives and terms independently) Ad group score, creative-term pair score # bid terms in the ad group Unigram entropy (cohesiveness) of the ad group Ratio of query words covered by the ad group text Fraction of the titles / terms / URLs that contain at least one query term Other features are possible! feature functions 31

32 Re-ranking retrieval performance Len 1 (143 queries) Len 2-3 (443 queries) Len 4+ (187 queries) Term index Structured t re-ranking (+ 0.95%) (+ 2.1%) (+ 4.6%) Structured re-ranking is superior for all query lengths Most notable improvements are obtained for longer queries Still very efficient! 32

33 To swing or not to swing: learning when (not) to advertise (CIKM 2008, w. Broder et al.) Repeatedly showing non- relevant ads can have detrimental long-term effects Should we show ads at all? Want to be able to predict when (not) to show individual ads or a set of ads ( swing ) Modeling actual short- and long-term costs of showing non-relevant ads is very difficult 33

34 Thresholding approach Decision made on individual ads based on ad scores Set a global score threshold Only retrieve ads with scores above it If none of the ad scores are above the threshold, then no ads are shown ( no swing ) Scores are not necessarily comparable across queries! 34

35 Machine learning approach Decision made on sets of ads based on a variety of features Learn a binary prediction model ( swing / no swing ) for sets of ads If we swing, then all ads are retrieved If we do not swing, then no ads are retrieved Features defined d over sets of ads, rather than individual ads 35

36 Features Relevance features Word overlap, cosine similarity between ad and query/page Vocabulary mismatch features Translation models PMI between query/page terms and bid terms Ad-based features Bid price (higher bids may indicate better ads) Result set cohesiveness features Coefficient of variation of ad scores (std/mean) Result set clarity If the set of ads is very cohesive and focused on 1-2 topics, the relevance language model is very different from the collection model Entropy 36

37 What happens after an ad click? Quantifying the impact of landing pages in Web advertising (CIKM 2009, w. Becker et al.) Can we optimally choose the landing gpage? 37

38 Conceptually: context transfer Search engine result page Click! Landing page Conversion (e.g., purchase of the product or service being advertised) User s activity on the advertiser s Web site 38

39 All landing pages are not created equal (and neither are the corresponding conversion rates) We propose a concise taxonomy of landing page types: I. Homepage (25%) top-level page of the advertiser s site (e.g., Verizon.com) II. Category browse (37.5%) main page of a sub-section section of the advertiser s site, which describes a category of related products III. Search transfer (26%) search within the advertiser s site OR on other Web sites IV. Other (11.5%) terminal pages (e.g., promotion pages or forms) 39

40 Examples: Homepage 40

41 Examples: Category browse 41

42 Examples: search transfer 42

43 Landing page classifier Features: bag of words, HTML patterns [ST] search results, found [CB] Home > Verizon > LG phones [HP] HTML overlap between given URL and base URL [O] ratio of form elements to text, few outgoing links Accuracy on the pilot dataset (10-fold xval): 83% Accuracy on additional 100 labeled pages: 80% Distribution of landing gpage types in a set of 20,000 landing pages from Yahoo! Toolbar logs: Homepage Search Category Other Transfer Browse 34.4% 22.3% 36.0% 7.3% 43

44 Using the landing page taxonomy Picking the right landing page type for each ad Improving the conversion rate Improving advertisers ROI! 44

45 Landing page type usage vs. conversion: breakdown by query frequency Navigational queries Category and search transfer become more popular p for rare queries Observed conversion rates are in sharp contrast with usage frequency of the different page types 45

46 Landing page type usage vs. conversion: breakdown by query price Category and search transfer are dominant for cheaper queries As the price goes up, so does the conversion rate (higher quality pages?) 46

47 What is the interplay between the organic and sponsored results? Competing for users attention: On the interplay between organic and sponsored search results (WWW 2010, w. Danescu-Niculescu-Mizil et al.) 47

48 The interplay between ads and organic results... in an information-rich world, the wealth of information means a dearth of something else: a scarcity of whatever it is that information consumes. What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it. -- Herbert Simon, Designing Organizations for an Information-Rich World, Is there competition for clicks between ads and organic results? Do users prefer ads that are similar to the organic results, or do they prefer diversity? We found that the nature of this interplay depends on the type of the query 48

49 Relation between the CTR of ads and the CTR of organic results Negative correlation (competition) Users are only willing to spend limited time and effort on each query Positive correlation (depends d on the quality of results) Easy query ( online radio ) decent ads and organic results clicks on both Hard query ( who is giving this talk? ) poor results on both sides no clicks on either Independence (null hypothesis) Users consider ads and organic results as two independent sources of information 49

50 Findings: competition + positive correlation 50

51 Decoupling the forces Users are willing to invest limited effort in each query competition In order to single out the competition effect, we tried to explicitly model the amount of effort the user is willing to invest Low effort = navigational i queries [Broder, 2002] (27% of queries) Pandora radio, Bank of America High effort = non-navigational queries Meaning of life, academia vs. industry 51

52 Competition clearly exists for navigational queries We also examined different degrees of navigationality: the less navigational the query is, the less competition we observed 52

53 Another viewpoint: Do users prefer ads that are more similar to the organic results or more diverse ads? Both have been argued for in prior work Preference for similarity Ads are more likely to be relevant This assumption is often made in query expansion for advertising i [Broder et al., 2008] Preference of diversity Diversity among organic search results has often been shown to be desirable (e.g., entire session on diversity WWW 2010) 53

54 We found evidence for users preferring both diversity and similarity So we need to dig deeper again... Overlap measured using the Jaccard coefficient between titles of ads and organic results 54

55 Let s break down by navigationality again 55

56 Break down by navigationality (cont d) 56

57 Counterintuitive? 57

58 Responsive and incidental ads Responsive ads directly address the user ss information need More likely to be similar to the organic results Incidental ads are only somewhat related to the user s information need Unreasonable as organic results, but ok for ads More likely to be different from the organic results Example: query = free internet radio Responsive: Pandora Internet Radio Incidental: Discount Bose Computer Speakers 58

59 Now it all make sense... Using the features that quantify this interplay, we improved the accuracy of CTR prediction by 5% 59

60 Summary 1. The financial scale is huge 2. Advertising is a form of information 3. Finding the best ad is an information retrieval problem Multiple, possibly contradictory utility functions Classical IR needs significant adaptation 4. The optimal solution requires extensive use of external knowledge 60

61 Thank you! 61

62 This talk is Copyright Yahoo! Yahoo! and the Author retain all rights, including copyright and distribution rights. No publication or further distribution in full or in part is permitted without explicit written permission. The opinions expressed herein are the responsibility of the author and do not necessarily reflect the opinion of Yahoo! Inc. This talk benefitted from the contributions of many colleagues and co-authors at Yahoo! and elsewhere. Their help is gratefully acknowledged. 62