Analysing Search Trends

Size: px

Start display at page:

Download "Analysing Search Trends"

Leslie Crawford
5 years ago
Views:

1 Data Mining in Business Intelligence 7 March 2013, Ben-Gurion University Analysing Search Trends Yair Shimshoni, Google R&D center, Tel-Aviv. shimsh@google.com

2 Outline What are search trends? The Google Trends tool Search Trends Analysis Using Search Trends for Business Intelligence Nowcasting with Google Trends & Google Correlate

3 Search Trends More than a billion people around the globe are searching the web using search engines. Through search, users express their interest: to know, to see, to hear, to do, to buy, to sell, to compare, to study, to travel,... Therefore, the aggregated search patterns of Internet users w.r.t. various terms convey valuable information on people s interests, preferences and intentions. These search patterns are represented by time series called Search Trends. Each data point represents the aggregated amount (or frequency) of searches for a given search term, in a given place, at each day/week/month. Search Trends represent what people are searching for, when, where and how much. Therefore, business questions can be examined through the proxy of Search Trends, that can answer questions like: Was my TV campaign effective? Is my brand s awareness growing? What is my share of voice w.r.t. my competitors? What do customers associate with my brand? What competitor products are considered by buyers along with my product? Where should I open my new store?

4 Search Trends Characteristics Search Trends share some fundamental characteristics which may vary, depending on the context of the corresponding search terms. Often Search Trends show regular behaviors like a constant increase, a seasonal pattern or recurring spikes, while other Trends may be highly irregular. We have conducted experiments that associated such characteristics with the notion of Predictability of Search Trends in Google. We showed that over a half of the Trends of the most popular search terms are predictable*.

5 Seasonal Search Trends (can you guess...?)

6 Seasonal Search Trends (examples) ski world cup הישרדות, כוכב נולד

7 Non-Seasonal Search Trends examples (can you guess...?)

8 Non-Seasonal Search Trends examples (examples) FTP facebook, myspace madonna, spice girls, lady gaga

9 The Google Trends Tool Google is sharing with the public the aggregated and anonymized search trends of users searches across its domains worldwide, and exposes this information via a free analytic tool called: Google Trends (f.k.a. Google Insights for Search) Google Trends analyzes a portion of all Google searches (from 2004 to the present) to show trends and patterns of what the world is searching for... Google Trends is updated on a daily basis with a delay of 1-2 days. Any exposed data is strictly based on multiple distinct users to protect privacy. Google Trends exposes search trends data and analysis in several dimensions: Time: Daily/Weekly/Monthly granularity. Geo Location: Countries, Regions and Cities (including Metro Areas in the US). Category: Search categories in a hierarchical taxonomy. Search property: Web, Image, News & Product Search. Associated searches: Top & Rising related searches Google Trends is simple & interactive, it can be embedded as a gadget and allows for data exports to CSV files. This tool and data can be useful to anyone, especially to: Analysts & researchers Journalists & bloggers Marketers & advertisers Scholars

10 The Google Trends Tool com/trends

11 Recurring Spikes of Search Interest

12 Alternating seasonality across geographies

13 YoY Seasonality: The Football season in the US

14 Classifying searches to categories

Taxonomy of Categories + ML classification engine Each search query is

it's nearby context (queries before & after).

the underlying patterns of search behavior in a specific category/market.

15 Taxonomy of Categories + ML classification engine Each search query is classified into a hierarchical taxonomy of search categories depending on it's nearby context (queries before & after). Filtering the trends by the category enables to 'clean' the data and observe the underlying patterns of search behavior in a specific category/market. Often, the aggregated trends of the entire search category represent the major underlying dynamics of the category. (C) Google Inc.

16 Aggregated Category Trends (can you guess...?)

17 Aggregated Category Trends (examples) Automotive Computers & Electronics Sports Entertainment

Search Trends Research Examine the regularity, seasonality & predictability of Search Trends. Conduct correlation analysis, clustering & profiling of the Trends space.

18 Search Trends Research Examine the regularity, seasonality & predictability of Search Trends. Conduct correlation analysis, clustering & profiling of the Trends space. Use time series prediction methodologies to forecast Search Trends. Forecast users interest & analyze business cycles using search trends. Examine the dynamics of Co-Searching of search terms. Define relatedness metrics & investigate association between search terms. Comparing & integrating query data with other online/offline data sources. Examine the 'flow' of a web phenomena and analyse its geo-propagation.

19 (C) Google Inc. Clustering Analysis: Common Patterns of Search Behavior

20 Example of yearly seasonality with peaks before the holidays (+ upward trend): (C) Google Inc. (standardized monthly search data for 30 popular US queries - automatically clustered)

21 Example of yearly seasonality with peaks in the summer and lows in the winter (+ downward trend): : (C) Google Inc. (standardized monthly search data for 30 popular US queries - automatically clustered)

22 Example of a long term downward trend: (C) Google Inc. (C) Google Inc. (standardized monthly search data for 30 popular US queries - automatically clustered)

23 Predictability of Search Trends which queries are predictable? Over half of the most popular Google search queries were found predictable in 12 month ahead forecast, with a mean absolute prediction error of approximately 12% on average. Some categories have particularly high fraction of predictable queries: Health (74%) Food & Drink (67%) Travel (65%) Some categories have particularly low fraction of predictable queries: Entertainment (35%) Online Communities & Social Networks (27%) Trends of aggregated categories were found even more predictable: 88% of the aggregated category trends of over 600 categories are predictable with a mean absolute prediction error of less than 6% on average.

24 Predictability of Search Trends (results) Formulation of Predictability w.r.t. a model, error def + thresholds. Explored trends from various countries and search categories. Examined the influence of different trends' characteristics. On the Predictability of Search Trends / Y. Shimshoni, N. Efron, Y. Matias, 2009

25 Analysing Search Trends for Business Intelligence on: Competitive Landscape & Market Share Analysis Measuring Brand Awareness and other brand tracking metrics Attributes Associations Targeting & Positioning Strategies Profiling of Customers' and Understanding their Preferences Planning, Media Mixing & Econometric Modeling Campaign Effectiveness & Benchmarking Business Development & New Product Decisions Detection of Market Changes and New Trends Prediction of Demand & Sales Nowcasting and Macroeconomic Monitoring

26 Examples: Brand & Campaign Analysis We use the proxy of searches for brand names/terms to estimate Market Shares: We can demonstrate the Campaign Effect and compare it to benchmark trends: (C) Google Inc.

Analyzing deviations from modeled prediction Model the aggregated trends of 4 sub-categories in the US Auto Industry. Forecast a period around the economic crisis: Aug 2008 July 2009.

27 Analyzing deviations from modeled prediction Model the aggregated trends of 4 sub-categories in the US Auto Industry. Forecast a period around the economic crisis: Aug 2008 July Examine the deviations of the prediction model in these 4 sub-categories. It seems evident that this industry went through a significant change: More search traffic than predicted Vehicle Maintenance Auto Parts Less search traffic than predicted Vehicle Shopping Auto Financing

28 Nowcasting with Search Trends Many real-life time series, like macro-economic indicators and many market signals can be modeled using Search Trends. Choi and Varian 2009,2011 have used Search Trends of single queries and aggregated search categories to predict the present, where they predict the present levels of indicators like car sales or unemployment levels, before their official publication. For example, the rate of unemployment y may be predicted by the query x job search or new auto purchases may be predicted by the search category vehicle shopping. So, we can use AR model: yt = a0+a1yt-1+ a12yt-12+ bxt + et Such Nowcasting models are much faster & cheaper to build and run than traditional measurement methodologies. In the recent years over 60 academic articles were published on using Google Search Trends for nowcasting in various fields (including several central banks). The overall finding is that Search Trends significantly improve the forecast accuracy of traditional models. Since there are millions of search queries and hundreds of search categories, comparing to about 100 monthly data observations in most real life time-series, One can use the Google Correlate Tool, to suggest correlating Trends.

29 Google Correlate

How App Ratings and Reviews Impact Rank on Google Play and the App Store

APP STORE OPTIMIZATION MASTERCLASS How App Ratings and Reviews Impact Rank on Google Play and the App Store BIG APPS GET BIG RATINGS 13,927 AVERAGE NUMBER OF RATINGS FOR TOP-RATED IOS APPS 196,833 AVERAGE