Web Dynamics. Part 1 - Introduction. 1.1 Dimensions of dynamics in the Web 1.2 Application examples. Summer Term 2010 Web Dynamics 1-1

Transcription

1 Web Dynamics Part 1 - Introduction 1.1 Dimensions of dynamics in the Web 1.2 Application examples Summer Term 2010 Web Dynamics 1-1

2 From Wikipedia: WhyWeb Dynamics? In physics the term dynamicscustomarily refers to the time evolution of physical processes. Summer Term 2010 Web Dynamics 1-2

3 Whichaspectsof theweb aredynamic? Size: sites/pages added and deleted all the time Summer Term 2010 Web Dynamics 1-3

4 Numberof siteson theweb 1998: 2,636,000 (IP addresses with HTTP server) 1999: 4,662, : 7,128,000, ~40% public, 40% dead 2001: 8,443, : 8,712, : 109 million sites(netcraft) 2007: 433 million hosts on Internet (ISC) : Summer Term 2010 Web Dynamics 1-4

5 Sizeestimatesforthe(indexable) Web 1995: ~11.4 million docs(bray) 1997: ~200 million docs(bharat&broder) (sampling based on Hotbot, Altavista, Excite and Infoseek, overlap ~2%) 1998: >800 million docs(lawrence&giles) January 2005: 11.5 billion docs(gulli&signorini) (sampling based on Google, MSN, Yahoo! and Ask/Teoma) 2005: 19.2 billion documents in Yahoo! index 2008: >1 trilliondocumentscountedbygoogle Summer Term 2010 Web Dynamics 1-5

6 Moresizeestimates Estimates based on overlap of search engine results (from [We will discuss this technique later in the course] Summer Term 2010 Web Dynamics 1-6

7 TheWeb isinfinite and growing Non-indexable Web not seen by search engines ( Deep Web behind forms): est. 550 billiondocs, est. 7.5 petabytes in 2000 (Bright Planet) User-generatedcontent(socialnetworks, communities, wikis, blogs, ) Pages created on demand ( next week link in online calendars) Summer Term 2010 Web Dynamics 1-7

8 Somesocialnetworks Flickr: (as of Oct2009) 4+ billionphotos(3 billionin Nov 2008, 2 billionin Nov 2007) 3 millionnewphotosper day Facebook: (as of Apr 2010) [ 3+ billionnewphotosper month, 60 millionstatusupdatesper day 400 millionactiveusers(120 millionin Nov 2008, 31 millionin Apr 2007) 150,000 newusersper dayin Nov 2008 (100,000/day in April 2007) Myspace: (as of Apr 2007) 135 million users(6th largest country on Earth) 2+ billion images(150,000 req/s), millions added daily 25 millionsongs 60TB videos StudiVZ.net: (as of Nov 2008) 11 millionusers 300 million images, 1 million added daily Summer Term 2010 Web Dynamics 1-8

9 Somesocialnetworks Flickr: (as of Oct2009) 4+ billionphotos(3 billionin Nov 2008, 2 billionin Nov 2007) 3 millionnewphotosper day Facebook: (as of Apr 2010) [ 3+ billionnewphotosper month, 60 millionstatusupdatesper day 400 millionactiveusers(120 millionin Nov 2008, 31 millionin Apr 2007) 150,000 newusersper dayin Nov 2008 (100,000/day in April 2007) Myspace: (as of Apr 2007) 135 million users(6th largest country on Earth) 2+ billion images(150,000 req/s), millions added daily 25 millionsongs 60TB videos StudiVZ.net: (as of Nov 2008) 11 millionusers 300 million images, 1 million added daily Flickr growth rate , from Summer Term 2010 Web Dynamics 1-9

10 Flickr: MySpace Infrastructure: (as of Oct2009) (as of 2008) Somesocialnetworks sending 100 gigabits of data per second to the Internet 10 gigabits HTML content 90 gigabits media (videos, pictures) 4500 web servers 4+ billionphotos(3 billionin Nov 2008, 2 billionin Nov 2007) 3 millionnewphotosper day Facebook: 1200 cache servers (as of Apr 2010) [ 500 database servers custom distributed file system (from and 3+ billionnewphotosper month, 60 millionstatusupdatesper day 400 millionactiveusers(120 millionin Nov 2008, 31 millionin Apr 2007) 150,000 newusersper dayin Nov 2008 (100,000/day in April 2007) Myspace: (as of Apr 2007) 135 million users(6th largest country on Earth) 2+ billion images(150,000 req/s), millions added daily 25 millionsongs 60TB videos StudiVZ.net: (as of Nov 2008) 11 millionusers 300 million images, 1 million added daily Summer Term 2010 Web Dynamics 1-10

11 Challenges: Sizedynamics How can a search engine deal with infinite Web? Massively parallel, distributed architecture (MapReduce, Hadoop, etc.) Detect and remove noise(duplicates, spam etc.) Summer Term 2010 Web Dynamics 1-11

12 Whichaspectsof theweb aredynamic? Size: pagesaddedand deletedall thetime Content: pageschangeall thetime Summer Term 2010 Web Dynamics 1-12

13 Lifetimeof versionson heise.de High-frequency crawl of heise.de over one week in January 2009 new version when news item added or removed [R. Schenkel, ECIR 2010] Summer Term 2010 Web Dynamics 1-13

14 Evolution of theweb (Ntoulaset al., 2004) Large-scale study: October2002 October2003 Weeklycrawlsof 154 large Web sites(up to 200,000 pages per site) Summer Term 2010 Web Dynamics 1-14

15 Averagepagecreationper week About8% newpagescreatedper week (A. Ntoulas, J. Cho, C. Olston: What snewon theweb? TheEvolution of theweb froma searchengineperspective, WWW Conference, 2004) Summer Term 2010 Web Dynamics 1-15

16 Howlongdo pageslive? About40% of thepagesstill availableafteroneyear (A. Ntoulas, J. Cho, C. Olston: What snewon theweb? TheEvolution of theweb froma searchengineperspective, WWW Conference, 2004) Summer Term 2010 Web Dynamics 1-16

17 Howfrequentlydoesa pagechange? Most pages never change, second most change at least weekly (A. Ntoulas, J. Cho, C. Olston: What snewon theweb? TheEvolution of theweb froma searchengineperspective, WWW Conference, 2004) Summer Term 2010 Web Dynamics 1-17

18 Howmuchdo pageschange? Most of thechangesareminor (A. Ntoulas, J. Cho, C. Olston: What snewon theweb? TheEvolution of theweb froma searchengineperspective, WWW Conference, 2004) Summer Term 2010 Web Dynamics 1-18

19 Howlarge arepages? Averagesizeraisedbyabout15% in oneyear (A. Ntoulas, J. Cho, C. Olston: What snewon theweb? TheEvolution of theweb froma searchengineperspective, WWW Conference, 2004) Summer Term 2010 Web Dynamics 1-19

20 Morerecentnumbers Average size of Web pages more than tripled since 2003 from 93.7K to over 312K Average number of objects per Web page nearly doubled from 25.7 to 49.9 Since 1995 average size of Web pages increased by 22 times Since 1995 average number of objects per Web page increased by 21.7 times (from Summer Term 2010 Web Dynamics 1-20

21 Morerecentcharts (from Summer Term 2010 Web Dynamics 1-21

22 Challenges: Contentdynamics How can a search engine maintain a reasonably accurate snapshot of the Web? Model how/when documents updated Recrawl policy based on expected changes Decide if a page s content changed(enough to replace old version in snapshot) HowcanwemaintaintheWeb of thepast? Web archiving Summer Term 2010 Web Dynamics 1-22

23 Whichaspectsof theweb aredynamic? Size: pagesaddedand deletedall thetime Content: pageschangeall thetime Structure: links added all the time (and dropped) Summer Term 2010 Web Dynamics 1-23

24 Howfrequentlydo links change? 25% newlinks createdper week, 80% of links replacedwithina year (A. Ntoulas, J. Cho, C. Olston: What snewon theweb? TheEvolution of theweb froma searchengineperspective, WWW Conference, 2004) Summer Term 2010 Web Dynamics 1-24

25 Challenges: Structuredynamics How can a search engine maintain a reasonably accurate snapshot of the Web graph? Massively parallel, distributed architecture (MapReduce, Hadoop, etc.) Distributed approximation algorithms for computing authority measures(pagerank) Summer Term 2010 Web Dynamics 1-25

26 Whichaspectsof theweb aredynamic? Size: pagesaddedand deletedall thetime Content: pageschangeall thetime Structure: links added all the time (and dropped) Usage: Behaviourof userschangesall thetime Summer Term 2010 Web Dynamics 1-26

27 Reasonswhyuserbehaviourchanges Global trendsand changes, Web 2.0 (Flickr, Youtube, social networks, twitter, ) Different situation/context Roles(private vs. professional) Locations(home vs. office vs. travelling) Date & Time Tasks(orderinga book, bookinga flight, ) influence browsing and search behaviour Summer Term 2010 Web Dynamics 1-27

28 Challenges: User dynamics Howcana searchengineadaptto changingusers? Identify user(e.g., Google s cookie) Collect user behaviour Personalize search results based on past actions Personalize based on current context Thiscanbedone For eachuser For groupsof users For all users( global user model ) Summer Term 2010 Web Dynamics 1-28

29 Web Dynamics Part 1 - Introduction 1.1 Dimensions of dynamics in the Web 1.2 Application examples Summer Term 2010 Web Dynamics 1-29

30 Live Searchin News Streams Summer Term 2010 Web Dynamics 1-30

31 Searchin PastNews Streams Summer Term 2010 Web Dynamics 1-31

32 Google Trends: Hot Searches Summer Term 2010 Web Dynamics 1-32

33 Google Trends: Searchstats Summer Term 2010 Web Dynamics 1-33

34 Google insights: Trends in searches Summer Term 2010 Web Dynamics 1-34

35 Google Website trends: accessstats Summer Term 2010 Web Dynamics 1-35

36 Google News Timeline: News trends Summer Term 2010 Web Dynamics 1-36

37 Google Web timeline: Date extraction Summer Term 2010 Web Dynamics 1-37

38 Google Zeitgeist: Frequentsearches Summer Term 2010 Web Dynamics 1-38

39 Internet Archive: Waybackmachine Summer Term 2010 Web Dynamics 1-39

40 Internet Archive: Waybackmachine Summer Term 2010 Web Dynamics 1-40

41 MoreWeb Archiving: Iterasi Summer Term 2010 Web Dynamics 1-41

42 References T. Bray: Measuring the Web, WWW Conference, K. Bharat, A. Broder: A technique for measuring the relative size and overlap of public web search engines, WWW Conference, 1998 A. Gulli, A. Signorini: The Indexable Web is more than 11.5 billion pages, WWW Conference, 2005 S. Lawrence and C. L. Giles: Accessibility of information on the web, Nature, 400: , 1999 J. Domenechet al.: A user-focused evaluation of web prefetchingalgorithms,computer Communications 30:10, , 2007 R. Sadre, B. Haverkort: Changes in the Web from 2000 to 2007, Workshop on Distributed Systems: Operations and Management, 2008 K.M. Risvik, R. Michelsen: Searchenginesand Web dynamics, Computer Networks 39, , 2002 Y. Keet al.: Web dynamicsand theirramificationsforthedevelopmentof Web searchengines, Computer Networks 50, , 2006 R. Baeza-Yates et al.: Web structure, dynamics and page quality, SPIRE Conference, 2002 V.N. Padmanabhan, L. Qiu: Thecontentand accessdynamicsof a busyweb site: Findingsand implications, SIGCOMM conference, 2000 L. Cherkasova, M. Karlsson: Dynamics and evolution of Web sites: Analysis, metrics and design issues, IEEE International Symposium on Computers and Communications, 2001 J. Cho, H. Garcia-Molina: Estimatingfrequencyof change, Transactionson Internet Technologies 3(3): , 2003 J. Cho, H. Garcia-Molina: Theevolutionof theweb and implicationsforan incrementalcrawler. VLDB Conference, 2000 A. Ntoulas, J. Cho, C. Olston: What snewon theweb? TheEvolution of theweb froma search engine perspective, WWW Conference, 2004 R. Schenkel: Temporal Shingling for Version Identification in Web Archives, ECIR Conference, Summer Term 2010 Web Dynamics 1-42