Plumbing the Web. Narayanan Shivakumar. Google Distinguished Entrepreneur & Director

Size: px

Start display at page:

Download "Plumbing the Web. Narayanan Shivakumar. Google Distinguished Entrepreneur & Director"

Brenda Lydia Watkins
6 years ago
Views:

2 Plumbing the Web Narayanan Shivakumar Google Distinguished Entrepreneur & Director

4 Developers and Google AJAX Search API Google Code Project Hosting Google Web Toolkit Calendar Data API Base Data API Blogger Data API Notebook Data API PicasaData API Spreadsheets Data API Google SOAP Search API Desktop API Sitemaps API Gadgets API AJAX Feed API Mashup Editor Mapplets Google Gears Copyright 2007, Google Inc

5 How much information is out there? How large is the Web? Hundreds of billions of documents? Trillions? ~10KB/doc => 100s of Terabytes Then there s everything else , personal files, closed databases, broadcast media, print, etc. Estimated 5 Exabytes/year (growing at 30%)* 800MB/year/person ~90% in magnetic media Web is just a tiny starting point Source: How much information Copyright 2007, Google Inc

8 Sitemaps ( ls for web) XML sitemaps auto-produced and maintained on webservers {<url>, <changerate> <lastmod> <priority> } Autodiscovery through robots.txt Log structured protocol Scalable from 50 ->50M+ urls 8 Copyright 2007, Google Inc

9 Sitemaps adoption Open protocol launched in Jun 05 under Creative-Commons Joint support announced by MSN, Yahoo in Nov 06 ( Auto-discovery thro robots.txt in Apr 07 (+IBM, Ask.com) Billions of URLs auto-produced by servers, tools, plugins 9 Copyright 2007, Google Inc

25 Google s Explosive Computational Requirements Every Google service sees continuing growth in computational needs More queries More users, happier users More data Bigger web, mailbox, blog, etc. Better results Find the right information, and find it faster better results more data more queries 25 Copyright 2007, Google Inc

26 Hardware Design Philosophy Prefer low-end server/pc-class designs Build lots of them! Why? Single machine performance is not interesting Our smaller problems are too large for any single system Large problems are easily partitioned into multiple threads Ultra-reliable hardware makes programmers lazy Most reliable platform will still fail fault-tolerant software needed Fault-tolerant software enables use of commodity components Interesting systems can be designed with commodity components 26 Copyright 2007, Google Inc

Current Design In-house rack design PC-class motherboards Low-end storage

35 Systems Infrastructure Goal: Create very large scale, high performance computing infrastructure Hardware + software systems to make it easy to build products Focus on price/performance, and ease of use Enables better products: indices containing more documents updated more often faster queries faster product development cycles 35 Copyright 2007, Google Inc

36 GFS: Google File System Why YADFS? Google has unique FS requirements Huge read/write bandwidth Reliability over thousands of nodes Mostly operating on large data blocks Need efficient distributed operations Unfair advantage We have control over applications, libraries and operating system 36 Copyright 2007, Google Inc

37 GFS Setup Masters Replicas GFS Master GFS Master Misc. servers Client Client C 0 C 1 C 1 C 0 C 5 C 5 C 2 C 5 C 3 C 2 Chunkserver 1 Chunkserver 2 Chunkserver N Master manages metadata Data transfers happen directly between clients/chunkservers Files broken into chunks (typically 64 MB) 37 Copyright 2007, Google Inc

38 MapReduce + BigTable Okay, GFS lets us store lots of data now what? We want to process that data in new and interesting ways! MapReduce: a programming model and library to simplify large-scale computations on large clusters BigTable: A large-scale storage system for semi-structured data Database-like model, but data stored on thousands of machines.. 38 Copyright 2007, Google Inc

39 Developers and Google AJAX Search API Google Code Project Hosting Google Web Toolkit Calendar Data API Base Data API Blogger Data API Notebook Data API PicasaData API Spreadsheets Data API Google SOAP Search API Desktop API Sitemaps API Gadgets API AJAX Feed API Mashup Editor Mapplets Google Gears Copyright 2007, Google Inc

CPSC 426/526. Cloud Computing. Ennan Zhai. Computer Science Department Yale University

CPSC 426/526. Cloud Computing. Ennan Zhai. Computer Science Department Yale University CPSC 426/526 Cloud Computing Ennan Zhai Computer Science Department Yale University Recall: Lec-7 In the lec-7, I talked about: - P2P vs Enterprise control - Firewall - NATs - Software defined network