Networked systems and their users

Transcription

1 Networked systems and their users q The expansion of the Google serving infrastructure q A personalized livestreaming system q A platform for crowdsourcing web QoE measurements

2 Mapping the expansion of Google s serving infrastructure M. Calder, X. Fan, Z. Hu, E. Katz- Bassett, J. Heidemann, R. Govindan IMC 2013

3 Why study Google? Content providers dominate Internet traffic Google serves an estimated 25% of North American Internet Traffic (Wired, July 2013) Google goes down for 5 minutes, Internet traffic drops 40% (CNET, August 2013) Complex infrastructure for performance Unique and valuable view of the Internet Properties of content serving infrastructure Improve new and existing systems 3

4 Google infrastructure Designed to reduce latency Datacenter 4

5 Google infrastructure Designed to reduce latency Public interface for web services Users interact directly with front-ends Relay between user and datacenter Datacenter Front-end 5

6 The work map front-ends Google s or ISP network More control from front-end to datacenter Financial motivation to map clients to nearby front-ends Front-end 6

7 Goal: Map content serving infrastructure Geographical and topological scope Client-server mapping Tracking expansion Client impact of expansion 7

8 Challenges and approach Insufficient vantage points limit mapping Existing geolocations inaccurate for servers Differentiate physical sites in metro area 8

9 Challenges and approach Use the fact that Google works hard to direct clients to nearby front-ends Insufficient vantage points limit mapping Complete enumeration using EDNS client subnet-prefix Existing geolocations inaccurate for servers Geolocate server based on its clients Differentiate physical sites in metro area Clustering technique 9

10 Background: Client to front-end mapping Hard case: Distributed users Google server A Google.com? Google server A Google server B Local DNS resolver 10

11 Background: Client to front-end mapping Solution: EDNS Client-subnet-prefix Google server A Google.com? /24 Google server B Google server B Local DNS resolver /24 11

12 Enumeration methodology Daily, we query from all routable prefixes using EDNS and find which front-end Google is directing it to For DNS, equivalent to having a vantage point in 10 million /24s 12

13 Client-Centric Geolocation Geolocates a server from the geographic mean of the (possibly noisy) locations for clients associated with that server Intuition If the provider is doing a good job, clients will be send to network and probably geographically close servers so if you know where clients are, you can approximate the server s location 13

14 Geolocation methodology Geolocate a front-end IP based on the location of the clients it serves 1. Given a front-end IP

15 Geolocation methodology Geolocate a front-end IP based on the location of the clients it serves 1. Given a front-end IP 2. Identify the client prefixes directed to it / / /24 15

16 Geolocation methodology Geolocate a front-end IP based on the location of the clients it serves 1. Given a front-end IP 2. Identify the client prefixes directed to it 3. Geolocate all client prefixes using MaxMind database /24 Sydney, Auz /24 Auckland, NZ /24 Sydney, Aus 16

17 Geolocation methodology Geolocate a front-end IP based on the location of the clients it serves 1. Given a front-end IP 2. Identify the client prefixes directed to it 3. Geolocate all client prefixes using MaxMind database 4. Discard outliers to improve accuracy 5. Location is geographic mean of remaining prefix locations /24 Sydney, Auz /24 Auckland, NZ /24 Sydney, Aus 17

18 Geolocation example ? 18

19 Geolocation example Sydney, AUS Ground truth from airport code sysd01s13-in-f16.1e100.net 19

20 Geolocation example All prefix locations 12944, error 580 Km Actual Estimated 20

21 Geolocation example All prefix locations 12944, error 580 Km Remove low confidence location -2227, error 320 Km Remove prefixes far from front-ends -7257, error 12 Km Actual Estimated 21

22 Speed of light filtering 1. Measure RTT 2. Establish feasibility region based on RTT 3. Discard locations outside intersection region Target front-end Vantage point 22

23 Geolocation example All prefix locations 12944, error 580 Km Remove low confidence locations -2227, error 320 Km Remove prefixes far from front-ends -7257, error 12 Km Additional filtering details in paper Actual Estimated 23

24 Validation: Geolocation accuracy Ground truth are Google IPs with airport codes MaxMind places everything in Mountain View, CA Speed of light needs nearby vantage points Our geolocation approach median error is 26Km 24

25 Overview Overcome infrastructure mapping challenges Limited vantage points Inaccurate geolocation for servers Enable longitudinal study of Google Expansion results 25

26 Results: Expansion 1,400+ serving sites as of Aug

27 Google.com servers in Oct sites in 60 countries and 100 ASes Large % are in Google s AS 27

28 Google.com servers in October sites (7x) In 130+ countries (2.3x) 800+ ASes (8x) Growth is outside Google s AS 28

29 Results: What is different for ASes? AS size 2012/11 ASes 2013/08 ASes Google 2 2 (+0%) Tier (+100%) Large (+310%) Small (+811%) Stub (+1133%) Most front-ends in low end of AS hierarchy 29

30 Results: What is different for users? AS size 2012/11 ASes 2013/08 ASes 2012/11 Clients 2013/08 Clients Google 2 2 (+0%) 9856K 9067K (-8%) Tier (+100%) K (+7278%) Large (+310%) 111K 410K (+270%) Small (+811%) 37K 359K (+870%) Stub (+1133%) 52K 180K (+240%) Most front-ends in low end of AS hierarchy 90% of prefixes still directed to servers in Google network Most prefixes directed to front ends in larger ASes 30

31 Results: Client Front-end distance Of prefixes now served by ISP-hosted sites Median Km to Google-hosted front-end Median within ~45 Km of ISP-hosted front-end 31

32 Expansion strategy Moving infrastructure closer to clients Reduce latency Split-TCP proxies Persistent, multiplexed connections Serve static content immediately Advanced error recovery Repurposing of Google Global Cache YouTube video caches 32

33 Client impact (ongoing work) PlanetLab node at Christchurch, NZ Nearest Google-hosted front-end in Sydney Query response time to ISP and Google hosted 33

34 Conclusion New techniques for mapping the content serving infrastructures for a major content provider, Google Longitudinal study of expansion of Google s infrastructure Shows shift in Google s strategy for reducing client latency Data available mappinggoogle.cs.usc.edu 34

35 Today The expansion of the Google serving infrastructure A personalized livestreaming system A platform for crowdsourcing web QoE measurements 35