Data Centers. Tom Anderson
|
|
- Brendan Harrington
- 5 years ago
- Views:
Transcription
1 Data Centers Tom Anderson
2 Transport Clarification RPC messages can be arbitrary size Ex: ok to send a tree or a hash table Can require more than one packet sent/received We assume messages can be dropped, duplicated, reordered It is packets that are dropped, duplicated, reordered Most RPCs are single packets TCP addresses some of these issues But we ll assume the general case
3 RPC Semantics At least once (NFS, DNS, lab 1b) true: executed at least once false: maybe executed, maybe multiple times At most once (lab 1c) true: executed once false: maybe executed, but never more than once Exactly once true: executed once false: never returns false
4 When does at-least-once work? If no side effects read-only operations (or idempotent ops) Example: MapReduce Example: NFS readfileblock (vs. Posix file read) writefileblock (but not: delete file)
5 At Most Once Client includes unique ID (UID) with each request use same UID for re-send Server RPC code detects duplicate requests return previous reply instead of re-running handler if seen[uid] { r = old[uid] } else { r = handler() old[uid] = r seen[uid] = true }
6 Some At-Most-Once Issues How do we ensure UID is unique? Big random number? Combine unique client ID (IP address?) with seq #? What if client crashes and restarts? Can it reuse the same UID? In labs, nodes never restart Equivalent to: every node gets new ID on start Client address + seq # is unique Seq # alone is NOT unique
7 When Can Server Discard Old RPCs? Option 1: Never? Option 2: unique client IDs per-client RPC sequence numbers client includes "seen all replies <= X" with every RPC Option 3: one client RPC at a time arrival of seq+1 allows server to discard all <= seq Labs use Option 3
8 What if Server Crashes? If at-most-once list of recent RPC results is stored in memory, server will forget and accept duplicate requests when it reboots Does server need to write the recent RPC results to disk? If replicated, does replica also need to store recent RPC results? In Labs, server gets new address on restart Client messages aren t delivered to restarted server (discard if sent to wrong version of server)
9 A Data Center
10 Inside a Data Center
11 Data center 10k - 100k servers: 250k 10M cores 1-100PB of DRAM 100PB - 10EB storage 1-10 Pbps bandwidth (>> Internet) MW power - 1-2% of global energy consumption 100s of millions of dollars
12 Servers Limits driven by the power consumption 1-4 multicore sockets cores/socket (150W each) 100s GB 1 TB of DRAM ( W) 40Gbps link to network
13 19 wide 1.75 tall (1u) (defined in 1922!) servers/rack network at top Servers in racks
14 Racks in rows
15 Rows in hot/cold pairs
16 Hot/cold pairs in data centers
17 Amazon, in the US: - Northern Virginia - Ohio - Oregon - Northern California Where is the cloud? Why those locations?
18 MTTF/MTTR Mean Time to Failure/Mean Time to Repair Disk failures (not reboots) per year ~ 2-4% At data center scale, that s about 2/hour. It takes 10 hours to restore a 10TB disk Server crashes 1/month * 30 seconds to reboot => 5 mins/year 100K+ servers
19 Typical Year in a Data Center (2008) ~0.5 overheating (power down most machines in <5 mins, ~1-2 days to recover) ~1 PDU failure (~ machines suddenly disappear, ~6 hours to come back) ~1 rack-move (plenty of warning, ~ machines powered down, ~6 hours) ~1 network rewiring (rolling ~5% of machines down over 2-day span) ~20 rack failures (40-80 machines instantly disappear, 1-6 hours to get back)
20 Typical Year in a Data Center (2008) ~5 racks go wonky (40-80 machines see 50% packetloss) ~8 network maintenances (4 might cause ~30-minute random connectivity losses) ~12 router reloads (takes out DNS and external vips for a couple minutes) ~3 router failures (have to immediately pull traffic for an hour) ~dozens of minor 30-second blips for dns ~1000 individual machine failures ~thousands of hard drive failures slow disks, bad memory, misconfigured machines, flaky machines, etc
21 This Week: Primary Backup How do we build systems that survive a single node failure? State replication: run two copies of server primary, backup different racks, different PDUs, different DCs! If primary fails, backup takes over How do we know primary failed vs. is slow?
22 Data Center Networks Every server wired to a ToR (top of rack) ToR s in neighboring aisles wired to an aggregation Agg. es wired to core es
23 Early data center networks 3 layers of es - Edge (ToR) - Aggregation - Core
24 Early data center networks 3 layers of es - Edge (ToR) - Aggregation - Core Optical Electrical
25 Early data center limitations Cost - Core, aggregation routers = high capacity, low volume - Expensive! Fault-tolerance - Failure of a single core or aggregation router = large bandwidth loss Bisection bandwidth limited by capacity of largest available router - Google s DC traffic doubles every year!
26 Clos networks How can I replace a big by many small es? Big
27 Clos networks How can I replace a big by many small es? Big
28 Clos Networks What about bigger es?
29
30
31 Multi-rooted tree Every pair of nodes has many paths Fault tolerant! But how do we pick a path?
32 Multipath routing Lots of bandwidth, split across many paths ECMP: hash on packet header to determine route - (5 tuple): Source IP, port, destination IP, port, prot. - Packets from client server usually take same route On or link failure, ECMP sends subsequent packets along a different route => Out of order packets!
33 Data Center Network Trends RT latency across data center ~ 10 usec 40 Gbps links common, 100 Gbps on the way 1KB packet every 80ns on a 100Gbps link Direct delivery into the on-chip cache (DDIO) Upper levels of tree are (expensive) optical links Thin tree to reduce costs Within rack > within aisle > within DC > cross DC Latency and bandwidth: keep communication local
34 Local Storage Magnetic disks for long term storage High latency (10ms), low bandwidth (250MB/s) Compressed and replicated for cost, resilience Solid state storage for persistence, cache layer 50us block access, multi-gb/s bandwidth Emerging NVM Low energy DRAM replacement Sub-microsecond persistence
35 Day in the Life of a Web Request User types google.com, what happens DNS = Domain Name System Global database of name->ip address DNS query to root name server Root name server is replicated (600+) Hard coded IP multicast addresses Updates happen async in background (rare) Root returns: IP address of google s name server Cached on client (for a configurable period) Can be out of date
36 Name Server to DC DNS returns google s name server Google name server returns local name server Name server in DC close to user s IP address Local name server returns IP of web front end Also in local data center With short timeout, reroute if front end failure Browser sends HTTP request to front end
37 Resilient Scalable Web Front End IP address of front end is an array of machines Packets first go through load balancer Actually, an array of load balancers (all the same) Load balancer hashes 3 or 5 tuple -> front end Source IP, port #, Destination IP, port #, protocol This way, all traffic from user goes to same server When front end fails, load balancer redirects incoming packets elsewhere
38 Challenge How do we build a hash function that s stable? That reroutes only the packets for the failed node, leaving the packets to other nodes alone.
39 Web Front End -> Cache, Storage Layer Key-value store Sharded across many cache and storage nodes Hash(key) -> cache, storage node If cache node fails, rehash to new cache node If storage node fails,???
Remote Procedure Call. Tom Anderson
Remote Procedure Call Tom Anderson Why Are Distributed Systems Hard? Asynchrony Different nodes run at different speeds Messages can be unpredictably, arbitrarily delayed Failures (partial and ambiguous)
More informationDatacenters. Mendel Rosenblum. CS142 Lecture Notes - Datacenters
Datacenters Mendel Rosenblum Evolution of datacenters 1960's, 1970's: a few very large time-shared computers 1980's, 1990's: heterogeneous collection of lots of smaller machines. Today and into the future:
More information1/10/16. RPC and Clocks. Tom Anderson. Last Time. Synchroniza>on RPC. Lab 1 RPC
RPC and Clocks Tom Anderson Go Synchroniza>on RPC Lab 1 RPC Last Time 1 Topics MapReduce Fault tolerance Discussion RPC At least once At most once Exactly once Lamport Clocks Mo>va>on MapReduce Fault Tolerance
More informationCOMP Parallel Computing. Lecture 22 November 29, Datacenters and Large Scale Data Processing
- Parallel Computing Lecture 22 November 29, 2018 Datacenters and Large Scale Data Processing Topics Parallel memory hierarchy extend to include disk storage Google web search Large parallel application
More informationData Centers and Cloud Computing
Data Centers and Cloud Computing CS677 Guest Lecture Tim Wood 1 Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises for server applications Internet
More informationData Centers and Cloud Computing. Slides courtesy of Tim Wood
Data Centers and Cloud Computing Slides courtesy of Tim Wood 1 Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises for server applications Internet
More informationWorse Is Better. Vijay Gill/Bikash Koley/Paul Schultz for Google Network Architecture Google. June 14, 2010 NANOG 49, San Francisco
Worse Is Better Vijay Gill/Bikash Koley/Paul Schultz for Google Network Architecture Google June 14, 2010 NANOG 49, San Francisco Worse is Better You see, everybody else is too afraid of looking stupid
More informationData Centers and Cloud Computing. Data Centers
Data Centers and Cloud Computing Slides courtesy of Tim Wood 1 Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises for server applications Internet
More informationAvailability, Reliability, and Fault Tolerance
Availability, Reliability, and Fault Tolerance Guest Lecture for Software Systems Security Tim Wood Professor Tim Wood - The George Washington University Distributed Systems have Problems Hardware breaks
More informationimage credit Fabien Hermenier Cloud compting 101
image credit http://eyepluscamera.files.wordpress.com/ Fabien Hermenier Cloud compting 101 1 ? was cloud computing needed 2 3 Mainframes Then came with affordable PCs Then we spread out the load for security,
More informationRouting Domains in Data Centre Networks. Morteza Kheirkhah. Informatics Department University of Sussex. Multi-Service Networks July 2011
Routing Domains in Data Centre Networks Morteza Kheirkhah Informatics Department University of Sussex Multi-Service Networks July 2011 What is a Data Centre? Large-scale Data Centres (DC) consist of tens
More informationimage credit Fabien Hermenier Cloud compting 101
image credit http://eyepluscamera.files.wordpress.com/ Fabien Hermenier Cloud compting 101 1 was cloud computing needed? 2 3 Mainframes Then came with affordable PCs Then we spread out the load for security,
More informationDistributed File Systems II
Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation
More informationLarge-Scale Web Applications
Large-Scale Web Applications Mendel Rosenblum Web Application Architecture Web Browser Web Server / Application server Storage System HTTP Internet CS142 Lecture Notes - Intro LAN 2 Large-Scale: Scale-Out
More informationDistributed System. Gang Wu. Spring,2018
Distributed System Gang Wu Spring,2018 Lecture7:DFS What is DFS? A method of storing and accessing files base in a client/server architecture. A distributed file system is a client/server-based application
More informationCS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring Lecture 21: Network Protocols (and 2 Phase Commit)
CS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring 2003 Lecture 21: Network Protocols (and 2 Phase Commit) 21.0 Main Point Protocol: agreement between two parties as to
More informationUtilizing Datacenter Networks: Centralized or Distributed Solutions?
Utilizing Datacenter Networks: Centralized or Distributed Solutions? Costin Raiciu Department of Computer Science University Politehnica of Bucharest We ve gotten used to great applications Enabling Such
More informationMore on Testing and Large Scale Web Apps
More on Testing and Large Scale Web Apps Testing Functionality Tests - Unit tests: E.g. Mocha - Integration tests - End-to-end - E.g. Selenium - HTML CSS validation - forms and form validation - cookies
More informationAdvanced Computer Networks Exercise Session 7. Qin Yin Spring Semester 2013
Advanced Computer Networks 263-3501-00 Exercise Session 7 Qin Yin Spring Semester 2013 1 LAYER 7 SWITCHING 2 Challenge: accessing services Datacenters are designed to be scalable Datacenters are replicated
More informationThemes. The Network 1. Energy in the DC: ~15% network? Energy by Technology
Themes The Network 1 Low Power Computing David Andersen Carnegie Mellon University Last two classes: Saving power by running more slowly and sleeping more. This time: Network intro; saving power by architecting
More informationDistributed Data Infrastructures, Fall 2017, Chapter 2. Jussi Kangasharju
Distributed Data Infrastructures, Fall 2017, Chapter 2 Jussi Kangasharju Chapter Outline Warehouse-scale computing overview Workloads and software infrastructure Failures and repairs Note: Term Warehouse-scale
More informationFault-Tolerant Computer System Design ECE 695/CS 590. Putting it All Together
Fault-Tolerant Computer System Design ECE 695/CS 590 Putting it All Together Saurabh Bagchi ECE/CS Purdue University ECE 695/CS 590 1 Outline Looking at some practical systems that integrate multiple techniques
More informationICS 351: Networking Protocols
ICS 351: Networking Protocols IP packet forwarding application layer: DNS, HTTP transport layer: TCP and UDP network layer: IP, ICMP, ARP data-link layer: Ethernet, WiFi 1 Networking concepts each protocol
More informationChapter 8 Fault Tolerance
DISTRIBUTED SYSTEMS Principles and Paradigms Second Edition ANDREW S. TANENBAUM MAARTEN VAN STEEN Chapter 8 Fault Tolerance 1 Fault Tolerance Basic Concepts Being fault tolerant is strongly related to
More informationTake Back Lost Revenue by Activating Virtuozzo Storage Today
Take Back Lost Revenue by Activating Virtuozzo Storage Today JUNE, 2017 2017 Virtuozzo. All rights reserved. 1 Introduction New software-defined storage (SDS) solutions are enabling hosting companies to
More informationDistributed Filesystem
Distributed Filesystem 1 How do we get data to the workers? NAS Compute Nodes SAN 2 Distributing Code! Don t move data to workers move workers to the data! - Store data on the local disks of nodes in the
More informationAgenda. 1 Web search. 2 Web search engines. 3 Web robots, crawler, focused crawling. 4 Web search vs Browsing. 5 Privacy, Filter bubble.
Agenda EITF25 Internet - Web Information (search, browse,...) Anders Ardö EIT Electrical and Information Technology, Lund University December 10, 2014 A. Ardö, EIT EITF25 Internet - Web Information (search,
More informationMODELS OF DISTRIBUTED SYSTEMS
Distributed Systems Fö 2/3-1 Distributed Systems Fö 2/3-2 MODELS OF DISTRIBUTED SYSTEMS Basic Elements 1. Architectural Models 2. Interaction Models Resources in a distributed system are shared between
More informationLecture 2 Distributed Filesystems
Lecture 2 Distributed Filesystems 922EU3870 Cloud Computing and Mobile Platforms, Autumn 2009 2009/9/21 Ping Yeh ( 葉平 ), Google, Inc. Outline Get to know the numbers Filesystems overview Distributed file
More informationPerformance Analysis for Crawling
Scalable Servers and Load Balancing Kai Shen Online Applications online applications Applications accessible to online users through. Examples Online keyword search engine: Google. Web email: Gmail. News:
More informationIntroduction. Network Architecture Requirements of Data Centers in the Cloud Computing Era
Massimiliano Sbaraglia Network Engineer Introduction In the cloud computing era, distributed architecture is used to handle operations of mass data, such as the storage, mining, querying, and searching
More informationBigtable: A Distributed Storage System for Structured Data By Fay Chang, et al. OSDI Presented by Xiang Gao
Bigtable: A Distributed Storage System for Structured Data By Fay Chang, et al. OSDI 2006 Presented by Xiang Gao 2014-11-05 Outline Motivation Data Model APIs Building Blocks Implementation Refinement
More informationEngineering Goals. Scalability Availability. Transactional behavior Security EAI... CS530 S05
Engineering Goals Scalability Availability Transactional behavior Security EAI... Scalability How much performance can you get by adding hardware ($)? Performance perfect acceptable unacceptable Processors
More informationCamdoop Exploiting In-network Aggregation for Big Data Applications Paolo Costa
Camdoop Exploiting In-network Aggregation for Big Data Applications costa@imperial.ac.uk joint work with Austin Donnelly, Antony Rowstron, and Greg O Shea (MSR Cambridge) MapReduce Overview Input file
More informationLecture 3 Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung, SOSP 2003
Lecture 3 Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung, SOSP 2003 922EU3870 Cloud Computing and Mobile Platforms, Autumn 2009 (2009/9/28) http://labs.google.com/papers/gfs.html
More informationThe Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung December 2003 ACM symposium on Operating systems principles Publisher: ACM Nov. 26, 2008 OUTLINE INTRODUCTION DESIGN OVERVIEW
More informationToday: Fault Tolerance. Replica Management
Today: Fault Tolerance Failure models Agreement in presence of faults Two army problem Byzantine generals problem Reliable communication Distributed commit Two phase commit Three phase commit Failure recovery
More informationMemory-Based Cloud Architectures
Memory-Based Cloud Architectures ( Or: Technical Challenges for OnDemand Business Software) Jan Schaffner Enterprise Platform and Integration Concepts Group Example: Enterprise Benchmarking -) *%'+,#$)
More informationComputer Network 2015 Mid-Term Exam.
Computer Network 2015 Mid-Term Exam. Question : ``Basic of Computer Networks and the Internet' Please fill into the blanks (15%) a) The amount of time required to push all of a packet s bits into a link
More informationGFS Overview. Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures
GFS Overview Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures Interface: non-posix New op: record appends (atomicity matters,
More informationCSE 124: Networked Services Lecture-16
Fall 2010 CSE 124: Networked Services Lecture-16 Instructor: B. S. Manoj, Ph.D http://cseweb.ucsd.edu/classes/fa10/cse124 11/23/2010 CSE 124 Networked Services Fall 2010 1 Updates PlanetLab experiments
More information02 - Distributed Systems
02 - Distributed Systems Definition Coulouris 1 (Dis)advantages Coulouris 2 Challenges Saltzer_84.pdf Models Physical Architectural Fundamental 2/58 Definition Distributed Systems Distributed System is
More informationFlat Datacenter Storage. Edmund B. Nightingale, Jeremy Elson, et al. 6.S897
Flat Datacenter Storage Edmund B. Nightingale, Jeremy Elson, et al. 6.S897 Motivation Imagine a world with flat data storage Simple, Centralized, and easy to program Unfortunately, datacenter networks
More information02 - Distributed Systems
02 - Distributed Systems Definition Coulouris 1 (Dis)advantages Coulouris 2 Challenges Saltzer_84.pdf Models Physical Architectural Fundamental 2/60 Definition Distributed Systems Distributed System is
More informationHadoop File System S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y 11/15/2017
Hadoop File System 1 S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y Moving Computation is Cheaper than Moving Data Motivation: Big Data! What is BigData? - Google
More informationStorage. CS 3410 Computer System Organization & Programming
Storage CS 3410 Computer System Organization & Programming These slides are the product of many rounds of teaching CS 3410 by Deniz Altinbuke, Kevin Walsh, and Professors Weatherspoon, Bala, Bracy, and
More informationThis project must be done in groups of 2 3 people. Your group must partner with one other group (or two if we have add odd number of groups).
1/21/2015 CS 739 Distributed Systems Fall 2014 PmWIki / Project1 PmWIki / Project1 The goal for this project is to implement a small distributed system, to get experience in cooperatively developing a
More informationCache and Forward Architecture
Cache and Forward Architecture Shweta Jain Research Associate Motivation Conversation between computers connected by wires Wired Network Large content retrieval using wireless and mobile devices Wireless
More informationMcAfee Network Security Platform
Revision B McAfee Network Security Platform (8.1.7.5-8.1.3.43 M-series Release Notes) Contents About this release New features Enhancements Resolved issues Installation instructions Known issues Product
More informationFault Tolerance. Distributed Systems IT332
Fault Tolerance Distributed Systems IT332 2 Outline Introduction to fault tolerance Reliable Client Server Communication Distributed commit Failure recovery 3 Failures, Due to What? A system is said to
More informationDistributed systems. Lecture 6: distributed transactions, elections, consensus and replication. Malte Schwarzkopf
Distributed systems Lecture 6: distributed transactions, elections, consensus and replication Malte Schwarzkopf Last time Saw how we can build ordered multicast Messages between processes in a group Need
More informationDistributed Systems
15-440 Distributed Systems 11 - Fault Tolerance, Logging and Recovery Tuesday, Oct 2 nd, 2018 Logistics Updates P1 Part A checkpoint Part A due: Saturday 10/6 (6-week drop deadline 10/8) *Please WORK hard
More informationThe Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google SOSP 03, October 19 22, 2003, New York, USA Hyeon-Gyu Lee, and Yeong-Jae Woo Memory & Storage Architecture Lab. School
More informationDistributed Systems Fault Tolerance
Distributed Systems Fault Tolerance [] Fault Tolerance. Basic concepts - terminology. Process resilience groups and failure masking 3. Reliable communication reliable client-server communication reliable
More informationMap-Reduce. Marco Mura 2010 March, 31th
Map-Reduce Marco Mura (mura@di.unipi.it) 2010 March, 31th This paper is a note from the 2009-2010 course Strumenti di programmazione per sistemi paralleli e distribuiti and it s based by the lessons of
More informationComputer Networks - Midterm
Computer Networks - Midterm October 28, 2016 Duration: 2h15m This is a closed-book exam Please write your answers on these sheets in a readable way, in English or in French You can use extra sheets if
More informationOutline. Spanner Mo/va/on. Tom Anderson
Spanner Mo/va/on Tom Anderson Outline Last week: Chubby: coordina/on service BigTable: scalable storage of structured data GFS: large- scale storage for bulk data Today/Friday: Lessons from GFS/BigTable
More informationCloud Computing and Hadoop Distributed File System. UCSB CS170, Spring 2018
Cloud Computing and Hadoop Distributed File System UCSB CS70, Spring 08 Cluster Computing Motivations Large-scale data processing on clusters Scan 000 TB on node @ 00 MB/s = days Scan on 000-node cluster
More informationInternet Content Distribution
Internet Content Distribution Chapter 1: Introduction Jussi Kangasharju Chapter Outline Introduction into content distribution Basic concepts TCP DNS HTTP Outline of the rest of the course Kangasharju:
More information6.033 Lecture Fault Tolerant Computing 3/31/2014
6.033 Lecture 14 -- Fault Tolerant Computing 3/31/2014 So far what have we seen: Modularity RPC Processes Client / server Networking Implements client/server Seen a few examples of dealing with faults
More informationCS 61C: Great Ideas in Computer Architecture (Machine Structures) Warehouse-Scale Computing
CS 61C: Great Ideas in Computer Architecture (Machine Structures) Warehouse-Scale Computing Instructors: Nicholas Weaver & Vladimir Stojanovic http://inst.eecs.berkeley.edu/~cs61c/ Coherency Tracked by
More information! Design constraints. " Component failures are the norm. " Files are huge by traditional standards. ! POSIX-like
Cloud background Google File System! Warehouse scale systems " 10K-100K nodes " 50MW (1 MW = 1,000 houses) " Power efficient! Located near cheap power! Passive cooling! Power Usage Effectiveness = Total
More informationHOW TO PLAN & EXECUTE A SUCCESSFUL CLOUD MIGRATION
HOW TO PLAN & EXECUTE A SUCCESSFUL CLOUD MIGRATION Steve Bertoldi, Solutions Director, MarkLogic Agenda Cloud computing and on premise issues Comparison of traditional vs cloud architecture Review of use
More informationWarehouse-Scale Computing
ecture 31 Computer Science 61C Spring 2017 April 7th, 2017 Warehouse-Scale Computing 1 New-School Machine Structures (It s a bit more complicated!) Software Hardware Parallel Requests Assigned to computer
More informationEMC RecoverPoint. EMC RecoverPoint Support
Support, page 1 Adding an Account, page 2 RecoverPoint Appliance Clusters, page 3 Replication Through Consistency Groups, page 4 Group Sets, page 22 System Tasks, page 24 Support protects storage array
More informationEECS 482 Introduction to Operating Systems
EECS 482 Introduction to Operating Systems Winter 2018 Harsha V. Madhyastha Multiple updates and reliability Data must survive crashes and power outages Assume: update of one block atomic and durable Challenge:
More informationCS 138: Google. CS 138 XVI 1 Copyright 2017 Thomas W. Doeppner. All rights reserved.
CS 138: Google CS 138 XVI 1 Copyright 2017 Thomas W. Doeppner. All rights reserved. Google Environment Lots (tens of thousands) of computers all more-or-less equal - processor, disk, memory, network interface
More informationTITLE: PRE-REQUISITE THEORY. 1. Introduction to Hadoop. 2. Cluster. Implement sort algorithm and run it using HADOOP
TITLE: Implement sort algorithm and run it using HADOOP PRE-REQUISITE Preliminary knowledge of clusters and overview of Hadoop and its basic functionality. THEORY 1. Introduction to Hadoop The Apache Hadoop
More informationLecture 10.1 A real SDN implementation: the Google B4 case. Antonio Cianfrani DIET Department Networking Group netlab.uniroma1.it
Lecture 10.1 A real SDN implementation: the Google B4 case Antonio Cianfrani DIET Department Networking Group netlab.uniroma1.it WAN WAN = Wide Area Network WAN features: Very expensive (specialized high-end
More informationNetwork Security Platform 8.1
8.1.7.5-8.1.3.43 M-series Release Notes Network Security Platform 8.1 Revision A Contents About this release New features Enhancements Resolved issues Installation instructions Known issues Product documentation
More informationMODELS OF DISTRIBUTED SYSTEMS
Distributed Systems Fö 2/3-1 Distributed Systems Fö 2/3-2 MODELS OF DISTRIBUTED SYSTEMS Basic Elements 1. Architectural Models 2. Interaction Models Resources in a distributed system are shared between
More informationMapReduce Spark. Some slides are adapted from those of Jeff Dean and Matei Zaharia
MapReduce Spark Some slides are adapted from those of Jeff Dean and Matei Zaharia What have we learnt so far? Distributed storage systems consistency semantics protocols for fault tolerance Paxos, Raft,
More informationCSE 124: Networked Services Fall 2009 Lecture-19
CSE 124: Networked Services Fall 2009 Lecture-19 Instructor: B. S. Manoj, Ph.D http://cseweb.ucsd.edu/classes/fa09/cse124 Some of these slides are adapted from various sources/individuals including but
More informationDistributed Computation Models
Distributed Computation Models SWE 622, Spring 2017 Distributed Software Engineering Some slides ack: Jeff Dean HW4 Recap https://b.socrative.com/ Class: SWE622 2 Review Replicating state machines Case
More informationLow-Latency Datacenters. John Ousterhout Platform Lab Retreat May 29, 2015
Low-Latency Datacenters John Ousterhout Platform Lab Retreat May 29, 2015 Datacenters: Scale and Latency Scale: 1M+ cores 1-10 PB memory 200 PB disk storage Latency: < 0.5 µs speed-of-light delay Most
More informationCSCI 466 Midterm Networks Fall 2013
CSCI 466 Midterm Networks Fall 2013 Name: This exam consists of 6 problems on the following 7 pages. You may use your single-sided hand-written 8 ½ x 11 note sheet and a calculator during the exam. No
More informationCloud Programming. Programming Environment Oct 29, 2015 Osamu Tatebe
Cloud Programming Programming Environment Oct 29, 2015 Osamu Tatebe Cloud Computing Only required amount of CPU and storage can be used anytime from anywhere via network Availability, throughput, reliability
More informationCS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring Lecture 20: Networks and Distributed Systems
S 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring 2003 Lecture 20: Networks and Distributed Systems 20.0 Main Points Motivation for distributed vs. centralized systems
More informationCS 138: Google. CS 138 XVII 1 Copyright 2016 Thomas W. Doeppner. All rights reserved.
CS 138: Google CS 138 XVII 1 Copyright 2016 Thomas W. Doeppner. All rights reserved. Google Environment Lots (tens of thousands) of computers all more-or-less equal - processor, disk, memory, network interface
More informationSecurity (and finale) Dan Ports, CSEP 552
Security (and finale) Dan Ports, CSEP 552 Today Security: what if parts of your distributed system are malicious? BFT: state machine replication Bitcoin: peer-to-peer currency Course wrap-up Security Too
More informationIntroduction to Database Services
Introduction to Database Services Shaun Pearce AWS Solutions Architect 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved Today s agenda Why managed database services? A non-relational
More informationCS370 Operating Systems
CS370 Operating Systems Colorado State University Yashwant K Malaiya Spring 2018 Lecture 24 Mass Storage, HDFS/Hadoop Slides based on Text by Silberschatz, Galvin, Gagne Various sources 1 1 FAQ What 2
More informationprecise rules that govern communication between two parties TCP/IP: the basic Internet protocols IP: Internet protocol (bottom level)
Protocols precise rules that govern communication between two parties TCP/IP: the basic Internet protocols IP: Internet protocol (bottom level) all packets shipped from network to network as IP packets
More informationGoogle File System and BigTable. and tiny bits of HDFS (Hadoop File System) and Chubby. Not in textbook; additional information
Subject 10 Fall 2015 Google File System and BigTable and tiny bits of HDFS (Hadoop File System) and Chubby Not in textbook; additional information Disclaimer: These abbreviated notes DO NOT substitute
More informationCS November 2017
Bigtable Highly available distributed storage Distributed Systems 18. Bigtable Built with semi-structured data in mind URLs: content, metadata, links, anchors, page rank User data: preferences, account
More informationCS155b: E-Commerce. Lecture 3: Jan 16, How Does the Internet Work? Acknowledgements: S. Bradner and R. Wang
CS155b: E-Commerce Lecture 3: Jan 16, 2001 How Does the Internet Work? Acknowledgements: S. Bradner and R. Wang Internet Protocols Design Philosophy ordered set of goals 1. multiplexed utilization of existing
More informationQuestion 1 (6 points) Compare circuit-switching and packet-switching networks based on the following criteria:
Question 1 (6 points) Compare circuit-switching and packet-switching networks based on the following criteria: (a) Reserving network resources ahead of data being sent: (2pts) In circuit-switching networks,
More informationScalability of web applications
Scalability of web applications CSCI 470: Web Science Keith Vertanen Copyright 2014 Scalability questions Overview What's important in order to build scalable web sites? High availability vs. load balancing
More informationInternet Technology. 06. Exam 1 Review Paul Krzyzanowski. Rutgers University. Spring 2016
Internet Technology 06. Exam 1 Review Paul Krzyzanowski Rutgers University Spring 2016 March 2, 2016 2016 Paul Krzyzanowski 1 Question 1 Defend or contradict this statement: for maximum efficiency, at
More informationNPTEL Course Jan K. Gopinath Indian Institute of Science
Storage Systems NPTEL Course Jan 2012 (Lecture 39) K. Gopinath Indian Institute of Science Google File System Non-Posix scalable distr file system for large distr dataintensive applications performance,
More informationFailure Models. Fault Tolerance. Failure Masking by Redundancy. Agreement in Faulty Systems
Fault Tolerance Fault cause of an error that might lead to failure; could be transient, intermittent, or permanent Fault tolerance a system can provide its services even in the presence of faults Requirements
More informationCS 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring Lecture 19: Networks and Distributed Systems
S 162 Operating Systems and Systems Programming Professor: Anthony D. Joseph Spring 2004 Lecture 19: Networks and Distributed Systems 19.0 Main Points Motivation for distributed vs. centralized systems
More informationLecture 16: Data Center Network Architectures
MIT 6.829: Computer Networks Fall 2017 Lecture 16: Data Center Network Architectures Scribe: Alex Lombardi, Danielle Olson, Nicholas Selby 1 Background on Data Centers Computing, storage, and networking
More informationCISC 7610 Lecture 5 Distributed multimedia databases. Topics: Scaling up vs out Replication Partitioning CAP Theorem NoSQL NewSQL
CISC 7610 Lecture 5 Distributed multimedia databases Topics: Scaling up vs out Replication Partitioning CAP Theorem NoSQL NewSQL Motivation YouTube receives 400 hours of video per minute That is 200M hours
More informationTHE DATACENTER AS A COMPUTER AND COURSE REVIEW
THE DATACENTER A A COMPUTER AND COURE REVIEW George Porter June 8, 2018 ATTRIBUTION These slides are released under an Attribution-NonCommercial-hareAlike 3.0 Unported (CC BY-NC-A 3.0) Creative Commons
More informationAgenda. 1 Web search. 2 Web search engines. 3 Web robots, crawler. 4 Focused Web crawling. 5 Web search vs Browsing. 6 Privacy, Filter bubble
Agenda EITF25 Internet - Web Search Anders Ardö EIT Electrical and Information Technology, Lund University November 28, 2013 A. Ardö, EIT EITF25 Internet - Web Search November 28, 2013 1 / 47 A. Ardö,
More informationCluster-Level Google How we use Colossus to improve storage efficiency
Cluster-Level Storage @ Google How we use Colossus to improve storage efficiency Denis Serenyi Senior Staff Software Engineer dserenyi@google.com November 13, 2017 Keynote at the 2nd Joint International
More informationFault Tolerance. Distributed Systems. September 2002
Fault Tolerance Distributed Systems September 2002 Basics A component provides services to clients. To provide services, the component may require the services from other components a component may depend
More informationPage 1. Key Value Storage" System Examples" Key Values: Examples "
CS162 Operating Systems and Systems Programming Lecture 15 Key-Value Storage, Network Protocols" October 22, 2012! Ion Stoica! http://inst.eecs.berkeley.edu/~cs162! Key Value Storage" Interface! put(key,
More informationDistributed Systems. 05r. Case study: Google Cluster Architecture. Paul Krzyzanowski. Rutgers University. Fall 2016
Distributed Systems 05r. Case study: Google Cluster Architecture Paul Krzyzanowski Rutgers University Fall 2016 1 A note about relevancy This describes the Google search cluster architecture in the mid
More information