CPSC 426/526. Cloud Computing. Ennan Zhai. Computer Science Department Yale University

Save this PDF as:

Size: px
Start display at page:

Download "CPSC 426/526. Cloud Computing. Ennan Zhai. Computer Science Department Yale University"

Transcription

1 CPSC 426/526 Cloud Computing Ennan Zhai Computer Science Department Yale University

2 Recall: Lec-7 In the lec-7, I talked about: - P2P vs Enterprise control - Firewall - NATs - Software defined network

3 Lecture Roadmap Cloud Computing Overview Challenges in the Clouds Distributed File Systems: GFS Data Process & Analysis: MapReduce Database: BigTable

4 What s the Cloud Computing

5 What s the Cloud Computing Cloud computing is a business model for enabling convenient network access to a shared pool of configurable resources which can be rapidly provisioned and released with minimal management effort or service provider interaction. --- according to NIST(National Institute of Standards and Technology)

6 Have You Used the Cloud?

7 Have You Used the Cloud?

8 Have You Used the Cloud?

9 Have You Used the Cloud?

10 Why We Like It?

11 Why We Like It? Why users like it? - Do not care where it is, it is just there - Access from any platform

12 Why We Like It? Why users like it? - Do not care where it is, it is just there - Access from any platform Cloud Services v.s. Traditional Distributed Systems

13 Why We Like It? Why users like it? - Do not care where it is, it is just there - Access from any platform Why CS researchers like it? - High-performance computation with less money - Lots of hard and interesting new challenges

14 Building Blocks What techniques are used to support cloud? - Internet - Smart and cheap personal devices - Robust and scalable software systems - Virtualization

15 Types of Cloud Services Three types of services: - Software as a Service (SaaS) - Analogy: Restaurant. Prepares&serves entire meal, does the dishes, etc - Platform as a Service (PaaS) - Analogy: Take-out food. Prepares meal but does not serve it. - Infrastructure as a Service (IaaS) - Analogy: Grocery store. Provides raw ingredients.

16 Types of Cloud Services Three types of services: - Software as a Service (SaaS) - Analogy: Restaurant. Prepares&serves entire meal, does the dishes, etc - Platform as a Service (PaaS) - Analogy: Take-out food. Prepares meal but does not serve it. - Infrastructure as a Service (IaaS) - Analogy: Grocery store. Provides raw ingredients.

17 Types of Cloud Services Three types of services: - Software as a Service (SaaS) - Analogy: Restaurant. Prepares&serves entire meal, does the dishes, etc - Platform as a Service (PaaS) - Analogy: Take-out food. Prepares meal but does not serve it. - Infrastructure as a Service (IaaS) - Analogy: Grocery store. Provides raw ingredients.

18 Types of Cloud Services Three types of services: - Software as a Service (SaaS) - Analogy: Restaurant. Prepares&serves entire meal, does the dishes, etc - Platform as a Service (PaaS) - Analogy: Take-out food. Prepares meal but does not serve it. - Infrastructure as a Service (IaaS) - Analogy: Grocery store. Provides raw ingredients.

19 Software as a Service (SaaS)

20 Software as a Service (SaaS) Cloud Provider (i.e., SaaS Provider) Application Middleware Hardware

21 Software as a Service (SaaS) Cloud Provider (i.e., SaaS Provider) Application Middleware Hardware SaaS provider offers an entire application - Word processor, spreadsheet, CRM software, etc. - Customer pays cloud provider - Example: Google Apps, Salesforce.com, etc.

22 Software as a Service (SaaS) Cloud Provider (i.e., SaaS Provider) Application Middleware Hardware SaaS provider offers an entire application - Word processor, spreadsheet, CRM software, etc. - Customer pays cloud provider - Example: Google Apps, Salesforce.com, etc.

23 Software as a Service (SaaS) Customer Cloud Provider (i.e., SaaS Provider) Application Middleware Hardware SaaS provider offers an entire application - Word processor, spreadsheet, CRM software, etc. - Customer pays cloud provider and uses the service - Example: Google Apps, Salesforce.com, etc.

24 Software as a Service (SaaS) Customer Cloud Provider (i.e., SaaS Provider) Application Middleware Hardware SaaS provider offers an entire application - Word processor, spreadsheet, CRM software, etc. - Customer pays cloud provider and uses the service - Example: Google Apps, Salesforce.com, etc.

25 Software as a Service (SaaS) Customer Cloud Provider (i.e., SaaS Provider) Application Middleware Hardware SaaS provider offers an entire application - Word processor, spreadsheet, CRM software, etc. - Customer pays cloud provider and uses the service - Example: Google Apps, Salesforce.com, etc.

26 SaaS Example: Gmail

27 SaaS Example: Gmail Gmail Provider Application Middleware Hardware

28 SaaS Example: Gmail Gmail Provider Application Middleware Hardware Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)

29 SaaS Example: Gmail Gmail Provider Application Middleware Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)

30 SaaS Example: Gmail Gmail Provider Application BigTable Middleware APIs Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)

31 SaaS Example: Gmail Gmail Provider Application Gmail BigTable Middleware APIs Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)

32 SaaS Example: Gmail Customer Gmail Provider Application Gmail BigTable Middleware APIs Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)

33 SaaS Example: Gmail Customer Gmail Provider Application Gmail BigTable Middleware APIs Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)

34 SaaS Example: Gmail Customer Gmail Provider Application Gmail BigTable Middleware APIs Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)

35 SaaS Example: Gmail Customer Gmail Provider Application Gmail BigTable Middleware APIs Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)

36 Platform as a Service (PaaS)

37 Platform as a Service (PaaS) Cloud Provider (i.e., PaaS Provider) Application Middleware Hardware Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - Customer pays SaaS provider for the service - SaaS provider pays the cloud for the platform - Example: Windows Azure, Google App Engine, etc.

38 Platform as a Service (PaaS) Cloud Provider (i.e., PaaS Provider) Application Middleware Hardware Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - Customer pays SaaS provider for the service - SaaS provider pays the cloud for the platform - Example: Windows Azure, Google App Engine, etc.

39 Platform as a Service (PaaS) App Provider Cloud Provider (i.e., PaaS Provider) Application Middleware Hardware Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - App provider pays the cloud for the platform - Customer pays App provider for the service - Example: Windows Azure, Google App Engine, etc.

40 Platform as a Service (PaaS) App Provider Cloud Provider (i.e., PaaS Provider) Application Middleware Hardware Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - App provider pays the cloud for the platform - Customer pays App provider for the service - Example: Windows Azure, Google App Engine, etc.

41 Platform as a Service (PaaS) App Provider Customer Application Middleware Hardware Cloud Provider (i.e., PaaS Provider) Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - App provider pays the cloud for the platform - Customer pays app provider for the service - Example: Windows Azure, Google App Engine, etc.

42 Platform as a Service (PaaS) App Provider Customer Application Middleware Hardware Cloud Provider (i.e., PaaS Provider) Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - App provider pays the cloud for the platform - Customer pays app provider for the service - Example: Windows Azure, Google App Engine, etc.

43 Platform as a Service (PaaS) App Provider Customer Application Middleware Hardware Cloud Provider (i.e., PaaS Provider) Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - App provider pays the cloud for the platform - Customer pays app provider for the service - Example: Windows Azure, Google App Engine, etc.

44 PaaS Example: Facebook

45 PaaS Example: Facebook Application Middleware Facebook Provider Hardware

46 PaaS Example: Facebook Facebook Provider Application Middleware Hardware Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - Third-party game applications - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations

47 PaaS Example: Facebook Facebook Provider Application Facebook Middleware APIs Facebook Hardware Clusters Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - Third-party game applications - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations

48 PaaS Example: Facebook Facebook Provider Application Facebook Middleware APIs Facebook Hardware Clusters Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - Third-party game applications - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations

49 PaaS Example: Facebook App Provider Facebook Application Game Facebook Middleware APIs Facebook Hardware Clusters Facebook Provider Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - App providers adopt their services (e.g., game) onto Facebook - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations

50 PaaS Example: Facebook App Provider Facebook Application Game Facebook Middleware APIs Facebook Hardware Clusters Facebook Provider Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - App providers adopt their services (e.g., game) onto Facebook - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations

51 PaaS Example: Facebook App Provider Customer Facebook Application Game Facebook Middleware APIs Facebook Hardware Clusters Facebook Provider Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - App providers adopt their services (e.g., game) onto Facebook - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations

52 PaaS Example: Facebook App Provider Customer Facebook Application Game Facebook Middleware APIs Facebook Hardware Clusters Facebook Provider Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - App providers adopt their services (e.g., game) onto Facebook - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations

53 PaaS Example: Facebook App Provider Customer Facebook Application Game Facebook Middleware APIs Facebook Hardware Clusters Facebook Provider Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - App providers adopt their services (e.g., game) onto Facebook - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations

54 Infrastructure as a Service (IaaS)

55 Infrastructure as a Service (IaaS) Cloud Provider (i.e., IaaS Provider) Application Middleware Hardware Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - Customer pays SaaS provider for the service - SaaS provider pays the cloud for the resources - Example: Amazon Web Services, Rackspace Cloud, etc.

56 Infrastructure as a Service (IaaS) Cloud Provider (i.e., IaaS Provider) Application Middleware Hardware Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - Customer pays SaaS provider for the service - SaaS provider pays the cloud for the resources - Example: Amazon Web Services, Rackspace Cloud, etc.

57 Infrastructure as a Service (IaaS) App Provider Cloud Provider (i.e., IaaS Provider) Application Middleware Hardware Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - App provider pays the cloud for the resources - Customer pays App provider for the service - Example: Amazon Web Services, Rackspace Cloud, etc.

58 Infrastructure as a Service (IaaS) App Provider Cloud Provider (i.e., IaaS Provider) Application Middleware Hardware Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - App provider pays the cloud for the resources - Customer pays App provider for the service - Example: Amazon Web Services, Rackspace Cloud, etc.

59 Infrastructure as a Service (IaaS) App Provider Customer Application Middleware Hardware Cloud Provider (i.e., IaaS Provider) Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - App provider pays the cloud for the resources - Customer pays App provider for the service - Example: Amazon Web Services, Rackspace Cloud, etc.

60 Infrastructure as a Service (IaaS) App Provider Customer Application Middleware Hardware Cloud Provider (i.e., IaaS Provider) Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - App provider pays the cloud for the resources - Customer pays App provider for the service - Example: Amazon Web Services, Rackspace Cloud, etc.

61 Infrastructure as a Service (IaaS) App Provider Customer Application Middleware Hardware Cloud Provider (i.e., IaaS Provider) Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - App provider pays the cloud for the resources - Customer pays App provider for the service - Example: Amazon Web Services, Rackspace Cloud, etc.

62 IaaS Example: EC2 and S3 Application Middleware Amazon Hardware

63 IaaS Example: EC2 and S3 Application Middleware Amazon EC2 Hardware S3

64 IaaS Example: EC2 and S3 Netflix Provider Amazon Application Middleware EC2 Hardware S3 Netflix (app) heavily depends on Amazon AWS: - Media files are stored in S3 - Transcoding to target devices (e.g., ipad) using EC2 - Analysis of streaming sessions based on Elastic MapReduce

65 IaaS Example: EC2 and S3 Netflix Provider Amazon Application Middleware EC2 Hardware S3 Netflix (app) heavily depends on Amazon AWS: - Media files are stored in S3 - Transcoding to target devices (e.g., ipad) using EC2 - Analysis of streaming sessions based on Elastic MapReduce

66 IaaS Example: EC2 and S3 Netflix Provider Application Netflix Middleware Amazon EC2 Hardware S3 Netflix (app) heavily depends on Amazon AWS: - Media files are stored in S3 - Transcoding to target devices (e.g., ipad) using EC2 - Analysis of streaming sessions based on Elastic MapReduce

67 IaaS Example: EC2 and S3 Netflix Provider Application Netflix Middleware Amazon EC2 Hardware S3 Netflix (app) heavily depends on Amazon AWS: - Media files are stored in S3 - Transcoding to target devices (e.g., ipad) using EC2 - Analysis of streaming sessions based on Elastic MapReduce

68 IaaS Example: EC2 and S3 Netflix Provider Customer Application Netflix Middleware Amazon EC2 Hardware S3 Netflix (app) heavily depends on Amazon AWS: - Media files are stored in S3 - Transcoding to target devices (e.g., ipad) using EC2 - Analysis of streaming sessions based on Elastic MapReduce

69 IaaS Example: EC2 and S3 Netflix Provider Customer Application Netflix Middleware Amazon EC2 Hardware S3 Netflix (app) heavily depends on Amazon AWS: - Media files are stored in S3 - Transcoding to target devices (e.g., ipad) using EC2 - Analysis of streaming sessions based on Elastic MapReduce

70 Types of Cloud Services Three types of services: - Software as a Service (SaaS) - Analogy: Restaurant. Prepares&serves entire meal, does the dishes, etc - Platform as a Service (PaaS) - Analogy: Take-out food. Prepares meal but does not serve it. - Infrastructure as a Service (IaaS) - Analogy: Grocery store. Provides raw ingredients.

71 Types of Cloud Services Three types of services: - Software as a Service (SaaS) - Analogy: Restaurant. Prepares&serves entire meal, does the dishes, etc Zoo? - Platform as a Service (PaaS) - Analogy: Take-out food. Prepares meal but does not serve it. - Infrastructure as a Service (IaaS) - Analogy: Grocery store. Provides raw ingredients.

72 The Major Cloud Providers Amazon is the big player: - Infrastructure as a service (e.g., EC2) - Storage as a service (e.g., S3) But there are many others: - Microsoft Azure: It has similar services to Amazon, with an emphasis on.net programming model - Google App Engine: It offers programming interface, Hadoop, also software as a service, e.g., Gmail and Google Docs - IBM, HP, Yahoo!: They seem to focus on enterprise scale cloud apps

73 The Major Cloud Providers Amazon is the big player: - Infrastructure as a service (e.g., EC2) - Storage as a service (e.g., S3) But there are many others: - Microsoft Azure: It has similar services to Amazon, with an emphasis on.net programming model - Google App Engine: It offers programming interface, Hadoop, also software as a service, e.g., Gmail and Google Docs - IBM, HP, Yahoo!: They seem to focus on enterprise scale cloud apps

74 Lecture Roadmap Cloud Computing Overview Challenges in the Clouds Distributed File Systems: GFS Data Process & Analysis: MapReduce Database: BigTable

75 Challenges? In the cloud, we have much more data and users than before

76 PC Data! Users! Traffic!

77 Data! Users! Traffic! PC What if one computer is not enough? - Buy a bigger (server-class) computer

78 Data! Users! Traffic! PC Server What if one computer is not enough? - Buy a bigger (server-class) computer

79 Data! Users! Traffic! PC Server What if one computer is not enough? - Buy a bigger (server-class) computer What if the biggest computer is not enough? - Buy many computers

80 Data! Users! Traffic! PC Server Cluster What if one computer is not enough? - Buy a bigger (server-class) computer What if the biggest computer is not enough? - Buy many computers

81 Data! Users! Traffic!

82 Rack Data! Users! Traffic!

83 Rack Data! Users! Traffic! Network switches (connects nodes with each other and with other racks)

84 Rack Data! Users! Traffic! Network switches (connects nodes with each other and with other racks) Many nodes/blades (often identical)

85 Rack Data! Users! Traffic! Network switches (connects nodes with each other and with other racks) Many nodes/blades (often identical) Storage device(s)

86 Data! Users! Traffic! PC Server Cluster What if cluster is too big to fit into machine room? - Build a separate building for the cluster - Building can have lots of cooling and power - Result: Data center

87 Data! Users! Traffic! PC Server Cluster What if cluster is too big to fit into machine room? - Build a separate building for the cluster - Building can have lots of cooling and power - Result: Data center

88 Data! Users! Traffic! PC Server Cluster Data center What if cluster is too big to fit into machine room? - Build a separate building for the cluster - Building can have lots of cooling and power - Result: Data center

89 Google s Datacenter in Oregon

90 Google s Datacenter in Oregon Data centers (size of a football field)

91 Google s Datacenter in Oregon Data centers (size of a football field) A warehouse-sized computer - A single data center can easily contain 10,000 racks with 100 cores in each rack (1,000,000 cores total)

92 Google s Datacenter Locations

93 Google s Datacenter Locations

94 Google s Datacenter Locations

95 Challenges? How to manage a huge group of data? - How to store the data? - How to process and extract something from the data? - How to handle multiple availability and consistency? - How to preserve the data privacy?

96 Example: Google How to manage a huge group of data? Google File System & BigTable - How to store the data? - How to process and extract something from the data? - How to handle multiple availability Paxos and consistency? - How to preserve the data privacy? MapReduce

97 Example: Google

98 Example: Google Google File System (GFS) - Lec 8

99 Example: Google MapReduce - Lec 9 BigTable - Lec 9 Google File System (GFS) - Lec 8

100 Example: Google Google Applications, e.g., Gmail and Google Map MapReduce - Lec 9 BigTable - Lec 9 Google File System (GFS) - Lec 8

101 Another OpenSource Example: Apache Apache Applications Apache MapReduce Apache HBase Hadoop Distributed File System (HDFS)

102 Lecture Roadmap Cloud Computing Overview Challenges in the Clouds Distributed File Systems: GFS Data Process & Analysis: MapReduce Database: BigTable

103 The Google File System [SOSP 03] GFS aims to offer file services for google applications: - Scalable distributed file system - Designed for large data-intensive applications - Fault-tolerant; runs on commodity hardware - Delivers high performance to a large number of clients

104 NFS vs GFS Single machine makes part of its file system available to other machines Sequential or random access PRO: Simplicity, generality, transparency CON: Storage capacity and throughput limited by single server Single virtual file system spread over many machines Optimized for sequential read and local accesses PRO: High throughput, high capacity CON: Specialized for particular types of applications

105 NFS vs GFS Single machine makes part of its file system available to other machines Sequential or random access PRO: Simplicity, generality, transparency CON: Storage capacity and throughput limited by single server Single virtual file system spread over many machines Optimized for sequential read and local accesses PRO: High throughput, high capacity CON: Specialized for particular types of applications

106 Design Assumptions Assumptions for conventional file systems do not work - e.g., most files are small and short lifetimes Component failures are norm: - File system = thousands of storage machines - Much more complex cases in practice Files are huge. n-gb/tb files are norm - I/O operations and block size choices are affected

107 Design Assumptions Most files are appended, not overwritten - Random writes in a file are quite rare - Once created, files are mostly read; often seq Workload: - Large streaming reads - Large appends - Hundreds of processes append to a file concurrently

108 Design Assumptions Most files are appended, not overwritten - Random writes in a file are quite rare - Once created, files are mostly read; often seq Workload: - Large streaming reads - Large appends - Hundreds of processes append to a file concurrently GFS was originally built for web indexing.

109 File System Interfaces Operations - Basic: create/delete/open/close/read/write - Additional: snapshot/append

110 File System Interfaces Operations - Basic: create/delete/open/close/read/write - Additional: snapshot/append Allow multi-clients to append atomically without locking

111 GFS Master & Chunkservers GFS Servers GFS cluster: N Chunkservers + 1 Master chunk server... File system metadata Maps files to chunks chunk server Master chunk server chunk server chunk server Data Storage: fixed-size chunks Chunks replicated on several system

112 GFS Master & Chunkservers GFS Servers GFS cluster: N Chunkservers + 1 Master chunk server... File system metadata Maps files to chunks chunk server Master chunk server chunk server chunk server Data Storage: fixed-size chunks Chunks replicated on several system

113 GFS Master & Chunkservers GFS Servers GFS cluster: N Chunkservers + 1 Master chunk server... File system metadata Maps files to chunks chunk server Master chunk server chunk server chunk server Data Storage: fixed-size chunks Chunks replicated on several system

114 Files in GFS GFS Files A File is made of 64MB chunks That are replicated for fault-tolerance chunkserver chunkserver chunkserver Chunks live on chunkservers master checkpoint image operation log In-memory FS metadata The master manages the file system namespace

115 Files in GFS GFS Files A File is made of 64MB chunks That are replicated for fault-tolerance chunkserver chunkserver chunkserver Chunks live on chunkservers master checkpoint image operation log In-memory FS metadata The master manages the file system namespace

116 Files in GFS GFS Files A File is made of 64MB chunks That are replicated for fault-tolerance chunkserver chunkserver chunkserver Chunks live on chunkservers master checkpoint image operation log In-memory FS metadata The master manages the file system namespace

117 Files in GFS GFS Files A File is made of 64MB chunks That are replicated for fault-tolerance chunkserver chunkserver chunkserver Chunks live on chunkservers master checkpoint image operation log In-memory FS metadata The master manages the file system namespace

118 Chunks and Chunkservers Chunk size = 64 MB (default) - 32-bit checksum with each chunk Chunk handler - Globally unique 64-bit number - Assigned by the master when creation Where are chunks stored? - Local disk as Linux files Each chunk is replicated across multiple nodes - Three replicas (default) - More replicas for popular files to avoid hotspots

119 Master Maintains all file system metadata - Namespace, access control info, filename to chunk mapping, current locations of chunks Manages - Chunk leases (locks), garbage collection, chunk migration Periodically communicates with all nodes - Via heartbeat messages - To get state and send commands

120 Client Interaction Model GFS client code linked into each application - No OS-level API - Interacts with mater for metadata operations - Interacts directly with chunkservers for file data Clients cache metadata - E.g., location of a file s chunks

121 Reading Files 1. Contact the master node 2. Get file s metadata: list chunk handlers 3. Get the location of each of the chunk handles - Multiple replicated chunkservers per chunk 4. Contact any available chunkserver for chunk data

122 Master: Stores metadata only Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, 5 Reading Files Master holds metadata for the files Chunkservers hold the actual chunks - Each chunk is replicated three times When a client wants to read file 5 3 Chunkserver: Store Chunks

123 Master: Stores metadata only Reading Files Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, 5 bar? Client Master holds metadata for the files Chunkservers hold the actual chunks - Each chunk is replicated three times When a client wants to read file 5 3 Chunkserver: Store Chunks

124 Master: Stores metadata only Reading Files Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, 5 bar? < Chunkserver IPs, 3, 5 > Client Master holds metadata for the files Chunkservers hold the actual chunks - Each chunk is replicated three times When a client wants to read file 5 3 Chunkserver: Store Chunks

125 Master: Stores metadata only Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, Reading Files < Chunkserver IPs, 3, 5 > Client Master holds metadata for the files Chunkservers hold the actual chunks - Each chunk is replicated three times When a client wants to read file 5 3 Chunkserver: Store Chunks

126 Writing to Files Less frequent than reading Master grants a chunk lease to one of the replicas - This replica will be the primary replica chunkserver - Primary can request lease extensions, if needed - Master increases the chunk version number and informs replicas

127 Writing to Files Phase 1: Send data (Deliver data but do not write to the file) A client is given a list of replicas - Identifying the primary and secondaries Client writes to the closest replica - Pipeline forwarding Chunkservers store this data in a cache Client Chunkserver 1 Chunkserver 2 Chunkserver 3

128 Writing to Files Phase 2: Write data (Commit it to the file) Client waits for replicas to ack. receiving data Send a write request to the primary The primary is responsible for serialization of writes (applying then forwarding) Once all ack. have been received, the primary ack. the client Client Primary Chunkserver Secondary Chunkserver 2 Secondary Chunkserver 3

129 Note: Writing to Files - Data Flow (phase 1) is different from Control Flow (phase 2) Data Flow: - Client to chunkserver to chunkserver to chunkserver... - Ordering does not matter Control Flow (commit): - Client to primary to all secondaries - Ordering maintained

130 Note: Writing to Files - Data Flow (phase 1) is different from Control Flow (phase 2) Data Flow: - Client to chunkserver to chunkserver to chunkserver... - Ordering Chunk version does not numbers matter are used to detect if any replica has stale data (was not updated because it was down) Control Flow (commit): - Client to primary to all secondaries - Ordering maintained

131 Files in GFS Google cluster environment - Core services: GFS + cluster scheduling system - Typically 100s to 1000s of active jobs clusters, many with 1000s of machines - Pools of 1000s of clients

132 Hadoop Distributed File System (HDFS) A project based on GFS An open-source distributed file system - Distributed file storage - The same assumption as GFS

133 Hadoop Distributed File System (HDFS) NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

134 Hadoop Distributed File System (HDFS) HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

135 Hadoop Distributed File System (HDFS) Data HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

136 Hadoop Distributed File System (HDFS) Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

137 Hadoop Distributed File System (HDFS) Data Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

138 Hadoop Distributed File System (HDFS) Data Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5

139 Lecture Roadmap Cloud Computing Overview Challenges in the Clouds Distributed File Systems: GFS Data Process & Analysis: MapReduce Database: BigTable

Cloud Computing. Ennan Zhai. Computer Science at Yale University

Cloud Computing. Ennan Zhai. Computer Science at Yale University Cloud Computing Ennan Zhai Computer Science at Yale University ennan.zhai@yale.edu About Final Project About Final Project Important dates before demo session: - Oct 31: Proposal v1.0 - Nov 7: Source code

More information

Distributed Systems 16. Distributed File Systems II

Distributed Systems 16. Distributed File Systems II Distributed Systems 16. Distributed File Systems II Paul Krzyzanowski pxk@cs.rutgers.edu 1 Review NFS RPC-based access AFS Long-term caching CODA Read/write replication & disconnected operation DFS AFS

More information

Distributed File Systems II

Distributed File Systems II Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation

More information

CA485 Ray Walshe Google File System

CA485 Ray Walshe Google File System Google File System Overview Google File System is scalable, distributed file system on inexpensive commodity hardware that provides: Fault Tolerance File system runs on hundreds or thousands of storage

More information

MapReduce. U of Toronto, 2014

MapReduce. U of Toronto, 2014 MapReduce U of Toronto, 2014 http://www.google.org/flutrends/ca/ (2012) Average Searches Per Day: 5,134,000,000 2 Motivation Process lots of data Google processed about 24 petabytes of data per day in

More information

Google File System (GFS) and Hadoop Distributed File System (HDFS)

Google File System (GFS) and Hadoop Distributed File System (HDFS) Google File System (GFS) and Hadoop Distributed File System (HDFS) 1 Hadoop: Architectural Design Principles Linear scalability More nodes can do more work within the same time Linear on data size, linear

More information

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective Part II: Data Center Software Architecture: Topic 1: Distributed File Systems GFS (The Google File System) 1 Filesystems

More information

BigData and Map Reduce VITMAC03

BigData and Map Reduce VITMAC03 BigData and Map Reduce VITMAC03 1 Motivation Process lots of data Google processed about 24 petabytes of data per day in 2009. A single machine cannot serve all the data You need a distributed system to

More information

GFS Overview. Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures

GFS Overview. Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures GFS Overview Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures Interface: non-posix New op: record appends (atomicity matters,

More information

! Design constraints. " Component failures are the norm. " Files are huge by traditional standards. ! POSIX-like

! Design constraints.  Component failures are the norm.  Files are huge by traditional standards. ! POSIX-like Cloud background Google File System! Warehouse scale systems " 10K-100K nodes " 50MW (1 MW = 1,000 houses) " Power efficient! Located near cheap power! Passive cooling! Power Usage Effectiveness = Total

More information

The Google File System. Alexandru Costan

The Google File System. Alexandru Costan 1 The Google File System Alexandru Costan Actions on Big Data 2 Storage Analysis Acquisition Handling the data stream Data structured unstructured semi-structured Results Transactions Outline File systems

More information

Cloud Computing and Hadoop Distributed File System. UCSB CS170, Spring 2018

Cloud Computing and Hadoop Distributed File System. UCSB CS170, Spring 2018 Cloud Computing and Hadoop Distributed File System UCSB CS70, Spring 08 Cluster Computing Motivations Large-scale data processing on clusters Scan 000 TB on node @ 00 MB/s = days Scan on 000-node cluster

More information

Distributed Systems. 15. Distributed File Systems. Paul Krzyzanowski. Rutgers University. Fall 2016

Distributed Systems. 15. Distributed File Systems. Paul Krzyzanowski. Rutgers University. Fall 2016 Distributed Systems 15. Distributed File Systems Paul Krzyzanowski Rutgers University Fall 2016 1 Google Chubby 2 Chubby Distributed lock service + simple fault-tolerant file system Interfaces File access

More information

Distributed Systems. 15. Distributed File Systems. Paul Krzyzanowski. Rutgers University. Fall 2017

Distributed Systems. 15. Distributed File Systems. Paul Krzyzanowski. Rutgers University. Fall 2017 Distributed Systems 15. Distributed File Systems Paul Krzyzanowski Rutgers University Fall 2017 1 Google Chubby ( Apache Zookeeper) 2 Chubby Distributed lock service + simple fault-tolerant file system

More information

Distributed Filesystem

Distributed Filesystem Distributed Filesystem 1 How do we get data to the workers? NAS Compute Nodes SAN 2 Distributing Code! Don t move data to workers move workers to the data! - Store data on the local disks of nodes in the

More information

The Google File System (GFS)

The Google File System (GFS) 1 The Google File System (GFS) CS60002: Distributed Systems Antonio Bruto da Costa Ph.D. Student, Formal Methods Lab, Dept. of Computer Sc. & Engg., Indian Institute of Technology Kharagpur 2 Design constraints

More information

MapReduce for Scalable and Cloud Computing

MapReduce for Scalable and Cloud Computing 1 MapReduce for Scalable and Cloud Computing CS6323 Adapted from NETS212, U. Penn, USA 2 Overview Networked computing The need for scalability; scale of current services Scaling up: From PCs to data centers

More information

CSE 124: Networked Services Lecture-16

CSE 124: Networked Services Lecture-16 Fall 2010 CSE 124: Networked Services Lecture-16 Instructor: B. S. Manoj, Ph.D http://cseweb.ucsd.edu/classes/fa10/cse124 11/23/2010 CSE 124 Networked Services Fall 2010 1 Updates PlanetLab experiments

More information

MapReduce for Scalable and Cloud Computing

MapReduce for Scalable and Cloud Computing 1 MapReduce for Scalable and Cloud Computing CS6323 Adapted from NETS212, U. Penn, USA 2 Overview Networked computing The need for scalability; scale of current services Scaling up: From PCs to data centers

More information

CS /30/17. Paul Krzyzanowski 1. Google Chubby ( Apache Zookeeper) Distributed Systems. Chubby. Chubby Deployment.

CS /30/17. Paul Krzyzanowski 1. Google Chubby ( Apache Zookeeper) Distributed Systems. Chubby. Chubby Deployment. Distributed Systems 15. Distributed File Systems Google ( Apache Zookeeper) Paul Krzyzanowski Rutgers University Fall 2017 1 2 Distributed lock service + simple fault-tolerant file system Deployment Client

More information

CSE 124: Networked Services Fall 2009 Lecture-19

CSE 124: Networked Services Fall 2009 Lecture-19 CSE 124: Networked Services Fall 2009 Lecture-19 Instructor: B. S. Manoj, Ph.D http://cseweb.ucsd.edu/classes/fa09/cse124 Some of these slides are adapted from various sources/individuals including but

More information

The Google File System

The Google File System The Google File System By Ghemawat, Gobioff and Leung Outline Overview Assumption Design of GFS System Interactions Master Operations Fault Tolerance Measurements Overview GFS: Scalable distributed file

More information

Georgia Institute of Technology ECE6102 4/20/2009 David Colvin, Jimmy Vuong

Georgia Institute of Technology ECE6102 4/20/2009 David Colvin, Jimmy Vuong Georgia Institute of Technology ECE6102 4/20/2009 David Colvin, Jimmy Vuong Relatively recent; still applicable today GFS: Google s storage platform for the generation and processing of data used by services

More information

Distributed Systems. Lec 10: Distributed File Systems GFS. Slide acks: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung

Distributed Systems. Lec 10: Distributed File Systems GFS. Slide acks: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Distributed Systems Lec 10: Distributed File Systems GFS Slide acks: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung 1 Distributed File Systems NFS AFS GFS Some themes in these classes: Workload-oriented

More information

GFS: The Google File System

GFS: The Google File System GFS: The Google File System Brad Karp UCL Computer Science CS GZ03 / M030 24 th October 2014 Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one

More information

Hadoop File System S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y 11/15/2017

Hadoop File System S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y 11/15/2017 Hadoop File System 1 S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y Moving Computation is Cheaper than Moving Data Motivation: Big Data! What is BigData? - Google

More information

CLOUD-SCALE FILE SYSTEMS

CLOUD-SCALE FILE SYSTEMS Data Management in the Cloud CLOUD-SCALE FILE SYSTEMS 92 Google File System (GFS) Designing a file system for the Cloud design assumptions design choices Architecture GFS Master GFS Chunkservers GFS Clients

More information

ΕΠΛ 602:Foundations of Internet Technologies. Cloud Computing

ΕΠΛ 602:Foundations of Internet Technologies. Cloud Computing ΕΠΛ 602:Foundations of Internet Technologies Cloud Computing 1 Outline Bigtable(data component of cloud) Web search basedonch13of thewebdatabook 2 What is Cloud Computing? ACloudis an infrastructure, transparent

More information

Distributed System. Gang Wu. Spring,2018

Distributed System. Gang Wu. Spring,2018 Distributed System Gang Wu Spring,2018 Lecture7:DFS What is DFS? A method of storing and accessing files base in a client/server architecture. A distributed file system is a client/server-based application

More information

GFS: The Google File System. Dr. Yingwu Zhu

GFS: The Google File System. Dr. Yingwu Zhu GFS: The Google File System Dr. Yingwu Zhu Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one big CPU More storage, CPU required than one PC can

More information

18-hdfs-gfs.txt Thu Oct 27 10:05: Notes on Parallel File Systems: HDFS & GFS , Fall 2011 Carnegie Mellon University Randal E.

18-hdfs-gfs.txt Thu Oct 27 10:05: Notes on Parallel File Systems: HDFS & GFS , Fall 2011 Carnegie Mellon University Randal E. 18-hdfs-gfs.txt Thu Oct 27 10:05:07 2011 1 Notes on Parallel File Systems: HDFS & GFS 15-440, Fall 2011 Carnegie Mellon University Randal E. Bryant References: Ghemawat, Gobioff, Leung, "The Google File

More information

Hadoop and HDFS Overview. Madhu Ankam

Hadoop and HDFS Overview. Madhu Ankam Hadoop and HDFS Overview Madhu Ankam Why Hadoop We are gathering more data than ever Examples of data : Server logs Web logs Financial transactions Analytics Emails and text messages Social media like

More information

Distributed Systems. GFS / HDFS / Spanner

Distributed Systems. GFS / HDFS / Spanner 15-440 Distributed Systems GFS / HDFS / Spanner Agenda Google File System (GFS) Hadoop Distributed File System (HDFS) Distributed File Systems Replication Spanner Distributed Database System Paxos Replication

More information

HDFS Architecture. Gregory Kesden, CSE-291 (Storage Systems) Fall 2017

HDFS Architecture. Gregory Kesden, CSE-291 (Storage Systems) Fall 2017 HDFS Architecture Gregory Kesden, CSE-291 (Storage Systems) Fall 2017 Based Upon: http://hadoop.apache.org/docs/r3.0.0-alpha1/hadoopproject-dist/hadoop-hdfs/hdfsdesign.html Assumptions At scale, hardware

More information

goals monitoring, fault tolerance, auto-recovery (thousands of low-cost machines) handle appends efficiently (no random writes & sequential reads)

goals monitoring, fault tolerance, auto-recovery (thousands of low-cost machines) handle appends efficiently (no random writes & sequential reads) Google File System goals monitoring, fault tolerance, auto-recovery (thousands of low-cost machines) focus on multi-gb files handle appends efficiently (no random writes & sequential reads) co-design GFS

More information

The Google File System

The Google File System October 13, 2010 Based on: S. Ghemawat, H. Gobioff, and S.-T. Leung: The Google file system, in Proceedings ACM SOSP 2003, Lake George, NY, USA, October 2003. 1 Assumptions Interface Architecture Single

More information

Authors : Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung Presentation by: Vijay Kumar Chalasani

Authors : Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung Presentation by: Vijay Kumar Chalasani The Authors : Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung Presentation by: Vijay Kumar Chalasani CS5204 Operating Systems 1 Introduction GFS is a scalable distributed file system for large data intensive

More information

The Google File System

The Google File System The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung SOSP 2003 presented by Kun Suo Outline GFS Background, Concepts and Key words Example of GFS Operations Some optimizations in

More information

Google File System. Arun Sundaram Operating Systems

Google File System. Arun Sundaram Operating Systems Arun Sundaram Operating Systems 1 Assumptions GFS built with commodity hardware GFS stores a modest number of large files A few million files, each typically 100MB or larger (Multi-GB files are common)

More information

Introduction to data centers

Introduction to data centers Introduction to data centers Paolo Giaccone Notes for the class on Switching technologies for data centers Politecnico di Torino December 2017 Cloud computing Section 1 Cloud computing Giaccone (Politecnico

More information

Google File System. By Dinesh Amatya

Google File System. By Dinesh Amatya Google File System By Dinesh Amatya Google File System (GFS) Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung designed and implemented to meet rapidly growing demand of Google's data processing need a scalable

More information

The Google File System

The Google File System The Google File System Sanjay Ghemawat, Howard Gobioff and Shun Tak Leung Google* Shivesh Kumar Sharma fl4164@wayne.edu Fall 2015 004395771 Overview Google file system is a scalable distributed file system

More information

Google is Really Different.

Google is Really Different. COMP 790-088 -- Distributed File Systems Google File System 7 Google is Really Different. Huge Datacenters in 5+ Worldwide Locations Datacenters house multiple server clusters Coming soon to Lenior, NC

More information

The Google File System

The Google File System The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google* 정학수, 최주영 1 Outline Introduction Design Overview System Interactions Master Operation Fault Tolerance and Diagnosis Conclusions

More information

The Google File System

The Google File System The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google SOSP 03, October 19 22, 2003, New York, USA Hyeon-Gyu Lee, and Yeong-Jae Woo Memory & Storage Architecture Lab. School

More information

Google File System 2

Google File System 2 Google File System 2 goals monitoring, fault tolerance, auto-recovery (thousands of low-cost machines) focus on multi-gb files handle appends efficiently (no random writes & sequential reads) co-design

More information

Yuval Carmel Tel-Aviv University "Advanced Topics in Storage Systems" - Spring 2013

Yuval Carmel Tel-Aviv University Advanced Topics in Storage Systems - Spring 2013 Yuval Carmel Tel-Aviv University "Advanced Topics in About & Keywords Motivation & Purpose Assumptions Architecture overview & Comparison Measurements How does it fit in? The Future 2 About & Keywords

More information

CS6030 Cloud Computing. Acknowledgements. Today s Topics. Intro to Cloud Computing 10/20/15. Ajay Gupta, WMU-CS. WiSe Lab

CS6030 Cloud Computing. Acknowledgements. Today s Topics. Intro to Cloud Computing 10/20/15. Ajay Gupta, WMU-CS. WiSe Lab CS6030 Cloud Computing Ajay Gupta B239, CEAS Computer Science Department Western Michigan University ajay.gupta@wmich.edu 276-3104 1 Acknowledgements I have liberally borrowed these slides and material

More information

Introduction To Cloud Computing

Introduction To Cloud Computing Introduction To Cloud Computing What is Cloud Computing? Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g.,

More information

Introduction to Cloud Computing

Introduction to Cloud Computing Introduction to Cloud Computing Distributed File Systems 15 319, spring 2010 12 th Lecture, Feb 18 th Majd F. Sakr Lecture Motivation Quick Refresher on Files and File Systems Understand the importance

More information

Map-Reduce. Marco Mura 2010 March, 31th

Map-Reduce. Marco Mura 2010 March, 31th Map-Reduce Marco Mura (mura@di.unipi.it) 2010 March, 31th This paper is a note from the 2009-2010 course Strumenti di programmazione per sistemi paralleli e distribuiti and it s based by the lessons of

More information

Introduction to MapReduce

Introduction to MapReduce Basics of Cloud Computing Lecture 4 Introduction to MapReduce Satish Srirama Some material adapted from slides by Jimmy Lin, Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google Distributed

More information

18-hdfs-gfs.txt Thu Nov 01 09:53: Notes on Parallel File Systems: HDFS & GFS , Fall 2012 Carnegie Mellon University Randal E.

18-hdfs-gfs.txt Thu Nov 01 09:53: Notes on Parallel File Systems: HDFS & GFS , Fall 2012 Carnegie Mellon University Randal E. 18-hdfs-gfs.txt Thu Nov 01 09:53:32 2012 1 Notes on Parallel File Systems: HDFS & GFS 15-440, Fall 2012 Carnegie Mellon University Randal E. Bryant References: Ghemawat, Gobioff, Leung, "The Google File

More information

Google File System. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google fall DIP Heerak lim, Donghun Koo

Google File System. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google fall DIP Heerak lim, Donghun Koo Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google 2017 fall DIP Heerak lim, Donghun Koo 1 Agenda Introduction Design overview Systems interactions Master operation Fault tolerance

More information

Hadoop/MapReduce Computing Paradigm

Hadoop/MapReduce Computing Paradigm Hadoop/Reduce Computing Paradigm 1 Large-Scale Data Analytics Reduce computing paradigm (E.g., Hadoop) vs. Traditional database systems vs. Database Many enterprises are turning to Hadoop Especially applications

More information

HDFS: Hadoop Distributed File System. Sector: Distributed Storage System

HDFS: Hadoop Distributed File System. Sector: Distributed Storage System GFS: Google File System Google C/C++ HDFS: Hadoop Distributed File System Yahoo Java, Open Source Sector: Distributed Storage System University of Illinois at Chicago C++, Open Source 2 System that permanently

More information

A Study of Comparatively Analysis for HDFS and Google File System towards to Handle Big Data

A Study of Comparatively Analysis for HDFS and Google File System towards to Handle Big Data A Study of Comparatively Analysis for HDFS and Google File System towards to Handle Big Data Rajesh R Savaliya 1, Dr. Akash Saxena 2 1Research Scholor, Rai University, Vill. Saroda, Tal. Dholka Dist. Ahmedabad,

More information

Hadoop Distributed File System(HDFS)

Hadoop Distributed File System(HDFS) Hadoop Distributed File System(HDFS) Bu eğitim sunumları İstanbul Kalkınma Ajansı nın 2016 yılı Yenilikçi ve Yaratıcı İstanbul Mali Destek Programı kapsamında yürütülmekte olan TR10/16/YNY/0036 no lu İstanbul

More information

MapReduce & BigTable

MapReduce & BigTable CPSC 426/526 MapReduce & BigTable Ennan Zhai Computer Science Department Yale University Lecture Roadmap Cloud Computing Overview Challenges in the Clouds Distributed File Systems: GFS Data Process & Analysis:

More information

Distributed Systems. CS422/522 Lecture17 17 November 2014

Distributed Systems. CS422/522 Lecture17 17 November 2014 Distributed Systems CS422/522 Lecture17 17 November 2014 Lecture Outline Introduction Hadoop Chord What s a distributed system? What s a distributed system? A distributed system is a collection of loosely

More information

The Google File System

The Google File System The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung December 2003 ACM symposium on Operating systems principles Publisher: ACM Nov. 26, 2008 OUTLINE INTRODUCTION DESIGN OVERVIEW

More information

Google File System and BigTable. and tiny bits of HDFS (Hadoop File System) and Chubby. Not in textbook; additional information

Google File System and BigTable. and tiny bits of HDFS (Hadoop File System) and Chubby. Not in textbook; additional information Subject 10 Fall 2015 Google File System and BigTable and tiny bits of HDFS (Hadoop File System) and Chubby Not in textbook; additional information Disclaimer: These abbreviated notes DO NOT substitute

More information

NPTEL Course Jan K. Gopinath Indian Institute of Science

NPTEL Course Jan K. Gopinath Indian Institute of Science Storage Systems NPTEL Course Jan 2012 (Lecture 39) K. Gopinath Indian Institute of Science Google File System Non-Posix scalable distr file system for large distr dataintensive applications performance,

More information

CS435 Introduction to Big Data FALL 2018 Colorado State University. 11/7/2018 Week 12-B Sangmi Lee Pallickara. FAQs

CS435 Introduction to Big Data FALL 2018 Colorado State University. 11/7/2018 Week 12-B Sangmi Lee Pallickara. FAQs 11/7/2018 CS435 Introduction to Big Data - FALL 2018 W12.B.0.0 CS435 Introduction to Big Data 11/7/2018 CS435 Introduction to Big Data - FALL 2018 W12.B.1 FAQs Deadline of the Programming Assignment 3

More information

Data Clustering on the Parallel Hadoop MapReduce Model. Dimitrios Verraros

Data Clustering on the Parallel Hadoop MapReduce Model. Dimitrios Verraros Data Clustering on the Parallel Hadoop MapReduce Model Dimitrios Verraros Overview The purpose of this thesis is to implement and benchmark the performance of a parallel K- means clustering algorithm on

More information

Introduction to Hadoop. Owen O Malley Yahoo!, Grid Team

Introduction to Hadoop. Owen O Malley Yahoo!, Grid Team Introduction to Hadoop Owen O Malley Yahoo!, Grid Team owen@yahoo-inc.com Who Am I? Yahoo! Architect on Hadoop Map/Reduce Design, review, and implement features in Hadoop Working on Hadoop full time since

More information

Data Centers and Cloud Computing

Data Centers and Cloud Computing Data Centers and Cloud Computing CS677 Guest Lecture Tim Wood 1 Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises for server applications Internet

More information

The Google File System

The Google File System The Google File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung ACM SIGOPS 2003 {Google Research} Vaibhav Bajpai NDS Seminar 2011 Looking Back time Classics Sun NFS (1985) CMU Andrew FS (1988) Fault

More information

4/9/2018 Week 13-A Sangmi Lee Pallickara. CS435 Introduction to Big Data Spring 2018 Colorado State University. FAQs. Architecture of GFS

4/9/2018 Week 13-A Sangmi Lee Pallickara. CS435 Introduction to Big Data Spring 2018 Colorado State University. FAQs. Architecture of GFS W13.A.0.0 CS435 Introduction to Big Data W13.A.1 FAQs Programming Assignment 3 has been posted PART 2. LARGE SCALE DATA STORAGE SYSTEMS DISTRIBUTED FILE SYSTEMS Recitations Apache Spark tutorial 1 and

More information

Google File System, Replication. Amin Vahdat CSE 123b May 23, 2006

Google File System, Replication. Amin Vahdat CSE 123b May 23, 2006 Google File System, Replication Amin Vahdat CSE 123b May 23, 2006 Annoucements Third assignment available today Due date June 9, 5 pm Final exam, June 14, 11:30-2:30 Google File System (thanks to Mahesh

More information

Map Reduce. Yerevan.

Map Reduce. Yerevan. Map Reduce Erasmus+ @ Yerevan dacosta@irit.fr Divide and conquer at PaaS 100 % // Typical problem Iterate over a large number of records Extract something of interest from each Shuffle and sort intermediate

More information

7680: Distributed Systems

7680: Distributed Systems Cristina Nita-Rotaru 7680: Distributed Systems GFS. HDFS Required Reading } Google File System. S, Ghemawat, H. Gobioff and S.-T. Leung. SOSP 2003. } http://hadoop.apache.org } A Novel Approach to Improving

More information

Distributed Systems. 31. The Cloud: Infrastructure as a Service Paul Krzyzanowski. Rutgers University. Fall 2013

Distributed Systems. 31. The Cloud: Infrastructure as a Service Paul Krzyzanowski. Rutgers University. Fall 2013 Distributed Systems 31. The Cloud: Infrastructure as a Service Paul Krzyzanowski Rutgers University Fall 2013 December 12, 2014 2013 Paul Krzyzanowski 1 Motivation for the Cloud Self-service configuration

More information

Replications and Consensus

Replications and Consensus CPSC 426/526 Replications and Consensus Ennan Zhai Computer Science Department Yale University Recall: Lec-8 and 9 In the lec-8 and 9, we learned: - Cloud storage and data processing - File system: Google

More information

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective

ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective Part II: Software Infrastructure in Data Centers: Distributed File Systems 1 Permanently stores data Filesystems

More information

Data Centers and Cloud Computing. Slides courtesy of Tim Wood

Data Centers and Cloud Computing. Slides courtesy of Tim Wood Data Centers and Cloud Computing Slides courtesy of Tim Wood 1 Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises for server applications Internet

More information

Data Centers and Cloud Computing. Data Centers

Data Centers and Cloud Computing. Data Centers Data Centers and Cloud Computing Slides courtesy of Tim Wood 1 Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises for server applications Internet

More information

GFS. CS6450: Distributed Systems Lecture 5. Ryan Stutsman

GFS. CS6450: Distributed Systems Lecture 5. Ryan Stutsman GFS CS6450: Distributed Systems Lecture 5 Ryan Stutsman Some material taken/derived from Princeton COS-418 materials created by Michael Freedman and Kyle Jamieson at Princeton University. Licensed for

More information

HDFS Architecture Guide

HDFS Architecture Guide by Dhruba Borthakur Table of contents 1 Introduction...3 2 Assumptions and Goals...3 2.1 Hardware Failure... 3 2.2 Streaming Data Access...3 2.3 Large Data Sets...3 2.4 Simple Coherency Model... 4 2.5

More information

A BigData Tour HDFS, Ceph and MapReduce

A BigData Tour HDFS, Ceph and MapReduce A BigData Tour HDFS, Ceph and MapReduce These slides are possible thanks to these sources Jonathan Drusi - SCInet Toronto Hadoop Tutorial, Amir Payberah - Course in Data Intensive Computing SICS; Yahoo!

More information

GOOGLE FILE SYSTEM: MASTER Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung

GOOGLE FILE SYSTEM: MASTER Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung ECE7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective (Winter 2015) Presentation Report GOOGLE FILE SYSTEM: MASTER Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung

More information

Lecture 11 Hadoop & Spark

Lecture 11 Hadoop & Spark Lecture 11 Hadoop & Spark Dr. Wilson Rivera ICOM 6025: High Performance Computing Electrical and Computer Engineering Department University of Puerto Rico Outline Distributed File Systems Hadoop Ecosystem

More information

What Is Datacenter (Warehouse) Computing. Distributed and Parallel Technology. Datacenter Computing Architecture

What Is Datacenter (Warehouse) Computing. Distributed and Parallel Technology. Datacenter Computing Architecture What Is Datacenter (Warehouse) Computing Distributed and Parallel Technology Datacenter, Warehouse and Cloud Computing Hans-Wolfgang Loidl School of Mathematical and Computer Sciences Heriot-Watt University,

More information

Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler Yahoo! Sunnyvale, California USA {Shv, Hairong, SRadia,

Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler Yahoo! Sunnyvale, California USA {Shv, Hairong, SRadia, Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler Yahoo! Sunnyvale, California USA {Shv, Hairong, SRadia, Chansler}@Yahoo-Inc.com Presenter: Alex Hu } Introduction } Architecture } File

More information

DISTRIBUTED SYSTEMS [COMP9243] Lecture 9b: Distributed File Systems INTRODUCTION. Transparency: Flexibility: Slide 1. Slide 3.

DISTRIBUTED SYSTEMS [COMP9243] Lecture 9b: Distributed File Systems INTRODUCTION. Transparency: Flexibility: Slide 1. Slide 3. CHALLENGES Transparency: Slide 1 DISTRIBUTED SYSTEMS [COMP9243] Lecture 9b: Distributed File Systems ➀ Introduction ➁ NFS (Network File System) ➂ AFS (Andrew File System) & Coda ➃ GFS (Google File System)

More information

Next-Generation Cloud Platform

Next-Generation Cloud Platform Next-Generation Cloud Platform Jangwoo Kim Jun 24, 2013 E-mail: jangwoo@postech.ac.kr High Performance Computing Lab Department of Computer Science & Engineering Pohang University of Science and Technology

More information

SEEM3450 Engineering Innovation and Entrepreneurship

SEEM3450 Engineering Innovation and Entrepreneurship SEEM3450 Engineering Innovation and Entrepreneurship Cloud Computing Guest Lecture Gabriel Fung, Ph.D. 2017-10-26 What is Cloud Computing? According to NIST (National Institute of Standards and Technology)

More information

Google Disk Farm. Early days

Google Disk Farm. Early days Google Disk Farm Early days today CS 5204 Fall, 2007 2 Design Design factors Failures are common (built from inexpensive commodity components) Files large (multi-gb) mutation principally via appending

More information

CS 138: Google. CS 138 XVI 1 Copyright 2017 Thomas W. Doeppner. All rights reserved.

CS 138: Google. CS 138 XVI 1 Copyright 2017 Thomas W. Doeppner. All rights reserved. CS 138: Google CS 138 XVI 1 Copyright 2017 Thomas W. Doeppner. All rights reserved. Google Environment Lots (tens of thousands) of computers all more-or-less equal - processor, disk, memory, network interface

More information

Staggeringly Large Filesystems

Staggeringly Large Filesystems Staggeringly Large Filesystems Evan Danaher CS 6410 - October 27, 2009 Outline 1 Large Filesystems 2 GFS 3 Pond Outline 1 Large Filesystems 2 GFS 3 Pond Internet Scale Web 2.0 GFS Thousands of machines

More information

DISTRIBUTED SYSTEMS [COMP9243] Lecture 8a: Cloud Computing WHAT IS CLOUD COMPUTING? 2. Slide 3. Slide 1. Why is it called Cloud?

DISTRIBUTED SYSTEMS [COMP9243] Lecture 8a: Cloud Computing WHAT IS CLOUD COMPUTING? 2. Slide 3. Slide 1. Why is it called Cloud? DISTRIBUTED SYSTEMS [COMP9243] Lecture 8a: Cloud Computing Slide 1 Slide 3 ➀ What is Cloud Computing? ➁ X as a Service ➂ Key Challenges ➃ Developing for the Cloud Why is it called Cloud? services provided

More information

11/5/2018 Week 12-A Sangmi Lee Pallickara. CS435 Introduction to Big Data FALL 2018 Colorado State University

11/5/2018 Week 12-A Sangmi Lee Pallickara. CS435 Introduction to Big Data FALL 2018 Colorado State University 11/5/2018 CS435 Introduction to Big Data - FALL 2018 W12.A.0.0 CS435 Introduction to Big Data 11/5/2018 CS435 Introduction to Big Data - FALL 2018 W12.A.1 Consider a Graduate Degree in Computer Science

More information

Cloud Programming. Programming Environment Oct 29, 2015 Osamu Tatebe

Cloud Programming. Programming Environment Oct 29, 2015 Osamu Tatebe Cloud Programming Programming Environment Oct 29, 2015 Osamu Tatebe Cloud Computing Only required amount of CPU and storage can be used anytime from anywhere via network Availability, throughput, reliability

More information

NPTEL Course Jan K. Gopinath Indian Institute of Science

NPTEL Course Jan K. Gopinath Indian Institute of Science Storage Systems NPTEL Course Jan 2012 (Lecture 40) K. Gopinath Indian Institute of Science Google File System Non-Posix scalable distr file system for large distr dataintensive applications performance,

More information

Staggeringly Large File Systems. Presented by Haoyan Geng

Staggeringly Large File Systems. Presented by Haoyan Geng Staggeringly Large File Systems Presented by Haoyan Geng Large-scale File Systems How Large? Google s file system in 2009 (Jeff Dean, LADIS 09) - 200+ clusters - Thousands of machines per cluster - Pools

More information

DISTRIBUTED FILE SYSTEMS CARSTEN WEINHOLD

DISTRIBUTED FILE SYSTEMS CARSTEN WEINHOLD Department of Computer Science Institute of System Architecture, Operating Systems Group DISTRIBUTED FILE SYSTEMS CARSTEN WEINHOLD OUTLINE Classical distributed file systems NFS: Sun Network File System

More information

Service and Cloud Computing Lecture 10: DFS2 Prof. George Baciu PQ838

Service and Cloud Computing Lecture 10: DFS2   Prof. George Baciu PQ838 COMP4442 Service and Cloud Computing Lecture 10: DFS2 www.comp.polyu.edu.hk/~csgeorge/comp4442 Prof. George Baciu PQ838 csgeorge@comp.polyu.edu.hk 1 Preamble 2 Recall the Cloud Stack Model A B Application

More information

Programming model and implementation for processing and. Programs can be automatically parallelized and executed on a large cluster of machines

Programming model and implementation for processing and. Programs can be automatically parallelized and executed on a large cluster of machines A programming model in Cloud: MapReduce Programming model and implementation for processing and generating large data sets Users specify a map function to generate a set of intermediate key/value pairs

More information

NPTEL Course Jan K. Gopinath Indian Institute of Science

NPTEL Course Jan K. Gopinath Indian Institute of Science Storage Systems NPTEL Course Jan 2012 (Lecture 41) K. Gopinath Indian Institute of Science Lease Mgmt designed to minimize mgmt overhead at master a lease initially times out at 60 secs. primary can request

More information

Cloud & container monitoring , Lars Michelsen Check_MK Conference #4

Cloud & container monitoring , Lars Michelsen Check_MK Conference #4 Cloud & container monitoring 04.05.2018, Lars Michelsen Some cloud definitions Applications Data Runtime Middleware O/S Virtualization Servers Storage Networking Software-as-a-Service (SaaS) Applications

More information