CPSC 426/526. Cloud Computing. Ennan Zhai. Computer Science Department Yale University
|
|
|
- Sophie Hoover
- 6 years ago
- Views:
Transcription
1 CPSC 426/526 Cloud Computing Ennan Zhai Computer Science Department Yale University
2 Recall: Lec-7 In the lec-7, I talked about: - P2P vs Enterprise control - Firewall - NATs - Software defined network
3 Lecture Roadmap Cloud Computing Overview Challenges in the Clouds Distributed File Systems: GFS Data Process & Analysis: MapReduce Database: BigTable
4 What s the Cloud Computing
5 What s the Cloud Computing Cloud computing is a business model for enabling convenient network access to a shared pool of configurable resources which can be rapidly provisioned and released with minimal management effort or service provider interaction. --- according to NIST(National Institute of Standards and Technology)
6 Have You Used the Cloud?
7 Have You Used the Cloud?
8 Have You Used the Cloud?
9 Have You Used the Cloud?
10 Why We Like It?
11 Why We Like It? Why users like it? - Do not care where it is, it is just there - Access from any platform
12 Why We Like It? Why users like it? - Do not care where it is, it is just there - Access from any platform Cloud Services v.s. Traditional Distributed Systems
13 Why We Like It? Why users like it? - Do not care where it is, it is just there - Access from any platform Why CS researchers like it? - High-performance computation with less money - Lots of hard and interesting new challenges
14 Building Blocks What techniques are used to support cloud? - Internet - Smart and cheap personal devices - Robust and scalable software systems - Virtualization
15 Types of Cloud Services Three types of services: - Software as a Service (SaaS) - Analogy: Restaurant. Prepares&serves entire meal, does the dishes, etc - Platform as a Service (PaaS) - Analogy: Take-out food. Prepares meal but does not serve it. - Infrastructure as a Service (IaaS) - Analogy: Grocery store. Provides raw ingredients.
16 Types of Cloud Services Three types of services: - Software as a Service (SaaS) - Analogy: Restaurant. Prepares&serves entire meal, does the dishes, etc - Platform as a Service (PaaS) - Analogy: Take-out food. Prepares meal but does not serve it. - Infrastructure as a Service (IaaS) - Analogy: Grocery store. Provides raw ingredients.
17 Types of Cloud Services Three types of services: - Software as a Service (SaaS) - Analogy: Restaurant. Prepares&serves entire meal, does the dishes, etc - Platform as a Service (PaaS) - Analogy: Take-out food. Prepares meal but does not serve it. - Infrastructure as a Service (IaaS) - Analogy: Grocery store. Provides raw ingredients.
18 Types of Cloud Services Three types of services: - Software as a Service (SaaS) - Analogy: Restaurant. Prepares&serves entire meal, does the dishes, etc - Platform as a Service (PaaS) - Analogy: Take-out food. Prepares meal but does not serve it. - Infrastructure as a Service (IaaS) - Analogy: Grocery store. Provides raw ingredients.
19 Software as a Service (SaaS)
20 Software as a Service (SaaS) Cloud Provider (i.e., SaaS Provider) Application Middleware Hardware
21 Software as a Service (SaaS) Cloud Provider (i.e., SaaS Provider) Application Middleware Hardware SaaS provider offers an entire application - Word processor, spreadsheet, CRM software, etc. - Customer pays cloud provider - Example: Google Apps, Salesforce.com, etc.
22 Software as a Service (SaaS) Cloud Provider (i.e., SaaS Provider) Application Middleware Hardware SaaS provider offers an entire application - Word processor, spreadsheet, CRM software, etc. - Customer pays cloud provider - Example: Google Apps, Salesforce.com, etc.
23 Software as a Service (SaaS) Customer Cloud Provider (i.e., SaaS Provider) Application Middleware Hardware SaaS provider offers an entire application - Word processor, spreadsheet, CRM software, etc. - Customer pays cloud provider and uses the service - Example: Google Apps, Salesforce.com, etc.
24 Software as a Service (SaaS) Customer Cloud Provider (i.e., SaaS Provider) Application Middleware Hardware SaaS provider offers an entire application - Word processor, spreadsheet, CRM software, etc. - Customer pays cloud provider and uses the service - Example: Google Apps, Salesforce.com, etc.
25 Software as a Service (SaaS) Customer Cloud Provider (i.e., SaaS Provider) Application Middleware Hardware SaaS provider offers an entire application - Word processor, spreadsheet, CRM software, etc. - Customer pays cloud provider and uses the service - Example: Google Apps, Salesforce.com, etc.
26 SaaS Example: Gmail
27 SaaS Example: Gmail Gmail Provider Application Middleware Hardware
28 SaaS Example: Gmail Gmail Provider Application Middleware Hardware Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)
29 SaaS Example: Gmail Gmail Provider Application Middleware Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)
30 SaaS Example: Gmail Gmail Provider Application BigTable Middleware APIs Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)
31 SaaS Example: Gmail Gmail Provider Application Gmail BigTable Middleware APIs Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)
32 SaaS Example: Gmail Customer Gmail Provider Application Gmail BigTable Middleware APIs Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)
33 SaaS Example: Gmail Customer Gmail Provider Application Gmail BigTable Middleware APIs Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)
34 SaaS Example: Gmail Customer Gmail Provider Application Gmail BigTable Middleware APIs Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)
35 SaaS Example: Gmail Customer Gmail Provider Application Gmail BigTable Middleware APIs Hardware BigTable Outsourcing your software: - Distributed, replicated message store in BigTable - Weak consistency model for some operations (e.g., msg read) - Stronger consistency for others (e.g., send msg)
36 Platform as a Service (PaaS)
37 Platform as a Service (PaaS) Cloud Provider (i.e., PaaS Provider) Application Middleware Hardware Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - Customer pays SaaS provider for the service - SaaS provider pays the cloud for the platform - Example: Windows Azure, Google App Engine, etc.
38 Platform as a Service (PaaS) Cloud Provider (i.e., PaaS Provider) Application Middleware Hardware Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - Customer pays SaaS provider for the service - SaaS provider pays the cloud for the platform - Example: Windows Azure, Google App Engine, etc.
39 Platform as a Service (PaaS) App Provider Cloud Provider (i.e., PaaS Provider) Application Middleware Hardware Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - App provider pays the cloud for the platform - Customer pays App provider for the service - Example: Windows Azure, Google App Engine, etc.
40 Platform as a Service (PaaS) App Provider Cloud Provider (i.e., PaaS Provider) Application Middleware Hardware Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - App provider pays the cloud for the platform - Customer pays App provider for the service - Example: Windows Azure, Google App Engine, etc.
41 Platform as a Service (PaaS) App Provider Customer Application Middleware Hardware Cloud Provider (i.e., PaaS Provider) Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - App provider pays the cloud for the platform - Customer pays app provider for the service - Example: Windows Azure, Google App Engine, etc.
42 Platform as a Service (PaaS) App Provider Customer Application Middleware Hardware Cloud Provider (i.e., PaaS Provider) Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - App provider pays the cloud for the platform - Customer pays app provider for the service - Example: Windows Azure, Google App Engine, etc.
43 Platform as a Service (PaaS) App Provider Customer Application Middleware Hardware Cloud Provider (i.e., PaaS Provider) Cloud provides middleware/infrastructure - For example, Microsoft Common Language Runtime (CLR) - App provider pays the cloud for the platform - Customer pays app provider for the service - Example: Windows Azure, Google App Engine, etc.
44 PaaS Example: Facebook
45 PaaS Example: Facebook Application Middleware Facebook Provider Hardware
46 PaaS Example: Facebook Facebook Provider Application Middleware Hardware Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - Third-party game applications - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations
47 PaaS Example: Facebook Facebook Provider Application Facebook Middleware APIs Facebook Hardware Clusters Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - Third-party game applications - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations
48 PaaS Example: Facebook Facebook Provider Application Facebook Middleware APIs Facebook Hardware Clusters Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - Third-party game applications - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations
49 PaaS Example: Facebook App Provider Facebook Application Game Facebook Middleware APIs Facebook Hardware Clusters Facebook Provider Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - App providers adopt their services (e.g., game) onto Facebook - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations
50 PaaS Example: Facebook App Provider Facebook Application Game Facebook Middleware APIs Facebook Hardware Clusters Facebook Provider Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - App providers adopt their services (e.g., game) onto Facebook - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations
51 PaaS Example: Facebook App Provider Customer Facebook Application Game Facebook Middleware APIs Facebook Hardware Clusters Facebook Provider Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - App providers adopt their services (e.g., game) onto Facebook - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations
52 PaaS Example: Facebook App Provider Customer Facebook Application Game Facebook Middleware APIs Facebook Hardware Clusters Facebook Provider Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - App providers adopt their services (e.g., game) onto Facebook - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations
53 PaaS Example: Facebook App Provider Customer Facebook Application Game Facebook Middleware APIs Facebook Hardware Clusters Facebook Provider Facebook offers PaaS capabilities to App provider - Facebook APIs allow access to social network properties - App providers adopt their services (e.g., game) onto Facebook - Facebook itself also uses PaaS provided by its company, e.g., log analysis for recommendations
54 Infrastructure as a Service (IaaS)
55 Infrastructure as a Service (IaaS) Cloud Provider (i.e., IaaS Provider) Application Middleware Hardware Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - Customer pays SaaS provider for the service - SaaS provider pays the cloud for the resources - Example: Amazon Web Services, Rackspace Cloud, etc.
56 Infrastructure as a Service (IaaS) Cloud Provider (i.e., IaaS Provider) Application Middleware Hardware Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - Customer pays SaaS provider for the service - SaaS provider pays the cloud for the resources - Example: Amazon Web Services, Rackspace Cloud, etc.
57 Infrastructure as a Service (IaaS) App Provider Cloud Provider (i.e., IaaS Provider) Application Middleware Hardware Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - App provider pays the cloud for the resources - Customer pays App provider for the service - Example: Amazon Web Services, Rackspace Cloud, etc.
58 Infrastructure as a Service (IaaS) App Provider Cloud Provider (i.e., IaaS Provider) Application Middleware Hardware Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - App provider pays the cloud for the resources - Customer pays App provider for the service - Example: Amazon Web Services, Rackspace Cloud, etc.
59 Infrastructure as a Service (IaaS) App Provider Customer Application Middleware Hardware Cloud Provider (i.e., IaaS Provider) Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - App provider pays the cloud for the resources - Customer pays App provider for the service - Example: Amazon Web Services, Rackspace Cloud, etc.
60 Infrastructure as a Service (IaaS) App Provider Customer Application Middleware Hardware Cloud Provider (i.e., IaaS Provider) Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - App provider pays the cloud for the resources - Customer pays App provider for the service - Example: Amazon Web Services, Rackspace Cloud, etc.
61 Infrastructure as a Service (IaaS) App Provider Customer Application Middleware Hardware Cloud Provider (i.e., IaaS Provider) Cloud provides raw computing resources - Virtual machines, blade servers, hard disk, etc. - App provider pays the cloud for the resources - Customer pays App provider for the service - Example: Amazon Web Services, Rackspace Cloud, etc.
62 IaaS Example: EC2 and S3 Application Middleware Amazon Hardware
63 IaaS Example: EC2 and S3 Application Middleware Amazon EC2 Hardware S3
64 IaaS Example: EC2 and S3 Netflix Provider Amazon Application Middleware EC2 Hardware S3 Netflix (app) heavily depends on Amazon AWS: - Media files are stored in S3 - Transcoding to target devices (e.g., ipad) using EC2 - Analysis of streaming sessions based on Elastic MapReduce
65 IaaS Example: EC2 and S3 Netflix Provider Amazon Application Middleware EC2 Hardware S3 Netflix (app) heavily depends on Amazon AWS: - Media files are stored in S3 - Transcoding to target devices (e.g., ipad) using EC2 - Analysis of streaming sessions based on Elastic MapReduce
66 IaaS Example: EC2 and S3 Netflix Provider Application Netflix Middleware Amazon EC2 Hardware S3 Netflix (app) heavily depends on Amazon AWS: - Media files are stored in S3 - Transcoding to target devices (e.g., ipad) using EC2 - Analysis of streaming sessions based on Elastic MapReduce
67 IaaS Example: EC2 and S3 Netflix Provider Application Netflix Middleware Amazon EC2 Hardware S3 Netflix (app) heavily depends on Amazon AWS: - Media files are stored in S3 - Transcoding to target devices (e.g., ipad) using EC2 - Analysis of streaming sessions based on Elastic MapReduce
68 IaaS Example: EC2 and S3 Netflix Provider Customer Application Netflix Middleware Amazon EC2 Hardware S3 Netflix (app) heavily depends on Amazon AWS: - Media files are stored in S3 - Transcoding to target devices (e.g., ipad) using EC2 - Analysis of streaming sessions based on Elastic MapReduce
69 IaaS Example: EC2 and S3 Netflix Provider Customer Application Netflix Middleware Amazon EC2 Hardware S3 Netflix (app) heavily depends on Amazon AWS: - Media files are stored in S3 - Transcoding to target devices (e.g., ipad) using EC2 - Analysis of streaming sessions based on Elastic MapReduce
70 Types of Cloud Services Three types of services: - Software as a Service (SaaS) - Analogy: Restaurant. Prepares&serves entire meal, does the dishes, etc - Platform as a Service (PaaS) - Analogy: Take-out food. Prepares meal but does not serve it. - Infrastructure as a Service (IaaS) - Analogy: Grocery store. Provides raw ingredients.
71 Types of Cloud Services Three types of services: - Software as a Service (SaaS) - Analogy: Restaurant. Prepares&serves entire meal, does the dishes, etc Zoo? - Platform as a Service (PaaS) - Analogy: Take-out food. Prepares meal but does not serve it. - Infrastructure as a Service (IaaS) - Analogy: Grocery store. Provides raw ingredients.
72 The Major Cloud Providers Amazon is the big player: - Infrastructure as a service (e.g., EC2) - Storage as a service (e.g., S3) But there are many others: - Microsoft Azure: It has similar services to Amazon, with an emphasis on.net programming model - Google App Engine: It offers programming interface, Hadoop, also software as a service, e.g., Gmail and Google Docs - IBM, HP, Yahoo!: They seem to focus on enterprise scale cloud apps
73 The Major Cloud Providers Amazon is the big player: - Infrastructure as a service (e.g., EC2) - Storage as a service (e.g., S3) But there are many others: - Microsoft Azure: It has similar services to Amazon, with an emphasis on.net programming model - Google App Engine: It offers programming interface, Hadoop, also software as a service, e.g., Gmail and Google Docs - IBM, HP, Yahoo!: They seem to focus on enterprise scale cloud apps
74 Lecture Roadmap Cloud Computing Overview Challenges in the Clouds Distributed File Systems: GFS Data Process & Analysis: MapReduce Database: BigTable
75 Challenges? In the cloud, we have much more data and users than before
76 PC Data! Users! Traffic!
77 Data! Users! Traffic! PC What if one computer is not enough? - Buy a bigger (server-class) computer
78 Data! Users! Traffic! PC Server What if one computer is not enough? - Buy a bigger (server-class) computer
79 Data! Users! Traffic! PC Server What if one computer is not enough? - Buy a bigger (server-class) computer What if the biggest computer is not enough? - Buy many computers
80 Data! Users! Traffic! PC Server Cluster What if one computer is not enough? - Buy a bigger (server-class) computer What if the biggest computer is not enough? - Buy many computers
81 Data! Users! Traffic!
82 Rack Data! Users! Traffic!
83 Rack Data! Users! Traffic! Network switches (connects nodes with each other and with other racks)
84 Rack Data! Users! Traffic! Network switches (connects nodes with each other and with other racks) Many nodes/blades (often identical)
85 Rack Data! Users! Traffic! Network switches (connects nodes with each other and with other racks) Many nodes/blades (often identical) Storage device(s)
86 Data! Users! Traffic! PC Server Cluster What if cluster is too big to fit into machine room? - Build a separate building for the cluster - Building can have lots of cooling and power - Result: Data center
87 Data! Users! Traffic! PC Server Cluster What if cluster is too big to fit into machine room? - Build a separate building for the cluster - Building can have lots of cooling and power - Result: Data center
88 Data! Users! Traffic! PC Server Cluster Data center What if cluster is too big to fit into machine room? - Build a separate building for the cluster - Building can have lots of cooling and power - Result: Data center
89 Google s Datacenter in Oregon
90 Google s Datacenter in Oregon Data centers (size of a football field)
91 Google s Datacenter in Oregon Data centers (size of a football field) A warehouse-sized computer - A single data center can easily contain 10,000 racks with 100 cores in each rack (1,000,000 cores total)
92 Google s Datacenter Locations
93 Google s Datacenter Locations
94 Google s Datacenter Locations
95 Challenges? How to manage a huge group of data? - How to store the data? - How to process and extract something from the data? - How to handle multiple availability and consistency? - How to preserve the data privacy?
96 Example: Google How to manage a huge group of data? Google File System & BigTable - How to store the data? - How to process and extract something from the data? - How to handle multiple availability Paxos and consistency? - How to preserve the data privacy? MapReduce
97 Example: Google
98 Example: Google Google File System (GFS) - Lec 8
99 Example: Google MapReduce - Lec 9 BigTable - Lec 9 Google File System (GFS) - Lec 8
100 Example: Google Google Applications, e.g., Gmail and Google Map MapReduce - Lec 9 BigTable - Lec 9 Google File System (GFS) - Lec 8
101 Another OpenSource Example: Apache Apache Applications Apache MapReduce Apache HBase Hadoop Distributed File System (HDFS)
102 Lecture Roadmap Cloud Computing Overview Challenges in the Clouds Distributed File Systems: GFS Data Process & Analysis: MapReduce Database: BigTable
103 The Google File System [SOSP 03] GFS aims to offer file services for google applications: - Scalable distributed file system - Designed for large data-intensive applications - Fault-tolerant; runs on commodity hardware - Delivers high performance to a large number of clients
104 NFS vs GFS Single machine makes part of its file system available to other machines Sequential or random access PRO: Simplicity, generality, transparency CON: Storage capacity and throughput limited by single server Single virtual file system spread over many machines Optimized for sequential read and local accesses PRO: High throughput, high capacity CON: Specialized for particular types of applications
105 NFS vs GFS Single machine makes part of its file system available to other machines Sequential or random access PRO: Simplicity, generality, transparency CON: Storage capacity and throughput limited by single server Single virtual file system spread over many machines Optimized for sequential read and local accesses PRO: High throughput, high capacity CON: Specialized for particular types of applications
106 Design Assumptions Assumptions for conventional file systems do not work - e.g., most files are small and short lifetimes Component failures are norm: - File system = thousands of storage machines - Much more complex cases in practice Files are huge. n-gb/tb files are norm - I/O operations and block size choices are affected
107 Design Assumptions Most files are appended, not overwritten - Random writes in a file are quite rare - Once created, files are mostly read; often seq Workload: - Large streaming reads - Large appends - Hundreds of processes append to a file concurrently
108 Design Assumptions Most files are appended, not overwritten - Random writes in a file are quite rare - Once created, files are mostly read; often seq Workload: - Large streaming reads - Large appends - Hundreds of processes append to a file concurrently GFS was originally built for web indexing.
109 File System Interfaces Operations - Basic: create/delete/open/close/read/write - Additional: snapshot/append
110 File System Interfaces Operations - Basic: create/delete/open/close/read/write - Additional: snapshot/append Allow multi-clients to append atomically without locking
111 GFS Master & Chunkservers GFS Servers GFS cluster: N Chunkservers + 1 Master chunk server... File system metadata Maps files to chunks chunk server Master chunk server chunk server chunk server Data Storage: fixed-size chunks Chunks replicated on several system
112 GFS Master & Chunkservers GFS Servers GFS cluster: N Chunkservers + 1 Master chunk server... File system metadata Maps files to chunks chunk server Master chunk server chunk server chunk server Data Storage: fixed-size chunks Chunks replicated on several system
113 GFS Master & Chunkservers GFS Servers GFS cluster: N Chunkservers + 1 Master chunk server... File system metadata Maps files to chunks chunk server Master chunk server chunk server chunk server Data Storage: fixed-size chunks Chunks replicated on several system
114 Files in GFS GFS Files A File is made of 64MB chunks That are replicated for fault-tolerance chunkserver chunkserver chunkserver Chunks live on chunkservers master checkpoint image operation log In-memory FS metadata The master manages the file system namespace
115 Files in GFS GFS Files A File is made of 64MB chunks That are replicated for fault-tolerance chunkserver chunkserver chunkserver Chunks live on chunkservers master checkpoint image operation log In-memory FS metadata The master manages the file system namespace
116 Files in GFS GFS Files A File is made of 64MB chunks That are replicated for fault-tolerance chunkserver chunkserver chunkserver Chunks live on chunkservers master checkpoint image operation log In-memory FS metadata The master manages the file system namespace
117 Files in GFS GFS Files A File is made of 64MB chunks That are replicated for fault-tolerance chunkserver chunkserver chunkserver Chunks live on chunkservers master checkpoint image operation log In-memory FS metadata The master manages the file system namespace
118 Chunks and Chunkservers Chunk size = 64 MB (default) - 32-bit checksum with each chunk Chunk handler - Globally unique 64-bit number - Assigned by the master when creation Where are chunks stored? - Local disk as Linux files Each chunk is replicated across multiple nodes - Three replicas (default) - More replicas for popular files to avoid hotspots
119 Master Maintains all file system metadata - Namespace, access control info, filename to chunk mapping, current locations of chunks Manages - Chunk leases (locks), garbage collection, chunk migration Periodically communicates with all nodes - Via heartbeat messages - To get state and send commands
120 Client Interaction Model GFS client code linked into each application - No OS-level API - Interacts with mater for metadata operations - Interacts directly with chunkservers for file data Clients cache metadata - E.g., location of a file s chunks
121 Reading Files 1. Contact the master node 2. Get file s metadata: list chunk handlers 3. Get the location of each of the chunk handles - Multiple replicated chunkservers per chunk 4. Contact any available chunkserver for chunk data
122 Master: Stores metadata only Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, 5 Reading Files Master holds metadata for the files Chunkservers hold the actual chunks - Each chunk is replicated three times When a client wants to read file 5 3 Chunkserver: Store Chunks
123 Master: Stores metadata only Reading Files Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, 5 bar? Client Master holds metadata for the files Chunkservers hold the actual chunks - Each chunk is replicated three times When a client wants to read file 5 3 Chunkserver: Store Chunks
124 Master: Stores metadata only Reading Files Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, 5 bar? < Chunkserver IPs, 3, 5 > Client Master holds metadata for the files Chunkservers hold the actual chunks - Each chunk is replicated three times When a client wants to read file 5 3 Chunkserver: Store Chunks
125 Master: Stores metadata only Metadata: /usr/data/foo -> 1, 2, 4 /usr/data/bar -> 3, Reading Files < Chunkserver IPs, 3, 5 > Client Master holds metadata for the files Chunkservers hold the actual chunks - Each chunk is replicated three times When a client wants to read file 5 3 Chunkserver: Store Chunks
126 Writing to Files Less frequent than reading Master grants a chunk lease to one of the replicas - This replica will be the primary replica chunkserver - Primary can request lease extensions, if needed - Master increases the chunk version number and informs replicas
127 Writing to Files Phase 1: Send data (Deliver data but do not write to the file) A client is given a list of replicas - Identifying the primary and secondaries Client writes to the closest replica - Pipeline forwarding Chunkservers store this data in a cache Client Chunkserver 1 Chunkserver 2 Chunkserver 3
128 Writing to Files Phase 2: Write data (Commit it to the file) Client waits for replicas to ack. receiving data Send a write request to the primary The primary is responsible for serialization of writes (applying then forwarding) Once all ack. have been received, the primary ack. the client Client Primary Chunkserver Secondary Chunkserver 2 Secondary Chunkserver 3
129 Note: Writing to Files - Data Flow (phase 1) is different from Control Flow (phase 2) Data Flow: - Client to chunkserver to chunkserver to chunkserver... - Ordering does not matter Control Flow (commit): - Client to primary to all secondaries - Ordering maintained
130 Note: Writing to Files - Data Flow (phase 1) is different from Control Flow (phase 2) Data Flow: - Client to chunkserver to chunkserver to chunkserver... - Ordering Chunk version does not numbers matter are used to detect if any replica has stale data (was not updated because it was down) Control Flow (commit): - Client to primary to all secondaries - Ordering maintained
131 Files in GFS Google cluster environment - Core services: GFS + cluster scheduling system - Typically 100s to 1000s of active jobs clusters, many with 1000s of machines - Pools of 1000s of clients
132 Hadoop Distributed File System (HDFS) A project based on GFS An open-source distributed file system - Distributed file storage - The same assumption as GFS
133 Hadoop Distributed File System (HDFS) NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
134 Hadoop Distributed File System (HDFS) HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
135 Hadoop Distributed File System (HDFS) Data HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
136 Hadoop Distributed File System (HDFS) Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
137 Hadoop Distributed File System (HDFS) Data Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
138 Hadoop Distributed File System (HDFS) Data Metadata HDFS NameNode DataNode1 DataNode2 DataNode3 DataNode4 DataNode5
139 Lecture Roadmap Cloud Computing Overview Challenges in the Clouds Distributed File Systems: GFS Data Process & Analysis: MapReduce Database: BigTable
Cloud Computing. Ennan Zhai. Computer Science at Yale University
Cloud Computing Ennan Zhai Computer Science at Yale University ennan.zhai@yale.edu About Final Project About Final Project Important dates before demo session: - Oct 31: Proposal v1.0 - Nov 7: Source code
Distributed Systems 16. Distributed File Systems II
Distributed Systems 16. Distributed File Systems II Paul Krzyzanowski pxk@cs.rutgers.edu 1 Review NFS RPC-based access AFS Long-term caching CODA Read/write replication & disconnected operation DFS AFS
Distributed File Systems II
Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation
CA485 Ray Walshe Google File System
Google File System Overview Google File System is scalable, distributed file system on inexpensive commodity hardware that provides: Fault Tolerance File system runs on hundreds or thousands of storage
MapReduce. U of Toronto, 2014
MapReduce U of Toronto, 2014 http://www.google.org/flutrends/ca/ (2012) Average Searches Per Day: 5,134,000,000 2 Motivation Process lots of data Google processed about 24 petabytes of data per day in
Google File System (GFS) and Hadoop Distributed File System (HDFS)
Google File System (GFS) and Hadoop Distributed File System (HDFS) 1 Hadoop: Architectural Design Principles Linear scalability More nodes can do more work within the same time Linear on data size, linear
ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective
ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective Part II: Data Center Software Architecture: Topic 1: Distributed File Systems GFS (The Google File System) 1 Filesystems
BigData and Map Reduce VITMAC03
BigData and Map Reduce VITMAC03 1 Motivation Process lots of data Google processed about 24 petabytes of data per day in 2009. A single machine cannot serve all the data You need a distributed system to
GFS Overview. Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures
GFS Overview Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures Interface: non-posix New op: record appends (atomicity matters,
! Design constraints. " Component failures are the norm. " Files are huge by traditional standards. ! POSIX-like
Cloud background Google File System! Warehouse scale systems " 10K-100K nodes " 50MW (1 MW = 1,000 houses) " Power efficient! Located near cheap power! Passive cooling! Power Usage Effectiveness = Total
The Google File System. Alexandru Costan
1 The Google File System Alexandru Costan Actions on Big Data 2 Storage Analysis Acquisition Handling the data stream Data structured unstructured semi-structured Results Transactions Outline File systems
Cloud Computing and Hadoop Distributed File System. UCSB CS170, Spring 2018
Cloud Computing and Hadoop Distributed File System UCSB CS70, Spring 08 Cluster Computing Motivations Large-scale data processing on clusters Scan 000 TB on node @ 00 MB/s = days Scan on 000-node cluster
Distributed Systems. 15. Distributed File Systems. Paul Krzyzanowski. Rutgers University. Fall 2016
Distributed Systems 15. Distributed File Systems Paul Krzyzanowski Rutgers University Fall 2016 1 Google Chubby 2 Chubby Distributed lock service + simple fault-tolerant file system Interfaces File access
Distributed Systems. 15. Distributed File Systems. Paul Krzyzanowski. Rutgers University. Fall 2017
Distributed Systems 15. Distributed File Systems Paul Krzyzanowski Rutgers University Fall 2017 1 Google Chubby ( Apache Zookeeper) 2 Chubby Distributed lock service + simple fault-tolerant file system
Distributed Filesystem
Distributed Filesystem 1 How do we get data to the workers? NAS Compute Nodes SAN 2 Distributing Code! Don t move data to workers move workers to the data! - Store data on the local disks of nodes in the
The Google File System (GFS)
1 The Google File System (GFS) CS60002: Distributed Systems Antonio Bruto da Costa Ph.D. Student, Formal Methods Lab, Dept. of Computer Sc. & Engg., Indian Institute of Technology Kharagpur 2 Design constraints
MapReduce for Scalable and Cloud Computing
1 MapReduce for Scalable and Cloud Computing CS6323 Adapted from NETS212, U. Penn, USA 2 Overview Networked computing The need for scalability; scale of current services Scaling up: From PCs to data centers
CSE 124: Networked Services Lecture-16
Fall 2010 CSE 124: Networked Services Lecture-16 Instructor: B. S. Manoj, Ph.D http://cseweb.ucsd.edu/classes/fa10/cse124 11/23/2010 CSE 124 Networked Services Fall 2010 1 Updates PlanetLab experiments
MapReduce for Scalable and Cloud Computing
1 MapReduce for Scalable and Cloud Computing CS6323 Adapted from NETS212, U. Penn, USA 2 Overview Networked computing The need for scalability; scale of current services Scaling up: From PCs to data centers
CS /30/17. Paul Krzyzanowski 1. Google Chubby ( Apache Zookeeper) Distributed Systems. Chubby. Chubby Deployment.
Distributed Systems 15. Distributed File Systems Google ( Apache Zookeeper) Paul Krzyzanowski Rutgers University Fall 2017 1 2 Distributed lock service + simple fault-tolerant file system Deployment Client
CSE 124: Networked Services Fall 2009 Lecture-19
CSE 124: Networked Services Fall 2009 Lecture-19 Instructor: B. S. Manoj, Ph.D http://cseweb.ucsd.edu/classes/fa09/cse124 Some of these slides are adapted from various sources/individuals including but
The Google File System
The Google File System By Ghemawat, Gobioff and Leung Outline Overview Assumption Design of GFS System Interactions Master Operations Fault Tolerance Measurements Overview GFS: Scalable distributed file
Georgia Institute of Technology ECE6102 4/20/2009 David Colvin, Jimmy Vuong
Georgia Institute of Technology ECE6102 4/20/2009 David Colvin, Jimmy Vuong Relatively recent; still applicable today GFS: Google s storage platform for the generation and processing of data used by services
Distributed Systems. Lec 10: Distributed File Systems GFS. Slide acks: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung
Distributed Systems Lec 10: Distributed File Systems GFS Slide acks: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung 1 Distributed File Systems NFS AFS GFS Some themes in these classes: Workload-oriented
GFS: The Google File System
GFS: The Google File System Brad Karp UCL Computer Science CS GZ03 / M030 24 th October 2014 Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one
Hadoop File System S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y 11/15/2017
Hadoop File System 1 S L I D E S M O D I F I E D F R O M P R E S E N T A T I O N B Y B. R A M A M U R T H Y Moving Computation is Cheaper than Moving Data Motivation: Big Data! What is BigData? - Google
CLOUD-SCALE FILE SYSTEMS
Data Management in the Cloud CLOUD-SCALE FILE SYSTEMS 92 Google File System (GFS) Designing a file system for the Cloud design assumptions design choices Architecture GFS Master GFS Chunkservers GFS Clients
ΕΠΛ 602:Foundations of Internet Technologies. Cloud Computing
ΕΠΛ 602:Foundations of Internet Technologies Cloud Computing 1 Outline Bigtable(data component of cloud) Web search basedonch13of thewebdatabook 2 What is Cloud Computing? ACloudis an infrastructure, transparent
Distributed System. Gang Wu. Spring,2018
Distributed System Gang Wu Spring,2018 Lecture7:DFS What is DFS? A method of storing and accessing files base in a client/server architecture. A distributed file system is a client/server-based application
GFS: The Google File System. Dr. Yingwu Zhu
GFS: The Google File System Dr. Yingwu Zhu Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one big CPU More storage, CPU required than one PC can
18-hdfs-gfs.txt Thu Oct 27 10:05: Notes on Parallel File Systems: HDFS & GFS , Fall 2011 Carnegie Mellon University Randal E.
18-hdfs-gfs.txt Thu Oct 27 10:05:07 2011 1 Notes on Parallel File Systems: HDFS & GFS 15-440, Fall 2011 Carnegie Mellon University Randal E. Bryant References: Ghemawat, Gobioff, Leung, "The Google File
Hadoop and HDFS Overview. Madhu Ankam
Hadoop and HDFS Overview Madhu Ankam Why Hadoop We are gathering more data than ever Examples of data : Server logs Web logs Financial transactions Analytics Emails and text messages Social media like
Distributed Systems. GFS / HDFS / Spanner
15-440 Distributed Systems GFS / HDFS / Spanner Agenda Google File System (GFS) Hadoop Distributed File System (HDFS) Distributed File Systems Replication Spanner Distributed Database System Paxos Replication
HDFS Architecture. Gregory Kesden, CSE-291 (Storage Systems) Fall 2017
HDFS Architecture Gregory Kesden, CSE-291 (Storage Systems) Fall 2017 Based Upon: http://hadoop.apache.org/docs/r3.0.0-alpha1/hadoopproject-dist/hadoop-hdfs/hdfsdesign.html Assumptions At scale, hardware
goals monitoring, fault tolerance, auto-recovery (thousands of low-cost machines) handle appends efficiently (no random writes & sequential reads)
Google File System goals monitoring, fault tolerance, auto-recovery (thousands of low-cost machines) focus on multi-gb files handle appends efficiently (no random writes & sequential reads) co-design GFS
The Google File System
October 13, 2010 Based on: S. Ghemawat, H. Gobioff, and S.-T. Leung: The Google file system, in Proceedings ACM SOSP 2003, Lake George, NY, USA, October 2003. 1 Assumptions Interface Architecture Single
Authors : Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung Presentation by: Vijay Kumar Chalasani
The Authors : Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung Presentation by: Vijay Kumar Chalasani CS5204 Operating Systems 1 Introduction GFS is a scalable distributed file system for large data intensive
The Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung SOSP 2003 presented by Kun Suo Outline GFS Background, Concepts and Key words Example of GFS Operations Some optimizations in
Google File System. Arun Sundaram Operating Systems
Arun Sundaram Operating Systems 1 Assumptions GFS built with commodity hardware GFS stores a modest number of large files A few million files, each typically 100MB or larger (Multi-GB files are common)
Introduction to data centers
Introduction to data centers Paolo Giaccone Notes for the class on Switching technologies for data centers Politecnico di Torino December 2017 Cloud computing Section 1 Cloud computing Giaccone (Politecnico
Google File System. By Dinesh Amatya
Google File System By Dinesh Amatya Google File System (GFS) Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung designed and implemented to meet rapidly growing demand of Google's data processing need a scalable
The Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff and Shun Tak Leung Google* Shivesh Kumar Sharma fl4164@wayne.edu Fall 2015 004395771 Overview Google file system is a scalable distributed file system
Google is Really Different.
COMP 790-088 -- Distributed File Systems Google File System 7 Google is Really Different. Huge Datacenters in 5+ Worldwide Locations Datacenters house multiple server clusters Coming soon to Lenior, NC
The Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google* 정학수, 최주영 1 Outline Introduction Design Overview System Interactions Master Operation Fault Tolerance and Diagnosis Conclusions
The Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google SOSP 03, October 19 22, 2003, New York, USA Hyeon-Gyu Lee, and Yeong-Jae Woo Memory & Storage Architecture Lab. School
Google File System 2
Google File System 2 goals monitoring, fault tolerance, auto-recovery (thousands of low-cost machines) focus on multi-gb files handle appends efficiently (no random writes & sequential reads) co-design
Yuval Carmel Tel-Aviv University "Advanced Topics in Storage Systems" - Spring 2013
Yuval Carmel Tel-Aviv University "Advanced Topics in About & Keywords Motivation & Purpose Assumptions Architecture overview & Comparison Measurements How does it fit in? The Future 2 About & Keywords
CS6030 Cloud Computing. Acknowledgements. Today s Topics. Intro to Cloud Computing 10/20/15. Ajay Gupta, WMU-CS. WiSe Lab
CS6030 Cloud Computing Ajay Gupta B239, CEAS Computer Science Department Western Michigan University ajay.gupta@wmich.edu 276-3104 1 Acknowledgements I have liberally borrowed these slides and material
Introduction To Cloud Computing
Introduction To Cloud Computing What is Cloud Computing? Cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g.,
Introduction to Cloud Computing
Introduction to Cloud Computing Distributed File Systems 15 319, spring 2010 12 th Lecture, Feb 18 th Majd F. Sakr Lecture Motivation Quick Refresher on Files and File Systems Understand the importance
Map-Reduce. Marco Mura 2010 March, 31th
Map-Reduce Marco Mura (mura@di.unipi.it) 2010 March, 31th This paper is a note from the 2009-2010 course Strumenti di programmazione per sistemi paralleli e distribuiti and it s based by the lessons of
Introduction to MapReduce
Basics of Cloud Computing Lecture 4 Introduction to MapReduce Satish Srirama Some material adapted from slides by Jimmy Lin, Christophe Bisciglia, Aaron Kimball, & Sierra Michels-Slettvet, Google Distributed
18-hdfs-gfs.txt Thu Nov 01 09:53: Notes on Parallel File Systems: HDFS & GFS , Fall 2012 Carnegie Mellon University Randal E.
18-hdfs-gfs.txt Thu Nov 01 09:53:32 2012 1 Notes on Parallel File Systems: HDFS & GFS 15-440, Fall 2012 Carnegie Mellon University Randal E. Bryant References: Ghemawat, Gobioff, Leung, "The Google File
Google File System. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google fall DIP Heerak lim, Donghun Koo
Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google 2017 fall DIP Heerak lim, Donghun Koo 1 Agenda Introduction Design overview Systems interactions Master operation Fault tolerance
Hadoop/MapReduce Computing Paradigm
Hadoop/Reduce Computing Paradigm 1 Large-Scale Data Analytics Reduce computing paradigm (E.g., Hadoop) vs. Traditional database systems vs. Database Many enterprises are turning to Hadoop Especially applications
HDFS: Hadoop Distributed File System. Sector: Distributed Storage System
GFS: Google File System Google C/C++ HDFS: Hadoop Distributed File System Yahoo Java, Open Source Sector: Distributed Storage System University of Illinois at Chicago C++, Open Source 2 System that permanently
A Study of Comparatively Analysis for HDFS and Google File System towards to Handle Big Data
A Study of Comparatively Analysis for HDFS and Google File System towards to Handle Big Data Rajesh R Savaliya 1, Dr. Akash Saxena 2 1Research Scholor, Rai University, Vill. Saroda, Tal. Dholka Dist. Ahmedabad,
Hadoop Distributed File System(HDFS)
Hadoop Distributed File System(HDFS) Bu eğitim sunumları İstanbul Kalkınma Ajansı nın 2016 yılı Yenilikçi ve Yaratıcı İstanbul Mali Destek Programı kapsamında yürütülmekte olan TR10/16/YNY/0036 no lu İstanbul
MapReduce & BigTable
CPSC 426/526 MapReduce & BigTable Ennan Zhai Computer Science Department Yale University Lecture Roadmap Cloud Computing Overview Challenges in the Clouds Distributed File Systems: GFS Data Process & Analysis:
Distributed Systems. CS422/522 Lecture17 17 November 2014
Distributed Systems CS422/522 Lecture17 17 November 2014 Lecture Outline Introduction Hadoop Chord What s a distributed system? What s a distributed system? A distributed system is a collection of loosely
The Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung December 2003 ACM symposium on Operating systems principles Publisher: ACM Nov. 26, 2008 OUTLINE INTRODUCTION DESIGN OVERVIEW
Google File System and BigTable. and tiny bits of HDFS (Hadoop File System) and Chubby. Not in textbook; additional information
Subject 10 Fall 2015 Google File System and BigTable and tiny bits of HDFS (Hadoop File System) and Chubby Not in textbook; additional information Disclaimer: These abbreviated notes DO NOT substitute
NPTEL Course Jan K. Gopinath Indian Institute of Science
Storage Systems NPTEL Course Jan 2012 (Lecture 39) K. Gopinath Indian Institute of Science Google File System Non-Posix scalable distr file system for large distr dataintensive applications performance,
CS435 Introduction to Big Data FALL 2018 Colorado State University. 11/7/2018 Week 12-B Sangmi Lee Pallickara. FAQs
11/7/2018 CS435 Introduction to Big Data - FALL 2018 W12.B.0.0 CS435 Introduction to Big Data 11/7/2018 CS435 Introduction to Big Data - FALL 2018 W12.B.1 FAQs Deadline of the Programming Assignment 3
Data Clustering on the Parallel Hadoop MapReduce Model. Dimitrios Verraros
Data Clustering on the Parallel Hadoop MapReduce Model Dimitrios Verraros Overview The purpose of this thesis is to implement and benchmark the performance of a parallel K- means clustering algorithm on
Introduction to Hadoop. Owen O Malley Yahoo!, Grid Team
Introduction to Hadoop Owen O Malley Yahoo!, Grid Team owen@yahoo-inc.com Who Am I? Yahoo! Architect on Hadoop Map/Reduce Design, review, and implement features in Hadoop Working on Hadoop full time since
Data Centers and Cloud Computing
Data Centers and Cloud Computing CS677 Guest Lecture Tim Wood 1 Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises for server applications Internet
The Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung ACM SIGOPS 2003 {Google Research} Vaibhav Bajpai NDS Seminar 2011 Looking Back time Classics Sun NFS (1985) CMU Andrew FS (1988) Fault
4/9/2018 Week 13-A Sangmi Lee Pallickara. CS435 Introduction to Big Data Spring 2018 Colorado State University. FAQs. Architecture of GFS
W13.A.0.0 CS435 Introduction to Big Data W13.A.1 FAQs Programming Assignment 3 has been posted PART 2. LARGE SCALE DATA STORAGE SYSTEMS DISTRIBUTED FILE SYSTEMS Recitations Apache Spark tutorial 1 and
Google File System, Replication. Amin Vahdat CSE 123b May 23, 2006
Google File System, Replication Amin Vahdat CSE 123b May 23, 2006 Annoucements Third assignment available today Due date June 9, 5 pm Final exam, June 14, 11:30-2:30 Google File System (thanks to Mahesh
Map Reduce. Yerevan.
Map Reduce Erasmus+ @ Yerevan dacosta@irit.fr Divide and conquer at PaaS 100 % // Typical problem Iterate over a large number of records Extract something of interest from each Shuffle and sort intermediate
7680: Distributed Systems
Cristina Nita-Rotaru 7680: Distributed Systems GFS. HDFS Required Reading } Google File System. S, Ghemawat, H. Gobioff and S.-T. Leung. SOSP 2003. } http://hadoop.apache.org } A Novel Approach to Improving
Distributed Systems. 31. The Cloud: Infrastructure as a Service Paul Krzyzanowski. Rutgers University. Fall 2013
Distributed Systems 31. The Cloud: Infrastructure as a Service Paul Krzyzanowski Rutgers University Fall 2013 December 12, 2014 2013 Paul Krzyzanowski 1 Motivation for the Cloud Self-service configuration
Replications and Consensus
CPSC 426/526 Replications and Consensus Ennan Zhai Computer Science Department Yale University Recall: Lec-8 and 9 In the lec-8 and 9, we learned: - Cloud storage and data processing - File system: Google
ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective
ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective Part II: Software Infrastructure in Data Centers: Distributed File Systems 1 Permanently stores data Filesystems
Data Centers and Cloud Computing. Slides courtesy of Tim Wood
Data Centers and Cloud Computing Slides courtesy of Tim Wood 1 Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises for server applications Internet
Data Centers and Cloud Computing. Data Centers
Data Centers and Cloud Computing Slides courtesy of Tim Wood 1 Data Centers Large server and storage farms 1000s of servers Many TBs or PBs of data Used by Enterprises for server applications Internet
GFS. CS6450: Distributed Systems Lecture 5. Ryan Stutsman
GFS CS6450: Distributed Systems Lecture 5 Ryan Stutsman Some material taken/derived from Princeton COS-418 materials created by Michael Freedman and Kyle Jamieson at Princeton University. Licensed for
HDFS Architecture Guide
by Dhruba Borthakur Table of contents 1 Introduction...3 2 Assumptions and Goals...3 2.1 Hardware Failure... 3 2.2 Streaming Data Access...3 2.3 Large Data Sets...3 2.4 Simple Coherency Model... 4 2.5
A BigData Tour HDFS, Ceph and MapReduce
A BigData Tour HDFS, Ceph and MapReduce These slides are possible thanks to these sources Jonathan Drusi - SCInet Toronto Hadoop Tutorial, Amir Payberah - Course in Data Intensive Computing SICS; Yahoo!
GOOGLE FILE SYSTEM: MASTER Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung
ECE7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective (Winter 2015) Presentation Report GOOGLE FILE SYSTEM: MASTER Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung
Lecture 11 Hadoop & Spark
Lecture 11 Hadoop & Spark Dr. Wilson Rivera ICOM 6025: High Performance Computing Electrical and Computer Engineering Department University of Puerto Rico Outline Distributed File Systems Hadoop Ecosystem
What Is Datacenter (Warehouse) Computing. Distributed and Parallel Technology. Datacenter Computing Architecture
What Is Datacenter (Warehouse) Computing Distributed and Parallel Technology Datacenter, Warehouse and Cloud Computing Hans-Wolfgang Loidl School of Mathematical and Computer Sciences Heriot-Watt University,
Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler Yahoo! Sunnyvale, California USA {Shv, Hairong, SRadia,
Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler Yahoo! Sunnyvale, California USA {Shv, Hairong, SRadia, Chansler}@Yahoo-Inc.com Presenter: Alex Hu } Introduction } Architecture } File
DISTRIBUTED SYSTEMS [COMP9243] Lecture 9b: Distributed File Systems INTRODUCTION. Transparency: Flexibility: Slide 1. Slide 3.
![DISTRIBUTED SYSTEMS [COMP9243] Lecture 9b: Distributed File Systems INTRODUCTION. Transparency: Flexibility: Slide 1. Slide 3.](/thumbs/86/93978710.jpg "DISTRIBUTED SYSTEMS [COMP9243] Lecture 9b: Distributed File Systems INTRODUCTION. Transparency: Flexibility: Slide 1. Slide 3.")
CHALLENGES Transparency: Slide 1 DISTRIBUTED SYSTEMS [COMP9243] Lecture 9b: Distributed File Systems ➀ Introduction ➁ NFS (Network File System) ➂ AFS (Andrew File System) & Coda ➃ GFS (Google File System)
Next-Generation Cloud Platform
Next-Generation Cloud Platform Jangwoo Kim Jun 24, 2013 E-mail: jangwoo@postech.ac.kr High Performance Computing Lab Department of Computer Science & Engineering Pohang University of Science and Technology
SEEM3450 Engineering Innovation and Entrepreneurship
SEEM3450 Engineering Innovation and Entrepreneurship Cloud Computing Guest Lecture Gabriel Fung, Ph.D. 2017-10-26 What is Cloud Computing? According to NIST (National Institute of Standards and Technology)
Google Disk Farm. Early days
Google Disk Farm Early days today CS 5204 Fall, 2007 2 Design Design factors Failures are common (built from inexpensive commodity components) Files large (multi-gb) mutation principally via appending
CS 138: Google. CS 138 XVI 1 Copyright 2017 Thomas W. Doeppner. All rights reserved.
CS 138: Google CS 138 XVI 1 Copyright 2017 Thomas W. Doeppner. All rights reserved. Google Environment Lots (tens of thousands) of computers all more-or-less equal - processor, disk, memory, network interface
Staggeringly Large Filesystems
Staggeringly Large Filesystems Evan Danaher CS 6410 - October 27, 2009 Outline 1 Large Filesystems 2 GFS 3 Pond Outline 1 Large Filesystems 2 GFS 3 Pond Internet Scale Web 2.0 GFS Thousands of machines
DISTRIBUTED SYSTEMS [COMP9243] Lecture 8a: Cloud Computing WHAT IS CLOUD COMPUTING? 2. Slide 3. Slide 1. Why is it called Cloud?
![DISTRIBUTED SYSTEMS [COMP9243] Lecture 8a: Cloud Computing WHAT IS CLOUD COMPUTING? 2. Slide 3. Slide 1. Why is it called Cloud?](/thumbs/78/77201529.jpg "DISTRIBUTED SYSTEMS [COMP9243] Lecture 8a: Cloud Computing WHAT IS CLOUD COMPUTING? 2. Slide 3. Slide 1. Why is it called Cloud?")
DISTRIBUTED SYSTEMS [COMP9243] Lecture 8a: Cloud Computing Slide 1 Slide 3 ➀ What is Cloud Computing? ➁ X as a Service ➂ Key Challenges ➃ Developing for the Cloud Why is it called Cloud? services provided
11/5/2018 Week 12-A Sangmi Lee Pallickara. CS435 Introduction to Big Data FALL 2018 Colorado State University
11/5/2018 CS435 Introduction to Big Data - FALL 2018 W12.A.0.0 CS435 Introduction to Big Data 11/5/2018 CS435 Introduction to Big Data - FALL 2018 W12.A.1 Consider a Graduate Degree in Computer Science
Cloud Programming. Programming Environment Oct 29, 2015 Osamu Tatebe
Cloud Programming Programming Environment Oct 29, 2015 Osamu Tatebe Cloud Computing Only required amount of CPU and storage can be used anytime from anywhere via network Availability, throughput, reliability
NPTEL Course Jan K. Gopinath Indian Institute of Science
Storage Systems NPTEL Course Jan 2012 (Lecture 40) K. Gopinath Indian Institute of Science Google File System Non-Posix scalable distr file system for large distr dataintensive applications performance,
Staggeringly Large File Systems. Presented by Haoyan Geng
Staggeringly Large File Systems Presented by Haoyan Geng Large-scale File Systems How Large? Google s file system in 2009 (Jeff Dean, LADIS 09) - 200+ clusters - Thousands of machines per cluster - Pools
DISTRIBUTED FILE SYSTEMS CARSTEN WEINHOLD
Department of Computer Science Institute of System Architecture, Operating Systems Group DISTRIBUTED FILE SYSTEMS CARSTEN WEINHOLD OUTLINE Classical distributed file systems NFS: Sun Network File System
Service and Cloud Computing Lecture 10: DFS2 Prof. George Baciu PQ838
COMP4442 Service and Cloud Computing Lecture 10: DFS2 www.comp.polyu.edu.hk/~csgeorge/comp4442 Prof. George Baciu PQ838 csgeorge@comp.polyu.edu.hk 1 Preamble 2 Recall the Cloud Stack Model A B Application
Programming model and implementation for processing and. Programs can be automatically parallelized and executed on a large cluster of machines
A programming model in Cloud: MapReduce Programming model and implementation for processing and generating large data sets Users specify a map function to generate a set of intermediate key/value pairs
NPTEL Course Jan K. Gopinath Indian Institute of Science
Storage Systems NPTEL Course Jan 2012 (Lecture 41) K. Gopinath Indian Institute of Science Lease Mgmt designed to minimize mgmt overhead at master a lease initially times out at 60 secs. primary can request
Cloud & container monitoring , Lars Michelsen Check_MK Conference #4
Cloud & container monitoring 04.05.2018, Lars Michelsen Some cloud definitions Applications Data Runtime Middleware O/S Virtualization Servers Storage Networking Software-as-a-Service (SaaS) Applications