2/27/2019 Week 6-B Sangmi Lee Pallickara

Transcription

1 2/27/ Spring 2019 Week 6-B-1 CS535 BIG DATA FAQs Participation scores will be collected separately Sign-up page is up PART A. BIG DATA TECHNOLOGY 5. SCALABLE DISTRIBUTED FILE SYSTEMS: GOOGLE FILE SYSTEM I AND II Google had 2.5 million servers in How does the software demonstration work? Building your cluster Running tasks Eplaining your results Sangmi Lee Pallickara Computer Science, Colorado State University 2/27/ Spring 2019 Week 6-B-2 2/27/ Spring 2019 Week 6-B-3 Two breaks in the communication lines Boston London Rome Relaed Consistency Chicago LA Paris Miami A single machine can t partition So it does not have to worry about partition tolerance There is only one node. If it s up, it s available Sydney 2/27/ Spring 2019 Week 6-B-4 Eventually consistent At any time nodes may have replication inconsistencies 2/27/ Spring 2019 Week 6-B-5 In GFS the state of file region after mutation depends on Type of the mutation If there are no more updates (or updates can be ordered), eventually all nodes will be updated to the same value Success/Failure of the mutation Whether there were concurrent mutations Spring 2019 Colorado State University, page 1

2 2/27/ Spring 2019 Week 6-B-6 2/27/ Spring 2019 Week 6-B-7 GFS has a relaed consistency model Inconsistent and undefined Operation A Consistent: See the same data On all replicas Defined: If it is consistent AND Clients see mutation writes in its entirety Operation B 2/27/ Spring 2019 Week 6-B-8 2/27/ Spring 2019 Week 6-B-9 Consistent but undefined Operation A Defined Operation A Operation B Operation B 2/27/ Spring 2019 Week 6-B-10 2/27/ Spring 2019 Week 6-B-11 File state region after a mutation Write Record Append Serial success Concurrent success Defined Consistent but undefined defined interspersed with inconsistent Handling writes and appends to a file Failure Inconsistent Spring 2019 Colorado State University, page 2

3 2/27/ Spring 2019 Week 6-B-12 GFS uses leases to maintain consistent mutation order across replicas 2/27/ Spring 2019 Week 6-B-13 Lease mechanism designed to minimize communications with the master Lease has initial timeout of 60 seconds Master grants lease to one of the replicas Primary As long as chunk is being mutated Primary can request and receive etensions Primary picks serial-order For all mutations to the chunk Other replicas follow this order When applying mutations Etension requests/grants piggybacked over heart-beat messages 2/27/ Spring 2019 Week 6-B-14 Revocation and transfer of leases Master may revoke a lease before it epires If communications lost with primary Master can safely give lease to another replica Only After the lease period for old primary elapses 2/27/ Spring 2019 Week 6-B-15 How a write is actually performed 4. Write request Client 1. Chunkserver holding the current lease for the chunk and the location of the other replica MASTER 3*. 2. Identity of the primary and the locations of other replicas Secondary Replica A 7. Final Reply Primary Replica 5. Write request/ 6. Acknowledgement Secondary Replica B 3. Client pushes the data to all the replicas 2/27/ Spring 2019 Week 6-B-16 Client pushes data to all the replicas [1/2] 2/27/ Spring 2019 Week 6-B-17 Client pushes data to all the replicas [2/2] Each chunk server stores data in an LRU buffer until Data is used Aged out When chunk servers acknowledge receipt of data Client sends a write request to primary Primary assigns consecutive serial numbers to mutations Forwards to replicas Spring 2019 Colorado State University, page 3

4 2/27/ Spring 2019 Week 6-B-18 Data flow is decoupled from the control flow to utilize network efficiently Utilize each machine s network bandwidth Avoid network bottlenecks Avoid high-latency links Leverage network topology Estimate distances from IP addresses Pipeline the data transfer Once a chunkserver receives some data, it starts forwarding immediately. For transferring B bytes to R replicas 2/27/ Spring 2019 Week 6-B-19 Append Ideal elapsed time will be B/T+RL where: T is the network throughput L is latency to transfer bytes between two machines 2/27/ Spring 2019 Week 6-B-20 2/27/ Spring 2019 Week 6-B-21 Append: Record sizes and fragmentation Size is restricted to ¼ the chunk size Maimum size Inconsistent Regions Data 1 Data 1 Data 1 Data 2 Data 2 Data 2 Minimizes worst-case fragmentation Internal fragmentation in each chunk Data 3 Data 3 User will re-try to store Data 3 Failed Data 1 Data 2 Data 1 Data 2 Data 1 Data 2 Empty Data 3 Data 3 Data 3 Data 3 Data 3 Data 3 2/27/ Spring 2019 Week 6-B-22 What if record append fails at one of the replicas Client must retry the operation Replicas of same chunk may contain Different data Duplicates of the same record In whole or in part Replicas of chunks are not bit-wise identical! In most systems, replicas are identical 2/27/ Spring 2019 Week 6-B-23 GFS only guarantees that the data will be written at least once as an atomic unit For an operation to return success Data must be written at the same offset on all the replicas After the write, all replicas are as long as the end of the record Any future record will be assigned a higher offset or a different chunk Spring 2019 Colorado State University, page 4

5 2/27/ Spring 2019 Week 6-B-24 GFS client code implements the file system API 2/27/ Spring 2019 Week 6-B-25 Communications with master and chunk servers done transparently On behalf of apps that read or write data Interact with master for metadata Data-bearing communications directly to chunk servers Creating Snapshots 2/27/ Spring 2019 Week 6-B-26 Snapshots Copying file or directory tree almost instantaneously Minimizing any interruptions of ongoing mutations Providing checkpoint Users can commit later Rollback 2/27/ Spring 2019 Week 6-B-27 Snapshots allow you to make a copy of a file very fast 1. Master revokes outstanding leases for any chunks of the file (source) to be snapshot 2. Log the operation to disk 3. Update in-memory state Duplicate metadata of the source file 4. Newly created file points to the same chunks as the source 2/27/ Spring 2019 Week 6-B-28 When a client wants to write to a chunk C after the snapshot operation Master sees the reference count to C > 1 Pick new chunk-handle C Ask chunk-server with current replica of C Create new chunk C Data is copied locally, not over the network From this point, chunk handling of C is no different 2/27/ Spring 2019 Week 6-B-29 GFS does not have a per-directory structure that lists files in the directory Name spaces represented as a lookup table Maps full pathnames to metadata Each node has an associated read/write lock File creation does not require a lock on the directory structure No inode needs to be protected from modification Spring 2019 Colorado State University, page 5

6 2/27/ Spring 2019 Week 6-B-30 Each master operation acquires a set of locks before it runs Read lock prevents a directory from being deleted, renamed, or snapshotted Write lock on file names serialize attempts to create a file with the same twice If operation involves /d1/d2/ /dn/leaf Acquire read locks on all of the directory names /d1, /d1/d2,, /d1/d2/ /dn Read or write lock on full pathname /d1/d2/ /dn/leaf 2/27/ Spring 2019 Week 6-B-31 Locks are used to prevent operations during snapshots How do we present creating /home/user/foo While /home/user is being snapshotted to /save/user? /home/user is being snapshotted to /save/user Read locks on /home and /save Write lock on /home/user and /save/user To create file Read lock on /home and /home/user Write lock on /home/user/foo The two operations will be serialized because they try to obtain /home/user File creation does not require write lock on parent directory there is no directory Read locks on /home and /home/user Write lock on /home/user/foo 2/27/ Spring 2019 Week 6-B-32 2/27/ Spring 2019 Week 6-B-33 Garbage collection in GFS After a file is deleted, GFS does not reclaim space immediately Deletion of Files and Garbage Collection Done lazily during garbage collection at File and chunk levels 2/27/ Spring 2019 Week 6-B-34 Master logs a file s deletion immediately 2/27/ Spring 2019 Week 6-B-35 Garbage collection: When Master scans its chunk namespace File is renamed to a hidden name Includes deletion timestamp Master scans the file system namespace Delete if hidden file eisted for more than 3 days Identifies orphaned chunks Not reachable from any file Erase metadata for these chunks When file is removed from namespace In memory metadata is also removed Severs links to all its chunks! Spring 2019 Colorado State University, page 6

7 2/27/ Spring 2019 Week 6-B-36 The role of heart-beats in garbage collection Chunk server reports subset of chunks it currently has Master replies with identity of chunks no longer present Chunk server is now free to delete its replica of such chunks 2/27/ Spring 2019 Week 6-B-37 Stale chunks and issues If a chunk server fails AND misses mutations to the chunk The chunk replica becomes stale Working with a stale replica causes problems with: Correctness Consistency 2/27/ Spring 2019 Week 6-B-38 Aiding the detection of stale chunks Master maintains a chunk version number for each chunk Distinguish between stale and up-to-date chunks When master grants a new lease on chunk Increase version number Occurs BEFORE any Inform replicas client mutate Record new version persistently chunk 2/27/ Spring 2019 Week 6-B-39 If a replica is unavailable its version number will not be advanced When a chunk server restarts, it reports to the Master with the following: Set of Chunks Corresponding version numbers Used to detect stale replicas Remove stale replicas in regular garbage collection 2/27/ Spring 2019 Week 6-B-40 2/27/ Spring 2019 Week 6-B-41 Additional safeguards against stale replicas Include chunk version number When client requests chunk information Client/Chunk server verify version to make sure things are up-to-date During cloning operations Clone the most up-to-date chunk Clients and chunk servers epected to verify versioning information Inefficiencies Spring 2019 Colorado State University, page 7

8 2/27/ Spring 2019 Week 6-B-42 The master server is a single point of failure Master server restart takes several seconds Complete recovery takes several minutes Shadow servers eist Can handle reads of files In place of the master Requires a massive main memory 2/27/ Spring 2019 Week 6-B-43 The system is optimized for large files But not for a very large number of very small files Primary operation on files Long, sequential reads/writes Large number of random overwrites will clog things up quite a bit 2/27/ Spring 2019 Week 6-B-44 Consistency Issues: GFS epects clients to resolve inconsistencies File chunks may have gaps or duplicates of some records The client has to be able to deal with this 2/27/ Spring 2019 Week 6-B-45 Security model Originally None Operation is epected to be in a trusted environment Imagine doing this for a scientific application Portions of a massive array are corrupted Clients would have to detect this Detection is possible of course, but onerous! NOTE: HDFS handles gaps and duplicates 2/27/ Spring 2019 Week 6-B-46 2/27/ Spring 2019 Week 6-B-47 Storage Software: Colossus (GFS2) Net-generation cluster-level file system Google File System II Colossus Automatically sharded metadata layer Distributed Masters (64MB block size à 1MB) Data typically written using Reed-Solomon (1.5) Client-driven replication, encoding and replication Metadata space has enabled availability Why Reed-Solomon? Cost Especially with cross cluster replication More fleible cost vs. availability choices Google File System II: Dawn of the Multiplying Master Nodes, Spring 2019 Colorado State University, page 8

9 2/27/ Spring 2019 Week 6-B-48 2/27/ Spring 2019 Week 6-B-49 Reed-Solomon Codes Block-based error correcting codes Digital communication and storage Google File System II Colossus Reed-Solomon Codes Storage devices (including tape, CD, DVD, barcodes, etc) Wireless or mobile communications Satellite communications Digital TV High-speed modems SOURCE: Solomon_codes_for_coders 2/27/ Spring 2019 Week 6-B-50 2/27/ Spring 2019 Week 6-B-51 What does the R-S code do? Takes a block of digital data Adds etra redundant bits If an error happens, the R-S decoder processes each block and recovers original data Google File System II Colossus Quick overview with an eample Noise, Errors Data source Reed-Solomon Encoder Communication channel or storage devices Reed-Solomon Decoder Data Sink 2/27/ Spring 2019 Week 6-B-52 2/27/ Spring 2019 Week 6-B coding Original files are broken into 4 pieces 2 parity pieces are added Applying coding matri First piece of data ABCD, second piece of data EFGH Original Data = 1b 1c c 1b Spring 2019 Colorado State University, page 9

10 2/27/ Spring 2019 Week 6-B-54 2/27/ Spring 2019 Week 6-B-55 Data loss 2 of 6 rows are lost Without 2 rows = 1b 1c = 1b 1c c 1b c 1b /27/ Spring 2019 Week 6-B-56 2/27/ Spring 2019 Week 6-B-57 Multiplying each side with the inverted matri The Inverse Matri and the Coding Matri Cancel Out 8d f6 7b 01 f6 8d 01 7b 1b 1c c 1b d f6 7b 01 f6 8d 01 7b 1b 1c c 1b = 8d f6 7b 01 f6 8d 01 7b = 8d f6 7b 01 f6 8d 01 7b 2/27/ Spring 2019 Week 6-B-58 2/27/ Spring 2019 Week 6-B-59 Reconstructing the Original Data = 8d f6 7b 01 f6 8d 01 7b Google File System II Colossus Properties Spring 2019 Colorado State University, page 10

11 2/27/ Spring 2019 Week 6-B-60 2/27/ Spring 2019 Week 6-B-61 Properties of Reed-Solomon codes RS(n,k) with s-bit symbols Encoder takes k data symbols (blocks) of s bits each Adds parity symbols to make n symbol code word Eample RS(255,223) with 8 bit symbols Each code word contains 255 code word bytes 223 bytes are data and 32 bytes are parity n=255, k=223, s=8, 2t = 32, t=16 There are n-k parity symbols of s bits each A Reed-Solomon decoder can correct up to t symbols that contain errors in a code word, where 2t = n-k. t= (n-k)/2 for n-k even t = (n-k-1)/2 for n-k odd The decoder can correct any 16 symbol errors in the code word Errors in up to 16 bytes anywhere in the codeword can be automatically corrected. n k 2t data Parity 2/27/ Spring 2019 Week 6-B-62 2/27/ Spring 2019 Week 6-B-63 The Maimum codeword length For a symbol size s The maimum codeword length n is n = 2 s-1 For eample, the ma length of a code with 8-bit symbols (s=8) is 255 bytes Coding Gain Coding Gain The probability of an error remaining in the decoded data is (usually) much lower than the probability of an error if Reed-Solomon is not used Eample A digital communication system is designed to operate at a Bit Error Ratio (BER) of 10-9 No more than 1 in 109 bits are received in error This can be achieved by boosting the power of the transmitter or by adding Reed-Solomon (or another type of Forward Error Correction) Reed-Solomon allows the system to achieve this target BER with a lower transmitter output power The power "saving" given by Reed-Solomon (in decibels) is the coding gain. 2/27/ Spring 2019 Week 6-B-64 Questions? Spring 2019 Colorado State University, page 11