9/26/2017 Sangmi Lee Pallickara Week 6- A. CS535 Big Data Fall 2017 Colorado State University

Transcription

1 CS535 Big Data - Fall 2017 Week 6-A-1 CS535 BIG DATA FAQs PA1: Use only one word query Deadends {{Dead end}} Hub value will be?? PART 1. BATCH COMPUTING MODEL FOR BIG DATA ANALYTICS 4. GOOGLE FILE SYSTEM AND COLOSSUS Computer Science, Colorado State University Iterations and Threshold? You should define own threshold: This is your software design. You can still give a maimum XYZ # of iterations Running Time Output Ranked list of the pages: with title Out of memory eception (related to GC) CS535 Big Data - Fall 2017 Week 6-A-2 CS535 Big Data - Fall 2017 Week 6-A-3 Today s topics Distributed File System GFS, GFS2 (Colossus) Google File System CS535 Big Data - Fall 2017 Week 6-A-4 File state region after a mutation CS535 Big Data - Fall 2017 Week 6-A-5 GFS guarantees the mutated file to be defined and to contain data written by the last mutation Write Record Append Applying mutations to a chunk in the same order on all its replicas Serial success Concurrent success Defined Consistent but undefined defined interspersed with inconsistent Using chunk version numbers to detect any replica that has become stale If the chunkserver was down Failure Inconsistent 1

2 CS535 Big Data - Fall 2017 Week 6-A-6 Clients cache chunk locations What if the chunk location points to a stale replica before that information is refreshed? The window is limited by: Cache entry s timeout Net open to the file It will purge all chunk information for that file from the cache CS535 Big Data - Fall 2017 Week 6-A-7 Implications for applications Rely on appends instead of overwrites Checkpoint Records should be Self-validating Self-identifying Append-only Stale replica usually returns a premature end of chunk rather than outdated data When a reader retries and contact master It will get current chuck locations CS535 Big Data - Fall 2017 Week 6-A-8 CS535 Big Data - Fall 2017 Week 6-A-9 GFS uses leases to maintain consistent mutation order across replicas Managing Mutations: Handling writes Master grants lease to one of the replicas Primary Primary picks serial-order For all mutations to the chunk Other replicas follow this order When applying mutations CS535 Big Data - Fall 2017 Week 6-A-10 Lease mechanism designed to minimize communications with the master Lease has initial timeout of 60 seconds As long as chunk is being mutated Primary can request and receive etensions Etension requests/grants piggybacked over heart-beat messages CS535 Big Data - Fall 2017 Week 6-A-11 Revocation and transfer of leases Master may revoke a lease before it epires If communications lost with primary Master can safely give lease to another replica Only After the lease period for old primary elapses 2

3 CS535 Big Data - Fall 2017 Week 6-A-12 How a write is actually performed 1. Chunkserver holding the current lease for the chunk and the location of the other replica 4. Write request MASTER Client 3*. Secondary Replica A 2. Identity of the primary and the locations of other replicas CS535 Big Data - Fall 2017 Week 6-A-13 Client pushes data to all the replicas (I) Each chunk server stores data in an LRU buffer until Data is used Aged out 7. Final Reply Primary Replica 5. Write request/ 6. Acknowledgement Secondary Replica B 3. Client pushes the data to all the replicas CS535 Big Data - Fall 2017 Week 6-A-14 Client pushes data to all the replicas (II) When chunk servers acknowledge receipt of data Client sends a write request to primary Primary assigns consecutive serial numbers to mutations Forwards to replicas CS535 Big Data - Fall 2017 Week 6-A-15 Data flow is decoupled from the control flow to utilize network efficiently Utilize each machine s network bandwidth Avoid network bottlenecks Avoid high-latency links Leverage network topology Estimate distances from IP addresses Pipeline the data transfer Once a chunkserver receives some data, it starts forwarding immediately. For transferring B bytes to R replicas Ideal elapsed time will be B/T+RL where: T is the network throughput L is latency to transfer bytes between two machines CS535 Big Data - Fall 2017 Week 6-A-16 CS535 Big Data - Fall 2017 Week 6-A-17 Append: Record sizes and fragmentation Size is restricted to ¼ the chunk size Maimum size Managing Mutations: Append Minimizes worst-case fragmentation Internal fragmentation in each chunk 3

4 CS535 Big Data - Fall 2017 Week 6-A-18 Inconsistent Regions CS535 Big Data - Fall 2017 Week 6-A-19 What if record append fails at one of the replicas Data 1 Data 1 Data 1 Data 2 Data 2 Data 2 Data 3 Data 3 User will re-try to store Data 3 Data 1 Data 1 Data 1 Data 2 Data 2 Data 2 Failed Empty Client must retry the operation Replicas of same chunk may contain Different data Duplicates of the same record In whole or in part Replicas of chunks are not bit-wise identical! In most systems, replicas are identical Data 3 Data 3 Data 3 Data 3 Data 3 Data 3 CS535 Big Data - Fall 2017 Week 6-A-20 GFS only guarantees that the data will be written at least once as an atomic unit CS535 Big Data - Fall 2017 Week 6-A-21 GFS client code implements the file system API For an operation to return success Data must be written at the same offset on all the replicas After the write, all replicas are as long as the end of the record Any future record will be assigned a higher offset or a different chunk Communications with master and chunk servers done transparently On behalf of apps that read or write data Interact with master for metadata Data-bearing communications directly to chunk servers CS535 Big Data - Fall 2017 Week 6-A-22 CS535 Big Data - Fall 2017 Week 6-A-23 Snapshots Copying file or directory tree almost instantaneously Creating Snapshots Minimizing any interruptions of ongoing mutations Providing checkpoint Users can commit later Rollback 4

5 CS535 Big Data - Fall 2017 Week 6-A-24 Snapshots allow you to make a copy of a file very fast 1. Master revokes outstanding leases for any chunks of the file (source) to be snapshot 2. Log the operation to disk 3. Update in-memory state Duplicate metadata of the source file 4. Newly created file points to the same chunks as the source CS535 Big Data - Fall 2017 Week 6-A-25 When a client wants to write to a chunk C after the snapshot operation Master sees the reference count to C > 1 Pick new chunk-handle C Ask chunk-server with current replica of C Create new chunk C Data is copied locally, not over the network From this point, chunk handling of C is no different CS535 Big Data - Fall 2017 Week 6-A-26 GFS does not have a per-directory structure that lists files in the directory Name spaces represented as a lookup table Maps full pathnames to metadata Each node has an associated read/write lock File creation does not require a lock on the directory structure No inode needs to be protected from modification CS535 Big Data - Fall 2017 Week 6-A-27 Each master operation acquires a set of locks before it runs Read lock prevents a directory from being deleted, renamed, or snapshotted Write lock on file names serialize attempts to create a file with the same twice If operation involves /d1/d2/ /dn/leaf Acquire read locks on all of the directory names /d1, /d1/d2,, /d1/d2/ /dn Read or write lock on full pathname /d1/d2/ /dn/leaf CS535 Big Data - Fall 2017 Week 6-A-28 Locks are used to prevent operations during snapshots How do we present creating /home/user/foo While /home/user is being snapshotted to /save/user? /home/user is being snapshotted to /save/user Read locks on /home and /save Write lock on /home/user and /save/user To create file Read lock on /home and /home/user Write lock on /home/user/foo The two operations will be serialized because they try to obtain /home/user File creation does not require write lock on parent directory there is no directory Read locks on /home and /home/user Write lock on /home/user/foo CS535 Big Data - Fall 2017 Week 6-A-29 Deletion of Files and Garbage Collection 5

6 CS535 Big Data - Fall 2017 Week 6-A-30 Garbage collection in GFS After a file is deleted, GFS does not reclaim space immediately Done lazily during garbage collection at File and chunk levels CS535 Big Data - Fall 2017 Week 6-A-31 Master logs a file s deletion immediately File is renamed to a hidden name Includes deletion timestamp Master scans the file system namespace Delete if hidden file eisted for more than 3 days When file is removed from namespace In memory metadata is also removed Severs links to all its chunks! CS535 Big Data - Fall 2017 Week 6-A-32 Garbage collection: When Master scans its chunk namespace CS535 Big Data - Fall 2017 Week 6-A-33 The role of heart-beats in garbage collection Identifies orphaned chunks Not reachable from any file Erase metadata for these chunks Chunk server reports subset of chunks it currently has Master replies with identity of chunks no longer present Chunk server is now free to delete its replica of such chunks CS535 Big Data - Fall 2017 Week 6-A-34 Stale chunks and issues If a chunk server fails AND misses mutations to the chunk The chunk replica becomes stale Working with a stale replica causes problems with: Correctness Consistency CS535 Big Data - Fall 2017 Week 6-A-35 Aiding the detection of stale chunks Master maintains a chunk version number for each chunk Distinguish between stale and up-to-date chunks When master grants a new lease on chunk Increase version number Inform replicas Record new version persistently Occurs BEFORE any client mutate chunk 6

7 CS535 Big Data - Fall 2017 Week 6-A-36 If a replica is unavailable its version number will not be advanced CS535 Big Data - Fall 2017 Week 6-A-37 Additional safeguards against stale replicas When a chunk server restarts, it reports to the Master with the following: Set of Chunks Corresponding version numbers Used to detect stale replicas Remove stale replicas in regular garbage collection Include chunk version number When client requests chunk information Client/Chunk server verify version to make sure things are up-to-date During cloning operations Clone the most up-to-date chunk Clients and chunk servers epected to verify versioning information CS535 Big Data - Fall 2017 Week 6-A-38 CS535 Big Data - Fall 2017 Week 6-A-39 Data Integrity Impractical to detect chunk corruptions across replicas Not bitwise identical in any case! Data Integrity Detection of corruption should be self-contained CS535 Big Data - Fall 2017 Week 6-A-40 CS535 Big Data - Fall 2017 Week 6-A-41 Data Integrity Break chunks into 64 KB data blocks Compute 32-bit checksum for block Keep in chunk server memory Store persistently, separate from the data Verify checksums of data blocks that overlap read range Inefficiencies 7

8 CS535 Big Data - Fall 2017 Week 6-A-42 The master server is a single point of failure Master server restart takes several seconds Complete recovery takes several minutes Shadow servers eist Can handle reads of files In place of the master Requires a massive main memory CS535 Big Data - Fall 2017 Week 6-A-43 The system is optimized for large files But not for a very large number of very small files Primary operation on files Long, sequential reads/writes Large number of random overwrites will clog things up quite a bit CS535 Big Data - Fall 2017 Week 6-A-44 Consistency Issues: GFS epects clients to resolve inconsistencies File chunks may have gaps or duplicates of some records The client has to be able to deal with this CS535 Big Data - Fall 2017 Week 6-A-45 Security model Originally None Operation is epected to be in a trusted environment Imagine doing this for a scientific application Portions of a massive array are corrupted Clients would have to detect this Detection is possible of course, but onerous! NOTE: HDFS handles gaps and duplicates CS535 Big Data - Fall 2017 Week 6-A-46 CS535 Big Data - Fall 2017 Week 6-A-47 Storage Software: Colossus (GFS2) Net-generation cluster-level file system Colossus: Google File System II Automatically sharded metadata layer Distributed Masters (64MB block size à 1MB) Data typically written using Reed-Solomon (1.5) Client-driven replication, encoding and replication Metadata space has enabled availability Why Reed-Solomon? Cost Especially with cross cluster replication More fleible cost vs. availability choices Google File System II: Dawn of the Multiplying Master Nodes, 8

9 CS535 Big Data - Fall 2017 Week 6-A-48 CS535 Big Data - Fall 2017 Week 6-A-49 Reed-Solomon Codes Block-based error correcting codes Digital communication and storage Colossus: Google File System II Reed-Solomon Codes Storage devices (including tape, CD, DVD, barcodes, etc) Wireless or mobile communications Satellite communications Digital TV High-speed modems SOURCE: Solomon_codes_for_coders CS535 Big Data - Fall 2017 Week 6-A-50 CS535 Big Data - Fall 2017 Week 6-A-51 What does the R-S code do? Takes a block of digital data Adds etra redundant bits If an error happens, the R-S decoder processes each block and recovers original data Colossus: Google File System II: Reed-Solomon code Quick overview with an eample Noise, Errors Data source Reed-Solomon Encoder Communication channel or storage devices Reed-Solomon Decoder Data Sink CS535 Big Data - Fall 2017 Week 6-A-52 A Quick Eample of the R-S encoding 4+2 coding Original files are broken into 4 pieces 2 parity pieces are added CS535 Big Data - Fall 2017 Week 6-A-53 A Quick Eample of the R-S encoding Applying coding matri First piece of data ABCD, second piece of data EFGH Original Data b 1c c 1b = 9

10 CS535 Big Data - Fall 2017 Week 6-A-54 A Quick Eample of the R-S encoding Data loss 2 of 6 rows are lost CS535 Big Data - Fall 2017 Week 6-A-55 A Quick Eample of the R-S encoding Without 2 rows b 1c c 1b = 1b 1c c 1b = CS535 Big Data - Fall 2017 Week 6-A-56 CS535 Big Data - Fall 2017 Week 6-A-57 A Quick Eample of the R-S encoding Multiplying each side with the inverted matri A Quick Eample of the R-S encoding The Inverse Matri and the Coding Matri Cancel Out 8d f6 7b 01 f6 8d 01 7b 1b 1c c 1b d f6 7b 01 f6 8d 01 7b 1b 1c c 1b = 8d f6 7b 01 f6 8d 01 7b = 8d f6 7b 01 f6 8d 01 7b CS535 Big Data - Fall 2017 Week 6-A-58 A Quick Eample of the R-S encoding Reconstructing the Original Data = 8d f6 7b 01 f6 8d 01 7b 10