Towards Automatically Checking Thousands of Failures with Micro-Specifications
|
|
|
- Coleen Catherine Benson
- 5 years ago
- Views:
Transcription
1 Towards Automatically Checking Thousands of Failures with Micro-Specifications Haryadi S. Gunawi, Thanh Do, Pallavi Joshi, Joseph M. Hellerstein, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Koushik Sen University of California, Berkeley University of Wisconsin, Madison 1
2 Cloud Era Solve bigger human problems Use cluster of thousands of machines 2 2
3 Failures in The Cloud 3 3
4 Failures in The Cloud The future is a world of failures everywhere - Garth Gibson 3 3
5 Failures in The Cloud The future is a world of failures everywhere - Garth Gibson Recovery must be a first-class operation - Raghu Ramakrishnan 3 3
6 Failures in The Cloud The future is a world of failures everywhere - Garth Gibson Recovery must be a first-class operation - Raghu Ramakrishnan Reliability has to come from the software - Jeffrey Dean 3 3
7 4 4
8 5 5
9 Why Failure Recovery Hard? Testing is not advanced enough against complex failures Diverse, frequent, and multiple failures FaceBook photo loss Recovery is under specified Need to specify failure recovery behaviors Customized well-grounded protocols Example: Paxos made live An engineering perspective [PODC 07] 6 6
10 Our Solutions 7 7
11 Our Solutions FTS ( FATE ) Failure Testing Service New abstraction for failure exploration Systematically exercise 40,000 unique combinations of failures 7 7
12 Our Solutions FTS ( FATE ) Failure Testing Service New abstraction for failure exploration Systematically exercise 40,000 unique combinations of failures DTS ( DESTINI ) Declarative Testing Specification Enable concise recovery specifications We have written 74 checks (3 lines / check) 7 7
13 Our Solutions FTS ( FATE ) Failure Testing Service New abstraction for failure exploration Systematically exercise 40,000 unique combinations of failures DTS ( DESTINI ) Declarative Testing Specification Enable concise recovery specifications We have written 74 checks (3 lines / check) Note: Names have changed since the paper 7 7
14 Summary of Findings Applied FATE and DESTINI to three cloud systems: HDFS, ZooKeeper, Cassandra Found 16 new bugs Reproduced 74 bugs Problems found Inconsistency Data loss Rack awareness broken Unavailability 8 8
15 Outline Introduction FATE DESTINI Evaluation Summary 9 9
16 10 10
17 No failures 10 10
18 Alloc. Req. No failures 10 10
19 Alloc. Req. Setup Stage Data Transfer Stage No failures 10 10
20 No failures 10 10
21 4 1 No failures Setup Stage Recovery: Recreate fresh pipeline 10 10
22 4 1 No failures Setup Stage Recovery: Recreate fresh pipeline 2 Data transfer Stage Recovery: Continue on surviving nodes 10 10
23 4 1 No failures Setup Stage Recovery: Recreate fresh pipeline 2 3 Data transfer Stage Recovery: Continue on surviving nodes Bug in Data Transfer Stage Recovery 10 10
24 4 1 Failures at No failures DIFFERENT STAGES lead to DIFFERENT FAILURE BEHAVIORS Setup Stage Recovery: Recreate fresh pipeline Goal: Exercise different 2 failure recovery path 3 Data transfer Stage Recovery: Continue on surviving nodes Bug in Data Transfer Stage Recovery 10 10
25 FATE A failure injection framework target IO points Systematically exploring failure Multiple failures New abstraction of failure scenario Remember injected failures Increase failure coverage 11 11
26 FATE A failure injection framework target IO points Systematically exploring failure Multiple failures New abstraction of failure scenario Remember injected failures Increase failure coverage 11 11
27 FATE A failure injection framework target IO points Systematically exploring failure Multiple failures New abstraction of failure scenario Remember injected failures Increase failure coverage 11 11
28 FATE A failure injection framework target IO points Systematically exploring failure Multiple failures New abstraction of failure scenario Remember injected failures Increase failure coverage 11 11
29 FATE A failure injection framework target IO points Systematically exploring failure Multiple failures New abstraction of failure scenario Remember injected failures Increase failure coverage 11 11
30 FATE A failure injection framework target IO points Systematically exploring failure Multiple failures New abstraction of failure scenario Remember injected failures Increase failure coverage 11 11
31 Failure ID
32 Failure ID 2 3 Fields Values Static Func. Call OutputStream.read() Source File BlockReceiver.java Dynamic Stack Track Domain Source Node 2 specific Destination Node 3 Net. Message Data Packet Failure Type Crash After Hash
33 How Developers Build Failure ID? FATE intercepts all I/Os Use aspectj to collect information at every I/O point I/O buffers (e.g file buffer, network buffer) Target I/O (e.g. file name, IP address) Reverse engineer for domain specific information 13 13
34 Failure ID
35 Failure ID 2 3 Fields Values Static Func. Call OutputStream.read() Source File BlockReceiver.java Dynamic Stack Track Domain Source Node 2 specific Destination Node 3 Net. Message Data Packet Failure Type Crash After Hash
36 Failure ID 2 3 Fields Values Static Func. Call OutputStream.read() Source File BlockReceiver.java Dynamic Stack Track Domain Source Node 2 specific Destination Node 3 Net. Message Data Packet Failure Type Crash After Hash
37 Exploring Failure Space 14 15
38 Exploring Failure Space Exp #1: A A 14 15
39 Exploring Failure Space Exp #1: A A Exp #2: B B A 14 15
40 Exploring Failure Space Exp #1: A A Exp #2: B B A Exp #3: C B A C 14 15
41 Exploring Failure Space Exp #1: A A AB B A Exp #2: B B A Exp #3: C A B C 14 15
42 Exploring Failure Space Exp #1: A A AB B A Exp #2: B B A AC B A C Exp #3: C A B C 14 15
43 Exploring Failure Space Exp #1: A A AB B A Exp #2: B B A AC B A C Exp #3: C B A C BC B A C 14 15
44 Outline Introduction FATE DESTINI Evaluation Summary 15 16
45 DESTINI Enable concise recovery specifications Check if expected behaviors match with actual behaviors Important elements: Expectations Facts Failure Events Check Timing Interpose network and disk protocols 16 17
46 Writing specifications 17 18
47 Writing specifications Violation if expectation is different from actual facts 17 18
48 Writing specifications Violation if expectation is different from actual facts 17 18
49 Writing specifications Violation if expectation is different from actual facts violationtable():- expectationtable(), NOT-IN actualtable() 17 18
50 Writing specifications Violation if expectation is different from actual facts violationtable():- expectationtable(), NOT-IN actualtable() 17 18
51 Writing specifications Violation if expectation is different from actual facts violationtable():- expectationtable(), NOT-IN actualtable() DataLog syntax: 17 18
52 Writing specifications Violation if expectation is different from actual facts violationtable():- expectationtable(), NOT-IN actualtable() DataLog syntax: :- derivation 17 18
53 Writing specifications Violation if expectation is different from actual facts violationtable():- expectationtable(), NOT-IN actualtable() DataLog syntax: :- derivation, AND 17 18
54 Correct recovery Incorrect Recovery 18 19
55 Correct recovery Incorrect Recovery incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N); 18 19
56 Correct recovery Incorrect Recovery Expected Nodes (Block, Node) B Node 1 B Node 2 incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N); 18 19
57 Correct recovery Incorrect Recovery Expected Nodes (Block, Node) B Node 1 B Node 2 actualnodes(block, Node) B Node 1 B Node 2 incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N); 18 19
58 Correct recovery Incorrect Recovery IncorrectNodes (Block, Node) Expected Nodes (Block, Node) B Node 1 B Node 2 actualnodes(block, Node) B Node 1 B Node 2 incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N); 18 19
59 Correct recovery Incorrect recovery incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N); 19 20
60 Correct recovery Incorrect recovery Expected Nodes (Block, Node) B Node 1 B Node 2 incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N); 19 20
61 Correct recovery Incorrect recovery Expected Nodes (Block, Node) B Node 1 actualnodes(block, Node) B Node 1 B Node 2 incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N); 19 20
62 Correct recovery Incorrect recovery IncorrectNodes (Block, Node) B Node 2 Expected Nodes (Block, Node) B Node 1 B Node 2 actualnodes(block, Node) B Node 1 incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N); 19 20
63 Correct recovery Incorrect recovery IncorrectNodes (Block, Node) B Node 2 Expected Nodes (Block, Node) B Node 1 B Node 2 actualnodes(block, Node) B Node 1 incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N); 19 20
64 Correct recovery Incorrect recovery IncorrectNodes (Block, Node) B Node 2 Expected Nodes (Block, Node) B Node 1 B Node 2 actualnodes(block, Node) B Node 1 incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N); BUILD EPECTATIONS 19 20
65 Correct recovery Incorrect recovery IncorrectNodes (Block, Node) B Node 2 Expected Nodes (Block, Node) B Node 1 B Node 2 actualnodes(block, Node) B Node 1 incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N); BUILD EPECTATIONS CAPTURE FACTS 19 20
66 Building Expectations 20 21
67 Building Expectations Master Client Give me list of nodes for B [Node 1, Node 2, Node 3] 20 21
68 Building Expectations Master Client Give me list of nodes for B [Node 1, Node 2, Node 3] expectednodes(b, N) :- getblockpipe(b, N); 20 21
69 Building Expectations Master Client Give me list of nodes for B [Node 1, Node 2, Node 3] expectednodes(b, N) :- getblockpipe(b, N); Expected Nodes(Block, Node) B Node 1 B Node 2 B Node
70 Updating Expectation Expected Nodes(Block, Node) B Node 1 B Node 2 B Node
71 Updating Expectation Expected Nodes(Block, Node) B Node 1 B Node 2 B Node 3 DEL expectednodes(b, N) :- fatecrashnode(n), writestage(b, Stage), Stage = Data Transfer, expectednode(b, N) 21 22
72 Updating Expectation Expected Nodes(Block, Node) B Node 1 B Node 2 B Node 3 DEL expectednodes(b, N) :- fatecrashnode(n), writestage(b, Stage), Stage = Data Transfer, expectednode(b, N) 21 22
73 Updating Expectation Expected Nodes(Block, Node) B Node 1 B Node 2 B Node 3 DEL expectednodes(b, N) :- fatecrashnode(n), writestage(b, Stage), Stage = Data Transfer, expectednode(b, N) 21 22
74 Updating Expectation Expected Nodes(Block, Node) B Node 1 B Node 2 B Node 3 DEL expectednodes(b, N) :- fatecrashnode(n), writestage(b, Stage), Stage = Data Transfer, expectednode(b, N) Client receives all acks from setup stage writestage enter Data Transfer stage 21 22
75 Updating Expectation setupacks (B, Pos, Ack) :- cdpsetupack Expected (B, Pos, Ack); Nodes(Block, Node) goodackscnt (B, COUNT<Ack>) :- setupacks B (B, Pos, Ack), Node Ack == 1 OK ; nodescnt (B, COUNT<Node>) :- pipenodes (B,, N, ); writestage (B, Stg) :- nodescnt (NCnt), goodackscnt B (ACnt), Node NCnt 2 == Acnt, Stg := Data Transfer ; B Node 3 DEL expectednodes(b, N) :- fatecrashnode(n), writestage(b, Stage), Stage = Data Transfer, expectednode(b, N) Client receives all acks from setup stage writestage enter Data Transfer stage 21 22
76 Updating Expectation Expected Nodes(Block, Node) B Node 1 B Node 2 B Node 3 DEL expectednodes(b, N) :- fatecrashnode(n), writestage(b, Stage), Stage = Data Transfer, expectednode(b, N) Client receives all acks from setup stage writestage enter Data Transfer stage 21 22
77 Updating Expectation Expected Nodes(Block, Node) B Node 1 B Node 2 B Node 3 DEL expectednodes(b, N) :- fatecrashnode(n), writestage(b, Stage), Stage = Data Transfer, expectednode(b, N) Client receives all acks from setup stage writestage enter Data Transfer stage Precise failure events 21 22
78 Updating Expectation Expected Nodes(Block, Node) B Node 1 B Node 2 B Node 3 DEL expectednodes(b, N) :- fatecrashnode(n), writestage(b, Stage), Stage = Data Transfer, expectednode(b, N) Client receives all acks from setup stage writestage enter Data Transfer stage Precise failure events - Different stages different recovery behaviors different specifications 21 22
79 Updating Expectation Expected Nodes(Block, Node) B Node 1 B Node 2 B Node 3 DEL expectednodes(b, N) :- fatecrashnode(n), writestage(b, Stage), Stage = Data Transfer, expectednode(b, N) Client receives all acks from setup stage writestage enter Data Transfer stage Precise failure events - Different stages different recovery behaviors different specifications - FATE and DESTINI must work hand in hand 21 22
80 Capture Facts Correct recovery Incorrect recovery B_gs2 B_gs1 B_gs
81 Capture Facts Correct recovery Incorrect recovery B_gs2 B_gs1 B_gs1 actualnodes(b, N) :- blockslocation(b, N, Gs), latestgenstamp(b, Gs) 22 23
82 Capture Facts Correct recovery Incorrect recovery B_gs2 B_gs1 B_gs1 actualnodes(b, N) :- blockslocation(b, N, Gs), latestgenstamp(b, Gs) blockslocations(b, N, Gs) B Node 1 2 B Node 2 1 B Node
83 Capture Facts Correct recovery Incorrect recovery B_gs2 B_gs1 B_gs1 actualnodes(b, N) :- blockslocation(b, N, Gs), latestgenstamp(b, Gs) blockslocations(b, N, Gs) B Node 1 2 latestgenstamp(b, Gs) B 2 B Node 2 1 B Node
84 Capture Facts Correct recovery Incorrect recovery B_gs2 B_gs1 B_gs1 actualnodes(b, N) :- blockslocation(b, N, Gs), latestgenstamp(b, Gs) blockslocations(b, N, Gs) B Node 1 2 latestgenstamp(b, Gs) B 2 B Node 2 1 B Node
85 Capture Facts Correct recovery Incorrect recovery B_gs2 B_gs1 B_gs1 actualnodes(b, N) :- blockslocation(b, N, Gs), latestgenstamp(b, Gs) blockslocations(b, N, Gs) B Node 1 2 latestgenstamp(b, Gs) B 2 B Node 2 1 B Node
86 Capture Facts Correct recovery Incorrect recovery B_gs2 B_gs1 B_gs1 actualnodes(b, N) :- blockslocation(b, N, Gs), latestgenstamp(b, Gs) actualnodes(block, Node) B Node 1 blockslocations(b, N, Gs) B Node 1 2 B Node 2 1 B Node 3 1 latestgenstamp(b, Gs) B
87 Violation and Check-Timing incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N), cnpcomplete(b) ; IncorrectNodes (Block, Node) B Node 2 ExpectedNodes (Block, Node) B Node 1 B Node 2 actualnodes(block, Node) B Node
88 Violation and Check-Timing incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N), cnpcomplete(b) ; IncorrectNodes (Block, Node) B Node 2 ExpectedNodes (Block, Node) B Node 1 B Node 2 actualnodes(block, Node) B Node 1 There is a point in time where recovery is ongoing, thus specifications are violated 23 24
89 Violation and Check-Timing incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N), cnpcomplete(b) ; IncorrectNodes (Block, Node) B Node 2 ExpectedNodes (Block, Node) B Node 1 B Node 2 actualnodes(block, Node) B Node 1 There is a point in time where recovery is ongoing, thus specifications are violated Need precise events to decide when the check should be done 23 24
90 Violation and Check-Timing incorrectnodes(b, N) :- expectednodes(b, N), NOT-IN actualnodes(b, N), cnpcomplete(b) ; IncorrectNodes (Block, Node) B Node 2 ExpectedNodes (Block, Node) B Node 1 B Node 2 actualnodes(block, Node) B Node 1 There is a point in time where recovery is ongoing, thus specifications are violated Need precise events to decide when the check should be done In this example, upon block completion 23 24
91 Rules r1 incorrectnodes (B, N) :- cnpcomplete (B), expectednodes (B, N), NOT-IN actualnodes (B, N); r2 pipenodes (B, Pos, N) :- getblkpipe (UFile, B, Gs, Pos, N); r3 expectednodes (B, N) :- getblkpipe (UFile, B, Gs, Pos, N); r4 DEL expectednodes (B, N) :- fatecrashnode (N), pipestage (B, Stg), Stg == 2, expectednodes (B, N); r5 setupacks (B, Pos, Ack) :- cdpsetupack (B, Pos, Ack); r6 goodackscnt (B, CUUNT<Ack>) :- setupacks (B, Pos, Ack), Ack == OK ; r7 nodescnt (B, COUNT<Node>) :- pipenodes (B,, N, ); r8 pipestage (B, Stg) :- nodescnt (NCnt), goodackscnt (ACnt), NCnt == Acnt, Stg := 2; r9 blkgenstamp (B, Gs) :- dnpnextgenstamp (B, Gs); r10 blkgenstamp (B, Gs) :- cnpgetblkpipe (UFile, B, Gs,, ); r11 diskfiles (N, File) :- fscreate (N, File); r12 diskfiles (N, Dst) :- fsrename (N, Src, Dst), diskfiles (N, Src, Type); r13 DEL diskfiles (N, Src) :- fsrename (N, Src, Dst), diskfiles (N, Src, Type); r14 filetypes (N, File, Type) :- diskfiles(n, File), Type := Util.getType(File); r15 blkmetas (N, B, Gs) :- filetypes (N, File, Type), Type == metafile, Gs := Util.getGs(File); r16 actualnodes (B, N) :- blkmetas (N, B, Gs), blkgenstamp (B, Gs); 24 25
92 Rules r1 incorrectnodes (B, N) :- cnpcomplete (B), expectednodes (B, N), NOT-IN actualnodes (B, N); r2 pipenodes (B, Pos, N) :- getblkpipe (UFile, B, Gs, Pos, N); r3 expectednodes (B, N) :- getblkpipe (UFile, B, Gs, Pos, N); r4 DEL expectednodes (B, N) :- fatecrashnode (N), pipestage (B, Stg), Stg == 2, expectednodes (B, N); r5 setupacks (B, Pos, Ack) :- cdpsetupack (B, Pos, Ack); r6 goodackscnt (B, CUUNT<Ack>) :- setupacks (B, Pos, Ack), Ack == OK ; r7 nodescnt (B, COUNT<Node>) :- pipenodes (B,, N, ); r8 pipestage (B, Stg) :- nodescnt (NCnt), goodackscnt (ACnt), NCnt == Acnt, Stg := 2; r9 blkgenstamp (B, Gs) :- dnpnextgenstamp (B, Gs); r10 blkgenstamp (B, Gs) :- cnpgetblkpipe (UFile, B, Gs,, ); r11 diskfiles (N, File) :- fscreate (N, File); r12 diskfiles (N, Dst) :- fsrename (N, Src, Dst), diskfiles (N, Src, Type); r13 DEL diskfiles (N, Src) :- fsrename (N, Src, Dst), diskfiles (N, Src, Type); r14 filetypes (N, File, Type) :- diskfiles(n, File), Type := Util.getType(File); r15 blkmetas (N, B, Gs) :- filetypes (N, File, Type), Type == metafile, Gs := Util.getGs(File); r16 actualnodes (B, N) :- blkmetas (N, B, Gs), blkgenstamp (B, Gs); 24 25
93 Rules r1 incorrectnodes (B, N) :- cnpcomplete (B), expectednodes (B, N), NOT-IN actualnodes (B, N); r2 pipenodes (B, Pos, N) :- getblkpipe (UFile, B, Gs, Pos, N); r3 expectednodes (B, N) :- getblkpipe (UFile, B, Gs, Pos, N); r4 DEL expectednodes (B, N) :- fatecrashnode (N), pipestage (B, Stg), Stg == 2, expectednodes (B, N); r5 setupacks (B, Pos, Ack) :- cdpsetupack (B, Pos, Ack); r6 goodackscnt (B, CUUNT<Ack>) :- setupacks (B, Pos, Ack), Ack == OK ; Capture Facts, Build Expectation from IO events - No need to interpose internal functions Specification Reuse - For the first check, # rules : #check is 16:1 - Overall, #rules: # check ratio is 3:1 r7 nodescnt (B, COUNT<Node>) :- pipenodes (B,, N, ); r8 pipestage (B, Stg) :- nodescnt (NCnt), goodackscnt (ACnt), NCnt == Acnt, Stg := 2; r9 blkgenstamp (B, Gs) :- dnpnextgenstamp (B, Gs); r10 blkgenstamp (B, Gs) :- cnpgetblkpipe (UFile, B, Gs,, ); r11 diskfiles (N, File) :- fscreate (N, File); r12 diskfiles (N, Dst) :- fsrename (N, Src, Dst), diskfiles (N, Src, Type); r13 DEL diskfiles (N, Src) :- fsrename (N, Src, Dst), diskfiles (N, Src, Type); r14 filetypes (N, File, Type) :- diskfiles(n, File), Type := Util.getType(File); r15 blkmetas (N, B, Gs) :- filetypes (N, File, Type), Type == metafile, Gs := Util.getGs(File); r16 actualnodes (B, N) :- blkmetas (N, B, Gs), blkgenstamp (B, Gs); 24 25
94 Outline Introduction FATE DESTINI Evaluation Summary 25 26
95 Evaluation FATE: 3900 lines, DESTINI: 1200 lines Applied FATE and DESTINI to three cloud systems HDFS, ZooKeeper, Cassandra 40,000 unique combination of failures Found 16 new bugs, reproduced 74 bugs 74 recovery specifications 3 lines / check 26 27
96 Bugs found Reduced availability and performance Data loss due to multiple failures Data loss in log recovery protocol Data loss in append protocol Rack awareness property is broken 27 28
97 Conclusion FATE explores multiple failure systematically DESTINI enables concise recovery specifications FATE and DESTINI: a unified framework Testing recovery specifications requires a failure service Failure service needs recovery specifications to catch recovery bugs 28 29
98 Thank you! QUESTIONS? Berkeley Orders of Magnitude The Advanced Systems Laboratory Downloads our full TR paper from these websites 29 30
FATE and DESTINI: A Framework for Cloud Recovery Testing
FATE and DESTINI: A Framework for Cloud Recovery Testing Haryadi S. Gunawi, Thanh Do, Pallavi Joshi, Peter Alvaro, Joseph M. Hellerstein, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Koushik Sen,
HARDFS: Hardening HDFS with Selective and Lightweight Versioning
HARDFS: Hardening HDFS with Selective and Lightweight Versioning Thanh Do, Tyler Harter, Yingchao Liu, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau Haryadi S. Gunawi 1 Cloud Reliability q Cloud
PreFail: A Programmable Failure-Injection Framework
PreFail: A Programmable Failure-Injection Framework Pallavi Joshi Haryadi S. Gunawi Koushik Sen Electrical Engineering and Computer Sciences University of California at Berkeley Technical Report No. UCB/EECS-2011-30
Towards Automatically Checking Thousands of Failures with Micro-specifications
Towards Automatically Checking Thousands of Failures with Micro-specifications Haryadi S. Gunawi, Thanh Do, Pallavi Joshi, Joseph M. Hellerstein, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and
PreFail: Programmable and Efficient Failure Testing Framework
PreFail: Programmable and Efficient Failure Testing Framework Pallavi Joshi Haryadi S. Gunawi Koushik Sen Electrical Engineering and Computer Sciences University of California at Berkeley Technical Report
Data Sharing Made Easier through Programmable Metadata. University of Wisconsin-Madison
Data Sharing Made Easier through Programmable Metadata Zhe Zhang IBM Research! Remzi Arpaci-Dusseau University of Wisconsin-Madison How do applications share data today? Syncing data between storage systems:
Protocol-Aware Recovery for Consensus-based Storage Ramnatthan Alagappan University of Wisconsin Madison
Protocol-Aware Recovery for Consensus-based Storage Ramnatthan Alagappan University of Wisconsin Madison 2018 Storage Developer Conference. University of Wisconsin - Madison. All Rights Reserved. 1 Failures
Towards Automatically Checking Thousands of Failures with Micro-specifications
Towards Automatically Checking Thousands of Failures with Micro-specifications Haryadi Gunawi Thanh Do Pallavi Joshi Joseph M. Hellerstein Andrea Arpaci-Dusseau Remzi Arpaci-Dusseau Koushik Sen Electrical
SAMC: Sema+c- Aware Model Checking for Fast Discovery of Deep Bugs in Cloud Systems
SAMC: Sema+c- Aware Model Checking for Fast Discovery of Deep Bugs in Cloud Systems Tanakorn Leesatapornwongsa, Mingzhe Hao, Pallavi Joshi *, Jeffrey F. Lukman, and Haryadi S. Gunawi * 1 Internet Services
Tolera'ng File System Mistakes with EnvyFS
Tolera'ng File System Mistakes with EnvyFS Lakshmi N. Bairavasundaram NetApp, Inc. Swaminathan Sundararaman Andrea C. Arpaci Dusseau Remzi H. Arpaci Dusseau University of Wisconsin Madison File Systems
Physical Disentanglement in a Container-Based File System
Physical Disentanglement in a Container-Based File System Lanyue Lu, Yupu Zhang, Thanh Do, Samer AI-Kiswany Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau University of Wisconsin Madison Motivation
*-Box (star-box) Towards Reliability and Consistency in Dropbox-like File Synchronization Services
*-Box (star-box) Towards Reliability and Consistency in -like File Synchronization Services Yupu Zhang, Chris Dragga, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University of Wisconsin - Madison 6/27/2013
Zettabyte Reliability with Flexible End-to-end Data Integrity
Zettabyte Reliability with Flexible End-to-end Data Integrity Yupu Zhang, Daniel Myers, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University of Wisconsin - Madison 5/9/2013 1 Data Corruption Imperfect
EIO: Error-handling is Occasionally Correct
EIO: Error-handling is Occasionally Correct Haryadi S. Gunawi, Cindy Rubio-González, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Ben Liblit University of Wisconsin Madison FAST 08 February 28, 2008
Optimistic Crash Consistency. Vijay Chidambaram Thanumalayan Sankaranarayana Pillai Andrea Arpaci-Dusseau Remzi Arpaci-Dusseau
Optimistic Crash Consistency Vijay Chidambaram Thanumalayan Sankaranarayana Pillai Andrea Arpaci-Dusseau Remzi Arpaci-Dusseau Crash Consistency Problem Single file-system operation updates multiple on-disk
All File Systems Are Not Created Equal: On the Complexity of Crafting Crash-Consistent Applications
All File Systems Are Not Created Equal: On the Compleity of Crafting Crash-Consistent Applications Thanumalayan Sankaranarayana Pillai Vijay Chidambaram Ramnatthan Alagappan, Samer Al-Kiswany Andrea Arpaci-Dusseau,
Principled Schedulability Analysis for Distributed Storage Systems Using Thread Architecture Models
Principled Schedulability Analysis for Distributed Storage Systems Using Thread Architecture Models Suli Yang*, Jing Liu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau * work done while at UW-Madison
Haryadi Gunawi and Andrew Chien
Haryadi Gunawi and Andrew Chien in collaboration with Gokul Soundararajan and Deepak Kenchammana (NetApp) Rob Ross and Dries Kimpe (Argonne National Labs) 2 q Complete fail-stop q Fail partial Rich literature
PERSISTENCE: FSCK, JOURNALING. Shivaram Venkataraman CS 537, Spring 2019
PERSISTENCE: FSCK, JOURNALING Shivaram Venkataraman CS 537, Spring 2019 ADMINISTRIVIA Project 4b: Due today! Project 5: Out by tomorrow Discussion this week: Project 5 AGENDA / LEARNING OUTCOMES How does
The objective. Atomic Commit. The setup. Model. Preserve data consistency for distributed transactions in the presence of failures
The objective Atomic Commit Preserve data consistency for distributed transactions in the presence of failures Model The setup For each distributed transaction T: one coordinator a set of participants
CloudLab. Updated: 5/24/16
2 The Need Addressed by Clouds are changing the way we look at a lot of problems Impacts go far beyond Computer Science but there's still a lot we don't know, from perspective of Researchers (those who
ViewBox. Integrating Local File Systems with Cloud Storage Services. Yupu Zhang +, Chris Dragga + *, Andrea Arpaci-Dusseau +, Remzi Arpaci-Dusseau +
ViewBox Integrating Local File Systems with Cloud Storage Services Yupu Zhang +, Chris Dragga + *, Andrea Arpaci-Dusseau +, Remzi Arpaci-Dusseau + + University of Wisconsin Madison *NetApp, Inc. 5/16/2014
Designing a True Direct-Access File System with DevFS
Designing a True Direct-Access File System with DevFS Sudarsun Kannan, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University of Wisconsin-Madison Yuangang Wang, Jun Xu, Gopinath Palani Huawei Technologies
Coordinating distributed systems part II. Marko Vukolić Distributed Systems and Cloud Computing
Coordinating distributed systems part II Marko Vukolić Distributed Systems and Cloud Computing Last Time Coordinating distributed systems part I Zookeeper At the heart of Zookeeper is the ZAB atomic broadcast
Coerced Cache Evic-on and Discreet- Mode Journaling: Dealing with Misbehaving Disks
Coerced Cache Evic-on and Discreet- Mode Journaling: Dealing with Misbehaving Disks Abhishek Rajimwale *, Vijay Chidambaram, Deepak Ramamurthi Andrea Arpaci- Dusseau, Remzi Arpaci- Dusseau * Data Domain
Announcements. Persistence: Crash Consistency
Announcements P4 graded: In Learn@UW by end of day P5: Available - File systems Can work on both parts with project partner Fill out form BEFORE tomorrow (WED) morning for match Watch videos; discussion
Yiying Zhang, Leo Prasath Arulraj, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau. University of Wisconsin - Madison
Yiying Zhang, Leo Prasath Arulraj, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau University of Wisconsin - Madison 1 Indirection Reference an object with a different name Flexible, simple, and
Warped Mirrors for Flash Yiying Zhang
Warped Mirrors for Flash Yiying Zhang Andrea C. Arpaci-Dusseau Remzi H. Arpaci-Dusseau 2 3 Flash-based SSDs in Storage Systems Using commercial SSDs in storage layer Good performance Easy to use Relatively
University of Wisconsin-Madison
Evolving RPC for Active Storage Muthian Sivathanu Andrea C. Arpaci-Dusseau Remzi H. Arpaci-Dusseau University of Wisconsin-Madison Architecture of the future Everything is active Cheaper, faster processing
Finding Consistency in an Inconsistent World: Towards Deep Semantic Understanding of Scale-out Distributed Databases
USENIX HotStorage2016 Finding Consistency in an Inconsistent World: Towards Deep Semantic Understanding of Scale-out Distributed Databases Neville Carvalho, Hyojun Kim, Maohua Lu, Prasenjit Sarkar, Rohit
[537] Journaling. Tyler Harter
![[537] Journaling. Tyler Harter](/thumbs/82/86808355.jpg "[537] Journaling. Tyler Harter")
[537] Journaling Tyler Harter FFS Review Problem 1 What structs must be updated in addition to the data block itself? [worksheet] Problem 1 What structs must be updated in addition to the data block itself?
Programming Distributed Systems
Annette Bieniusa Programming Distributed Systems Summer Term 2018 1/ 26 Programming Distributed Systems 09 Testing Distributed Systems Annette Bieniusa AG Softech FB Informatik TU Kaiserslautern Summer
Configuring the Oracle Network Environment. Copyright 2009, Oracle. All rights reserved.
Configuring the Oracle Network Environment Objectives After completing this lesson, you should be able to: Use Enterprise Manager to: Create additional listeners Create Oracle Net Service aliases Configure
Announcements. Persistence: Log-Structured FS (LFS)
Announcements P4 graded: In Learn@UW; email 537-help@cs if problems P5: Available - File systems Can work on both parts with project partner Watch videos; discussion section Part a : file system checker
Active Testing for Concurrent Programs
Active Testing for Concurrent Programs Pallavi Joshi, Mayur Naik, Chang-Seo Park, Koushik Sen 12/30/2008 ROPAS Seminar ParLab, UC Berkeley Intel Research Overview ParLab The Parallel Computing Laboratory
Distributed Systems Final Exam
15-440 Distributed Systems Final Exam Name: Andrew: ID December 12, 2011 Please write your name and Andrew ID above before starting this exam. This exam has 14 pages, including this title page. Please
CS 345A Data Mining. MapReduce
CS 345A Data Mining MapReduce Single-node architecture CPU Machine Learning, Statistics Memory Classical Data Mining Disk Commodity Clusters Web data sets can be very large Tens to hundreds of terabytes
Advanced Computer Networks. End Host Optimization
Oriana Riva, Department of Computer Science ETH Zürich 263 3501 00 End Host Optimization Patrick Stuedi Spring Semester 2017 1 Today End-host optimizations: NUMA-aware networking Kernel-bypass Remote Direct
Outline. Distributed File System Map-Reduce The Computational Model Map-Reduce Algorithm Evaluation Computing Joins
MapReduce 1 Outline Distributed File System Map-Reduce The Computational Model Map-Reduce Algorithm Evaluation Computing Joins 2 Outline Distributed File System Map-Reduce The Computational Model Map-Reduce
Optimistic Crash Consistency. Vijay Chidambaram Thanumalayan Sankaranarayana Pillai Andrea Arpaci-Dusseau Remzi Arpaci-Dusseau
Optimistic Crash Consistency Vijay Chidambaram Thanumalayan Sankaranarayana Pillai ndrea rpaci-dusseau Remzi rpaci-dusseau Crash Consistency Problem Single file-system operation updates multiple on-disk
Chapter 4 Network Layer: The Data Plane
Chapter 4 Network Layer: The Data Plane A note on the use of these Powerpoint slides: We re making these slides freely available to all (faculty, students, readers). They re in PowerPoint form so you see
Hadoop and HDFS Overview. Madhu Ankam
Hadoop and HDFS Overview Madhu Ankam Why Hadoop We are gathering more data than ever Examples of data : Server logs Web logs Financial transactions Analytics Emails and text messages Social media like
System Malfunctions. Implementing Atomicity and Durability. Failures: Crash. Failures: Abort. Log. Failures: Media
System Malfunctions Implementing Atomicity and Durability Chapter 22 Transaction processing systems have to maintain correctness in spite of malfunctions Crash Abort Media Failure 1 2 Failures: Crash Processor
PACM: A Prediction-based Auto-adaptive Compression Model for HDFS. Ruijian Wang, Chao Wang, Li Zha
PACM: A Prediction-based Auto-adaptive Compression Model for HDFS Ruijian Wang, Chao Wang, Li Zha Hadoop Distributed File System Store a variety of data http://popista.com/distributed-filesystem/distributed-file-system:/125620
Parallel DBs. April 25, 2017
Parallel DBs April 25, 2017 1 Sending Hints Rk B Si Strategy 3: Bloom Filters Node 1 Node 2 2 Sending Hints Rk B Si Strategy 3: Bloom Filters Node 1 with
Optimistic Crash Consistency. Vijay Chidambaram Thanumalayan Sankaranarayana Pillai Andrea Arpaci-Dusseau Remzi Arpaci-Dusseau
Optimistic Crash Consistency Vijay Chidambaram Thanumalayan Sankaranarayana Pillai ndrea rpaci-dusseau Remzi rpaci-dusseau Crash Consistency Problem Single file-system operation updates multiple on-disk
Cloud-Native File Systems
Cloud-Native File Systems Remzi H. Arpaci-Dusseau Andrea C. Arpaci-Dusseau University of Wisconsin-Madison Venkat Venkataramani Rockset, Inc. How And What We Build Is Always Changing Earliest days Assembly
lecture 18: network virtualization platform (NVP) 5590: software defined networking anduo wang, Temple University TTLMAN 401B, R 17:30-20:00
lecture 18: network virtualization platform (NVP) 5590: software defined networking anduo wang, Temple University TTLMAN 401B, R 17:30-20:00 Network Virtualization in multi-tenant Datacenters Teemu Koponen.,
Analyzing Flight Data
IBM Analytics Analyzing Flight Data Jeff Carlson Rich Tarro July 21, 2016 2016 IBM Corporation Agenda Spark Overview a quick review Introduction to Graph Processing and Spark GraphX GraphX Overview Demo
Overview AEG Conclusion CS 6V Automatic Exploit Generation (AEG) Matthew Stephen. Department of Computer Science University of Texas at Dallas
CS 6V81.005 Automatic Exploit Generation (AEG) Matthew Stephen Department of Computer Science University of Texas at Dallas February 20 th, 2012 Outline 1 Overview Introduction Considerations 2 AEG Challenges
Software Based Fault Injection Framework For Storage Systems Vinod Eswaraprasad Smitha Jayaram Wipro Technologies
Software Based Fault Injection Framework For Storage Systems Vinod Eswaraprasad Smitha Jayaram Wipro Technologies The agenda Reliability in Storage systems Types of errors/faults in distributed storage
EIO: ERROR CHECKING IS OCCASIONALLY CORRECT
Department of Computer Science Institute of System Architecture, Operating Systems Group EIO: ERROR CHECKING IS OCCASIONALLY CORRECT HARYADI S. GUNAWI, CINDY RUBIO-GONZÁLEZ, ANDREA C. ARPACI-DUSSEAU, REMZI
Fault-Tolerance, Fast and Slow: Exploiting Failure Asynchrony in Distributed Systems
Fault-Tolerance, Fast and Slow: Exploiting Failure Asynchrony in Distributed Systems Ramnatthan Alagappan, Aishwarya Ganesan, Jing Liu, Andrea C. Arpaci-Dusseau, and Remzi H. Arpaci-Dusseau, University
Distributed Systems 16. Distributed File Systems II
Distributed Systems 16. Distributed File Systems II Paul Krzyzanowski pxk@cs.rutgers.edu 1 Review NFS RPC-based access AFS Long-term caching CODA Read/write replication & disconnected operation DFS AFS
Tackling the Reproducibility Problem in Systems Research with Declarative Experiment Specifications
Tackling the Reproducibility Problem in Systems Research with Declarative Experiment Specifications Ivo Jimenez, Carlos Maltzahn (UCSC) Adam Moody, Kathryn Mohror (LLNL) Jay Lofstead (Sandia) Andrea Arpaci-Dusseau,
CSE 444: Database Internals. Section 9: 2-Phase Commit and Replication
CSE 444: Database Internals Section 9: 2-Phase Commit and Replication 1 Today 2-Phase Commit Replication 2 Two-Phase Commit Protocol (2PC) One coordinator and many subordinates Phase 1: Prepare Phase 2:
data corruption in the storage stack: a closer look
L a k s h m i B a i r a v a s u n d a r a m, G a r t h G o o d s o n, B i a n c a S c h r o e d e r, A n d r e a A r p a c i - D u s s e a u, a n d R e m z i A r p a c i - D u s s e a u data corruption
Fast-Response Multipath Routing Policy for High-Speed Interconnection Networks
HPI-DC 09 Fast-Response Multipath Routing Policy for High-Speed Interconnection Networks Diego Lugones, Daniel Franco, and Emilio Luque Leonardo Fialho Cluster 09 August 31 New Orleans, USA Outline Scope
MapReduce: Simplified Data Processing on Large Clusters 유연일민철기
MapReduce: Simplified Data Processing on Large Clusters 유연일민철기 Introduction MapReduce is a programming model and an associated implementation for processing and generating large data set with parallel,
Dynamo: Key-Value Cloud Storage
Dynamo: Key-Value Cloud Storage Brad Karp UCL Computer Science CS M038 / GZ06 22 nd February 2016 Context: P2P vs. Data Center (key, value) Storage Chord and DHash intended for wide-area peer-to-peer systems
A Study of Linux File System Evolution
A Study of Linux File System Evolution Lanyue Lu Andrea C. Arpaci-Dusseau Remzi H. Arpaci-Dusseau Shan Lu University of Wisconsin - Madison Local File Systems Are Important Local File Systems Are Important
Engineering Fault-Tolerant TCP/IP servers using FT-TCP. Dmitrii Zagorodnov University of California San Diego
Engineering Fault-Tolerant TCP/IP servers using FT-TCP Dmitrii Zagorodnov University of California San Diego Motivation Reliable network services are desirable but costly! Extra and/or specialized hardware
Outline. Data Definitions and Templates Syntax and Semantics Defensive Programming
Outline Data Definitions and Templates Syntax and Semantics Defensive Programming 1 Data Definitions Question 1: Are both of the following data definitions ok? ; A w-grade is either ; - num ; - posn ;
End-to-end Data Integrity for File Systems: A ZFS Case Study
End-to-end Data Integrity for File Systems: A ZFS Case Study Yupu Zhang, Abhishek Rajimwale Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau University of Wisconsin - Madison 2/26/2010 1 End-to-end Argument
Outline. Spanner Mo/va/on. Tom Anderson
Spanner Mo/va/on Tom Anderson Outline Last week: Chubby: coordina/on service BigTable: scalable storage of structured data GFS: large- scale storage for bulk data Today/Friday: Lessons from GFS/BigTable
How Facebook Got Consistency with MySQL in the Cloud Sam Dunster
How Facebook Got Consistency with MySQL in the Cloud Sam Dunster Production Engineer Consistency Replication Replication for High Availability Facebook Replicaset Region A Slave Slave Region B Region
MEMBRANE: OPERATING SYSTEM SUPPORT FOR RESTARTABLE FILE SYSTEMS
Department of Computer Science Institute of System Architecture, Operating Systems Group MEMBRANE: OPERATING SYSTEM SUPPORT FOR RESTARTABLE FILE SYSTEMS SWAMINATHAN SUNDARARAMAN, SRIRAM SUBRAMANIAN, ABHISHEK
Membrane: Operating System support for Restartable File Systems
Membrane: Operating System support for Restartable File Systems Membrane Swaminathan Sundararaman, Sriram Subramanian, Abhishek Rajimwale, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Michael M.
A Distributed System Case Study: Apache Kafka. High throughput messaging for diverse consumers
A Distributed System Case Study: Apache Kafka High throughput messaging for diverse consumers As always, this is not a tutorial Some of the concepts may no longer be part of the current system or implemented
Distributed Computation Models
Distributed Computation Models SWE 622, Spring 2017 Distributed Software Engineering Some slides ack: Jeff Dean HW4 Recap https://b.socrative.com/ Class: SWE622 2 Review Replicating state machines Case
The Unwritten Contract of Solid State Drives
The Unwritten Contract of Solid State Drives Jun He, Sudarsun Kannan, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau Department of Computer Sciences, University of Wisconsin - Madison Enterprise SSD
Is Power State Table Golden?
Is Power State Table Golden? Harsha Vardhan #1, Ankush Bagotra #2, Neha Bajaj #3 # Synopsys India Pvt. Ltd Bangalore, India 1 dhv@synopsys.com 2 ankushb@synopsys.com 3 nehab@synopsys.com Abstract: Independent
UNIT-IV HDFS. Ms. Selva Mary. G
UNIT-IV HDFS HDFS ARCHITECTURE Dataset partition across a number of separate machines Hadoop Distributed File system The Design of HDFS HDFS is a file system designed for storing very large files with
Simple testing can prevent most critical failures -- An analysis of production failures in distributed data-intensive systems
Simple testing can prevent most critical failures -- An analysis of production failures in distributed data-intensive systems Ding Yuan, Yu Luo, Xin Zhuang, Guilherme Rodrigues, Xu Zhao, Yongle Zhang,
Active Testing for Concurrent Programs
Active Testing for Concurrent Programs Pallavi Joshi Mayur Naik Chang-Seo Park Koushik Sen 1/8/2009 ParLab Retreat ParLab, UC Berkeley Intel Research Overview Checking correctness of concurrent programs
Virtual memory Paging
Virtual memory Paging M1 MOSIG Operating System Design Renaud Lachaize Acknowledgments Many ideas and slides in these lectures were inspired by or even borrowed from the work of others: Arnaud Legrand,
RECOVERY CHAPTER 21,23 (6/E) CHAPTER 17,19 (5/E)
RECOVERY CHAPTER 21,23 (6/E) CHAPTER 17,19 (5/E) 2 LECTURE OUTLINE Failures Recoverable schedules Transaction logs Recovery procedure 3 PURPOSE OF DATABASE RECOVERY To bring the database into the most
Wenchao Zhou *, Micah Sherr *, Tao Tao *, Xiaozhou Li *, Boon Thau Loo *, and Yun Mao +
Wenchao Zhou *, Micah Sherr *, Tao Tao *, Xiaozhou Li *, Boon Thau Loo *, and Yun Mao + * University of Pennsylvania + AT&T Labs Research Introduction Developing correct large-scale distributed systems
Deconstructing Commodity Storage Clusters
Deconstructing Commodity Storage Clusters Haryadi S. Gunawi, Nitin Agrawal, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and Jiri Schindler Computer Sciences Department, University of Wisconsin,
COMMENTS. AC-1: AC-1 does not require all processes to reach a decision It does not even require all correct processes to reach a decision
ATOMIC COMMIT Preserve data consistency for distributed transactions in the presence of failures Setup one coordinator a set of participants Each process has access to a Distributed Transaction Log (DT
The challenges of non-stable predicates. The challenges of non-stable predicates. The challenges of non-stable predicates
The challenges of non-stable predicates Consider a non-stable predicate Φ encoding, say, a safety property. We want to determine whether Φ holds for our program. The challenges of non-stable predicates
Georgia Institute of Technology ECE6102 4/20/2009 David Colvin, Jimmy Vuong
Georgia Institute of Technology ECE6102 4/20/2009 David Colvin, Jimmy Vuong Relatively recent; still applicable today GFS: Google s storage platform for the generation and processing of data used by services
COMP211 Chapter 4 Network Layer: The Data Plane
COMP211 Chapter 4 Network Layer: The Data Plane All material copyright 1996-2016 J.F Kurose and K.W. Ross, All Rights Reserved Computer Networking: A Top Down Approach 7 th edition Jim Kurose, Keith Ross
Midterm Exam #3 Solutions November 30, 2016 CS162 Operating Systems
University of California, Berkeley College of Engineering Computer Science Division EECS Fall 2016 Anthony D. Joseph Midterm Exam #3 Solutions November 30, 2016 CS162 Operating Systems Your Name: SID AND
Che-Wei Chang Department of Computer Science and Information Engineering, Chang Gung University
Che-Wei Chang chewei@mail.cgu.edu.tw Department of Computer Science and Information Engineering, Chang Gung University Chapter 10: File System Chapter 11: Implementing File-Systems Chapter 12: Mass-Storage
CS 318 Principles of Operating Systems
CS 318 Principles of Operating Systems Fall 2018 Lecture 16: Advanced File Systems Ryan Huang Slides adapted from Andrea Arpaci-Dusseau s lecture 11/6/18 CS 318 Lecture 16 Advanced File Systems 2 11/6/18
Dynamic Reconfiguration of Primary/Backup Clusters
Dynamic Reconfiguration of Primary/Backup Clusters (Apache ZooKeeper) Alex Shraer Yahoo! Research In collaboration with: Benjamin Reed Dahlia Malkhi Flavio Junqueira Yahoo! Research Microsoft Research
CIS 601 Graduate Seminar Presentation Introduction to MapReduce --Mechanism and Applicatoin. Presented by: Suhua Wei Yong Yu
CIS 601 Graduate Seminar Presentation Introduction to MapReduce --Mechanism and Applicatoin Presented by: Suhua Wei Yong Yu Papers: MapReduce: Simplified Data Processing on Large Clusters 1 --Jeffrey Dean
CSE Lecture 11: Map/Reduce 7 October Nate Nystrom UTA
CSE 3302 Lecture 11: Map/Reduce 7 October 2010 Nate Nystrom UTA 378,000 results in 0.17 seconds including images and video communicates with 1000s of machines web server index servers document servers
TungstenFabric (Contrail) at Scale in Workday. Mick McCarthy, Software Workday David O Brien, Software Workday
TungstenFabric (Contrail) at Scale in Workday Mick McCarthy, Software Engineer @ Workday David O Brien, Software Engineer @ Workday Agenda Introduction Contrail at Workday Scale High Availability Weekly
Virtualizing Memory: Paging. Announcements
CS 537 Introduction to Operating Systems UNIVERSITY of WISCONSIN-MADISON Computer Sciences Department Andrea C. Arpaci-Dusseau Remzi H. Arpaci-Dusseau Virtualizing Memory: Paging Questions answered in
Design and Synthesis for Test
TDTS 80 Lecture 6 Design and Synthesis for Test Zebo Peng Embedded Systems Laboratory IDA, Linköping University Testing and its Current Practice To meet user s quality requirements. Testing aims at the
18-hdfs-gfs.txt Thu Oct 27 10:05: Notes on Parallel File Systems: HDFS & GFS , Fall 2011 Carnegie Mellon University Randal E.
18-hdfs-gfs.txt Thu Oct 27 10:05:07 2011 1 Notes on Parallel File Systems: HDFS & GFS 15-440, Fall 2011 Carnegie Mellon University Randal E. Bryant References: Ghemawat, Gobioff, Leung, "The Google File
Quantifying Violations of Destination-based Forwarding on the Internet
Quantifying Violations of Destination-based Forwarding on the Internet Tobias Flach, Ethan Katz-Bassett, and Ramesh Govindan University of Southern California November 14, 2012 Destination-based Routing
CSE 544 Principles of Database Management Systems. Alvin Cheung Fall 2015 Lecture 14 Distributed Transactions
CSE 544 Principles of Database Management Systems Alvin Cheung Fall 2015 Lecture 14 Distributed Transactions Transactions Main issues: Concurrency control Recovery from failures 2 Distributed Transactions
ANSI X (Purchase Order Acknowledgment) Outbound (from Eclipse) Version 4010
ANSI X12 855 (Purchase Order Acknowledgment) Outbound (from Eclipse) Version 4010 Eclipse 855 4010 (Customer) 1 10/11/2012 855 Purchase Order Acknowledgment Functional Group=PR Purpose: This Draft Standard
Warming up Storage-level Caches with Bonfire
Warming up Storage-level Caches with Bonfire Yiying Zhang Gokul Soundararajan Mark W. Storer Lakshmi N. Bairavasundaram Sethuraman Subbiah Andrea C. Arpaci-Dusseau Remzi H. Arpaci-Dusseau 2 Does on-demand
Secure Routing for Mobile Ad-hoc Networks
Department of Computer Science IIT Kanpur CS625: Advanced Computer Networks Outline 1 2 3 4 Outline 1 2 3 4 Need Often setting up an infrastructure is infeasible Disaster relief Community networks (OLPC)
CSE-E5430 Scalable Cloud Computing Lecture 9
CSE-E5430 Scalable Cloud Computing Lecture 9 Keijo Heljanko Department of Computer Science School of Science Aalto University keijo.heljanko@aalto.fi 15.11-2015 1/24 BigTable Described in the paper: Fay
Chapter 11: File System Implementation. Objectives
Chapter 11: File System Implementation Objectives To describe the details of implementing local file systems and directory structures To describe the implementation of remote file systems To discuss block