Building Scalable Stateful Services. Craft Conf 2016
|
|
|
- Reginald Bishop
- 6 years ago
- Views:
Transcription
1 Building Scalable Stateful s Craft Conf 2016
2 Caitie McCaffrey Distributed Systems caitiem.com
3 Stateless s
4 Stateless s
5 Stateless s
6 Stateless s
7 Stateless s
8 Stateless s
9 Stateless s
10 Stateless s
11 Stateless s
12 Stateless s
13 Stateless s
14 Stateless s
15 Stateless s
16 Data Shipping Paradigm
17 Data Shipping Paradigm
18 Data Shipping Paradigm
19 Data Shipping Paradigm
20 Data Shipping Paradigm
21 Data Shipping Paradigm
22 Data Shipping Paradigm
23 Data Shipping Paradigm
24 Data Shipping Paradigm
25 Data Shipping Paradigm
26 Data Shipping Paradigm
27 Data Shipping Paradigm
28 Data Shipping Paradigm
29 Data Shipping Paradigm
30 Data Shipping Paradigm
31 Benefits Overview Building Real World Caution
32 Data Locality For Low Latency & Data Intensive s
33 Function Shipping Paradigm
34 Function Shipping Paradigm
35 Function Shipping Paradigm
36 Function Shipping Paradigm
37 Function Shipping Paradigm
38 Function Shipping Paradigm
39 Function Shipping Paradigm
40 Function Shipping Paradigm
41 Function Shipping Paradigm
42 Function Shipping Paradigm
43 Function Shipping Paradigm
44 Sticky Connections & Consistency Additional Available Consistency Models
45 Sticky Connections
46 Sticky Connections
47 Consistency Models Linearizable Sequential CP Consistency AP Consistency Pipelined Random Access Memory Causal Write From Read Read Your Write Monotonic Read Monotonic Write
48 Consistency Models Linearizable Sequential CP Consistency AP Consistency AP Consistency w/ Sticky Connections Pipelined Random Access Memory Causal Write From Read Read Your Write Monotonic Read Monotonic Write
49 Whether or not read-your-write, session and monotonic consistency can be achieved depends in general on the "stickiness" of clients to the server that executes the distributed protocol for them Using sessions, which are sticky, makes this explicit and provides an exposure level that clients can reason about. - Werner Vogel 2007
50 Building Sticky Connections For Low Latency & Data Intensive s
51 Building Sticky Connections
52 Building Sticky Connections
53 Persistent Connections Problems Load Balancing Problems No Stickiness Once Connection Breaks
54 Persistent Connections Problems Load Balancing Problems No Stickiness Once Connection Breaks
55 Routing Logic Problems to Solve Cluster Membership Work Distribution
56 Routing Logic Problems to Solve Cluster Membership Work Distribution
57 Routing Logic Problems to Solve Cluster Membership Work Distribution
58 Cluster Membership Static Dynamic
59 Static Cluster Membership Machine Outage means Partial Outage Downtime to Add Capacity
60 Static Cluster Membership Machine Outage means Partial Outage Downtime to Add Capacity
61 Static Cluster Membership Machine Outage means Partial Outage Downtime to Add Capacity
62 Dynamic Cluster Membership Fault Tolerance
63 Dynamic Cluster Membership Fault Tolerance
64 Dynamic Cluster Membership Fault Tolerance
65 Dynamic Cluster Membership Fault Tolerance
66 Dynamic Cluster Membership Fault Tolerance
67 Dynamic Cluster Membership Gossip Protocols Consensus Systems Availability vs Consistency
68 Consensus Systems for Consistency Zookeeper Paxos ETCD Raft
69 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
70 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
71 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
72 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
73 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
74 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
75 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
76 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
77 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
78 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
79 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
80 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
81 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
82 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
83 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
84 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
85 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
86 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
87 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
88 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
89 Gossip Protocols for Availability Assumes Non-Reliable Networks Information Dissemination Pairwise Communication
90 Work Distribution Random Placement Consistent Hashing Distributed Hash Tables
91 Random Placement Write Anywhere Read from Everywhere
92 Random Placement Write Anywhere Read from Everywhere
93 Random Placement Write Anywhere Read from Everywhere
94 Random Placement Write Anywhere Read from Everywhere
95 Random Placement Write Anywhere Read from Everywhere
96 Random Placement Write Anywhere Read from Everywhere
97 Random Placement Write Anywhere Read from Everywhere
98 Consistent Hashing Node B Deterministic Placement Node A Consistent Hashing & Random Trees: Node C Distributed caching protocols for relieving hot spots on the World Wide Web NodeD
99 Distributed Hash Table Non- Deterministic Node A Placement Node B Node C
100 Distributed Hash Table Non- Deterministic Node A Placement Node B Node C
101 Distributed Hash Table Non- Deterministic Node A Placement Node B Node C
102 Distributed Hash Table Non- Deterministic Node A Placement Node B Node C
103 Distributed Hash Table Non- Deterministic Node A Placement Node B Node C
104 Stateful s In the Real World
105 Scuba is a fast, scalable, distributed, in-memory database built at Facebook. It is the workhorse Fan-out request to all machines in the cluster behind code regression analysis & bug report, revenue, and performance debugging Compose Results Return Results and Completeness
106 Scuba is a fast, scalable, distributed, in-memory database built at Facebook. It is the workhorse Fan-out request to all machines in the cluster behind code regression analysis & bug report, revenue, and performance debugging Compose Results Return Results and Completeness
107 Scuba is a fast, scalable, distributed, in-memory database built at Facebook. It is the workhorse Fan-out request to all machines in the cluster behind code regression analysis & bug report, revenue, and performance debugging Compose Results Return Results and Completeness
108 Twitter Nuthatch is a temporal set indexing service. Helps resolve glob queries in metric names. Static Cluster Membership + Hashing
109 Twitter Nuthatch is a temporal set indexing service. Helps resolve glob queries in metric names. Static Cluster Membership + Hashing
110 Twitter Nuthatch is a temporal set indexing service. Helps resolve glob queries in metric names. Static Cluster Membership + Hashing
111 Uber Ringpop is an opensource Node.js library that brings application-layer sharding to many of their dispatching platform services. Swim Gossip Protocol + Consistent Hashing
112 Uber Ringpop is an opensource Node.js library that brings application-layer sharding to many of their dispatching platform services. Swim Gossip Protocol + Consistent Hashing
113 Uber Ringpop is an opensource Node.js library that brings application-layer sharding to many of their dispatching platform services. Swim Gossip Protocol + Consistent Hashing
114 Orleans is a runtime and Programming model for building distributed systems based on the Actor Model from the extreme Computing Group at MSR Orleans Cluster Actor Gossip Protocol + Consistent Hashing + Distributed Hash Table Actor Actor Actor Actor
115 Orleans is a runtime and Programming model for building distributed systems based on the Actor Model from the extreme Computing Group at MSR Orleans Cluster Actor Gossip Protocol + Consistent Hashing + Distributed Hash Table Actor Actor Actor Actor
116 Orleans is a runtime and Programming model for building distributed systems based on the Actor Model from the extreme Computing Group at MSR Orleans Cluster Actor Gossip Protocol + Consistent Hashing + Distributed Hash Table Actor Actor Actor Actor
117 Orleans Distributed Hash Table Orleans Cluster
118 Orleans Distributed Hash Table Orleans Cluster
119 Orleans Distributed Hash Table Orleans Cluster
120 Orleans Distributed Hash Table Orleans Cluster
121 Orleans Distributed Hash Table Orleans Cluster
122 Orleans Distributed Hash Table Orleans Cluster
123 Orleans Distributed Hash Table Orleans Cluster
124 Orleans Distributed Hash Table Orleans Cluster
125 Caution Stateful s Ahead
126 Unbounded Data Structures Implicit Assumptions are the Killer of Distributed Systems
127 Memory Management Get Ready to Make Friends with the Garbage Collector Profiler
128 Reloading State First Connection Recovering From Crashes Deploying New Code
129 Fast Restarts at Facebook Our Key Observation is that we can decouple the memory lifetime from the process lifetime. When we shutdown a server for a planned upgrade.
130 Conclusion Data Locality & Available Consistency Cluster Membership & Work Distribution Successful Stateful Real World Systems Caution: Some New Challenges
131 Thank You Kyle Kingsbury Chris Meiklejohn Tom Santero Ines
132 Questions
133
Distributed Systems. Day 9: Replication [Part 1]
![Distributed Systems. Day 9: Replication [Part 1]](/thumbs/94/121965696.jpg "Distributed Systems. Day 9: Replication [Part 1]")
Distributed Systems Day 9: Replication [Part 1] Hash table k 0 v 0 k 1 v 1 k 2 v 2 k 3 v 3 ll Facebook Data Does your client know about all of F s servers? Security issues? Performance issues? How do clients
Orleans. Actors for High-Scale Services. Sergey Bykov extreme Computing Group, Microsoft Research
Orleans Actors for High-Scale Services Sergey Bykov extreme Computing Group, Microsoft Research 3-Tier Architecture Frontends Middle Tier Storage Stateless frontends Stateless middle tier Storage is the
More on Testing and Large Scale Web Apps
More on Testing and Large Scale Web Apps Testing Functionality Tests - Unit tests: E.g. Mocha - Integration tests - End-to-end - E.g. Selenium - HTML CSS validation - forms and form validation - cookies
羅仲成 ROY LOU 17MEDIA 分散式緩存服務實踐 DISTRIBUTED CACHING SERVICE
羅仲成 ROY LOU 17MEDIA 分散式緩存服務實踐 DISTRIBUTED CACHING SERVICE ABOUT ME 17media architect Past: HTC, Google, NVIDIA 2-year-old monster s dad Jogging, basketball, snowboarding There are only two hard things
10. Replication. Motivation
10. Replication Page 1 10. Replication Motivation Reliable and high-performance computation on a single instance of a data object is prone to failure. Replicate data to overcome single points of failure
Ovid A Software-Defined Distributed Systems Framework. Deniz Altinbuken, Robbert van Renesse Cornell University
Ovid A Software-Defined Distributed Systems Framework Deniz Altinbuken, Robbert van Renesse Cornell University Ovid Build distributed systems that are easy to evolve easy to reason about easy to compose
Goals. Facebook s Scaling Problem. Scaling Strategy. Facebook Three Layer Architecture. Workload. Memcache as a Service.
Goals Memcache as a Service Tom Anderson Rapid application development - Speed of adding new features is paramount Scale Billions of users Every user on FB all the time Performance Low latency for every
Distributed Systems. replication Johan Montelius ID2201. Distributed Systems ID2201
Distributed Systems ID2201 replication Johan Montelius 1 The problem The problem we have: servers might be unavailable The solution: keep duplicates at different servers 2 Building a fault-tolerant service
A Brief History of Distributed Programming: RPC. YOW Brisbane 2016
A Brief History of Distributed Programming: RPC YOW Brisbane 2016 Christopher Meiklejohn Université catholique de Louvain @cmeik christophermeiklejohn.com Caitie McCaffrey Distributed Systems Engineer
Distributed systems. Lecture 6: distributed transactions, elections, consensus and replication. Malte Schwarzkopf
Distributed systems Lecture 6: distributed transactions, elections, consensus and replication Malte Schwarzkopf Last time Saw how we can build ordered multicast Messages between processes in a group Need
Distributed Systems (5DV147)
Distributed Systems (5DV147) Replication and consistency Fall 2013 1 Replication 2 What is replication? Introduction Make different copies of data ensuring that all copies are identical Immutable data
Consistency in Distributed Storage Systems. Mihir Nanavati March 4 th, 2016
Consistency in Distributed Storage Systems Mihir Nanavati March 4 th, 2016 Today Overview of distributed storage systems CAP Theorem About Me Virtualization/Containers, CPU microarchitectures/caches, Network
DEMYSTIFYING BIG DATA WITH RIAK USE CASES. Martin Schneider Basho Technologies!
DEMYSTIFYING BIG DATA WITH RIAK USE CASES Martin Schneider Basho Technologies! Agenda Defining Big Data in Regards to Riak A Series of Trade-Offs Use Cases Q & A About Basho & Riak Basho Technologies is
Distributed Systems. Characteristics of Distributed Systems. Lecture Notes 1 Basic Concepts. Operating Systems. Anand Tripathi
1 Lecture Notes 1 Basic Concepts Anand Tripathi CSci 8980 Operating Systems Anand Tripathi CSci 8980 1 Distributed Systems A set of computers (hosts or nodes) connected through a communication network.
Distributed Systems. Characteristics of Distributed Systems. Characteristics of Distributed Systems. Goals in Distributed System Designs
1 Anand Tripathi CSci 8980 Operating Systems Lecture Notes 1 Basic Concepts Distributed Systems A set of computers (hosts or nodes) connected through a communication network. Nodes may have different speeds
Intuitive distributed algorithms. with F#
Intuitive distributed algorithms with F# Natallia Dzenisenka Alena Hall @nata_dzen @lenadroid A tour of a variety of intuitivedistributed algorithms used in practical distributed systems. and how to prototype
Tools for Social Networking Infrastructures
Tools for Social Networking Infrastructures 1 Cassandra - a decentralised structured storage system Problem : Facebook Inbox Search hundreds of millions of users distributed infrastructure inbox changes
How we build TiDB. Max Liu PingCAP Amsterdam, Netherlands October 5, 2016
How we build TiDB Max Liu PingCAP Amsterdam, Netherlands October 5, 2016 About me Infrastructure engineer / CEO of PingCAP Working on open source projects: TiDB: https://github.com/pingcap/tidb TiKV: https://github.com/pingcap/tikv
Replication in Distributed Systems
Replication in Distributed Systems Replication Basics Multiple copies of data kept in different nodes A set of replicas holding copies of a data Nodes can be physically very close or distributed all over
Scaling Up from 1000 to 10 Nodes
Scaling Up from 1000 to 10 Nodes Anton Lavrik Alert Logic, Inc. 1 Alert Logic vs Twitter: daily numbers (2012) Alert Logic Twitter Events received 11.5B 0.5B Volume received 5TB 70GB Logs produced? 100TB
XTP, Scalability and Data Grids An Introduction to Coherence
XTP, Scalability and Data Grids An Introduction to Coherence Tom Stenström Principal Sales Consultant Oracle Presentation Overview The challenge of scalability The Data Grid What
Configuring CWMP Service
CHAPTER 12 This chapter describes how to configure the CWMP service in Cisco Broadband Access Center (BAC). Topics covered are:, page 12-1 Configuring Service Ports on the DPE, page 12-2 Disabling Connection
Above the Clouds: Introducing Akka. Jonas Bonér Scalable Solutions
Above the Clouds: Introducing Akka Jonas Bonér CEO @ Scalable Solutions Twitter: @jboner The problem It is way too hard to build: 1. correct highly concurrent systems 2. truly scalable systems 3. fault-tolerant
InfluxDB and the Raft consensus protocol. Philip O'Toole, Director of Engineering (SF) InfluxDB San Francisco Meetup, December 2015
InfluxDB and the Raft consensus protocol Philip O'Toole, Director of Engineering (SF) InfluxDB San Francisco Meetup, December 2015 Agenda What is InfluxDB and why is it distributed? What is the problem
Dynamo. Smruti R. Sarangi. Department of Computer Science Indian Institute of Technology New Delhi, India. Motivation System Architecture Evaluation
Dynamo Smruti R. Sarangi Department of Computer Science Indian Institute of Technology New Delhi, India Smruti R. Sarangi Leader Election 1/20 Outline Motivation 1 Motivation 2 3 Smruti R. Sarangi Leader
Page 1. Key Value Storage"
Key Value Storage CS162 Operating Systems and Systems Programming Lecture 14 Key Value Storage Systems March 12, 2012 Anthony D. Joseph and Ion Stoica http://inst.eecs.berkeley.edu/~cs162 Handle huge volumes
Distributed Data Management Replication
Felix Naumann F-2.03/F-2.04, Campus II Hasso Plattner Institut Distributing Data Motivation Scalability (Elasticity) If data volume, processing, or access exhausts one machine, you might want to spread
CAP Theorem. March 26, Thanks to Arvind K., Dong W., and Mihir N. for slides.
C A CAP Theorem P March 26, 2018 Thanks to Arvind K., Dong W., and Mihir N. for slides. CAP Theorem It is impossible for a web service to provide these three guarantees at the same time (pick 2 of 3):
Distributed Hash Tables Chord and Dynamo
Distributed Hash Tables Chord and Dynamo (Lecture 19, cs262a) Ion Stoica, UC Berkeley October 31, 2016 Today s Papers Chord: A Scalable Peer-to-peer Lookup Service for Internet Applications, Ion Stoica,
CS 655 Advanced Topics in Distributed Systems
Presented by : Walid Budgaga CS 655 Advanced Topics in Distributed Systems Computer Science Department Colorado State University 1 Outline Problem Solution Approaches Comparison Conclusion 2 Problem 3
CS 470 Spring Fault Tolerance. Mike Lam, Professor. Content taken from the following:
CS 47 Spring 27 Mike Lam, Professor Fault Tolerance Content taken from the following: "Distributed Systems: Principles and Paradigms" by Andrew S. Tanenbaum and Maarten Van Steen (Chapter 8) Various online
Dynamo: Amazon s Highly Available Key-value Store
Dynamo: Amazon s Highly Available Key-value Store Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall and
Cluster-Based Scalable Network Services
Cluster-Based Scalable Network Services Suhas Uppalapati INFT 803 Oct 05 1999 (Source : Fox, Gribble, Chawathe, and Brewer, SOSP, 1997) Requirements for SNS Incremental scalability and overflow growth
EECS 498 Introduction to Distributed Systems
EECS 498 Introduction to Distributed Systems Fall 2017 Harsha V. Madhyastha Replicated State Machines Logical clocks Primary/ Backup Paxos? 0 1 (N-1)/2 No. of tolerable failures October 11, 2017 EECS 498
PARALLEL CONSENSUS PROTOCOL
CANOPUS: A SCALABLE AND MASSIVELY PARALLEL CONSENSUS PROTOCOL Bernard Wong CoNEXT 2017 Joint work with Sajjad Rizvi and Srinivasan Keshav CONSENSUS PROBLEM Agreement between a set of nodes in the presence
Using the SDACK Architecture to Build a Big Data Product. Yu-hsin Yeh (Evans Ye) Apache Big Data NA 2016 Vancouver
Using the SDACK Architecture to Build a Big Data Product Yu-hsin Yeh (Evans Ye) Apache Big Data NA 2016 Vancouver Outline A Threat Analytic Big Data product The SDACK Architecture Akka Streams and data
GOSSIP ARCHITECTURE. Gary Berg css434
GOSSIP ARCHITECTURE Gary Berg css434 WE WILL SEE Architecture overview Consistency models How it works Availability and Recovery Performance and Scalability PRELIMINARIES Why replication? Fault tolerance
Presented By: Devarsh Patel
: Amazon s Highly Available Key-value Store Presented By: Devarsh Patel CS5204 Operating Systems 1 Introduction Amazon s e-commerce platform Requires performance, reliability and efficiency To support
The Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff and Shun Tak Leung Google* Shivesh Kumar Sharma fl4164@wayne.edu Fall 2015 004395771 Overview Google file system is a scalable distributed file system
Motivation There are applications for which it is critical to establish certain availability, consistency, performance etc.
1 Motivation Motivation There are applications for which it is critical to establish certain availability, consistency, performance etc. Banking Web mail KOS, CourseWare (to some degree) Questions How
Coherence An Introduction. Shaun Smith Principal Product Manager
Coherence An Introduction Shaun Smith Principal Product Manager About Me Product Manager for Oracle TopLink Involved with object-relational and object-xml mapping technology for over 10 years. Co-Lead
Applications of Paxos Algorithm
Applications of Paxos Algorithm Gurkan Solmaz COP 6938 - Cloud Computing - Fall 2012 Department of Electrical Engineering and Computer Science University of Central Florida - Orlando, FL Oct 15, 2012 1
Important Lessons. Today's Lecture. Two Views of Distributed Systems
Important Lessons Replication good for performance/ reliability Key challenge keeping replicas up-to-date Wide range of consistency models Will see more next lecture Range of correctness properties L-10
Large-Scale Key-Value Stores Eventual Consistency Marco Serafini
Large-Scale Key-Value Stores Eventual Consistency Marco Serafini COMPSCI 590S Lecture 13 Goals of Key-Value Stores Export simple API put(key, value) get(key) Simpler and faster than a DBMS Less complexity,
Linearizability CMPT 401. Sequential Consistency. Passive Replication
Linearizability CMPT 401 Thursday, March 31, 2005 The execution of a replicated service (potentially with multiple requests interleaved over multiple servers) is said to be linearizable if: The interleaved
Pimp My Data Grid. Brian Oliver Senior Principal Solutions Architect <Insert Picture Here>
Pimp My Data Grid Brian Oliver Senior Principal Solutions Architect (brian.oliver@oracle.com) Oracle Coherence Oracle Fusion Middleware Agenda An Architectural Challenge Enter the
Huge market -- essentially all high performance databases work this way
11/5/2017 Lecture 16 -- Parallel & Distributed Databases Parallel/distributed databases: goal provide exactly the same API (SQL) and abstractions (relational tables), but partition data across a bunch
Hi! NET Developer Group Braunschweig!
Hi! NET Developer Group Braunschweig! Über Tobias Dipl. Informatiker (FH) Passionated Software Developer Clean Code Developer.NET Junkie.NET User Group Lead Microsoft PFE Software Development Twitter @Blubern
Distributed DNS Name server Backed by Raft
Distributed DNS Name server Backed by Raft Emre Orbay Gabbi Fisher December 13, 2017 Abstract We implement a asynchronous distributed DNS server backed by Raft. Our DNS server system supports large cluster
Building Web-Scale Applications with AWS
#reinvent Building Web-Scale Applications with AWS Simon Elisha / Principal Solution Architect / @simon_elisha James Hamilton / Vice President & Distinguished Engineer 8 Months of Travel 7 Minutes of Terror
Introduction to riak_ensemble. Joseph Blomstedt Basho Technologies
Introduction to riak_ensemble Joseph Blomstedt (@jtuple) Basho Technologies riak_ensemble Paxos framework for scalable consistent system 2 node node node node node node node node 3 What about state? 4
Ceph: A Scalable, High-Performance Distributed File System
Ceph: A Scalable, High-Performance Distributed File System S. A. Weil, S. A. Brandt, E. L. Miller, D. D. E. Long Presented by Philip Snowberger Department of Computer Science and Engineering University
Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams
Photon: Fault-tolerant and Scalable Joining of Continuous Data Streams Rajagopal Ananthanarayanan, Venkatesh Basker, Sumit Das, Ashish Gupta, Haifeng Jiang, Tianhao Qiu, Alexey Reznichenko, Deomid Ryabkov,
CPS 512 midterm exam #1, 10/7/2016
CPS 512 midterm exam #1, 10/7/2016 Your name please: NetID: Answer all questions. Please attempt to confine your answers to the boxes provided. If you don t know the answer to a question, then just say
How you can benefit from using. javier
How you can benefit from using I was Lois Lane redis has super powers myth: the bottleneck redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop,mset -P 16 -q On my laptop: SET: 513610 requests
Cloud-Native Applications. Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0
Cloud-Native Applications Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0 Cloud-Native Characteristics Lean Form a hypothesis, build just enough to validate or disprove it. Learn
Building Microservices with the 12 Factor App Pattern
Building Microservices with the 12 Factor App Pattern Context This documentation will help introduce Developers to implementing MICROSERVICES by applying the TWELVE- FACTOR PRINCIPLES, a set of best practices
CS 347 Parallel and Distributed Data Processing
CS 347 Parallel and Distributed Data Processing Spring 2016 Notes 12: Distributed Information Retrieval CS 347 Notes 12 2 CS 347 Notes 12 3 CS 347 Notes 12 4 CS 347 Notes 12 5 Web Search Engine Crawling
Last time. Distributed systems Lecture 6: Elections, distributed transactions, and replication. DrRobert N. M. Watson
Distributed systems Lecture 6: Elections, distributed transactions, and replication DrRobert N. M. Watson 1 Last time Saw how we can build ordered multicast Messages between processes in a group Need to
CS 347 Parallel and Distributed Data Processing
CS 347 Parallel and Distributed Data Processing Spring 2016 Notes 12: Distributed Information Retrieval CS 347 Notes 12 2 CS 347 Notes 12 3 CS 347 Notes 12 4 Web Search Engine Crawling Indexing Computing
Transactions and ACID
Transactions and ACID Kevin Swingler Contents Recap of ACID transactions in RDBMSs Transactions and ACID in MongoDB 1 Concurrency Databases are almost always accessed by multiple users concurrently A user
Distributed Meta-data Servers: Architecture and Design. Sarah Sharafkandi David H.C. Du DISC
Distributed Meta-data Servers: Architecture and Design Sarah Sharafkandi David H.C. Du DISC 5/22/07 1 Outline Meta-Data Server (MDS) functions Why a distributed and global Architecture? Problem description
TANDBERG Management Suite - Redundancy Configuration and Overview
Management Suite - Redundancy Configuration and Overview TMS Software version 11.7 TANDBERG D50396 Rev 2.1.1 This document is not to be reproduced in whole or in part without the permission in writing
Cassandra - A Decentralized Structured Storage System. Avinash Lakshman and Prashant Malik Facebook
Cassandra - A Decentralized Structured Storage System Avinash Lakshman and Prashant Malik Facebook Agenda Outline Data Model System Architecture Implementation Experiments Outline Extension of Bigtable
Designing and Evaluating a Distributed Computing Language Runtime. Christopher Meiklejohn Université catholique de Louvain, Belgium
Designing and Evaluating a Distributed Computing Language Runtime Christopher Meiklejohn (@cmeik) Université catholique de Louvain, Belgium R A R B R A set() R B R A set() set(2) 2 R B set(3) 3 set() set(2)
Distributed PostgreSQL with YugaByte DB
Distributed PostgreSQL with YugaByte DB Karthik Ranganathan PostgresConf Silicon Valley Oct 16, 2018 1 CHECKOUT THIS REPO: github.com/yugabyte/yb-sql-workshop 2 About Us Founders Kannan Muthukkaruppan,
Strong Consistency & CAP Theorem
Strong Consistency & CAP Theorem CS 240: Computing Systems and Concurrency Lecture 15 Marco Canini Credits: Michael Freedman and Kyle Jamieson developed much of the original material. Consistency models
Distributed Systems. Day 13: Distributed Transaction. To Be or Not to Be Distributed.. Transactions
Distributed Systems Day 13: Distributed Transaction To Be or Not to Be Distributed.. Transactions Summary Background on Transactions ACID Semantics Distribute Transactions Terminology: Transaction manager,,
Microservices at Netflix Scale. First Principles, Tradeoffs, Lessons Learned Ruslan
Microservices at Netflix Scale First Principles, Tradeoffs, Lessons Learned Ruslan Meshenberg @rusmeshenberg Microservices: all benefits, no costs? Netflix is the world s leading Internet television network
Deep Learning Inference as a Service
Deep Learning Inference as a Service Mohammad Babaeizadeh Hadi Hashemi Chris Cai Advisor: Prof Roy H. Campbell Use case 1: Model Developer Use case 1: Model Developer Inference Service Use case
The Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google* 정학수, 최주영 1 Outline Introduction Design Overview System Interactions Master Operation Fault Tolerance and Diagnosis Conclusions
Performance, Scalability and High-availability of Enterprise Applications
Performance, Scalability and High-availability of Enterprise Applications Miroslav Blaško miroslav.blasko@fel.cvut.cz Winter Term 2017 Miroslav Blaško (miroslav.blasko@fel.cvut.cz)performance, Scalability
Efficient HTTP based I/O on very large datasets for high performance computing with the Libdavix library
Efficient HTTP based I/O on very large datasets for high performance computing with the Libdavix library Authors Devresse Adrien (CERN) Fabrizio Furano (CERN) Typical HPC architecture Computing Cluster
Lecture XIII: Replication-II
Lecture XIII: Replication-II CMPT 401 Summer 2007 Dr. Alexandra Fedorova Outline Google File System A real replicated file system Paxos Harp A consensus algorithm used in real systems A replicated research
Programming Distributed Systems
Annette Bieniusa Programming Distributed Systems Summer Term 2018 1/ 26 Programming Distributed Systems 09 Testing Distributed Systems Annette Bieniusa AG Softech FB Informatik TU Kaiserslautern Summer
How to go serverless with AWS Lambda
How to go serverless with AWS Lambda Roman Plessl, nine (AWS Partner) Zürich, AWSomeDay 12. September 2018 About myself and nine Roman Plessl Working for nine as a Solution Architect, Consultant and Leader.
Cloud Computing CS
Cloud Computing CS 15-319 Distributed File Systems and Cloud Storage Part I Lecture 12, Feb 22, 2012 Majd F. Sakr, Mohammad Hammoud and Suhail Rehman 1 Today Last two sessions Pregel, Dryad and GraphLab
Causal Consistency. CS 240: Computing Systems and Concurrency Lecture 16. Marco Canini
Causal Consistency CS 240: Computing Systems and Concurrency Lecture 16 Marco Canini Credits: Michael Freedman and Kyle Jamieson developed much of the original material. Consistency models Linearizability
EECS 498 Introduction to Distributed Systems
EECS 498 Introduction to Distributed Systems Fall 2017 Harsha V. Madhyastha Dynamo Recap Consistent hashing 1-hop DHT enabled by gossip Execution of reads and writes Coordinated by first available successor
Microsoft Azure Databricks for data engineering. Building production data pipelines with Apache Spark in the cloud
Microsoft Azure Databricks for data engineering Building production data pipelines with Apache Spark in the cloud Azure Databricks As companies continue to set their sights on making data-driven decisions
Large-Scale Web Applications
Large-Scale Web Applications Mendel Rosenblum Web Application Architecture Web Browser Web Server / Application server Storage System HTTP Internet CS142 Lecture Notes - Intro LAN 2 Large-Scale: Scale-Out
Staggeringly Large File Systems. Presented by Haoyan Geng
Staggeringly Large File Systems Presented by Haoyan Geng Large-scale File Systems How Large? Google s file system in 2009 (Jeff Dean, LADIS 09) - 200+ clusters - Thousands of machines per cluster - Pools
Self-healing Data Step by Step
Self-healing Data Step by Step Uwe Friedrichsen (codecentric AG) NoSQL matters Cologne, 29. April 2014 @ufried Uwe Friedrichsen uwe.friedrichsen@codecentric.de http://slideshare.net/ufried http://ufried.tumblr.com
One Trillion Edges. Graph processing at Facebook scale
One Trillion Edges Graph processing at Facebook scale Introduction Platform improvements Compute model extensions Experimental results Operational experience How Facebook improved Apache Giraph Facebook's
Today CSCI Coda. Naming: Volumes. Coda GFS PAST. Instructor: Abhishek Chandra. Main Goals: Volume is a subtree in the naming space
Today CSCI 5105 Coda GFS PAST Instructor: Abhishek Chandra 2 Coda Main Goals: Availability: Work in the presence of disconnection Scalability: Support large number of users Successor of Andrew File System
Google File System. By Dinesh Amatya
Google File System By Dinesh Amatya Google File System (GFS) Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung designed and implemented to meet rapidly growing demand of Google's data processing need a scalable
Building Durable Real-time Data Pipeline
Building Durable Real-time Data Pipeline Apache BookKeeper at Twitter @sijieg Twitter Background Layered Architecture Agenda Design Details Performance Scale @Twitter Q & A Publish-Subscribe Online services
Dynamo: Amazon s Highly Available Key-value Store. ID2210-VT13 Slides by Tallat M. Shafaat
Dynamo: Amazon s Highly Available Key-value Store ID2210-VT13 Slides by Tallat M. Shafaat Dynamo An infrastructure to host services Reliability and fault-tolerance at massive scale Availability providing
CSE-E5430 Scalable Cloud Computing Lecture 10
CSE-E5430 Scalable Cloud Computing Lecture 10 Keijo Heljanko Department of Computer Science School of Science Aalto University keijo.heljanko@aalto.fi 23.11-2015 1/29 Exam Registering for the exam is obligatory,
Achieving Scalability and High Availability for clustered Web Services using Apache Synapse. Ruwan Linton WSO2 Inc.
Achieving Scalability and High Availability for clustered Web Services using Apache Synapse Ruwan Linton [ruwan@apache.org] WSO2 Inc. Contents Introduction Apache Synapse Web services clustering Scalability/Availability
Consistency and Replication
Topics to be covered Introduction Consistency and Replication Consistency Models Distribution Protocols Consistency Protocols 1 2 + Performance + Reliability Introduction Introduction Availability: proportion
Distributed Systems Intro and Course Overview
Distributed Systems Intro and Course Overview COS 418: Distributed Systems Lecture 1 Wyatt Lloyd Distributed Systems, What? 1) Multiple computers 2) Connected by a network 3) Doing something together Distributed
The Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Google SOSP 03, October 19 22, 2003, New York, USA Hyeon-Gyu Lee, and Yeong-Jae Woo Memory & Storage Architecture Lab. School
The Next Generation of Extreme OLTP Processing with Oracle TimesTen
The Next Generation of Extreme OLTP Processing with TimesTen Tirthankar Lahiri Redwood Shores, California, USA Keywords: TimesTen, velocity scaleout elastic inmemory relational database cloud Introduction
MapReduce Spark. Some slides are adapted from those of Jeff Dean and Matei Zaharia
MapReduce Spark Some slides are adapted from those of Jeff Dean and Matei Zaharia What have we learnt so far? Distributed storage systems consistency semantics protocols for fault tolerance Paxos, Raft,
Scalability of web applications
Scalability of web applications CSCI 470: Web Science Keith Vertanen Copyright 2014 Scalability questions Overview What's important in order to build scalable web sites? High availability vs. load balancing
Data-Intensive Distributed Computing
Data-Intensive Distributed Computing CS 451/651 (Fall 2018) Part 7: Mutable State (2/2) November 13, 2018 Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo These slides are
Key-value store with eventual consistency without trusting individual nodes
basementdb Key-value store with eventual consistency without trusting individual nodes https://github.com/spferical/basementdb 1. Abstract basementdb is an eventually-consistent key-value store, composed
Database Availability and Integrity in NoSQL. Fahri Firdausillah [M ]
![Database Availability and Integrity in NoSQL. Fahri Firdausillah [M ]](/thumbs/83/88388270.jpg "Database Availability and Integrity in NoSQL. Fahri Firdausillah [M ]")
Database Availability and Integrity in NoSQL Fahri Firdausillah [M031010012] What is NoSQL Stands for Not Only SQL Mostly addressing some of the points: nonrelational, distributed, horizontal scalable,
MINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES
MINIMIZING TRANSACTION LATENCY IN GEO-REPLICATED DATA STORES Divy Agrawal Department of Computer Science University of California at Santa Barbara Joint work with: Amr El Abbadi, Hatem Mahmoud, Faisal