0-1 Million in 46 Days Scaling a Facebook Application in Rails
|
|
|
- Bethany Rice
- 6 years ago
- Views:
Transcription
1 0-1 Million in 46 Days Scaling a Facebook Application in Rails Ikai Lan Linkedin Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 1
2 Hi! I m Ikai Lan Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 2
3 What this talk is not One-size-fits all solution to scaling problems Example source code An in-depth discussion of Ruby on Rails Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 3
4 What this talk is A look into the challenges we faced with scaling a rapid growth web application Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 4
5 This is a detective story. Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 5
6 Rather than telling you best practices Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 6
7 We ll talk about how we came to define them. Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 7
8 Bumper Sticker? Investigation of the Facebook platform Investigation of Ruby on Rails as a serious development platform Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 8
9 Reid Hoffman s wager Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 9
10 1 million users in 45 days! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 10
11 Personal Badging Create and display badges Share badges Find badges Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 11
12 = Bumper Sticker! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 12
13 Success! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 13
14 1 million in 46 days Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 14
15 Trouble in Paradise Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 15
16 Throw hardware at it! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 16
17 More hardware! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 17
18 This is not working. Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 18
19 Current state of affairs 7 million adds 20 million pageviews 3 million timeouts a day Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 19
20 Strategy Analyze MySQL slow queries log Application log Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 20
21 D oh! No logging! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 21
22 Logging -> Syslog Rails does not buffer writes to filesystem Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 22
23 Clue 1: Slaves ran several hours behind Master Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 23
24 Clue 2: Updates to add count One by one updates Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 24
25 Counter Cache SQL Updates not batched Very inefficient! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 25
26 Result: less slow queries Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 26
27 Lesson 1: Buffer everything! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 27
28 Still had lots of timeouts! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 28
29 Clue 3: 80% of traffic to static content Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 29
30 Next attempt: edge caching! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 30
31 Edge caching using fb:ref Facebook mechanism for storing data Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 31
32 Fb:ref on the load balancer! F5 BigIP set to conditionally respond to static URIs with fb:ref Background process to constantly update fb:ref in Facebook cache Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 32
33 Result: Static pages = perfect Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 33
34 Lesson 2: Edge cache when possible Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 34
35 But we were still serving millions of timeouts! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 35
36 Clue 4: Most CPU time accepting stickers Request taking up to 10 seconds during peak Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 36
37 Slow database writes Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 37
38 1st attempt: Background process writes Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 38
39 Result: Timeouts dropped significantly! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 39
40 Success? Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 40
41 No. Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 41
42 Application would not seem responsive to user actions and seem out of sync. Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 42
43 2nd attempt: Two step write Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 43
44 Step 1: Synchronous write to MemCache Step 2: Asynchronous write to database (queue) Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 44
45 Result: Success! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 45
46 Lesson 3: Use asynchronous processing when possible Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 46
47 Lesson 3.1: Keeping data in sync is not trivial! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 47
48 Lesson 3.2: Lack of concurrency in Rails presents problem earlier Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 48
49 Lesson 3.3: Very few problems can be reduced to a linear equation Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 49
50 QED: Rails can scale. Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 50
51 Latest stats 16 million installs 30 million pageviews a day Average response time of <80ms Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 51
52 Scaling has a price! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 52
53 Complexity Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 53
54 No new functionality! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 54
55 Key takeaways Make long running tasks asynchronous Use edge caching Buffer long operations when possible Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 55
56 The bigger picture Language choice is less important than architecture Be creative brute force doesn t work Be ready to be wrong Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 56
57 Questions? Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 57
58 Thanks for coming! Ikai Lan From 0 to 1,000,000 in 46 Days: Scaling a Facebook Application in Rails Slide 58
MySQL Performance Improvements
Taking Advantage of MySQL Performance Improvements Baron Schwartz, Percona Inc. Introduction About Me (Baron Schwartz) Author of High Performance MySQL 2 nd Edition Creator of Maatkit, innotop, and so
MySQL HA Solutions. Keeping it simple, kinda! By: Chris Schneider MySQL Architect Ning.com
MySQL HA Solutions Keeping it simple, kinda! By: Chris Schneider MySQL Architect Ning.com What we ll cover today High Availability Terms and Concepts Levels of High Availability What technologies are there
Scaling App Engine Applications. Justin Haugh, Guido van Rossum May 10, 2011
Scaling App Engine Applications Justin Haugh, Guido van Rossum May 10, 2011 First things first Justin Haugh Software Engineer Systems Infrastructure jhaugh@google.com Guido Van Rossum Software Engineer
High-resolution Measurement of Data Center Microbursts
High-resolution Measurement of Data Center Microbursts Qiao Zhang (University of Washington) Vincent Liu (University of Pennsylvania) Hongyi Zeng (Facebook) Arvind Krishnamurthy (University of Washington)
CISC 7610 Lecture 2b The beginnings of NoSQL
CISC 7610 Lecture 2b The beginnings of NoSQL Topics: Big Data Google s infrastructure Hadoop: open google infrastructure Scaling through sharding CAP theorem Amazon s Dynamo 5 V s of big data Everyone
The Right Read Optimization is Actually Write Optimization. Leif Walsh
The Right Read Optimization is Actually Write Optimization Leif Walsh leif@tokutek.com The Right Read Optimization is Write Optimization Situation: I have some data. I want to learn things about the world,
MySQL Database Scalability
MySQL Database Scalability Nextcloud Conference 2016 TU Berlin Oli Sennhauser Senior MySQL Consultant at FromDual GmbH oli.sennhauser@fromdual.com 1 / 14 About FromDual GmbH Support Consulting remote-dba
How to pimp high volume PHP websites. 27. September 2008, PHP conference Barcelona. By Jens Bierkandt
How to pimp high volume PHP websites 27. September 2008, PHP conference Barcelona By Jens Bierkandt 1 About me Jens Bierkandt Working with PHP since 2000 From Germany, living in Spain, speaking English
IX: A Protected Dataplane Operating System for High Throughput and Low Latency
IX: A Protected Dataplane Operating System for High Throughput and Low Latency Belay, A. et al. Proc. of the 11th USENIX Symp. on OSDI, pp. 49-65, 2014. Reviewed by Chun-Yu and Xinghao Li Summary In this
MSG: An Overview of a Messaging System for the Grid
MSG: An Overview of a Messaging System for the Grid Daniel Rodrigues Presentation Summary Current Issues Messaging System Testing Test Summary Throughput Message Lag Flow Control Next Steps Current Issues
MYSQL TROUBLESHOOTING. Or, what to do when MySQL starts throwing a fit
MYSQL TROUBLESHOOTING Or, what to do when MySQL starts throwing a fit ABOUT THE CLASS 24 hours over three days Very Short Lecture and Lots of Labs Hours: 8:30am - 5:00pm Lunch: 11:45am - 1:00pm ABOUT THE
ScaleArc Performance Benchmarking with sysbench
MySQL Performance Blog ScaleArc Performance Benchmarking with sysbench Peter Boros, 2014 1/31 Author: Peter Boros Revision: 2.0 Date: Mar 28, 2014 Customer: ScaleArc Contents 1 Executive Summary 3 2 About
ò Server can crash or be disconnected ò Client can crash or be disconnected ò How to coordinate multiple clients accessing same file?
Big picture (from Sandberg et al.) NFS Don Porter CSE 506 Intuition Challenges Instead of translating VFS requests into hard drive accesses, translate them into remote procedure calls to a server Simple,
InnoDB Scalability Limits. Peter Zaitsev, Vadim Tkachenko Percona Inc MySQL Users Conference 2008 April 14-17, 2008
InnoDB Scalability Limits Peter Zaitsev, Vadim Tkachenko Percona Inc MySQL Users Conference 2008 April 14-17, 2008 -2- Who are the Speakers? Founders of Percona Inc MySQL Performance and Scaling consulting
NFS. Don Porter CSE 506
NFS Don Porter CSE 506 Big picture (from Sandberg et al.) Intuition ò Instead of translating VFS requests into hard drive accesses, translate them into remote procedure calls to a server ò Simple, right?
DB2 is a complex system, with a major impact upon your processing environment. There are substantial performance and instrumentation changes in
DB2 is a complex system, with a major impact upon your processing environment. There are substantial performance and instrumentation changes in versions 8 and 9. that must be used to measure, evaluate,
Drupal Hosting. April 19, Northeast Ohio Drupal User Group 1
Northeast Ohio Drupal User Group 1 Security: PSA came out Monday regarding a d8 release for Wednesday. The notice suggested that it was a serious flaw and exploits were expected within short order after
Facebook Tao Distributed Data Store for the Social Graph
L. Lancia, G. Salillari Cloud Computing Master Degree in Data Science Sapienza Università di Roma Facebook Tao Distributed Data Store for the Social Graph L. Lancia & G. Salillari 1 / 40 Table of Contents
Low Latency Data Grids in Finance
Low Latency Data Grids in Finance Jags Ramnarayan Chief Architect GemStone Systems jags.ramnarayan@gemstone.com Copyright 2006, GemStone Systems Inc. All Rights Reserved. Background on GemStone Systems
Distributed Systems. 29. Distributed Caching Paul Krzyzanowski. Rutgers University. Fall 2014
Distributed Systems 29. Distributed Caching Paul Krzyzanowski Rutgers University Fall 2014 December 5, 2014 2013 Paul Krzyzanowski 1 Caching Purpose of a cache Temporary storage to increase data access
Using PostgreSQL in Tantan - From 0 to 350bn rows in 2 years
Using PostgreSQL in Tantan - From 0 to 350bn rows in 2 years Victor Blomqvist vb@viblo.se Tantan ( 探探 ) December 2, PGConf Asia 2016 in Tokyo tantanapp.com 1 Sweden - Tantan - Tokyo 10 Million 11 Million
Life as a Service. Scalability and Other Aspects. Dino Esposito JetBrains ARCHITECT, TRAINER AND CONSULTANT
Life as a Service Scalability and Other Aspects Dino Esposito JetBrains ARCHITECT, TRAINER AND CONSULTANT PART I Scalability and Measurable Tasks SCALABILITY Scalability is the ability of a system to expand
Inside the PostgreSQL Shared Buffer Cache
Truviso 07/07/2008 About this presentation The master source for these slides is http://www.westnet.com/ gsmith/content/postgresql You can also find a machine-usable version of the source code to the later
Bringing Multi-Threading to the I/O Level
Bringing Multi-Threading to the I/O Level Rodrigo Medeiros Solutions Architect, Data Platforms @rationalrodrigo @DataCore Copyright 2018 DataCore Software Corp. All Rights Reserved. Objectives Discuss
MySQL Performance Optimization and Troubleshooting with PMM. Peter Zaitsev, CEO, Percona
MySQL Performance Optimization and Troubleshooting with PMM Peter Zaitsev, CEO, Percona In the Presentation Practical approach to deal with some of the common MySQL Issues 2 Assumptions You re looking
Data-Intensive Distributed Computing
Data-Intensive Distributed Computing CS 451/651 (Fall 2018) Part 7: Mutable State (2/2) November 13, 2018 Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo These slides are
ECE 550D Fundamentals of Computer Systems and Engineering. Fall 2017
ECE 550D Fundamentals of Computer Systems and Engineering Fall 2017 The Operating System (OS) Prof. John Board Duke University Slides are derived from work by Profs. Tyler Bletsch and Andrew Hilton (Duke)
TIME AND SYNCHRONIZATION. I. Physical Clock Synchronization: Motivation and Challenges
TIME AND SYNCHRONIZATION In previous lectures, we discussed some important concepts and goals of distributed. One important concept is remote procedure calls, where we saw how failures creep up into semantics
<Insert Picture Here> MySQL Web Reference Architectures Building Massively Scalable Web Infrastructure
MySQL Web Reference Architectures Building Massively Scalable Web Infrastructure Mario Beck (mario.beck@oracle.com) Principal Sales Consultant MySQL Session Agenda Requirements for
Goals. Facebook s Scaling Problem. Scaling Strategy. Facebook Three Layer Architecture. Workload. Memcache as a Service.
Goals Memcache as a Service Tom Anderson Rapid application development - Speed of adding new features is paramount Scale Billions of users Every user on FB all the time Performance Low latency for every
Distributed Systems Exam 1 Review Paul Krzyzanowski. Rutgers University. Fall 2016
Distributed Systems 2015 Exam 1 Review Paul Krzyzanowski Rutgers University Fall 2016 1 Question 1 Why did the use of reference counting for remote objects prove to be impractical? Explain. It s not fault
COTS Integration and Debugging Challenges - RBSP Lessons Learned. Subodh Harmalkar Joseph Hennawy Samuel Fix Debbie Clancy
COTS Integration and Debugging Challenges - RBSP Lessons Learned Subodh Harmalkar Joseph Hennawy Samuel Fix Debbie Clancy Agenda Background Testbed Architecture Intel PCI bus architecture Testbed modifications
Inside Relational Databases with Examples in Access
Inside Relational Databases with Examples in Access Inside Relational Databases with Examples in Access Mark Whitehorn and Bill Marklyn 123 Mark Whitehorn Applied Computing Division, University of Dundee,
Massive Scalability With InterSystems IRIS Data Platform
Massive Scalability With InterSystems IRIS Data Platform Introduction Faced with the enormous and ever-growing amounts of data being generated in the world today, software architects need to pay special
Servers & Developers. Julian Nadeau Production Engineer
Servers & Developers Julian Nadeau Production Engineer Provisioning & Orchestration of Servers Setting a server up Packer - one server at a time Chef - all servers at once Containerization What are Containers?
Effective Testing for Live Applications. March, 29, 2018 Sveta Smirnova
Effective Testing for Live Applications March, 29, 2018 Sveta Smirnova Table of Contents Sometimes You Have to Test on Production Wrong Data SELECT Returns Nonsense Wrong Data in the Database Performance
Big Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2017)
Big Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2017) Week 10: Mutable State (2/2) March 16, 2017 Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo These
Real World Web Scalability. Ask Bjørn Hansen Develooper LLC
Real World Web Scalability Ask Bjørn Hansen Develooper LLC Hello. 28 brilliant methods to make your website keep working past $goal requests/transactions/sales per second/hour/day Requiring minimal extra
Lesson 11 Transcript: Concurrency and locking
Lesson 11 Transcript: Concurrency and locking Slide 1: Cover Welcome to Lesson 11 of the DB2 on Campus Lecture Series. We are going to talk today about concurrency and locking. My name is Raul Chong and
The Care and Feeding of a MySQL Database for Linux Adminstrators. Dave Stokes MySQL Community Manager
The Care and Feeding of a MySQL Database for Linux Adminstrators Dave Stokes MySQL Community Manager David.Stokes@Oracle.com Simple Introduction This is a general introduction to running a MySQL database
COMP9242 Advanced OS. Copyright Notice. The Memory Wall. Caching. These slides are distributed under the Creative Commons Attribution 3.
Copyright Notice COMP9242 Advanced OS S2/2018 W03: s: What Every OS Designer Must Know @GernotHeiser These slides are distributed under the Creative Commons Attribution 3.0 License You are free: to share
Distributed Systems. Lec 9: Distributed File Systems NFS, AFS. Slide acks: Dave Andersen
Distributed Systems Lec 9: Distributed File Systems NFS, AFS Slide acks: Dave Andersen (http://www.cs.cmu.edu/~dga/15-440/f10/lectures/08-distfs1.pdf) 1 VFS and FUSE Primer Some have asked for some background
Twitch Plays Pokémon: Twitch s Chat Architecture. John Rizzo Sr Software Engineer
Twitch Plays Pokémon: Twitch s Chat Architecture John Rizzo Sr Software Engineer About Me Twitch Introduction Twitch Introduction Twitch Introduction Over 800k concurrent users Tens of BILLIONS of daily
Thursday, March 8, 12. Architecture at Scale
Architecture at Scale About the speaker Manny Pelarinos- Sr. Director of Engineering, ESPN manole.pelarinos@espn.com Been with ESPN for 5 years First time at QCon! 2 Agenda High Level Architecture Overview
COMP9242 Advanced OS. S2/2017 W03: Caches: What Every OS Designer Must
COMP9242 Advanced OS S2/2017 W03: Caches: What Every OS Designer Must Know @GernotHeiser Copyright Notice These slides are distributed under the Creative Commons Attribution 3.0 License You are free: to
Buses. Disks PCI RDRAM RDRAM LAN. Some slides adapted from lecture by David Culler. Pentium 4 Processor. Memory Controller Hub.
es > 100 MB/sec Pentium 4 Processor L1 and L2 caches Some slides adapted from lecture by David Culler 3.2 GB/sec Display Memory Controller Hub RDRAM RDRAM Dual Ultra ATA/100 24 Mbit/sec Disks LAN I/O Controller
Optimizing a News Website for Speed and Stability
Optimizing a News Website for Speed and Stability Julian Dunn SecondMarket, Inc. Blake Crosby Canadian Broadcasting Corp. - Who has worked for a news website - Who has configured a CDN - familiar with
From Single Purpose to Multi Purpose Data Lakes. Thomas Niewel Technical Sales Director DACH Denodo Technologies March, 2019
From Single Purpose to Multi Purpose Data Lakes Thomas Niewel Technical Sales Director DACH Denodo Technologies March, 2019 Agenda Data Lakes Multiple Purpose Data Lakes Customer Example Demo Takeaways
WALK MS Fundraise with. Guide To Fundraising with Facebook Created by the Georgia Chapter
WALK MS Fundraise with Fundraise with Facebook Using CharityDynamics new boundlessfundraising TM Application for Facebook, you can extent your fundraising efforts beyond the National MS Society s Participant
To Shard or Not to Shard That is the question! Peter Zaitsev April 21, 2016
To Shard or Not to Shard That is the question! Peter Zaitsev April 21, 2016 Story Let s start with the story 2 First things to decide Before you decide how to shard you d best understand whether or not
Improve WordPress performance with caching and deferred execution of code. Danilo Ercoli Software Engineer
Improve WordPress performance with caching and deferred execution of code Danilo Ercoli Software Engineer http://daniloercoli.com Agenda PHP Caching WordPress Page Caching WordPress Object Caching Deferred
Fixing Twitter.... and Finding your own Fail Whale. John Adams Twitter Operations
Fixing Twitter... and Finding your own Fail Whale John Adams Twitter Operations Operations Small team, growing rapidly. What do we do? Software Performance (back-end) Availability Capacity
Building a Scalable Architecture for Web Apps - Part I (Lessons Directi)
Intelligent People. Uncommon Ideas. Building a Scalable Architecture for Web Apps - Part I (Lessons Learned @ Directi) By Bhavin Turakhia CEO, Directi (http://www.directi.com http://wiki.directi.com http://careers.directi.com)
MySQL. The Right Database for GIS Sometimes
MySQL The Right Database for GIS Sometimes Who am I? Web/GIS Software Engineer with Cimbura.com BS in IT, MGIS Michael Moore I like making and using tools (digital or physical) GIS Web Services I m most
Lecture 17 Introduction to Memory Hierarchies" Why it s important " Fundamental lesson(s)" Suggested reading:" (HP Chapter
Processor components" Multicore processors and programming" Processor comparison" vs." Lecture 17 Introduction to Memory Hierarchies" CSE 30321" Suggested reading:" (HP Chapter 5.1-5.2)" Writing more "
MapReduce: Simplified Data Processing on Large Clusters. By Stephen Cardina
MapReduce: Simplified Data Processing on Large Clusters By Stephen Cardina The Problem You have a large amount of raw data, such as a database or a web log, and you need to get some sort of derived data
1. A synchronous FIFO can only be used if both the read and write interface share the same clock.
FIFO Answers 1. A synchronous FIFO can only be used if both the read and write interface share the same clock. 2. The depth of the FIFO is determined by the number of addressable locations in the memory
Map Reduce. Yerevan.
Map Reduce Erasmus+ @ Yerevan dacosta@irit.fr Divide and conquer at PaaS 100 % // Typical problem Iterate over a large number of records Extract something of interest from each Shuffle and sort intermediate
Migrating To MySQL The Live Database Upgrade Guide
Migrating To MySQL 5.7 - The Live Database Upgrade Guide October 4, 2016 Krzysztof Książek Severalnines krzysztof@severalnines.com 1 Agenda! Why upgrading to MySQL 5.7?! Preparing an upgrade - changes
MongoDB - a No SQL Database What you need to know as an Oracle DBA
MongoDB - a No SQL Database What you need to know as an Oracle DBA David Burnham Aims of this Presentation To introduce NoSQL database technology specifically using MongoDB as an example To enable the
Cloud Programming on Java EE Platforms. mgr inż. Piotr Nowak
Cloud Programming on Java EE Platforms mgr inż. Piotr Nowak Distributed data caching environment Hadoop Apache Ignite "2 Cache what is cache? how it is used? "3 Cache - hardware buffer temporary storage
Computer Networks. Course Reference Model. Topic. Congestion What s the hold up? Nature of Congestion. Nature of Congestion 1/5/2015.
Course Reference Model Computer Networks 7 Application Provides functions needed by users Zhang, Xinyu Fall 204 4 Transport Provides end-to-end delivery 3 Network Sends packets over multiple links School
Learning with Purpose
Network Measurement for 100Gbps Links Using Multicore Processors Xiaoban Wu, Dr. Peilong Li, Dr. Yongyi Ran, Prof. Yan Luo Department of Electrical and Computer Engineering University of Massachusetts
All you need is fun. Cons T Åhs Keeper of The Code
All you need is fun Cons T Åhs Keeper of The Code cons@klarna.com Cons T Åhs Keeper of The Code at klarna Architecture - The Big Picture Development - getting ideas to work Code Quality - care about the
Extreme Computing. NoSQL.
Extreme Computing NoSQL PREVIOUSLY: BATCH Query most/all data Results Eventually NOW: ON DEMAND Single Data Points Latency Matters One problem, three ideas We want to keep track of mutable state in a scalable
Traffic is coming! OMG moments
CASE STUDY Traffic is coming! OMG moments How Distinct helped evomag.ro handle Blackfriday 2013 traffic spike Summary: 1. Who 2. The Challenge 3. The Work 4. Aftermath 1. Who :: Bogdan Belu - Distinct
CS18000: Programming I
CS18000: Programming I Synchronization 22 February, 2010 Prof. Chris Clifton Concurrency Example: Banking class ATM { public void withdrawcash(acct a) { Scanner sc = new Scanner(System.in); int amount
From the event loop to the distributed system. Martyn 3rd November, 2011
From the event loop to the distributed system Martyn Loughran martyn@pusher.com @mloughran 3rd November, 2011 From the event loop to the distributed system From the event loop to the distributed system
Basic Reliable Transport Protocols
Basic Reliable Transport Protocols Do not be alarmed by the length of this guide. There are a lot of pictures. You ve seen in lecture that most of the networks we re dealing with are best-effort : they
HBase Solutions at Facebook
HBase Solutions at Facebook Nicolas Spiegelberg Software Engineer, Facebook QCon Hangzhou, October 28 th, 2012 Outline HBase Overview Single Tenant: Messages Selection Criteria Multi-tenant Solutions
Consistency: Relaxed. SWE 622, Spring 2017 Distributed Software Engineering
Consistency: Relaxed SWE 622, Spring 2017 Distributed Software Engineering Review: HW2 What did we do? Cache->Redis Locks->Lock Server Post-mortem feedback: http://b.socrative.com/ click on student login,
Distributed Architectures & Microservices. CS 475, Spring 2018 Concurrent & Distributed Systems
Distributed Architectures & Microservices CS 475, Spring 2018 Concurrent & Distributed Systems GFS Architecture GFS Summary Limitations: Master is a huge bottleneck Recovery of master is slow Lots of success
Architecture and Design of MySQL Powered Applications. Peter Zaitsev CEO, Percona Highload Moscow, Russia 31 Oct 2014
Architecture and Design of MySQL Powered Applications Peter Zaitsev CEO, Percona Highload++ 2014 Moscow, Russia 31 Oct 2014 About Percona 2 Open Source Software for MySQL Ecosystem Percona Server Percona
Media-Ready Network Transcript
Media-Ready Network Transcript Hello and welcome to this Cisco on Cisco Seminar. I m Bob Scarbrough, Cisco IT manager on the Cisco on Cisco team. With me today are Sheila Jordan, Vice President of the
Scaling Without Sharding. Baron Schwartz Percona Inc Surge 2010
Scaling Without Sharding Baron Schwartz Percona Inc Surge 2010 Web Scale!!!! http://www.xtranormal.com/watch/6995033/ A Sharding Thought Experiment 64 shards per proxy [1] 1 TB of data storage per node
#1593: The top 10 things that can go wrong with an IBM Traveler Server
#1593: The top 10 things that can go wrong with an IBM Traveler Server plus how to detect and correct them Alan Forbes Acknowledgements and Disclaimer. Copyright IBM Corporation 2016. All rights reserved.
LAMP Tuning. Sean Walberg Network guy, server ninja
LAMP Tuning Sean Walberg Network guy, server ninja LAMP Linux server Apache Web Server PHP/Perl application MySQL server Started off with 2 servers, 10 now 65m pageviews/month 20mbit/s of web traffic Guiding
Ubiquitous and Mobile Computing CS 525M: Virtually Unifying Personal Storage for Fast and Pervasive Data Accesses
Ubiquitous and Mobile Computing CS 525M: Virtually Unifying Personal Storage for Fast and Pervasive Data Accesses Pengfei Tang Computer Science Dept. Worcester Polytechnic Institute (WPI) Introduction:
War Stories from the Cloud Going Behind the Web Security Headlines. Emmanuel Mace Security Expert
War Stories from the Cloud Going Behind the Web Security Headlines Emmanuel Mace Security Expert The leading cloud platform for enabling secure, high-performing user experiences on any device, anywhere.
Identifying Workloads for the Cloud
Identifying Workloads for the Cloud 1 This brief is based on a webinar in RightScale s I m in the Cloud Now What? series. Browse our entire library for webinars on cloud computing management. Meet our
Tutorial 8 Build resilient, responsive and scalable web applications with SocketPro
Tutorial 8 Build resilient, responsive and scalable web applications with SocketPro Contents: Introduction SocketPro ways for resilient, responsive and scalable web applications Vertical scalability o
Highly Available Database Architectures in AWS. Santa Clara, California April 23th 25th, 2018 Mike Benshoof, Technical Account Manager, Percona
Highly Available Database Architectures in AWS Santa Clara, California April 23th 25th, 2018 Mike Benshoof, Technical Account Manager, Percona Hello, Percona Live Attendees! What this talk is meant to
Scaling with mongodb
Scaling with mongodb Ross Lawley Python Engineer @ 10gen Web developer since 1999 Passionate about open source Agile methodology email: ross@10gen.com twitter: RossC0 Today's Talk Scaling Understanding
Introduction Storage Processing Monitoring Review. Scaling at Showyou. Operations. September 26, 2011
Scaling at Showyou Operations September 26, 2011 I m Kyle Kingsbury Handle aphyr Code http://github.com/aphyr Email kyle@remixation.com Focus Backend, API, ops What the hell is Showyou? Nontrivial complexity
Choosing a MySQL HA Solution Today. Choosing the best solution among a myriad of options
Choosing a MySQL HA Solution Today Choosing the best solution among a myriad of options Questions...Questions...Questions??? How to zero in on the right solution You can t hit a target if you don t have
CISC 7610 Lecture 5 Distributed multimedia databases. Topics: Scaling up vs out Replication Partitioning CAP Theorem NoSQL NewSQL
CISC 7610 Lecture 5 Distributed multimedia databases Topics: Scaling up vs out Replication Partitioning CAP Theorem NoSQL NewSQL Motivation YouTube receives 400 hours of video per minute That is 200M hours
8/24/2017 Week 1-B Instructor: Sangmi Lee Pallickara
Week 1-B-0 Week 1-B-1 CS535 BIG DATA FAQs Slides are available on the course web Wait list Term project topics PART 0. INTRODUCTION 2. DATA PROCESSING PARADIGMS FOR BIG DATA Sangmi Lee Pallickara Computer
Polishing your cache with Varnish
Polishing your cache with Varnish David Smalley, Co-Founder of Litmus I m David Smalley - co-founder of Litmus Talking about our newest site, Doctype. http://doctype.com Doctype is the newest project from
(Refer Slide Time 00:01:09)
Computer Organization Part I Prof. S. Raman Department of Computer Science & Engineering Indian Institute of Technology Lecture 3 Introduction to System: Hardware In the previous lecture I said that I
Brief introduction of SocketPro continuous SQL-stream sending and processing system (Part 1: SQLite)
Brief introduction of SocketPro continuous SQL-stream sending and processing system (Part 1: SQLite) Introduction Most of client server database systems only support synchronous communication between client
ANALYZING THE MOST COMMON PERFORMANCE AND MEMORY PROBLEMS IN JAVA. 18 October 2017
ANALYZING THE MOST COMMON PERFORMANCE AND MEMORY PROBLEMS IN JAVA 18 October 2017 Who am I? Working in Performance and Reliability Engineering Team at Hotels.com Part of Expedia Inc, handling $72billion
Data Modeling and Databases Ch 14: Data Replication. Gustavo Alonso, Ce Zhang Systems Group Department of Computer Science ETH Zürich
Data Modeling and Databases Ch 14: Data Replication Gustavo Alonso, Ce Zhang Systems Group Department of Computer Science ETH Zürich Database Replication What is database replication The advantages of
Lesson 2: Using the Performance Console
Lesson 2 Lesson 2: Using the Performance Console Using the Performance Console 19-13 Windows XP Professional provides two tools for monitoring resource usage: the System Monitor snap-in and the Performance
Data Center TCP (DCTCP)
Data Center TCP (DCTCP) Mohammad Alizadeh, Albert Greenberg, David A. Maltz, Jitendra Padhye Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, Murari Sridharan Microsoft Research Stanford University 1
In the following chapter two most common WAFS architectures are presented and the most common and simple set of used techniques is shown.
Structure: 1. Motivation a. Preview With appearing and developing of the Internet, spreading over the world for many enterprises became possible. Enterprises have a possibility to open branch offices that
The Gearman Cookbook OSCON Eric Day Senior Software Rackspace
The Gearman Cookbook OSCON 2010 Eric Day http://oddments.org/ Senior Software Engineer @ Rackspace Thanks for being here! OSCON 2010 The Gearman Cookbook 2 Ask questions! Grab a mic for long questions.
EECS 394. Software Development. Communicating. Chris Riesbeck
EECS 394 Software Development Chris Riesbeck Communicating 1 The Problem I never have a clue what the developers are talking about. When I ask what they're doing, they say they're XMLing the CSS SQL RSS
Joomla, 40 Million Page Views, and You. Mitch Pirtle NYPHP, January 2009
Joomla, 40 Million Page Views, and You Mitch Pirtle NYPHP, January 2009 Summary Overview MTV brought Mitch Pirtle in to recommend a platform for a very high-traffic website - he chose an open source platform
TAIL LATENCY AND PERFORMANCE AT SCALE
TAIL LATENCY AND PERFORMANCE AT SCALE George Porter May 21, 2018 ATTRIBUTION These slides are released under an Attribution-NonCommercial-ShareAlike 3.0 Unported (CC BY-NC-SA 3.0) Creative Commons license
SQLite vs. MongoDB for Big Data
SQLite vs. MongoDB for Big Data In my latest tutorial I walked readers through a Python script designed to download tweets by a set of Twitter users and insert them into an SQLite database. In this post