Running MongoDB in Production, Part II
|
|
- Gavin Watts
- 5 years ago
- Views:
Transcription
1 Running MongoDB in Production, Part II Tim Vaillancourt Sr Technical Operations Architect, Percona Speaker Name
2 `whoami` { name: tim, lastname: vaillancourt, employer: percona, techs: [ mongodb, mysql, cassandra, redis, rabbitmq, solr, mesos kafka, couch*, python, golang ] }
3 Agenda Architecture and High-Availability Hardware Tuning MongoDB Tuning Linux
4 Architecture and High-Availability
5 High Availability Replication Asynchronous Write Concerns can provide psuedo-synchronous replication Changelog based, using the Oplog Maximum 50 members Maximum 7 voting members Use vote:0 for members $gt 7 Oplog The oplog.rs capped-collection in local storing changes to data Read by secondary members for replication Written to by local node after apply of operation
6 Architecture Datacenter Recommendations Minimum of 3 x physical servers required for High-Availability Ensure only 1 x member per Replica Set is on a single physical server!!! EC2 / Cloud Recommendations Place Replica Set members in 3 Availability Zones, same region Use a hidden-secondary node for Backup and Disaster Recovery in another region Entire Availability Zones have been lost before!
7 Hardware
8 Hardware: Mainframe vs Commodity Databases: The Past Buy some really amazing, expensive hardware Buy some crazy expensive license Don t run a lot of servers due to above Scale up: Buy even more amazing hardware for monolithic host Hardware came on a truck HA: When it rains, it pours
9 Hardware: Mainframe vs Commodity Databases: A New Era Everything fails, nothing is precious Elastic infrastructures ( The cloud, Mesos, etc) Scale up: add more cheap, commodity servers HA: lots of cheap, commodity servers - still up!
10 Hardware: Block Devices Isolation RAID Level Run Mongod dbpaths on separate volume Optionally, run Mongod journal on separate volume RAID 10 == performance/durability sweet spot RAID 0 == fast and dangerous SSDs Benefit MMAPv1 a lot Benefit WT and RocksDB a bit less Keep about 20-30% free space for internal GC
11 Hardware: Block Devices EBS / NFS / iscsi Risks / Drawbacks Exponentially more things to break (more on this) Block device requests wrapped in TCP is extremely slow You probably already paid for some fast local disks More difficult (sometimes nearly-impossible) to troubleshoot MongoDB doesn t really benefit from remote storage features/flexibility Built-in High-Availability of data via replication MongoDB replication can bootstrap new members Strong write concerns can be specified for critical data
12 Hardware: Block Devices EBS / NFS / iscsi Things to break or troubleshoot Application needs a block from disk System call to kernel for block Kernel frames block request in TCP No logic to align block sizes TCP connection/handshake (if not pooled) TCP packet moves across wire, routers, switches Ethernet is exponentially slower than SATA/SAS/SCSI Storage server parses TCP to block device Storage server system calls to kernel for block Storage server storage driver calls RAID/storage controller Block is returned (finally!)
13 Hardware: CPUs Cores vs Core Speed Lots of cores > faster cores (4 CPU minimum recommended) Thread-per-connection Model CPU Frequency Scaling cpufreq : a daemon for dynamic scaling of the CPU frequency Terrible idea for databases or any predictability! Disable or set governor to 100% frequency always, i.e mode: performance Disable any BIOS-level performance/efficiency tuneable Set ENERGY_PERF_BIAS to performance on CentOS/Red Hat
14 Hardware: Network Infrastructure Datacenter Tiers Network Edge Public Server VLAN Servers with Public NAT and/or port forwards from Network Edge Examples: Proxies, Static Content, etc Calls backends in Backend VLAN Backend Server VLAN Servers with port forwarding from Public Server VLAN (w/source IP ACLs) Optional load balancer for stateless backends Examples: Webserver, Application Server/Worker, etc Calls data stores in Data VLAN
15 Hardware: Network Infrastructure Datacenter Tiers Data VLAN Servers, filers, etc with port forwarding from Backend Server VLAN (w/source IP ACLs) Examples: Databases, Queues, Filers, Caches, HDFS, etc
16 Hardware: Network Infrastructure Network Fabric Try to use 10GBe for low latency Use Jumbo Frames for efficiency Try to keep all MongoDB nodes on the same segment Goal: few or no network hops between nodes Check with traceroute Outbound / Public Access Databases don t need to talk to the internet* Store a copy of your Yum, DockerHub, etc repos locally Deny any access to Public internet or have no route to it Hackers will try to upload a dump of your data out of the network!!
17 Hardware: Why So Quick? MongoDB allows you to scale reads and writes with more nodes Single-instance performance is important, but deal-breaking You are the most expensive resource! Not hardware anymore
18 Tuning MongoDB
19 Tuning MongoDB: MMAPv1 A kernel-level function to map file blocks to memory MMAPv1 syncs data to disk once per 60 seconds (default) If a server with no journal crashes it can lose 1 min of data!!! In memory buffering of Journal Synced every 30ms journal is on a different disk Or every 100ms Or 1/3rd of above if change uses j:true WC
20 Tuning MongoDB: MMAPv1 Fragmentation Can cause serious slowdowns on scans, range queries, etc WiredTiger and RocksDB have little-no fragmentation due to checkpoints / compaction
21 Tuning MongoDB: WiredTiger WT syncs data to disk in a process called Checkpointing : Every 60 seconds or >= 2GB data changes In-memory buffering of Journal Journal buffer size 128kb Synced every 50 ms (as of 3.2) Or every change with Journaled write concern
22 Tuning MongoDB: RocksDB Deprecated in PSDMB 3.6+ Level-based strategy using immutable data level files Built-in Compression Block and Filesystem caches RocksDB uses compaction to apply changes to data files Tiered level compaction Follows same logic as MMAPv1 for journal buffering
23 Tuning MongoDB: Storage Engine Caches WiredTiger In heap 50% available system memory Uncompressed WT pages Filesystem Cache 50% available system memory Compressed pages RocksDB Internal testing planned from Percona in the future 30% in-heap cache recommended by Facebook / Parse Platform
24 Tuning MongoDB: Durability storage.journal.enabled = <true/false> Default since 2.0 on 64-bit builds Always enable unless data is transient Always enable on cluster config servers storage.journal.commitintervalms = <ms> Max time between journal syncs storage.syncperiodsecs = <secs> Max time between data file flushes
25 Tuning MongoDB: Don t Enable! cpu External monitoring is recommended rest Will be deprecated in 3.6+ smallfiles In most situations this is not necessary unless You use MMAPv1, and It is a Development / Test environment You have 100s-1000s of databases with very little data inside (unlikely) Profiling mode 2 Unless troubleshooting an issue / intentional
26 Tuning Linux
27 Tuning Linux: Love your OS! I can login via SSH...we re done! The database is only as fast as as the kernel Expect a default Linux install to be optimised for a cheap laptop, not your $$$ hardware
28 Tuning Linux: The Linux Kernel Avoid Linux earlier than 3.10.x x Large improvements in parallel efficiency in (for Free!) More:
29 Tuning Linux: NUMA A memory architecture that takes into account the locality of memory, caches and CPUs for lower latency But no databases want to use it :( MongoDB codebase is not NUMA aware Unbalanced memory allocations to a single zone
30 Tuning Linux: NUMA Disable NUMA In the Server BIOS Using numactl in init scripts BEFORE mongod command (recommended for future compatibility): numactl --interleave=all /usr/bin/mongod <other flags>
31 Tuning Linux: Transparent HugePages Introduced in RHEL/CentOS 6, Linux Merges memory pages in background (Khugepaged process) AnonHugePages in /proc/meminfo shows usage Disable TransparentHugePages! Add transparent_hugepage=never to kernel command-line (GRUB) Reboot the system Disabling online does not clear previous TH pages Rebooting tests your system will come back up!
32 Tuning Linux: Time Source Replication and Clustering needs consistent clocks mongodb_consistent_backup relies on time sync, for example! Use a consistent time source/server It s ok if everyone is equally wrong Non-Virtualized Run NTP daemon on all MongoDB and Monitoring hosts Enable service so it starts on reboot
33 Tuning Linux: Time Source Virtualised Check if your VM platform has an agent syncing time VMWare and Xen are known to have their own time sync If no time sync provided install NTP daemon
34 Tuning Linux: I/O Scheduler Algorithm kernel uses to commit reads and writes to disk CFQ Completely Fair Queue Perhaps too clever/inefficient for database workloads Probably good for a laptop, assume multi-use Deadline Best general default IMHO Predictable I/O request latencies
35 Tuning Linux: I/O Scheduler Noop Use with virtualised servers Use with real-hardware BBU RAID controllers
36 Tuning Linux: Filesystems Types Use XFS or EXT4, not EXT3 EXT3 has very poor pre-allocation performance Use XFS only on WiredTiger EXT4 data=ordered mode recommended Btrfs not tested, yet! Options Set noatime on MongoDB data volumes in /etc/fstab : Remount the filesystem after an options change, or reboot
37 Tuning Linux: Block Device Readahead Tuning that causes data ahead of a block on disk to be read and then cached Assumption: There is a sequential read pattern Something will benefit from the extra cached blocks Risk Too high waste cache space Increases eviction work MongoDB tends to have very random disk patterns A good start for MongoDB volumes is a 32 (16kb) read-ahead Let MongoDB worry about optimising the pattern
38 Tuning Linux: Block Device Readahead Change ReadAhead Add file to /etc/udev/rules.d /etc/udev/rules.d/60-mongodb-disk.rules: # set deadline scheduler and 32/16kb read-ahead for /dev/sda ACTION=="add change", KERNEL=="sda", ATTR{queue/scheduler}="deadline", ATTR{bdi/read_ahead_kb}="16" Reboot (or use CLI tools to apply)
39 Tuning Linux: Virtual Memory Dirty Pages Dirty Pages Pages stored in-cache, but needs to be written to storage Dirty Ratio Max percent of total memory that can be dirty VM stalls and flushes when this limit is reached Start with 10, default (30) too high
40 Tuning Linux: Virtual Memory Dirty Pages Dirty Background Ratio Separate threshold forbackground dirty page flushing Flushes without pauses Start with 3, default (15) too high
41 Tuning Linux: Swappiness A Linux kernel sysctl setting for preferring RAM or disk for swap Linux default: 60 To avoid disk-based swap: 1 (not zero!) To allow some disk-based swap: 10 0 can cause more swapping than 1 on recent kernels More on this here: elation-vm-swappiness0-new-kernel/
42 Tuning Linux: Ulimit Allows per-linux-user resource constraints Number of User-level Processes Number of Open Files CPU Seconds Scheduling Priority And others
43 Tuning Linux: Ulimit MongoDB Should probably have a dedicated VM, container or server Creates a new process For every new connection to the Database Plus various background tasks / threads Creates an open file for each active data file on disk 64,000 open files and 64,000 max processes is a good start
44 Tuning Linux: Ulimit Setting ulimits /etc/security/limits.d file Systemd Service Init script Ulimits are set by Percona and MongoDB packages! Example on left: PSMDB RPM (Systemd)
45 Tuning Linux: Network Stack Defaults are not good for > 100mbps Ethernet Suggested starting point: Set Network Tunings: Add the above sysctl tunings to /etc/sysctl.conf Run /sbin/sysctl -p as root to set the tunings Run /sbin/sysctl -a to verify the changes
46 Tuning Linux: More on this...
47 Tuning Linux: Tuned Tuned A framework for applying tunings to Linux RedHat/CentOS 7 only for now Debian added tuned, not sure if compatible yet Cannot tune NUMA, file system type or fs mount opts Syctls, THP, I/O sched, etc My apology to the community for writing Tuning Linux for MongoDB :
48 To be continued... Speaker Name May 3rd, 2018
49 Questions? Speaker Name
50 50
MongoDB Backup and Recovery Field Guide. Tim Vaillancourt Sr Technical Operations Architect, Percona
MongoDB Backup and Recovery Field Guide Tim Vaillancourt Sr Technical Operations Architect, Percona `whoami` { name: tim, lastname: vaillancourt, employer: percona, techs: [ mongodb, mysql, cassandra,
More informationMongoDB Backup & Recovery Field Guide
MongoDB Backup & Recovery Field Guide Tim Vaillancourt Percona Speaker Name `whoami` { name: tim, lastname: vaillancourt, employer: percona, techs: [ mongodb, mysql, cassandra, redis, rabbitmq, solr, mesos
More informationMongoDB Security Checklist
MongoDB Security Checklist Tim Vaillancourt Sr Technical Operations Architect, Percona Speaker Name `whoami` { name: tim, lastname: vaillancourt, employer: percona, techs: [ mongodb, mysql, cassandra,
More informationDeploying MongoDB in Production. Monday, November 5, :00 AM - 12:00 PM Bull
Deploying MongoDB in Production Monday, November 5, 2018 9:00 AM - 12:00 PM Bull About us 4 Agenda Hardware and OS configuration MongoDB in Production Backups and Monitoring Q&A 5 Terminology Data Document:
More informationRunning MongoDB in Production, Part I
Running MongoDB in Production, Part I Tim Vaillancourt Sr Technical Operations Architect, Percona Speaker Name `whoami` { name: tim, lastname: vaillancourt, employer: percona, techs: [ mongodb, mysql,
More informationWhy Do Developers Prefer MongoDB?
Why Do Developers Prefer MongoDB? Tim Vaillancourt Software Engineer, Percona Speaker Name `whoami` { name: tim, lastname: vaillancourt, employer: percona, techs: [ mongodb, mysql, cassandra, redis, rabbitmq,
More informationMonitoring MongoDB s Engines in the Wild. Tim Vaillancourt Sr. Technical Operations Architect
Monitoring MongoDB s Engines in the Wild Tim Vaillancourt Sr. Technical Operations Architect About Me Joined Percona in January 2016 Sr Technical Operations Architect for MongoDB Previous: EA DICE (MySQL
More informationMike Kania Truss
Mike Kania Engineer @ Truss http://truss.works/ MongoDB on AWS With Minimal Suffering + Topics Provisioning MongoDB Replica Sets on AWS Choosing storage and a storage engine Backups Monitoring Capacity
More informationScaling MongoDB. Percona Webinar - Wed October 18th 11:00 AM PDT Adamo Tonete MongoDB Senior Service Technical Service Engineer.
caling MongoDB Percona Webinar - Wed October 18th 11:00 AM PDT Adamo Tonete MongoDB enior ervice Technical ervice Engineer 1 Me and the expected audience @adamotonete Intermediate - At least 6+ months
More informationChoosing Hardware and Operating Systems for MySQL. Apr 15, 2009 O'Reilly MySQL Conference and Expo Santa Clara,CA by Peter Zaitsev, Percona Inc
Choosing Hardware and Operating Systems for MySQL Apr 15, 2009 O'Reilly MySQL Conference and Expo Santa Clara,CA by Peter Zaitsev, Percona Inc -2- We will speak about Choosing Hardware Choosing Operating
More informationCaching and reliability
Caching and reliability Block cache Vs. Latency ~10 ns 1~ ms Access unit Byte (word) Sector Capacity Gigabytes Terabytes Price Expensive Cheap Caching disk contents in RAM Hit ratio h : probability of
More informationInnoDB Scalability Limits. Peter Zaitsev, Vadim Tkachenko Percona Inc MySQL Users Conference 2008 April 14-17, 2008
InnoDB Scalability Limits Peter Zaitsev, Vadim Tkachenko Percona Inc MySQL Users Conference 2008 April 14-17, 2008 -2- Who are the Speakers? Founders of Percona Inc MySQL Performance and Scaling consulting
More informationLinux Internals For MySQL DBAs. Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros
Linux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros Linux Kernel It s big (almost 20 million lines of code) It ll take you YEARS to
More informationHow to Scale MongoDB. Apr
How to Scale MongoDB Apr-24-2018 About me Location: Skopje, Republic of Macedonia Education: MSc, Software Engineering Experience: Lead Database Consultant (since 2016) Database Consultant (2012-2016)
More informationInnodb Performance Optimization
Innodb Performance Optimization Most important practices Peter Zaitsev CEO Percona Technical Webinars December 20 th, 2017 1 About this Presentation Innodb Architecture and Performance Optimization 3h
More informationMySQL Performance Optimization and Troubleshooting with PMM. Peter Zaitsev, CEO, Percona
MySQL Performance Optimization and Troubleshooting with PMM Peter Zaitsev, CEO, Percona In the Presentation Practical approach to deal with some of the common MySQL Issues 2 Assumptions You re looking
More informationOperating Systems. File Systems. Thomas Ropars.
1 Operating Systems File Systems Thomas Ropars thomas.ropars@univ-grenoble-alpes.fr 2017 2 References The content of these lectures is inspired by: The lecture notes of Prof. David Mazières. Operating
More informationMongoDB Storage Engine with RocksDB LSM Tree. Denis Protivenskii, Software Engineer, Percona
MongoDB Storage Engine with RocksDB LSM Tree Denis Protivenskii, Software Engineer, Percona Contents - What is MongoRocks? 2 Contents - What is MongoRocks? - RocksDB overview 3 Contents - What is MongoRocks?
More informationAdvanced SUSE Linux Enterprise Server Administration (Course 3038) Chapter 8 Perform a Health Check and Performance Tuning
Advanced SUSE Linux Enterprise Server Administration (Course 3038) Chapter 8 Perform a Health Check and Performance Tuning Objectives Find Performance Bottlenecks Reduce System and Memory Load Optimize
More informationdavidklee.net heraflux.com linkedin.com/in/davidaklee
@kleegeek davidklee.net heraflux.com linkedin.com/in/davidaklee Specialties / Focus Areas / Passions: Performance Tuning & Troubleshooting Virtualization Cloud Enablement Infrastructure Architecture Health
More informationFile System Performance (and Abstractions) Kevin Webb Swarthmore College April 5, 2018
File System Performance (and Abstractions) Kevin Webb Swarthmore College April 5, 2018 Today s Goals Supporting multiple file systems in one name space. Schedulers not just for CPUs, but disks too! Caching
More informationMongoDB and Mysql: Which one is a better fit for me? Room 204-2:20PM-3:10PM
MongoDB and Mysql: Which one is a better fit for me? Room 204-2:20PM-3:10PM About us Adamo Tonete MongoDB Support Engineer Agustín Gallego MySQL Support Engineer Agenda What are MongoDB and MySQL; NoSQL
More informationMongoDB Monitoring and Performance for The Savvy DBA
MongoDB Monitoring and Performance for The Savvy DBA Key metrics to focus on for day-to-day MongoDB operations Bimal Kharel Senior Technical Services Engineer Percona Webinar 2017-05-23 1 What I ll cover
More informationNutanix Tech Note. Virtualizing Microsoft Applications on Web-Scale Infrastructure
Nutanix Tech Note Virtualizing Microsoft Applications on Web-Scale Infrastructure The increase in virtualization of critical applications has brought significant attention to compute and storage infrastructure.
More informationMesosphere and Percona Server for MongoDB. Jeff Sandstrom, Product Manager (Percona) Ravi Yadav, Tech. Partnerships Lead (Mesosphere)
Mesosphere and Percona Server for MongoDB Jeff Sandstrom, Product Manager (Percona) Ravi Yadav, Tech. Partnerships Lead (Mesosphere) Mesosphere DC/OS MICROSERVICES, CONTAINERS, & DEV TOOLS DATA SERVICES,
More informationMySQL Database Scalability
MySQL Database Scalability Nextcloud Conference 2016 TU Berlin Oli Sennhauser Senior MySQL Consultant at FromDual GmbH oli.sennhauser@fromdual.com 1 / 14 About FromDual GmbH Support Consulting remote-dba
More informationDell EMC CIFS-ECS Tool
Dell EMC CIFS-ECS Tool Architecture Overview, Performance and Best Practices March 2018 A Dell EMC Technical Whitepaper Revisions Date May 2016 September 2016 Description Initial release Renaming of tool
More informationCIT 668: System Architecture. Amazon Web Services
CIT 668: System Architecture Amazon Web Services Topics 1. AWS Global Infrastructure 2. Foundation Services 1. Compute 2. Storage 3. Database 4. Network 3. AWS Economics Amazon Services Architecture Regions
More informationExt3/4 file systems. Don Porter CSE 506
Ext3/4 file systems Don Porter CSE 506 Logical Diagram Binary Formats Memory Allocators System Calls Threads User Today s Lecture Kernel RCU File System Networking Sync Memory Management Device Drivers
More informationSystem Requirements ENTERPRISE
System Requirements ENTERPRISE Hardware Prerequisites You must have a single bootstrap node, Mesos master nodes, and Mesos agent nodes. Bootstrap node 1 node with 2 cores, 16 GB RAM, 60 GB HDD. This is
More informationGFS: The Google File System
GFS: The Google File System Brad Karp UCL Computer Science CS GZ03 / M030 24 th October 2014 Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one
More informationRunning MySQL on AWS. Michael Coburn Wednesday, April 15th, 2015
Running MySQL on AWS Michael Coburn Wednesday, April 15th, 2015 Who am I? 2 Senior Architect with Percona 3 years on Friday! Canadian but I now live in Costa Rica I see 3-10 different customer environments
More informationMesosphere and Percona Server for MongoDB. Peter Schwaller, Senior Director Server Eng. (Percona) Taco Scargo, Senior Solution Engineer (Mesosphere)
Mesosphere and Percona Server for MongoDB Peter Schwaller, Senior Director Server Eng. (Percona) Taco Scargo, Senior Solution Engineer (Mesosphere) Mesosphere DC/OS MICROSERVICES, CONTAINERS, & DEV TOOLS
More informationMyRocks deployment at Facebook and Roadmaps. Yoshinori Matsunobu Production Engineer / MySQL Tech Lead, Facebook Feb/2018, #FOSDEM #mysqldevroom
MyRocks deployment at Facebook and Roadmaps Yoshinori Matsunobu Production Engineer / MySQL Tech Lead, Facebook Feb/2018, #FOSDEM #mysqldevroom Agenda MySQL at Facebook MyRocks overview Production Deployment
More informationDatabase Hardware Selection Guidelines
Database Hardware Selection Guidelines BRUCE MOMJIAN Database servers have hardware requirements different from other infrastructure software, specifically unique demands on I/O and memory. This presentation
More informationTopics. File Buffer Cache for Performance. What to Cache? COS 318: Operating Systems. File Performance and Reliability
Topics COS 318: Operating Systems File Performance and Reliability File buffer cache Disk failure and recovery tools Consistent updates Transactions and logging 2 File Buffer Cache for Performance What
More informationInfrastructure Tuning
Infrastructure Tuning For SQL Server Performance SQL PASS Performance Virtual Chapter 2014.07.24 About David Klee @kleegeek davidklee.net gplus.to/kleegeek linked.com/a/davidaklee Specialties / Focus Areas
More informationCourse 55187B Linux System Administration
Course Outline Module 1: System Startup and Shutdown This module explains how to manage startup and shutdown processes in Linux. Understanding the Boot Sequence The Grand Unified Boot Loader GRUB Configuration
More informationMySQL HA Solutions Selecting the best approach to protect access to your data
MySQL HA Solutions Selecting the best approach to protect access to your data Sastry Vedantam sastry.vedantam@oracle.com February 2015 Copyright 2015, Oracle and/or its affiliates. All rights reserved
More information@joerg_schad Nightmares of a Container Orchestration System
@joerg_schad Nightmares of a Container Orchestration System 2017 Mesosphere, Inc. All Rights Reserved. 1 Jörg Schad Distributed Systems Engineer @joerg_schad Jan Repnak Support Engineer/ Solution Architect
More informationDISTRIBUTED FILE SYSTEMS & NFS
DISTRIBUTED FILE SYSTEMS & NFS Dr. Yingwu Zhu File Service Types in Client/Server File service a specification of what the file system offers to clients File server The implementation of a file service
More informationUsing DC/OS for Continuous Delivery
Using DC/OS for Continuous Delivery DevPulseCon 2017 Elizabeth K. Joseph, @pleia2 Mesosphere 1 Elizabeth K. Joseph, Developer Advocate, Mesosphere 15+ years working in open source communities 10+ years
More informationOPERATING SYSTEM. Chapter 12: File System Implementation
OPERATING SYSTEM Chapter 12: File System Implementation Chapter 12: File System Implementation File-System Structure File-System Implementation Directory Implementation Allocation Methods Free-Space Management
More informationMySQL Performance Optimization and Troubleshooting with PMM. Peter Zaitsev, CEO, Percona Percona Technical Webinars 9 May 2018
MySQL Performance Optimization and Troubleshooting with PMM Peter Zaitsev, CEO, Percona Percona Technical Webinars 9 May 2018 Few words about Percona Monitoring and Management (PMM) 100% Free, Open Source
More informationAurora, RDS, or On-Prem, Which is right for you
Aurora, RDS, or On-Prem, Which is right for you Kathy Gibbs Database Specialist TAM Katgibbs@amazon.com Santa Clara, California April 23th 25th, 2018 Agenda RDS Aurora EC2 On-Premise Wrap-up/Recommendation
More informationCS3600 SYSTEMS AND NETWORKS
CS3600 SYSTEMS AND NETWORKS NORTHEASTERN UNIVERSITY Lecture 11: File System Implementation Prof. Alan Mislove (amislove@ccs.neu.edu) File-System Structure File structure Logical storage unit Collection
More informationPebblesDB: Building Key-Value Stores using Fragmented Log Structured Merge Trees
PebblesDB: Building Key-Value Stores using Fragmented Log Structured Merge Trees Pandian Raju 1, Rohan Kadekodi 1, Vijay Chidambaram 1,2, Ittai Abraham 2 1 The University of Texas at Austin 2 VMware Research
More information<Insert Picture Here> Filesystem Features and Performance
Filesystem Features and Performance Chris Mason Filesystems XFS Well established and stable Highly scalable under many workloads Can be slower in metadata intensive workloads Often
More informationAdvanced Continuous Delivery Strategies for Containerized Applications Using DC/OS
Advanced Continuous Delivery Strategies for Containerized Applications Using DC/OS ContainerCon @ Open Source Summit North America 2017 Elizabeth K. Joseph @pleia2 1 Elizabeth K. Joseph, Developer Advocate
More informationA fields' Introduction to SUSE Enterprise Storage TUT91098
A fields' Introduction to SUSE Enterprise Storage TUT91098 Robert Grosschopff Senior Systems Engineer robert.grosschopff@suse.com Martin Weiss Senior Consultant martin.weiss@suse.com Joao Luis Senior Software
More informationEvaluating Cloud Storage Strategies. James Bottomley; CTO, Server Virtualization
Evaluating Cloud Storage Strategies James Bottomley; CTO, Server Virtualization Introduction to Storage Attachments: - Local (Direct cheap) SAS, SATA - Remote (SAN, NAS expensive) FC net Types - Block
More informationApache Cassandra. Tips and tricks for Azure
Apache Cassandra Tips and tricks for Azure Agenda - 6 months in production Introduction to Cassandra Design and Test Getting ready for production The first 6 months 1 Quick introduction to Cassandra Client
More informationIBM V7000 Unified R1.4.2 Asynchronous Replication Performance Reference Guide
V7 Unified Asynchronous Replication Performance Reference Guide IBM V7 Unified R1.4.2 Asynchronous Replication Performance Reference Guide Document Version 1. SONAS / V7 Unified Asynchronous Replication
More informationGFS: The Google File System. Dr. Yingwu Zhu
GFS: The Google File System Dr. Yingwu Zhu Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one big CPU More storage, CPU required than one PC can
More informationPractical MySQL Performance Optimization. Peter Zaitsev, CEO, Percona July 02, 2015 Percona Technical Webinars
Practical MySQL Performance Optimization Peter Zaitsev, CEO, Percona July 02, 2015 Percona Technical Webinars In This Presentation We ll Look at how to approach Performance Optimization Discuss Practical
More informationEffective Use of CSAIL Storage
Effective Use of CSAIL Storage How to get the most out of your computing infrastructure Garrett Wollman, Jonathan Proulx, and Jay Sekora The Infrastructure Group Introduction Outline of this talk 1. Introductions
More informationNoSQL BENCHMARKING AND TUNING. Nachiket Kate Santosh Kangane Ankit Lakhotia Persistent Systems Ltd. Pune, India
NoSQL BENCHMARKING AND TUNING Nachiket Kate Santosh Kangane Ankit Lakhotia Persistent Systems Ltd. Pune, India Today large variety of available NoSQL options has made it difficult for developers to choose
More informationAt course completion. Overview. Audience profile. Course Outline. : 55187B: Linux System Administration. Course Outline :: 55187B::
Module Title Duration : 55187B: Linux System Administration : 4 days Overview This four-day instructor-led course is designed to provide students with the necessary skills and abilities to work as a professional
More informationCS5460: Operating Systems Lecture 20: File System Reliability
CS5460: Operating Systems Lecture 20: File System Reliability File System Optimizations Modern Historic Technique Disk buffer cache Aggregated disk I/O Prefetching Disk head scheduling Disk interleaving
More information"Charting the Course... MOC B: Linux System Administration. Course Summary
Description Course Summary This four-day instructor-led course is designed to provide students with the necessary skills and abilities to work as a professional Linux system administrator. The course covers
More informationò Very reliable, best-of-breed traditional file system design ò Much like the JOS file system you are building now
Ext2 review Very reliable, best-of-breed traditional file system design Ext3/4 file systems Don Porter CSE 506 Much like the JOS file system you are building now Fixed location super blocks A few direct
More informationInstalling and configuring Apache Kafka
3 Installing and configuring Date of Publish: 2018-08-13 http://docs.hortonworks.com Contents Installing Kafka...3 Prerequisites... 3 Installing Kafka Using Ambari... 3... 9 Preparing the Environment...9
More informationMySQL Performance Troubleshooting
MySQL Performance Troubleshooting Best Practices Francisco Bordenave - Architect, Percona Agenda Who am I? Introduction Identifying the source of problem We know where the problem is, now what? Best practices
More information416 Distributed Systems. Distributed File Systems 2 Jan 20, 2016
416 Distributed Systems Distributed File Systems 2 Jan 20, 2016 1 Outline Why Distributed File Systems? Basic mechanisms for building DFSs Using NFS and AFS as examples NFS: network file system AFS: andrew
More informationSoftware Engineering at VMware Dan Scales May 2008
Software Engineering at VMware Dan Scales May 2008 Eng_BC_Mod 1.Product Overview v091806 The Challenge Suppose that you have a very popular software platform: that includes hardware-level and OS code that
More informationThe Care and Feeding of a MySQL Database for Linux Adminstrators. Dave Stokes MySQL Community Manager
The Care and Feeding of a MySQL Database for Linux Adminstrators Dave Stokes MySQL Community Manager David.Stokes@Oracle.com Simple Introduction This is a general introduction to running a MySQL database
More informationBen Walker Data Center Group Intel Corporation
Ben Walker Data Center Group Intel Corporation Notices and Disclaimers Intel technologies features and benefits depend on system configuration and may require enabled hardware, software or service activation.
More informationChoosing Storage for MySQL. Peter Zaitsev CEO, Percona Inc Percona Live, Washington,DC 11 January 2012
Choosing Storage for MySQL Peter Zaitsev CEO, Percona Inc Percona Live, Washington,DC 11 January 2012 Storage for MySQL Storage vs Memory Aspects of Choosing Storage for MySQL Directly Attaches Storage
More informationScaling Without Sharding. Baron Schwartz Percona Inc Surge 2010
Scaling Without Sharding Baron Schwartz Percona Inc Surge 2010 Web Scale!!!! http://www.xtranormal.com/watch/6995033/ A Sharding Thought Experiment 64 shards per proxy [1] 1 TB of data storage per node
More informationChapter 11: Implementing File Systems
Chapter 11: Implementing File Systems Operating System Concepts 99h Edition DM510-14 Chapter 11: Implementing File Systems File-System Structure File-System Implementation Directory Implementation Allocation
More informationPreventing and Resolving MySQL Downtime. Jervin Real, Michael Coburn Percona
Preventing and Resolving MySQL Downtime Jervin Real, Michael Coburn Percona About Us Jervin Real, Technical Services Manager Engineer Engineering Engineers APAC Michael Coburn, Principal Technical Account
More informationUsing MySQL in a Virtualized Environment. Scott Seighman Systems Engineer Sun Microsystems
Using MySQL in a Virtualized Environment Scott Seighman Systems Engineer Sun Microsystems 1 Agenda Virtualization Overview > Why Use Virtualization > Options > Considerations MySQL & Virtualization Best
More informationGLUSTER CAN DO THAT! Architecting and Performance Tuning Efficient Gluster Storage Pools
GLUSTER CAN DO THAT! Architecting and Performance Tuning Efficient Gluster Storage Pools Dustin Black Senior Architect, Software-Defined Storage @dustinlblack 2017-05-02 Ben Turner Principal Quality Engineer
More informationPRESENTATION TITLE GOES HERE
Enterprise Storage PRESENTATION TITLE GOES HERE Leah Schoeb, Member of SNIA Technical Council SNIA EmeraldTM Training SNIA Emerald Power Efficiency Measurement Specification, for use in EPA ENERGY STAR
More informationDisaster Recovery-to-the- Cloud Best Practices
Disaster Recovery-to-the- Cloud Best Practices HOW TO EFFECTIVELY CONFIGURE YOUR OWN SELF-MANAGED RECOVERY PLANS AND THE REPLICATION OF CRITICAL VMWARE VIRTUAL MACHINES FROM ON-PREMISES TO A CLOUD SERVICE
More informationOpen Source Storage. Ric Wheeler Architect & Senior Manager April 30, 2012
Open Source Storage Architect & Senior Manager rwheeler@redhat.com April 30, 2012 1 Linux Based Systems are Everywhere Used as the base for commercial appliances Enterprise class appliances Consumer home
More informationChapter 11: Implementing File
Chapter 11: Implementing File Systems Chapter 11: Implementing File Systems File-System Structure File-System Implementation Directory Implementation Allocation Methods Free-Space Management Efficiency
More informationExam LFCS/Course 55187B Linux System Administration
Exam LFCS/Course 55187B Linux System Administration About this course This four-day instructor-led course is designed to provide students with the necessary skills and abilities to work as a professional
More informationSQL, NoSQL, MongoDB. CSE-291 (Cloud Computing) Fall 2016 Gregory Kesden
SQL, NoSQL, MongoDB CSE-291 (Cloud Computing) Fall 2016 Gregory Kesden SQL Databases Really better called Relational Databases Key construct is the Relation, a.k.a. the table Rows represent records Columns
More informationChapter 11: Implementing File Systems. Operating System Concepts 9 9h Edition
Chapter 11: Implementing File Systems Operating System Concepts 9 9h Edition Silberschatz, Galvin and Gagne 2013 Chapter 11: Implementing File Systems File-System Structure File-System Implementation Directory
More informationRecall: Address Space Map. 13: Memory Management. Let s be reasonable. Processes Address Space. Send it to disk. Freeing up System Memory
Recall: Address Space Map 13: Memory Management Biggest Virtual Address Stack (Space for local variables etc. For each nested procedure call) Sometimes Reserved for OS Stack Pointer Last Modified: 6/21/2004
More informationCONFIGURING SQL SERVER FOR PERFORMANCE LIKE A MICROSOFT CERTIFIED MASTER
CONFIGURING SQL SERVER FOR PERFORMANCE LIKE A MICROSOFT CERTIFIED MASTER TIM CHAPMAN PREMIERE FIELD ENGINEER MICROSOFT THOMAS LAROCK HEAD GEEK SOLARWINDS A LITTLE ABOUT TIM Tim is a Microsoft Dedicated
More informationWhy Choose Percona Server for MongoDB? Tyler Duzan
Why Choose Percona Server for MongoDB? Tyler Duzan Product Manager Who Am I? My name is Tyler Duzan Formerly an operations engineer for more than 12 years focused on security and automation Now a Product
More informationUsing the SDACK Architecture to Build a Big Data Product. Yu-hsin Yeh (Evans Ye) Apache Big Data NA 2016 Vancouver
Using the SDACK Architecture to Build a Big Data Product Yu-hsin Yeh (Evans Ye) Apache Big Data NA 2016 Vancouver Outline A Threat Analytic Big Data product The SDACK Architecture Akka Streams and data
More informationBuilding a Data-Friendly Platform for a Data- Driven Future
Building a Data-Friendly Platform for a Data- Driven Future Benjamin Hindman - @benh 2016 Mesosphere, Inc. All Rights Reserved. INTRO $ whoami BENJAMIN HINDMAN Co-founder and Chief Architect of Mesosphere,
More informationScaling with mongodb
Scaling with mongodb Ross Lawley Python Engineer @ 10gen Web developer since 1999 Passionate about open source Agile methodology email: ross@10gen.com twitter: RossC0 Today's Talk Scaling Understanding
More informationGridGain and Apache Ignite In-Memory Performance with Durability of Disk
GridGain and Apache Ignite In-Memory Performance with Durability of Disk Dmitriy Setrakyan Apache Ignite PMC GridGain Founder & CPO http://ignite.apache.org #apacheignite Agenda What is GridGain and Ignite
More informationCascade Mapping: Optimizing Memory Efficiency for Flash-based Key-value Caching
Cascade Mapping: Optimizing Memory Efficiency for Flash-based Key-value Caching Kefei Wang and Feng Chen Louisiana State University SoCC '18 Carlsbad, CA Key-value Systems in Internet Services Key-value
More informationDesigning Next Generation FS for NVMe and NVMe-oF
Designing Next Generation FS for NVMe and NVMe-oF Liran Zvibel CTO, Co-founder Weka.IO @liranzvibel Santa Clara, CA 1 Designing Next Generation FS for NVMe and NVMe-oF Liran Zvibel CTO, Co-founder Weka.IO
More informationSQL Saturday Jacksonville Aug 12, 2017
Virtualize FCI and AGs What to know before you decide SQL Saturday Jacksonville Aug 12, 2017 Shawn Meyers Principal Architect (@1DizzyGoose) Sponsors About Me Shawn Meyers @1dizzygoose linkedin.com/in/shawnmeyers42
More informationBeyond 1001 Dedicated Data Service Instances
Beyond 1001 Dedicated Data Service Instances Introduction The Challenge Given: Application platform based on Cloud Foundry to serve thousands of apps Application Runtime Many platform users - who don
More informationBlock Device Scheduling. Don Porter CSE 506
Block Device Scheduling Don Porter CSE 506 Logical Diagram Binary Formats Memory Allocators System Calls Threads User Kernel RCU File System Networking Sync Memory Management Device Drivers CPU Scheduler
More informationBlock Device Scheduling
Logical Diagram Block Device Scheduling Don Porter CSE 506 Binary Formats RCU Memory Management File System Memory Allocators System Calls Device Drivers Interrupts Net Networking Threads Sync User Kernel
More informationSolidFire and Ceph Architectural Comparison
The All-Flash Array Built for the Next Generation Data Center SolidFire and Ceph Architectural Comparison July 2014 Overview When comparing the architecture for Ceph and SolidFire, it is clear that both
More informationVirtualization with VMware ESX and VirtualCenter SMB to Enterprise
Virtualization with VMware ESX and VirtualCenter SMB to Enterprise This class is an intense, five-day introduction to virtualization using VMware s immensely popular Virtual Infrastructure suite including
More informationLies, Damn Lies and Benchmarks: How to Accurately Measure Distributed Application Performance. Heinz Schaffner
Lies, Damn Lies and Benchmarks: How to Accurately Measure Distributed Application Performance Heinz Schaffner Science Projects vs. Production Testing to Destruction vs. Distressed Processing Latency Schemes
More informationNPTEL Course Jan K. Gopinath Indian Institute of Science
Storage Systems NPTEL Course Jan 2012 (Lecture 39) K. Gopinath Indian Institute of Science Google File System Non-Posix scalable distr file system for large distr dataintensive applications performance,
More informationCost-Effective Virtual Petabytes Storage Pools using MARS. FrOSCon 2017 Presentation by Thomas Schöbel-Theuer
Cost-Effective Virtual Petabytes Storage Pools using MARS FrOSCon 2017 Presentation by Thomas Schöbel-Theuer 1 Virtual Petabytes Storage Pools: Agenda Scaling Properties of Storage Architectures Reliability
More informationLeveraging Traditional Technologies in Non-Traditional Ways
Leveraging Traditional Technologies in Non-Traditional Ways Ben Rockwood Director of Systems Joyent, Inc. SNIA Winter Symposium 2009 Cloud Hype Cloud is marketing hype (and everyone knows it)... but so
More informationXen and CloudStack. Ewan Mellor. Director, Engineering, Open-source Cloud Platforms Citrix Systems
Xen and CloudStack Ewan Mellor Director, Engineering, Open-source Cloud Platforms Citrix Systems Agenda What is CloudStack? Move to the Apache Foundation CloudStack architecture on Xen The future for CloudStack
More information