Linux Internals For MySQL DBAs. Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

Transcription

1 Linux Internals For MySQL DBAs Ryan Lowe Marcos Albe Chris Giard Daniel Nichter Syam Purnam Emily Slocombe Le Peter Boros

2 Linux Kernel It s big (almost 20 million lines of code) It ll take you YEARS to be an expert Resources: linuxfromscratch.org kernel.org man pages lwn.net

3

4 What do we care about? Networking Storage Memory Processing

5 What do we care about? Storage Memory Processing Networking

6 /proc

7 /proc Processes and other system information in a hierarchical file-like structure Interaction between kernel space and user space Exposes kernel knobs and sliders Plain text files echo 'value' > /proc/you/want/adjusted

8 /proc cpuinfo meminfo interrupts diskstats fs/ net/ sys/

9 ulimits Considered part of security system For performance/operations we only care about open files (-n) For debugging might need to set core dump size limit to Unlimited (-c)

10 Storage

11 Virtual File System (VFS) Abstraction layer to allow Linux to handle many filesystems. Provides a common interface to make your life easier. Introduces the Common File Model

12 Common File Model Superblock Inode File Dentry

13 Filesystems Btrfs ext2/3/4 ReiserFS XFS ZFS

14 ext2/3/4 ext2: Old; No Journaling (SSD/Flash, maybe) ext3: ext2 + journaling + HTree Indexing ext4 Large Volumes Extents Checksummed Journal Nanosecond Timestamps

15 XFS Standard recommendation for DB workloads Highly proficient with parallel IO Current (started 1998, but development is active)

16 AIO A method for performing IO operations so that the process that issued an IO request is not blocked till the data is available. Instead, after an IO request is submitted, the process continues to execute its code and can later check the status of the submitted request.

17 Mount Options no[dir]atime nobarrier discard! %> mount -o remount,rw,new,options,here

18 IO Schedulers noop FIFO Queue w/request Merging deadline cfq Impose a deadline on all operations Completely Fair Queueing anticipatory Anticipates synchronous read operations

19 Storage: Disks Rotational Latency: The delay waiting for the rotation of the disk to bring the required disk sector under the read-write head. Seek time: Time to move the Read/Write Head from current position to the desired track location Access time (Response time): How fast we can locate a position of a file Transfer time (Throughput): How fast we can get bytes from disk to RAM

20 IOPS 1 / (average latency in ms + average seek time in ms)

21 IOPS Example Model: Western Digital VelociRaptor 2.5" SATA hard drive Rotational speed: 10,000 RPM Average latency: 3 ms (0.003 seconds) Average seek time: 4.2 (r)/4.7 (w) = 4.45 ms ( seconds) Calculated IOPS for this disk: 1/( ) = about 133 IOPS

22 RAID Hardware RAID w/bbu (or NVRAM) RAID 0, 1, 5, 6, 10, 50, 60, etc

23 RAID Alignment

24 RAID Alignment

25 InnoDB Flush Method fdatasync O_DSYNC O_DIRECT O_DIRECT_NO_FSYNC

26 Memory

27 Memory: Swap

28

29 InnoDB Buffer Pool Size Allocated Dynamically unless you use innodb_buffer_pool_populate InnoDB checks on boot if enough RAM is available 10% overhead

30 Huge Pages Translation Lookaside Buffer (TLB) Default 4k page size Larger pages = Smaller TLB Huge Pages: 2MB - commodity HW 1GB - High End (1TB+) Huge Pages are good for huge Buffer Pool

31 NUMA Each physical die is a NUMA node introspection: numactl --hardware / numactl --show O(log(N)) O(1) for local O(log(N)) for remote

32 NUMA Disable MySQL NUMA affinity / pinning except: When you run multiple mysqld instances on the same system AND you have data for each instance separated on different PCIe cards AND the PCIe cards are local to different CPU sockets Percona implements Jeremy Cole s suggestions from

33 malloc

34 Processing

35 CPU Time

36 Interrupts

37 Context Switching

38 CPU Governors Control CPU speed and power consumption # cat /sys/devices/system/cpu/cpu0/cpufreq/ scaling_available_governors control through /proc (or /etc/init.d/cpuspeed or cpupower) More: Most distros use on-demand by default (/etc/defaults/cpupower) We care about on-demand, performance, powersave. We usually want performance;

39 Networking

40 Backlog Queue for new TCP connections MySQL: back_log Linux: tcp_max_syn_backlog sysctl -w net.ipv4.tcp_max_syn_backlog = 4096 sysctl -w net.core.somaxconn = 1024

41 Slow Links /proc/net/core/wmem_max = /proc/net/core/rmem_max = /proc/net/ipv4/tcp_wmem = /proc/net/ipv4/tcp_rmem = /proc/net/ipv4/tcp_mem = /proc/net/ipv4/tcp_window_scaling = 1 /proc/net/ipv4/tcp_sack = 1 /proc/net/ipv4/tcp_timestamps = 1 /proc/net/ipv4/tcp_no_metrics_save = 0 /proc/net/ipv4/tcp_moderate_rcvbuf = 1

42 High QPS /proc/net/ipv4/ip_local_port_range = /proc/net/ipv4/tcp_max_tw_buckets = /proc/net/ipv4/tcp_tw_reuse = 1 /proc/net/ipv4/tcp_syncookies = 1 /proc/net/core/wmem_default = /proc/net/core/rmem_default = /proc/net/ipv4/tcp_wmem = /proc/net/ipv4/tcp_rmem = /proc/net/ipv4/tcp_mem = /proc/net/core/rmem_max = /proc/net/core/wmem_max =

43 Questions