Eric Bollengier Director of Engineering Bacula Systems. Bacula Performance & Tuning

Transcription

1 Eric Bollengier Director of Engineering Bacula Systems Bacula Performance & Tuning

2 First Bacula patch in Feb 2003 French Translation BWeb GUI Accurate Mode, RunScript, Batch Insert, Base Jobs, Deadlock detection system... First Deployment Version 1.39 custom, 500 Clients, 2 SD, 1 DIR using PostgreSQL, DRBD and HALinux 2

3 Journey to the Center of your Backup 3

4 Journey to the Center of your Backup From simple disk (40 MB/s 80MB/s), to large array (100MB/s MB/s) Fragmented FS = Sequential I/O Random I/O 4

5 Journey to the Center of your Backup Compression? Encryption? 5

6 Journey to the Center of your Backup 100Mb/s? (35GB/h) 1Gb/s? 10 Gb/s? FD Using Bonding? Dedicated Link? Duplex problem? 6

7 Journey to the Center of your Backup? FD 100 Mb/s? ADSL link? 7

8 Journey to the Center of your Backup? FD SD 8

9 Journey to the Center of your Backup Interface shared between X jobs SD Ex : 1Gb/s, 6 jobs 16MB/s per job. 9

10 Journey to the Center of your Backup SD Packing, Checksum, etc... 10

11 Journey to the Center of your Backup 1Gb/s shared (100MB/s) between X jobs Disks (80MB/s) SD Tape Drive (80MB/s 200MB/s) 11

12 Journey to the Center of your Backup - Special case with Data Spooling (Disk to Tape using large chunks) Multiple Jobs writing at the same time Despooling to tape at constant high speed rate Disk structure should allow to Read AND Write with a good throughput 12

13 Journey to the Center of your Backup Done! 13

14 Journey to the Center of your Backup 14

15 Journey to the Center of your Backup - Test all compoments - Believe only what you see - Test the whole chain Network : netcat, ntop Disks : tar to /dev/null, iostat, bench CPU : top Tape : btape 15

16 Client Side Resolve a bottleneck Not always easy Split the FileSet Change Compression to LZO Network Configure properly Add interfaces, switches Avoid firewalls and routers Storage Daemon Add a new Storage Daemon 16

17 Bacula Servers System Tuning Memory (> 8GB) vm.dirty_ratio = 2 vm.dirty_background_ratio = 1 vm.swappiness = 10 vm.zone_reclaim_node = 0 Disks Use good RAID (ARECA) controler with battery Use deadline disk scheduler Use XFS, stay away from ext3 on Redhat5 17

18 Network Tuning Use network bonding and Jumbo Frames Configure Kernel net.core.wmem_max net.core.rmem_max net.ipv4.tcp_rmem net.ipv4.tcp_wmem - net.ipv4.tcp_sack - net.core.netdev_max_backlog - net.ipv4.tcp_timestamps - net.ipv4.tcp_window_scaling Use a Storage Daemon in your File Daemon LAN/VLAN, avoid Firewall and routers 18

19 Bacula Tuning 1/2 Turn off «AutoPruning» in Client and Job ressources Prune volumes once a day Turn on «Attribute Spooling» for all Jobs Do not try VTL to Tape Job migration 19

20 Bacula Tuning 2/2 Use concurrent jobs, specially when using spooling and tapes Use batch insert and configure your database Run more Incremental, less Full For tapes, «Maximum Block Size» (256 to 512k), «Maximum File Size» (10 to 20GB) 20

21 Catalog Tuning : Why? 21

22 Catalog Tuning : Why Is what you Expect from Bacula? 22

23 Catalog Tuning : Why? Is what you get with default settings... 23

24 Catalog Tuning : Why PostgreSQL work_mem = 1MB wal_buffer = 64k MySQL sort_buffer_size = 2MB innodb_buffer_pool = 128MB File record is about 100 bytes Jobs can despool several GB of attributes Even a small Catalog can contain GB of data 24

25 Catalog Tuning : Why Typical Bacula Job Large, sequential insertion Job Batch Filename Path One record per file File 25

26 Catalog Tuning : PostgreSQL shared_buffers = 1GB # up to 8GB work_mem = 64MB effective_cache_size = 2GB checkpoint_segments = 64 checkpoint_timeout = 20min checkpoint_completion_target = 0.9 maintenance_work_mem = 256MB wal_buffers = 4MB # No longer needed

27 Catalog Tuning : MySQL Good Question! key_buffer_size sort_buffer_size join_buffer_size myisam_sort_buffer_size innodb_log_buffer_size innodb_log_file_size innodb_buffer_pool_size binlog_cache_size thread_cache_size query_cache_size query_cache_limit tmp_table_size max_heap_table_size innodb_additional_mem_p ool_size innodb_rollback_segments... 27

28 Catalog Tuning : MySQL mysqltuner 28

29 Some Catalog Numbers Big Hosting Company 3.3 billion records 1.4 TB of Postgresql data 1200 client defined 4 to 10 weeks of retention 8 SAS disks, no SSD, standard raid controler 29

30 Some Numbers 1/3 Jobs / day 30

31 Some Numbers 2/3 200,000,000 files in one day Files / Day 31

32 Some Numbers 3/3 32

33 Thank you! Eric Bollengier