OpenZFS Collaboration

Transcription

1 OpenZFS Collaboration OpenZFS User Conference March 16, 2017 Brian Behlendorf Lawrence Livermore National Laboratory This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA Lawrence Livermore National Security, LLC

2 OpenZFS is Available on Multiple Platforms Developers from all platforms contribute to OpenZFS 2

3 ZFS on Linux Releases Official Release (v0.6.x) Critical bug fixes Linux kernel compatibility Low-risk update for distributions v released Feb 3 rd, 2017 Development Branch (v0.7.x) New features Performance improvements Frequent release candidates v0.7.0-rc3 released Jan 20 th, 2017 ZFS on Linux provides tagged releases for Linux distributions 3

4 Extended Attribute (File Fork) Performance LLNL / Onlight / OSNexus Problem: Extended attributes (xattrs) are used extensively by Linux SELinux security contexts Lustre, Ceph and Gluster file layouts ACLs Posix, Samba Three IOs per xattr 16K DNODE BLOCK 512B DNODE ZAP BONUS SPACE 4

5 Property xattr=sa 16K DNODE BLOCK ZAP 512B DNODE BONUS SPACE SPILL BLOCK Small xattrs may be stored in the system attribute space Two IOs for multiple xattrs Multiple variable-length SA corner cases Tim Chase Xattr=sa/dir David Chen Solution: Store xattrs as a system attribute (SA) 5

6 Feature large_dnodes 16K DNODE BLOCK BONUS SPACE 512B-16K DNODE BONUS SPACE ZAP SPILL BLOCK ZAP SPILL BLOCK Ned Bass One IO for all small xattrs Solution: Store SAs in the dnode by increasing its size 6

7 Vectorization RAIDZ and Checksums FIAS / Atos / Intel Problem: No SIMD optimizations when available (SSE, AVX, NEON) RAIDZ and checksums Highly desirable for JBOD systems Faster parity generation Faster reconstruction of missing data Faster scrub and resilver times Increased reliability Gvozden Neskovic (@ironmann) FAIR - An International Facility for Antiproton and Ion Research 7

8 Vectorization of RAIDZ - Goethe University 8

9 Vectorization RAIDZ Developed flexible infrastructure for SIMD optimizations Micro-benchmark determines fastest method for the system Rigorous correctness tests added to the ZFS Test Suite RAIDZ SIMD implementations: sse2, ssse3, avx2 - Gvozden Neskovic neon, neonx2, avx512f, avx512bw Romain Dolbeau Solution: Detect, benchmark and use fastest available SIMD instructions 9

10 Scalar SSE AVX2 AVX512 NEON P generate P reconstruct PQ generate Q reconstruct PQ reconstruct PQR generate R reconstruct PR reconstruct QR reconstruct PRQ reconstruct RAIDZ - Speed-up relative to original methods 10

11 Vectorization Checksums Jinshan Xiong adapted RAIDZ infrastructure for Fletcher 4 Micro-benchmarks, ZFS Test Suite, etc Fletcher4 checksum SIMD optimizations: avx2 - Jinshan Xiong sse2, ssse3 - Tyler Stachecki superscaler, neon, avx512f - Romain Dolbeau 11

12 ARC Buffer Data (ABD) OSNexus / Delphix / Intel / FIAS / LLNL Problems: Inefficient use of memory Wasted Memory ARC Collapse Irregular performance Out-of-memory events Restricted maximum block size 32-bit system ARC size 12

13 Slab Caches 256K 512K 128K 128K 128K 64K 64K 64K 64K 64K 64K ARC - 1M allocated System - 2M allocated 256K 64K 128K 64K Large buffers for data blocks Slab allocator Reduces allocation cost Fragmentation is complicated Slabs can distort the ARCs internal memory accounting ARC collapse occurs because slabs cannot be free until all buffers are returned 13

14 ARC Buffer Data (ABD) Version 1 Proposed by David Chen - worked great! All buffers are vectors of pages Minimal waste Fast allocations 4K 4K 4K 4K 4K 4K Similar to the page cache Changes were extensive 16M Concerns about maintainability Solution: Construct all buffers using a vector of uniformly sized pages 14

15 ARC Buffer Data (ABD) Version 2 Modified by Dan Kimmel and Matt Ahrens to support Illumos Simplified to only apply to data buffers (not meta data) Integrated with George Wilson s compressed ARC feature Optimized IO pipeline for ABD A version of ABD for all platforms 4K 4K 4K 4K 4K 4K 16M Data Buffer 16K Meta Data Buffer Solution: Construct all data buffers using a vector of uniformly sized pages 15

16 Back to Linux ABD Version 2 applied to Linux - David Quigley Vectorized RAIDZ and checksums - Gvozden Neskovic Linux block layer optimizations to avoid copies Isaac Huang Linux page allocation optimizations David Chen ABD is the result of contributions from many developers 16

17 Take Away OpenZFS is advancing at a rapid pace with many companies and organizations contributing to its development 17

18 Additional Features In-Progress Device Removal/Evacuation - Delphix TRIM/Discard - Nexenta Native Encryption - Datto Declustered Parity (DRAID) Intel Channel Programs Delphix Scrub/Resilver Performance Nexenta Pool Allocation Classes Intel 18

19