Lustre at Scale The LLNL Way

Transcription

1 Lustre at Scale The LLNL Way D. Marc Stearman Lustre Administration Lead Livermore uting - LLNL This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA LLNL-PRES

2 Topics Project Structure LLNL uting Platforms Network Design and Topology Software Release Methodology Operation and Management Next Steps Hyperion Concerns 2

3 Project Structure - Who are all these people, and what are they doing here? Project Lead - Mark Gary Administration/Operations Team - Marc Stearman Responsible for daily system administration, cluster integration, upgrades, hardware repair, user application support 4 Full Time Employees + HW Repair Team Software Development - Jim Garlick Responsible for bug fixing, build, QA, and tool development 4 Full Time Employees 3

4 HPC ute/storage Pairing Philosophies Islands of Storage Many HPC clusters, with dedicated attached storage Data access internal to cluster High performance - Uses local high-speed interconnect Many copies of data running on multiple clusters Peninsulas of utation Many HPC clusters using shared network storage User convenience - Fewer copies of data Redundant - if one file system is down, others are still available Extra network latency Interactiv e Network Interactive Network Login Gateway Gateway Login Login Gateway..... High Speed Interconnect High Speed Interconnect Gateway Storage Network 4

5 Peninsulas Explained Login Gateway... High Speed Interconnect Gateway Edge Router Lustre: /p/lscratcha. All storage visible via storage network ute clusters have some Lustre file systems more local than others Lustre Network Core Routers Edge Router Gateway. Gateway.. Login.... High Speed Interconnect Lustre: /p/lscratchb 5

6 Livermore uting - Open uting Facility IGS (test) ALC 9 TFLOPS Atlas 44 TFLOPS Thunder 23 TFLOPS 95TB Zeus 11 TFLOPS Prism 1 TFLOP BGL-Dev 6 TFLOPS Lustre Federated Ethernet Core Network Yana 3 TFLOPS lscratcha lscratchb 1.2PB 12 GB/s Total: ~100 TFLOPS ~2PB 744TB 8 GB/s 6

7 Livermore uting - Open uting Facility IGS (test) ALC 9 TFLOPS Atlas 44 TFLOPS Thunder 23 TFLOPS 95TB Zeus 11 TFLOPS Prism 1 TFLOP BGL-Dev 6 TFLOPS Lustre Federated Ethernet Core Network Yana 3 TFLOPS lscratcha lscratchb 1.2PB Total: ~100 TFLOPS ~2PB 744TB 7

8 LNET view of the world - Open uting Facility tcp0 elan0,elan3 o2ib1 elan4 95TB o2ib0 tcp0 LNET tcp0 tcp0 tcp0 tcp0 tcp0 1.2PB 744TB 8

9 Livermore uting - Secure uting Facility Total: 665 TFLOPS 3.5 PB BGL 596 TFLOPS Minos 33 TFLOPS Rhea 22 TFLOPS Gauss 2 TFLOPS Hopi 3 TFLOPS Lustre Federated Ethernet Core Network Lilac 9 TFLOPS lscratch1 lscratch3 1PB 24 GB/s 2.5PB 26 GB/s 9

10 Livermore uting - Secure uting Facility Total: 665 TFLOPS 3.5 PB BGL 596 TFLOPS Minos 33 TFLOPS Gauss 2 TFLOPS 15GB/s 75GB/s Rhea 22 TFLOPS Hopi 3 TFLOPS 2.5GB/s 45GB/s Lustre Federated Ethernet Core Network 30GB/s Lilac 9 TFLOPS lscratch1 lscratch3 1PB 2.5PB 10

11 LNET view of the world - Secure uting Facility Total: 665 TFLOPS 3.5 PB tcp0 o2ib1 75GB/s o2ib0 tcp0 15GB/s tcp0 2.5GB/s Lustre Federated Ethernet Core Network 30GB/s elan0 45GB/s tcp0 tcp0 1PB 2.5PB 11

12 Livermore uting - Next Steps (LNET) TCP0 Lustre Federated Ethernet Core Network TCP1 TCP2 Edge Routers Gateway. Gateway Login High Speed Interconnect.... Lustre File System Expansion Servers Expansion Servers 12

13 Livermore uting - Next Steps (LNET) Benefits Reduced network traffic through the core switches Keeps traffic local to switch backplanes and reducess latency Saves Money! Issues Need dynamic NIDs Routes can be added dynamically via lctl on cluster with routers, but tcp-only clients on tcp0 need unload of lustre modules to add tcp1 and tcp2 NIDs With multiple lustre filesystems mounted, ALL filesystems must be unmounted so all jobs are killed limiting production work Makes LNET configuration more complex (But we already have that) 13

14 SLIC - Storage Lustre Interface Cluster Lustre FS Lustre Net SLIC Archive Lustre FS Storage Net Archive 14

15 Software Release - Testing Methodology Sanity test in Build Farm Small scale testing in Testbed Mid scale testing on BGL-Dev, and new 64 node cluster Large scale testing on ALC (~400 clients) Giant scale testing on Atlas (1100 clients) during DST (Dedicated System Time) Wide variety of tests - IOR, iozone, fsx, MIB, mdtest, simul, PIOS, various reproducers Developers are using Lustre for their /home file system We carry 150+ patches from base releases to make lustre work on our production systems - This requires a great deal of testing resources 15

16 Software Release - Rollout Methodology Midway through migration from 1.4 to 1.6 Servers before Clients All servers now running chaos ( patches) All x86_64 clients and BGL moving to chaos All i686 and ia64 clients will remain at chaos ( patches) until they retire (about 6-9 months from now) The 1.4 clients can mount the 1.6 file systems because they were created under 1.4, and migrated to 1.6 If we field a new file system we have three options for our legacy clients Create it at 1.4 and migrate to 1.6 Develop a tool that will read the 1.6 on-disk configuration, and write out 1.4 style config Not mount legacy clients 16

17 Operations and Management - Sameness Storage Scalable Units (SSUs) Concept borrowed from ute Clusters Building blocks used to build/expand file systems Makes deployment smooth and quick Hardware Repair team Operations Personnel handle all HW repair actions with minimal SysAdmin intervention Operator training Training courses developed locally Knowledge Base on local wiki Goal to reduce off-hours pages Numerous scripts for testing and problem determination LMT v2 17

18 LMT v2 - Start with xwatch-lustre functionality, then add: New views (OSS, Filesystem, Router Group, ) Plotting capability (historical trends, heart-beat, ) Customization features Full-system health at a glance 18

19 Livermore uting - Next Steps Filesystem Requirements Dawn (.5 PFLOPS): 96 GB/s, ~4PB Sequoia (10 PFLOPS): 512 GB/s, ~50PB Router Cluster Sequoia may have an Infiniband network Need to bridge Infiniband with legacy 10GE Net Evaluating ZFS (Both User Space and In Kernel) Failover 19

20 Hyperion Will replace ALC as large scale lustre test cluster Hyperion is a partnership with the vendor community, including Sun/CFS Will test and evaluate emerging technologies - OFED, Virtualization, Lustre, QDR Infiniband, 40/100 GE, etc 1152 nodes, 9K cores, 120 TFLOPS 1.6 PB >36 GB/s Two Lustre Networks Infiniband 10 Gigabit Ethernet 20

21 Concerns Space Management How do you manage a 50 PB file system? Quotas? Purges? Conventional tools do not scale ls, tar, cp, rsync, etc Moving data to Archive efficiently ZFS Performance: User space vs. Kernel space Migration from ldiskfs to ZFS Scaling concerns Metadata Performance Adaptive Timeouts Multi/Many core parallelization Seemless Failover 21

22 Questions 22