Lustre Update. Brent Gorda. CEO Whamcloud, Inc. Linux Foundation Collaboration Summit San Francisco, CA April 4, 2012

Transcription

1

2 Linux Foundation Collaboration Summit San Francisco, CA April 4, 2012 Lustre Update Brent Gorda CEO Whamcloud, Inc.

3 Agenda Lustre History / Whamcloud introduction Lustre status Community Roadmap Current development at Whamcloud Looking forward Growing the Lustre market What to expect in 2012 Exascale thoughts 3

4 What is Lustre? Lustre (file system) From Wikipedia, the free encyclopedia Lustre is a massively parallel distributed file system, generally used for large scale cluster computing Available under the GNU GPL, the project provides a high performance file system for clusters of tens of thousands of nodes with petabytes of storage capacity.

5 A Lustre Example

6 Lustre dominant in HPC of top100 (and growing) 8 of top 10 6 Lustre China Workshop November 2011

7 Lustre Major Releases 1999 Lustre project startup 2001 ASCI Pathforward 2003 V1.0 CFS 2007 V1.6 Sun 2009 V1.8 Sun 2010 V2.0 Oracle 2011 V2.1 Whamcloud (9/30) 2012 V2.2 Whamcloud (3/30) 7 Lustre China Workshop November 2011

8 Whamcloud introduction VC-backed California Corp, formed July 2010 Responsible for ~90% of Lustre Development Hundreds of supported sites worldwide Whamcloud offerings: Worldwide L3 Lustre support Lustre development Training Chroma - management Whamcloud is widely recognized as the source for Lustre We have the only HW vendor-neutral offering 8

9 Whamcloud management team CEO Brent Gorda (> 25 yrs HPC) DOE program manager for Lustre, BlueGene, TLCC Entrepreneur: Myrias Research, Bonsai Software, MetaExchange CTO Eric Barton (> 25 yrs in HPC) LNet developer since 2002, Lustre architect since 2008 Entrepreneur: Meiko Scientific founder, Quadrics & consulting Daniel Ferber Cray/SGI/Sun/Oracle, bizdev Peter Jones Lustre support manager Bryon Neitzel Lustre development manager Jessica Popp Lustre project management Robert Read Lustre 2.0 lead engineer, Chroma 9

10 Lustre community Whamcloud community membership Whamcloud maintains the community assets Wiki + roadmap: All Lustre releases: Jira bug tracker: Git repositories: Gerrit code review: Build: No copyright assignment on source contributions Ensures no single entity can own whole copyright on Lustre Has support of OpenSFS and EOFS 10

11 Lustre releases Single community-wide source tree Hosted at Whamcloud Formally recognized by community Release trains Releases on a regular schedule Features not ready to land don t delay release Scheduled for next release instead Feature releases Major Lustre releases containing new features High level of focus on quality but production exposure may be limited Maintenance releases Proven reliability for production environments Bugfixes only in maintenance updates 11

12 Lustre Development under the hood Lustre development Majority development at Whamcloud Same engineers Long range reliable roadmap Lustre tree maintenance Stability prioritized Lustre test automation & QA Testing at scale Test results database Lustre support Direct access to developers Experienced Lustre engineers worldwide Minimum time to incident resolution Lustre 2.2 patch landings Lustre 2.2 bugs reported 12

13 On Lustre quality "The Spider parallel file systems at Oak Ridge National Laboratory, three of the largest parallel file systems in the world, powered by Data Direct Networks, Dell, Mellanox, and Lustre supported by Whamcloud, have provided unprecedented reliability in supporting the I/O needs of all major compute systems at our Leadership Computing Facility. On our largest file system we have achieved 100% availability in 8 of the 12 months of 2011 with scheduled availability of 99.26% over the entire year. The stability of this system is unprecedented given its scale. -- Galen M. Shipman, Technology Integration Group Leader, National Center for Computational Sciences (NCCS) at ORNL 13

14 Quality achieved by planning/ testing 14

15 Agile as a Quality Development Process Using GreenHopper plugin from Jira Focus is on taking large chunks of development work and splitting them into small definable groups of work Demonstrable items for every Sprint Design doc, test plan, prototype code, testable code Something that moves the bar forward in a measurable way Allows us to see if we veer off track before it s too late No longer bugging engineers for (useless) statuses Sprint Retrospectives: re-visit, analyze, re-plan if necessary We don t just plow forward, or try to make up time 15

16 Community Lustre Roadmap 16

17 The Lustre Ecosystem whamcloud Lustre Storage Compute Integration Development 17

18 Lustre development at Whamcloud Imperative recovery Reduced server recovery latency Dirop SMP scaling Improve MDS throughput Server stack SMP scaling Improve SMP node affinity for LNET and ptlrpc Statahead performance Fast ls l, du etc Wide striping Allow files to stripe over ~1500 OSTs OSD restructuring Single backend storage abstraction (ext4, ZFS, btrfs?, others?) Online check/scrub Filesystem consistency checking and repair Distributed Namespace Horizontal scale for metadata 18

19 Lustre 2.3 and Beyond Lustre 2.3 (Q3 2012) Server Stack SMP Scaling Online check/scrub Lustre 2.4 (Q1 2013) OSD Restructuring Distributed Namespace HSM A great deal of other development is underway but not tied to a specific release yet Roadmap refreshes now happening regularly 19

20 Chroma Manager Administration Provisioning Maintenance HA Setup Fault diagnosis Management information Performance Utilization Alerts Intuitive interfaces GUI (single pane of glass) Scriptable CLI (automation) System integration Multi-vendor storage management Multi-vendor cluster/site management Partners can build their own appliance 20

21 Chroma Manager interfaces Management console Simplify provisioning, monitoring and maintenance of Lustre storage Monitor and analyze filesystem performance Deep integration with storage vendor partner products Appliance toolkit Scriptable user interface Command line Web service REST API for Lustre management Architecture for using standard web protocols Separate front-end and server logic Used internally by GUI and CLI Deep integration with 3rd party site/cluster management products 21

22 Exascale filesystem /projects /Legacy /HPC /BigData Conventional namespace Works at human scale Administration, security, accounting Supports legacy data and applications DAOS Container Works at exascale Embedded in conventional namespace Provides storage for application domain specific object schemas (data + metadata) Posix striped file a b c a b c a b c a Simulation data OODB OODB OODB OODB OODB metadata data data data data data data data data data data data data data data data MapReduce data Blocksequence data data data data data data data data data 22

23 Exascale challenges Application data + metadata Model 1 OODB OODB OODB OODB metadata /project1 Model 2 /user /project2 App metadata Explosive growth OODB OODB data OODB Large, sophisticated models data data data OODB data data data App metadata data data data data data Model 3 data data data Uncertainty Qualification data data data OODB data data data OODB data data data OODB metadata Billions trillions of Leaf data objects data data data data data data App metadata Complex analysis data data data Filesystem namespace pollution Keep filesystem namespace for storage management / administration Separate namespace for application data + metadata Distributed Application Object Storage (DAOS) containers Preserve model integrity in the face of all possible failures Very large atomic, durable transactions Integrity APIs at all levels of the I/O stack Search / query / analysis Non-resident index maintenance & traversal / non-sequential data traversal Move query processing to global storage Same programming model as apps? OODB metadata data data data data data data data data data data data data 23

24 Main Points Lustre is stable today Technically: v1.8.x is rock solid, 2.x adds features Politically: the community has stepped up Lustre Ecosystem is growing EOFS + OpenSFS + Whamcloud Single tree from which to pull Active dev community: Whamcloud + others More storage vendors shipping Lustre today 24

25 "The maturation of the Lustre community, anchored by OpenSFS and Whamcloud, was the most significant trend in HPC in Addison Snell, CEO of Intersect360 Research 25

26 Brent Gorda CEO Whamcloud, Inc.