5.4 - DAOS Demonstration and Benchmark Report

Transcription

1 5.4 - DAOS Demonstration and Benchmark Report Johann LOMBARDI on behalf of the DAOS team September 25 th, 2013 Livermore (CA) NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL UNDER ITS SUBCONTRACT WITH LAWRENCE LIVERMORE NATIONAL SECURITY, LLC WHO IS THE OPERATOR AND MANAGER OF LAWRENCE LIVERMORE NATIONAL LABORATORY UNDER CONTRACT NO. DE-AC52-07NA27344 WITH THE U.S. DEPARTMENT OF ENERGY. THE UNITED STATES GOVERNMENT RETAINS AND THE PUBLISHER, BY ACCEPTING THE ARTICLE OF PUBLICATION, ACKNOWLEDGES THAT THE UNITED STATES GOVERNMENT RETAINS A NON-EXCLUSIVE, PAID-UP, IRREVOCABLE, WORLD-WIDE LICENSE TO PUBLISH OR REPRODUCE THE PUBLISHED FORM OF THIS MANUSCRIPT, OR ALLOW OTHERS TO DO SO, FOR UNITED STATES GOVERNMENT PURPOSES. THE VIEWS AND OPINIONS OF AUTHORS EXPRESSED HEREIN DO NOT NECESSARILY REFLECT THOSE OF THE UNITED STATES GOVERNMENT OR LAWRENCE LIVERMORE NATIONAL SECURITY, LLC.

2 Agenda 1. Functional Demonstrations 2. Performance & Scalability 2

3 Functional Demonstrations DAOS Demonstration and Benchmark Report 3

4 Testing Infrastructure Reuse existing Lustre* test infrastructure Patch pushed to gerrit, built by jenkins, tested by autotest and results available in maloo Usual test suites (sanity, conf-sanity, sanityn, ) are run New sanity-daos test suite Invoke binary tests (e.g. event and event queue testing) Execute tests through a tool (daosop) using the DAOS API 52 regression tests New tests added each time we land a major patch Intel High Performance Data Division * Other names and brands may be claimed as the property of others. 4

5 Test Distribution Component Number of Tests Test range Event & EQ 4 1 5b System Container Container 7 20a - 24 Container + Shard Objects Collective Open Epoch Protocol

6 Testing Configurations Autotest configuration 2 clients, 2 OSSs & 1 MDT in VMs results available here: sanity-daos to be run live during this presentation in a virtual machine on this laptop 6

7 Event & Event Queue (EQ) Criteria Associated Test # Maximum number of EQs test 1 Maximum number of child events test 2 Finalize in-progress event test 2 Parent event test 2 Maximum number of events per EQ test 3 Non-blocking operations / EQ Poll & Query Benchmark #events handled with a single core Benchmark #events handled with multiple cores test 4a test 5a test 5b 7

8 System Container Criteria Associated Test # System container open/close test 10 System container query test 11 System container query target test 12 8

9 Container & Shard Criteria Associated Test # Container open/create/unlink/query tests 20[a,b] Container POSIX stat/unlink test 21 No POSIX open on container test 22 Exclusive open for write test 24 Shard addition & list tests 25/26/28/29 Shard Query test 30 9

10 Object I/O Criteria Associated Test # Object open/close + mode check tests 40[a,b] 44 Object direct read/write Object cached write + flush Read on non-existent object Object punch tests 45[a-h] tests 45[i-l] tests 45[c,d] tests 46[a,b] Shard object index listing test 47 Object created on write test 47 10

11 Collective Open Criteria Associated Test # Collective open in same mountpoint test 60 Collective open in different mountpoint test 61 Layout refresh through loca2global2local test 62 11

12 Epoch Protocol Criteria Associated Test # Epoch reference on open test 70 Epoch slip test 71 Epoch commit test 72 Epoch wait test 73 12

13 Performance & Scalability DAOS Demonstration and Benchmark Report 13

14 Performance Testbed - lola Intel s FF internal cluster called lola 5 OSS nodes connected to JBODs 1 OST per OSS (5 OSTs in all) each OST has a dedicated RAIDZ2 pool composed of 10 JBOD disks 32 GB of memory 1 MDS connected to a JBOD 1 MDT in a dedicated zpool of 1 JBOD disk 32 GB of memory 20 client nodes Infiniband network 14

15 Scalability Testbed - hyperion 8 OSSs connected to NetApp E-5460 arrays 1 OST per OSS (8 OSTs in all) Each OST has a dedicated zpool composed of 1 RAID6 LUN across 10 disks 1 MDS connected to NetApp E-5460 array 1 MDT on a dedicated zpool composed of 1 RAID10 LUN across 4 disks 120 client nodes Infiniband network 15

16 Collective Container Open/Close (1/3) DAOSBENCH in container open/close mode Process leader opens the container and generates global handle Global handle broadcasted to all tasks Slave tasks call local to global Measure how many such operations can be completed per second Each container has one shard per target ssfopen used to compare with POSIX similar to daosbench except that each client node performs a POSIX open(2) 16

17 Collective Open/Close lola (2/3) Collective open rate (per sec) DAOS POSIX number of client nodes 17

18 Collective Open/Close hyperion (3/3) Collective open rate (per sec) DAOS POSIX Number of client nodes 18

19 Shard Addition (1/2) DAOSBENCH in shard-add mode Process leader creates a container and broadcasts handle Each task creates multiple shards in the same container --shards=n option means that a container gets N shards in total Those N shards are mapped to processes as even as possible. e.g. N = 7 ntasks = 3, rank #0 adds shards 0, 3 & 6, rank #1 adds 1 & 4 and rank #2 adds 2 & 5. Process leader refreshes layout information and broadcasts updated global handle Slave processes re-execute global2local to update local layout 19

20 Shard Addition - hyperion (2/2) Shard create rate (per sec) DAOS Shard Creation Number of client nodes 20

21 Small I/O: 4K Object Creation (1/3) New DAOSBENCH benchmark, object-create mode Create a container with a shard on each OST Each task is assigned a shard and writes 4KB into multiple objects in the shard Standard mdsrate used to compare with POSIX Use the write=4096 option to create non-empty files Each task creates files in its own directory 21

22 Small I/O - lola (2/3) KB object create rate (per sec) DAOS (DAOSBENCH) POSIX (mdsrate) Number of client nodes 22

23 Small I/O - hyperion (3/3) KB object create rate (per sec) DAOS POSIX Number of client nodes 23

24 Streaming I/O (1/5) IOR benchmark modified to use the DAOS API Process leader opens containers, creates one shard per task and broadcasts global handle Each task is assigned a shard and submits async I/Os to an object in that shard No application-level striping/mirroring support yet (in our TODO list) DAOS runs includes: direct writes & reads buffered writes For comparison, POSIX runs include: file per process buffered reads & writes direct I/O not benchmarked because too slow could likely be improved by using ZIL 24

25 Streaming I/O - single client - lola (2/5) Throughput (MiB/s) DAOS Direct Write DAOS Read DAOS Buffered Write Maximum number of I/Os in flight 25

26 Streaming I/O - mutli clients - lola (3/5) Throughput (MiB/s) DAOS Direct Write DAOS Direct Read POSIX FPP Write POSIX FPP Read DAOS Buffered Write Number of client nodes 26

27 Streaming I/O single client - hyperion (4/5) Throughput (MiB/s) DAOS Direct Write DAOS Direct Read Maximum number of I/Os in flight 27

28 Streaming I/O multi client - hyperion (5/5) Throughput (MiB/s) DAOS Direct Write DAOS Direct Read POSIX FPP Write POSIX FPP Read Number of client nodes 28

29 Conclusions DAOS prototype turns out to be fairly stable when tested at scale DAOS model seems to scale pretty well Still room for improvements Speed up shard creation rate on a single OST A ZFS dataset creation involves a txg sync ZFS Read performance drop not yet understood e.g. reduce amount of seeks by increasing block size to 1MB Limit impact of asynchronous dataset removal Disabling ZFS feature@async_destroy seems to help Add striping support to IOR/DAOS and test with unaligned I/Os Groundwork in place for epoch recovery implementation 29

30 Fast Forward Project DAOS