NEXPReS WP8 Progress Meeting

Transcription

1 NEXPReS WP8 Progress Meeting Preparations for Multi Site Testing Ari Mujunen Aalto University Metsähovi Radio Observatory VLBI2SKA Workshop in Aveiro, Portugal, Research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/ ) under grant agreement n RI This presentation reflects only the author's views. The European Union is not liable for any use that may be made of the information contained therein.

2 Partner Focus Areas AALTO Coordination, basic technologies ASTRON Long-term archival & reprocessing INAF Global/local allocation/deallocation schemes JIVE Augmenting correlation capabilities /w buffering OSO Trial-site performance & applicability testing PSNC Role & trials of computing center buffering UMAN Trial-site performance & applicability testing 2 (44)

3 WP8 Objective Determine the best practical mix of solutions What kind of storage Where located & packaged Connected in which ways How storage is allocated/deallocated and accessed Which will serve the needs of evolving (>1Gbps) VLBI data acquisition and processing 3 (44)

4 WP8 Schedule 4 (44)

5 D8.1 Interface design document of storage element API Delivered in December 2010 Main results: One buffer one computer, many disks Control API through SSH Simple SSH-protected CLI in/out, potentially opening up UDP/TCP ports in public 10GE Simple (UDP-based/Tsunami-like) transfer protocol Separation of data and metadata Though no sw cleanup of native data formats of data sources (Mk4 timecode, VDIF headers, UDP packet numbers,...) Data chunks in memory and on individual disks 5 (44)

6 D8.2: Hardware design document for simultaneous I/O storage elements For the highest capacity and bandwidth at the lowest cost, 3.5 hard disks are still (for a while) the best match To fill one 10GE port we need disks We presented two alternative designs 6 (44)

7 Current Issues MS801: Hardware decision for subsequent prototyping and integration tests D8.2 alternative solution with server motherboard and CPUs selected D8.3: Design document of storage element allocation methods Due ~now... Should describe how to manage buffer space across stations Could describe how SSH keys (or WP5 sshbroker?) are used to allow cross-access 7 (44)

8 Multi Station Tests are Scheduled for Summer 2011 Striving to deliver 21-Sep-2011 as scheduled Although it is a milestone, not a deliverable... Mh, On, Jb JIVE D8.4: Design document of transparent local/remote application programming interface for storage elements MS802: Review of prototype test results, decision of the architecture and scope of subsequent integration tests 8 (44)

9 Platform Selection 9 (44)

10 Platform Selection in D8.2 Two alternatives: Very basic COTS chassis with customization COTS chassis (more expensive) 10 (44)

11 Lowest-Cost Solution by Customized Chassis Suggested a basic 4U rack enclosure Chieftec UNC-410F-B 11 (44)

12 Customized Chassis Was a Great Idea, but... Wiring needs more work than predicted 12 (44)

13 Adding ~+30% to Cost: 2*Intel Xeon QP More disks (36) 2*1400W Uses 420W Neat, /w 3yr warranty 13 (44)

14 Final Recommendation as MS801 We ended up using D8.2 alternative solution and we also chose a (potentially?) more powerful motherboard/cpu from the same vendor: This fulfills the recommendations in D8.2 of COTS implementation of 8Gbps capable 36 disk storage unit in a fairly standard 4U rack enclosure 72TB or 20 hours of 8Gbps Add a heavy-duty 2x4-core MB with 10GE and disk controllers With room for expansion with external disk controllers, cf. the recent Mark6 proposal 14 (44)

15 Parts List 15 (44)

16 36 Disks + 7 (low-profile) PCIe x8 slots Still only 4U 16 (44)

17 D8.2 Recommendation OK, too Though now recommending the SC disk case... Asus Crosshair IV Formula (+846TQ) 3 PCIe x8 16+6=22 disks 10GE No fans Extreme (+846A) 4 PCI x8 24(+6) disks 10GE Fans, Hydralogix 17 (44)

18 Pictures from Mh 18 (44)

19 Linux Environment, Software Algorithms 19 (44)

20 Linux Environment 64-bit Ubuntu/Debian server booted from network recommended 64-bit: 16GB (and up; >4GB) needed for latency buffers Ubuntu/Debian: familiarity within our community Server: avoiding extra services (X, net) in storage nodes Netboot: central management of (potentially) multiple storage nodes; maximize the number of data disks Netboot via the motherboard 1GE interfaces Local.vlbi VLAN for that while 10GE connected to Internet/LP 20 (44)

21 Netboot Server TFTP+NFS small server 21 (44)

22 Lessons Learned from EXPReS 6Gbps/20-disk RAID0 Total throughput bottlenecked in a single kernel thread flushing all data from block buffer cache to the single raid0 Tried avoiding the buffer cache with O_DIRECT, got abysmal performance in EXPReS trial tests Did not understand the importance of O_DIRECT writing large blocks in a single write() (2 64MB or more) Substituted two raid0s; still all CPU being consumed in copying data from user programs to buffer cache Giving each disk spindle its own process/thread doing O_DIRECT write()s with large enough block size seems fine With 22 disks attained >17Gbps using only 20% of CPUs 22 (44)

23 RAID Arrays? For high bandwidth and high capacity at the same time, one cannot really use other RAIDs but raid0 (striping) Raid5/Raid6 slow down in streaming writing, especially in degraded mode (with failed disks) False illusion of guard against disk failures Cannot afford losing half the disk space with raid10 But raid0 is vulnerable to single-disk failures Why use any RAIDs at all for astronomy data? Why use any form of redundancy for volatile, non-archival data? 23 (44)

24 Writing to a set of disks Divide incoming data stream into consecutive largish (50 500MB) chunks of memory buffer Write chunks into files on a set of stand-alone disks Local or even remote disks Disk handler processes/threads 24 (44)

25 Naming Files on Disks Prefer naming conventions which ls-list alphabetically in time/sequence order Determining file name list in correct order becomes as easy as 'ls /*/<experiment>/*'... Detecting missing files or holes For example: /d001/n11k2mh/ m5a /d001/n11k2mh/ m5a 25 (44)

26 Managing Local Disk Space Initially thought writing chunks in round-robin fashion The current idea is to select among idle disks the one with the largest amount of free space Round-robin (or longest idle time?) for disks with the same (largest) amount of free space Balances disk consumption automatically, allows different disk sizes & usage levels Also allows subsets of disks Tracking the number of idle disks & idle time gives an idea of how much headroom still left 26 (44)

27 Minimizing CPU Usage & Memory Bandwidth At 10Gbps rates, avoiding unnecessary memory copies becomes essential to avoid exhausting CPU & memory Current memory bandwidth is only 30 60Gbps in total The standard Linux disk r/w model makes one extra memcpy() between user mode and kernel buffer cache Must use O_DIRECT with large enough transfer sizes to allow disk drives to queue large enough DMA request / NCQ queues By default the network stack makes another memcpy() when adding/removing IP packet headers Can be circumvented with splice() or raw packet ring buffer 27 (44)

28 Networking CPU Overhead Planning to tolerate one memcpy() with the simplest UDP packet processing Though avoiding IP sw checksumming & fragmentation Retaining user headers (like packet #) Assuming UDP mostly in order ; attempting UDP receive at the assumed packet location and copying out-of-order packets back If zero-copy needed PacketShader raw ring buffer driver Though dictates using the latest Intel cards... mmap()ed PCAP raw ring buffer turned out to be one memcpy() plus regular IP stack processing (44)

29 10GE Cards Now recommending latest Intel cards PacketShader raw ring buffer driver compatible 29 (44)

30 Reading from a set of disks Initiate preloading of memory buffers from a set of disks Once all disks in a set have had a chance to preload, consuming the streaming data can commence at full rate 30 (44)

31 Simultaneous Read and Write Two problems, can be solved with interleaving: HDD seek time, slows down using more than one spot of disk Even without, double data streaming bandwidth required throughout the internal data paths Duration of one sequential r/w of the order of ms, >>10ms, one disk rotation 31 (44)

32 Simultaneous Read and Write Avoiding hard disk 8 12ms seek times Must stream in either read or write direction for much longer than seek time (or full disk rotation time, approx ms) E.g. read streaming for about 120ms gives the data from 10 rotations, then wastes the time of only one rotation, 90% efficiency So nothing really fancy is required but: Usually simultaneous read and write will halve the total bandwidth If the bottleneck at one subsystem can be removed (e.g. hard disks SSDs), another bottleneck will surface in another subsystem Memory and/or disk controller (PCIe) bandwidth limits are the most probable bottleneck candidates 32 (44)

33 Error Recovery If one disk handler does not complete in allotted time, it can hand over the memory block to another hander A spare disk Or even to another disk in the current set, provided there is enough slack time in the set, so it can eventually catch up Disk handler processes/threads Testing/recovery of failed disk Spare 33 (44)

34 Read Error Recovery Failure to read a chunk file from one disk causes the loss of a well-defined (0.1 1sec) time interval in the data stream In worst case (a total disk failure) the loss repeats at n*(0.1 1s) intervals Can adjust read retry strategy according to whether the reader/consumer can wait or it need quasi-realtime data 34 (44)

35 Turning Recordings into Archives If some time after a recording is made it is found that It needs to be retained/archived for a longer period of time And there is spare disk capacity available And there is some idle time for the data disk set available Then redundancy (aka raid5/raid6) can be calculated afterwards (file based) and the ECC information stored onto the spare disks If later a disk of the disk set fails, it can be replaced and its content files restored by recovering from the ECC files 35 (44)

36 Multi Site Test Setup 36 (44)

37 Test Cases We divide the test into two cases: Receiving data to the local buffer Transferring data from local buffer to remote buffer/processing Later, if this works, we can test doing thee two simultaneously == simultaneous buffering Possibly even multiple concurrent reads? To support multiple (slower) reads by several SFX instances Divide main memory roughly in half for write / read Memory chunk == file size, half of memory divided by # of disks Divide read memory chunks to concurrent readers, minimum of 2 3 chunks per reader 37 (44)

38 Boundary Conditions Although we talked about transparent access in DoW, Transparent remote writing over the network makes very little sense As it is effectively real-time e-vlbi, defeating the very purpose of buffering, namely independence of network performance Transparent remote reading looks fine, esp. with sw correlators Thus most of the buffer disk space will be located at stations, close to data sources Fits the practical economic model, too: nobody is going to buy disk space needed by a given station except the station itself... Transparent teleporting of data from one buffer to another rare (contingency?) 38 (44)

39 More Boundary Conditions Our available data sources: Test 10GE Linux PC: dummy data up to 9Gbps Could devise something to generate artificial noise data for testing? ibobs: 8Gbps UDP packet stream (1Gsps/8-bit) Not readily correlatable (nor super-interesting, too many bits/sample) Though the only real data source readily available Jivemk5 software: numbered UDP packets up to 512/896Mbps via the std Mk5 1GE Or up to 1024Mbps via an additional Mk5 10Ge card... Unlikely that JIVE can have lots of disks in a buffer machine already during summer But could use JIVE Mk5s to access station buffers (Mh, On, Jb) and feed the data to Either SFX or the Mk4 correlator 39 (44)

40 Summer Test Scenario Due to boundary conditions, combining high data rate and buffered correlation seems unlikely this summer Thus recording 22GHz UDP packet stream from Jivemk5 locally and correlating that with SFX (with UDT or even TCP remote access) seems like the only viable real correlation option Could spice that up with recording more scans while SFX is reading the previous scan(s)... Even better if there could be multiple SFX instances running concurrently, from different locations, processing several past scans Any way to show high-rate performance? >1Gbps remote reads Mh<->Jb<->On? 40 (44)

41 What Do I Have to Do Next? Test partners need a similar test system 36, 24, or 22 disks + 10GE Install Linux to the test system Preferably with a small ( leftover PC/HP Microserver) TFTP+NFS netboot server Check suitable schedule for 10GE test traffic with your operator (or WP6?) Agree test schedule with partners 41 (44)

42 Tasks 1 AALTO Create UDP stream writer + UDT(/TCP) reader sw plus Debianbased installation for OSO, UMAN JIVE Remote UDT(/TCP) reader component for SFX OSO, UMAN Acquire test systems (36/24/22/6? disks) 42 (44)

43 Tasks 2 PSNC Role & trials of computing center buffering Could we run SFX from there, accessing station buffers over the network? Could that be combined with a contingency transfer of some observation from Mh/On/Jb to PSNC buffer disks? INAF Global/local allocation/deallocation schemes The deliverable D8.3 ASTRON Long-term archival & reprocessing? Start scheduled for (44)

44 The Inconvenient Truths :-) About networked data streaming: A given station cannot really sustain recording bandwidth larger than their network connectivity unless given an unlimited disk buffer or long enough breaks between recordings. For the VLBI case: A single slow (or high-latency, like shipping) connection in a given e-vlbi network will force others (or some buffering party) to buffer most of the VLBI data of the whole network, if not all. 44 (44)