Data Streaming. Efficient Handling of Large and Small (Detector) Data at the Paul Scherrer Institute

Transcription

1 Data Streaming Efficient Handling of Large and Small (Detector) Data at the Paul Scherrer Institute S. Ebner 1, H. Billich 1, H. Brands 1, Ezequiel Panepucci 1, L. Sala 1 1 Paul Scherrer Institut (PSI), CH-5232 Villigen PSI, Switzerland

2 The Paul Scherrer Institute, PSI, is the largest research centre for natural and engineering sciences within Switzerland SwissFEL Swiss Free Electron Laser SLS Swiss Light Source SINQ Swiss Spallation Neutron Source HIPA The high-intensity proton accelerator...

3 Outline Motivation Objectives Design Lessons Learned Outlook Questions

4 Motivation

5 Motivation Detector Viewer File

6 Motivation Detector Viewer File Detector X Viewer File

7 Motivation File Formats Detector Viewer File Detector X Viewer File

8 Motivation Detector Viewer File

9 Motivation Analysis Detector Viewer File

10 Motivation Analysis Detector Viewer File Processing

11 Motivation Analysis Detector Viewer File I/O Limited Processing

12 Motivation Detector HDF5 Viewer File

13 Motivation Detector HDF5 Viewer File Delay / Concurrency

14 Motivation

15 Objectives

16 Objectives... Dependencies Minimize Code Complexity Design Conflicts

17 Objectives Stability Maintainability Programming Language Exchangeability Parallelization Simplicity Flexibility Maximize Decoupling (Code) Reuse Design Scaleability...

18 Design

19 Design Principles

20 Design FRAMEWORKS SUCK Principles

21 Design FRAMEWORKS SUCK Principles Data on Disks is DEAD

22 Design Control/Data Separation of Concerns Data taking, viewing, analysis, persistence, Data on Disks is DEAD Principles FRAMEWORKS SUCK

23 Design Control/Data Separation of Concerns Data taking, viewing, analysis, persistence, Data on Disks is DEAD Principles FRAMEWORKS SUCK Standardisation/ APIs on the Network

24 Design Detector Persistence

25 Design Detector Persistence

26 Design Detector ZMQ Persistence

27 Design Delivery Scheme: PUSH/PULL PUB/SUB Detector ZMQ Persistence

28 Design Delivery Scheme: PUSH/PULL PUB/SUB Detector ZMQ Persistence HWM (HighWaterMark) HWM

29 Design Control Delivery Scheme: PUSH/PULL PUB/SUB Control Detector ZMQ Persistence HWM (HighWaterMark) HWM

30 Main Header (JSON) {"htype":"array-1.0", "shape":[10,20], "type":"uint16", "frame":0} Sub Message htype defines content of main header as well as the structure of the whole message (sub messages)... Sub Message(s) can be binary or JSON 0..* required Design - Message

31 Design Control Delivery Scheme: PUSH/PULL PUB/SUB Control Detector ZMQ Persistence HWM (HighWaterMark) HWM

32 Design - Receiver Example import zmq context = zmq.context.instance() sock = context.socket(zmq.pull) sock.connect('tcp://hostname:9999') while True: header = sock.recv() print header while socket.getsockopt(zmq.rcvmore) data = sock.recv()

33 Design - Persistence Example import zmq context = zmq.context.instance() sock = context.socket(zmq.pull) sock.connect('tcp://hostname:9999') counter = 0 while True: header = sock.recv() print header while socket.getsockopt(zmq.rcvmore) data = sock.recv() open('file_{}.raw'.format(counter),'wb') f.write(data) f.close() counter += 1

34 Design Control Delivery Scheme: PUSH/PULL PUB/SUB Control Detector ZMQ Persistence HWM (HighWaterMark) HWM

35 Design Detector Persistence

36 Design Detector Persistence

37 Design Detector Splitter PUSH/PULL Persistence PUSH/PULL

38 Design Detector Splitter PUSH/PULL Persistence PUSH/PULL PUB/SUB Viewer(s)

39 Design Analysis PUSH/PULL PUSH/PULL or PUB/SUB Detector Splitter PUSH/PULL Persistence PUSH/PULL PUB/SUB Viewer(s)

40 Design - Advantages No delay in online analysis and viewing Re-use of components (e.g. file writing) Adding/exchange of components (e.g. detector, processing) Dynamic and automatic scaling (e.g. of processing) due to push/pull scheme Easy debugging through JSON header...

41 Design - Disadvantages It is a distributed system with all its challenges...

42 Detectors GigaFROST Pixels: 2016*2016 / 12 bit Exposure: 2µs exposure 1255 Hz Data Rate: 61.21Gbit/s GByte/s Write Stream: 2 GByte/s Preview Stream: 5 Hz PCO Edge Pixels: 2560x2160 / 16 bit Speed: 100 Hz mit dem 286Mhz ADC Rolling Shutter Data Rate: 1GByte/s Write Stream: 1GByte/s Preview Stream

43 Lessons Learned

44 Lessons Learned Keep code of components simple (no all-in-one-components) Failure tolerant design of components/software Automatic restart of components... Take special care when starting and closing a connection Setting ZMQ high watermark (on client and server) "right" is very important Central logging is a must Statistics for all components are needed Messages received Messages dropped...

45 Outlook

46 Outlook - Improvements Improve tooling for Python, Java, Improve (central) debugging and logging Work on the easy configuration and orchestration of the components/modules (black box concept )

47 Interested?

48 Contact Us

49 Questions?

50 Slides Contact Simon Ebner Paul Scherrer Institute WBGB/ Villigen PSI

51 Colors