The Design of a State Machine of Controlling the ALICE s Online-Offline Compute Platform. Sirapop Na Ranong

Size: px

Start display at page:

Download "The Design of a State Machine of Controlling the ALICE s Online-Offline Compute Platform. Sirapop Na Ranong"

Reynold Ball
5 years ago
Views:

1 The Design of a State Machine of Controlling the ALICE s Online-Offline Compute Platform Sirapop Na Ranong

2 Control, Configuration and Monitoring

3 The Functional Requirement of Control System Responsible for managing the communication and state machine that control the heterogeneous computing infrastructure. Should be able to start, pause and stop running processes during the experimental period. Required to able to monitor and handle the failures occurring in the local processes. Also required to start and stop a data taking run with an arbitrary number of detectors, data migration, recovery action, physics operations and etc.

4 The Performance Requirement of Control System Warm up time of O2 system: < 5 minutes Processes starting period : < 20 seconds at least 140K commands at least 400K state transitions Processes stopping period: < 10 seconds

5 The Constraints of the Control System Function Number of Nodes FLPs 250 EPNs ,000 processes Control servers 5 Network servers 5 Operator terminals 25

6 REVIEW ON EXISTING TECHNOLOGIES AND TOOLS FOR COMPUTER CLUSTER CONTROLLING

7 HTCondor The main objective of HTCondor is to manage the workload of computationally intensive jobs. Provides resource management, job scheduling, job queueing mechanism and priority scheme. Able to use the shared file system across machine to transfer data More information: research.cs.wisc.edu/htcondor

8 HTCondor Concern: Cannot access and control particular process

9 Globus toolkits An open source toolkit for controlling and managing grid system Aims at letting people share their computing power and data storage across the world without sacrificing their local autonomies. Includes several services and libraries for resource monitoring, discovery, management and security. More information: toolkit.globus.org

10 Globus toolkits Concern: Cannot access and control particular process toolkit.globus.org

11 We decide to review on lower level tools Communication State machine controlling

12 Message Passing Interface (MPI) An open source library focusing on the message passing paradigm of the cluster. Provides wide range of features such as point-to-point send/receive operations exchanging data between a pair of processes aggregating the partial results of computations synchronization Supports either in synchronous and asynchronous mode More information:

13 Message Passing Interface (MPI) Concern Connect and bind port function of MPI are not independent Allocate much resource for each connection (using heap) linuxcourse.rutgers.edu

14 ZooKeeper A high performance coordination service mostly used to manage the atomic broadcast in distributed system. Provides an API for implementing a distributed application in the high level. Example provided services naming and configuration management synchronization group services More information: zookeeper.apache.org

15 ZooKeeper Concern Pretty heavy and have a lot of complexity Have to access the disk storage to change and store its setting values to its configuration file (I/O cost)

16 ZeroMQ A messaging library which is designed to build distributed systems Acts itself as an abstraction over the cluster socket Allows chunks of data to be efficiently sent. Therefore, it minimizes the overhead and system calls Easy to use comparing to the aforementioned tools since it automatically handles all the socket implementations. Able to handle sending thousands of concurrent connection while ensuring that ant parts of these messages do not get lost. The delivered messages will be sent in order exactly as its original message More information:

17 ZeroMQ Sockets - tcp protocol - connect and bind are independent - asynchronous process (with queues) Simple communication pattern Request - Response Request Response Publisher - Subscriber Push - Pull

18 ZeroMQ Library Sockets - tcp protocol - connect and bind are independent - asynchronous process (with queues) Simple communication pattern Request - Response Publisher - Subscriber Publisher Subscriber Push - Pull

19 ZeroMQ Library Sockets - tcp protocol - connect and bind are independent - asynchronous process (with queues) Simple communication pattern Request - Response Publisher - Subscriber Push - Pull Pull Push

20 We decide to review on lower level tools Communication State machine controlling

21 Boost State Chart Boost Statechart C++ library for state machine definition Support for asynchronous state machines and multi-threading Manually define state and transition Example of define state Example of define transition

22 The proposed work

23 The system design Pub Pull Detector #1 Detector #2 Detector #20 Detector Sub Pub Push Pull Sub Pub Push Pull Sub Pub Push Pull 20 FLP/EPN Sub Pub Push Pull Sub Pub Push Pull Sub Pub Push Pull 1,500 Local Process Sub Push State Machine Sub Push State Machine Sub Push State Machine 70,000

24 Base State Machine v4 / CONFIGURE STANDBY / CONFIGURE / RESET ERROR / GO_ERROR 5 STATES: STANDBY CONFIGURED RUNNING PAUSED ERROR / EXIT / STOP CONFIGURED / START RUNNING / STOP PAUSED / RESUME / PAUSE 7 TRANSITIONS: CONFIGURE START STOP RESUME PAUSE EXIT GO_ERROR

25 @Running State No Thread Non - Blocking State */ While (running) { if (not receive command yet) { subscriber.recv(&message, ZMQ_DONTWAIT); } if (already received command && at cycle % 10) { state machine changing } processing }

26 Local Controller Local Controller Able to start LV.n controller at any host Each local controller communicate in different port Subscribe command from higher level Push state status to higher level Sub Push Pub Pull Publish command to lower level Pull response of all process in lower level

27 Central Controller Central Controller Top level controller Receive command from control room - Which command? (configure, start, pause, ) - Which detector? - Which node? command Generate message Pub Pull Display process status

28 Evaluation Method Experimental Setup 1. The measurement of the response time of the whole system Measure the response time of the local processes. Compare the time between the control machine (the 1st level) to the local processes (the 4th level) of the 1 process control and multiple processes control. Test the experiment with - 5,000 processes - 20,000 processes - 40,000 processes - 75,000 processes

29 Evaluation Method Experimental Setup 2. The measurement of the shortest time that system is able to handle - Test the system by continuously submitting a set of commands to the system in different frequency - Observe the shortest time that the system able to run without any error.

30 Thank you

MapReduce. U of Toronto, 2014

MapReduce. U of Toronto, 2014 MapReduce U of Toronto, 2014 http://www.google.org/flutrends/ca/ (2012) Average Searches Per Day: 5,134,000,000 2 Motivation Process lots of data Google processed about 24 petabytes of data per day in