Streaming data Model is opposite Queries are usually fixed and data are flows through the system.

Transcription

1 1

2 2

3 3

4 Main difference is: Static Data Model (For related database or Hadoop) Data is stored, and we just send some query. Streaming data Model is opposite Queries are usually fixed and data are flows through the system. 4

5 News, blogs, weather alerts, earthquakes, sales, traffic social, etc 5

6 6

7 7

8 8

9 Actors programming model: Each processing unity sending message across other processing units. That makes pairs processing very simple. cool features: ( Flexible deployment: Application packages are standard jar files (suffixed.s4r) Platform modules for customizing the platform are standard jar files By default keys are homogeneously sparsed over the cluster: helps balance the load, especially for fine grained partitioning Modular design: both the platform and the applications are built by dependency injection, and configured through independent modules. makes it easy to customize the system according to specific requirements pluggable event serving policies: load shedding, throttling, blocking Dynamic and loose coupling of S4 applications: through a pub-sub mechanism makes it easy to: assemble subsystems into larger systems reuse applications 9

10 separate pre-processing provision, control and update subsystems independently Fault tolerant Fail-over mechanism for high availability Checkpointing and recovery mechanism for minimizing state loss Pure Java: statically typed, easy to understand, to refactor, and to extend 9

11 1. Data streams coming in 2. Cluster for adaptors (read in stream and out put standardized event format) 3. Load balancer: to going to the clusters by send event to the servers (the clusters). 4. With zoom in: each node has Event listener, Emitter (for event come in and out); Processing Element Container in the middle. PE (processing Element). With each PE Prototype, it has extra PE instances. 10

12 1. Loss failover is acceptable. Upon a server failure, processes are automatically moved to a standby server. The state of the processes, which is stored in local memory, is lost during the handoff. The state is regenerated using the input streams. Downstream systems must degrade gracefully. 3. Data processing is based on Processing Elements (PE) - corresponding to Actor model of concurrent computation 11

13 Processing Elements (PEs) are the basic computational units in S4 Each instance of PE is uniquely identified by four components: 1. it's functionality as defined by PE class and associated configuration 2. the types of events that it consumes 3. the keyed attribute in those events 4. the value of the keyed attribute in events which it consumes 5. Every PE consumes exactly those events which correspond to the value on which it is keyed and can produce output events 6. PE is instantiated for each value of the key attribute 7. Special class of PEs is the set of keyless PEs with no keyed attribute or value. These PEs consume all events of the type with which they are associated 8. Keyless PEs are typically used at the input layer of an S4 cluster where events are assigned a key 9. "Garbage Collection" of PEs represents a major challenge for the platform 10. State of the PE is lost after the "cleanup 12

14 1. Processing Nodes (PNs) are the logical hosts to PEs 2. They are responsible for listening to events, executing operations on the incoming events, dispatching events with the assistance of communication layer and emitting output events 3. S4 routes each event to PN based on a hash function of the values of all known keyed attributes in that event 4. A single event may be routed to multiple PNs 5. The set of all possible keying attributes is known from the configuration of S4 cluster 6. An event listener in the PN passes incoming events to the processing element container (PEC) which invokes the appropriate PEs in appropriate order 7. There is a special type of PE object : the PE prototype 8. It has the first three components of its identity : functionality event type keyed attribute 9. The PE prototype does not have the attribute value assigned 10. This object is configured upon initialization and for any value V is capable of cloning itself to create fully qualified PEs of that class with identical configuration and value V for the keyed attribute 11. This operation is triggered by the PN once for each unique value of the keyed attribute 13

15 that it encounters 12. As a result of this design - all events with a particular value of a keyed attribute are guaranteed to arrive at particular corresponding PN and be routed to the corresponding PE instance within it 13. Every keyed PE can be mapped to exactly one PN based on the value of the hash function applied to the value of keyed attribute of that PE 14. Keyless PEs may be instantiated on every PN 13

16 How does it work? Some definitions Platform S4 provides a runtime distributed platform that handles communication, scheduling and distribution across containers. Distributed containers are called S4 nodes S4 nodes are deployed on S4 clusters S4 clusters define named ensembles of S4 nodes by default, the size of the cluster is fixed the size of an S4 cluster corresponds to the number of logical partitions (sometimes referred to as tasks) an ongoing integration with Apache Helix will remove these limitations and allow a variable number of nodes and rebalancing the partitions Applications Users develop applications and deploy them on S4 clusters Applications are built as a graph of: Processing elements (PEs) Streams that interconnect PEs PEs communicate asynchronously by sending events on streams. 14

17 Events are dispatched to nodes according to their key External streams are a special kind of stream that: send events outside of the application receive events from external sources can interconnect and assemble applications into larger systems. Adapters are S4 applications that can convert external streams into streams of S4 events. Since adapters are also S4 applications, they can be scaled easily. 14

18 15

19 16

20 Example: we want to Count Msgs by User 1. First thing we need is a counter: 1. We will reuse that class count PE 2. This could be a java class 3. Should be very simple api 2. Once have the class, the next thing is to write the prototype 1. It is a single version of the PE for which will be cloned 2. In this example. The counter class will be extended to Counter PE class 3. It will be configured to use specific parameters 17

21 18

22 19

23 20

24 21

25 22

26 23

27 24

28 25

29 26

30 27

31 28