ASTRI Distributed Stream Computing Platform

Transcription

1 ASTRI Distributed Stream Computing Platform Dr. Kent Wu Feb. 1, 2013 ASTRI Proprietary

2 Introduction Motivation A general-purpose, distributed, scalable, fault-tolerant, managed platform that allows programmers to easily develop applications for processing continuous unbounded streams of data. Fills the gap between complex proprietary systems and batchoriented computing platforms, such as Hadoop. Applications Real-time data analysis (financial data, twitter feeds, news data ) High-frequency trading Network intrusion detection Complex Event Processing Real-time Search, Social Networks 2 ASTRI Proprietary

3 What is stream data? Runner up Finalist Title 2012 DEBS Grand Challenge Competition The 6th ACM International Conference On Distributed Event-Based Systems in Berlin, Germany Large Hi-tech Manufacturing Equipment Monitoring 1000 equipment, 50 Terabytes every day, 5 Million events per second Competition 18 non-consecutive days Monitoring range of parameters and triggering conditions 77,576,214 events total 14.2 GB total data size 3 ASTRI Proprietary

4 Traditional base Architecture Problem The high latency of the response (which currently reaches 30 minutes) is the major factor increasing the severity of the KPI (Key Performance Indicators) violations and their direct monetary costs. 4 ASTRI Proprietary

5 Stream-Based Architecture Challenges Accuracy Throughput Latency 5 ASTRI Proprietary

6 Query 1 6 ASTRI Proprietary

7 Query 2 7 ASTRI Proprietary

8 Stream Processing System Stream of Event 8 ASTRI Proprietary

9 Stream Processing System Stream of Event 9 ASTRI Proprietary

10 Distributed Stream Processing System node1 Querie s node2 Querie s noden Stream of Event 10 ASTRI Proprietary

11 Problems Scaling is painful Poor fault-tolerance Coding is tedious 11 ASTRI Proprietary

12 MapReduce Scalability to large data volumes: Scan 100 TB on 1 50 MB/s = 24 days Scan on 1000-node cluster = 35 minutes Cost-efficiency: Commodity nodes (cheap, but unreliable) Commodity network (low bandwidth) Automatic fault-tolerance (fewer admins) Easy to use (fewer programmers) 12 ASTRI Proprietary

13 MapReduce Programming Model type: key-value records (key, value) Map function: Map(k1,v1) list(k2,v2) Reduce function: Reduce(k2, list (v2)) list(v3) 13 ASTRI Proprietary

14 Example: Word Count The prototypical MapReduce example counts the appearance of each word in a set of documents function map(string name, String document): // name: document name // document: document contents for each word w in document: emit (w, 1) function reduce(string word, Iterator partialcounts): // word: a word // partialcounts: a list of aggregated partial counts sum = 0 for each pc in partialcounts: sum += ParseInt(pc) emit (word, sum) 14 ASTRI Proprietary

15 Word Count Execution Input Map Shuffle & Sort Reduce Output the quick brown fox the fox ate the mouse Map Map the, 1 fox, 1 the, 1 the, 1 brown, 1 fox, 1 quick, 1 Reduce brown, 2 fox, 2 how, 1 now, 1 the, 3 how now brown cow how, 1 now, 1 brown, 1 Map ate, 1 mouse, 1 cow, 1 Reduce ate, 1 cow, 1 mouse, 1 quick, 1 15 ASTRI Proprietary

16 Twitter trending example Compute trendy Twitter topics by listening to the spritzer stream from the Twitter API CountPE1 Tweet Adapter ExtractorPE CountApp1 CountPEn CountPEn+1 TopNTopicP E CountApp2 CountPEn+m 16 ASTRI Proprietary

17 Twitter-adapter public class TwitterInputAdapter extends LoadControlAdapterApp { private LinkedBlockingQueue<Status> messagequeue = new LinkedBlockingQueue<Status>(); public void connectandread() throws Exception { messagequeue.add(status); public void initqueue() { loadcontroller.initblockingqueue(messagequeue); public void initloadcontroller() { loadcontroller = new EventQueueLoadController(); class Dequeuer implements Runnable { public void run() { while (true) { try { LoadControlObject lco = (LoadControlObject) messagequeue.take(); Status status = (Status)lco.getObj(); Event event = new Event(); event.put("statustext", String.class, status.gettext()); getremotestream().put(event); 17 ASTRI Proprietary catch (Exception e) {

18 Twitter-counter public class TopicExtractorPE extends LoadControlProcessingElement { Streamable<Event> downstream; public void setdownstream(streamable<event> stream) { this.downstream = stream; public void onevent(event event) { String text = event.get("statustext", String.class); Iterable<String> split = Splitter.on(' ').omitemptystrings().trimresults().split(text); for (String topic : split) { if (!topic.startswith("#")) { continue; String topiconly = topic.substring(1); if (topiconly.length() == 0 topiconly.contains("#")) { continue; super.throughmsgdropper(new TopicEvent(topicOnly, 1)); protected void eventprocessing(object arg0) { downstream.put((topicevent)arg0); 18 ASTRI Proprietary

19 Twitter-counter public class TopicCountAndReportPE extends ProcessingElement { transient Stream<TopicEvent> downstream; transient int threshold = 10; public void setdownstream(stream<topicevent> aggregatedtopicstream) { this.downstream = aggregatedtopicstream; public void onevent(topicevent event) { if (firstevent) { logger.info("handling new topic [{]", getid()); firstevent = false; count += event.getcount(); public void ontime() { if (count < threshold) { return; downstream.put(new TopicEvent(getId(), count)); 19 ASTRI Proprietary

20 Twitter-counter public class TopNTopicPE extends ProcessingElement { Map<String, Integer> countedtopics = Maps.newHashMap(); public void onevent(topicevent event) { countedtopics.put(event.gettopic(), event.getcount()); public void ontime() { TreeSet<TopNEntry> sortedtopics = Sets.newTreeSet(); for (Map.Entry<String, Integer> topiccount : countedtopics.entryset()) { sortedtopics.add(new TopNEntry(topicCount.getKey(), topiccount.getvalue())); StringBuilder sb = new StringBuilder(); Iterator<TopNEntry> iterator = sortedtopics.iterator(); while (iterator.hasnext() && i < 10) { TopNEntry entry = iterator.next(); sb.append("topic [" + entry.topic + "] count [" + entry.count + "]\n"); i++; sb.append("\n"); //write sb into files 20 ASTRI Proprietary

21 Twitter-counter public class TwitterCounterApp extends App { protected void oninit() { //init TopNTopicPE TopNTopicPE topntopicpe = createpe(topntopicpe.class); topntopicpe.settimerinterval(10, TimeUnit.SECONDS); // we checkpoint this PE every 20s topntopicpe.setcheckpointingconfig(new Stream<TopicEvent> aggregatedtopicstream = createstream("aggregatedtopicseen", new KeyFinder<TopicEvent>() public List<String> get(final TopicEvent arg0) { return ImmutableList.of("aggregationKey");, topntopicpe); //init TopicCountAndReportPE //init TopicExtractorPE 21 ASTRI Proprietary

22 Twitter-counter public class TwitterCounterApp extends App { protected void oninit() { //init TopNTopicPE //init TopicCountAndReportPE TopicCountAndReportPE topiccountandreportpe = createpe(topiccountandreportpe.class); topiccountandreportpe.setdownstream(aggregatedtopicstream); topiccountandreportpe.settimerinterval(10, TimeUnit.SECONDS); // we checkpoint instances every 2 events topiccountandreportpe.setcheckpointingconfig(new CheckpointingConfig.Builder(CheckpointingMode.EVENT_COUNT).frequency(2).build()); Stream<TopicEvent> topicseenstream = createstream("topicseen", new KeyFinder<TopicEvent>() public List<String> get(final TopicEvent arg0) { return ImmutableList.of(arg0.getTopic());, topiccountandreportpe); //init TopicExtractorPE 22 ASTRI Proprietary

23 Twitter-counter public class TwitterCounterApp extends LoadControlApp { protected void oninit() { //init TopNTopicPE //init TopicCountAndReportPE //init TopicExtractorPE TopicExtractorPE topicextractorpe = createpe(topicextractorpe.class); topicextractorpe.setdownstream(topicseenstream); topicextractorpe.setsingleton(true); createinputstream("rawstatus", topicextractorpe); if(this.gethadoopapplicationid()!=null) { // enable LS & LB by AppMaster super.initloadcontroller(topicextractorpe); 23 ASTRI Proprietary

24 An Example: High Frequency Algorithmic Trading CRA-INVESTIGATOR (Transaction Cost Analysis), Charles River Advisors Limited One tick per 1ms per security (million ticks/events per second) Complex data/event processing for HF algorithmic trading Distributed computational security model GUI Fault tolerance / failover protection Challenges: Regulations/risk management Aggregation Analytics Latency Throughput Scalability Performance Execution Analytics base Market Analytics Order & Execution Service Market Service Algorithmic Trading Engine Market Provider (Real-time / Historical) 24 ASTRI Proprietary

25 CRA-INVESTIGATOR (Transaction Cost Analysis) 25 ASTRI Proprietary

26 CRA-INVESTIGATOR (Transaction Cost Analysis) 26 ASTRI Proprietary

27 Conclusion Proven Decentralized Scalable / Elasticity Extensible Cluster management Fault-tolerance Load balancing Guaranteed data processing No intermediate message broker Higher level abstraction than message transmission Just works 27 ASTRI Proprietary

28 Thank You 28 ASTRI Proprietary