(incubating) Introduction. Swapnil Bawaskar.

Size: px

Start display at page:

Download "(incubating) Introduction. Swapnil Bawaskar."

Randolph McDonald
5 years ago
Views:

(incubating) Introduction William Markito

1 (incubating) Introduction William Swapnil

2 Agenda Introduction What? Who? Why? How? DEBS Roadmap Q&A 2

3 3 Introduction

4 Introduction A distributed, memory-based data management platform for data oriented apps that need: high performance, scalability, resiliency and continuous availability fast access to critical data set location aware distributed data processing event driven data architecture 4

5 Incubating but rock solid systems in production (real customers) Cutting edge use cases Massive increase in data volumes Falling margins per transaction Increasing cost of IT maintenance Need for elasticity in systems Real Time response needs Time to market constraints Need for flexible data models across enterprise Distributed development Persistence + In-memory Global data visibility needs Fast Ingest needs for data Need to allow devices to hook into enterprise data Always on Financial Services Providers (every major Wall Street bank) Department of Defense Largest travel Portal Airlines Trade clearing Online gambling Largest Telcos Large mfrers Largest Payroll processor Auto insurance giants Largest rail systems on earth 5

6 Incubating but rock solid 17 billion records in memory GE Power & Water's Remote Monitoring & Diagnostics Center 3 TB operational data in-memory, 400 TB archived China Railways 4.6 Million transactions a day / 40K transactions a second China Railways 120,000 Concurrent Users Indian Railways 6

7 China Railway Corporation Indian Railways Population: 1,401,586,609 1,251,695,616 World: ~7,349,000,000 ~36% of the world population

8 Incubating but rock solid operations per second Cassandra Geode 0 A Reads A Updates B Reads B Updates C Reads D Inserts D Reads F Reads F Updates YCSB Workloads 8

9 Horizontal scaling for reads, consistent latency and CPU Speedup speedup latency (ms) CPU % Server Hosts Scaled from 256 clients and 2 servers to 1280 clients and 10 servers Partitioned region with redundancy and 1K data size 0

10 What makes it go fast? Minimize copying Minimize contention points Run user code in-process Partitioning and parallelism Avoid disk seeks Automated benchmarks

Hands-on: Build & run Clone & Build git clone

com/apache/incubator-geode cd incubator-geode.

/bin/gfsh gfsh> start locator --name=locator gfsh>

11 Hands-on: Build & run Clone & Build git clone cd incubator-geode./gradlew build -Dskip.tests=true Start a server cd gemfire-assembly/build/install/apache-geode./bin/gfsh gfsh> start locator --name=locator gfsh> start server --name=server gfsh> create region --name=myregion --type=replicate Docker $ docker run -it apachegeode/geode:1.0.0-incubating.m1 gfsh 11

12 Hands on

13 Concepts Cache Region Member Client Cache Functions Listeners High Availability Serialization 13

14 Concepts Cache In-memory storage and management for your data Configurable through XML, Java API or CLI Region Region Region Cache JVM Collection of Region 14

15 Concepts Region Distributed java.util.map on steroids (Key/Value) Consistent API regardless of where or how data is stored Observable (reactive) Highly available, redundant on cache Member (s). Region java.util.map Cache JVM Key K01 K02 Value May Tim 15

16 Concepts Region Key K01 Value May Region Key K01 Value May Region K02 java.util.map Cache Tim K02 java.util.map Cache Tim JVM JVM Local, Replicated or Partitioned In-memory or persistent Redundant LRU Overflow LOCAL LOCAL_HEAP_LRU LOCAL_OVERFLOW LOCAL_PERSISTENT LOCAL_PERSISTENT_OVERFLOW PARTITION PARTITION_HEAP_LRU PARTITION_OVERFLOW PARTITION_PERSISTENT PARTITION_PERSISTENT_OVERFLOW PARTITION_PROXY PARTITION_PROXY_REDUNDANT PARTITION_REDUNDANT PARTITION_REDUNDANT_HEAP_LRU PARTITION_REDUNDANT_OVERFLOW PARTITION_REDUNDANT_PERSISTENT PARTITION_REDUNDANT_PERSISTENT_OVERFLOW REPLICATE REPLICATE_HEAP_LRU REPLICATE_OVERFLOW REPLICATE_PERSISTENT REPLICATE_PERSISTENT_OVERFLOW REPLICATE_PROXY 16

17 Concepts Persistent Regions Put k4->v7 Member 1 Durability Oplog3.crf Modify k4->v7 WAL for efficient writing Oplog2.crf Modify k1->v5 Create k6->v6 Create k2->v2 Create k4->v4 Consistent recovery Compaction Region Key K01 Value May Region Key K01 Value May K02 Tim K02 Tim java.util.map Cache java.util.map Cache JVM JVM Server 1 Server N 17

18 Persistence - Shared Nothing Server 1 Server 2 Server 3 18

19 Persistence - Shared Nothing Server 1 Server 2 Server 3 B1 B1 Primary B2 B2 Secondary B3 B3 19

20 Persistence - Shared Nothing Server 1 Server 2 Server 3 B1 B1 Primary B2 B2 Secondary B3 B3 20

21 Persistence - Shared Nothing Server 1 Server 2 Server 3 B1 B2 B1 B2 Primary Secondary B3 B3 21

22 Persistence - Shared Nothing Server 1 Server 2 Server 3 B1 B2 B1 B2 Primary Secondary B3 B3 22

23 Persistence - Shared Nothing Server 1 waits for others when it starts Server 1 Server 2 Server 3 B1 B1 Primary B2 B2 B2 Secondary B3 B3 B3 23

24 Persistence - Shared Nothing Fetches missed operations on restart Server 1 Server 2 Server 3 B1 B1 Primary B2 B2 Secondary B3 B3 24

25 Persistence - Operational Logs Put k6->v6 Member 1 Append to operation log Oplog2.crf Modify k1->v5 Create k6->v6 Oplog1.crf Create k1->v1 Create k2->v2 Modify k1->v3 Create k4->v4 25

26 Persistence - Operational Logs - Compaction Put k6->v6 Member 1 Append to operation log Oplog2.crf Modify k1->v5 Create k6->v6 Create k2->v2 Create k4->v4 Oplog1.crf Create k1->v1 Modify k1->v3 Copy live data forward 26

27 Concepts Member A process that has a connection to the system A process that has created a cache Embeddable within your application Client Locator Server 27

28 Concepts Client cache A process connected to the Geode server(s) Can have a local copy of the data Run OQL queries on local data Can be notified about events on the servers GemFire Server Application Client Cache Region Region Region 28

29 Concepts Client Notifications Register Interest Individual Keys OR RegEx for Keys Updates Local Copy Examples: region.registerinterest( key-1 ); region1.registerinterestregex( [a-z]+ ); Continuous Query Receive Notification when Query condition met on server Example: SELECT * FROM /tradeorder t WHERE t.price > Can be DURABLE 29

30 Concepts Functions Used for distributed concurrent processing (Map/Reduce, stored procedure) Highly available Data oriented Member oriented Execute Functions f 1, f 2, f n Submit (f1) 30

31 Concepts Functions Server Partitioned Region Data Store - X Server Distributed System Server Partitioned Region Data Store - Y Server Partitioned Region Data Store - Z Server Partitioned Region Data Accessor 4 execute execute result result Server Partitioned Region Data Accessor 1 Client Region filter = Keys X, Y FunctionService.onRegion.withFilter.execute 2 execute 6 ResultCollector.getResult 5 result 31

32 Concepts Listeners CacheWriter / CacheListener AsyncEventListener (queue / batch) Parallel or Serial Conflation 32

33 33 Concepts - HA

34 Fixed or flexible schema? id name age pet_id or { } id : 1, name : Fred, age : 42, pet : { name : Barney, type : dino }

35 C#, C++, Java, JSON Portable Data exchange header data pdx length dsid typeid fields offsets No IDL, no schemas, no hand-coding Schema evolution (Forward and Backward Compatible) * domain object classes not required

36 Efficient for queries SELECT p.name FROM /Person p WHERE p.pet.type = dino { } id : 1, name : Fred, age : 42, pet : { name : Barney, type : dino } single field deserialization

37 But how to serialize data? Benchmark:

38 Schema evolution Application #1 Member A Member B Application #2 v2 objects preserve data from missing fields v1 objects use default values to fill in new fields PDX provides forwards and backwards compatibility, no code required v1 v2 Distributed Type Definitions

39 39 Adapters

40 memcached write-through as opposed to cache-aside Stale Cache Inconsistent Cache Thundering Heards 40

41 Redis Scalable Data-Structures Use All Cores WAN Replication 41

42 TAXI TRIP ANALYSIS (INCUBATING) (DEBS GRAND-CHALLENGE) WITH APACHE GEODE TAXI TRIP ANALYSIS (DEBS GRAND-CHALLENGE) WITH APACHE GEODE Swapnil Bawaskar William Markito Oliveira

43 INTRODUCTION DEBS Distributed Event-Based Systems Grand challenges (2013, 2014, 2015, 2016 ) Analyze NY Taxi Trip information 2013* 12 GB in size and ~173 million events. Most profitable areas Most frequent routes * FOIL (The Freedom of Information Law)

44 INTRODUCTION DEBS

46 IMPLEMENTATION

47 IMPLEMENTATION HOW AsyncEvent Listener Parallel or Serial public class FrequentRouterListener implements AsyncEventListener, Declarable { public boolean processevents(list<asyncevent> list) { // PDX object deserializing single field pickupdatetime = (Date) taxitrip.getfield("pickup_datetime"); // some processing with events } } - Memory - Threads - Persistence - Batch size - Batch interval

48 IMPLEMENTATION HOW SELECT e.key,e.value FROM /FrequentRoute.entries e ORDER BY e.value.numtrips DESC LIMIT 10 1' 2' CLIENT n F_ROUTES Area Area 1.1 x.y 2.1 x'.y' { 3 { TRIPS Taxi Area 1 x.y 2 x.y' CACHING_PROXY F_ROUTES N x.y Area Area 1.1 x.y 2.1 x'.y' Update routes SELECT AVG(getFarePlusTip()) as avgtotal, pickup_cell.tostring() FROM /TaxiTrip t GROUP BY pickup_cell.tostring() ORDER BY avgtotal DESC LIMIT 10" NOT SQL!*

49 IMPLEMENTATION HOW F_ROUTES Area Area 1.1 x.y 2.1 x'.y' Evict entries older than 15 seconds Replicated Listener attached Taxi TRIPS Area 1 x.y 2 x.y' N x.y Historical with memory eviction to disk Partitioned across nodes Async listener with queue

50 Demo

51 Roadmap Off-heap memory storage Cloud Foundry service Lucene Search* HDFS Persistence* Spark Integration* * -Experimental and waiting community feedback 51

52 How to Contribute Code New features Bug fixes Writing tests Documentation Wiki Web site Community Join the mailing list Ask or answer Join our HipChat Become a speaker Finding bugs Testing an RC/Beta User guide 52

53 Links Website JIRA Wiki cwiki.apache.org/confluence/display/geode GitHub Mailing lists mail-archives.apache.org/mod_mbox/incubator-geode-dev/ 53

54 Thank you

IoT Platform using Geode and ActiveMQ

IoT Platform using Geode and ActiveMQ Scalable IoT Platform Swapnil Bawaskar @sbawaskar sbawaskar@apache.org Agenda Introduction IoT MQTT Apache ActiveMQ Artemis Apache Geode Real world use case Q&A 2