Home of Redis. Redis for Fast Data Ingest

Size: px

Start display at page:

Download "Home of Redis. Redis for Fast Data Ingest"

Emory Watts
5 years ago
Views:

1 Home of Redis Redis for Fast Data Ingest

2 Agenda Fast Data Ingest and its challenges Redis for Fast Data Ingest Pub/Sub List Sorted Sets as a Time Series Database The Demo Scaling with Redis e Flash 2

3 Fast Data Ingest Scenarios

4 IOT 4

5 Network Traffic Inspection 5

6 Social Media Analysis 6

7 More Scenarios User Activity Tracking Log Collection Multi-player Gaming And more Fintech 7

8 Fast Data Ingest Challenges Keeping up with the pace of data arrival Data from multiple sources with no standard data format Filter, analyze, and transform data in real-time Managing data arriving from sources distributed geographically 8

9 Requirements for Fast Data Ingest Physical infrastructure network, computational resources, etc. Software stack to: Filter Aggregate Transform Distribute data in real-time with sub-millisecond latency 9

10 Fast Data Ingest with Redis

11 About Redis Open source. The leading in-memory database platform, supporting any high performance operational, analytics or hybrid use case. The open source home and commercial provider of Redis Enterprise (Redis e ) technology, platform, products & services. 11

12 Redis for Fast Data Ingest 12

13 Redis for Fast Data Ingest Subscriber 1 Strings Sets Geospatial Indexes Subscriber 2 Sorted Sets Lists Bitmaps Publisher Channel Subscriber 3 Hashes Hyperloglog Bit field Subscriber n Redis Data Structures Redis Pub/Sub 13

14 Common Ingest Techniques in Redis

15 Pub/Sub Subscriber 1 Publisher Channel Subscriber 2 Subscriber 3 Subscriber n Commands Publisher: Subscriber: publish <channel name> <message> subscribe <channel name> 15

16 List Subscriber 1 Subscriber 2 Publisher Subscriber 3 Subscriber n Commands Publisher: Subscriber: lpush <list name> <message> brpop <list name> <timeout> 16

17 Sorted Set Subscriber 1 Subscriber 2 Publisher Subscriber 3 Subscriber n Commands Publisher: Subscriber: zadd <timeseries name> <timestamp> <message> zrangebyscore <timeseries name> <last timestamp> <current timestamp> WITHSCORES 17

18 The Demo

19 Demo: Problem Description English Tweets Filter "lang":"en" Popular hashtags among English tweets Match pattern #(\\w+) Increment count for that pattern All Tweets Influencer Tweets Filter Influencer Catalog Sample Tweet Message in the JSON format: { "created_at":"tue Jul 11 17:06: ", "id": , } followers_count > Map influencer id to profile Sorted Set: follower count -> id "text":"usgs reports a M2 #earthquake 31km WSW of Enterprise, Utah on 17:01:53 UTC #quake", "user":{ "id": , "name":"every Earthquake", "screen_name":"everyearthquake", "location":"earth", "followers_count":18978, "friends_count":17, "lang":"en" } 19

20 Demo Setup Service Provider for Messages Programming Language for the demo IDE Redis container on Docker 20

21 The Three Data Ingest Techniques Fast Data Ingest Technique Pros Cons Pub/Sub Lists Easy Decoupled setup Good for geographically distributed setup Easy Resilient to connection loss Not resilient to connection loss Requires many connections Tightly coupled producers and consumers Data duplication Sorted Sets Resilient to connection loss Least chance of losing data Access to historical data Loosely coupled producers and consumers Consumes space Complex logic 21

22 Technique 1: Fast Data Ingest with Pub/Sub

23 Fast Data Ingest with Pub/Sub EnglishTweetsFilter English Tweets HashTagCollector Ingest PubSub AllTweets InfluencerTweetsFilter Influencer Tweets InfluencerCollector Advantages Easy Decoupled setup Good for geographically distributed setup 23

24 Class Diagrams and Sample Code 24

25 Technique 2: Fast Data Ingest with Lists

26 Fast Data Ingest with Lists EnglishTweetsFilter englishtweets EnglishTweets Listener HashTagFilter Ingest Stream alldata AllTweets Listener InfluencerFilter Advantages Easy Resilient to connection loss 26

27 Class Diagrams and Sample Code 27

28 Technique 3: Fast Data Ingest with Sorted Sets

29 Fast Data Ingest with Sorted Sets EnglishTweetsFilter HashTagFilter Ingest Stream englishtweets alltweets InfluencerFilter Advantages Resilient to connection loss Least chance of losing data Access to historical data Loosely coupled producers and consumers 29

30 Class Diagrams and Sample Code 30

31 Redis e for Fast Data Ingest

32 Redis e Technology Redis Database Instances 32

33 Redis e Technology Enterprise Layer Zero latency proxy Cluster Manager REST API Open Source Layer 33

34 Redis e Technology Redis e Node Enterprise Layer Zero latency proxy Cluster Manager REST API Open Source Layer 34

35 Redis e Technology Redis e Cluster Shared nothing cluster architecture Fully compatible with open source commands & data structures 35

36 Redis e - Shared Nothing Symmetric Architecture Distributed Proxies Single or Multiple Endpoints Cluster Management Path Data Path Proxies Node Watchdog Cluster Watchdog Redis Shards Node 1 Node 2 Node N (odd number) Unique multi-tenant Docker like architecture enables running hundreds of databases over a single, average cloud instance without performance degradation and with maximum security provisions 36

37 Redis e Benefits for Data Ingest Effortless Scaling Always On Availability Substantially Lower Costs Simple, Seamless Clustering. Linear scaling Instant Failure Recovery, No Data loss Run on Flash as a RAM extension ACID Compliance in Cluster Architecture Stable and Predictable High Performance Top notch 24x7 expert support 37

Pluggable storage engine Available on SATA-based SSD, NVMe-based SSD, NVDIMM like 3D

38 Redis e Flash Near-RAM performance at 70%+ lower costs 2048 GB RAM Technology treats Flash as a RAM replacement (or extension) RAM/Flash ratio can be easily configured Pluggable storage engine Available on SATA-based SSD, NVMe-based SSD, NVDIMM like 3D XPoint/SCM on x86 and P8 platforms Keys & hot values 204 GB RAM Cold values 1844 GB Flash 10% 90% 38

39 Redis e Flash - 10TB Redis Deployment on EC2 Redis on RAM Redis e Flash Dataset size 10 TB 10 TB Database size with replication 30 TB 20 TB AWS instance type x1.32xlarge i3.16xlarge Actual instance size (RAM, and RAM+Flash) 1.46 TB 3.66 TB # of instances needed (for quorum) Persistent Storage (EBS) 154 TB 110 TB 1 year cost (reserved instances) $1,595,643 $298,896 Savings % * Redis Enterprise only needs 1 copy of the data because quorum issues are solved at the node level * 39

40 Questions??????????? 40

41 One more thing. redis.conf setting: client-output-buffer-limit pubsub 32mb 8mb 60 With this setting, Redis will force the clients to disconnect under two situations: If the output buffer grows more than 32mb If the output buffer holds 8mb of data consistently for 60 seconds 41

42 Thank You Roshan Redis 42

Home of Redis. April 24, 2017

Home of Redis. April 24, 2017 Home of Redis April 24, 2017 Introduction to Redis and Redis Labs Redis with MySQL Data Structures in Redis Benefits of Redis e 2 Redis and Redis Labs Open source. The leading in-memory database platform,