THE FLEXIBLE DATA-STRUCTURE SERVER THAT COULD.

Size: px

Start display at page:

Download "THE FLEXIBLE DATA-STRUCTURE SERVER THAT COULD."

Jayson Adams
6 years ago
Views:

1 REDIS THE FLEXIBLE DATA-STRUCTURE SERVER THAT

2 REDIS

3 REDIS April 10, 2009; 6 years old Founding Author: Salvatore Sanfilippo Stable release: / June 4, 2015; 3 months ago

4 Fundamental data structures for a fundamental API. The Redis API is a direct consequence of fundamental data structures. SALVATORE SANFILIPPO

5 We're against complexity. We believe designing systems is a fight against complexity. SALVATORE SANFILIPPO

6 We optimize for joy. We believe writing code is a lot of hard work, and the only way it can be worth it is by enjoying it. SALVATORE SANFILIPPO

7 Memory storage is #1. Memory is fast, and allows Redis to have very predictable performance. Datasets composed of 10k or 40 millions keys will perform similarly. SALVATORE SANFILIPPO

8 REDIS Written in: C License: BSD Protocol: Telnet-like, binary safe Disk-backed in-memory database Master-slave replication, automatic failover Simple values or data structures by keys Lua scripting Transactions Publish/Subscribe

9 PERSISTANCE Append-only file (logical log) Snapshotting save <seconds> <changes> save save save

10 REDIS: BEST USED RAPIDLY CHANGING DATA WITH A FORESEEABLE DATABASE SIZE

11 DATA-STRUCTURES Hashes Sets Lists Sorted Sets Bitmaps

12 LUA SCRIPTING Evaluate scripts on the server directly

13 LUA SCRIPTING: EXAMPLES > eval "return redis.call('set','foo','bar')" 0 > eval "return redis.call('set',keys[1],'bar')" 1 foo

14 SCRIPT LOAD / EVALSHA $ redis-cli SCRIPT LOAD "return 'hello perconaconf'" > " c6b470dc5a6d65de91" $ redis-cli EVALSHA c6b470dc5a6d65de91 0 > hello perconaconf"

15 THE BANDWIDTH PENALTY Using EVAL sends the script to the server every time it is invoked. Redis has an internal caching mechanism, but you can avoid the bandwidth overhead altogether using EVALSHA

16 EVALSHA PRODUCTION PATTERN SCRIPT LOAD all your Lua scripts when your application boots (or when you deploy), then call the scripts later with EVALSHA and their unique SHA-1 identifier.

17 LUA SCRIPTING: BENEFITS REDUCE ROUND-TRIPS REDUCED LATENCY 7S -> 850MS BUILD ABSTRACTIONS AROUND MULTIPLE COMMANDS

18 LUA SCRIPTING: WATCH OUT When Redis is running a Lua script, it will not run anything else. Bugs in Lua scripts can lock up a Redis server forever Keep things short and easy to debug.

19 REDIS APPLICATION ARCHITECTURE

20 FUNDAMENTAL ARCHITECTURE client REDIS PROXY APP

21 REDIS ANALYTICS

22 ANALYTICS Data-informed decisions will save yourself and your company time and money.

23 REALTIME VS OFFLINE Metrics calculated from some job running once a day Use bitmaps to calculate these same metrics in realtime Make business decisions faster

24 ANALYTICS WITH REDIS Bitmaps <key> : redis.setbit( <key>, 6, 1) redis.getbit( <key>, 6) > 1

25 PERFORMANCE logged_in:yyyymmdd : redis.bitcount( logged_in:yyyymmdd ) > 5 period latency(ms) daily 50.2 weekly 392 monthly

26 BITWISE OPERATIONS logged_in:yyyymmdd : is_mobile:yyyymmdd : AND, OR, XOR, NOT

27 BITWISE OPERATIONS (CTND) redis.bitop( AND, loggedinmobile:yyyymmdd, logged_in:yyyymmdd, is_mobile:yyyymmdd ) loggedinmobile:yyyymmdd : " O(N)

28 REDIS RATE-LIMITING

29 RATE-LIMITING Limit the effect that someone can have on a given platform.

30 WHAT FOR? Resource utilization Prohibit spam (blog comments) Prohibit fraud (credit-card forms) Generate revenue (paid api s)

31 REDIS: INCR :<timestamp> : <count> redis.incr( #{clientip}:now() ) > 1 redis.incr( #{clientip}:now() ) > 2

32 REDIS: INCR/EXPIRE :<timestamp> : <count> key = #{clientip}:now() redis.incr(key) > 1 redis.expire(key, 3) > 1

33 REDIS INCRBY # some calls might be worth more than others if request_type is EXPENSIVE: redis.incrby(key, 3) else: redis.incrby(key, 1)

34 REDIS AUTHENTICATION

35 FUNDAMENTAL ARCHITECTURE client REDIS PROXY APP

36 REDIS: SESSION MGMT LOGIN SERVER REDIS (M) clients PROXY (NGINX) REDIS (S) APP

37 LOGIN SERVER userdata = UserTable.getUser(id) sessionid = uuid(32) redis.hmset(sessionid, userdata) resp.set_header( Cookie, moz_id=#{sessionid} )

38 REDIS: HASH osidfowb138foj2nf : { userid : chris.whitten@moz.com, level : pro_plus }

39 HTTP HEADERS sessionid = request.get_header( moz_id ) sessiondata = redis.hgetall(sessionid) for key, value in sessiondata: request.set_header( x-#{key}, value) cookie: moz_id=osidfowb138foj2nf x-userid: x- chris.whitten@moz.com x-level: pro_plus

40 REDIS: SETS How to track user s active sessions across their browsers or devices?

41 REDIS: ZADD userdata = UserTable.getUser(id) sessionid = uuid(32) redis.hmset(sessionid, userdata) redis.zadd(userdata[ id ], sessionid)

42 REDIS: EXPIRE userdata = UserTable.getUser(id) sessionid = uuid(32) redis.hmset(sessionid, userdata) redis.expire(sessionid, moment().add(2, weeks ))

43 REDIS: EXPIRE (CTND) sessiondata = redis.hgetall( osidfowb138foj2nf ) for key, value in sessiondata: request.set_header( x-#{key}, value) expiration = moment().add(2, weeks ) redis.expire( osidfowb138foj2nf, expiration)

44 PERFORMANCE client NGINX REDIS PROXY (NGINX) Openresty Lua co-socket APi lua-resty-redis APP

45 PERFORMANCE: HTTPERF 1500 req/sec (fresh HTTP connections) Total: connections requests replies test-duration s Connection rate: conn/s (0.7 ms/conn, <=114 concurrent connections) Connection time [ms]: min 0.3 avg 4.8 max median 0.5 stddev 8.7 Connection time [ms]: connect 0.2 Connection length [replies/conn]: Request rate: req/s (0.7 ms/req) Request size [B]: 62.0 Reply rate [replies/s]: min avg max stddev 4.3 (12 samples) Reply time [ms]: response 4.6 transfer 0.0 Reply size [B]: header content footer 2.0 (total ) Reply status: 1xx=0 2xx= xx=0 4xx=0 5xx=0 CPU time [s]: user 7.22 system (user 12.0% system 35.1% total 47.2%) Net I/O: KB/s (39.8*10^6 bps)

46 THANKS!

COSC Redis. Paul Moore, Stephen Smithbower, William Lee. March 11, 2013

COSC Redis. Paul Moore, Stephen Smithbower, William Lee. March 11, 2013 March 11, 2013 What is Redis? - Redis is an in-memory key-value data store. - Can be a middle-ware solution between your expensive persistent data-store (Oracle), and your application. - Provides PubSub,