Spring Session Redis

Transcription

1 Spring Session Redis Why, How and Production Considerations March 2018 Oded Shopen

2 About Me Oded Shopen Software Amdocs Israel Working on Event-Driven, Cloud Native Microservices Redis Geek on twitter 2 Information Security Level 2 Sensitive

3 Why? 3

4 Why Spring Session? Once upon a time, there was the servlet container. And it was good = David 4

5 Why Spring Session? = Larry = David Load Balancer 5

6 Why Spring Session? = Larry =? Load Balancer 6

7 Why Spring Session? 7

8 Why Spring Session? Load Balancer 8

9 Why Spring Session? Load Balancer Jetty Jetty Jetty 9

10 Why Spring Session? Session Data 10

11 Why Spring Session? Jetty Session Data 11

12 Why Spring Session? Replaces the built-in HttpSession with another implementation Transparent drop-in replacement when using Spring Boot Makes your servers truly stateless Sessions survive application restarts No need for Load Balancer sticky sessions Conforms to the cloud-native apps 12-factor principals 12

13 Twelve-factor processes are stateless and share-nothing. Any data that needs to persist must be stored in a stateful backing service, typically a database. 13

14 Why Spring Session Redis? 14

15 Why Spring Session Redis? The application requires frequent, fast access to the session A fast database is critical Redis is really fast No user experience degradation by externalizing the session to Redis Sessions needs to expire after some time Redis expiring keys are a great solution With sharding and clustering, Redis scales when your user base scales 15

16 How? 16

17 Demo 17

18 Production Considerations 18

19 #1 Be Prepared to Scale At the very minimum, use a master/slave setup 19

20 #1 Be Prepared to Scale Consider sharding your sessions in a Redis Cluster RedisLabs + DNS Proxy will hide the cluster topology and let you focus on a simple configuration on the client side No need to configure sentinels Node 1 spring.redis.sentinel.master=masternode spring.redis.url=proxy:6379 Proxy spring.redis.sentinel.nodes=node1:6379,node2:6379,node3:6379 Node 2 Node 3 20

21 #2 Pool Settings Spring Data Redis uses a JedisConnectionFactory with the following defaults: spring.redis.pool.max-active=8 spring.redis.pool.max-idle=8 spring.redis.pool.max-wait=-1 spring.redis.pool.min-idle=0 spring.redis.timeout=0 Behind the scenes - an Apache Commons GenericObjectPool. Use max-active based on your tomcat/jetty threads. Use max-wait based on your setup Too high - all tomcat threads can get stuck, waiting Too low - your consumers may get errors Use min-idle of at least 8 to offset sudden load issues (such as mass login on a business day). Set spring.redis.timeout to a reasonably low number (such as 1000ms) Too high - your connection pool can become full quickly. Too low - your consumers may get errors 21

22 #3 Server-Side Metrics Spring Boot Actuator provides a production-ready monitoring metrics. Simply add spring-boot-starter-actuator to your gradle/maven dependencies /metrics endpoint provides valuable monitoring data However, Spring Data Redis is not available Luckily, actuator is extensible! 22

23 #4 Monitor Redis Redislabs monitoring dashboards provide excellent visibility to identify issues, as we ll soon see. If you use open source and monitor yourself on-prem, keep in mind that Redis is a singlethreaded database. A 2-core virtual machine would report 50% CPU utilization However, redis itself might be at 100% at that time Monitor the process, not the VM 23

24 #5 Please Monitor Responsibly Be careful how you monitor KEYS * runs over all the keys in the DB. Your DB is unresponsive until operation is over! SMEMBERS returns all keys in a given set. Some sets may contain all keys in the system. In our use case, we kept track of all logins from a certain vendor, which is pretty much O(n) cardinality, so same as KEYS command SCARD simply returns the size of a given set with cardinality of O(1), so it can be used safely in production SCARD spring:session:index:org.springframework.session.findbyindexnamesessionrepository.principal_name_index_name:ui_users" INFO provides valuable information (such as instantaneous_ops_per_sec) and can be safely used in production. 24

25 25

26 26

27 #6 Inside Spring Session Redis What happens when your system is completely idle (0 users)? You would expect redis to be idle as well However 27

28 #6 Inside Spring Session Redis Multiple logins using: for ((i=1;i<= ;i++)); do curl -s -u gateway:password localhost:8080/user >/dev/null; done One would expect consistent load on redis, but in reality, we got: 28

29 #6 Inside Spring Session Redis 29

30 #6 Inside Spring Session Redis Time to dig into the code (thanks, open source!) Inside = "${spring.session.cleanup.cron.expression:0 * * * * *}") public void cleanupexpiredsessions() { this.expirationpolicy.cleanexpiredsessions(); } cleanupexpiredsessions() would loop over expired keys, delete them, and touch them to make sure they are immediately deleted. This means that by default, every minute on the minute, Spring will call delete on every expired key in Redis. 30

31 #6 Inside Spring Session Redis Why? Redis can self-expire keys just fine, thank you very much The main reason: Spring wants to conduct cleanup activities when a session is expired. The API exposes SessionDeletedEvent and SessionExpiredEvent to let the program cleanup a user s session (close web sockets, notify SSO server etc.) Spring needs the session details available when the cleanup activity is performed 31

32 #6 Inside Spring Session Redis To solve this issue, two separate keys are managed for each session An expiration notification key expires after 30 minutes The actual session details key expires 5 minutes later HMSET spring:session:sessions:33fdd1b6-b496-4b33-9f7d-df96679d32fe creationtime \ maxinactiveinterval 1800 \ lastaccessedtime \ sessionattr:attrname someattrvalue \ sessionattr2:attrname someattrvalue2 EXPIRE spring:session:sessions:33fdd1b6-b496-4b33-9f7d-df96679d32fe 2100 APPEND spring:session:sessions:expires:33fdd1b6-b496-4b33-9f7d-df96679d32fe "" EXPIRE spring:session:sessions:expires:33fdd1b6-b496-4b33-9f7d-df96679d32fe 1800 SADD spring:session:expirations: expires:33fdd1b6-b496-4b33-9f7ddf96679d32fe EXPIRE spring:session:expirations

33 #6 Inside Spring Session Redis Redis guarantees expired keys would be passively cleaned, but makes no guarantees on when Spring Session Redis wants to make sure the expire key is expired within the 5 minutes window. For this reason, the cronjob updates the expired keys every minute But, there is a risk here 33

34 #6 Inside Spring Session Redis Jetty Session Data 34

35 #6 Inside Spring Session Redis Jetty Session Data 35

36 #6 Inside Spring Session Redis To recap: Every minute, each server tries to delete all expires keys. A lot of redundant calls, since only the first instance would actually achieve this purpose. Redis would then notify every instance that the expires key was deleted. Every instance would then connect to Redis to get the session details about the expiring session to handle the expiration. A little bit of math: 100 servers * 2500 expiring sessions = 500,000 access at the same second The more you scale, the bigger the problem 36

37 #6 Inside Spring Session Redis Solutions: If you don t care about session expiration events, set the cron job to an impossible value such as February 31st: spring.session.cleanup.cron.expression= ? You can also disable registering to those expiration and deletion events. Note that all servers need to disable the registering for this to public class RedisOperationsSessionRepositoryConfigNoOp public static ConfigureRedisAction configureredisaction() { return ConfigureRedisAction.NO_OP; } } 37

38 #6 Inside Spring Session Redis If you do care about session expiration events, set only a specific instance(s) to handle those events. On all other servers, disable receiving the public class RedisOperationsSessionRepositoryConfigNoListener public RedisMessageListenerContainer redismessagelistenercontainer( RedisConnectionFactory connectionfactory, RedisOperationsSessionRepository messagelistener) { } } return new RedisMessageListenerContainer(); 38

39 To Recap Be prepared to scale Monitor well, monitor with caution Open source rocks. Get involved! Tweak the framework for your own use case. 39

40 Thank you