Scaling Your Cache & Caching at Scale. Alex

Size: px

Start display at page:

Download "Scaling Your Cache & Caching at Scale. Alex"

Lenard Reed
6 years ago
Views:

1 Scaling Your Cache & Caching at Scale Alex

2 Mission Why does caching work? What s hard about caching? How do we make choices as we design a caching architecture? How do we test a cache for performance?

3 What is caching?

4 Lots of data

5 Memory Hierarchy Clock cycles to access Register 1 L1 cache 3 L2 cache 15 RAM 200 Disk Remote disk E+00 1E+01 1E+02 1E+03 1E+04 1E+05 1E+06 1E+07 1E+08 1E+09

6 Facts of Life Register Fast Small Expensive L1 Cache L2 Cache Main Memory Local Disk Remote Disk Slow Big Cheap

7 Caching to the rescue!

8 Temporal Locality Hits: 0% Cache: Stream:

9 Temporal Locality Hits: 0% Cache: Stream: Stream: Cache: Hits: 65%

10 Non-uniform distribution 3200 Web page hits, ordered by rank 100% % % % 0 Page views, ordered by rank 0% Pageviews per rank % of total hits per rank

11 Temporal locality + Non-uniform distribution

12 17000 pageviews assume avg load = 250 ms cache 17 pages / 80% of views cached page load = 10 ms new avg load = 58 ms trade memory for latency reduction

13 The hidden benefit: reduces database load Memory Database line of over provisioning

14 A brief aside... What is Ehcache? What is Terracotta?

15 Ehcache Example CacheManager manager = new CacheManager(); Ehcache cache = manager.getehcache("employees"); cache.put(new Element(employee.getId(), employee)); Element element = cache.get(employee.getid()); <cache name="employees" maxelementsinmemory="1000" memorystoreevictionpolicy="lru" eternal="false" timetoidleseconds="600" timetoliveseconds="3600" overflowtodisk="false"/>

16 Terracotta App Node App Node App Node App Node Terracotta Server Terracotta Server App Node App Node App Node App Node

17 But things are not always so simple...

18 Pain of Large Data Sets How do I choose which elements stay in memory and which go to disk? How do I choose which elements to evict when I have too many? How do I balance cache size against other memory uses?

19 Eviction When cache memory is full, what do I do? Delete - Evict elements Overflow to disk - Move to slower, bigger storage Delete local - But keep remote data

20 Eviction in Ehcache Evict with Least Recently Used policy: <cache name="employees" maxelementsinmemory="1000" memorystoreevictionpolicy="lru" eternal="false" timetoidleseconds="600" timetoliveseconds="3600" overflowtodisk="false"/>

21 Spill to Disk in Ehcache Spill to disk: <diskstore path="java.io.tmpdir"/> <cache name="employees" maxelementsinmemory="1000" memorystoreevictionpolicy="lru" eternal="false" timetoidleseconds="600" timetoliveseconds="3600" overflowtodisk="true" maxelementsondisk=" " diskexpirythreadintervalseconds="120" diskspoolbuffersizemb="30" />

22 Terracotta Clustering Terracotta configuration: <terracottaconfig url="server1:9510,server2:9510"/> <cache name="employees" maxelementsinmemory="1000" memorystoreevictionpolicy="lru" eternal="false" timetoidleseconds="600" timetoliveseconds="3600" overflowtodisk="false"> <terracotta/> </cache>

23 Pain of Stale Data How tolerant am I of seeing values changed on the underlying data source? How tolerant am I of seeing values changed by another node?

24 Expiration TTI= TTL=4

25 TTI and TTL in Ehcache <cache name="employees" maxelementsinmemory="1000" memorystoreevictionpolicy="lru" eternal="false" timetoidleseconds="600" timetoliveseconds="3600" overflowtodisk="false"/>

26 Replication in Ehcache <cachemanagerpeerproviderfactory class="net.sf.ehcache.distribution. RMICacheManagerPeerProviderFactory" properties="hostname=fully_qualified_hostname_or_ip, peerdiscovery=automatic, multicastgroupaddress= , multicastgroupport=4446, timetolive=32"/> <cache name="employees"...> <cacheeventlistenerfactory class="net.sf.ehcache.distribution.rmicachereplicatorfactory properties="replicateasynchronously=true, replicateputs=true, replicateputsviacopy=false, replicateupdates=true, replicateupdatesviacopy=true, replicateremovals=true asynchronousreplicationintervalmillis=1000"/> </cache>

27 Terracotta Clustering Still use TTI and TTL to manage stale data between cache and data source Coherent by default but can relax with coherentreads= false

28 Pain of Loading How do I pre-load the cache on startup? How do I avoid re-loading the data on every node?

29 Persistent Disk Store <diskstore path="java.io.tmpdir"/> <cache name="employees" maxelementsinmemory="1000" memorystoreevictionpolicy="lru" eternal="false" timetoidleseconds="600" timetoliveseconds="3600" overflowtodisk="true" maxelementsondisk=" " diskexpirythreadintervalseconds="120" diskspoolbuffersizemb="30" diskpersistent="true" />

30 Bootstrap Cache Loader Bootstrap a new cache node from a peer: <bootstrapcacheloaderfactory class="net.sf.ehcache.distribution. RMIBootstrapCacheLoaderFactory" properties="bootstrapasynchronously=true, maximumchunksizebytes= " propertyseparator=", /> On startup, create background thread to pull the existing cache data from another peer.

31 Terracotta Persistence Nothing needed beyond setting up Terracotta clustering. Terracotta will automatically bootstrap: - the cache key set on startup - cache values on demand

32 Pain of Duplication How do I get failover capability while avoiding excessive duplication of data?

33 Partitioning + Terracotta Virtual Memory Each node (mostly) holds data it has seen Use load balancer to get app-level partitioning Use fine-grained locking to get concurrency Use memory flush/fault to handle memory overflow and availability Use causal ordering to guarantee coherency

34 Scaling Your Cache

Scalability Continuum causal ordering YES NO NO YES YES # JVMs 1 JVM 2 or more 2 JVMS or more JVMs 2 or more JVMS 2 or more big JVMs 2 or more 2 JVMS or more JVMs 2 or more JVMS 2 or more 2 JVMs or

35 Scalability Continuum causal ordering YES NO NO YES YES # JVMs 1 JVM 2 or more 2 JVMS or more JVMs 2 or more JVMS 2 or more big JVMs 2 or more 2 JVMS or more JVMs 2 or more JVMS 2 or more 2 JVMs or more JVMs lots of JVMs 2 or more 2 JVMs or more 2 JVMs or more JVMs lots of JVMs runtime Ehcache Ehcache RMI Ehcache disk store Terracotta OSS Terracotta FX Ehcache FX Terracotta FX Ehcache FX Ehcache DX management and control Ehcache EX and FX management and control more scale 21

36 Caching at Scale

37 Know Your Use Case Is your data partitioned (sessions) or not (reference data)? Do you have a hot set or uniform access distribution? Do you have a very large data set? Do you have a high write rate (50%)? How much data consistency do you need?

38 Types of caches Name Communication Advantage Broadcast invalidation multicast low latency Replicated multicast offloads db Datagrid point-to-point scalable Distributed 2-tier point-topoint all of the above

39 Common Data Patterns I/O pattern Locality Hot set Rate of change Catalog/ customer low low low Inventory high high high Conversations high high low Catalogs/customers warm all the data into cache High TTL Inventory fine-grained locking write-behind to DB Conversations sticky load balancer disconnect conversations from DB

40 Build a Test As realistic as possible Use real data (or good fake data) Verify test does what you think Ideal test run is minutes

41 Cache Warming

42 Cache Warming Explicitly record cache warming or loading as a testing phase Possibly multiple warming phases

43 Lots o Knobs

44 Things to Change Cache size Read / write / other mix Key distribution Hot set Key / value size and structure # of nodes

45 Lots o Gauges

46 Things to Measure Application throughput (TPS) Application

47 Benchmark and Tune Create a baseline Run and modify parameters Test, observe, hypothesize, verify Keep a run log

48 Bottleneck Analysis

49 Pushing It If CPUs are not all busy... Can you push more load? Waiting for I/O or resources If CPUs are all busy... Latency analysis

50 I/O Waiting Database Connection pooling Database tuning Lazy connections Remote services

51 Locking and Concurrency Threads Locks Key Value 1 get 2 get 2 put 8 put

52 Locking and Concurrency Threads Locks Key Value get 2 get put put

53 Objects and GC Unnecessary object churn Tune GC Concurrent vs parallel collectors Max heap...and so much more Watch your GC pauses!!!

54 Cache Efficiency Watch hit rates and latencies Cache hit - should be fast Unless concurrency issue Cache miss Miss local vs Miss disk / cluster

55 Cache Sizing Expiration and eviction tuning TTI - manage moving hot set TTL - manage max staleness Max in memory - keep hot set resident Max on disk / cluster - manage total disk / clustered cache

56 Cache Coherency No replication (fastest) RMI replication (loose coupling) Terracotta replication (causal ordering) - way faster than strict ordering

57 Latency Analysis Profilers Custom timers Tier timings Tracer bullets

58 mumble-mumble* It s time to add it to Terracotta. * lawyers won t let me say more

59 Thanks! Twitter Blog - Terracotta - Ehcache -

The new Ehcache and Hibernate Caching SPI Provider

The new Ehcache and Hibernate Caching SPI Provider The new Ehcache 2.0.0 and Hibernate Caching SPI Provider Presented by: Chris Dennis, Software Engineer, Terracotta Inc. May 12, 2010 Agenda Intro to Ehcache and Terracotta Ehcache and Ehcache + Terracotta