CIT 668: System Architecture. Caching

Transcription

1 CIT 668: System Architecture Caching

2 Topics 1. Cache Types 2. Web Caching 3. Replacement Algorithms 4. Distributed Caches 5. memcached

3 A cache is a system component that stores data so that future requests for that data can be served faster. The original data may be the result of an expensive computation or access to a slower storage medium. Cache

4 Why Caches Work: Locality Two common patterns of data access: Temporal locality: Recently referenced items are likely to be referenced in the near future. Spatial locality: Items with nearby addresses tend to be referenced close together in time. The hit rate of a cache is the percentage of data accesses that find data in the cache.

5 Cache Types Write-Back Write operations finish when data in cache. Disk writes happen at a later time (asynchronous.) Faster, but if power fails written data disappears. Write-Through Write operations finish when data on disk. Disk writes happen at same time (synchronous.) Slower, but no data will be lost after write complete.

6 Memory Hierarchy

7 Disk Caching Operating System Cache Typically gigabytes in size (uses spare RAM.) Power cutoff causes cached data to be lost. Use sync command to force write to disk. Disk Controller Cache Motherboard controllers typically have no cache. Server RAID controllers often have 128MB-1GB write cache. Some are battery-backed to preserve data if power fails. Disk Cache All SATA/SAS disks have 8-64MB cache. Read prefetching grabs data blocks before requested in anticipation of sequential reads. Write ordering re-orders writes to minimize head movement (only disk firmware knows drive geometry so higher level caches can t.)

8 Database Caching

9 Web Caching: Static and Dynamic Content Static Content The origin of the content sent by the server comes from an existing file lying on a filesystem Files such as html, jpeg Dynamic Content The origin of the content sent by the server is dynamically generated by some other program or script or application programming interface (API) called by the web server. Serving static content is usually much faster (from 2 to 100 times) than serving dynamic content, especially if the latter involves data access from a database.

10 Types of Web Caches Client 1 CC Client 2 CC Client n CC Client-side Cache 1.Limited capacities 2.Security issue CP ES CP Internet ES CP Client-side Proxy 1.Very close to clients 2. Reduce web server load reduce communication latency and network traffic 3. Can only cache static web content 4. Can NOT cache web processing components Edge Server 1.Reduce web server load 2.Reduce communication latency and network traffic if appropriately placed 3. Can not be close to clients RP WS Reverse Proxy CC: Client-side Cache CP: Client-side Proxy 1.Reduce web server load ES: Edge Server 2.Cannot reduce communication RP: Reverse Proxy latency and network traffic WS: Web Server

11 Reverse Proxy Caching

12 Cache Replacement Which data item should be removed from cache when the cache is full? Base decision on metrics, such as hit rate saved bandwidth latency reduction CPU performance.

13 Classic Replacement Algorithms Least Recently Used (LRU) It evicts the cached object which was requested the least recently. Least-Frequently-Used (LFU) It evicts the cached object which is accessed least frequently. Size Algorithm It evicts the largest object.

14 Key-Based Replacement Algorithms LRU-Threshold It first presets a threshold and if the objects are larger than that threshold, the objects are not cached. LRU-Minimum It first checks whether there are any objects larger than or equal to the requested object. If yes, one of them is replaced using LRU. Otherwise, it checks all objects larger than half the size of requested object. Repeat this procedure until there exists enough cache space. Lowest Latency First It minimizes the average latency by replacing the object with the lowest download latency first

15 Cost-Based Replacement Algorithms GreedyDual-Size (GDS) algorithm Cost is time/bandwidth to get new copy of object. When a replacement needs to be made, the object with the lowest cost/size ratio will be replaced. GreedyDual-Size Popularity (GDSP) Incorporates frequency of object access into ratio when deciding which object to replace.

16 Cache Validation Cache should provide same version of object as is currently on the server. HTTP headers tell caches how to validate Expires: do not use object after specified date. Cache-Control: flexible HTTP/1.1 headers max-age: do not use object after specified # seconds have passed since retrieval. private: cached data is specific to authenticated user, so only browser caches should cache object. public: marks authenticated responses as cacheable. Etag: unique identifier generated by server changed every time object on server changes. Cache can be sure object is up to date with a HEAD request with If-None-Match header.

17 Why Use Distributed Caches? Why not use DB server cache? DB server is a single machine with limited RAM. It is faster to save application level data, perhaps with complete HTML formatting. Why not have each web server cache data? Servers will cache some of the same data, wasting RAM on redundant caching.

18 Distributed Caches

19 memcached Free & open source, highperformance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load. Memcached is an in-memory key-value store for small chunks of arbitrary data (strings, objects) from results of database calls, API calls, or page rendering. -- memcached.org

20 Client software Memcached components Configured with list of memcached servers. Client-based hashing algorithm Chooses server based on key. Server software Stores key/value pairs in in-memory hash table. Servers don t communicate with each other. Server algorithms Cache replacement algorithms, LRU by default.

21 Hashing A hash table is a data structure that uses a hash function to map identifiers known as keys to their associated values. A hash function is a mathematical function that takes a large input and converts it to a small number that can be used to index into an array. Key feature: it takes the same amount of time to store or fetch an item no matter how large the table is.

22 memcached process

23 Memcached example diagram

24 Original code Using memcached function get_foo(int userid) { result = db_select("select * FROM users WHERE userid =?", userid); return result; } Updated code with memcached function get_foo(int userid) { /* first try the cache */ data = memcached_fetch("userrow:" + userid); if (!data) { /* not found : request database */ data = db_select("select * FROM users WHERE userid =?", userid); /* then store in cache until next get */ memcached_add("userrow:" + userid, data); } return data; }

25 Key Points 1. Locality principles 1. Spatial 2. Temporal 2. Cache types: write through vs. write back 3. Web caches: client, proxy, edge(cdn), reverse proxy 4. Cache replacement algorithms 1. LRU, LFU, size 2. LRU-minimum, LRU-threshold, lowest latency 3. GDS, GDSP 5. Distributed caching advantages 6. Memcached 1. Distributed in-memory key/value store 2. Hashing algorithm to distribute data across servers

26 References 1. John Allspaw, The Art of Capacity Planning, O Reilly, Brad Fitzpatrick, LiveJournal: Behind the Scenes, USENIX, Abhijit Gadkari, Caching in the Distributed Environment, The Architecture Journal, 4. Mark Nottingham, Caching Tutorial, Theo Schlossnagle, Scalable Internet Architectures, Sams Publishing,