MATH is Hard: TTL Index Configuration and Considerations. Kimberly Wilkins Sr.

Transcription

1 MATH is Hard: TTL Index Configuration and Considerations Kimberly Wilkins Sr.

2 Drowning in Data?

3 TTL s are your lifeboat

4 Sources?

5 Amounts? 600 TB 115 million 58 million avg 1EB -2 yrs ^^ 5 B s IoT/IoE 1ZB

6 What do you KEEP? How do you REMOVE?

7 TTL s Time To Live Indexes by

8 What Collections Indexes Why 2.2 +, normal collection - special TTL index type Defines # seconds a document exists Control collection/database size Automatically remove based on BSON date type No MANUAL mass/batch deletes or crons

9 Event data Logs Time Series Session Info Use Cases User Activity Logs gaming, social Queuing Any time data needs to persist only for a short, specific period of time

10 What is Your Goal? Think Before Creating Save space? Improve Performance? Data retention policies? Business logic or requirements? System load? More Reads vs. Writes?

11 Use Expire AFTER vs. AT? Global expiration for all documents expireafterseconds (after 3 days) { "createdat": 1 }, { expireafterseconds: } Global policy determines Per document expiration for each document - Certain Clock Time Things to Think cont d {"expiresat": 1 }, { expireafterseconds: 0 } Application sets FUTURE time Combo - {"expiresat": 1 }, { expireafterseconds: } application and global

12 Numbers to Know Sample seconds minutes hours days math seconds / 60 seconds = 1440 minutes / 60 minutes = 24 hours = 1 day 1 hour = 3600 seconds 1 day = seconds 3 days = seconds * 3 = seconds 7 days = seconds * 7 = seconds -Offsets 10.5 days = seconds * 10.5 = seconds

13 How to Create TTL s Create a new index with ensureindex Specify expireafterseconds option -REQUIRED use "createdat or "expiresat leader 2 real Types : global & per document/application expireafterseconds -global type all docs expire X seconds after CREATED

14 How to Create TTL s cont d Expire AT Certain Clock Time -application set - a date set in future by the application code - expireafterseconds set to ZERO combo type - set an expires_at date in application code - but apply a global type expiration policy with "expireafterseconds Messages documents set to expire 24 hours after creation

15 Changing Your TTL How to Modify your TTL Time Index must be created as a TTL index - -from the beginning {"expiresat" : 1 } alone!= a TTL index Must use the collmod command to change seconds Example increase to a 3 day TTL db.runcommand({"collmod" : "messages", "index" : { "keypattern" : {"expiresat" : 1 }, "expireafterseconds" : } })

16 Considerations and Gotchas Aggressiv e deletes Space Usage TTL Instance Isolation* usepowerof2siz es limits fragmentation No _Id field No nulls Driver issues No compounds Stale timestamps downstream clients language s timezon e

17 More Considerations and Gotchas -Very heavy writes/inserts - turn off TTL -Busy app? Hard to catch up (^7-15M inserts per day) -Version Diffs - 2.6^ ttl monitor, index builds* -TTLMonitor thread runs ~60 seconds -Background or Foreground Build implications* Replica sets, Primary cascade, thread runs on all Check oplog size increase for busy systems *

18 More Considerations and Gotchas Warning: The TTL index does not guarantee that expired data will be deleted immediately. Duration of removal operation depends on workload hitting your mongod. Q^ ww-dww-d Expired data may exist beyond the 60 second period between runs of the TTL monitor. - <=N + 60 top mongod deletes

19 Is My TTL Working Pre-check gather info, prepare, know what to look for -expect

20 TTL Working? cont d -Confirm ttl index exists AND is correctly defined -Define a variable to check with ex. ttltime -Set new Date for checking

21 MATH and Associated Queries : TTL Working? cont d *ISODate is in MICROSECONDS; TTL is in SECONDS* db.messages.find({expires_at: {$lt:isodate(" t03:11:00.013z") }}).count()

22 Other Associated Queries Adjust to expire docs after 1 day : set ttltime and adjust ex. mongos> ttltime = new Date(ISODate()+3600*1000) ISODate(" :04:35:34.936Z") If older than exact ttl exists, wait & re-query get oldest documents in a collection using the _id as the determiner

23 Questions to Ask How Aggressive? How Busy? Time or Space Crunch? #docs to delete? Reads vs Writes? Hardware and Architecture? Busy system => Slower = Safer = Less Performance Impact Lots to remove => greater impact System architecture => more smaller shards = more write scopes, distributes impact TIP: Use gettime turns ISODates into numbers aka milliseconds

24 THE MATH- Determine Starting TTL seconds Good Rule +20% buffer in TTL seconds = > Math.round(1.2 * )

25 Formula After Known Start Time Formula : SS -ES = Secs2Reduce /StepinSecs =#ofhops(iterations) SS =Starting Seconds ES =Ending Seconds Step = How many docs/secs you want to delete during per run Hop = # of times to iterate through reduction Goal: Desire 60.5 day TTL Oldest doc time = >Starting Secs : / = 65 days Goal TTL Secs time = >Ending Secs : = 86400*60.5 days Step : 60 secs Hops : aka 60 secs worth of docs at a time

26 Explanation SS =Starting Seconds ES =Ending Seconds Step = How many docs you want to delete during per run Hop = # of times to iterate through reduction Formula: Starting Secs - Ending Secs = Seconds to Reduce / Your step increment in seconds = # of Hops (# of times you will have to iterate through the reducer) ex. Starting Seconds Ending Seconds = Total Seconds needed to Reduce to your Deired TTL Time / 60 sec step increment = 6480 hops or iterations

27 db = db.getsiblingdb("kw_events"); var step=60; var index_info = db.system.indexes.findone({ns:"kw_events.events","expireafterseconds": {$exists:1}}); print("starting index TTL spot") printjson(index_info); var doc_count = db.events.count(); print("starting doc count") printjson(doc_count); var endingsecs= ; ##// var endingsecs=index_info.expireafterseconds - (6480*step); while ( index_info.expireafterseconds > endingsecs ){ var collmodobj ={collmod: "events", index: {keypattern: index_info.key, expireafterseconds: (index_info.expireafterseconds-step)} }; printjson(db.runcommand(collmodobj)); index_info = db.system.indexes.findone({ns:"kw_events.events","expireafterseconds": {$exists:1}}); sleep(20000); } print("ending index TTL spot"); printjson(index_info);

28 Explained in layman s Terms See sublime english-ified More examples next.. Beg, mid, end

29 TTL Reducer Tips -This is a Process Run in a screen!!! - small batch deletes allows other work - Create new ttl with 20% padded seconds to start - Monitor impact with mongostat - Minimize dblock% on Primary to a few seconds - Beginning script - high padded ttl time until deletes start more steps and a low hops/interval ex. s-2000, h-30 - Mid js Adjust # of seconds in step expect docs, minimize impact, run this one longer ex. s-500, h End js Be very specific in iterations impact already known, remaining known ex. S-500, h Be exact to not pass desired TTL

30 ttl_reducer_begin.js beginning -Start with a larger step (interval 2000 secs) and fewer hops/iterations (30) -High initial threshold (+20%) -might take time to 1 st start hitting the deletes

31 ttl_reducer_mid.js mid-way - Midway - do fewer steps(500 secs) and more hops (2930) - Caution: deletes may negatively impact performance

32 ttl_reducer_end.js end - At end, optimize step (seconds to be deleted) and # hops to reach desired permanent TTL expireafterseconds

33 A Mass Pre-TTL Delete Loop Useful MATH, Queries, and Loop for non-ttl based deletes Run the below query and delete loop for a 30 minute period defined by x below

34 Enabling / Disabling TTL Instance Wide -Single global setting so no need to drop all TTL indexes -Version 2.4.x + : TTLMonitor thread - enabled by default -Via a command-line parameter and/or server command. To disable at startup: setparameter ttlmonitorenabled=false #config file of the MONGOD's only setparameter=ttlmonitorenabled=false ## To turn it back on (just remove that line) To disable at runtime: on ALL replica set members - primary and secondary on secondaries - -- rs.slaveok() db.admincommand({setparameter:1, ttlmonitorenabled:false}) To re-enable when ready: db.admincommand({setparameter:1, ttlmonitorenabled:true})

35 Links - slideshare of this deck: blog on this topic will be up eventually here: free ObjectRocket MongoDB 30 day Trial Rackspace Managed Data Services - Hadoop, MySQL Contacts:

36 Links Slideshare: Gist of the ttl_reducer.js scriptlets : Gist of additional queries and examples : Contacts:

37 Questions?