LOGGING WITH. Alex Stupka, Developer

Transcription

1 LOGGING WITH Alex Stupka, Developer 1.Introduction 1.Developer at Customink 2.If you have any questions, feel free to stop me and ask

2 CUSTOMINK USAGE Products and Services, EComm, and Operations Technology are putting finishing touches on a Catalog service which uses Mongoid in place of an RDBMS & ActiveRecord. Operations Technology uses the Central Logger in production for all back end services and order processing. Mongoid is used on Operations Technology s Supplier and Decorator Service for tracking supplier information/blanks inventory and printer/vendor data respectively. Catalog manages products, styles clothing, and other items we sell. The primary functions of the supplier service are to: store general supplier information store blanks inventory data (e.g. availability and pricing) support automated blanks ordering support ordering logistics Decorator Service This service maintains all relevant data to describe decorator's (aka printer/vendor). Vendor's details Capacity Supply inventory Capabilities

3 PROBLEM Complex Production Environments Orchestrated Services Need Hoptoad Complementary Auditing 1.Problem 1.Load balanced applications 2.Orchestrated services

4 LOGGING OPTIONS Network File System Syslog Database Hoptoad 1. Solutions 1. Network File System logs 2. Syslog 1. File based 3. Database 1. Relational 4. Hoptoad 1. Exceptions 2. Throttles logging

5 WHY MONGO? Introduces the Company to MongoDB Capped Collections are Great Easy to Add New Fields Easy to Install and Deploy Lightweight Development 1. Good way to introduce developers and the company to Mongo 1. not-mission critical. Logs are teed to file as backup 2. model is tinker-friendly 2. Capped collections 1. emulate rolling logs 2. Never get larger than their configured size 3. natural ordering 4. fast, can handle large amount of audit data 3. Easy to add new fields 4. Easy to install and deploy

6 MONGO LOGGER Phil Burrows Original idea & code

7 CENTRAL LOGGER Subclasses BufferedLogger It s a Gem, Loaded as a Railtie Captures Useful Metadata 1. Subclasses the Buffered Logger in Rails in order to tee logs to a mongo database 2. Its a gem 1. Works with Rails to Rail-Tie (replaces the buffered logger during rails boot) 3. Works with Heroku by using separate config 4. database.yml 5. central_logger.yml 3. default captures metadata you re probably interested in 1. fields (log message, request_time, params, application, controller, elapsed execution time)

8 RECORD STRUCTURE Go over individual records

9 ACTUAL DATA Go over individual elements

10 INSTALLATION gem install central-logger Update ApplicationController Create Config File Estimate the Size of Your Capped Collection 1. gem install 2. Update ApplicationController 1. uses around_filter to audit all logging that occurs in an action 3. config file 1. database.yml - if using RDBMS 2. mongoid.yml - for ODM 3. central_logger.yml 4. Do some work to figure out how large your capped collection needs to be. 1. How many days of log data do you want? 2. count number of apps, current size of logs divided by days (assuming current load is normal) and multiply by number of machines for each app and the number of days you need 3. May want index for tail functionality 1. index uses gt to do this 2. link to tail article

11 APPLICATION CONTROLLER Module that goes around

12 Mongo key in your db yaml or central logger yml or mongoid yml CONFIG FILE

13 AROUND FILTER Benchmark call in around filter wraps call to the controller Exceptions are logged Insert always attempted

14 IS IT WORKING? 1. rails console - look at type of Rails.logger 2. Go to an action you want to log, such as a listing page that extends ActionController 3. mongo console 1. use system_log (or whatever database you specified 2. show collections 3. db.development_log.count() 4. Falls back to Buffered Logger if logging isn t working.

15 EXTRAS Replica Set Support Mongo Auth 1. Replica set support - 1. in case your logs are extra important 2. Mongo Auth 1. Mongo recommends no auth, and its just basic 2. should use inside trusted network or over vpn (secure at other network layer)

16 GOTCHAS Beware of Unserializable Types Exceptions will have Central Logger in the call stack 1. Beware of buried BSON unserializable data types in your log statements 1. Will add this feature 2. Exceptions will usually have central_logger in the stack trace because of the around filter

17 CENTRAL LOG VIEWER Need Flattened Log Messages Need Log Messages Across Servers Grep Tail -f Functionality 1. Central Log Viewer 1. Want to see flattened log messages inside a document rooted at the action 2. Need to see log messages orchestrated across multiple services 3. Grep 4. Tail -f while making request

18 ARCHITECTURE Trivial Rails Controller JQuery Handlebars CoffeeScript rails controller uses eval (oh no)! no ODM or ORM installed, just jason returned to the client Handlebars - Templating DSL based on Mustache, but runs on client Coffeescript - concise language that eliminates complexity and ceremony of JS

19 INSTALLATION Just a Rails App Modify Config File Add Indexes 1. Currently just a rails app piggybacking off the central_logger 2. Change the config file to point to your central logger 3. Add indexes

20 HANDLEBARS relative path specified in JSON document rooted at action compile step - creates an AST consisting of jquery functions that evaluate incoming JSON

21 1. Quick and dirty query generation 2. find logs for particular apps 3. Grep through messages 4. Tail FIRE IT UP

22 Images

23 QUESTIONS? Alex Stupka twitter: astupka

24 THANKS!