Deep Dive into Logging and Error Handling. Miro Dietiker (miro_dietiker) Eric Peterson (iameap)

Transcription

1 Deep Dive into Logging and Error Handling By: Miro Dietiker (miro_dietiker) Eric Peterson (iameap)

2 WSOD & friends How to know when your kittens just die()d.

3 PHP Exception handler set_exception_handler( callable $exception_handler ) throw new Exception('Sorry kitty'); Catches uncaught exceptions Stacktraces available _drupal_decode_exception()

4 All the PHP Errors E_ERROR E_PARSE E_NOTICE E_CORE_ERROR, E_CORE_WARNING E_COMPILE_ERROR, E_COMPILE_WARNING E_USER_ERROR, E_USER_WARNING, E_USER_NOTICE E_STRICT E_RECOVERABLE_ERROR E_DEPRECATED, E_USER_DEPRECATED

5 PHP Error handler set_error_handler( callable $error_handler [, int $error_types = E_ALL E_STRICT ] ) trigger_error('3 kittens killed'); Fired on PHP error Stacktraces available Can't deal with E_ERROR, E_PARSE, E_CORE_ERROR, E_CORE_WARNING, E_COMPILE_ERROR, E_COMPILE_WARNING

6 Devel module Improved error handler admin/config/development/devel Krumo backtrace Development only!

7 PHP Shutdown function register_shutdown_function ( callable $callback ); Captures E_ERROR + friends (Non-existing functions,...) Missing in Drupal 7! No backtrace, sorry!

8 PHP log apache error.log E_PARSE Backtraces With xdebug

9 Code fails drupal_set_message('sad', 'error'); Remaining invisible errors No error thrown No watchdog

10 Past (Log) module Log interface, with structured data Exception handler With backtrace hook_watchdog With backtrace Shutdown handler Error log grabber Alternative storage backends

11 Conclusion error_reporting(e_all); settings.php: $conf['error_level'] = 2; Enable verbosity admin/config/development/logging Get rid off all errors Check logs in production... Shutdown handler for Drupal 8

12 New Relic Show me the truth

13

14 Architecture The PHP plugin Agent SaaS UI The agent The SaaS UI Apache PHP plugin Drupal

15 Reporting features Overall statistics Aggregated values Execution Time (CPU / Waiting) Specific period in time Slow traces Deep details about a single call Error reporting Fatal only

16 Reporting perspectives Web transactions (foreground) Functions SQL queries External services Drupal (modules) Background tasks

17 Deeper integration Drupal Module Exceptions Watchdog Background detection (cron, drush) Deployments

18 Demo Let's play...

19 Logging, Drupal and Beyond Eric Peterson (iameap)

20 Goals 1. Understand the fundamentals of logging in a generic sense. 2. Assess Drupal's system and access log implementations and their weaknesses. 3. Become familiar with log management software and the problems it solves. 4. Learn how to solve real-world problems with Splunk.

21 What is logging, why do we do it? Logging is the process of recording events in order to provide an audit trail that can be used to understand system activity and diagnose problems. For example, an apache access log entry: [23/Jun/2013:12:07: ] "GET /robots.txt HTTP/1.1" "" "Googlebot/2.1"

22 Traditional log creation The IETF maintains the "syslog" standard, including facilities, severities, etc. Facilities indicate program type. Standard severity levels: "Do I Notice When Evenings Come Around Early" For example, a UNIX tool for logging: logger -p local7.notice -t teapot "I'm a little teapot."

23 Traditional log management Configure log routing based upon facility, severity. Route to: file, user, host, program. Commands for rotating, archiving, storing, deleting stale logs. # Sample syslog.conf # Send all local7 to file. local7.* /var/log/local7.log # Send all alerts via *.alert my_ _script # Sample logrotate.conf /var/log/local7.log { size 1m create 700 root root rotate 4 compress }

24 Traditional log analysis What are the top user agents visiting my site? awk -F\" '{print $6}' /var/log/httpd/access.log sort uniq -c sort -fr 633 Mozilla/5.0 (compatible; Ezooms/1.0; 287 Mozilla/5.0 (compatible; Googlebot/2.1; + html) 266 Mozilla/5.0 (compatible; AhrefsBot/4.0; Amazon CloudFront tail -f /var/log/httpd/access.log

25 Drupal system log creation watchdog('type', 'MESSAGE: %message_var', array('%message_var => 'MESSAGE VAR VALUE'), WATCHDOG_NOTICE); Watchdog API and syslog are very similar. Severity levels, able to specify type/application. Default enabled "Database Logging" module writes all data to the watchdog database table. hook_watchdog provides flexibility in routing log messages in custom ways.

26 Drupal access log creation Functionality contained within the Core Statistics module; disabled by default. When enabled, writes all accesses to the accesslog database table. No API.

27 Drupal log management Drupal cron "rotates" (deletes) stale data; configure the number of records to keep.

28 Drupal system log analysis /admin/reports/dblog Filterable by type or severity. Click out to an individual log event

29 Drupal access log analysis /admin/reports/hits Additional "one-off" reports for top visitors, pages, referrers.

30 Problems with log data analysis It can be extremely cumbersome to track down the relevant log entry; let alone the root cause. What about searching? What about filtering by user or IP or date or path? We can't see IP or path on the overview? A paged view is terrible when log events flow in copiously. Still, Views exposed filters were built for end-user interactions, not deep data analysis.

31 Problems alerting/reporting on logs Why didn't we catch this problem earlier? Still, this only helps in alerting on specific events. What about alerts based on aggregates (e.g. count thresholds, averages, etc)?

32 Problems with log rotation/retention Didn't we have this same problem last week? (DBlog Selective Cron) Configuring this on medium-to-large sites is very cumbersome (what types and severities even exist, let alone are useful to you?) Still no archiving.

33 Log performance/scalability problems Multi-host sites: accesslog/watchdog quickly force the DB into the biggest bottleneck. Shared hosting: latency and/or overloaded DB server can be a performance killer. Try an alternate backend: Write logs to syslog with core Syslog module.

34 Log management and OI There's no clear strategic advantage to using Drupal as your log management platform. Why not use something else more fully featured and built specifically for the purpose? Splunk Loggly Papertrail Logstash (a FOSS alternative/complement)

35 Connecting Drupal + Syslog + Splunk 1. Enable syslog (optionally, Syslog Access) 2. Configure facility, application tag, message format 3. Add the following to syslog/rsyslog.conf, restart. 4. Configure Splunk to accept events from your host. 5. Once verified, disable database logging. Other configs possible (log to file, Splunk's universal forwarder, API calls...

36 Splunk Search Fundamentals

37 Splunk Fields

38 Splunk Field Extraction In many cases, splunk automatically extracts "interesting fields" based on message syntax You can also extract custom fields using regex, or by selecting in the Splunk GUI

39 Splunk Commands & IO Redirection Similar to a command line, you can pipe Splunk search results to commands. process="drupal" eval userstatus=(if(uid=0, "Anonymous", "Authenticated")) process="drupal" eval userstatus=(if(uid=0, "Anonymous", "Authenticated")) chart count by userstatus sort count

40 Splunk Visualization Basics process="drupal" eval userstatus=(if(uid=0, "Anonymous", "Authenticated")) chart count by userstatus

41 Splunk Visualization (cont) Analyzing measures over time with timechart process="drupal-access" timechart min(timer) as "Min", avg(timer) as "Avg", max(timer) as "Max"

42 Splunk Visualization (cont) 100% stacked area charts process="drupal-access" timechart count by cache_status

43 Splunk Reporting [XML of the above dashboard]

44 Splunk Alerting

45 A few ideas for inspiration Find the top occurring PHP errors Alert when cron hasn't run recently Track service outages in 3rd party APIs. Find all pages/urls referring to a given page Detect code sync issues in a multi-host setup Monitor errors, memory usage, etc. during deployment. Trace events chronologically for a given IP, user. DOS or infinite redirect from IE?

46 Conclusion, questions Don't use Drupal as your log management platform; there is no advantage. Whatever solution you do use, be aware of its analysis and reporting capabilities. You can better understand your log data by visualizing it; when reporting, be aware of what you're communicating and how. You know your application best.