talisker Documentation

Transcription

1 talisker Documentation Release Simon Davy Apr 18, 2018

2

3 Contents 1 Talisker - an opinionated WSGI app platform Quick Start Elevator Pitch Overview Goals FAQ Development using Talisker Devel Mode Development Logging Logging Configuring Logging Logger Class Root Handler Root Logger Limitations Debug Logging Log Format Log Suppression Additional logging configuration Gunicorn Logs Grok filters RSyslog Django Request Id Tracing Non-http Id Tracing Gunicorn Basic Usage Python 3.6, Async and Requests Sentry Error Data Sentry Configuration i

4 8 Status Endpoints Revision Requests Enhanced session Session lifecycle Using a talisker session Statsd Configuration Integration Testing Prometheus Installation Configuration Integration Celery Logging Metrics Sentry Flask Usage Sentry Details Django Logging Sentry Management Tasks Postgresql Query Security Slow query Logging Sentry Breadcrumbs Contract Logging Endpoints Statsd Sentry Contributing Types of Contributions Get Started! Pull Request Guidelines Credits Development Lead Contributors Release History ( ) ( ) ( ) ii

5 ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) Indices and tables 55 iii

6 iv

7 Contents: Contents 1

8 2 Contents

9 CHAPTER 1 Talisker - an opinionated WSGI app platform Talisker is an enhanced runtime for your WSGI application that aims to provide a common operational platform for your python microservices. It integrates with many standard python libraries to give you out-of-the-box logging, metrics, error reporting, status urls and more. 1.1 Quick Start Simply install Talisker via pip: pip install talisker And then run your WSGI app with Talisker (as if it was regular gunicorn).: talisker.gunicorn app:wsgi -c config.py... This gives you 80% of the benefits of Talisker: structured logging, metrics, sentry error handling, standardised status endpoints and more. 1.2 Elevator Pitch Talisker integrates and configures standard python libraries into a single tool, useful in both development and production. It provides: structured logging for stdlib logging module (with grok filter) gunicorn as a wsgi runner request id tracing standard status endpoints 3

10 statsd/prometheus metrics for incoming/outgoing http requests and more. deep sentry integration It also optionally supports the same level of logging/metrics/sentry integration for: celery workers general python scripts, like cron jobs or management tasks. Talisker is opinionated, and designed to be simple to use. As such, it is not currently very configurable. However, PR s are very welcome! For more information, see The Documentation, which should be found at: 4 Chapter 1. Talisker - an opinionated WSGI app platform

11 CHAPTER 2 Overview 2.1 Goals Talisker is designed with the following high level goals: to provide a standard platform for running wsgi apps to support both operational and developer workflows in the same tool to be easy as possible to integrate with any wsgi app and framework The original motivation was to standardise tooling across a number of different wsgi services, and improve the status quo along the way. Some of the inspiration came from Finagle, Twitter s service framework for the JVM, particularly the operational standardisation it provides. In particular, we wanted to standardise how we monitor applications in production, including logging, metrics, errors and alerts. Talisker provides a single library to do this for all our apps, that our operations tooling can configure easily, and means the application doesn t need to care about it. Also we wanted to provide best practice features for applications to use. So we added support for structured logging, integrated with the python stdlib logging. This allows developers to add custom tags to their logs, as well as add operational tags. We also provide easy to use helpers for best practice usage of things like statsd clients and requests sessions, which were used inconsistently across our projects, with repeated performance problems. 2.2 FAQ Some questions that have actually been asked, if not particularly frequently. 1. Why does talisker use a custom entry point? Wouldn t it be better to just be some WSGI middleware? There are 3 reasons for using a talisker specific entry point (a) to configure stdlib logging early, before any loggers are created (b) to allow for easy configuration of gunicorn for logging, statsd and other things. 5

12 (c) to automatically wrap your wsgi in app in some simple standard middleware, to provide request id tracing and other things. If it was just middleware, logging would get configured too late for gunicorn s logs to be affected, and you would need to add explicit middleware and config to your app and its gunicorn config. Doing it as an alternate entry point means you literally just switch out gunicorn for talisker, and you are good to go. 2. Why just gunicorn? Why not twistd, or waitress, etc? Simply because we use gunicorn currently. Integrating with other wsgi application runners is totally possible, just haven t done it yet, due to no need. Things like django channels and HTTP/2 might change that. 3. Why is it called talisker? WSGI sort of sounds like whisky if you say it quick. One of my favourite whiskies is Talisker, I ve even visited the distillery on the Isle of Skye. Also, Talisker is a heavily peated malt, which is not to everyone s taste, which seemed to fit thematically with a WSGI runtime that is also very opinionated and probably not to everyone s taste. Also, it has 8 characters just like gunicorn, and it wasn t taken on PyPI. 6 Chapter 2. Overview

13 CHAPTER 3 Development using Talisker Talisker has been designed with the goal of working in development and production. This is to try and encourage the same tool used throughout. Talisker s default configuration is designed for production usage, e.g.: only INFO level logs and above go to stderr python s warning system is disabled 3.1 Devel Mode If stderr is a tty, Talisker will run in devel mode. Additionally, if the DEVEL env var is set, it will run in devel mode. What this means varies on which tool you are using, but at a base level it enables python s warning logs, as you generally want these in development. For Gunicorn, devel mode means a few more things: enables access logs to stderr sets timeout to 99999, to avoid timeouts when debugging it enables auto reloading on code changes Also, for convenience, it you manually set Gunicorn s debug level to DEBUG, when in devel mode, Talisker will actually log debug level messages to stderr. 3.2 Development Logging Talisker logs have been designed to be readable in development. This includes: preserving the common first 4 fields in python logging for developer familiarity. 7

14 tags are rendered most-specific to least specific. This means that the tags a developer is interested in are likely first. if stderr is an interactive tty, then logs are colorized, to aid human reading. 8 Chapter 3. Development using Talisker

15 CHAPTER 4 Logging Talisker configures a specific carefully-designed logging set up. At a high level, it configures the following with the stdlib logging module: logger class that can collect structured data formatter that supports structured data via logfmt root handler that logs everything at INFO level to stderr Talisker also provides additional debugging options that can be used for more information. The stdlib logging module is old, based on a Java API, and makes heavy use of global state. Some of the implementation details for talisker are work arounds for these limitations. But stdlib logging is everywhere, and is actually very flexible, so we can make it work how we want to. 4.1 Configuring Logging If you use Talisker s entry points, like talisker.gunicorn or talisker.celery, then logging will be configured by default. You can also use the more generic entrypoints that talisker provides to run any script with Talisker logging enabled. talisker.run script.py... or python -m talisker script.py... If you are using your own entry point, you can configure Talisker with: import talisker talisker.initialise() Note that this should be done before any loggers are created via logging.getlogger() or similar, as they will not be altered. 9

16 This will set up logging, taking into account environment variables like DEBUGLOG. If you want to configure those parameters explcitly, you can do: config = {'devel': True, 'debuglog': '/path'} talisker.logs.configure(config) For testing, you can use special test configuration, which sets up talisker style logging, except adds a single NullHandler to the root logger: talisker.logs.configure_test_logging() This means logging will work as per talisker s set up, but you will get no log output. You can always add a logging.handlers.bufferinghandler temporarily to capture log messages in tests, e.g. for pytest: import def log(): handler = logging.handlers.bufferinghandler(10000) try: logs.add_talisker_handler(logging.notset, handler) yield handler.buffer finally: handler.flush() logging.getlogger().handlers.remove(handler) and use like so: def test_something(log): something() # log is a list of logging.logrecord items assert type(log[0]) is logging.logrecord 4.2 Logger Class Talisker sets a custom base logger class via logging.setloggerclass(). Its only difference from logger.logger is that it supports more explicitly storing extra arguments to the log call. This allows the StructuredFormatter class to append an arbitrary number of flags to the formatted message. Without this, there is no way to know which fields of a LogRecord are supposed to be added as tags. It also supports pulling in additional extra data from the current context, which is primarily used for providing request_id data for the log message. 4.3 Root Handler Talisker simply adds a handler to the root logger to log to stderr, at the INFO log level. Simple, self-contained application, no log file config No file permissions needed by app System handles buffering, synchronisation, persistence, rotation and shipping Works in development PaaS friendly 10 Chapter 4. Logging

17 4.4 Root Logger Limitations If you log messages against the root logger object (i.e. logging.getlogger()), then Talisker s structured logging enhancements will not work. This is due to the fact that the stdlib logging module instanciates the root logger object at import, and has no facility for safely switching it to be another instance or class. For libraries, it is not recommended practice to use the root logger, you should be specific about your library s top level logger anyway. A note about log levels Go read Dave Cheney s excellent post Let s talk about logging. Its focus is on golang logging, but is universally applicable. There are two intended users of logs: users and developers. In a WSGI service setting the user is someone in an operations role, trying to debug something in a production setting, where security and scale preclude logging everything. This is the INFO level. There is no need for anything more really (as argued in the post above), but this will of course include any logs at a higher level, as many libraries do use those levels. Anything going to stderr is designed to be shipped, so log with that in mind, regarding PII or secrets. Note, if you put sensitive information as an extra, then it s easier for your log shipping/aggregation tool to mask. But, perhaps it is better not to log it the first place, or only at DEBUG level? 4.5 Debug Logging Talisker also supports adding an additional root handler that logs to disk at DEBUG level. The stderr logging output is unchanged. To enable, just set the DEBUGLOG envvar to the path you want the log file to go to: DEBUGLOG=/path/to/logfile talisker... If talisker can open that file, it will add a handler to log to it at DEBUG level, and log a message at the start of your log output to say it is doing do. If it cannot open that file, it will log a message saying so, but not fail. The handler is a TimedRotatingFileHandler, set to 24 hour period with no backup copies, i.e. logs last for 24 hours at most. This is designed to support development and production use cases. In development, typically usage of DEBUG logs is by grepping a file, rather than viewing in the console, given the verbosity. So we write to disk where the developer has told us to, and they can grep/view the file there. In production, operators sometimes want to turn on more logging for limited period, to debug a specific problem. But we generally don t want to ship that extra logging. This is in part due to scaling - debug logs can be 10x more verbose than INFO, this could lead to a 10x traffic spike on your log aggregation service. Additionally, debug logs often include details that are sensitive, and that you don t want stored centrally. So this mechanism of writing to a temporary log file helps in that scenario too, as the INFO logging on stderr that is shipped is unchanged. 4.6 Log Format Talisker uses a default format that is designed to be human readable in development, but still structured for richer data Root Logger Limitations 11

18 Why hybrid format? Why not just use json in production, and text in dev? The motivation for the hybrid format is to have one format used in both development and production. This means when developers look at on-disk logs in production, they look familiar and are readable. This is as opposed to json or similar. Now, in actual production, this should be rare, as developers should really be using a log aggregation tool like Kibana to view the logs. However, we have found that when developing our infrastructure-as-code locally, we don t have a full ELK stack to process logs, so we have to fall back to on-disk logs on the actual machines to debug issues, so this feature is very useful then. The talisker logging format is as follows: format = '%(asctime)s.%(msecs)03dz %(levelname)s %(name)s "%(message)s"' datefmt = "%Y-%m-%d %H:%M:%S" which should look like this: :02:03.456Z INFO app "hello" This provides: the default data python logging usually has a more ISOish timestamp (uses. for msecs rather than, but we omit the T for readability) explicit UTC timestamps (logging module uses local time by default /o) explicitly quoted message (embedded are escaped) Talisker can also append an arbitrary number of tags on the end of the log line, following the logfmt idea. e.g.: :02:03.456Z INFO app "hello" foo=bar baz="some value" Defining logfmt logfmt is very loosely specified, and our target parser (logstash) has some limitations, so we define it as: keys: any string, except: values:, ``., and = are replaced by _, due to logstash limitations. " characters are stripped. always unquoted in log message. a python string, int, float or boolean strings are double quoted, with " stripped. Both keys and values are truncated after 2kb, to avoid accidental huge log messages. The reason for stripping characters in values is to do with the limitations of logstash s kv filter, which cannot currently cope with them, even when escaped. See issue 2 for more info. If this issue is fixed, talisker may in future escape characters in values rather than strip them. These extra tags can be specified in 2 main ways: 12 Chapter 4. Logging

19 1. By the developer at the call site: logger.info('something happened', extra={'foo': 'bar', 'context': 'I can haz it '}) would output:: :24:07.357Z INFO app "something happened" foo=bar, svc.context="i can haz it" 2. For a specific context, e.g. for a request. Talisker uses this to add request_id to every log message for a specific request. e.g.: logger.info('something happened') would output: :24:07.357Z INFO app "something happened" request_id=<request id> You can add your own temporary context variables with a context manager: with talisker.logs.logging_context(foo="bar"): logger.info('my important message') would output: :24:07.357Z INFO app "my important message" foo=bar Additionally, it would be expected that your log shipper should add additional tags, like hostname or service group, to the logfmt tags when shipping. If there are any global or context keys, these will take precedence if there is a collision with developer supplied keys. The developer keys will be suffixed with a _ to preserve the info, with out stomping on the other keys. 4.7 Log Suppression By default, talisker suppresses some loggers. The python python py.warnings logger is set not to propagate, as these are just noise in production. Additionally, talisker also configures the requests logger to WARNING level. This is because the INFO level on requests is particularly verbose, and we use requests everywhere. If you prefer to have full requests logs, you can simply set the level yourself. e.g.: logging.getlogger('requests').setlevel(logging.info) 4.8 Additional logging configuration Talisker just sets a root handler with formatter. You are free to add your own additional loggers and handlers as needed via the normal methods, if you need to Log Suppression 13

20 You can still benefit from the structured logging provided by talisker if you set your handler s formatter to be an instance of talisker.logs.structuredformatter. This is a standard formatter, except it uses UTC for the time and adds the logfmt tags on the end. The default format is as specified in Log Format. For example, suppose you want to enable debug logs for django s db logger. e.g: handler = logging.filehandler('db.log') handler.setformatter(talisker.logs.structuredformatter()) handler.setlevel(logging.debug) db = logging.getlogger('django.db.backends') db.setlevel(logging.debug) db.sethandler(handler) 4.9 Gunicorn Logs Gunicorn s error logs use talisker s logging setup. Gunicorn s access logs use the same format, but are disabled by default, as per gunicorn s defaults. The reasons for using the talisker format are: 1. Can use the same log shipping/aggregation (e.g. grok filter) 2. Can mix access logs and error logs in same stream. To enable access logs on stderr, with the error logs, use the normal gunicorn method: $ talisker --access-logfile=- To log to a file: $ talisker --access-logfile=/path/to/file Talisker overrides some config options for gunicorn, mainly to do with logging. It issues warnings if the user specifies any of these configs, as they will not be applied. Specifically, the following gunicorn config items are ignored by talisker: error-logfile/ log-file, as talisker logs everything to stderr logger-class, talisker uses its custom class statsd-host and statsd-port, as talisker uses the STATSD_DSN env var. If you run talisker.gunicorn in devel mode, and specify log-level=debug, it will output debug logs to the console Grok filters Talisker includes a filter and patterns for parsing the logformat into logstash with grok. These are in the talisker/logstash/ directory of the source tree. They are also included in the python package as resources RSyslog TODO 14 Chapter 4. Logging

21 4.12 Django TODO Django 15

22 16 Chapter 4. Logging

23 CHAPTER 5 Request Id Tracing A request id is used for log tracing. Talisker will use a request id provided in an request s X-Request-Id header, or generate one if needed. For that request, talisker will then: add in all log messages generated during that request add it as a response header include in outgoing HTTP requests using talisker requests Session in the WSGI environ ( REQUEST_ID ) in the raven error data (TODO) This is implemented using werkzeug.locals, to provide thread/greenlet local storage. 5.1 Non-http Id Tracing For other uses, you can use the following methods to inject a request id in logging messages: def get_id(arg, *args, **kwargs): return def my_task(arg, any, other='stuff'):... log.info('interesting thing') Or: def my_task(arg, any, other='stuff'): with request_id.context(arg.request_id):... log.info('interesting thing') 17

24 This is useful for background tasks like celery. 18 Chapter 5. Request Id Tracing

25 CHAPTER 6 Gunicorn 6.1 Basic Usage Gunicorn is the wsgi server used by Talisker. To use it, simply use the Talisker wrapper in place of regular gunicorn script. $ talisker.gunicorn -c config.py myapp:wsgi_app This wrapper simply initialises Talisker before passing control to Gunicorn s entrypoint. As such, it takes exactly the same command line arguments, and behaves in the same way. Talisker supports the sync, gevent, and eventlet workers. Others workers may work, but have not been tested. The only place it matters is in the context-local storage of Talsiker log tags and request ids. Talisker will use greenlet based contexts if it finds itself running in a greenlet context, or else a thread local object. 6.2 Python 3.6, Async and Requests Due to changes in the SSL api in python 3.6, requests currently has a bug with https endpoints in monkeypatched async context. The details are at but basically the monkeypatching must be done before requests is imported. Normally, this would not affect gunicorn, as your app would only import requests in a worker process after the monkeypatch has been applied. However, because Talisker enables some integrations in the main process, before the gunicorn code is run, it triggers this bug. Specfically, we import the raven library to get early error handling, and raven imports requests. We provide two special entrypoints to work around this problem, if you are using python 3.6 and eventlet or gevent workers with the requests library. They simply apply the appropriate monkeypatching first, before then just initialising talikser and running gunicorn as normal. $ talisker.gunicorn.eventlet --worker-class eventlet -c config.py myapp:wsgi_app $ talisker.gunicorn.gevent --worker-class gevent -c config.py myapp:wsgi_app 19

26 20 Chapter 6. Gunicorn

27 CHAPTER 7 Sentry Talisker provides out-of-the-box integration with sentry. Specifically, Talisker adds: some default configuration of the sentry client sentry wsgi middleware sentry error log handler (for logged exception messages) log message breadcrumbs (more breadcrumbs is a TODO) sentry integration with flask, django, and celery To get the current sentry client, simply use: talisker.sentry.get_client() 7.1 Error Data Talisker configures sentry breadcrumbs for log messages at INFO or higher level. It also adds the request id as a tag to the sentry message. If you want to add some custom error context, you can use the client above as you would use the sentry client as normal. For example: client = talisker.sentry.get_client() client.context.merge({'tags': my_tags}) 7.2 Sentry Configuration Talisker uses the default SENTRY_DSN env var to configure sentry by default. Simply setting this will enable sentry for wsgi and logging. 21

28 In addition, Talisker configures the sentry client by default as follows: sets install_logging_hook=false, as Talisker handles it sets release to the current Revision sets hook_libraries=[], disabling breadcrumbs for request/httplib sets environment to TALISKER_ENVIRONMENT envvar sets name to TALISKER_UNIT envvar sets site to TALISKER_DOMAIN envvar ensures the RemovePostDataProcessor, SanitizePasswordsProcessor, and RemoveStackLocalsProcessor processors are always included, to be safe by default. If you are using Talisker s Flask or Django integration, you can configure your sentry client further via the usual config methods for those frameworks. If you wish to manually configure the sentry client, use the following: talisker.sentry.configure_client(**config) This will reconfigure and reset the sentry client used by the wsgi middleware and logging handler that Talisker sets up. If you want to set your own client instance, do: talisker.sentry.set_client(client) Whichever way you wish to configure sentry, talisker will honour your configuration except for 2 things 1. install_logging_hook will always be set to false, or else you ll get duplicate exceptions logged to sentry. 2. the processors will always include the base 3, although you can add more. If you really need to remove the default processors, you can modify the list at talisker.sentry.default_processors and then call talisker.sentry.set_client(). 22 Chapter 7. Sentry

29 CHAPTER 8 Status Endpoints Talisker provides a set of app-agnostic standard endpoints for your app for querying its status. This is designed so that in production you have standard ways to investigate problems, and to configure load balancer health checks and nagios checks. /_status/ping A simple check designed for use with haproxy s httpcheck option, returns 200, responds to GET, HEAD, or OPTIONS, the body content being the application s Revision. /_status/check For use with nagios check_http plugin, or similar. It tries to hit /_status/check in your app. If that is not found, it just returns a 200, as a basic proxy for the application being up. /_status/test/sentry (/_status/error for backwards compatibility) Raise a test error, designed to test sentry/raven integration. /_status/test/statsd Send a test metric value. Designed to test statsd integration. /_status/test/prometheus Increment a test counter. Designed to test Prometheus integration. /_status/metrics Exposes prometheus metrics in Prometheus text format. /_status/info Return some useful information about server status (TODO). 8.1 Revision It is often useful to know what revision of your software is running, either for manual checking, or automatic deploy tooling. Talisker returns this in the body of a /_status/ping request, and also adds it to every response with the header X-VCS-Revision: Talisker does its best to figure out the revision of your code. It tries the following methods to discover the revision. output of git rev-parse HEAD output of bzr revno bzr version-info: versioninfo.version_info[ revno ] 23

30 output of hg id -i It falls back to the string unknown if none of the above work. To supply a custom revision, call the following in your startup code:: talisker.revision.set(my_revision) 24 Chapter 8. Status Endpoints

31 CHAPTER 9 Requests 9.1 Enhanced session Talisker provides a way to upgrade a request.session instance with a few extra features. Firstly, the X-Request-Id header will be added to the outgoing request headers. This can be used by other services to track the originating request id. We usually append incoming request id to one generated for that request, e.g.: X-Request-Id: <generated request id>-<incoming request id>. This allows simple searches to uncover all related sub requests for a specific request, also known as fanout. Secondly, we also add statsd metrics for outgoing requests durations. Each request will generate one timer statsd metric for request duration, with a name in the form:: <prefix>.requests.<hostname>.<method>.<status code> In the hostname, a. is replaced with a -. So an http request like this: session.post(' data=...) would result in the duration of that request in ms being sent to statsd with the following metric name: my.prefix.requests.somehost-com.post.200 You can customise the name of this metric in a few ways. Firstly, if you are using IP addresses in your urls, you can give them a human friendly name for the metric: talisker.requests.register_ip(' ', 'myhost') Now, a request to will emit a metric like so: my.prefix.requests.myhost.post

32 Talisker does not include the url path in the metric name by default, as it could be highly variable, and thus create too many distinct metrics. But you can optionally allow a number of path components to be included in the name: session.get(' metrics_path_len=2) will output a metric name: my.prefix.requests.myhost.some.url.post Session lifecycle We found many of our services were not using session objects properly, often creating/destroying them per-request, thus not benefiting from the default connection pooling provided by requests. This is especially painful for latency when your upstream services are https, as nearly all ours are. But sessions are not thread-safe (see this issue for details), sadly, so a global session is risky. So, talisker helps by providing a simple way to have thread local sessions. 9.3 Using a talisker session To get a base requests.session thread local session with metrics and request id tracing:: session = talisker.requests.get_session() or use the wsgi environ: session = environ['requests'] If you wish to use a custom subclass of Session rather than the default requests.session, just pass the session class as an argument. Talisker will ensure there is one instance of this session subclass per thread.: session = talisker.requests.get_session(mycustomsessionclass) This works because talisker does not subclass Session to add metrics or requests id tracing. Instead, it adds a response hook to the session object for metrics, and decorates the send method to inject the header (ugh, but I couldn t find a better way). If you wish to use talisker s enhancements, but not the lifecycle management, you can do: session = MySession() talisker.requests.configure(session) and session will now have metrics and id tracing. 26 Chapter 9. Requests

33 CHAPTER 10 Statsd Talisker provides statsd integration and configuration Configuration Statsd can be configured by the STATSD_DSN envvar, patterned after the SENTRY_DSN. This combines all statsd config into a single DSN url. For example:.. code-block:: bash # talk udp on port 1234 to host statsd, using a prefix of my.prefix STATSD_DSN=udp://statsd:1234/my.prefix # can also use / for prefix separators, the / converted to. STATSD_DSN=udp://statsd:1234/my/prefix # ipv6 STATSD_DSN=udp6://statsd:1234/my.prefix # custom max udp size of 1024 STATSD_DSN=udp://statsd:1234/my.prefix?maxudpsize=1024 Currently, only the udp statsd client is supported. If no config is provided, a dummy client is used that does nothing. TODO: contribute this to upstream statsd module 10.2 Integration If statsd is configured, talisker will configure gunicorn s statsd functionality to use it. Additionally, it will enable statsd metrics for talisker s requests sessions. Your app code can get a statsd client by simply calling:: statsd = talisker.statsd.get_client() 27

34 Note: this next section is still WIP. Document intended usage, following DDD. Additionally, talisker supports a more efficient statsd flushing mechanism: pipelines. It creates a statsd pipeline per request, and puts in the wsgi environment. This pipeline client can be used by anything during that request, and is then flushed when the request is completed. This results in fewer udp packets being sent, as several can be packed in to the same 512 byte packet. See the statsd module s pipeline documentation for more information Testing To assist in testing, the dummy client can temporarily collect metrics for checking against: statsd = talisker.statsd.get_client() with statsd.collect() as metrics: do_something() assert metrics[0] == 'test:1 c' 28 Chapter 10. Statsd

35 CHAPTER 11 Prometheus Talisker provides optional prometheus_client integration and configuration Installation The package supports extras args to install prometheus_client: $ pip install talisker[prometheus] 11.2 Configuration prometheus_client integration needs no extra configuration if running in single-process mode. If the app is running in multiprocess mode (ie. with multiple workers), the prometheus_multiproc_dir envvar should be set to a writable directory that can be used for metrics. If this envvar is not found, a temporary directory will be created and used instead (but note that metrics will be reset on restarts). A default worker_exit server hook is automatically set up for cleaning up the shared prometheus directory as well. Note that this can be overridden by user supplied configuration, and in that case the custom worker_exit function should perform the cleanup itself Integration If prometheus_client is installed, Talisker will expose metrics collected by the app using the default registry. Custom registries are only supported in multiprocess mode. Metrics are available at /_status/metrics in Prometheus text format. 29

36 30 Chapter 11. Prometheus

37 CHAPTER 12 Celery Talisker provides some optional integration with celery. If your project does not have celery installed, this will not be used. To run a Celery worker with Talisker, use the provided Celery entrypoint: $ talisker.celery worker -A myapp 12.1 Logging Talisker configures Celery logging to use Talisker s logging mechanisms. It ensures that the task_id and task_name will be logged with every log message for a Celery job. Additionally, if the job is triggered by a Talisker process (e.g. a Talisker gunicorn worker) it will add the request_id to the logging tags for the celery job when it executes. This allows you to track jobs initiated by a specific request id Metrics Talisker will enable basic celery task metrics by default. Talisker sets up statsd timers for celery.<task_name>.enqueue (time to publish to queue) celery.<task_name>.run (time to run task) And statsd counters for celery.<task_name>.retry celery.<task_name>.success celery.<task_name>.failure celery.<task_name>.revoked 31

38 Note: talisker supports celery>= dependencies: If you need to be sure, the package supports extras args to install celery $ pip install talisker[celery] 12.3 Sentry Talisker integrates Sentry with Celery, so Celery exceptions will be reported to Sentry. This uses the standard support in raven for integrating Celery and Sentry. 32 Chapter 12. Celery

39 CHAPTER 13 Flask 13.1 Usage Talisker provides some opt-in support for flask apps. This does two main things currently. 1. enable sentry flask support for your app. This means you will get more information in your sentry errors, as well as being able to configure sentry via your app config as normal. 2. disable flask default app logger configuration, and just use talisker s configuration. This avoids double-logged exception messages. To enable, you can either use a special Talisker flask app: app = talisker.flask.taliskerapp( name ) or register your app with Talisker afterwards: talisker.flask.register(app) 13.2 Sentry Details Talisker integrates the flask support in raven.contrib.flask. See the raven flask documentation for more details. The sentry flask extension is configured to work with talisker. logging=false as Talisker has already set this up. This means the other possible logging config is ignored. wrap_wsgi=false as Talisker has already set this up register_signal=true, which is the default If for some reason you wish to configure the flask sentry extension yourself: talisker.flask.sentry(app, **config) 33

40 This has the same api as the default raven.contrib.flask.sentry object, but with the above configuration. 34 Chapter 13. Flask

41 CHAPTER 14 Django Talisker provides opt-in support for django apps Logging To integrate with Talisker, you should at a minimum disable django s default logging in your settings.py: LOGGING_CONFIG = None If you don t, you ll get Django s default logging configuration in addition to Talisker s, leading to some duplicated logs in development, and possibly s of errors in production, which is often not a good idea. If you have custom logging you want to add on top of Talisker s, you can follow the Django documentation for configuring logging yourself, with something like: LOGGING = {...} LOGGING_CONFIG = None import logging.config logging.config.dictconfig(logging) which is excactly what Django does, but without the default logging Sentry To integrate with Talisker s sentry support, add raven to INSTALLED_APPS as normal, and also set SEN- TRY_CLIENT in your settings.py: INSTALLED_APPS = [ 'raven.contrib.django.raven_compat',... (continues on next page) 35

42 ] SENTRY_CLIENT = 'talisker.django.sentryclient' (continued from previous page) This will ensure the extra info Talsiker adds to Sentry messages will be applied, and also that the WSGI and logging handlers will use your Sentry configuration in settings.py. It will also set install_sql_hook=false, as that leaks raw SQL to the sentry server for every query. This will hopefully be addressed in a future release Management Tasks If you use management tasks, and want them to run with Talisker logging, you can use the generic talisker runner: talisker.run manage.py... or python -m talisker manage.py Chapter 14. Django

43 CHAPTER 15 Postgresql Talisker provides some optional integration with postgresql via pyscopg2, in the form of a custom connection/cursor implementation that integrates with logging and with sentry breadcrumbs. Ensure you have the correct dependencies by using the pg extra: pip install talisker[pg] To use it in Django: DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql',..., 'OPTIONS': { 'connection_factory': talisker.postgresql.taliskerconnection } } } To use with sqlalchemy: engine = sqlalchemy.create_engine(..., connect_args={'connection_factory': talisker.postgresql.taliskerconnection}, ) 15.1 Query Security Given the security sensitive nature of raw sql queries, Talisker is very cautious about what it might communicate externally, either via logs or via sentry. 37

44 Talisker will only ever log the query string with placeholders, and never the query parameters. This avoids leakage of sensitive information altogether, while still providing enough info to be useful to users trying to debug problems. If a query does not use query parameter, the query string is not sent, as there is no way to determine if it is sensitive or not. One exception to this is stored procedures with parameters. The only access to the query is via the raw query that was actually run, which has already merged the query parameters, so we never send the raw query. Note: in the future, we plan to add support for customised query sanitizing Slow query Logging The connection logs slow queries to the talisker.slowqueries logger. The default timeout is -1, which disables slow query logging, but can be controlled with the TALISKER_SLOWQUERY_THRESHOLD env var. If DEVEL envvar is set, the default is 0, and Talisker will log every query Sentry Breadcrumbs Talisker will capture all queries run during a request or other context as Sentry breadcrumbs. In the case of an error, they will include them in the report to sentry. 38 Chapter 15. Postgresql

45 CHAPTER 16 Contract There is nothing particularly Python-specific about what Talisker offers. The basic contract of its behaviour and configuration could be provided in other languages as well. This contract outlines the basic requirements of an implementation in order to be considered compliant. Note: this section of documentation is under development Logging The DEBUGLOG environment variable tells Talisker where to write the debug log file. 1. Log format: FORMAT = '%(asctime)s.%(msecs)03dz %(levelname)s %(name)s "%(message)s"' DATEFMT = "%Y-%m-%d %H:%M:%S" Where message is the formatted log message, with any extra parameters appended as key=value pairs. The resulting log entry should look like this: :02:03.456Z INFO app "hello" key=value key2=value2 2. Access logs to stderr: :54:14.124Z INFO gunicorn.access "GET /" method=get path=/ qs= status=400 ip= proto=http/1.1 length=121 referrer=none ua=curl/ duration= request_id=00cf39ce-47a2-402d d2fd Error logs to stderr: :59:18.237Z ERROR gunicorn.error "Error handling request /_status/ error" request_id=5baf01d a734-fbcdbf7b8e10 Traceback (most recent call last): File "/home/talisker/src/talisker-service/env/lib/python3.4/site-packages/ gunicorn/workers/sync.py", line 135, in handle (continues on next page) 39

46 (continued from previous page) self.handle_request(listener, req, client, addr) File "/home/talisker/src/talisker-service/env/lib/python3.4/site-packages/ gunicorn/workers/sync.py", line 176, in handle_request respiter = self.wsgi(environ, resp.start_response) File "/home/talisker/src/talisker-service/env/lib/python3.4/site-packages/ werkzeug/local.py", line 228, in application return ClosingIterator(app(environ, start_response), self.cleanup) File "/home/talisker/src/talisker-service/env/lib/python3.4/site-packages/ talisker/wsgi.py", line 52, in middleware return app(environ, custom_start_response) File "/home/talisker/src/talisker-service/env/lib/python3.4/site-packages/ talisker/request_id.py", line 105, in call return self.app(environ, add_id_header) File "/home/talisker/src/talisker-service/env/lib/python3.4/site-packages/ talisker/wsgi.py", line 42, in middleware return app(environ, start_response) File "/home/talisker/src/talisker-service/env/lib/python3.4/site-packages/ talisker/endpoints.py", line 110, in call response = func(request) File "/home/talisker/src/talisker-service/env/lib/python3.4/site-packages/ talisker/endpoints.py", line 71, in wrapper return f(self, request) File "/home/talisker/src/talisker-service/env/lib/python3.4/site-packages/ talisker/endpoints.py", line 161, in error raise TestException('this is a test, ignore') talisker.endpoints.testexception: this is a test, ignore 4. Generates a request ID if one isn t in the X-Request-ID header in the incoming request Endpoints 1. /_status/ping A very simple server status check, to determine if the service is actually running. An HTTP 200 response indicates success. 2. /_status/check A more thorough status check, to determine if the service is healthy. This is intended to be consumed by something like Nagios to issue alerts in the event of failures. Things to check here might include database connectivity, disk space and memory utilisation. An HTTP 200 response indicates that the service is healthy. 3. /_status/test/sentry (/_status/error for backwards compatibility) Generate an error that can be traced in the logs and error reporting backend. 4. /_status/test/statsd Generate a metric that can be traced in a statsd collector such as graphite or grafana. 5. /_status/test/prometheus Increment a test counter. Designed to test Prometheus integration. 6. /_status/metrics Expose prometheus metrics in Prometheus text format. 40 Chapter 16. Contract

47 7. /_status/info Service info? 16.3 Statsd Integration with statsd may be configured using the environment variable STATSD_DSN, which should be a standard URL, e.g.: udp://statsd-host:8125/prefix 16.4 Sentry Configured using the environment variable SENTRY_DSN Statsd 41

48 42 Chapter 16. Contract

49 CHAPTER 17 Contributing Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given. You can contribute in many ways: 17.1 Types of Contributions Report Bugs Report bugs at If you are reporting a bug, please include: Your operating system name and version. Any details about your local setup that might be helpful in troubleshooting. Detailed steps to reproduce the bug Fix Bugs Look through the GitHub issues for bugs. Anything tagged with bug is open to whoever wants to implement it Implement Features Look through the GitHub issues for features. Anything tagged with feature is open to whoever wants to implement it. 43

50 Write Documentation talisker could always use more documentation, whether as part of the official talisker docs, in docstrings, or even on the web in blog posts, articles, and such Submit Feedback The best way to send feedback is to file an issue at If you are proposing a feature: Explain in detail how it would work. Keep the scope as narrow as possible, to make it easier to implement. Remember that this is a volunteer-driven project, and that contributions are welcome :) 17.2 Get Started! Ready to contribute? Here s how to set up talisker for local development. 1. Fork the talisker repo on GitHub. 2. Clone your fork locally: $ git clone git@github.com:your_name_here/talisker.git 3. run make with no arguments. This will create a virtualenv env in.env and install dependencies and run tests. 4. Create a branch for local development: $ git checkout -b name-of-your-bugfix-or-feature Now you can make your changes locally. 5. When you re done making changes, check that your changes pass flake8 and the tests, including testing other Python versions with tox: $ make # runs python3 tests and lint $ make tox # runs all python tests and lint 6. Commit your changes and push your branch to GitHub: $ git add. $ git commit -m "Your detailed description of your changes." $ git push origin name-of-your-bugfix-or-feature 7. Submit a pull request through the GitHub website Pull Request Guidelines Before you submit a pull request, check that it meets these guidelines: 1. The pull request should include tests. 44 Chapter 17. Contributing

51 2. If the pull request adds functionality, the docs should be updated. Put your new functionality into a function with a docstring, and add the feature to the list in README.rst. 3. The pull request should work for Python 2.7, 3.4 and 3.5, and for PyPy. Check talisker/pull_requests and make sure that the tests pass for all supported Python versions Pull Request Guidelines 45

52 46 Chapter 17. Contributing

53 CHAPTER 18 Credits 18.1 Development Lead Simon Davy 18.2 Contributors None yet. Why not be the first? 47

54 48 Chapter 18. Credits

55 CHAPTER 19 Release History ( ) The main work in this release has been improvments to metrics. Collect prometheus metrics as well as statsd for gunicorn, requests, and celery (#172) Support flask/django adding X-View-Name header to indicate view function (#151) Control over naming requests metrics (#171) Gunicorn logging enhancements (#165) Gather better metadata from OSError exceptions Fixed some small logging issues ( ) The logfmt output has been reworked to explictly quote strings, and test coverage much improved in the process. This allows for more robust parsing in logstash, such as allowing numeric fields. New talisker.testing module, which has helpers for functionally testing talisker servers and related talisker tools. Added a functional test suite using the new talisker.testing helpers Custom ruby logstash filter to handle parsing of numeric values and escaped quotes ( ) add support for postgresql via psycopg2 (#85). This will add breadcrumbs to sentry reports, as slow query logs. See for more info Access log cleanups (#94). We no longer include the querystring in the logmsg, just as a field. 49