CID Documentation Release 0.2.0 Francis Reyes Sep 30, 2017
Contents 1 Django Correlation IDs 1 1.1 Features.................................................. 1 Python Module Index 9 i
ii
CHAPTER 1 Django Correlation IDs Logging is important. Anyone who has had a call at 3am to say the site is down knows this. Without quality logging it is almost impossible to work out what on earth is happening. Even with plenty of logs it can be hard to track down exactly what the effects of a particular request are. Enter Django Correlation IDs. The approach is quite a simple one. Incoming requests are assigned a unique id (a uuid). This can either happen in say your public facing web server (e.g. nginx) or be applied as soon as it hits django. This cid is then available throught the django request/response cycle. We provide filters for logging witch adds the cid to the logging record so you can add it to your formatting string. We also provide wrappers around all the standard database backends which adds the cid as a comment before each SQL request. Features Processing/Generation of a correlation id Database wrappers to add correlation id to each sql call Logging filter to inject the correlation id into logs A template context processe to make correlation id available in templates Output correlation id as a header Documentation can be found at: http://django-correlation-id.readthedocs.org/ Contents: Installation At the command line: $ pip install django-cid You will need to add cid to your list of installed apps. 1
INSTALLED_APPS = ( # some apps 'cid', # some other apps ) For more information on how to actually make use of Django Correlation Id see the Usage section. Correlation ID Generation Django correlation id can handle the generation of correlation id in a couple of different ways. Either internally or from an incoming header. To use the internal generator you will need to set CID_GENERATE to true in you settings file: CID_GENERATE = True This approach is perfectly acceptable but does suffer one drawback. If you host your django application behind another web server such as nginx, then the nginx logs won t contain the correlation id. Django Correlation Id can handle this situation by extracting a correlation id created in nginx and passed through as a header in the HTTP request. By default Django Correlation Id will look for the header X_CORRELATION_ID, but you can change this with the CID_HEADER settings. A simple way to generate the correlation id in nginx is to use one of the available built in scripting languages. e.g. the embedded perl. One Ubuntu and Debian systems this requires the installation of nginx-extras and libossp-uuid-perl. sudo apt-get install nginx-extras libossp-uuid-perl And then a couple of configuration changes. in /etc/nginx.conf you will need to add the following within the http section: http { perl_require "Data/UUID.pm"; perl_set $uuid 'sub { $ug = new Data::UUID; $str = $ug->create_str(); return $str; }'; } # Remaining http config This tell nginx to generate uuids. The next step is to assign them to your proxy or uwsgi pass through for you application. This is done within a server block of your site configuration server { proxy_set_header X-Correlation-Id $uuid; } # Remainder of site config 2 Chapter 1. Django Correlation IDs
Usage Django Correlation Ids have a number of options for usage covering logging, SQL comments, and a template context processor. All of these rely on the usage of a piece of middleware. To configure the middleware simply add the following to your list of MIDDLEWARE_CLASSES in your settings file: MIDDLEWARE_CLASSES = ( 'cid.middleware.cidmiddleware', # other middleware ) By default the middleware will look for the correlation id in an incoming request header called X_CORRELATION_ID. An alternative header name can be used by putting a value into settings for the CID_HEADER. e.g.: CID_HEADER = 'X_MY_CID_HEADER' Note: Most WSGI implementations sanitize HTTP headers by appending an HTTP_ prefix and replacing - by _. For example, an incoming X-Correlation-Id header would be available as HTTP_X_CORRELATION_ID in Django. When using such a WSGI server in front of Django, the latter, sanitized value should be used in the settings. You can also configure Django Correlation Id to generate its own correlation id if one is not found in the header. For this set CID_GENERATE to true in your settings file: CID_GENERATE = True By default, Django Correlation Id sets an HTTP header in the HTTP response with the same name as configured in CID_HEADER. You may customize it with CID_RESPONSE_HEADER in the settings: CID_RESPONSE_HEADER = 'X-Something-Completely-Different' Note: As indicated in the note above, if Django is behind a WSGI server that sanitizes HTTP headers, you need to customize CID_RESPONSE_HEADER to have the same header name in the response as in the request. # The header is ``X-Correlation-Id`` but is sanitized by the WSGI server. CID_HEADER = 'HTTP_X_CORRELATION_ID' # Don't use the default value (equal to CID_HEADER) for the response header. CID_RESPONSE_HEADER = 'X-Correlation-Id' If you don t want the header to appear in the HTTP response, you must explicitly set CID_RESPONSE_HEADER to None. # Don't include the header in the HTTP response. CID_RESPONSE_HEADER = None SQL Comments To make use of the SQL comments support you will need to change your database backend to one of the cid wrapped database backends. e.g. for sqlite3 you will need to use the following: 1.1. Features 3
DATABASES = { 'default': { 'ENGINE': 'cid.backends.sqlite3', 'NAME': location('db.sqlite3'), } } The are database backend wrappers for all the currently support database backends found in Django. mysql cid.backends.mysql oracle cid.backends.oracle postgresql cid.backends.postgresql sqlite3 cid.backends.sqlite3 Logging To make use of the correlation id in logs you will need to add the cid log filter to your settings and apply it to each logger. e.g. LOGGING = { 'version': 1, 'disable_existing_loggers': False, 'formatters': { 'verbose': { 'format': '[cid: %(cid)s] %(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s' 'simple': { 'format': '[cid: %(cid)s] %(levelname)s %(message)s' 'handlers': { 'file': { 'level': 'DEBUG', 'class': 'logging.filehandler', 'filename': '/path/to/django/debug.log', 'formatter': 'verbose', 'filters': { 'correlation': { '()': 'cid.log.cidcontextfilter' 'loggers': { 'django.request': { 'handlers': ['file'], 'level': 'DEBUG', 'propagate': True, 'filters': ['correlation'] } 4 Chapter 1. Django Correlation IDs
You can then use your loggers as normal, safe in the knowledge that you can tie them all back to the correlation id. Template Context Processor Django Correlation Id provides a template context processor which adds the correlation id to the template context if it is available. To enable this you will need to add the context processor to your settings: TEMPLATE_CONTEXT_PROCESSORS = ( "django.contrib.auth.context_processors.auth", "django.core.context_processors.debug", "django.core.context_processors.i18n", "django.core.context_processors.media", "django.core.context_processors.static", "django.core.context_processors.tz", "django.contrib.messages.context_processors.messages", "cid.context_processors.cid_context_processor", ) This will place the context variable correlation_id in your template context if a correlation id is available. For example you could add it as a meta tag in your templates with the follwing snippet: {% if correlation_id %} <meta name="correlation_id" content="{{ correlation_id }}" /> {% endif %} Settings CID_GENERATE Tell the cid middleware to generate a correlation id if it doesn t already exist. Default value: False. CID_HEADER The HTTP header to extract the correlation id from. Default value: X_CORRELATION_ID CID_RESPONSE_HEADER The HTTP response header where the correlation id will be set. Default value: same name as CID_HEADER. Set to None if you do not want the response to include the header. Contributing Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given. You can contribute in many ways: Types of Contributions Report Bugs Report bugs at https://github.com/snowball-one/cid/issues. If you are reporting a bug, please include: Your operating system name and version. Any details about your local setup that might be helpful in troubleshooting. Detailed steps to reproduce the bug. 1.1. Features 5
Fix Bugs Look through the GitHub issues for bugs. Anything tagged with bug is open to whoever wants to implement it. Implement Features Look through the GitHub issues for features. Anything tagged with feature is open to whoever wants to implement it. Write Documentation CID could always use more documentation, whether as part of the official CID docs, in docstrings, or even on the web in blog posts, articles, and such. Submit Feedback The best way to send feedback is to file an issue at https://github.com/snowball-one/cid/issues. If you are proposing a feature: Explain in detail how it would work. Keep the scope as narrow as possible, to make it easier to implement. Remember that this is a volunteer-driven project, and that contributions are welcome :) Get Started! Ready to contribute? Here s how to set up cid for local development. 1. Fork the cid repo on GitHub. 2. Clone your fork locally: $ git clone git@github.com:your_name_here/cid.git 3. Install your local copy into a virtualenv. Assuming you have virtualenvwrapper installed, this is how you set up your fork for local development: $ mkvirtualenv cid $ cd cid/ $ pip install -e. $ pip install -r requirement/ci.txt 4. Create a branch for local development: $ git checkout -b name-of-your-bugfix-or-feature Now you can make your changes locally. 5. When you re done making changes, check that your changes pass flake8 and the tests, including testing other Python versions with tox: $ py.test --pep8 tests 6 Chapter 1. Django Correlation IDs
6. Commit your changes and push your branch to GitHub: $ git add. $ git commit -m "Your detailed description of your changes." $ git push origin name-of-your-bugfix-or-feature 7. Submit a pull request through the GitHub website. Pull Request Guidelines Before you submit a pull request, check that it meets these guidelines: 1. The pull request should include tests. 2. If the pull request adds functionality, the docs should be updated. Put your new functionality into a function with a docstring, and add the feature to the list in README.rst. 3. The pull request should work for Python 2.6, 2.7, 3.3, 3.4, 3.5 and for PyPy. Check https://travis-ci.org/ snowball-one/cid/pull_requests and make sure that the tests pass for all supported Python versions. Credits Jonathan Moss <jonathan.moss@snowballone.com.au> Francis Reyes <francis.reyes@snowballone.com.au> Damien Baty <github: @dbaty> BEY Quentin <github: @qbey> History 0.1.0 (2014-08-05) First release on PyPI. 0.1.2 (2016-12-01) Made CID repsonse header configurable, and optional (thanks @dbaty) 0.2.0 (2016-12-06) Added support for Django 1.10 middleware (thanks @qbey) Modules cid package cid.context_processors module cid.context_processors.cid_context_processor(request) Adds the correlation id as correlation_id to template contexts 1.1. Features 7
cid.cursor module class cid.cursor.cidcursorwrapper(cursor) Bases: object A cursor wrapper that attempts to add a cid comment to each query add_comment(sql) callproc(procname, params=none) execute(sql, params=none) executemany(sql, param_list) cid.locals module cid.locals.get_cid() Retrieves the currently set Correlation Id cid.locals.set_cid(cid) Sets the Correlation Id for a a request cid.middleware module Warning: Django Correlation Ids currently makes use of threading.local. If this offends your sensibilities please let an know any alternative you may have because it makes us a bit itchy too. :-) 8 Chapter 1. Django Correlation IDs
Python Module Index c cid.context_processors, 7 cid.cursor, 8 cid.locals, 8 9
10 Python Module Index
Index A add_comment() (cid.cursor.cidcursorwrapper method), 8 C callproc() (cid.cursor.cidcursorwrapper method), 8 cid.context_processors (module), 7 cid.cursor (module), 8 cid.locals (module), 8 cid_context_processor() (in module cid.context_processors), 7 CID_GENERATE, 5 CID_HEADER, 5 CID_RESPONSE_HEADER, 5 CidCursorWrapper (class in cid.cursor), 8 E execute() (cid.cursor.cidcursorwrapper method), 8 executemany() (cid.cursor.cidcursorwrapper method), 8 G get_cid() (in module cid.locals), 8 S set_cid() (in module cid.locals), 8 11