Kinto Documentation Release 0.2.2 Mozilla Services Da French Team June 23, 2015
Contents 1 In short 3 2 Table of content 5 2.1 API Endpoints.............................................. 5 2.2 Installation................................................ 6 2.3 Configuration............................................... 10 2.4 Contributing............................................... 10 2.5 Changelog................................................ 10 3 Indices and tables 13 i
ii
Kinto is a server allowing you to store and synchronize arbitrary data, attached to your Firefox account. Online documentation Issue tracker Contributing Fig. 1: Kinto-Un is a magical, yellow cloud that serves as a way of transportation. Contents 1
2 Contents
CHAPTER 1 In short It s as simple as that: 1. Pick up a name for your records collection; 2. Push the records on the server; 3. Pull the records from the server (ordered, filtered, paginated) The server doesn t impose anything about the records data model. Kinto is based on top of cliquet. 3
4 Chapter 1. In short
CHAPTER 2 Table of content 2.1 API Endpoints See cliquet API documentation for an exhaustive list of features and endpoints of the Kinto API. 2.1.1 /collections/<collection_id>/records Requires authentication End-point for the collection of records: Create record Fetch, sort and filter records Note: A collection is considered empty by default. In other words, no error will be thrown if the collection id is unknown. Note: Records are private by user. See cliquet resource documentation for more details on available operations. 2.1.2 /collections/<collection_id>/records/<id> Requires authentication End-point for a single record of the collection: Fetch Modify Replace Delete See cliquet record documentation for more details on available operations. 5
2.2 Installation Kinto is based on top of the cliquet project, and as such, please refer to cliquet s documentation if details seem to be missing here. 2.2.1 Run locally For development By default, Kinto persists the records and internal cache in a PostgreSQL database. The default configuration will connect to the postgres database on localhost:5432, with user/password postgres/postgres. See more details below about installation and setup of PostgreSQL. make serve Using Docker Kinto uses Docker Compose, which takes care of running and connecting PostgreSQL: docker-compose up Authentication By default, Kinto relies on Firefox Account OAuth2 Bearer tokens to authenticate users. See cliquet documentation to configure authentication options. 2.2.2 Running in production Recommended settings Most default setting values in the application code base are suitable for production. But the set of settings mentionned below might deserve some review or adjustments: cliquet.http_scheme = https cliquet.paginate_by = 100 cliquet.batch_max_requests = 25 cliquet.storage_pool_maxconn = 50 cliquet.cache_pool_maxconn = 50 fxa-oauth.cache_ttl_seconds = 3600 note For an exhaustive list of available settings and their default values, refer to the source code. Monitoring # Heka cliquet.logging_renderer = cliquet.logs.mozillahekarenderer # StatsD cliquet.statsd_url = udp://carbon.server:8125 6 Chapter 2. Table of content
With the following configuration, all logs are structured in JSON and redirected to standard output (See 12factor app). A Sentry logger is also enabled. [loggers] keys = root, kinto, cliquet [handlers] keys = console, sentry [formatters] keys = generic, heka [logger_root] level = INFO handlers = console, sentry [logger_kinto] level = INFO handlers = console, sentry qualname = kinto [logger_cliquet] level = INFO handlers = console, sentry qualname = cliquet [handler_console] class = StreamHandler args = (sys.stdout,) level = INFO formatter = heka [handler_sentry] class = raven.handlers.logging.sentryhandler args = ('http://public:secret@example.com/1',) level = INFO formatter = generic [formatter_generic] format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadname)s] %(message)s [formatter_heka] format = %(message)s PostgreSQL setup In production, it is wise to run the application with a dedicated database and user. postgres=# CREATE USER prod; postgres=# CREATE DATABASE prod OWNER prod; CREATE DATABASE Once storage and cache are modified in.ini, the tables need to be created with the cliquet command-line tool: $ cliquet --ini production.ini migrate note Alternatively the SQL initialization files can be found in the Cliquet source code (cliquet/cache/postgresql/schemal.sql and 2.2. Installation 7
cliquet/storage/postgresql/schemal.sql). Running with uwsgi To run the application using uwsgi, an app.wsgi file is provided. This command can be used to run it: uwsgi --ini config/kinto.ini uwsgi configuration can be tweaked in the ini file in the dedicated [uwsgi] section. Here s an example: [uwsgi] wsgi-file = app.wsgi enable-threads = true http-socket = 127.0.0.1:8000 processes = 3 master = true module = kinto harakiri = 120 uid = kinto gid = kinto virtualenv =. lazy = true lazy-apps = true single-interpreter = true buffer-size = 65535 post-buffering = 65535 To use a different ini file, the KINTO_INI environment variable should be present with a path to it. 2.2.3 Install and setup PostgreSQL (requires PostgreSQL 9.3 or higher). Client libraries only Install PostgreSQL client headers: sudo apt-get install libpq-dev Full server In Ubuntu/Debian based: sudo apt-get install postgresql By default, the postgres user has no password and can hence only connect if ran by the postgres system user. The following command will assign it: sudo -u postgres psql -c "ALTER USER postgres PASSWORD 'postgres';" 8 Chapter 2. Table of content
Server using Docker Install docker: On Ubuntu you can do: sudo apt-get install docker.io Run the official PostgreSQL container locally: postgres=$(sudo docker run -e POSTGRES_PASSWORD=postgres -d -p 5432:5432 postgres) Tag and save the current state with: sudo docker commit $postgres kinto-db In the future, run the tagged version of the container kinto=$(sudo docker run -e POSTGRES_PASSWORD=postgres -d -p 5432:5432 kinto-db)... sudo docker stop $kinto OS X Assuming brew is installed: brew update brew install postgresql Create the initial database: initdb /usr/local/var/postgres 2.2.4 Cryptography libraries Linux On Debian / Ubuntu based systems: apt-get install libffi-dev libssl-dev On RHEL-derivatives: apt-get install libffi-devel openssl-devel OS X Assuming brew is installed: brew install libffi openssl pkg-config 2.2. Installation 9
2.3 Configuration See cliquet settings documentation. 2.3.1 Storage backend In order to use Kinto as a storage backend for an application built with cliquet, some settings must be set carefully. Firefox Account In order to avoid double-verification of FxA OAuth tokens, the cliquet.cache_url should be the same in Kinto and in the application. This way the verification cache will be shared between the two components. 2.4 Contributing Thank you for considering to contribute to Kinto! note No contribution is too small; please submit as many fixes for typos and grammar bloopers as you can! note Open a pull-request even if your contribution is not ready yet! It can be discussed and improved collaboratively! 2.4.1 Run tests make tests 2.4.2 IRC channel Join #storage on irc.mozilla.org! 2.5 Changelog This document describes changes between each past release. 2.5.1 0.2.2 (2015-06-04) Upgraded to cliquet 1.8.+ Breaking changes PostgreSQL database initialization process is not run automatically in production. Add this command to deployment procedure: cliquet --ini config/kinto.ini migrate Internal changes Improved documentation (#29) 10 Chapter 2. Table of content
Require 100% coverage during tests (#27) Basic Auth is now enabled by default in example config 2.5.2 0.2.1 (2015-03-25) Upgraded to cliquet 1.4.1 Bug fixes Rely on Pyramid API to build pagination Next-Url (#147) 2.5.3 0.2 (2015-03-24) Upgraded to cliquet 1.4 Bug fixes Fix behaviour of CloudStorage with backslashes in querystring (mozilla-services/cliquet#142) Force PostgreSQl session timezone to UTC (mozilla-services/cliquet#122) Fix basic auth ofuscation and prefix (mozilla-services/cliquet#128) Make sure the paginate_by setting overrides the passed limit argument (mozilla-services/cliquet#129) Fix crash of classic logger with unicode (mozilla-services/cliquet#142) Fix crash of CloudStorage backend when remote returns 500 (mozilla-services/cliquet#142) Fix python3.4 segmentation fault (mozilla-services/cliquet#142) Add missing port in Next-Page header (mozilla-services/cliquet#147) 2.5.4 0.1 (2015-03-20) Initial version Schemaless storage of records Firefox Account authentication Kinto as a storage backend for cliquet applications 2.5. Changelog 11
12 Chapter 2. Table of content
CHAPTER 3 Indices and tables genindex modindex search 13