dql Release gadaccee-dirty

Transcription

1 dql Release gadaccee-dirty February 05, 2014

2

3 Contents i

4 ii

5 A simple, SQL-ish language for DynamoDB Contents 1

6 2 Contents

7 CHAPTER 1 User Guide 1.1 Getting Started Install DQL with pip: pip install dql Since DQL uses boto under the hood, the authentication mechanism is the same. You may either set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, or pass them in on the command line: $ dql -a <access key> -s <secret key> DQL uses us-west-1 as the default region. You can change this by setting the AWS_REGION variable or passing it in on the command line: $ dql -r us-east-1 You can begin using DQL immediately. Try creating a table and inserting some data us-west-1> CREATE TABLE posts (username STRING HASH KEY, > postid NUMBER RANGE KEY, > ts NUMBER INDEX( ts-index )) > THROUGHPUT (5, 5); us-west-1> INSERT INTO posts (username, postid, ts, text) > VALUES ( steve, 1, , Hey guys! ), > ( steve, 2, , Guys? ), > ( drdice, 3, , Fun fact: dolphins are dicks ); us-west-1> ls Name Status Read Write posts ACTIVE 5 5 You can query this data in a couple of ways. The first should look familiar us-west-1> SELECT * FROM posts WHERE username = steve ; This performs a table query using the hash key, so it should run relatively quickly. The second way of accessing data is by using a scan us-west-1> SCAN posts FILTER username = steve ; This returns the same data as the SELECT, but it retrieves it with a table scan instead, which is much slower. You can also perform updates to the data in a familiar way 3

8 us-west-1> UPDATE posts SET text = Hay gusys!!11 WHERE > username = steve AND postid = 1; The Queries section has more details. Check it out for more information on the performance implications and query options. 1.2 Advanced Usage This section is devoted to some of the advanced uses of DQL API DQL execution is available as a python library. For example: import dql engine = dql.engine() engine.connect_to_region( us-west-1 ) results = engine.execute("scan mytable LIMIT 10") for item in results: print dict(item) The return value will vary based on the type of query Embedded Python DQL supports the use of python expressions anywhere that you would otherwise have to specify a data type. Just surround the python with backticks. Create your variable scope as a dict and pass it to the engine with the commands: >>> engine.scope = { foo1 : 1, foo2 : 2} >>> engine.execute("insert INTO foobars (foo) VALUES ( foo1 ), ( foo2 )") The interactive client has a special way to modify the scope. You can switch into code mode to execute python, and then use that scope as the engine scope: us-west-1> code >>> foo1 = 1 >>> import time >>> endcode us-west-1> INSERT INTO foobars (foo) VALUES ( foo1 ), ( time.time() ) success You can also spread the expressions across multiple lines by using the format m <expr>. If you do this, you will need to return a value. The difference here is between using eval and exec in python. us-west-1> UPDATE foobars SET foo = m > if bar % 2 == 0: > return bar**2 > else: > return bar - 1 > ; 4 Chapter 1. User Guide

9 1.3 Queries ALTER Synopsis ALTER TABLE tablename SET [INDEX index] THROUGHPUT throughput Examples ALTER TABLE foobars SET THROUGHPUT (4, 8); ALTER TABLE foobars SET THROUGHPUT (7, *); ALTER TABLE foobars SET INDEX ts-index THROUGHPUT (5, *); Description Alter changes the read/write throughput on a table. Dynamo stipulates that you may only increase the throughput by a factor of 2 at a time. If you attempt to increase the throughput by more than that, DQL will have to make multiple calls, waiting for the updates to go through in the interim. You may only decrease the throughput twice per day. If you wish to change one of the throughput values and not the other, pass in 0 or * for the value you wish to remain unchanged. Parameters tablename The name of the table throughput The read/write throughput in the form (read_throughput, write_throughput) index The name of the global index COUNT Synopsis COUNT [ CONSISTENT ] tablename WHERE expression [ USING index ] Examples COUNT foobars WHERE foo = bar ; COUNT CONSISTENT foobars WHERE foo!= bar AND baz >= 3; COUNT foobars WHERE (foo = bar AND baz >= 3) USING baz-index ; 1.3. Queries 5

10 Description This counts the number of matching items in your table. It is making a query, so you must search using the hash key, and you may optionally also provide the range key or an index. The WHERE clause is mandatory. If you want a count of all elements in a table, look at the table description. Parameters CONSISTENT If this is present, use a read-consistent query tablename The name of the table index When the WHERE expression uses an indexed attribute, this allows you to manually specify which index name to use for the query. It should generally not be needed, as the DQL engine will automatically detect the correct index to use for a query. Where Clause Examples WHERE hkey = a AND bar > 5 AND baz <= 16 WHERE hkey = 1 AND bar BEGINS WITH "prefix" WHERE hkey = 1 AND bar BETWEEN (1, 100) CREATE Synopsis CREATE TABLE [IF NOT EXISTS] tablename attributes [GLOBAL [ALL KEYS INCLUDE] INDEX global_index] Examples CREATE TABLE foobars (id STRING HASH KEY); CREATE TABLE IF NOT EXISTS foobars (id STRING HASH KEY); CREATE TABLE foobars (id STRING HASH KEY, foo BINARY RANGE KEY, THROUGHPUT (1, 1)); CREATE TABLE foobars (id STRING HASH KEY, foo BINARY RANGE KEY, ts NUMBER INDEX( ts-index ), views NUMBER INDEX( views-index )); CREATE TABLE foobars (id STRING HASH KEY, bar STRING) GLOBAL INDEX ( bar-index, bar, THROUGHPUT (1, 1)); CREATE TABLE foobars (id STRING HASH KEY, bar STRING, baz NUMBER, THROUGHPUT (2, 2)) GLOBAL INDEX ( bar-index, bar, baz) GLOBAL INDEX ( baz-index, baz, THROUGHPUT (4, 2)); 6 Chapter 1. User Guide

11 Description Create a new table. You must have exactly one hash key, zero or one range keys, and up to five indexes. You must have a range key in order to have any indexes. Parameters IF NOT EXISTS If present, do not through an exception if the table already exists. tablename The name of the table that you want to alter attributes A list of attribute declarations of the format (name data type [key type]) The available data types are STRING, NUMBER, and BINARY. You will not need to specify SET types, because these fields are only used for index creation and it is (presently) impossible to index a SET. The available key types are HASH KEY, RANGE KEY, and [ALL KEYS INCLUDE] INDEX(name). At the end of the attribute list you may specify the THROUGHPUT, which is in the form of (read_throughput, write_throughput). If throughput is not specified it will default to (5, 5). global_index A global index for the table. You may provide up to 5. The format is (name, hash key, [range key], [non-key attributes], [throughput]). The hash key and range key must be in the attributes declaration. non-key attributes should only be provided if it is an INCLUDE index. If throughput is not specified it will default to (5, 5). Schema Design at a Glance When DynamoDB scales, it partitions based on the hash key. For this reason, all of your table queries must include the hash key in the where clause (and optionally the range key or a local index). So keep that in mind as you design your schema. The keypair formed by the hash key and range key is referred to as the primary key. If there is no range key, it s just the hash key. The primary key is unique among items in the table. No two items may have the same hash key and range key. From a query standpoint, local indexes behave nearly the same as a range key. The main difference is that they do not need to be unique. Global indexes can be thought of as adding additional hash and range keys to the table. They allow you to query a table on a different hash key than the one defined on the table. Global indexes have throughput that is managed independently of the table they are on. Global index keys do not have a uniqueness constraint (there may be multiple items in the table that have the same hash and range key in a global index). Read Amazon s documentation for Create Table for more information DELETE Synopsis DELETE FROM tablename WHERE expression [ USING index ] 1.3. Queries 7

12 Examples DELETE FROM foobars WHERE foo = bar ; DELETE FROM foobars WHERE foo!= bar AND baz >= 3; DELETE FROM foobars WHERE KEYS IN ( hkey1 ), ( hkey2 ); DELETE FROM foobars WHERE KEYS IN ( hkey1, rkey1 ), ( hkey2, rkey2 ); DELETE FROM foobars WHERE (foo = bar AND baz >= 3) USING baz-index ; Description Parameters tablename The name of the table index When the WHERE expression uses an indexed attribute, this allows you to manually specify which index name to use for the query. It should generally not be needed, as the DQL engine will automatically detect the correct index to use for a query. Where Clause Examples WHERE hkey = a AND bar > 5 AND baz <= 16 WHERE hkey = 1 AND bar BEGINS WITH "prefix" WHERE hkey = 1 AND bar BETWEEN (1, 100) WHERE KEYS IN ( hkey1, rkey1 ), ( hkey2, rkey2 ) WHERE KEYS IN ( hkey ), ( hkey2 ) Notes Using the KEYS IN form is much more efficient because DQL will not have to perform a query first to get the primary keys DROP Synopsis DROP TABLE [ IF EXISTS ] tablename Examples DROP TABLE foobars; DROP TABLE IF EXISTS foobars; 8 Chapter 1. User Guide

13 Description Deletes a table and all its items. Warning: This is not reversible! Use with extreme caution! Parameters IF EXISTS If present, do not through an exception if the table does not exist. tablename The name of the table DUMP Synopsis DUMP SCHEMA [ tablename [,...] ] Examples DUMP SCHEMA; DUMP SCHEMA foobars, widgets; Description Print out the matching CREATE statements for your tables. Parameters tablename The name of the table(s) whose schema you want to dump. If no tablenames are present, it will dump all table schemas INSERT Synopsis INSERT INTO tablename attributes VALUES values Examples INSERT INTO foobars (id) VALUES (1); INSERT INTO foobars (id, bar) VALUES (1, hi ), (2, yo ); 1.3. Queries 9

14 Description Insert data into a table Parameters tablename The name of the table attributes Comma-separated list of attribute names values Comma-separated list of data to insert. The data is of the form (var [, var]...) and must contain the same number of items as the attributes parameter. Data Types Below are examples of how to represent the different dynamo data types in DQL NUMBER: 123 STRING: abc or "abc" BINARY: b abc or b"abc" NUMBER SET: (1, 2, 3) STRING SET: ( a, b, c ) BINARY SET: (b a, b b, b c ) SCAN Synopsis SCAN tablename [ FILTER expression ] [ LIMIT limit ] Examples SCAN foobars; SCAN foobars FILTER id = a AND foo = 4; SCAN foobars FILTER id = a AND foo CONTAINS 4 LIMIT 100; Description Sequentially iterate over items in a table. This does not use the indexes, so it can be significantly slower than a query. Parameters tablename The name of the table limit Maximum number of results to return Filter Clause Examples 10 Chapter 1. User Guide

15 FILTER hkey = a AND bar > 5 AND baz!= 16 FILTER hkey = 1 AND bar BEGINS WITH "prefix" FILTER hkey = 1 AND bar BETWEEN (1, 100) FILTER hkey = 1 AND bar IS NULL AND baz IS NOT NULL FILTER hkey = 1 AND bar CONTAINS 5 AND baz NOT CONTAINS 5 FILTER hkey = 1 AND bar IN (1, 3, 5, 7, 9) Notes See the AWS docs for more information on scan parameters SELECT Synopsis SELECT [ CONSISTENT ] attributes FROM tablename WHERE expression [ USING index ] [ LIMIT limit ] [ ASC DESC ] Examples SELECT * FROM foobars WHERE foo = bar ; SELECT CONSISTENT * foobars WHERE foo = bar AND baz >= 3; SELECT foo, bar FROM foobars WHERE id = a AND ts < 100 USING ts-index ; SELECT * FROM foobars WHERE foo = bar AND baz >= 3 LIMIT 50 DESC; Description Query a table for items. Parameters CONSISTENT If this is present, use a read-consistent query attributes Comma-separated list of item attributes to fetch. * is a special case meaning all attributes. tablename The name of the table index When the WHERE expression uses an indexed attribute, this allows you to manually specify which index name to use for the query. It should generally not be needed, as the DQL engine will automatically detect the correct index to use for a query. limit Maximum number of results to return ASC DESC Sort the results in ASCending (the default) or DESCending order Queries 11

16 Where Clause Examples WHERE hkey = a AND bar > 5 AND baz <= 16 WHERE hkey = 1 AND bar BEGINS WITH "prefix" WHERE hkey = 1 AND bar BETWEEN (1, 100) WHERE KEYS IN ( hkey1, rkey1 ), ( hkey2, rkey2 ) WHERE KEYS IN ( hkey ), ( hkey2 ) Notes When using the KEYS IN form, DQL will perform a batch get instead of a table query. See the AWS docs for more information on query parameters UPDATE Synopsis UPDATE tablename SET values [ WHERE expression ] [ RETURNS (NONE ( ALL UPDATED) (NEW OLD)) ] Examples UPDATE foobars SET foo = a ; UPDATE foobars SET foo = a, bar += 4 WHERE id = 1 AND foo = b ; UPDATE foobars SET foo = a, bar += 4 RETURNS ALL NEW; UPDATE foobars SET myset << (5, 6, 7), mytags << new tag WHERE KEYS IN ( a, b ); UPDATE foobars SET foo = bar + 1 ; Description Update the attributes on items in your table. Parameters tablename The name of the table values Comma-separated list of attribute assignments. The supported operators are =, +=, -=, <<, and >>. The shovel operators (<< & >>) are used to atomically add/remove items to/from a set. Likewise, the += and -= perform atomic inc/decrement and may only be used on NUMBER types. expression Only return elements that match this expression. The supported operators are =,!=, >, >=, <, and <=. The only conjunction allowed is AND. There is another form available that looks like KEYS IN (pkey) [, (pkey)]... The pkey is a single value if the table only has a hash key, or two comma-separated values if there is also a range key. 12 Chapter 1. User Guide

17 When using the first form, DQL does a table query to find matching items and then performs the updates. The second form performs the updates with no reads necessary. If no expression is specified, DQL will perform a full-table scan. RETURNS Return the items that were operated on. Default is RETURNS NONE. See the Amazon docs for UpdateItem for more detail. Notes When using python expressions to set values, you may reference attributes on the table row: UPDATE foobars SET foo = bar + 1 If you aren t sure if the attribute will exist or not, you can reference the row dict directly: us-west-1> UPDATE foobars SET foo = m if row.get( bar ): > return bar + 1 > else: > return 1 ; This syntax will NOT WORK if you are using the KEYS IN form of the query, as that performs the update without doing any table reads. 1.4 Developing To get started developing dql, run the following command: wget && \ python unbox.py git@github.com:mathcamp/dql This will clone the repository and install the package into a virtualenv Running Tests The command to run tests is python setup.py nosetests. Some of these tests require DynamoDB Local. There is a nose plugin that will download and run the DynamoDB Local service during the tests. It requires the java 6/7 runtime, so make sure you have that installed Developing 13

18 14 Chapter 1. User Guide

19 CHAPTER 2 API Reference 2.1 dql package Subpackages dql.grammar package Submodules dql.grammar.common module Common use grammars dql.grammar.common.upkey(name) Shortcut for creating an uppercase keyword dql.grammar.query module Grammars for parsing query strings dql.grammar.query.create_filter() Create a grammar for filtering on table scans dql.grammar.query.create_filter_constraint() Create a constraint for a scan FILTER clause dql.grammar.query.create_limit() Create the gramar for a limit clause dql.grammar.query.create_query_constraint() Create a constraint for a query WHERE clause dql.grammar.query.create_select_where() Create a grammar for the where clause used by select dql.grammar.query.create_where() Create the grammar for a where clause Module contents DQL language parser dql.grammar.create_alter() Create the grammar for the alter statement 15

20 dql.grammar.create_count() Create the grammar for the count statement dql.grammar.create_create() Create the grammar for the create statement dql.grammar.create_delete() Create the grammar for the delete statement dql.grammar.create_drop() Create the grammar for the drop statement dql.grammar.create_dump() Create the grammar for the dump statement dql.grammar.create_insert() Create the grammar for the insert statement dql.grammar.create_parser() Create the language parser dql.grammar.create_scan() Create the grammar for the scan statement dql.grammar.create_select() Create the grammar for the select statement dql.grammar.create_throughput(variable=primitive) Create a throughput specification dql.grammar.create_update() Create the grammar for the update statement Submodules dql.cli module Interative DQL client class dql.cli.dqlclient(completekey= tab, stdin=none, stdout=none) Bases: cmd.cmd Interactive commandline interface Attributes running ddb engine bool(x) -> bool complete_display(text, *_) Autocomplete for display complete_file(text, line, *_) Autocomplete DQL file lookup complete_ls(text, *_) Autocomplete for ls 16 Chapter 2. API Reference

21 ddb = None default(command) display = None do_eof(arglist) Exit do_code(arglist) Switch to executing python code do_display(arglist) Get or set the type of display to use when printing results do_endcode(arglist) Stop executing python code do_exit(arglist) Exit do_file(arglist) Read and execute a.dql file do_ls(arglist) List all tables or print details of one table do_pagesize(arglist) Get or set the page size of the query output do_shell(arglist) Run a shell command do_use(arglist) Switch the AWS region You may also specify use local host=localhost port=8000 to use the DynamoDB Local service do_width(arglist) Get or set the width of the formatted output do_x(arglist) Toggle expanded display format You can set smart formatting with x smart emptyline() engine = None formatter = None help_alter() Print the help text for ALTER help_count() Print the help text for COUNT help_create() Print the help text for CREATE help_delete() Print the help text for DELETE help_drop() Print the help text for DROP 2.1. dql package 17

22 help_dump() Print the help text for DUMP help_help() Print the help text for help help_insert() Print the help text for INSERT help_scan() Print the help text for SCAN help_select() Print the help text for SELECT help_update() Print the help text for UPDATE initialize(region= us-west-1, host= localhost, port=8000, access_key=none, secret_key=none) Set up the repl for execution load_config() Load your configuration settings from a file postcmd(stop, line) region = None running = False save_config_value(key, value) Save your configuration settings to a file start() Start running the interactive session (blocking) update_prompt() Update the prompt dql.cli.connect(region, host= localhost, port=8000, access_key=none, secret_key=none) Create a DynamoDB connection dql.cli.repl_command(fxn) Decorator for cmd methods Parses arguments from the arg string and passes them to the method as *args and **kwargs. dql.engine module Execution engine class dql.engine.engine(connection=none) Bases: object DQL execution engine Parameters connection : DynamoDBConnection, optional If not present, you will need to call Engine.connect_to_region() 18 Chapter 2. API Reference

23 Notes One of the advanced features of the engine is the ability to set variables which can later be referenced in queries. Whenever a STRING or NUMBER is expected in a query, you may alternatively supply a variable name. That name will be looked up in the engine s scope. This allows you to do things like: engine.scope[ myfoo ] = abc engine.execute("select * FROM foobars WHERE foo = myfoo") Attributes scope dict Lookup scope for variables cloudwatch_connection Lazy create a connection to cloudwatch connect_to_region(region, **kwargs) Connect the engine to an AWS region connection Get the dynamo connection describe(tablename, refresh=false, metrics=false) Get the TableMeta for a table describe_all() Describe all tables in the connected region execute(commands) Parse and run a DQL string Parameters commands : str The DQL command string get_capacity(tablename) Get the consumed read/write capacity resolve(val, scope=none) Resolve a value into a string or number class dql.engine.fragmentengine(connection) Bases: dql.engine.engine A DQL execution engine that can handle query fragments execute(fragment) Run or aggregate a query fragment Concat the fragment to any stored fragments. If they form a complete query, run it and return the result. If not, store them and return None. partial True if there is a partial query stored pformat_exc(exc) Format an exception message for the last query s parse error reset() Clear any query fragments from the engine 2.1. dql package 19

24 dql.engine.float_to_decimal(f ) Monkey-patched replacement for boto s broken version dql.help module Help text for the CLI dql.models module Data containers class dql.models.globalindex(name, index_type, status, hash_key, range_key, read_throughput, write_throughput, size, item_count, includes=none) Bases: object Container for global index data classmethod from_description(description, attrs) Create an object from a boto response pformat() Pretty format for insertion into table pformat schema The DQL fragment for constructing this index class dql.models.indexfield(name, data_type, index_type, index_name, includes=none) Bases: dql.models.tablefield A TableField that is also part of a Local Secondary Index schema The DQL syntax for creating this item class dql.models.tablefield(name, data_type, key_type=none) Bases: object A DynamoDB table attribute Parameters name : str data_type : str The type of object (e.g. STRING, NUMBER, etc) key_type : str, optional The type of key (e.g. RANGE, HASH, INDEX ) index_name : str, optional If the key_type is INDEX, this will be the name of the index that uses the field as a range key. schema The DQL syntax for creating this item to_index(index_type, index_name, includes=none) Create an index field from this field 20 Chapter 2. API Reference

25 class dql.models.tablemeta(name, status, attrs, global_indexes, read_throughput, write_throughput, decreases_today, size, item_count) Bases: object Container for table metadata Parameters name : str status : str attrs : dict Mapping of attribute name to :class:.tablefield s global_indexes : dict Mapping of hash key to GlobalIndex read_throughput : int write_throughput : int decreases_today : int consumed_read_capacity : int May be None if unknown consumed_write_capacity : int May be None if unknown size : int Size of the table in bytes item_count : int Number of items in the table decreases_remaining Number of remaining times you may decrease throughput today classmethod from_description(description) Factory method that uses the boto describe return value pformat() Pretty string format primary_key(hkey, rkey=none) Construct a primary key dictionary You can either pass in a (hash_key[, range_key]) as the arguments, or you may pass in an Item itself schema The DQL query that will construct this table s schema total_read_throughput Combined read throughput of table and global indexes total_write_throughput Combined write throughput of table and global indexes 2.1. dql package 21

26 dql.output module Formatting and displaying output class dql.output.baseformat(width=100, pagesize=1000) Bases: object Base class for formatters format(result, ostream) Format a single result and stick it in an output stream format_field(field) Format a single Dynamo value write(results, ostream) Write results to an output stream class dql.output.columnformat(width=100, pagesize=1000) Bases: dql.output.baseformat A layout that puts item attributes in columns format(results, columns, ostream) write(results, ostream) class dql.output.expandedformat(width=100, pagesize=1000) Bases: dql.output.baseformat A layout that puts item attributes on separate lines format(result, ostream) class dql.output.smartformat(*args, **kwargs) Bases: dql.output.columnformat A layout that chooses column/expanded format intelligently format(results, columns, ostream) dql.output.get_default_display() Get the default display function for this system dql.output.less_display(*args, **kwds) Use smoke and mirrors to acquire less for pretty paging dql.output.stdout_display(*args, **kwds) Print results straight to stdout dql.output.truncate(string, length, ellipsis=u... ) Truncate a string to a length, ending with... if it overflows dql.output.wrap(string, length, indent) Wrap a string at a line length Module contents Simple SQL-like query language for dynamo dql.main() Start the DQL client 22 Chapter 2. API Reference

27 CHAPTER 3 Versions Version Build Coverage master Code lives here: 23

28 24 Chapter 3. Versions

29 CHAPTER 4 Changelog First public release 25

30 26 Chapter 4. Changelog

31 CHAPTER 5 Indices and tables genindex modindex search 27

32 28 Chapter 5. Indices and tables

33 Python Module Index d dql,?? dql.cli,?? dql.engine,?? dql.grammar,?? dql.grammar.common,?? dql.grammar.query,?? dql.help,?? dql.models,?? dql.output,?? 29

34 30 Python Module Index

35 Python Module Index d dql,?? dql.cli,?? dql.engine,?? dql.grammar,?? dql.grammar.common,?? dql.grammar.query,?? dql.help,?? dql.models,?? dql.output,?? 31