Transactions for web developers

Size: px

Start display at page:

Download "Transactions for web developers"

Coleen Logan
6 years ago
Views:

1 Transactions for web developers Aymeric Augustin DjangoCon Europe - May 17th,

2 Transaction management tools are often made to seem like a black art. Christophe Pettus (2011) Life with Object-Relational Mappers 2

3 Today s talk Framework sqlite3 psycopg2 Interface Database 3

4 SQL-92 Framework Interface 4 Database

of SQLstatements that is atomic with respect to

5 Definition An SQL-transaction (sometimes simply called a "transaction") is a sequence of executions of SQLstatements that is atomic with respect to recovery. 5 Source:

6 Definition An SQL-transaction (sometimes simply called a "transaction") is a sequence of executions of SQLstatements that is atomic with respect to recovery. 6 Source:

7 Example BEGIN; UPDATE cars SET status = 'available' WHERE id = 1; UPDATE rentals SET end = CURRENT_TIMESTAMP WHERE car_id = 1 AND end IS NULL; COMMIT; 7

8 Lifecycle A transaction begins with a transaction-initiating statement. A transaction ends with a commit, an explicit rollback, or an implicit rollback. 8

9 Isolation levels At this level, READ UNCOMMITTED READ COMMITTED REPEATABLE READ SERIALIZABLE (default) these phenomena cannot happen: - dirty read read a row that is never committed non repeatable read read different values for the same row phantom read read different rows for the same query SERIALIZABLE also guarantees that transactions are serializable. 9

10 Statements COMMIT ROLLBACK (advanced) SET TRANSACTION... SET CONSTRAINTS... 10

11 SQL:1999 Framework Interface 11 Database

12 Savepoints S1 S2 becomes either S1 S2 S3 after SAVEPOINT S3; or S1 RELEASE SAVEPOINT S2; or S1 S2 ROLLBACK TO SAVEPOINT S2; 12

13 Explicit transaction start START TRANSACTION; UPDATE cars SET status = 'available' WHERE id = 1; UPDATE rentals SET end = CURRENT_TIMESTAMP WHERE car_id = 1 AND end IS NULL; COMMIT; 13

14 Statements SAVEPOINT <svpt> RELEASE SAVEPOINT <svpt> ROLLBACK TO SAVEPOINT <svpt> (advanced) START TRANSACTION... 14

15 Key learnings Statements always run in a transaction. Transactions are opened automatically. Transactions are advanced technology. 15

16 A brief interlude Framework Interface 16 Database

17 The dreaded PostgreSQL error: current transaction is aborted, commands ignored until end of transaction block What it actually means: a previous statement failed, the application must perform a rollback Any auto-recovery scheme breaks transactional integrity! 17

18 Autocommit Framework Interface 18 Database

19 What does it mean? Commit implicitly after each statement. Wrap each statement in its own transaction. Just execute my query KTHXBYE! Most databases default to autocommit. 19

20 Transactions in autocommit START TRANSACTION BEGIN (alias) COMMIT ROLLBACK Autocommit ON Autocommit OFF Autocommit ON Transaction 20

21 Autocommit in PostgreSQL The server always uses autocommit. Clients can emulate standard behavior by inserting implicit BEGIN statements. In psql: \set autocommit off 21

22 Autocommit in SQLite Transaction semantics are tightly related to the implementation of atomic commit. SQLite automatically starts a transaction before all statements except. It automatically commits such transactions as soon as all statements finish executing. 22

23 Python client libraries Framework Interface 23 Database

24 DB API PEP 249 Connection Performs commits and rollbacks Cursor Executes queries Fetches results 24

25 Transactions in PEP 249 If the database supports an auto-commit feature, this must be initially off. An interface method may be provided to turn it back on. Closing a connection without committing the changes first will cause an implicit rollback to be performed. 25

26 Example >>> from psycopg2 import connect >>> cnx = connect(...); cur = cnx.cursor() >>> cur.execute("insert INTO cars "... "VALUES (1, 'available');") >>> cnx = connect(...); cur = cnx.cursor() >>> cur.execute(" * FROM cars;") >>> cur.fetchall() [] 26

27 Example >>> from psycopg2 import connect >>> cnx = connect(...); cur = cnx.cursor() >>> cur.execute("insert INTO cars "... "VALUES (1, 'available');") >>> cnx.commit() >>> cnx = connect(...); cur = cnx.cursor() >>> cur.execute(" * FROM cars;") >>> cur.fetchall() [(1, 'available')] 27

28 Transactions in psycopg2 psycopg2 tracks the transaction state. It inserts a BEGIN before each statement, unless there s already a transaction in progress. Even before statements. Idle in transaction. cnx.autocommit = True disables this behavior. 28

29 Transactions in sqlite3 sqlite3 tracks the transaction state. It parses statements to insert BEGIN or COMMIT: : INSERT, UPDATE, DELETE, REPLACE: BEGIN Any other statement: COMMIT cnx.isolation_level = None disables this behavior. This is totally unrelated to the isolation level! 29

30 Transactions in sqlite3 sqlite3 tracks the transaction state. It parses statements and inserts before: : INSERT, UPDATE, DELETE, REPLACE: BEGIN Any other statement: COMMIT This is broken by design. SAVEPOINT foo triggers a commit! 30

31 Key learnings The DB API requires the same transactional behavior as the SQL standard. Client libraries for databases that always autocommit have to emulate this behavior. But you can turn it off and use autocommit. 31

32 Problems in Django 1.5 Framework Interface 32 Database

33 Default behavior ( 1.5) Database connection opened Database connection closed Transaction ROLLBACK Django runs with an open transaction. 33

34 Default behavior ( 1.5) Database connection opened Database connection closed COMMIT COMMIT COMMIT ROLLBACK UPDATE INSERT DELETE Django auto-commits in save(), delete(), update() and bulk_create(). 34

35 Transaction middleware Database connection opened Database connection closed COMMIT UPDATE DELETE INSERT One HTTP request one transaction. Commit on success, roll back on exception. Only for the default database! May or may not apply to middleware! 35

36 High-level APIs from django.db import transaction with transaction.autocommit(): # Like Django s default behavior. with transaction.commit_on_success(): # Like the transaction middleware. with transaction.commit_manually(): # Don t use it. 36

37 Low-level APIs from django.db import transaction transaction.commit() transaction.rollback() sid = transaction.savepoint() transaction.savepoint_commit(sid) transaction.savepoint_rollback(sid) How do they interact with Django s transaction management? 37

38 Behind the scenes Django maintains a stack of transaction management states: Auto (False): the ORM commits every change. Managed (True): Django doesn t commit. Django maintains a dirty flag: Set automatically by the ORM after writes. Must be set manually after raw SQL writes. 38

39 Nesting doesn t work well from django.db import transaction from music.models import Guitar guitars = (Guitar.objects.filter(brand="Fender")) with transaction.commit_on_success(): guitars.update(brand="gibson") with transaction.autocommit(): guitars.compute_ratings() compute_ratings() raises an exception. Is the brand change saved? 39

40 Nesting doesn t work well from django.db import transaction from music.models import Guitar guitars = (Guitar.objects.filter(brand="Fender")) with transaction.commit_on_success(): guitars.update(brand="gibson") with transaction.commit_manually(): guitars.compute_ratings() compute_ratings() raises an exception. Is the brand change saved? 40

41 Key learnings If you never understood transaction management in Django 1.5, that s fine. The transaction middleware is a reasonable idea. The decorators/context managers don t work well, especially when they re nested. 41

42 Solution in Django 1.6 Framework Interface 42 Database

43 Default behavior ( 1.6) Database connection opened Database connection closed Django 1.6 uses database-level autocommit. 43

44 Default behavior ( 1.6) Database connection opened Database connection closed UPDATE INSERT DELETE Django 1.6 uses database-level autocommit. 44

45 ATOMIC_REQUESTS Database connection opened Database connection closed COMMIT UPDATE DELETE INSERT One HTTP request = one transaction. Commit on success, roll back on exception. Per database. Only for the view function. 45

46 High-level API: atomic Usable as a decorator or as a context manager. Commits on success, rolls back on exceptions. Can be nested without any restrictions. Concept stolen from xact by Christophe Pettus. Guarantees atomicity. Don t Give Users Guns Aimed At Feet principle. 46

47 High-level API: atomic from django.db import IntegrityError from django.db import def viewfunc(request): do_stuff() try: with transaction.atomic(): do_stuff_that_could_fail() except IntegrityError: handle_exception() do_more_stuff() 47

48 Low-level APIs In case you want to implement your own transaction management. There aren t many sane ways to combine them. 48

49 Key learnings If you don t understand transaction management in Django 1.6, get to work! ATOMIC_REQUESTS is still a reasonable idea. Use atomic whenever you need atomicity. 49

50 Thank you! Questions? 50

Transaction Management Chapter 11. Class 9: Transaction Management 1

Transaction Management Chapter 11 Class 9: Transaction Management 1 The Concurrent Update Problem To prevent errors from being introduced when concurrent updates are attempted, the application logic must