redis-lock Release 3.2.0

Transcription

1 redis-lock Release Sep 05, 2018

2

3 Contents 1 Overview Usage Features Implementation Documentation Development Requirements Similar projects Installation 7 3 Usage 9 4 Reference redis_lock Contributing Bug reports Documentation improvements Feature requests and feedback Development Authors 15 7 Changelog ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) i

4 8 Indices and tables 21 Python Module Index 23 ii

5 CHAPTER 1 Overview docs tests package Lock context manager implemented via redis SETNX/BLPOP. Free software: BSD license Interface targeted to be exactly like threading.lock. 1.1 Usage Because we don t want to require users to share the lock instance across processes you will have to give them names. Eg: conn = StrictRedis() with redis_lock.lock(conn, "name-of-the-lock"): print("got the lock. Doing some work...") time.sleep(5) Eg: lock = redis_lock.lock(conn, "name-of-the-lock") if lock.acquire(blocking=false): print("got the lock.") else: print("someone else has the lock.") 1

6 You can also associate an identifier along with the lock so that it can be retrieved later by the same process, or by a different one. This is useful in cases where the application needs to identify the lock owner (find out who currently owns the lock). Eg: import socket host_id = "owned-by-%s" % socket.gethostname() lock = redis_lock.lock(conn, "name-of-the-lock", id=host_id) if lock.acquire(blocking=false): print("got the lock.") else: if lock.get_owner_id() == host_id: print("i already acquired this in another process.") else: print("the lock is held on another machine.") Avoid dogpile effect in django The dogpile is also known as the thundering herd effect or cache stampede. Here s a pattern to avoid the problem without serving stale data. The work will be performed a single time and every client will wait for the fresh data. To use this you will need django-redis, however, python-redis-lock provides you a cache backend that has a cache method for your convenience. Just install python-redis-lock like this: pip install "python-redis-lock[django]" Now put something like this in your settings: CACHES = { 'default': { 'BACKEND': 'redis_lock.django_cache.rediscache', 'LOCATION': 'redis:// :6379/1', 'OPTIONS': { 'CLIENT_CLASS': 'django_redis.client.defaultclient' } } } Note: If using a django-redis < 3.8.x, you ll probably need redis_cache which has been deprecated in favor to django_redis. The redis_cache module is removed in django-redis versions > 3.9.x. See django-redis notes. This backend just adds a convenient.lock(name, expire=none) function to django-redis s cache backend. You would write your functions like this: from django.core.cache import cache def function(): val = cache.get(key) if val: return val else: with cache.lock(key): val = cache.get(key) if val: return val (continues on next page) 2 Chapter 1. Overview

7 else: # DO EXPENSIVE WORK val =... (continued from previous page) cache.set(key, value) return val Troubleshooting In some cases, the lock remains in redis forever (like a server blackout / redis or application crash / an unhandled exception). In such cases, the lock is not removed by restarting the application. One solution is to turn on the auto_renewal parameter in combination with expire to set a time-out on the lock, but let Lock() automatically keep resetting the expire time while your application code is executing: # Get a lock with a 60-second lifetime but keep renewing it automatically # to ensure the lock is held for as long as the Python process is running. with redis_lock.lock('my-lock', expire=60, auto_renewal=true): # Do work... Another solution is to use the reset_all() function when the application starts: # On application start/restart import redis_lock redis_lock.reset_all() Alternativelly, you can reset individual locks via the reset method. Use these carefully, if you understand what you do. 1.2 Features based on the standard SETNX recipe optional expiry optional timeout optional lock renewal (use a low expire but keep the lock active) no spinloops at acquire 1.3 Implementation redis_lock will use 2 keys for each lock named <name>: lock:<name> - a string value for the actual lock lock-signal:<name> - a list value for signaling the waiters when the lock is released This is how it works: 1.2. Features 3

8 4 Chapter 1. Overview

9 1.4 Documentation Development To run the all tests run: tox 1.6 Requirements OS Any Runtime Python 2.7, 3.3 or later, or PyPy Services Redis or later. 1.7 Similar projects bbangert/retools - acquire does spinloop distributing-locking-python-and-redis - acquire does polling cezarsa/redis_lock - acquire does not block andymccurdy/redis-py - acquire does spinloop mpessas/python-redis-lock - blocks fine but no expiration 1.4. Documentation 5

10 6 Chapter 1. Overview

11 CHAPTER 2 Installation At the command line: pip install python-redis-lock 7

12 8 Chapter 2. Installation

13 CHAPTER 3 Usage To use redis-lock in a project: import redis_lock Blocking lock: conn = StrictRedis() lock = redis_lock.lock(conn, "name-of-the-lock"): if lock.acquire(): print("got the lock. Doing some work...") time.sleep(5) Blocking lock with timeout: conn = StrictRedis() lock = redis_lock.lock(conn, "name-of-the-lock"): if lock.acquire(timeout=3): print("got the lock. Doing some work...") time.sleep(5) else: print("someone else has the lock.") Non-blocking lock: conn = StrictRedis() lock = redis_lock.lock(conn, "name-of-the-lock"): if lock.acquire(blocking=false): print("got the lock. Doing some work...") time.sleep(5) else: print("someone else has the lock.") Releasing previously acquired lock: 9

14 conn = StrictRedis() lock = redis_lock.lock(conn, "name-of-the-lock") lock.acquire() print("got the lock. Doing some work...") time.sleep(5) lock.release() The above example could be rewritten using context manager: conn = StrictRedis() with redis_lock.lock(conn, "name-of-the-lock"): print("got the lock. Doing some work...") time.sleep(5) In cases, where lock not necessarily in acquired state, and user need to ensure, that it has a matching id, example: lock1 = Lock(conn, "foo") lock1.acquire() lock2 = Lock(conn, "foo", id=lock1.id) lock2.release() 10 Chapter 3. Usage

15 CHAPTER 4 Reference 4.1 redis_lock exception redis_lock.alreadyacquired exception redis_lock.alreadystarted exception redis_lock.invalidtimeout class redis_lock.lock(redis_client, name, expire=none, id=none, auto_renewal=false, strict=true) A Lock context manager implemented via redis SETNX/BLPOP. acquire(blocking=true, timeout=none) Parameters blocking Boolean value specifying whether lock should be blocking or not. timeout An integer value specifying the maximum number of seconds to block. extend(expire=none) Extends expiration time of the lock. Parameters expire New expiration time. If None - expire provided during lock initialization will be taken. release() Releases the lock, that was acquired with the same object. Note: If you want to release a lock that you acquired in a different place you have two choices: Use Lock("name", id=id_from_other_place).release() Use Lock("name").reset() 11

16 reset() Forcibly deletes the lock. Use this with care. exception redis_lock.notacquired exception redis_lock.notexpirable exception redis_lock.timeoutnotusable exception redis_lock.timeouttoolarge redis_lock.reset_all(redis_client) Forcibly deletes all locks if its remains (like a crash reason). Use this with care. 12 Chapter 4. Reference

17 CHAPTER 5 Contributing Contributions are welcome, and they are greatly appreciated! Every little bit helps, and credit will always be given. 5.1 Bug reports When reporting a bug please include: Your operating system name and version. Any details about your local setup that might be helpful in troubleshooting. Detailed steps to reproduce the bug. 5.2 Documentation improvements redis-lock could always use more documentation, whether as part of the official redis-lock docs, in docstrings, or even on the web in blog posts, articles, and such. 5.3 Feature requests and feedback The best way to send feedback is to file an issue at If you are proposing a feature: Explain in detail how it would work. Keep the scope as narrow as possible, to make it easier to implement. Remember that this is a volunteer-driven project, and that code contributions are welcome :) 13

18 5.4 Development To set up python-redis-lock for local development: 1. Fork python-redis-lock on GitHub. 2. Clone your fork locally: git clone 3. Create a branch for local development: git checkout -b name-of-your-bugfix-or-feature Now you can make your changes locally. 4. When you re done making changes, run all the checks, doc builder and spell checker with tox one command: tox 5. Commit your changes and push your branch to GitHub: git add. git commit -m "Your detailed description of your changes." git push origin name-of-your-bugfix-or-feature 6. Submit a pull request through the GitHub website Pull Request Guidelines If you need some code review or feedback while you re developing the code just make the pull request. For merging, you should: 1. Include passing tests (run tox) Update documentation when there s new API, functionality etc. 3. Add a note to CHANGELOG.rst about the changes. 4. Add yourself to AUTHORS.rst Tips To run a subset of tests: tox -e envname -- py.test -k test_myfeature To run all the test environments in parallel (you need to pip install detox): detox 1 If you don t have all the necessary python versions available locally you can rely on Travis - it will run the tests for each change you add in the pull request. It will be slower though Chapter 5. Contributing

19 CHAPTER 6 Authors Ionel Cristian Mărieș - Rob Terhaar - Corey Farwell - Yokotoka - Jardel Weyrich - Victor Torres - Andrew Pashkin - Tero Vuotila - Joel Höner

20 16 Chapter 6. Authors

21 CHAPTER 7 Changelog ( ) Changed the signal key cleanup operation do be done without any expires. This prevents lingering keys around for some time. Contributed by Andrew Pashkin in #38. Allow locks with given id to acquire. Previously it assumed that if you specify the id then the lock was already acquired. See #44 and #39. Allow using other redis clients with a strict=false. Normally you re expected to pass in an instance of redis.strictredis. Added convenience method locked_get_or_set to Django cache backend ( ) Changed the auto renewal to automatically stop the renewal thread if lock gets garbage collected. Contributed by Andrew Pashkin in # ( ) Changed release so that it expires signal-keys immediately. Contributed by Andrew Pashkin in #28. Resetting locks (reset or reset_all) will release the lock. If there s someone waiting on the reset lock now it will acquire it. Contributed by Andrew Pashkin in #29. Added the extend method on Lock objects. Contributed by Andrew Pashkin in #24. Documentation improvements on release method. Contributed by Andrew Pashkin in #22. Fixed acquire(block=true) handling when expire option was used (it wasn t blocking indefinitely). Contributed by Tero Vuotila in #35. 17

22 Changed release to check if lock was acquired with he same id. If not, NotAcquired will be raised. Previously there was just a check if it was acquired with the same instance (self._held). BACKWARDS IN- COMPATIBLE Removed the force option from release - it wasn t really necessary and it only encourages sloppy programming. See #25. BACKWARDS INCOMPATIBLE Dropped tests for Python 2.6. It may work but it is unsupported ( ) Added the timeout option. Contributed by Victor Torres in # ( ) Added the auto_renewal option. Contributed by Nick Groenen in # ( ) New specific exception classes: AlreadyAcquired and NotAcquired. Slightly improved efficiency when non-waiting acquires are used ( ) Rename Lock.token to Lock.id. Now only allowed to be set via constructor. Contributed by Jardel Weyrich in # ( ) Fix Django integration. (reported by Jardel Weyrich) Reorganize tests to use py.test. Add test for Django integration. Add reset_all functionality. Contributed by Yokotoka in #7. Add Lock.reset functionality. Expose the Lock.token attribute ( )? 18 Chapter 7. Changelog

23 ( )? ( )? ( ) First release on PyPI ( ) 19

24 20 Chapter 7. Changelog

25 CHAPTER 8 Indices and tables genindex modindex search 21

26 22 Chapter 8. Indices and tables

27 Python Module Index r redis_lock, 11 23

28 24 Python Module Index

29 Index A acquire() (redis_lock.lock method), 11 AlreadyAcquired, 11 AlreadyStarted, 11 E extend() (redis_lock.lock method), 11 I InvalidTimeout, 11 L Lock (class in redis_lock), 11 N NotAcquired, 12 NotExpirable, 12 R redis_lock (module), 11 release() (redis_lock.lock method), 11 reset() (redis_lock.lock method), 11 reset_all() (in module redis_lock), 12 T TimeoutNotUsable, 12 TimeoutTooLarge, 12 25