Kuyruk Documentation. Release 0. Cenk Altı

Transcription

1 Kuyruk Documentation Release 0 Cenk Altı Mar 07, 2018

2

3 Contents 1 About Kuyruk 3 2 User s Guide 5 3 API Reference 17 4 Indices and tables 21 Python Module Index 23 i

4 ii

5 Welcome to Kuyruk s documentation. This documentation covers the usage of the library and describes how to run workers. Contents 1

6 2 Contents

7 CHAPTER 1 About Kuyruk Kuyruk is a simple and easy way of distributing tasks to run on servers. It uses RabbitMQ as message broker and depends on amqp which is a pure-python RabbitMQ client library. Compatible with Python All design decisions is based on simplicity. Speed is not first priority. Kuyruk only supports RabbitMQ and does not plan to support any other backend. Kuyruk requires no change in client code other than adding a decorator on top of a function in order to convert it to a Task. Also, sensible defaults have been set for configuration options and can be run without configuring during the development of your application. Kuyruk is designed for long running mission critical jobs. For this reason Kuyruk sends acknowledgement only after the task is fully processed. 3

8 4 Chapter 1. About Kuyruk

9 CHAPTER 2 User s Guide 2.1 Getting Started Running a function in background requires only few steps with Kuyruk. This tutorial assumes that you have a running RabbitMQ server on localhost with default configuration. Following files and commands are in example directory for convenience Installing Kuyruk is available on PyPI. You can install it via pip. $ pip install kuyruk Defining Tasks Instantiate a Kuyruk object and put a task() decorator on top of your function. This will convert your function into a Task object. # tasks.py from kuyruk import Kuyruk kuyruk = def echo(message): print message You can specify some options when defining task. See task() for details. 5

10 2.1.3 Sending the Task to RabbitMQ When you call the Task object, Kuyruk will serialize the task as JSON and will send it to a queue on RabbitMQ instead of running it. import tasks tasks.echo("hello, World!") Running a Worker Run the following command to process tasks in default queue. $ kuyruk --app tasks.kuyruk worker 2.2 Worker Worker processes tasks from a queue Usage $ kuyruk --app <path.to.kuyruk.instance> --queue <queue_name> If queue_name is not given default queue name( kuyruk ) is used. Example: $ kuyruk --app tasks.kuyruk --queue download_file OS Signals Description of how worker processes react to OS signals. SIGINT Worker exits after completing the current task. This is the signal sent when you press CTRL-C on your keyboard. SIGTERM Worker exits after completing the current task. SIGQUIT Worker quits immediately. This is unclean shutdown. If worker is running a task it will be requeued by RabbitMQ. This is the signal sent when you press CTRL-on your keyboard. SIGUSR1 Prints stacktrace. Useful for debugging stuck tasks or seeing what the worker is doing. SIGUSR2 Discard current task and proceed to next one. Discarded task will not be requeued by RabbitMQ. SIGHUP Used internally to fail the task when connection to RabbitMQ is lost during the execution of a long running task. Do not use it. 6 Chapter 2. User s Guide

11 2.3 Signals Kuyruk can be extended via signals. Kuyruk has a signalling support via Blinker library Example Here is an example for clearing the SQLAlchemy s scoped_session before executing the function and commiting it after the task is executed: from kuyruk import Kuyruk from kuyruk.signals import task_prerun, task_postrun from myapp.orm import Session kuyruk = def open_session(sender, task=none, **extra): task.session = def close_session(sender, task=none, **extra): def task_with_a_session(): session = task_with_a_session.session # Work with the session... session.commit() List of Signals kuyruk.signals.task_init = <blinker.base.signal object> Sent when the task decorator is applied. Arguments: sender: Kuyruk object task: Task object kuyruk.signals.task_preapply = <blinker.base.signal object> Sent before the task is applied. Arguments: sender: Kuyruk object task: Task object args: Positional arguments to the task kwargs: Keyword arguments to the task kuyruk.signals.task_postapply = <blinker.base.signal object> Sent after the task is applied. Arguments: 2.3. Signals 7

12 sender: Kuyruk object task: Task object args: Positional arguments to the task kwargs: Keyword arguments to the task kuyruk.signals.task_prerun = <blinker.base.signal object> Sent before the wrapped function is executed. Arguments: sender: Kuyruk object task: Task object args: Positional arguments to the task kwargs: Keyword arguments to the task kuyruk.signals.task_postrun = <blinker.base.signal object> Sent after the wrapped function is executed. Arguments: sender: Kuyruk object task: Task object args: Positional arguments to the task kwargs: Keyword arguments to the task kuyruk.signals.task_success = <blinker.base.signal object> Sent when the wrapped function is returned. Arguments: sender: Kuyruk object task: Task object args: Positional arguments to the task kwargs: Keyword arguments to the task kuyruk.signals.task_error = <blinker.base.signal object> Sent when the wrapped function raises an exception. Arguments: sender: Kuyruk object task: Task object args: Positional arguments to the task kwargs: Keyword arguments to the task exc_info: Return value of sys.exc_info() kuyruk.signals.task_failure = <blinker.base.signal object> Sent when the task fails after all retries(if any). Arguments: sender: Kuyruk object task: Task object 8 Chapter 2. User s Guide

13 args: Positional arguments to the task kwargs: Keyword arguments to the task exc_info: Return value of sys.exc_info() kuyruk.signals.task_presend = <blinker.base.signal object> Sent before the task is sent to queue. Arguments: sender: Kuyruk object task: Task object args: Positional arguments to the task kwargs: Keyword arguments to the task description: dict representation of the task kuyruk.signals.task_postsend = <blinker.base.signal object> Sent after the task is sent to queue. Arguments: sender: Kuyruk object task: Task object args: Positional arguments to the task kwargs: Keyword arguments to the task description: dict representation of the task kuyruk.signals.worker_failure = <blinker.base.signal object> Sent when the task fails. Arguments: sender: Kuyruk object worker: The Worker object task: Task object args: Positional arguments to the task kwargs: Keyword arguments to the task description: dict representation of the task exc_info: Return value of sys.exc_info() kuyruk.signals.worker_init = <blinker.base.signal object> Sent when the worker is initialized. Arguments: sender: Kuyruk object worker: The Worker object kuyruk.signals.worker_start = <blinker.base.signal object> Sent when the worker is started. Arguments: sender: Kuyruk object 2.3. Signals 9

14 worker: The Worker object kuyruk.signals.worker_shutdown = <blinker.base.signal object> Sent when the worker shuts down. Arguments: sender: Kuyruk object worker: The Worker object 2.4 Extensions These are some extensions that adds a functionality to Kuyruk Kuyruk-Sentry Sends exceptions in Kuyruk workers to Sentry Kuyruk-Requeue Save failed tasks to Redis and requeue them Kuyruk-Manager See and manage Kuyruk workers Changelog Here you can see the full list of changes between each Kuyruk release Version Released on Added Windows support Version Released on Added Kuyruk.send_tasks_to_queue method for batch sending. 10 Chapter 2. User s Guide

15 2.5.3 Version Released on Added delay for failed and rejected taks Version Released on Added RabbitMQ connection timeout values to config Version Released on Removed queue key from task description dictionary. Improve error handling when closing connections and channels Version Released on Removed local argument from Kuyruk.task decorator Version Released on Changed WORKER_MAX_LOAD behavior. None disables the feature. Set to -1 for number of CPUs on host. Add argumens to worker command to override WORKER_MAX_LOAD and WORKER_MAX_RUN_TIME config values. Renamed ConnectionError to HeartbeatError. Removed Task.run_in_queue method. Use Task.send_to_queue with wait_result argument instead. Removed local argument from Task.send_to_queue method. Pass host="localhost" for sending to local queue. Removed local argument from worker command. If queue name ends with ".localhost" hostname will be appended to queue name. Removed deprecated Worker.queue property. Removed WORKER_LOGGING_CONFIG configuration value. Added --logging-level to worker command arguments. Removed Worker.config property. Added message_ttl argument to Task.send_to_queue method Changelog 11

16 2.5.8 Version Released on Added Task.run_in_queue context manager for getting task results Version Released on Removed kuyruk_host and kuyruk_local arguments from task calls. Exported Task.send_to_queue method Version Released on Fixed 1 second delay after processing a task. Fixed a depreciation warning from recent version of amqp library Version Released on Workers can consume tasks from multiple queues Version Released on Export Task.name property for fixing a bug in kuyruk-manager Version Released on Fixed a bug related with Python , uwsgi and setproctitle Version Released on This is major rewrite of Kuyruk and it is not backwards compatible. Added Python 3 support. Replaced pika with amqp. Fixed multi-threading issues. Removed master subcommand. Removed scheduler subcommand. 12 Chapter 2. User s Guide

17 Removed requeue subcommand. Removed manager subcommand. Exceptions are not sent to Sentry. Failed tasks are not saved to Redis anymore. Failed tasks are retried in the same worker. Unknown keys in config are now errors. Changed some config variable names. Worker command takes Kuyruk instance instead of config file Version Released on Fixed a worker startup bug happens when running workers as another user Version Released on Added periodic task scheduler feature Version Released on Added Task.delay() function alias for easy migration from Celery Version Released on Use rpyc library for manager communication Version Released on Reverted the option to give Task class from configuration. This caused master to import from user code. Added sleep after respawn_worker to prevent cpu burning Version Released on Added the option to give Task class from configuration Changelog 13

18 Version Released on Prevented close to be called on a nonexistent connection Version Released on Fix the bug about freezing processes on exit Version Released on Fix unclosed socket error on manager Version Released on Removed InvalidCall exception type. TypeError or AttributeError is raised instead. If a kuyruk process exits with a signal, the exit code will be Version Released on Master uses os.wait() instead of polling workers every second Version Released on Use forking again instead Popen after fixing import issue. Add Quit Task button to Manager interface Version Released on Drop support for Python 2.6. Switch back to subprocess module from forking Version Released on Use fork() directly instead of subprocess.popen() when starting workers from master. 14 Chapter 2. User s Guide

19 Version Released on First public release Changelog 15

20 16 Chapter 2. User s Guide

21 CHAPTER 3 API Reference 3.1 Kuyruk Class class kuyruk.kuyruk(config: kuyruk.config.config = None) None Provides task() decorator to convert a function into a Task. Provides channel() context manager for opening a new channel on the connection. Connection is opened when the first channel is created. Parameters config Must be an instance of Config. If None, default config is used. See Config for default values. channel() typing.iterator[amqp.channel.channel] Returns a new channel from a new connection as a context manager. connection() typing.iterator[amqp.connection.connection] Returns a new connection as a context manager. task(queue: str = kuyruk, **kwargs: typing.any) typing.callable Wrap functions with this decorator to convert them to tasks. After wrapping, calling the function will send a message to a queue instead of running the function. Parameters queue Queue name for the tasks. kwargs Keyword arguments will be passed to Task constructor. Returns Callable Task object wrapping the original function. 3.2 Task Class class kuyruk.task Calling a Task object serializes the task to JSON and sends it to the queue. Parameters 17

22 retry Retry this times before give up. The failed task will be retried in the same worker. max_run_time Maximum allowed time in seconds for task to complete. apply(*args: typing.any, **kwargs: typing.any) typing.any Called by workers to run the wrapped function. You may call it yourself if you want to run the task in current process without sending to the queue. If task has a retry property it will be retried on failure. If task has a max_run_time property the task will not be allowed to run more than that. name Full path to the task in the form of <module>.<function>. Workers find and import tasks by this path. send_to_queue(args: typing.tuple = (), kwargs: typing.dict[str, typing.any] = {}, host: str = None, wait_result: typing.union[int, float] = None, message_ttl: typing.union[int, float] = None) typing.any Sends a message to the queue. A worker will run the task s function when it receives the message. Parameters args Arguments that will be passed to task on execution. kwargs Keyword arguments that will be passed to task on execution. host Send this task to specific host. host will be appended to the queue name. If host is localhost, hostname of the server will be appended to the queue name. wait_result Wait for result from worker for wait_result seconds. If timeout occurs, ResultTimeout is raised. If excecption occurs in worker, RemoteException is raised. message_ttl If set, message will be destroyed in queue after message_ttl seconds. Returns Result from worker if wait_result is set, else None. 3.3 Config Class class kuyruk.config Kuyruk configuration object. Default values are defined as class attributes. Additional attributes may be added by extensions. EAGER = False Run tasks in the process without sending to queue. Useful in tests. RABBIT_HOST = 'localhost' RabbitMQ host. RABBIT_PASSWORD = 'guest' RabbitMQ password. RABBIT_PORT = 5672 RabbitMQ port. RABBIT_USER = 'guest' RabbitMQ user. RABBIT_VIRTUAL_HOST = '/' RabbitMQ virtual host. 18 Chapter 3. API Reference

23 WORKER_LOGGING_LEVEL = 'INFO' Logging level of root logger. WORKER_MAX_LOAD = None Pause consuming queue when the load goes above this level. WORKER_MAX_RUN_TIME = None Gracefully shutdown worker after running this seconds. from_dict(d: typing.dict[str, typing.any]) None Load values from a dict. from_env_vars() None Load values from environment variables. Keys must start with KUYRUK_. from_object(obj: typing.any) None Load values from an object. from_pyfile(filename: str) None Load values from a Python file. 3.4 Exceptions exception kuyruk.exceptions.discard The task may raise this if it does not want to process the message. The message will be dropped. typ- exception kuyruk.exceptions.heartbeaterror(exc_info: ing.tuple[typing.type[baseexception], BaseException, traceback]) None Raised when there is problem while sending heartbeat during task execution. exception kuyruk.exceptions.kuyrukerror Base class for Kuyruk exceptions. exception kuyruk.exceptions.reject The task may raise this if it does not want to process the message. The message will be requeued and delivered to another worker. exception kuyruk.exceptions.remoteexception(type_: typing.type, value: Exception, traceback: traceback) None Raised from kuyruk.task.send_to_queue() if wait_result is set and exception is raised on the worker while running the task. exception kuyruk.exceptions.resulttimeout Raised from kuyruk.task.send_to_queue() if wait_result is set and reply is not received in wait_result seconds. exception kuyruk.exceptions.timeout Raised if a task exceeds it s allowed run time Exceptions 19

24 20 Chapter 3. API Reference

25 CHAPTER 4 Indices and tables genindex search 21

26 22 Chapter 4. Indices and tables

27 Python Module Index k kuyruk.exceptions, 19 kuyruk.signals, 7 23

28 24 Python Module Index

29 Index A apply() (kuyruk.task method), 18 C channel() (kuyruk.kuyruk method), 17 Config (class in kuyruk), 18 connection() (kuyruk.kuyruk method), 17 D Discard, 19 E EAGER (kuyruk.config attribute), 18 F from_dict() (kuyruk.config method), 19 from_env_vars() (kuyruk.config method), 19 from_object() (kuyruk.config method), 19 from_pyfile() (kuyruk.config method), 19 H HeartbeatError, 19 K Kuyruk (class in kuyruk), 17 kuyruk.exceptions (module), 19 kuyruk.signals (module), 7 KuyrukError, 19 N name (kuyruk.task attribute), 18 R RABBIT_HOST (kuyruk.config attribute), 18 RABBIT_PASSWORD (kuyruk.config attribute), 18 RABBIT_PORT (kuyruk.config attribute), 18 RABBIT_USER (kuyruk.config attribute), 18 RABBIT_VIRTUAL_HOST (kuyruk.config attribute), 18 Reject, 19 RemoteException, 19 ResultTimeout, 19 S send_to_queue() (kuyruk.task method), 18 T Task (class in kuyruk), 17 task() (kuyruk.kuyruk method), 17 task_error (in module kuyruk.signals), 8 task_failure (in module kuyruk.signals), 8 task_init (in module kuyruk.signals), 7 task_postapply (in module kuyruk.signals), 7 task_postrun (in module kuyruk.signals), 8 task_postsend (in module kuyruk.signals), 9 task_preapply (in module kuyruk.signals), 7 task_prerun (in module kuyruk.signals), 8 task_presend (in module kuyruk.signals), 9 task_success (in module kuyruk.signals), 8 Timeout, 19 W worker_failure (in module kuyruk.signals), 9 worker_init (in module kuyruk.signals), 9 WORKER_LOGGING_LEVEL (kuyruk.config attribute), 18 WORKER_MAX_LOAD (kuyruk.config attribute), 19 WORKER_MAX_RUN_TIME (kuyruk.config attribute), 19 worker_shutdown (in module kuyruk.signals), 10 worker_start (in module kuyruk.signals), 9 25