Implementation of Synchronizers

Transcription

1 Implementation of Synchronizers This document describes the implementation of synchronizers, including the synchronizer core that all synchronizers share, and the implementation of service synchronizers. For an overview of the principles involved in the Synchronizer and its design, refer to Design of Synchronizers. There are three types of synchronizers: Work-based Synchronizers, Event-based Synchronizers, and Hybrid Synchronizers (the last of which subsume the functionalities of the first two). Work-based synchronizers are somewhat cumbersome to implement, but offer strong robustness guarantees such as causal consistency, retries in the face of failure, model-dependency analysis and concurrent scheduling of synchronization modules. Event-based synchronizers are simpler to implement, but lack the aforementioned guarantees. Difference between Work-based and Event-based Synchronizers Mechanism Work-based Synchronizers Event-based Synchronizers Control-logic Binding Check if models are up-to-date based on their content React to events notifying of model updates Implementation constraints Modules have to be idempotent Modules are not required to be idempotent Dependencies Modules are executed in dependency order Modules are executed reactively in an arbitrary order Concurrency Error Handling Non-dependent modules are executed concurrently Errors are propagated to dependencies, retries on failure Modules are executed sequentially No error dependency. It's up to the Synchronizer to cope with event loss Ease of implementation Moderate Easy Implementing an Event-based Synchronizer Note: The following instructions assume that you have created one or more models for your for service, and placed them in a django app with the name of your service. An Event-based Synchronizer is a collection of "Watcher" modules. Each Watcher module listens for ("watches") events pertaining to a particular model. The Synchronizer developer must provide the set of these modules. The steps for assembling a synchronizer once these modules have been implemented, are as follows: Run the gen_watcher.py script: gen_watcher.py <name of your app> Set your Synchronizer-specific config options in the config file, and also set observer_enable_watchers to true. FIXME Install python-redis by running "pip install redis" in your Synchronizer container FIXME Link the redis container that comes packaged with xos with your Synchronizer container as "redis'" FIXME Drop your watcher modules in the directory /opt/xos/synchronizers/<your synchronizer>/steps Run your synchronizer by running /opt/xos/synchronizers/<your synchronizer>/run-synchronizer.sh Watcher Module API Type Description handle_watched_object def handle_watched_object(self, o) This method is called every time a watched object is added, deleted, or updated. watch_degree int Defines the set of watched models implicitly. If this module synchronizes models A and B, then the watched set is defined by the models that are a distance of watch_degree from A or from B in the model dependency graph.

2 watched list of type ModelLink Defines the set of watched models explicitly. If this is defined, then watch_degree is ignored. synchronizes list of type Model The model that this module synchronizes. The main body of a watcher module is the function handle_watched_object, which responds to operations on objects that the module synchronizes. If the module responds to multiple object types, then it must determine the type of object, and proceed to process it accordingly. def handle_watched_object(self, o): if (type(o) is Slice): self.handle_changed_slice(o) elif (type(o) is Node): self.handle_changed_node(o) Linking the Watcher into the Synchronizer There are two ways of linking in a Watcher. Using them both does not hurt. The first method is complex but robust, and involves making the declaration in the data model, by ensuring that the model that your synchronizer would like to watch is linked to the model that it actuates. For instance, if your synchronizer actuates a service model called Fabric, which links the Instance model, then you would ensure that Instance is a dependency of Fabric by making the following annotation in the Fabric model: class Fabric(Service): xos_links = [ModelLink(Instance,via='instance',into='ip')] There can be several ModelLink specifications in a single xos_links declaration, each encapsulating the referenced model, the field in the current model that links to it, and the destination field in which the watcher is interested. If into is omitted, then the watcher is notified of all changes in the linked model, irrespective of the fields that change. The above change needs to be backed up with an instruction to the synchronizer that the watcher is interested in being notified of changes to its dependencies. This is done through a watch_degree annotation. class SyncFabricService(SyncStep): watch_degree=1 By default, watch_degree is 0, meaning that the Synchronizer watches nothing. When watch degree is 1, it watches 1 level of dependencies removed, and so on. If the watch_degree in the above code were 2, then this module would also get notified of changes in dependencies of the Instance model. The second way of linking in a watcher is to hardcode the watched model directly in the synchronizer: class SyncFabricService(SyncStep): watched = [ModelLink(Instance,via='instance',into='ip')] Linking the Watcher into the Synchronizer Set the observer_enable_watchers option to true in your xos synchronizer config file Add a link between your synchronizer container and the redis container by including the following lines in the definition of your synchronizer's docker-compose file. You may need to adapt these to the name of the project used (e.g. cordpod) external_links: xos_redis:redis

3 Ensure that there is a similar link between your XOS UI container and the redis container In addition to the above development tasks, you also need to make the following changes to your configuration to activate watchers. Implementing a Work-based Synchronizer Note: The following instructions assume that you have created one or more models for your for service, and placed them in a django app with the name of your service. A work-based Synchronizer is a collection of "Actuator" modules. Each Actuator module is invoked when a model is found to be outdated relative to its last synchronization. An actuator module can be self-contained and written entirely in Python, or it can be broken into a "dispatcher" and "payload", with the dispatcher implemented in Python and the payload implemented using Ansible. The Synchronizer core has built-in support for the dispatch of Ansible modules and helps extract parameters from the synchronized model and translate them into the parameters required by the corresponding Ansible script. It also tracks an hierarchically structured list of such ansible scripts on the filesystem, for operators to use to inspect and debug a system. The procedure for building a work-based synchronizer is as follows: Run the gen_workbased.py script. gen_workbased <app name>. Set your Synchronizer-specific config options in the config file, and also set observer_enable_watchers to False. FIXME Drop your actuator modules in the directory /opt/xos/synchronizers/<your synchronizer>/steps Run your synchronizer by running /opt/xos/synchronizers/<your synchronizer>/run-synchronizer.sh Actuator Module API Type Description synchronizes list of type Model The set of models that the module synchronizes sync_record def sync_record(self, object) A coarse-grained method that handles outdated objects. delete_record def delete_record(self, object) A coarse-grained method that handles object deletion in the back end get_extra_attributes def get_extra_attributes(self, object) A method that maps an object to the parameters required by its Ansible payload. Returns a dict with those parameters and their values. fetch_pending def fetch_pending(self, deleted) A method that fetches the set of pending objects from the database. The synchronizer core provides a default implementation. Override only if you have a reason to do so. template_name string The name of the Ansible script that directly interacts with the underlying substrate. Implementing a Step with Ansible To implement a step using Ansible, a developer must provide two things: an Ansible recipe, and a get_extra_attributes method, which maps attributes of the object into a dictionary that configures that Ansible recipe. The Ansible recipe comes in two parts, an inner payload and a wrapper that delivers that payload to the VMs associated with the service. The wrapper itself comes in two parts. A part that sets up the preliminaries:

4 --- - hosts: "{{ instance_name }}" connection: ssh user: ubuntu sudo: yes gather_facts: no vars: - package_location: "{{ package_location }}" - server_port: "{{ server_port }}" The template variables package_location and server_port come out of the Python part of the Synchronizer implementation (discussed shortly). The outer wrapper then includes a set of Ansible roles that perform the required actions: roles: - download_packages - configure_packages - start_server The "payload" of the Ansible recipe contains an implementation of the roles, in this case, download_packages, configure_packages and start_server. The concrete values of parameters required by the Ansible payload are provided in the implementation of the "get_extra_attributes" method in the Python part of the Synchronizer. This method receives an object from the data model and is charged with the task of converting the properties of that object into the set of properties required by the Ansible recipe, which are returned as a Python dictionary. def get_extra_attributes(self, o): fields = {} fields['package_location'] = o.package_location fields['server_port'] = o.server_port return fields Implementing a Step without Ansible To implement a step without using Ansible, a developer need only implement the sync_record and delete_record methods, which get called for every pending object. These methods interact directly with the underlying substrate. Managing Dependencies If your data models have dependencies between them, so that for one to be synchronized, another must already have been synchronized, then you can define such dependencies in your data model. The Synchronizer automatically picks up such dependencies and ensures that the steps corresponding to the models in questions are executed in a valid order. It also ensures that any errors that arise propagate from the affected objects to its dependents, and that the dependents are held up until the errors have been resolved and the dependencies have been successfully synchronized. In the absence of failures, the Synchronizer tries to execute your synchronization steps concurrently to whatever extent this is possible while still honoring dependencies.

5 <in the definition of your model> xos_links = [ModelLink(dest=MyServiceUser,via='user'),ModelLink (dest=myservicedevice,via='device')] In the above example, the xos_links field declares two dependencies. The name "xos_links" is key, and so the field should be named as such. The dependencies are contained in a list of type ModelLink, each of which defines a type of object (a model) and an "accessor" field via which a related object of that type can be accessed. Handling Errors To fault synchronization, you can raise an exception in any of the methods of your step that are automatically called by the synchronizer core. These include fetch_pending, sync_record and delete_record. The outcome of such exceptions has multiple parts: The synchronization of the present object is deferred. The synchronization of dependent objects is deferred, if those objects are accessible via the current object (see the 'via' field) A string representation of your exception is propagated into a scratchpad in your model, which in turn appears in your UI. When you click the object in question, in the UI, you should see the error message. The synchronization state of your object, and of dependent objects changes to "Error" and a red icon appears next to it. If the object repeatedly fails to synchronize, then its synchronization interval is increased exponentially. Sometimes, you may encounter a temporary error, which you think may be resolved shortly, by the time the Synchronizer runs again. In these cases, you can raise a "DeferredException." This error type differs from a general exception in two ways: It does not put your object in error state It disabled exponential backoff - the Synchronizer tries to synchronize your object every single time. Synchronizer Configuration Options Option Default Purpose observer_disabled False A directive to run the XOS server in Observer-less mode. Events are not relayed to the Observer and no bookkeeping is done. observer_steps_dir n/a The path of the directory in which the Synchronizer will look for your watcher and actuator modules. observer_sys_dir n/a The path of the directory that enlists backend objects your synchronizer creates. This is like the /sys directory in an operating system. Each entry is a file that contains an Ansible recipe that creates, updates or deletes your object. When you debug your synchronizer, you can run these files manually. observer_pretend False This option runs the Synchronizer in "pretend" mode, in which synchronizer modules that use Ansible run in emulated mode, and do not actually execute backend API calls. observer_proxy_ssh n/a observer_name n/a The name of your Synchronizer. This is a re quired option. observer_applist core A list consisting of the Django apps that your Synchronizer uses. observer_dependency_graph /opt/xos/model-deps Dependencies between various models that your Synchronizer services. These are generated automatically by the Synchronizer utility "dmdot."

6 observer_backoff_disabled True Models whose synchronization fails are reexecuted, but with intervals that increase exponentially. This option disables the exponential growth of the intervals. observer_logstash_hostport n/a The host name and port number (e.g. xosmo nitor.org:4132) to which the Synchronizer streams its logs, on which a logstash server is running. observer_log_file n/a The log file into which the Synchronizer logs are published. observer_model_policies_dir n/a The directory in which model policies are stored.