josync Documentation Release 1.0 Joel Goop and Jonas Einarsson May 10, 2014
Contents 1 Contents 3 1.1 Getting started.............................................. 3 1.2 Jobs.................................................... 5 1.3 Failure notifications........................................... 6 1.4 Logging.................................................. 7 1.5 Command-line options.......................................... 7 1.6 Authors & license............................................ 8 1.7 Code documentation........................................... 8 2 Indices and tables 11 Python Module Index 13 i
ii
Josync is a light-weight Python application for Windows backup using rsync/cygwin, using shadow copies for read access to open files. Josync is open-source, come join us on GitHub https://github.com/jeinarsson/josync Features: open-source (MIT License, see Authors & license) backup jobs are specified in minimal JSON configuration files, e-mail failure notifications, extensive and configurable logging, translation of mapped network drive letters to UNC paths in order to avoid problems with inaccessible mounted drives on startup It s not difficult, it s just simple. Get started! If you find bugs, or have feature suggestions, please use the Issue Tracker at Github: https://github.com/jeinarsson/josync/issues Contents 1
2 Contents
CHAPTER 1 Contents 1.1 Getting started 1.1.1 Prerequisites Running Josync 1.x requires, in addition to Josync itself, Python 2.7, Download and install from https://www.python.org/downloads/ Cygwin with rsync and cygpath binaries, Download and install from http://cygwin.com/install.html Cygwin has its own installer offering many, many packages. For Josync, you need only to check rsync in the category net. The installer will find the dependencies automatically. A copy of vshadow.exe for your operating system. Either (recommended) Or * download a zip with binaries from the GitHub 1.x release https://github.com/jeinarsson/josync/releases and find the version appropriate to your OS. * install the Windows SDK for your Windows version and find vshadow.exe from the Samples section. 1.1.2 Getting Josync Either Download the latest 1.x release zip file from https://github.com/jeinarsson/josync/releases Or, if you re familiar with git, checkout the stable_1 branch. git clone https://github.com/jeinarsson/josync.git -b stable_1 3
1.1.3 Basic configuration Josync reads its configuration from a JSON-formatted files. To get started Josync needs to know where to find the external binaries. By default Josync looks for them in the current working directory. If they are available elsewhere, specify the corresponding paths in default.josync-config. For example: { } "cygwin_bin_path": "C:/cygwin64/bin", "vshadow_bin": "d:/projects/josync/ext/vshadow.exe", Note: you may also create a file called user.josync-config containing your settings. Any configuration options specified there will override the options in default.josync-config. We use this to avoid Git replacing our settings. 1.1.4 Running a basic sync job Josync reads job descriptions from JSON-formatted job-files. Josync 1.x contains a sample job description in syncjob-example.josync-job, it looks like this: { } "type": "sync", "sources": [ {"path": "d:/phd", "excludes": []}, {"path": "d:/projects", "excludes": ["*.pyc;*.pyo"]} ], "global_excludes": ["*.hdf5"], "target": "g:/josync Backups/work" Job descriptions consist of a job type, a list of source paths and a target path. Optionally, a job description may contain exclusion patterns. Excludes may be specified either globally or per source. Modify the job description to make sense on your system, and try running this job: python josync.py syncjob-example 1.1.5 Scheduling When we schedule Josync to run using the built-in Task Scheduler in Windows, we execute with pythonw.exe instead of python.exe. The only difference is that pythonw.exe does not open a command-line window, but runs quietly in the background. The Action of the task can look like this (click for larger version): 4 Chapter 1. Contents
Screenshot of Task Scheduler 1.1.6 Next steps Now that Josync 1.0 runs you probably want to check out Failure notifications, know more about and configure the log files written by Josync: Logging, read the details about the available job types 1.2 Jobs This version of Josync provides two types of jobs: 1. sync 2. add All job types support optional Failure notifications. 1.2.1 sync A sync job does what you expect from a backup program: it mirrors the source onto the target, and only updates changed files. In particular, when files are removed on the source, they are also removed on the target. Example jobfile: { } "type": "sync", "sources": [ {"path": "d:/phd", "excludes": []}, {"path": "d:/projects", "excludes": ["*.pyc;*.pyo"]} ], "global_excludes": ["*.hdf5"], "target": "g:/josync Backups/work" 1.2. Jobs 5
1.2.2 add An add job copies a source onto a target, but never removes anything from the target. The typical use-case is a music library stored centrally. Imagine that you keep a subset of the music library on your local computer, and you add some new music to your local library. You would like to copy this new music to the main library, but you cannot use a sync job, because that will remove everything on the target that is not on your local computer. A Josync add job will add your new addition to the main library store, but it will never remove anything. Example jobfile: { } "type": "add", "sources": [ {"path": "d:/music", "excludes": []} ], "global_excludes": ["*.hdf5"], "target": "g:/music" 1.3 Failure notifications Josync can send an e-mail to you if a job fails, for example if the sync target is not found, or is full. The e-mail receipient and notification trigger is configured per job, in the.josync-job-file. The required settings for sending e-mail must be configured in default.josync-config. If you have configured failure notifications but would like to run Josync with notifications disabled, use the command line option --nonotifications. 1.3.1 Configuring notification for a job The notifications are configured in the job description. For example: { } "type": "sync", "sources": [ {"path": "d:/phd", "excludes": []}, {"path": "d:/projects", "excludes": ["*.pyc;*.pyo"]} ], "global_excludes": ["*.hdf5"], "target": "g:/josync Backups/work", "failure_notification": { "hours_since_success": 4, "e-mail": "me@domain.com" } In this example a notification e-mail will be sent to me@domain.com if the job fails and it was more than 4 hours since the last successful run. If hours_since_success is omitted, or set to 0, an e-mail is sent on every failure. 6 Chapter 1. Contents
1.3.2 Configuring SMTP settings In order to send e-mail, Josync requires an SMTP-server. The SMTP settings goes in default.josync-config, like this: { } "cygwin_bin_path": ".", "vshadow_bin": "./vshadow.exe", "smtp": { "host": "mysmtp.com", "port": 465, "username": "username", "password": "password", "from_address": "me@domain.com" } You can test your settings by running: python testmail.py your@address.com This command will try to send an e-mail to your@address.com with the Josync settings. 1.4 Logging Josync provides extensive logging through Python s built in logging modules. By default a summarized log documenting success or failure of each run is appended to a common log file named main.josync-log. More detailed logs are written to {job file name}.josync-job-log. If you are happy with the log file names and formatting you don t have to touch the logging settings! 1.4.1 Settings Logging is configured through the JSON file logging.josync-config. Settings are read as a dictionary and fed to the logging module through logging.config.dictconfig(). One handler is created for each log file and a separate handler is created for logging to stdout. The Python logging module documentation describes the the handlers and formatters that determine where log messages are sent and how they are formatted for each handler. These can be defined in the configuration file and added either to the root logger handling all log messages or to specific loggers defined in the file. Josync uses one logger for each module (python file) and a separate logger josync_run for the main log. If you want to customize the logging configuration, have a look at the example dictionary config found in the Python logging cookbook. 1.5 Command-line options usage: josync.py [-h] [--debug] [--nonotifications] [--dry-run] jobfile positional arguments: jobfile path to job file specifying josync job optional arguments: -h, --help show this help message and exit 1.4. Logging 7
--debug --nonotifications --dry-run set all loggers to debug level disable notifications on backup failure send dry-run to rsync and do not actually transfer any files 1.6 Authors & license Josync is developed by Joel Goop & Jonas Einarsson. Josync is released under the MIT License. The MIT License (MIT) Copyright (c) 2014 Joel Goop & Jonas Einarsson Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the Software ), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARIS- ING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 1.7 Code documentation Documentation of code for developers. 1.7.1 jobs.py class jobs.additivejob(params) Updating target with new files from sources. class jobs.basesyncjob(params) Base class for sync-type jobs. run() Run rsync to sync one or more sources with one target directory. class jobs.job(params) Parent class for backup jobs. excludes_to_options(excludes) Convert a list of strings to a list of exclude options to rsync. Parameters excludes List of excludes. 8 Chapter 1. Contents
class jobs.syncjob(params) Simple backup syncing multiple sources to a target directory with full tree structure. jobs.create_job_from_file(job_file) Creates a job from a JSON job specification. Parameters job_file (str) Path to job file. Returns Job object of specified type. 1.7.2 utils.py class utils.failurenotifier(job_file) Keeps track of when the last time a job was successfully run. Notifies user per e-mail. notify() Check if conditions for notification are fulfilled, and send e-mail. record_successful_run() Create an empty.josync-job-success file to mark a successful run. class utils.rsync(source, target, options=none) Sub-class of subprocess.popen to run rsync process. output_thread(pipe, send) Start thread handling output from rsync. utils.enumerate_net_drives() Runs NET USE and parses output. Returns List of drive letters and corresponding UNC path as dictionary. utils.get_cygwin_path(path) Return cygwin path for a given windows path. Parameters path (str) The windows path to convert. Returns str The cygwin path to path. Raises IOError utils.get_file_modification_date(filename) Read last modified metadata of file Parameters filename (str) filename (with path) Returns Datetime object with last modified datetime utils.is_net_drive(drive) Looks up drive in net_drives Returns True if drive is a net drive (is in net_drives list) utils.read_config(default_cfg, user_cfg) Reads config file and checks values. Parameters default_cfg (str) Path to default config file. user_cfg (str) Path to user config file. Raises KeyError, IOError 1.7. Code documentation 9
utils.send_email(msg_address, msg_subject, msg_body) Sends an e-mail notification using SMTP settings in config. Parameters msg_subject (str) Message subject. msg_body (str) Message body. user_email (str) E-mail to send notification to. utils.shell_execute(command) Run command with subprocess.popen Parameters command (str) Command to run. Returns 2-tuple with return code and stdout. utils.update_config(config_file) Update the module config dict from a config file. Parameters config_file (str) Path to JSON config file. utils.volume_shadow(drive) Creates a shadow copy of a drive and mounts it at a temporary directory. Implemented with contextmanager to be used through the python with statement. Parameters drive (str) Drive to shadow copy. Yields str Path to temp folder where shadow copy is mounted. Raises OSError 10 Chapter 1. Contents
CHAPTER 2 Indices and tables genindex modindex search 11
12 Chapter 2. Indices and tables
Python Module Index j jobs, 8 u utils, 9 13