pyftpsync Documentation

Transcription

1 pyftpsync Documentation Release dev Martin Wendt June 09, 2015

2

3 Contents 1 pyftpsync Architecture Classes ftpsync ftpsync package Features Known limitations Quickstart 13 5 Command line syntax Upload files syntax Example: Upload files Synchronize files syntax Example: Synchronize folders Script examples 19 7 Contribute 21 8 Indices and tables 23 Python Module Index 25 i

4 ii

5 Synchronize local directories with FTP server. Project License The MIT License Author Martin Wendt Version 1.0 Date June 09, 2015 Contents 1

6 2 Contents

7 CHAPTER 1 pyftpsync Architecture 1.1 Classes _Target FsTarget DirectoryEntry _Resource FileEntry DirMetadata BiDirSynchronizer BaseSynchronizer DownloadSynchronizer UploadSynchronizer 3

8 4 Chapter 1. pyftpsync Architecture

9 CHAPTER 2 ftpsync 2.1 ftpsync package Submodules ftpsync.ftp_target module (c) Martin Wendt; see Licensed under the MIT license: class ftpsync.ftp_target.ftptarget(path, host, port=none, username=none, password=none, extra_opts=none) Bases: ftpsync.targets._target Represents a synchronisation target on a FTP server. path str ftp FTP host str port int Current working directory on FTP server. Instance of ftplib.ftp. hostname of FTP server FTP port (defaults to 21) username str password str close() cwd(dir_name) get_base_name() 5

10 get_dir() get_id() mkdir(dir_name) open() open_readable(name) Open cur_dir/name for reading. pwd() remove_file(name) Remove cur_dir/name. rmdir(dir_name) set_mtime(name, mtime, size) write_file(name, fp_src, blocksize=8192, callback=none) ftpsync.pyftpsync module Simple folder synchronization using FTP. (c) Martin Wendt; see Licensed under the MIT license: Usage examples: > pyftpsync.py help > pyftpsync.py upload. ftp://example.com/myfolder ftpsync.pyftpsync.namespace_to_dict(o) Convert an argparse namespace object to a dictionary. ftpsync.pyftpsync.run() ftpsync.resources module (c) Martin Wendt; see Licensed under the MIT license: class ftpsync.resources.directoryentry(target, rel_path, name, size, mtime, unique) Bases: ftpsync.resources._resource is_dir() class ftpsync.resources.fileentry(target, rel_path, name, size, mtime, unique) Bases: ftpsync.resources._resource EPS_TIME = 2.01 get_sync_info() Get mtime/size when this resource was last synchronized with remote. is_file() was_modified_since_last_sync() Return True if this resource was modified since last sync. None is returned if we don t know (because of missing meta data). 6 Chapter 2. ftpsync

11 2.1.5 ftpsync.synchronizers module (c) Martin Wendt; see Licensed under the MIT license: class ftpsync.synchronizers.basesynchronizer(local, remote, options) Bases: object Synchronizes two target instances in dry_run mode (also base class for other synchronizers). get_stats() run() sync_conflict(local, remote) sync_equal_dir(local_dir, remote_dir) Return False to prevent visiting of children sync_equal_file(local_file, remote_file) sync_missing_local_dir(remote_dir) Return False to prevent visiting of children sync_missing_local_file(remote_file) sync_missing_remote_dir(local_dir) sync_missing_remote_file(local_file) sync_newer_local_file(local_file, remote_file) sync_older_local_file(local_file, remote_file) class ftpsync.synchronizers.bidirsynchronizer(local, remote, options) Bases: ftpsync.synchronizers.basesynchronizer Synchronizer that performs up- and download operations as required. Newer files override unmodified older files When both files are newer than last sync -> conflict! Conflicts may be resolved by these options resolve=old: use the older version resolve=new: use the newer version resolve=local: use the local file resolve=remote: use the remote file resolve=ask: prompt mode When a file is missing: check if it existed in the past. If so, delete it. Otherwise copy it. In order to know if a file was modified, deleted, or created since last sync, we store a snapshot of the directory in the local directory. get_info_strings() sync_conflict(local_entry, remote_entry) sync_missing_local_dir(remote_dir) sync_missing_local_file(remote_file) sync_missing_remote_dir(local_dir) sync_missing_remote_file(local_file) sync_newer_local_file(local_file, remote_file) sync_older_local_file(local_file, remote_file) 2.1. ftpsync package 7

12 class ftpsync.synchronizers.downloadsynchronizer(local, remote, options) Bases: ftpsync.synchronizers.basesynchronizer get_info_strings() sync_equal_dir(local_dir, remote_dir) Return False to prevent visiting of children sync_equal_file(local_file, remote_file) sync_missing_local_dir(remote_dir) sync_missing_local_file(remote_file) sync_missing_remote_dir(local_dir) sync_missing_remote_file(local_file) sync_newer_local_file(local_file, remote_file) sync_older_local_file(local_file, remote_file) class ftpsync.synchronizers.uploadsynchronizer(local, remote, options) Bases: ftpsync.synchronizers.basesynchronizer get_info_strings() sync_equal_dir(local_dir, remote_dir) Return False to prevent visiting of children sync_equal_file(local_file, remote_file) sync_missing_local_dir(remote_dir) sync_missing_local_file(remote_file) sync_missing_remote_dir(local_dir) sync_missing_remote_file(local_file) sync_newer_local_file(local_file, remote_file) sync_older_local_file(local_file, remote_file) ftpsync.targets module (c) Martin Wendt; see Licensed under the MIT license: class ftpsync.targets.dirmetadata(target) Bases: object DEBUG = False DEBUG_META_FILE_NAME = _pyftpsync-meta.json LOCK_FILE_NAME =.pyftpsync-lock.json META_FILE_NAME =.pyftpsync-meta.json PRETTY = False VERSION = 1 flush() read() 8 Chapter 2. ftpsync

13 remove(filename) set_mtime(filename, mtime, size) Store real file mtime in meta data. This is needed, because FTP targets don t allow to set file mtime, but use to the upload time instead. We also record size and upload time, so we can detect if the file was changed by other means and we have to discard our meta data. set_sync_info(filename, mtime, size) Store mtime/size when local and remote file was last synchronized. This is stored in the local file s folder as meta data. The information is used to detect conflicts, i.e. if both source and remote had been modified by other means since last synchronization. class ftpsync.targets.fstarget(root_dir, extra_opts=none) Bases: ftpsync.targets._target close() cwd(dir_name) flush_meta() Write additional meta information for current directory. get_dir() mkdir(dir_name) open() open_readable(name) pwd() remove_file(name) Remove cur_dir/name. rmdir(dir_name) Remove cur_dir/name. set_mtime(name, mtime, size) Set modification time on file. write_file(name, fp_src, blocksize=8192, callback=none) ftpsync.targets.ansi_code(name) Return ansi color or style codes or if colorama is not available. ftpsync.targets.get_credentials_for_url(url, 2-tuple (username, password) or None ftpsync.targets.make_target(url, extra_opts=none) Factory that creates _Target objects from URLs. ftpsync.targets.prompt_for_password(url, user=none) ftpsync.targets.save_password(url, username, password) ftpsync.targets.to_binary(s) Convert unicode (text strings) to binary data on Python 2 and 3. ftpsync.targets.to_str(s) Convert unicode to native str on Python 2 and ftpsync package 9

14 ftpsync.targets.to_text(s) Convert binary data to unicode (text strings) on Python 2 and Module contents 10 Chapter 2. ftpsync

15 CHAPTER 3 Features This is a command line tool and a library for use in your Python projects. Upload, download, and bi-directional synch mode. Allows FTP-to-FTP and Filesystem-to-Filesystem synchronization as well. Architecture is open to add other target types. 3.1 Known limitations The FTP server must support the MLST command. pyftpsync uses file size and modification dates to detect file changes. This is efficient, but not as robust as CRC checksums could be. pyftpsync tries to detect conflicts (i.e. simultaneous modifications of local and remote targets) by storing last sync time and size in a separate meta data file inside the local folders. This is not bullet proof and may fail under some conditions. In short: pyftpsync is not (nor tries to be a replacement for) a distributed version control system. Make sure you have backups. 11

16 12 Chapter 3. Features

17 CHAPTER 4 Quickstart Todo There will be a MSI installer available for Windows. Requirements: Python 2.6+ or 3 is required. Releases are hosted on PyPI and can be installed using pip or EasyInstall: $ pip install -U pyftpsync After that the pyftpsync command is available: $ pyftpsync --version and the ftpsync package can be used in Python code: $ python >>> from ftpsync import version >>> version '1.0.3' 13

18 14 Chapter 4. Quickstart

19 CHAPTER 5 Command line syntax Use the --help or -h argument to get help: $ pyftpsync -h usage: pyftpsync [-h] [--verbose --quiet] [--version] [--progress] {upload,download,sync}... Synchronize folders over FTP. positional arguments: {upload,download,sync} sub-command help upload copy new and modified files to remote folder download copy new and modified files from remote folder to local target sync synchronize new and modified files between remote folder and local target optional arguments: -h, --help show this help message and exit --verbose, -v increment verbosity by one (default: 3, range: 0..5) --quiet, -q decrement verbosity by one --version show program's version number and exit --progress, -p show progress info, even if redirected or verbose < 3 See also $ 5.1 Upload files syntax Command specific help is available like so: $ pyftpsync upload --help usage: pyftpsync upload [-h] [-x] [-f INCLUDE_FILES] [-o OMIT] [--store-password] [--no-prompt] [--no-color] [--force] [--delete] [--delete-unmatched] LOCAL REMOTE positional arguments: LOCAL path to local folder (default:.) REMOTE path to remote folder 15

20 optional arguments: -h, --help -x, --execute $ show this help message and exit turn off the dry-run mode (which is ON by default), that would just print status messages but does not change anything -f INCLUDE_FILES, --include-files INCLUDE_FILES wildcard for file names (default: all, separate multiple values with ',') -o OMIT, --omit OMIT wildcard of files and directories to exclude (applied after --include) --store-password --no-prompt --no-color --force --delete --delete-unmatched save password to keyring if login succeeds prevent prompting for missing credentials prevent use of ansi terminal color codes overwrite different remote files, even if the target is newer remove remote files if they don't exist locally remove remote files if they don't exist locally or don't match the current filter (implies '--delete' option) 5.2 Example: Upload files Upload all new and modified files from user s temp folder to an FTP server. No files are changed on the local directory: $ pyftpsync upload ~/temp ftp://example.com/target/folder Add the --delete option to remove all files from the remote target that don t exist locally: $ pyftpsync upload ~/temp ftp://example.com/target/folder --delete Add the -x option to switch from DRY-RUN mode to real execution: $ pyftpsync upload ~/temp ftp://example.com/target/folder --delete -x 5.3 Synchronize files syntax $ pyftpsync sync --help usage: pyftpsync sync [-h] [-x] [-f INCLUDE_FILES] [-o OMIT] [--store-password] [--no-prompt] [--no-color] [--resolve {old,new,local,remote,ask}] LOCAL REMOTE positional arguments: LOCAL path to local folder (default:.) REMOTE path to remote folder optional arguments: -h, --help -x, --execute show this help message and exit turn off the dry-run mode (which is ON by default), that would just print status messages but does not change anything -f INCLUDE_FILES, --include-files INCLUDE_FILES wildcard for file names (default: all, separate 16 Chapter 5. Command line syntax

21 multiple values with ',') -o OMIT, --omit OMIT wildcard of files and directories to exclude (applied after --include) --store-password save password to keyring if login succeeds --no-prompt prevent prompting for missing credentials --no-color prevent use of ansi terminal color codes --resolve {old,new,local,remote,ask} conflict resolving strategy $ 5.4 Example: Synchronize folders Two-way synchronization of a local folder with an FTP server: $ pyftpsync sync --store-password --resolve=ask --execute ~/temp ftp://example.com/target/folder 5.4. Example: Synchronize folders 17

22 18 Chapter 5. Command line syntax

23 CHAPTER 6 Script examples Upload changes from local folder to FTP server: from ftpsync.targets import FsTarget, UploadSynchronizer from ftpsync.ftp_target import FtpTarget local = FsTarget("~/temp") user ="joe" passwd = "secret" remote = FtpTarget("/temp", "example.com", user, passwd) opts = {"force": False, "delete_unmatched": True, "verbose": 3, "dry_run" : False} s = UploadSynchronizer(local, remote, opts) s.run() Synchronize local folder with FTP server: from ftpsync.targets import FsTarget, BiDirSynchronizer from ftpsync.ftp_target import FtpTarget local = FsTarget("~/temp") user ="joe" passwd = "secret" remote = FtpTarget("/temp", "example.com", user, passwd) opts = {"resolve": "skip", "verbose": 1, "dry_run" : False} s = BiDirSynchronizer(local, remote, opts) s.run() 19

24 20 Chapter 6. Script examples

25 CHAPTER 7 Contribute If you plan to debug or contribute, install to run directly from the source: $ python setup.py develop $ python setup.py test The use of virtualenv is recommended. 21

26 22 Chapter 7. Contribute

27 CHAPTER 8 Indices and tables genindex modindex search 23

28 24 Chapter 8. Indices and tables

29 Python Module Index f ftpsync, 10 ftpsync.ftp_target, 5 ftpsync.pyftpsync, 6 ftpsync.resources, 6 ftpsync.synchronizers, 7 ftpsync.targets, 8 25

30 26 Python Module Index

31 Index A ansi_code() (in module ftpsync.targets), 9 B BaseSynchronizer (class in ftpsync.synchronizers), 7 BiDirSynchronizer (class in ftpsync.synchronizers), 7 C close() (ftpsync.ftp_target.ftptarget method), 5 close() (ftpsync.targets.fstarget method), 9 cwd() (ftpsync.ftp_target.ftptarget method), 5 cwd() (ftpsync.targets.fstarget method), 9 D DEBUG (ftpsync.targets.dirmetadata attribute), 8 DEBUG_META_FILE_NAME (ftpsync.targets.dirmetadata attribute), 8 DirectoryEntry (class in ftpsync.resources), 6 DirMetadata (class in ftpsync.targets), 8 DownloadSynchronizer (class in ftpsync.synchronizers), 7 E EPS_TIME (ftpsync.resources.fileentry attribute), 6 F FileEntry (class in ftpsync.resources), 6 flush() (ftpsync.targets.dirmetadata flush_meta() (ftpsync.targets.fstarget method), 9 FsTarget (class in ftpsync.targets), 9 ftp (ftpsync.ftp_target.ftptarget attribute), 5 ftpsync (module), 10 ftpsync.ftp_target (module), 5 ftpsync.pyftpsync (module), 6 ftpsync.resources (module), 6 ftpsync.synchronizers (module), 7 ftpsync.targets (module), 8 FtpTarget (class in ftpsync.ftp_target), 5 G get_base_name() (ftpsync.ftp_target.ftptarget method), 5 get_credentials_for_url() (in module ftpsync.targets), 9 get_dir() (ftpsync.ftp_target.ftptarget method), 6 get_dir() (ftpsync.targets.fstarget method), 9 get_id() (ftpsync.ftp_target.ftptarget method), 6 get_info_strings() (ftpsync.synchronizers.bidirsynchronizer method), 7 get_info_strings() (ftpsync.synchronizers.downloadsynchronizer get_info_strings() (ftpsync.synchronizers.uploadsynchronizer get_stats() (ftpsync.synchronizers.basesynchronizer method), 7 get_sync_info() (ftpsync.resources.fileentry method), 6 H host (ftpsync.ftp_target.ftptarget attribute), 5 I is_dir() (ftpsync.resources.directoryentry method), 6 is_file() (ftpsync.resources.fileentry method), 6 L LOCK_FILE_NAME (ftpsync.targets.dirmetadata attribute), 8 M make_target() (in module ftpsync.targets), 9 META_FILE_NAME (ftpsync.targets.dirmetadata attribute), 8 mkdir() (ftpsync.ftp_target.ftptarget method), 6 mkdir() (ftpsync.targets.fstarget method), 9 N namespace_to_dict() (in module ftpsync.pyftpsync), 6 O open() (ftpsync.ftp_target.ftptarget method), 6 27

32 open() (ftpsync.targets.fstarget method), 9 open_readable() (ftpsync.ftp_target.ftptarget method), 6 open_readable() (ftpsync.targets.fstarget method), 9 P password (ftpsync.ftp_target.ftptarget attribute), 5 path (ftpsync.ftp_target.ftptarget attribute), 5 port (ftpsync.ftp_target.ftptarget attribute), 5 PRETTY (ftpsync.targets.dirmetadata attribute), 8 prompt_for_password() (in module ftpsync.targets), 9 pwd() (ftpsync.ftp_target.ftptarget method), 6 pwd() (ftpsync.targets.fstarget method), 9 R read() (ftpsync.targets.dirmetadata remove() (ftpsync.targets.dirmetadata remove_file() (ftpsync.ftp_target.ftptarget method), 6 remove_file() (ftpsync.targets.fstarget method), 9 rmdir() (ftpsync.ftp_target.ftptarget method), 6 rmdir() (ftpsync.targets.fstarget method), 9 run() (ftpsync.synchronizers.basesynchronizer method), 7 run() (in module ftpsync.pyftpsync), 6 S save_password() (in module ftpsync.targets), 9 set_mtime() (ftpsync.ftp_target.ftptarget method), 6 set_mtime() (ftpsync.targets.dirmetadata method), 9 set_mtime() (ftpsync.targets.fstarget method), 9 set_sync_info() (ftpsync.targets.dirmetadata method), 9 sync_conflict() (ftpsync.synchronizers.basesynchronizer method), 7 sync_conflict() (ftpsync.synchronizers.bidirsynchronizer method), 7 sync_equal_dir() (ftpsync.synchronizers.basesynchronizer method), 7 sync_equal_dir() (ftpsync.synchronizers.downloadsynchronizer sync_missing_remote_file() (ftpsync.synchronizers.downloadsynchronizer sync_equal_dir() (ftpsync.synchronizers.uploadsynchronizer sync_missing_remote_file() (ftpsync.synchronizers.uploadsynchronizer sync_equal_file() (ftpsync.synchronizers.basesynchronizer method), 7 sync_equal_file() (ftpsync.synchronizers.downloadsynchronizer sync_equal_file() (ftpsync.synchronizers.uploadsynchronizer sync_missing_local_dir() (ftpsync.synchronizers.basesynchronizer method), 7 sync_missing_local_dir() (ftpsync.synchronizers.bidirsynchronizer method), 7 sync_missing_local_dir() (ftpsync.synchronizers.downloadsynchronizer sync_missing_local_dir() (ftpsync.synchronizers.uploadsynchronizer sync_missing_local_file() (ftpsync.synchronizers.basesynchronizer method), 7 sync_missing_local_file() (ftpsync.synchronizers.bidirsynchronizer method), 7 sync_missing_local_file() (ftpsync.synchronizers.downloadsynchronizer sync_missing_local_file() (ftpsync.synchronizers.uploadsynchronizer sync_missing_remote_dir() (ftpsync.synchronizers.basesynchronizer method), 7 sync_missing_remote_dir() (ftpsync.synchronizers.bidirsynchronizer method), 7 sync_missing_remote_dir() (ftpsync.synchronizers.downloadsynchronizer sync_missing_remote_dir() (ftpsync.synchronizers.uploadsynchronizer sync_missing_remote_file() (ftpsync.synchronizers.basesynchronizer method), 7 sync_missing_remote_file() (ftpsync.synchronizers.bidirsynchronizer method), 7 sync_newer_local_file() (ftpsync.synchronizers.basesynchronizer method), 7 sync_newer_local_file() (ftpsync.synchronizers.bidirsynchronizer method), 7 sync_newer_local_file() (ftpsync.synchronizers.downloadsynchronizer sync_newer_local_file() (ftpsync.synchronizers.uploadsynchronizer 28 Index

33 sync_older_local_file() (ftpsync.synchronizers.basesynchronizer method), 7 sync_older_local_file() (ftpsync.synchronizers.bidirsynchronizer method), 7 sync_older_local_file() (ftpsync.synchronizers.downloadsynchronizer sync_older_local_file() (ftpsync.synchronizers.uploadsynchronizer T to_binary() (in module ftpsync.targets), 9 to_str() (in module ftpsync.targets), 9 to_text() (in module ftpsync.targets), 9 U UploadSynchronizer (class in ftpsync.synchronizers), 8 username (ftpsync.ftp_target.ftptarget attribute), 5 V VERSION (ftpsync.targets.dirmetadata attribute), 8 W was_modified_since_last_sync() (ftpsync.resources.fileentry method), 6 write_file() (ftpsync.ftp_target.ftptarget method), 6 write_file() (ftpsync.targets.fstarget method), 9 Index 29