Making Python a better scripting language

Transcription

1 Making Python a better scripting language Nicola Musatti Agenda Why are we here? What is a scripting language? A programming language of sorts Has some form of flow control Provides a convenient way to execute commands Is Python a scripting language? It is a programming language Has many flow control statements Provides a not so convenient way to execute commands A simple command Any command interpreter: ls -la Python: from subprocess import call call(("ls", "-la")) What's wrong with that? Manageable for simple, fixed command lines Handling complex commands becomes awkward

2 No support from the language syntax Hard to come out with reusable code A not so simple command svnadmin create repo if [ $? == 0 ]; then p="file://$(pwd)/repo/project" svn mkdir $p/branches $p/tags --parents \ -m"create project tree" svn import backup $p/trunk -m"first import" fi mkdir sandbox cd sandbox svn checkout $p/trunk project In the days before nxpy.command import os from subprocess import call p = "file://" + os.getcwd() + "/repo/project" res = call("svnadmin create repo", shell=true) if res == 0: call("svn mkdir " + p + "/branches " + p + "/tags " "--parents -m=\"create project tree\"", shell=true) call("svn import \"backup\" " + p + "/trunk " "-m=\"first import\"", shell=true) os.mkdir("sandbox") os.chdir("sandbox") call("svn checkout " + p + "/trunk \"project\"", shell=true) Do we really want... import os from nxpy.command.command import Error from magic import run_commands p = "file://" + os.getcwd() + "/repo/project" try: svnadmin create repo svn mkdir $p/branches $p/tags --parents \ -m"create project tree" svn import backup $p/trunk -m"first import" except Error: pass os.mkdir("sandbox") os.chdir("sandbox") svn checkout $p/trunk project Ideally

3 from ls import ls print ls(long=true, all=true) It can be done! Launching commands A not so simple command (reprise) import os from nxpy.command.command import Error from nxpy.svn.svnadmin import SvnAdmin from svn import Svn svn_ = Svn() p = "file://" + os.getcwd() + "/repo/project" try: SvnAdmin().create("repo") svn_.mkdir(p + "/branches", p + "/tags", parents=true, message="\"create project tree\"") svn_.import_("backup", p + "/trunk", message="\"first import\"") except Error as e: print e os.mkdir("sandbox") os.chdir("sandbox") svn_.checkout(p + "/trunk", "project") The nxpy.command package svn_.mkdir(p + "/branches", p + "/tags", parents=true, message="\"create project tree\"") Commands are expressed as method calls Command arguments are passed as plain arguments Options are passed as keyword arguments The nxpy.command package (cont.) Supported options may be specified declaratively Enforces custom constraints on option and argument combinations Builds the resulting command lines and executes them Returns the command's output and error Works with both non-interactive and interactive commands Most effective when many subcommands share the same options e.g. svn, cleartool

4 The nxpy.command package (cont.) package nxpy + package command + module option + class Config + class Parser + module command + class Command + module interpreter + class Interpreter The nxpy.command.option.config class Specify which options are supported and how they should appear on the command line: bool_opts: value_opts: iterable_opts: format_opts: mapped_opts: opposite_options: must be specified when True take a single argument may take several arguments have their syntax specified by a format string are mapped to a specific syntax must be specified on the command line when they are not present The nxpy.command.option.config class (cont.) _config = Config( bool_opts = ( "ignore_externals", "parents" ), value_opts = ( "message", ), mapped_opts = { "ignore_externals" : "--ignore-externals" } ) Other arguments are: prefix: option prefix. Default is -- separator: option argument separator. Default is blank The nxpy.command.option.parser class def import_(self, src, dest, **options): op = Parser(_config, "import", ( src, dest ), options, message="") op.checkmandatoryoptions("message") Check that passed options and arguments are valid Build the actual command line Keyword arguments indicate supported options with their defaults The nxpy.command.command.command class

5 class Svn(Command): def init (self): super(svn, self). init ("svn", None) def import_(self, src, dest, **options): op = Parser(_config, "import", ( src, dest ), options, message="") op.checkmandatoryoptions("message") self.run(op) Puts all the pieces together Executes (or echoes) the resulting command line Returns the command's output and error contents Wrapping the ls command from nxpy.command.command import Command from nxpy.command.option import Config, Parser _config = Config( bool_opts = ( "all", "long" ), mapped_opts = { "all" : "-a", "long" : "-l" } ) class Ls(Command): def init (self, debug=false): super(ls, self). init ("/bin/ls", debug) def call (self, *args, **options): op = Parser(_config, None, args, options, all=false, long=false) return self.run(op)[0] ls = Ls() Controlling interpreters Dealing with interactive programs Subcommand options are handled in the same way Non-blocking I/O is used to avoid deadlocks Must indicate how to detect completion of current command A polling algorithm is used to check for termination Might need to handle excessive duration An example: the cleartool interpreter class FailedCommand(BadCommand): def init (self, cmd, err_code=0, err=""):

6 message = [] if err: message.append(err) if err_code!= 0: message.append("error: " + err_code) super(failedcommand, self). init (cmd, "\n".join(message)) command_line = "cleartool -status" _result_re = re.compile(r"command \d+ returned status (\d)\r\n") An example: the cleartool interpreter (cont.) class ClearTool(Interpreter): def _run(self, parser, **kwargs): if isinstance(parser, Parser): cmd = parser.getcommandline() else: cmd = parser raise_on_failure = kwargs.get("raise_on_failure", False) try: kwargs["cond"] = RegexpWaiter(self._result_re, EXP_OUT) out, err = self.run(cmd, **kwargs) if raise_on_failure: err_code = ClearTool._result_re.search(out).group(1) if err_code > 0: raise FailedCommand(cmd, err_code=err_code) return ClearTool._result_re.sub("", out), err, cmd except BadCommand as e: raise FailedCommand(e.command, err=e.stderr) Expressing termination conditions Interpreters may issue the same message after each command Otherwise specific output must be sought "Waiters" help discover termination Available waiters The interpreter module provides ways to wait for: Any output / error A specific number of lines A specific string A regular expression A different example: the ftp client class Ftp(Interpreter): def init (self): super(ftp, self). init ("ftp -v") self.setlog(true) def open(self, host, port, user, password): op = Parser(None, "open", ( host, port ), {}) self.send_cmd(op.getcommandline()) self.expect_regexp("220")

7 self.send_cmd(user) self.expect_string("password:", EXP_ERR, raise_on_error=false) self.send_cmd(password) self.expect_regexp("230") Tweaking the polling algorithm Sometimes there's no identifiable end of output Commands may hang You want to bail out of commands taking too long Tweaking the polling algorithm (cont.) You can specify: An overall timeout A maximum number of retries A fixed interval between retries A lower bound for the delay between retries An example: the cleartool interpreter (reprise) def update(self, **options): op = Parser(_config, "update", (), options, print_report=false, force=false, nolog=false, log="") op.checkexclusiveoptions("log", "nolog") self._run(op, raise_on_error=false, timeout=300, interval=0.5, quantum=0.5) Wrap up Python as a scripting language - Does it matter? The agility of the language and the wealth of available libraries often make it more convenient to write full-fledged Python programs rather than any scripting approach. However: User commands may be more stable Sometimes there isn't an API! Tips & tricks Compacting command implementation kills self-documentation Make sure you find the very end of subcommands' output/error Before using EAFP ensure your command is forgiving! Open issues

8 Porting to Python 3 More tests! Split from library? Directions The nxpy.command package is part of the Nxpy library Project homepage: It's available on PyPI: pip install Nxpy Colophon sphinx sphinx-intl hieroglyph No stylesheet was harmed in the making of this presentation! Q & A Q & A [Slide intentionally left blank]