Single Source Python 2 3

Transcription

1 Single Source Python 2 3 May 21, Single Source Python 2/3 1.1 A Tutorial at PyData Berlin Dr. Mike Müller Python Academy GmbH & Co. KG PyData Berlin 2016 Berlin, Germany 2 Overview Motivation Major differences between Python 2 and 3 Conversion strategies Single source Roll your own Use a library python-future.org 3 Motivation Python 3.0 was released 2008 Python 2 (2.7) still widely used Supporting only major Python version is often not an option Single source can be a good solution 1

2 4 Overview of important Python 2/3 differences Many small changes You might not encounter many of them Lot s of cleanups of warts Many (partially) backported to Python 2.7 Very few real syntax incompatibility between 2 and 3 5 print is a function likely the most obvious one but is pretty easy to fix from future import print function print(value,..., sep=, end= \n, file=sys.stdout, flush=false) In [3]: for x in range(10): print(x, end= ) In [4]: print(*range(10), sep= ) Iterators are everywhere zip() map() In [6]: zip( abc, xyz ) Out[6]: <zip at 0x1050ae548> In [7]: list(zip( abc, xyz )) Out[7]: [( a, x ), ( b, y ), ( c, z )] In [8]: map(str.upper, abc ) Out[8]: <map at 0x1050ad198> In [9]: list(map(str.upper, abc )) Out[9]: [ A, B, C ] 7 range() a.k.a. xrange() ++ range() just like xrange() no limit slice-able In [10]: range(10) Out[10]: range(0, 10) 2

3 In [13]: range(int(1e100)) Out[13]: range(0, In [14]: %%python2 xrange(int(1e100)) Traceback (most recent call last): File "<stdin>", line 2, in <module> OverflowError: Python int too large to convert to C long In [15]: r = range(10) r[2:7] Out[15]: range(2, 7) In [16]: %%python2 xr = xrange(10) xr[2:7] Traceback (most recent call last): File "<stdin>", line 3, in <module> TypeError: sequence index must be integer, not slice 8 Key views dict.keys() returns a view Python 2: list views are more efficient they update In [28]: d = { a : 100, b : 200, c : 300} k = d.keys() k Out[28]: dict keys([ b, a, c ]) In [29]: d[ x ] = 400 k Out[29]: dict keys([ b, x, a, c ]) 9 No apple-orange comparisons only defined comparisons for many the most important change In [18]: 1 < TypeError Traceback (most recent call last) 3

4 <ipython-input-18-9ee3bb6f2cf3> in <module>() ----> 1 1 < 1 TypeError: unorderable types: int() < str() In [20]: %%python2 print(1 < 1 ) True In [21]: 100 < None TypeError Traceback (most recent call last) <ipython-input-21-ab68f0474b07> in <module>() ----> < None TypeError: unorderable types: int() < NoneType() In [22]: %%python2 print(100 < None) False 10 Integers without limits long became int no big change difference has been small in Python 2 In [23]: Out[23]: In [25]: %%python2 print(repr( )) L 11 Mathematical division divide integers just like in math by default from future import division In [30]: 1 / 2 Out[30]: 0.5 4

5 In [32]: 1 // 2 Out[32]: 0 In [33]: %%python2 0 0 print(1 / 2) print(1 // 2) In [34]: %%python from future import division print(1 / 2) print(1 // 2) 12 Strings are always unicode Maybe the biggest change, depending on the use case Python 2: weak distinction between bytes and strings 13 Prefixing Python 2: abc : bytestring u abc : unicode string Python 3: abc : string unicode b abc : bytestring both: u abc : string unicode b abc : bytestring 14 Source code encoding default encodings: Python 2 ascii Python 3 utf-8 both, explicit: # -*- coding: utf-8 -*- 5

6 15 Cleanup - Modern Python 2 can do no classic classes anymore no backticks, use repr() <> gone exception instances only with as, old comma syntax is gone literal octal numbers with 0o prefix, e.g. 0o7 open() is now io.open() use next() not obj.next() use functools.reduce() instead of buil-in reduce() use func(*args) instead of apply(func, args) 16 Cleanup - Modern Python 2 can do with a little help raw input() > input(), input() > eval(input()) importlib.reload() instead of reload() 17 Cleanup - Modern Python 2 can do with help of a library exec statement turned into a function metaclasses as keyword argument in class definition, no metaclass anymore super() without class and self round() uses a different rule round(2.5) == 2, instead of round(2.5) == 3 18 New syntax - Python 3 only keyword-only arguments nonlocal extended unpacking of iterables True, False, and None are keywords, yeah variables from list comprehensions don t leak anymore, great exception instances only available in except block 19 Re-structuring of the standard library Removal of old, unused modules gopherlib renaming to comply with PEP8, ConfigParser > configparser, StringIO moved into io grouping of related modules like http or dbm 20 Write your own compatibility layer use modern Python 2 (exceptions, next) future version testing re-defining built-ins standard lib packports handling strings and bytes IO 6

7 21 Never use classic classes In [73]: class A(object): pass 22 Don t use ancient features no backticks, use repr() never use <> prefix literal octal numbers with 0o prefix, e.g. 0o7 use functools.reduce() instead of buil-in reduce() use func(*args) instead of apply(func, args) In [136]: %%python2 a = a print( a ) a In [79]: a = a print( a ) File "<ipython-input-79-2c78c9df6889>", line 2 print( a ) ^ SyntaxError: invalid syntax In [81]: a = a print(repr(a)) a 23 Exceptions always with as In [55]: %%python2 try: 1 / 0 except ZeroDivisionError, err: pass print(err) integer division or modulo by zero In [56]: try: 1 / 0 except ZeroDivisionError, err: pass print(err) 7

8 File "<ipython-input-56-d4e7da7453b9>", line 3 except ZeroDivisionError, err: ^ SyntaxError: invalid syntax In [82]: try: 1 / 0 except ZeroDivisionError as err: pass In [83]: %%python2 try: 1 / 0 except ZeroDivisionError as err: pass 24 Use io.open you can specify an encoding In [84]: import io io.open is open Out[84]: True In [87]: %%python2 from io import open In [88]: open(file, mode= r, buffering=-1, encoding=none, errors=none, newline=none, closefd=true, open 25 Use next() In [93]: i = iter( abc ) print(next(i)) a In [92]: %%python2 i = iter( abc ) print(next(i)) a 26 Travel to the future add this line at the beginning of all your source files In [137]: from future import absolute_import, division, print_function 8

9 27 Unicode litertals can be problematic In [138]: from future import unicode_literals not always recommended good for 3 > 2 much less useful for 2 > 3 e.g., may cause problems with library calls (e.g. paths) 28 Version Testing always test for < 3 or > 2 never == 2 or == 3 Python 4 will be comming 29 Re-defining built-ins In [106]: %%python2 from future import absolute_import, division, print_function import sys if sys.version_info.major < 3: input = raw_input range = xrange from io import open from itertools import izip as zip, imap as map, ifilter as filter print(filter) <type itertools.ifilter > 30 Standard library packports several libraries are backported lru cache: backports.functools lru cache pathlib2 In [107]: try: from functools import lru_cache except ImportError: from backports.functools_lru_cache import lru_cache 31 Your own compatibility layer use of all seems justified In [ ]: # %load compat.py from future import absolute_import, division, print_function 9

10 import sys all = [] if sys.version_info.major < 3: input = raw_input range = xrange from io import open from itertools import (izip as zip, imap as map, ifilter as filter) all = [ input, filter, map, open, range, zip ] In [109]: %%python3 <class range > from compat import * print(range) In [110]: %%python2 <type xrange > from compat import * print(range) 32 Using help six Python 2.5 and up, latest version only 2.6 future Python 2.6 and up 33 Using python-future.org does the work for you and is much better at it makes Python 2 behave largely like Python 3 34 Options starting from scratch starting from Python 2 starting from Python 3 automatic conversion porting philosophy 35 Starting from scratch at the beginng of your source files add: 10

11 In [112]: from future import (absolute_import, division, print_function, unicode_literals) from builtins import * write standard Python 3 without th Python 3 only parts 36 Starting from Python 2 In [ ]: # %load py2sample.py import StringIO fobj = StringIO.StringIO("""Hello world line1 = fobj.next() print Done In [ ]: $ futurize -w py2sample.py RefactoringTool: Skipping optional fixer: idioms RefactoringTool: Skipping optional fixer: ws_comma RefactoringTool: Refactored py2sample.py --- py2sample.py (original) +++ py2sample.py -1,10 from future import print_function +from future import standard_library +standard_library.install_aliases() +from builtins import next -import StringIO +import io -fobj = StringIO.StringIO("""Hello world +fobj = io.stringio("""hello world line1 = next(fobj) RefactoringTool: Files that were modified: RefactoringTool: py2sample.py In [ ]: # %load py2sample.py from future import print_function from future import standard_library standard_library.install_aliases() import io 11

12 fobj = io.stringio("""hello world line1 = next(fobj) print( Done ) 37 Backup py2sample.py.bak 38 Two-stage approach first stage applies only save changes no imports from future produces modern, 2.6-compatible code result may not run with Python 3 yet In [ ]: $ futurize -w --stage1 py2sample.py RefactoringTool: Skipping optional fixer: idioms RefactoringTool: Skipping optional fixer: ws_comma RefactoringTool: Refactored py2sample.py --- py2sample.py (original) +++ py2sample.py -1,3 +from future import print_function import -6,6 fobj = StringIO.StringIO("""Hello world -line1 = fobj.next() +line1 = next(fobj) -print Done +print( Done ) RefactoringTool: Files that were modified: RefactoringTool: py2sample.py In [ ]: # %load py2sample.py from future import print_function import StringIO fobj = StringIO.StringIO("""Hello world 12

13 line1 = next(fobj) print( Done ) In [ ]: $ futurize -w --stage2 py2sample.py RefactoringTool: Refactored py2sample.py --- py2sample.py (original) +++ py2sample.py -1,10 from future import print_function +from future import standard_library +standard_library.install_aliases() +from builtins import next -import StringIO +import io -fobj = StringIO.StringIO("""Hello world +fobj = io.stringio("""hello world line1 = next(fobj) RefactoringTool: Files that were modified: RefactoringTool: py2sample.py In [ ]: # %load py2sample.py from future import print_function from future import standard_library standard_library.install_aliases() from builtins import next import io fobj = io.stringio("""hello world line1 = next(fobj) print( Done ) 39 Going fix-by-fix more than fifty different fixes futurize --list-fixes In [ ]: $ futurize -f lib2to3.fixes.fix_next py2sample.py RefactoringTool: Refactored py2sample.py --- py2sample.py (original) +++ py2sample.py -4,6 13

14 fobj = StringIO.StringIO("""Hello world -line1 = fobj.next() +line1 = next(fobj) print Done RefactoringTool: Files that need to be modified: RefactoringTool: py2sample.py In [ ]: futurize -f lib2to3.fixes.fix_has_key py2sample.py RefactoringTool: No files need to be modified. 40 Strategies need to have tests (coverage) create environments for Python 2 and 3 (conda) version control all steps either go fix-by-fix (>50) or go stage-by-stage (2 steps) 41 Starting from Python 3 In [116]: # %load py3sample.pya zip( abc, range(3)) Out[116]: <zip at 0x1051d3f88> pasteurize -w py3sample.py In [ ]: # %load py3sample.py from future import unicode_literals from future import print_function from future import division from future import absolute_import from builtins import zip from builtins import range from future import standard_library standard_library.install_aliases() zip( abc, range(3)) old version in py3sample.py.bak In [ ]: 14