[%]%async_run. an IPython notebook* magic for asynchronous (code) cell execution. Valerio Maggio Researcher

Transcription

1 [%]%async_run an IPython notebook* magic for asynchronous (code) cell execution Valerio Maggio

2 Premises

3 Jupyter Notebook

4 Jupyter Notebook

5 Jupyter Notebook Example Multiple Kernels

6 Currently in Use at

7

8 Jupyter Architecture Notebook Document Format Jupyter Notebooks are an open document format based on JSON. They contain a complete record of the user's sessions and embed code, narrative text, equations and rich output. Interactive Computing Protocol The Notebook communicates with computational Kernels using the Interactive Computing Protocol, an open network protocol based on JSON data over ZMQ and WebSockets. The Kernel Kernels are processes that run interactive code in a particular programming language and return output to the user. Kernels also respond to tab completion and introspection requests.

9 >50 Kernels github.com/ipython/ipython/wiki/ipython-kernels-for-other-languages

10 Reproducible Research

11 Motivations Sometimes it may be required to apply for heavy computations computationally intensive code cells Moreover, sometimes may be required that this computation is actually executed on a remote server machine reminder: Jupyter Notebook Server In the general case, this could work but since

12 Anything that can possibly go wrong, does. Murphy s Law, 1952

13 Main Goal Try to define a strategy to cope with this kind of situation keeping the following requirements in mind: Allow the execution on a remote machine (also) Avoid the client machine to busy waiting Keep the interactivity of the notebook as much as possible

14

15 [%]%async_run an IPython notebook* magic for asynchronous (code) cell execution What I learned during my adventures in the world of Jupyter, Multiprocessing and Asynchronous I/O

16 Jupyter Ecosystem

17

18

19 [%]%async_run an IPython notebook* magic for asynchronous (code) cell execution What I learned during my adventures in the world of Jupyter, Multiprocessing and Asynchronous I/O

20 IPython Magics (since IPython 3.x) IPython has a system of commands we call magics provide effectively a mini command language that is orthogonal to the syntax of Python easily extensible by the user with new commands. Magics are meant to be typed interactively i.e. command-line conventions e.g. whitespace for separating arguments, dashes for options. Magics come in two kinds: Line magics: prepended by one % character Cell magics: two percent characters as a marker (%%)

21 [%]%timeit Line Magic Cell Magic

22 Activate matplotlib inline-backend to have charts displayed inline with notebook cells

23 Custom Magics: how to

24

25

26

27 Notebook Data Format

28 Notebook Data Format

29

30

31

32

33 Back to our issue to solve Anything that can possibly go wrong, does. Murphy s Law, 1952

34 Why that?

35 First Idea (very early stage) run the heavy computation using the write API to add a new cell to the notebook and that s it. Drawbacks: No interactivity No way to auto-refresh the content y to check for existing

36 Try to see if there s any already existing solution to this! Take away: avoid reinventing the wheel!

37 %run to the rescue (?) ipython.org/ipython-doc/3/interactive/magics.html#magic-run

38 %run

39

40 Test in the notebook

41 A bit more complicated

42 test it! Blocking Call No interactivity

43 runipy to the rescue (?)

44 runipy to the rescue (?)

45 A closer look

46 A closer look

47 Notebook Runner

48 runipy features (+) Notebook APIs (+) Kernel Protocol Messaging (+) Support for multiple document formats nbformat.versions (-) No interactivity (-) No support for online/non-blocking execution (~) No support for multi-processing

49 Idea: try to borrow some code from runipy and re-implement it as an IPython Magic (w/ steroids)

50

51 But if you : Hangs on protocol communication and it has no link with the current shell

52

53 [%]%async_run an IPython notebook* magic for asynchronous (code) cell execution What I learned during my adventures in the world of Jupyter, Multiprocessing and Asynchronous I/O

54 IPython is based on Tornado!

55 Reference

56

57

58

59 Client-side

60

61

62

63 Limitations and Future Works Pickle/JSON Serialisation Dependency Major Flaw of Python Multiprocessing Module Try to use dill multiprocessing* Improve the infrastructure to handle errors not really handled yet apart from IPython/JS Integration

64 Not going to be any demo Due to aforementioned Murphy s Laws :P

65 Thanks a lot for your kind valeriomaggio@gmail.com +ValerioMaggio it.linkedin.com/in/valeriomaggio