Using jupyter notebooks on Blue Waters. Roland Haas (NCSA / University of Illinois)

Transcription

1 Using jupyter notebooks on Blue Waters Roland Haas (NCSA / University of Illinois) rhaas@ncsa.illinois.edu

2 Jupyter notebooks 2/18 interactive, browser based interface to Python freely mix python commands shell commands graphics markdown explore data and debug scripts interactively more memory (128GB) than your laptop no need to copy data off Blue Waters export result as Python script share notebook with collaborators

3 Three ways to use jupyter on Blue Waters on login node simple to set up on multiple compute nodes large scale python applications using MPI, e.g. yt task farms with many jobs workflow systems like parsl complex to set up on a single compute node 3/18 good for quick, inexpensive tasks for tasks longer than what the process killer allows for tasks using large amounts of resources (IO, memory, CPU) somewhat harder to set up ssh Notebook server TCP Python kernel laptop notebook file Jupyter starts a notebook server that one connects to using a browser Blue Waters

4 Common setup jupyter notebook server is provided by bwpy module module load bwpy notebook server listens for connections using TCP anyone on Blue Waters can connect provides full access to your account protect access using a password password is set in from socket import * ips = gethostbyname_ex(gethostname())[2] for ip in ips: if(ip.startswith("10.")): internal_ip = ip break from IPython.lib import passwd c.notebookapp.password = passwd("$passwd$") c.notebookapp.open_browser = False c.notebookapp.ip = internal_ip thonnotebooks jupyter_notebook_config.py 4/18 file in ~/.jupyter directory config file available on portal notebook server listens on internal network interface only default password is $passwd$

5 Jupyter on login nodes bw$ module load bwpy bw$ jupyter notebook The Jupyter Notebook is running at: laptop% ssh -L 8888: :8981 bw.ncsa.illinois.edu laptop% open 5/18 Notebook server accessible Blue Waters wide but not from the public internet jupyter outputs connection information to stdout on startup use second ssh connection to any login node to forward ports on Windows ssh -L :8888: :8981 bw may be required

6 General issues and suggestions 6/18 bwpy provides a large number of python modules useful to explore data numpy, scipy, sympy h5py, netcdf, gdal, pandas PostCactus matplotlib, yt, plotly pip list shows all packages use %matplotlib notebook magic command to show plots jupyter auto-saves notebooks in case connection is dropped

7 Jupyter on compute nodes sample PBS script (ccmrun!) #!/bin/bash #PBS -l nodes=1:xe:ppn=32 #PBS -l walltime=2:0:0 module load ccm module load bwpy ccmrun jupyter-notebook uses same jupyter_notebook_config.py as on login node CCM used to provide cluster like environment on compute node use xe or xk nodes connection information written to <JOBID>.bw.OU create port forwarding as before, this time to compute node laptop% ssh -L 8888: :8981 bw.ncsa.illinois.edu laptop% open 7/18

8 Parallelism in python bag-of-jobs workload use multiprocessing module cessing.html use multi-threading in compiled code more complex workflows use parsl ( install in virtualenv using pip caveats request all 32 (16) cores in PBS script avoid core binding (ccmrun is fine) 8/18 check if you are compute or IO bound Multiprocessing import multiprocessing as mp pool = mp.pool(processes=8) def sqr(x): return x*x data = range(21) squared = pool.map(sqr, data) print(squared) Parsl from parsl import App, DataFlowKernel import parsl.configs.local as lc dfk = dfk) def sqr(x): return x*x data = range(21) squared = map(sqr, data) print([i.result() for i in squared])

9 Multiple compute nodes using MPI PBS starts notebook server ssh ccmrun jupyter-notebook this does not work with aprun aprun -n2 jupyter-notebook starts 2 copies of the notebook server, not two copies of the python kernel need to start two copies of kernel and hook up to server handled by ipyparallel 9/18 TCP Python kernel laptop notebook file Notebook server Blue Waters TCP MPI workers ipyparallel lets you use clusters with jupyter multiple parallelism backends local processes good for testing MPI (and PBS) Blue Waters supported in bwpy/1.1.1

10 Setting up ipyparallel in your account bw$ module load bwpy bw$ jupyter serverextension enable --py ipyparallel bw$ jupyter nbextension install --user --py ipyparallel bw$ jupyter nbextension enable --py ipyparallel bw$ ipython profile create --parallel --profile=pbs installs to $HOME/.local cannot use virtualenv pbs profile is in $HOME/.ipython/profile_pbs ipcluster_config.py contains launcher class (PBS) ipcluster_config.py PBS script template ipengine_config.py port to listen for engines 10/18 enable MPI in ipengine_config.py: c.mpi.use = 'mpi4py' listen only to internal BW network

11 Sample ipcluster_config.py file ipcluster_config.py: from socket import * ips = gethostbyname_ex(\ gethostname())[2] for ip in ips: if(ip.startswith("10.")): internal_ip = ip break c.hubfactory.ip = \ internal_ip c.batchsystemlauncher.\ queue = 'debug' 11/18 c.ipclusterengines.\ engine_launcher_class = \ 'PBSEngineSetLauncher' c.pbsenginesetlauncher.\ batch_template = """#!/bin/bash #PBS -q {queue} #PBS -l walltime=00:30:00 #PBS -l nodes={n//4}:ppn=32:xe module load bwpy bwpy-mpi OMP_NUM_THREADS=8 aprun -n{n} -d$omp_num_threads \ bwpy-environ -- ipengine \ --profile-dir={profile_dir} """

12 Starting ipyparallel on compute nodes ipcluster command in bwpy creates parallel workers bw$ module load bwpy bw$ bwpy-environ ipcluster start --n=4 --profile=pbs alternatively can use IPython Clusters tab in jupyter notebook for same purpose connect to workers in jupyter notebook import ipyparallel as ipp ranks = ipp.client(profile='pbs') ranks.ids if this is empty then workers are not yet ready wait and try again ipcluster has --PBSLauncher.queue option which accepts any string for the {queue} placeholder. This allows for creative abuse. 12/18

13 MPI parallel notebooks examples (1/4) 13/18 two sets of nodes notebook and ipython kernel MPI workers use %%px cell magic command to execute code on MPI ranks use mpi4py to access MPI ranks[rank]['var'] accesses var from rank rank worker variables persist between %%px cells output to stdout and stderr is forwarded

14 MPI parallel notebooks examples (2/4) %%px import numpy as np, numpy.random as rd x = np.arange(10); y = rd.rand(len(x)) import matplotlib as mpl mpl.use('agg') import matplotlib.pyplot as plt if MPI.COMM_WORLD.Get_rank() == 3: p = plt.plot(x, y) %matplotlib notebook ranks[3]['p']; 14/18 can use matplotlib to visualize results runs on all nodes by default use MPI rank to select single node plots need to be stored and pulled to display use %matplotlib notebook magic

15 MPI parallel notebooks examples (3/4) %%px import yt yt.enable_parallelism() ds = yt.load(\ "enzo_tiny_cosmology/d0046/d0046") p = yt.projectionplot(ds, "y", "density") if yt.is_root() print ("Redshift =", ds.current_redshift) p.show() 15/18

16 MPI parallel notebooks examples (4/4) %%px ds = yt.load(\ "IsolatedGalaxy/galaxy0030/galaxy0030") t2 = time.time() sc = yt.create_scene(ds) sc.camera.set_width(ds.quan(20, 'kpc')) source = sc.sources['source_00'] tf = \ yt.colortransferfunction((-28, -24)) tf.add_layers(4, w=0.01) source.set_transfer_function(tf) sc.render() if yt.is_root(): sc.show() 16/18

17 Question? This research is part of the Blue Waters sustained-petascale computing project, which is supported by the National Science Foundation (awards OCI and ACI ) and the state of Illinois. Blue Waters is a joint effort of the University of Illinois at Urbana-Champaign and its National Center for Supercomputing Applications.

18 Using compatible Python versions bwpy/1.1.0 offers python 2.7, 3.5 and 3.6 and defaults to python 3.5 jupyter notebook uses python 3.6 ipyparallel workers use python 3.5 causes hang or crash when accessing worker variables adjust jupyter kernel config 18/18 $HOME/.local/share/jupyter/kernels/ python3/kernel.json { "argv": [ "python3.5", "-m", "ipykernel_launcher", "-f", "{connection_file}" ], "display_name": "Python 3", "language": "python" }