Guillimin HPC Users Meeting December 14, 2017

Transcription

1 Guillimin HPC Users Meeting December 14, 2017 McGill University / Calcul Québec / Compute Canada Montréal, QC Canada

2 Please be kind to your fellow user meeting attendees Limit to two slices of pizza per person to start please And please recycle your pop cans. Thank you! 2

3 Compute Canada News Guillimin Decommissioning and Data Migration System Status Software Update Training News Special Topic How to Install Python Packages in your Account Outline 3

4 Compute Canada News 2018 Resource Allocation Competitions Technical reviews are undergoing No more allocation on Guillimin Any Fast Track and RPP compute allocation on Guillimin will be moved to another cluster Groups having a storage allocation on Guillimin could request NDC-GP4 storage allocation for

5 Guillimin Decommissioning Guillimin Decommissioning Installation of GP4 will require that Guillimin compute and storage be decommissioned and removed Guillimin Phase 1 (2011) and Phase 2 (2013) Core integration at the level of the InfiniBand Network Guillimin Compute Phase 1 Compute - 14,400 cores Phase 2 Compute - 8,000 cores and 116 GPUs, 100 Phis Guillimin Storage Phase 1 Storage - 2 PB Phase 2 Storage - 3 PB Tape Storage PB Includes large amounts of hosted CFI nodes and storage 5

6 Guillimin Decommissioning GP4 Timeline - Tentative and preliminary (!) 6

7 Guillimin Decommissioning Multi-step Approach Guillimin Storage Migration Guillimin Code and Application Migration Separation of Guillimin Phases 1 and 2 Separation at the level of the InfiniBand Network Approximately 2 weeks full downtime anticipated Phase 1 Compute Shutdown and Removal Phase 2 Compute Shutdown and Removal Some or all of Phase 2 Compute Nodes will be kept online and in operation so as to overlap with GP4 7

8 Guillimin Decommissioning Contributed CFI Resources in Guillimin Contributed CFI resources are part of Phase 2 and not scheduled for decommissioning until January 2019 January 2019 = the expected 5 year duration for operations of CFI funded infrastructure Includes compute, GPUs and storage The contributed Phase 2 resources will either: Be kept online and operational until January 2019 An equivalent portion of GP4 compute and/or storage resources will be assigned to the CFI project as a bridge until January 2019 More information to be communicated to award holders 8

9 Guillimin Migration Guillimin Data Migration Approximately 5 PB of GP4 disk storage will be purchased in advance so as to host all current Guillimin data (Feb - March 2018) This copy of Guillimin data will be visible on GP4 Migrations will be performed centrally by staff Aim is to reduce time required to complete migration Ensure data is copied to the new storage platform Some migrations can be performed directly by users Clean-up and remove any unused data files Migrate any of your current Guillimin files to elsewhere in Compute Canada (Cedar or Graham) All migrations efficiently managed through Globus Please start ASAP the clean-up and migrations that can be made 9

10 Guillimin Migration Guillimin Code and Application Migration GP4 not ready for production use until ~Sept 2018 So no GP4 Compute for start of RAC 2018 Compute Canada CVMFS software stack is the default across all GP platforms including Cedar, Graham and GP4 Groups can access the software stack already on Guillimin source /software/soft.computecanada.ca.sh Start testing your codes based on the new stack and start copying your applications and codes to Cedar and/or Graham Default compute & storage resource amounts are larger on Cedar / Graham (2017) than Guillimin (2011/2013) 10

11 Guillimin Migration Questions regarding Guillimin compute and storage migration Contact us: Information regarding the migration will be updated and communicated to all users as the timeline and actions become more concrete Documentation for Cedar and Graham a_documentation Support inquiries: 11

12 System Status About recent storage system issues Dec. 6: Maintenance activities on the storage system We reduced the load on GPFS by pausing job scheduling and reserving full racks hours in advance We stopped GPFS on the problematic storage node; the redundant storage node got the full (reduced) load Maintenance done quickly without any GPFS issue Storage: need to cleanup /sb and /lb Final deadline : by the end of January 2018 For groups using exclusively /sb, we will open access to a project space on /gs Storage: /gs is 97% full as of December 12 12

13 Software Update Software on Compute Canada systems McGill users can now use Matlab on Cedar and Graham To use modules and non-commercial packages on Guillimin: source /software/soft.computecanada.ca.sh 13

14 Training News All upcoming events: calculquebec.eventbrite.ca --- Suggestions for training? Please let us know! Recently completed: Nov Développement d'intelligence artificielle sur grappes de calcul: Dec. 7 - Introduction to R for Scientific Computing (McGill) All materials from previous workshops are available online: wiki.calculquebec.ca/w/formations/en All user meeting presentations online at 14

15 Other News Support Level Activity during the Holiday Period December 22 to January 2nd inclusive - returning January 3 Reduced level of access to general user support All systems and services available and will be closely monitored Priority and critical issues will be addressed Happy Holidays! Joyeuses Fêtes! Guillimin HPC Users Meeting - December

16 User Feedback and Discussion Questions? Comments? We value your feedback. Contact us at: Guillimin Operational News for Users Status Pages (all CQ systems) Follow us on Twitter 16

17 How to Install Python Packages in your Account December 14, 2017 McGill University / Calcul Québec / Compute Canada Montréal, QC Canada

18 Outline By using the Compute Canada software stack: scipy-stack module Using Virtual Environments, pip and wheels Using python setup.py Other methods : make,... Most frequent problems How to force a package upgrade How to uninstall a package How to fix library paths (setrpaths.sh) Documentation: 18

19 Compute Canada software stack, Python versions By using the Compute Canada software stack: On Cedar and Graham: default On Guillimin: source /software/soft.computecanada.ca.sh $ module avail python python/ (t,2:2.7) python/ (t) python/3.5.2 (t,d:3.5) python/3.5.4 (t) python/3.6.3 (t,3:3.6) Recommended: or (3.6.3 is catching up) module load python/ Or module load python/3.5.4 These are bare-bones python modules (basic Python+ setuptools, pip, virtualenv, wheel, readline, pysqlite) 19

20 scipy-stack module scipy-stack module on top of python module Following: module load scipy-stack/2017b brings you the following packages and their dependencies: numpy scipy matplotlib dateutil-2.6.0, pytz default backend: agg, for non-interactive batch plotting ipython pyzmq , tornado pandas nose-1.37 sympy Cython

21 Using virtualenv Almost anything else: use virtualenv (or pyvenv for Python 3) and pip $ virtualenv ENV New python executable in /home/oldeman/env/bin/python Installing setuptools, pip, wheel...done. $ source ENV/bin/activate (ENV) $ pip install scikit-learn Collecting scikit-learn Installing collected packages: scikit-learn Successfully installed scikit-learn (ENV) $ python -c import sklearn (ENV) $ deactivate $ 21

22 Python wheels pip install will try to use our wheel store before downloading from the internet Avoids random binaries and unoptimized compilations For example: The numpy wheel uses MKL for optimized linear algebra. The tensorflow wheels properly integrate in our setup and works with all GPUs. Wheels are here: /cvmfs/soft.computecanada.ca/custom/python/wheelhouse Contact us if a wheel is missing. Or DIY development numpy: pip install git+git://github.com/numpy/numpy.git 22

23 Exception: mpi4py mpi4py depends on the MPI module: (default=openmpi 2.1.1) $ module load python/ \ scipy-stack/2017b mpi4py/3.0.0 Guillimin specific: may need to set mpi4py.rc.recv_mprobe = False After import mpi4py Depending on the MPI that is used 23

24 Using python setup.py If pip does not work $ module load python/ $ virtualenv ENV $ source ENV/bin/activate (ENV) $ tar xvf package-version.tar.gz (ENV) $ cd package-version (ENV) $ python setup.py build (ENV) $ python setup.py install 24

25 Other methods: make... If there is no setup.py script, may need to use make Look for README* and INSTALL* files, or ask for help by writing us. 25

26 What about --user? Using --user after pip install or python setup.py install installs under ~/.local. Hence it behaves like a default virtual environment. We prefer separate virtual environments: easier to manage, one virtualenv per project. 26

27 What about Anaconda and Jupyter? Some prefer Anaconda instead: please see instead of trying to install it manually by downloading binaries from the internet. Jupyter: see 27

28 Most frequent problems Examples: Running pip install tensorflow outside a virtualenv: Read-only file system: '/cvmfs/soft.computecanada.ca/easybuild/software /2017/Core/python/2.7.14/lib/python2.7/site-pack ages/backports' Trying sudo pip install tensorflow Sorry, user oldeman is not allowed to execute '/bin/pip install tensorflow' as root on cedar5.cedar.computecanada.ca. Solution: see (use tensorflow-cpu and tensorflow-gpu wheels) Similar for pytorch and dlib. 28

29 How to force a package upgrade pip install --upgrade --force-reinstall <package> OR pip install --ignore-installed <package> OR deactivate rm -rf ENV # name of virtual environment virtualenv ENV pip install... 29

30 How to uninstall a package pip uninstall package Or remove (rm -rf) the virtualenv as on the previous slide. 30

31 How to fix library paths (setrpaths.sh) Sometimes a binary wheel from the internet will get installed. In generally this should be avoided but it there is no other possibility you can do setrpaths.sh --path /path/to/env #name of virtualenv Which make sure all binaries are linked correctly within our setup (see re_in_your_home_directory) 31