HPC at UZH: status and plans

Transcription

1 HPC at UZH: status and plans Dec. 4, 2013

2 This presentation s purpose Meet the sysadmin team. Update on what s coming soon in Schroedinger s HW. Review old and new usage policies. Discussion (later on). UZH University of Zurich, GC3: Grid Computing Competence Center Dec. 4, 2013

3 We want your feedback! How do the policies impact your usage and productivity? What would you like to see introduced, and what removed? Would like changes introduced to Schroedinger to converge to the new HPC system? UZH University of Zurich, GC3: Grid Computing Competence Center Dec. 4, 2013

4 Meet the team

5 GC3 sysadmin team Tyanko Aleksiev Antonio Messina Riccardo Murri

6 Contact us or

7 Today s HW status

8 Status of current HW Schroedinger s HW has been in 24 7 use 4 years long. Pieces are now starting to fail: 39% of the total HW failures happened in 2013 every 2 weeks, 3 Lustre disks fail 1 storage blade on the Panasas system failed 16 compute nodes failures in the last two months

9 Status of current HW Schroedinger s HW has been in 24 7 use 4 years long. Pieces are now starting to fail: 39% of the total HW failures happened in 2013 every 2 weeks, 3 Lustre disks fail 1 storage blade on the Panasas system failed 16 compute nodes failures in the last two months Replacing storage is the most important task, as it holds data.

10 Review of current storage policies /home: 5GB quota, nightly back-ups on Tivoli /data: 50GB quota, no backups /lustre: no quota, no backups, originally meant as scratch space

11 Status of support contracts Support contract with Oracle expires mid-december. HW support on the compute nodes will not be renewed. Expect a little capacity degradation during next year (estimate around 5%)

12 New storage HW

13 New storage HW: Panasas New Panasas PAS8 already bought by Informatikdienste Arrives this week (likely) two storage shelves, total 76TB raw fully redundant configuration

14 New Panasas: deployment (December 2013) Try to switch to the new PAS8 during this month s scheduled maintenance window. (Dec. 18) 1. Migrate all data (estimated time: 7 10 days) The old Panasas still serves /home and /data directories for IDES We might need to throttle the migration bandwidth in order not to interfere with normal cluster operations 2. Do a final rsync during the downtime filesystem must be quiescent: cluster must be free from users and jobs might take longer than expected!

15 New storage HW: Lustre The Informatikdienste already bought a complete replacement for the Lustre filesystem. HW will be delivered in December, but it won t be ready for productive use until after Xmas

16 New Lustre: deployment (February 2014) 1. Burn-in and tune servers and new Lustre software until February 18, In the meanwhile, you have to copy your important data from /lustre into /home or /data 3. Switch-over to new Lustre filesystem during the scheduled maintenance in February. No Lustre files will be preserved across the switch!

17 New Lustre: deployment (February 2014) 1. Burn-in and tune servers and new Lustre software until February 18, In the meanwhile, you have to copy your important data from /lustre into /home or /data 3. Switch-over to new Lustre filesystem during the scheduled maintenance in February. No Lustre files will be preserved across the switch!

18 New Lustre: deployment plan rationale 1. Current Lustre s capacity is 230TB, over 70% full. Migrating all data will take ages! 2. Lustre was conceived as a scratch, but is in fact used as a data storage. Given the filesystem size, we cannot guarantee data safety (backups, redundancy). So we would like to enforce the policy that /lustre is for scratch (i.e., temporary) files only! 3. Lustre version bumps from to 2.4: possible co-existence problems

19 Review of future storage policies /home: 10GB quota, nightly back-ups on Tivoli /data: 100GB quota, no backups /lustre: no quota, no backups, scratch only files older than 60 days will be automatically deleted

20 Review of future storage policies /home: 10GB quota, nightly back-ups on Tivoli /data: 100GB quota, no backups /lustre: no quota, no backups, scratch only files older than 60 days will be automatically deleted

21 SW and policy changes

22 SW changes: Operating System No planned changes to the operating system. But we may need to update to SLES 11 SP2 because of the new Panasas. (So maybe you ll end up recompiling your applications anyway.)

23 modulefiles reboot (March 2014) Remove current module files and start with a new set: C/C++ compilers (GNU + Intel) FORTRAN compilers (GNU + Intel) OpenMPI FFTW MatLab what toolboxes do you use/need? R Need more supported software? Ask!

24 Standardize on OpenMPI (April 2014) OpenMPI 1.6 becomes the only MPI library supported by the Schroedinger admin team. Provided via module load openmpi ParastationMPI remains available, but the support contract with Par-Tec will be rescinded. Other MPI libraries will be removed (they have not been updated nor used since quite a while).

25 SW changes: batch system? (May 2014) Oracle is not developing GridEngine any more, nor supporting it. Switch to SLURM? already used at CSCS and in the zbox4 big changes in the usage and command-line organize a 1-day training session on the new batch system test-drive SLURM cluster available starting March 2014 what should the user-level documentation cover? Alternative: keep GridEngine keep submitting as usual known bugs will not be fixed

26 Timeline 2013 December New /home and /data 2014 February New /lustre March module files reboot April standardize on OpenMPI May new batch system?

27 Thank you! Any questions?

28 Timeline 2013 December New /home and /data 2014 February New /lustre March module files reboot April standardize on OpenMPI May new batch system?

29 Appendix

30 modulefiles usage Jun. Nov mpi/openmpi/gcc java/ intel/mkl 4556 intel/comp 4507 intel/comp/ mpi/openmpi-1.4.5/gcc mpi/openmpi-1.4.5/gcc R/ matlab/r2011a 1213 intel/comp/ intel/mkl/ mpi/parastationmpi/intel 889 fftw/3.2.2-double 422 binutils/ mpi/openmpi-1.6.2/gcc gcc/ mpi/openmpi/intel 181 mpi/parastationmpi/gcc 105 gcc/4.5.0-system 87 matlab/r2012a 40 gcc/ gcc/ R/ mpi/openmpi/gcc mpi/openmpi-1.4.3/gcc mpi/mvapich2 6 gcc/ mpi 2 matlab 2 gcc/ gcc 2 intel 1 gcc/ binutils 1 Back to Modulesfiles reboot