J O U R N É E D E R E N C O N T R E D E S U T I L I S A T E U R S D U P Ô L E D E C A L C U L I N T E N S I F P O U R L A M E R P I E R R E C O T T Y

Transcription

1 J O U R N É E D E R E N C O N T R E D E S U T I L I S A T E U R S D U P Ô L E D E C A L C U L I N T E N S I F P O U R L A M E R P I E R R E C O T T Y

2 Le programme de la journée 9 h h 10 DATARMOR: nouveau périmètre, nouvelles perspectives Pause déjeuner : buffet 14 h h 30 DATARMOR: nouveaux champs d'applications scientifiques Poster et Discussions autour d un pot 17/05/2018 2

3 D ATA R M O R A N D N E W T O O L S Tina ODAKA

4 DATARMOR 17/05/2018 4

5 ACCES CLIENTS (10 Gigabit Ethernet connected to IFREMER and INFUSER network (SHOM / IUEM / ENSTA Bretagne Cluster HPC 426 Tflops cores 396 Nodes Login nodes PBS Pro; sharing computing resources Batch system: PBS Pro InfiniBand FDR 17/05/2018 5

6 DATARMOR: not only HPC Cluster SMP Cluster HPC Cluster VIS SCRATCH High speed Network Infini Band Cluster WEB DATAREF DATA HOME DATA WORK 17/05/2018 6

7 DISK DATAREF (LUSTRE, 1.5PB = 1395TiB) Reference data (next presentation) DATAWORK (GPFS, 5PB = 4500TiB) Work data User space and project space SCRATCH (LUSTRE 0.5PB) Scratch file for HPC computation. 15 days max one kibibit DATAHOME (NFS, 0.04PB) PB? TiB?? comparisons with International System of Units (SI) prefixes 1 Kibit = 2 10 bit = 1024 bit one kilobit 1 kbit = 10 3 bit = 1000 bit one mebibyte 1 MiB = (2 10 ) 2 B = B one megabyte 1 MB = (10 3 ) 2 B = B one gibibyte 1 GiB = (2 10 ) 3 B = B one gigabyte 1 GB = (10 3 ) 3 B = B byte = octet( French) TiB (tebibyte) = Tio (French tébioctet) Source codes and softwares with backup ( tape, in a different building) 17/05/2018 7

8 Clustre WEB, SMP, VIS Cluster SMP 1 node with 240 cores, 5 TB RAM PBS Pro Jobs only for huge memory, or many CPU cores. Cluster VIS 2 nodes each with: 2 GPU (NVIDIA Quadro K6000: 2880 cuda cores, 12GB memory) 2 Intel Cpu 2.2 Ghz each with 10 cores PBS Pro Visualisation. Cluster WEB 10 servers Based on vmware virtual machine, and soon in docker/kubernetes Hosting web services for datarmor, like VRE 17/05/2018 8

9 New tools on DATARMOR 17/05/2018 9

10 VRE: virtual research enviroment Complex system but simple access SeaDatacloud, Emodnet 17/05/

11 Traditional access ssh : terminal, command line,,, Must have basic linux knowledge. 17/05/

12 Challenge DATARMOR more flexible (web, click,,, ) access Less of : vi, ssh, fortran, mpi Flexible, and easy access should speed up the learning/development/analysis time for users. It can lower the barrier for new users non traditional users. It might slowdown computing time, but if it globally wins the time from development through having result in a graph to be putted in a paper, whats the harm of slowing down a bit of computing time? How to co-habite traditional HPC users and new users? 17/05/

13 first step for VRE: example Visit PBS Pro controls the access. Cohabitation with traditional usage Remote access to GPU Co-working possibilities 17/05/

14 Jupyter notebook: interactive usage of computing power through PBS Pro Netcdf related graphical tools (python) Julia high performance, altenative of python?? DIVA (fortran-mpi to Julia) Pyspark session spark end of this morning! Machine learning tools through conda Or other GUI tools Compmression of image: interface by nomachine software. 17/05/

15 Virtual environment: integration with PBSPRO Benefits: Light weight Can share environment between different architectures. Keep same environments as your PC/datarmor DATARMOR Bioinformatics web-tools conda IFREMER developed Docker integration to PBS pro. PBS Pro controls the access. Cohabitation with traditional usage

16 DATARMOR: status of use 17/05/

17 Reserved resources en K hours/month mai-17 juin-17 juil.-17 août-17 sept.-17 oct.-17 nov.-17 déc.-17 janv.-18 févr.-18 mars-18 avr /05/

18 Reserved CPU cores resources (%) 80,00% 70,00% 60,00% 50,00% 40,00% 30,00% 20,00% 10,00% 0,00% juil.-17 août-17 sept.-17 oct.-17 nov.-17 déc.-17 janv.-18 févr.-18 mars-18 avr.-18 HPCSIMPLE HPC HYPERHREAD SMP VISU 17/05/

19 Use of DATAWORK (TiB) MAX 4500 TiB juil.-17 août-17 sept.-17 oct.-17 nov.-17 déc.-17 janv.-18 févr.-18 mars-18 avr.-18 Datawork alloué (Tio) Datawork occupé (Tio) 17/05/

20 Upgrade 2018 Increase the capacity of datawork 1.5PB =1364TiB cluster-web Summer /05/

21 DATARMOR is not only HPC DATARMOR and (Virtual) research environment Upgrade 2018 coming SOON T H A N K Y O U 17/05/