SURFSARA, TECHNOLOGICAL EXPERTISE & SURF E- INFRASTRUCTURE Axel Berg Information event etec-big call NLeSC-SURFsara Amsterdam, April 9, 2019

Transcription

1 SURFSARA, TECHNOLOGICAL EXPERTISE & SURF E- INFRASTRUCTURE Axel Berg Information event etec-big call NLeSC-SURFsara Amsterdam, April 9, 2019

2 We are SURF

3 Fields of work Education Flexible education Diverse learning resources Using study data Research Unlimited access World-class facilities Stimulating Open Science Cooperative facilities On campus Security in the digital world User-centred

4 Worldwide LHC Computing Grid National supercomputer service Cartesius 4

5 It s far more than just BIG systems As scientific problems become more complex: - collaborations will grow - data sizes will grow - e-infrastructure demands will grow - requires usually integration of compute and data For future e-infrastructure development this means advances and collaborations in the complete e-infrastructure ecosystem, through cocreation: community services + software (system, middleware, libraries, applications) + algorithms + programming models + workflows + hardware (compute, data, network) + datacenter + operations + support + data management + training + education + integration + federation + you name it LAWRENCE LIVERMORE EXAMPLE 5

6 Photo: SETI Institute Example SETI, the Search for Extraterrestrial Intelligence Exploratory science that seeks evidence of life in the universe by looking for some signature of its technology. Access to national research ICT facilities Data processing of LOFAR telescope data on GRID Applying complex queries on Hadoop cluster using Spark and Scala In-depth scientific and technological expertise to set up data pipelines

7 Convergence of machine learning and largescale numerical simulations Could machine learning approaches enhance traditional HPC workloads, like numerical simulations? Predict simulation outcomes instead of calculating a full simulation using numerical analysis Use data from simulation to train a deep neural network à use for inference analysis to simulate the system being studied simulate coarse grid, input to make predictions at refined grid 7

8 Machine learning enhanced HPC applications Climate modeling Dr. Chiel van Heerwaarden, Wageningen University: Machine-learned turbulence in next-generation weather models High-energy physics Dr. Sacha Caron, Radboud University: Generating physics without an event generator Life Sciences Prof. Alexandre M.J.J. Bonvin, Utrecht University: Distinguishing biological interfaces from crystal artifacts in biomolecular complexes using deep learning Astrophysics Prof. Simon Portegies Zwart, Leiden University: Machine learning for accelerating planetary dynamics in stellar clusters Article Transforming HPC research with AI approaches on The Next Platform -hpc-research-with-ai-approaches/ 8

9 Collaboration SURF research community in service development RESEARCH DOMAIN EXPERTISE.. Data services Software services Nationale ICT infrastructure services CO-CREATION ICT INFRASTRUCTURE EXPERTISE 9

10 Example collaborations

11 Essential component of e-infrastructure: Expert Support Matching e-infrastructure (development) to scientific problem à effective use of the e- infrastructure Co-creation: discipline expertise and e- infrastructure development and implementation expertise Scalability of e-infrastructure and applications Close collaboration with scientists, NLeSC and SURF Performance optimisation and parallelisation Algorithm improvements I/O optimisation Accelerator optimization Treatment of large datasets Distributed computing Scalable machine learning Data analytics Streaming data pipelines Research data management Visualisation (remote, in-situ) Etc. 11

12 Connecting research challenges and ICT solutions Which ICT service is most suited for my research challenge? 5 Areas of Support National and International Access Mode Routing How do I gain access to the national and international research facilities and which access mode is most suited (institutional access, excellence-driven, wide access, international)? Optimalizing ICT services SaaS > How to optimise software? PaaS > How can I make use of discipline specific research environment platforms? IaaS > How to optimise the ICT infrastructure with parallel processing and job batching? Analysing & visualising How to gain insight and meaning from research data? How to get results from complex and unstructured data sets? How do I visualise my research data? Managing research data How to handle with data before, during and after my research? How to collect, collaborate, manage, transfer, store, find and reuse data?

13 The national digital infrastructure Access to a wide range of reliable services to process, analyse, store and share data Available to all researchers, independent of the scientific domain Superior digital infrastructure is crucial to the Dutch knowledge economy

14 Digital infrastructure for research in Netherlands LARGE-SCALE SCIENTIFIC INFRASTRUCTURE: Report Access Compute Storage Network Visualization Authentication & Authorization Research ICT support NL e-infra Research Institutes Store 14

15 What SURF does for research (Re)search High-end Compute services Supercomputing Cluster computing Visualisation Report Access Data processing Scalable data analytics Grid services HPC cloud Analyse RESEARCH DATA LIFECYCLE Store Data services Online data services Data preservation services Data management services Network & collaboration services Optimise Translate Fast networks for data transfers and compute scale out Federated access & identity management solutions Integrate Compute 15

16 Computing services Dutch Supercomputer Cartesius large scale parallel applications Lisa Computer Cluster processing power with user friendliness Visualisation powerfull remote visualization Grid high-end distributed data processing HPC Cloud high-performance cloud for research Available compute power >2 PFlops On > CPU cores >450 GPUs 16

17 Scalable Data Analytics Exploration / Mining of Big Data Streaming data (real time analysis) 2 to 6 million tweets/day Prof. Antal van den Bosch Radboud University, Meertens Institute Spark framework, flexible infrastructure built on Kubernetes, training with Jupyter notebooks 17

18 SURF Data services Service Features Intended for Long/ short term SURF Research Drive One view on data Collaborative working and sharing of Short/ research data long term SURFdrive DropBox like Online collaboration Short term SURFfilesender WeTransfer like with encryption HPC central data archive Tiered, Dual copies, supports NFs/ ssh/scp/gridftp PID service Identifier service for datasets Online collaboration For use with super compute facilities Publishing data Short term Long term preservation Long term SURF Object Store S3/REST API Online storage for large data volumes Short term SURF Data Management platform Metadata handling Policy based data orchestration Long term SURF object storage

19 The collaborative environment that supports open science Analyse Describe Manage HPC Process National Super Computer HPC Cloud Services Publish Collect Access Find PID Service Object Store Data Archive Persistent Identifiers Online S3/REST API Offline sharekit Data Preservation Future Concept

20 Identity Management Services Identity Providers Service Providers SURFconext - access to cloud services - federative login (Institutional account, log in once, receive access everywhere) Science Collaboration Zone (pilot) - access to (research) specific services - federative login AND - guests - teams - international 20

21 ICT, expertise & collaboration ICT services Knowledge & expertise Collaboration -Large-scale computations, simulations & modelling on Cartesius supercomputer, Lisa cluster, HPC cloud* -Accessing the Grid for large-scale data processing -PaaS environment tools, compilers, libraries -Performance optimization on computing facilities -Running your software in parallel for faster processing -Machine learning, big data analysis, visualization -Training -consolidation of research communities in the Netherlands -multidisciplinary approach -collaboration local and national research infrastructures -Network-connectivity -Lightpath: very quickly send and receive data; direct connection shielded from the Internet; extra secure -security, trust & identity -Integrating your virtual infrastructure into your work processes -Science Collaboration Zone for authorization & authentication -International collaboration -Strong international orientation: high level of participation in European research programs -Accessing the Grid for data processing worldwide -Data storage and sharing services, long term archiving, - -FAIR datamanagement, datastewardship -Methods for approaching your data -The exact design of your data storage system -How to organise your data infrastructure -Meeting the FAIR data principles -public-private cooperation -datahubs, strategic partnerships with private sector *Regular Compute Call via NWO (duration 2 yrs, start within 2 months, combination of SURF compute services allowed. Cartesius supercomputer >500,000 core hours, Lisa Compute cluster >100,000 core hours, HPC Cloud >50,000 core hours, Grid >500,000 core hours(max data 200TB disk/300tb tape) *Compute access via SURF (Cartesius supercomputer <500,000 core hours, Lisa Cluster <100,000, Grid Data processing <500,000, HPC Cloud <50,000). Max 1 application per research project per year. More info about services: SURF.nl/research or via support4research@surf.nl

22 THANK YOU! SURF Open Innovation Lab 22 Driving innovationtogether

23 Twi-NL Project of Meertens Institute (previously NL-eSC and Radboud university) Collection of Dutch tweets for scientific research About 40% of all Dutch tweets Used by CBS and others

24 The Green Village Innovation lab at Delft Multiple projects, lots of sensor data Many projects share (streaming) data SURFsara offers platform based on Apache Kafka

25 Project MinE: ALS disease DNA samples 2 million gigabytes data 8 million compute hours Scalability of application Data analysis expertise

26 Project MinE is a large-scale research initiative devoted to discovering the genetic cause of ALS Data from 20 participating countries Data upload to SURFsara Data analysis at SURFsara Plan to map the full DNA profiles of at least 15,000 people with ALS Compare these to DNA profiles of 7,500 control subjects 46% DNA profiles collected so far Data is uploaded to SURFsara stored on Tape (dcache) Data uploaded through third party (e.g., illumina) or directly by the participating country/contributing institute More than 1PB of data already uploded Data Analysis is run on Grid (Gina and Nikhef) Users from various countries can run the analysis themselves More than 4 million core hours of compute already performed

27 Our support model for the project A ProjectMinE dedicated user interface is made available with: * All software tools to submit jobs to grid (e.g., grid middleware) * Grid Storage clients (incl. setting up of Globus connect personal endpoint) * Read only NFS mount of dcache to access data On the dcache storage system data directories have ACLs so that * Only specified users can access the data in each country s directory * Users can have their own private directories For analysis on grid * RCauth proxy authentication for all users * Access to long queue (96 hours walltime) * Dedicated shared softdrive space for software installation * Individual space on softdrive for their own pipelines * Access to pilot job frameworks (PicaS)

28 CERN Openlab Dell EMC COLLABORATIONS UNDER IPCC GENCI/CINES/INRIA Radboud University Medical Center Max Planck Institute Netherlands Cancer Institute (NKI) 28

29 29

30 Dell-SURFsara-Intel CheXNet CheXNet (from Stanford) is a model for identifying thoracic pathologies from the NIH ChestXray14 dataset DenseNet121 topology Pretrained on ImageNet Dataset contains 112K images Multicategory / Multilabel Unbalanced 30

31 ResNet-50 and beyond ResNet-50 provides great scalability and improved results ResNet-59, with 896x896 input improves accuracy further It takes 50 minutes to train ResNet-59 on ChestXRay-14 using 240 nodes for higher accuracy Challenge encountered: Large memory usage for largeinput models

32 Exploring scalable, accurate AI radiology models High computational and memory requirements! DenseNet-121, 224x224 AmoebaNet-D architecture ResNet-50, 224x224 Mean AUROC ResNet-59, 896x896 AmoebaNet AmoebaNet (4,256), 299x299 (2,512), 480x480 ResNet50 ResNet59 AmoebaNet (4,256) No. parameters 22M 90M 80M 168M AmoebaNet (2,512) Input size 224x x x x480 Training throughput [img/s/node] Memory consumption/ batch of 64 images [GB] Approximate training time on 256 nodes [minutes] Top-1 accuracy on ImageNet-1K % 78.1% 79.9% 80.9%

33 33

34 World flora classification Encountered difficulties - Protobuf library is limited to 2GB files: - Impossible to serialize ResNet-50 model with around 300K classes - Fully connected layer gets very large (~2GB) - Communication bottlenecks at large node counts - Dataset is very large (1.5TB, 11.5M images) and poses problems to Lustre filesystems Used CEA Irene cluster (1600 Skylake 2S nodes) 34

35 294K classes 294K classes solutions Protobuf 2GB files - dimensionality reduction trick ResNet-50 Network size: 2.3GB ResNet-50 Network size: 1.8GB No striping 47.5 img/s (23.7 img/s/node) Improving Lustre performance 2 nodes 32 nodes Scaling efficiency 303 img/s (9.5 img/s/node) 40.1% Lustre striping (SC: 64,SS: 32M) 47.5 img/s (23.7 img/s/node) 688 img/s (21.5 img/s/node) 90.7% 35

36 results (Skylake) Top1 accuracy [%] Top5 accuracy [%] Training time Scalability on Irene Irene: 512 skylake nodes - ResNet-50 Irene: 1320 skylake nodes - ResNet-50 Irene: 1024 skylake nodes - ResNet-50 - Collapsed Ensemble minutes/epoch minutes/epoch minutes/epoch Huge improvements in model accuracy! images/second Ideal 36