Portfolio of Services. NATIONAL COMPUTATIONAL Portfolio INFRASTRUCTURE

Size: px

Start display at page:

Download "Portfolio of Services. NATIONAL COMPUTATIONAL Portfolio INFRASTRUCTURE"

Justin Ryan
5 years ago
Views:

1 Portfolio of Services NATIONAL COMPUTATIONAL Portfolio INFRASTRUCTURE of Services 1

2 National Computational Infrastructure The Australian National University 143 Ward Road Acton ACT 2601 T E enquiries@nci.org.au W nci.org.au National Computational NCINationalFacility National Computational Infrastructure July 2015 This work is copyright. Apart from any use permitted under the Copyright Act 1968, all rights are reserved. Requests for authorisation and enquiries concerning the contents of this document should be directed to enquiries@nci.org.au The document is also accessible from the NCI website Produced by National Computational Infrastructure The Australian National University 143 Ward Road Acton ACT 2601 Designed, typeset and printed by CanPrint Communications Pty Ltd.

3 RESEARCH Portfolio of Services

4 NCI provides world-class high-end computing services for Australian research and innovation. We are unique in the world, offering Australia s first petaflop supercomputer, Australia s fastest research cloud and Australia s largest research data repository, tightly integrated with the Southern Hemisphere s fastest filesystems. 2

WELCOME FROM THE DIRECTOR As Australia s most highly integrated, high-performance e-infrastructure environment, NCI offers a tailor-made facility for supporting world-class research and innovation.

5 WELCOME FROM THE DIRECTOR As Australia s most highly integrated, high-performance e-infrastructure environment, NCI offers a tailor-made facility for supporting world-class research and innovation. Our unique integration of Australia s first petaflop supercomputer with the nation s fastest research cloud and the Southern Hemisphere s fastest filesystems, housed beside the nation s largest research data repository, allows NCI to support innovative research that would otherwise not be possible. We cater for the full research spectrum, from fundamental investigation to applied, strategic and industry-driven research. Based on campus at The Australian National University, NCI provides services to three of the nation s science agencies CSIRO, Geoscience Australia, and the Australian Bureau of Meteorology as well as a further 30 Australian universities, five ARC Centres of Excellence, and our industry partners. The diversity of our partner organisations and users demonstrates our ability to accommodate and adapt to a range of requirements, from those of a single researcher based at a university, to a start-up company looking to upscale, or the policy commitments of an entire government agency. This portfolio sets out NCI s base level of services, which can be tailored to suit your needs. Please don t hesitate to contact me if you would like to discuss further the services NCI provides. I look forward to hearing from you. Professor Lindsay Botten Director enquiries@nci.org.au Portfolio of Services 3

6 NCI offers everything Australian research institutions, government agencies and companies need to SIMULATE AND ANALYSE, ACCESS, SHARE AND PUBLISH, and VISUALISE data. HPC Cloud Digital labs Data storage Visualisation 4

7 SIMULATE AND ANALYSE NCI offers a comprehensive platform for high performance simulation and data analysis. Raijin Tenjin Supercomputer High performance cloud 20PB Data Repository Portfolio of Services Digital laboratories 5

8 Raijin 1.2 petaflop peak performance supercomputer Fujitsu Primergy cluster 57,472 cores (Intel Xeon Sandy Bridge, 2.6GHz) in 3,592 nodes Mellanox FDR 56Gb/sec Infiniband full fat tree interconnect at 9TB/sec bandwidth 160TB main memory 10PB working storage Access to the 20PB global high-speed parallel data repository Raijin was specifically designed to handle the highest performance and most highly parallel workloads. At any one time more than 95% of Raijin s compute cores are in action, running up to 3,500 different jobs. Raijin is a robust system, with greater than 99% uptime since installation. A custom monitoring system allows support staff to work with users to optimise their jobs to increase throughput. 6

9 Tenjin 67 teraflop peak performance Dell OpenStack Cloud 1,600 cores (Intel Xeon Sandy Bridge, 2.6GHz) in 200 nodes Mellanox FDR 56Gb/sec Infiniband full fat tree interconnect 25TB main memory 160TB Solid State Disk Access to the 20PB global high-speed parallel data repository Tenjin is NCI s supercomputer-grade cloud, available to our partner organisations. Built on the same hardware foundation as Raijin, Tenjin is the most high-performance research cloud in Australia. Tenjin is specifically designed for data-intensive research, such as genomics and geophysics. It is an ideal environment for research requiring access to the datasets held in the NCI Environmental Research Data Collection. My laboratory is engaged in harnessing groundbreaking technology to sequence large numbers of whole human genomes which could not have been dreamt of a decade ago. NCI has driven our successful genome and exome informatics through providing access to high performance data storage and computation that has allowed us to scale at the rate of data growth in our exploding field. Professor Chris Goodnow, Deputy Director, Garvan Institute of Medical Research Portfolio of Services 7

$Get more done in a fraction of the time CASE STUDY: EVATT HAWKES, UNSW The key to supercomputing is parallelism the ability to split a large compute job into lots of smaller ones that can run in$

10 Get more done in a fraction of the time CASE STUDY: EVATT HAWKES, UNSW The key to supercomputing is parallelism the ability to split a large compute job into lots of smaller ones that can run in parallel, allowing vast amounts of data to be processed in a fraction of the time. ARC Future Fellow Associate Professor Evatt Hawkes runs the largest compute job Raijin has tackled so far. He and his team at UNSW are modelling turbulent combustion to design fuel-efficient engines that emit fewer pollutants. We use about one billion grid points and about 200,000 time steps, with seven variables per grid point and time step, explains Professor Hawkes. A single simulation takes approximately one million hours. If it were run on a single core, it would take more than 100 years. By splitting the computation across thousands of cores simultaneously, the task can be done in just a few weeks. We typically run jobs in the 4,000 to 8,000 core range, and we have scaled this right out to at least 25,000 cores, says Professor Hawkes. The team is using their first-principles models to develop an industry-suitable version, says Professor Hawkes. Industry need models that they can turn around in 100 different configurations within a week, he says. That s what we re aiming for. Expert Support A team of PhD-qualified scientists is on hand during business hours to help you get the most out of your compute hours. The NCI Help Desk provides training and specialised advice on parallel programming, algorithms, and other aspects of high performance computing and data-intensive services. NCI also employs a team of specialists who work with our major partners to optimise large-scale models. We re leading the world in our area of research and that is in large part due to the professionalism of the very dedicated team at NCI. Dr Wenju Cai, Research Director, Ocean & Atmosphere, CSIRO 8

Scaling massive models CASE STUDY: BOM, FUJITSU, CSIRO, ARCCSS As part of a collaboration with Fujitsu, the NCI code optimisation (BoM) has been working closely with NCI to

Centre of Excellence for Climate System Science, to increase We re trialling our next-generation models on Raijin in preparation for them becoming operational in a year or

Every day at NCI we run Australian Community Climate and Earth System Simulator (ACCESS) numerical weather prediction the efficiency of the ACCESS component models.

Already collaboration on asynchronous input/output processing has realised a time saving of up to 20% using 2,000 cores.

11 Scaling massive models CASE STUDY: BOM, FUJITSU, CSIRO, ARCCSS As part of a collaboration with Fujitsu, the NCI code optimisation (BoM) has been working closely with NCI to test and refine their team have been working with BoM, Fujitsu, CSIRO and the ARC next-generation national weather prediction models. Centre of Excellence for Climate System Science, to increase We re trialling our next-generation models on Raijin in preparation for them becoming operational in a year or two, explains Dr Michael Naughton from BoM. Every day at NCI we run Australian Community Climate and Earth System Simulator (ACCESS) numerical weather prediction the efficiency of the ACCESS component models. The faster they run, the larger the model that can be supported, and the greater the detail and accuracy provided. Already collaboration on asynchronous input/output processing has realised a time saving of up to 20% using 2,000 cores. suites, which support research and evaluation of prototype The collaborators have worked very effectively together to systems, with a special focus on severe and high-impact realise these improvements. The expertise of the NCI team weather events in the Australian region, he says. has been essential to solving a number of critical software Portfolio of Services For the past two years, the Australian Bureau of Meteorology problems, says Dr Naughton. 9

Digital Labs One-stop computing shops for the non-coding scientist Drawing on the power of Raijin, Tenjin, and our 10 petabyte data collection, NCI has created a number of custom field-specific

12 Digital Labs One-stop computing shops for the non-coding scientist Drawing on the power of Raijin, Tenjin, and our 10 petabyte data collection, NCI has created a number of custom field-specific online research environments, called digital laboratories. These include the Australian Government NeCTAR-funded Climate & Weather Science Laboratory, the All Sky Virtual Observatory and the Virtual Geophysics Laboratory. Digital labs bring together in a single place everything you need for research computing: vast data libraries, high-performance computing resources, and analysis and visualisation tools. These one-stop-shop online labs provide easy access to the power of Raijin and Tenjin. As part of the Southern Sky Survey we re producing torrents of data as much as 225 terabytes in all which is why we need the phenomenal processing power of the NCI supercomputer. Nobel Laureate Professor Brian Schmidt, ANU, principal investigator of the key dataset served by the All Sky Virtual Observatory. Paving the way for future digital labs We have developed an innovative configuration system, using Puppet bundles, that provides the building blocks for constructing new, sustainable digital laboratory environments at NCI. This means more time can be spent developing customised tools and applications specific to each laboratory s needs. If you are interested in working with us to develop a digital laboratory, contact enquiries@nci.org.au 10

13 ACCESS, SHARE AND PUBLISH The NCI National Research Data Collection NCI is world leading in its integration of a vast, publicly funded data repository with petascale computing facilities. The result is the ability to query, analyse and publish datasets in situ, reducing the need for time-consuming data transfer. The NCI National Research Data Collection holds more than 10 petabytes of nationally and internationally significant research data, in a wide range of disciplines. Through an extensive catalogue, our users can easily query multiple collections simultaneously to select and analyse data subsets, thus advancing interdisciplinary research. Browse the Collection at nci.org.au/data-collections Outer Space Stratosphere Atmosphere Environment Crust Upper Mantle Mantle Core Portfolio of Services Data from every part of the Universe, all in one searchable database 11

14 Data Management NCI has a team of expert data managers who can work with you to catalogue and curate your data collection for optimal accessibility, NCI is no longer a research infrastructure facility that stands on its own; it has become integrated into Geoscience Australia s business-as-usual model. analysis and data sharing. If you represent a university, company or institution that generates, Dr Clinton Foster, Chief Scientist, Geoscience Australia owns or requires access to big data, contact us at DATA COLLEC ED NS TIO CUR AT enquiries@nci.org.au to find out how we can help you. n DISPARATE DATA COLLECTIONS Government agencies Universities Research institutes 12 em ac ma rea ti o n D ata ag ent p t la n - M e ad at PUBLISHED DATA COLLECTIONS DOI minting Data services

Unlocking the Landsat Archive The result is the Australian Geoscience Data Cube, an interactive 3D CASE STUDY: THE AUSTRALIAN GEOSCIENCE DATA CUBE continent at a resolution of 25 square kilometres.

The project s legacy was hundreds of terabytes of data that could provide vital insights into Australia s future climate and resources.

the Australian landscape, including bushfire and flood paths, has changed over time and predict what might change in the future.

15 Unlocking the Landsat Archive The result is the Australian Geoscience Data Cube, an interactive 3D CASE STUDY: THE AUSTRALIAN GEOSCIENCE DATA CUBE continent at a resolution of 25 square kilometres. For three decades the United States Geological Survey s Landsat satellites captured breathtaking images of Earth from space. The project s legacy was hundreds of terabytes of data that could provide vital insights into Australia s future climate and resources. platform of organised, calibrated image tiles that covers the entire Tiles showing the same area at different time points are stacked on top of each other, allowing researchers to visually track how the Australian landscape, including bushfire and flood paths, has changed over time and predict what might change in the future. For example, using the Data Cube we can look at how often surface However, computing and data facilities around the world lacked the water has been seen from satellites at locations across Australia, tools to process and analyse such a vast dataset in a useful way. says Dr Lewis. With most scientific datasets, a limiting factor is how many people Manually, this would take about 8 years to do. Using NCI s high- can look at, analyse, and extract information from the data, says Dr performance computing facilities we can now run the analysis in a Adam Lewis at Geoscience Australia. few hours. We wanted to begin applying the Landsat Archive to big problems, such as the National Flood Risk Information project and the MurrayDarling Basin Vegetation Assessment. Once we knew this is what we wanted to do, it became clear we needed to organise the data differently. Since 2013 Geoscience Australia has worked closely with NCI hundreds of thousands of images of Australia. Portfolio of Services to develop a way to unlock the potential of the Landsat Archive s 13

Data Connectivity at NCI NCI s filesystems are the fastest in the Southern Hemisphere. Both Raijin and Tenjin access the filesystems at up to 140 Gigabytes/sec.

16 Data Connectivity at NCI NCI s filesystems are the fastest in the Southern Hemisphere. Both Raijin and Tenjin access the filesystems at up to 140 Gigabytes/sec. We can also ingest data from external organisations across Australia and the globe at high speed. FILESYSTEM CONFIGURATION Internet NCI data movers VMware Cloud Raijin login + data movers 10 GigE 56 Gb FDR IB FaBRIC Raijin compute 56 Gb FDR IB FaBRIC Archival tape storage Active project storage Active job storage mass data 12PB up to 140MB/sec /g/data 22.1PB up to 140GB/sec /short 7.6PB up to 150GB/sec Superfast International Data Transfer CASE STUDY: CMIP5 DATASET REPLICATION NCI is working with international data managers to securely and reliably replicate and transfer very large datasets to NCI. These datasets will be used by the Australian research community, including the national science agencies, the university sector and other government departments. So far NCI has ingested almost 1PB of the Coupled Model Intercomparison Project (CMIP) 5 dataset from the Lawrence Livermore National Laboratory in the USA using high speed global networks. The project is paving the way for the next generation of CMIP data, which needs to be available for analysis and synchronised across the globe. The CMIP6 dataset is anticipated to be in excess of 10PB and will underpin the next major activities in international climate research. This undertaking is only possible because of the expertise of NCI staff, who have achieved a very high fraction of the theoretical bandwidth of the global networks. 14

VISUALISE NCI has a team of specialist visualisation programmers the NCI VizLab who work closely with our users to visually investigate research datasets.

The VizLab team are experts in applying visualisation as a research and investigation tool to gain deeper insight into complex datasets.

17 VISUALISE NCI has a team of specialist visualisation programmers the NCI VizLab who work closely with our users to visually investigate research datasets. Scientific visualisation is not just about transforming data into pretty pictures. The VizLab team are experts in applying visualisation as a research and investigation tool to gain deeper insight into complex datasets. The VizLab team have also developed two unique platforms for data visualisation: a user-friendly web-based package called Voluminous, and the more sophisticated Drishti software for more complex datasets. If you represent a university, company or institution that would benefit from scientific visualisation services, contact us at enquiries@nci.org.au Drishti is amazing. We ve looked at this slater a thousand times, yet with a few adjustments, we are still seeing parts we have never seen before. Erica Seccombe, PhD candidate, ANU Winner of the 2015 Paramor Prize for Art and Innovation Portfolio of Services Garden slater Sea urchin 15

18 Marine protists 16

Seeing the unexpected CASE STUDY: ANDY HOGG, ANU & ARC CENTRE OF EXCELLENCE FOR CLIMATE SYSTEM SCIENCE ARC Future Fellow Associate Professor Andy Hogg is at the forefront of global efforts to

The NCI VizLab team have helped Professor Hogg to visualise the vast ocean current models that he and his team run on the NCI supercomputer.

19 Seeing the unexpected CASE STUDY: ANDY HOGG, ANU & ARC CENTRE OF EXCELLENCE FOR CLIMATE SYSTEM SCIENCE ARC Future Fellow Associate Professor Andy Hogg is at the forefront of global efforts to understand the processes that govern the Southern Ocean. The NCI VizLab team have helped Professor Hogg to visualise the vast ocean current models that he and his team run on the NCI supercomputer. 3D visualisation of these large and complex datasets is an essential part of the scientific process to both understand the dynamics in the models, and to communicate that understanding to scientific and public audiences, says Professor Hogg. Working with the NCI VizLab team has enabled me to discover results from my models that would otherwise have been obscured. Portfolio of Services 17

20 HOW WE PROVIDE SERVICES TO YOU DEPENDS ON WHO YOU ARE MERIT-BASED NCMAS NCI is secretariat for the National Computational Merit Allocation Scheme (NCMAS). If you are a researcher at a publicly funded Australian university or research institute you are eligible for NCMAS. A number of special allocations are reserved for Early Career Researchers. Application rounds open in September each year. ncmas.nci.org.au Flagship Scheme The Flagship Scheme currently supports a number of ARC Centres of Excellence and NCRIS capabilities in environment and astronomy science. INDUSTRY SHARE NCI supports a number of industry partners in meeting their Research and Development needs. PARTNER SHARES The majority of NCI s HPC resources are provided through partnership contracts. If you are a researcher or employee at the following organisations, contact your NCI Allocation Manager (see nci.org.au/access for contact information): Current NCI partners: 18

21 IF YOU REPRESENT A UNIVERSITY, COMPANY OR INSTITUTION NOT LISTED ABOVE AND ARE INTERESTED IN WHAT NCI HAS TO OFFER, CONTACT ENQUIRIES@ NCI.ORG.AU Portfolio of Services 19

22 20

23 FEEDBACK FROM THE NCI USER SURVEY 2014 For us, NCI is about as important as a parachute is to someone jumping out of a plane. NCI has been crucial to the success of my research, which has now moved into the commercialisation stage. We can get our workhorse calculations done without NCI but serious science requires NCI s facilities. National Computational Infrastructure The Australian National University 143 Ward Road Acton ACT 2601 T E enquiries@nci.org.au W nci.org.au National Computational NCINationalFacility Professor Lindsay Botten Director director@nci.org.au

Clare Richards, Benjamin Evans, Kate Snow, Chris Allen, Jingbo Wang, Kelsey A Druken, Sean Pringle, Jon Smillie and Matt Nethery. nci.org.

The important role of HPC and data-intensive infrastructure facilities in supporting a diversity of Virtual Research Environments (VREs): working with Climate Clare Richards, Benjamin Evans, Kate Snow,