Darren D Souza ICT Manager, UQ Diamantina Institute
Who is UQDI? The University of Queensland Diamantina Institute was established as UQ s sixth research institute in 2007 The Institute was formed through the amalgamation of UQ's Centre for Immunology and Cancer Research (CICR) and the Centre for Diabetes and Endocrinology Research (CDER). UQDI has grown to include over 200 researchers, students and support staff Supports world-changing discoveries such as the world s first cervical cancer vaccine.
Where we are.
TRI PAH PACE
What we do.. Researchers focus on turning their scientific discoveries into better treatments for diseases including a variety of cancers, osteoporosis, arthritis and other autoimmune diseases. Research Clusters: Autoimmunity Cancer Biology Epithelial Cancer Melanoma Bioinformatics, statistical genetics, flow cytometry and imaging (live cell microscopes and PET-CT scanner) are particularly compute and data intensive By the end of the year we will have 4 large Next Generation Genome Sequencers, cost up to $750,000 each and can each consume up to $500,000 in reagents a year. Huge datasets, each sequence run can generate hundreds of gigabytes of data.
ICT Staff Small team of four IT staff Manager Systems Specialist Senior Systems Administrator Computing Support Officer
ICT Systems Spiceworks Helpdesk system (freeware) Use WPKG (opensource) application deployment on Windows Using Apple Remote Desktop for our small Apple fleet Putting in place tools for remote assistance, server/service monitoring Approximately 200 desktops Mainly Windows/Dell Around 15 Linux desktops Less than 20 Macs (but growing )
ICT Systems cont d Hitachi AMS 2500 SAN for sequencing data ~80TB SGI DMF tiered storage system, ~10TB online, 86TB offline Spectra Logic T120, 120-slot LTO-4 tape library 2-Node Vmware Cluster with Dell iscsi SAN ~20TB raw Small HPC Cluster with 312 cores & 2.3TB of RAM
Who is TRI? UQ Faculty of Health Sciences
What s the TRI? UQDI moved into the Translational Research Institute (TRI) in late 2012 TRI is an Australian first and will see Australia play a leading role in international efforts to address infectious disease, cancer and diabetes. The $195-million seven-storey TRI building comprises four floors of laboratory research plus facilities for research support, administration and teaching it has 32,000 m 2 of floor space (nearly 8 acres) TRI building will house up to 900 researchers and support staff. Facilities include PC2 and PC3 laboratories, a vivarium, a Cell Therapies clean room facility, staff support spaces, a large lecture theatre, advanced education facilities and datacenter Additionally a bio-pharmaceutical manufacturing facility is located in an adjoining building and there will be clinical spaces at the Mater and Princess Alexandra Hospitals
Before (2011)
After (2013)
Aims To enable world-class research by significantly expanding ICT resources All partners have equality of access to resources Assist in collaboration between stakeholders Alleviate current capacity constraints
ICT Resources All partners to be able to share resources easily HPC Facility Large Data Storage Capability Excellent Communications Advanced Datacenter
Networks Eduroam wireless to all building areas (voice-grade) All desks have at least two network points The building is flood patched Utilises Cisco Identity Services Engine (ISE) Discrete & shared networks including shared physical infrastructure with QHealth
Communications & Networks IP-based phones leveraging the UQ telephone system Free extension-based dialling back to all stakeholders Future plans to push out Wireless DECT to entire building using existing APs
Communications Cont.. All main meeting rooms have Cisco video conference capabilities
Cisco Telepresence Suite on Level 7
TRI Datacenter Capacity for 110 racks of computer equipment, will house TRI stakeholder IT equipment and also likely to host some UQ,QUT and RDSI & Nectar systems Located on Level 7 Gas fire suppression, UPS backup, redundant power,cooling and backup generator Aim to meet Tier-2 Datacenter Standards Max useable power per rack averages 6 kw/rack. Racks APC AR3347 48 RU high 1200mm deep 750 mm wide PDU s: APC AP8881 (2 per rack) 3 phase input 36 x 10 Amp IEC, output connection 6 x 15 Amp IEC, output connection 16 Amps per phase, output Floor loading is around1200 kg/sqm
TRI Storage & Compute Diagram removed for security, please contact if you would like more information.
Compute HPC Common Nodes 35x SGI H2106 servers each with: 64 Cores AMD 6278 256GB memory Total 2,240 cores, 8.9TB RAM HPC High Memory System 1x SGI UV2000 server 256 Intel Xeon cores Single 4TB RAM node or 2 X 2TB Nodes
Storage SGI DMF ~1.1PB disk, 3PB tape Aim to have an off-site tape copy as well Lustre ~200TB The benchmark results show an average write of 4481 MB/sec and an average read of 13277 MB/sec. Bandwidth calculated at: 8.879GB/sec. SGI NAS ZFS (based on Nexenta Platform) ~200TB Useable Oracle SL3500 tape Library 6 X 10KC Drives 600 cartridges available (5TB uncompressed each) 1600 slots capacity Total TRI Storage 4.5PB+
Interconnects & Networking Infiniband FDR 56Gb/s Fabric 2 X Mellanox SX6036 36-ports 7 X Mellanox SX6025 36-ports 8Gb/s Fibre Chanel Fabric 2 X Brocade 5100 24-ports, mainly for tape library and storage 10GbE Converged Fabric TRI Core Nexus 7K Routers with Nexus 5000 EoR and Nexus 2000 ToR switches, fully redundant Separate gigabit management switches