April 15 th, 2011 CC-IN2P3: A High Performance Data Center for Research Toward a partnership with DELL Dominique Boutigny
Agenda Welcome Introduction to CC-IN2P3 Visit of the computer room Lunch Discussion about future partnership Discussion about a possible common event in September 2
CC-IN2P3 within the French scientific framework Main scientific organization in France Dedicated Computing Center National Institute for Nuclear Physics and Particle Physics CC-IN2P3 federates the main computing resources For : High energy physics Nuclear physics Astroparticle physics + some opening to other sciences Manpower: 82 people Budget: 10-12 M with salaries Irfu Atomic Energy Commission 3
A diversity of scientific applications CC-IN2P3 is part of a worldwide network of datacenters for High Energy Physics 4
LHC = Large Hadron Collider The LHC is built at CERN in a 27 km tunnel, 100 m under the Geneva region 2 proton beams circulating in opposite directions Collisions in 4 experimental areas 10 years of construction Cost of the accelerator~3 B More than 900 superconducting magnet Several 1000 physicists and engineers involved
The Large Hadron Collider (LHC): the today s challenge ATLAS 4 experiments with a level of complexity never reached before CMS 10 billions collisions recorded each year Up to 15 PB/year of raw data 6
7
A global grid infrastructure CC-IN2P3 Annecy Nantes CC-IN2P3 FZK TRIUMF RAL CC-IN2P3 T1 (11) Grenoble T3 (many) ASCC CNAF Île de France Lyon Strasbourg Brookhaven NIKHEF T0 NDGF PIC Fermilab Clermont T2 (~60) Marseille 8
9
Grid activities CC-IN2P3 is the main node of a French multidisciplinary grid (France Grille) which is part of the European infrastructure EGI Operation Training Distributed data management Support Interoperability tools and portals based on standards Evolution toward the cloud world Regional initiative 10
CC-IN2P3 current status Today, computing power at CC-IN2P3 is dominated by LHC But several other experiments (~40) relies on CC-IN2P3 for their computing 12 000 cores 10 PB of disk storage 4 robotic silos (up to 40 Pbytes capacity) 11
CC-IN2P3 specificities Very complex infrastructure with a strong specificity on data management, data movement, data mining We start to see limitations related to multicore architecture strong impact on CPU efficiency We need to evaluate new architecture and to propose new solutions At the moment we are leading this effort in the high energy physics community Partnership with DELL 12
Infrastructure evolution The very fast increase of the computing equipments installed at CC-IN2P3 created a huge load on the infrastructure 13
The infrastructure is at its limit A lot of money invested since 6 years But not enough to fulfill our long term scientific commitments 14
Build a new datacenter The design parameters used to define the size of the new datacenter has been the following: Serve ~40 experiments with "standards needs" Fulfill LHC computing commitments and provide first class analysis capability Expect a very significant growth of the Astroparticle community needs: LSST, EUCLID) Add some capacity for network, services, etc. ~ 80 000 today s cores 15
New datacenter Assume Moore law is still valid Extrapolate up to 2019 End up with: indicated power is for computing only power for cooling has to be added 2011 50 racks 600 kw 2015 125 racks 1.5 MW 2019 216 racks 3.2 MW On top of the existing computer room (1 MW) Due to budget constraints the new computer room will start with a limited power (600 kw) Modular design Easily scalable Chilled water and electricity distribution has been designed for the 2019 target value Equipment: transformers, chillers, UPS etc will be added later 16
The new datacenter 17
Computer room in the 2 nd floor (850 m 2 ) All the cooling equipment is on the roof Connection between the 2 computer rooms + elevator Electrical equipment in the basement 18
19
New datacenter The new computer room is available and first computers are being installed An upgrade plan do exist to bring it to full power by 2019 (4.2 MW total capacity for computing equipment) Plenty of room, cooling and power CC-IN2P3 is the only datacenter in the HEP field with such a perspective Huge possibilities for worldwide scientific projects 20
Theoretical physics Lattice QCD Lattice Quantum ChromoDynamics is an area of theoretical physics that needs intensive computation The scientific challenge is huge as many experimental results cannot be interpreted without LQCD inputs Requires massively parallel computation with high speed interconnection Initially the IBM Blue Gene architecture as been developed by former LQCD physicist in Watson labs LQCD need dedicated petaflop computers GPU are very good candidates to build a dedicated LQCD computer at acceptable cost Partnership with DELL 21