High Performance Datacenter Networks

Transcription

1 M & C Morgan & Claypool Publishers High Performance Datacenter Networks Architectures, Algorithms, and Opportunity Dennis Abts John Kim SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE Mark D. Hill, Series Editor

2 High Performance Datacenter Networks Architectures, Algorithms, and Opportunities

3 Synthesis Lectures on Computer Architecture Editor Mark D. Hill, University of Wisconsin Synthesis Lectures on Computer Architecture publishes 50- to 100-page publications on topics pertaining to the science and art of designing, analyzing, selecting and interconnecting hardware components to create computers that meet functional, performance and cost goals. The scope will largely follow the purview of premier computer architecture conferences, such as ISCA, HPCA, MICRO, and ASPLOS. High Performance Datacenter Networks: Architectures, Algorithms, and Opportunities Dennis Abts and John Kim 2011 Quantum Computing for Architects, Second Edition Tzvetan Metodi, Fred Chong, and Arvin Faruque 2011 Processor Microarchitecture: An Implementation Perspective Antonio González, Fernando Latorre, and Grigorios Magklis 2010 Transactional Memory, 2nd edition Tim Harris, James Larus, and Ravi Rajwar 2010 Computer Architecture Performance Evaluation Methods Lieven Eeckhout 2010 Introduction to Reconfigurable Supercomputing Marco Lanzagorta, Stephen Bique, and Robert Rosenberg 2009 On-Chip Networks Natalie Enright Jerger and Li-Shiuan Peh 2009

4 The Memory System: You Can t Avoid It, You Can t Ignore It, You Can t Fake It Bruce Jacob 2009 iii Fault Tolerant Computer Architecture Daniel J. Sorin 2009 The Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines free access Luiz André Barroso and Urs Hölzle 2009 Computer Architecture Techniques for Power-Efficiency Stefanos Kaxiras and Margaret Martonosi 2008 Chip Multiprocessor Architecture: Techniques to Improve Throughput and Latency Kunle Olukotun, Lance Hammond, and James Laudon 2007 Transactional Memory James R. Larus and Ravi Rajwar 2006 Quantum Computing for Computer Architects Tzvetan S. Metodi and Frederic T. Chong 2006

5 Copyright 2011 by Morgan & Claypool All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means electronic, mechanical, photocopy, recording, or any other except for brief quotations in printed reviews, without the prior permission of the publisher. High Performance Datacenter Networks: Architectures, Algorithms, and Opportunities Dennis Abts and John Kim ISBN: ISBN: paperback ebook DOI /S00341ED1V01Y201103CAC014 A Publication in the Morgan & Claypool Publishers series SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE Lecture #14 Series Editor: Mark D. Hill, University of Wisconsin Series ISSN Synthesis Lectures on Computer Architecture Print Electronic

6 High Performance Datacenter Networks Architectures, Algorithms, and Opportunities Dennis Abts Google Inc. John Kim Korea Advanced Institute of Science and Technology (KAIST) SYNTHESIS LECTURES ON COMPUTER ARCHITECTURE #14 & M C Morgan & claypool publishers

7 ABSTRACT Datacenter networks provide the communication substrate for large parallel computer systems that form the ecosystem for high performance computing (HPC) systems and modern Internet applications. The design of new datacenter networks is motivated by an array of applications ranging from communication intensive climatology, complex material simulations and molecular dynamics to such Internet applications asweb search, language translation, collaborative Internet applications, streaming video and voice-over-ip. For both Supercomputing and Cloud Computing the network enables distributed applications to communicate and interoperate in an orchestrated and efficient way. This book describes the design and engineering tradeoffs of datacenter networks. It describes interconnection networks from topology and network architecture to routing algorithms, and presents opportunities for taking advantage of the emerging technology trends that are influencing router microarchitecture. With the emergence of many-core processor chips, it is evident that we will also need many-port routing chips to provide a bandwidth-rich network to avoid the performance limiting effects of Amdahl s Law. We provide an overview of conventional topologies and their routing algorithms and show how technology, signaling rates and cost-effective optics are motivating new network topologies that scale up to millions of hosts. The book also provides detailed case studies of two high performance parallel computer systems and their networks. KEYWORDS network architecture and design, topology, interconnection networks, fiber optics, parallel computer architecture, system design

8 vii Contents Preface... xi Acknowledgments...xiii Note to the Reader...xv 1 Introduction From Supercomputing to Cloud Computing Beowulf: The Cluster is Born Overview of Parallel Programming Models Putting it all together Quality of Service (QoS) requirements Flowcontrol Lossy flow control Lossless flow control The rise of ethernet Summary Background Interconnection networks Technology trends Topology, Routing and Flow Control Communication Stack Topology Basics Introduction TypesofNetworks Mesh, Torus, and Hypercubes Node identifiers k-ary n-cube tradeoffs... 22

9 viii 4 High-Radix Topologies Towards High-radix Topologies Technology Drivers Pin Bandwidth Economical Optical Signaling High-Radix Topology High-Dimension Hypercube, Mesh, Torus Butterfly High-Radix Folded-Clos Flattened Butterfly Dragonfly HyperX Routing Routing Basics Objectives of a Routing Algorithm Minimal Routing Deterministic Routing Oblivious Routing Non-minimal Routing Valiant s algorithm (VAL) Universal Global Adaptive Load-Balancing (UGAL) Progressive Adaptive Routing (PAR) Dimensionally-Adaptive, Load-balanced (DAL) Routing Indirect Adaptive Routing Routing Algorithm Examples Example 1: Folded-Clos Example 2: Flattened Butterfly Example 3: Dragonfly Scalable Switch Microarchitecture Router Microarchitecture Basics Scaling baseline microarchitecture to high radix Fully Buffered Crossbar Hierarchical Crossbar Architecture Examples of High-Radix Routers... 57

10 6.5.1 Cray YARC Router Mellanox InfiniScale IV System Packaging Packaging hierarchy Power delivery and cooling Topology and Packaging Locality Case Studies Cray BlackWidow Multiprocessor BlackWidow Node Organization High-radix Folded-Clos Network System Packaging High-radix Fat-tree Packet Format Network Layer Flow Control Data-link Layer Protocol Serializer/Deserializer Cray XT Multiprocessor D torus Routing Flow Control SeaStar Router Microarchitecture Summary Closing Remarks Programming models Wire protocols Opportunities Bibliography Authors Biographies ix