Domain-Specific Accelerators: What They Are & Why They Should Matter to Cloud & Network Providers

Transcription

1 Independent market research and cmpetitive analysis f next-generatin business and technlgy slutins fr service prviders and vendrs Dmain-Specific Acceleratrs: What They Are & Why They Shuld Matter t Clud & Netwrk Prviders A Heavy Reading white paper prduced fr Netrnme AUTHOR: SIMON STANLEY, ANALYST AT LARGE, HEAVY READING

2 EXECUTIVE SUMMARY The bandwidth demands n server prcessrs have significantly utgrwn the increases in central prcessing unit (CPU) perfrmance ver the past few years. This is already having an impact n clud and netwrk service prviders, which are installing a grwing number f servers and data centers t meet the demand frm custmers. Virtualizatin is key t delivering many f these services but als increases the wrklad n server CPUs. Server perfrmance can be a critical factr in prviding cst-effective clud and netwrking services. Dmain-specific acceleratrs such as SmartNICs, and machine learning and inference cprcessrs can fflad prcessing frm CPU cres, significantly increasing applicatin perfrmance and releasing additinal CPU cres fr ther revenue earning wrklads. T deliver the best benefits these acceleratrs shuld integrate best-f-breed cmpnents such as prcessrs, hardware engines, memry and I/O peripherals. Dmain-specific acceleratrs are prgrammable applicatin-specific integrated circuits (ASICs) usually implemented as a mnlithic system-n-chip (SC). The shift t smaller silicn gemetries has enabled larger SC devices with big caches and many cres but each new silicn technlgy generatin requires huge investment and results in significantly higher develpment and die csts. Develping dmain-specific acceleratrs fr many applicatins is nt cmmercially attractive due t the high develpment csts fr each device. Chiplets and multi-chip mdules (MCM) with multi-dies n a single substrate prvide an attractive alternative t mnlithic designs with a single die. Chiplets have been widely used where an integrated device uses cmpnents develped using different silicn technlgy and are increasingly being used t prvide mdular slutins. Ntable uses f chiplets have been High Bandwidth Memry (HBM), mbile phne prcessrs and the AMD EPYC prcessrs. The use f chiplets ffers a cst-effective apprach t dmain-specific acceleratrs that shuld enable a wider range f slutins t becme available. Mst chiplets s far have been used in a clsed develpment envirnment, where all the chiplets and the integrated system have been defined by a single cmpany r grup wrking n a single design. An pen develpment envirnment where chiplets frm different cmpanies have cmmn interfaces and supprt a cmmn prtcl stack will enable the wider use f chiplets. The Open Dmain-Specific Architecture (ODSA) Wrkgrup was launched in Octber 2018 by seven cmpanies t develp an pen architecture and related specificatins fr develping chiplets. The grup is wrking n a reference design and a cmplete prtcl stack t supprt dmain-specific acceleratr develpment. This white paper is based n inputs frm the ODSA Wrkgrup and individual member cmpanies including Achrnix, Avera Semicnductr, Aquantia, ESnet, Kandu, Netrnme, NXP, SamTec, Sarcina and SiFive. WHY USE OPEN DOMAIN-SPECIFIC ACCELERATORS? Key Market Areas General-purpse CPUs alne cannt sustain the demands f server wrklads in many key markets. The demise f Mre's Law (shrinking length f transistrs and number f transistrs bught per dllar) nly exacerbates the situatin. Silicn devices based n dmainspecific architectures have cme t the rescue. Key applicatins fr these dmain-specific HEAVY READING NETRONOME OPEN DOMAIN-SPECIFIC ACCELERATORS 2

3 acceleratrs are netwrking and security cprcessrs as used in SmartNICs, and machine learning and inferencing cprcessrs as used in PCIe adapters r appliances. Other applicatins include cryptcurrency/blckchain, database acceleratin and streaming data prcessing fr the Internet f Things (IT). Dmain-specific architectures, as the name implies, are tailred t a class f wrklads that share a cmmn characteristic. The devices are prgrammable, nt hardwired as are traditinal ASICs. Dmain-specific architectures integrate applicatin and deplyment-aware functins and supprt dmain-specific languages fr ease f use. Key attributes f a dmainspecific architecture are parallelized data prcessing, functin-specific lgic, applicatinaware data management and cntrl. Tw dmain-specific silicn-based examples are shwn in Figure 1, namely Ggle's Tensr Prcessing Unit (TPU) fr machine learning and AI, and Netrnme's Netwrk Flw Prcessr (NFP) fr netwrking and security. The charts shw significantly better perfrmance/watt fr these slutins relative t CPUs and FPGAs. Figure 1: Dmain-Specific Silicn Delivers Higher Perfrmance/Watt Surces: "An in-depth lk at Ggle's first Tensr Prcessing Unit (TPU)," Ggle Clud, May 2017 (left); Netrnme, based n internal benchmarks and industry reprts related t Xen CPUs and Arris FPGAs (right) HEAVY READING NETRONOME OPEN DOMAIN-SPECIFIC ACCELERATORS 3

4 Silicn Develpment Challenges Dmain-specific acceleratrs require a specific cmbinatin f prcessing cres and hardware engines. As can be seen in Figure 2, as technlgy becmes mre cmplex, the cst fr SC develpment is rising expnentially frm nde t nde. The ROI revenue required t embark n silicn develpment at smaller, mre advanced prcess ndes is astunding. Figure 2: Skyrcketing Csts f Silicn Develpment Surce: Keith Flamm, "Measuring Mre's Law; Evidence frm Price, Cst & Quality Indices," Glbal Fundries, semiengineering.cm ("Hw much will that chip cst?") and Netrnme Due t the significant cst in develping ASICs in advanced ndes, nly extrardinarily well funded prjects can be brught t cmpletin. Many acceleratr develpers, with applicatins that serve a limited market, are simply unable t justify the expense even thugh they can prvide a significant cst and pwer advantage versus using prcessrs and FPGAs t implement the needed functin. In the past, a designer culd ften cmbine tw ASICs in their system int a single mnlithic design in the next prcess nde, with a ptential increase in frequency as well. As multiple parts are cmbined int a single device interface pwer cnsumptin is reduced, an additinal benefit beynd the active pwer imprvement f mving t a smaller gemetry. While this has area and pwer benefits ver having multiple devices, it ften results in ASIC designs being larger than they actually need t be as they are designed with a superset f needed functins fr a variety f applicatins. Given the increasing investments required t build an ASIC, alng with decreasing cst benefits f ding s, it seems that an bvius chice wuld be t build very large mnlithic die in lder technlgy ndes. Unfrtunately, this has cst implicatins driven by defectivity f large die, technical limitatins due t the limit f the reticle used in lithgraphy tls and limits f reliable large die attached t laminates. HEAVY READING NETRONOME OPEN DOMAIN-SPECIFIC ACCELERATORS 4

5 Benefits fr End Users and Develpers The use f pen dmain-specific acceleratrs ffers a third path with hetergeneus integratin f multiple cmpnent die r chiplets using lwer pwer interfaces such as USR, Bunch f Wires (BW) r emerging 112G SiP standards. By layering a cmmn prtcl ver different interfaces, a cmmn "building blck"-based apprach can be leveraged t create systems n a substrate by simply varying the Bill f Materials (BM) fr an MCM. This chiplet apprach can prduce many f the benefits f integratin at a fractin f the develpment cst. The integrated system n a substrate als prvides a majr savings in bard real-estate and ruting layers, significant system csts that can ften dwarf any increased cst t design and integrate the multi-die package. The ecnmics f building silicn using chiplets with smaller die sizes has been established by multiple cmpanies and prducts. High Bandwidth Memry (HBM) is an example f a chiplet-based prduct that uses pen interfaces. AMD, with its successful EPYC CPUs, has shwn hw the use f chiplets with CPU cres can reduce silicn develpment and manufacturing csts f a 32-cre CPU by up t 40 percent. The strategy enables a rapid g-t-market fr ther versins f CPUs with fewer cres. One significant advantage f a chiplet-based apprach is that unlike a mnlithic design, all the cmpnents need nt be prted t a new prcess nde n each shrink. Fr example, interface cmpnent blcks (such as a Lng Reach SerDes tile r Electrical t Optical Interface) may remain in cst-effective ndes t reduce verall investment. As shwn in Figure 3, while nt attaining the same area and pwer advantage f a technlgy shrink, this third path prvides a cnsiderable area and pwer savings ver mnlithic integratin in mre cst-effective ndes by reducing interface area and pwer significantly. While multi-chip systems typically have a higher cst than individual die, the incremental investment can be smewhat ffset by these area and pwer savings. Figure 3: Area fr Multi-Die Integratin vs. Technlgy Shrink Surce: Avera Semicnductr HEAVY READING NETRONOME OPEN DOMAIN-SPECIFIC ACCELERATORS 5

6 KEY FEATURES OF OPEN DOMAIN-SPECIFIC ACCELERATORS Architecture The pen dmain-specific acceleratr architecture being develped in the ODSA Wrkgrup enables chiplet-based silicn design t be cmpsed using best-f-breed cmpnents such as prcessrs, hardware engines, and memry and I/O peripherals using ptimal prcess ndes. The pen architecture will prvide a cmplete stack f cmpnents (knwn gd die, packaging, intercnnect netwrk, sftware integratin stack) that lwers the hardware and sftware csts f develping and deplying dmain-specific acceleratr slutins. By implementing pen specificatins, any vendr's silicn die can becme a building blck that can be utilized in a chiplet-based SC design. Dmain-specific acceleratr silicn design needs t cmprehend bth the develper prgramming mdel fr firmware and hw the acceleratr is integrated int the system-level applicatin data flw and cntrl management. The ODSA architecture shwn in Figure 4 defines a reference multi-die architecture that is derived frm elements cmmn t a wide range f cmmercial and academic acceleratrs. Figure 4: ODSA Reference Multi-Die Architecture Surce: ODSA HEAVY READING NETRONOME OPEN DOMAIN-SPECIFIC ACCELERATORS 6

7 The reference design cnsists f the fllwing chiplets: A netwrk I/O chiplet A RISC CPU chiplet A dmain-specific acceleratin chiplet, which may be implemented as ne r mre instances f: an FPGA; a many-cre RISC prcessr; r dmain-specific lgic A switching and interface chiplet t which all the ther chiplets are cnnected These chiplets are packaged tgether n a cmmn substrate t frm an SC slutin. Key benefits f using pen dmain-specific acceleratrs and chiplets include reduced NRE and time t market, reduced circuit bard area and pwer cnsumptin. By using an pen architecture, such as that being develped by the ODSA Wrkgrup, acceleratr develpers can assemble systems integrating best-f-breed cmpnents frm multiple vendrs. Develpment effrt and investment can therefre be fcused n their wn dmain expertise. Prtcl Stack T present a mnlithic IC-like develpment envirnment, the chiplets in the ODSA implement cmmunicatin agents t supprt inter-die netwrking. The netwrk infrastructure ffers a memry mdel and perfrmance cmparable t a mnlithic architecture. T cmpensate fr the difference between mnlithic and hetergeneus implementatins, the ODSA reference architecture will supprt a cmplete prtcl stack, as shwn in Figure 5. Figure 5: ODSA Prtcl Stack Surce: ODSA HEAVY READING NETRONOME OPEN DOMAIN-SPECIFIC ACCELERATORS 7

8 This stack addresses the fllwing issues: A netwrk layer t mve data arund with three sublayers: PHY layer: Prtcls fr physical inter-die cmmunicatin Link layer: A prtcl fr lgical cmmunicatin between physically cnnected die Ruting layer: A prtcl t rute transactins between devices nt directly cnnected A memry layer that is the infrastructure fr all prcessing activity in the ODSA Data Transactin layer: A cmmn transactin mdel t transprt data Data Transprt layer: Prtcls fr memry management acrss prcessing cmpnents. The architecture will supprt cherent memry access and a nvel apprach t instructin-driven data mvement. The applicatin layer: Firmware and sftware t integrate the acceleratr with a hst. With this apprach, higher layers can be preserved even as the design evlves. Fr example, the PHY prtcl can change frm PCIe in a prttype t an Ultra Shrt SerDes in a prductin part, withut impacting higher-layer sftware. Perfrmance is achieved with a PHY cnnectivity technlgy that enables the architecture t be implemented with cst-efficient rganic substrate packaging technlgy. Large semicnductr cmpanies, including Intel, Xilinx, AMD and Nvidia, have recently annunced chiplet-based prducts that use prprietary prtcls at the PHY layer and higher layers f the n-package netwrk. With the pen ODSA architecture, custmers can chse best-f-breed prducts frm all the vendrs that supprt the architecture (see Figure 6). Figure 6: Open & Clsed Chiplet Ecsystems Surce: ODSA The U.S. gvernment agency DARPA recently annunced a chiplet initiative. The agency's effrt fcuses n pening the PHY layer prtcl. The ODSA specifies an entire prtcl stack abve the PHY layer, greatly simplifying and accelerating prduct develpment. The HEAVY READING NETRONOME OPEN DOMAIN-SPECIFIC ACCELERATORS 8

9 ODSA als supprts SerDes technlgies that reduce wire cunt, enabling prducts t be built with simple rganic substrates. DELIVERING OPEN DOMAIN-SPECIFIC ACCELERATORS The ODSA Wrkgrup prpses t make a platfrm prttype, built frm currently existing chiplets, available with the fllwing cmpnents: A netwrking chiplet with multiple USR SerDes prts t implement message-based cmmunicatin A lng-range SerDes chiplet with a USR SerDes prt A RISC CPU chiplet with a USR SerDes prt A multi-chip package with rm fr an additinal acceleratr die Hst integratin sftware fr netwrking acceleratin There are a number f chiplet-based slutins already available including: Achrnix ffers FPGA chiplets tday alngside their FPGA chips and embedded FPGA cres. Chiplets allw cmpanies t create very high-perfrmance and pwer-efficient slutins while keeping die size smaller t achieve gd yield. Netrnme can supprt the develpment f ODSA prttypes with the NFP-4000 and 6000 devices. These devices supprt up t fur independent PCIe Gen 3 interfaces. Netrnme intends t develp a prttype implementatin f the ODSA with the PCIe as the inter-chiplet interface. Sarcina is a semicnductr packaging and testing cmpany that supprts C4 bump-npad, Cu pillar bump n-pad, Cu pillar bump-n-trace, BGA based package, wirebnd die, QFN package, TQFP package, die stacking, SMD cap, 0201M silicn cap, discrete devices and ther ptins. Averasemi/GLOBALFOUNDRIES is in prductin with multi-chip slutins tday, including multi-die with chip scale package, HBM and ther memry integratin using rganic laminates and silicn interpsers. The visin f chiplets is a brad ecsystem f thusands f interperable chiplets built in varius fundries that deliver a wide variety f functinality fr lwer csts, faster time-tmarket and mre cst-effective innvatin. Business mdels will need t supprt this visin. In rder fr this methd t succeed, nvel business mdels need t be established. Integrated ASIC prviders have already put in place effective mdels fr integratin f HBM mdules, memry devices and knwn gd die systems. This mdel can be extended t prvide much mre cmplex integratin with cmpnents frm multiple surces. Technical Challenges The key technical challenge fr pen dmain-specific acceleratrs are ensuring the cnnectivity between chiplets is pen and standardized and that applicatins can be develped and executed n the reference multi-die architecture in a similar way t a mnlithic architecture. HEAVY READING NETRONOME OPEN DOMAIN-SPECIFIC ACCELERATORS 9

10 The transactin layer used between chiplets is the key enabler fr this integratin. In rder fr an integrated CPU r acceleratr unit t realize maximum efficiency, access t external data interfaces (PCI Express, Ethernet, Flash memry) must appear as if they are "lcal" t the prcessing unit and be ttally transparent t the sftware mdel. By leveraging the ODSA mdel, develpers are free t chse the ptimal slutin fr each chiplet based n perfrmance needs, IP availability and cst. As an example, a slutin with relatively limited external bandwidth requirements but a need fr premium perfrmance in an AI acceleratr may chse t implement the acceleratr and CPU chiplets in an advanced FinFET nde (16/14 nm r beynd) while chsing t use a lwer-cst 28 nm technlgy t implement a small handful f PCI Express and 10G Ethernet links. With the ODSA architecture, the CPU and acceleratr chiplets will see these links as lcal resurces that appear t the system as if they were attached t the same internal bus. Beynd this, multi-chiplet packaging will have t deal with the well-knwn prblem f selecting knwn gd die t integrate int a package. Business Challenges Althugh ffering a similar functinality in a system, chiplet technlgy will require a different business mdel than that used fr silicn IP. The reasn fr this is that chiplets, unlike silicn IP, will need t be prcessed, manufactured and quality-guaranteed fr years, if nt decades. Silicn IP requires a very intensive engagement during the design prcess, but relatively little need fr cntinuing supprt after prductin. A chiplet business will need t prvide manufacturing guarantees regarding hw lng the supplier will guarantee that a chiplet will be in prductin, alternatively a chiplet prvider may ffer the transfer f manufacturing rights t the MCM develper in exchange fr ryalty payments. This apprach wuld simplify the selectin f chiplets by a vendr with regard t cncerns abut lng-term availability. ODSA WORKGROUP The Imprtance f the ODSA Wrkgrup Thrugh the ODSA, the silicn industry has the pprtunity t expand the use f chiplets frm single vendrs t a cmplete ecsystem, enabling system develpers and service prviders t develp and deply advanced SC slutins. The fllwing qutes frm leading industry figures prvide sme insight int current thinking: "We are extremely excited t cllabrate with industry leaders and cntribute significant intellectual prperty and related pen specificatins derived frm the prven NFP prducts and apply that effectively t the pen and cmpsable chiplet-based architecture being develped in the ODSA Wrkgrup." said Niel Viljen, funder and CEO at Netrnme. HEAVY READING NETRONOME OPEN DOMAIN-SPECIFIC ACCELERATORS 10

11 "We are delighted t jin and bring ur embedded FPGA technlgy t the ODSA Wrkgrup t enable custmers t bring pen, cst-efficient acceleratr prducts t market." said Steve Mensr, vice president f marketing at Achrnix. "Our cllabratin effrts with the ODSA Wrkgrup ensure an additinal ptin t enable data center SC acceleratr technlgy supprting applicatins frm deep learning fr artificial intelligence t next-generatin 5G netwrks." said Kevin O'Buckley, general manager f Avera Semicnductr. "With unprecedented bandwidth and ultra-lw pwer, Glasswing enables cmpanies t quickly and efficiently build flexible yet ptimized slutins fr wrklad-specific applicatins," said Dr. Amin Shkrllahi, funder and CEO at Kandu. "Kandu supprts the ODSA Wrkgrup and delivering Glasswing as a critical cmpnent." "NXP is pleased t jin the ODSA Wrkgrup and prvide its Multicre Arm SC slutins t enable lw-pwer, lw-latency, pen acceleratr slutins that deliver greater cst and perfrmance efficiencies." said Sam Fuller, directr f marketing at NXP. "We are pleased t be a member f the ODSA Wrkgrup and lk frward t SiFive's leading RISC-V Cre IP being available in chiplet frm, ptentially via ur silicn capabilities, t enable custmers t create pen, hetergeneus, best-in-class acceleratrs at lw cst." said Dr. Naveed Sherwani, president and CEO at SiFive. Planned Develpments Specificatins and cntributins within the ODSA Wrkgrup relate t pen and standardsbased cnnectivity between chiplets. They are in the fllwing areas: Applicatin layer: Open surce firmware infrastructure and hst sftware fr acceleratr use and hst integratin; new pen IP/specificatin Memry management layer: Prtcls fr synchrnus memry traffic based n an pen prtcl leveraging CCIX and TileLink; existing pen standard Prtcls fr asynchrnus memry traffic such as Instructin-driven Switch Fabric (ISF); new pen IP/specificatin based n Netrnme's n-chip cmmunicatin prtcl Link layer: A light link layer prtcl based n and leveraging PCIe and TileLink; existing pen standard Multiple PHY layer interfaces: Based n and leveraging PCIe XSR; existing pen standard Based n USR SerDes, and Bunch f Wires (BW) abstracted thrugh an pen, cmmn interface such as PCIe PIPE Substrate layer: Based n rganic substrate packaging and intercnnect; new pen IP/specificatin HEAVY READING NETRONOME OPEN DOMAIN-SPECIFIC ACCELERATORS 11

12 CONCLUSIONS CPU perfrmance cannt keep up with the prcessing required t deliver clud and netwrk services and many ther applicatins. Dmain-specific acceleratrs can dramatically increase the perfrmance f servers fr specific applicatins, reducing the number f servers required and thus saving cst and pwer cnsumptin. T gain the best benefits frm dmain-specific acceleratrs, these must be ptimized fr specific applicatins and integrate best-f-breed cmpnents. Chiplets ffer an attractive alternative t mnlithic designs fr dmain-specific acceleratrs and ther applicatins. The develpment and manufacturing csts f multiple die and MCM assembly can be significantly smaller than the csts f develping and manufacturing cmplete SC slutins n 7 nm r 10 nm silicn technlgy. Dmain-specific acceleratrs are ptimized fr specific applicatins and therefre cmpanies may want t use multiple acceleratrs with different cmbinatins f chiplets. Open standards are key t a strng chiplet ecsystem. As mre chiplets becme available with cmpatible interfaces, cmpanies will be able t create slutins mre quickly and csteffectively. The ODSA architecture and related specificatins will help the develpment f the chiplets with pen interfaces and a cmmn prtcl stack that are required t build this strng ecsystem. If yu wish t help build this ecsystem r partner with the ODSA and its members, please cntact inf@dsachiplets.rg. HEAVY READING NETRONOME OPEN DOMAIN-SPECIFIC ACCELERATORS 12