RDMA in Embedded Fabrics

Size: px
Start display at page:

Download "RDMA in Embedded Fabrics"

Transcription

1 RDMA in Embedded Fabrics Ken Cain, Mercury Computer Systems 06 April Mercury Computer Systems, Inc. Uncontrolled for Export Purposes 1

2 Outline Embedded Systems Architecture / OpenVPX Open MPI, OFED for Serial RapidIO (srio) Embedded Data Flow Use Cases OFED Enhancement Opportunities Conclusions 2

3 Embedded Systems Architecture Model OpenVPX Mercury Computer Systems, Inc. Uncontrolled for Export Purposes 3

4 Typical Embedded System Architecture Backplane Backplane Front Panel Front Panel RADAR, sonar, video, etc. sensors Sensor compute modules (FPGA) A to D converters, RF tuners VITA 41, 46/48, OpenVPX 3U, 6U, SBC modules DSP modules (PPC / x86 multicore, GPU) Switch module (e.g., serial RapidIO srio) Dataflows among sensor I/O and heterogeneous compute devices Size/weight/power constraints embedded servers? Compute, fabric efficiency smaller (or more capable) systems 4

5 OpenVPX Model 5

6 I/O, Accel CPU I/O, Accel I/O, Accel CPU I/O, Accel OpenVPX Model System Example... Connectivity Expansion Plane PCIe to Adjacent Slots Examples: GPGPU, FPGA Sensor/Compute 6

7 SWITCH I/O, Accel CPU I/O, Accel I/O, Accel CPU I/O, Accel OpenVPX Model System Example... Connectivity Data Plane Switched Fabric High Performance Interconnect Examples: srio, Ethernet, IB, PCIe Other Options: Distributed Daisy-Chained / Mesh 7

8 OFED for Mercury srio CPU/Fabric HW Reference, Examples Mercury POET Engine Performance Data Mercury Computer Systems, Inc. Uncontrolled for Export Purposes 8

9 CPU, Switch Board Model 9

10 Mercury Examples CPU Freescale 8641D dual-core SoC srio DMA Core i7 dual core Xeon dual-quad core 10

11 Mercury POET Protocol Offload Engine Tech. Data Plane for Intel Embedded Nodes PCIe srio DMA L2 Ethernet (Planned) PCIe Port 1 PCIe Port 2 POET POET PCIe / srio Bridge + Multi-channel DMA srio Port 1 srio Port 2 11

12 OFA Stack 12

13 OFA Stack + Mercury Software Open MPI PPC/sRIO MCS Provider SW Intel + POET/sRIO 13

14 Performance Data Next Mercury Open MPI, OFED Software Freescale 8641D / srio Intel + POET/sRIO Sample 10GbE iwarp (same rate as srio) Mercury-developed Benchmark SW Your mileage may vary Intel/POET/sRIO Data is Preliminary 14

15 OFED Latency srio, iwarp 15

16 OFED Data Rate srio, iwarp 16

17 OMPI Latency srio, iwarp 8641D: Write Rendezvous POET: < 4K Eager Limit (Work in Progress Temporary) 17

18 OMPI Data Rate srio, iwarp 18

19 OFED + Native Latency srio Native SW Lightweight, close to the DMA HW OFED SW Queue Pair State Machine Data, Control Transfers / DMA HW 19

20 OFED + Native Data Rate srio 20

21 Embedded Data Flow Use Cases Fabric I/O Memory Copies Sensor I/O Streaming Mode Heterogeneous Compute Devices Mercury Computer Systems, Inc. Uncontrolled for Export Purposes 21

22 Embedded Fabric IPC Backplane Front Panel Front Panel Backplane 22

23 Embedded Fabric IPC Good Opportunity to use OFED Front Panel Backplane Accelerators Present Challenges Front Panel Ex: FPGA direct connect to data plane Minimal Mem BW, CPU is Essential Backplane 23

24 Embedded Sensor I/O + IPC Backplane Front Panel Front Panel Backplane Sensor Cluster: Need SW for IPC (or a Wire Protocol) Sensor Tx: Minimize CPU/SW Role Ex: SW set up sensor IPC, then autonomous inner-loop Cluster Rx: Like Fabric IPC, Minimize Mem BW, CPU Mercury Computer Systems, Inc. Uncontrolled for Export Purposes 24

25 OFED* Enhancement Opportunities * Software API Enhancements Mercury Computer Systems, Inc. Uncontrolled for Export Purposes 25

26 Fabric IPC: HPC / Real-Time * Persistent WQEs (Transfer) Issue Reinterpretation of WRs Each Time? Improvement Pre-plan / Persistent Fast (Modify +) Queue * Figure From InfiniBand specification 26

27 Now Multi-Node (Shared SQ?) Xfer 1 (Dest A) Xfer 2 (Dest B) Xfer 3 (Dest C) Xfer 1 / A Xfer 2 / B Xfer 3 / C Outer Loop Setup Separate Transfers One SW call per-destination Group Transfers into 1 Chain Inner Loop Fast (Modify +) Queue 27

28 Contiguous / Physical Memory Contiguous Memory Benefits DMA Performance PPC Ex: Block Address Translation (BAT) Mappings External Driver Provides non-linux Managed Pool MPI exposes to application via MPI_Alloc_mem() OFED Memory Registration Challenge Patch libibverbs / avoid Linux madvise() issues MCS provider distinguishes Linux / Non-Linux MEM Common Approach Too for Accelerator Memory? 28

29 Conclusions Mercury Computer Systems, Inc. Uncontrolled for Export Purposes 29

30 Perspective Embedded Market Demands Flexibility It s not just about IB, iwarp, RoCE, srio, PCIe It is about open standard SW and performance Cost of Open / Efficiency Always a Factor OFED RDMA performance promising initial results MPI model adds performance challenges Hybrid Data Plane Programming Environments HPC / Exascale: MPI investment focus MPI+PGAS HPEC: Vendor RDMA MPI+RDMA-based middleware 30

31 Backup Mercury Computer Systems, Inc. Uncontrolled for Export Purposes 31

32 OFED Alignment with Embedded Architectural Flexibility Offload / RDMA Open Systems, Industry SW Ecosystem Mercury Computer Systems, Inc. Uncontrolled for Export Purposes 32

33 33

34 34

35 Miscellaneous (RDMA) CM Devices Mercury Has Integrated with OFED not: IB iwarp Linux netdev Patched RDMA CM Associate names (IP addr) with srio L2 addr RDMA CM Connection Support (for now): Emulate just enough IB RDMA CM layer over QP1 with send/recv verbs 35

Open Fabrics Workshop 2013

Open Fabrics Workshop 2013 Open Fabrics Workshop 2013 OFS Software for the Intel Xeon Phi Bob Woodruff Agenda Intel Coprocessor Communication Link (CCL) Software IBSCIF RDMA from Host to Intel Xeon Phi Direct HCA Access from Intel

More information

Creating High Performance Clusters for Embedded Use

Creating High Performance Clusters for Embedded Use Creating High Performance Clusters for Embedded Use 1 The Hype.. The Internet of Things has the capacity to create huge amounts of data Gartner forecasts 35ZB of data from things by 2020 etc Intel Putting

More information

Performance Analysis and Evaluation of Mellanox ConnectX InfiniBand Architecture with Multi-Core Platforms

Performance Analysis and Evaluation of Mellanox ConnectX InfiniBand Architecture with Multi-Core Platforms Performance Analysis and Evaluation of Mellanox ConnectX InfiniBand Architecture with Multi-Core Platforms Sayantan Sur, Matt Koop, Lei Chai Dhabaleswar K. Panda Network Based Computing Lab, The Ohio State

More information

RapidIO.org Update. Mar RapidIO.org 1

RapidIO.org Update. Mar RapidIO.org 1 RapidIO.org Update rickoco@rapidio.org Mar 2015 2015 RapidIO.org 1 Outline RapidIO Overview & Markets Data Center & HPC Communications Infrastructure Industrial Automation Military & Aerospace RapidIO.org

More information

RoCE Update. Liran Liss, Mellanox Technologies March,

RoCE Update. Liran Liss, Mellanox Technologies March, RoCE Update Liran Liss, Mellanox Technologies March, 2012 www.openfabrics.org 1 Agenda RoCE Ecosystem QoS Virtualization High availability Latest news 2 RoCE in the Data Center Lossless configuration recommended

More information

BUILDING A BLOCK STORAGE APPLICATION ON OFED - CHALLENGES

BUILDING A BLOCK STORAGE APPLICATION ON OFED - CHALLENGES 3rd ANNUAL STORAGE DEVELOPER CONFERENCE 2017 BUILDING A BLOCK STORAGE APPLICATION ON OFED - CHALLENGES Subhojit Roy, Tej Parkash, Lokesh Arora, Storage Engineering [May 26th, 2017 ] AGENDA Introduction

More information

NTRDMA v0.1. An Open Source Driver for PCIe NTB and DMA. Allen Hubbe at Linux Piter 2015 NTRDMA. Messaging App. IB Verbs. dmaengine.h ntb.

NTRDMA v0.1. An Open Source Driver for PCIe NTB and DMA. Allen Hubbe at Linux Piter 2015 NTRDMA. Messaging App. IB Verbs. dmaengine.h ntb. Messaging App IB Verbs NTRDMA dmaengine.h ntb.h DMA DMA DMA NTRDMA v0.1 An Open Source Driver for PCIe and DMA Allen Hubbe at Linux Piter 2015 1 INTRODUCTION Allen Hubbe Senior Software Engineer EMC Corporation

More information

Voltaire. Fast I/O for XEN using RDMA Technologies. The Grid Interconnect Company. April 2005 Yaron Haviv, Voltaire, CTO

Voltaire. Fast I/O for XEN using RDMA Technologies. The Grid Interconnect Company. April 2005 Yaron Haviv, Voltaire, CTO Voltaire The Grid Interconnect Company Fast I/O for XEN using RDMA Technologies April 2005 Yaron Haviv, Voltaire, CTO yaronh@voltaire.com The Enterprise Grid Model and ization VMs need to interact efficiently

More information

Improving Application Performance and Predictability using Multiple Virtual Lanes in Modern Multi-Core InfiniBand Clusters

Improving Application Performance and Predictability using Multiple Virtual Lanes in Modern Multi-Core InfiniBand Clusters Improving Application Performance and Predictability using Multiple Virtual Lanes in Modern Multi-Core InfiniBand Clusters Hari Subramoni, Ping Lai, Sayantan Sur and Dhabhaleswar. K. Panda Department of

More information

RapidIO.org Update.

RapidIO.org Update. RapidIO.org Update rickoco@rapidio.org June 2015 2015 RapidIO.org 1 Outline RapidIO Overview Benefits Interconnect Comparison Ecosystem System Challenges RapidIO Markets Data Center & HPC Communications

More information

Advanced Computer Networks. End Host Optimization

Advanced Computer Networks. End Host Optimization Oriana Riva, Department of Computer Science ETH Zürich 263 3501 00 End Host Optimization Patrick Stuedi Spring Semester 2017 1 Today End-host optimizations: NUMA-aware networking Kernel-bypass Remote Direct

More information

Welcome to the IBTA Fall Webinar Series

Welcome to the IBTA Fall Webinar Series Welcome to the IBTA Fall Webinar Series A four-part webinar series devoted to making I/O work for you Presented by the InfiniBand Trade Association The webinar will begin shortly. 1 September 23 October

More information

In-Network Computing. Sebastian Kalcher, Senior System Engineer HPC. May 2017

In-Network Computing. Sebastian Kalcher, Senior System Engineer HPC. May 2017 In-Network Computing Sebastian Kalcher, Senior System Engineer HPC May 2017 Exponential Data Growth The Need for Intelligent and Faster Interconnect CPU-Centric (Onload) Data-Centric (Offload) Must Wait

More information

MELLANOX EDR UPDATE & GPUDIRECT MELLANOX SR. SE 정연구

MELLANOX EDR UPDATE & GPUDIRECT MELLANOX SR. SE 정연구 MELLANOX EDR UPDATE & GPUDIRECT MELLANOX SR. SE 정연구 Leading Supplier of End-to-End Interconnect Solutions Analyze Enabling the Use of Data Store ICs Comprehensive End-to-End InfiniBand and Ethernet Portfolio

More information

Embracing Open Technologies in the HPEC Market

Embracing Open Technologies in the HPEC Market Embracing Open Technologies in the HPEC Market Embedded Tech Trends January 2013 Andrew Shieh Director of Engineering CSPI, MultiComputer Division 1 CSPI Company Overview Diversified multinational technology

More information

Chelsio 10G Ethernet Open MPI OFED iwarp with Arista Switch

Chelsio 10G Ethernet Open MPI OFED iwarp with Arista Switch PERFORMANCE BENCHMARKS Chelsio 10G Ethernet Open MPI OFED iwarp with Arista Switch Chelsio Communications www.chelsio.com sales@chelsio.com +1-408-962-3600 Executive Summary Ethernet provides a reliable

More information

PARAVIRTUAL RDMA DEVICE

PARAVIRTUAL RDMA DEVICE 12th ANNUAL WORKSHOP 2016 PARAVIRTUAL RDMA DEVICE Aditya Sarwade, Adit Ranadive, Jorgen Hansen, Bhavesh Davda, George Zhang, Shelley Gong VMware, Inc. [ April 5th, 2016 ] MOTIVATION User Kernel Socket

More information

Comparing Ethernet and Soft RoCE for MPI Communication

Comparing Ethernet and Soft RoCE for MPI Communication IOSR Journal of Computer Engineering (IOSR-JCE) e-issn: 7-66, p- ISSN: 7-77Volume, Issue, Ver. I (Jul-Aug. ), PP 5-5 Gurkirat Kaur, Manoj Kumar, Manju Bala Department of Computer Science & Engineering,

More information

Introduction to High-Speed InfiniBand Interconnect

Introduction to High-Speed InfiniBand Interconnect Introduction to High-Speed InfiniBand Interconnect 2 What is InfiniBand? Industry standard defined by the InfiniBand Trade Association Originated in 1999 InfiniBand specification defines an input/output

More information

Birds of a Feather Presentation

Birds of a Feather Presentation Mellanox InfiniBand QDR 4Gb/s The Fabric of Choice for High Performance Computing Gilad Shainer, shainer@mellanox.com June 28 Birds of a Feather Presentation InfiniBand Technology Leadership Industry Standard

More information

OpenFabrics Interface WG A brief introduction. Paul Grun co chair OFI WG Cray, Inc.

OpenFabrics Interface WG A brief introduction. Paul Grun co chair OFI WG Cray, Inc. OpenFabrics Interface WG A brief introduction Paul Grun co chair OFI WG Cray, Inc. OFI WG a brief overview and status report 1. Keep everybody on the same page, and 2. An example of a possible model for

More information

OCP Engineering Workshop - Telco

OCP Engineering Workshop - Telco OCP Engineering Workshop - Telco Low Latency Mobile Edge Computing Trevor Hiatt Product Management, IDT IDT Company Overview Founded 1980 Workforce Approximately 1,800 employees Headquarters San Jose,

More information

1 of 5 10/6/2014 3:22 PM

1 of 5 10/6/2014 3:22 PM 1 of 5 10/6/2014 3:22 PM HOME ARCHIVES RESOURCES ABOUT US ADVERTISE TECH RECON SYSTEM DEVELOPMENT TECHNOLOGY FOCUS SPECIAL FEATURE PRODUCTS PUBLISHER'S NOTEBOOK ALL Search Articles SPECIAL FEATURE KEYWORDS

More information

Enabling Efficient Use of UPC and OpenSHMEM PGAS models on GPU Clusters

Enabling Efficient Use of UPC and OpenSHMEM PGAS models on GPU Clusters Enabling Efficient Use of UPC and OpenSHMEM PGAS models on GPU Clusters Presentation at GTC 2014 by Dhabaleswar K. (DK) Panda The Ohio State University E-mail: panda@cse.ohio-state.edu http://www.cse.ohio-state.edu/~panda

More information

SR-IOV Support for Virtualization on InfiniBand Clusters: Early Experience

SR-IOV Support for Virtualization on InfiniBand Clusters: Early Experience SR-IOV Support for Virtualization on InfiniBand Clusters: Early Experience Jithin Jose, Mingzhe Li, Xiaoyi Lu, Krishna Kandalla, Mark Arnold and Dhabaleswar K. (DK) Panda Network-Based Computing Laboratory

More information

Unified Runtime for PGAS and MPI over OFED

Unified Runtime for PGAS and MPI over OFED Unified Runtime for PGAS and MPI over OFED D. K. Panda and Sayantan Sur Network-Based Computing Laboratory Department of Computer Science and Engineering The Ohio State University, USA Outline Introduction

More information

Open Fabrics Interfaces Architecture Introduction. Sean Hefty Intel Corporation

Open Fabrics Interfaces Architecture Introduction. Sean Hefty Intel Corporation Open Fabrics Interfaces Architecture Introduction Sean Hefty Intel Corporation Current State of Affairs OFED software Widely adopted low-level RDMA API Ships with upstream Linux but OFED SW was not designed

More information

Solutions for Scalable HPC

Solutions for Scalable HPC Solutions for Scalable HPC Scot Schultz, Director HPC/Technical Computing HPC Advisory Council Stanford Conference Feb 2014 Leading Supplier of End-to-End Interconnect Solutions Comprehensive End-to-End

More information

Application Acceleration Beyond Flash Storage

Application Acceleration Beyond Flash Storage Application Acceleration Beyond Flash Storage Session 303C Mellanox Technologies Flash Memory Summit July 2014 Accelerating Applications, Step-by-Step First Steps Make compute fast Moore s Law Make storage

More information

The rcuda middleware and applications

The rcuda middleware and applications The rcuda middleware and applications Will my application work with rcuda? rcuda currently provides binary compatibility with CUDA 5.0, virtualizing the entire Runtime API except for the graphics functions,

More information

Interconnect Your Future

Interconnect Your Future Interconnect Your Future Smart Interconnect for Next Generation HPC Platforms Gilad Shainer, August 2016, 4th Annual MVAPICH User Group (MUG) Meeting Mellanox Connects the World s Fastest Supercomputer

More information

Octopus: A Multi-core implementation

Octopus: A Multi-core implementation Octopus: A Multi-core implementation Kalpesh Sheth HPEC 2007, MIT, Lincoln Lab Export of this products is subject to U.S. export controls. Licenses may be required. This material provides up-to-date general

More information

MOVING FORWARD WITH FABRIC INTERFACES

MOVING FORWARD WITH FABRIC INTERFACES 14th ANNUAL WORKSHOP 2018 MOVING FORWARD WITH FABRIC INTERFACES Sean Hefty, OFIWG co-chair Intel Corporation April, 2018 USING THE PAST TO PREDICT THE FUTURE OFI Provider Infrastructure OFI API Exploration

More information

Highly Scalable, Non-RDMA NVMe Fabric. Bob Hansen,, VP System Architecture

Highly Scalable, Non-RDMA NVMe Fabric. Bob Hansen,, VP System Architecture A Cost Effective,, High g Performance,, Highly Scalable, Non-RDMA NVMe Fabric Bob Hansen,, VP System Architecture bob@apeirondata.com Storage Developers Conference, September 2015 Agenda 3 rd Platform

More information

jverbs: Java/OFED Integration for the Cloud

jverbs: Java/OFED Integration for the Cloud jverbs: Java/OFED Integration for the Cloud Authors: Bernard Metzler, Patrick Stuedi, Animesh Trivedi. IBM Research Zurich Date: 03/27/12 www.openfabrics.org 1 Motivation The commodity Cloud is Flexible

More information

2008 International ANSYS Conference

2008 International ANSYS Conference 2008 International ANSYS Conference Maximizing Productivity With InfiniBand-Based Clusters Gilad Shainer Director of Technical Marketing Mellanox Technologies 2008 ANSYS, Inc. All rights reserved. 1 ANSYS,

More information

URDMA: RDMA VERBS OVER DPDK

URDMA: RDMA VERBS OVER DPDK 13 th ANNUAL WORKSHOP 2017 URDMA: RDMA VERBS OVER DPDK Patrick MacArthur, Ph.D. Candidate University of New Hampshire March 28, 2017 ACKNOWLEDGEMENTS urdma was initially developed during an internship

More information

2017 Storage Developer Conference. Mellanox Technologies. All Rights Reserved.

2017 Storage Developer Conference. Mellanox Technologies. All Rights Reserved. Ethernet Storage Fabrics Using RDMA with Fast NVMe-oF Storage to Reduce Latency and Improve Efficiency Kevin Deierling & Idan Burstein Mellanox Technologies 1 Storage Media Technology Storage Media Access

More information

Unified Communication X (UCX)

Unified Communication X (UCX) Unified Communication X (UCX) Pavel Shamis / Pasha ARM Research SC 18 UCF Consortium Mission: Collaboration between industry, laboratories, and academia to create production grade communication frameworks

More information

Chelsio Communications. Meeting Today s Datacenter Challenges. Produced by Tabor Custom Publishing in conjunction with: CUSTOM PUBLISHING

Chelsio Communications. Meeting Today s Datacenter Challenges. Produced by Tabor Custom Publishing in conjunction with: CUSTOM PUBLISHING Meeting Today s Datacenter Challenges Produced by Tabor Custom Publishing in conjunction with: 1 Introduction In this era of Big Data, today s HPC systems are faced with unprecedented growth in the complexity

More information

Support for GPUs with GPUDirect RDMA in MVAPICH2 SC 13 NVIDIA Booth

Support for GPUs with GPUDirect RDMA in MVAPICH2 SC 13 NVIDIA Booth Support for GPUs with GPUDirect RDMA in MVAPICH2 SC 13 NVIDIA Booth by D.K. Panda The Ohio State University E-mail: panda@cse.ohio-state.edu http://www.cse.ohio-state.edu/~panda Outline Overview of MVAPICH2-GPU

More information

When evaluating choices between

When evaluating choices between Fabric Bandwidth Comparisons on Backplanes Open Backplane Fabric Choice Calls for Careful Analyses There s a host of factors to consider when evaluating the system of various Open fabrics. A detailed comparison

More information

USING OPEN FABRIC INTERFACE IN INTEL MPI LIBRARY

USING OPEN FABRIC INTERFACE IN INTEL MPI LIBRARY 14th ANNUAL WORKSHOP 2018 USING OPEN FABRIC INTERFACE IN INTEL MPI LIBRARY Michael Chuvelev, Software Architect Intel April 11, 2018 INTEL MPI LIBRARY Optimized MPI application performance Application-specific

More information

The Exascale Architecture

The Exascale Architecture The Exascale Architecture Richard Graham HPC Advisory Council China 2013 Overview Programming-model challenges for Exascale Challenges for scaling MPI to Exascale InfiniBand enhancements Dynamically Connected

More information

10-Gigabit iwarp Ethernet: Comparative Performance Analysis with InfiniBand and Myrinet-10G

10-Gigabit iwarp Ethernet: Comparative Performance Analysis with InfiniBand and Myrinet-10G 10-Gigabit iwarp Ethernet: Comparative Performance Analysis with InfiniBand and Myrinet-10G Mohammad J. Rashti and Ahmad Afsahi Queen s University Kingston, ON, Canada 2007 Workshop on Communication Architectures

More information

CPMD Performance Benchmark and Profiling. February 2014

CPMD Performance Benchmark and Profiling. February 2014 CPMD Performance Benchmark and Profiling February 2014 Note The following research was performed under the HPC Advisory Council activities Special thanks for: HP, Mellanox For more information on the supporting

More information

ehca Virtualization on System p

ehca Virtualization on System p ehca Virtualization on System p Christoph Raisch Technical Lead ehca Infiniband and HEA device drivers 2008-04-04 Trademark Statements IBM, the IBM logo, ibm.com, System p, System p5 and POWER Hypervisor

More information

OPEN MPI WITH RDMA SUPPORT AND CUDA. Rolf vandevaart, NVIDIA

OPEN MPI WITH RDMA SUPPORT AND CUDA. Rolf vandevaart, NVIDIA OPEN MPI WITH RDMA SUPPORT AND CUDA Rolf vandevaart, NVIDIA OVERVIEW What is CUDA-aware History of CUDA-aware support in Open MPI GPU Direct RDMA support Tuning parameters Application example Future work

More information

Performance Optimizations via Connect-IB and Dynamically Connected Transport Service for Maximum Performance on LS-DYNA

Performance Optimizations via Connect-IB and Dynamically Connected Transport Service for Maximum Performance on LS-DYNA Performance Optimizations via Connect-IB and Dynamically Connected Transport Service for Maximum Performance on LS-DYNA Pak Lui, Gilad Shainer, Brian Klaff Mellanox Technologies Abstract From concept to

More information

OCTOPUS Performance Benchmark and Profiling. June 2015

OCTOPUS Performance Benchmark and Profiling. June 2015 OCTOPUS Performance Benchmark and Profiling June 2015 2 Note The following research was performed under the HPC Advisory Council activities Special thanks for: HP, Mellanox For more information on the

More information

Ensemble 6000 Series OpenVPX HCD6210 Dual QorIQ T4240 Processing Module

Ensemble 6000 Series OpenVPX HCD6210 Dual QorIQ T4240 Processing Module Ensemble 6000 Series OpenVPX HCD6210 Dual QorIQ T4240 Processing Module Next-Generation High Density Processing With I/O in a Single VPX slot OpenVPX The Ensemble 6000 Series OpenVPX HCD6210 High Compute

More information

FiPS and M2DC: Novel Architectures for Reconfigurable Hyperscale Servers

FiPS and M2DC: Novel Architectures for Reconfigurable Hyperscale Servers FiPS and M2DC: Novel Architectures for Reconfigurable Hyperscale Servers Rene Griessl, Meysam Peykanu, Lennart Tigges, Jens Hagemeyer, Mario Porrmann Center of Excellence Cognitive Interaction Technology

More information

Scaling with PGAS Languages

Scaling with PGAS Languages Scaling with PGAS Languages Panel Presentation at OFA Developers Workshop (2013) by Dhabaleswar K. (DK) Panda The Ohio State University E-mail: panda@cse.ohio-state.edu http://www.cse.ohio-state.edu/~panda

More information

Support for Smart NICs. Ian Pratt

Support for Smart NICs. Ian Pratt Support for Smart NICs Ian Pratt Outline Xen I/O Overview Why network I/O is harder than block Smart NIC taxonomy How Xen can exploit them Enhancing Network device channel NetChannel2 proposal I/O Architecture

More information

Concurrent Support of NVMe over RDMA Fabrics and Established Networked Block and File Storage

Concurrent Support of NVMe over RDMA Fabrics and Established Networked Block and File Storage Concurrent Support of NVMe over RDMA Fabrics and Established Networked Block and File Storage Ásgeir Eiriksson CTO Chelsio Communications Inc. August 2016 1 Introduction API are evolving for optimal use

More information

Accessing NVM Locally and over RDMA Challenges and Opportunities

Accessing NVM Locally and over RDMA Challenges and Opportunities Accessing NVM Locally and over RDMA Challenges and Opportunities Wendy Elsasser Megan Grodowitz William Wang MSST - May 2018 Emerging NVM A wide variety of technologies with varied characteristics Address

More information

iscsi or iser? Asgeir Eiriksson CTO Chelsio Communications Inc

iscsi or iser? Asgeir Eiriksson CTO Chelsio Communications Inc iscsi or iser? Asgeir Eiriksson CTO Chelsio Communications Inc Introduction iscsi is compatible with 15 years of deployment on all OSes and preserves software investment iser and iscsi are layered on top

More information

RoCE vs. iwarp Competitive Analysis

RoCE vs. iwarp Competitive Analysis WHITE PAPER February 217 RoCE vs. iwarp Competitive Analysis Executive Summary...1 RoCE s Advantages over iwarp...1 Performance and Benchmark Examples...3 Best Performance for Virtualization...5 Summary...6

More information

iwarp Learnings and Best Practices

iwarp Learnings and Best Practices iwarp Learnings and Best Practices Author: Michael Fenn, Penn State Date: March 28, 2012 www.openfabrics.org 1 Introduction Last year, the Research Computing and Cyberinfrastructure group at Penn State

More information

Ensemble 3000 Series OpenVPX Intel Xeon Processor D Family LDS3506 Server-Class with FPGA Module

Ensemble 3000 Series OpenVPX Intel Xeon Processor D Family LDS3506 Server-Class with FPGA Module DATASHEET Ensemble 3000 Series OpenVPX Intel Xeon Processor D Family LDS3506 Server-Class with FPGA Module Powerful combination of Intel Xeon D family server-class and FPGA processing in a rugged, 3U OpenVPX

More information

IO virtualization. Michael Kagan Mellanox Technologies

IO virtualization. Michael Kagan Mellanox Technologies IO virtualization Michael Kagan Mellanox Technologies IO Virtualization Mission non-stop s to consumers Flexibility assign IO resources to consumer as needed Agility assignment of IO resources to consumer

More information

Inspur AI Computing Platform

Inspur AI Computing Platform Inspur Server Inspur AI Computing Platform 3 Server NF5280M4 (2CPU + 3 ) 4 Server NF5280M5 (2 CPU + 4 ) Node (2U 4 Only) 8 Server NF5288M5 (2 CPU + 8 ) 16 Server SR BOX (16 P40 Only) Server target market

More information

Interconnect Your Future

Interconnect Your Future Interconnect Your Future Gilad Shainer 2nd Annual MVAPICH User Group (MUG) Meeting, August 2014 Complete High-Performance Scalable Interconnect Infrastructure Comprehensive End-to-End Software Accelerators

More information

Containing RDMA and High Performance Computing

Containing RDMA and High Performance Computing Containing RDMA and High Performance Computing Liran Liss ContainerCon 2015 Agenda High Performance Computing (HPC) networking RDMA 101 Containing RDMA Challenges Solution approach RDMA network namespace

More information

Enabling the NVMe CMB and PMR Ecosystem

Enabling the NVMe CMB and PMR Ecosystem Architected for Performance Enabling the NVMe CMB and PMR Ecosystem Stephen Bates, PhD. CTO, Eideticom Oren Duer. Software Architect, Mellanox NVM Express Developers Day May 1, 2018 Outline 1. Intro to

More information

CPU Agnostic Motherboard design with RapidIO Interconnect in Data Center

CPU Agnostic Motherboard design with RapidIO Interconnect in Data Center Agnostic Motherboard design with RapidIO Interconnect in Data Center Devashish Paul Senior Product Manager IDT Chairman RapidIO Trade Association: Marketing Council 2013 RapidIO Trade Association Agenda

More information

RDMA Container Support. Liran Liss Mellanox Technologies

RDMA Container Support. Liran Liss Mellanox Technologies RDMA Container Support Liran Liss Mellanox Technologies Agenda Containers 101 RDMA isolation Namespace support Controller support Putting it all together Status Conclusions March 15 18, 2015 #OFADevWorkshop

More information

High Performance Computing

High Performance Computing High Performance Computing Dror Goldenberg, HPCAC Switzerland Conference March 2015 End-to-End Interconnect Solutions for All Platforms Highest Performance and Scalability for X86, Power, GPU, ARM and

More information

Oncilla - a Managed GAS Runtime for Accelerating Data Warehousing Queries

Oncilla - a Managed GAS Runtime for Accelerating Data Warehousing Queries Oncilla - a Managed GAS Runtime for Accelerating Data Warehousing Queries Jeffrey Young, Alex Merritt, Se Hoon Shon Advisor: Sudhakar Yalamanchili 4/16/13 Sponsors: Intel, NVIDIA, NSF 2 The Problem Big

More information

GEN-2 / GEN-1 OpenVPX BACKPLANES

GEN-2 / GEN-1 OpenVPX BACKPLANES BENEFITS GEN-2 / GEN-1 VPX data/expansion plane support for Gen-2, backplanes compatible with: Serial RapidIO (SRIO) Gen-2 InfiniBand DDR PCI Express (PCIe) Gen-2 VPX data/expansion plane support for Gen-1,

More information

Multifunction Networking Adapters

Multifunction Networking Adapters Ethernet s Extreme Makeover: Multifunction Networking Adapters Chuck Hudson Manager, ProLiant Networking Technology Hewlett-Packard 2004 Hewlett-Packard Development Company, L.P. The information contained

More information

NVM Express Awakening a New Storage and Networking Titan Shaun Walsh G2M Research

NVM Express Awakening a New Storage and Networking Titan Shaun Walsh G2M Research NVM Express Awakening a New Storage and Networking Titan Shaun Walsh G2M Research Acronyms and Definition Check Point Term Definition NVMe Non-Volatile Memory Express NVMe-oF Non-Volatile Memory Express

More information

Interconnect Your Future

Interconnect Your Future #OpenPOWERSummit Interconnect Your Future Scot Schultz, Director HPC / Technical Computing Mellanox Technologies OpenPOWER Summit, San Jose CA March 2015 One-Generation Lead over the Competition Mellanox

More information

Using FPGAs to accelerate NVMe-oF based Storage Networks

Using FPGAs to accelerate NVMe-oF based Storage Networks Using FPGAs to accelerate NVMe-oF based Storage Networks Deboleena Sakalley IP & Solutions Architect, Xilinx Santa Clara, CA 1 Agenda NVMe-oF Offload in FPGA NVMe-oF Integrated Solution Solution Architecture

More information

Building supercomputers from embedded technologies

Building supercomputers from embedded technologies http://www.montblanc-project.eu Building supercomputers from embedded technologies Alex Ramirez Barcelona Supercomputing Center Technical Coordinator This project and the research leading to these results

More information

Networking at the Speed of Light

Networking at the Speed of Light Networking at the Speed of Light Dror Goldenberg VP Software Architecture MaRS Workshop April 2017 Cloud The Software Defined Data Center Resource virtualization Efficient services VM, Containers uservices

More information

Intra-MIC MPI Communication using MVAPICH2: Early Experience

Intra-MIC MPI Communication using MVAPICH2: Early Experience Intra-MIC MPI Communication using MVAPICH: Early Experience Sreeram Potluri, Karen Tomko, Devendar Bureddy, and Dhabaleswar K. Panda Department of Computer Science and Engineering Ohio State University

More information

FUJITSU PHI Turnkey Solution

FUJITSU PHI Turnkey Solution FUJITSU PHI Turnkey Solution Integrated ready to use XEON-PHI based platform Dr. Pierre Lagier ISC2014 - Leipzig PHI Turnkey Solution challenges System performance challenges Parallel IO best architecture

More information

ANSYS Fluent 14 Performance Benchmark and Profiling. October 2012

ANSYS Fluent 14 Performance Benchmark and Profiling. October 2012 ANSYS Fluent 14 Performance Benchmark and Profiling October 2012 Note The following research was performed under the HPC Advisory Council activities Special thanks for: HP, Mellanox For more information

More information

Latest Advances in MVAPICH2 MPI Library for NVIDIA GPU Clusters with InfiniBand

Latest Advances in MVAPICH2 MPI Library for NVIDIA GPU Clusters with InfiniBand Latest Advances in MVAPICH2 MPI Library for NVIDIA GPU Clusters with InfiniBand Presentation at GTC 2014 by Dhabaleswar K. (DK) Panda The Ohio State University E-mail: panda@cse.ohio-state.edu http://www.cse.ohio-state.edu/~panda

More information

OFED Storage Protocols

OFED Storage Protocols OFED Storage Protocols R. Pearson System Fabric Works, Inc. Agenda Why OFED Storage Introduction to OFED Storage Protocols OFED Storage Protocol Update 2 Why OFED Storage 3 Goals of I/O Consolidation Cluster

More information

CRFS: A Lightweight User-Level Filesystem for Generic Checkpoint/Restart

CRFS: A Lightweight User-Level Filesystem for Generic Checkpoint/Restart CRFS: A Lightweight User-Level Filesystem for Generic Checkpoint/Restart Xiangyong Ouyang, Raghunath Rajachandrasekar, Xavier Besseron, Hao Wang, Jian Huang, Dhabaleswar K. Panda Department of Computer

More information

SNAP Performance Benchmark and Profiling. April 2014

SNAP Performance Benchmark and Profiling. April 2014 SNAP Performance Benchmark and Profiling April 2014 Note The following research was performed under the HPC Advisory Council activities Participating vendors: HP, Mellanox For more information on the supporting

More information

Why AI Frameworks Need (not only) RDMA?

Why AI Frameworks Need (not only) RDMA? Why AI Frameworks Need (not only) RDMA? With Design and Implementation Experience of Networking Support on TensorFlow GDR, Apache MXNet, WeChat Amber, and Tencent Angel Bairen Yi (byi@connect.ust.hk) Jingrong

More information

Messaging Overview. Introduction. Gen-Z Messaging

Messaging Overview. Introduction. Gen-Z Messaging Page 1 of 6 Messaging Overview Introduction Gen-Z is a new data access technology that not only enhances memory and data storage solutions, but also provides a framework for both optimized and traditional

More information

Study. Dhabaleswar. K. Panda. The Ohio State University HPIDC '09

Study. Dhabaleswar. K. Panda. The Ohio State University HPIDC '09 RDMA over Ethernet - A Preliminary Study Hari Subramoni, Miao Luo, Ping Lai and Dhabaleswar. K. Panda Computer Science & Engineering Department The Ohio State University Introduction Problem Statement

More information

Open Compute Stack (OpenCS) Overview. D.D. Nikolić Updated: 20 August 2018 DAE Tools Project,

Open Compute Stack (OpenCS) Overview. D.D. Nikolić Updated: 20 August 2018 DAE Tools Project, Open Compute Stack (OpenCS) Overview D.D. Nikolić Updated: 20 August 2018 DAE Tools Project, http://www.daetools.com/opencs What is OpenCS? A framework for: Platform-independent model specification 1.

More information

Low latency, high bandwidth communication. Infiniband and RDMA programming. Bandwidth vs latency. Knut Omang Ifi/Oracle 2 Nov, 2015

Low latency, high bandwidth communication. Infiniband and RDMA programming. Bandwidth vs latency. Knut Omang Ifi/Oracle 2 Nov, 2015 Low latency, high bandwidth communication. Infiniband and RDMA programming Knut Omang Ifi/Oracle 2 Nov, 2015 1 Bandwidth vs latency There is an old network saying: Bandwidth problems can be cured with

More information

Exploiting Full Potential of GPU Clusters with InfiniBand using MVAPICH2-GDR

Exploiting Full Potential of GPU Clusters with InfiniBand using MVAPICH2-GDR Exploiting Full Potential of GPU Clusters with InfiniBand using MVAPICH2-GDR Presentation at Mellanox Theater () Dhabaleswar K. (DK) Panda - The Ohio State University panda@cse.ohio-state.edu Outline Communication

More information

MVAPICH-Aptus: Scalable High-Performance Multi-Transport MPI over InfiniBand

MVAPICH-Aptus: Scalable High-Performance Multi-Transport MPI over InfiniBand MVAPICH-Aptus: Scalable High-Performance Multi-Transport MPI over InfiniBand Matthew Koop 1,2 Terry Jones 2 D. K. Panda 1 {koop, panda}@cse.ohio-state.edu trj@llnl.gov 1 Network-Based Computing Lab, The

More information

ABySS Performance Benchmark and Profiling. May 2010

ABySS Performance Benchmark and Profiling. May 2010 ABySS Performance Benchmark and Profiling May 2010 Note The following research was performed under the HPC Advisory Council activities Participating vendors: AMD, Dell, Mellanox Compute resource - HPC

More information

High Performance Distributed Lock Management Services using Network-based Remote Atomic Operations

High Performance Distributed Lock Management Services using Network-based Remote Atomic Operations High Performance Distributed Lock Management Services using Network-based Remote Atomic Operations S. Narravula, A. Mamidala, A. Vishnu, K. Vaidyanathan, and D. K. Panda Presented by Lei Chai Network Based

More information

Informatix Solutions INFINIBAND OVERVIEW. - Informatix Solutions, Page 1 Version 1.0

Informatix Solutions INFINIBAND OVERVIEW. - Informatix Solutions, Page 1 Version 1.0 INFINIBAND OVERVIEW -, 2010 Page 1 Version 1.0 Why InfiniBand? Open and comprehensive standard with broad vendor support Standard defined by the InfiniBand Trade Association (Sun was a founder member,

More information

NFS/RDMA over 40Gbps iwarp Wael Noureddine Chelsio Communications

NFS/RDMA over 40Gbps iwarp Wael Noureddine Chelsio Communications NFS/RDMA over 40Gbps iwarp Wael Noureddine Chelsio Communications Outline RDMA Motivating trends iwarp NFS over RDMA Overview Chelsio T5 support Performance results 2 Adoption Rate of 40GbE Source: Crehan

More information

Paving the Road to Exascale

Paving the Road to Exascale Paving the Road to Exascale Gilad Shainer August 2015, MVAPICH User Group (MUG) Meeting The Ever Growing Demand for Performance Performance Terascale Petascale Exascale 1 st Roadrunner 2000 2005 2010 2015

More information

GPU-centric communication for improved efficiency

GPU-centric communication for improved efficiency GPU-centric communication for improved efficiency Benjamin Klenk *, Lena Oden, Holger Fröning * * Heidelberg University, Germany Fraunhofer Institute for Industrial Mathematics, Germany GPCDP Workshop

More information

A Plugin-based Approach to Exploit RDMA Benefits for Apache and Enterprise HDFS

A Plugin-based Approach to Exploit RDMA Benefits for Apache and Enterprise HDFS A Plugin-based Approach to Exploit RDMA Benefits for Apache and Enterprise HDFS Adithya Bhat, Nusrat Islam, Xiaoyi Lu, Md. Wasi- ur- Rahman, Dip: Shankar, and Dhabaleswar K. (DK) Panda Network- Based Compu2ng

More information

UCX: An Open Source Framework for HPC Network APIs and Beyond

UCX: An Open Source Framework for HPC Network APIs and Beyond UCX: An Open Source Framework for HPC Network APIs and Beyond Presented by: Pavel Shamis / Pasha ORNL is managed by UT-Battelle for the US Department of Energy Co-Design Collaboration The Next Generation

More information

VM Migration Acceleration over 40GigE Meet SLA & Maximize ROI

VM Migration Acceleration over 40GigE Meet SLA & Maximize ROI VM Migration Acceleration over 40GigE Meet SLA & Maximize ROI Mellanox Technologies Inc. Motti Beck, Director Marketing Motti@mellanox.com Topics Introduction to Mellanox Technologies Inc. Why Cloud SLA

More information

Mercury Computer Systems & The Cell Broadband Engine

Mercury Computer Systems & The Cell Broadband Engine Mercury Computer Systems & The Cell Broadband Engine Georgia Tech Cell Workshop 18-19 June 2007 About Mercury Leading provider of innovative computing solutions for challenging applications R&D centers

More information