Achieving 98Gbps of Crosscountry TCP traffic using 2.5 hosts, 10 x 10G NICs, and 10 TCP streams

Size: px
Start display at page:

Download "Achieving 98Gbps of Crosscountry TCP traffic using 2.5 hosts, 10 x 10G NICs, and 10 TCP streams"

Transcription

1 Achieving 98Gbps of Crosscountry TCP traffic using 2.5 hosts, 10 x 10G NICs, and 10 TCP streams Eric Pouyoul, Brian Tierney ESnet January 25, 2012

2 ANI 100G Testbed ANI Middleware Testbed NERSC To ESnet 10G To ESnet ANL 1GE nersc-asw1 Site Router (nersc-mr2) ANI 100G Network 10G 1GE anl-asw1 nersc-c2940 switch ANL Site Router 100G 100G anl-c2940 switch nersc-app 100G 100G nersc-diskpt-1 NICs: 2: 2x10G Myricom 1: 4x10G HotLava nersc-diskpt-2 NICs: 1: 2x10G Myricom 1: 2x10G Chelsio 1: 6x10G HotLava nersc-diskpt-1 10GE (MM) 10GE (MM) 4x10GE (MM) nersc-diskpt-2 4x10GE (MM) eth2-5 4x10GE (MM) eth2-5 ANI 100G Router ANI 100G Router eth2-5 4x 10GE (MM) anl-app anl-mempt-1 NICs: 2: 2x10G Myricom nersc-diskpt-3 NICs: 1: 2x10G Myricom 1: 2x10G Mellanox 1: 6x10G HotLava nersc-diskpt-3 4x10GE (MM) eth2-5 eth2-5 4x10GE (MM) anl-mempt-1 anl-mempt-2 NICs: 2: 2x10G Myricom anl-mempt-2 Note: ANI 100G routers and 100G wave available till summer 2012; Testbed resources after that subject funding availability. 4x10GE (MM) eth2-5 anl-mempt-3 anl-mempt-3 NICs: 1: 2x10G Myricom 1: 2x10G Mellanox Updated December 11, /25/12 2

3 Test Configuration Details 48.6ms RTT, 97.9Gbps aggregate TCP throughput* with 10 TCP streams 1/25/12 3

4 Per-Stream Results 1/25/12 4

5 Hosts Must be Tuned BIOS Firmware Device Drivers NIC/TCP Process Affinity 7/11/10 Joint Techs, Summer

6 BIOS Tuning: Free the Hardware Hyperthreading: disable, we want real cores. CPU frequency scaling: disable, as well as all energy saving features: we want the full power all the time. Check memory bus speed (force to max.) 7/11/10 Joint Techs, Summer

7 NIC/TCP Tuning We are using Myricom 10G NIC Download latest drive/firmware from vendor site Version of driver in RHEL/CentOS fairly old Enable MSI-X Increase txgueuelen /sbin/ifconfig eth2 txqueuelen 10000! Increase Interrupt coalescence! /usr/sbin/ethtool -C eth2 rx-usecs 100! Standard TCP Tuning: net.core.rmem_max = ! net.core.wmem_max = ! net.core.netdev_max_backlog = /25/12 7

8 100G = It s full of frames! Problem: Interrupts are very expensive Even with jumbo frames and driver optimization, there is still too many interrupts. Solution: Turn off Linux irqbalance (chkconfig irqbalance off) Use /proc/interrupt to get the list of interrupts Dedicate an entire processor core for each 10G interface Use /proc/irq/<irq-number>/smp_affinity to bind rx/tx queues to a specific core. 1/25/12 8

9 Application Tuning nuttcp from NASA provides command line option to bind to a processor core. Server command: nuttcp -S -P5200 -p5201 &; nuttcp -S -P5300 -p5301 &; nuttcp -S -P p5401 &; nuttcp -S -P5500 -p5501 Client command: nuttcp -xc2/2 -I1 -P5200 -p5201 -i1 -T30 -w50m &; nuttcp - xc3/3 -I2 -P5300 -p5301 -i1 -T30 -w50m &; nuttcp -xc4/4 -I3 -P5400 -p5401 -i1 -T30 -w50m & ; nuttcp -xc5/5 -I4 -P p5501 -i1 -T30 -w50m & Note: hand tune TCP window to 50M 1/25/12 9

10 More Information ANI 100G Testbed: Next round of proposals is due April 1 st 2012 Host tuning: Even more information: lomax@es.net

11 Extra Slides 1/25/12 11

12 PCI optimization: MSI-X - Extension to MSI (Message Signaled Interrupts) - Increases the number of interrupt pins per card - Associates rx/tx queues to a given core - Allows to stitch together on the same core, the thread that runs the program and the asynchronous event it may receive (incoming network packets, asynchronous I/O ), resulting in maximizing L1 cache hit. - Requires Chipset, card, and operating support. - Optimized for Linux kernel > This is a major optimization: on a system with 4 x 10G ethernet, performance gain can be up to 20% 1/25/12 12

13 Testbed Access Proposal process to gain access described at: Testbed is available to anyone: DOE researchers Other government agencies Industry Must submit a short proposal to the testbed review committee Committee is made up of members from the R&E community and industry Goal is to accept roughly five proposals every 6 month review cycle Next round of proposals is due April 1,

Experiences with 40G/100G Applications

Experiences with 40G/100G Applications Experiences with 40G/100G Applications Brian L Tierney ESnet, Internet2 Global Summit, April 2014 Outline Review of packet loss Overview SC13 high-bandwidth demos ESnet s 100G testbed Sample of results

More information

Improving Performance of 100G Data Transfer Nodes

Improving Performance of 100G Data Transfer Nodes Improving Performance of 100G Data Transfer Nodes Brian Tierney, Consultant Nate Hanford, ESnet bltierney@gmail.com http://fasterdata.es.net APAN, Singapore March 28, 2018 Recent TCP changes 2 Observation

More information

Zhengyang Liu University of Virginia. Oct 29, 2012

Zhengyang Liu University of Virginia. Oct 29, 2012 SDCI Net: Collaborative Research: An integrated study of datacenter networking and 100 GigE wide-area networking in support of distributed scientific computing Zhengyang Liu University of Virginia Oct

More information

10GE network tests with UDP. Janusz Szuba European XFEL

10GE network tests with UDP. Janusz Szuba European XFEL 10GE network tests with UDP Janusz Szuba European XFEL Outline 2 Overview of initial DAQ architecture Slice test hardware specification Initial networking test results DAQ software UDP tests Summary 10GE

More information

A System-Level Optimization Framework For High-Performance Networking. Thomas M. Benson Georgia Tech Research Institute

A System-Level Optimization Framework For High-Performance Networking. Thomas M. Benson Georgia Tech Research Institute A System-Level Optimization Framework For High-Performance Networking Thomas M. Benson Georgia Tech Research Institute thomas.benson@gtri.gatech.edu 1 Why do we need high-performance networking? Data flow

More information

UNIVERSITY OF CALIFORNIA

UNIVERSITY OF CALIFORNIA UNIVERSITY OF CALIFORNIA Evaluating switch buffers for high BDP flows (10G x 50ms) Michael Smitasin Network Engineer LBLnet Services Group Lawrence Berkeley National Laboratory Brian Tierney Staff Scientist

More information

Linux Network Tuning Guide for AMD EPYC Processor Based Servers

Linux Network Tuning Guide for AMD EPYC Processor Based Servers Linux Network Tuning Guide for AMD EPYC Processor Application Note Publication # 56224 Revision: 1.00 Issue Date: November 2017 Advanced Micro Devices 2017 Advanced Micro Devices, Inc. All rights reserved.

More information

Linux Network Tuning Guide for AMD EPYC Processor Based Servers

Linux Network Tuning Guide for AMD EPYC Processor Based Servers Linux Network Tuning Guide for AMD EPYC Processor Application Note Publication # 56224 Revision: 1.10 Issue Date: May 2018 Advanced Micro Devices 2018 Advanced Micro Devices, Inc. All rights reserved.

More information

IsoStack Highly Efficient Network Processing on Dedicated Cores

IsoStack Highly Efficient Network Processing on Dedicated Cores IsoStack Highly Efficient Network Processing on Dedicated Cores Leah Shalev Eran Borovik, Julian Satran, Muli Ben-Yehuda Outline Motivation IsoStack architecture Prototype TCP/IP over 10GE on a single

More information

Optimizing the GigE transfer What follows comes from company Pleora.

Optimizing the GigE transfer What follows comes from company Pleora. Optimizing the GigE transfer What follows comes from company Pleora. Selecting a NIC and Laptop Based on our testing, we recommend Intel NICs. In particular, we recommend the PRO 1000 line of Intel PCI

More information

ANI Testbed Project Update

ANI Testbed Project Update ANI Testbed Project Update Brian Tierney ESnet Feb 2, 2011 Testbed Overview Progression Start out as a tabletop testbed Move to Long Island MAN when dark fiber is available Extend to WAN when 100 Gbps

More information

Achieving the Science DMZ

Achieving the Science DMZ Achieving the Science DMZ Eli Dart, Brian Tierney and Eric Pouyoul, ESnet Raj Kettimuthu, ANL Jason Zurawski, Internet2 Guy Almes: Texas A&M TIP2013, Honolulu Hawaii, January, 2013 Section 2: Bulk Data

More information

Performance Optimisations for HPC workloads. August 2008 Imed Chihi

Performance Optimisations for HPC workloads. August 2008 Imed Chihi Performance Optimisations for HPC workloads August 2008 Imed Chihi Agenda The computing model The assignment problem CPU sets Priorities Disk IO optimisations gettimeofday() Disabling services Memory management

More information

Programmable Information Highway (with no Traffic Jams)

Programmable Information Highway (with no Traffic Jams) Programmable Information Highway (with no Traffic Jams) Inder Monga Energy Sciences Network Scientific Networking Division Lawrence Berkeley National Lab Exponential Growth ESnet Accepted Traffic: Jan

More information

Traffic engineering and GridFTP log analysis. Jan 17, 2013 Project web site:

Traffic engineering and GridFTP log analysis. Jan 17, 2013 Project web site: Traffic engineering and GridFTP log analysis Zhenzhen Yan, Z. Liu, M. Veeraraghavan University of Virginia mvee@virginia.edu Chris Tracy, Chin Guok ESnet ctracy@es.net Jan 17, 2013 Project web site: http://www.ece.virginia.edu/mv/research/doe09/index.html

More information

DTN End Host performance and tuning

DTN End Host performance and tuning DTN End Host performance and tuning 1 Gigabit Ethernet & NVME Disks Richard- Hughes Jones Senior Network Advisor, Office of the CTO GEANT AssociaOon - Cambridge Workshop: Moving My Data at High Speeds

More information

IBM POWER8 100 GigE Adapter Best Practices

IBM POWER8 100 GigE Adapter Best Practices Introduction IBM POWER8 100 GigE Adapter Best Practices With higher network speeds in new network adapters, achieving peak performance requires careful tuning of the adapters and workloads using them.

More information

Achieving the Science DMZ

Achieving the Science DMZ Achieving the Science DMZ Eli Dart, Brian Tierney and Eric Pouyoul, ESnet Raj Kettimuthu, ANL Jason Zurawski, Internet2 Guy Almes: Texas A&M TIP2013, Honolulu Hawaii, January, 2013 Section 2: Bulk Data

More information

FPGAs and Networking

FPGAs and Networking FPGAs and Networking Marc Kelly & Richard Hughes-Jones University of Manchester 12th July 27 1 Overview of Work Looking into the usage of FPGA's to directly connect to Ethernet for DAQ readout purposes.

More information

The Missing Piece of Virtualization. I/O Virtualization on 10 Gb Ethernet For Virtualized Data Centers

The Missing Piece of Virtualization. I/O Virtualization on 10 Gb Ethernet For Virtualized Data Centers The Missing Piece of Virtualization I/O Virtualization on 10 Gb Ethernet For Virtualized Data Centers Agenda 10 GbE Adapters Built for Virtualization I/O Throughput: Virtual & Non-Virtual Servers Case

More information

ntop Users Group Meeting

ntop Users Group Meeting ntop Users Group Meeting PF_RING Tutorial Alfredo Cardigliano Overview Introduction Installation Configuration Tuning Use cases PF_RING Open source packet processing framework for

More information

打造 Linux 下的高性能网络 北京酷锐达信息技术有限公司技术总监史应生.

打造 Linux 下的高性能网络 北京酷锐达信息技术有限公司技术总监史应生. 打造 Linux 下的高性能网络 北京酷锐达信息技术有限公司技术总监史应生 shiys@solutionware.com.cn BY DEFAULT, LINUX NETWORKING NOT TUNED FOR MAX PERFORMANCE, MORE FOR RELIABILITY Trade-off :Low Latency, throughput, determinism Performance

More information

Measuring a 25 Gb/s and 40 Gb/s data plane

Measuring a 25 Gb/s and 40 Gb/s data plane Measuring a 25 Gb/s and 40 Gb/s data plane Christo Kleu Pervaze Akhtar 1 Contents Preliminaries Equipment Traffic generators Test topologies Host and VM configuration NUMA Architecture CPU allocation BIOS

More information

Data transfer over the wide area network with a large round trip time

Data transfer over the wide area network with a large round trip time Journal of Physics: Conference Series Data transfer over the wide area network with a large round trip time To cite this article: H Matsunaga et al 1 J. Phys.: Conf. Ser. 219 656 Recent citations - A two

More information

OneCore Storage Performance Tuning

OneCore Storage Performance Tuning OneCore Storage Performance Tuning Overview To improve Emulex adapter performance while using the OneCore Storage Linux drivers in a multi-core CPU environment, multiple performance tuning features can

More information

Tales of the Tail Hardware, OS, and Application-level Sources of Tail Latency

Tales of the Tail Hardware, OS, and Application-level Sources of Tail Latency Tales of the Tail Hardware, OS, and Application-level Sources of Tail Latency Jialin Li, Naveen Kr. Sharma, Dan R. K. Ports and Steven D. Gribble February 2, 2015 1 Introduction What is Tail Latency? What

More information

Agilio CX 2x40GbE with OVS-TC

Agilio CX 2x40GbE with OVS-TC PERFORMANCE REPORT Agilio CX 2x4GbE with OVS-TC OVS-TC WITH AN AGILIO CX SMARTNIC CAN IMPROVE A SIMPLE L2 FORWARDING USE CASE AT LEAST 2X. WHEN SCALED TO REAL LIFE USE CASES WITH COMPLEX RULES TUNNELING

More information

Traffic Characteristics of Bulk Data Transfer using TCP/IP over Gigabit Ethernet

Traffic Characteristics of Bulk Data Transfer using TCP/IP over Gigabit Ethernet Traffic Characteristics of Bulk Data Transfer using TCP/IP over Gigabit Ethernet Aamir Shaikh and Kenneth J. Christensen Department of Computer Science and Engineering University of South Florida Tampa,

More information

NFS on the Fast track - fine tuning and futures

NFS on the Fast track - fine tuning and futures NFS on the Fast track - fine tuning and futures Bikash Roy Choudhury Solutions Architect, NetApp Agenda Overview of NFS layers Linux Client Why is the NFS performance Slow? Understanding application behavior

More information

An Intelligent NIC Design Xin Song

An Intelligent NIC Design Xin Song 2nd International Conference on Advances in Mechanical Engineering and Industrial Informatics (AMEII 2016) An Intelligent NIC Design Xin Song School of Electronic and Information Engineering Tianjin Vocational

More information

Steven Carter. Network Lead, NCCS Oak Ridge National Laboratory OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 1

Steven Carter. Network Lead, NCCS Oak Ridge National Laboratory OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 1 Networking the National Leadership Computing Facility Steven Carter Network Lead, NCCS Oak Ridge National Laboratory scarter@ornl.gov 1 Outline Introduction NCCS Network Infrastructure Cray Architecture

More information

Application Acceleration Beyond Flash Storage

Application Acceleration Beyond Flash Storage Application Acceleration Beyond Flash Storage Session 303C Mellanox Technologies Flash Memory Summit July 2014 Accelerating Applications, Step-by-Step First Steps Make compute fast Moore s Law Make storage

More information

Lighting the Blue Touchpaper for UK e-science - Closing Conference of ESLEA Project The George Hotel, Edinburgh, UK March, 2007

Lighting the Blue Touchpaper for UK e-science - Closing Conference of ESLEA Project The George Hotel, Edinburgh, UK March, 2007 Working with 1 Gigabit Ethernet 1, The School of Physics and Astronomy, The University of Manchester, Manchester, M13 9PL UK E-mail: R.Hughes-Jones@manchester.ac.uk Stephen Kershaw The School of Physics

More information

Optimizing Performance: Intel Network Adapters User Guide

Optimizing Performance: Intel Network Adapters User Guide Optimizing Performance: Intel Network Adapters User Guide Network Optimization Types When optimizing network adapter parameters (NIC), the user typically considers one of the following three conditions

More information

HKG net_mdev: Fast-path userspace I/O. Ilias Apalodimas Mykyta Iziumtsev François-Frédéric Ozog

HKG net_mdev: Fast-path userspace I/O. Ilias Apalodimas Mykyta Iziumtsev François-Frédéric Ozog HKG18-110 net_mdev: Fast-path userspace I/O Ilias Apalodimas Mykyta Iziumtsev François-Frédéric Ozog Why userland I/O Time sensitive networking Developed mostly for Industrial IOT, automotive and audio/video

More information

Fast packet processing in the cloud. Dániel Géhberger Ericsson Research

Fast packet processing in the cloud. Dániel Géhberger Ericsson Research Fast packet processing in the cloud Dániel Géhberger Ericsson Research Outline Motivation Service chains Hardware related topics, acceleration Virtualization basics Software performance and acceleration

More information

Nova Scheduler: Optimizing, Configuring and Deploying NFV VNF's on OpenStack

Nova Scheduler: Optimizing, Configuring and Deploying NFV VNF's on OpenStack Nova Scheduler: Optimizing, Configuring and Deploying NFV VNF's on OpenStack Ian Jolliffe, Chris Friesen WHEN IT MATTERS, IT RUNS ON WIND RIVER. 2017 WIND RIVER. ALL RIGHTS RESERVED. Ian Jolliffe 2 2017

More information

MiAMI: Multi-Core Aware Processor Affinity for TCP/IP over Multiple Network Interfaces

MiAMI: Multi-Core Aware Processor Affinity for TCP/IP over Multiple Network Interfaces MiAMI: Multi-Core Aware Processor Affinity for TCP/IP over Multiple Network Interfaces Hye-Churn Jang Hyun-Wook (Jin) Jin Department of Computer Science and Engineering Konkuk University Seoul, Korea {comfact,

More information

Linux Kernel Hacking Free Course

Linux Kernel Hacking Free Course Linux Kernel Hacking Free Course 3 rd edition G.Grilli, University of me Tor Vergata IRQ DISTRIBUTION IN MULTIPROCESSOR SYSTEMS April 05, 2006 IRQ distribution in multiprocessor systems 1 Contents: What

More information

DPDK Performance Report Release Test Date: Nov 16 th 2016

DPDK Performance Report Release Test Date: Nov 16 th 2016 Test Date: Nov 16 th 2016 Revision History Date Revision Comment Nov 16 th, 2016 1.0 Initial document for release 2 Contents Audience and Purpose... 4 Test setup:... 4 Intel Xeon Processor E5-2699 v4 (55M

More information

Advanced Computer Networks. End Host Optimization

Advanced Computer Networks. End Host Optimization Oriana Riva, Department of Computer Science ETH Zürich 263 3501 00 End Host Optimization Patrick Stuedi Spring Semester 2017 1 Today End-host optimizations: NUMA-aware networking Kernel-bypass Remote Direct

More information

The Power of Batching in the Click Modular Router

The Power of Batching in the Click Modular Router The Power of Batching in the Click Modular Router Joongi Kim, Seonggu Huh, Keon Jang, * KyoungSoo Park, Sue Moon Computer Science Dept., KAIST Microsoft Research Cambridge, UK * Electrical Engineering

More information

Netchannel 2: Optimizing Network Performance

Netchannel 2: Optimizing Network Performance Netchannel 2: Optimizing Network Performance J. Renato Santos +, G. (John) Janakiraman + Yoshio Turner +, Ian Pratt * + HP Labs - * XenSource/Citrix Xen Summit Nov 14-16, 2007 2003 Hewlett-Packard Development

More information

High bandwidth, Long distance. Where is my throughput? Robin Tasker CCLRC, Daresbury Laboratory, UK

High bandwidth, Long distance. Where is my throughput? Robin Tasker CCLRC, Daresbury Laboratory, UK High bandwidth, Long distance. Where is my throughput? Robin Tasker CCLRC, Daresbury Laboratory, UK [r.tasker@dl.ac.uk] DataTAG is a project sponsored by the European Commission - EU Grant IST-2001-32459

More information

ClearStream. Prototyping 40 Gbps Transparent End-to-End Connectivity. Cosmin Dumitru! Ralph Koning! Cees de Laat! and many others (see posters)!

ClearStream. Prototyping 40 Gbps Transparent End-to-End Connectivity. Cosmin Dumitru! Ralph Koning! Cees de Laat! and many others (see posters)! ClearStream Prototyping 40 Gbps Transparent End-to-End Connectivity Cosmin Dumitru! Ralph Koning! Cees de Laat! and many others (see posters)! University of Amsterdam! more data! Speed! Volume! Internet!

More information

Interrupt Swizzling Solution for Intel 5000 Chipset Series based Platforms

Interrupt Swizzling Solution for Intel 5000 Chipset Series based Platforms Interrupt Swizzling Solution for Intel 5000 Chipset Series based Platforms Application Note August 2006 Document Number: 314337-002 Notice: This document contains information on products in the design

More information

The latency of user-to-user, kernel-to-kernel and interrupt-to-interrupt level communication

The latency of user-to-user, kernel-to-kernel and interrupt-to-interrupt level communication The latency of user-to-user, kernel-to-kernel and interrupt-to-interrupt level communication John Markus Bjørndalen, Otto J. Anshus, Brian Vinter, Tore Larsen Department of Computer Science University

More information

CIS Test 1- Practice - Fall 2011

CIS Test 1- Practice - Fall 2011 CIS 192 - Test 1- Practice - Fall 2011 Name Each question worth 2 points: Tip: When not logged in as root, you can still use many of the network commands but you must preface them with /sbin/ because you

More information

Data Path acceleration techniques in a NFV world

Data Path acceleration techniques in a NFV world Data Path acceleration techniques in a NFV world Mohanraj Venkatachalam, Purnendu Ghosh Abstract NFV is a revolutionary approach offering greater flexibility and scalability in the deployment of virtual

More information

Using (Suricata over) PF_RING for NIC-Independent Acceleration

Using (Suricata over) PF_RING for NIC-Independent Acceleration Using (Suricata over) PF_RING for NIC-Independent Acceleration Luca Deri Alfredo Cardigliano Outlook About ntop. Introduction to PF_RING. Integrating PF_RING with

More information

NTRDMA v0.1. An Open Source Driver for PCIe NTB and DMA. Allen Hubbe at Linux Piter 2015 NTRDMA. Messaging App. IB Verbs. dmaengine.h ntb.

NTRDMA v0.1. An Open Source Driver for PCIe NTB and DMA. Allen Hubbe at Linux Piter 2015 NTRDMA. Messaging App. IB Verbs. dmaengine.h ntb. Messaging App IB Verbs NTRDMA dmaengine.h ntb.h DMA DMA DMA NTRDMA v0.1 An Open Source Driver for PCIe and DMA Allen Hubbe at Linux Piter 2015 1 INTRODUCTION Allen Hubbe Senior Software Engineer EMC Corporation

More information

An FPGA-Based Optical IOH Architecture for Embedded System

An FPGA-Based Optical IOH Architecture for Embedded System An FPGA-Based Optical IOH Architecture for Embedded System Saravana.S Assistant Professor, Bharath University, Chennai 600073, India Abstract Data traffic has tremendously increased and is still increasing

More information

NIC-PCIE-4RJ45-PLU PCI Express x4 Quad Port Copper Gigabit Server Adapter (Intel I350 Based)

NIC-PCIE-4RJ45-PLU PCI Express x4 Quad Port Copper Gigabit Server Adapter (Intel I350 Based) NIC-PCIE-4RJ45-PLU PCI Express x4 Quad Port Copper Gigabit Server Adapter (Intel I350 Based) Quad-port Gigabit Ethernet server adapters designed with performance enhancing features and new power management

More information

Network Test and Monitoring Tools

Network Test and Monitoring Tools ajgillette.com Technical Note Network Test and Monitoring Tools Author: A.J.Gillette Date: December 6, 2012 Revision: 1.3 Table of Contents Network Test and Monitoring Tools...1 Introduction...3 Link Characterization...4

More information

OpenFlow Software Switch & Intel DPDK. performance analysis

OpenFlow Software Switch & Intel DPDK. performance analysis OpenFlow Software Switch & Intel DPDK performance analysis Agenda Background Intel DPDK OpenFlow 1.3 implementation sketch Prototype design and setup Results Future work, optimization ideas OF 1.3 prototype

More information

PEARL. Programmable Virtual Router Platform Enabling Future Internet Innovation

PEARL. Programmable Virtual Router Platform Enabling Future Internet Innovation PEARL Programmable Virtual Router Platform Enabling Future Internet Innovation Hongtao Guan Ph.D., Assistant Professor Network Technology Research Center Institute of Computing Technology, Chinese Academy

More information

D1.1 Server Scalibility

D1.1 Server Scalibility D1.1 Server Scalibility Ronald van der Pol and Freek Dijkstra SARA Computing & Networking Services, Science Park 121, 1098 XG Amsterdam, The Netherlands March 2010 ronald.vanderpol@sara.nl,freek.dijkstra@sara.nl

More information

46PaQ. Dimitris Miras, Saleem Bhatti, Peter Kirstein Networks Research Group Computer Science UCL. 46PaQ AHM 2005 UKLIGHT Workshop, 19 Sep

46PaQ. Dimitris Miras, Saleem Bhatti, Peter Kirstein Networks Research Group Computer Science UCL. 46PaQ AHM 2005 UKLIGHT Workshop, 19 Sep 46PaQ Dimitris Miras, Saleem Bhatti, Peter Kirstein Networks Research Group Computer Science UCL 46PaQ AHM 2005 UKLIGHT Workshop, 19 Sep 2005 1 Today s talk Overview Current Status and Results Future Work

More information

High-Performance TCP Tips and Tricks

High-Performance TCP Tips and Tricks High-Performance TCP Tips and Tricks TNC 16 June 12, 2015 Brian Tierney, ESnet blgerney@es.net A small amount of packet loss makes a huge difference in TCP performance Local (LAN) Metro Area With loss,

More information

DPDK Intel NIC Performance Report Release 18.02

DPDK Intel NIC Performance Report Release 18.02 DPDK Intel NIC Performance Report Test Date: Mar 14th 2018 Author: Intel DPDK Validation team Revision History Date Revision Comment Mar 15th, 2018 1.0 Initial document for release 2 Contents Audience

More information

Network and Host Design to Facilitate High Performance Data Transfer

Network and Host Design to Facilitate High Performance Data Transfer Network and Host Design to Facilitate High Performance Data Transfer Jason Zurawski - ESnet Engineering & Outreach engage@es.net globusworld 2014 April 15 th 2014 With contributions from S. Balasubramanian,

More information

DXE-810S. Manual. 10 Gigabit PCI-EXPRESS-Express Ethernet Network Adapter V1.01

DXE-810S. Manual. 10 Gigabit PCI-EXPRESS-Express Ethernet Network Adapter V1.01 DXE-810S 10 Gigabit PCI-EXPRESS-Express Ethernet Network Adapter Manual V1.01 Table of Contents INTRODUCTION... 1 System Requirements... 1 Features... 1 INSTALLATION... 2 Unpack and Inspect... 2 Software

More information

Accelerating 4G Network Performance

Accelerating 4G Network Performance WHITE PAPER Accelerating 4G Network Performance OFFLOADING VIRTUALIZED EPC TRAFFIC ON AN OVS-ENABLED NETRONOME SMARTNIC NETRONOME AGILIO SMARTNICS PROVIDE A 5X INCREASE IN vepc BANDWIDTH ON THE SAME NUMBER

More information

Xilinx Answer QDMA Performance Report

Xilinx Answer QDMA Performance Report Xilinx Answer 71453 QDMA Performance Report Important Note: This downloadable PDF of an Answer Record is provided to enhance its usability and readability. It is important to note that Answer Records are

More information

Analysis of CPU Pinning and Storage Configuration in 100 Gbps Network Data Transfer

Analysis of CPU Pinning and Storage Configuration in 100 Gbps Network Data Transfer Analysis of CPU Pinning and Storage Configuration in 100 Gbps Network Data Transfer International Center for Advanced Internet Research Northwestern University Se-young Yu Jim Chen, Joe Mambretti, Fei

More information

Networking Servers made for BSD and Linux systems

Networking Servers made for BSD and Linux systems Networking Servers made for BSD and Linux systems presents NETMAP L-800 high-end 1U rack networking server for mission critical operations ServerU Netmap L-800 is our best offer for an embedded network-centric

More information

Optimized Distributed Data Sharing Substrate in Multi-Core Commodity Clusters: A Comprehensive Study with Applications

Optimized Distributed Data Sharing Substrate in Multi-Core Commodity Clusters: A Comprehensive Study with Applications Optimized Distributed Data Sharing Substrate in Multi-Core Commodity Clusters: A Comprehensive Study with Applications K. Vaidyanathan, P. Lai, S. Narravula and D. K. Panda Network Based Computing Laboratory

More information

DPDK Intel NIC Performance Report Release 18.05

DPDK Intel NIC Performance Report Release 18.05 DPDK Intel NIC Performance Report Test Date: Jun 1th 2018 Author: Intel DPDK Validation team Revision History Date Revision Comment Jun 4th, 2018 1.0 Initial document for release 2 Contents Audience and

More information

May 1, Foundation for Research and Technology - Hellas (FORTH) Institute of Computer Science (ICS) A Sleep-based Communication Mechanism to

May 1, Foundation for Research and Technology - Hellas (FORTH) Institute of Computer Science (ICS) A Sleep-based Communication Mechanism to A Sleep-based Our Akram Foundation for Research and Technology - Hellas (FORTH) Institute of Computer Science (ICS) May 1, 2011 Our 1 2 Our 3 4 5 6 Our Efficiency in Back-end Processing Efficiency in back-end

More information

DPDK Vhost/Virtio Performance Report Release 18.11

DPDK Vhost/Virtio Performance Report Release 18.11 DPDK Vhost/Virtio Performance Report Test Date: December 3st 2018 Author: Intel DPDK Validation Team Revision History Date Revision Comment December 3st, 2018 1.0 Initial document for release 2 Contents

More information

TCP Tuning Domenico Vicinanza DANTE, Cambridge, UK

TCP Tuning Domenico Vicinanza DANTE, Cambridge, UK TCP Tuning Domenico Vicinanza DANTE, Cambridge, UK domenico.vicinanza@dante.net EGI Technical Forum 2013, Madrid, Spain TCP! Transmission Control Protocol (TCP)! One of the original core protocols of the

More information

DPDK Intel NIC Performance Report Release 17.08

DPDK Intel NIC Performance Report Release 17.08 DPDK Intel NIC Performance Report Test Date: Aug 23th 2017 Author: Intel DPDK Validation team Revision History Date Revision Comment Aug 24th, 2017 1.0 Initial document for release 2 Contents Audience

More information

perfsonar Host Hardware

perfsonar Host Hardware perfsonar Host Hardware This document is a result of work by the perfsonar Project (http://www.perfsonar.net) and is licensed under CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/). Event

More information

Ultra high-speed transmission technology for wide area data movement

Ultra high-speed transmission technology for wide area data movement Ultra high-speed transmission technology for wide area data movement Michelle Munson, president & co-founder Aspera Outline Business motivation Moving ever larger file sets over commodity IP networks (public,

More information

Performance Evaluation of Myrinet-based Network Router

Performance Evaluation of Myrinet-based Network Router Performance Evaluation of Myrinet-based Network Router Information and Communications University 2001. 1. 16 Chansu Yu, Younghee Lee, Ben Lee Contents Suez : Cluster-based Router Suez Implementation Implementation

More information

Mellanox NIC s Performance Report with DPDK Rev 1.0

Mellanox NIC s Performance Report with DPDK Rev 1.0 Mellanox NIC s Performance Report with DPDK 18.02 Rev 1.0 www.mellanox.com Mellanox Technologies Copyright 2018. Mellanox Technologies Ltd. All Rights Reserved. Mellanox, Mellanox logo, Accelio, BridgeX,

More information

Operating System: Chap13 I/O Systems. National Tsing-Hua University 2016, Fall Semester

Operating System: Chap13 I/O Systems. National Tsing-Hua University 2016, Fall Semester Operating System: Chap13 I/O Systems National Tsing-Hua University 2016, Fall Semester Outline Overview I/O Hardware I/O Methods Kernel I/O Subsystem Performance Application Interface Operating System

More information

QuickSpecs. HP Z 10GbE Dual Port Module. Models

QuickSpecs. HP Z 10GbE Dual Port Module. Models Overview Models Part Number: 1Ql49AA Introduction The is a 10GBASE-T adapter utilizing the Intel X722 MAC and X557-AT2 PHY pairing to deliver full line-rate performance, utilizing CAT 6A UTP cabling (or

More information

Managing Network Adapters

Managing Network Adapters Managing Network Adapters This chapter includes the following sections: Overview of the Cisco UCS C-Series Network Adapters, page 1 Viewing Network Adapter Properties, page 3 Configuring Network Adapter

More information

Large Receive Offload implementation in Neterion 10GbE Ethernet driver

Large Receive Offload implementation in Neterion 10GbE Ethernet driver Large Receive Offload implementation in Neterion 10GbE Ethernet driver Leonid Grossman Neterion, Inc. leonid@neterion.com Abstract 1 Introduction The benefits of TSO (Transmit Side Offload) implementation

More information

NIC TEAMING IEEE 802.3ad

NIC TEAMING IEEE 802.3ad WHITE PAPER NIC TEAMING IEEE 802.3ad NIC Teaming IEEE 802.3ad Summary This tech note describes the NIC (Network Interface Card) teaming capabilities of VMware ESX Server 2 including its benefits, performance

More information

Intel PRO/1000 PT and PF Quad Port Bypass Server Adapters for In-line Server Appliances

Intel PRO/1000 PT and PF Quad Port Bypass Server Adapters for In-line Server Appliances Technology Brief Intel PRO/1000 PT and PF Quad Port Bypass Server Adapters for In-line Server Appliances Intel PRO/1000 PT and PF Quad Port Bypass Server Adapters for In-line Server Appliances The world

More information

perfsonar Deployment on ESnet

perfsonar Deployment on ESnet perfsonar Deployment on ESnet Brian Tierney ESnet ISMA 2011 AIMS-3 Workshop on Active Internet Measurements Feb 9, 2011 Why does the Network seem so slow? Where are common problems? Source Campus Congested

More information

DYNES: DYnamic NEtwork System

DYNES: DYnamic NEtwork System DYNES: DYnamic NEtwork System Artur Barczyk California Institute of Technology / US LHCNet TERENA e2e Workshop Prague, November 29 th, 2010 1 OUTLINE What is DYNES Status Deployment Plan 2 DYNES Overview

More information

Improving DPDK Performance

Improving DPDK Performance Improving DPDK Performance Data Plane Development Kit (DPDK) was pioneered by Intel as a way to boost the speed of packet API with standard hardware. DPDK-enabled applications typically show four or more

More information

Research on DPDK Based High-Speed Network Traffic Analysis. Zihao Wang Network & Information Center Shanghai Jiao Tong University

Research on DPDK Based High-Speed Network Traffic Analysis. Zihao Wang Network & Information Center Shanghai Jiao Tong University Research on DPDK Based High-Speed Network Traffic Analysis Zihao Wang Network & Information Center Shanghai Jiao Tong University Outline 1 Background 2 Overview 3 DPDK Based Traffic Analysis 4 Experiment

More information

Improving Cluster Performance

Improving Cluster Performance Improving Cluster Performance Service Offloading Larger clusters may need to have special purpose node(s) to run services to prevent slowdown due to contention (e.g. NFS, DNS, login, compilation) In cluster

More information

NFS/RDMA over 40Gbps iwarp Wael Noureddine Chelsio Communications

NFS/RDMA over 40Gbps iwarp Wael Noureddine Chelsio Communications NFS/RDMA over 40Gbps iwarp Wael Noureddine Chelsio Communications Outline RDMA Motivating trends iwarp NFS over RDMA Overview Chelsio T5 support Performance results 2 Adoption Rate of 40GbE Source: Crehan

More information

Performance Enhancement for IPsec Processing on Multi-Core Systems

Performance Enhancement for IPsec Processing on Multi-Core Systems Performance Enhancement for IPsec Processing on Multi-Core Systems Sandeep Malik Freescale Semiconductor India Pvt. Ltd IDC Noida, India Ravi Malhotra Freescale Semiconductor India Pvt. Ltd IDC Noida,

More information

Chapter 4. Routers with Tiny Buffers: Experiments. 4.1 Testbed experiments Setup

Chapter 4. Routers with Tiny Buffers: Experiments. 4.1 Testbed experiments Setup Chapter 4 Routers with Tiny Buffers: Experiments This chapter describes two sets of experiments with tiny buffers in networks: one in a testbed and the other in a real network over the Internet2 1 backbone.

More information

PCI Express x8 Single Port SFP+ 10 Gigabit Server Adapter (Intel 82599ES Based) Single-Port 10 Gigabit SFP+ Ethernet Server Adapters Provide Ultimate

PCI Express x8 Single Port SFP+ 10 Gigabit Server Adapter (Intel 82599ES Based) Single-Port 10 Gigabit SFP+ Ethernet Server Adapters Provide Ultimate NIC-PCIE-1SFP+-PLU PCI Express x8 Single Port SFP+ 10 Gigabit Server Adapter (Intel 82599ES Based) Single-Port 10 Gigabit SFP+ Ethernet Server Adapters Provide Ultimate Flexibility and Scalability in Virtual

More information

TCP Tuning for the Web

TCP Tuning for the Web TCP Tuning for the Web Jason Cook - @macros - jason@fastly.com Me Co-founder and Operations at Fastly Former Operations Engineer at Wikia Lots of Sysadmin and Linux consulting The Goal Make the best use

More information

Internet data transfer record between CERN and California. Sylvain Ravot (Caltech) Paolo Moroni (CERN)

Internet data transfer record between CERN and California. Sylvain Ravot (Caltech) Paolo Moroni (CERN) Internet data transfer record between CERN and California Sylvain Ravot (Caltech) Paolo Moroni (CERN) Summary Internet2 Land Speed Record Contest New LSR DataTAG project and network configuration Establishing

More information

Much Faster Networking

Much Faster Networking Much Faster Networking David Riddoch driddoch@solarflare.com Copyright 2016 Solarflare Communications, Inc. All rights reserved. What is kernel bypass? The standard receive path The standard receive path

More information

Tuning Intelligent Data Lake Performance

Tuning Intelligent Data Lake Performance Tuning Intelligent Data Lake Performance 2016 Informatica LLC. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without

More information

NetFPGA Hardware Architecture

NetFPGA Hardware Architecture NetFPGA Hardware Architecture Jeffrey Shafer Some slides adapted from Stanford NetFPGA tutorials NetFPGA http://netfpga.org 2 NetFPGA Components Virtex-II Pro 5 FPGA 53,136 logic cells 4,176 Kbit block

More information

DPDK Vhost/Virtio Performance Report Release 18.05

DPDK Vhost/Virtio Performance Report Release 18.05 DPDK Vhost/Virtio Performance Report Test Date: Jun 1 2018 Author: Intel DPDK Validation Team Revision History Date Revision Comment Jun 1st, 2018 1.0 Initial document for release 2 Release 18.02 Contents

More information

TCP Wirespeed Verify TCP Throughput up to 10G Using the T-BERD /MTS-6000A Multi- Services Application Module (MSAM)

TCP Wirespeed Verify TCP Throughput up to 10G Using the T-BERD /MTS-6000A Multi- Services Application Module (MSAM) Ordering Information: Use Case: TCP Wirespeed Verify TCP Throughput up to 10G Using the T-BERD /MTS-6000A Multi- Services Application Module (MSAM) CTLAYER4 10M to 1GigE TCP Wirespeed CT10GLAYER4 10G TCP

More information

Low-Latency Datacenters. John Ousterhout Platform Lab Retreat May 29, 2015

Low-Latency Datacenters. John Ousterhout Platform Lab Retreat May 29, 2015 Low-Latency Datacenters John Ousterhout Platform Lab Retreat May 29, 2015 Datacenters: Scale and Latency Scale: 1M+ cores 1-10 PB memory 200 PB disk storage Latency: < 0.5 µs speed-of-light delay Most

More information

Total Cost of Ownership Analysis for a Wireless Access Gateway

Total Cost of Ownership Analysis for a Wireless Access Gateway white paper Communications Service Providers TCO Analysis Total Cost of Ownership Analysis for a Wireless Access Gateway An analysis of the total cost of ownership of a wireless access gateway running

More information