Object Storage: Redefining Bandwidth for Linux Clusters

Size: px
Start display at page:

Download "Object Storage: Redefining Bandwidth for Linux Clusters"

Transcription

1 Object Storage: Redefining Bandwidth for Linux Clusters Brent Welch Principal Architect, Inc. November 18, 2003

2 Blocks, Files and Objects Block-base architecture: fast but private Traditional SCSI and FC approaches Expensive fabric, difficult to share between hosts File-based architecture: sharable, but bottlenecked performance NAS storage (NFS, CIFS, AFS and DFS) Filer CPU and memory system between clients and disks Object-based architecture: fast and sharable Storage nodes directly accessible by clients via GbE Out-of-band metadata servers make policy decisions for a file system Storage nodes enforce access control to allow safe sharing Page 2

3 Object Storage System Architecture Moves low-level storage functions into the storage device itself Key Object Storage Features Intelligent space management in storage layer Media geometry aware placement Late binding allocation Data aware prefetching, caching & recovery Encapsulation of data and attributes Native object interface, good programming model Storage interpreted attributes for per file properties Key Object Storage Advantages Robust, shared access by many clients Scalable performance via an offloaded data path Strong fine-grained end-to-end security Page 3

4 What is an Object? Object Comprised of: User Data Attributes Interface: ID <dev#,grp#,obj#> Read/Write Create/Delete Getattr/Setattr Capability-based File Component: Stripe files across component objects Page 4

5 Scalability: Capacity Balanced storage node CPU, SDRAM, GE NIC and 2 spindles Commodity parts drive low cost Drive linear performance gains Simply add StorageBlades Single Seamless Namespace! Page 5

6 Scalability: Management Single filesystem namespace ActiveScale Architecture Removes physical & logical boundaries Dynamic load-balancing Interoperability Gateway for NFS/CIFS Free clustered NAS Internal cluster management Single Global Namespace Fault tolerance Environmental/thermal monitoring Software upgrades Service and Support Personalized extranet for bugs, SRs, orders Eng. Developers Marketing Eng. QA Page 6

7 Scalability: Metadata Scaling Block-level metadata controlled by Storage Blades (OSDs) Client caching with callbacks to reduce load for file-level metadata Clustered servers (Director Blades) with active/active failover Metadata provides file system semantics over objects Chunk ownership over collections of files and directories For really large directories, hash into different collections Store metadata with the objects on storage nodes Page 7

8 Bandwidth Sustained Throughput 60 seconds, N clients to N files 1 Client, 10 OSDs: 95 MB/s read, 77 MB/s write 10 Clients, 10 OSDs: 415 MB/s read, 335 MB/s write 151 Clients, 299 OSDs: MB/s read Barrier synchronized 1 TB move (MPI IO min time) 151 Clients, 299 OSDs: N to N, 7486 MB/s read, 6506 MB/s write 151 Clients, 198 OSDs: 2775 MB/s concurrent write to one file Clients are mostly 2.4 GHz uni-processors Large tests had a mix, some duals, some faster Page 8

9 GE Networking GE NICs part of commodity-based storage and compute clusters Cluster-specific interconnects optimized for bandwidth, latency Storage interconnects optimized for cost, longevity Cluster nodes get devoted to being I/O routers Multiprotocol switch: bridge inside cluster switch Eliminates I/O node, two switch ports Page 9

10 Object Storage Acceptance Life Science Gov t Science Oil and Gas Los Alamos Labs buying up to 620 TB through FY04 Business Objective 5X capability at 10% the cost of today s system Requirements Linux commodity cluster 100+ Teraflops Throughput Goal: 1 GB/sec per Teraflop = 100 GB/Sec Object storage testing at scale 120 TB storage installed Option to buy up to 500 TB in FY04 Page 10

11 The premier storage system for scalable Linux clusters November 18, 2003

12 Fine Grain Access Enforcement State of art is VPN of all out-of-band clients, all sharable data and metadata Accident prone & vulnerable to subverted client; analogy to single-address space computing Private Communication File Manager StorageBlade Integrity/Privacy Secret Key 2: CapArgs, CapKey 1: Request for access Object Storage uses digitally signed, object-specific capabilities on each request CapKey = MAC SecretKey (CapArgs) CapArgs = ObjID, Version, Rights, Expiry, ReqMAC = MAC CapKey (Req, Nonceln) Client 3: CapArgs, Req, Nonceln, ReqMAC NASD 4: Reply, NonceOut, ReplyMAC Secret Key ReplyMAC = MAC CapKey (Reply,NonceOut) Page 12

13 Objects: Performance & Scalability Breakthrough Data Throughput AND Random I/O Random I/O Data Throughput 32-shelf, 600 spindles: 305,805 SFS ops/sec, 10 GB/sec Page 13

14 Standardization Timeline CMU NASD Lustre NSIC NASD OSD market T10/SNIA OSD SNIA TWG is nearing completion of proposed OSD standard Great participation by leading storage industry vendors ANSI X3 T10 V1 standard should be in review November 03 Next step for the OSD spec is under development Roadmap includes SMIS support & Information Life Cycle management Page 14

15 Ease of Management Problem: Management is 80% of Storage TCO Multiple physical & logical management sets Ongoing adjustments to maintain efficiency Security breaches System backup, downtime and recovery 80% Initial Purchase cost Ongoing Management costs redefines Appliance-like simplicity Single Namespace Dynamic Load Balancing Quality of Service Page 15

NASD: Network-Attached Secure Disks

NASD: Network-Attached Secure Disks NASD: Network-Attached Secure Disks Garth Gibson garth.gibson@cmu.edu also David Nagle, Khalil Amiri,Jeff Butler, Howard Gobioff, Charles Hardin, Nat Lanza, Erik Riedel, David Rochberg, Chris Sabol, Marc

More information

Storage Aggregation for Performance & Availability:

Storage Aggregation for Performance & Availability: Storage Aggregation for Performance & Availability: The Path from Physical RAID to Virtual Objects Univ Minnesota, Third Intelligent Storage Workshop Garth Gibson CTO, Panasas, and Assoc Prof, CMU ggibson@panasas.com

More information

Got Isilon? Need IOPS? Get Avere.

Got Isilon? Need IOPS? Get Avere. Got Isilon? Need IOPS? Get Avere. Scalable I/O Performance to Complement Any EMC Isilon Environment By: Jeff Tabor, Director of Product Marketing Achieving Performance Scaling Overcoming Random I/O and

More information

RAIDIX Data Storage Solution. Clustered Data Storage Based on the RAIDIX Software and GPFS File System

RAIDIX Data Storage Solution. Clustered Data Storage Based on the RAIDIX Software and GPFS File System RAIDIX Data Storage Solution Clustered Data Storage Based on the RAIDIX Software and GPFS File System 2017 Contents Synopsis... 2 Introduction... 3 Challenges and the Solution... 4 Solution Architecture...

More information

iscsi Technology: A Convergence of Networking and Storage

iscsi Technology: A Convergence of Networking and Storage HP Industry Standard Servers April 2003 iscsi Technology: A Convergence of Networking and Storage technology brief TC030402TB Table of Contents Abstract... 2 Introduction... 2 The Changing Storage Environment...

More information

Parallel File Systems for HPC

Parallel File Systems for HPC Introduction to Scuola Internazionale Superiore di Studi Avanzati Trieste November 2008 Advanced School in High Performance and Grid Computing Outline 1 The Need for 2 The File System 3 Cluster & A typical

More information

The Oracle Database Appliance I/O and Performance Architecture

The Oracle Database Appliance I/O and Performance Architecture Simple Reliable Affordable The Oracle Database Appliance I/O and Performance Architecture Tammy Bednar, Sr. Principal Product Manager, ODA 1 Copyright 2012, Oracle and/or its affiliates. All rights reserved.

More information

Introducing Panasas ActiveStor 14

Introducing Panasas ActiveStor 14 Introducing Panasas ActiveStor 14 SUPERIOR PERFORMANCE FOR MIXED FILE SIZE ENVIRONMENTS DEREK BURKE, PANASAS EUROPE INTRODUCTION TO PANASAS Storage that accelerates the world s highest performance and

More information

CA485 Ray Walshe Google File System

CA485 Ray Walshe Google File System Google File System Overview Google File System is scalable, distributed file system on inexpensive commodity hardware that provides: Fault Tolerance File system runs on hundreds or thousands of storage

More information

Isilon Performance. Name

Isilon Performance. Name 1 Isilon Performance Name 2 Agenda Architecture Overview Next Generation Hardware Performance Caching Performance Streaming Reads Performance Tuning OneFS Architecture Overview Copyright 2014 EMC Corporation.

More information

The UnAppliance provides Higher Performance, Lower Cost File Serving

The UnAppliance provides Higher Performance, Lower Cost File Serving The UnAppliance provides Higher Performance, Lower Cost File Serving The UnAppliance is an alternative to traditional NAS solutions using industry standard servers and storage for a more efficient and

More information

Lustre A Platform for Intelligent Scale-Out Storage

Lustre A Platform for Intelligent Scale-Out Storage Lustre A Platform for Intelligent Scale-Out Storage Rumi Zahir, rumi. May 2003 rumi.zahir@intel.com Agenda Problem Statement Trends & Current Data Center Storage Architectures The Lustre File System Project

More information

HPC File Systems and Storage. Irena Johnson University of Notre Dame Center for Research Computing

HPC File Systems and Storage. Irena Johnson University of Notre Dame Center for Research Computing HPC File Systems and Storage Irena Johnson University of Notre Dame Center for Research Computing HPC (High Performance Computing) Aggregating computer power for higher performance than that of a typical

More information

Isilon Scale Out NAS. Morten Petersen, Senior Systems Engineer, Isilon Division

Isilon Scale Out NAS. Morten Petersen, Senior Systems Engineer, Isilon Division Isilon Scale Out NAS Morten Petersen, Senior Systems Engineer, Isilon Division 1 Agenda Architecture Overview Next Generation Hardware Performance Caching Performance SMB 3 - MultiChannel 2 OneFS Architecture

More information

IBM Spectrum NAS. Easy-to-manage software-defined file storage for the enterprise. Overview. Highlights

IBM Spectrum NAS. Easy-to-manage software-defined file storage for the enterprise. Overview. Highlights IBM Spectrum NAS Easy-to-manage software-defined file storage for the enterprise Highlights Reduce capital expenditures with storage software on commodity servers Improve efficiency by consolidating all

More information

Customer Success Story Los Alamos National Laboratory

Customer Success Story Los Alamos National Laboratory Customer Success Story Los Alamos National Laboratory Panasas High Performance Storage Powers the First Petaflop Supercomputer at Los Alamos National Laboratory Case Study June 2010 Highlights First Petaflop

More information

Brent Welch. Director, Architecture. Panasas Technology. HPC Advisory Council Lugano, March 2011

Brent Welch. Director, Architecture. Panasas Technology. HPC Advisory Council Lugano, March 2011 Brent Welch Director, Architecture Panasas Technology HPC Advisory Council Lugano, March 2011 Panasas Background Technology based on Object Storage concepts invented by company founder, Garth Gibson, a

More information

Sun Lustre Storage System Simplifying and Accelerating Lustre Deployments

Sun Lustre Storage System Simplifying and Accelerating Lustre Deployments Sun Lustre Storage System Simplifying and Accelerating Lustre Deployments Torben Kling-Petersen, PhD Presenter s Name Principle Field Title andengineer Division HPC &Cloud LoB SunComputing Microsystems

More information

Microsoft Office SharePoint Server 2007

Microsoft Office SharePoint Server 2007 Microsoft Office SharePoint Server 2007 Enabled by EMC Celerra Unified Storage and Microsoft Hyper-V Reference Architecture Copyright 2010 EMC Corporation. All rights reserved. Published May, 2010 EMC

More information

Parallel File Systems Compared

Parallel File Systems Compared Parallel File Systems Compared Computing Centre (SSCK) University of Karlsruhe, Germany Laifer@rz.uni-karlsruhe.de page 1 Outline» Parallel file systems (PFS) Design and typical usage Important features

More information

HPC Considerations for Scalable Multidiscipline CAE Applications on Conventional Linux Platforms. Author: Correspondence: ABSTRACT:

HPC Considerations for Scalable Multidiscipline CAE Applications on Conventional Linux Platforms. Author: Correspondence: ABSTRACT: HPC Considerations for Scalable Multidiscipline CAE Applications on Conventional Linux Platforms Author: Stan Posey Panasas, Inc. Correspondence: Stan Posey Panasas, Inc. Phone +510 608 4383 Email sposey@panasas.com

More information

Architecting Storage for Semiconductor Design: Manufacturing Preparation

Architecting Storage for Semiconductor Design: Manufacturing Preparation White Paper Architecting Storage for Semiconductor Design: Manufacturing Preparation March 2012 WP-7157 EXECUTIVE SUMMARY The manufacturing preparation phase of semiconductor design especially mask data

More information

GFS: The Google File System

GFS: The Google File System GFS: The Google File System Brad Karp UCL Computer Science CS GZ03 / M030 24 th October 2014 Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one

More information

Dell Fluid Data solutions. Powerful self-optimized enterprise storage. Dell Compellent Storage Center: Designed for business results

Dell Fluid Data solutions. Powerful self-optimized enterprise storage. Dell Compellent Storage Center: Designed for business results Dell Fluid Data solutions Powerful self-optimized enterprise storage Dell Compellent Storage Center: Designed for business results The Dell difference: Efficiency designed to drive down your total cost

More information

STORAGE CONSOLIDATION WITH IP STORAGE. David Dale, NetApp

STORAGE CONSOLIDATION WITH IP STORAGE. David Dale, NetApp STORAGE CONSOLIDATION WITH IP STORAGE David Dale, NetApp SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA. Member companies and individuals may use this material in

More information

An Oracle White Paper December Accelerating Deployment of Virtualized Infrastructures with the Oracle VM Blade Cluster Reference Configuration

An Oracle White Paper December Accelerating Deployment of Virtualized Infrastructures with the Oracle VM Blade Cluster Reference Configuration An Oracle White Paper December 2010 Accelerating Deployment of Virtualized Infrastructures with the Oracle VM Blade Cluster Reference Configuration Introduction...1 Overview of the Oracle VM Blade Cluster

More information

Chelsio Communications. Meeting Today s Datacenter Challenges. Produced by Tabor Custom Publishing in conjunction with: CUSTOM PUBLISHING

Chelsio Communications. Meeting Today s Datacenter Challenges. Produced by Tabor Custom Publishing in conjunction with: CUSTOM PUBLISHING Meeting Today s Datacenter Challenges Produced by Tabor Custom Publishing in conjunction with: 1 Introduction In this era of Big Data, today s HPC systems are faced with unprecedented growth in the complexity

More information

IBM Storwize V7000 Unified

IBM Storwize V7000 Unified IBM Storwize V7000 Unified Pavel Müller IBM Systems and Technology Group Storwize V7000 Position Enterprise Block DS8000 For clients requiring: Advanced disaster recovery with 3-way mirroring and System

More information

STORAGE CONSOLIDATION WITH IP STORAGE. David Dale, NetApp

STORAGE CONSOLIDATION WITH IP STORAGE. David Dale, NetApp STORAGE CONSOLIDATION WITH IP STORAGE David Dale, NetApp SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA. Member companies and individuals may use this material in

More information

The Path to Lower-Cost, Scalable, Highly Available Windows File Serving

The Path to Lower-Cost, Scalable, Highly Available Windows File Serving The Path to Lower-Cost, Scalable, Highly Available Windows File Serving Higher Performance, Modular Expansion, Fault Tolerance at a Lower Cost The Challenges of Cost Effective, Scalable File Services for

More information

70-414: Implementing an Advanced Server Infrastructure Course 01 - Creating the Virtualization Infrastructure

70-414: Implementing an Advanced Server Infrastructure Course 01 - Creating the Virtualization Infrastructure 70-414: Implementing an Advanced Server Infrastructure Course 01 - Creating the Virtualization Infrastructure Slide 1 Creating the Virtualization Infrastructure Slide 2 Introducing Microsoft System Center

More information

Coordinating Parallel HSM in Object-based Cluster Filesystems

Coordinating Parallel HSM in Object-based Cluster Filesystems Coordinating Parallel HSM in Object-based Cluster Filesystems Dingshan He, Xianbo Zhang, David Du University of Minnesota Gary Grider Los Alamos National Lab Agenda Motivations Parallel archiving/retrieving

More information

Sun N1: Storage Virtualization and Oracle

Sun N1: Storage Virtualization and Oracle OracleWorld 2003 Session 36707 - Sun N1: Storage Virtualization and Oracle Glenn Colaco Performance Engineer Sun Microsystems Performance and Availability Engineering September 9, 2003 Background PAE works

More information

A GPFS Primer October 2005

A GPFS Primer October 2005 A Primer October 2005 Overview This paper describes (General Parallel File System) Version 2, Release 3 for AIX 5L and Linux. It provides an overview of key concepts which should be understood by those

More information

A High-Performance Storage and Ultra- High-Speed File Transfer Solution for Collaborative Life Sciences Research

A High-Performance Storage and Ultra- High-Speed File Transfer Solution for Collaborative Life Sciences Research A High-Performance Storage and Ultra- High-Speed File Transfer Solution for Collaborative Life Sciences Research Storage Platforms with Aspera Overview A growing number of organizations with data-intensive

More information

DELL EMC ISILON F800 AND H600 I/O PERFORMANCE

DELL EMC ISILON F800 AND H600 I/O PERFORMANCE DELL EMC ISILON F800 AND H600 I/O PERFORMANCE ABSTRACT This white paper provides F800 and H600 performance data. It is intended for performance-minded administrators of large compute clusters that access

More information

The vsphere 6.0 Advantages Over Hyper- V

The vsphere 6.0 Advantages Over Hyper- V The Advantages Over Hyper- V The most trusted and complete virtualization platform SDDC Competitive Marketing 2015 Q2 VMware.com/go/PartnerCompete 2015 VMware Inc. All rights reserved. v3b The Most Trusted

More information

LustreFS and its ongoing Evolution for High Performance Computing and Data Analysis Solutions

LustreFS and its ongoing Evolution for High Performance Computing and Data Analysis Solutions LustreFS and its ongoing Evolution for High Performance Computing and Data Analysis Solutions Roger Goff Senior Product Manager DataDirect Networks, Inc. What is Lustre? Parallel/shared file system for

More information

Deploying Celerra MPFSi in High-Performance Computing Environments

Deploying Celerra MPFSi in High-Performance Computing Environments Deploying Celerra MPFSi in High-Performance Computing Environments Best Practices Planning Abstract This paper describes the EMC Celerra MPFSi approach to scalable high-performance data sharing, with a

More information

Performance Evaluation Using Network File System (NFS) v3 Protocol. Hitachi Data Systems

Performance Evaluation Using Network File System (NFS) v3 Protocol. Hitachi Data Systems P E R F O R M A N C E B R I E F Hitachi NAS Platform 3080 Cluster Using the Hitachi Adaptable Modular Aciduisismodo Dolore Eolore Storage 2500: SPEC SFS2008 Dionseq Uatummy Odolorem Vel Performance Analysis

More information

The advantages of architecting an open iscsi SAN

The advantages of architecting an open iscsi SAN Storage as it should be The advantages of architecting an open iscsi SAN Pete Caviness Lefthand Networks, 5500 Flatiron Parkway, Boulder CO 80301, Ph: +1-303-217-9043, FAX: +1-303-217-9020 e-mail: pete.caviness@lefthandnetworks.com

More information

The Google File System

The Google File System The Google File System Sanjay Ghemawat, Howard Gobioff and Shun Tak Leung Google* Shivesh Kumar Sharma fl4164@wayne.edu Fall 2015 004395771 Overview Google file system is a scalable distributed file system

More information

Introduction to iscsi

Introduction to iscsi Introduction to iscsi As Ethernet begins to enter into the Storage world a new protocol has been getting a lot of attention. The Internet Small Computer Systems Interface or iscsi, is an end-to-end protocol

More information

Storage Industry Resource Domain Model

Storage Industry Resource Domain Model Storage Industry Resource Domain Model A Technical Proposal from the SNIA Technical Council Topics Abstract Data Storage Interfaces Storage Resource Domain Data Resource Domain Information Resource Domain

More information

Configuring and Managing Virtual Storage

Configuring and Managing Virtual Storage Configuring and Managing Virtual Storage Module 6 You Are Here Course Introduction Introduction to Virtualization Creating Virtual Machines VMware vcenter Server Configuring and Managing Virtual Networks

More information

GFS: The Google File System. Dr. Yingwu Zhu

GFS: The Google File System. Dr. Yingwu Zhu GFS: The Google File System Dr. Yingwu Zhu Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one big CPU More storage, CPU required than one PC can

More information

Object-based Storage Devices (OSD) T10 Standard

Object-based Storage Devices (OSD) T10 Standard Object-based Storage Devices (OSD) T10 Standard Erik Riedel Seagate Research Motivation for OSD Improved device and data sharing Platform-dependent metadata moved to device Systems need only agree on naming

More information

As storage networking technology

As storage networking technology Chapter 10 Storage As storage networking technology matures, larger and complex implementations are becoming more common. The heterogeneous nature of storage infrastructures has further added to the complexity

More information

access addresses/addressing advantages agents allocation analysis

access addresses/addressing advantages agents allocation analysis INDEX A access control of multipath port fanout, LUN issues, 122 of SAN devices, 154 virtualization server reliance on, 173 DAS characteristics (table), 19 conversion to SAN fabric storage access, 105

More information

PCI Express x8 Single Port SFP+ 10 Gigabit Server Adapter (Intel 82599ES Based) Single-Port 10 Gigabit SFP+ Ethernet Server Adapters Provide Ultimate

PCI Express x8 Single Port SFP+ 10 Gigabit Server Adapter (Intel 82599ES Based) Single-Port 10 Gigabit SFP+ Ethernet Server Adapters Provide Ultimate NIC-PCIE-1SFP+-PLU PCI Express x8 Single Port SFP+ 10 Gigabit Server Adapter (Intel 82599ES Based) Single-Port 10 Gigabit SFP+ Ethernet Server Adapters Provide Ultimate Flexibility and Scalability in Virtual

More information

Write a technical report Present your results Write a workshop/conference paper (optional) Could be a real system, simulation and/or theoretical

Write a technical report Present your results Write a workshop/conference paper (optional) Could be a real system, simulation and/or theoretical Identify a problem Review approaches to the problem Propose a novel approach to the problem Define, design, prototype an implementation to evaluate your approach Could be a real system, simulation and/or

More information

LS-DYNA Best-Practices: Networking, MPI and Parallel File System Effect on LS-DYNA Performance

LS-DYNA Best-Practices: Networking, MPI and Parallel File System Effect on LS-DYNA Performance 11 th International LS-DYNA Users Conference Computing Technology LS-DYNA Best-Practices: Networking, MPI and Parallel File System Effect on LS-DYNA Performance Gilad Shainer 1, Tong Liu 2, Jeff Layton

More information

IBM IBM Open Systems Storage Solutions Version 4. Download Full Version :

IBM IBM Open Systems Storage Solutions Version 4. Download Full Version : IBM 000-742 IBM Open Systems Storage Solutions Version 4 Download Full Version : https://killexams.com/pass4sure/exam-detail/000-742 Answer: B QUESTION: 156 Given the configuration shown, which of the

More information

Lessons learned from Lustre file system operation

Lessons learned from Lustre file system operation Lessons learned from Lustre file system operation Roland Laifer STEINBUCH CENTRE FOR COMPUTING - SCC KIT University of the State of Baden-Württemberg and National Laboratory of the Helmholtz Association

More information

Object-based Storage (OSD) Architecture and Systems

Object-based Storage (OSD) Architecture and Systems Object-based Storage (OSD) Architecture and Systems Erik Riedel, Seagate Technology April 2007 Abstract Object-based Storage (OSD) Architecture and Systems The Object-based Storage Device interface standard

More information

Data Protection for Cisco HyperFlex with Veeam Availability Suite. Solution Overview Cisco Public

Data Protection for Cisco HyperFlex with Veeam Availability Suite. Solution Overview Cisco Public Data Protection for Cisco HyperFlex with Veeam Availability Suite 1 2017 2017 Cisco Cisco and/or and/or its affiliates. its affiliates. All rights All rights reserved. reserved. Highlights Is Cisco compatible

More information

Hitachi Virtual Storage Platform Family

Hitachi Virtual Storage Platform Family Hitachi Virtual Storage Platform Family Advanced Storage Capabilities for All Organizations Andre Lahrmann 23. November 2017 Hitachi Vantara Vorweg: Aus Hitachi Data Systems wird Hitachi Vantara The efficiency

More information

Red Hat Enterprise 7 Beta File Systems

Red Hat Enterprise 7 Beta File Systems Red Hat Enterprise 7 Beta File Systems New Scale, Speed & Features Ric Wheeler Director Red Hat Kernel File & Storage Team Red Hat Storage Engineering Agenda Red Hat Enterprise Linux 7 Storage Features

More information

Omneon MediaGrid Technical Overview

Omneon MediaGrid Technical Overview Omneon MediaGrid Technical Overview Introduction To MediaGrid Property Architecture Scalability Networking Client Access High Availability Application Integration Attributes Scale-out; MediaFS clustered

More information

The Last Bottleneck: How Parallel I/O can attenuate Amdahl's Law

The Last Bottleneck: How Parallel I/O can attenuate Amdahl's Law The Last Bottleneck: How Parallel I/O can attenuate Amdahl's Law ERESEARCH AUSTRALASIA, NOVEMBER 2011 REX TANAKIT DIRECTOR OF INDUSTRY SOLUTIONS AGENDA Parallel System Parallel processing goes mainstream

More information

Shared Object-Based Storage and the HPC Data Center

Shared Object-Based Storage and the HPC Data Center Shared Object-Based Storage and the HPC Data Center Jim Glidewell High Performance Computing BOEING is a trademark of Boeing Management Company. Computing Environment Cray X1 2 Chassis, 128 MSPs, 1TB memory

More information

Distributed Systems. Lec 10: Distributed File Systems GFS. Slide acks: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung

Distributed Systems. Lec 10: Distributed File Systems GFS. Slide acks: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung Distributed Systems Lec 10: Distributed File Systems GFS Slide acks: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung 1 Distributed File Systems NFS AFS GFS Some themes in these classes: Workload-oriented

More information

Evaluating Cloud Storage Strategies. James Bottomley; CTO, Server Virtualization

Evaluating Cloud Storage Strategies. James Bottomley; CTO, Server Virtualization Evaluating Cloud Storage Strategies James Bottomley; CTO, Server Virtualization Introduction to Storage Attachments: - Local (Direct cheap) SAS, SATA - Remote (SAN, NAS expensive) FC net Types - Block

More information

Oracle Database 11g Direct NFS Client Oracle Open World - November 2007

Oracle Database 11g Direct NFS Client Oracle Open World - November 2007 Oracle Database 11g Client Oracle Open World - November 2007 Bill Hodak Sr. Product Manager Oracle Corporation Kevin Closson Performance Architect Oracle Corporation Introduction

More information

Distributed Filesystem

Distributed Filesystem Distributed Filesystem 1 How do we get data to the workers? NAS Compute Nodes SAN 2 Distributing Code! Don t move data to workers move workers to the data! - Store data on the local disks of nodes in the

More information

Advanced iscsi Management April, 2008

Advanced iscsi Management April, 2008 April, 2008 Gene Nagle, istor Networks SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA. Member companies and individuals may use this material in presentations and

More information

Network Design Considerations for Grid Computing

Network Design Considerations for Grid Computing Network Design Considerations for Grid Computing Engineering Systems How Bandwidth, Latency, and Packet Size Impact Grid Job Performance by Erik Burrows, Engineering Systems Analyst, Principal, Broadcom

More information

The File Systems Evolution. Christian Bandulet, Sun Microsystems

The File Systems Evolution. Christian Bandulet, Sun Microsystems The s Evolution Christian Bandulet, Sun Microsystems SNIA Legal Notice The material contained in this tutorial is copyrighted by the SNIA. Member companies and individuals may use this material in presentations

More information

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters

COSC 6374 Parallel Computation. Parallel I/O (I) I/O basics. Concept of a clusters COSC 6374 Parallel I/O (I) I/O basics Fall 2010 Concept of a clusters Processor 1 local disks Compute node message passing network administrative network Memory Processor 2 Network card 1 Network card

More information

I/O Virtualization: Enabling New Architectures in the Data Center for Server and I/O Connectivity Sujoy Sen Micron

I/O Virtualization: Enabling New Architectures in the Data Center for Server and I/O Connectivity Sujoy Sen Micron I/O Virtualization: Enabling New Architectures in the Data Center for Server and I/O Connectivity Sujoy Sen Micron Server Evolution Performance 1970s-80s (mainframe) Very Powerful 1990s 2000s 2010s Powerful

More information

InfiniBand Networked Flash Storage

InfiniBand Networked Flash Storage InfiniBand Networked Flash Storage Superior Performance, Efficiency and Scalability Motti Beck Director Enterprise Market Development, Mellanox Technologies Flash Memory Summit 2016 Santa Clara, CA 1 17PB

More information

Distributed File Systems II

Distributed File Systems II Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation

More information

SwiftStack and python-swiftclient

SwiftStack and python-swiftclient SwiftStack and python-swiftclient Summary Storage administrators around the world are turning to object storage and particularly SwiftStack s enterprise-ready object storage built on OpenStack Swift for

More information

DAFS Storage for High Performance Computing using MPI-I/O: Design and Experience

DAFS Storage for High Performance Computing using MPI-I/O: Design and Experience DAFS Storage for High Performance Computing using MPI-I/O: Design and Experience Vijay Velusamy, Anthony Skjellum MPI Software Technology, Inc. Email: {vijay, tony}@mpi-softtech.com Arkady Kanevsky *,

More information

The Last Bottleneck: How Parallel I/O can improve application performance

The Last Bottleneck: How Parallel I/O can improve application performance The Last Bottleneck: How Parallel I/O can improve application performance HPC ADVISORY COUNCIL STANFORD WORKSHOP; DECEMBER 6 TH 2011 REX TANAKIT DIRECTOR OF INDUSTRY SOLUTIONS AGENDA Panasas Overview Who

More information

Storage Area Network (SAN)

Storage Area Network (SAN) Storage Area Network (SAN) 1 Outline Shared Storage Architecture Direct Access Storage (DAS) SCSI RAID Network Attached Storage (NAS) Storage Area Network (SAN) Fiber Channel and Fiber Channel Switch 2

More information

iscsi Technology Brief Storage Area Network using Gbit Ethernet The iscsi Standard

iscsi Technology Brief Storage Area Network using Gbit Ethernet The iscsi Standard iscsi Technology Brief Storage Area Network using Gbit Ethernet The iscsi Standard On February 11 th 2003, the Internet Engineering Task Force (IETF) ratified the iscsi standard. The IETF was made up of

More information

EMC Celerra CNS with CLARiiON Storage

EMC Celerra CNS with CLARiiON Storage DATA SHEET EMC Celerra CNS with CLARiiON Storage Reach new heights of availability and scalability with EMC Celerra Clustered Network Server (CNS) and CLARiiON storage Consolidating and sharing information

More information

Experiences with HP SFS / Lustre in HPC Production

Experiences with HP SFS / Lustre in HPC Production Experiences with HP SFS / Lustre in HPC Production Computing Centre (SSCK) University of Karlsruhe Laifer@rz.uni-karlsruhe.de page 1 Outline» What is HP StorageWorks Scalable File Share (HP SFS)? A Lustre

More information

The Future of Storage

The Future of Storage The Future of Storage A Technical Discussion Replacing Your Proprietary Scale-out NAS With GlusterFS [Presenter name] Solutions Architect Jacob Shucart SA, Red Hat January 2012 October, 2011 1 Agenda Introduction

More information

BeoLink.org. Design and build an inexpensive DFS. Fabrizio Manfredi Furuholmen. FrOSCon August 2008

BeoLink.org. Design and build an inexpensive DFS. Fabrizio Manfredi Furuholmen. FrOSCon August 2008 Design and build an inexpensive DFS Fabrizio Manfredi Furuholmen FrOSCon August 2008 Agenda Overview Introduction Old way openafs New way Hadoop CEPH Conclusion Overview Why Distributed File system? Handle

More information

EditShare XStream EFS Shared Storage

EditShare XStream EFS Shared Storage WHITE PAPER EditShare XStream EFS Shared Storage Advantages of EFS Native Client network protocol in media intensive storage workflows 2018 EditShare LLC. All rights reserved. EditShare is a registered

More information

Demartek September Intel 10GbE Adapter Performance Evaluation for FCoE and iscsi. Introduction. Evaluation Environment. Evaluation Summary

Demartek September Intel 10GbE Adapter Performance Evaluation for FCoE and iscsi. Introduction. Evaluation Environment. Evaluation Summary Intel 10GbE Adapter Performance Evaluation for FCoE and iscsi Evaluation report prepared under contract with Intel Corporation Introduction As the interest in converged networks grows, and as the vendors

More information

Building a Highly Scalable and Performant SMB Protocol Server

Building a Highly Scalable and Performant SMB Protocol Server Building a Highly Scalable and Performant SMB Protocol Server Dr. Sunu Engineer CTO, Ryussi Technologies About Ryussi Creator of MoSMB SMB with Mojo. Specializes in Storage and Systems Software Product

More information

Storage Update and Storage Best Practices for Microsoft Server Applications. Dennis Martin President, Demartek January 2009 Copyright 2009 Demartek

Storage Update and Storage Best Practices for Microsoft Server Applications. Dennis Martin President, Demartek January 2009 Copyright 2009 Demartek Storage Update and Storage Best Practices for Microsoft Server Applications Dennis Martin President, Demartek January 2009 Copyright 2009 Demartek Agenda Introduction Storage Technologies Storage Devices

More information

GFS Overview. Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures

GFS Overview. Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures GFS Overview Design goals/priorities Design for big-data workloads Huge files, mostly appends, concurrency, huge bandwidth Design for failures Interface: non-posix New op: record appends (atomicity matters,

More information

Vendor: IBM. Exam Code: Exam Name: IBM Midrange Storage Technical Support V3. Version: Demo

Vendor: IBM. Exam Code: Exam Name: IBM Midrange Storage Technical Support V3. Version: Demo Vendor: IBM Exam Code: 000-451 Exam Name: IBM Midrange Storage Technical Support V3 Version: Demo QUESTION NO: 1 On the Storwize V7000, which IBM utility analyzes the expected compression savings for an

More information

How to Increase VMware Application Availability with Shared Storage

How to Increase VMware Application Availability with Shared Storage WHITE PAPER How to Increase VMware Application Availability with Shared Storage The Essential Guide to Using Shared Storage with VMware to Achieve High Availability While Reducing Operating Costs INTRODUCTION

More information

! Design constraints. " Component failures are the norm. " Files are huge by traditional standards. ! POSIX-like

! Design constraints.  Component failures are the norm.  Files are huge by traditional standards. ! POSIX-like Cloud background Google File System! Warehouse scale systems " 10K-100K nodes " 50MW (1 MW = 1,000 houses) " Power efficient! Located near cheap power! Passive cooling! Power Usage Effectiveness = Total

More information

Oracle RAC 10g Celerra NS Series NFS

Oracle RAC 10g Celerra NS Series NFS Oracle RAC 10g Celerra NS Series NFS Reference Architecture Guide Revision 1.0 EMC Solutions Practice/EMC NAS Solutions Engineering. EMC Corporation RTP Headquarters RTP, NC 27709 www.emc.com Oracle RAC

More information

Simplified and Consolidated Parallel Media File System Solution

Simplified and Consolidated Parallel Media File System Solution Simplified and Consolidated Parallel Media File System Solution Presenters : Dr. M. K. Jibbe, Technical Director NetApp ESG Joey Parnell, Software Architect NetApp, ESG 2018 Storage Developer Conference.

More information

NPTEL Course Jan K. Gopinath Indian Institute of Science

NPTEL Course Jan K. Gopinath Indian Institute of Science Storage Systems NPTEL Course Jan 2012 (Lecture 39) K. Gopinath Indian Institute of Science Google File System Non-Posix scalable distr file system for large distr dataintensive applications performance,

More information

Cisco 4000 Series Integrated Services Routers: Architecture for Branch-Office Agility

Cisco 4000 Series Integrated Services Routers: Architecture for Branch-Office Agility White Paper Cisco 4000 Series Integrated Services Routers: Architecture for Branch-Office Agility The Cisco 4000 Series Integrated Services Routers (ISRs) are designed for distributed organizations with

More information

SFA12KX and Lustre Update

SFA12KX and Lustre Update Sep 2014 SFA12KX and Lustre Update Maria Perez Gutierrez HPC Specialist HPC Advisory Council Agenda SFA12KX Features update Partial Rebuilds QoS on reads Lustre metadata performance update 2 SFA12KX Features

More information

The Design and Implementation of AQuA: An Adaptive Quality of Service Aware Object-Based Storage Device

The Design and Implementation of AQuA: An Adaptive Quality of Service Aware Object-Based Storage Device The Design and Implementation of AQuA: An Adaptive Quality of Service Aware Object-Based Storage Device Joel Wu and Scott Brandt Department of Computer Science University of California Santa Cruz MSST2006

More information

virtual machine block storage with the ceph distributed storage system sage weil xensummit august 28, 2012

virtual machine block storage with the ceph distributed storage system sage weil xensummit august 28, 2012 virtual machine block storage with the ceph distributed storage system sage weil xensummit august 28, 2012 outline why you should care what is it, what it does how it works, how you can use it architecture

More information

IBM EXAM QUESTIONS & ANSWERS

IBM EXAM QUESTIONS & ANSWERS IBM 000-452 EXAM QUESTIONS & ANSWERS Number: 000-452 Passing Score: 800 Time Limit: 120 min File Version: 68.8 http://www.gratisexam.com/ IBM 000-452 EXAM QUESTIONS & ANSWERS Exam Name: IBM Storwize V7000

More information

A product by CloudFounders. Wim Provoost Open vstorage

A product by CloudFounders. Wim Provoost Open vstorage A product by CloudFounders Wim Provoost (@wimpers_be) Open vstorage (@openvstorage) http://www.openvstorage.com CloudFounders vrun Converged infrastructure that combines the benefits of the hyperconverged

More information

BIG DATA AND HADOOP ON THE ZFS STORAGE APPLIANCE

BIG DATA AND HADOOP ON THE ZFS STORAGE APPLIANCE BIG DATA AND HADOOP ON THE ZFS STORAGE APPLIANCE BRETT WENINGER, MANAGING DIRECTOR 10/21/2014 ADURANT APPROACH TO BIG DATA Align to Un/Semi-structured Data Instead of Big Scale out will become Big Greatest

More information

NFSv4.1 Using pnfs PRESENTATION TITLE GOES HERE. Presented by: Alex McDonald CTO Office, NetApp

NFSv4.1 Using pnfs PRESENTATION TITLE GOES HERE. Presented by: Alex McDonald CTO Office, NetApp NFSv4.1 Using pnfs PRESENTATION TITLE GOES HERE Presented by: Alex McDonald CTO Office, NetApp Webcast Presenter Alex McDonald joined NetApp in 2005, after more than 30 years in a variety of roles with

More information