WORK PROJECT REPORT: TAPE STORAGE AND CRC PROTECTION

Size: px
Start display at page:

Download "WORK PROJECT REPORT: TAPE STORAGE AND CRC PROTECTION"

Transcription

1 WORK PROJECT REPORT: TAPE STORAGE AND CRC PROTECTION CERN Summer Student Programme 2014 Student: Main supervisor: Second supervisor: Division: Karel Ha Julien Marcel Leduc German Cancio Melia IT-DSS-TAB Project ID: Project name: Introduction Tape Storage and CRC Protection Date: Wed Sep My name is Karel Ha and I am a student of Computer Science/Applied Mathematics at Charles University in the Czech Republic. I got selected for CERN Summer Student internship From June to September, I worked the IT-DSS-TAB section under the supervision of Julien Leduc and German Cancio Melia. This is the story of my work project: its goal, progress, tools used and final results. CERN and IT-DSS-TAB CERN is an international particle physics laboratory. Its greatest instrument, the Large Hadron Collider (LHC), accelerates protons at almost the speed of light and collides them at dedicated points along the 27km long accelerator tunnel. At these collision points, four monumental detectors record and selectively filter data from 40 million collisions per second. This huge amount of data travels afterwards to Computer Center, which is maintained by the IT department. The Data & Storage Services group (DSS) takes care of storing the data on various physical media. One of them is the tape storage system, where the data are stored for the final archiving. This is the responsibility of Tapes, Archives and Backups section (TAB). Data corruption and the importance of CRC Over 100 Petabytes of data is stored on several kind of physical support, namely disks and tapes. Data on any physical support or traveling on a data link (network, fibre channel...) can be subject to silent data corruption. A possible improvement is introducing end-to-end data integrity from the filesystem down to the tape layer using the T10 Data Protection Information (T10 PI) standard. Modern filesystems like ZFS or BTRFS are using CRCs to enforce data integrity at the filesystem level, the SCSI communication channels are supporting the same T10 PI CRC's to ensure data integrity on the communication link, and

2 new-generation tape drives are able to read on the fly the written data to check these CRC's with little performance penalty. The goals of the project enable CRC protection on tape storage system design and implement speedup techniques to increase throughput by improving CRC algorithm itself by computing each block's CRC using parallel programming perform and analyze benchmarks of tape-write speed using previous techniques The previous work Re-using previous code, we measured following speeds tape-write: 250 MB/s tape-write with Reed-Solomon CRC: 130 MB/s Reed-Solomon CRC with output to /dev/null: 200 MB/s Thus, it was (previously) a bottleneck to have CRC protection enabled! Progress of the project Initial weeks For first couple of weeks, I was getting familiar with CERN and CASTOR infrastructure: 1. I studied how to create a puppet-maintained virtual machine using OpenStack infrastructure. This was due to the initial attempt to test on virtual tape library. The result of this endeavor is kha-vtl.cern.ch, a VM running Scientific Linux After that I learned how to manipulate with tape drive and perform basic tape operations, such as mount, rewind, read... This was achieved by mt tool and/or by sending SCSI CDB to sg3_utils. 3. Enabling Logical Block Protection on the IBM tape drive using sg3_utils and sdparm. 4. Obtain, recompile, run and understand the previous code, which is located on official CASTOR Git repository. 5. After a little struggle, I found out that this would be impossible on my local machine (as all CASTOR code is designed for Scientific Linux). Whence, I decided to develop remotely on tpsrv212 using ssh, vim and gcc. I created a bash script git-sync.sh, which: git-pulls source code from official CASTOR's repo to work-space directory, hard-resetting all the previous changes in the local directory switches to appropriate branch and commit copies minimal working set of source code files from local GitLab folder structure to the work-space runs cmake and make to compile the code This routine is (re)launched every time one needs to compile new changes in the code. The repo and work-space are located in tpsrv212:/root/kha 6. As soon as the code was compilable and working, I performed baseline speed measurements (see section The previous work)

3 7. Having measurement numbers, we started to optimize. There are 2 possible ways to go: improve CRC algorithm itself compute CRC for blocks concurrently (i.e. parallel programming) 8. Having chosen the first approach, I studied CRC manual A Painless Guide to CRC Error Detection Algorithms and implemented a~speedup of Reed-Solomon CRC. Namely, creating larger pre-computed table (XOR constants for every 16 bits instead of original 8 bits). However, the gain in speed was not sufficient enough: from 200 MB/s to cca. 230 MB/s. 9. Realizing this fact, we opted for the multithreaded approach, which became eventually successful. See the next section. Multithreaded implementation A diagram of the whole process: Input CRCs Compute Threads Ouput There are two types of threads: the master thread handles all the IO operations: reading blocks of data to buffer, writing ready blocks with their CRCs to the output and managing compute threads compute threads only computes CRC: they receive pointers to assigned input blocks, calculate their CRCs and store these CRCs into corresponding space in the CRC buffer The queue of tasks is implemented as a ring buffer, which scales to number of compute threads and is always larger than necessary. For our tests, the buffer had the size of 5 * number of threads * number of blocks per single thread. Infrastructure The following machines were assigned to me, so as to develop and test my code: tpsrv212.cern.ch: a tape server with 4 cores (in case of enabled HyperThreading, this rises to 8 cores) and with IBM tape drive tpsrv100.cern.ch: a tape server with 6 cores (12 with HT) We found out that tpsrv212 had HT disabled as opposed to tpsrv100, which is an important discovery for future tape operations. Results Benchmarks Scalability benchmarks of raw performance (output to /dev/null):

4 1400 Reed-Solomon on tpsrv100, output to /dev/null 3500 CRC32-C on tpsrv100, output to /dev/null MB/s MB/s n threads n threads There are 6 cores on tpsrv100, therefore the increase of speed for 1 to 6 compute threads is practically linear. In case of Reed-Solomon, from 6 to 12 compute threads the HyperThreading is taking effect, resulting in the speed gain of +30 %. Therefore, the HyperThreading has substantial impact on the speed and it should be enabled during Logical Block Protection. In case of CRC32-C, the algorithm is much faster, hence, the speed is saturated for already 5 threads and HyperThreading doesn't impact the results anymore. Note Blue dashed line shows the speed of tape drive: 250 MB/s, i.e. speed that is necessary to reach. Otherwise enabling CRC protection would slow down tape-write speed. Also, n_threads is the count of COMPUTE threads, hence, one needs to add +1 master thread for the total number of threads. The benchmark of performance with writing to tape (output to /dev/tape): 300 Reed-Solomon on tpsrv212, output to tape MB/s n threads From the plot we can observe that already two compute threads are enough to reach the original speed of the tape drive.

5 Technologies During this internship I used these technologies: C/C++ The most common programming language at CERN. Especially for low-level programming concerning tape drivers, C/C++ is a must. Besides, the previous source code, which handles tape drives, was as well written in these languages. Git CERN uses Git as the standard source code management and revision control system. With my supervisor, we create and maintained a Git repository tape-summer2014 on CERN's GitLab (currently set as a private project). It contains: source codes and scripts manual and documentations (IBM and Oracle tape drive manuals, CRC manual) logs of my work progress benchmark outputs final presentation (LaTeX source code, images, plots) CMake CMake manages the build process using a compiler-independent method. In my case, this was the only way to re-compile the previous C++ code and develop additional functionality on top of it. Pthreads library For multithreaded programming, POSIX Threads library was used. Other Other tools I used, which are also worth mentioning: Vim restructuredtext bash and GNU tools gcc and make gdb gnuplot Presentation In the final week of my internship, I had the opportunity to give a talk on my project during a section meeting. The presentation slides are available for download on page:

6 Conclusion Regarding the project, I was able to implement multithreaded computation for CRC algorithms and to measure their raw and tape drive performance. These benchmarks showed that, in case of Reed-Solomon algorithm, 1-2 compute threads would suffice to reach original tape drive speed. Nevertheless, the modern CRC32-C algorithm is far more advanced in terms of hardware acceleration capabilities. Hence, with current tape drive speeds there is no need to use multithreading for CRC32-C. For me personally, the gained experience of my summer internship lies in data handling, modern data protection techniques, and disk/tape storage management at the world's largest scientific data archive. And also in implementing multithreaded applications.

Storage and I/O requirements of the LHC experiments

Storage and I/O requirements of the LHC experiments Storage and I/O requirements of the LHC experiments Sverre Jarp CERN openlab, IT Dept where the Web was born 22 June 2006 OpenFabrics Workshop, Paris 1 Briefly about CERN 22 June 2006 OpenFabrics Workshop,

More information

Conference The Data Challenges of the LHC. Reda Tafirout, TRIUMF

Conference The Data Challenges of the LHC. Reda Tafirout, TRIUMF Conference 2017 The Data Challenges of the LHC Reda Tafirout, TRIUMF Outline LHC Science goals, tools and data Worldwide LHC Computing Grid Collaboration & Scale Key challenges Networking ATLAS experiment

More information

LHCb Computing Resources: 2018 requests and preview of 2019 requests

LHCb Computing Resources: 2018 requests and preview of 2019 requests LHCb Computing Resources: 2018 requests and preview of 2019 requests LHCb-PUB-2017-009 23/02/2017 LHCb Public Note Issue: 0 Revision: 0 Reference: LHCb-PUB-2017-009 Created: 23 rd February 2017 Last modified:

More information

Linux System Management with Puppet, Gitlab, and R10k. Scott Nolin, SSEC Technical Computing 22 June 2017

Linux System Management with Puppet, Gitlab, and R10k. Scott Nolin, SSEC Technical Computing 22 June 2017 Linux System Management with Puppet, Gitlab, and R10k Scott Nolin, SSEC Technical Computing 22 June 2017 Introduction I am here to talk about how we do Linux configuration management at the Space Science

More information

Storage Resource Sharing with CASTOR.

Storage Resource Sharing with CASTOR. Storage Resource Sharing with CASTOR Olof Barring, Benjamin Couturier, Jean-Damien Durand, Emil Knezo, Sebastien Ponce (CERN) Vitali Motyakov (IHEP) ben.couturier@cern.ch 16/4/2004 Storage Resource Sharing

More information

Revision control. INF5750/ Lecture 2 (Part I)

Revision control. INF5750/ Lecture 2 (Part I) Revision control INF5750/9750 - Lecture 2 (Part I) Problem area Software projects with multiple developers need to coordinate and synchronize the source code Approaches to version control Work on same

More information

CS 390 Software Engineering Lecture 3 Configuration Management

CS 390 Software Engineering Lecture 3 Configuration Management CS 390 Software Engineering Lecture 3 Configuration Management Includes slides from the companion website for Sommerville, Software Engineering, 10/e. Pearson Higher Education, 2016. All rights reserved.

More information

Overview. About CERN 2 / 11

Overview. About CERN 2 / 11 Overview CERN wanted to upgrade the data monitoring system of one of its Large Hadron Collider experiments called ALICE (A La rge Ion Collider Experiment) to ensure the experiment s high efficiency. They

More information

CERN openlab II. CERN openlab and. Sverre Jarp CERN openlab CTO 16 September 2008

CERN openlab II. CERN openlab and. Sverre Jarp CERN openlab CTO 16 September 2008 CERN openlab II CERN openlab and Intel: Today and Tomorrow Sverre Jarp CERN openlab CTO 16 September 2008 Overview of CERN 2 CERN is the world's largest particle physics centre What is CERN? Particle physics

More information

New strategies of the LHC experiments to meet the computing requirements of the HL-LHC era

New strategies of the LHC experiments to meet the computing requirements of the HL-LHC era to meet the computing requirements of the HL-LHC era NPI AS CR Prague/Rez E-mail: adamova@ujf.cas.cz Maarten Litmaath CERN E-mail: Maarten.Litmaath@cern.ch The performance of the Large Hadron Collider

More information

Revision Control. An Introduction Using Git 1/15

Revision Control. An Introduction Using Git 1/15 Revision Control An Introduction Using Git 1/15 Overview 1. What is revision control? 2. 30,000 foot view 3. Software - git and gitk 4. Setting up your own repository on onyx 2/15 What is version control?

More information

Virtualizing a Batch. University Grid Center

Virtualizing a Batch. University Grid Center Virtualizing a Batch Queuing System at a University Grid Center Volker Büge (1,2), Yves Kemp (1), Günter Quast (1), Oliver Oberst (1), Marcel Kunze (2) (1) University of Karlsruhe (2) Forschungszentrum

More information

Grid Computing a new tool for science

Grid Computing a new tool for science Grid Computing a new tool for science CERN, the European Organization for Nuclear Research Dr. Wolfgang von Rüden Wolfgang von Rüden, CERN, IT Department Grid Computing July 2006 CERN stands for over 50

More information

Pushing the Limits. ADSM Symposium Sheelagh Treweek September 1999 Oxford University Computing Services 1

Pushing the Limits. ADSM Symposium Sheelagh Treweek September 1999 Oxford University Computing Services 1 Pushing the Limits ADSM Symposium Sheelagh Treweek sheelagh.treweek@oucs.ox.ac.uk September 1999 Oxford University Computing Services 1 Overview History of ADSM services at Oxford October 1995 - started

More information

RADU POPESCU IMPROVING THE WRITE SCALABILITY OF THE CERNVM FILE SYSTEM WITH ERLANG/OTP

RADU POPESCU IMPROVING THE WRITE SCALABILITY OF THE CERNVM FILE SYSTEM WITH ERLANG/OTP RADU POPESCU IMPROVING THE WRITE SCALABILITY OF THE CERNVM FILE SYSTEM WITH ERLANG/OTP THE EUROPEAN ORGANISATION FOR PARTICLE PHYSICS RESEARCH (CERN) 2 THE LARGE HADRON COLLIDER THE LARGE HADRON COLLIDER

More information

Shared snapshots. 1 Abstract. 2 Introduction. Mikulas Patocka Red Hat Czech, s.r.o. Purkynova , Brno Czech Republic

Shared snapshots. 1 Abstract. 2 Introduction. Mikulas Patocka Red Hat Czech, s.r.o. Purkynova , Brno Czech Republic Shared snapshots Mikulas Patocka Red Hat Czech, s.r.o. Purkynova 99 612 45, Brno Czech Republic mpatocka@redhat.com 1 Abstract Shared snapshots enable the administrator to take many snapshots of the same

More information

Geant4 on Azure using Docker containers

Geant4 on Azure using Docker containers http://www.geant4.org Geant4 on Azure using Docker containers Andrea Dotti (adotti@slac.stanford.edu) ; SD/EPP/Computing 1 Outlook Motivation/overview Docker + G4 Azure + G4 Conclusions 2 Motivation/overview

More information

Grid Computing: dealing with GB/s dataflows

Grid Computing: dealing with GB/s dataflows Grid Computing: dealing with GB/s dataflows Jan Just Keijser, Nikhef janjust@nikhef.nl David Groep, NIKHEF 21 March 2011 Graphics: Real Time Monitor, Gidon Moont, Imperial College London, see http://gridportal.hep.ph.ic.ac.uk/rtm/

More information

The CMS Computing Model

The CMS Computing Model The CMS Computing Model Dorian Kcira California Institute of Technology SuperComputing 2009 November 14-20 2009, Portland, OR CERN s Large Hadron Collider 5000+ Physicists/Engineers 300+ Institutes 70+

More information

CC-IN2P3: A High Performance Data Center for Research

CC-IN2P3: A High Performance Data Center for Research April 15 th, 2011 CC-IN2P3: A High Performance Data Center for Research Toward a partnership with DELL Dominique Boutigny Agenda Welcome Introduction to CC-IN2P3 Visit of the computer room Lunch Discussion

More information

IEPSAS-Kosice: experiences in running LCG site

IEPSAS-Kosice: experiences in running LCG site IEPSAS-Kosice: experiences in running LCG site Marian Babik 1, Dusan Bruncko 2, Tomas Daranyi 1, Ladislav Hluchy 1 and Pavol Strizenec 2 1 Department of Parallel and Distributed Computing, Institute of

More information

CERN Tape Archive (CTA) :

CERN Tape Archive (CTA) : CERN Tape Archive (CTA) : From Development to Production Deployment Michael Davis, Vladimír Bahyl, Germán Cancio, Eric Cano, Julien Leduc and Steven Murray CHEP 2018, Sofia, Bulgaria 9 July 2018 Changing

More information

ECFS: A decentralized, distributed and faulttolerant FUSE filesystem for the LHCb online farm

ECFS: A decentralized, distributed and faulttolerant FUSE filesystem for the LHCb online farm Journal of Physics: Conference Series OPEN ACCESS ECFS: A decentralized, distributed and faulttolerant FUSE filesystem for the LHCb online farm To cite this article: Tomasz Rybczynski et al 2014 J. Phys.:

More information

Volunteer Computing at CERN

Volunteer Computing at CERN Volunteer Computing at CERN BOINC workshop Sep 2014, Budapest Tomi Asp & Pete Jones, on behalf the LHC@Home team Agenda Overview Status of the LHC@Home projects Additional BOINC projects Service consolidation

More information

LGTM Enterprise System Requirements. Release , August 2018

LGTM Enterprise System Requirements. Release , August 2018 Release 1.17.2, August 2018 Semmle Inc 180 Sansome St San Francisco, CA 94104 Copyright 2018, Semmle Ltd. All rights reserved. LGTM Enterprise release 1.17.2 Document published August 30, 2018 Contents

More information

ISTITUTO NAZIONALE DI FISICA NUCLEARE

ISTITUTO NAZIONALE DI FISICA NUCLEARE ISTITUTO NAZIONALE DI FISICA NUCLEARE Sezione di Perugia INFN/TC-05/10 July 4, 2005 DESIGN, IMPLEMENTATION AND CONFIGURATION OF A GRID SITE WITH A PRIVATE NETWORK ARCHITECTURE Leonello Servoli 1,2!, Mirko

More information

Lab 2: Threads and Processes

Lab 2: Threads and Processes CS333: Operating Systems Lab Lab 2: Threads and Processes Goal The goal of this lab is to get you comfortable with writing basic multi-process / multi-threaded applications, and understanding their performance.

More information

CESSDA Expert Seminar 13 & 14 September 2016 Prague, Czech Republic

CESSDA Expert Seminar 13 & 14 September 2016 Prague, Czech Republic CESSDA Expert Seminar 13 & 14 September 2016 Prague, Czech Republic - basics Matthäus Zloch GESIS Outline for this session Git introduction and some theory Git command basics (plus some little advanced)

More information

New coding practices for LSDALTON

New coding practices for LSDALTON New coding practices for LSDALTON Simen Reine Centre for Theoretical and Computational Chemistry (CTCC), Department of Chemistry, University of Oslo, Norway November 20th, 2015 Simen Reine (CTCC, University

More information

Evaluation of the Huawei UDS cloud storage system for CERN specific data

Evaluation of the Huawei UDS cloud storage system for CERN specific data th International Conference on Computing in High Energy and Nuclear Physics (CHEP3) IOP Publishing Journal of Physics: Conference Series 53 (4) 44 doi:.88/74-6596/53/4/44 Evaluation of the Huawei UDS cloud

More information

High Throughput WAN Data Transfer with Hadoop-based Storage

High Throughput WAN Data Transfer with Hadoop-based Storage High Throughput WAN Data Transfer with Hadoop-based Storage A Amin 2, B Bockelman 4, J Letts 1, T Levshina 3, T Martin 1, H Pi 1, I Sfiligoi 1, M Thomas 2, F Wuerthwein 1 1 University of California, San

More information

CHAPTER 11: IMPLEMENTING FILE SYSTEMS (COMPACT) By I-Chen Lin Textbook: Operating System Concepts 9th Ed.

CHAPTER 11: IMPLEMENTING FILE SYSTEMS (COMPACT) By I-Chen Lin Textbook: Operating System Concepts 9th Ed. CHAPTER 11: IMPLEMENTING FILE SYSTEMS (COMPACT) By I-Chen Lin Textbook: Operating System Concepts 9th Ed. File-System Structure File structure Logical storage unit Collection of related information File

More information

Long term data preservation and virtualization

Long term data preservation and virtualization Long term data preservation and virtualization - Why preserving data? What to do with that old data? - Possible usages of virtualization - Some work done Yves Kemp, DESY IT 2nd Workshop on adapting applications

More information

LHCb Computing Resources: 2019 requests and reassessment of 2018 requests

LHCb Computing Resources: 2019 requests and reassessment of 2018 requests LHCb Computing Resources: 2019 requests and reassessment of 2018 requests LHCb-PUB-2017-019 09/09/2017 LHCb Public Note Issue: 0 Revision: 0 Reference: LHCb-PUB-2017-019 Created: 30 th August 2017 Last

More information

Your desktop or laptop computer consists of several hardware components:

Your desktop or laptop computer consists of several hardware components: Appendix A VirtualBox This appendix describes the role of an operating system on your desktop or laptop computer, how virtualization packages enable you to simultaneously run multiple operating systems

More information

CS370 Operating Systems

CS370 Operating Systems CS370 Operating Systems Colorado State University Yashwant K Malaiya Spring 2018 Lecture 2 Slides based on Text by Silberschatz, Galvin, Gagne Various sources 1 1 2 What is an Operating System? What is

More information

SSIM Collection & Archiving Infrastructure Scaling & Performance Tuning Guide

SSIM Collection & Archiving Infrastructure Scaling & Performance Tuning Guide SSIM Collection & Archiving Infrastructure Scaling & Performance Tuning Guide April 2013 SSIM Engineering Team Version 3.0 1 Document revision history Date Revision Description of Change Originator 03/20/2013

More information

GUIDE TO MAKE A REAL CONTRIBUTION TO AN OPEN SOURCE PROJECT 1. 1

GUIDE TO MAKE A REAL CONTRIBUTION TO AN OPEN SOURCE PROJECT 1. 1 GUIDE TO MAKE A REAL CONTRIBUTION TO AN OPEN SOURCE PROJECT 1. 1 WHO AM I? @tushar_rishav GSoC'16 student contributing to coala - a static code analysis tool, under Python So ware Foundation. A senior

More information

Monte Carlo Production on the Grid by the H1 Collaboration

Monte Carlo Production on the Grid by the H1 Collaboration Journal of Physics: Conference Series Monte Carlo Production on the Grid by the H1 Collaboration To cite this article: E Bystritskaya et al 2012 J. Phys.: Conf. Ser. 396 032067 Recent citations - Monitoring

More information

Operating Systems. Lecture Course in Autumn Term 2015 University of Birmingham. Eike Ritter. September 22, 2015

Operating Systems. Lecture Course in Autumn Term 2015 University of Birmingham. Eike Ritter. September 22, 2015 Lecture Course in Autumn Term 2015 University of Birmingham September 22, 2015 Course Details Overview Course Details What is an Operating System? OS Definition and Structure Lecture notes and resources:

More information

Today s presentation. Git gdb Project 1

Today s presentation. Git gdb Project 1 CS3214: Project 1 Today s presentation Git gdb Project 1 Project 1 Due Monday, February 20 at 11:59 PM Office hours are on course website Check Piazza for updates Git Version Control System Keep snapshots

More information

A WEB-BASED SOLUTION TO VISUALIZE OPERATIONAL MONITORING LINUX CLUSTER FOR THE PROTODUNE DATA QUALITY MONITORING CLUSTER

A WEB-BASED SOLUTION TO VISUALIZE OPERATIONAL MONITORING LINUX CLUSTER FOR THE PROTODUNE DATA QUALITY MONITORING CLUSTER A WEB-BASED SOLUTION TO VISUALIZE OPERATIONAL MONITORING LINUX CLUSTER FOR THE PROTODUNE DATA QUALITY MONITORING CLUSTER BADISA MOSESANE EP-NU Supervisor: Nektarios Benekos Department: EP-NU Table of Contents

More information

The INFN Tier1. 1. INFN-CNAF, Italy

The INFN Tier1. 1. INFN-CNAF, Italy IV WORKSHOP ITALIANO SULLA FISICA DI ATLAS E CMS BOLOGNA, 23-25/11/2006 The INFN Tier1 L. dell Agnello 1), D. Bonacorsi 1), A. Chierici 1), M. Donatelli 1), A. Italiano 1), G. Lo Re 1), B. Martelli 1),

More information

The COMPASS Event Store in 2002

The COMPASS Event Store in 2002 The COMPASS Event Store in 2002 V. Duic INFN, Trieste, Italy M. Lamanna CERN, Switzerland and INFN, Trieste, Italy COMPASS, the fixed-target experiment at CERN studying the structure of the nucleon and

More information

Scientific data processing at global scale The LHC Computing Grid. fabio hernandez

Scientific data processing at global scale The LHC Computing Grid. fabio hernandez Scientific data processing at global scale The LHC Computing Grid Chengdu (China), July 5th 2011 Who I am 2 Computing science background Working in the field of computing for high-energy physics since

More information

The Fusion Distributed File System

The Fusion Distributed File System Slide 1 / 44 The Fusion Distributed File System Dongfang Zhao February 2015 Slide 2 / 44 Outline Introduction FusionFS System Architecture Metadata Management Data Movement Implementation Details Unique

More information

Software and computing evolution: the HL-LHC challenge. Simone Campana, CERN

Software and computing evolution: the HL-LHC challenge. Simone Campana, CERN Software and computing evolution: the HL-LHC challenge Simone Campana, CERN Higgs discovery in Run-1 The Large Hadron Collider at CERN We are here: Run-2 (Fernando s talk) High Luminosity: the HL-LHC challenge

More information

CERN s Business Computing

CERN s Business Computing CERN s Business Computing Where Accelerated the infinitely by Large Pentaho Meets the Infinitely small Jan Janke Deputy Group Leader CERN Administrative Information Systems Group CERN World s Leading Particle

More information

10 Gbit/s Challenge inside the Openlab framework

10 Gbit/s Challenge inside the Openlab framework 10 Gbit/s Challenge inside the Openlab framework Sverre Jarp IT Division CERN SJ Feb 2003 1 Agenda Introductions All Overview Sverre Feedback Enterasys HP Intel Further discussions Elaboration of plan

More information

New Contributor Tutorial and Best Practices

New Contributor Tutorial and Best Practices New Contributor Tutorial and Best Practices Vicențiu Ciorbaru Software Engineer @ MariaDB Foundation * 2018 MariaDB Foundation * Goal of this session Most attendees here are highly experienced devs Let's

More information

Embedded Systems Dr. Santanu Chaudhury Department of Electrical Engineering Indian Institute of Technology, Delhi

Embedded Systems Dr. Santanu Chaudhury Department of Electrical Engineering Indian Institute of Technology, Delhi Embedded Systems Dr. Santanu Chaudhury Department of Electrical Engineering Indian Institute of Technology, Delhi Lecture - 13 Virtual memory and memory management unit In the last class, we had discussed

More information

Unit 2 : Computer and Operating System Structure

Unit 2 : Computer and Operating System Structure Unit 2 : Computer and Operating System Structure Lesson 1 : Interrupts and I/O Structure 1.1. Learning Objectives On completion of this lesson you will know : what interrupt is the causes of occurring

More information

itesla Power System Tools The open-source project for power grid simulations

itesla Power System Tools The open-source project for power grid simulations itesla Power System Tools The open-source project for power grid simulations ipst session RTE Tech Rain 1 st June, 2017 Paris La Défense, France ipst open-source project itesla platform source code available

More information

Symantec Design of DP Solutions for UNIX using NBU 5.0. Download Full Version :

Symantec Design of DP Solutions for UNIX using NBU 5.0. Download Full Version : Symantec 250-421 Design of DP Solutions for UNIX using NBU 5.0 Download Full Version : http://killexams.com/pass4sure/exam-detail/250-421 B. Applications running on the Windows clients will be suspended

More information

Misc. Third Generation Batch Multiprogramming. Fourth Generation Time Sharing. Last Time Evolution of OSs

Misc. Third Generation Batch Multiprogramming. Fourth Generation Time Sharing. Last Time Evolution of OSs Third Generation Batch Multiprogramming Misc. Problem: but I/O still expensive; can happen in middle of job Idea: have a pool of ready jobs in memory, switch to one when another needs I/O When one job

More information

A Brief Git Primer for CS 350

A Brief Git Primer for CS 350 A Brief Git Primer for CS 350 Tyler Szepesi (shamelessly stolen by Ben Cassell) University of Waterloo becassel@uwaterloo.ca September 8, 2017 Overview 1 Introduction 2 One-Time Setup 3 Using Git Git on

More information

Submitting your Work using GIT

Submitting your Work using GIT Submitting your Work using GIT You will be using the git distributed source control system in order to manage and submit your assignments. Why? allows you to take snapshots of your project at safe points

More information

CouchDB-based system for data management in a Grid environment Implementation and Experience

CouchDB-based system for data management in a Grid environment Implementation and Experience CouchDB-based system for data management in a Grid environment Implementation and Experience Hassen Riahi IT/SDC, CERN Outline Context Problematic and strategy System architecture Integration and deployment

More information

:59:32 PM PST

:59:32 PM PST Page 1 of 5 1 Group Database PHP workflow 2 3 The Linux side of the CS Lab machines is setup exactly as the Virtual 4 Box images in Scott. You have access to /srv/www/htdocs/php/punetid/ 5 and there is

More information

Git. Ľubomír Prda. IT4Innovations.

Git. Ľubomír Prda. IT4Innovations. Git Ľubomír Prda IT4Innovations lubomir.prda@vsb.cz support@it4i.cz VCS Version Control System Versioning - creation and management of multiple releases of a product, all of which have the same general

More information

PoS(High-pT physics09)036

PoS(High-pT physics09)036 Triggering on Jets and D 0 in HLT at ALICE 1 University of Bergen Allegaten 55, 5007 Bergen, Norway E-mail: st05886@alf.uib.no The High Level Trigger (HLT) of the ALICE experiment is designed to perform

More information

Section 1: Tools. Contents CS162. January 19, Make More details about Make Git Commands to know... 3

Section 1: Tools. Contents CS162. January 19, Make More details about Make Git Commands to know... 3 CS162 January 19, 2017 Contents 1 Make 2 1.1 More details about Make.................................... 2 2 Git 3 2.1 Commands to know....................................... 3 3 GDB: The GNU Debugger

More information

Database Services at CERN with Oracle 10g RAC and ASM on Commodity HW

Database Services at CERN with Oracle 10g RAC and ASM on Commodity HW Database Services at CERN with Oracle 10g RAC and ASM on Commodity HW UKOUG RAC SIG Meeting London, October 24 th, 2006 Luca Canali, CERN IT CH-1211 LCGenève 23 Outline Oracle at CERN Architecture of CERN

More information

What is version control? (discuss) Who has used version control? Favorite VCS? Uses of version control (read)

What is version control? (discuss) Who has used version control? Favorite VCS? Uses of version control (read) 1 For the remainder of the class today, I want to introduce you to a topic we will spend one or two more classes discussing and that is source code control or version control. What is version control?

More information

Data oriented job submission scheme for the PHENIX user analysis in CCJ

Data oriented job submission scheme for the PHENIX user analysis in CCJ Journal of Physics: Conference Series Data oriented job submission scheme for the PHENIX user analysis in CCJ To cite this article: T Nakamura et al 2011 J. Phys.: Conf. Ser. 331 072025 Related content

More information

Fault Detection using Advanced Analytics at CERN's Large Hadron Collider

Fault Detection using Advanced Analytics at CERN's Large Hadron Collider Fault Detection using Advanced Analytics at CERN's Large Hadron Collider Antonio Romero Marín Manuel Martin Marquez USA - 27/01/2016 BIWA 16 1 What s CERN USA - 27/01/2016 BIWA 16 2 What s CERN European

More information

GETTING STARTED WITH. Michael Lessard Senior Solutions Architect June 2017

GETTING STARTED WITH. Michael Lessard Senior Solutions Architect June 2017 GETTING STARTED WITH Michael Lessard Senior Solutions Architect June 2017 Agenda What is Git? Installation of Git Git basis Github First steps with Git 2 WHAT IS GIT? What is Git? Started in 2005 Created

More information

1Z0-433

1Z0-433 1Z0-433 Passing Score: 800 Time Limit: 0 min Exam A QUESTION 1 What is the function of the samfsdump utility? A. It provides a metadata backup of the file names, directory structure, inode information,

More information

Stephen J. Gowdy (CERN) 12 th September 2012 XLDB Conference FINDING THE HIGGS IN THE HAYSTACK(S)

Stephen J. Gowdy (CERN) 12 th September 2012 XLDB Conference FINDING THE HIGGS IN THE HAYSTACK(S) Stephen J. Gowdy (CERN) 12 th September 2012 XLDB Conference FINDING THE HIGGS IN THE HAYSTACK(S) Overview Large Hadron Collider (LHC) Compact Muon Solenoid (CMS) experiment The Challenge Worldwide LHC

More information

Benchmarking Over The Grid Using SEE Virtual Organization.

Benchmarking Over The Grid Using SEE Virtual Organization. Benchmarking Over The Grid Using SEE Virtual Organization. January 2007 Technical Report Ioannis Kouvakis 1 and Fotis Georgatos 2 1 Student of Univerity of Aegean, Department of Mathematics, gkouvakis@hep.ntua.gr

More information

CSCE UVM Hands-on Session-1 Pre-Work

CSCE UVM Hands-on Session-1 Pre-Work CSCE489-689 UVM Hands-on Session-1 Pre-Work Please complete the following steps before the lecture on Feb-16. These steps will help set-up the environment and tools necessary for the hands-on session.

More information

The Nios II Family of Configurable Soft-core Processors

The Nios II Family of Configurable Soft-core Processors The Nios II Family of Configurable Soft-core Processors James Ball August 16, 2005 2005 Altera Corporation Agenda Nios II Introduction Configuring your CPU FPGA vs. ASIC CPU Design Instruction Set Architecture

More information

Big Computing and the Mitchell Institute for Fundamental Physics and Astronomy. David Toback

Big Computing and the Mitchell Institute for Fundamental Physics and Astronomy. David Toback Big Computing and the Mitchell Institute for Fundamental Physics and Astronomy Texas A&M Big Data Workshop October 2011 January 2015, Texas A&M University Research Topics Seminar 1 Outline Overview of

More information

CHARON-VAX application note

CHARON-VAX application note CHARON-VAX application note AN-036 Testing various Shared Disk VAX-cluster Author: Ralf van Diesen Updated: 16 September 2005 CLUSTERING WITH DIRECT ACCESS USING SHARED STORAGE VAX/VMS clusters can be

More information

PORTAL. A Case Study. Dr. Kristin Tufte Mark Wong September 23, Linux Plumbers Conference 2009

PORTAL. A Case Study. Dr. Kristin Tufte Mark Wong September 23, Linux Plumbers Conference 2009 PORTAL A Case Study Dr. Kristin Tufte (tufte@cecs.pdx.edu) Mark Wong (markwkm@postgresql.org) Linux Plumbers Conference 2009 September 23, 2009 Overview What is PORTAL? How PORTAL works Improving PORTAL

More information

Grid Computing: dealing with GB/s dataflows

Grid Computing: dealing with GB/s dataflows Grid Computing: dealing with GB/s dataflows Jan Just Keijser, Nikhef janjust@nikhef.nl David Groep, NIKHEF 3 May 2012 Graphics: Real Time Monitor, Gidon Moont, Imperial College London, see http://gridportal.hep.ph.ic.ac.uk/rtm/

More information

Laboratorio di Programmazione. Prof. Marco Bertini

Laboratorio di Programmazione. Prof. Marco Bertini Laboratorio di Programmazione Prof. Marco Bertini marco.bertini@unifi.it http://www.micc.unifi.it/bertini/ Code versioning: techniques and tools Software versions All software has multiple versions: Each

More information

Exam : S Title : Snia Storage Network Management/Administration. Version : Demo

Exam : S Title : Snia Storage Network Management/Administration. Version : Demo Exam : S10-200 Title : Snia Storage Network Management/Administration Version : Demo 1. A SAN architect is asked to implement an infrastructure for a production and a test environment using Fibre Channel

More information

ATLAS NOTE. December 4, ATLAS offline reconstruction timing improvements for run-2. The ATLAS Collaboration. Abstract

ATLAS NOTE. December 4, ATLAS offline reconstruction timing improvements for run-2. The ATLAS Collaboration. Abstract ATLAS NOTE December 4, 2014 ATLAS offline reconstruction timing improvements for run-2 The ATLAS Collaboration Abstract ATL-SOFT-PUB-2014-004 04/12/2014 From 2013 to 2014 the LHC underwent an upgrade to

More information

Extraordinary HPC file system solutions at KIT

Extraordinary HPC file system solutions at KIT Extraordinary HPC file system solutions at KIT Roland Laifer STEINBUCH CENTRE FOR COMPUTING - SCC KIT University of the State Roland of Baden-Württemberg Laifer Lustre and tools for ldiskfs investigation

More information

At course completion. Overview. Audience profile. Course Outline. : 55187B: Linux System Administration. Course Outline :: 55187B::

At course completion. Overview. Audience profile. Course Outline. : 55187B: Linux System Administration. Course Outline :: 55187B:: Module Title Duration : 55187B: Linux System Administration : 4 days Overview This four-day instructor-led course is designed to provide students with the necessary skills and abilities to work as a professional

More information

Front-End Electronics Configuration System for CMS. Philippe Gras CERN - University of Karlsruhe

Front-End Electronics Configuration System for CMS. Philippe Gras CERN - University of Karlsruhe Front-End Electronics Configuration System for CMS Philippe Gras CERN - University of Karlsruhe Outline Introduction Tracker electronics parameters Tracker beam test DCS overview Electronics configuration

More information

Using Arcserve Backup-R17 product with Amazon Web Services(AWS) Storage Gateway-VTL

Using Arcserve Backup-R17 product with Amazon Web Services(AWS) Storage Gateway-VTL Using Arcserve Backup-R17 product with Amazon Web Services(AWS) Storage Gateway-VTL 1. Section-1: Using Arcserve Backup-R17 product with Amazon Web Services (AWS) Storage Gateway-VTL. 2. Section-2: Workarounds

More information

(Cloud9) and to the Remote Repository (GitHub)

(Cloud9) and to the Remote Repository (GitHub) 1 2 3 Add Commit Push Steps to Move files into the Local Repository (Cloud9) and to the Remote Repository (GitHub) Assignment Steps: Step #1: Create a GitHub account Step #2: Link Cloud9 account to GitHub

More information

Initial Explorations of ARM Processors for Scientific Computing

Initial Explorations of ARM Processors for Scientific Computing Initial Explorations of ARM Processors for Scientific Computing Peter Elmer - Princeton University David Abdurachmanov - Vilnius University Giulio Eulisse, Shahzad Muzaffar - FNAL Power limitations for

More information

Lecture 01 - Working with Linux Servers and Git

Lecture 01 - Working with Linux Servers and Git Jan. 9, 2018 Working with Linux Servers: SSH SSH (named for Secure SHell) is a protocol commonly used for remote login. You can use it from a command line interface with the following syntax ssh username@server_url

More information

SCAP Security Guide Questions / Answers. Contributor WorkShop Volume #2

SCAP Security Guide Questions / Answers. Contributor WorkShop Volume #2 SCAP Security Guide Questions / Answers Contributor WorkShop Volume #2 Ján Lieskovský January 2016 Agenda Introductory Notes Source Code / Repository Notes (Moved to Appendix for self-study) SCAP Security

More information

"Charting the Course... MOC B: Linux System Administration. Course Summary

Charting the Course... MOC B: Linux System Administration. Course Summary Description Course Summary This four-day instructor-led course is designed to provide students with the necessary skills and abilities to work as a professional Linux system administrator. The course covers

More information

Nexus Builder Developing a Graphical User Interface to create NeXus files

Nexus Builder Developing a Graphical User Interface to create NeXus files Nexus Builder Developing a Graphical User Interface to create NeXus files Lilit Grigoryan, Yerevan State University, Armenia September 9, 2014 Abstract This report describes a project which main purpose

More information

Shared Memory Programming With OpenMP Exercise Instructions

Shared Memory Programming With OpenMP Exercise Instructions Shared Memory Programming With OpenMP Exercise Instructions John Burkardt Interdisciplinary Center for Applied Mathematics & Information Technology Department Virginia Tech... Advanced Computational Science

More information

Software Project (Lecture 4): Git & Github

Software Project (Lecture 4): Git & Github Software Project (Lecture 4): Git & Github Wouter Swierstra, Atze Dijkstra Feb 2016 Wouter Swierstra, Atze Dijkstra Software Project (Lecture 4): Git & Github Feb 2016 1 / 45 Wouter Swierstra, Atze Dijkstra

More information

Vaango Installation Guide

Vaango Installation Guide Vaango Installation Guide Version Version 17.10 October 1, 2017 The Utah Vaango team and Biswajit Banerjee Copyright 2015-2017 Parresia Research Limited The contents of this manual can and will change

More information

Software Revision Control for MASS. Git Installation / Configuration / Use

Software Revision Control for MASS. Git Installation / Configuration / Use Software Revision Control for MASS Git Installation / Configuration / Use Matthew Sell, CSSE Student MASS Research Participant, February 2014 Overview Download / execute installer Initial configuration

More information

Singularity in CMS. Over a million containers served

Singularity in CMS. Over a million containers served Singularity in CMS Over a million containers served Introduction The topic of containers is broad - and this is a 15 minute talk! I m filtering out a lot of relevant details, particularly why we are using

More information

Chapter 03. Authors: John Hennessy & David Patterson. Copyright 2011, Elsevier Inc. All rights Reserved. 1

Chapter 03. Authors: John Hennessy & David Patterson. Copyright 2011, Elsevier Inc. All rights Reserved. 1 Chapter 03 Authors: John Hennessy & David Patterson Copyright 2011, Elsevier Inc. All rights Reserved. 1 Figure 3.3 Comparison of 2-bit predictors. A noncorrelating predictor for 4096 bits is first, followed

More information

The JINR Tier1 Site Simulation for Research and Development Purposes

The JINR Tier1 Site Simulation for Research and Development Purposes EPJ Web of Conferences 108, 02033 (2016) DOI: 10.1051/ epjconf/ 201610802033 C Owned by the authors, published by EDP Sciences, 2016 The JINR Tier1 Site Simulation for Research and Development Purposes

More information

Draft: MLDesigner and LinCVS

Draft: MLDesigner and LinCVS Draft: MLDesigner and LinCVS 17th April 2003 Daniel Zinn zinn@mldesigner.com MLDesign Technologies, Inc. 2230 St. Francis Drive Palo Alto, CA 94303 support : www.mldesigner.com/support http : www.mldesigner.com

More information

Scalasca support for Intel Xeon Phi. Brian Wylie & Wolfgang Frings Jülich Supercomputing Centre Forschungszentrum Jülich, Germany

Scalasca support for Intel Xeon Phi. Brian Wylie & Wolfgang Frings Jülich Supercomputing Centre Forschungszentrum Jülich, Germany Scalasca support for Intel Xeon Phi Brian Wylie & Wolfgang Frings Jülich Supercomputing Centre Forschungszentrum Jülich, Germany Overview Scalasca performance analysis toolset support for MPI & OpenMP

More information

Data services for LHC computing

Data services for LHC computing Data services for LHC computing SLAC 1 Xavier Espinal on behalf of IT/ST DAQ to CC 8GB/s+4xReco Hot files Reliable Fast Processing DAQ Feedback loop WAN aware Tier-1/2 replica, multi-site High throughout

More information

Triton file systems - an introduction. slide 1 of 28

Triton file systems - an introduction. slide 1 of 28 Triton file systems - an introduction slide 1 of 28 File systems Motivation & basic concepts Storage locations Basic flow of IO Do's and Don'ts Exercises slide 2 of 28 File systems: Motivation Case #1:

More information