FACADE Financial Analysis Computing Architecture in Distributed Environment
|
|
|
- Jared York
- 5 years ago
- Views:
Transcription
1 FACADE Financial Analysis Computing Architecture in Distributed Environment V. Motoška, L. Slebodník, M. Jurečko, M. Zvada May 4, 2011
2 Outline Motivation CADE Middleware Future work 2 / 19
3 What? Motivation financial markets prediction and analysis simulations of statistical and computational finance models From the Currency Rate Quotations onto Strings and Brane World Scenarios 1 (Phys.Rev.E) Agent Based Simulations of Prediction Algorithms Memory Prediction Framework Model for Clustering the Financial Market Behaviors / 19
4 How? Motivation Hardware Intel Xeon E5530 quad-core based worker nodes NVIDIA GeForce GTX 480, GTX 285, Tesla C1060 Software CentOS 5.5 x86_64 managed by Puppet Latest Red Hat MRG Condor with our middleware central authentication and account management via LDAP and Kerberos GNU Octave, MATLAB, ROOT 2 custom applications developed in Python, Perl, C++, OpenCL / 19
5 Why? Motivation Condor huge processing power (HTC) high customizability community support MRG messaging for our Java based trading application almost real-time computation (algorithmic trading) rebuilt under CentOS in mock environment / 19
6 Why? Motivation NVIDIA GPUs according to our tests 4 best price vs. performance OpenCL maturity and completeness Linux drivers stability CentOS RHEL clone MRG compatible / 19
7 CADE Middleware Challenges long running jobs processing quite large amount of WORM data producing comparable amount of results data manipulation portable (data locality independence) reliable (fault tolerance during computing) optimal (file transfer control) isolate users from unnecessary technical details obtain a higher degree of Condor customizability 7 / 19
8 CADE Middleware CADE URL To solve storage related problems we introduced CADE URL cade:// sors/ users/ foo/bar pool name storage relative path cade://sors/users/jdoe/math1 cade://sors/pse/oanda/eurusd/?ds= &de= &closing cade://sors/bin/gcc?version=3.4 taking care not to overwrite concurrently modified data possibility to append data to the existing files warning on any possibly unsafe action map concrete storage according to configuration file 8 / 19
9 CADE Middleware UI Submit /home/users/johndoe/data/file translate cade://sors/users/johndoe/data/file Stage In WN Exec Stage Out 9 / 19
10 CADE Middleware UI Submit /home/users/johndoe/data/file translate cade://sors/users/johndoe/data/file translate Stage In 1. /export/home/users/johndoe/data/file (on NFS) 2. copy out to Condor sandbox (/var/execute/1234/ /data/file) WN Exec Stage Out 10 / 19
11 CADE Middleware UI Submit /home/users/johndoe/data/file translate cade://sors/users/johndoe/data/file translate Stage In 1. /export/home/users/johndoe/data/file (on NFS) 2. copy out to Condor sandbox (/var/execute/1234/ /data/file) WN Exec Executing job... Stage Out 11 / 19
12 CADE Middleware UI Submit /home/users/johndoe/data/file translate cade://sors/users/johndoe/data/file translate Stage In 1. /export/home/users/johndoe/data/file (on NFS) 2. copy out to Condor sandbox (/var/execute/1234/ /data/file) WN Exec Executing job... Stage Out 1. output URL (cade://sors/results/strings/1.csv) SOURCE: /var/execute/1234/home/ /1.csv TARGET: ftp://carina.sors.local/strings/1.csv 2. copy in SOURCE to TARGET 12 / 19
13 Benefits of CADE URLs CADE Middleware portability GRID/SRM ready can access various data back-ends NFS/dCache/PSE/Xrootd/gFTP... reliability protection against accidental overwrites no storage SPOF during job execution optimization possibilities storage side filtering/preprocessing online data compression caching 13 / 19
14 CADE Middleware CADE Job Description Language similar to condor_submit syntax (no need for backward compatibility) domain specific syntax additional layer covering various submission methods condor_submit/web Services/QMF created for user convenience 14 / 19
15 Example.jdl file CADE Middleware #!/usr/bin/env cade_submit arguments = subdir%ix%/infile %o1% executable = copyprogram # o1 alias input = $HOME/subdir%ix%/ # stage in output = "output file %ix%" # $(ProcID) + 2 output = common.txt :append # do not overwrite stdout = %ix%/stdout.txt parametric = 2:5 15 / 19
16 Example.csf file CADE Middleware arguments = subdir%ix%/infile cade://sors/.../work_dir/outfile%25ix%25 +cade_argument0 = "c3vizglyjwl4js9pbmzpbgu=" +cade_argument1 = "Y2FkZTovL3NvcnMvdXNlcnMvc2xlYm9kbmlrL3N2bmtvL2NhZGUvdWkvdGVzdHMvMDNfY2FkZV9zdWJtaXQ=" +cade_arguments = true +cade_cwd = "cade://sors/.../work_dir/" +cade_executable = "cade://sors/.../work_dir/copyprogram" +cade_input0 = "cade://sors/.../home/.../subdir%25ix%25" +cade_orig_cwd = "/home/.../work_dir/" +cade_orig_executable = "copyprogram" +cade_orig_input0 = "/home/users/.../subdir%ix%/" +cade_orig_jdl = "cade://sors/.../work_dir/03_positional_templates.jdl" +cade_orig_output0 = "outfile%ix%" +cade_orig_output1 = "common.txt:append" +cade_output0 = "cade://sors/.../work_dir/outfile%25ix%25" +cade_output1 = "cade://sors/.../work_dir/common.txt%3aappend" +cade_parametric_end = 5 +cade_parametric_start = 2 +cade_stdout = "cade://sors/.../work_dir/%25ix%25/stdout.txt" +cade_submit_directory = "cade://sors/service/submits/ui01dev.cade.sors.local/000083_ /" executable = 03_positional_templates:copyProgram log = /service/submits/ui01dev.cade.sors.local/000083_ /logs/condor_user notification = Complete notify_user = slebodnik@sors.com transfer_executable = false transfer_input_files = /service/submits/ui01dev.cade.sors.local/000083_ /csf transfer_output_files = universe = vanilla wantremoteio = false whentotransferoutput = ON_EXIT queue 4 16 / 19
17 CADE Middleware CADE Service directory structure /submits/ <submitter>/ _day/ job/ /jobs/ <submitter>/ <cluster id>/ submit/ stages/ 0001_<stage name>/ <section id>/<worker node>/0001_<stage name>/ /users/ <user>/ jobs/ <submitter>/ <cluster id>/ submits/ _day/ contents and purpose middleware logs, Condor user and failure logs working directory snapshots for each stage job and machine classadds 17 / 19
18 CADE Development CADE Middleware development environment XCP based (fast re-provisioning) Condor instances and configuration per developer $ dev_env_reset $ dev_env_run all query $ dev_env_run head install $ dev_env_run ui01dev start integration tests following whole job life-cycle from submission (condor_sumbit interoperation) through run on worker nodes to results checking 18 / 19
19 Future work Future work migrate Cfengine to Puppet move service directory from NFS Cumin LDAP integration (ready on Dev env) cade_query interactive monitoring status of each executed stage history of stages on worker user requests: enhanced mail notifications, application checkpointing (DMTCP) / 19
Introduction to HTCondor
Introduction to HTCondor Kenyi Hurtado June 17, 2016 1 Covered in this presentation What is HTCondor? How to run a job (and multiple ones) Monitoring your queue 2 What is HTCondor? A specialized workload
HTCONDOR USER TUTORIAL. Greg Thain Center for High Throughput Computing University of Wisconsin Madison
HTCONDOR USER TUTORIAL Greg Thain Center for High Throughput Computing University of Wisconsin Madison gthain@cs.wisc.edu 2015 Internet2 HTCondor User Tutorial CONTENTS Overview Basic job submission How
Day 9: Introduction to CHTC
Day 9: Introduction to CHTC Suggested reading: Condor 7.7 Manual: http://www.cs.wisc.edu/condor/manual/v7.7/ Chapter 1: Overview Chapter 2: Users Manual (at most, 2.1 2.7) 1 Turn In Homework 2 Homework
Introduction to Condor. Jari Varje
Introduction to Condor Jari Varje 25. 27.4.2016 Outline Basics Condor overview Submitting a job Monitoring jobs Parallel jobs Advanced topics Host requirements Running MATLAB jobs Checkpointing Case study:
Tutorial 4: Condor. John Watt, National e-science Centre
Tutorial 4: Condor John Watt, National e-science Centre Tutorials Timetable Week Day/Time Topic Staff 3 Fri 11am Introduction to Globus J.W. 4 Fri 11am Globus Development J.W. 5 Fri 11am Globus Development
HPC Resources at Lehigh. Steve Anthony March 22, 2012
HPC Resources at Lehigh Steve Anthony March 22, 2012 HPC at Lehigh: Resources What's Available? Service Level Basic Service Level E-1 Service Level E-2 Leaf and Condor Pool Altair Trits, Cuda0, Inferno,
Support for Vanilla Universe Checkpointing. Thomas Downes University of Wisconsin-Milwaukee (LIGO)
Support for Vanilla Universe Checkpointing Thomas Downes University of Wisconsin-Milwaukee (LIGO) Experimental feature! All features discussed are present in the official 8.5 releases. The Morgridge Institute
Grid Compute Resources and Job Management
Grid Compute Resources and Job Management How do we access the grid? Command line with tools that you'll use Specialised applications Ex: Write a program to process images that sends data to run on the
An introduction to checkpointing. for scientific applications
damien.francois@uclouvain.be UCL/CISM - FNRS/CÉCI An introduction to checkpointing for scientific applications November 2013 CISM/CÉCI training session What is checkpointing? Without checkpointing: $./count
Architecture Proposal
Nordic Testbed for Wide Area Computing and Data Handling NORDUGRID-TECH-1 19/02/2002 Architecture Proposal M.Ellert, A.Konstantinov, B.Kónya, O.Smirnova, A.Wäänänen Introduction The document describes
Introduction to HPC Using zcluster at GACRC
Introduction to HPC Using zcluster at GACRC Georgia Advanced Computing Resource Center University of Georgia Zhuofei Hou, HPC Trainer zhuofei@uga.edu Outline What is GACRC? What is HPC Concept? What is
The Why and How of HPC-Cloud Hybrids with OpenStack
The Why and How of HPC-Cloud Hybrids with OpenStack OpenStack Australia Day Melbourne June, 2017 Lev Lafayette, HPC Support and Training Officer, University of Melbourne lev.lafayette@unimelb.edu.au 1.0
History of SURAgrid Deployment
All Hands Meeting: May 20, 2013 History of SURAgrid Deployment Steve Johnson Texas A&M University Copyright 2013, Steve Johnson, All Rights Reserved. Original Deployment Each job would send entire R binary
Red Hat enterprise virtualization 3.0
Red Hat enterprise virtualization 3.0 feature comparison at a glance Red Hat Enterprise is the first fully open source, enterprise ready virtualization platform Compare the functionality of RHEV to VMware
High Performance Computing (HPC) Using zcluster at GACRC
High Performance Computing (HPC) Using zcluster at GACRC On-class STAT8060 Georgia Advanced Computing Resource Center University of Georgia Zhuofei Hou, HPC Trainer zhuofei@uga.edu Outline What is GACRC?
Center for Mathematical Modeling University of Chile HPC 101. HPC systems basics and concepts. Juan Carlos Maureira B.
Center for Mathematical Modeling University of Chile HPC 101 HPC systems basics and concepts By Juan Carlos Maureira B. BioMedicina II Calculo Masivo en Biomedicina CMM - FCFM - University
Project Blackbird. U"lizing Condor and HTC to address archiving online courses at Clemson on a weekly basis. Sam Hoover
Project Blackbird U"lizing Condor and HTC to address archiving online courses at Clemson on a weekly basis Sam Hoover shoover@clemson.edu 1 Project Blackbird Blackboard at Clemson End of Semester archives
Red Hat enterprise virtualization 3.1 feature comparison
Red Hat enterprise virtualization 3.1 feature comparison at a glance Red Hat Enterprise Virtualization 3.1 is first fully open source, enterprise ready virtualization platform Compare functionality of
Introduction to HPC Using zcluster at GACRC
Introduction to HPC Using zcluster at GACRC On-class PBIO/BINF8350 Georgia Advanced Computing Resource Center University of Georgia Zhuofei Hou, HPC Trainer zhuofei@uga.edu Outline What is GACRC? What
Day 1 : August (Thursday) An overview of Globus Toolkit 2.4
An Overview of Grid Computing Workshop Day 1 : August 05 2004 (Thursday) An overview of Globus Toolkit 2.4 By CDAC Experts Contact :vcvrao@cdacindia.com; betatest@cdacindia.com URL : http://www.cs.umn.edu/~vcvrao
30 Nov Dec Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy
Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy Why the Grid? Science is becoming increasingly digital and needs to deal with increasing amounts of
Grid Compute Resources and Grid Job Management
Grid Compute Resources and Job Management March 24-25, 2007 Grid Job Management 1 Job and compute resource management! This module is about running jobs on remote compute resources March 24-25, 2007 Grid
Introduction to HPC Using zcluster at GACRC
Introduction to HPC Using zcluster at GACRC On-class STAT8330 Georgia Advanced Computing Resource Center University of Georgia Suchitra Pakala pakala@uga.edu Slides courtesy: Zhoufei Hou 1 Outline What
Eurogrid: a glideinwms based portal for CDF data analysis - 19th January 2012 S. Amerio. (INFN Padova) on behalf of Eurogrid support group
Eurogrid: a glideinwms based portal for CDF data analysis - 19th January 2012 S. Amerio (INFN Padova) on behalf of Eurogrid support group CDF computing model CDF computing model is based on Central farm
Presented by: Jon Wedell BioMagResBank
HTCondor Tutorial Presented by: Jon Wedell BioMagResBank wedell@bmrb.wisc.edu Background During this tutorial we will walk through submitting several jobs to the HTCondor workload management system. We
The GridWay. approach for job Submission and Management on Grids. Outline. Motivation. The GridWay Framework. Resource Selection
The GridWay approach for job Submission and Management on Grids Eduardo Huedo Rubén S. Montero Ignacio M. Llorente Laboratorio de Computación Avanzada Centro de Astrobiología (INTA - CSIC) Associated to
ITNPBD7 Cluster Computing Spring Using Condor
The aim of this practical is to work through the Condor examples demonstrated in the lectures and adapt them to alternative tasks. Before we start, you will need to map a network drive to \\wsv.cs.stir.ac.uk\datasets
Introduction. Distributed Systems IT332
Introduction Distributed Systems IT332 2 Outline Definition of A Distributed System Goals of Distributed Systems Types of Distributed Systems 3 Definition of A Distributed System A distributed systems
Update on EZ-Grid. Priya Raghunath University of Houston. PI : Dr Barbara Chapman
Update on EZ-Grid Priya Raghunath University of Houston PI : Dr Barbara Chapman chapman@cs.uh.edu Outline Campus Grid at the University of Houston (UH) Functionality of EZ-Grid Design and Implementation
Zadara Enterprise Storage in
Zadara Enterprise Storage in Google Cloud Platform (GCP) Deployment Guide March 2017 Revision A 2011 2017 ZADARA Storage, Inc. All rights reserved. Zadara Storage / GCP - Deployment Guide Page 1 Contents
Introduction Matlab Compiler Matlab & Condor Conclusion. From Compilation to Distribution. David J. Herzfeld
Matlab R and MUGrid From Compilation to Distribution David J. Herzfeld david.herzfeld@marquette.edu Marquette University Department of Biomedical Engineering July 15, 2010 David J. Herzfeld (Marquette
Introduction to HPC Using zcluster at GACRC On-Class GENE 4220
Introduction to HPC Using zcluster at GACRC On-Class GENE 4220 Georgia Advanced Computing Resource Center University of Georgia Suchitra Pakala pakala@uga.edu Slides courtesy: Zhoufei Hou 1 OVERVIEW GACRC
Grid Computing Fall 2005 Lecture 5: Grid Architecture and Globus. Gabrielle Allen
Grid Computing 7700 Fall 2005 Lecture 5: Grid Architecture and Globus Gabrielle Allen allen@bit.csc.lsu.edu http://www.cct.lsu.edu/~gallen Concrete Example I have a source file Main.F on machine A, an
FROM HPC TO THE CLOUD WITH AMQP AND OPEN SOURCE SOFTWARE
FROM HPC TO THE CLOUD WITH AMQP AND OPEN SOURCE SOFTWARE Carl Trieloff cctrieloff@redhat.com Red Hat Lee Fisher lee.fisher@hp.com Hewlett-Packard High Performance Computing on Wall Street conference 14
Centre de Calcul de l Institut National de Physique Nucléaire et de Physique des Particules. Singularity overview. Vanessa HAMAR
Centre de Calcul de l Institut National de Physique Nucléaire et de Physique des Particules Singularity overview Vanessa HAMAR Disclaimer } The information in this presentation was compiled from different
Red Hat Virtualization 4.1 Product Guide
Red Hat Virtualization 4.1 Product Guide Introduction to Red Hat Virtualization 4.1 Red Hat Virtualization Documentation TeamRed Hat Red Hat Virtualization 4.1 Product Guide Introduction to Red Hat Virtualization
Introduction to High Performance Computing Using Sapelo2 at GACRC
Introduction to High Performance Computing Using Sapelo2 at GACRC Georgia Advanced Computing Resource Center University of Georgia Suchitra Pakala pakala@uga.edu 1 Outline High Performance Computing (HPC)
Copyright 2016 Pivotal. All rights reserved. Cloud Native Design. Includes 12 Factor Apps
1 Cloud Native Design Includes 12 Factor Apps Topics 12-Factor Applications Cloud Native Design Guidelines 2 http://12factor.net Outlines architectural principles and patterns for modern apps Focus on
First evaluation of the Globus GRAM Service. Massimo Sgaravatto INFN Padova
First evaluation of the Globus GRAM Service Massimo Sgaravatto INFN Padova massimo.sgaravatto@pd.infn.it Draft version release 1.0.5 20 June 2000 1 Introduction...... 3 2 Running jobs... 3 2.1 Usage examples.
! " # " $ $ % & '(()
!"# " $ $ % &'(() First These slides are available from: http://www.cs.wisc.edu/~roy/italy-condor/ 2 This Morning s Condor Topics * +&, & - *.&- *. & * && - * + $ 3 Part One Matchmaking: Finding Machines
Clustering. Research and Teaching Unit
Clustering Research and Teaching Unit Disclaimer...though it cannot hope to be useful or informative on all matters, it does at least make the reassuring claim, that where it is inaccurate it is at least
ONTAP 9 Cluster Administration. Course outline. Authorised Vendor e-learning. Guaranteed To Run. DR Digital Learning. Module 1: ONTAP Overview
ONTAP 9 Cluster Administration Course Code: Duration: 3 Days Product Page: https://digitalrevolver.com/product/ontap-9-cluster-administration-2/ This 3-day, instructor led course uses lecture and hands-on
An Introduction to Using HTCondor Karen Miller
An Introduction to Using HTCondor 2015 Karen Miller The Team - 2014 Established in 1985, to do research and development of distributed high-throughput computing 2 HT Stands for High Throughput Throughput:
Introduction to High Performance Computing (HPC) Resources at GACRC
Introduction to High Performance Computing (HPC) Resources at GACRC Georgia Advanced Computing Resource Center University of Georgia Zhuofei Hou, HPC Trainer zhuofei@uga.edu Outline What is GACRC? Concept
The rcuda middleware and applications
The rcuda middleware and applications Will my application work with rcuda? rcuda currently provides binary compatibility with CUDA 5.0, virtualizing the entire Runtime API except for the graphics functions,
Outline. ASP 2012 Grid School
Distributed Storage Rob Quick Indiana University Slides courtesy of Derek Weitzel University of Nebraska Lincoln Outline Storage Patterns in Grid Applications Storage
PARALLEL PROGRAM EXECUTION SUPPORT IN THE JGRID SYSTEM
PARALLEL PROGRAM EXECUTION SUPPORT IN THE JGRID SYSTEM Szabolcs Pota 1, Gergely Sipos 2, Zoltan Juhasz 1,3 and Peter Kacsuk 2 1 Department of Information Systems, University of Veszprem, Hungary 2 Laboratory
Bright Cluster Manager Advanced HPC cluster management made easy. Martijn de Vries CTO Bright Computing
Bright Cluster Manager Advanced HPC cluster management made easy Martijn de Vries CTO Bright Computing About Bright Computing Bright Computing 1. Develops and supports Bright Cluster Manager for HPC systems
Ganga The Job Submission Tool. WeiLong Ueng
Ganga The Job Submission Tool WeiLong Ueng wlueng@twgrid.org Objectives This tutorial gives users to understand Why require Ganga in Grid environment What advantages of Ganga The Architecture of Ganga
Parallel and Distributed Computing
Parallel and Distributed Computing NUMA; OpenCL; MapReduce José Monteiro MSc in Information Systems and Computer Engineering DEA in Computational Engineering Department of Computer Science and Engineering
Deploying virtualisation in a production grid
Deploying virtualisation in a production grid Stephen Childs Trinity College Dublin & Grid-Ireland TERENA NRENs and Grids workshop 2 nd September 2008 www.eu-egee.org EGEE and glite are registered trademarks
CloudBATCH: A Batch Job Queuing System on Clouds with Hadoop and HBase. Chen Zhang Hans De Sterck University of Waterloo
CloudBATCH: A Batch Job Queuing System on Clouds with Hadoop and HBase Chen Zhang Hans De Sterck University of Waterloo Outline Introduction Motivation Related Work System Design Future Work Introduction
Getting Started with OSG Connect ~ an Interactive Tutorial ~
Getting Started with OSG Connect ~ an Interactive Tutorial ~ Emelie Harstad , Mats Rynge , Lincoln Bryant , Suchandra Thapa ,
NOVELL NETWARE 5.1 ADMINISTRATION CONTENTS
CNA Certification Guide: Novell NetWare 5.1 Administration CERTIFICATION GUIDE: NOVELL NETWARE 5.1 ADMINISTRATION CONTENTS AUTHOR ACKNOWLEDGEMENTS... XIII INTRODUCTION... XV UNIT 1 NETWARE ACCESS... 1-1
NUSGRID a computational grid at NUS
NUSGRID a computational grid at NUS Grace Foo (SVU/Academic Computing, Computer Centre) SVU is leading an initiative to set up a campus wide computational grid prototype at NUS. The initiative arose out
Developing and Deploying vsphere Solutions, vservices, and ESX Agents. 17 APR 2018 vsphere Web Services SDK 6.7 vcenter Server 6.7 VMware ESXi 6.
Developing and Deploying vsphere Solutions, vservices, and ESX Agents 17 APR 2018 vsphere Web Services SDK 6.7 vcenter Server 6.7 VMware ESXi 6.7 You can find the most up-to-date technical documentation
Developing and Deploying vsphere Solutions, vservices, and ESX Agents
Developing and Deploying vsphere Solutions, vservices, and ESX Agents vsphere 6.0 This document supports the version of each product listed and supports all subsequent versions until the document is replaced
Checkpointing using DMTCP, Condor, Matlab and FReD
Checkpointing using DMTCP, Condor, Matlab and FReD Gene Cooperman (presenting) High Performance Computing Laboratory College of Computer and Information Science Northeastern University, Boston gene@ccs.neu.edu
What s new in HTCondor? What s coming? HTCondor Week 2018 Madison, WI -- May 22, 2018
What s new in HTCondor? What s coming? HTCondor Week 2018 Madison, WI -- May 22, 2018 Todd Tannenbaum Center for High Throughput Computing Department of Computer Sciences University of Wisconsin-Madison
Proactive maintenance and adaptive power management using Dell OpenManage Systems Management for VMware DRS Clusters
Proactive maintenance and adaptive power management using Dell OpenManage Systems Management for ware DRS Clusters White Paper Balasubramanian Chandrasekaran, Puneet Dhawan Dell Virtualization Solutions
Developing and Deploying vsphere Solutions, vservices, and ESX Agents
Developing and Deploying vsphere Solutions, vservices, and ESX Agents vsphere 5.0 This document supports the version of each product listed and supports all subsequent versions until the document is replaced
Architecture of the WMS
Architecture of the WMS Dr. Giuliano Taffoni INFORMATION SYSTEMS UNIT Outline This presentation will cover the following arguments: Overview of WMS Architecture Job Description Language Overview WMProxy
DMTCP: Fixing the Single Point of Failure of the ROS Master
DMTCP: Fixing the Single Point of Failure of the ROS Master Tw i n k l e J a i n j a i n. t @ h u s k y. n e u. e d u G e n e C o o p e r m a n g e n e @ c c s. n e u. e d u C o l l e g e o f C o m p u
Developing and Deploying vsphere Solutions, vservices, and ESX Agents
Developing and Deploying vsphere Solutions, vservices, and ESX Agents Modified on 27 JUL 2017 vsphere Web Services SDK 6.5 vcenter Server 6.5 VMware ESXi 6.5 Developing and Deploying vsphere Solutions,
GPU Clouds IGT Cloud Computing Summit Mordechai Butrashvily, CEO 2009 (c) All rights reserved
GPU Clouds IGT 2009 Cloud Computing Summit Mordechai Butrashvily, CEO moti@hoopoe-cloud.com 02/12/2009 Agenda Introduction to GPU Computing Future GPU architecture GPU on a Cloud: Visualization Computing
Scientific Cluster Deployment and Recovery Using puppet to simplify cluster management
Journal of Physics: Conference Series Scientific Cluster Deployment and Recovery Using puppet to simplify cluster management To cite this article: Val Hendrix et al 2012 J. Phys.: Conf. Ser. 396 042027
HIGH-THROUGHPUT COMPUTING AND YOUR RESEARCH
HIGH-THROUGHPUT COMPUTING AND YOUR RESEARCH Christina Koch, Research Computing Facilitator Center for High Throughput Computing STAT679, October 29, 2018 1 About Me I work for the Center for High Throughput
Concurrency User Guide
Concurrency User Guide Release 1.0 Dylan Hackers January 26, 2019 CONTENTS 1 Basic Abstractions 3 1.1 Executors................................................. 3 1.2 Queues..................................................
! " #$%! &%& ' ( $ $ ) $*+ $
! " #$%! &%& ' ( $ $ ) $*+ $ + 2 &,-)%./01 ) 2 $ & $ $ ) 340 %(%% 3 &,-)%./01 ) 2 $& $ $ ) 34 0 %(%% $ $ %% $ ) 5 67 89 5 & % % %$ %%)( % % ( %$ ) ( '$ :!"#$%%&%'&( )!)&(!&( *+,& )- &*./ &*( ' 0&/ 1&2
Building a Virtualized Desktop Grid. Eric Sedore
Building a Virtualized Desktop Grid Eric Sedore essedore@syr.edu Why create a desktop grid? One prong of an three pronged strategy to enhance research infrastructure on campus (physical hosting, HTC grid,
An Introduction to Using HTCondor Karen Miller
An Introduction to Using HTCondor 2014 Karen Miller The Team - 2013 Established in 1985, to do research and development of distributed high-throughput computing 2 HT Stands for High Throughput Throughput:
Accelerating Financial Applications on the GPU
Accelerating Financial Applications on the GPU Scott Grauer-Gray Robert Searles William Killian John Cavazos Department of Computer and Information Science University of Delaware Sixth Workshop on General
Cycle Sharing Systems
Cycle Sharing Systems Jagadeesh Dyaberi Dependable Computing Systems Lab Purdue University 10/31/2005 1 Introduction Design of Program Security Communication Architecture Implementation Conclusion Outline
Lecture 11 Hadoop & Spark
Lecture 11 Hadoop & Spark Dr. Wilson Rivera ICOM 6025: High Performance Computing Electrical and Computer Engineering Department University of Puerto Rico Outline Distributed File Systems Hadoop Ecosystem
A Compact Computing Environment For A Windows PC Cluster Towards Seamless Molecular Dynamics Simulations
A Compact Computing Environment For A Windows PC Cluster Towards Seamless Molecular Dynamics Simulations Yuichi Tsujita Abstract A Windows PC cluster is focused for its high availabilities and fruitful
Venafi Server Agent Agent Overview
Venafi Server Agent Agent Overview Venafi Server Agent Agent Intro Agent Architecture Agent Grouping Agent Prerequisites Agent Registration Process What is Venafi Agent? The Venafi Agent is a client/server
Bright Cluster Manager
Bright Cluster Manager Using Slurm for Data Aware Scheduling in the Cloud Martijn de Vries CTO About Bright Computing Bright Computing 1. Develops and supports Bright Cluster Manager for HPC systems, server
HTCondor Essentials. Index
HTCondor Essentials 31.10.2017 Index Login How to submit a job in the HTCondor pool Why the -name option? Submitting a job Checking status of submitted jobs Getting id and other info about a job
NVIDIA Tesla C Installation Guide of C-2075 Driver on Linux
Installation Guide NVIDIA Tesla C-2075 1.1 Installation Guide of C-2075 Driver on Linux English 1 Version Date Description 0.1 29 February 2012 First release 0.2 1 March 2012 - Add the chapter 2.2. Software
AixViPMaP towards an open simulation platform for microstructure modelling
AixViPMaP towards an open simulation platform for microstructure modelling Aachen, den 7. September 2018 Lukas Koschmieder, ICME Group, Steel Institute IEHK, RWTH Aachen University ICME Cloud Service Web-based
Cloud Computing. Up until now
Cloud Computing Lectures 3 and 4 Grid Schedulers: Condor, Sun Grid Engine 2012-2013 Introduction. Up until now Definition of Cloud Computing. Grid Computing: Schedulers: Condor architecture. 1 Summary
bootmachine Documentation
bootmachine Documentation Release 0.6.0 Thomas Schreiber April 20, 2015 Contents 1 bootmachine 3 1.1 Configuration Management Tools.................................... 3 1.2 Providers.................................................
HTCondor on Titan. Wisconsin IceCube Particle Astrophysics Center. Vladimir Brik. HTCondor Week May 2018
HTCondor on Titan Wisconsin IceCube Particle Astrophysics Center Vladimir Brik HTCondor Week May 2018 Overview of Titan Cray XK7 Supercomputer at Oak Ridge Leadership Computing Facility Ranked #5 by TOP500
goals monitoring, fault tolerance, auto-recovery (thousands of low-cost machines) handle appends efficiently (no random writes & sequential reads)
Google File System goals monitoring, fault tolerance, auto-recovery (thousands of low-cost machines) focus on multi-gb files handle appends efficiently (no random writes & sequential reads) co-design GFS
Resource Management Systems
Resource Management Systems RMS DCC/FCUP Grid Computing 1 NQE (Network Queue Environment) DCC/FCUP Grid Computing 2 NQE #QSUB eo #QSUB J m #QSUB o %fred@gale/nppa_latte:/home/gale/fred/mary.jjob.output
Improve Web Application Performance with Zend Platform
Improve Web Application Performance with Zend Platform Shahar Evron Zend Sr. PHP Specialist Copyright 2007, Zend Technologies Inc. Agenda Benchmark Setup Comprehensive Performance Multilayered Caching
Distributed File Systems II
Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation
Part 1 : Getting Familiar with Linux. Hours. Part II : Administering Red Hat Enterprise Linux
Part 1 : Getting Familiar with Linux Chapter 1 : Getting started with Red Hat Enterprise Linux Chapter 2 Finding Your Way on the Command Line Hours Part II : Administering Red Hat Enterprise Linux Linux,
Linux Virtual Machine (VM) Provisioning on the Hyper-V Platform
I T S E R V I C E S Son Truong Systems & Operations Unix Technical Lead November 2016 SERVER PROVISIONING: Linux Virtual Machine (VM) Provisioning on the Hyper-V Platform Standard Linux VM Introduction
MERCED CLUSTER BASICS Multi-Environment Research Computer for Exploration and Discovery A Centerpiece for Computational Science at UC Merced
MERCED CLUSTER BASICS Multi-Environment Research Computer for Exploration and Discovery A Centerpiece for Computational Science at UC Merced Sarvani Chadalapaka HPC Administrator University of California
g-eclipse A Framework for Accessing Grid Infrastructures Nicholas Loulloudes Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.
g-eclipse A Framework for Accessing Grid Infrastructures Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.cy) EGEE Training the Trainers May 6 th, 2009 Outline Grid Reality The Problem g-eclipse
Note: Who is Dr. Who? You may notice that YARN says you are logged in as dr.who. This is what is displayed when user
Run a YARN Job Exercise Dir: ~/labs/exercises/yarn Data Files: /smartbuy/kb In this exercise you will submit an application to the YARN cluster, and monitor the application using both the Hue Job Browser
Michigan Grid Research and Infrastructure Development (MGRID)
Michigan Grid Research and Infrastructure Development (MGRID) Abhijit Bose MGRID and Dept. of Electrical Engineering and Computer Science The University of Michigan Ann Arbor, MI 48109 abose@umich.edu
Data Protection Guide
SnapCenter Software 4.0 Data Protection Guide For Oracle Databases May 2018 215-12930_D0 doccomments@netapp.com Table of Contents 3 Contents Deciding whether to read the SnapCenter Data Protection Guide
Overview Job Management OpenVZ Conclusions. XtreemOS. Surbhi Chitre. IRISA, Rennes, France. July 7, Surbhi Chitre XtreemOS 1 / 55
XtreemOS Surbhi Chitre IRISA, Rennes, France July 7, 2009 Surbhi Chitre XtreemOS 1 / 55 Surbhi Chitre XtreemOS 2 / 55 Outline What is XtreemOS What features does it provide in XtreemOS How is it new and
Introduction to Discovery.
Introduction to Discovery http://discovery.dartmouth.edu The Discovery Cluster 2 Agenda What is a cluster and why use it Overview of computer hardware in cluster Help Available to Discovery Users Logging
An Empirical Study of High Availability in Stream Processing Systems
An Empirical Study of High Availability in Stream Processing Systems Yu Gu, Zhe Zhang, Fan Ye, Hao Yang, Minkyong Kim, Hui Lei, Zhen Liu Stream Processing Model software operators (PEs) Ω Unexpected machine
NFSv4 as the Building Block for Fault Tolerant Applications
NFSv4 as the Building Block for Fault Tolerant Applications Alexandros Batsakis Overview Goal: To provide support for recoverability and application fault tolerance through the NFSv4 file system Motivation:
Red Hat Enterprise Linux MRG Red Hat Network Satellite Red Hat Enterprise Virtualization JBoss Cloud
1 Red Hat Enterprise Linux MRG Red Hat Satellite Red Hat Enterprise Virtualization JBoss Cloud 2 Red Hat Enterprise Linux 3 Proven development model Red Hat collaborates with the open source community
Quick Start Guide. Table of Contents
Quick Start Guide Table of Contents Account Registration... 2 Signup Request... 2 Account Activation... 4 Running FLOW-3D on POD... 9 Launching the GUI... 9 Running Simulations... 11 Collaborating with