Logic Networks on the Grid: Handling 15 Million Jobs
|
|
- Simon McCarthy
- 5 years ago
- Views:
Transcription
1 Logic Networks on the Grid: Handling 15 Million Jobs Jan Bot, Delft Bioinformatics Lab Delft University of Technology Challenge the future
2 Overview Explanation of the application Challenges for the grid Custom grid solution design & implementation More challenges (aka problems) Adding a desktop cluster Errors and statistics Discussion 2
3 But first... Does anybody not know what these are: Life Science Grid Grid middleware ToPoS ORM (Object Relational Mapper) 3
4 The application: overview Input data: ~100 mouse tumors 4
5 Grid pipeline Prepare inputs: prepare data for future grid runs Multiple parameter settings are tested, output of these tests contains the 'real' data Choose best parameter settings, it should still be feasible to do at least 100 permutations Do permutations, 10 permutations per run 5
6 Run properties Number of jobs per run is fairly large: 24 * 6228 = Run time is, due to the optimization algorithm, unpredictable: jobs can take anywhere between 2 seconds and 14 hours Outputs are small, both for the real runs and for the permutations 6
7 Middleware problems Scheduling: This amount of jobs cannot be scheduled using the normal (glite) middleware Overhead of scheduling could out-weight the run time Bookkeeping No method of tracking this amount of jobs Output handling: No grid resource can store large amounts of small files (dcache is not an option) Other solutions (such as ToPoS) are slow when retrieving output 7
8 Scheduling jobs with ToPoS ToPoS takes care of the first two categories of problems but presents some new challenges: ToPoS does not scale beyond jobs per pool No client software which facilitates spreading the tokens over multiple pools 8
9 Python ToPoS clients To deal with the limitations of ToPoS two clients were implemented: Grid client: uses the most basic Python httplib module can fetch, lock and delete tokens has a generator to transparently handle tokens in multiple pools Local client: uses the more advanced UrlLib2 module can create and delete pools, spread tokens over multiple pools, delete all locks in a pool, gather ToPoS statistics, etc. 9
10 Dealing with the outputs Outputs: are small and well defined why not just flush them to a database? Proposed solution: Python as language SQLAlchemy as ORM XML-RPC as communication channel MySQL (for now) as database 10
11 Client design send results Bash script start python Python script Matlab (MCR) call Matlab Set environment variables Fetch input data Make binaries executable Load modules Start python script Loop: Fetch token from ToPoS Call Matlab (MCR) Parse output & send to result server Perform algorithm 11
12 Application design ORM DB App XML-RPC Clients Design overview Listen loop: listen for incoming calls XML-RPC server Client Fetch token Do work Upload output main DB DB code Flush loop: flush once every minute Thread model 12
13 Implementation & the weakest link Implemented in Python Hosted on a P4 in a broom closet in our department On power failure: everything collapses (but that's not very likely, right?) 13
14 Getting ready to run: data replication Getting the data from one (remote) site is expensive Use data replication across all sites to minimize external traffic and divide the load over multiple SRMs Data replication can be done easily with the V-Browser Manual approach: Register file: lcg-cr -l lfn:///grid/lsgrid/jridder/mgtest/mg_perm2_5_datapack.zip MG_Perm2_5_Datapack.zip Replicate file (in this example to nikhef): lcg-rep --vo lsgrid -d tbn18.nikhef.nl srm://gb-setud.ewi.tudelft.nl/dpm/ewi.tudelft.nl/home/lsgrid/generated/ /file006bff9b-49ef-46bd80cd-5b On a WN, retrieve a local copy: DATAPACK=lfn:/grid/lsgrid/jridder/MGtest/MG_Perm2_5_Datapack.zip echo $VO_LSGRID_DEFAULT_SE TDATA=`lcg-lr --vo lsgrid $DATAPACK grep $VO_LSGRID_DEFAULT_SE` lcg-cp --verbose $TDATA $DATAPACK 14
15 Adding a desktop cluster Practical (student) pcs are not doing anything at night Use these computers to increase computation power Compute at night & in weekends Our scenario (using ToPoS and an external output server) is ideal for testing such a cluster Use condor to manage the work 15
16 Desktop cluster locations Two locations Drebbelweg: 250 practical pcs Mekelweg: pcs distributed throughout the building Different locations means different vlans: use two condor queues 16
17 Problems during run Many jobs seemed to quit prematurely while most of them ran fine Errors could be traced back to Deimos and Nikhef The middleware doesn't really provide statistics to the enduser Output files cannot always be retrieved 17
18 Gathering statistics Add run information (e.g. start & end times) to the job-output Add an additional XML-RPC method to capture error information Uploading error info is easy: Use return status of external program Use Pythons internal error handling capabilities All error messages (of the entire job) are located in one text file 18
19 Job Running times (1) 19
20 Job running times (2) One permutation run (10 permutations) takes: seconds hours days years Now, repeat 9 times (yes, that's a century) 20
21 Work done per site 21
22 Nikhef and Deimos mortality 22
23 Gathering error info Gathering error information on the grid is prone to error Again, work around the middleware: Implement additional XML-RPC call to gather error information 23
24 Error & Fix Jobs failed due to one error: Could not access the MCR component cache" Fix: export MCR_CACHE_ROOT=$( mktemp -d ) basically tells the MCR to store all temporary information in a new tmpdir Will be included in the next POC environment 24
25 Mortality after fix 25
26 Discussion We can schedule millions of jobs and capture their outputs on the grid, it just takes a custom solution Other fields (such as pattern recognition) can benefit from this solution Is their similar work being done? If not, can we design and implement a generic solution which does the same? 26
27 Thanks Jeroen de Ridder Roeland van Ochten Marcel Reinders Jeroen Engelberts Pieter van Beek Evert Lammerts Jan Just Keijzer 27
28 Life Science Grid Site CPUs SARA 2000 NIKHEF 5000 Philips 1500 RUG 160 Erasmus 32 Keygene 32 TU Delft 32 RUG 32 AMS 32 NKI 16 AMC 16 LUMC 16 WUR 16 UU 16 kun 16 Total
29 Grid Middleware The glue (or spaghetti) that unifies job management across clusters Middleware gina: condor keygene: pbs TU Delft: lsf... RUG: SGE Different sites with different job scheduling applications Heterogeneous compute resources 29
30 ToPoS: Token Pool Server Add work Submit jobs 1. Get token 2. Do work: Translate token Call function Upload output 3. Delete token Fetch work ToPoS: a pilot job framework. Pilot job: one job / thread which keeps running until all the work has been done. A 'token' represents one unit of work. Tokens can be locked to prevent other jobs from doing the same work twice. Why is ToPoS needed: Problems with grid middleware Inability to deal with large amounts of jobs Failing jobs Job accounting Etc. 30
31 ORM: Object-Relational Mapper Mapper for persistent storage of objects into a database Saves you from having to write any DB code yourself Examples: Python: SQL Alchemy, Storm Java: Hybernate, Cayenne Ruby: ActiveRecord 31
32 Why not Molgenis? Familiar with Python, which already has all the tools to make this Design XML-ify generate rol-out to cumbersome 32
glite Grid Services Overview
The EPIKH Project (Exchange Programme to advance e-infrastructure Know-How) glite Grid Services Overview Antonio Calanducci INFN Catania Joint GISELA/EPIKH School for Grid Site Administrators Valparaiso,
More informationThe Wuppertal Tier-2 Center and recent software developments on Job Monitoring for ATLAS
The Wuppertal Tier-2 Center and recent software developments on Job Monitoring for ATLAS DESY Computing Seminar Frank Volkmer, M. Sc. Bergische Universität Wuppertal Introduction Hardware Pleiades Cluster
More informationBOSCO Architecture. Derek Weitzel University of Nebraska Lincoln
BOSCO Architecture Derek Weitzel University of Nebraska Lincoln Goals We want an easy to use method for users to do computational research It should be easy to install, use, and maintain It should be simple
More informationThe glite middleware. Ariel Garcia KIT
The glite middleware Ariel Garcia KIT Overview Background The glite subsystems overview Security Information system Job management Data management Some (my) answers to your questions and random rumblings
More informationCloud Computing. Up until now
Cloud Computing Lecture 4 and 5 Grid: 2012-2013 Introduction. Up until now Definition of Cloud Computing. Grid Computing: Schedulers: Condor SGE 1 Summary Core Grid: Toolkit Condor-G Grid: Conceptual Architecture
More informationWMS overview and Proposal for Job Status
WMS overview and Proposal for Job Status Author: V.Garonne, I.Stokes-Rees, A. Tsaregorodtsev. Centre de physiques des Particules de Marseille Date: 15/12/2003 Abstract In this paper, we describe briefly
More informationOutline. Definition of a Distributed System Goals of a Distributed System Types of Distributed Systems
Distributed Systems Outline Definition of a Distributed System Goals of a Distributed System Types of Distributed Systems What Is A Distributed System? A collection of independent computers that appears
More informationDatabase Server. 2. Allow client request to the database server (using SQL requests) over the network.
Database Server Introduction: Client/Server Systems is networked computing model Processes distributed between clients and servers. Client Workstation (usually a PC) that requests and uses a service Server
More informationI Tier-3 di CMS-Italia: stato e prospettive. Hassen Riahi Claudio Grandi Workshop CCR GRID 2011
I Tier-3 di CMS-Italia: stato e prospettive Claudio Grandi Workshop CCR GRID 2011 Outline INFN Perugia Tier-3 R&D Computing centre: activities, storage and batch system CMS services: bottlenecks and workarounds
More informationReal World Web Scalability. Ask Bjørn Hansen Develooper LLC
Real World Web Scalability Ask Bjørn Hansen Develooper LLC Hello. 28 brilliant methods to make your website keep working past $goal requests/transactions/sales per second/hour/day Requiring minimal extra
More informationBOSCO Architecture. Derek Weitzel University of Nebraska Lincoln
BOSCO Architecture Derek Weitzel University of Nebraska Lincoln Goals We want an easy to use method for users to do computational research It should be easy to install, use, and maintain It should be simple
More information30 Nov Dec Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy
Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy Why the Grid? Science is becoming increasingly digital and needs to deal with increasing amounts of
More informationEdinburgh (ECDF) Update
Edinburgh (ECDF) Update Wahid Bhimji On behalf of the ECDF Team HepSysMan,10 th June 2010 Edinburgh Setup Hardware upgrades Progress in last year Current Issues June-10 Hepsysman Wahid Bhimji - ECDF 1
More informationBeob Kyun KIM, Christophe BONNAUD {kyun, NSDC / KISTI
2010. 6. 17 Beob Kyun KIM, Christophe BONNAUD {kyun, cbonnaud}@kisti.re.kr NSDC / KISTI 1 1 2 Belle Data Transfer Metadata Extraction Scalability Test Metadata Replication Grid-awaring of Belle data Test
More informationVMs at a Tier-1 site. EGEE 09, Sander Klous, Nikhef
VMs at a Tier-1 site EGEE 09, 21-09-2009 Sander Klous, Nikhef Contents Introduction Who are we? Motivation Why are we interested in VMs? What are we going to do with VMs? Status How do we approach this
More informationBiG Grid HPC Cloud Beta
BiG Grid HPC Cloud Beta Floris Sluiter SARA Computing and Networking services Amsterdam www.cloud.sara.nl About BiG Grid and Sara The BiG Grid project is a collaboration between NCF, Nikhef and NBIC, and
More informationArchitecture Proposal
Nordic Testbed for Wide Area Computing and Data Handling NORDUGRID-TECH-1 19/02/2002 Architecture Proposal M.Ellert, A.Konstantinov, B.Kónya, O.Smirnova, A.Wäänänen Introduction The document describes
More informationg-eclipse A Framework for Accessing Grid Infrastructures Nicholas Loulloudes Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.
g-eclipse A Framework for Accessing Grid Infrastructures Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.cy) EGEE Training the Trainers May 6 th, 2009 Outline Grid Reality The Problem g-eclipse
More informationOverview of HEP software & LCG from the openlab perspective
Overview of HEP software & LCG from the openlab perspective Andreas Unterkircher, CERN openlab February 2005 Andreas Unterkircher 1 Contents 1. Opencluster overview 2. High Energy Physics (HEP) software
More informationThe LGI Pilot job portal. EGI Technical Forum 20 September 2011 Jan Just Keijser Willem van Engen Mark Somers
The LGI Pilot job portal EGI Technical Forum 20 September 2011 Jan Just Keijser Willem van Engen Mark Somers Outline What? Why? How? Pro's and Cons What's next? Credits 2 What is LGI? LGI Project Server
More informationNUSGRID a computational grid at NUS
NUSGRID a computational grid at NUS Grace Foo (SVU/Academic Computing, Computer Centre) SVU is leading an initiative to set up a campus wide computational grid prototype at NUS. The initiative arose out
More informationDistributed Systems. 29. Distributed Caching Paul Krzyzanowski. Rutgers University. Fall 2014
Distributed Systems 29. Distributed Caching Paul Krzyzanowski Rutgers University Fall 2014 December 5, 2014 2013 Paul Krzyzanowski 1 Caching Purpose of a cache Temporary storage to increase data access
More informationReal Life Web Development. Joseph Paul Cohen
Real Life Web Development Joseph Paul Cohen joecohen@cs.umb.edu Index 201 - The code 404 - How to run it? 500 - Your code is broken? 200 - Someone broke into your server? 400 - How are people using your
More informationTextual Description of webbioc
Textual Description of webbioc Colin A. Smith October 13, 2014 Introduction webbioc is a web interface for some of the Bioconductor microarray analysis packages. It is designed to be installed at local
More informationWork Queue + Python. A Framework For Scalable Scientific Ensemble Applications
Work Queue + Python A Framework For Scalable Scientific Ensemble Applications Peter Bui, Dinesh Rajan, Badi Abdul-Wahid, Jesus Izaguirre, Douglas Thain University of Notre Dame Distributed Computing Examples
More informationMonitoring System for the GRID Monte Carlo Mass Production in the H1 Experiment at DESY
Journal of Physics: Conference Series OPEN ACCESS Monitoring System for the GRID Monte Carlo Mass Production in the H1 Experiment at DESY To cite this article: Elena Bystritskaya et al 2014 J. Phys.: Conf.
More informationJquery Ajax Json Php Mysql Data Entry Example
Jquery Ajax Json Php Mysql Data Entry Example Then add required assets in head which are jquery library, datatable js library and css By ajax api we can fetch json the data from employee-grid-data.php.
More information1 of 5 3/28/2010 8:01 AM Unit Testing Notes Home Class Info Links Lectures Newsgroup Assignmen [Jump to Writing Clear Tests, What about Private Functions?] Testing The typical approach to testing code
More informationParallel Job Support in the Spanish NGI! Enol Fernández del Cas/llo Ins/tuto de Física de Cantabria (IFCA) Spain
Parallel Job Support in the Spanish NGI! Enol Fernández del Cas/llo Ins/tuto de Física de Cantabria (IFCA) Spain Introduction (I)! Parallel applica/ons are common in clusters and HPC systems Grid infrastructures
More informationX Grid Engine. Where X stands for Oracle Univa Open Son of more to come...?!?
X Grid Engine Where X stands for Oracle Univa Open Son of more to come...?!? Carsten Preuss on behalf of Scientific Computing High Performance Computing Scheduler candidates LSF too expensive PBS / Torque
More informationTuning Intelligent Data Lake Performance
Tuning Intelligent Data Lake Performance 2016 Informatica LLC. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording or otherwise) without
More informationMonitoring the Usage of the ZEUS Analysis Grid
Monitoring the Usage of the ZEUS Analysis Grid Stefanos Leontsinis September 9, 2006 Summer Student Programme 2006 DESY Hamburg Supervisor Dr. Hartmut Stadie National Technical
More informationglideinwms Training Glidein Internals How they work and why by Igor Sfiligoi, Jeff Dost (UCSD) glideinwms Training Glidein internals 1
Glidein Internals How they work and why by Igor Sfiligoi, Jeff Dost (UCSD) Glidein internals 1 Refresher glidein_startup the glidein_startup script configures and starts Condor on the worker node Glidein
More informationMultiprocessor Scheduling. Multiprocessor Scheduling
Multiprocessor Scheduling Will consider only shared memory multiprocessor or multi-core CPU Salient features: One or more caches: cache affinity is important Semaphores/locks typically implemented as spin-locks:
More informationMultiprocessor Scheduling
Multiprocessor Scheduling Will consider only shared memory multiprocessor or multi-core CPU Salient features: One or more caches: cache affinity is important Semaphores/locks typically implemented as spin-locks:
More informationScaling Slack. Bing Wei
Scaling Slack Bing Wei Infrastructure@Slack 2 3 Our Mission: To make people s working lives simpler, more pleasant, and more productive. 4 From supporting small teams To serving gigantic organizations
More informationMonitoring the ALICE Grid with MonALISA
Monitoring the ALICE Grid with MonALISA 2008-08-20 Costin Grigoras ALICE Workshop @ Sibiu Monitoring the ALICE Grid with MonALISA MonALISA Framework library Data collection and storage in ALICE Visualization
More informationWLCG Lightweight Sites
WLCG Lightweight Sites Mayank Sharma (IT-DI-LCG) 3/7/18 Document reference 2 WLCG Sites Grid is a diverse environment (Various flavors of CE/Batch/WN/ +various preferred tools by admins for configuration/maintenance)
More informationChapter 18: Database System Architectures.! Centralized Systems! Client--Server Systems! Parallel Systems! Distributed Systems!
Chapter 18: Database System Architectures! Centralized Systems! Client--Server Systems! Parallel Systems! Distributed Systems! Network Types 18.1 Centralized Systems! Run on a single computer system and
More informationChapter 1: Distributed Information Systems
Chapter 1: Distributed Information Systems Contents - Chapter 1 Design of an information system Layers and tiers Bottom up design Top down design Architecture of an information system One tier Two tier
More informationLessons Learned in the NorduGrid Federation
Lessons Learned in the NorduGrid Federation David Cameron University of Oslo With input from Gerd Behrmann, Oxana Smirnova and Mattias Wadenstein Creating Federated Data Stores For The LHC 14.9.12, Lyon,
More informationHow to pimp high volume PHP websites. 27. September 2008, PHP conference Barcelona. By Jens Bierkandt
How to pimp high volume PHP websites 27. September 2008, PHP conference Barcelona By Jens Bierkandt 1 About me Jens Bierkandt Working with PHP since 2000 From Germany, living in Spain, speaking English
More informationand the GridKa mass storage system Jos van Wezel / GridKa
and the GridKa mass storage system / GridKa [Tape TSM] staging server 2 Introduction Grid storage and storage middleware dcache h and TSS TSS internals Conclusion and further work 3 FZK/GridKa The GridKa
More informationGrid Scheduling Architectures with Globus
Grid Scheduling Architectures with Workshop on Scheduling WS 07 Cetraro, Italy July 28, 2007 Ignacio Martin Llorente Distributed Systems Architecture Group Universidad Complutense de Madrid 1/38 Contents
More informationWeb Applications. Software Engineering 2017 Alessio Gambi - Saarland University
Web Applications Software Engineering 2017 Alessio Gambi - Saarland University Based on the work of Cesare Pautasso, Christoph Dorn, Andrea Arcuri, and others ReCap Software Architecture A software system
More informationEasy Access to Grid Infrastructures
Easy Access to Grid Infrastructures Dr. Harald Kornmayer (NEC Laboratories Europe) On behalf of the g-eclipse consortium WP11 Grid Workshop Grenoble, France 09 th of December 2008 Background in astro particle
More informationBookkeeping and submission tools prototype. L. Tomassetti on behalf of distributed computing group
Bookkeeping and submission tools prototype L. Tomassetti on behalf of distributed computing group Outline General Overview Bookkeeping database Submission tools (for simulation productions) Framework Design
More informationVulkan: Scaling to Multiple Threads. Kevin sun Lead Developer Support Engineer, APAC PowerVR Graphics
Vulkan: Scaling to Multiple Threads Kevin sun Lead Developer Support Engineer, APAC PowerVR Graphics www.imgtec.com Introduction Who am I? Kevin Sun Working at Imagination Technologies Take responsibility
More informationHigh Throughput WAN Data Transfer with Hadoop-based Storage
High Throughput WAN Data Transfer with Hadoop-based Storage A Amin 2, B Bockelman 4, J Letts 1, T Levshina 3, T Martin 1, H Pi 1, I Sfiligoi 1, M Thomas 2, F Wuerthwein 1 1 University of California, San
More informationGeant4 on Azure using Docker containers
http://www.geant4.org Geant4 on Azure using Docker containers Andrea Dotti (adotti@slac.stanford.edu) ; SD/EPP/Computing 1 Outlook Motivation/overview Docker + G4 Azure + G4 Conclusions 2 Motivation/overview
More informationCERN: LSF and HTCondor Batch Services
Batch @ CERN: LSF and HTCondor Batch Services Iain Steers, Jérôme Belleman, Ulrich Schwickerath IT-PES-PS INFN Visit: Batch Batch @ CERN 2 Outline The Move Environment Grid Pilot Local Jobs Conclusion
More informationSetup Desktop Grids and Bridges. Tutorial. Robert Lovas, MTA SZTAKI
Setup Desktop Grids and Bridges Tutorial Robert Lovas, MTA SZTAKI Outline of the SZDG installation process 1. Installing the base operating system 2. Basic configuration of the operating system 3. Installing
More informationVL-e PoC: What it is and what it isn t
VL-e PoC: What it is and what it isn t Jan Just Keijser VL-e P4 Scaling and Validation Team TU Delft Grid Meeting, December 11th, 2008 The VL-e vision (aka Bob's stoomboot ) What is the VL-e PoC Environment?
More informationMicrosoft Access Vba Copy Table Structure Only
Microsoft Access Vba Copy Table Structure Only My setup is I have a design copy of the database with a backup that is only Thus, whichever copy, of whichever version of the FE, assuming table structure
More informationCreating an Online Catalogue Search for CD Collection with AJAX, XML, and PHP Using a Relational Database Server on WAMP/LAMP Server
CIS408 Project 5 SS Chung Creating an Online Catalogue Search for CD Collection with AJAX, XML, and PHP Using a Relational Database Server on WAMP/LAMP Server The catalogue of CD Collection has millions
More informationPARALLEL PROGRAM EXECUTION SUPPORT IN THE JGRID SYSTEM
PARALLEL PROGRAM EXECUTION SUPPORT IN THE JGRID SYSTEM Szabolcs Pota 1, Gergely Sipos 2, Zoltan Juhasz 1,3 and Peter Kacsuk 2 1 Department of Information Systems, University of Veszprem, Hungary 2 Laboratory
More informationLesson 4 Transcript: DB2 Architecture
Lesson 4 Transcript: DB2 Architecture Slide 1: Cover Welcome to Lesson 4 of the DB2 on campus series. Today we are going to talk about the DB2 architecture. My name is Raul Chong and I am the DB2 on Campus
More informationMonitoring Grid Virtual Machine deployments
University of Victoria Faculty of Engineering Fall 2008 Work Term Report Monitoring Grid Virtual Machine deployments Department of Physics University of Victoria Victoria, BC Michael Paterson 0031209 Work
More informationSchema Validate With Oracle 10g Database Status
Schema Validate With Oracle 10g Database Status This exam has been validated against Oracle Database 10g, Oracle Database 11g, Oracle Database 11g Release 2, and Oracle Database 12c Release 1. Copy whole
More informationName Department/Research Area Have you used the Linux command line?
Please log in with HawkID (IOWA domain) Macs are available at stations as marked To switch between the Windows and the Mac systems, press scroll lock twice 9/27/2018 1 Ben Rogers ITS-Research Services
More informationPart2: Let s pick one cloud IaaS middleware: OpenStack. Sergio Maffioletti
S3IT: Service and Support for Science IT Cloud middleware Part2: Let s pick one cloud IaaS middleware: OpenStack Sergio Maffioletti S3IT: Service and Support for Science IT, University of Zurich http://www.s3it.uzh.ch/
More informationVoldemort. Smruti R. Sarangi. Department of Computer Science Indian Institute of Technology New Delhi, India. Overview Design Evaluation
Voldemort Smruti R. Sarangi Department of Computer Science Indian Institute of Technology New Delhi, India Smruti R. Sarangi Leader Election 1/29 Outline 1 2 3 Smruti R. Sarangi Leader Election 2/29 Data
More informationNAREGI PSE with ACS. S.Kawata 1, H.Usami 2, M.Yamada 3, Y.Miyahara 3, Y.Hayase 4, S.Hwang 2, K.Miura 2. Utsunomiya University 2
NAREGI PSE with ACS S.Kawata 1, H.Usami 2, M.Yamada 3, Y.Miyahara 3, Y.Hayase 4, S.Hwang 2, K.Miura 2 1 Utsunomiya University 2 National Institute of Informatics 3 FUJITSU Limited 4 Toyama College National
More informationLab 4 : Caching Locks. Introduction. Getting Started
Lab 4 : Caching Locks Introduction In this lab you will build a lock server and client that cache locks at the client, reducing the load on the server and improving client performance. For example, suppose
More informationManual Trigger Sql Server 2008 Examples Insert Update
Manual Trigger Sql Server 2008 Examples Insert Update blog.sqlauthority.com/2011/03/31/sql-server-denali-a-simple-example-of you need to manually delete this trigger or else you can't get into master too
More informationOutline. Distributed Computing Systems. The Rise of Distributed Systems. Depiction of a Distributed System 4/15/2014
Outline Distributed Computing Systems Overview of Distributed Systems Overview Goals Software Client Server Andrew Tanenbaum and Marten van Steen, Distributed Systems Principles and Paradigms, Prentice
More informationTroubleshooting Guide and FAQs Community release
Troubleshooting Guide and FAQs Community 1.5.1 release This guide details the common problems which might occur during Jumbune execution, help you to diagnose the problem, and provide probable solutions.
More informationInformatica Developer Tips for Troubleshooting Common Issues PowerCenter 8 Standard Edition. Eugene Gonzalez Support Enablement Manager, Informatica
Informatica Developer Tips for Troubleshooting Common Issues PowerCenter 8 Standard Edition Eugene Gonzalez Support Enablement Manager, Informatica 1 Agenda Troubleshooting PowerCenter issues require a
More informationThe INFN Tier1. 1. INFN-CNAF, Italy
IV WORKSHOP ITALIANO SULLA FISICA DI ATLAS E CMS BOLOGNA, 23-25/11/2006 The INFN Tier1 L. dell Agnello 1), D. Bonacorsi 1), A. Chierici 1), M. Donatelli 1), A. Italiano 1), G. Lo Re 1), B. Martelli 1),
More informationComputer Basics 1/24/13. Computer Organization. Computer systems consist of hardware and software.
Hardware and Software Computer Basics TOPICS Computer Organization Data Representation Program Execution Computer Languages Computer systems consist of hardware and software. Hardware includes the tangible
More informationDistributed production managers meeting. Armando Fella on behalf of Italian distributed computing group
Distributed production managers meeting Armando Fella on behalf of Italian distributed computing group Distributed Computing human network CNAF Caltech SLAC McGill Queen Mary RAL LAL and Lyon Bari Legnaro
More informationDeveloping Solutions for Google Cloud Platform (CPD200) Course Agenda
Developing Solutions for Google Cloud Platform (CPD200) Course Agenda Module 1: Developing Solutions for Google Cloud Platform Identify the advantages of Google Cloud Platform for solution development
More informationMonitoring and Analytics With HTCondor Data
Monitoring and Analytics With HTCondor Data William Strecker-Kellogg RACF/SDCC @ BNL 1 RHIC/ATLAS Computing Facility (SDCC) Who are we? See our last two site reports from the HEPiX conference for a good
More informationIntegration and State Summarization. Prof. Holly Yanco Munjal Desai Dan Hestand
Integration and State Summarization Prof. Holly Yanco Munjal Desai Dan Hestand Pragmatics (Alchemy) World Database GUI State Summarization Pragmatics Integration Pragmatics (Alchemy) World Database GUI
More informationBuilding loosely coupled and scalable systems using Event-Driven Architecture. Jonas Bonér Patrik Nordwall Andreas Källberg
Building loosely coupled and scalable systems using Event-Driven Architecture Jonas Bonér Patrik Nordwall Andreas Källberg Why is EDA Important for Scalability? What building blocks does EDA consists of?
More informationInstruction Decode In Oracle Sql Loader Control File Example Csv
Instruction Decode In Oracle Sql Loader Control File Example Csv The examples in this article require the SCOTT schema. SET PAGESIZE 0 LINESIZE 100 TRIMSPOOL ON FEEDBACK OFF SPOOL such that all the default
More informationChapter 20: Database System Architectures
Chapter 20: Database System Architectures Chapter 20: Database System Architectures Centralized and Client-Server Systems Server System Architectures Parallel Systems Distributed Systems Network Types
More informationGustavo Alonso, ETH Zürich. Web services: Concepts, Architectures and Applications - Chapter 1 2
Chapter 1: Distributed Information Systems Gustavo Alonso Computer Science Department Swiss Federal Institute of Technology (ETHZ) alonso@inf.ethz.ch http://www.iks.inf.ethz.ch/ Contents - Chapter 1 Design
More informationTroubleshooting Grid authentication from the client side
Troubleshooting Grid authentication from the client side By Adriaan van der Zee RP1 presentation 2009-02-04 Contents The Grid @NIKHEF The project Grid components and interactions X.509 certificates, proxies
More informationMotivation. Threads. Multithreaded Server Architecture. Thread of execution. Chapter 4
Motivation Threads Chapter 4 Most modern applications are multithreaded Threads run within application Multiple tasks with the application can be implemented by separate Update display Fetch data Spell
More informationProgramming Assignment Multi-Threading and Debugging 2
Programming Assignment Multi-Threading and Debugging 2 Due Date: Friday, June 1 @ 11:59 pm PAMT2 Assignment Overview The purpose of this mini-assignment is to continue your introduction to parallel programming
More informationELEC 377 Operating Systems. Week 9 Class 3
ELEC 377 Operating Systems Week 9 Class 3 Last Week I/O Systems Block and Character Devices Today I/O Systems Block and Character Devices Network Devices Kernel Services Distributed Systems /dev filesystem
More informationThe EU DataGrid Testbed
The EU DataGrid Testbed The European DataGrid Project Team http://www.eudatagrid.org DataGrid is a project funded by the European Union Grid Tutorial 4/3/2004 n 1 Contents User s Perspective of the Grid
More informationMining for insight. Osma Ahvenlampi, CTO, Sulake Implementing business intelligence for Habbo
Mining for insight Osma Ahvenlampi, CTO, Sulake Implementing business intelligence for Habbo Virtual world 3 Social Play 4 Habbo Countries 5 Leading virtual world» 129 million registered Habbo-characters
More informationManual Trigger Sql Server 2008 Update Inserted Rows
Manual Trigger Sql Server 2008 Update Inserted Rows Am new to SQL scripting and SQL triggers, any help will be appreciated Does it need to have some understanding of what row(s) were affected, sql-serverperformance.com/2010/transactional-replication-2008-r2/
More informationPegasus Workflow Management System. Gideon Juve. USC Informa3on Sciences Ins3tute
Pegasus Workflow Management System Gideon Juve USC Informa3on Sciences Ins3tute Scientific Workflows Orchestrate complex, multi-stage scientific computations Often expressed as directed acyclic graphs
More informationArchitectural Design. Architectural Design. Software Architecture. Architectural Models
Architectural Design Architectural Design Chapter 6 Architectural Design: -the design the desig process for identifying: - the subsystems making up a system and - the relationships between the subsystems
More informationCloud Computing. Summary
Cloud Computing Lectures 2 and 3 Definition of Cloud Computing, Grid Architectures 2012-2013 Summary Definition of Cloud Computing (more complete). Grid Computing: Conceptual Architecture. Condor. 1 Cloud
More informationGrid Experiment and Job Management
Grid Experiment and Job Management Week #6 Basics of Grid and Cloud computing University of Tartu March 20th 2013 Hardi Teder hardi@eenet.ee Overview Grid Jobs Simple Jobs Pilot Jobs Workflows Job management
More informationParallel Computing Ideas
Parallel Computing Ideas K. 1 1 Department of Mathematics 2018 Why When to go for speed Historically: Production code Code takes a long time to run Code runs many times Code is not end in itself 2010:
More informationUtilizing Databases in Grid Engine 6.0
Utilizing Databases in Grid Engine 6.0 Joachim Gabler Software Engineer Sun Microsystems http://sun.com/grid Current status flat file spooling binary format for jobs ASCII format for other objects accounting
More informationLecture 11 Hadoop & Spark
Lecture 11 Hadoop & Spark Dr. Wilson Rivera ICOM 6025: High Performance Computing Electrical and Computer Engineering Department University of Puerto Rico Outline Distributed File Systems Hadoop Ecosystem
More informationBy Ludovic Duvaux (27 November 2013)
Array of jobs using SGE - an example using stampy, a mapping software. Running java applications on the cluster - merge sam files using the Picard tools By Ludovic Duvaux (27 November 2013) The idea ==========
More information3C05 - Advanced Software Engineering Thursday, April 29, 2004
Distributed Software Architecture Using Middleware Avtar Raikmo Overview Middleware What is middleware? Why do we need middleware? Types of middleware Distributed Software Architecture Business Object
More informationThe University of Oxford campus grid, expansion and integrating new partners. Dr. David Wallom Technical Manager
The University of Oxford campus grid, expansion and integrating new partners Dr. David Wallom Technical Manager Outline Overview of OxGrid Self designed components Users Resources, adding new local or
More informationDigital Humanities. Tutorial Regular Expressions. March 10, 2014
Digital Humanities Tutorial Regular Expressions March 10, 2014 1 Introduction In this tutorial we will look at a powerful technique, called regular expressions, to search for specific patterns in corpora.
More informationLCG-2 and glite Architecture and components
LCG-2 and glite Architecture and components Author E.Slabospitskaya www.eu-egee.org Outline Enabling Grids for E-sciencE What are LCG-2 and glite? glite Architecture Release 1.0 review What is glite?.
More informationTHE CPU SPENDS ALMOST ALL of its time fetching instructions from memory
THE CPU SPENDS ALMOST ALL of its time fetching instructions from memory and executing them. However, the CPU and main memory are only two out of many components in a real computer system. A complete system
More informationGrid Programming: Concepts and Challenges. Michael Rokitka CSE510B 10/2007
Grid Programming: Concepts and Challenges Michael Rokitka SUNY@Buffalo CSE510B 10/2007 Issues Due to Heterogeneous Hardware level Environment Different architectures, chipsets, execution speeds Software
More informationThe Evolution of a Data Project
The Evolution of a Data Project The Evolution of a Data Project Python script The Evolution of a Data Project Python script SQL on live DB The Evolution of a Data Project Python script SQL on live DB SQL
More information