Grid Mashups. Gluing grids together with Condor and BOINC
|
|
|
- Amos Parsons
- 5 years ago
- Views:
Transcription
1 Grid Mashups Gluing grids together with Condor and BOINC, Artyom Sharov, Assaf Schuster, Dan Geiger Technion Israel Institute of Technology 1
2 Problem... 2
3 Problem... 3
4 Problem... 4
5 Parallelization From tens to millions of subtasks 5
6 Parallelization From tens to millions of subtasks Where to find the computers????? 6
7 Let's build a cluster!!! 7
8 Let's build a cluster!!! 8
9 Let's use several clusters 9
10 Let's use several clusters 10
11 Let's also ask for help from... 11
12 Let's also ask for help 12
13 Let's also ask for help community! 13
14 Can we glue these grids together?? 14
15 Can we glue these grids together? Yes? 15
16 We also want to hide the complexity.. 16
17 By building virtual supercomputer for domain researchers 17
18 Virtual supercomputer for geneticists Superlink-online: 18
19 Main components BOINC Server HTTP frontend Scheduler Database jobs, monitoring, system statistics 19
20 Main components BOINC Server HTTP frontend Scheduler Database jobs, monitoring, system statistics 20
21 Main components BOINC Server HTTP frontend Scheduler Database BOINC clients submittor for EGEE Submittor to Technion Condor jobs, monitoring, system statistics BOINC clients submittor for Condor in Madison 21
22 Main components BOINC Server HTTP frontend Scheduler Database BOINC clients submittor for Condor in Madison BOINC clients submittor for EGEE Submittor to Technion Condor jobs, monitoring, system statistics Web Portal Task state 22
23 Main components BOINC Server HTTP frontend Scheduler Database BOINC clients submittor for Condor in Madison BOINC clients submittor for EGEE Submittor to Technion Condor jobs, monitoring, system statistics Web Portal Task state Task execution and monitoring workflow 23
24 Main components BOINC Server HTTP frontend Scheduler Database BOINC clients submittor for Condor in Madison Virtual cluster maintainer jobs, monitoring, system statistics BOINC clients submittor for EGEE Submittor to Technion Condor Web Portal Task state Task execution and monitoring workflow 24
25 Main components BOINC Server HTTP frontend Scheduler Database Web Portal BOINC clients submittor for EGEE Virtual cluster maintainer jobs, monitoring, system statistics Task state BOINC clients submittor for Condor in Madison Task execution and monitoring workflow Submittor to Technion Condor Dedicated cluster fallback 25
26 BOINC Server HTTP frontend Scheduler Database BOINC clients submittor for EGEE Virtual cluster maintainer jobs, monitoring, system statistics Web Portal Task state BOINC clients submittor for Condor in Madison Task execution and monitoring workflow Submittor to Technion Condor Dedicated cluster fallback Condor INSIDE Condor DAGman reliability 26
27 BOINC Server HTTP frontend Scheduler Database Web Portal Task execution and monitoring workflow Condor INSIDE Condor DAGman reliability BOINC clients submittor for EGEE Virtual cluster maintainer jobs, monitoring, system statistics Task state BOINC clients submittor for Condor in Madison Submittor to Technion Condor Dedicated cluster fallback Condor INSIDE Condor priority management 27
28 Condor STARTD as a stand-alone resource manager (Condor over BOINC) Condor INSIDE BOINC Server HTTP frontend Scheduler Database Web Portal Task execution and monitoring workflow Condor INSIDE Condor DAGman reliability BOINC clients submittor for EGEE Virtual cluster maintainer jobs, monitoring, system statistics Task state BOINC clients submittor for Condor in Madison Submittor to Technion Condor Dedicated cluster fallback Condor INSIDE Condor priority management 28
29 29
30 Preliminary results 110 CPU Years consumed and a few millions jobs completed in 4 months ~ 350 CPUs working for us around the clock 49 (clusters: EGEE, UW Madison, Technion CS > 20,000 CPUs ) 61 (Contributors of Superlink@Technion > 8000 CPUs ) 30
31 31
32 ~CPU time ~CPU time 32
33 Single framework for grid performance analysis environment is more predictable! Jobs left in queue Jobs left in queue Since clusters are much more volatile because of policy Eviction rate Eviction rate 33
34 Single framework for grid performance analysis EGEE vs. Madison pool EGEE behaves more than Madison pool. Why? Jobs left High load, complex policies and centralized control result in rapid (statistically unpredictable) changes in resource allocation MADISON Jobs left EGEE is much less centralized evictions are less correlated, hence have steady state EGEE 34
35 Lesson 1: Track the system performance from the application perspective Simple criterion - compare with the time it would take on a dedicated cluster Active hosts for 24 hours Actual throughput Error rate Eviction rate Using BOINC (glide-ins) helps separate application performance from batch system performance 35
36 Lesson 2: Learn the system parameters Example: Average running time of a BOINC client until preempted Wrong job granularity The same run with proper job granularity Using BOINC allows to tailor job size with no additional overhead : few minutes long jobs run as fast as few hours long 36
37 Lesson 3: Distinguish between high throughput and high performance runs Many-jobs: high throughput runs number of jobs much more than number of running machines Less sensitive to failures - overlapped with the execution of other jobs Few-jobs: high performance runs number of jobs is about the number of running machines Performance very sensitive to failures Dedicated cluster comes handy!!! High throughput High performance 37
38 Thinking out loud Will grids become less useful to opportunistic users when they become easy to use (higher utilized)? Should Condor policy be hostile to opportunistic users, or some guarantees are better to be provided? And if above is true, and you are going to have a lot of jobs maybe it's better to buy your own cluster... 38
39 Future work... Venus.. Moon But before that... If your grid can handle backfill let us in! If you want to contribute your PC(s) to the search for disease-provoking genetic mutations Join 39
Day 9: Introduction to CHTC
Day 9: Introduction to CHTC Suggested reading: Condor 7.7 Manual: http://www.cs.wisc.edu/condor/manual/v7.7/ Chapter 1: Overview Chapter 2: Users Manual (at most, 2.1 2.7) 1 Turn In Homework 2 Homework
A Distributed System for Genetic Linkage Analysis. Mark Silberstein
A Distributed System for Genetic Linkage Analysis Mark Silberstein A Distributed System for Genetic Linkage Analysis Research thesis Submitted in Partial Fulfillment of the Requirement for the degree
What s new in HTCondor? What s coming? HTCondor Week 2018 Madison, WI -- May 22, 2018
What s new in HTCondor? What s coming? HTCondor Week 2018 Madison, WI -- May 22, 2018 Todd Tannenbaum Center for High Throughput Computing Department of Computer Sciences University of Wisconsin-Madison
XSEDE High Throughput Computing Use Cases
XSEDE High Throughput Computing Use Cases 31 May 2013 Version 0.3 XSEDE HTC Use Cases Page 1 XSEDE HTC Use Cases Page 2 Table of Contents A. Document History B. Document Scope C. High Throughput Computing
The Lattice BOINC Project Public Computing for the Tree of Life
The Lattice BOINC Project Public Computing for the Tree of Life Presented by Adam Bazinet Center for Bioinformatics and Computational Biology Institute for Advanced Computer Studies University of Maryland
Grid Compute Resources and Job Management
Grid Compute Resources and Job Management How do we access the grid? Command line with tools that you'll use Specialised applications Ex: Write a program to process images that sends data to run on the
OPERATING SYSTEMS. Systems with Multi-programming. CS 3502 Spring Chapter 4
OPERATING SYSTEMS CS 3502 Spring 2018 Systems with Multi-programming Chapter 4 Multiprogramming - Review An operating system can support several processes in memory. While one process receives service
g-eclipse A Framework for Accessing Grid Infrastructures Nicholas Loulloudes Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.
g-eclipse A Framework for Accessing Grid Infrastructures Trainer, University of Cyprus (loulloudes.n_at_cs.ucy.ac.cy) EGEE Training the Trainers May 6 th, 2009 Outline Grid Reality The Problem g-eclipse
One Pool To Rule Them All The CMS HTCondor/glideinWMS Global Pool. D. Mason for CMS Software & Computing
One Pool To Rule Them All The CMS HTCondor/glideinWMS Global Pool D. Mason for CMS Software & Computing 1 Going to try to give you a picture of the CMS HTCondor/ glideinwms global pool What s the use case
SZDG, ecom4com technology, EDGeS-EDGI in large P. Kacsuk MTA SZTAKI
SZDG, ecom4com technology, EDGeS-EDGI in large P. Kacsuk MTA SZTAKI The EDGI/EDGeS projects receive(d) Community research funding 1 Outline of the talk SZTAKI Desktop Grid (SZDG) SZDG technology: ecom4com
Grid Compute Resources and Grid Job Management
Grid Compute Resources and Job Management March 24-25, 2007 Grid Job Management 1 Job and compute resource management! This module is about running jobs on remote compute resources March 24-25, 2007 Grid
DB2 is a complex system, with a major impact upon your processing environment. There are substantial performance and instrumentation changes in
DB2 is a complex system, with a major impact upon your processing environment. There are substantial performance and instrumentation changes in versions 8 and 9. that must be used to measure, evaluate,
Improved 3G Bridge scalability to support desktop grid executions
Improved 3G Bridge scalability to support desktop grid executions Zoltán Farkas zfarkas@sztaki.hu MTA SZTAKI LPDS 09/01/2010 09/01/2010 3G Bridge Scalability 2 Outline Introduction The scalability problem
Announcements. Reading. Project #1 due in 1 week at 5:00 pm Scheduling Chapter 6 (6 th ed) or Chapter 5 (8 th ed) CMSC 412 S14 (lect 5)
Announcements Reading Project #1 due in 1 week at 5:00 pm Scheduling Chapter 6 (6 th ed) or Chapter 5 (8 th ed) 1 Relationship between Kernel mod and User Mode User Process Kernel System Calls User Process
Scheduling of processes
Scheduling of processes Processor scheduling Schedule processes on the processor to meet system objectives System objectives: Assigned processes to be executed by the processor Response time Throughput
Solving Hard Integer Programs with MW
Solving Hard Integer Programs with MW Jeff Linderoth ISE Department COR@L Lab Lehigh University jtl3@lehigh.edu 2007 Condor Jamboree Madison, WI May 2, 2007 Thanks! NSF OCI-0330607, CMMI-0522796, DOE DE-FG02-05ER25694
Condor and BOINC. Distributed and Volunteer Computing. Presented by Adam Bazinet
Condor and BOINC Distributed and Volunteer Computing Presented by Adam Bazinet Condor Developed at the University of Wisconsin-Madison Condor is aimed at High Throughput Computing (HTC) on collections
UW-ATLAS Experiences with Condor
UW-ATLAS Experiences with Condor M.Chen, A. Leung, B.Mellado Sau Lan Wu and N.Xu Paradyn / Condor Week, Madison, 05/01/08 Outline Our first success story with Condor - ATLAS production in 2004~2005. CRONUS
Corral: A Glide-in Based Service for Resource Provisioning
: A Glide-in Based Service for Resource Provisioning Gideon Juve USC Information Sciences Institute juve@usc.edu Outline Throughput Applications Grid Computing Multi-level scheduling and Glideins Example:
Lecture Topics. Announcements. Today: Uniprocessor Scheduling (Stallings, chapter ) Next: Advanced Scheduling (Stallings, chapter
Lecture Topics Today: Uniprocessor Scheduling (Stallings, chapter 9.1-9.3) Next: Advanced Scheduling (Stallings, chapter 10.1-10.4) 1 Announcements Self-Study Exercise #10 Project #8 (due 11/16) Project
A Survey on Grid Scheduling Systems
Technical Report Report #: SJTU_CS_TR_200309001 A Survey on Grid Scheduling Systems Yanmin Zhu and Lionel M Ni Cite this paper: Yanmin Zhu, Lionel M. Ni, A Survey on Grid Scheduling Systems, Technical
UK Tier-2 site evolution for ATLAS. Alastair Dewhurst
UK Tier-2 site evolution for ATLAS Alastair Dewhurst Introduction My understanding is that GridPP funding is only part of the story when it comes to paying for a Tier 2 site. Each site is unique. Aim to
Part IV. Workflow Mapping and Execution in Pegasus. (Thanks to Ewa Deelman)
AAAI-08 Tutorial on Computational Workflows for Large-Scale Artificial Intelligence Research Part IV Workflow Mapping and Execution in Pegasus (Thanks to Ewa Deelman) 1 Pegasus-Workflow Management System
Changing landscape of computing at BNL
Changing landscape of computing at BNL Shared Pool and New Users and Tools HTCondor Week May 2018 William Strecker-Kellogg Shared Pool Merging 6 HTCondor Pools into 1 2 What? Current Situation
Clouds: An Opportunity for Scientific Applications?
Clouds: An Opportunity for Scientific Applications? Ewa Deelman USC Information Sciences Institute Acknowledgements Yang-Suk Ki (former PostDoc, USC) Gurmeet Singh (former Ph.D. student, USC) Gideon Juve
CSE 120 Principles of Operating Systems
CSE 120 Principles of Operating Systems Spring 2018 Lecture 15: Multicore Geoffrey M. Voelker Multicore Operating Systems We have generally discussed operating systems concepts independent of the number
CS370 Operating Systems
CS370 Operating Systems Colorado State University Yashwant K Malaiya Fall 2017 Lecture 23 Virtual memory Slides based on Text by Silberschatz, Galvin, Gagne Various sources 1 1 FAQ Is a page replaces when
Scheduling Bits & Pieces
Scheduling Bits & Pieces 1 Windows Scheduling 2 Windows Scheduling Priority Boost when unblocking Actual boost dependent on resource Disk (1), serial (2), keyboard (6), soundcard (8).. Interactive, window
Introduction to Grid Computing
Milestone 2 Include the names of the papers You only have a page be selective about what you include Be specific; summarize the authors contributions, not just what the paper is about. You might be able
Grid Scheduling Architectures with Globus
Grid Scheduling Architectures with Workshop on Scheduling WS 07 Cetraro, Italy July 28, 2007 Ignacio Martin Llorente Distributed Systems Architecture Group Universidad Complutense de Madrid 1/38 Contents
Pegasus Workflow Management System. Gideon Juve. USC Informa3on Sciences Ins3tute
Pegasus Workflow Management System Gideon Juve USC Informa3on Sciences Ins3tute Scientific Workflows Orchestrate complex, multi-stage scientific computations Often expressed as directed acyclic graphs
Determining the Number of CPUs for Query Processing
Determining the Number of CPUs for Query Processing Fatemah Panahi Elizabeth Soechting CS747 Advanced Computer Systems Analysis Techniques The University of Wisconsin-Madison fatemeh@cs.wisc.edu, eas@cs.wisc.edu
BOSCO Architecture. Derek Weitzel University of Nebraska Lincoln
BOSCO Architecture Derek Weitzel University of Nebraska Lincoln Goals We want an easy to use method for users to do computational research It should be easy to install, use, and maintain It should be simple
Multiprocessor Scheduling. Multiprocessor Scheduling
Multiprocessor Scheduling Will consider only shared memory multiprocessor or multi-core CPU Salient features: One or more caches: cache affinity is important Semaphores/locks typically implemented as spin-locks:
Multiprocessor Scheduling
Multiprocessor Scheduling Will consider only shared memory multiprocessor or multi-core CPU Salient features: One or more caches: cache affinity is important Semaphores/locks typically implemented as spin-locks:
Subject Name:Operating system. Subject Code:10EC35. Prepared By:Remya Ramesan and Kala H.S. Department:ECE. Date:
Subject Name:Operating system Subject Code:10EC35 Prepared By:Remya Ramesan and Kala H.S. Department:ECE Date:24-02-2015 UNIT 1 INTRODUCTION AND OVERVIEW OF OPERATING SYSTEM Operating system, Goals of
Operating System Review Part
Operating System Review Part CMSC 602 Operating Systems Ju Wang, 2003 Fall Virginia Commonwealth University Review Outline Definition Memory Management Objective Paging Scheme Virtual Memory System and
Li Yu. University of Notre Dame
Li Yu University of Notre Dame 1 Distributed systems are hard to use! An abstraction is a regular structure that can be efficiently scaled up to large problem sizes. We have implemented abstractions such
Preview. Process Scheduler. Process Scheduling Algorithms for Batch System. Process Scheduling Algorithms for Interactive System
Preview Process Scheduler Short Term Scheduler Long Term Scheduler Process Scheduling Algorithms for Batch System First Come First Serve Shortest Job First Shortest Remaining Job First Process Scheduling
Low Latency Data Grids in Finance
Low Latency Data Grids in Finance Jags Ramnarayan Chief Architect GemStone Systems jags.ramnarayan@gemstone.com Copyright 2006, GemStone Systems Inc. All Rights Reserved. Background on GemStone Systems
CS370 Operating Systems
CS370 Operating Systems Colorado State University Yashwant K Malaiya Fall 2016 Lecture 33 Virtual Memory Slides based on Text by Silberschatz, Galvin, Gagne Various sources 1 1 FAQ How does the virtual
Operating Systems. Overview. Dr Alun Moon. Computing, Engineering and Information Sciences. 27th September 2011
Operating Systems Overview Dr Alun Moon Computing, Engineering and Information Sciences 27th September 2011 Dr Alun Moon (ceis:nu) Operating Systems 27th September 2011 1 / 16 Chapters 1 & 2 Galvin Silberschatz.
Special Topics: CSci 8980 Edge History
Special Topics: CSci 8980 Edge History Jon B. Weissman (jon@cs.umn.edu) Department of Computer Science University of Minnesota P2P: What is it? No always-on server Nodes are at the network edge; come and
Properties of Processes
CPU Scheduling Properties of Processes CPU I/O Burst Cycle Process execution consists of a cycle of CPU execution and I/O wait. CPU burst distribution: CPU Scheduler Selects from among the processes that
CHTC Policy and Configuration. Greg Thain HTCondor Week 2017
CHTC Policy and Configuration Greg Thain HTCondor Week 2017 Image credit: flickr user shanelin cc Image credit: wikipedia CHTC Pool Mission CHTC Pool Mission To improve computational research on campus
Announcements. Program #1. Program #0. Reading. Is due at 9:00 AM on Thursday. Re-grade requests are due by Monday at 11:59:59 PM.
Program #1 Announcements Is due at 9:00 AM on Thursday Program #0 Re-grade requests are due by Monday at 11:59:59 PM Reading Chapter 6 1 CPU Scheduling Manage CPU to achieve several objectives: maximize
Page Replacement Algorithms
Page Replacement Algorithms MIN, OPT (optimal) RANDOM evict random page FIFO (first-in, first-out) give every page equal residency LRU (least-recently used) MRU (most-recently used) 1 9.1 Silberschatz,
The LHC Computing Grid
The LHC Computing Grid Gergely Debreczeni (CERN IT/Grid Deployment Group) The data factory of LHC 40 million collisions in each second After on-line triggers and selections, only 100 3-4 MB/event requires
Pegasus WMS Automated Data Management in Shared and Nonshared Environments
Pegasus WMS Automated Data Management in Shared and Nonshared Environments Mats Rynge USC Information Sciences Institute Pegasus Workflow Management System NSF funded project and developed
Building Campus HTC Sharing Infrastructures. Derek Weitzel University of Nebraska Lincoln (Open Science Grid Hat)
Building Campus HTC Sharing Infrastructures Derek Weitzel University of Nebraska Lincoln (Open Science Grid Hat) HCC: Campus Grids Motivation We have 3 clusters in 2 cities. Our largest (4400 cores) is
Memory - Paging. Copyright : University of Illinois CS 241 Staff 1
Memory - Paging Copyright : University of Illinois CS 241 Staff 1 Physical Frame Allocation How do we allocate physical memory across multiple processes? What if Process A needs to evict a page from Process
Job Scheduler Simulator Extension for Evaluating Queue Mapping to Computing Node
Job Scheduler Simulator Extension for Evaluating Queue Mapping to Computing Node Susumu Date, Yuki Matsui, Yasuhiro Watashiba, Takashi Yoshikawa, Shinji Shimojo Cybermedia Center, Osaka University [Background]
Using GPUaaS in Cloud Foundry
Using GPUaaS in Cloud Foundry Agenda Introduction GPUaaS Cloud Foundry Integration 2 Technology Research Innovation Group Innovation Advanced Research Proof of Concept User Feedback Agile Roadmap 3 Technology
Cycle Sharing Systems
Cycle Sharing Systems Jagadeesh Dyaberi Dependable Computing Systems Lab Purdue University 10/31/2005 1 Introduction Design of Program Security Communication Architecture Implementation Conclusion Outline
Grid Computing Competence Center Large Scale Computing Infrastructures (MINF 4526 HS2011)
Grid Computing Competence Center Large Scale Computing Infrastructures (MINF 4526 HS2011) Sergio Maffioletti Grid Computing Competence Centre, University of Zurich http://www.gc3.uzh.ch/
CPU Scheduling. Rab Nawaz Jadoon. Assistant Professor DCS. Pakistan. COMSATS, Lahore. Department of Computer Science
CPU Scheduling Rab Nawaz Jadoon DCS COMSATS Institute of Information Technology Assistant Professor COMSATS, Lahore Pakistan Operating System Concepts Objectives To introduce CPU scheduling, which is the
ayaz ali Micro & Macro Scheduling Techniques Ayaz Ali Department of Computer Science University of Houston Houston, TX
ayaz ali Micro & Macro Scheduling Techniques Ayaz Ali Department of Computer Science University of Houston Houston, TX 77004 ayaz@cs.uh.edu 1. INTRODUCTION Scheduling techniques has historically been one
Interactive Scheduling
Interactive Scheduling 1 Two Level Scheduling Interactive systems commonly employ two-level scheduling CPU scheduler and Memory Scheduler Memory scheduler was covered in VM We will focus on CPU scheduling
Two Level Scheduling. Interactive Scheduling. Our Earlier Example. Round Robin Scheduling. Round Robin Schedule. Round Robin Schedule
Two Level Scheduling Interactive Scheduling Interactive systems commonly employ two-level scheduling CPU scheduler and Memory Scheduler Memory scheduler was covered in VM We will focus on CPU scheduling
Scientific data processing at global scale The LHC Computing Grid. fabio hernandez
Scientific data processing at global scale The LHC Computing Grid Chengdu (China), July 5th 2011 Who I am 2 Computing science background Working in the field of computing for high-energy physics since
Grid Computing Systems: A Survey and Taxonomy
Grid Computing Systems: A Survey and Taxonomy Material for this lecture from: A Survey and Taxonomy of Resource Management Systems for Grid Computing Systems, K. Krauter, R. Buyya, M. Maheswaran, CS Technical
Cloud Computing. Summary
Cloud Computing Lectures 2 and 3 Definition of Cloud Computing, Grid Architectures 2012-2013 Summary Definition of Cloud Computing (more complete). Grid Computing: Conceptual Architecture. Condor. 1 Cloud
Cluster Computing. Resource and Job Management for HPC 16/08/2010 SC-CAMP. ( SC-CAMP) Cluster Computing 16/08/ / 50
Cluster Computing Resource and Job Management for HPC SC-CAMP 16/08/2010 ( SC-CAMP) Cluster Computing 16/08/2010 1 / 50 Summary 1 Introduction Cluster Computing 2 About Resource and Job Management Systems
Multiprocessor scheduling
Chapter 10 Multiprocessor scheduling When a computer system contains multiple processors, a few new issues arise. Multiprocessor systems can be categorized into the following: Loosely coupled or distributed.
Scalable, Reliable Marshalling and Organization of Distributed Large Scale Data Onto Enterprise Storage Environments *
Scalable, Reliable Marshalling and Organization of Distributed Large Scale Data Onto Enterprise Storage Environments * Joesph JaJa joseph@ Mike Smorul toaster@ Fritz McCall fmccall@ Yang Wang wpwy@ Institute
High Throughput WAN Data Transfer with Hadoop-based Storage
High Throughput WAN Data Transfer with Hadoop-based Storage A Amin 2, B Bockelman 4, J Letts 1, T Levshina 3, T Martin 1, H Pi 1, I Sfiligoi 1, M Thomas 2, F Wuerthwein 1 1 University of California, San
CS418 Operating Systems
CS418 Operating Systems Lecture 9 Processor Management, part 1 Textbook: Operating Systems by William Stallings 1 1. Basic Concepts Processor is also called CPU (Central Processing Unit). Process an executable
First evaluation of the Globus GRAM Service. Massimo Sgaravatto INFN Padova
First evaluation of the Globus GRAM Service Massimo Sgaravatto INFN Padova massimo.sgaravatto@pd.infn.it Draft version release 1.0.5 20 June 2000 1 Introduction...... 3 2 Running jobs... 3 2.1 Usage examples.
Towards Ensuring Collective Availability in Volatile Resource Pools via Forecasting
Towards CloudComputing@home: Ensuring Collective Availability in Volatile Resource Pools via Forecasting Artur Andrzejak Berlin (ZIB) andrzejak[at]zib.de Zuse-Institute Derrick Kondo David P. Anderson
High Throughput Urgent Computing
Condor Week 2008 High Throughput Urgent Computing Jason Cope jason.cope@colorado.edu Project Collaborators Argonne National Laboratory / University of Chicago Pete Beckman Suman Nadella Nick Trebon University
Tutorial 4: Condor. John Watt, National e-science Centre
Tutorial 4: Condor John Watt, National e-science Centre Tutorials Timetable Week Day/Time Topic Staff 3 Fri 11am Introduction to Globus J.W. 4 Fri 11am Globus Development J.W. 5 Fri 11am Globus Development
ECE519 Advanced Operating Systems
IT 540 Operating Systems ECE519 Advanced Operating Systems Prof. Dr. Hasan Hüseyin BALIK (10 th Week) (Advanced) Operating Systems 10. Multiprocessor, Multicore and Real-Time Scheduling 10. Outline Multiprocessor
CS 578 Software Architectures Fall 2014 Homework Assignment #1 Due: Wednesday, September 24, 2014 see course website for submission details
CS 578 Software Architectures Fall 2014 Homework Assignment #1 Due: Wednesday, September 24, 2014 see course website for submission details The Berkeley Open Infrastructure for Network Computing (BOINC)
Managing large-scale workflows with Pegasus
Funded by the National Science Foundation under the OCI SDCI program, grant #0722019 Managing large-scale workflows with Pegasus Karan Vahi ( vahi@isi.edu) Collaborative Computing Group USC Information
Scheduling. Scheduling. Scheduling. Scheduling Criteria. Priorities. Scheduling
scheduling: share CPU among processes scheduling should: be fair all processes must be similarly affected no indefinite postponement aging as a possible solution adjust priorities based on waiting time
glideinwms: Quick Facts
glideinwms: Quick Facts glideinwms is an open-source Fermilab Computing Sector product driven by CMS Heavy reliance on HTCondor from UW Madison and we work closely with them http://tinyurl.com/glideinwms
Operating Systems. Memory Management. Lecture 9 Michael O Boyle
Operating Systems Memory Management Lecture 9 Michael O Boyle 1 Memory Management Background Logical/Virtual Address Space vs Physical Address Space Swapping Contiguous Memory Allocation Segmentation Goals
EGEE and Interoperation
EGEE and Interoperation Laurence Field CERN-IT-GD ISGC 2008 www.eu-egee.org EGEE and glite are registered trademarks Overview The grid problem definition GLite and EGEE The interoperability problem The
A Comparative Study of Various Computing Environments-Cluster, Grid and Cloud
Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology IJCSMC, Vol. 4, Issue. 6, June 2015, pg.1065
High Performance Computing Course Notes HPC Fundamentals
High Performance Computing Course Notes 2008-2009 2009 HPC Fundamentals Introduction What is High Performance Computing (HPC)? Difficult to define - it s a moving target. Later 1980s, a supercomputer performs
Improving Peer-to-Peer Resource Usage Through Idle Cycle Prediction
Improving Peer-to-Peer Resource Usage Through Idle Cycle Prediction Elton Nicoletti Mathias, Andrea Schwertner Charão, Marcelo Pasin LSC - Laboratório de Sistemas de Computação UFSM - Universidade Federal
Grid Programming: Concepts and Challenges. Michael Rokitka CSE510B 10/2007
Grid Programming: Concepts and Challenges Michael Rokitka SUNY@Buffalo CSE510B 10/2007 Issues Due to Heterogeneous Hardware level Environment Different architectures, chipsets, execution speeds Software
Programming Techniques for Supercomputers. HPC RRZE University Erlangen-Nürnberg Sommersemester 2018
Programming Techniques for Supercomputers HPC Services @ RRZE University Erlangen-Nürnberg Sommersemester 2018 Outline Login to RRZE s Emmy cluster Basic environment Some guidelines First Assignment 2
Storage and Compute Resource Management via DYRE, 3DcacheGrid, and CompuStore
Storage and Compute Resource Management via DYRE, 3DcacheGrid, and CompuStore Ioan Raicu Distributed Systems Laboratory Computer Science Department University of Chicago DSL Seminar November st, 006 Analysis
Parallel Programming & Cluster Computing High Throughput Computing
Parallel Programming & Cluster Computing High Throughput Computing Henry Neeman, University of Oklahoma Charlie Peck, Earlham College Tuesday October 11 2011 Outline What is High Throughput Computing?
Future Generation Computer Systems. Computational models and heuristic methods for Grid scheduling problems
Future Generation Computer Systems 26 (2010) 608 621 Contents lists available at ScienceDirect Future Generation Computer Systems journal homepage: www.elsevier.com/locate/fgcs Computational models and
Scheduling. The Basics
The Basics refers to a set of policies and mechanisms to control the order of work to be performed by a computer system. Of all the resources in a computer system that are scheduled before use, the CPU
Predictable Time-Sharing for DryadLINQ Cluster. Sang-Min Park and Marty Humphrey Dept. of Computer Science University of Virginia
Predictable Time-Sharing for DryadLINQ Cluster Sang-Min Park and Marty Humphrey Dept. of Computer Science University of Virginia 1 DryadLINQ What is DryadLINQ? LINQ: Data processing language and run-time
Chapter 5: CPU Scheduling
Chapter 5: CPU Scheduling Basic Concepts Scheduling Criteria Scheduling Algorithms Thread Scheduling Multiple-Processor Scheduling Operating Systems Examples Algorithm Evaluation Chapter 5: CPU Scheduling
Operating Systems. studykorner.org
Operating Systems Outlines What are Operating Systems? All components Description, Types of Operating Systems Multi programming systems, Time sharing systems, Parallel systems, Real Time systems, Distributed
COLLIN LEE INITIAL DESIGN THOUGHTS FOR A GRANULAR COMPUTING PLATFORM
COLLIN LEE INITIAL DESIGN THOUGHTS FOR A GRANULAR COMPUTING PLATFORM INITIAL DESIGN THOUGHTS FOR A GRANULAR COMPUTING PLATFORM GOAL OF THIS TALK Introduce design ideas and issues for a granular computing
Module 1: Introduction
Module 1: Introduction What is an operating system? Simple Batch Systems Multiprogramming Batched Systems Time-Sharing Systems Personal-Computer Systems Parallel Systems Distributed Systems Real-Time Systems
The LGI Pilot job portal. EGI Technical Forum 20 September 2011 Jan Just Keijser Willem van Engen Mark Somers
The LGI Pilot job portal EGI Technical Forum 20 September 2011 Jan Just Keijser Willem van Engen Mark Somers Outline What? Why? How? Pro's and Cons What's next? Credits 2 What is LGI? LGI Project Server
CS420: Operating Systems
Virtual Memory James Moscola Department of Physical Sciences York College of Pennsylvania Based on Operating System Concepts, 9th Edition by Silberschatz, Galvin, Gagne Background Code needs to be in memory
CS10 The Beauty and Joy of Computing
CS10 The Beauty and Joy of Computing Lecture #19 Distributed Computing UC Berkeley EECS Lecturer SOE Dan Garcia 2010-11-08 Researchers at Indiana U used data mining techniques to uncover evidence that
Processes and Threads
OPERATING SYSTEMS CS3502 Spring 2018 Processes and Threads (Chapter 2) Processes Two important types of dynamic entities in a computer system are processes and threads. Dynamic entities only exist at execution
3. CPU Scheduling. Operating System Concepts with Java 8th Edition Silberschatz, Galvin and Gagn
3. CPU Scheduling Operating System Concepts with Java 8th Edition Silberschatz, Galvin and Gagn S P O I L E R operating system CPU Scheduling 3 operating system CPU Scheduling 4 Long-short-medium Scheduler
CPU Scheduling Algorithms
CPU Scheduling Algorithms Notice: The slides for this lecture have been largely based on those accompanying the textbook Operating Systems Concepts with Java, by Silberschatz, Galvin, and Gagne (2007).
STORING DATA: DISK AND FILES
STORING DATA: DISK AND FILES CS 564- Spring 2018 ACKs: Dan Suciu, Jignesh Patel, AnHai Doan WHAT IS THIS LECTURE ABOUT? How does a DBMS store data? disk, SSD, main memory The Buffer manager controls how
ALL the assignments (A1, A2, A3) and Projects (P0, P1, P2) we have done so far.
Midterm Exam Reviews ALL the assignments (A1, A2, A3) and Projects (P0, P1, P2) we have done so far. Particular attentions on the following: System call, system kernel Thread/process, thread vs process