Cloud Computing. Summary

Size: px
Start display at page:

Download "Cloud Computing. Summary"

Transcription

1 Cloud Computing Lectures 2 and 3 Definition of Cloud Computing, Grid Architectures Summary Definition of Cloud Computing (more complete). Grid Computing: Conceptual Architecture. Condor. 1

2 Cloud computing as an Utility Computing as an utility (water, electricity, internet): Illusion of infinite resources on demand: 1 CPU x 100 h = 100 CPU x 1h Removal of an a priori commitment: No (or very low) initial costs. Small granularity payments: Computing paid by CPU hour, web request, storage or bandwidth. CC as an Extension of Web Hosting (1) Shared hosting: One server with multiple websites. Virtual dedicated server: Resource sharing among different websites on the same machine (possibly proportional to payment). Managed dedicated hosting: One server per website with partial admin rights (for good QoS). 2

3 CC as an Extension of Web Hosting (2) Dedicated hosting: One server per client (simple and flexible). Collocated hosting: Client owns the machine (Provider guarantees network QoS). Cluster hosting: Multiple rented machines (for scalability). Grid hosting: Multiple geographically distributed and replicated machines (scalability, speed, fault tolerance). The Cloud for the End-User Utility computing: I just want to rent CPU cycles. Example: Amazon EC2. Platform as a Service (PaaS): Give me an API and I ll handle the implementation. Example: Google App Engine. Software as a Service(SaaS): I want a finished application available off-site. Example: Gmail. 3

4 3 Visions of CC: Who Will Win? Amazon Web Services Microsoft Azure Google App Engine Computing x86 CLR (VM) Applicational Framework (Python, Java) Storage (initial approach) Disk blocks SQL Server BigTable API Network IP address ranges Declarative but automatic 3 level app hosting The Successes of Cloud Computing Scalability, grow as you go along: Grow the company: Start-ups with no HW risk. Grow the application, e.g. Google indexes an evergrowing set of data formats. Utility aspects: No system administrators. Low granularity payments. Varied development paradigms (VM, web app, etc ) 4

5 The Failures of Cloud Computing 1. Availability 2. Lock-In 3. Privacy and Auditing 4. Cost of data transfer 5. Performance reliability 6. Scalable storage 7. Large scale bugs 8. Scale-up time 9. Reputation propagation 10. Compatible licensing (See Above the clouds ) Let s roll back the clock 30 years! Cloud -> Grid -> Clusters 5

6 Before the Grid: the 80s... Parallel computing was quite widespread Separation between machines blurred by technologies like: MPI, Message Passing Interface: MPI_Send, MPI_Recv. PVM, similar. OpenMP, Open Multi-Processing: API for concurrent programming. A Growing Need for Large Scale Computing Computer scientists: Simulation of computational systems (HW, networking). Natural sciences: processing of experimental data. Companies: Data mining, prototyping. Climate and environmental simulations. Teaching: immersive environments. 6

7 Example of a Large Debit Computing Infrastructure ~PBytes/sec online system ~100 MBytes/sec offline cluster ~100 MBytes/sec or air mail ~622 Mbits/sec Level 0 CERN computing center Level 1 Regional Center France Regional Center Germany Regional Center Italy Fermilab ~622 Mbits/sec ~622 Mbits/sec Level 2 Level 2 Center Level 2 Center Level 2 Center Level 2 Center Level 2 Center ~1 TIPS ~1 TIPS ~1 TIPS ~1 TIPS ~1 TIPS Institute Institute ~0.25TIPS Institute Institute Level 3 Local Data Cache ~1 MBytes/sec Personal PCs Level 4 The Beginning of the Grid Initially intended as an aggregation of cluster. but,... It became a much more sophisticated integration mechanism for heterogeneous applications and resources. Resulting in a trans-national inter-organizational Integrated Computing platform. 7

8 The Components Local networks (LAN) and large scale networks (WAN). Computational nodes: PCs, large servers and clusters. Partnerships / Virtual organizations: members from multiple organizations of different types (univ., companies, labs.) Existing service implementations. Other HW and SW. Implications How to share resources: It s always conditional: issues of trust, policies, negotiation, payment, reputation. How to coordinate the participants: Distribute data and computation ensuring coordination, fairness, availability. In a nutshell, how to implement complex policies in virtual organizations: I want to execute program X at site Y using a particular resource consumption policy Z, accessing data at site W with permissions P. In a way that is: User friendly, Does not compromise performance, Ensures security. 8

9 Grid: Conceptual Architecture Tools and Applications Aplications Descovery, negotiation, diagnostics & monitoring Secure access to resources Colective Services Communication and Resource Management Protocols Resources: CPU, storage, networking Fabric Fabric: CPU Computational resources: Task scheduling and monitoring. Control the resources attached to each task. Watch hardware and software metrics. 9

10 Storage resources: Fabric: Storage Access mechanisms (open, close, read, write, seek). Resource management (disk quotas and bandwidth) and corresponding reservations. Monitoring tools to assess available resources. Fabric: Connectivity Communication resources: Transfer scheduling and priorities. Monitoring available capabilities: bandwidth and load. 10

11 Communication Management Protocols TCP/IP stack and WS / WSDL / UDDI Wanted properties: Single sign-on. Delegation of permissions. Integration of multiple security solutions and implementations. User centred trust. Resource Management Protocols Information: What resources are available? What is the current/possible configurations? What is the current load? Management: Negotiate and reserve access. Specify operations/tasks. Monitor job progress. 11

12 Colective Services Resource discovery in virtual organizations. Scheduling and negotiation. Task execution monitoring. Data Replication. Authorization and auditing. Payment. Some Current Technologies Schedulers: Condor. Sun Grid Engine. Nimrod-G (parallel/concurrent workflow engine). Core Grid Services: discovery, security, job management, data transfer: Globus. Legion. 12

13 Condor An opportunistic scheduler for local networks. Implemented as a set of daemons spread over a central nodes and a pool of distributed worker nodes. Features Distributed job submission. Job suspension and restart. ClassAds: policy specification/advertising/negotiation among nodes. User priorities. Task priorities. Task dependencies. Flocking: federation of different Condor clusters. 13

14 Condor Daemons Condor local master startd negotiator schedd collector condor_master General coordinator of the system. Defines the master node. Starts all other daemons (the only one started manually) Restarts any daemon that fails or is updated (by checking file dates) and notifies the system administrator. 14

15 condor_master(2) It is the daemon that executes most administrative commands: condor_reconfig, condor_restart, condor_off, condor_on, condor_config_val, etc. condor_startd Represents a computing node. Manages jobs: starting, suspending, restarting, stopping. Applies the machine s policies (see later Class Ads). 15

16 condor_schedd Represents a user. Keeps a persistent job queue. Sends jobs to available nodes. Executes job queue commands: condor_submit,condor_rm, condor_q, condor_hold, condor_release, condor_prio, condor_collector Runs only in the master node. Gathers information from all nodes in the pool: The pool s directory/database. Each pool node periodically sends a ClassAdto the collector. Responds to information requests: From other daemons. From users (condor_status). 16

17 condor_negotiator Runs in the central master node only. Matches jobs to Condor nodes. Asks the collector for the list of free nodes and the list of idle jobs. Tries to match jobs to nodes taking into account the job request and the nodes characteristics and policy. Starting a Job Schedd negotiator (@ master) Startd 5. Starter Shadow Customer Job Submit Condor Syscall Lib 17

18 General Condor Commands condor_status Info on machine state condor_q List request queue condor_submit Submit new jobs condor_rm Remove jobs condor_prio Manage job priority condor_history List job history condor_submit_dag Specify job dependencies condor_checkpoint Execute a job checkpoint Condor: Compiling tasks. Next time Submitting tasks:.sub files. ClassAds, policies: specification and match making. DAGMan. 18

HTCondor overview. by Igor Sfiligoi, Jeff Dost (UCSD)

HTCondor overview. by Igor Sfiligoi, Jeff Dost (UCSD) HTCondor overview by Igor Sfiligoi, Jeff Dost (UCSD) Acknowledgement These slides are heavily based on the presentation Todd Tannenbaum gave at CERN in Feb 2011 https://indico.cern.ch/event/124982/timetable/#20110214.detailed

More information

Tutorial 4: Condor. John Watt, National e-science Centre

Tutorial 4: Condor. John Watt, National e-science Centre Tutorial 4: Condor John Watt, National e-science Centre Tutorials Timetable Week Day/Time Topic Staff 3 Fri 11am Introduction to Globus J.W. 4 Fri 11am Globus Development J.W. 5 Fri 11am Globus Development

More information

Cloud Computing. Up until now

Cloud Computing. Up until now Cloud Computing Lectures 3 and 4 Grid Schedulers: Condor, Sun Grid Engine 2012-2013 Introduction. Up until now Definition of Cloud Computing. Grid Computing: Schedulers: Condor architecture. 1 Summary

More information

Cloud Computing. Up until now

Cloud Computing. Up until now Cloud Computing Lecture 4 and 5 Grid: 2012-2013 Introduction. Up until now Definition of Cloud Computing. Grid Computing: Schedulers: Condor SGE 1 Summary Core Grid: Toolkit Condor-G Grid: Conceptual Architecture

More information

Condor and BOINC. Distributed and Volunteer Computing. Presented by Adam Bazinet

Condor and BOINC. Distributed and Volunteer Computing. Presented by Adam Bazinet Condor and BOINC Distributed and Volunteer Computing Presented by Adam Bazinet Condor Developed at the University of Wisconsin-Madison Condor is aimed at High Throughput Computing (HTC) on collections

More information

Grid Compute Resources and Grid Job Management

Grid Compute Resources and Grid Job Management Grid Compute Resources and Job Management March 24-25, 2007 Grid Job Management 1 Job and compute resource management! This module is about running jobs on remote compute resources March 24-25, 2007 Grid

More information

Grid Compute Resources and Job Management

Grid Compute Resources and Job Management Grid Compute Resources and Job Management How do we access the grid? Command line with tools that you'll use Specialised applications Ex: Write a program to process images that sends data to run on the

More information

Condor-G and DAGMan An Introduction

Condor-G and DAGMan An Introduction Condor-G and DAGMan An Introduction Condor Project Computer Sciences Department University of Wisconsin-Madison condor-admin@cs.wisc.edu / tutorials/miron-condor-g-dagmantutorial.html 2 Outline Overview

More information

Condor-G Stork and DAGMan An Introduction

Condor-G Stork and DAGMan An Introduction Condor-G Stork and DAGMan An Introduction Condor Project Computer Sciences Department University of Wisconsin-Madison condor-admin@cs.wisc.edu Outline Background and principals The Story of Frieda, the

More information

What s new in HTCondor? What s coming? HTCondor Week 2018 Madison, WI -- May 22, 2018

What s new in HTCondor? What s coming? HTCondor Week 2018 Madison, WI -- May 22, 2018 What s new in HTCondor? What s coming? HTCondor Week 2018 Madison, WI -- May 22, 2018 Todd Tannenbaum Center for High Throughput Computing Department of Computer Sciences University of Wisconsin-Madison

More information

DISTRIBUTED SYSTEMS [COMP9243] Lecture 8a: Cloud Computing WHAT IS CLOUD COMPUTING? 2. Slide 3. Slide 1. Why is it called Cloud?

DISTRIBUTED SYSTEMS [COMP9243] Lecture 8a: Cloud Computing WHAT IS CLOUD COMPUTING? 2. Slide 3. Slide 1. Why is it called Cloud? DISTRIBUTED SYSTEMS [COMP9243] Lecture 8a: Cloud Computing Slide 1 Slide 3 ➀ What is Cloud Computing? ➁ X as a Service ➂ Key Challenges ➃ Developing for the Cloud Why is it called Cloud? services provided

More information

M. Roehrig, Sandia National Laboratories. Philipp Wieder, Research Centre Jülich Nov 2002

M. Roehrig, Sandia National Laboratories. Philipp Wieder, Research Centre Jülich Nov 2002 Category: INFORMATIONAL Grid Scheduling Dictionary WG (SD-WG) M. Roehrig, Sandia National Laboratories Wolfgang Ziegler, Fraunhofer-Institute for Algorithms and Scientific Computing Philipp Wieder, Research

More information

Distributed and Cloud Computing

Distributed and Cloud Computing Jiří Kašpar, Pavel Tvrdík (ČVUT FIT) Distributed and Cloud Computing MI-POA, 2011, Lecture 12 1/28 Distributed and Cloud Computing Ing. Jiří Kašpar prof. Ing. Pavel Tvrdík CSc. Department of Computer Systems

More information

Outline. Definition of a Distributed System Goals of a Distributed System Types of Distributed Systems

Outline. Definition of a Distributed System Goals of a Distributed System Types of Distributed Systems Distributed Systems Outline Definition of a Distributed System Goals of a Distributed System Types of Distributed Systems What Is A Distributed System? A collection of independent computers that appears

More information

Grid Computing Fall 2005 Lecture 5: Grid Architecture and Globus. Gabrielle Allen

Grid Computing Fall 2005 Lecture 5: Grid Architecture and Globus. Gabrielle Allen Grid Computing 7700 Fall 2005 Lecture 5: Grid Architecture and Globus Gabrielle Allen allen@bit.csc.lsu.edu http://www.cct.lsu.edu/~gallen Concrete Example I have a source file Main.F on machine A, an

More information

Work Queue + Python. A Framework For Scalable Scientific Ensemble Applications

Work Queue + Python. A Framework For Scalable Scientific Ensemble Applications Work Queue + Python A Framework For Scalable Scientific Ensemble Applications Peter Bui, Dinesh Rajan, Badi Abdul-Wahid, Jesus Izaguirre, Douglas Thain University of Notre Dame Distributed Computing Examples

More information

PROOF-Condor integration for ATLAS

PROOF-Condor integration for ATLAS PROOF-Condor integration for ATLAS G. Ganis,, J. Iwaszkiewicz, F. Rademakers CERN / PH-SFT M. Livny, B. Mellado, Neng Xu,, Sau Lan Wu University Of Wisconsin Condor Week, Madison, 29 Apr 2 May 2008 Outline

More information

HTCONDOR USER TUTORIAL. Greg Thain Center for High Throughput Computing University of Wisconsin Madison

HTCONDOR USER TUTORIAL. Greg Thain Center for High Throughput Computing University of Wisconsin Madison HTCONDOR USER TUTORIAL Greg Thain Center for High Throughput Computing University of Wisconsin Madison gthain@cs.wisc.edu 2015 Internet2 HTCondor User Tutorial CONTENTS Overview Basic job submission How

More information

Monitoring and Analytics With HTCondor Data

Monitoring and Analytics With HTCondor Data Monitoring and Analytics With HTCondor Data William Strecker-Kellogg RACF/SDCC @ BNL 1 RHIC/ATLAS Computing Facility (SDCC) Who are we? See our last two site reports from the HEPiX conference for a good

More information

Cycle Sharing Systems

Cycle Sharing Systems Cycle Sharing Systems Jagadeesh Dyaberi Dependable Computing Systems Lab Purdue University 10/31/2005 1 Introduction Design of Program Security Communication Architecture Implementation Conclusion Outline

More information

Introduction to Distributed Systems. INF5040/9040 Autumn 2018 Lecturer: Eli Gjørven (ifi/uio)

Introduction to Distributed Systems. INF5040/9040 Autumn 2018 Lecturer: Eli Gjørven (ifi/uio) Introduction to Distributed Systems INF5040/9040 Autumn 2018 Lecturer: Eli Gjørven (ifi/uio) August 28, 2018 Outline Definition of a distributed system Goals of a distributed system Implications of distributed

More information

! " # " $ $ % & '(()

!  #  $ $ % & '(() !"# " $ $ % &'(() First These slides are available from: http://www.cs.wisc.edu/~roy/italy-condor/ 2 This Morning s Condor Topics * +&, & - *.&- *. & * && - * + $ 3 Part One Matchmaking: Finding Machines

More information

Adaptive Cluster Computing using JavaSpaces

Adaptive Cluster Computing using JavaSpaces Adaptive Cluster Computing using JavaSpaces Jyoti Batheja and Manish Parashar The Applied Software Systems Lab. ECE Department, Rutgers University Outline Background Introduction Related Work Summary of

More information

Cloud Programming. Programming Environment Oct 29, 2015 Osamu Tatebe

Cloud Programming. Programming Environment Oct 29, 2015 Osamu Tatebe Cloud Programming Programming Environment Oct 29, 2015 Osamu Tatebe Cloud Computing Only required amount of CPU and storage can be used anytime from anywhere via network Availability, throughput, reliability

More information

Cloud Computing 4/17/2016. Outline. Cloud Computing. Centralized versus Distributed Computing Some people argue that Cloud Computing. Cloud Computing.

Cloud Computing 4/17/2016. Outline. Cloud Computing. Centralized versus Distributed Computing Some people argue that Cloud Computing. Cloud Computing. Cloud Computing By: Muhammad Naseem Assistant Professor Department of Computer Engineering, Sir Syed University of Engineering & Technology, Web: http://sites.google.com/site/muhammadnaseem105 Email: mnaseem105@yahoo.com

More information

Care and Feeding of HTCondor Cluster. Steven Timm European HTCondor Site Admins Meeting 8 December 2014

Care and Feeding of HTCondor Cluster. Steven Timm European HTCondor Site Admins Meeting 8 December 2014 Care and Feeding of HTCondor Cluster Steven Timm European HTCondor Site Admins Meeting 8 December 2014 Disclaimer Some HTCondor configuration and operations questions are more religion than science. There

More information

Grid Programming: Concepts and Challenges. Michael Rokitka CSE510B 10/2007

Grid Programming: Concepts and Challenges. Michael Rokitka CSE510B 10/2007 Grid Programming: Concepts and Challenges Michael Rokitka SUNY@Buffalo CSE510B 10/2007 Issues Due to Heterogeneous Hardware level Environment Different architectures, chipsets, execution speeds Software

More information

Introducing the HTCondor-CE

Introducing the HTCondor-CE Introducing the HTCondor-CE CHEP 2015 Presented by Edgar Fajardo 1 Introduction In summer 2012, OSG performed an internal review of major software components, looking for strategic weaknesses. One highlighted

More information

Special Topics: CSci 8980 Edge History

Special Topics: CSci 8980 Edge History Special Topics: CSci 8980 Edge History Jon B. Weissman (jon@cs.umn.edu) Department of Computer Science University of Minnesota P2P: What is it? No always-on server Nodes are at the network edge; come and

More information

30 Nov Dec Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy

30 Nov Dec Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy Why the Grid? Science is becoming increasingly digital and needs to deal with increasing amounts of

More information

CPSC 426/526. Cloud Computing. Ennan Zhai. Computer Science Department Yale University

CPSC 426/526. Cloud Computing. Ennan Zhai. Computer Science Department Yale University CPSC 426/526 Cloud Computing Ennan Zhai Computer Science Department Yale University Recall: Lec-7 In the lec-7, I talked about: - P2P vs Enterprise control - Firewall - NATs - Software defined network

More information

Developing Enterprise Cloud Solutions with Azure

Developing Enterprise Cloud Solutions with Azure Developing Enterprise Cloud Solutions with Azure Java Focused 5 Day Course AUDIENCE FORMAT Developers and Software Architects Instructor-led with hands-on labs LEVEL 300 COURSE DESCRIPTION This course

More information

Tools and Methodology for Ensuring HPC Programs Correctness and Performance. Beau Paisley

Tools and Methodology for Ensuring HPC Programs Correctness and Performance. Beau Paisley Tools and Methodology for Ensuring HPC Programs Correctness and Performance Beau Paisley bpaisley@allinea.com About Allinea Over 15 years of business focused on parallel programming development tools Strong

More information

Advanced School in High Performance and GRID Computing November Introduction to Grid computing.

Advanced School in High Performance and GRID Computing November Introduction to Grid computing. 1967-14 Advanced School in High Performance and GRID Computing 3-14 November 2008 Introduction to Grid computing. TAFFONI Giuliano Osservatorio Astronomico di Trieste/INAF Via G.B. Tiepolo 11 34131 Trieste

More information

Cloud Computing. What is cloud computing. CS 537 Fall 2017

Cloud Computing. What is cloud computing. CS 537 Fall 2017 Cloud Computing CS 537 Fall 2017 What is cloud computing Illusion of infinite computing resources available on demand Scale-up for most apps Elimination of up-front commitment Small initial investment,

More information

Introduction to Grid Computing

Introduction to Grid Computing Milestone 2 Include the names of the papers You only have a page be selective about what you include Be specific; summarize the authors contributions, not just what the paper is about. You might be able

More information

AutoPyFactory: A Scalable Flexible Pilot Factory Implementation

AutoPyFactory: A Scalable Flexible Pilot Factory Implementation ATL-SOFT-PROC-2012-045 22 May 2012 Not reviewed, for internal circulation only AutoPyFactory: A Scalable Flexible Pilot Factory Implementation J. Caballero 1, J. Hover 1, P. Love 2, G. A. Stewart 3 on

More information

Day 9: Introduction to CHTC

Day 9: Introduction to CHTC Day 9: Introduction to CHTC Suggested reading: Condor 7.7 Manual: http://www.cs.wisc.edu/condor/manual/v7.7/ Chapter 1: Overview Chapter 2: Users Manual (at most, 2.1 2.7) 1 Turn In Homework 2 Homework

More information

Cloud Computing. Technologies and Types

Cloud Computing. Technologies and Types Cloud Computing Cloud Computing Technologies and Types Dell Zhang Birkbeck, University of London 2017/18 The Technological Underpinnings of Cloud Computing Data centres Virtualisation RESTful APIs Cloud

More information

OSG Lessons Learned and Best Practices. Steven Timm, Fermilab OSG Consortium August 21, 2006 Site and Fabric Parallel Session

OSG Lessons Learned and Best Practices. Steven Timm, Fermilab OSG Consortium August 21, 2006 Site and Fabric Parallel Session OSG Lessons Learned and Best Practices Steven Timm, Fermilab OSG Consortium August 21, 2006 Site and Fabric Parallel Session Introduction Ziggy wants his supper at 5:30 PM Users submit most jobs at 4:59

More information

UW-ATLAS Experiences with Condor

UW-ATLAS Experiences with Condor UW-ATLAS Experiences with Condor M.Chen, A. Leung, B.Mellado Sau Lan Wu and N.Xu Paradyn / Condor Week, Madison, 05/01/08 Outline Our first success story with Condor - ATLAS production in 2004~2005. CRONUS

More information

! " #$%! &%& ' ( $ $ ) $*+ $

!  #$%! &%& ' ( $ $ ) $*+ $ ! " #$%! &%& ' ( $ $ ) $*+ $ + 2 &,-)%./01 ) 2 $ & $ $ ) 340 %(%% 3 &,-)%./01 ) 2 $& $ $ ) 34 0 %(%% $ $ %% $ ) 5 67 89 5 & % % %$ %%)( % % ( %$ ) ( '$ :!"#$%%&%'&( )!)&(!&( *+,& )- &*./ &*( ' 0&/ 1&2

More information

HTCondor Essentials. Index

HTCondor Essentials. Index HTCondor Essentials 31.10.2017 Index Login How to submit a job in the HTCondor pool Why the -name option? Submitting a job Checking status of submitted jobs Getting id and other info about a job

More information

What s new in HTCondor? What s coming? European HTCondor Workshop June 8, 2017

What s new in HTCondor? What s coming? European HTCondor Workshop June 8, 2017 What s new in HTCondor? What s coming? European HTCondor Workshop June 8, 2017 Todd Tannenbaum Center for High Throughput Computing Department of Computer Sciences University of Wisconsin-Madison Release

More information

Chapter 3. Design of Grid Scheduler. 3.1 Introduction

Chapter 3. Design of Grid Scheduler. 3.1 Introduction Chapter 3 Design of Grid Scheduler The scheduler component of the grid is responsible to prepare the job ques for grid resources. The research in design of grid schedulers has given various topologies

More information

Lecture 23 Database System Architectures

Lecture 23 Database System Architectures CMSC 461, Database Management Systems Spring 2018 Lecture 23 Database System Architectures These slides are based on Database System Concepts 6 th edition book (whereas some quotes and figures are used

More information

Chapter 4:- Introduction to Grid and its Evolution. Prepared By:- NITIN PANDYA Assistant Professor SVBIT.

Chapter 4:- Introduction to Grid and its Evolution. Prepared By:- NITIN PANDYA Assistant Professor SVBIT. Chapter 4:- Introduction to Grid and its Evolution Prepared By:- Assistant Professor SVBIT. Overview Background: What is the Grid? Related technologies Grid applications Communities Grid Tools Case Studies

More information

Chapter 20: Database System Architectures

Chapter 20: Database System Architectures Chapter 20: Database System Architectures Chapter 20: Database System Architectures Centralized and Client-Server Systems Server System Architectures Parallel Systems Distributed Systems Network Types

More information

Lecture 11 Hadoop & Spark

Lecture 11 Hadoop & Spark Lecture 11 Hadoop & Spark Dr. Wilson Rivera ICOM 6025: High Performance Computing Electrical and Computer Engineering Department University of Puerto Rico Outline Distributed File Systems Hadoop Ecosystem

More information

Things you may not know about HTCondor. John (TJ) Knoeller Condor Week 2017

Things you may not know about HTCondor. John (TJ) Knoeller Condor Week 2017 Things you may not know about HTCondor John (TJ) Knoeller Condor Week 2017 -limit not just for condor_history condor_q -limit Show no more than jobs. Ignored if Schedd is before 8.6 condor_status

More information

Distributed Systems COMP 212. Lecture 18 Othon Michail

Distributed Systems COMP 212. Lecture 18 Othon Michail Distributed Systems COMP 212 Lecture 18 Othon Michail Virtualisation & Cloud Computing 2/27 Protection rings It s all about protection rings in modern processors Hardware mechanism to protect data and

More information

Look What I Can Do: Unorthodox Uses of HTCondor in the Open Science Grid

Look What I Can Do: Unorthodox Uses of HTCondor in the Open Science Grid Look What I Can Do: Unorthodox Uses of HTCondor in the Open Science Grid Mátyás Selmeci Open Science Grid Software Team / Center for High- Throughput Computing HTCondor Week 2015 More Than a Batch System

More information

Cloud Computing. Ennan Zhai. Computer Science at Yale University

Cloud Computing. Ennan Zhai. Computer Science at Yale University Cloud Computing Ennan Zhai Computer Science at Yale University ennan.zhai@yale.edu About Final Project About Final Project Important dates before demo session: - Oct 31: Proposal v1.0 - Nov 7: Source code

More information

Building Campus HTC Sharing Infrastructures. Derek Weitzel University of Nebraska Lincoln (Open Science Grid Hat)

Building Campus HTC Sharing Infrastructures. Derek Weitzel University of Nebraska Lincoln (Open Science Grid Hat) Building Campus HTC Sharing Infrastructures Derek Weitzel University of Nebraska Lincoln (Open Science Grid Hat) HCC: Campus Grids Motivation We have 3 clusters in 2 cities. Our largest (4400 cores) is

More information

Grid Scheduling Architectures with Globus

Grid Scheduling Architectures with Globus Grid Scheduling Architectures with Workshop on Scheduling WS 07 Cetraro, Italy July 28, 2007 Ignacio Martin Llorente Distributed Systems Architecture Group Universidad Complutense de Madrid 1/38 Contents

More information

CSE6331: Cloud Computing

CSE6331: Cloud Computing CSE6331: Cloud Computing Leonidas Fegaras University of Texas at Arlington c 2019 by Leonidas Fegaras Cloud Computing Fundamentals Based on: J. Freire s class notes on Big Data http://vgc.poly.edu/~juliana/courses/bigdata2016/

More information

Things you may not know about HTCondor. John (TJ) Knoeller Condor Week 2017

Things you may not know about HTCondor. John (TJ) Knoeller Condor Week 2017 Things you may not know about HTCondor John (TJ) Knoeller Condor Week 2017 -limit not just for condor_history condor_q -limit Show no more than jobs. Ignored if Schedd is before 8.6 condor_status

More information

COMP6511A: Large-Scale Distributed Systems. Windows Azure. Lin Gu. Hong Kong University of Science and Technology Spring, 2014

COMP6511A: Large-Scale Distributed Systems. Windows Azure. Lin Gu. Hong Kong University of Science and Technology Spring, 2014 COMP6511A: Large-Scale Distributed Systems Windows Azure Lin Gu Hong Kong University of Science and Technology Spring, 2014 Cloud Systems Infrastructure as a (IaaS): basic compute and storage resources

More information

Programming model and implementation for processing and. Programs can be automatically parallelized and executed on a large cluster of machines

Programming model and implementation for processing and. Programs can be automatically parallelized and executed on a large cluster of machines A programming model in Cloud: MapReduce Programming model and implementation for processing and generating large data sets Users specify a map function to generate a set of intermediate key/value pairs

More information

First Principles Vulnerability Assessment. Motivation

First Principles Vulnerability Assessment. Motivation First Principles Vulnerability Assessment James A. Kupsch Barton P. Miller Computer Sciences Department University of Wisconsin Elisa Heymann Eduardo César Computer Architecture and Operating Systems Department

More information

Introduction to Distributed Systems (DS)

Introduction to Distributed Systems (DS) Introduction to Distributed Systems (DS) INF5040/9040 autumn 2014 lecturer: Frank Eliassen Frank Eliassen, Ifi/UiO 1 Outline Ø What is a distributed system? Ø Challenges and benefits of distributed systems

More information

glideinwms architecture by Igor Sfiligoi, Jeff Dost (UCSD)

glideinwms architecture by Igor Sfiligoi, Jeff Dost (UCSD) glideinwms architecture by Igor Sfiligoi, Jeff Dost (UCSD) Outline A high level overview of the glideinwms Description of the components 2 glideinwms from 10k feet 3 Refresher - HTCondor A Condor pool

More information

glideinwms UCSD Condor tunning by Igor Sfiligoi (UCSD) UCSD Jan 18th 2012 Condor Tunning 1

glideinwms UCSD Condor tunning by Igor Sfiligoi (UCSD) UCSD Jan 18th 2012 Condor Tunning 1 glideinwms Training @ UCSD Condor tunning by Igor Sfiligoi (UCSD) UCSD Jan 18th 2012 Condor Tunning 1 Regulating User Priorities UCSD Jan 18th 2012 Condor Tunning 2 User priorities By default, the Negotiator

More information

Chapter 18: Database System Architectures.! Centralized Systems! Client--Server Systems! Parallel Systems! Distributed Systems!

Chapter 18: Database System Architectures.! Centralized Systems! Client--Server Systems! Parallel Systems! Distributed Systems! Chapter 18: Database System Architectures! Centralized Systems! Client--Server Systems! Parallel Systems! Distributed Systems! Network Types 18.1 Centralized Systems! Run on a single computer system and

More information

CMS Tier-2 Program for user Analysis Computing on the Open Science Grid Frank Würthwein UCSD Goals & Status

CMS Tier-2 Program for user Analysis Computing on the Open Science Grid Frank Würthwein UCSD Goals & Status CMS Tier-2 Program for user Analysis Computing on the Open Science Grid Frank Würthwein UCSD Goals & Status High Level Requirements for user analysis computing Code Development Environment Compile, run,

More information

Technical Memorandum

Technical Memorandum Technical Memorandum Strong Mobile Agent Architecture based on the Beowulf System Written and Prepared by Sungwoo Tak Passakon Prathombutr Donghoon Lee E.K. Park Jerrold Stach Table of Contents 1 Beowulf

More information

Cloud & container monitoring , Lars Michelsen Check_MK Conference #4

Cloud & container monitoring , Lars Michelsen Check_MK Conference #4 Cloud & container monitoring 04.05.2018, Lars Michelsen Some cloud definitions Applications Data Runtime Middleware O/S Virtualization Servers Storage Networking Software-as-a-Service (SaaS) Applications

More information

Grid Computing. MCSN - N. Tonellotto - Distributed Enabling Platforms

Grid Computing. MCSN - N. Tonellotto - Distributed Enabling Platforms Grid Computing 1 Resource sharing Elements of Grid Computing - Computers, data, storage, sensors, networks, - Sharing always conditional: issues of trust, policy, negotiation, payment, Coordinated problem

More information

Forget about the Clouds, Shoot for the MOON

Forget about the Clouds, Shoot for the MOON Forget about the Clouds, Shoot for the MOON Wu FENG feng@cs.vt.edu Dept. of Computer Science Dept. of Electrical & Computer Engineering Virginia Bioinformatics Institute September 2012, W. Feng Motivation

More information

Operating systems Architecture

Operating systems Architecture Operating systems Architecture 1 Operating Systems Low level software system that manages all applications implements an interface between applications and resources manages available resources Resource

More information

Cloud Essentials for Architects using OpenStack

Cloud Essentials for Architects using OpenStack Cloud Essentials for Architects using OpenStack Course Overview Start Date 5th March 2015 Duration 2 Days Location Dublin Course Code SS15-13 Programme Overview Cloud Computing is gaining increasing attention

More information

Introduction to Grid Technology

Introduction to Grid Technology Introduction to Grid Technology B.Ramamurthy 1 Arthur C Clarke s Laws (two of many) Any sufficiently advanced technology is indistinguishable from magic." "The only way of discovering the limits of the

More information

Introduction to Condor. Jari Varje

Introduction to Condor. Jari Varje Introduction to Condor Jari Varje 25. 27.4.2016 Outline Basics Condor overview Submitting a job Monitoring jobs Parallel jobs Advanced topics Host requirements Running MATLAB jobs Checkpointing Case study:

More information

Apache Hadoop 3. Balazs Gaspar Sales Engineer CEE & CIS Cloudera, Inc. All rights reserved.

Apache Hadoop 3. Balazs Gaspar Sales Engineer CEE & CIS Cloudera, Inc. All rights reserved. Apache Hadoop 3 Balazs Gaspar Sales Engineer CEE & CIS balazs@cloudera.com 1 We believe data can make what is impossible today, possible tomorrow 2 We empower people to transform complex data into clear

More information

FACULTY OF ENGINEERING B.E. 4/4 (CSE) II Semester (Old) Examination, June Subject : Information Retrieval Systems (Elective III) Estelar

FACULTY OF ENGINEERING B.E. 4/4 (CSE) II Semester (Old) Examination, June Subject : Information Retrieval Systems (Elective III) Estelar B.E. 4/4 (CSE) II Semester (Old) Examination, June 2014 Subject : Information Retrieval Systems Code No. 6306 / O 1 Define Information retrieval systems. 3 2 What is precision and recall? 3 3 List the

More information

Technical Brief. A Checklist for Every API Call. Managing the Complete API Lifecycle

Technical Brief. A Checklist for Every API Call. Managing the Complete API Lifecycle Technical Brief A Checklist for Table of Contents Introduction: The API Lifecycle 2 3 Security professionals API developers Operations engineers API product or business owners Apigee Edge 7 A Checklist

More information

Chapter 1: Introduction 1/29

Chapter 1: Introduction 1/29 Chapter 1: Introduction 1/29 What is a Distributed System? A distributed system is a collection of independent computers that appears to its users as a single coherent system. 2/29 Characteristics of a

More information

VMware Join the Virtual Revolution! Brian McNeil VMware National Partner Business Manager

VMware Join the Virtual Revolution! Brian McNeil VMware National Partner Business Manager VMware Join the Virtual Revolution! Brian McNeil VMware National Partner Business Manager 1 VMware By the Numbers Year Founded Employees R&D Engineers with Advanced Degrees Technology Partners Channel

More information

UVA HPC & BIG DATA COURSE. Cloud Computing. Adam Belloum

UVA HPC & BIG DATA COURSE. Cloud Computing. Adam Belloum UVA HPC & BIG DATA COURSE Cloud Computing Adam Belloum outline Cloud computing: Approach and vision Resource Provisioning in Cloud systems: Cloud Systems: IaaS, PaaS, SaaS Using Cloud Systems in practice

More information

Whiteboard 6 feet by 4 feet (minimum) Whiteboard markers Red, Blue, Green, Black Video Projector (1024 X 768 resolutions)

Whiteboard 6 feet by 4 feet (minimum) Whiteboard markers Red, Blue, Green, Black Video Projector (1024 X 768 resolutions) Workshop Name Windows Azure Platform as a Service (PaaS) Duration 6 Days Objective Build development skills on the cloud platform from Microsoft Windows Azure Platform Participants Entry Profile Participants

More information

ECE Enterprise Storage Architecture. Fall ~* CLOUD *~. Tyler Bletsch Duke University

ECE Enterprise Storage Architecture. Fall ~* CLOUD *~. Tyler Bletsch Duke University ECE590-03 Enterprise Storage Architecture Fall 2017.~* CLOUD *~. Tyler Bletsch Duke University Includes material adapted from the course Information Storage and Management v2 (module 13), published by

More information

Lecture 09: VMs and VCS head in the clouds

Lecture 09: VMs and VCS head in the clouds Lecture 09: VMs and VCS head in the Hands-on Unix system administration DeCal 2012-10-29 1 / 20 Projects groups of four people submit one form per group with OCF usernames, proposed project ideas, and

More information

DISTRIBUTED SYSTEMS Principles and Paradigms Second Edition ANDREW S. TANENBAUM MAARTEN VAN STEEN. Chapter 1. Introduction

DISTRIBUTED SYSTEMS Principles and Paradigms Second Edition ANDREW S. TANENBAUM MAARTEN VAN STEEN. Chapter 1. Introduction DISTRIBUTED SYSTEMS Principles and Paradigms Second Edition ANDREW S. TANENBAUM MAARTEN VAN STEEN Chapter 1 Introduction Modified by: Dr. Ramzi Saifan Definition of a Distributed System (1) A distributed

More information

Grid Middleware and Globus Toolkit Architecture

Grid Middleware and Globus Toolkit Architecture Grid Middleware and Globus Toolkit Architecture Lisa Childers Argonne National Laboratory University of Chicago 2 Overview Grid Middleware The problem: supporting Virtual Organizations equirements Capabilities

More information

Managing large-scale workflows with Pegasus

Managing large-scale workflows with Pegasus Funded by the National Science Foundation under the OCI SDCI program, grant #0722019 Managing large-scale workflows with Pegasus Karan Vahi ( vahi@isi.edu) Collaborative Computing Group USC Information

More information

High Performance Computing Course Notes Grid Computing I

High Performance Computing Course Notes Grid Computing I High Performance Computing Course Notes 2008-2009 2009 Grid Computing I Resource Demands Even as computer power, data storage, and communication continue to improve exponentially, resource capacities are

More information

Mission-Critical Databases in the Cloud. Oracle RAC in Microsoft Azure Enabled by FlashGrid Software.

Mission-Critical Databases in the Cloud. Oracle RAC in Microsoft Azure Enabled by FlashGrid Software. Mission-Critical Databases in the Cloud. Oracle RAC in Microsoft Azure Enabled by FlashGrid Software. White Paper rev. 2017-10-16 2017 FlashGrid Inc. 1 www.flashgrid.io Abstract Ensuring high availability

More information

Course Overview. ECE 1779 Introduction to Cloud Computing. Marking. Class Mechanics. Eyal de Lara

Course Overview. ECE 1779 Introduction to Cloud Computing. Marking. Class Mechanics. Eyal de Lara ECE 1779 Introduction to Cloud Computing Eyal de Lara delara@cs.toronto.edu www.cs.toronto.edu/~delara/courses/ece1779 Course Overview Date Topic Sep 14 Introduction Sep 21 Python Sep 22 Tutorial: Python

More information

Clouds in High Energy Physics

Clouds in High Energy Physics Clouds in High Energy Physics Randall Sobie University of Victoria Randall Sobie IPP/Victoria 1 Overview Clouds are integral part of our HEP computing infrastructure Primarily Infrastructure-as-a-Service

More information

High Throughput, Low Impedance e-science on Microsoft Azure

High Throughput, Low Impedance e-science on Microsoft Azure High Throughput, Low Impedance e-science on Microsoft Azure David Abramson & Blair Bethwaite Monash e-science and Grid Engineering Lab (MeSsAGE Lab) Faculty of Information Technology Monash e-research

More information

Integrate MATLAB Analytics into Enterprise Applications

Integrate MATLAB Analytics into Enterprise Applications Integrate Analytics into Enterprise Applications Aurélie Urbain MathWorks Consulting Services 2015 The MathWorks, Inc. 1 Data Analytics Workflow Data Acquisition Data Analytics Analytics Integration Business

More information

The Intersection of Cloud & Solid State Storage

The Intersection of Cloud & Solid State Storage The Intersection of Cloud & Solid State Storage Val Bercovici Cloud Czar, NetApp Office of the CTO SNIA Cloud Storage Initiative SNIA Solid State Storage Initiative Cloud Backdrop Worldwide IT spending

More information

New Directions and BNL

New Directions and BNL New Directions and HTCondor @ BNL USATLAS TIER-3 & NEW COMPUTING DIRECTIVES William Strecker-Kellogg RHIC/ATLAS Computing Facility (RACF) Brookhaven National Lab May 2016 RACF Overview 2 RHIC Collider

More information

Developing Microsoft Azure Solutions (70-532) Syllabus

Developing Microsoft Azure Solutions (70-532) Syllabus Developing Microsoft Azure Solutions (70-532) Syllabus Cloud Computing Introduction What is Cloud Computing Cloud Characteristics Cloud Computing Service Models Deployment Models in Cloud Computing Advantages

More information

Cloud platforms. T Mobile Systems Programming

Cloud platforms. T Mobile Systems Programming Cloud platforms T-110.5130 Mobile Systems Programming Agenda 1. Motivation 2. Different types of cloud platforms 3. Popular cloud services 4. Open-source cloud 5. Cloud on this course 6. Mobile Edge Computing

More information

Integrate MATLAB Analytics into Enterprise Applications

Integrate MATLAB Analytics into Enterprise Applications Integrate Analytics into Enterprise Applications Lyamine Hedjazi 2015 The MathWorks, Inc. 1 Data Analytics Workflow Preprocessing Data Business Systems Build Algorithms Smart Connected Systems Take Decisions

More information

2010 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media,

2010 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, 2010 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising

More information

A Simulation Model for Large Scale Distributed Systems

A Simulation Model for Large Scale Distributed Systems A Simulation Model for Large Scale Distributed Systems Ciprian M. Dobre and Valentin Cristea Politechnica University ofbucharest, Romania, e-mail. **Politechnica University ofbucharest, Romania, e-mail.

More information

Developing Microsoft Azure Solutions (70-532) Syllabus

Developing Microsoft Azure Solutions (70-532) Syllabus Developing Microsoft Azure Solutions (70-532) Syllabus Cloud Computing Introduction What is Cloud Computing Cloud Characteristics Cloud Computing Service Models Deployment Models in Cloud Computing Advantages

More information

ALHAD G. APTE, BARC 2nd GARUDA PARTNERS MEET ON 15th & 16th SEPT. 2006

ALHAD G. APTE, BARC 2nd GARUDA PARTNERS MEET ON 15th & 16th SEPT. 2006 GRID COMPUTING ACTIVITIES AT BARC ALHAD G. APTE, BARC 2nd GARUDA PARTNERS MEET ON 15th & 16th SEPT. 2006 Computing Grid at BARC Computing Grid system has been set up as a Test-Bed using existing Grid Technology

More information