GridNEWS: A distributed Grid platform for efficient storage, annotating, indexing and searching of large audiovisual news content
|
|
- Maximillian Porter
- 6 years ago
- Views:
Transcription
1 1st HellasGrid User Forum 10-11/1/2008 GridNEWS: A distributed Grid platform for efficient storage, annotating, indexing and searching of large audiovisual news content Ioannis Konstantinou School of ECE Computing Systems Laboratory National Technical University of Athens
2 Concept Text based search in audiovisual content Search results: Portions of video files containing selected keywords Example User searches for keyword Acropolis Video portions containing the spoken word Acropolis are located and presented in the user YouTube like functionality
3 Objectives Keyword extraction from video files using automatic speech recognition algorithms (ASR) Efficient and scalable distributed storage of large media content Indexing of extracted metadata for efficient keyword search YouTube like user interface for video searching/downloading Contribution to existing Grid Middleware using GGF standardized components
4 Addressed Issues Execution Distributed execution of CPU/Data intensive Speech Recognition Algorithms Storage Server load balancing using performance metrics Client transfer time optimization using bittorrent like algorithms Increase data availability Multi-organizational data storage support using Virtual Organizations (VOs)
5 Video file keyword extraction procedure MetaScheduler Application Software Storage Cluster Cluster Schedule Execution CVSP tool Data Client GET/PUT File Ops SE Export Cluster Import Metadata Parser SE SE Metadata File Update Metadata DB
6 Distributed Execution Platform Architecture MDS : Managing and Discovery Service PBS : Portable Batch Scheduler WSGRAM : Grid Resource and Allocation Management WN: Worker Node Central MDS Server User Interface GridWay Metascheduler Globus Toolkit 4 MDS Publish aggregate Cluster information SOAP/XML messages Computing Element PBS Scheduler Globus Toolkit 4 MDS + WSGRAM PBS client service Ganglia Tool Local MDS Server Local MDS Server Local MDS Server Ganglia Stats WN WN WN WN WN WN WN WN WN
7 Distributed Storage Platform Architecture MDS: Managing and Discovery Service GridFTP: Grid File Transfer Protocol DRLS: Distributed Replica Location Service SE: Storage Element PFN: Physical File Name LFN: Logical File Name DRLS Get/Put LFN to PFN mappings SOAP/XML Messages Pastry Peer to Peer Storage Subsystem Client Machine Query Storage Subsystem to obtain candidate storage servers using SOAP/XML Messages GridFTP Get/put SE SE Replication Algorithm using NWS stats SE GridNews DataClient component GridFTP Client Parallel Downloader SE GridFTP Server Network Weather Service (NWS) DRLSCom Service Globus WS Apache Axis Container
8 Distributed Replica Location Service Contains mappings of LFNs to PFNs DHT used: Pastry P2P Logarithmic routing In a network with n nodes, a query needs only log(n) messages (hops) Plaxton s algorithm minimizes query latencies Redundancy through replication Eliminates single point of failure situations Inherent load balancing capabilities Consistent hashing algorithms
9 Load Balancing Servers exchange load metrics CPU Bandwidth Free Disk Space Prediction algorithms (e.g. Linear regression) forecast future metrics from history data Weighted Normalized Metric WNM : Wm X(M t / M max ) Servers maintain sorted WNM list : [WNM 1,..WNM n ] WNM list periodically refreshed
10 Replication Upon a STORE client request: Top k servers are selected from WNM list k: configurable static replication factor Most suitable server is returned to the client Client initiates a single GridFTP file upload Server replicates the new file according to WNM list and factor k DRLS is informed about the new LFN->PFN mappings Client is informed Upon completion
11 Parallel Downloader Upon a GET client request: Server contacts DRLS and retrieves replica locations Client establishes N GridFTP connections Client initiates N parallel (threaded) small data chunk requests After each successful retrieval, client reinitiates another request Optimum file transfer time: The greater file portion is retrieved from the faster storage nodes
12 GridTorrent Metadata fields Current Id File size Piece size Hashes Distributed RLS instead of Tracker Partial GridFTP for actual transfer BitTorrent replica selection and tit-for-tat algorithm. Compatible with plain GridFTP servers PFN s prefix determines protocol (gtp://site.fully.qualified.d omain.name/path/to/file) size,piece_size, hashes, currentid Peer that wants to download a file New connection Bittorrent piece Peer1 GridTorrent client Bittorrent Bittorrent New connection piece publish pfns Striped GridFTP Peer2 GridTorrent client New connection Striped GridFTP piece piece piece Peer3 GridFTP Server Striped GridFTP piece Distributed RLS Peer set CurrentID :pfn1 pfn2 pfn3
13 EXISTING MIDDLEWARE CONTRIBUTION Design and development of a dynamic replica selection/placement algorithm Added support for multiple clusters using Gridway Metascheduler Enhanced bittorrent like file downloading from multiple sources Replace centralized replica location service with a scalable distributed peer to peer solution
14 Development Testbed Hardware 4X Dual Core AMD Opteron(tm) Processor GHz 8 virtual CPU 16Gb Ram Deployment of 5 Xen virtual machines, 2GB ram Software Globus Toolkit v4.0 Globus WS Core (Apache AXIS WS Container) Rice Pastry P2P (Java) Network Weather Service Torque (OpenPBS) scheduler GridWay Metascheduler
15 Virtualization Xen Hypervisor Paravirtualization tequnique Guest OS use special xen aware kernel Direct utilization of special CPU instructions Faster than full virtualization (VmWare) Use of Xen Hypervisor Easy prototype management/administration Simple control of the node lifecycle Facilitate prototype deployment in many actual nodes
16 Currently working Replace ParallelDownloader with GridTorrent Deploy prototype in the PlanetLab testbed Run experiments Fine-tune designed algorithms
17 Gridnews Portal Users can: Perform keyword search in the autoannotated multimedia content View the video from their browser in a youtube style Download only a fragment of the video where this keyword exists
18 Screenshots keyword Video URLs time position
19 Questions
Architecture Proposal
Nordic Testbed for Wide Area Computing and Data Handling NORDUGRID-TECH-1 19/02/2002 Architecture Proposal M.Ellert, A.Konstantinov, B.Kónya, O.Smirnova, A.Wäänänen Introduction The document describes
More information30 Nov Dec Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy
Advanced School in High Performance and GRID Computing Concepts and Applications, ICTP, Trieste, Italy Why the Grid? Science is becoming increasingly digital and needs to deal with increasing amounts of
More informationComputer Science Section. Computational and Information Systems Laboratory National Center for Atmospheric Research
Computer Science Section Computational and Information Systems Laboratory National Center for Atmospheric Research My work in the context of TDD/CSS/ReSET Polynya new research computing environment Polynya
More informationCloud Computing. Up until now
Cloud Computing Lecture 4 and 5 Grid: 2012-2013 Introduction. Up until now Definition of Cloud Computing. Grid Computing: Schedulers: Condor SGE 1 Summary Core Grid: Toolkit Condor-G Grid: Conceptual Architecture
More informationvcloud Automation Center Reference Architecture vcloud Automation Center 5.2
vcloud Automation Center Reference Architecture vcloud Automation Center 5.2 This document supports the version of each product listed and supports all subsequent versions until the document is replaced
More informationGrid Architectural Models
Grid Architectural Models Computational Grids - A computational Grid aggregates the processing power from a distributed collection of systems - This type of Grid is primarily composed of low powered computers
More informationGrid Middleware and Globus Toolkit Architecture
Grid Middleware and Globus Toolkit Architecture Lisa Childers Argonne National Laboratory University of Chicago 2 Overview Grid Middleware The problem: supporting Virtual Organizations equirements Capabilities
More informationChapter 4:- Introduction to Grid and its Evolution. Prepared By:- NITIN PANDYA Assistant Professor SVBIT.
Chapter 4:- Introduction to Grid and its Evolution Prepared By:- Assistant Professor SVBIT. Overview Background: What is the Grid? Related technologies Grid applications Communities Grid Tools Case Studies
More informationThe Integration of Grid Technology with OGC Web Services (OWS) in NWGISS for NASA EOS Data
The Integration of Grid Technology with OGC Web Services (OWS) in NWGISS for NASA EOS Data Liping Di, Aijun Chen, Wenli Yang and Peisheng Zhao achen6@gmu.edu; achen@laits.gmu.edu Lab for Advanced Information
More informationIoan Raicu. Everyone else. More information at: Background? What do you want to get out of this course?
Ioan Raicu More information at: http://www.cs.iit.edu/~iraicu/ Everyone else Background? What do you want to get out of this course? 2 Data Intensive Computing is critical to advancing modern science Applies
More informationA Performance Evaluation of WS-MDS in the Globus Toolkit
A Performance Evaluation of WS-MDS in the Globus Toolkit Ioan Raicu * Catalin Dumitrescu * Ian Foster +* * Computer Science Department The University of Chicago {iraicu,cldumitr}@cs.uchicago.edu Abstract
More informationvsan Remote Office Deployment January 09, 2018
January 09, 2018 1 1. vsan Remote Office Deployment 1.1.Solution Overview Table of Contents 2 1. vsan Remote Office Deployment 3 1.1 Solution Overview Native vsphere Storage for Remote and Branch Offices
More informationIntegration of Cloud and Grid Middleware at DGRZR
D- of International Symposium on Computing 2010 Stefan Freitag Robotics Research Institute Dortmund University of Technology March 12, 2010 Overview D- 1 D- Resource Center Ruhr 2 Clouds in the German
More informationglite Grid Services Overview
The EPIKH Project (Exchange Programme to advance e-infrastructure Know-How) glite Grid Services Overview Antonio Calanducci INFN Catania Joint GISELA/EPIKH School for Grid Site Administrators Valparaiso,
More informationIntroduction to Grid Computing
Milestone 2 Include the names of the papers You only have a page be selective about what you include Be specific; summarize the authors contributions, not just what the paper is about. You might be able
More informationVirtualization. Michael Tsai 2018/4/16
Virtualization Michael Tsai 2018/4/16 What is virtualization? Let s first look at a video from VMware http://www.vmware.com/tw/products/vsphere.html Problems? Low utilization Different needs DNS DHCP Web
More informationNative vsphere Storage for Remote and Branch Offices
SOLUTION OVERVIEW VMware vsan Remote Office Deployment Native vsphere Storage for Remote and Branch Offices VMware vsan is the industry-leading software powering Hyper-Converged Infrastructure (HCI) solutions.
More informationFREE SCIENTIFIC COMPUTING
Institute of Physics, Belgrade Scientific Computing Laboratory FREE SCIENTIFIC COMPUTING GRID COMPUTING Branimir Acković March 4, 2007 Petnica Science Center Overview 1/2 escience Brief History of UNIX
More informationWrite a technical report Present your results Write a workshop/conference paper (optional) Could be a real system, simulation and/or theoretical
Identify a problem Review approaches to the problem Propose a novel approach to the problem Define, design, prototype an implementation to evaluate your approach Could be a real system, simulation and/or
More informationEsgynDB Enterprise 2.0 Platform Reference Architecture
EsgynDB Enterprise 2.0 Platform Reference Architecture This document outlines a Platform Reference Architecture for EsgynDB Enterprise, built on Apache Trafodion (Incubating) implementation with licensed
More informationThe Hadoop Distributed File System Konstantin Shvachko Hairong Kuang Sanjay Radia Robert Chansler
The Hadoop Distributed File System Konstantin Shvachko Hairong Kuang Sanjay Radia Robert Chansler MSST 10 Hadoop in Perspective Hadoop scales computation capacity, storage capacity, and I/O bandwidth by
More informationGrid services. Enabling Grids for E-sciencE. Dusan Vudragovic Scientific Computing Laboratory Institute of Physics Belgrade, Serbia
Grid services Dusan Vudragovic dusan@phy.bg.ac.yu Scientific Computing Laboratory Institute of Physics Belgrade, Serbia Sep. 19, 2008 www.eu-egee.org Set of basic Grid services Job submission/management
More informationIntroduction to SRM. Riccardo Zappi 1
Introduction to SRM Grid Storage Resource Manager Riccardo Zappi 1 1 INFN-CNAF, National Center of INFN (National Institute for Nuclear Physic) for Research and Development into the field of Information
More informationThe course modules of MongoDB developer and administrator online certification training:
The course modules of MongoDB developer and administrator online certification training: 1 An Overview of the Course Introduction to the course Table of Contents Course Objectives Course Overview Value
More informationVoldemort. Smruti R. Sarangi. Department of Computer Science Indian Institute of Technology New Delhi, India. Overview Design Evaluation
Voldemort Smruti R. Sarangi Department of Computer Science Indian Institute of Technology New Delhi, India Smruti R. Sarangi Leader Election 1/29 Outline 1 2 3 Smruti R. Sarangi Leader Election 2/29 Data
More informationGrid Programming: Concepts and Challenges. Michael Rokitka CSE510B 10/2007
Grid Programming: Concepts and Challenges Michael Rokitka SUNY@Buffalo CSE510B 10/2007 Issues Due to Heterogeneous Hardware level Environment Different architectures, chipsets, execution speeds Software
More informationData Management 1. Grid data management. Different sources of data. Sensors Analytic equipment Measurement tools and devices
Data Management 1 Grid data management Different sources of data Sensors Analytic equipment Measurement tools and devices Need to discover patterns in data to create information Need mechanisms to deal
More informationDatabase Assessment for PDMS
Database Assessment for PDMS Abhishek Gaurav, Nayden Markatchev, Philip Rizk and Rob Simmonds Grid Research Centre, University of Calgary. http://grid.ucalgary.ca 1 Introduction This document describes
More informationEfficient On-Demand Operations in Distributed Infrastructures
Efficient On-Demand Operations in Distributed Infrastructures Steve Ko and Indranil Gupta Distributed Protocols Research Group University of Illinois at Urbana-Champaign 2 One-Line Summary We need to design
More informationOverview of HEP software & LCG from the openlab perspective
Overview of HEP software & LCG from the openlab perspective Andreas Unterkircher, CERN openlab February 2005 Andreas Unterkircher 1 Contents 1. Opencluster overview 2. High Energy Physics (HEP) software
More informationUnderstanding StoRM: from introduction to internals
Understanding StoRM: from introduction to internals 13 November 2007 Outline Storage Resource Manager The StoRM service StoRM components and internals Deployment configuration Authorization and ACLs Conclusions.
More informationPerformance Analysis of Applying Replica Selection Technology for Data Grid Environments*
Performance Analysis of Applying Replica Selection Technology for Data Grid Environments* Chao-Tung Yang 1,, Chun-Hsiang Chen 1, Kuan-Ching Li 2, and Ching-Hsien Hsu 3 1 High-Performance Computing Laboratory,
More informationXen and CloudStack. Ewan Mellor. Director, Engineering, Open-source Cloud Platforms Citrix Systems
Xen and CloudStack Ewan Mellor Director, Engineering, Open-source Cloud Platforms Citrix Systems Agenda What is CloudStack? Move to the Apache Foundation CloudStack architecture on Xen The future for CloudStack
More information70-414: Implementing an Advanced Server Infrastructure Course 01 - Creating the Virtualization Infrastructure
70-414: Implementing an Advanced Server Infrastructure Course 01 - Creating the Virtualization Infrastructure Slide 1 Creating the Virtualization Infrastructure Slide 2 Introducing Microsoft System Center
More informationDeploying virtualisation in a production grid
Deploying virtualisation in a production grid Stephen Childs Trinity College Dublin & Grid-Ireland TERENA NRENs and Grids workshop 2 nd September 2008 www.eu-egee.org EGEE and glite are registered trademarks
More informationEmpirical Evaluation of Latency-Sensitive Application Performance in the Cloud
Empirical Evaluation of Latency-Sensitive Application Performance in the Cloud Sean Barker and Prashant Shenoy University of Massachusetts Amherst Department of Computer Science Cloud Computing! Cloud
More informationCopyright 2012, Oracle and/or its affiliates. All rights reserved.
1 ! 2 Oracle VM Introduction Adam Hawley, Senior Director Virtualization, Oracle January 15, 2013 Safe Harbor Statement The following is intended to outline our general product direction. It is intended
More informationPeer to Peer Systems and Probabilistic Protocols
Distributed Systems 600.437 Peer to Peer Systems & Probabilistic Protocols Department of Computer Science The Johns Hopkins University 1 Peer to Peer Systems and Probabilistic Protocols Lecture 11 Good
More informationKnowledge Discovery Services and Tools on Grids
Knowledge Discovery Services and Tools on Grids DOMENICO TALIA DEIS University of Calabria ITALY talia@deis.unical.it Symposium ISMIS 2003, Maebashi City, Japan, Oct. 29, 2003 OUTLINE Introduction Grid
More informationHigh Throughput WAN Data Transfer with Hadoop-based Storage
High Throughput WAN Data Transfer with Hadoop-based Storage A Amin 2, B Bockelman 4, J Letts 1, T Levshina 3, T Martin 1, H Pi 1, I Sfiligoi 1, M Thomas 2, F Wuerthwein 1 1 University of California, San
More informationParallel & Cluster Computing. cs 6260 professor: elise de doncker by: lina hussein
Parallel & Cluster Computing cs 6260 professor: elise de doncker by: lina hussein 1 Topics Covered : Introduction What is cluster computing? Classification of Cluster Computing Technologies: Beowulf cluster
More informationPerformance & Scalability Testing in Virtual Environment Hemant Gaidhani, Senior Technical Marketing Manager, VMware
Performance & Scalability Testing in Virtual Environment Hemant Gaidhani, Senior Technical Marketing Manager, VMware 2010 VMware Inc. All rights reserved About the Speaker Hemant Gaidhani Senior Technical
More informationGrid Computing Middleware. Definitions & functions Middleware components Globus glite
Seminar Review 1 Topics Grid Computing Middleware Grid Resource Management Grid Computing Security Applications of SOA and Web Services Semantic Grid Grid & E-Science Grid Economics Cloud Computing 2 Grid
More informationBy Ian Foster. Zhifeng Yun
By Ian Foster Zhifeng Yun Outline Introduction Globus Architecture Globus Software Details Dev.Globus Community Summary Future Readings Introduction Globus Toolkit v4 is the work of many Globus Alliance
More informationNetwork Platform for Creating Services over Virtualized Networks
Network Platform for Creating Services over Virtualized Networks August 24, 2012 Toru Hasegawa KDDI R&D Labs. Outline Introduction Toy Programming Environment Network Platform Example Service Conclusion
More informationOpen-source library and tools to support the OVF.
Open OVF Proposal Open-source library and tools to support the OVF http://xml.coverpages.org/ni2007-09-11-a.html IBM will be donating code and invites all to form an open community around OVF Eclipse Public
More informationProfiling Grid Data Transfer Protocols and Servers. George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA
Profiling Grid Data Transfer Protocols and Servers George Kola, Tevfik Kosar and Miron Livny University of Wisconsin-Madison USA Motivation Scientific experiments are generating large amounts of data Education
More informationCourse Content MongoDB
Course Content MongoDB 1. Course introduction and mongodb Essentials (basics) 2. Introduction to NoSQL databases What is NoSQL? Why NoSQL? Difference Between RDBMS and NoSQL Databases Benefits of NoSQL
More informationData Staging: Moving large amounts of data around, and moving it close to compute resources
Data Staging: Moving large amounts of data around, and moving it close to compute resources PRACE advanced training course on Data Staging and Data Movement Helsinki, September 10 th 2013 Claudio Cacciari
More informationThe glite middleware. Ariel Garcia KIT
The glite middleware Ariel Garcia KIT Overview Background The glite subsystems overview Security Information system Job management Data management Some (my) answers to your questions and random rumblings
More informationCS555: Distributed Systems [Fall 2017] Dept. Of Computer Science, Colorado State University
CS 555: DISTRIBUTED SYSTEMS [P2P SYSTEMS] Shrideep Pallickara Computer Science Colorado State University Frequently asked questions from the previous class survey Byzantine failures vs malicious nodes
More informationSharePoint 2010 Technical Case Study: Microsoft SharePoint Server 2010 Enterprise Intranet Collaboration Environment
SharePoint 2010 Technical Case Study: Microsoft SharePoint Server 2010 Enterprise Intranet Collaboration Environment This document is provided as-is. Information and views expressed in this document, including
More informationSphinx: A Scheduling Middleware for Data Intensive Applications on a Grid
Sphinx: A Scheduling Middleware for Data Intensive Applications on a Grid Richard Cavanaugh University of Florida Collaborators: Janguk In, Sanjay Ranka, Paul Avery, Laukik Chitnis, Gregory Graham (FNAL),
More informationVMs at a Tier-1 site. EGEE 09, Sander Klous, Nikhef
VMs at a Tier-1 site EGEE 09, 21-09-2009 Sander Klous, Nikhef Contents Introduction Who are we? Motivation Why are we interested in VMs? What are we going to do with VMs? Status How do we approach this
More informationGridFTP Scalability and Performance Results Ioan Raicu Catalin Dumitrescu -
GridFTP Scalability and Performance Results 2/12/25 Page 1 of 13 GridFTP Scalability and Performance Results Ioan Raicu iraicu@cs.uchicago.edu Catalin Dumitrescu - catalind@cs.uchicago.edu 1. Introduction
More informationA Decentralized Content-based Aggregation Service for Pervasive Environments
A Decentralized Content-based Aggregation Service for Pervasive Environments Nanyan Jiang, Cristina Schmidt, Manish Parashar The Applied Software Systems Laboratory Rutgers, The State University of New
More informationSystem Support for Internet of Things
System Support for Internet of Things Kishore Ramachandran (Kirak Hong - Google, Dave Lillethun, Dushmanta Mohapatra, Steffen Maas, Enrique Saurez Apuy) Overview Motivation Mobile Fog: A Distributed
More informationFusion Architecture. Planning for an on-premise deployment
Planning for an on-premise deployment Fusion Architecture Part number: VV/GUIDE/CUST/62/D Copyright 2017 VoiceVault Inc. All rights reserved. This document may not be copied, reproduced, transmitted or
More informationHDFS: Hadoop Distributed File System. CIS 612 Sunnie Chung
HDFS: Hadoop Distributed File System CIS 612 Sunnie Chung What is Big Data?? Bulk Amount Unstructured Introduction Lots of Applications which need to handle huge amount of data (in terms of 500+ TB per
More informationGlobus Toolkit 4 Execution Management. Alexandra Jimborean International School of Informatics Hagenberg, 2009
Globus Toolkit 4 Execution Management Alexandra Jimborean International School of Informatics Hagenberg, 2009 2 Agenda of the day Introduction to Globus Toolkit and GRAM Zoom In WS GRAM Usage Guide Architecture
More informationAdvanced School in High Performance and GRID Computing November Introduction to Grid computing.
1967-14 Advanced School in High Performance and GRID Computing 3-14 November 2008 Introduction to Grid computing. TAFFONI Giuliano Osservatorio Astronomico di Trieste/INAF Via G.B. Tiepolo 11 34131 Trieste
More informationGridMonitor: Integration of Large Scale Facility Fabric Monitoring with Meta Data Service in Grid Environment
GridMonitor: Integration of Large Scale Facility Fabric Monitoring with Meta Data Service in Grid Environment Rich Baker, Dantong Yu, Jason Smith, and Anthony Chan RHIC/USATLAS Computing Facility Department
More informationCLOUD COMPUTING IT0530. G.JEYA BHARATHI Asst.Prof.(O.G) Department of IT SRM University
CLOUD COMPUTING IT0530 G.JEYA BHARATHI Asst.Prof.(O.G) Department of IT SRM University What is virtualization? Virtualization is way to run multiple operating systems and user applications on the same
More informationA Case for High Performance Computing with Virtual Machines
A Case for High Performance Computing with Virtual Machines Wei Huang*, Jiuxing Liu +, Bulent Abali +, and Dhabaleswar K. Panda* *The Ohio State University +IBM T. J. Waston Research Center Presentation
More informationGrid Computing Systems: A Survey and Taxonomy
Grid Computing Systems: A Survey and Taxonomy Material for this lecture from: A Survey and Taxonomy of Resource Management Systems for Grid Computing Systems, K. Krauter, R. Buyya, M. Maheswaran, CS Technical
More informationSSIM Collection & Archiving Infrastructure Scaling & Performance Tuning Guide
SSIM Collection & Archiving Infrastructure Scaling & Performance Tuning Guide April 2013 SSIM Engineering Team Version 3.0 1 Document revision history Date Revision Description of Change Originator 03/20/2013
More informationSynergy: Sharing-Aware Component Composition for Distributed Stream Processing Systems
Synergy: Sharing-Aware Component Composition for Distributed Stream Processing Systems Thomas Repantis 1, Xiaohui Gu 2, and Vana Kalogeraki 1, 1 Dept. of Computer Science & Engineering, University of California,
More informationCompSci 356: Computer Network Architectures Lecture 21: Overlay Networks Chap 9.4. Xiaowei Yang
CompSci 356: Computer Network Architectures Lecture 21: Overlay Networks Chap 9.4 Xiaowei Yang xwy@cs.duke.edu Overview Problem Evolving solutions IP multicast Proxy caching Content distribution networks
More informationMapReduce. U of Toronto, 2014
MapReduce U of Toronto, 2014 http://www.google.org/flutrends/ca/ (2012) Average Searches Per Day: 5,134,000,000 2 Motivation Process lots of data Google processed about 24 petabytes of data per day in
More informationGAIA CU6 Bruxelles Meeting (12-13 october 2006)
GAIA CU6 Bruxelles Meeting (12-13 october 2006) Preparation of CNES DPC Infrastructure Technology studies prepared by F. Jocteur Monrozier Context: GAIA CNES Infrastructure: Functional blocks import /
More informationXen Summit Spring 2007
Xen Summit Spring 2007 Platform Virtualization with XenEnterprise Rich Persaud 4/20/07 Copyright 2005-2006, XenSource, Inc. All rights reserved. 1 Xen, XenSource and XenEnterprise
More informationStorage Virtualization. Eric Yen Academia Sinica Grid Computing Centre (ASGC) Taiwan
Storage Virtualization Eric Yen Academia Sinica Grid Computing Centre (ASGC) Taiwan Storage Virtualization In computer science, storage virtualization uses virtualization to enable better functionality
More informationVIProf: A Vertically Integrated Full-System Profiler
VIProf: A Vertically Integrated Full-System Profiler NGS Workshop, April 2007 Hussam Mousa Chandra Krintz Lamia Youseff Rich Wolski RACELab Research Dynamic software adaptation As program behavior or resource
More informationKenneth A. Hawick P. D. Coddington H. A. James
Student: Vidar Tulinius Email: vidarot@brandeis.edu Distributed frameworks and parallel algorithms for processing large-scale geographic data Kenneth A. Hawick P. D. Coddington H. A. James Overview Increasing
More informationZhengyang Liu! Oct 25, Supported by NSF Grant OCI
SDCI Net: Collaborative Research: An integrated study of datacenter networking and 100 GigE wide-area networking in support of distributed scientific computing Zhengyang Liu! Oct 25, 2013 Supported by
More informationICAT Job Portal. a generic job submission system built on a scientific data catalog. IWSG 2013 ETH, Zurich, Switzerland 3-5 June 2013
ICAT Job Portal a generic job submission system built on a scientific data catalog IWSG 2013 ETH, Zurich, Switzerland 3-5 June 2013 Steve Fisher, Kevin Phipps and Dan Rolfe Rutherford Appleton Laboratory
More informationIntel Enterprise Edition Lustre (IEEL-2.3) [DNE-1 enabled] on Dell MD Storage
Intel Enterprise Edition Lustre (IEEL-2.3) [DNE-1 enabled] on Dell MD Storage Evaluation of Lustre File System software enhancements for improved Metadata performance Wojciech Turek, Paul Calleja,John
More informationWhite Paper. Major Performance Tuning Considerations for Weblogic Server
White Paper Major Performance Tuning Considerations for Weblogic Server Table of Contents Introduction and Background Information... 2 Understanding the Performance Objectives... 3 Measuring your Performance
More informationSeasonal forecast modeling application on the GARUDA Grid infrastructure
Seasonal forecast modeling application on the GARUDA Grid infrastructure 1, S. Janakiraman, Mohit Ved and B. B. Prahlada Rao Centre for Development of Advanced Computing CDAC Knowledge Park, Byappanahalli,
More informationOpenNebula on VMware: Cloud Reference Architecture
OpenNebula on VMware: Cloud Reference Architecture Version 1.2, October 2016 Abstract The OpenNebula Cloud Reference Architecture is a blueprint to guide IT architects, consultants, administrators and
More informationPredicting the Performance of a GRID Environment: An Initial Effort to Increase Scheduling Efficiency
Predicting the Performance of a GRID Environment: An Initial Effort to Increase Scheduling Efficiency Nuno Guerreiro and Orlando Belo Department of Informatics, School of Engineering, University of Minho
More informationUX - User Experience: Multi-Cloud Network Visibility
Data Sheet UX - User Experience: Multi-Cloud Network Visibility UX - User Experience monitoring User Experience dashboard summary application performance User Experience is a software module available
More informationCLIENT DATA NODE NAME NODE
Volume 6, Issue 12, December 2016 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Efficiency
More informationCisco HyperFlex Hyperconverged Infrastructure Solution for SAP HANA
Cisco HyperFlex Hyperconverged Infrastructure Solution for SAP HANA Learn best practices for running SAP HANA on the Cisco HyperFlex hyperconverged infrastructure (HCI) solution. 2018 Cisco and/or its
More informationSDS: A Scalable Data Services System in Data Grid
SDS: A Scalable Data s System in Data Grid Xiaoning Peng School of Information Science & Engineering, Central South University Changsha 410083, China Department of Computer Science and Technology, Huaihua
More informationGrid Scheduling Architectures with Globus
Grid Scheduling Architectures with Workshop on Scheduling WS 07 Cetraro, Italy July 28, 2007 Ignacio Martin Llorente Distributed Systems Architecture Group Universidad Complutense de Madrid 1/38 Contents
More informationLayered Architecture
The Globus Toolkit : Introdution Dr Simon See Sun APSTC 09 June 2003 Jie Song, Grid Computing Specialist, Sun APSTC 2 Globus Toolkit TM An open source software toolkit addressing key technical problems
More informationSuccessMaker Learning Management System System Requirements v1.0
SuccessMaker Learning Management System System Requirements v1.0 Released July 2008 Copyright @ 2008 Pearson Education, Inc. or its affiliates. All rights reserved. SuccessMaker is a registered trademark,
More informationS.No QUESTIONS COMPETENCE LEVEL UNIT -1 PART A 1. Illustrate the evolutionary trend towards parallel distributed and cloud computing.
VALLIAMMAI ENGNIEERING COLLEGE SRM Nagar, Kattankulathur 603203. DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING Year & Semester : IV & VII Section : CSE -1& 2 Subject Code : CS6703 Subject Name : Grid
More informationTHE GLOBUS PROJECT. White Paper. GridFTP. Universal Data Transfer for the Grid
THE GLOBUS PROJECT White Paper GridFTP Universal Data Transfer for the Grid WHITE PAPER GridFTP Universal Data Transfer for the Grid September 5, 2000 Copyright 2000, The University of Chicago and The
More informationDynamic Load-Balanced Multicast for Data-Intensive Applications on Clouds 1
Dynamic Load-Balanced Multicast for Data-Intensive Applications on Clouds 1 Contents: Introduction Multicast on parallel distributed systems Multicast on P2P systems Multicast on clouds High performance
More informationPivot3 Acuity with Microsoft SQL Server Reference Architecture
Pivot3 Acuity with Microsoft SQL Server 2014 Reference Architecture How to Contact Pivot3 Pivot3, Inc. General Information: info@pivot3.com 221 West 6 th St., Suite 750 Sales: sales@pivot3.com Austin,
More informationA QoS-Aware Middleware for Fault Tolerant Web Services
A QoS-Aware Middleware for Fault Tolerant Web Services Zibin Zheng and Michael R. Lyu Department of Computer Science & Engineering The Chinese University of Hong Kong Hong Kong, China ISSRE 2008, Seattle,
More informationCluster Setup and Distributed File System
Cluster Setup and Distributed File System R&D Storage for the R&D Storage Group People Involved Gaetano Capasso - INFN-Naples Domenico Del Prete INFN-Naples Diacono Domenico INFN-Bari Donvito Giacinto
More informationSuccessMaker. System Requirements SuccessMaker v1.0
SuccessMaker System Requirements SuccessMaker v1.0 Contents Overview... 1 Stand-alone Configuration...2 Server/Network Configuration...2 District Configuration...3 Automatic Updates...3 System Requirements...
More informationSystematic Cooperation in P2P Grids
29th October 2008 Cyril Briquet Doctoral Dissertation in Computing Science Department of EE & CS (Montefiore Institute) University of Liège, Belgium Application class: Bags of Tasks Bag of Task = set of
More informationPeer-to-Peer Internet Applications: A Review
Peer-to-Peer Internet Applications: A Review Davide Quaglia 01/14/10 Introduction Key points Lookup task Outline Centralized (Napster) Query flooding (Gnutella) Distributed Hash Table (Chord) Simulation
More informationToward Scalable Monitoring on Large-Scale Storage for Software Defined Cyberinfrastructure
Toward Scalable Monitoring on Large-Scale Storage for Software Defined Cyberinfrastructure Arnab K. Paul, Ryan Chard, Kyle Chard, Steven Tuecke, Ali R. Butt, Ian Foster Virginia Tech, Argonne National
More informationDay 1 : August (Thursday) An overview of Globus Toolkit 2.4
An Overview of Grid Computing Workshop Day 1 : August 05 2004 (Thursday) An overview of Globus Toolkit 2.4 By CDAC Experts Contact :vcvrao@cdacindia.com; betatest@cdacindia.com URL : http://www.cs.umn.edu/~vcvrao
More informationAMGA metadata catalogue system
AMGA metadata catalogue system Hurng-Chun Lee ACGrid School, Hanoi, Vietnam www.eu-egee.org EGEE and glite are registered trademarks Outline AMGA overview AMGA Background and Motivation for AMGA Interface,
More information