Ioan Raicu. Distributed Systems Laboratory Computer Science Department University of Chicago
|
|
- Lambert Harmon
- 5 years ago
- Views:
Transcription
1 The Quest for Scalable Support of Data Intensive Applications in Distributed Systems Ioan Raicu Distributed Systems Laboratory Computer Science Department University of Chicago In Collaboration with: Ian Foster, University of Chicago and Argonne National Laboratory Alex Szalay, The Johns Hopkins University Yong Zhao, Microsoft Corporation Philip Little, Christopher Moretti, Amitabh Chaudhary, Douglas Thain, University of Notre Dame IEEE/ACM Supercomputing 28 Argonne National Laboratory Booth November 2 th, 28
2 Many-Core Growth Rates Increasing 9 attention to nm parallel chips Many plans for 65 cores with In-Order nm execution On-chip shared 45 memory nm Cores Far faster to access on-chip 32 memory than DRAM nm Interesting challenges in synchronization 22 (e.g. 64 locking) nm 64 2 Cores 4 8 Inexpensive 6 Cores Low-Power Cores Cores 32 Parallel Chips Amazing amounts of computing very cheap nm Cores 32 Cores Slower (or same) sequential speed! Cores Cores 6 nm nm 8 nm Pat Helland, Microsoft, The Irresistible Forces Meet the Movable Objects, November 9 th, 27 Slide 2
3 What will we do with + Exaflops and M+ cores?
4 Storage Resource Scalability Th hroughput (Mb/s) GPFS R LOCAL R GPFS R+W LOCAL R+W Number of Nodes IBM BlueGene/P GPFS vs. LOCAL Read Throughput node:.48gb/s vs..3gb/s 2.5x 6 nodes: 3.4Gb/s vs. 65Gb/s 48x Mb/s per CPU vs. 55Mb/s per CPU Read+Write Throughput: node:.2gb/s vs..39gb/s.95x 6 nodes:.gb/s vs. 62Gb/s 55x Metadata (mkdir / rm -rf) node: 5/sec vs. 99/sec.3x 6 nodes: 2/sec vs. 384/sec 56x 6K CPU cores GPFS 8GB/s I/O rates (6 servers) Experiments on 6K CPU BG/P achieved.3mb/s per CPU core Experiments on 5.7K CPU SiCortex achieved.6mb/s per CPU core /3/28 The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 4
5 Programming Model Issues Multicore processors Massive task parallelism Massive data parallelism Integrating black box applications Complex task dependencies (task graphs) Failure, and other execution management issues Dynamic task graphs Documenting provenance of data products Data management: input, intermediate, output Dynamic data access involving large amounts of data The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 5
6 Programming Model Issues Multicore processors Massive task parallelism Massive data parallelism Integrating black box applications Complex task dependencies (task graphs) Failure, and other execution management issues Dynamic task graphs Documenting provenance of data products Data management: input, intermediate, output Dynamic data access involving large amounts of data The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 6
7 Problem Types Input Hi Data Size Med Data Analysis, Mining Big Data and Many Tasks Heroic MPI Tasks Many Loosely Coupled Apps Low K M Number of Tasks The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 7
8 An Incomplete and Simplistic View of Programming Models and Tools The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 8
9 MTC: Many Task Computing Loosely coupled applications High-performance computations comprising of multiple distinct activities, coupled via file system operations or message passing Emphasis on using many resources over short time periods Tasks can be: small or large, independent and dependent, uniprocessor or multiprocessor, compute-intensive or data-intensive, static or dynamic, homogeneous or heterogeneous, loosely or tightly coupled, large number of tasks, large quantity of computing, and large volumes of data The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 9
10 Motivating Example: AstroPortal Stacking Service Purpose On-demand stacks of random locations within ~TB dataset Challenge Processing Costs: O(ms) per object Data Intensive: 4MB:sec Rapid access to -K random files Time-varying load AP The Quest for Scalable Support of Data Intensive Applications in Distributed Systems = Sloan Data
11 Hypothesis Significant performance improvements can be obtained in the analysis of large dataset by leveraging information about data analysis workloads rather than individual data analysis tasks. Important concepts related to the hypothesis Workload: a complex query (or set of queries) decomposable into simpler tasks to answer broader analysis questions Data locality is crucial to the efficient use of large scale distributed systems for scientific and data-intensive applications Allocate computational and caching storage resources, co-scheduled to optimize workload performance The Quest for Scalable Support of Data Intensive Applications in Distributed Systems
12 Abstract Model AMDASK: An Abstract Model for DAta-centric task farms Task Farm: A common parallel pattern that drives independent computational tasks Models the efficiency of data analysis workloads for the MTC class of applications Captures the following data diffusion properties Resources are acquired in response to demand Data and applications diffuse from archival storage to new resources Resource caching allows faster responses to subsequent requests Resources are released when demand drops Considers both data and computations to optimize performance The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 2
13 AMDASK: Base Definitions Data Stores: Persistent & Transient Store capacity, load, ideal bandwidth, available bandwidth Data Objects: Data object size, data object s storage location(s), copy time Transient resources: compute speed, resource state Task: application, input/output data The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 3
14 AMDASK: Execution Model Concepts Dispatch Policy next-available, first-available, max-compute-util, max-cache-hit Caching Policy random, FIFO, LRU, LFU Replay policy Data Fetch Policy Just-in-Time, Spatial Locality Resource Acquisition Policy one-at-a-time, additive, exponential, all-at-once, optimal Resource Release Policy distributed, centralized The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 4
15 AMDASK: Performance Efficiency Model B: Average Task Execution Time: K: Stream of tasks Β = µ ( κ ) µ(k): Task k execution time Κ k Κ Y: Average Task Execution Time with Overheads: ο(k): Dispatch overhead ς(δ,τ): Time to get data V: Workload Execution Time: A: Arrival rate of tasks T: Transient Resources Y = Κ W: Workload Execution Time with Overheads V [ µ ( κ ) + o ( κ )], δ φ ( τ ), δ Ω Κ [ µ ( κ) + o( κ) + ζ ( δ, τ )], B = max, * Κ Τ Α Υ W = max, * Κ Τ Α δ φ( τ ), δ Ω The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 5 κ Κ κ Κ
16 AMDASK: Performance Efficiency Model Efficiency Ε = Speedup V W E = max B Y Y, T Τ, Α* Y, A Y T > A S = E* T Optimizing Efficiency Easy to maximize either efficiency or speedup independently Harder to maximize both at the same time Find the smallest number of transient resources T while maximizing The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 6 speedup*efficiency
17 Model Validation Stacking service (large scale astronomy application) 92 experiments 558K files Compressed: 2MB each.tb Un-compressed: 6MB each 33TB 3.3TB Model Error Model Error 3% 2% % GPFS (GZ) Data Diffusion (FIT) - Locality GPFS (FIT) Data Diffusion (GZ) - Locality Data Diffusion (FIT) - Locality.38 Data Diffusion (FIT) - Locality 3 Data Diffusion (GZ) - Locality.38 Data Diffusion (GZ) - Locality 3 % Number of CPUs 3% 2% % GPFS (GZ) GPFS (FIT) Data Diffusion (FIT) Data Diffusion (GZ) % Data Locality 7
18 Falkon: a Fast and Light-weight task execution framework Goal: enable the rapid and efficient execution of many independent jobs on large compute clusters Combines three components: a streamlined task dispatcher resource provisioning i i through h multi-level l l scheduling techniques data diffusion and data-aware scheduling to leverage the co-located computational and storage resources Integration into Swift to leverage many applications Applications cover many domains: astronomy, astro-physics, medicine, chemistry, economics, climate modeling, etc The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 8
19 Falkon: a Fast and Light-weight task execution framework Goal: enable the rapid and efficient execution of many independent jobs on large compute clusters Combines three components: a streamlined task dispatcher resource provisioning i i through h multi-level l l scheduling techniques data diffusion and data-aware scheduling to leverage the co-located computational and storage resources Integration into Swift to leverage many applications Applications cover many domains: astronomy, astro-physics, medicine, chemistry, economics, climate modeling, etc The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 9
20 Falkon Overview The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 2
21 Data Diffusion Resource acquired in response to demand Data and applications diffuse from archival storage to newly acquired resources Resource caching allows faster responses to subsequent requests Cache Eviction Strategies: RANDOM, FIFO, LRU, LFU Resources are released when demand drops Task Dispatcher Data-Aware Scheduler Idle Resources Persistent Storage Shared File System text Provisioned Resources The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 2
22 Data Diffusion Considers both data and computations to optimize performance Supports data-aware scheduling Can optimize compute utilization, cache hit performance, or a mixture of the two Decrease dependency of a shared file system Theoretical linear scalability with compute resources Significantly increases meta-data creation and/or modification performance Central for data-centric task farm realization The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 22
23 Scheduling Policies first-available: simple load balancing max-cache-hit maximize cache hits max-compute-util maximize processor utilization good-cache-compute maximize both cache hit and processor utilization at the same time The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 23
24 Data-Aware Scheduler Profiling Task (ms) CPU Time per Task Submit Notification for Task Availability Task Dispatch (data-aware scheduler) Task Results (data-aware scheduler) Notification for Task Results WS Communication Throughput (tasks/sec) Throughput (t tasks/sec) firstavailable without I/O firstavailable with I/O max- max-cachecompute-utihit goodcachecompute The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 24
25 AstroPortal Stacking Service Purpose On-demand stacks of random locations within ~TB dataset Challenge Rapid access to -K random files Time-varying load Sample Workloads Locality Number of Objects Number of Files S 4 Web page or Web Service = Sloan Data The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 27
26 Purpose AstroPortal Stacking Service 45 On-demand stacks of random locations 4 within ~TB dataset 35 Challenge 3 Rapid access to -K random files Time (ms) 25 Time-varying load 2 Sample Workloads 5 Locality Number of Objects Number of Files open radec2xy readhdu+gettile+curl+convertarray calibration+interpolation+dostacking writestacking The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 28 S 4 Web page or Web Service GPFS GZ LOCAL GZ GPFS FIT LOCAL FIT Filesystem and Image Format = Sloan Data
27 Time (ms) per stack per CPU AstroPortal Stacking Service with Data Diffusion Low data locality Similar (but better) performance to GPFS Data Diffusion (GZ) Data Diffusion i (FIT) GPFS (GZ) GPFS (FIT) Time (ms s) per stack per CPU Data Diffusion (GZ) Data Diffusion (FIT) GPFS (GZ) GPFS (FIT) Number of CPUs High data locality Near perfect scalability Number of CPUs The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 29
28 AstroPortal Stacking Service with Data Diffusion Aggregate throughput: 39Gb/s X higher than GPFS Reduced load on GPFS Time (ms) per stack per CPU Throughput (Gb/s) Gb/s 2 / of the original load 5 Data Diffusion (GZ) Data Diffusion (FIT) GPFS (GZ) GPFS (FIT) Ideal Locality Aggregate 5 Data Diffusion Throughput Local Data Diffusion Throughput Cache-to-Cache Data Diffusion Throughput GPFS GPFS Throughput (FIT) GPFS Throughput (GZ) Locality Big performance gains as locality increases 3
29 Monotonically Increasing Workload 25K tasks MB reads ms compute Vary arrival rate: Min: task/sec Increment function: CEILING(*.3) Max: tasks/sec 28 processors Ideal case: 45 sec 8Gb/s peak throughput Arrival Rate per second Arrival Rate per sec Tasks completed Ideal Throughput Mb/s Time (sec) Tasks Comp pleted Ideal Throughpu ut (Mb/s) The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 3
30 Data Diffusion: First-available (GPFS) GPFS vs. ideal: 5 sec vs. 45 sec. Number of No odes Throughput (G Gb/s) Queue Length (xk) Time (sec) The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 32 Ideal Throughput (Gb/s) Throughput (Gb/s) Wait Queue Length Number of Nodes
31 Data Diffusion: Max-compute-util & max-cache-hit Max-compute-util Max-cache-hit Queue Length (xk) Cache Hit/Miss % Number of Nodes Aggregate Throughput (G Gb/s) Throughput per Node (M B/s) Queue Length (xk) CPU Utilization % Cache Hit/Miss % Time (sec) Cache Hit Local % Cache Hit Global % Cache Miss % Ideal Throughput (Gb/s) Throughput (Gb/s) Wait Queue Length Number of Nodes Time (sec) Cache Hit Local % Cache Hit Global % Cache Miss % Ideal Throughput (Gb/s) Throughput (Gb/s) Wait Queue Length Number of Nodes CPU Utilization The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 33 Number of Nodes Throughput (Gb/s)
32 Data Diffusion: Good-cache-compute GB.5GB Number of Nodes Aggregate Throughput (Gb/s) Throughput per Node (MB/s) Queue Length (xk) Cache Hit/Miss %.. Number of Nodes Aggregate Throughput (Gb/s) Throughput per Node (MB/s) Queue Length (xk) Cache Hit/Miss % Time (sec) Cache Hit Local % Cache Hit Global % Cache Miss % Ideal Throughput (Gb/s) Throughput (Gb/s) Wait Queue Length Number of Nodes Time (sec) Cache Hit Local % Cache Hit Global % Cache Miss % Ideal Throughput (Gb/s) Throughput (Gb/s) Wait Queue Length Number of Nodes GB 4GB Time (sec) Number of Nodes Aggregate Throughput (Gb/s) Throughput per Node (MB/s) Queue Length (xk) Cache Hit/Miss %.. Number of Nodes Aggregate Throughput (Gb/s) Throughput per Node (MB/s) Queue Length (xk) Cache Hit/Miss % Cache Hit Local % Cache Hit Global % Cache Miss % Ideal Throughput (Gb/s) Throughput (Gb/s) Wait Queue Length Number of Nodes Time (sec) Cache Hit Local % Cache Hit Global % Cache Miss % Ideal Throughput (Gb/s) Throughput (Gb/s) Wait Queue Length Number of Nodes
33 Data Diffusion: Good-cache-compute Data Diffusion vs. ideal: 436 sec vs 45 sec Number of No odes Aggregate Through hput (Gb/s) Throughput per No ode (MB/s) Queue Length (xk) Cache Hit/Mis ss % Time (sec) Cache Hit Local % Cache Hit Global % Cache Miss % 35 Ideal Throughput (Gb/s) Throughput (Gb/s) Wait Queue Length Number of Nodes
34 Data Diffusion: Throughput and Response Time Throughput (Gb/s) Local Worker Caches (Gb/s) Remote Worker Caches (Gb/s) GPFS Throughput (Gb/s) Throughput: Average: 4Gb/s vs 4Gb/s Peak: Gb/s vs. 6Gb/s. Ideal firstavailable goodcachecompute, GB goodcachecompute,.5gb goodcachecompute, 2GB goodcachecompute, 4GB maxcache-hit, 4GB maxcomputeutil, 4GB Response Time 3 sec vs 569 sec 56X Average Response Time (sec) 569 firstavailable 84 goodcachecompute, GB 4 goodcachecompute,.5gb 3.4 goodcachecompute, 2GB 3. goodcachecompute, 4GB max-cachehit, 4GB 36 maxcomputeutil, 4GB
35 Data Diffusion: Performance Index, Slowdown, and Speedup Performance Index: 34X higher Speedup 3.5X faster than GPFS Slowdown first-available good-cache-compute, GB good-cache-compute,.5gb good-cache-compute, 2GB good-cache-compute, 4GB max-cache-hit, 4GB max-compute-util, 4GB Arrival Rate per Second Performance Index Performance Index Speedup (compared to first-available) firstavailable goodcachecompute, GB goodcachecompute,.5gb goodcachecompute, 2GB goodcachecompute, 4GB good- maxcachecache-hitcompute, 4GB 4GB, SRP Slowdown: 8X slowdown for GPFS Near ideal X slowdown for large enough caches maxcomputeutil, 4GB Speedu up (compared to LAN GPFS)
36 Sin-Wave Workload 2M tasks MB reads ms compute Vary arrival rate: Min: task/sec Arrival rate function: Max: tasks/sec 2 processors Ideal case: 655 sec 8Gb/s peak throughput A = (sin( sqrt( time+.)* ) + )*( time+.)*5.75 Arrival Rate (per sec) Arrival Rate 9 Number of Tasks Time (sec) Number of Tasks Complete ed
37 Sin-Wave Workload GPFS 5.7 hrs, ~8Gb/s, 38 CPU hrs DF+SRP.8 hrs, ~25Gb/s, 36 CPU hrs DF+DRP.86 hrs, ~24Gb/s, 253 CPU hrs Number of Nodes Throughput (Gb/s) Queue Length (xk) % 9% 8% 7% 6% 5% 4% 3% 2% % % Cache Hit/Miss Number of Nodes Throughput (Gb/s) Queue Length (xk) % 9% 8% 7% 6% 5% 4% 3% 2% % % Cache Hit/Miss % Time (sec) Cache Miss % Cache Hit Global % Cache Hit Local % Demand (Gb/s) Throughput (Gb/s) Wait Queue Length Number of Nodes Time (sec) Cache Miss % Cache Hit Global % Cache Hit 39 Local % Demand (Gb/s) Throughput (Gb/s) Wait Queue Length Number of Nodes
38 Sin-Wave Workload Number of Nodes Through hput (Gb/s) Queue Le ength (xk) % 9% 8% 7% 6% 5% 4% 3% 2% % % Cache Hit/Miss % Time (sec) Cache Miss % Cache Hit Global % Cache Hit Local % 4 Demand (Gb/s) Throughput (Gb/s) Wait Queue Length Number of Nodes
39 5x5 25K tasks 24MB reads ms compute 2 CPUs x M tasks 24MB reads 4sec compute 496 CPUs Ideal case: 655 sec 8Gb/s peak throughput All-Pairs Workload All-Pairs( set A, set B, function F ) returns matrix M: Compare all elements of set A to all elements of set B via function F, yielding matrix M, such that M[i,j] = F(A[i],B[j]) foreach $i in A 2 foreach $j in B 3 submit_job F $i $j 4 end 5 end The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 4
40 Throughput (Gb/s) All-Pairs Workload 5x5 on 2 CPUs Efficiency: 75% Cache Miss % Cache Hit Global % Cache Hit Local % Throughput (Data Diffusion) Maximum Throughput (GPFS) Maximum Throughput (Local Disk) % 9% 8% 7% 6% 5% 4% 3% 2% % % Miss Cache Hit/ Time (sec) 42
41 All-Pairs Workload x on 4K emulated CPUs Throughput (Gb/s) Efficiency: 86% Cache Miss % Cache Hit Global % Cache Hit Local % Throughput (Data Diffusion) Maximum Throughput (GPFS) Maximum Throughput (Local Memory) Time (sec) % 9% 8% 7% 6% 5% 4% 3% 2% % % Cache Hit/ /Miss 43
42 All-Pairs Workload Data Diffusion vs. Active Storage Push vs. Pull Active Storage: % 9% 8% 7% 6% 5% Pushes workload 4% 3% working set to all nodes 2% Static spanning tree Data Diffusion Pulls task working set Incremental spanning forest Efficiency % % Experiment 5x5 2 CPUs sec 5x5 2 CPUs. sec x 496 CPUs 4 sec 5x5 2 CPUs sec Approach Best Case (active storage) Falkon (data diffusion) Best Case (active storage) Falkon (data diffusion) Best Case (active storage) Falkon (data diffusion) Best Case (active storage) Falkon (data diffusion) Falkon (GPFS) 5x5 x 2 CPUs 496 CPUs. sec 4 sec Shared Local Network File Disk/Memory (node-to-node) System (GB) (GB) (GB)
43 All-Pairs Workload Data Diffusion vs. Active Storage Best to use active storage if Slow data source Workload working set fits on local node storage Good aggregate network bandwidth Best to use data diffusion if Medium to fast data source Task working set << workload working set Task working set fits on local node storage Good aggregate network bandwidth If task working set does not fit on local node storage Use parallel file system (i.e. GPFS, Lustre, PVFS, etc)
44 Limitations of Data Diffusion Needs Java.4+ Needs IP connectivity between hosts Needs local storage (disk, memory, etc) Per task workings set must fit in local storage Task definition must include input/output files metadata Data access patterns: write once, read many The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 46
45 Related Work: Data Management [Beynon]: DataCutter [Ranganathan3]: Simulations [Ghemawat3,Dean4,Chang6]: BigTable, GFS, MapReduce [Liu4]: GridDB [Chervenak4,Chervenak6]: RLS (Replica Location Service), DRS (Data Replication Service) [Tatebe4,Xiaohui5]: GFarm [Branco4,Adams6]: DIAL/ATLAS [Kosar6]: Stork [Thain8]: Chirp/Parrot Conclusion: None focused on the co-location of storage and generic black box computations with data-aware scheduling while operating in a dynamic environment The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 47
46 Scaling from K to K CPUs without Data Diffusion At K CPUs: Server to manage all K CPUs Use shared file system extensively Invoke application from shared file system Read/write data from/to shared file system At K CPUs: N Servers to manage K CPUs (:256 ratio) Don t trust the application I/O access patterns to behave optimally Copy applications and input data to RAM Read input data from RAM, compute, and write results to RAM Archive all results in a single file in RAM Copy result file from RAM back to GPFS Great potential for improvements Could leverage the Torus network for high aggregate bandwidth Collective I/O (CIO) Primitives Roadblocks: machine global IP connectivity, Java support, and time 48
47 Mythbusting Embarrassingly Happily parallel apps are trivial to run Logistical problems can be tremendous Loosely coupled apps do not require supercomputers Total computational requirements can be enormous Individual tasks may be tightly coupled Workloads frequently involve large amounts of I/O Make use of idle resources from supercomputers via backfilling Costs to run supercomputers per FLOP is among the best BG/P:.35 gigaflops/watt (higher is better) SiCortex:.32 gigaflops/watt BG/L:.23 gigaflops/watt x86-based HPC systems: an order of magnitude lower Loosely coupled apps do not require specialized system software Shared file systems are good for all applications They don t scale proportionally with the compute resources The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 49 Data intensive applications don t perform and scale well
48 Conclusions & Contributions Defined an abstract model for performance efficiency of data analysis workloads using data-centric task farms Provide a reference implementation (Falkon) Use a streamlined dispatcher to increase task throughput by several orders of magnitude over traditional LRMs Use multi-level l l scheduling to reduce perceived wait queue time for tasks to execute on remote resources Address data diffusion through co-scheduling of storage and computational resources to improve performance and scalability Provide the benefits of dedicated hardware without the associated high cost Show effectiveness of data diffusion: real large-scale astronomy application and a variety of synthetic workloads The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 5
49 More Information More information: Related Projects: Falkon: AstroPortal: Swift: Funding: NASA: Ames Research Center, Graduate Student Research Program (GSRP) DOE: Mathematical, Information, and Computational Sciences Division subprogram of the Office of Advanced Scientific Computing Research, Office of Science, U.S. Dept. of Energy NSF: TeraGrid The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 5
50 In conjunction with IEEE/ACM SuperComputing 28 Location: Austin Texas Date: November 7 th, 28 The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 52
51 In conjunction with IEEE/ACM SuperComputing 28 Location: Austin Texas Date: November 8 th, 28 The Quest for Scalable Support of Data Intensive Applications in Distributed Systems 53
NFS, GPFS, PVFS, Lustre Batch-scheduled systems: Clusters, Grids, and Supercomputers Programming paradigm: HPC, MTC, and HTC
Segregated storage and compute NFS, GPFS, PVFS, Lustre Batch-scheduled systems: Clusters, Grids, and Supercomputers Programming paradigm: HPC, MTC, and HTC Co-located storage and compute HDFS, GFS Data
More informationNFS, GPFS, PVFS, Lustre Batch-scheduled systems: Clusters, Grids, and Supercomputers Programming paradigm: HPC, MTC, and HTC
Segregated storage and compute NFS, GPFS, PVFS, Lustre Batch-scheduled systems: Clusters, Grids, and Supercomputers Programming paradigm: HPC, MTC, and HTC Co-located storage and compute HDFS, GFS Data
More informationIoan Raicu Distributed Systems Laboratory Computer Science Department University of Chicago
Running 1 Million Jobs in 10 Minutes via the Falkon Fast and Light-weight Ioan Raicu Distributed Systems Laboratory Computer Science Department University of Chicago In Collaboration with: Ian Foster,
More informationIoan Raicu Distributed Systems Laboratory Computer Science Department University of Chicago
Falkon, a Fast and Light-weight task execution framework for Clusters, Grids, and Supercomputers Ioan Raicu Distributed Systems Laboratory Computer Science Department University of Chicago In Collaboration
More informationManaging and Executing Loosely-Coupled Large-Scale Applications on Clusters, Grids, and Supercomputers
Managing and Executing Loosely-Coupled Large-Scale Applications on Clusters, Grids, and Supercomputers Ioan Raicu Distributed Systems Laboratory Computer Science Department University of Chicago Collaborators:
More informationA Data Diffusion Approach to Large Scale Scientific Exploration
A Data Diffusion Approach to Large Scale Scientific Exploration Ioan Raicu Distributed Systems Laboratory Computer Science Department University of Chicago Joint work with: Yong Zhao: Microsoft Ian Foster:
More informationSynonymous with supercomputing Tightly-coupled applications Implemented using Message Passing Interface (MPI) Large of amounts of computing for short
Synonymous with supercomputing Tightly-coupled applications Implemented using Message Passing Interface (MPI) Large of amounts of computing for short periods of time Usually requires low latency interconnects
More informationTypically applied in clusters and grids Loosely-coupled applications with sequential jobs Large amounts of computing for long periods of times
Typically applied in clusters and grids Loosely-coupled applications with sequential jobs Large amounts of computing for long periods of times Measured in operations per month or years 2 Bridge the gap
More informationData Diffusion: Dynamic Resource Provision and Data-Aware Scheduling for Data-Intensive Applications
Data Diffusion: Dynamic Resource Provision and Data-Aware Scheduling for Data-Intensive Applications Ioan Raicu, Yong Zhao 2, Ian Foster,3,4, Alex Szalay 5 iraicu@cs.uchicago.edu, yozha@microsoft.com,
More informationStorage and Compute Resource Management via DYRE, 3DcacheGrid, and CompuStore
Storage and Compute Resource Management via DYRE, 3DcacheGrid, and CompuStore Ioan Raicu Distributed Systems Laboratory Computer Science Department University of Chicago DSL Seminar November st, 006 Analysis
More informationHarnessing Grid Resources to Enable the Dynamic Analysis of Large Astronomy Datasets
Page 1 of 5 1 Year 1 Proposal Harnessing Grid Resources to Enable the Dynamic Analysis of Large Astronomy Datasets Year 1 Progress Report & Year 2 Proposal In order to setup the context for this progress
More informationThe Quest for Scalable Support of Data-Intensive Workloads in Distributed Systems
The Quest for Scalable Support of Data-Intensive Workloads in Distributed Systems Ioan Raicu, 1 Ian T. Foster, 1,2,3 Yong Zhao 4 Philip Little, 5 Christopher M. Moretti, 5 Amitabh Chaudhary, 5 Douglas
More informationOverview Past Work Future Work. Motivation Proposal. Work-in-Progress
Overview Past Work Future Work Motivation Proposal Work-in-Progress 2 HPC: High-Performance Computing Synonymous with supercomputing Tightly-coupled applications Implemented using Message Passing Interface
More informationAccelerating Large Scale Scientific Exploration through Data Diffusion
Accelerating Large Scale Scientific Exploration through Data Diffusion Ioan Raicu *, Yong Zhao *, Ian Foster #*+, Alex Szalay - {iraicu,yongzh }@cs.uchicago.edu, foster@mcs.anl.gov, szalay@jhu.edu * Department
More informationMOHA: Many-Task Computing Framework on Hadoop
Apache: Big Data North America 2017 @ Miami MOHA: Many-Task Computing Framework on Hadoop Soonwook Hwang Korea Institute of Science and Technology Information May 18, 2017 Table of Contents Introduction
More informationIan Foster, An Overview of Distributed Systems
The advent of computation can be compared, in terms of the breadth and depth of its impact on research and scholarship, to the invention of writing and the development of modern mathematics. Ian Foster,
More informationArguably one of the most fundamental discipline that touches all other disciplines and people
The scientific and mathematical approach in information technology and computing Started in the 1960s from Mathematics or Electrical Engineering Today: Arguably one of the most fundamental discipline that
More informationKeywords: many-task computing; MTC; high-throughput computing; resource management; Falkon; Swift
Editorial Manager(tm) for Cluster Computing Manuscript Draft Manuscript Number: Title: Middleware Support for Many-Task Computing Article Type: HPDC Special Issue Section/Category: Keywords: many-task
More informationThe Fusion Distributed File System
Slide 1 / 44 The Fusion Distributed File System Dongfang Zhao February 2015 Slide 2 / 44 Outline Introduction FusionFS System Architecture Metadata Management Data Movement Implementation Details Unique
More informationStorage and Compute Resource Management via DYRE, 3DcacheGrid, and CompuStore Ioan Raicu, Ian Foster
Storage and Compute Resource Management via DYRE, 3DcacheGrid, and CompuStore Ioan Raicu, Ian Foster. Overview Both the industry and academia have an increase demand for good policies and mechanisms to
More informationRevealing Applications Access Pattern in Collective I/O for Cache Management
Revealing Applications Access Pattern in for Yin Lu 1, Yong Chen 1, Rob Latham 2 and Yu Zhuang 1 Presented by Philip Roth 3 1 Department of Computer Science Texas Tech University 2 Mathematics and Computer
More informationIoan Raicu. Everyone else. More information at: Background? What do you want to get out of this course?
Ioan Raicu More information at: http://www.cs.iit.edu/~iraicu/ Everyone else Background? What do you want to get out of this course? 2 Data Intensive Computing is critical to advancing modern science Applies
More informationCase Studies in Storage Access by Loosely Coupled Petascale Applications
Case Studies in Storage Access by Loosely Coupled Petascale Applications Justin M Wozniak and Michael Wilde Petascale Data Storage Workshop at SC 09 Portland, Oregon November 15, 2009 Outline Scripted
More informationTowards Data Intensive Many-Task Computing
Towards Data Intensive Many-Task Computing Ioan Raicu 1, Ian Foster 1,2,3, Yong Zhao 4, Alex Szalay 5 Philip Little 6, Christopher M. Moretti 6, Amitabh Chaudhary 6, Douglas Thain 6 iraicu@cs.uchicago.edu,
More informationIntroduction & Motivation Problem Statement Proposed Work Evaluation Conclusions Future Work
Introduction & Motivation Problem Statement Proposed Work Evaluation Conclusions Future Work Introduction & Motivation Problem Statement Proposed Work Evaluation Conclusions Future Work Today (2014):
More informationWrite a technical report Present your results Write a workshop/conference paper (optional) Could be a real system, simulation and/or theoretical
Identify a problem Review approaches to the problem Propose a novel approach to the problem Define, design, prototype an implementation to evaluate your approach Could be a real system, simulation and/or
More informationExtreme-scale scripting: Opportunities for large taskparallel applications on petascale computers
Extreme-scale scripting: Opportunities for large taskparallel applications on petascale computers Michael Wilde, Ioan Raicu, Allan Espinosa, Zhao Zhang, Ben Clifford, Mihael Hategan, Kamil Iskra, Pete
More informationIntroduction to Grid Computing
Milestone 2 Include the names of the papers You only have a page be selective about what you include Be specific; summarize the authors contributions, not just what the paper is about. You might be able
More informationIME (Infinite Memory Engine) Extreme Application Acceleration & Highly Efficient I/O Provisioning
IME (Infinite Memory Engine) Extreme Application Acceleration & Highly Efficient I/O Provisioning September 22 nd 2015 Tommaso Cecchi 2 What is IME? This breakthrough, software defined storage application
More informationChapter 4:- Introduction to Grid and its Evolution. Prepared By:- NITIN PANDYA Assistant Professor SVBIT.
Chapter 4:- Introduction to Grid and its Evolution Prepared By:- Assistant Professor SVBIT. Overview Background: What is the Grid? Related technologies Grid applications Communities Grid Tools Case Studies
More informationAssistant Professor at Illinois Institute of Technology (CS) Director of the Data-Intensive Distributed Systems Laboratory (DataSys)
Current position: Assistant Professor at Illinois Institute of Technology (CS) Director of the Data-Intensive Distributed Systems Laboratory (DataSys) Guest Research Faculty, Argonne National Laboratory
More informationToday (2010): Multicore Computing 80. Near future (~2018): Manycore Computing Number of Cores Processing
Number of Cores Manufacturing Process 300 250 200 150 100 50 0 2004 2006 2008 2010 2012 2014 2016 2018 100 Today (2010): Multicore Computing 80 1~12 cores commodity architectures 70 60 80 cores proprietary
More informationAssistant Professor at Illinois Institute of Technology (CS) Director of the Data-Intensive Distributed Systems Laboratory (DataSys)
Current position: Assistant Professor at Illinois Institute of Technology (CS) Director of the Data-Intensive Distributed Systems Laboratory (DataSys) Guest Research Faculty, Argonne National Laboratory
More informationCharacterizing Storage Resources Performance in Accessing the SDSS Dataset Ioan Raicu Date:
Characterizing Storage Resources Performance in Accessing the SDSS Dataset Ioan Raicu Date: 8-17-5 Table of Contents Table of Contents...1 Table of Figures...1 1 Overview...4 2 Experiment Description...4
More informationData Management in Parallel Scripting
Data Management in Parallel Scripting Zhao Zhang 11/11/2012 Problem Statement Definition: MTC applications are those applications in which existing sequential or parallel programs are linked by files output
More informationLecture 2: Memory Systems
Lecture 2: Memory Systems Basic components Memory hierarchy Cache memory Virtual Memory Zebo Peng, IDA, LiTH Many Different Technologies Zebo Peng, IDA, LiTH 2 Internal and External Memories CPU Date transfer
More informationMATE-EC2: A Middleware for Processing Data with Amazon Web Services
MATE-EC2: A Middleware for Processing Data with Amazon Web Services Tekin Bicer David Chiu* and Gagan Agrawal Department of Compute Science and Engineering Ohio State University * School of Engineering
More informationMULTIPROCESSORS AND THREAD-LEVEL. B649 Parallel Architectures and Programming
MULTIPROCESSORS AND THREAD-LEVEL PARALLELISM B649 Parallel Architectures and Programming Motivation behind Multiprocessors Limitations of ILP (as already discussed) Growing interest in servers and server-performance
More informationMULTIPROCESSORS AND THREAD-LEVEL PARALLELISM. B649 Parallel Architectures and Programming
MULTIPROCESSORS AND THREAD-LEVEL PARALLELISM B649 Parallel Architectures and Programming Motivation behind Multiprocessors Limitations of ILP (as already discussed) Growing interest in servers and server-performance
More informationLecture 9: MIMD Architectures
Lecture 9: MIMD Architectures Introduction and classification Symmetric multiprocessors NUMA architecture Clusters Zebo Peng, IDA, LiTH 1 Introduction MIMD: a set of general purpose processors is connected
More informationDisk Cache-Aware Task Scheduling
Disk ache-aware Task Scheduling For Data-Intensive and Many-Task Workflow Masahiro Tanaka and Osamu Tatebe University of Tsukuba, JST/REST Japan Science and Technology Agency IEEE luster 2014 2014-09-24
More informationGFS: The Google File System
GFS: The Google File System Brad Karp UCL Computer Science CS GZ03 / M030 24 th October 2014 Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one
More informationEffect of memory latency
CACHE AWARENESS Effect of memory latency Consider a processor operating at 1 GHz (1 ns clock) connected to a DRAM with a latency of 100 ns. Assume that the processor has two ALU units and it is capable
More informationLecture 9: MIMD Architectures
Lecture 9: MIMD Architectures Introduction and classification Symmetric multiprocessors NUMA architecture Clusters Zebo Peng, IDA, LiTH 1 Introduction A set of general purpose processors is connected together.
More informationExtreme scripting and other adventures in data-intensive computing
Extreme scripting and other adventures in data-intensive computing Ian Foster Allan Espinosa, Ioan Raicu, Mike Wilde, Zhao Zhang Computation Institute Argonne National Lab & University of Chicago How data
More informationCrossing the Chasm: Sneaking a parallel file system into Hadoop
Crossing the Chasm: Sneaking a parallel file system into Hadoop Wittawat Tantisiriroj Swapnil Patil, Garth Gibson PARALLEL DATA LABORATORY Carnegie Mellon University In this work Compare and contrast large
More informationThe Google File System
The Google File System Sanjay Ghemawat, Howard Gobioff and Shun Tak Leung Google* Shivesh Kumar Sharma fl4164@wayne.edu Fall 2015 004395771 Overview Google file system is a scalable distributed file system
More informationIan Foster, CS554: Data-Intensive Computing
The advent of computation can be compared, in terms of the breadth and depth of its impact on research and scholarship, to the invention of writing and the development of modern mathematics. Ian Foster,
More informationCloud Programming. Programming Environment Oct 29, 2015 Osamu Tatebe
Cloud Programming Programming Environment Oct 29, 2015 Osamu Tatebe Cloud Computing Only required amount of CPU and storage can be used anytime from anywhere via network Availability, throughput, reliability
More informationSupport for resource sharing Openness Concurrency Scalability Fault tolerance (reliability) Transparence. CS550: Advanced Operating Systems 2
Support for resource sharing Openness Concurrency Scalability Fault tolerance (reliability) Transparence CS550: Advanced Operating Systems 2 Share hardware,software,data and information Hardware devices
More informationHigh Performance Computing on GPUs using NVIDIA CUDA
High Performance Computing on GPUs using NVIDIA CUDA Slides include some material from GPGPU tutorial at SIGGRAPH2007: http://www.gpgpu.org/s2007 1 Outline Motivation Stream programming Simplified HW and
More informationThe Google File System. Alexandru Costan
1 The Google File System Alexandru Costan Actions on Big Data 2 Storage Analysis Acquisition Handling the data stream Data structured unstructured semi-structured Results Transactions Outline File systems
More informationLecture 9: MIMD Architecture
Lecture 9: MIMD Architecture Introduction and classification Symmetric multiprocessors NUMA architecture Cluster machines Zebo Peng, IDA, LiTH 1 Introduction MIMD: a set of general purpose processors is
More informationDistributed NoSQL Storage for Extreme-scale System Services
Distributed NoSQL Storage for Extreme-scale System Services Short Bio 6 th year PhD student from DataSys Lab, CS Department, Illinois Institute oftechnology,chicago Academic advisor: Dr. Ioan Raicu Research
More informationGFS: The Google File System. Dr. Yingwu Zhu
GFS: The Google File System Dr. Yingwu Zhu Motivating Application: Google Crawl the whole web Store it all on one big disk Process users searches on one big CPU More storage, CPU required than one PC can
More informationAdvances of parallel computing. Kirill Bogachev May 2016
Advances of parallel computing Kirill Bogachev May 2016 Demands in Simulations Field development relies more and more on static and dynamic modeling of the reservoirs that has come a long way from being
More informationMemory. Objectives. Introduction. 6.2 Types of Memory
Memory Objectives Master the concepts of hierarchical memory organization. Understand how each level of memory contributes to system performance, and how the performance is measured. Master the concepts
More informationTransparent Throughput Elas0city for IaaS Cloud Storage Using Guest- Side Block- Level Caching
Transparent Throughput Elas0city for IaaS Cloud Storage Using Guest- Side Block- Level Caching Bogdan Nicolae (IBM Research, Ireland) Pierre Riteau (University of Chicago, USA) Kate Keahey (Argonne National
More informationCorrelation based File Prefetching Approach for Hadoop
IEEE 2nd International Conference on Cloud Computing Technology and Science Correlation based File Prefetching Approach for Hadoop Bo Dong 1, Xiao Zhong 2, Qinghua Zheng 1, Lirong Jian 2, Jian Liu 1, Jie
More informationScaling Distributed Machine Learning with the Parameter Server
Scaling Distributed Machine Learning with the Parameter Server Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J. Shekita, and Bor-Yiing Su Presented
More informationCSE 591/392: GPU Programming. Introduction. Klaus Mueller. Computer Science Department Stony Brook University
CSE 591/392: GPU Programming Introduction Klaus Mueller Computer Science Department Stony Brook University First: A Big Word of Thanks! to the millions of computer game enthusiasts worldwide Who demand
More informationDistributed Filesystem
Distributed Filesystem 1 How do we get data to the workers? NAS Compute Nodes SAN 2 Distributing Code! Don t move data to workers move workers to the data! - Store data on the local disks of nodes in the
More informationWAN Optimized Replication of Backup Datasets Using Stream-Informed Delta Compression
WAN Optimized Replication of Backup Datasets Using Stream-Informed Delta Compression Philip Shilane, Mark Huang, Grant Wallace, & Windsor Hsu Backup Recovery Systems Division EMC Corporation Introduction
More informationIBM Spectrum Scale IO performance
IBM Spectrum Scale 5.0.0 IO performance Silverton Consulting, Inc. StorInt Briefing 2 Introduction High-performance computing (HPC) and scientific computing are in a constant state of transition. Artificial
More informationDistributed File Systems II
Distributed File Systems II To do q Very-large scale: Google FS, Hadoop FS, BigTable q Next time: Naming things GFS A radically new environment NFS, etc. Independence Small Scale Variety of workloads Cooperation
More informationSDSS Dataset and SkyServer Workloads
SDSS Dataset and SkyServer Workloads Overview Understanding the SDSS dataset composition and typical usage patterns is important for identifying strategies to optimize the performance of the AstroPortal
More informationFlat Datacenter Storage. Edmund B. Nightingale, Jeremy Elson, et al. 6.S897
Flat Datacenter Storage Edmund B. Nightingale, Jeremy Elson, et al. 6.S897 Motivation Imagine a world with flat data storage Simple, Centralized, and easy to program Unfortunately, datacenter networks
More informationCA485 Ray Walshe Google File System
Google File System Overview Google File System is scalable, distributed file system on inexpensive commodity hardware that provides: Fault Tolerance File system runs on hundreds or thousands of storage
More informationAdapted from: TRENDS AND ATTRIBUTES OF HORIZONTAL AND VERTICAL COMPUTING ARCHITECTURES
Adapted from: TRENDS AND ATTRIBUTES OF HORIZONTAL AND VERTICAL COMPUTING ARCHITECTURES Tom Atwood Business Development Manager Sun Microsystems, Inc. Takeaways Understand the technical differences between
More informationRAIDIX Data Storage Solution. Clustered Data Storage Based on the RAIDIX Software and GPFS File System
RAIDIX Data Storage Solution Clustered Data Storage Based on the RAIDIX Software and GPFS File System 2017 Contents Synopsis... 2 Introduction... 3 Challenges and the Solution... 4 Solution Architecture...
More informationLeveraging Software-Defined Storage to Meet Today and Tomorrow s Infrastructure Demands
Leveraging Software-Defined Storage to Meet Today and Tomorrow s Infrastructure Demands Unleash Your Data Center s Hidden Power September 16, 2014 Molly Rector CMO, EVP Product Management & WW Marketing
More informationCPU Scheduling. Operating Systems (Fall/Winter 2018) Yajin Zhou ( Zhejiang University
Operating Systems (Fall/Winter 2018) CPU Scheduling Yajin Zhou (http://yajin.org) Zhejiang University Acknowledgement: some pages are based on the slides from Zhi Wang(fsu). Review Motivation to use threads
More informationLecture 11 Hadoop & Spark
Lecture 11 Hadoop & Spark Dr. Wilson Rivera ICOM 6025: High Performance Computing Electrical and Computer Engineering Department University of Puerto Rico Outline Distributed File Systems Hadoop Ecosystem
More informationHDFS: Hadoop Distributed File System. Sector: Distributed Storage System
GFS: Google File System Google C/C++ HDFS: Hadoop Distributed File System Yahoo Java, Open Source Sector: Distributed Storage System University of Illinois at Chicago C++, Open Source 2 System that permanently
More informationWarehouse- Scale Computing and the BDAS Stack
Warehouse- Scale Computing and the BDAS Stack Ion Stoica UC Berkeley UC BERKELEY Overview Workloads Hardware trends and implications in modern datacenters BDAS stack What is Big Data used For? Reports,
More informationLecture 7: Parallel Processing
Lecture 7: Parallel Processing Introduction and motivation Architecture classification Performance evaluation Interconnection network Zebo Peng, IDA, LiTH 1 Performance Improvement Reduction of instruction
More informationPERFORMANCE ANALYSIS AND OPTIMIZATION OF MULTI-CLOUD COMPUITNG FOR LOOSLY COUPLED MTC APPLICATIONS
PERFORMANCE ANALYSIS AND OPTIMIZATION OF MULTI-CLOUD COMPUITNG FOR LOOSLY COUPLED MTC APPLICATIONS V. Prasathkumar, P. Jeevitha Assiatant Professor, Department of Information Technology Sri Shakthi Institute
More informationFalkon: a Fast and Light-weight task execution framework
: a Fast and Light-weight task execution framework Ioan Raicu *, Yong Zhao *, Catalin Dumitrescu *, Ian Foster #*+, Mike Wilde #+ * Department of Computer Science, University of Chicago, IL, USA + Computation
More informationCS 590: High Performance Computing. Parallel Computer Architectures. Lab 1 Starts Today. Already posted on Canvas (under Assignment) Let s look at it
Lab 1 Starts Today Already posted on Canvas (under Assignment) Let s look at it CS 590: High Performance Computing Parallel Computer Architectures Fengguang Song Department of Computer Science IUPUI 1
More informationShort Talk: System abstractions to facilitate data movement in supercomputers with deep memory and interconnect hierarchy
Short Talk: System abstractions to facilitate data movement in supercomputers with deep memory and interconnect hierarchy François Tessier, Venkatram Vishwanath Argonne National Laboratory, USA July 19,
More informationParaMEDIC: Parallel Metadata Environment for Distributed I/O and Computing
ParaMEDIC: Parallel Metadata Environment for Distributed I/O and Computing Prof. Wu FENG Department of Computer Science Virginia Tech Work smarter not harder Overview Grand Challenge A large-scale biological
More informationChapter 6 Objectives
Chapter 6 Memory Chapter 6 Objectives Master the concepts of hierarchical memory organization. Understand how each level of memory contributes to system performance, and how the performance is measured.
More information6.2 DATA DISTRIBUTION AND EXPERIMENT DETAILS
Chapter 6 Indexing Results 6. INTRODUCTION The generation of inverted indexes for text databases is a computationally intensive process that requires the exclusive use of processing resources for long
More informationForget about the Clouds, Shoot for the MOON
Forget about the Clouds, Shoot for the MOON Wu FENG feng@cs.vt.edu Dept. of Computer Science Dept. of Electrical & Computer Engineering Virginia Bioinformatics Institute September 2012, W. Feng Motivation
More informationData Partitioning on Heterogeneous Multicore and Multi-GPU systems Using Functional Performance Models of Data-Parallel Applictions
Data Partitioning on Heterogeneous Multicore and Multi-GPU systems Using Functional Performance Models of Data-Parallel Applictions Ziming Zhong Vladimir Rychkov Alexey Lastovetsky Heterogeneous Computing
More informationDELL EMC ISILON F800 AND H600 I/O PERFORMANCE
DELL EMC ISILON F800 AND H600 I/O PERFORMANCE ABSTRACT This white paper provides F800 and H600 performance data. It is intended for performance-minded administrators of large compute clusters that access
More informationWas ist dran an einer spezialisierten Data Warehousing platform?
Was ist dran an einer spezialisierten Data Warehousing platform? Hermann Bär Oracle USA Redwood Shores, CA Schlüsselworte Data warehousing, Exadata, specialized hardware proprietary hardware Introduction
More informationCAVA: Exploring Memory Locality for Big Data Analytics in Virtualized Clusters
2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing : Exploring Memory Locality for Big Data Analytics in Virtualized Clusters Eunji Hwang, Hyungoo Kim, Beomseok Nam and Young-ri
More informationThe RAMDISK Storage Accelerator
The RAMDISK Storage Accelerator A Method of Accelerating I/O Performance on HPC Systems Using RAMDISKs Tim Wickberg, Christopher D. Carothers wickbt@rpi.edu, chrisc@cs.rpi.edu Rensselaer Polytechnic Institute
More informationImproved Solutions for I/O Provisioning and Application Acceleration
1 Improved Solutions for I/O Provisioning and Application Acceleration August 11, 2015 Jeff Sisilli Sr. Director Product Marketing jsisilli@ddn.com 2 Why Burst Buffer? The Supercomputing Tug-of-War A supercomputer
More informationTwo-Choice Randomized Dynamic I/O Scheduler for Object Storage Systems. Dong Dai, Yong Chen, Dries Kimpe, and Robert Ross
Two-Choice Randomized Dynamic I/O Scheduler for Object Storage Systems Dong Dai, Yong Chen, Dries Kimpe, and Robert Ross Parallel Object Storage Many HPC systems utilize object storage: PVFS, Lustre, PanFS,
More informationSAS and Grid Computing Maximize Efficiency, Lower Total Cost of Ownership Cheryl Doninger, SAS Institute, Cary, NC
Paper 227-29 SAS and Grid Computing Maximize Efficiency, Lower Total Cost of Ownership Cheryl Doninger, SAS Institute, Cary, NC ABSTRACT IT budgets are declining and data continues to grow at an exponential
More informationGrid Scheduling Architectures with Globus
Grid Scheduling Architectures with Workshop on Scheduling WS 07 Cetraro, Italy July 28, 2007 Ignacio Martin Llorente Distributed Systems Architecture Group Universidad Complutense de Madrid 1/38 Contents
More informationDesign and Evaluation of a Collective IO Model for Loosely Coupled Petascale Programming
Design and Evaluation of a Collective IO Model for Loosely Coupled Petascale Programming Zhao Zhang +, Allan Espinosa *, Kamil Iskra #, Ioan Raicu *, Ian Foster #*+, Michael Wilde #+ + Computation Institute,
More informationData Intensive Scalable Computing
Data Intensive Scalable Computing Randal E. Bryant Carnegie Mellon University http://www.cs.cmu.edu/~bryant Examples of Big Data Sources Wal-Mart 267 million items/day, sold at 6,000 stores HP built them
More informationUniprocessor Scheduling. Basic Concepts Scheduling Criteria Scheduling Algorithms. Three level scheduling
Uniprocessor Scheduling Basic Concepts Scheduling Criteria Scheduling Algorithms Three level scheduling 2 1 Types of Scheduling 3 Long- and Medium-Term Schedulers Long-term scheduler Determines which programs
More informationEMC Business Continuity for Microsoft Applications
EMC Business Continuity for Microsoft Applications Enabled by EMC Celerra, EMC MirrorView/A, EMC Celerra Replicator, VMware Site Recovery Manager, and VMware vsphere 4 Copyright 2009 EMC Corporation. All
More informationIssues in Parallel Processing. Lecture for CPSC 5155 Edward Bosworth, Ph.D. Computer Science Department Columbus State University
Issues in Parallel Processing Lecture for CPSC 5155 Edward Bosworth, Ph.D. Computer Science Department Columbus State University Introduction Goal: connecting multiple computers to get higher performance
More informationDistributed OS and Algorithms
Distributed OS and Algorithms Fundamental concepts OS definition in general: OS is a collection of software modules to an extended machine for the users viewpoint, and it is a resource manager from the
More informationAlgorithm Engineering with PRAM Algorithms
Algorithm Engineering with PRAM Algorithms Bernard M.E. Moret moret@cs.unm.edu Department of Computer Science University of New Mexico Albuquerque, NM 87131 Rome School on Alg. Eng. p.1/29 Measuring and
More information