The HDF Group Q5 Demo
|
|
- Adelia Dawson
- 5 years ago
- Views:
Transcription
1 The HDF Group The HDF Group Q5 Demo 5.6 HDF5 Transaction API 5.7 Full HDF5 Dynamic Data Structure NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL UNDER ITS SUBCONTRACT WITH LAWRENCE LIVERMORE NATIONAL SECURITY, LLC WHO IS THE OPERATOR AND MANAGER OF LAWRENCE LIVERMORE NATIONAL LABORATORY UNDER CONTRACT NO. DE-AC52-07NA27344 WITH THE U.S. DEPARTMENT OF ENERGY. THE UNITED STATES GOVERNMENT RETAINS AND THE PUBLISHER, BY ACCEPTING THE ARTICLE OF PUBLICATION, ACKNOWLEDGES THAT THE UNITED STATES GOVERNMENT RETAINS A NON-EXCLUSIVE, PAID-UP, IRREVOCABLE, WORLD-WIDE LICENSE TO PUBLISH OR REPRODUCE THE PUBLISHED FORM OF THIS MANUSCRIPT, OR ALLOW OTHERS TO DO SO, FOR UNITED STATES GOVERNMENT PURPOSES. THE VIEWS AND OPINIONS OF AUTHORS EXPRESSED HEREIN DO NOT NECESSARILY REFLECT THOSE OF THE UNITED STATES GOVERNMENT OR LAWRENCE LIVERMORE NATIONAL SECURITY, LLC. Copyright 2013 The HDF Group. All Rights Reserved 1
2 Q5 Highlights (I) New features added since Q4 (other than 5.6 & 5.7): Support for variable length datatypes Some deferred items from Q4 (mostly metadata get routines) Data Integrity for metadata; although not helpful at the moment without a real storage backend. Enable/Disable Data integrity checks through a property. Can be done for metadata and raw data. Still working with skeletal IOD on a local Linux box: EMC has their first code demo this quarter. Many tests (especially reads) are faked for now. This is the main limitation for adding automated regression testing framework and benchmarks. Accounts for new Cray system at LANL pending. Copyright 2013 The HDF Group. All Rights Reserved 2
3 Testing: Q5 Highlights (II) More tests are always added with new features. Tests verify correctness of asynchronous execution, axe dependencies, function shipping, and HDF5 to IOD translation (to a simple extent). Try to cover all routines added. Automated with a simple script to run tests on a local machine. Still working more on a larger automated regression testing framework. (should be available in Q6). This Demo will focus on 2 main and new features added: Transactions and Read Contexts Dynamic Data Structure support (Map objects) Copyright 2013 The HDF Group. All Rights Reserved 3
4 5.6 HDF5 Transaction API: Outline Transactions and Versions recap from earlier technical presentation. Diving more into semantics and usage. Go through some pseudo code & actual code. Run Demo. 5.7 Full HDF5 Dynamic Data Structure: Changes from Q4 New Map objects and routines Code example and Run Demo. Copyright 2013 The HDF Group. All Rights Reserved 4
5 FastForward Transactions (I) A transaction consists of a set of updates to a container. container file Updates are added to a transaction, not made directly to a container. Updates include additions, deletions, and modifications. Any number and size of updates may be included in a single transaction. Tiny transactions may have high overhead. Large transactions will amortize the overhead. Multiple processes can add updates to a single transaction. Copyright 2013 The HDF Group. All Rights Reserved 5
6 FastForward Transactions (II) A transaction is finished when no more updates will be added to the transaction. Transactions can finish in any order. The updates for a finished transaction are not visible in the container. A finished transaction must be committed in order for its updates to become visible in the container. Transactions are committed in strict numerical order. When a transaction is committed, all updates in the transaction are applied atomically to the container and become visible. If all updates cannot be applied, none are applied, and the transaction is discarded. Copyright 2013 The HDF Group. All Rights Reserved 6
7 FastForward Container Versions A version identifies a container at a given state. The version number equals the number of the committed transaction that transitioned the container into the state. A read context can be created for a container version. The read context allows access to the contents of the container version. A given container version is guaranteed to remain readable until all associated read contexts are closed. Copyright 2013 The HDF Group. All Rights Reserved 7
8 Acquiring a Container Version This is important, so that IOD will not flatten this version, and it is available to read from. Only one process is required to acquire, but any number can. IOD will ref. count the acquired context. If one process acquires a version, it can communicate the acquired version to other processes so that they do need to acquire it themselves. The same number of release calls need to be issued as acquire calls. [although different processes can release than acquired] Copyright 2013 The HDF Group. All Rights Reserved 8
9 Transactions and Read Contexts in Q5 Two new set of routines added to the HDF5 FastForward API: H5TR for transactions H5RC for Read Contexts All new APIs are asynchronous. All HDF5 read/get routines take Read Context IDs. All HDF5 write/update routines take Transaction IDs. Note that transactions specify a read context, so writes/updates happen within one. H5Fcreate_ff() will always use up transaction 0, i.e. application starts using transaction 1. H5Fopen_ff() will return an acquired read context with the latest readable version of the container. Copyright 2013 The HDF Group. All Rights Reserved 9
10 Container Version Acquire/Release semantics /* Any Leader Process */ version = 15; /* acquire container version 15 */ rid = H5RCacquire(file_id, &version, H5P_DEFAULT, event_q); /* Any Delegate process */ MPI_Ibcast()/MPI_Irecv() MPI_Wait(); /* wait for the acquire to complete. This is not necessary, but user must live with the consequences that the acquire on a flattened version will fail and so all subsequent reads and the release call on rid will fail. */ /* client received a version x = 15 ; create a read context ID. This is a local immediate operation. */ rid = H5RCcreate(file_id, x); /* If Leader has delegates that it wants to tell that it has acquired the container version, it has to wait for the acquire to complete before informing them */ H5EQwait(event_q, &num_requests, &status); MPI_Ibcast ()/ MPI_Isend() /* Read from Container */ /* If other processes were informed to use this version, must wait to hear from them before releasing */ MPI_Barrier()/MPI_Bcast()/MPI_Recv() /* Read from Container */ /* Wait for all reads to complete */ H5EQwait(event_q, &num_requests, &status); /* Tell leader I am done with my reads.*/ MPI_Barrier()/MPI_Bcast()/MPI_Secv() /* Close RC ID. This is a local operation that just frees resources. */ H5RCclose(rid1); /* release container version 15. This is async. */ H5RCrelease(rid, event_q); /* Close RC ID. This is a local operation that just frees resources. */ H5RCclose(rid); /* Wait on all Events, Everything was asynchronous thus far */ H5EQwait(event_q, &num_requests, &status); Copyright 2013 The HDF Group. All Rights Reserved 10
11 Hints to H5RCacquire It is possible that the user is not interested with one Exact version to acquire; i.e. does not want the acquire to fail if the version specified is not valid. Through the Read Context Acquire property list, the user can specify the following hints to acquire: H5RC_EXACT: Fail if the current version is not available (default). H5RC_PREV: Acquire the highest version smaller than the one specified if it is not available. H5RC_NEXT: Acquire the lowest version greater than the one specified if it is not available. H5RC_LAST: Acquire the last readable version; this will ignore the version specified in acquire. Copyright 2013 The HDF Group. All Rights Reserved 11
12 Other ways to Acquire a Version When opening a container, the user can optionally ask to also acquire the last readable version of the container: file_id = H5Fopen_ff(file_name, H5F_ACC_RDONLY, fapl_id, &rid, H5_EVENT_QUEUE_NULL); When finishing a transaction, the user can also optionally ask to acquire it into a read context: H5TRfinish(tid, H5P_DEFAULT, &rid, H5_EVENT_QUEUE_NULL); Copyright 2013 The HDF Group. All Rights Reserved 12
13 Creating a Transaction All operations that write/update the container must be part of a started transaction. All transactions must be created within a read context, because some updates require reading from the container. Transactions may be started by x leader processes and communicated to other delegate processes, or started by all processes. Those are two different models of operation. In the former case, it is not required that the processes who started the transaction be the one who finishes or aborts it. Copyright 2013 The HDF Group. All Rights Reserved 13
14 Operating Models Leaders (Red) Will start transaction. Will communicate that transaction has started and is available to do updates on to the delegates (blue). Will hear back from delegates that they are done updating. Will finish transaction; or designate a delegate to do it. Same number of start and finish calls. Delegates (Blue) Will hear from leaders that transactions can be used. Can write to transactions. Have to inform leaders when done with updates to transactions. Copyright 2013 The HDF Group. All Rights Reserved 14
15 Leaders/Delegates Model /* create transaction object with an already acquired read context 15 this is a local immediate operation */ tid = H5TRcreate(file_id, rid, (uint64_t)20); /* start transaction 20 with default model, i.e. Leader Model. */ if( I am a leader process) /* This is asynchronous, but here we make it synchronous so we can tell the delegates that it has been started */ ret = H5TRstart(tid, H5P_DEFAULT, H5_EVENT_QUEUE_NULL); trans_num = 20; /* Tell other procs that transaction 20 is started */ MPI_Ibcast(&trans_num, ); /* Leader processes can continue writing to transaction 20, while others wait for the ibcast to complete */ if(i am a delegate process) MPI_Wait(&mpi_req, MPI_STATUS_IGNORE); /* Write to container */. /* Delegate processes have to complete operations before notifying the leader */ if(i am a delegate process) H5EQwait(event_q, &num_requests, &status); /* Each Leader synchronizes with its delegates that they are done writing */ MPI_Barrier()/Bcast()/ if( I am a leader process) /* Finish the started transaction. */ ret = H5TRfinish(tid, H5P_DEFAULT, NULL, event_q); /* Note that leader does not have to wait for its updates to complete before issuing the finish. We track this internally. */ /* Wait on all Events*/ H5EQwait(event_q, &num_requests, &status); Copyright 2013 The HDF Group. All Rights Reserved 15
16 Multiple Leaders/No Delegates Model /* All processes are considered equal participants in the transaction semantics; no inter-process communication is required More communication with the IONs will be done though. */ /* create & start transaction 20 with num_peers = n */ tid2 = H5TRcreate(file_id, rid2, (uint64_t)20); trspl_id = H5Pcreate (H5P_TR_START); H5Pset_trspl_num_peers(trspl_id, n); H5TRstart(tid2, trspl_id, event_q); H5Pclose(trspl_id); /* Update/write to the container */ /* finish transaction 20. */ H5TRfinish(tid2, H5P_DEFAULT, NULL, event_q); /* Wait on all Events; everything was asynchronous thus far. */ H5EQwait(event_q, &num_requests, &status); Copyright 2013 The HDF Group. All Rights Reserved 16
17 Operating Within Transactions HDF5 has metadata and raw data operations. Operations that occur inside a transaction are typically to update the contents of the container, i.e. create a group, create a dataset within a group, write to the dataset, etc Is it required, for example, if I create a dataset to commit the transaction that the dataset was created in to be able to write to the dataset? No! But there are limitations/rules that must be followed Copyright 2013 The HDF Group. All Rights Reserved 17
18 Operating Inside a Transaction If I start transaction 1: Create a group G1: gid1 = H5Gcreate_ff(file_id, "G1", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT, tid1, event_q); Create a group G2 in G1: gid2 = H5Gcreate_ff(gid1, "G2", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT, tid1, event_q); This is possible because a read is not required to write to G1. If we happen to do something like this : gid2 = H5Gcreate_ff(file_id, G1/G2", H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT, tid1, event_q); the operation will fail because it will require a read to get to G1 from the root group. But since G1 is not readable in the transaction it has been created in, the result will be a failure. Using a path in an access operation requires all objects in the path to be readable from the read context that is used by the transaction. Copyright 2013 The HDF Group. All Rights Reserved 18
19 Access to Objects created by Leader But what if 1 process created the object, and other processes need to write to the object in the same transaction? The leader process will need to retrieve a token representing the object when it created it. This is a local operation: H5Oget_token(object_id, &token); Send the token to the other processes. The other processes will open the object using the token they received: obj_id = H5Oopen_by_token(token); This is commonly referred to as local-to-global/global-tolocal operation. This is not currently supported. Deferred to Q6. Copyright 2013 The HDF Group. All Rights Reserved 19
20 Quick Summary Transactions provide a mechanism for making atomic updates to a container. Committed transactions result in container versions. Writes are done to the future. To uncommitted transactions Reads are made from the past. From container versions Copyright 2013 The HDF Group. All Rights Reserved 20
21 Demo Look at some example code. Run Tests. Copyright 2013 The HDF Group. All Rights Reserved 21
22 Dynamic Data Structures The main purpose is to support ACG s need to access the FastForward stack. We added a new HDF5 object called a Map object with a new set of routines that should fully support the Dynamic Data Structure use case. H5DO routines to conveniently append to datasets and fast-append mechanism: Realized that implementation with IOD will not be possible without an atomic append feature. Will drop from current project as it wasn t high priority as other features. Support for variable length data is done with some limitations: No nested VL types, or VL types as fields of compound types. Neither are very commonly seen. Copyright 2013 The HDF Group. All Rights Reserved 22
23 Map objects A Map object is a direct mapping to a KV store. We wanted to expose that type of access to the application. New routines added: hid_t H5Mcreate_ff(hid_t loc_id, const char *name, hid_t keytype, hid_t valtype, hid_t lcpl_id, hid_t mcpl_id, hid_t mapl_id, hid_t trans_id, hid_t eq_id); hid_t H5Mopen_ff(hid_t loc_id, const char *name, hid_t mapl_id, hid_t rcxt_id, hid_t eq_id); herr_t H5Mset_ff(hid_t map_id, hid_t key_mem_type_id, const void *key, hid_t val_mem_type_id, const void *value, hid_t dxpl_id, hid_t trans_id, hid_t eq_id); herr_t H5Mget_ff(hid_t map_id, hid_t key_mem_type_id, const void *key, hid_t val_mem_type_id, void *value, hid_t dxpl_id, hid_t rcxt_id, hid_t eq_id); herr_t H5Mget_types_ff(hid_t map_id, hid_t *key_type_id, hid_t *val_type_id, hid_t rcxt_id, hid_t eq_id); herr_t H5Mget_count_ff(hid_t map_id, hsize_t *count, hid_t rcxt_id, hid_t eq_id); herr_t H5Mexists_ff(hid_t map_id, hid_t key_mem_type_id, const void *key, hbool_t *exists, hid_t rcxt_id, hid_t eq_id); herr_t H5Miterate_ff(hid_t map_id, hid_t key_mem_type_id, hid_t value_mem_type_id, H5M_iterate_func_t callback_func, void *context, hid_t rcxt_id); herr_t H5Mdelete_ff(hid_t map_id, hid_t key_mem_type_id, const void *key, hid_t trans_id, hid_t eq_id); herr_t H5Mclose_ff(hid_t map_id, hid_t eq_id); Copyright 2013 The HDF Group. All Rights Reserved 23
24 Demo Look at some example code. Run Tests. Copyright 2013 The HDF Group. All Rights Reserved 24
Milestone 6.3: Basic Analysis Shipping Demonstration
The HDF Group Milestone 6.3: Basic Analysis Shipping Demonstration Ruth Aydt, Mohamad Chaarawi, Ivo Jimenez, Quincey Koziol, Jerome Soumagne 12/17/2013 NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL
More informationMilestone 8.1: HDF5 Index Demonstration
The HDF Group Milestone 8.1: HDF5 Index Demonstration Ruth Aydt, Mohamad Chaarawi, Quincey Koziol, Aleksandar Jelenak, Jerome Soumagne 06/30/2014 NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY THE HDF GROUP
More informationHigh Level Design IOD KV Store FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O
Date: January 10, 2013 High Level Design IOD KV Store FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O LLNS Subcontract No. Subcontractor Name Subcontractor Address B599860
More informationDesign Document (Historical) HDF5 Dynamic Data Structure Support FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O
Date: July 24, 2013 Design Document (Historical) HDF5 Dynamic Data Structure Support FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O LLNS Subcontract No. Subcontractor
More informationFastForward I/O and Storage: ACG 8.6 Demonstration
FastForward I/O and Storage: ACG 8.6 Demonstration Kyle Ambert, Jaewook Yu, Arnab Paul Intel Labs June, 2014 NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL UNDER ITS SUBCONTRACT WITH LAWRENCE LIVERMORE
More informationFastForward I/O and Storage: ACG 5.8 Demonstration
FastForward I/O and Storage: ACG 5.8 Demonstration Jaewook Yu, Arnab Paul, Kyle Ambert Intel Labs September, 2013 NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL UNDER ITS SUBCONTRACT WITH LAWRENCE
More informationEFF-IO M7.5 Demo. Semantic Migration of Multi-dimensional Arrays
EFF-IO M7.5 Demo Semantic Migration of Multi-dimensional Arrays John Bent, Sorin Faibish, Xuezhao Liu, Harriet Qui, Haiying Tang, Jerry Tirrell, Jingwang Zhang, Kelly Zhang, Zhenhua Zhang NOTICE: THIS
More informationHigh Level Design Client Health and Global Eviction FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O MILESTONE: 4.
Date: 2013-06-01 High Level Design Client Health and Global Eviction FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O MILESTONE: 4.1 LLNS Subcontract No. Subcontractor
More information5.4 - DAOS Demonstration and Benchmark Report
5.4 - DAOS Demonstration and Benchmark Report Johann LOMBARDI on behalf of the DAOS team September 25 th, 2013 Livermore (CA) NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL UNDER ITS SUBCONTRACT WITH
More information8.5 End-to-End Demonstration Exascale Fast Forward Storage Team June 30 th, 2014
8.5 End-to-End Demonstration Exascale Fast Forward Storage Team June 30 th, 2014 NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL, THE HDF GROUP, AND EMC UNDER INTEL S SUBCONTRACT WITH LAWRENCE LIVERMORE
More informationParallel I/O and Portable Data Formats
Parallel I/O and Portable Data Formats Sebastian Lührs s.luehrs@fz-juelich.de Jülich Supercomputing Centre Forschungszentrum Jülich GmbH Reykjavík, August 25 th, 2017 Overview I/O can be the main bottleneck
More informationRFC: HDF5 File Space Management: Paged Aggregation
RFC: HDF5 File Space Management: Paged Aggregation Vailin Choi Quincey Koziol John Mainzer The current HDF5 file space allocation accumulates small pieces of metadata and raw data in aggregator blocks.
More informationParallel I/O and Portable Data Formats HDF5
Parallel I/O and Portable Data Formats HDF5 Sebastian Lührs s.luehrs@fz-juelich.de Jülich Supercomputing Centre Forschungszentrum Jülich GmbH Jülich, March 13th, 2018 Outline Introduction Structure of
More informationHierarchical Data Format 5:
Hierarchical Data Format 5: Giusy Muscianisi g.muscianisi@cineca.it SuperComputing Applications and Innovation Department May 17th, 2013 Outline What is HDF5? Overview to HDF5 Data Model and File Structure
More informationParallel I/O CPS343. Spring Parallel and High Performance Computing. CPS343 (Parallel and HPC) Parallel I/O Spring / 22
Parallel I/O CPS343 Parallel and High Performance Computing Spring 2018 CPS343 (Parallel and HPC) Parallel I/O Spring 2018 1 / 22 Outline 1 Overview of parallel I/O I/O strategies 2 MPI I/O 3 Parallel
More informationHDF5 File Space Management. 1. Introduction
HDF5 File Space Management 1. Introduction The space within an HDF5 file is called its file space. When a user first creates an HDF5 file, the HDF5 library immediately allocates space to store information
More informationCS /15/16. Paul Krzyzanowski 1. Question 1. Distributed Systems 2016 Exam 2 Review. Question 3. Question 2. Question 5.
Question 1 What makes a message unstable? How does an unstable message become stable? Distributed Systems 2016 Exam 2 Review Paul Krzyzanowski Rutgers University Fall 2016 In virtual sychrony, a message
More informationReduction Network Discovery Design Document FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O
Date: May 01, 2014 Reduction Network Discovery Design Document FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O LLNS Subcontract No. Subcontractor Name Subcontractor
More informationLustre* - Fast Forward to Exascale High Performance Data Division. Eric Barton 18th April, 2013
Lustre* - Fast Forward to Exascale High Performance Data Division Eric Barton 18th April, 2013 DOE Fast Forward IO and Storage Exascale R&D sponsored by 7 leading US national labs Solutions to currently
More informationState of OpenMP & Outlook on OpenMP 4.1
State of OpenMP & Outlook on OpenMP 4.1 Thursday, October 11, 2015 Bronis R. de Supinski Chair, OpenMP Language Committee This work has been authored by Lawrence Livermore National Security, LLC under
More informationSynchronization Part 2. REK s adaptation of Claypool s adaptation oftanenbaum s Distributed Systems Chapter 5 and Silberschatz Chapter 17
Synchronization Part 2 REK s adaptation of Claypool s adaptation oftanenbaum s Distributed Systems Chapter 5 and Silberschatz Chapter 17 1 Outline Part 2! Clock Synchronization! Clock Synchronization Algorithms!
More informationVersioning Object Storage Device (VOSD) Design Document FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O
Date: June 4, 2014 Versioning Object Storage Device (VOSD) Design Document FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O LLNS Subcontract No. Subcontractor Name Subcontractor
More informationAPPLICATION NOTE. Atmel AT03261: SAM D20 System Interrupt Driver (SYSTEM INTERRUPT) SAM D20 System Interrupt Driver (SYSTEM INTERRUPT)
APPLICATION NOTE Atmel AT03261: SAM D20 System Interrupt Driver (SYSTEM INTERRUPT) ASF PROGRAMMERS MANUAL SAM D20 System Interrupt Driver (SYSTEM INTERRUPT) This driver for SAM D20 devices provides an
More informationCOSC 6374 Parallel Computation. Scientific Data Libraries. Edgar Gabriel Fall Motivation
COSC 6374 Parallel Computation Scientific Data Libraries Edgar Gabriel Fall 2013 Motivation MPI I/O is good It knows about data types (=> data conversion) It can optimize various access patterns in applications
More informationNew Features in HDF5. Why new features? September 9, 2008 SPEEDUP Workshop - HDF5 Tutorial
New Features in HDF5 September 9, 2008 SPEEDUP Workshop - HDF5 Tutorial 1 Why new features? September 9, 2008 SPEEDUP Workshop - HDF5 Tutorial 2 1 Why new features? HDF5 1.8.0 was released in February
More informationDAOS Epoch Recovery Design FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O
Date: June 4, 2014 DAOS Epoch Recovery Design FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O LLNS Subcontract No. Subcontractor Name Subcontractor Address B599860 Intel
More informationRFC: HDF5 Virtual Dataset
RFC: HDF5 Virtual Dataset Quincey Koziol (koziol@hdfgroup.org) Elena Pourmal (epourmal@hdfgroup.org) Neil Fortner (nfortne2@hdfgroup.org) This document introduces Virtual Datasets (VDS) for HDF5 and summarizes
More informationProgress on OpenMP Specifications
Progress on OpenMP Specifications Wednesday, November 13, 2012 Bronis R. de Supinski Chair, OpenMP Language Committee This work has been authored by Lawrence Livermore National Security, LLC under contract
More informationMilestone Burst Buffer & Data Integrity Demonstra>on Milestone End- to- End Epoch Recovery Demonstra>on
he HF Group ilestone 7.2 - Burst Buffer & ata Integrity emonstra>on ilestone 7.3 - End- to- End Epoch Recovery emonstra>on NOICE: HIS ANUSCRIP HAS BEEN AUHORE BY HE HF GROUP UNER HE INEL SUBCONRAC WIH
More informationHDF5 User s Guide. HDF5 Release November
HDF5 User s Guide HDF5 Release 1.8.8 November 2011 http://www.hdfgroup.org Copyright Notice and License Terms for HDF5 (Hierarchical Data Format 5) Software Library and Utilities HDF5 (Hierarchical Data
More informationQuestions from last time
Questions from last time Pthreads vs regular thread? Pthreads are POSIX-standard threads (1995). There exist earlier and newer standards (C++11). Pthread is probably most common. Pthread API: about a 100
More informationNIF ICCS Test Controller for Automated & Manual Testing
UCRL-CONF-235325 NIF ICCS Test Controller for Automated & Manual Testing J. S. Zielinski October 5, 2007 International Conference on Accelerator and Large Experimental Physics Control Systems Knoxville,
More informationIntroduction to HDF5
The HDF Group Introduction to HDF5 Quincey Koziol Director of Core Software & HPC The HDF Group October 15, 2014 Blue Waters Advanced User Workshop 1 Why HDF5? Have you ever asked yourself: How will I
More informationCTF: State-of-the-Art and Building the Next Generation ASE 2017
CTF: State-of-the-Art and Building the Next Generation ASE 2017 August 15, 2017 Clark Taylor Lawrence Livermore National Laboratory University of Arizona 737334 LLNL-PRES-XXXXXX This work was performed
More informationHDF5 I/O Performance. HDF and HDF-EOS Workshop VI December 5, 2002
HDF5 I/O Performance HDF and HDF-EOS Workshop VI December 5, 2002 1 Goal of this talk Give an overview of the HDF5 Library tuning knobs for sequential and parallel performance 2 Challenging task HDF5 Library
More informationHDF- A Suitable Scientific Data Format for Satellite Data Products
HDF- A Suitable Scientific Data Format for Satellite Data Products Sk. Sazid Mahammad, Debajyoti Dhar and R. Ramakrishnan Data Products Software Division Space Applications Centre, ISRO, Ahmedabad 380
More informationSynchronization. Chapter 5
Synchronization Chapter 5 Clock Synchronization In a centralized system time is unambiguous. (each computer has its own clock) In a distributed system achieving agreement on time is not trivial. (it is
More information(Pessimistic) Timestamp Ordering. Rules for read and write Operations. Read Operations and Timestamps. Write Operations and Timestamps
(Pessimistic) stamp Ordering Another approach to concurrency control: Assign a timestamp ts(t) to transaction T at the moment it starts Using Lamport's timestamps: total order is given. In distributed
More informationTransactions in a Distributed Key-Value Store
Transactions in a Distributed Key-Value Store 6.824 Final Project James Thomas, Deepak Narayanan, Arjun Srinivasan May 11, 2014 Introduction Over the last several years, NoSQL databases such as Redis,
More informationSynchronization Part II. CS403/534 Distributed Systems Erkay Savas Sabanci University
Synchronization Part II CS403/534 Distributed Systems Erkay Savas Sabanci University 1 Election Algorithms Issue: Many distributed algorithms require that one process act as a coordinator (initiator, etc).
More informationCommon Persistent Memory POSIX* Runtime (CPPR) API Reference (MS21) API Reference High Performance Data Division
Common Persistent Memory POSIX* Runtime (CPPR) API Reference High Performance Data Division INTEL FEDERAL, LLC PROPRIETARY December 2017 Generated under Argonne Contract number: B609815 DISTRIBUTION STATEMENT:
More informationAPI and Usage of libhio on XC-40 Systems
API and Usage of libhio on XC-40 Systems May 24, 2018 Nathan Hjelm Cray Users Group May 24, 2018 Los Alamos National Laboratory LA-UR-18-24513 5/24/2018 1 Outline Background HIO Design HIO API HIO Configuration
More informationIntroduction to MySQL InnoDB Cluster
1 / 148 2 / 148 3 / 148 Introduction to MySQL InnoDB Cluster MySQL High Availability made easy Percona Live Europe - Dublin 2017 Frédéric Descamps - MySQL Community Manager - Oracle 4 / 148 Safe Harbor
More information(Pessimistic) Timestamp Ordering
(Pessimistic) Timestamp Ordering Another approach to concurrency control: Assign a timestamp ts(t) to transaction T at the moment it starts Using Lamport's timestamps: total order is given. In distributed
More informationAdding a System Call to Plan 9
Adding a System Call to Plan 9 John Floren (john@csplan9.rit.edu) Sandia National Laboratories Livermore, CA 94551 DOE/NNSA Funding Statement Sandia is a multiprogram laboratory operated by Sandia Corporation,
More informationHDF5: An Introduction. Adam Carter EPCC, The University of Edinburgh
HDF5: An Introduction Adam Carter EPCC, The University of Edinburgh What is HDF5? Hierarchical Data Format (version 5) From www.hdfgroup.org: HDF5 is a unique technology suite that makes possible the management
More informationPOSIX Threads and OpenMP tasks
POSIX Threads and OpenMP tasks Jimmy Aguilar Mena February 16, 2018 Introduction Pthreads Tasks Two simple schemas Independent functions # include # include void f u n c t i
More informationEnd-to-End Data Integrity in the Intel/EMC/HDF Group Exascale IO DOE Fast Forward Project
End-to-End Data Integrity in the Intel/EMC/HDF Group Exascale IO DOE Fast Forward Project As presented by John Bent, EMC and Quincey Koziol, The HDF Group Truly End-to-End App provides checksum buffer
More informationFast Forward I/O & Storage
Fast Forward I/O & Storage Eric Barton Lead Architect 1 Department of Energy - Fast Forward Challenge FastForward RFP provided US Government funding for exascale research and development Sponsored by 7
More informationDatabase Management System Prof. D. Janakiram Department of Computer Science & Engineering Indian Institute of Technology, Madras Lecture No.
Database Management System Prof. D. Janakiram Department of Computer Science & Engineering Indian Institute of Technology, Madras Lecture No. # 20 Concurrency Control Part -1 Foundations for concurrency
More informationProject Name. The Eclipse Integrated Computational Environment. Jay Jay Billings, ORNL Parent Project. None selected yet.
Project Name The Eclipse Integrated Computational Environment Jay Jay Billings, ORNL 20140219 Parent Project None selected yet. Background The science and engineering community relies heavily on modeling
More informationA Fast Review of C Essentials Part II
A Fast Review of C Essentials Part II Structural Programming by Z. Cihan TAYSI Outline Fixed vs. Automatic duration Scope Global variables The register specifier Storage classes Dynamic memory allocation
More informationCSP IN JAVASCRIPT.
CSP IN JAVASCRIPT Page 1 of 31 VINCENZO CHIANESE BUGS INTRODUCER AT APIARY https://github.com/xvincentx @D3DVincent https://vncz.js.org Page 2 of 31 COMMUNICATING SEQUENTIAL PROCESSES Page 3 of 31 THE
More informationFastForward I/O and Storage: IOD M5 Demonstration (5.2, 5.3, 5.9, 5.10)
FastForward I/O and Storage: IOD M5 Demonstration (5.2, 5.3, 5.9, 5.10) 1 EMC September, 2013 John Bent john.bent@emc.com Sorin Faibish faibish_sorin@emc.com Xuezhao Liu xuezhao.liu@emc.com Harriet Qiu
More informationCaching and Buffering in HDF5
Caching and Buffering in HDF5 September 9, 2008 SPEEDUP Workshop - HDF5 Tutorial 1 Software stack Life cycle: What happens to data when it is transferred from application buffer to HDF5 file and from HDF5
More informationRicardo Rocha. Department of Computer Science Faculty of Sciences University of Porto
Ricardo Rocha Department of Computer Science Faculty of Sciences University of Porto Slides based on the book Operating System Concepts, 9th Edition, Abraham Silberschatz, Peter B. Galvin and Greg Gagne,
More informationCSE 530A ACID. Washington University Fall 2013
CSE 530A ACID Washington University Fall 2013 Concurrency Enterprise-scale DBMSs are designed to host multiple databases and handle multiple concurrent connections Transactions are designed to enable Data
More informationAPIs for Persistent Memory Programming
APIs for Persistent Memory Programming MSST 2018 Andy Rudoff NVM Software Architect Intel Corporation Data Center Group A Full-Stack Example Using a key-value store as an example App Unmodified App, uses
More informationLecture X: Transactions
Lecture X: Transactions CMPT 401 Summer 2007 Dr. Alexandra Fedorova Transactions A transaction is a collection of actions logically belonging together To the outside world, a transaction must appear as
More informationOperating Systems, Assignment 2 Threads and Synchronization
Operating Systems, Assignment 2 Threads and Synchronization Responsible TA's: Zohar and Matan Assignment overview The assignment consists of the following parts: 1) Kernel-level threads package 2) Synchronization
More informationSnapshots and Repeatable reads for HBase Tables
Snapshots and Repeatable reads for HBase Tables Note: This document is work in progress. Contributors (alphabetical): Vandana Ayyalasomayajula, Francis Liu, Andreas Neumann, Thomas Weise Objective The
More informationTotalView Debugger New Features Guide. version 8.4.0
TotalView Debugger New Features Guide version 8.4.0 Copyright 2007, 2008 by TotalView Technologies. All rights reserved Copyright 1998 2007 by Etnus LLC. All rights reserved. Copyright 1996 1998 by Dolphin
More informationOracle Enterprise Manager Ops Center. Introduction. Creating Oracle Solaris 11 Zones Guide 12c Release 1 ( )
Oracle Enterprise Manager Ops Center Creating Oracle Solaris 11 Zones Guide 12c Release 1 (12.1.0.0.0) E27336-01 April 2012 This guide provides an end-to-end example for how to use Oracle Enterprise Manager
More informationAsynchronous Termination Detection Module User s Guide
Asynchronous Termination Detection Module User s Guide William McLon III Sandia National Laboratories wcmclen@sandia.gov 1 Introduction Interprocessor communications are performed through point-to-point
More informationZooKeeper Recipes and Solutions
by Table of contents 1 A Guide to Creating Higher-level Constructs with ZooKeeper...2 1.1 Out of the Box Applications: Name Service, Configuration, Group Membership... 2 1.2 Barriers... 2 1.3 Queues...
More informationQCOM Reference Guide
QCOM Reference Guide Lars Wirfelt 2002 06 10 Copyright 2005 2016 SSAB EMEA AB Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License,
More informationMPI Runtime Error Detection with MUST
MPI Runtime Error Detection with MUST At the 27th VI-HPS Tuning Workshop Joachim Protze IT Center RWTH Aachen University April 2018 How many issues can you spot in this tiny example? #include #include
More informationExtreme I/O Scaling with HDF5
Extreme I/O Scaling with HDF5 Quincey Koziol Director of Core Software Development and HPC The HDF Group koziol@hdfgroup.org July 15, 2012 XSEDE 12 - Extreme Scaling Workshop 1 Outline Brief overview of
More informationVisual Profiler. User Guide
Visual Profiler User Guide Version 3.0 Document No. 06-RM-1136 Revision: 4.B February 2008 Visual Profiler User Guide Table of contents Table of contents 1 Introduction................................................
More informationLA-UR Approved for public release; distribution is unlimited.
LA-UR-15-27727 Approved for public release; distribution is unlimited. Title: Survey and Analysis of Multiresolution Methods for Turbulence Data Author(s): Pulido, Jesus J. Livescu, Daniel Woodring, Jonathan
More informationCommon Persistent Memory POSIX Runtime (CPPR) API Reference Manual. Reference Manual High Performance Data Division
Common Persistent Memory POSIX Runtime (CPPR) Reference Manual High Performance Data Division INTEL FEDERAL, LLC PROPRIETARY October 2016 Generated under Argonne Contract number: B609815 DISTRIBUTION STATEMENT:
More informationRFC: Metadata Journaling to Improve Crash Survivability
RFC: Metadata Journaling to Improve Crash Survivability John Mainzer Introduction HDF5 files written via calls to the HDF5 library are susceptible to corruption in the event of application or system crash.
More informationTroubleshooting SCA Problems in WebSphere Process Server Open Mic
IBM Software Group Troubleshooting SCA Problems in WebSphere Process Server Open Mic 4 January 2011 WebSphere Support Technical Exchange Agenda Introduce the panel of experts Introduce Troubleshooting
More informationP1202R1: Asymmetric Fences
Document number: P1202R1 Date: 2018-01-20 (pre-kona) Reply-to: David Goldblatt Audience: SG1 P1202R1: Asymmetric Fences Overview Some types of concurrent algorithms can be split
More informationEL6483: Brief Overview of C Programming Language
EL6483: Brief Overview of C Programming Language EL6483 Spring 2016 EL6483 EL6483: Brief Overview of C Programming Language Spring 2016 1 / 30 Preprocessor macros, Syntax for comments Macro definitions
More informationAutomatic trace analysis with the Scalasca Trace Tools
Automatic trace analysis with the Scalasca Trace Tools Ilya Zhukov Jülich Supercomputing Centre Property Automatic trace analysis Idea Automatic search for patterns of inefficient behaviour Classification
More informationParallel HDF5 (PHDF5)
Parallel HDF5 (PHDF5) Giusy Muscianisi g.muscianisi@cineca.it SuperComputing Applications and Innovation Department May 17th, 2013 Outline Overview of Parallel HDF5 design Programming model for Creating
More informationLast Time. Think carefully about whether you use a heap Look carefully for stack overflow Especially when you have multiple threads
Last Time Cost of nearly full resources RAM is limited Think carefully about whether you use a heap Look carefully for stack overflow Especially when you have multiple threads Embedded C Extensions for
More informationCSC369H1 S2016 Midterm Test Instructor: Bogdan Simion. Duration 50 minutes Aids allowed: none. Student number:
CSC369H1 S2016 Midterm Test Instructor: Bogdan Simion Duration 50 minutes Aids allowed: none Student number: Last name: First name: Lecture section: L0101(day) L5101(evening) (circle only one) Do NOT turn
More informationOperating Systems. Operating System Structure. Lecture 2 Michael O Boyle
Operating Systems Operating System Structure Lecture 2 Michael O Boyle 1 Overview Architecture impact User operating interaction User vs kernel Syscall Operating System structure Layers Examples 2 Lower-level
More informationHigh Scalability Resource Management with SLURM Supercomputing 2008 November 2008
High Scalability Resource Management with SLURM Supercomputing 2008 November 2008 Morris Jette (jette1@llnl.gov) LLNL-PRES-408498 Lawrence Livermore National Laboratory What is SLURM Simple Linux Utility
More informationDeveloper's Reference Release 12.0
[1]Oracle Communications Billing and Revenue Management Developer's Reference Release 12.0 E51025-01 December 2017 Oracle Communications Billing and Revenue Management Developer's Reference, Release 12.0
More informationDeviations are things that modify a thread s normal flow of control. Unix has long had signals, and these must be dealt with in multithreaded
Deviations are things that modify a thread s normal flow of control. Unix has long had signals, and these must be dealt with in multithreaded improvements to Unix. There are actually two fairly different
More informationThe Performance Data Collector for hp OpenVMS
The Performance Data Collector for hp OpenVMS TDC Version 2.1 Technical Overview November 2004 Copyright 2003-2005 Hewlett-Packard Development Company, L.P. Contents Background... 4 Insufficient OpenVMS
More information! How is a thread different from a process? ! Why are threads useful? ! How can POSIX threads be useful?
Chapter 2: Threads: Questions CSCI [4 6]730 Operating Systems Threads! How is a thread different from a process?! Why are threads useful?! How can OSIX threads be useful?! What are user-level and kernel-level
More informationThe State and Needs of IO Performance Tools
The State and Needs of IO Performance Tools Scalable Tools Workshop Lake Tahoe, CA August 6 12, 2017 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National
More informationZooKeeper Recipes and Solutions
by Table of contents 1 A Guide to Creating Higher-level Constructs with ZooKeeper...2 1.1 Important Note About Error Handling... 2 1.2 Out of the Box Applications: Name Service, Configuration, Group Membership...2
More informationThe HDF Group. Parallel HDF5. Extreme Scale Computing Argonne.
The HDF Group Parallel HDF5 Advantage of Parallel HDF5 Recent success story Trillion particle simulation on hopper @ NERSC 120,000 cores 30TB file 23GB/sec average speed with 35GB/sec peaks (out of 40GB/sec
More informationLaboratory 1: Eclipse and Karel the Robot
Math 121: Introduction to Computing Handout #2 Laboratory 1: Eclipse and Karel the Robot Your first laboratory task is to use the Eclipse IDE framework ( integrated development environment, and the d also
More informationApplying Code Generation Approach in Fabrique Kirill Kalishev, JetBrains
november 2004 Applying Code Generation Approach in Fabrique This paper discusses ideas on applying the code generation approach to help the developer to focus on high-level models rather than on routine
More information!! How is a thread different from a process? !! Why are threads useful? !! How can POSIX threads be useful?
Chapter 2: Threads: Questions CSCI [4 6]730 Operating Systems Threads!! How is a thread different from a process?!! Why are threads useful?!! How can OSIX threads be useful?!! What are user-level and kernel-level
More informationWhat is DARMA? DARMA is a C++ abstraction layer for asynchronous many-task (AMT) runtimes.
DARMA Janine C. Bennett, Jonathan Lifflander, David S. Hollman, Jeremiah Wilke, Hemanth Kolla, Aram Markosyan, Nicole Slattengren, Robert L. Clay (PM) PSAAP-WEST February 22, 2017 Sandia National Laboratories
More informationTable of Contents. Cilk
Table of Contents 212 Introduction to Parallelism Introduction to Programming Models Shared Memory Programming Message Passing Programming Shared Memory Models Cilk TBB HPF Chapel Fortress Stapl PGAS Languages
More informationBlackBerry Java Development Environment (JDE)
1 BlackBerry Java Applications for Accessing SAP Applications BlackBerry Java Development Environment The BlackBerry Java Development Environment (JDE) is a fully integrated development and simulation
More informationCA IdentityMinder. Glossary
CA IdentityMinder Glossary 12.6.3 This Documentation, which includes embedded help systems and electronically distributed materials, (hereinafter referred to as the Documentation ) is for your informational
More informationCSE 486/586: Distributed Systems
CSE 486/586: Distributed Systems Concurrency Control (part 3) Ethan Blanton Department of Computer Science and Engineering University at Buffalo Lost Update Some transaction T1 runs interleaved with some
More informationStorageGRID Webscale NAS Bridge Management API Guide
StorageGRID Webscale NAS Bridge 2.0.3 Management API Guide January 2018 215-12414_B0 doccomments@netapp.com Table of Contents 3 Contents Understanding the NAS Bridge management API... 4 RESTful web services
More informationExam TI2720-C/TI2725-C Embedded Software
Exam TI2720-C/TI2725-C Embedded Software Wednesday April 16 2014 (18.30-21.30) Koen Langendoen In order to avoid misunderstanding on the syntactical correctness of code fragments in this examination, we
More informationDHANALAKSHMI COLLEGE OF ENGINEERING, CHENNAI
DHANALAKSHMI COLLEGE OF ENGINEERING, CHENNAI Department of Computer Science and Engineering CS6302- DATABASE MANAGEMENT SYSTEMS Anna University 2 & 16 Mark Questions & Answers Year / Semester: II / III
More informationDESCRIPTION OF NEW FUNCTIONS. Version 18.0
DESCRIPTION OF NEW FUNCTIONS Version 18.0 This document describes new functions in Automation version 18. Descriptions of Component Wizard, Software License, PC SCHEMATIC Panelrouter and PC SCHEMATIC Service
More information