Milestone 8.1: HDF5 Index Demonstration
|
|
- Shonda Peters
- 5 years ago
- Views:
Transcription
1 The HDF Group Milestone 8.1: HDF5 Index Demonstration Ruth Aydt, Mohamad Chaarawi, Quincey Koziol, Aleksandar Jelenak, Jerome Soumagne 06/30/2014 NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY THE HDF GROUP UNDER INTEL S SUBCONTRACT WITH LAWRENCE LIVERMORE NATIONAL SECURITY, LLC WHO IS THE OPERATOR AND MANAGER OF LAWRENCE LIVERMORE NATIONAL LABORATORY UNDER CONTRACT NO. DE-AC52-07NA27344 WITH THE U.S. DEPARTMENT OF ENERGY. THE UNITED STATES GOVERNMENT RETAINS AND THE PUBLISHER, BY ACCEPTING THE ARTICLE OF PUBLICATION, ACKNOWLEDGES THAT THE UNITED STATES GOVERNMENT RETAINS A NON-EXCLUSIVE, PAID-UP, IRREVOCABLE, WORLD-WIDE LICENSE TO PUBLISH OR REPRODUCE THE PUBLISHED FORM OF THIS MANUSCRIPT, OR ALLOW OTHERS TO DO SO, FOR UNITED STATES GOVERNMENT PURPOSES. THE VIEWS AND OPINIONS OF AUTHORS EXPRESSED HEREIN DO NOT NECESSARILY REFLECT THOSE OF THE UNITED STATES GOVERNMENT OR LAWRENCE LIVERMORE NATIONAL SECURITY, LLC The HDF Group 1
2 Background Analysis Shipping Queries sent to I/O node for enumeration of the data that is present locally on each node Queries operate on dataset elements Indexing support Add index to datasets Added indexing capabilities to H5Datasets, using a plugin mechanism Indexing packages may create, write and read index information anonymously within the H5Container, storing it outside of group hierarchy 2
3 Previous Indexing Demo What we demo ed (brute force index) A simple and not-to-smart index plugin that verifies access to H5Dataset information within an H5Container Index creation, access and updates within that container An application using a chosen plugin to create an index for an H5Dataset, querying its associated index, and verification that the index is used to help fulfill the query inside the view H5D Application H5Xplugin H5Xdummy H5 Container H5X 3
4 Indexing Demo Two indexing plugins added FastBit (bitmap index) ALACRITY (inverted index) Demo using these two plugins Creation of index / Answer of query Data update / Index update / Answer of query Current Limitation Incremental update of data is not supported by indexing packages Entire data has to be read to rebuild index 4
5 Acknowledgments FastBit team (LBNL) John Wu (many thanks for modifications to FastBit!) Suren Byna Bin Dong Prabhat ALACRITY team (NCSU) Nagiza Samatova David Boyuka Xiaocheng Zou 5
6 Index Plugin Class typedef struct { unsigned version; /* Version number of the index plugin class struct */ /* (Should always be set to H5X_CLASS_VERSION, which * may vary between releases of HDF5 library) */ unsigned id; /* Index ID (assigned by The HDF Group, for now) */ const char *idx_name; /* Index name (for debugging only, currently) */ H5X_type_t type; /* Type of data indexed by this plugin */ /* Callbacks */ void *(*create)(hid_t file_id, hid_t dataset_id, hid_t xcpl_id, hid_t xapl_id, size_t *metadata_size, void **metadata); herr_t (*remove)(hid_t file_id, hid_t dataset_id, size_t metadata_size, void *metadata); void *(*open)(hid_t file_id, hid_t dataset_id, hid_t xapl_id, size_t metadata_size, void *metadata); herr_t (*close)(void *idx_handle); herr_t (*pre_update)(void *idx_handle, hid_t dataspace_id, hid_t xxpl_id); herr_t (*post_update)(void *idx_handle, const void *buf, hid_t dataspace_id, hid_t xxpl_id); herr_t (*query)(void *idx_handle, hid_t query_id, hid_t xxpl_id, hid_t *dataspace_id); herr_t (*refresh)(void *idx_handle, size_t *metadata_size, void **metadata); } H5X_class_t; 6
7 Index API H5_DLL herr_t H5Xregister(const H5X_class_t *idx_class); H5_DLL herr_t H5Xunregister(unsigned plugin_id); H5_DLL herr_t H5Xcreate(hid_t file_id, unsigned plugin_id, hid_t scope_id, hid_t xcpl_id); H5_DLL herr_t H5Xcreate_ff(hid_t file_id, unsigned plugin_id, hid_t scope_id, hid_t xcpl_id, hid_t trans_id, hid_t estack_id); H5_DLL herr_t H5Xremove(hid_t file_id, unsigned plugin_id, hid_t scope_id); H5_DLL herr_t H5Xremove_ff(hid_t file_id, unsigned plugin_id, hid_t scope_id, hid_t trans_id, hid_t estack_id); H5_DLL herr_t H5Xget_count(hid_t scope_id, hsize_t *idx_count); H5_DLL herr_t H5Xget_count_ff(hid_t scope_id, hsize_t *idx_count, hid_t rcxt_id, hid_t estack_id); 7
8 FastBit Plugin Use FastBit in-memory API <iapi.h> Modifications to API to integrate into plugin API fastbit_iapi_build_index (build index) fastbit_iapi_decontruct_index (serialize index into key, offset and bitmap arrays) fastbit_iapi_attach_index (rebuild index from arrays) Create callback Build index from entire data Deconstruct index into arrays Create anonymous HDF5 datasets from arrays and store array dataset IDs into plugin metadata 8
9 FastBit Plugin Query callback: Reconstruct index if not reconstructed Create fastbit selection from query Evaluate selection Get coordinates from selection Return dataspace selection from coordinates Post-update callback: Gather/scatter data to merge new data with original data Update index and arrays used to store it (Refresh index metadata) 9
10 ALACRITY Plugin Use main ALACRITY API Create callback Build index from entire data using ALencode Store ALACRITY metadata and ALACRITY index into 2 anonymous datasets Store dataset IDs into plugin metadata Query callback: Read ALACRITY metadata if not present Translate HDF5 query into range Find bins that are touched by the query (all elements in the query range will fall into these bins) If bins are found, then read index from bin offsets and get coordinates, return dataspace selection from coordinates 10
11 ALACRITY Plugin Post-update callback: Gather/scatter data to merge new data with original data Update index and arrays used to store it (Refresh index metadata) 11
12 Demo example Dataset D0 (512 x 3) Create datasets using H5Dcreate/H5Dwrite etc filename: eff_index_file.h5 obj_name: D CNs ION Process 0 HDF5/IOD Server (Mercury Server) D
13 Demo example CNs Create index using H5Xcreate (call index create, create anonymous datasets for index information) filename: eff_index_file.h5 obj_name: D0 ION IDX0 Process 0 HDF5/IOD Server (Mercury Server) D0 set_index_info with index metadata created (stored on server using KVs for FF) 13
14 Demo example H5Dquery (call index query) filename: eff_index_file.h5 obj_name: D0 simple query: 38.8 < x < 42.1 Should return [39:0] [42:2] CNs ION IDX0 Process 0 HDF5/IOD Server (Mercury Server) D0 14
15 Demo example Dataset D0 (512 x 3) CNs Process 0 Process 1 Modify data and do incremental update (call post_update) / 1 write per transaction per process filename: eff_index_file.h5 obj_name: D0 ION HDF5/IOD Server (Mercury Server) D0 IDX0 15
16 Demo example H5Dquery (call index query) filename: eff_index_file.h5 obj_name: D0 simple query: 38.8 < x < 42.1 Should return [39:0] [511:2] CNs ION IDX0 Process 0 HDF5/IOD Server (Mercury Server) D0 16
17 Performance Number of Tuples , ,288 Brute Force (ms) FastBit (ms) ALACRITY (ms)
18 Time (ms) Performance Brute force FastBit ALACRITY k 8k 32k 128k 512k Number of tuples 18
19 Demo time 19
20 Conclusion Indexing capabilities integrated into analysis shipping Two Third-Party Index Package Plugins: FastBit ALACRITY Still room for optimization Pass minimal query range to index plugins Query optimizer Real incremental update Better use of pre_update/post_update calls 20
21 Questions 21
Milestone 6.3: Basic Analysis Shipping Demonstration
The HDF Group Milestone 6.3: Basic Analysis Shipping Demonstration Ruth Aydt, Mohamad Chaarawi, Ivo Jimenez, Quincey Koziol, Jerome Soumagne 12/17/2013 NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL
More informationThe HDF Group Q5 Demo
The HDF Group The HDF Group Q5 Demo 5.6 HDF5 Transaction API 5.7 Full HDF5 Dynamic Data Structure NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL UNDER ITS SUBCONTRACT WITH LAWRENCE LIVERMORE NATIONAL
More informationDesign Document (Historical) HDF5 Dynamic Data Structure Support FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O
Date: July 24, 2013 Design Document (Historical) HDF5 Dynamic Data Structure Support FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O LLNS Subcontract No. Subcontractor
More informationHigh Level Design IOD KV Store FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O
Date: January 10, 2013 High Level Design IOD KV Store FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O LLNS Subcontract No. Subcontractor Name Subcontractor Address B599860
More informationFastForward I/O and Storage: ACG 8.6 Demonstration
FastForward I/O and Storage: ACG 8.6 Demonstration Kyle Ambert, Jaewook Yu, Arnab Paul Intel Labs June, 2014 NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL UNDER ITS SUBCONTRACT WITH LAWRENCE LIVERMORE
More informationRFC: HDF5 File Space Management: Paged Aggregation
RFC: HDF5 File Space Management: Paged Aggregation Vailin Choi Quincey Koziol John Mainzer The current HDF5 file space allocation accumulates small pieces of metadata and raw data in aggregator blocks.
More informationFastForward I/O and Storage: ACG 5.8 Demonstration
FastForward I/O and Storage: ACG 5.8 Demonstration Jaewook Yu, Arnab Paul, Kyle Ambert Intel Labs September, 2013 NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL UNDER ITS SUBCONTRACT WITH LAWRENCE
More information5.4 - DAOS Demonstration and Benchmark Report
5.4 - DAOS Demonstration and Benchmark Report Johann LOMBARDI on behalf of the DAOS team September 25 th, 2013 Livermore (CA) NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL UNDER ITS SUBCONTRACT WITH
More informationEFF-IO M7.5 Demo. Semantic Migration of Multi-dimensional Arrays
EFF-IO M7.5 Demo Semantic Migration of Multi-dimensional Arrays John Bent, Sorin Faibish, Xuezhao Liu, Harriet Qui, Haiying Tang, Jerry Tirrell, Jingwang Zhang, Kelly Zhang, Zhenhua Zhang NOTICE: THIS
More informationHigh Level Design Client Health and Global Eviction FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O MILESTONE: 4.
Date: 2013-06-01 High Level Design Client Health and Global Eviction FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O MILESTONE: 4.1 LLNS Subcontract No. Subcontractor
More informationParallel I/O and Portable Data Formats HDF5
Parallel I/O and Portable Data Formats HDF5 Sebastian Lührs s.luehrs@fz-juelich.de Jülich Supercomputing Centre Forschungszentrum Jülich GmbH Jülich, March 13th, 2018 Outline Introduction Structure of
More informationHierarchical Data Format 5:
Hierarchical Data Format 5: Giusy Muscianisi g.muscianisi@cineca.it SuperComputing Applications and Innovation Department May 17th, 2013 Outline What is HDF5? Overview to HDF5 Data Model and File Structure
More informationParallel I/O and Portable Data Formats
Parallel I/O and Portable Data Formats Sebastian Lührs s.luehrs@fz-juelich.de Jülich Supercomputing Centre Forschungszentrum Jülich GmbH Reykjavík, August 25 th, 2017 Overview I/O can be the main bottleneck
More informationParallel I/O CPS343. Spring Parallel and High Performance Computing. CPS343 (Parallel and HPC) Parallel I/O Spring / 22
Parallel I/O CPS343 Parallel and High Performance Computing Spring 2018 CPS343 (Parallel and HPC) Parallel I/O Spring 2018 1 / 22 Outline 1 Overview of parallel I/O I/O strategies 2 MPI I/O 3 Parallel
More informationRFC: Reading Bit field Values from NPOESS Product File
April7,2009 RFCTHG2009 04 07.v3 RFC:ReadingBit fieldvaluesfromnpoessproductfile ElenaPourmal M.ScotBreitenfeld ThisRFCdescribesahelperAPIthatextractsbit fieldvaluesfromadatasetstoredinan NPOESSproductfile.
More informationTaming Parallel I/O Complexity with Auto-Tuning
Taming Parallel I/O Complexity with Auto-Tuning Babak Behzad 1, Huong Vu Thanh Luu 1, Joseph Huchette 2, Surendra Byna 3, Prabhat 3, Ruth Aydt 4, Quincey Koziol 4, Marc Snir 1,5 1 University of Illinois
More informationHDF5 File Space Management. 1. Introduction
HDF5 File Space Management 1. Introduction The space within an HDF5 file is called its file space. When a user first creates an HDF5 file, the HDF5 library immediately allocates space to store information
More informationTHE HYPERDYADIC INDEX AND GENERALIZED INDEXING AND QUERY WITH PIQUE
THE HYPERDYADIC INDEX AND GENERALIZED INDEXING AND QUERY WITH PIQUE David A. Boyuka II, Houjun Tang, Kushal Bansal, Xiaocheng Zou, Scott Klasky, Nagiza F. Samatova 6/30/2015 1 OVERVIEW Motivation Formal
More informationCOSC 6374 Parallel Computation. Scientific Data Libraries. Edgar Gabriel Fall Motivation
COSC 6374 Parallel Computation Scientific Data Libraries Edgar Gabriel Fall 2013 Motivation MPI I/O is good It knows about data types (=> data conversion) It can optimize various access patterns in applications
More information8.5 End-to-End Demonstration Exascale Fast Forward Storage Team June 30 th, 2014
8.5 End-to-End Demonstration Exascale Fast Forward Storage Team June 30 th, 2014 NOTICE: THIS MANUSCRIPT HAS BEEN AUTHORED BY INTEL, THE HDF GROUP, AND EMC UNDER INTEL S SUBCONTRACT WITH LAWRENCE LIVERMORE
More informationSDS: A Framework for Scientific Data Services
SDS: A Framework for Scientific Data Services Bin Dong, Suren Byna*, John Wu Scientific Data Management Group Lawrence Berkeley National Laboratory Finding Newspaper Articles of Interest Finding news articles
More informationIntroduction to HDF5
The HDF Group Introduction to HDF5 Quincey Koziol Director of Core Software & HPC The HDF Group October 15, 2014 Blue Waters Advanced User Workshop 1 Why HDF5? Have you ever asked yourself: How will I
More informationAPI and Usage of libhio on XC-40 Systems
API and Usage of libhio on XC-40 Systems May 24, 2018 Nathan Hjelm Cray Users Group May 24, 2018 Los Alamos National Laboratory LA-UR-18-24513 5/24/2018 1 Outline Background HIO Design HIO API HIO Configuration
More informationObject-Oriented Programming for Scientific Computing
Object-Oriented Programming for Scientific Computing Traits and Policies Ole Klein Interdisciplinary Center for Scientific Computing Heidelberg University ole.klein@iwr.uni-heidelberg.de 11. Juli 2017
More informationMilestone Burst Buffer & Data Integrity Demonstra>on Milestone End- to- End Epoch Recovery Demonstra>on
he HF Group ilestone 7.2 - Burst Buffer & ata Integrity emonstra>on ilestone 7.3 - End- to- End Epoch Recovery emonstra>on NOICE: HIS ANUSCRIP HAS BEEN AUHORE BY HE HF GROUP UNER HE INEL SUBCONRAC WIH
More informationReduction Network Discovery Design Document FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O
Date: May 01, 2014 Reduction Network Discovery Design Document FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O LLNS Subcontract No. Subcontractor Name Subcontractor
More informationCommon Persistent Memory POSIX* Runtime (CPPR) API Reference (MS21) API Reference High Performance Data Division
Common Persistent Memory POSIX* Runtime (CPPR) API Reference High Performance Data Division INTEL FEDERAL, LLC PROPRIETARY December 2017 Generated under Argonne Contract number: B609815 DISTRIBUTION STATEMENT:
More informationHDF5 User s Guide. HDF5 Release November
HDF5 User s Guide HDF5 Release 1.8.8 November 2011 http://www.hdfgroup.org Copyright Notice and License Terms for HDF5 (Hierarchical Data Format 5) Software Library and Utilities HDF5 (Hierarchical Data
More informationState of OpenMP & Outlook on OpenMP 4.1
State of OpenMP & Outlook on OpenMP 4.1 Thursday, October 11, 2015 Bronis R. de Supinski Chair, OpenMP Language Committee This work has been authored by Lawrence Livermore National Security, LLC under
More informationHigh Scalability Resource Management with SLURM Supercomputing 2008 November 2008
High Scalability Resource Management with SLURM Supercomputing 2008 November 2008 Morris Jette (jette1@llnl.gov) LLNL-PRES-408498 Lawrence Livermore National Laboratory What is SLURM Simple Linux Utility
More informationNIF ICCS Test Controller for Automated & Manual Testing
UCRL-CONF-235325 NIF ICCS Test Controller for Automated & Manual Testing J. S. Zielinski October 5, 2007 International Conference on Accelerator and Large Experimental Physics Control Systems Knoxville,
More informationArrayUDF Explores Structural Locality for Faster Scientific Analyses
ArrayUDF Explores Structural Locality for Faster Scientific Analyses John Wu 1 Bin Dong 1, Surendra Byna 1, Jialin Liu 1, Weijie Zhao 2, Florin Rusu 1,2 1 LBNL, Berkeley, CA 2 UC Merced, Merced, CA Two
More informationThe State and Needs of IO Performance Tools
The State and Needs of IO Performance Tools Scalable Tools Workshop Lake Tahoe, CA August 6 12, 2017 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National
More informationDRAFT. HDF5 Data Flow Pipeline for H5Dread. 1 Introduction. 2 Examples
This document describes the HDF5 library s data movement and processing activities when H5Dread is called for a dataset with chunked storage. The document provides an overview of how memory management,
More informationOvercoming Distributed Debugging Challenges in the MPI+OpenMP Programming Model
Overcoming Distributed Debugging Challenges in the MPI+OpenMP Programming Model Lai Wei, Ignacio Laguna, Dong H. Ahn Matthew P. LeGendre, Gregory L. Lee This work was performed under the auspices of the
More informationCommon Persistent Memory POSIX Runtime (CPPR) API Reference Manual. Reference Manual High Performance Data Division
Common Persistent Memory POSIX Runtime (CPPR) Reference Manual High Performance Data Division INTEL FEDERAL, LLC PROPRIETARY October 2016 Generated under Argonne Contract number: B609815 DISTRIBUTION STATEMENT:
More informationLA-UR Approved for public release; distribution is unlimited.
LA-UR-15-27727 Approved for public release; distribution is unlimited. Title: Survey and Analysis of Multiresolution Methods for Turbulence Data Author(s): Pulido, Jesus J. Livescu, Daniel Woodring, Jonathan
More informationPDF Document structure, that need for managing of PDF file. It uses in all functions from EMF2PDF SDK.
EMF2PDF SDK Pilot Structures struct pdf_document { PDFDocument4 *pdfdoc; }; PDF Document structure, that need for managing of PDF file. It uses in all functions from EMF2PDF SDK. typedef enum { conone
More informationOne-Sided Append: A New Communication Paradigm For PGAS Models
One-Sided Append: A New Communication Paradigm For PGAS Models James Dinan and Mario Flajslik Intel Corporation {james.dinan, mario.flajslik}@intel.com ABSTRACT One-sided append represents a new class
More informationPower Bounds and Large Scale Computing
1 Power Bounds and Large Scale Computing Friday, March 1, 2013 Bronis R. de Supinski 1 Tapasya Patki 2, David K. Lowenthal 2, Barry L. Rountree 1 and Martin Schulz 1 2 University of Arizona This work has
More informationJialin Liu, Evan Racah, Quincey Koziol, Richard Shane Canon, Alex Gittens, Lisa Gerhardt, Suren Byna, Mike F. Ringenburg, Prabhat
H5Spark H5Spark: Bridging the I/O Gap between Spark and Scien9fic Data Formats on HPC Systems Jialin Liu, Evan Racah, Quincey Koziol, Richard Shane Canon, Alex Gittens, Lisa Gerhardt, Suren Byna, Mike
More informationAdding a System Call to Plan 9
Adding a System Call to Plan 9 John Floren (john@csplan9.rit.edu) Sandia National Laboratories Livermore, CA 94551 DOE/NNSA Funding Statement Sandia is a multiprogram laboratory operated by Sandia Corporation,
More informationEnd-to-End Data Integrity in the Intel/EMC/HDF Group Exascale IO DOE Fast Forward Project
End-to-End Data Integrity in the Intel/EMC/HDF Group Exascale IO DOE Fast Forward Project As presented by John Bent, EMC and Quincey Koziol, The HDF Group Truly End-to-End App provides checksum buffer
More informationJava Based Open Architecture Controller
Preprint UCRL-JC- 137092 Java Based Open Architecture Controller G. Weinet? This article was submitted to World Automation Conference, Maui, HI, June 1 I- 16,200O U.S. Department of Energy January 13,200O
More informationFastForward I/O and Storage: IOD M5 Demonstration (5.2, 5.3, 5.9, 5.10)
FastForward I/O and Storage: IOD M5 Demonstration (5.2, 5.3, 5.9, 5.10) 1 EMC September, 2013 John Bent john.bent@emc.com Sorin Faibish faibish_sorin@emc.com Xuezhao Liu xuezhao.liu@emc.com Harriet Qiu
More informationHDF Product Designer: A tool for building HDF5 containers with granule metadata
The HDF Group HDF Product Designer: A tool for building HDF5 containers with granule metadata Lindsay Powers Aleksandar Jelenak, Joe Lee, Ted Habermann The HDF Group Data Producer s Conundrum 2 HDF Features
More informationDAOS Epoch Recovery Design FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O
Date: June 4, 2014 DAOS Epoch Recovery Design FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O LLNS Subcontract No. Subcontractor Name Subcontractor Address B599860 Intel
More informationMellanox Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) API Guide. Version 1.0
Mellanox Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) API Guide Version 1.0 Table of Contents Copyright... 3 Introduction... 4 Class Index... 5 File Index... 6 Class Documentation...
More informationRFC: HDF5 Virtual Dataset
RFC: HDF5 Virtual Dataset Quincey Koziol (koziol@hdfgroup.org) Elena Pourmal (epourmal@hdfgroup.org) Neil Fortner (nfortne2@hdfgroup.org) This document introduces Virtual Datasets (VDS) for HDF5 and summarizes
More informationEMPRESS Extensible Metadata PRovider for Extreme-scale Scientific Simulations
EMPRESS Extensible Metadata PRovider for Extreme-scale Scientific Simulations Photos placed in horizontal position with even amount of white space between photos and header Margaret Lawson, Jay Lofstead,
More informationDAOS Server Collectives Design FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O
Date: June 05, 2013 DAOS Server Collectives Design FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O LLNS Subcontract No. Subcontractor Name Subcontractor Address B599860
More informationand opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof.
'4 L NMAS CORE: UPDATE AND CURRENT DRECTONS DSCLAMER This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any
More informationzorder-lib: Library API for Z-Order Memory Layout
zorder-lib: Library API for Z-Order Memory Layout E. Wes Bethel Lawrence Berkeley National Laboratory Berkeley, CA, USA, 94720 April, 2015 i Acknowledgment This work was supported by the Director, Office
More informationIntermediate Programming, Spring 2017*
600.120 Intermediate Programming, Spring 2017* Misha Kazhdan *Much of the code in these examples is not commented because it would otherwise not fit on the slides. This is bad coding practice in general
More informationMETADATA REGISTRY, ISO/IEC 11179
LLNL-JRNL-400269 METADATA REGISTRY, ISO/IEC 11179 R. K. Pon, D. J. Buttler January 7, 2008 Encyclopedia of Database Systems Disclaimer This document was prepared as an account of work sponsored by an agency
More informationAdapting Software to NetCDF's Enhanced Data Model
Adapting Software to NetCDF's Enhanced Data Model Russ Rew UCAR Unidata EGU, May 2010 Overview Background What is netcdf? What is the netcdf classic data model? What is the netcdf enhanced data model?
More informationIntel Atom x3-c3200rk Processor (Formerly SoFIA 3G R) Simple V4L2- based Capture Method for MIPI-CSI2 Smart Camera Sensors
Intel Atom x3-c3200rk Processor (Formerly SoFIA 3G R) Simple V4L2- based Capture Method for MIPI-CSI2 Smart Camera Sensors Application developer s Guide June 2017 Maintenance Release 002 Document Number:
More informationUse of a new I/O stack for extreme-scale systems in scientific applications
1 Use of a new I/O stack for extreme-scale systems in scientific applications M. Scot Breitenfeld a, Quincey Koziol b, Neil Fortner a, Jerome Soumagne a, Mohamad Chaarawi a a The HDF Group, b Lawrence
More informationAn exceedingly high-level overview of ambient noise processing with Spark and Hadoop
IRIS: USArray Short Course in Bloomington, Indian Special focus: Oklahoma Wavefields An exceedingly high-level overview of ambient noise processing with Spark and Hadoop Presented by Rob Mellors but based
More informationFall 2017 :: CSE 306. File Systems Basics. Nima Honarmand
File Systems Basics Nima Honarmand File and inode File: user-level abstraction of storage (and other) devices Sequence of bytes inode: internal OS data structure representing a file inode stands for index
More informationProgress on OpenMP Specifications
Progress on OpenMP Specifications Wednesday, November 13, 2012 Bronis R. de Supinski Chair, OpenMP Language Committee This work has been authored by Lawrence Livermore National Security, LLC under contract
More informationCaching and Buffering in HDF5
Caching and Buffering in HDF5 September 9, 2008 SPEEDUP Workshop - HDF5 Tutorial 1 Software stack Life cycle: What happens to data when it is transferred from application buffer to HDF5 file and from HDF5
More informationMemory management. Johan Montelius KTH
Memory management Johan Montelius KTH 2017 1 / 22 C program # include int global = 42; int main ( int argc, char * argv []) { if( argc < 2) return -1; int n = atoi ( argv [1]); int on_stack
More informationData Structures Unit 02
Data Structures Unit 02 Bucharest University of Economic Studies Memory classes, Bit structures and operators, User data types Memory classes Define specific types of variables in order to differentiate
More informationLA-UR Approved for public release; distribution is unlimited.
LA-UR-15-27727 Approved for public release; distribution is unlimited. Title: Survey and Analysis of Multiresolution Methods for Turbulence Data Author(s): Pulido, Jesus J. Livescu, Daniel Woodring, Jonathan
More informationAET60 BioCARDKey. Application Programming Interface. Subject to change without prior notice
AET60 BioCARDKey Application Programming Interface Subject to change without prior notice Table of Contents 1.0. Introduction... 3 2.0. Application Programming Interface... 4 2.1. Overview...4 2.2. Data
More informationImplementing HDF5 in MATLAB
Implementing HDF5 in MATLAB Jeff Mather & Alec Rogers The MathWorks, Inc. 2006 The MathWorks, Inc. 29 November 2006 HDF4 1-1 mapping of C API first. (1998) Customer requests for high-level functions. HDFREAD,
More informationFile Open, Close, and Flush Performance Issues in HDF5 Scot Breitenfeld John Mainzer Richard Warren 02/19/18
File Open, Close, and Flush Performance Issues in HDF5 Scot Breitenfeld John Mainzer Richard Warren 02/19/18 1 Introduction Historically, the parallel version of the HDF5 library has suffered from performance
More informationrobotics/ openel.h File Reference Macros Macro Definition Documentation Typedefs Functions
openel.h File Reference Macros #define EL_TRUE 1 #define EL_FALSE 0 #define EL_NXT_PORT_A 0 #define EL_NXT_PORT_B 1 #define EL_NXT_PORT_C 2 #define EL_NXT_PORT_S1 0 #define EL_NXT_PORT_S2 1 #define EL_NXT_PORT_S3
More informationFCFS: On-Disk Design Revision: 1.8
Revision: 1.8 Date: 2003/07/06 12:26:43 1 Introduction This document describes the on disk format of the FCFSobject store. 2 Design Constraints 2.1 Constraints from attributes of physical disks The way
More informationGetting Insider Information via the New MPI Tools Information Interface
Getting Insider Information via the New MPI Tools Information Interface EuroMPI 2016 September 26, 2016 Kathryn Mohror This work was performed under the auspices of the U.S. Department of Energy by Lawrence
More informationFile Management. Information Structure 11/5/2013. Why Programmers Need Files
File Mgr Device Mgr Memory Mgr Process Mgr File Mgr Device Mgr Memory Mgr Process Mgr 11/5/2013 Slide 13-1 Slide 13-2 File Management 13 Fig 13-2: The External View of the File Manager Slide 13-3 Why Programmers
More informationAET60 API version 1.4 February Introduction...3 Features...3 Application Programming Interface...4
Version 1.4 02-2007 Unit 1008, 10th Floor, Hongkong International Trade and Exhibition Centre 1 Trademart Drive, Kowloon Bay, Hong Kong Tel: +852 2796 7873 Fax: +852 2796 1286 Email: info@acs.com.hk Website:
More informationStackwalkerAPI Programmer s Guide
Paradyn Parallel Performance Tools StackwalkerAPI Programmer s Guide Release 2.0 March 2011 Paradyn Project www.paradyn.org Computer Sciences Department University of Wisconsin Madison, WI 53706-1685 Computer
More informationHDF- A Suitable Scientific Data Format for Satellite Data Products
HDF- A Suitable Scientific Data Format for Satellite Data Products Sk. Sazid Mahammad, Debajyoti Dhar and R. Ramakrishnan Data Products Software Division Space Applications Centre, ISRO, Ahmedabad 380
More informationTIP120-SW-42. VxWorks Device Driver. Motion Controller with Incremental Encoder Interface. Version 2.1.x. User Manual. Issue 2.1.
The Embedded I/O Company TIP120-SW-42 VxWorks Device Driver Motion Controller with Incremental Encoder Interface Version 2.1.x User Manual Issue 2.1.0 May 2010 TEWS TECHNOLOGIES GmbH Am Bahnhof 7 25469
More informationlibhio: Optimizing IO on Cray XC Systems With DataWarp
libhio: Optimizing IO on Cray XC Systems With DataWarp May 9, 2017 Nathan Hjelm Cray Users Group May 9, 2017 Los Alamos National Laboratory LA-UR-17-23841 5/8/2017 1 Outline Background HIO Design Functionality
More informationint fnvgetconfig(handle h, UINT32 id, const void *cfg, size_t sz);... 4
RP-VL-UTIL-V1 Developer s Guide [ Contents ] 1. Introduction... 1 2. Building Environment... 1 3. Operating Environment... 1 4. Function Explanation... 2 4.1. Common API for Transmitting and Receiving...
More informationTIP570-SW-95 QNX-Neutrino Device Driver TIP570 16/8 Channel 12 Bit ADC and 8 Channel 12 Bit DAC on SBS PCI40 Carrier
TIP570-SW-95 QNX-Neutrino Device Driver TIP570 16/8 Channel 12 Bit ADC and 8 Channel 12 Bit DAC on SBS PCI40 Carrier Version 1.0.x Reference Manual Issue 1.0 January 2002 TEWS TECHNOLOGIES GmbH Am Bahnhof
More informationTIP670-SW-95. QNX-Neutrino Device Driver. User Manual. The Embedded I/O Company. Digital I/O. Version 1.0.x. Issue August 2008.
The Embedded I/O Company TIP670-SW-95 QNX-Neutrino Device Driver Digital I/O Version 1.0.x User Manual Issue 1.0.0 August 2008 TEWS TECHNOLOGIES GmbH Am Bahnhof 7 Phone: +49 (0) 4101 4058 0 25469 Halstenbek,
More informationVersioning Object Storage Device (VOSD) Design Document FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O
Date: June 4, 2014 Versioning Object Storage Device (VOSD) Design Document FOR EXTREME-SCALE COMPUTING RESEARCH AND DEVELOPMENT (FAST FORWARD) STORAGE AND I/O LLNS Subcontract No. Subcontractor Name Subcontractor
More informationTIP610-SW-95 QNX-Neutrino Device Driver TIP610 16/20 Channel Digital I/O on SBS PCI40 Carrier
TIP610-SW-95 QNX-Neutrino Device Driver TIP610 16/20 Channel Digital I/O on SBS PCI40 Carrier Version 1.0.x Reference Manual Issue 1.0 January 2002 TEWS TECHNOLOGIES GmbH Am Bahnhof 7 D-25469 Halstenbek
More informationThe HDF Group. Parallel HDF5. Quincey Koziol Director of Core Software & HPC The HDF Group.
The HDF Group Parallel HDF5 Quincey Koziol Director of Core Software & HPC The HDF Group Parallel HDF5 Success Story Recent success story Trillion particle simulation on hopper @ NERSC 120,000 cores 30TB
More informationQNX SDK for Apps and Media 1.1. Multimedia Playlist Library Reference
QNX SDK for Apps and Media 1.1 Multimedia Playlist Library Reference 2013 2015, QNX Software Systems Limited, a subsidiary of BlackBerry Limited. All rights reserved. QNX Software Systems Limited 1001
More informationProgress on Efficient Integration of Lustre* and Hadoop/YARN
Progress on Efficient Integration of Lustre* and Hadoop/YARN Weikuan Yu Robin Goldstone Omkar Kulkarni Bryon Neitzel * Some name and brands may be claimed as the property of others. MapReduce l l l l A
More informationTPMC500-SW-42. VxWorks Device Driver. User Manual. The Embedded I/O Company. 32 Channel 12 Bit ADC. Version 2.0.x. Issue 2.0.
The Embedded I/O Company TPMC500-SW-42 VxWorks Device Driver 32 Channel 12 Bit ADC Version 2.0.x User Manual Issue 2.0.0 October 2004 TEWS TECHNOLOGIES GmbH Am Bahnhof 7 e-mail: info@tews.com 25469 Halstenbek
More informationOptimizing Local File Accesses for FUSE-Based Distributed Storage
Optimizing Local File Accesses for FUSE-Based Distributed Storage Shun Ishiguro 1, Jun Murakami 1, Yoshihiro Oyama 1,3, Osamu Tatebe 2,3 1. The University of Electro-Communications, Japan 2. University
More informationTIP675-SW-82. Linux Device Driver. 48 TTL I/O Lines with Interrupts Version 1.2.x. User Manual. Issue November 2013
The Embedded I/O Company TIP675-SW-82 Linux Device Driver 48 TTL I/O Lines with Interrupts Version 1.2.x User Manual Issue 1.2.5 November 2013 TEWS TECHNOLOGIES GmbH Am Bahnhof 7 25469 Halstenbek, Germany
More informationSection 3: File I/O, JSON, Generics. Meghan Cowan
Section 3: File I/O, JSON, Generics Meghan Cowan POSIX Family of standards specified by the IEEE Maintains compatibility across variants of Unix-like OS Defines API and standards for basic I/O: file, terminal
More informationPusleIR Multitouch Screen Software SDK Specification. Revision 4.0
PusleIR Multitouch Screen Software SDK Specification Revision 4.0 Table of Contents 1. Overview... 3 1.1. Diagram... 3 1.1. PulseIR API Hierarchy... 3 1.2. DLL File... 4 2. Data Structure... 5 2.1 Point
More informationThe bigger picture. File systems. User space operations. What s a file. A file system is the user space implementation of persistent storage.
The bigger picture File systems Johan Montelius KTH 2017 A file system is the user space implementation of persistent storage. a file is persistent i.e. it survives the termination of a process a file
More informationCA Data Protection. External Agent COM API Reference Guide. Release 15.0
CA Data Protection External Agent COM API Reference Guide Release 15.0 This Documentation, which includes embedded help systems and electronically distributed materials (hereinafter referred to as the
More informationParallel NetCDF. Rob Latham Mathematics and Computer Science Division Argonne National Laboratory
Parallel NetCDF Rob Latham Mathematics and Computer Science Division Argonne National Laboratory robl@mcs.anl.gov I/O for Computational Science Application Application Parallel File System I/O Hardware
More informationGalica OSPERT Verification of OS-level Cache Management. Renato Mancuso Creative template. Sagar Chaki
Galica Verification of OS-level Cache Management Renato Mancuso Creative template Sagar Chaki OSPERT 2018 Goal + Approach Colored Lockdown for deterministic cache management C source code via CBMC Linux
More informationCSE 509: Computer Security
CSE 509: Computer Security Date: 2.16.2009 BUFFER OVERFLOWS: input data Server running a daemon Attacker Code The attacker sends data to the daemon process running at the server side and could thus trigger
More informationSystemVerilog Essentials Simulation & Synthesis
SystemVerilog Essentials Simulation & Synthesis Course Description This course provides all necessary theoretical and practical know-how to design programmable logic devices using SystemVerilog standard
More informationException Namespaces C Interoperability Templates. More C++ David Chisnall. March 17, 2011
More C++ David Chisnall March 17, 2011 Exceptions A more fashionable goto Provides a second way of sending an error condition up the stack until it can be handled Lets intervening stack frames ignore errors
More informationECP Alpine: Algorithms and Infrastructure for In Situ Visualization and Analysis
ECP Alpine: Algorithms and Infrastructure for In Situ Visualization and Analysis Presented By: Matt Larsen LLNL-PRES-731545 This work was performed under the auspices of the U.S. Department of Energy by
More informationCS-537: Midterm Exam (Spring 2009) The Future of Processors, Operating Systems, and You
CS-537: Midterm Exam (Spring 2009) The Future of Processors, Operating Systems, and You Please Read All Questions Carefully! There are 15 total numbered pages. Please put your NAME and student ID on THIS
More informationDeveloper manual. (Onvif Client Library) Happytimesoft Technology Co.,LTD
Developer manual (Onvif Client Library) Happytimesoft Technology Co.,LTD Declaration All rights reserved. No part of this publication may be excerpted, reproduced, translated, annotated or edited, in any
More information