Ariadne: a Tracking System for Relationships in LHCb Metadata
|
|
- Norman Cook
- 6 years ago
- Views:
Transcription
1 Ariadne: a Tracking System for Relationships in LHCb Metadata Illya Shapoval 1,2,3,4, Marco Clemencic 1, Marco Cattaneo 1 1 CERN, 2 KIPT, 3 UNIFE, 4 INFN-FE 20 th International Conference on Computing in High Energy and Nuclear Physics October 2013 Amsterdam
2 Content Objectives Approach Choice of DBMS Ariadne: design Interaction with the system CHEP 2013, October, Amsterdam 2
3 Introduction LHCb data processing implies handling of heterogeneous metadata entities Versions of applications (in 2 dimensions) Conditions Database states (in 2x3 dimensions) Real data reconstruction types MC data simulation types Many others: trigger configurations, arch. specificators, etc CHEP 2013, October, Amsterdam 3
4 Objectives: how did it start What entities are compatible with a CondDB state? Is a CondDB state consistent? What entities are compatible with an application? What entities are compatible with a data processing type? What entities are compatible? How are entities related? CHEP 2013, October, Amsterdam 4
5 Requirements An operational space with generic way of Expressing relationship constraints Tracking relationships Extracting solutions Ease of data management Flexibility (to extend the area of application) CHEP 2013, October, Amsterdam 5
6 Approach: property graphs Modeling structured metadata A:1 as nodes with attributes (key+value) A:2 A:3 B: W R: T S: CHEP 2013, October, Amsterdam 6
7 Approach: property graphs Modeling structured metadata as nodes with attributes (key+value) with typed connections A:1 A:2 A:3 xyz B: W R: T S: CHEP 2013, October, Amsterdam 7
8 Approach: property graphs Modeling structured metadata as nodes with attributes (key+value) with typed connections A:1 A:2 A:3 xyz B: W R: T S:99 Tracking? Extracting? CHEP 2013, October, Amsterdam 8
9 Choice of DBMS model Relevant characteristics Relational solution Graph solution Object-relational impedance mismatch problem [1] [2] Suffers of [2] Free of Flexibility of schema No Yes Performance of structural queries Poor [3] Better [3] Scaling to data complexity Poor [4] Better [4] Ease of data management Doable Better [1] C. Ireland et al, A Classification of Object-Relational Impedance Mismatch, DBKDA 09. [2] Correlated with the Performance of structural queries row of the table [3] C. Vicknair et al, A Comparison of a Graph Database and a Relational Database, ACM SE 10, NY. [4] See the next slide CHEP 2013, October, Amsterdam 9
10 Other NoSQL DBMS models Data size Key-value dbs Column dbs Document dbs Relational dbs Graph dbs Still billions of nodes Data complexity CHEP 2013, October, Amsterdam 10
11 Choice of graph DBMS Neo4j Taken from the Knowledge Base of Relational and NoSQL DBMS at: CHEP 2013, October, Amsterdam 11
12 Neo4j customers CHEP 2013, October, Amsterdam 12
13 Ariadne: system design A tracking system for relationships in LHCb metadata Metadata Ariadne Structural metadata Context diagram (CL tools, web FE) (Ariadne Python API, CL tools, web FE) Publish Neo4j database Query Level 0 (Leveled data flow diagram, shown in the Yourdon-DeMarco DFD notation) CHEP 2013, October, Amsterdam 13
14 Ariadne: current knowledge graph Metadata entities that current graph contains (>500) Applications A CondDB tags T (or TD, TC and TQ to specify partitions) DetectorTypes( DataTypes) D, RecoTypes R, SimTypes S Platforms, GRID sites Relationships between those entities (~50k),,,,,, P G A T A D A R A S A P D T R T S T CHEP 2013, October, Amsterdam 14
15 Tracking Relationships: Matching Patterns (1) What is the full compatible set of entities for concrete real data processing type? TQ TD Query A R D TC Ariadne TD TQ TD S Solution A R D D A TC TC TC CHEP 2013, October, Amsterdam 15
16 Tracking Relationships: Matching Patterns (2) What is the full compatible set of entities for concrete application and MC data processing type? TQ TD Query A S D TC Ariadne TD TQ TD S Solution A S D D A TC TC TC CHEP 2013, October, Amsterdam 16
17 Tracking Relationships: filtering multiple solutions (3) What is the full compatible set of entities for concrete application and MC data processing type? Query A TQ A S Set of solutions A TQ TD D S TC Ariadne TQ TD A S D TQ TD TC D S TC Ariadne filters multiple results according to the criterion provided by a user. TD D TC CHEP 2013, October, Amsterdam 17
18 Other application domains What entities are compatible with a CondDB state? Is a CondDB state consistent? What entities are compatible with an application? What entities are compatible? What entities are compatible with a data processing type? What does an application depend on? Where an entity is deployed to? What platform an application was built for?? How are entities related? CHEP 2013, October, Amsterdam 18
19 Summary Ariadne a generic tracking system for relationships in LHCb metadata Provides generic UI layer for heterogeneous metadata; Based on the novel Neo4j graph database; Provides powerful expressiveness when dealing with complex data (lots of relationships); Scalable and high performant solution for complex data CHEP 2013, October, Amsterdam 19
20 BACKUP CHEP 2013, October, Amsterdam 20
21 Ariadne: system requirements (narrows down to the Neo4j s requirements) Minimum Recommended Actual CPU Intel Core i3 Intel Core i7 Intel(R) Xeon(R) CPU 2.27GHz Memory 2GB 16 32GB or more 48GB Disk SATA SSD w/ SATA III SATA II Filesystem ext4 (or similar) ext4, ZFS ext4 Software Oracle Java 7, OpenJDK 7 Oracle Java 7 Oracle Java CHEP 2013, October, Amsterdam 21
22 Compatibility of Entities: Definition Two entities are declared compatible if a job, that uses them simultaneously: A. does not crash because of the combination B. is configured by the combination to work exactly in the way a developer anticipated it. Implication: A set of entities is declared compatible if and only if each pair of the entities (the compatibility is tracked between) out of the set is compatible CHEP 2013, October, Amsterdam 22
23 Ariadne management - Admin command line tools Ariadne: interaction - Visual interaction with the Ariadne graph 'DDDBConf' already got an Ariadne-based feature (>=LHCB_v35r6) from Configurables import DDDBConf DDDBConf( DataType = "2012", AutoTags = True ) Advanced interaction for users through the XMLRPC endpoint: from xmlrpclib import ServerProxy proxy = ServerProxy(" allow_none = True) response = proxy.queryariadne(question = {...}, criterion = '...') CHEP 2013, October, Amsterdam 23
24 Ariadne: interaction (2) criterion = 'latest_lhcbcond_dddb' question = { 'application': 'Brunel', 'application_version':'v43r2p2', 'detector_type': '2012', 'reco_type': 'Reco14', 'DDDBTag': None, 'LHCBCONDTag': None, 'DQFLAGSTag': None } response = { 'pathexists': True 'application': 'Brunel', 'application_version': 'v43r2p2', 'detector_type': '2012', 'reco_type': 'Reco14', 'DDDBTag': 'dddb ', 'LHCBCONDTag': 'cond ', 'DQFLAGSTag': 'dq ' } question = { 'application': 'Brunel', 'application_version': 'v43r2p2', 'detector_type': '2012', 'reco_type': 'Reco14', 'DDDBTag': 'dddb ', 'LHCBCONDTag': 'cond ', 'DQFLAGSTag': 'dq ' } question = { 'detector_type': '2012', 'reco_type': 'Reco14', 'DDDBTag': None, 'LHCBCONDTag': None, 'DQFLAGSTag': None } question = { 'detector_type': '2012', 'DDDBTag': None, 'LHCBCONDTag': None, 'DQFLAGSTag': None } question = { 'reco_type': 'Reco14', 'DDDBTag': None, 'LHCBCONDTag': None } CHEP 2013, October, Amsterdam 24
LHCb Distributed Conditions Database
LHCb Distributed Conditions Database Marco Clemencic E-mail: marco.clemencic@cern.ch Abstract. The LHCb Conditions Database project provides the necessary tools to handle non-event time-varying data. The
More informationUnderstanding StoRM: from introduction to internals
Understanding StoRM: from introduction to internals 13 November 2007 Outline Storage Resource Manager The StoRM service StoRM components and internals Deployment configuration Authorization and ACLs Conclusions.
More informationThe ATLAS EventIndex: an event catalogue for experiments collecting large amounts of data
The ATLAS EventIndex: an event catalogue for experiments collecting large amounts of data D. Barberis 1*, J. Cranshaw 2, G. Dimitrov 3, A. Favareto 1, Á. Fernández Casaní 4, S. González de la Hoz 4, J.
More informationBenchmarking the ATLAS software through the Kit Validation engine
Benchmarking the ATLAS software through the Kit Validation engine Alessandro De Salvo (1), Franco Brasolin (2) (1) Istituto Nazionale di Fisica Nucleare, Sezione di Roma, (2) Istituto Nazionale di Fisica
More informationAMGA metadata catalogue system
AMGA metadata catalogue system Hurng-Chun Lee ACGrid School, Hanoi, Vietnam www.eu-egee.org EGEE and glite are registered trademarks Outline AMGA overview AMGA Background and Motivation for AMGA Interface,
More informationExperience with ATLAS MySQL PanDA database service
Journal of Physics: Conference Series Experience with ATLAS MySQL PanDA database service To cite this article: Y Smirnov et al 2010 J. Phys.: Conf. Ser. 219 042059 View the article online for updates and
More informationCIB Session 12th NoSQL Databases Structures
CIB Session 12th NoSQL Databases Structures By: Shahab Safaee & Morteza Zahedi Software Engineering PhD Email: safaee.shx@gmail.com, morteza.zahedi.a@gmail.com cibtrc.ir cibtrc cibtrc 2 Agenda What is
More informationThe High-Level Dataset-based Data Transfer System in BESDIRAC
The High-Level Dataset-based Data Transfer System in BESDIRAC T Lin 1,2, X M Zhang 1, W D Li 1 and Z Y Deng 1 1 Institute of High Energy Physics, 19B Yuquan Road, Beijing 100049, People s Republic of China
More informationNOSQL Databases and Neo4j
NOSQL Databases and Neo4j Database and DBMS Database - Organized collection of data The term database is correctly applied to the data and their supporting data structures. DBMS - Database Management System:
More informationMySQL Cluster Web Scalability, % Availability. Andrew
MySQL Cluster Web Scalability, 99.999% Availability Andrew Morgan @andrewmorgan www.clusterdb.com Safe Harbour Statement The following is intended to outline our general product direction. It is intended
More informationDatabase Management Systems
DATABASE CONCEPTS & APPLICATIONS Database Management Systems A Database Management System (DBMS) is a software package designed to store and manage databases through database applications. User Database
More informationCIT 668: System Architecture. Amazon Web Services
CIT 668: System Architecture Amazon Web Services Topics 1. AWS Global Infrastructure 2. Foundation Services 1. Compute 2. Storage 3. Database 4. Network 3. AWS Economics Amazon Services Architecture Regions
More informationOracle Enterprise Manager Ops Center. Overview. What You Need. Create Oracle Solaris 10 Zones 12c Release 3 ( )
Oracle Enterprise Manager Ops Center Create Oracle Solaris 10 Zones 12c Release 3 (12.3.0.0.0) E60027-01 June 2015 This guide provides an end-to-end example for how to use Oracle Enterprise Manager Ops
More informationLNE. Vlaamse Milieuoverheid Flemish environmental government. Author: Patrik Uytterhoeven
Zabbix @ LNE Vlaamse Milieuoverheid Flemish environmental government Author: Patrik Uytterhoeven Who Am I Name: Patrik Uytterhoeven Job: Open-source consultant Company: Open-future Zabbix Trainer Author:
More informationDISTRIBUTED DATABASES
DISTRIBUTED DATABASES INTRODUCTION: Database technology has taken us from a paradigm of data processing in which each application defined and maintained its own data, i.e. one in which data is defined
More informationEvent cataloguing and other database applications in ATLAS
Event cataloguing and other database applications in ATLAS Dario Barberis Genoa University/INFN 1 Topics Event cataloguing: the new EventIndex DB Database usage by ATLAS in LHC Run2 PCD WS Ideas for a
More informationATLAS Oracle database applications and plans for use of the Oracle 11g enhancements
Database TEG workshop, Nov 2011 ATLAS Oracle database applications and plans for use of the Oracle 11g enhancements Gancho Dimitrov 1 Outline Some facts about the ATLAS databases at CERN Plan for upgrade
More informationTHE ATLAS DISTRIBUTED DATA MANAGEMENT SYSTEM & DATABASES
1 THE ATLAS DISTRIBUTED DATA MANAGEMENT SYSTEM & DATABASES Vincent Garonne, Mario Lassnig, Martin Barisits, Thomas Beermann, Ralph Vigne, Cedric Serfon Vincent.Garonne@cern.ch ph-adp-ddm-lab@cern.ch XLDB
More informationDeferred High Level Trigger in LHCb: A Boost to CPU Resource Utilization
Deferred High Level Trigger in LHCb: A Boost to Resource Utilization The use of periods without beam for online high level triggers Introduction, problem statement Realization of the chosen solution Conclusions
More informationMotivation and basic concepts Storage Principle Query Principle Index Principle Implementation and Results Conclusion
JSON Schema-less into RDBMS Most of the material was taken from the Internet and the paper JSON data management: sup- porting schema-less development in RDBMS, Liu, Z.H., B. Hammerschmidt, and D. McMahon,
More informationOracle Enterprise Manager Ops Center. Introduction. What You Will Need. Configure and Install Root Domains 12c Release 3 (
Oracle Enterprise Manager Ops Center Configure and Install Root Domains 12c Release 3 (12.3.0.0.0) E60040-01 June 2015 This guide provides an end-to-end example for how to use Oracle Enterprise Manager
More informationNew Oracle NoSQL Database APIs that Speed Insertion and Retrieval
New Oracle NoSQL Database APIs that Speed Insertion and Retrieval O R A C L E W H I T E P A P E R F E B R U A R Y 2 0 1 6 1 NEW ORACLE NoSQL DATABASE APIs that SPEED INSERTION AND RETRIEVAL Introduction
More informationThe Wuppertal Tier-2 Center and recent software developments on Job Monitoring for ATLAS
The Wuppertal Tier-2 Center and recent software developments on Job Monitoring for ATLAS DESY Computing Seminar Frank Volkmer, M. Sc. Bergische Universität Wuppertal Introduction Hardware Pleiades Cluster
More informationThe ATLAS Distributed Analysis System
The ATLAS Distributed Analysis System F. Legger (LMU) on behalf of the ATLAS collaboration October 17th, 2013 20th International Conference on Computing in High Energy and Nuclear Physics (CHEP), Amsterdam
More informationEvolution of Cloud Computing in ATLAS
The Evolution of Cloud Computing in ATLAS Ryan Taylor on behalf of the ATLAS collaboration 1 Outline Cloud Usage and IaaS Resource Management Software Services to facilitate cloud use Sim@P1 Performance
More informationCopyright 2016 Ramez Elmasri and Shamkant B. Navathe
Chapter 10 Outline Database Programming: Techniques and Issues Embedded SQL, Dynamic SQL, and SQLJ Database Programming with Function Calls: SQL/CLI and JDBC Database Stored Procedures and SQL/PSM Comparing
More informationDatabase System Concepts and Architecture
CHAPTER 2 Database System Concepts and Architecture Copyright 2017 Ramez Elmasri and Shamkant B. Navathe Slide 2-2 Outline Data Models and Their Categories History of Data Models Schemas, Instances, and
More information4th National Conference on Electrical, Electronics and Computer Engineering (NCEECE 2015)
4th National Conference on Electrical, Electronics and Computer Engineering (NCEECE 2015) Benchmark Testing for Transwarp Inceptor A big data analysis system based on in-memory computing Mingang Chen1,2,a,
More informationScott Meder Senior Regional Sales Manager
www.raima.com Scott Meder Senior Regional Sales Manager scott.meder@raima.com Short Introduction to Raima What is Data Management What are your requirements? How do I make the right decision? - Architecture
More informationCISC 7610 Lecture 4 Approaches to multimedia databases
CISC 7610 Lecture 4 Approaches to multimedia databases Topics: Metadata Loose vs tight coupling of data and metadata Object (oriented) databases Graph databases Object-relational mapping Homework 1 Entity-relationship
More informationAccessibility Features in the SAS Intelligence Platform Products
1 CHAPTER 1 Overview of Common Data Sources Overview 1 Accessibility Features in the SAS Intelligence Platform Products 1 SAS Data Sets 1 Shared Access to SAS Data Sets 2 External Files 3 XML Data 4 Relational
More informationHyperDex. A Distributed, Searchable Key-Value Store. Robert Escriva. Department of Computer Science Cornell University
HyperDex A Distributed, Searchable Key-Value Store Robert Escriva Bernard Wong Emin Gün Sirer Department of Computer Science Cornell University School of Computer Science University of Waterloo ACM SIGCOMM
More informationMySQL Database Administrator Training NIIT, Gurgaon India 31 August-10 September 2015
MySQL Database Administrator Training Day 1: AGENDA Introduction to MySQL MySQL Overview MySQL Database Server Editions MySQL Products MySQL Services and Support MySQL Resources Example Databases MySQL
More informationOracle Enterprise Manager Ops Center
Oracle Enterprise Manager Ops Center Configure and Install Guest Domains 12c Release 3 (12.3.2.0.0) E60042-03 June 2016 This guide provides an end-to-end example for how to use Oracle Enterprise Manager
More informationGoogle Cloud Platform for Systems Operations Professionals (CPO200) Course Agenda
Google Cloud Platform for Systems Operations Professionals (CPO200) Course Agenda Module 1: Google Cloud Platform Projects Identify project resources and quotas Explain the purpose of Google Cloud Resource
More informationUsing Oracle NoSQL Database Workshop Ed 1
Using Oracle NoSQL Database Workshop Ed 1 Duration 4 Days What you will learn In the Using Oracle NoSQL Database Workshop, you ll learn how Oracle NoSQL Database is used by administrators as well as developers.
More informationWhat is database? Types and Examples
What is database? Types and Examples Visit our site for more information: www.examplanning.com Facebook Page: https://www.facebook.com/examplanning10/ Twitter: https://twitter.com/examplanning10 TABLE
More informationHadoop An Overview. - Socrates CCDH
Hadoop An Overview - Socrates CCDH What is Big Data? Volume Not Gigabyte. Terabyte, Petabyte, Exabyte, Zettabyte - Due to handheld gadgets,and HD format images and videos - In total data, 90% of them collected
More informationPractical MySQL Performance Optimization. Peter Zaitsev, CEO, Percona July 20 th, 2016 Percona Technical Webinars
Practical MySQL Performance Optimization Peter Zaitsev, CEO, Percona July 20 th, 2016 Percona Technical Webinars In This Presentation We ll Look at how to approach Performance Optimization Discuss Practical
More informationOracle Enterprise Manager Ops Center. Introduction. Creating Oracle Solaris 11 Zones 12c Release 2 ( )
Oracle Enterprise Manager Ops Center Creating Oracle Solaris 11 Zones 12c Release 2 (12.2.2.0.0) E41230-04 February 2015 This guide provides an end-to-end example for how to use Oracle Enterprise Manager
More informationDatabase Systems. Sven Helmer. Database Systems p. 1/567
Database Systems Sven Helmer Database Systems p. 1/567 Chapter 1 Introduction and Motivation Database Systems p. 2/567 Introduction What is a database system (DBS)? Obviously a system for storing and managing
More informationCorrelX: A Cloud-Based VLBI Correlator
CorrelX: A Cloud-Based VLBI Correlator V. Pankratius, A. J. Vazquez, P. Elosegui Massachusetts Institute of Technology Haystack Observatory pankrat@mit.edu, victorpankratius.com 5 th International VLBI
More informationLCG Conditions Database Project
Computing in High Energy and Nuclear Physics (CHEP 2006) TIFR, Mumbai, 13 Feb 2006 LCG Conditions Database Project COOL Development and Deployment: Status and Plans On behalf of the COOL team (A.V., D.Front,
More informationThe ATLAS EventIndex: Full chain deployment and first operation
The ATLAS EventIndex: Full chain deployment and first operation Álvaro Fernández Casaní Instituto de Física Corpuscular () Universitat de València CSIC On behalf of the ATLAS Collaboration 1 Outline ATLAS
More informationA Web Service for Scholarly Big Data Information Extraction
A Web Service for Scholarly Big Data Information Extraction Kyle Williams, Lichi Li, Madian Khabsa, Jian Wu, Patrick C. Shih and C. Lee Giles Information Sciences and Technology Computer Science and Engineering
More informationInstalling the Product Software
CHAPTER 1 The Cisco Configuration Engine is a network management software that acts as a configuration service for automating the deployment, management, and upgrading of network devices and services.
More informationAn SQL-based approach to physics analysis
Journal of Physics: Conference Series OPEN ACCESS An SQL-based approach to physics analysis To cite this article: Dr Maaike Limper 2014 J. Phys.: Conf. Ser. 513 022022 View the article online for updates
More informationTesting storage and metadata backends
Testing storage and metadata backends Hugo González Labrador, Arno Formella LIA2, University of Vigo CS3: Cloud Storage Services for Novel Applications and Workflows Zürich, January 2016 Outline Origin
More informationScaling for Humongous amounts of data with MongoDB
Scaling for Humongous amounts of data with MongoDB Alvin Richards Technical Director, EMEA alvin@10gen.com @jonnyeight alvinonmongodb.com From here... http://bit.ly/ot71m4 ...to here... http://bit.ly/oxcsis
More informationData and Analysis preservation in LHCb
Data and Analysis preservation in LHCb - March 21, 2013 - S.Amerio (Padova), M.Cattaneo (CERN) Outline 2 Overview of LHCb computing model in view of long term preservation Data types and software tools
More informationWhere is Database Management System (DBMS) being Used?
The main objective of DBMS (Database Management System) is to provide a structured way to store and retrieve information that is both convenient and efficient. By data, we mean known facts that can be
More informationA Non-Relational Storage Analysis
A Non-Relational Storage Analysis Cassandra & Couchbase Alexandre Fonseca, Anh Thu Vu, Peter Grman Cloud Computing - 2nd semester 2012/2013 Universitat Politècnica de Catalunya Microblogging - big data?
More informationOracle Enterprise Manager Ops Center. Introduction. Creating Oracle Solaris 11 Zones Guide 12c Release 1 ( )
Oracle Enterprise Manager Ops Center Creating Oracle Solaris 11 Zones Guide 12c Release 1 (12.1.0.0.0) E27336-01 April 2012 This guide provides an end-to-end example for how to use Oracle Enterprise Manager
More informationNoSQL Databases An efficient way to store and query heterogeneous astronomical data in DACE. Nicolas Buchschacher - University of Geneva - ADASS 2018
NoSQL Databases An efficient way to store and query heterogeneous astronomical data in DACE DACE https://dace.unige.ch Data and Analysis Center for Exoplanets. Facility to store, exchange and analyse data
More informationUtilizing Databases in Grid Engine 6.0
Utilizing Databases in Grid Engine 6.0 Joachim Gabler Software Engineer Sun Microsystems http://sun.com/grid Current status flat file spooling binary format for jobs ASCII format for other objects accounting
More informationAnalytics Platform for ATLAS Computing Services
Analytics Platform for ATLAS Computing Services Ilija Vukotic for the ATLAS collaboration ICHEP 2016, Chicago, USA Getting the most from distributed resources What we want To understand the system To understand
More informationA Case Study of Real-World Porting to the Itanium Platform
A Case Study of Real-World Porting to the Itanium Platform Jeff Byard VP, Product Development RightOrder, Inc. Agenda RightOrder ADS Product Description Porting ADS to Itanium 2 Testing ADS on Itanium
More informationISTITUTO NAZIONALE DI FISICA NUCLEARE
ISTITUTO NAZIONALE DI FISICA NUCLEARE Sezione di Padova INFN/CCR-07/9 November 30, 2007 CCR/16/07/P PERFORMANCE OF DUAL PROCESSOR SERVER ON HEP CODE Michele Michelotto 1) 1) INFN-Sezione di Padova, Via
More informationOral Questions and Answers (DBMS LAB) Questions & Answers- DBMS
Questions & Answers- DBMS https://career.guru99.com/top-50-database-interview-questions/ 1) Define Database. A prearranged collection of figures known as data is called database. 2) What is DBMS? Database
More informationPractical MySQL Performance Optimization. Peter Zaitsev, CEO, Percona July 02, 2015 Percona Technical Webinars
Practical MySQL Performance Optimization Peter Zaitsev, CEO, Percona July 02, 2015 Percona Technical Webinars In This Presentation We ll Look at how to approach Performance Optimization Discuss Practical
More informationOutline. Database Management Systems (DBMS) Database Management and Organization. IT420: Database Management and Organization
Outline IT420: Database Management and Organization Dr. Crăiniceanu Capt. Balazs www.cs.usna.edu/~adina/teaching/it420/spring2007 Class Survey Why Databases (DB)? A Problem DB Benefits In This Class? Admin
More informationTransformation-free Data Pipelines by combining the Power of Apache Kafka and the Flexibility of the ESB's
Building Agile and Resilient Schema Transformations using Apache Kafka and ESB's Transformation-free Data Pipelines by combining the Power of Apache Kafka and the Flexibility of the ESB's Ricardo Ferreira
More informationA Tool for Conditions Tag Management in ATLAS
A Tool for Conditions Tag Management in ATLAS A. Sharmazanashvili 1, G. Batiashvili 1, G. Gvaberidze 1, L. Shekriladze 1, A. Formica 2 on behalf of ATLAS collaboration 1 Georgian CADCAM Engineering Center
More informationTechno Expert Solutions An institute for specialized studies!
Course Content of Data Integration and ETL with Oracle Warehouse Builder: Part 1: Installing and Setting Up the Warehouse Builder Environment What Is Oracle Warehouse Builder? Basic Process Flow of Design
More informationOracle NoSQL, ready for the Cloud
Oracle NoSQL, ready for the Cloud Nowadays everybody is talking about the Cloud. Enterprises are making a great effort to move many applications into the Cloud. There is a big concern about how the new
More informationHow we build TiDB. Max Liu PingCAP Amsterdam, Netherlands October 5, 2016
How we build TiDB Max Liu PingCAP Amsterdam, Netherlands October 5, 2016 About me Infrastructure engineer / CEO of PingCAP Working on open source projects: TiDB: https://github.com/pingcap/tidb TiKV: https://github.com/pingcap/tikv
More informationOracle Enterprise Manager Ops Center E Introduction
Oracle Enterprise Manager Ops Center Discover an Oracle ZFS Storage Appliance and Configure Storage Libraries 12c Release 2 (12.2.2.0.0) E40770-03 December 2014 This guide provides an end-to-end example
More informationAN introduction to nosql databases
AN introduction to nosql databases Terry McCann @SQLshark Purpose of this presentation? It is important for a data scientist / data engineer to have the right tool for the right job. We will look at an
More informationChapter 11: Implementing File
Chapter 11: Implementing File Systems Chapter 11: Implementing File Systems File-System Structure File-System Implementation Directory Implementation Allocation Methods Free-Space Management Efficiency
More informationW b b 2.0. = = Data Ex E pl p o l s o io i n
Hypertable Doug Judd Zvents, Inc. Background Web 2.0 = Data Explosion Web 2.0 Mt. Web 2.0 Traditional Tools Don t Scale Well Designed for a single machine Typical scaling solutions ad-hoc manual/static
More informationThe Application of DAQ-Middleware to the J-PARC E16 Experiment
Track 1 Session: #3 (Data acquisition and electronics) April 14, 2015 18:00 Village Center The Application of DAQ-Middleware to the J-PARC E16 Experiment E Hamada 1, M Ikeno 1, D Kawama 2, Y Morino 1,
More informationChapter 11: Implementing File Systems. Operating System Concepts 9 9h Edition
Chapter 11: Implementing File Systems Operating System Concepts 9 9h Edition Silberschatz, Galvin and Gagne 2013 Chapter 11: Implementing File Systems File-System Structure File-System Implementation Directory
More informationApplication development with relational and non-relational databases
Application development with relational and non-relational databases Mario Lassnig European Organization for Nuclear Research (CERN) mario.lassnig@cern.ch About me Software Engineer Data Management for
More informationTPC Benchmark H Full Disclosure Report. SPARC T4-4 Server Using Oracle Database 11g Release 2 Enterprise Edition with Partitioning
TPC Benchmark H Full Disclosure Report SPARC T4-4 Server Using Oracle Database 11g Release 2 Enterprise Edition with Partitioning Submitted for Review September 26, 2011 First Printing September 26, 2011
More informationFast Track Model Based Design and Development with Oracle9i Designer. An Oracle White Paper August 2002
Fast Track Model Based Design and Development with Oracle9i Designer An Oracle White Paper August 2002 Fast Track Model Based Design and Development with Oracle9i Designer Executive Overivew... 3 Introduction...
More informationCS370 Operating Systems
CS370 Operating Systems Colorado State University Yashwant K Malaiya Spring 2018 Lecture 22 File Systems Slides based on Text by Silberschatz, Galvin, Gagne Various sources 1 1 Disk Structure Disk can
More informationSocrates: A System for Scalable Graph Analytics C. Savkli, R. Carr, M. Chapman, B. Chee, D. Minch
Socrates: A System for Scalable Graph Analytics C. Savkli, R. Carr, M. Chapman, B. Chee, D. Minch September 10, 2014 Cetin Savkli Cetin.Savkli@jhuapl.edu 240 228 0115 Challenges of Big Data & Analytics
More information<Insert Picture Here> Oracle NoSQL Database A Distributed Key-Value Store
Oracle NoSQL Database A Distributed Key-Value Store Charles Lamb The following is intended to outline our general product direction. It is intended for information purposes only,
More informationGrid Services and the Globus Toolkit
Grid Services and the Globus Toolkit Lisa Childers childers@mcs.anl.gov The Globus Alliance Copyright (C) 2003 University of Chicago and The University of Southern California. All Rights Reserved. This
More informationEI 338: Computer Systems Engineering (Operating Systems & Computer Architecture)
EI 338: Computer Systems Engineering (Operating Systems & Computer Architecture) Dept. of Computer Science & Engineering Chentao Wu wuct@cs.sjtu.edu.cn Download lectures ftp://public.sjtu.edu.cn User:
More informationA Fast and High Throughput SQL Query System for Big Data
A Fast and High Throughput SQL Query System for Big Data Feng Zhu, Jie Liu, and Lijie Xu Technology Center of Software Engineering, Institute of Software, Chinese Academy of Sciences, Beijing, China 100190
More informationDatabase Assessment for PDMS
Database Assessment for PDMS Abhishek Gaurav, Nayden Markatchev, Philip Rizk and Rob Simmonds Grid Research Centre, University of Calgary. http://grid.ucalgary.ca 1 Introduction This document describes
More informationA Comparison of MapReduce Join Algorithms for RDF
A Comparison of MapReduce Join Algorithms for RDF Albert Haque 1 and David Alves 2 Research in Bioinformatics and Semantic Web Lab, University of Texas at Austin 1 Department of Computer Science, 2 Department
More informationChapter 10: File System Implementation
Chapter 10: File System Implementation Chapter 10: File System Implementation File-System Structure" File-System Implementation " Directory Implementation" Allocation Methods" Free-Space Management " Efficiency
More informationMySQL for Database Administrators Ed 3.1
Oracle University Contact Us: 1.800.529.0165 MySQL for Database Administrators Ed 3.1 Duration: 5 Days What you will learn The MySQL for Database Administrators training is designed for DBAs and other
More informationMigrate from Netezza Workload Migration
Migrate from Netezza Automated Big Data Open Netezza Source Workload Migration CASE SOLUTION STUDY BRIEF Automated Netezza Workload Migration To achieve greater scalability and tighter integration with
More informationASG-Rochade Reconciliation Toolkit Release Notes
ASG-Rochade Reconciliation Toolkit Release Notes Version 1.76.002 January 29, 2016 RRT1100-176 This publication introduces changes made to ASG-Rochade Reconciliation Toolkit (herein called Reconciliation
More informationGetting to know. by Michelle Darling August 2013
Getting to know by Michelle Darling mdarlingcmt@gmail.com August 2013 Agenda: What is Cassandra? Installation, CQL3 Data Modelling Summary Only 15 min to cover these, so please hold questions til the end,
More informationCreating Ultra-fast Realtime Apps and Microservices with Java. Markus Kett, CEO Jetstream Technologies
Creating Ultra-fast Realtime Apps and Microservices with Java Markus Kett, CEO Jetstream Technologies #NoDBMSApplications #JetstreamDB About me: Markus Kett Living in Regensburg, Germany Working with Java
More informationVeeam Availability for Nutanix AHV
Veeam Availability for Nutanix AHV Patric Hermann Systems Engineer Patric.Hermann@veeam.com HQ in Baar, Switzerland 320,000 Customers worldwide 4,000 additional per month 17.6M+ s protected in 200+ countries
More informationDIRAC data management: consistency, integrity and coherence of data
Journal of Physics: Conference Series DIRAC data management: consistency, integrity and coherence of data To cite this article: M Bargiotti and A C Smith 2008 J. Phys.: Conf. Ser. 119 062013 Related content
More informationClarityNow Best Practices Guide
ClarityNow Best Practices Guide Abstract A guide containing ClarityNow best practices and recommendations for common deployment to help avoid difficulties. Document includes descriptions of some default
More informationOptimizing Testing Performance With Data Validation Option
Optimizing Testing Performance With Data Validation Option 1993-2016 Informatica LLC. No part of this document may be reproduced or transmitted in any form, by any means (electronic, photocopying, recording
More informationThe LHCb Alignment Framework
The LHCb Alignment Framework LHCb detector description principles and elements Applying misalignments to detector components Some examples and uses in LHCb Conclusions Juan P. Palacios CERN CHEP, Mumbai,
More informationHBase vs Neo4j. Technical overview. Name: Vladan Jovičić CR09 Advanced Scalable Data (Fall, 2017) Ecolé Normale Superiuere de Lyon
HBase vs Neo4j Technical overview Name: Vladan Jovičić CR09 Advanced Scalable Data (Fall, 2017) Ecolé Normale Superiuere de Lyon 12th October 2017 1 Contents 1 Introduction 3 2 Overview of HBase and Neo4j
More informationHPSS Treefrog Summary MARCH 1, 2018
HPSS Treefrog Summary MARCH 1, 2018 Disclaimer Forward looking information including schedules and future software reflect current planning that may change and should not be taken as commitments by IBM
More informationDATABASE DESIGN II - 1DL400
DATABASE DESIGN II - 1DL400 Fall 2016 A second course in database systems http://www.it.uu.se/research/group/udbl/kurser/dbii_ht16 Kjell Orsborn Uppsala Database Laboratory Department of Information Technology,
More information1
1 3 4 6 7 8 9 Link to Storage Benchmarking Primer Running sysbench and preparing data Use the prepare option to generate the data. Experiments Run sysbench with different storage systems and instance
More informationOracle NoSQL Database Enterprise Edition, Version 18.1
Oracle NoSQL Database Enterprise Edition, Version 18.1 Oracle NoSQL Database is a scalable, distributed NoSQL database, designed to provide highly reliable, flexible and available data management across
More informationA Simplified Access to Grid Resources for Virtual Research Communities
Consorzio COMETA - Progetto PI2S2 UNIONE EUROPEA A Simplified Access to Grid Resources for Virtual Research Communities Roberto BARBERA (1-3), Marco FARGETTA (3,*) and Riccardo ROTONDO (2) (1) Department
More information