Lazy Big Data Integration
|
|
- Juliet Merritt
- 5 years ago
- Views:
Transcription
1 Lazy Big Integration Prof. Dr. Andreas Thor Hochschule für Telekommunikation Leipzig (HfTL) Martin-Luther-Universität Halle-Wittenberg
2 Agenda Integration analytics for domain-specific questions Use cases: Bibliometrics & Life Sciences Big Integration Techniques for efficient big data management Exploiting cloud infrastructures (MapReduce, NoSQL data stores) Lazy Big Integration (Efficient and) effective goal-oriented data integration Integrated analytical approach for big data analytics 2
3 Use Case: Bibliometrics Does the peer review process actually work? Does it select the best papers? from reviewing process (e.g., Easy Chair) Bibliographic information (title, authors, ) of submitted papers Review score(s) incl. editorial decision from bibliographic data sources (e.g., Google Scholar) Accepted papers and rejected papers that are published elsewhere Number of citations Determine covariance between review score(s) and #citations 3
4 Integration Combining data residing at different sources and providing the user with a unified view of this data 4 Added value by linking & merging data Queries that can only be answered using multiple sources Schema Matching Finding mappings of corresponding attributes Entity Matching Finding equivalent data objects
5 Quality Can/should we use Google Scholar citations for ranking Papers Researchers Institutions etc. Google Scholar Web of Science Coverage Huge Limited quality Medium (fully automatic) High (manually curated) Costs Free Expensive Convergent validity of citation analyses? Comparison of analysis results for source overlap 5
6 Use Case: Life Sciences Gene Annotation Graph Genes are annotated with Gene Ontology (GO) and Plant Ontology (PO) terms Links form a graph that captures meaningful biological knowledge Sense making of graph is important Prediction of new annotations hypothesis for wet lab experiments 6 Is this annotation likely to be added in the future?
7 Graph Summarization + Link Prediction Graph summary = Signature + Corrections Signature: graph pattern / structure Super nodes = partitioning of nodes Super edges = edges between super nodes = all edges between nodes of super nodes PO_9006 PO_20030 HY5 PHOT1 CIB5 Corrections: edges e between individual nodes Additions: e G but e signature Deletions: e G but e signature PO_37 PO_20038 = CRY2 COP1 CRY1 p(po_20030, CIB5) High prediction score because it is the only missing piece to a perfect 4x6 pattern PO_9006 PO_20030 PO_37 PO_20038 HY5 PHOT1 CIB5 CRY2 COP1 CRY1
8 (Big) Analytics Pipeline Bibliometrics Life Sciences Acquisition Extraction/ Cleaning Schema & Entity Matching Fusion/ Aggregation Analytics / Visualization 8
9 Agenda Integration analytics for domain-specific questions Use cases: Bibliometrics & Life Sciences Big Integration Techniques for efficient big data management Exploiting cloud infrastructures (MapReduce, NoSQL data stores) Lazy Big Integration (Efficient and) effective goal-oriented data integration Integrated analytical approach for big data analytics 9
10 (Big) Analytics Pipeline Bibliometrics Life Sciences Acquisition Extraction/ Cleaning Schema & Entity Matching Fusion/ Aggregation Analytics / Visualization Query Strategies Product Code Extraction Load Balancing NoSQL Sharding 10 Dedoop (Deduplication with Hadoop)
11 How to speed up entity matching? Entity matching is expensive (due to pair-wise comparisons) Blocking to reduce search space Group similar entities within blocks based on blocking key Restrict matching to entities from the same block Parallelization Split match computation in sub-tasks to be executed in parallel Exploitation of cloud infrastructures and frameworks like MapReduce 11
12 Input Split Partitioning: BlockKey mod #Reduce Tasks Blocking + MapReduce: Naïve skew leads to unbalanced workload S A w B w Cx D x E x F z G z H w Iw K y L y M z N z O z 12 Π 1 A w B w Cx D x E x F z G z Π 2 H w Iw K y L y M z N z O z Map: Blocking map 1 map 2 Key Value w A w B x C x D x E z F z G Key Value w H w I y K y L z M z N z O Key Value w A w B w H w I z F z G z M z N z O Key Value y K y L Key Value x C x D x E Reduce: Matching reduce 1 reduce 2 reduce 3 Pairs A-B, A-H, A-I, B-H, B-I, H-I F-G, F-M, F-N, F-O, G-M, G-N, G-O, M-N, M-O, N-O Pairs K-L Pairs C-D, C-E, D-E
13 Blocks Load Balancing for MR-based EM Partition Π 1 Π 2 Entity A B C D E F G H I K L M N O Blocking Key w w x x x z z w w y y z z z Analysis to identify blocking key distribution Global load balancing algorithms assigning similar number of pairs to reduce tasks 13 Input Entities MR Job1: Analysis Blocking Count Occurrences Π 1 map 1 reduce 1 Input Partitions Π 2 map 2 reduce 2 Π 1 Π 2 map 1 map 2 Load Balancing z Φ Block Distribution Matrix MR Job2: Matching reduce 1 reduce 2 reduce 3 Similarity Computation Partition Match Partitions M 1 M 2 M 3 Overall Π 1 Π 2 #E #P w Φ y Φ x Φ Match Result
14 Match Tasks Blocks BlockSplit Large blocks split into m sub-blocks according to m input partitions large if #P Block > #P Overall / #Reducer Two types of match tasks Single (small blocks and sub-blocks) Two sub-blocks Greedy load balancing Sort match tasks by number of pairs in descending order Assign match task to reducer with lowest number of pairs Example r=3 reduce tasks, split Φ 4 in m=2 sub-blocks Partition Overall Π 1 Π 2 #E #P w Φ y Φ x Φ z Φ #P Reducer Φ Φ Φ Φ Φ Φ Φ 4 s match tasks: Φ 4.1, Φ 4.2, and Φ
15 Partitioning by Reducer-Index BlockSplit: MR-flow MapReduce Techniques MapKey = ReducerIndex + MatchTask Replicate entities of sub-blocks 15 Π 1 A w B w Cx D x E x F z G z Π 2 H w Iw K y L y M z N z O z map 1 map 2 Map Key Value 1.1.* A 1.1.* B 3.3.* C 3.3.* D 3.3.* E F F G G 1 Key Value 1.1.* H 1.1.* I 1.2.* K 1.2.* L M M N N O O Reduce Key Value 1.1.* A 1.1.* B 1.1.* H 1.1.* I 1.2.* K 1.2.* L Key Value F G M N O F G Key Value 3.3.* C 3.3.* D 3.3.* E M N O reduce 1 reduce 2 reduce 3 Pairs A-B, A-H, A-I, B-H, B-I, H-I K-L Pairs F-M, F-N, F-O, G-M, G-N, G-O F-G Pairs C-D, C-E, D-E M-N, M-O, N-O
16 Evaluation: Robustness + Scalability Robust against data skew Uniform distribution All entities in a single block Scalable 16
17 Agenda Integration analytics for domain-specific questions Use cases: Bibliometrics & Life Sciences Big Integration Techniques for efficient big data management Exploiting cloud infrastructures (MapReduce, NoSQL data stores) Lazy Big Integration (Efficient and) effective goal-oriented data integration Integrated analytical approach for big data analytics 17
18 (Big) Analytics Pipeline Acquisition Extraction/ Cleaning Schema & Entity Matching Fusion/ Aggregation Analytics / Visualization Integration Interpretation 18
19 Citation Analysis Pipeline For a given set of Bibtex entries Find matching Google Scholar entries Determine aggregated citation counts Analytical questions for a researcher Complete publication list + #citations Top-5 publications H-Index, Average Number of citations Analytical questions for comparing Institutions Research fields 19
20 Lazy Machine : Effectiveness Do the right thing! Do only things that are needed! Priorization / filtering of data objects to be processed Example: Top-5 publications of a researcher Entity Matching for highly cited Google Scholar entries Cutoff data that does not contribute to the analytical result (anymore) does not is not likely to Pipeline stages Akquisition: query strategies Extraction: on-demand Matching: relevant entities only 20
21 Lazy User : Quality Automatic data integration does not give 100% data quality acquisition might miss relevant data Matching is imperfect (precision, recall) Pipeline & integrated result should effectively point the user to the weak points Examples What (non-)matching pairs have the most effect on the analytical result? Outlier detection What pipeline stage caused the effect? 21
22 Lazy Big Integration Integrated approach for both integration workflow Analytical query Current work based on Gradoop (Graph Analytics on Hadoop) Graph model + operators for analytical pipelines Efficient execution in distributed environment Next steps Operators for complex analytical queries / statistics (e.g., h-index) provenance model for measuring the impact of data objects to specific results 22
23 Summary Integration analytics for domain-specific questions Use cases: Bibliometrics & Life Sciences Big Integration Techniques for efficient big data management Exploiting cloud infrastructures (MapReduce, NoSQL data stores) Lazy Big Integration (Efficient and) effective goal-oriented data integration Integrated analytical approach for big data analytics 23
Efficient Entity Matching over Multiple Data Sources with MapReduce
Efficient Entity Matching over Multiple Data Sources with MapReduce Demetrio Gomes Mestre, Carlos Eduardo Pires Universidade Federal de Campina Grande, Brazil demetriogm@gmail.com, cesp@dsc.ufcg.edu.br
More informationAnalyzing Big Data with Microsoft R
Analyzing Big Data with Microsoft R 20773; 3 days, Instructor-led Course Description The main purpose of the course is to give students the ability to use Microsoft R Server to create and run an analysis
More informationOverview. Audience profile. At course completion. Course Outline. : 20773A: Analyzing Big Data with Microsoft R. Course Outline :: 20773A::
Module Title Duration : 20773A: Analyzing Big Data with Microsoft R : 3 days Overview The main purpose of the course is to give students the ability to use Microsoft R Server to create and run an analysis
More informationLoad Balancing for Entity Matching over Big Data using Sorted Neighborhood
San Jose State University SJSU ScholarWorks Master's Projects Master's Theses and Graduate Research Fall 2015 Load Balancing for Entity Matching over Big Data using Sorted Neighborhood Yogesh Wattamwar
More informationLoad Balancing for MapReduce-based Entity Resolution
Load Balancing for MapReduce-based Entity Resolution Lars Kolb, Andreas Thor, Erhard Rahm Database Group, University of Leipzig Leipzig, Germany {kolb,thor,rahm}@informatik.uni-leipzig.de Abstract The
More informationChallenges for Data Driven Systems
Challenges for Data Driven Systems Eiko Yoneki University of Cambridge Computer Laboratory Data Centric Systems and Networking Emergence of Big Data Shift of Communication Paradigm From end-to-end to data
More informationCISC 7610 Lecture 2b The beginnings of NoSQL
CISC 7610 Lecture 2b The beginnings of NoSQL Topics: Big Data Google s infrastructure Hadoop: open google infrastructure Scaling through sharding CAP theorem Amazon s Dynamo 5 V s of big data Everyone
More informationOracle Big Data Connectors
Oracle Big Data Connectors Oracle Big Data Connectors is a software suite that integrates processing in Apache Hadoop distributions with operations in Oracle Database. It enables the use of Hadoop to process
More informationCopyright 2012, Oracle and/or its affiliates. All rights reserved.
1 Big Data Connectors: High Performance Integration for Hadoop and Oracle Database Melli Annamalai Sue Mavris Rob Abbott 2 Program Agenda Big Data Connectors: Brief Overview Connecting Hadoop with Oracle
More informationPLATFORM AND SOFTWARE AS A SERVICE THE MAPREDUCE PROGRAMMING MODEL AND IMPLEMENTATIONS
PLATFORM AND SOFTWARE AS A SERVICE THE MAPREDUCE PROGRAMMING MODEL AND IMPLEMENTATIONS By HAI JIN, SHADI IBRAHIM, LI QI, HAIJUN CAO, SONG WU and XUANHUA SHI Prepared by: Dr. Faramarz Safi Islamic Azad
More informationHadoop Map Reduce 10/17/2018 1
Hadoop Map Reduce 10/17/2018 1 MapReduce 2-in-1 A programming paradigm A query execution engine A kind of functional programming We focus on the MapReduce execution engine of Hadoop through YARN 10/17/2018
More informationAdvanced Data Management Technologies Written Exam
Advanced Data Management Technologies Written Exam 02.02.2016 First name Student number Last name Signature Instructions for Students Write your name, student number, and signature on the exam sheet. This
More informationEmbedded Technosolutions
Hadoop Big Data An Important technology in IT Sector Hadoop - Big Data Oerie 90% of the worlds data was generated in the last few years. Due to the advent of new technologies, devices, and communication
More informationECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective
ECE 7650 Scalable and Secure Internet Services and Architecture ---- A Systems Perspective Part II: Data Center Software Architecture: Topic 3: Programming Models RCFile: A Fast and Space-efficient Data
More informationIntroduction to NoSQL
Introduction to NoSQL Agenda History What is NoSQL Types of NoSQL The CAP theorem History - RDBMS Relational DataBase Management Systems were invented in the 1970s. E. F. Codd, "Relational Model of Data
More informationIBM Data Science Experience White paper. SparkR. Transforming R into a tool for big data analytics
IBM Data Science Experience White paper R Transforming R into a tool for big data analytics 2 R Executive summary This white paper introduces R, a package for the R statistical programming language that
More informationCS555: Distributed Systems [Fall 2017] Dept. Of Computer Science, Colorado State University
CS 555: DISTRIBUTED SYSTEMS [MAPREDUCE] Shrideep Pallickara Computer Science Colorado State University Frequently asked questions from the previous class survey Bit Torrent What is the right chunk/piece
More informationParallel Programming Principle and Practice. Lecture 10 Big Data Processing with MapReduce
Parallel Programming Principle and Practice Lecture 10 Big Data Processing with MapReduce Outline MapReduce Programming Model MapReduce Examples Hadoop 2 Incredible Things That Happen Every Minute On The
More informationCSE 544 Principles of Database Management Systems. Alvin Cheung Fall 2015 Lecture 10 Parallel Programming Models: Map Reduce and Spark
CSE 544 Principles of Database Management Systems Alvin Cheung Fall 2015 Lecture 10 Parallel Programming Models: Map Reduce and Spark Announcements HW2 due this Thursday AWS accounts Any success? Feel
More informationAccelerating Analytical Workloads
Accelerating Analytical Workloads Thomas Neumann Technische Universität München April 15, 2014 Scale Out in Big Data Analytics Big Data usually means data is distributed Scale out to process very large
More informationMapReduce: Simplified Data Processing on Large Clusters 유연일민철기
MapReduce: Simplified Data Processing on Large Clusters 유연일민철기 Introduction MapReduce is a programming model and an associated implementation for processing and generating large data set with parallel,
More informationIn-memory data pipeline and warehouse at scale using Spark, Spark SQL, Tachyon and Parquet
In-memory data pipeline and warehouse at scale using Spark, Spark SQL, Tachyon and Parquet Ema Iancuta iorhian@gmail.com Radu Chilom radu.chilom@gmail.com Big data analytics / machine learning 6+ years
More informationTime Complexity and Parallel Speedup to Compute the Gamma Summarization Matrix
Time Complexity and Parallel Speedup to Compute the Gamma Summarization Matrix Carlos Ordonez, Yiqun Zhang Department of Computer Science, University of Houston, USA Abstract. We study the serial and parallel
More informationAbstractions for Data Intensive Computing
Severin Hacker, Robert J. Simmons, and Carsten Varming Carnegie Mellon University December 19, 2007 Data-Intensive Computing The Netezza Platform Hypothesis Outline Anatomy of a data-intensive computing
More informationCSE 344 MAY 2 ND MAP/REDUCE
CSE 344 MAY 2 ND MAP/REDUCE ADMINISTRIVIA HW5 Due Tonight Practice midterm Section tomorrow Exam review PERFORMANCE METRICS FOR PARALLEL DBMSS Nodes = processors, computers Speedup: More nodes, same data
More informationMap-Reduce. Marco Mura 2010 March, 31th
Map-Reduce Marco Mura (mura@di.unipi.it) 2010 March, 31th This paper is a note from the 2009-2010 course Strumenti di programmazione per sistemi paralleli e distribuiti and it s based by the lessons of
More informationA Fast and High Throughput SQL Query System for Big Data
A Fast and High Throughput SQL Query System for Big Data Feng Zhu, Jie Liu, and Lijie Xu Technology Center of Software Engineering, Institute of Software, Chinese Academy of Sciences, Beijing, China 100190
More information/ Cloud Computing. Recitation 10 March 22nd, 2016
15-319 / 15-619 Cloud Computing Recitation 10 March 22nd, 2016 Overview Administrative issues Office Hours, Piazza guidelines Last week s reflection Project 3.3, OLI Unit 4, Module 15, Quiz 8 This week
More informationStages of Data Processing
Data processing can be understood as the conversion of raw data into a meaningful and desired form. Basically, producing information that can be understood by the end user. So then, the question arises,
More informationAn introduction to Big Data. Presentation by Devesh Sharma, Zubair Asghar & Andreas Aalsaunet
An introduction to Big Data Presentation by Devesh Sharma, Zubair Asghar & Andreas Aalsaunet About the presenters Andreas Oven Aalsaunet Bachelor s Degree in Computer Engineering from HiOA (2011-2014)
More informationHadoop/MapReduce Computing Paradigm
Hadoop/Reduce Computing Paradigm 1 Large-Scale Data Analytics Reduce computing paradigm (E.g., Hadoop) vs. Traditional database systems vs. Database Many enterprises are turning to Hadoop Especially applications
More informationWhat is the maximum file size you have dealt so far? Movies/Files/Streaming video that you have used? What have you observed?
Simple to start What is the maximum file size you have dealt so far? Movies/Files/Streaming video that you have used? What have you observed? What is the maximum download speed you get? Simple computation
More informationIntroduction p. 1 What is the World Wide Web? p. 1 A Brief History of the Web and the Internet p. 2 Web Data Mining p. 4 What is Data Mining? p.
Introduction p. 1 What is the World Wide Web? p. 1 A Brief History of the Web and the Internet p. 2 Web Data Mining p. 4 What is Data Mining? p. 6 What is Web Mining? p. 6 Summary of Chapters p. 8 How
More informationCONSOLIDATING RISK MANAGEMENT AND REGULATORY COMPLIANCE APPLICATIONS USING A UNIFIED DATA PLATFORM
CONSOLIDATING RISK MANAGEMENT AND REGULATORY COMPLIANCE APPLICATIONS USING A UNIFIED PLATFORM Executive Summary Financial institutions have implemented and continue to implement many disparate applications
More informationBig Data Hadoop Developer Course Content. Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours
Big Data Hadoop Developer Course Content Who is the target audience? Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours Complete beginners who want to learn Big Data Hadoop Professionals
More informationComparative Analysis of Range Aggregate Queries In Big Data Environment
Comparative Analysis of Range Aggregate Queries In Big Data Environment Ranjanee S PG Scholar, Dept. of Computer Science and Engineering, Institute of Road and Transport Technology, Erode, TamilNadu, India.
More informationCS 61C: Great Ideas in Computer Architecture. MapReduce
CS 61C: Great Ideas in Computer Architecture MapReduce Guest Lecturer: Justin Hsia 3/06/2013 Spring 2013 Lecture #18 1 Review of Last Lecture Performance latency and throughput Warehouse Scale Computing
More informationApache Spark is a fast and general-purpose engine for large-scale data processing Spark aims at achieving the following goals in the Big data context
1 Apache Spark is a fast and general-purpose engine for large-scale data processing Spark aims at achieving the following goals in the Big data context Generality: diverse workloads, operators, job sizes
More informationLarge-Scale Duplicate Detection
Large-Scale Duplicate Detection Potsdam, April 08, 2013 Felix Naumann, Arvid Heise Outline 2 1 Freedb 2 Seminar Overview 3 Duplicate Detection 4 Map-Reduce 5 Stratosphere 6 Paper Presentation 7 Organizational
More informationParallel Computing: MapReduce Jin, Hai
Parallel Computing: MapReduce Jin, Hai School of Computer Science and Technology Huazhong University of Science and Technology ! MapReduce is a distributed/parallel computing framework introduced by Google
More informationIntroduction to Data Management CSE 344
Introduction to Data Management CSE 344 Lecture 24: MapReduce CSE 344 - Fall 2016 1 HW8 is out Last assignment! Get Amazon credits now (see instructions) Spark with Hadoop Due next wed CSE 344 - Fall 2016
More informationI am: Rana Faisal Munir
Self-tuning BI Systems Home University (UPC): Alberto Abelló and Oscar Romero Host University (TUD): Maik Thiele and Wolfgang Lehner I am: Rana Faisal Munir Research Progress Report (RPR) [1 / 44] Introduction
More informationMap/Reduce. Large Scale Duplicate Detection. Prof. Felix Naumann, Arvid Heise
Map/Reduce Large Scale Duplicate Detection Prof. Felix Naumann, Arvid Heise Agenda 2 Big Data Word Count Example Hadoop Distributed File System Hadoop Map/Reduce Advanced Map/Reduce Stratosphere Agenda
More informationOLTP vs. OLAP Carnegie Mellon Univ. Dept. of Computer Science /615 - DB Applications
OLTP vs. OLAP Carnegie Mellon Univ. Dept. of Computer Science 15-415/615 - DB Applications C. Faloutsos A. Pavlo Lecture#25: OldSQL vs. NoSQL vs. NewSQL On-line Transaction Processing: Short-lived txns.
More informationMapReduce, Hadoop and Spark. Bompotas Agorakis
MapReduce, Hadoop and Spark Bompotas Agorakis Big Data Processing Most of the computations are conceptually straightforward on a single machine but the volume of data is HUGE Need to use many (1.000s)
More informationData Management Glossary
Data Management Glossary A Access path: The route through a system by which data is found, accessed and retrieved Agile methodology: An approach to software development which takes incremental, iterative
More informationArchitectural Solutions for Next Generation Software Systems
Architectural Solutions for Next Generation Software Systems Presenter: Faheem Ullah & Nguyen K. Tran PhD Students Supervisor: M. Ali Babar The Centre for Research on Engineering Software Technologies
More informationDATA SCIENCE USING SPARK: AN INTRODUCTION
DATA SCIENCE USING SPARK: AN INTRODUCTION TOPICS COVERED Introduction to Spark Getting Started with Spark Programming in Spark Data Science with Spark What next? 2 DATA SCIENCE PROCESS Exploratory Data
More informationGlobal Journal of Engineering Science and Research Management
A FUNDAMENTAL CONCEPT OF MAPREDUCE WITH MASSIVE FILES DATASET IN BIG DATA USING HADOOP PSEUDO-DISTRIBUTION MODE K. Srikanth*, P. Venkateswarlu, Ashok Suragala * Department of Information Technology, JNTUK-UCEV
More informationBatch Processing Basic architecture
Batch Processing Basic architecture in big data systems COS 518: Distributed Systems Lecture 10 Andrew Or, Mike Freedman 2 1 2 64GB RAM 32 cores 64GB RAM 32 cores 64GB RAM 32 cores 64GB RAM 32 cores 3
More informationMicrosoft Big Data and Hadoop
Microsoft Big Data and Hadoop Lara Rubbelke @sqlgal Cindy Gross @sqlcindy 2 The world of data is changing The 4Vs of Big Data http://nosql.mypopescu.com/post/9621746531/a-definition-of-big-data 3 Common
More informationMAPREDUCE FOR BIG DATA PROCESSING BASED ON NETWORK TRAFFIC PERFORMANCE Rajeshwari Adrakatti
International Journal of Computer Engineering and Applications, ICCSTAR-2016, Special Issue, May.16 MAPREDUCE FOR BIG DATA PROCESSING BASED ON NETWORK TRAFFIC PERFORMANCE Rajeshwari Adrakatti 1 Department
More informationSOURCERER: MINING AND SEARCHING INTERNET- SCALE SOFTWARE REPOSITORIES
SOURCERER: MINING AND SEARCHING INTERNET- SCALE SOFTWARE REPOSITORIES Introduction to Information Retrieval CS 150 Donald J. Patterson This content based on the paper located here: http://dx.doi.org/10.1007/s10618-008-0118-x
More informationHuge market -- essentially all high performance databases work this way
11/5/2017 Lecture 16 -- Parallel & Distributed Databases Parallel/distributed databases: goal provide exactly the same API (SQL) and abstractions (relational tables), but partition data across a bunch
More informationBig Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2017)
Big Data Infrastructure CS 489/698 Big Data Infrastructure (Winter 2017) Week 10: Mutable State (1/2) March 14, 2017 Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo These
More informationAn InterSystems Guide to the Data Galaxy. Benjamin De Boe Product Manager
An InterSystems Guide to the Data Galaxy Benjamin De Boe Product Manager Analytics 3 InterSystems Corporation. All rights reserved. 4 InterSystems Corporation. All rights reserved. 5 InterSystems Corporation.
More informationPart I: Data Mining Foundations
Table of Contents 1. Introduction 1 1.1. What is the World Wide Web? 1 1.2. A Brief History of the Web and the Internet 2 1.3. Web Data Mining 4 1.3.1. What is Data Mining? 6 1.3.2. What is Web Mining?
More informationBig Data Development CASSANDRA NoSQL Training - Workshop. November 20 to (5 days) 9 am to 5 pm HOTEL DUBAI GRAND DUBAI
Big Data Development CASSANDRA NoSQL Training - Workshop November 20 to 24 2016 (5 days) 9 am to 5 pm HOTEL DUBAI GRAND DUBAI ISIDUS TECH TEAM FZE PO Box 9798 Dubai UAE, email training-coordinator@isidusnet
More informationData Mining. Part 2. Data Understanding and Preparation. 2.4 Data Transformation. Spring Instructor: Dr. Masoud Yaghini. Data Transformation
Data Mining Part 2. Data Understanding and Preparation 2.4 Spring 2010 Instructor: Dr. Masoud Yaghini Outline Introduction Normalization Attribute Construction Aggregation Attribute Subset Selection Discretization
More informationImproving the MapReduce Big Data Processing Framework
Improving the MapReduce Big Data Processing Framework Gistau, Reza Akbarinia, Patrick Valduriez INRIA & LIRMM, Montpellier, France In collaboration with Divyakant Agrawal, UCSB Esther Pacitti, UM2, LIRMM
More informationBIG DATA COURSE CONTENT
BIG DATA COURSE CONTENT [I] Get Started with Big Data Microsoft Professional Orientation: Big Data Duration: 12 hrs Course Content: Introduction Course Introduction Data Fundamentals Introduction to Data
More informationTowards Automatic Optimization of MapReduce Programs (Position Paper) Shivnath Babu Duke University
Towards Automatic Optimization of MapReduce Programs (Position Paper) Shivnath Babu Duke University Roadmap Call to action to improve automatic optimization techniques in MapReduce frameworks Challenges
More informationHierarchy of knowledge BIG DATA 9/7/2017. Architecture
BIG DATA Architecture Hierarchy of knowledge Data: Element (fact, figure, etc.) which is basic information that can be to be based on decisions, reasoning, research and which is treated by the human or
More informationFROM PEER TO PEER...
FROM PEER TO PEER... Dipartimento di Informatica, Università degli Studi di Pisa HPC LAB, ISTI CNR Pisa in collaboration with: Alessandro Lulli, Emanuele Carlini, Massimo Coppola, Patrizio Dazzi 2 nd HPC
More information/ Cloud Computing. Recitation 7 October 10, 2017
15-319 / 15-619 Cloud Computing Recitation 7 October 10, 2017 Overview Last week s reflection Project 3.1 OLI Unit 3 - Module 10, 11, 12 Quiz 5 This week s schedule OLI Unit 3 - Module 13 Quiz 6 Project
More informationTable of Index Hadoop for Developers Hibernate: Using Hibernate For Java Database Access HP FlexNetwork Fundamentals, Rev. 14.21 HP Navigating the Journey to Cloud, Rev. 15.11 HP OneView 1.20 Rev.15.21
More informationStatistical Analysis of Metabolomics Data. Xiuxia Du Department of Bioinformatics & Genomics University of North Carolina at Charlotte
Statistical Analysis of Metabolomics Data Xiuxia Du Department of Bioinformatics & Genomics University of North Carolina at Charlotte Outline Introduction Data pre-treatment 1. Normalization 2. Centering,
More informationIntroduction to Hadoop and MapReduce
Introduction to Hadoop and MapReduce Antonino Virgillito THE CONTRACTOR IS ACTING UNDER A FRAMEWORK CONTRACT CONCLUDED WITH THE COMMISSION Large-scale Computation Traditional solutions for computing large
More informationDistributed Computing.
Distributed Computing at Hai.Thai@rackspace.com About: Me ME About: Me ME 09 Tech grad B.S. Computer Engineering 4 years at rackspace About: Rackspace About: Rackspace Managed + Cloud hosting Cloud Applications:
More informationCassandra, MongoDB, and HBase. Cassandra, MongoDB, and HBase. I have chosen these three due to their recent
Tanton Jeppson CS 401R Lab 3 Cassandra, MongoDB, and HBase Introduction For my report I have chosen to take a deeper look at 3 NoSQL database systems: Cassandra, MongoDB, and HBase. I have chosen these
More informationBig Data Architect.
Big Data Architect www.austech.edu.au WHAT IS BIG DATA ARCHITECT? A big data architecture is designed to handle the ingestion, processing, and analysis of data that is too large or complex for traditional
More informationCIB Session 12th NoSQL Databases Structures
CIB Session 12th NoSQL Databases Structures By: Shahab Safaee & Morteza Zahedi Software Engineering PhD Email: safaee.shx@gmail.com, morteza.zahedi.a@gmail.com cibtrc.ir cibtrc cibtrc 2 Agenda What is
More informationFusion iomemory PCIe Solutions from SanDisk and Sqrll make Accumulo Hypersonic
WHITE PAPER Fusion iomemory PCIe Solutions from SanDisk and Sqrll make Accumulo Hypersonic Western Digital Technologies, Inc. 951 SanDisk Drive, Milpitas, CA 95035 www.sandisk.com Table of Contents Executive
More informationMapReduce Algorithms
Large-scale data processing on the Cloud Lecture 3 MapReduce Algorithms Satish Srirama Some material adapted from slides by Jimmy Lin, 2008 (licensed under Creation Commons Attribution 3.0 License) Outline
More informationDon t Match Twice: Redundancy-free Similarity Computation with MapReduce
Don t Match Twice: Redundancy-free Similarity Computation with MapReduce Lars Kolb kolb@informatik.unileipzig.de Andreas Thor thor@informatik.unileipzig.de Erhard Rahm rahm@informatik.unileipzig.de ABSTRACT
More informationBuilding a Scalable Recommender System with Apache Spark, Apache Kafka and Elasticsearch
Nick Pentreath Nov / 14 / 16 Building a Scalable Recommender System with Apache Spark, Apache Kafka and Elasticsearch About @MLnick Principal Engineer, IBM Apache Spark PMC Focused on machine learning
More informationData Informatics. Seon Ho Kim, Ph.D.
Data Informatics Seon Ho Kim, Ph.D. seonkim@usc.edu HBase HBase is.. A distributed data store that can scale horizontally to 1,000s of commodity servers and petabytes of indexed storage. Designed to operate
More informationCS 655 Advanced Topics in Distributed Systems
Presented by : Walid Budgaga CS 655 Advanced Topics in Distributed Systems Computer Science Department Colorado State University 1 Outline Problem Solution Approaches Comparison Conclusion 2 Problem 3
More informationTutorial Outline. Map/Reduce vs. DBMS. MR vs. DBMS [DeWitt and Stonebraker 2008] Acknowledgements. MR is a step backwards in database access
Map/Reduce vs. DBMS Sharma Chakravarthy Information Technology Laboratory Computer Science and Engineering Department The University of Texas at Arlington, Arlington, TX 76009 Email: sharma@cse.uta.edu
More informationRanked Retrieval. Evaluation in IR. One option is to average the precision scores at discrete. points on the ROC curve But which points?
Ranked Retrieval One option is to average the precision scores at discrete Precision 100% 0% More junk 100% Everything points on the ROC curve But which points? Recall We want to evaluate the system, not
More informationDesign and Implementation of Agricultural Information Resources Vertical Search Engine Based on Nutch
619 A publication of CHEMICAL ENGINEERING TRANSACTIONS VOL. 51, 2016 Guest Editors: Tichun Wang, Hongyang Zhang, Lei Tian Copyright 2016, AIDIC Servizi S.r.l., ISBN 978-88-95608-43-3; ISSN 2283-9216 The
More informationPutting it all together: Creating a Big Data Analytic Workflow with Spotfire
Putting it all together: Creating a Big Data Analytic Workflow with Spotfire Authors: David Katz and Mike Alperin, TIBCO Data Science Team In a previous blog, we showed how ultra-fast visualization of
More informationHow we build TiDB. Max Liu PingCAP Amsterdam, Netherlands October 5, 2016
How we build TiDB Max Liu PingCAP Amsterdam, Netherlands October 5, 2016 About me Infrastructure engineer / CEO of PingCAP Working on open source projects: TiDB: https://github.com/pingcap/tidb TiKV: https://github.com/pingcap/tikv
More informationScience 2.0 VU Big Science, e-science and E- Infrastructures + Bibliometric Network Analysis
W I S S E N n T E C H N I K n L E I D E N S C H A F T Science 2.0 VU Big Science, e-science and E- Infrastructures + Bibliometric Network Analysis Elisabeth Lex KTI, TU Graz WS 2015/16 u www.tugraz.at
More informationG(B)enchmark GraphBench: Towards a Universal Graph Benchmark. Khaled Ammar M. Tamer Özsu
G(B)enchmark GraphBench: Towards a Universal Graph Benchmark Khaled Ammar M. Tamer Özsu Bioinformatics Software Engineering Social Network Gene Co-expression Protein Structure Program Flow Big Graphs o
More informationLarge-Scale Web Applications
Large-Scale Web Applications Mendel Rosenblum Web Application Architecture Web Browser Web Server / Application server Storage System HTTP Internet CS142 Lecture Notes - Intro LAN 2 Large-Scale: Scale-Out
More informationBig Data with Hadoop Ecosystem
Diógenes Pires Big Data with Hadoop Ecosystem Hands-on (HBase, MySql and Hive + Power BI) Internet Live http://www.internetlivestats.com/ Introduction Business Intelligence Business Intelligence Process
More informationMAGE-ML: MicroArray Gene Expression Markup Language
MAGE-ML: MicroArray Gene Expression Markup Language Links: - Full MAGE specification: http://cgi.omg.org/cgi-bin/doc?lifesci/01-10-01 - MAGE-ML Document Type Definition (DTD): http://cgi.omg.org/cgibin/doc?lifesci/01-11-02
More informationCIS 601 Graduate Seminar. Dr. Sunnie S. Chung Dhruv Patel ( ) Kalpesh Sharma ( )
Guide: CIS 601 Graduate Seminar Presented By: Dr. Sunnie S. Chung Dhruv Patel (2652790) Kalpesh Sharma (2660576) Introduction Background Parallel Data Warehouse (PDW) Hive MongoDB Client-side Shared SQL
More informationREVIEW ON BIG DATA ANALYTICS AND HADOOP FRAMEWORK
REVIEW ON BIG DATA ANALYTICS AND HADOOP FRAMEWORK 1 Dr.R.Kousalya, 2 T.Sindhupriya 1 Research Supervisor, Professor & Head, Department of Computer Applications, Dr.N.G.P Arts and Science College, Coimbatore
More informationHadoop & Big Data Analytics Complete Practical & Real-time Training
An ISO Certified Training Institute A Unit of Sequelgate Innovative Technologies Pvt. Ltd. www.sqlschool.com Hadoop & Big Data Analytics Complete Practical & Real-time Training Mode : Instructor Led LIVE
More informationMassive Scalability With InterSystems IRIS Data Platform
Massive Scalability With InterSystems IRIS Data Platform Introduction Faced with the enormous and ever-growing amounts of data being generated in the world today, software architects need to pay special
More information1 Dulcian, Inc., 2001 All rights reserved. Oracle9i Data Warehouse Review. Agenda
Agenda Oracle9i Warehouse Review Dulcian, Inc. Oracle9i Server OLAP Server Analytical SQL Mining ETL Infrastructure 9i Warehouse Builder Oracle 9i Server Overview E-Business Intelligence Platform 9i Server:
More informationAn Introduction to Big Data Formats
Introduction to Big Data Formats 1 An Introduction to Big Data Formats Understanding Avro, Parquet, and ORC WHITE PAPER Introduction to Big Data Formats 2 TABLE OF TABLE OF CONTENTS CONTENTS INTRODUCTION
More informationAn overview of Graph Categories and Graph Primitives
An overview of Graph Categories and Graph Primitives Dino Ienco (dino.ienco@irstea.fr) https://sites.google.com/site/dinoienco/ Topics I m interested in: Graph Database and Graph Data Mining Social Network
More informationBig Data Analytics. Izabela Moise, Evangelos Pournaras, Dirk Helbing
Big Data Analytics Izabela Moise, Evangelos Pournaras, Dirk Helbing Izabela Moise, Evangelos Pournaras, Dirk Helbing 1 Big Data "The world is crazy. But at least it s getting regular analysis." Izabela
More informationTOOLS FOR INTEGRATING BIG DATA IN CLOUD COMPUTING: A STATE OF ART SURVEY
Journal of Analysis and Computation (JAC) (An International Peer Reviewed Journal), www.ijaconline.com, ISSN 0973-2861 International Conference on Emerging Trends in IOT & Machine Learning, 2018 TOOLS
More information1. Introduction to MapReduce
Processing of massive data: MapReduce 1. Introduction to MapReduce 1 Origins: the Problem Google faced the problem of analyzing huge sets of data (order of petabytes) E.g. pagerank, web access logs, etc.
More informationMitigating Data Skew Using Map Reduce Application
Ms. Archana P.M Mitigating Data Skew Using Map Reduce Application Mr. Malathesh S.H 4 th sem, M.Tech (C.S.E) Associate Professor C.S.E Dept. M.S.E.C, V.T.U Bangalore, India archanaanil062@gmail.com M.S.E.C,
More informationBeyond Online Aggregation: Parallel and Incremental Data Mining with MapReduce Joos-Hendrik Böse*, Artur Andrzejak, Mikael Högqvist
Beyond Online Aggregation: Parallel and Incremental Data Mining with MapReduce Joos-Hendrik Böse*, Artur Andrzejak, Mikael Högqvist *ICSI Berkeley Zuse Institut Berlin 4/26/2010 Joos-Hendrik Boese Slide
More information