Automated Debugging In Data Intensive Scalable Computing Systems
|
|
|
- Erica Green
- 5 years ago
- Views:
Transcription
1 Automated Debugging In Data Intensive Scalable Computing Systems Muhammad Ali Gulzar 1, Matteo Interlandi 3, Xueyuan Han 2, Mingda Li 1, Tyson Condie 1, and Miryung Kim 1 1 University of California, Los Angeles 2 Harvard University 3 Mircrosoft 1
2 Big Data Debugging in the Dark Develop locally Hope it works Run in cloud Bug! Guesswork Map Reduce 2
3 Motivating Example Alice writes a Spark program that identifies, for each state in the US, the delta between the minimum and the maximum snowfall reading for each day of any year and for any particular year. Zip Code Date Snow Fall /01/ mm /01/ mm /01/1991 0mm 3
4 Problem Definition Using a test function, a user can specify incorrect results TextFile FlatMap GroupByKey Map Output 99504, 01/01/1992, 1ft 99504, 03/01/1992, 0.1ft 99504, 01/01/1993, 70in 99504, 03/01/1993, 145mm 99504, 01/01/1994, 245mm 99504, 01/01/1993, 85mm 90031, 02/01/1991, 0mm } AK, 01/01, AK, 1992, AK, 03/01, 30.5 AK, 1992, 30.5 AK, 01/01, AK, 01/01, [304.8, 21336, 245, 85] AK, 03/01, [30.5, 145] AK, 1992, [304.8, 30.5] AK, 1993, [21336, 145, 85] AK, 1994, [245] CA, 02/01, [0] CA, 1991, [0] def test(key:string, delta: Float) : Boolean = { AK, 1993, delta < 6000 AK, 03/01, 145 AK, 1993, 145 AK, 01/01, 245 AK, 1994, AK, 01/01, AK, 03/01, AK, 1992, AK, 1993, AK, 1994, 0 CA, 02/01, 0 CA, 1991, 0 Given a test function, the goal is to identify a minimum subset of the input that is able to reproduce the same test failure. 4
5 Existing Approach 1: Data Provenance for Spark TextFile FlatMap GroupByKey Map Output 99504, 01/01/1992, 1ft 99504, 03/01/1992, 0.1ft 99504, 01/01/1993, 70in 99504, 03/01/1993, 145mm 99504, 01/01/1994, 245mm 99504, 01/01/1993, 85mm 90031, 02/01/1991, 0mm AK, 01/01, AK, 1992, AK, 03/01, 30.5 AK, 1992, 30.5 AK, 01/01, AK, 1993, AK, 03/01, 145 AK, 1993, 145 AK, 01/01, 245 AK, 1994, AK, 01/01, [304.8, 21336, 245, 85] AK, 03/01, [30.5, 145] AK, 1992, [304.8, 30.5] AK, 1993, [21336, 145, 85] AK, 1994, [245] CA, 02/01, [0] CA, 1991, [0] AK, 01/01, AK, 03/01, AK, 1992, AK, 1993, AK, 1994, 0 CA, 02/01, 0 CA, 1991, 0 It over-approximates the scope of failure-inducing inputs i.e. records in the faulty key-group are all marked as faulty 5
6 Existing Approach 2: Delta Debugging Delta Debugging performs a systematic binary search-like procedure on the input dataset using a test oracle function TextFile FlatMap GroupByKey Map Output , 01/01/1992, 1ft 99504, 03/01/1992, 0.1ft 99504, 01/01/1993, 70in 99504, 03/01/1993, 145mm 99504, 01/01/1994, 245mm 99504, 01/01/1993, 85mm 90031, 02/01/1991, 0mm AK,01/01, AK,1992, AK,03/01, 30.5 AK,1992, 30.5 AK,01/01, AK,1993, AK,03/01, 145 AK,1993, 145 AK,01/01, 245 AK,1994, AK,01/01, [304.8, 21336, 245, 85] AK,03/01, [30.5, 145] AK,1992, [304.8, 30.5] AK,1993, [21336, 145, 85] AK,1994, [245] CA,02/01, [0] CA,1991, [0] AK, 01/01, AK, 03/01, AK, 1992, AK, 1993, AK, 1994, 0 CA, 02/01, 0 CA, 1991, 0 It does not prune input records known to be irrelevant because of the lack of semantic understanding of data-flow operators 6
7 Existing Approach 2: Delta Debugging Delta Debugging performs a systematic binary-like search on the input dataset using a test oracle function TextFile FlatMap GroupByKey Map Output , 01/01/1992, 1ft 99504, 03/01/1992, 0.1ft 99504, 01/01/1993, 70in AK,01/01, AK,1992, AK,03/01, 30.5 AK,1992, 30.5 AK,01/01, AK,1993, AK, 01/01, [304.8, 21336] AK,03/01, [30.5] AK,1992, [304.8, 30.5] AK,1993, [21336] AK, 01/01, AK, 03/01, 0 AK, 1992, AK, 1993, 0 Run 2 It does not prune input records known to be irrelevant because of the lack of semantic understanding of data-flow operators 7
8 Existing Approach 2: Delta Debugging Delta Debugging performs a systematic binary-like search on the input dataset using a test oracle function TextFile FlatMap GroupByKey Map Output 99504, 01/01/1992, 1ft 99504, 03/01/1992, 0.1ft 99504, 01/01/1993, 70in AK,01/01, AK,1992, AK,03/01, 30.5 AK,1992, 30.5 AK, 01/01, [304.8] AK,03/01, [30.5] AK,1992, [304.8, 30.5] AK, 01/01, 0 AK, 03/01, 0 AK, 1992, Run 3 It does not prune input records known to be irrelevant because of the lack of semantic understanding of data-flow operators 8
9 Existing Approach 2: Delta Debugging Delta Debugging performs a systematic binary-like search on the input dataset using a test oracle function TextFile FlatMap GroupByKey Map Output 99504, 01/01/1992, 1ft 99504, 03/01/1992, 0.1ft 99504, 01/01/1993, 70in AK,01/01, AK,1993, AK, 01/01, [21336] AK,1993, [21336] AK, 01/01, 0 AK, 1993, 0 Run 4 It does not prune input records known to be irrelevant because of the lack of semantic understanding of data-flow operators 9
10 Existing Approach 2: Delta Debugging Delta Debugging performs a systematic binary-like search on the input dataset using a test oracle function TextFile FlatMap GroupByKey Map Output 99504, 01/01/1992, 1ft 99504, 03/01/1992, 0.1ft 99504, 01/01/1993, 70in AK,01/01, AK,1992, AK, 01/01, [304.8] AK,1992, [304.8] AK, 01/01, 0 AK, 1992, 0 Run 5 It does not prune input records known to be irrelevant because of the lack of semantic understanding of data-flow operators 10
11 Existing Approach 2: Delta Debugging Delta Debugging performs a systematic binary-like search on the input dataset using a test oracle function TextFile FlatMap GroupByKey Map Output 99504, 01/01/1992, 1ft 99504, 03/01/1992, 0.1ft 99504, 01/01/1993, 70in AK,03/01, 30.5 AK,1992, 30.5 AK,03/01, [30.5] AK,1992, [30.5] AK, 03/01, 0 AK, 1992, 0 Run 6 It does not prune input records known to be irrelevant because of the lack of semantic understanding of data-flow operators 11
12 Existing Approach 2: Delta Debugging Delta Debugging performs a systematic binary-like search on the input dataset using a test oracle function TextFile FlatMap GroupByKey Map Output 99504, 01/01/1992, 1ft 99504, 03/01/1992, 0.1ft 99504, 01/01/1993, 70in AK,01/01, AK,1993, AK, 01/01, [21336] AK,1993, [21336] AK, 01/01, 0 AK, 1993, 0 Run 7 It does not prune input records known to be irrelevant because of the lack of semantic understanding of data-flow operators 12
13 Existing Approach 2: Delta Debugging Delta Debugging performs a systematic binary-like search on the input dataset using a test oracle function TextFile FlatMap GroupByKey Map Output 99504, 01/01/1992, 1ft 99504, 03/01/1992, 0.1ft 99504, 01/01/1993, 70in AK,03/01, 30.5 AK,1992, 30.5 AK,01/01, AK,1993, AK, 01/01, [21336] AK,03/01, [30.5] AK,1992, [30.5] AK,1993, [21336] AK, 01/01, 0 AK, 03/01, 0 AK, 1992, 0 AK, 1993, 0 Run 8 It does not prune input records known to be irrelevant because of the lack of semantic understanding of data-flow operators 13
14 Existing Approach 2: Delta Debugging Delta Debugging performs a systematic binary-like search on the input dataset using a test oracle function TextFile FlatMap GroupByKey Map Output 99504, 01/01/1992, 1ft 99504, 03/01/1992, 0.1ft 99504, 01/01/1993, 70in AK,01/01, AK,1992, AK,01/01, AK,1993, AK, 01/01, [304.8, 21336] AK,1992, [304.8] AK,1993, [21336] AK, 01/01, AK, 1992, 0 AK, 1993, 0 Run 9 It does not prune input records known to be irrelevant because of the lack of semantic understanding of data-flow operators 14
15 Automated Debugging in DISC with BigSift Input: A Spark Program, A Test Function Output: Minimum Fault-Inducing Input Records Data Provenance + Delta Debugging Test Predicate Pushdown Prioritizing Backward Traces Bitmap based Test Memoization 15
16 Optimization 1: Test Predicate Pushdown Observation: During backward tracing, data provenance traces through all partitions even though only a few partitions contain faulty intermediate data. Test Without Test Pushdown Test Test Test Test Test Test With Test Pushdown If applicable, BigSift pushes down the test function to test the output of combiners in order to isolate the faulty partitions. 16
17 Optimization 2: Prioritizing Backward Traces Observation: The same faulty input record may contribute to multiple faulty output due to operators such as Join or Flatmap TextFile FlatMap GroupByKey Map Output 99504, 01/01/1992, 1ft 99504, 03/01/1992, 0.1ft 99504, 01/01/1993, 70in 99504, 03/01/1993, 145mm 99504, 01/01/1994, 245mm 99504, 01/01/1993, 85mm 90031, 02/01/1991, 0mm AK, 01/01, AK, 1992, AK, 03/01, 30.5 AK, 1992, 30.5 AK, 01/01, AK, 1993, AK, 03/01, 145 AK, 1993, 145 AK, 01/01, 245 AK, 1994, AK, 01/01, [304.8, 21336, 245, 85] AK, 03/01, [30.5, 145] AK, 1992, [304.8, 30.5] AK, 1993, [21336, 145, 85] AK, 1994, [245] CA, 02/01, [0] CA, 1991, [0] AK, 01/01, AK, 03/01, AK, 1992, AK, 1993, AK, 1994, 0 CA, 02/01, 0 CA, 1991, 0 In case of multiple faulty outputs, BigSift overlaps two backward traces to minimize the scope of fault-inducing input records 17
18 Optimization 3: Bitmap Based Test Memoization Observation: Delta debugging may try running a program on the same subset of input redundantly. 0 BigSift leverages bitmap to compactly encode the offsets of original input to refer to an input subset Input Data We use a bitmap based test memoization technique to avoid redundant testing of the same input dataset Bitmap X Test Outcome 18
19 Evaluation Questions RQ1: How much improvement in the debugging time does BigSift provide in comparison to delta debugging? RQ2: How long is the debugging time of BigSift in comparison to original running time of a job? RQ3: How much improvement in the precision of fault-inducing input records does BigSift provide in comparison to data provenance?
20 RQ1: Performance Improvement over Delta Debugging Subject Program Running Time (sec) Debugging Time (sec) Subject Program Fault Original Job DD BigSift Improvement Movie Histogram Code X Inverted Index Code X Rating Histogram Code X Sequence Count Code X Rating Frequency Code X College Student Data X Weather Analysis Data X Transit Analysis Code X BigSift provides up to a 66X speed up in isolating the precise faultinducing input records, in comparison to the baseline DD
21 RQ2: Debugging Time vs. Original Job Time Subject Program Running Time (sec) Debugging Time (sec) Subject Program Fault Original Job DD BigSift Improvement Movie Histogram Code X Inverted Index Code X Rating Histogram Code X Sequence Count Code X Rating Frequency Code X College Student Data X Weather Analysis Data X Transit Analysis Code X On average, BigSift takes 62% less time to debug a single faulty output than the time taken for a single run on the entire data.
22 RQ2: Debugging Time vs. Original Job Time Sequence Count # of fault-inducing input records 1E Fault Localization Time (s) Delta Debugging Test Driven Data Provenance BigSift Data Provenance On average, BigSift takes 62% less time to debug a single faulty output than the time taken for a single run on the entire data.
23 RQ3: Fault Localizability over Data Provenance # of fault-inducing input records Movie Historgram Inverted Index Rating Histogram Sequence Count Rating Frequency College Students Weather Analysis Data Provenance Test Driven Data Provenance BigSift & DD BigSift leverages DD after DP to continue fault isolation, achieving several orders of magnitude 10 3 to 10 7 better precision.
24 Conclusion BigSift is the first piece of work in automated debugging of big data analytics in DISC. BigSift provides 10 3 X 10 7 X more precision than data provenance in terms of fault localizability. It provides up to 66X speed up in debugging time over baseline Delta Debugging. In our evaluation we have observed that, on average, BigSift finds the faulty input in 62% less than the original job execution time.
25 Questions?
Interactive Debugging for Big Data Analytics
Interactive Debugging for Big Data Analytics Muhammad Ali Gulzar, Xueyuan Han, Matteo Interlandi, Shaghayegh Mardani, Sai Deep Tetali, Tyson Condie, Todd Millstein, Miryung Kim University of California,
BigDebug: Interactive Debugger for Big Data Analytics in Apache Spark
BigDebug: Interactive Debugger for Big Data Analytics in Apache Spark Muhammad Ali Gulzar, Matteo Interlandi, Tyson Condie, Miryung Kim University of California, Los Angeles, USA {gulzar, minterlandi,
BigDebug: Interactive Debugger for Big Data Analytics in Apache Spark
BigDebug: Interactive Debugger for Big Data Analytics in Apache Spark Muhammad Ali Gulzar, Matteo Interlandi, Tyson Condie, Miryung Kim University of California, Los Angeles ABSTRACT To process massive
Locating Faults Through Automated Predicate Switching
Locating Faults Through Automated Predicate Switching Authored by Xiangyu Zhang, Neelam Gupta, Rajiv Gupta The University of Arizona ICSE 2006 Presented by Jing Pu Authors 1 Neelam Gupta The University
BigDebug: Debugging Primitives for Interactive Big Data Processing in Spark
2016 IEEE/ACM 38th IEEE International Conference on Software Engineering BigDebug: Debugging Primitives for Interactive Big Data Processing in Spark Muhammad Ali Gulzar, Matteo Interlandi, Seunghyun Yoo,
Adding data provenance support to Apache Spark
The VLDB Journal DOI 10.1007/s00778-017-0474-5 SPECIAL ISSUE PAPER Adding data provenance support to Apache Spark Matteo Interlandi 1 Ari Ekmekji 3 Kshitij Shah 2 Muhammad Ali Gulzar 2 Sai Deep Tetali
Kafka Streams: Hands-on Session A.A. 2017/18
Università degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Kafka Streams: Hands-on Session A.A. 2017/18 Matteo Nardelli Laurea Magistrale in Ingegneria Informatica
(See related materials in textbook.) CSE 435: Software Engineering (slides adapted from Ghezzi et al & Stirewalt
Verification (See related materials in textbook.) Outline What are the goals of verification? What are the main approaches to verification? What kind of assurance do we get through testing? How can testing
Part I: Preliminaries 24
Contents Preface......................................... 15 Acknowledgements................................... 22 Part I: Preliminaries 24 1. Basics of Software Testing 25 1.1. Humans, errors, and testing.............................
CSE 444: Database Internals. Lecture 23 Spark
CSE 444: Database Internals Lecture 23 Spark References Spark is an open source system from Berkeley Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. Matei
Hive and Shark. Amir H. Payberah. Amirkabir University of Technology (Tehran Polytechnic)
Hive and Shark Amir H. Payberah amir@sics.se Amirkabir University of Technology (Tehran Polytechnic) Amir H. Payberah (Tehran Polytechnic) Hive and Shark 1393/8/19 1 / 45 Motivation MapReduce is hard to
Efficient, Scalable, and Provenance-Aware Management of Linked Data
Efficient, Scalable, and Provenance-Aware Management of Linked Data Marcin Wylot 1 Motivation and objectives of the research The proliferation of heterogeneous Linked Data on the Web requires data management
Security and Performance advances with Oracle Big Data SQL
Security and Performance advances with Oracle Big Data SQL Jean-Pierre Dijcks Oracle Redwood Shores, CA, USA Key Words SQL, Oracle, Database, Analytics, Object Store, Files, Big Data, Big Data SQL, Hadoop,
Oracle Big Data Connectors
Oracle Big Data Connectors Oracle Big Data Connectors is a software suite that integrates processing in Apache Hadoop distributions with operations in Oracle Database. It enables the use of Hadoop to process
Automated Documentation Inference to Explain Failed Tests
Automated Documentation Inference to Explain Failed Tests Sai Zhang University of Washington Joint work with: Cheng Zhang, Michael D. Ernst A failed test reveals a potential bug Before bug-fixing, programmers
Good Practices in Parallel and Scientific Software. Gabriel Pedraza Ferreira
Good Practices in Parallel and Scientific Software Gabriel Pedraza Ferreira gpedraza@uis.edu.co Parallel Software Development Research Universities Supercomputing Centers Oil & Gas 2004 Time Present CAE
A Toolbox for Counter-Example Analysis and Optimization
A Toolbox for Counter-Example Analysis and Optimization Alan Mishchenko Niklas Een Robert Brayton Department of EECS, University of California, Berkeley {alanmi, een, brayton}@eecs.berkeley.edu Abstract
Compile-Time Code Generation for Embedded Data-Intensive Query Languages
Compile-Time Code Generation for Embedded Data-Intensive Query Languages Leonidas Fegaras University of Texas at Arlington http://lambda.uta.edu/ Outline Emerging DISC (Data-Intensive Scalable Computing)
Titian: Data Provenance Support in Spark
Titian: Data Provenance Support in Matteo Interlandi Kshitij Shah Sai Deep Tetali Muhammad Ali Gulzar Seunghyun Yoo Miryung Kim Todd Millstein Tyson Condie University of California, Los Angeles ABSTRACT
Resilient Distributed Datasets
Resilient Distributed Datasets A Fault- Tolerant Abstraction for In- Memory Cluster Computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael Franklin,
In this Lecture you will Learn: Testing in Software Development Process. What is Software Testing. Static Testing vs.
In this Lecture you will Learn: Testing in Software Development Process Examine the verification and validation activities in software development process stage by stage Introduce some basic concepts of
Assignment 5: Collaborative Filtering
Assignment 5: Collaborative Filtering Arash Vahdat Fall 2015 Readings You are highly recommended to check the following readings before/while doing this assignment: Slope One Algorithm: https://en.wikipedia.org/wiki/slope_one.
Unifying Big Data Workloads in Apache Spark
Unifying Big Data Workloads in Apache Spark Hossein Falaki @mhfalaki Outline What s Apache Spark Why Unification Evolution of Unification Apache Spark + Databricks Q & A What s Apache Spark What is Apache
Research challenges in data-intensive computing The Stratosphere Project Apache Flink
Research challenges in data-intensive computing The Stratosphere Project Apache Flink Seif Haridi KTH/SICS haridi@kth.se e2e-clouds.org Presented by: Seif Haridi May 2014 Research Areas Data-intensive
Memory Bus Encoding for Low Power: A Tutorial
Memory Bus Encoding for Low Power: A Tutorial Wei-Chung Cheng and Massoud Pedram University of Southern California Department of EE-Systems Los Angeles CA 90089 Outline Background Memory Bus Encoding Techniques
Joint Entity Resolution
Joint Entity Resolution Steven Euijong Whang, Hector Garcia-Molina Computer Science Department, Stanford University 353 Serra Mall, Stanford, CA 94305, USA {swhang, hector}@cs.stanford.edu No Institute
Shark. Hive on Spark. Cliff Engle, Antonio Lupher, Reynold Xin, Matei Zaharia, Michael Franklin, Ion Stoica, Scott Shenker
Shark Hive on Spark Cliff Engle, Antonio Lupher, Reynold Xin, Matei Zaharia, Michael Franklin, Ion Stoica, Scott Shenker Agenda Intro to Spark Apache Hive Shark Shark s Improvements over Hive Demo Alpha
Apache Spark Internals
Apache Spark Internals Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Apache Spark Internals 1 / 80 Acknowledgments & Sources Sources Research papers: https://spark.apache.org/research.html Presentations:
Data Preprocessing Yudho Giri Sucahyo y, Ph.D , CISA
Obj ti Objectives Motivation: Why preprocess the Data? Data Preprocessing Techniques Data Cleaning Data Integration and Transformation Data Reduction Data Preprocessing Lecture 3/DMBI/IKI83403T/MTI/UI
IOTA ARCHITECTURE: DATA VIRTUALIZATION AND PROCESSING MEDIUM DR. KONSTANTIN BOUDNIK DR. ALEXANDRE BOUDNIK
IOTA ARCHITECTURE: DATA VIRTUALIZATION AND PROCESSING MEDIUM DR. KONSTANTIN BOUDNIK DR. ALEXANDRE BOUDNIK DR. KONSTANTIN BOUDNIK DR.KONSTANTIN BOUDNIK EPAM SYSTEMS CHIEF TECHNOLOGIST BIGDATA, OPEN SOURCE
Java Software Solutions for AP Computer Science 3rd Edition, Lewis et al. 2011
A Correlation of AP Computer Science 3rd Edition, Lewis et al. 2011 To the INTRODUCTION This document demonstrates how AP (Advanced Placement) Computer Science, 3rd Edition 2011, Lewis et al. meets the
Data-intensive computing systems
Data-intensive computing systems University of Verona Computer Science Department Damiano Carra Acknowledgements q Credits Part of the course material is based on slides provided by the following authors
Software Testing. Minsoo Ryu. Hanyang University. Real-Time Computing and Communications Lab., Hanyang University
Software Testing Minsoo Ryu Hanyang University Topics covered 1. Testing Goals and Principles 2. Testing Process 3. Testing Strategies Component testing Integration testing Validation/system testing 4.
Spark.jl: Resilient Distributed Datasets in Julia
Spark.jl: Resilient Distributed Datasets in Julia Dennis Wilson, Martín Martínez Rivera, Nicole Power, Tim Mickel [dennisw, martinmr, npower, tmickel]@mit.edu INTRODUCTION 1 Julia is a new high level,
Programming Models MapReduce
Programming Models MapReduce Majd Sakr, Garth Gibson, Greg Ganger, Raja Sambasivan 15-719/18-847b Advanced Cloud Computing Fall 2013 Sep 23, 2013 1 MapReduce In a Nutshell MapReduce incorporates two phases
Bolt: I Know What You Did Last Summer In the Cloud
Bolt: I Know What You Did Last Summer In the Cloud Christina Delimitrou 1 and Christos Kozyrakis 2 1 Cornell University, 2 Stanford University ASPLOS April 12 th 2017 Executive Summary Problem: cloud resource
Sub-millisecond Stateful Stream Querying over Fast-evolving Linked Data
Sub-millisecond Stateful Stream Querying over Fast-evolving Linked Data Yunhao Zhang, Rong Chen, Haibo Chen Institute of Parallel and Distributed Systems (IPADS) Shanghai Jiao Tong University Stream Query
Replaying and Isolating Failing Multi-Object Interactions. Martin Burger Andreas Zeller Saarland University
Replaying and Isolating Failing Multi-Object Interactions Martin Burger Andreas Zeller Saarland University e-mail client written in Java 100,200 LOC ~ 1,600 Java classes 17 developers Actively developed
Motivation. Overview. Scalable Dynamic Analysis for Automated Fault Location and Avoidance. Rajiv Gupta. Program Execution
Motivation Scalable Dynamic Analysis for Automated Fault Location and Avoidance Rajiv Gupta Funded by NSF grants from CPA, CSR, & CRI programs and grants from Microsoft Research Software bugs cost the
Stream Processing Platforms Storm, Spark,.. Batch Processing Platforms MapReduce, SparkSQL, BigQuery, Hive, Cypher,...
Data Ingestion ETL, Distcp, Kafka, OpenRefine, Query & Exploration SQL, Search, Cypher, Stream Processing Platforms Storm, Spark,.. Batch Processing Platforms MapReduce, SparkSQL, BigQuery, Hive, Cypher,...
Generalized Order-By and Group-By. joint work with Min Wang Lipyeow Lim Haixun Wang (IBM) Kevin Chang (UIUC)
Supporting Ranking and Clustering as Generalized Order-By and Group-By Chengkai Li (UIUC) joint work with Min Wang Lipyeow Lim Haixun Wang (IBM) Kevin Chang (UIUC) Boolean database queries (Relational
Dremel: Interactive Analysis of Web- Scale Datasets
Dremel: Interactive Analysis of Web- Scale Datasets S. Melnik, A. Gubarev, J. Long, G. Romer, S. Shivakumar, M. Tolton Google Inc. VLDB 200 Presented by Ke Hong (slide adapted from Melnik s) Outline Problem
Databricks Delta: Bringing Unprecedented Reliability and Performance to Cloud Data Lakes
Databricks Delta: Bringing Unprecedented Reliability and Performance to Cloud Data Lakes AN UNDER THE HOOD LOOK Databricks Delta, a component of the Databricks Unified Analytics Platform*, is a unified
Lecture 19. Delta Debugging Cooperative Bug Isolation. EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim
Lecture 19 Delta Debugging Cooperative Bug Isolation EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim Today s Agenda Presentation: Chris on Cooperative Bug Isolation Quiz on Delta Debugging
Apache Hive for Oracle DBAs. Luís Marques
Apache Hive for Oracle DBAs Luís Marques About me Oracle ACE Alumnus Long time open source supporter Founder of Redglue (www.redglue.eu) works for @redgluept as Lead Data Architect @drune After this talk,
Oasis: An Active Storage Framework for Object Storage Platform
Oasis: An Active Storage Framework for Object Storage Platform Yulai Xie 1, Dan Feng 1, Darrell D. E. Long 2, Yan Li 2 1 School of Computer, Huazhong University of Science and Technology Wuhan National
Query Processing & Optimization
Query Processing & Optimization 1 Roadmap of This Lecture Overview of query processing Measures of Query Cost Selection Operation Sorting Join Operation Other Operations Evaluation of Expressions Introduction
Cloud Computing CS
Cloud Computing CS 15-319 Programming Models- Part III Lecture 6, Feb 1, 2012 Majd F. Sakr and Mohammad Hammoud 1 Today Last session Programming Models- Part II Today s session Programming Models Part
Big Data. Big Data Analyst. Big Data Engineer. Big Data Architect
Big Data Big Data Analyst INTRODUCTION TO BIG DATA ANALYTICS ANALYTICS PROCESSING TECHNIQUES DATA TRANSFORMATION & BATCH PROCESSING REAL TIME (STREAM) DATA PROCESSING Big Data Engineer BIG DATA FOUNDATION
Spark, Shark and Spark Streaming Introduction
Spark, Shark and Spark Streaming Introduction Tushar Kale tusharkale@in.ibm.com June 2015 This Talk Introduction to Shark, Spark and Spark Streaming Architecture Deployment Methodology Performance References
Improving Efficiency and Scalability of Formula-based Debugging
Improving Efficiency and Scalability of Formula-based Debugging Wei Jin and Alessandro Orso Georgia Institute of Technology {weijin orso}@gatech.edu arxiv:1409.1989v1 [cs.se] 6 Sep 2014 Abstract Formula-based
A Relational Platform for Efficient Large-Scale Video Analytics. Yao Lu, Aakanksha Chowdhery, Srikanth Kandula
A Relational Platform for Efficient Large-Scale Video Analytics Yao Lu, Aakanksha Chowdhery, Srikanth Kandula Cameras are ubiquitous; video analysis is a big-data problem Cameras are ubiquitous; video
Technical Sheet NITRODB Time-Series Database
Technical Sheet NITRODB Time-Series Database 10X Performance, 1/10th the Cost INTRODUCTION "#$#!%&''$!! NITRODB is an Apache Spark Based Time Series Database built to store and analyze 100s of terabytes
Testing, Debugging, Program Verification
Testing, Debugging, Program Verification Debugging Programs, Part II Wolfgang Ahrendt & Vladimir Klebanov & Moa Johansson & Gabriele Paganelli 14 November 2012 TDV: Debugging II /GU 2011-11-09 1 / 32 Today
SparkBench: A Comprehensive Spark Benchmarking Suite Characterizing In-memory Data Analytics
SparkBench: A Comprehensive Spark Benchmarking Suite Characterizing In-memory Data Analytics Min LI,, Jian Tan, Yandong Wang, Li Zhang, Valentina Salapura, Alan Bivens IBM TJ Watson Research Center * A
Lecture 18. Delta Debugging-- Yesterday my program worked, it does not. Why? EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim
Lecture 18 Delta Debugging-- Yesterday my program worked, it does not. Why? EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim This Week - Fault Localization Debugging is a process of finding
Oracle Data Integrator 12c: Integration and Administration
Oracle University Contact Us: +34916267792 Oracle Data Integrator 12c: Integration and Administration Duration: 5 Days What you will learn Oracle Data Integrator is a comprehensive data integration platform
Bolt: I Know What You Did Last Summer In the Cloud
Bolt: I Know What You Did Last Summer In the Cloud Christina Delimitrou1 and Christos Kozyrakis2 1Cornell University, 2Stanford University Platform Lab Review February 2018 Executive Summary Problem: cloud
MultiNyx: A Multi-Level Abstraction Framework for Systematic Analysis of Hypervisors. Pedro Fonseca, Xi Wang, Arvind Krishnamurthy
MultiNyx: A Multi-Level Abstraction Framework for Systematic Analysis of Hypervisors Pedro Fonseca, Xi Wang, Arvind Krishnamurthy Hypervisor correctness is critical Hypervisors need to virtualize correctly
Interruptible Tasks: Treating Memory Pressure as Interrupts for Highly Scalable Data-Parallel Programs
Interruptible s: Treating Pressure as Interrupts for Highly Scalable Data-Parallel Programs Lu Fang 1, Khanh Nguyen 1, Guoqing(Harry) Xu 1, Brian Demsky 1, Shan Lu 2 1 University of California, Irvine
CS 406/534 Compiler Construction Putting It All Together
CS 406/534 Compiler Construction Putting It All Together Prof. Li Xu Dept. of Computer Science UMass Lowell Fall 2004 Part of the course lecture notes are based on Prof. Keith Cooper, Prof. Ken Kennedy
Stream Processing Platforms Storm, Spark,.. Batch Processing Platforms MapReduce, SparkSQL, BigQuery, Hive, Cypher,...
Data Ingestion ETL, Distcp, Kafka, OpenRefine, Query & Exploration SQL, Search, Cypher, Stream Processing Platforms Storm, Spark,.. Batch Processing Platforms MapReduce, SparkSQL, BigQuery, Hive, Cypher,...
A Case Study of Automatically Creating Test Suites from Web Application Field Data. Sara Sprenkle, Emily Gibson, Sreedevi Sampath, and Lori Pollock
A Case Study of Automatically Creating Test Suites from Web Application Field Data Sara Sprenkle, Emily Gibson, Sreedevi Sampath, and Lori Pollock Evolving Web Applications Code constantly changing Fix
Software Engineering Fall 2015 (CSC 4350/6350) TR. 5:30 pm 7:15 pm. Rao Casturi 11/10/2015
Software Engineering Fall 2015 (CSC 4350/6350) TR. 5:30 pm 7:15 pm Rao Casturi 11/10/2015 http://cs.gsu.edu/~ncasturi1 Class announcements Final Exam date - Dec 1 st. Final Presentations Dec 3 rd. And
An Introduction to Big Data Formats
Introduction to Big Data Formats 1 An Introduction to Big Data Formats Understanding Avro, Parquet, and ORC WHITE PAPER Introduction to Big Data Formats 2 TABLE OF TABLE OF CONTENTS CONTENTS INTRODUCTION
New Developments in Spark
New Developments in Spark And Rethinking APIs for Big Data Matei Zaharia and many others What is Spark? Unified computing engine for big data apps > Batch, streaming and interactive Collection of high-level
Software Engineering
Software Engineering Lecture 15: Testing and Debugging Debugging Peter Thiemann University of Freiburg, Germany SS 2014 Motivation Debugging is unavoidable and a major economical factor Software bugs cost
TENSORFLOW: LARGE-SCALE MACHINE LEARNING ON HETEROGENEOUS DISTRIBUTED SYSTEMS. by Google Research. presented by Weichen Wang
TENSORFLOW: LARGE-SCALE MACHINE LEARNING ON HETEROGENEOUS DISTRIBUTED SYSTEMS by Google Research presented by Weichen Wang 2016.11.28 OUTLINE Introduction The Programming Model The Implementation Single
Software Engineering Fall 2014
Software Engineering Fall 2014 (CSC 4350/6350) Mon.- Wed. 5:30 pm 7:15 pm ALC : 107 Rao Casturi 11/10/2014 Final Exam date - Dec 10 th? Class announcements Final Presentations Dec 3 rd. And Dec 8 th. Ability
Spark Streaming. Guido Salvaneschi
Spark Streaming Guido Salvaneschi 1 Spark Streaming Framework for large scale stream processing Scales to 100s of nodes Can achieve second scale latencies Integrates with Spark s batch and interactive
Microsoft. [MS20762]: Developing SQL Databases
![Microsoft. [MS20762]: Developing SQL Databases](/thumbs/77/75057809.jpg "Microsoft. [MS20762]: Developing SQL Databases")
[MS20762]: Developing SQL Databases Length : 5 Days Audience(s) : IT Professionals Level : 300 Technology : Microsoft SQL Server Delivery Method : Instructor-led (Classroom) Course Overview This five-day
Performance Optimization for Informatica Data Services ( Hotfix 3)
Performance Optimization for Informatica Data Services (9.5.0-9.6.1 Hotfix 3) 1993-2015 Informatica Corporation. No part of this document may be reproduced or transmitted in any form, by any means (electronic,
Oracle Database In-Memory
Oracle Database In-Memory Mark Weber Principal Sales Consultant November 12, 2014 Row Format Databases vs. Column Format Databases Row SALES Transactions run faster on row format Example: Insert or query
Reminders. Lecture Outline. Lecture Agenda. Lecture Outline. Remember This? COMP10001 Foundations of Computing Testing and Debugging
COMP10001 Foundations of Computing Testing and Debugging Reminders Semester 1, 2017 Tim Baldwin & Egemen Tanin version: 1103, date: March 30, 2017 Grok Worksheets 8 11 due at the end of this week Advanced/revision
Boosting DWH Performance with SQL Server ColumnStore Index
Boosting DWH Performance with SQL Server 2016 ColumnStore Index Thank you to our AWESOME sponsors! Introduction Markus Ehrenmüller-Jensen Business Intelligence Architect markus.ehrenmueller@gmail.com @MEhrenmueller
Isolating Failure Causes. Andreas Zeller
Isolating Failure Causes Andreas Zeller Isolating Causes Actual world Alternate world Test? Mixed world 2 Isolating Causes Actual world Alternate world + 1.0 How can we automate this? Test? Mixed world
Big Data Using Hadoop
IEEE 2016-17 PROJECT LIST(JAVA) Big Data Using Hadoop 17ANSP-BD-001 17ANSP-BD-002 Hadoop Performance Modeling for JobEstimation and Resource Provisioning MapReduce has become a major computing model for
Verification Overview Testing Theory and Principles Testing in Practice. Verification. Miaoqing Huang University of Arkansas 1 / 80
1 / 80 Verification Miaoqing Huang University of Arkansas Outline 1 Verification Overview 2 Testing Theory and Principles Theoretical Foundations of Testing Empirical Testing Principles 3 Testing in Practice
Developing SQL Databases
Course 20762B: Developing SQL Databases Page 1 of 9 Developing SQL Databases Course 20762B: 4 days; Instructor-Led Introduction This four-day instructor-led course provides students with the knowledge
StackMine Performance Debugging in the Large via Mining Millions of Stack Traces
StackMine Performance Debugging in the Large via Mining Millions of Stack Traces Shi Han 1, Yingnong Dang 1, Song Ge 1, Dongmei Zhang 1, and Tao Xie 2 Software Analytics Group, Microsoft Research Asia
COMP10001 Foundations of Computing Testing and Debugging
COMP10001 Foundations of Computing Testing and Debugging Semester 1, 2017 Tim Baldwin & Egemen Tanin version: 1103, date: March 30, 2017 2017 The University of Melbourne Reminders Grok Worksheets 8 11
Lecture 20. Delta Debugging Regression Testing. EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim
Lecture 20 Delta Debugging Regression Testing EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim Today s Agenda Regression Testing Delta Debugging: Presentation by Divya (advocate) Presentation
Dremel: Interac-ve Analysis of Web- Scale Datasets
Dremel: Interac-ve Analysis of Web- Scale Datasets Google Inc VLDB 2010 presented by Arka BhaEacharya some slides adapted from various Dremel presenta-ons on the internet The Problem: Interactive data
Shark: SQL and Rich Analytics at Scale. Michael Xueyuan Han Ronny Hajoon Ko
Shark: SQL and Rich Analytics at Scale Michael Xueyuan Han Ronny Hajoon Ko What Are The Problems? Data volumes are expanding dramatically Why Is It Hard? Needs to scale out Managing hundreds of machines
CloudExpo November 2017 Tomer Levi
CloudExpo November 2017 Tomer Levi About me Full Stack Engineer @ Intel s Advanced Analytics group. Artificial Intelligence unit at Intel. Responsible for (1) Radical improvement of critical processes
Most real programs operate somewhere between task and data parallelism. Our solution also lies in this set.
for Windows Azure and HPC Cluster 1. Introduction In parallel computing systems computations are executed simultaneously, wholly or in part. This approach is based on the partitioning of a big task into
Testing: Test design and testing process
Testing: Test design and testing process Zoltán Micskei Based on István Majzik s slides Dept. of Measurement and Information Systems Budapest University of Technology and Economics Department of Measurement
Software Engineering
Software Engineering Lecture 12: Testing and Debugging Debugging Peter Thiemann University of Freiburg, Germany 13.06.2013 Today s Topic Last Lecture Bug tracking Program control Design for Debugging Input
CLUSTERING BIG DATA USING NORMALIZATION BASED k-means ALGORITHM
Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology ISSN 2320 088X IMPACT FACTOR: 5.258 IJCSMC,
Improving the Efficiency of Fast Using Semantic Similarity Algorithm
International Journal of Scientific and Research Publications, Volume 4, Issue 1, January 2014 1 Improving the Efficiency of Fast Using Semantic Similarity Algorithm D.KARTHIKA 1, S. DIVAKAR 2 Final year
In-Memory Data Management
In-Memory Data Management Martin Faust Research Assistant Research Group of Prof. Hasso Plattner Hasso Plattner Institute for Software Engineering University of Potsdam Agenda 2 1. Changed Hardware 2.
COSC 6339 Big Data Analytics. Hadoop MapReduce Infrastructure: Pig, Hive, and Mahout. Edgar Gabriel Fall Pig
COSC 6339 Big Data Analytics Hadoop MapReduce Infrastructure: Pig, Hive, and Mahout Edgar Gabriel Fall 2018 Pig Pig is a platform for analyzing large data sets abstraction on top of Hadoop Provides high
From Single Purpose to Multi Purpose Data Lakes. Thomas Niewel Technical Sales Director DACH Denodo Technologies March, 2019
From Single Purpose to Multi Purpose Data Lakes Thomas Niewel Technical Sales Director DACH Denodo Technologies March, 2019 Agenda Data Lakes Multiple Purpose Data Lakes Customer Example Demo Takeaways
Forward Chaining Reasoning Tool for Rya
Forward Chaining Reasoning Tool for Rya Rya Working Group, 6/29/2016 Forward Chaining Reasoning Tool for Rya 6/29/2016 1 / 11 OWL Reasoning OWL (the Web Ontology Language) facilitates rich ontology definition
SCOPE: A Stochastic Computing Engine for DRAM-based In-situ Accelerator
SCOPE: A Stochastic Computing Engine for DRAM-based In-situ Accelerator Shuangchen Li, Alvin Oliver Glova, Xing Hu, Peng Gu, Dimin Niu*, Krishna T. Malladi*, Hongzhong Zheng*, Bob Brennan*, and Yuan Xie
Using the SDACK Architecture to Build a Big Data Product. Yu-hsin Yeh (Evans Ye) Apache Big Data NA 2016 Vancouver
Using the SDACK Architecture to Build a Big Data Product Yu-hsin Yeh (Evans Ye) Apache Big Data NA 2016 Vancouver Outline A Threat Analytic Big Data product The SDACK Architecture Akka Streams and data
Lecture 21. Regression Testing Path Spectra. EE 382V Spring 2009 Software Evolution - Instructor Miryung Kim
Lecture 21 Regression Testing Path Spectra Today s Agenda (1) Regression Test Selection Path Spectra Presentation by David (skeptic) Presentation by Sidd (advocate) Presentation by Srinivas (skeptic) Today
Oracle Big Data SQL. Release 3.2. Rich SQL Processing on All Data
Oracle Big Data SQL Release 3.2 The unprecedented explosion in data that can be made useful to enterprises from the Internet of Things, to the social streams of global customer bases has created a tremendous
Apache Flink Big Data Stream Processing
Apache Flink Big Data Stream Processing Tilmann Rabl Berlin Big Data Center www.dima.tu-berlin.de bbdc.berlin rabl@tu-berlin.de XLDB 11.10.2017 1 2013 Berlin Big Data Center All Rights Reserved DIMA 2017
Shark: Hive (SQL) on Spark
Shark: Hive (SQL) on Spark Reynold Xin UC Berkeley AMP Camp Aug 29, 2013 UC BERKELEY Stage 0:M ap-shuffle-reduce M apper(row ) { fields = row.split("\t") em it(fields[0],fields[1]); } Reducer(key,values)