Massive Data Analysis

Size: px
Start display at page:

Download "Massive Data Analysis"

Transcription

1 Professor, Department of Electrical and Computer Engineering Tennessee Technological University February 25, 2015

2 Big Data This talk is based on the report [1]. The growth of big data is changing that paradigm, especially in cases in which massive amounts of data are distributed across locations. We expect to see the emergence of a new class of engineers whose skill is the management of such platforms in the context of the solution of real-world problems. We embrace complexity and make use of the best ally we have: the unreasonable effectiveness of data [2].

3 Inference Inference is the problem of turning data into knowledge, where knowledge often is expressed in terms of entities that are not present in the data per se but are present in models that one uses to interpret the data. Statistical rigor is necessary to justify the inferential leap from data to knowledge, and many difficulties arise in attempting to bring statistical principles to bear on massive data. Computer scientists involved in building big-data systems must develop a deeper awareness of inferential issues, while statisticians must concern themselves with scalability, algorithmic issues, and real-time decision-making. Mathematicians also have important roles to play, because areas such as applied linear algebra and optimization theory (already contributing to large-scale data analysis) are likely to continue to grow in importance.

4 (1) Massive data analysis is not the province of any one field, but is rather a thoroughly interdisciplinary enterprise. Solutions to massive data problems will require an intimate blending of ideas from computer science and statistics, with essential contributions also needed from applied and pure mathematics, from optimization theory, and from various engineering areas, notably signal processing and information theory. In general, by bringing interdisciplinary perspectives to bear on massive data analysis, it will be possible to discuss trade-offs that arise when one jointly considers the computational, statistical, scientific, and human-centric constraints that frame a problem.

5 (2) One might hope that general, standardized procedures might emerge that can be used as a default for any massive data set, in much the way that the Fast Fourier Transform is a default procedure in classical signal processing. We are pessimistic that such procedures exist in general. Nevertheless, some useful general procedures and pipelines will surely emerge. We emphasize the need for flexibility and for tools that are sensitive to the overall goals of an analysis. The underlying intellectual issue is that of finding general laws that are applicable at a variety of scales, or ideally, that are scale-free. We want to link measures of inferential accuracy with intrinsic characteristics of the data-generating process and with computational resources such as time, space, and energy. Perhaps these principles can be uncovered once and for all, such that each successive generation of researchers does not

6 Scaling the Infrastructure for Data Management Most work in the past has focused on a setting that involves a single processor with the entire data set fitting in random access memory (RAM). The streaming setting, in which data arrive in quick succession, and only a subset can be stored; The disk-based setting, in which the data are too large to store in RAM but fit on one machine s disk; The distributed setting, in which the data are distributed over multiple machines RAMs or disks; and The multi-threaded setting, in which the data lie on one machine having multiple processors that share RAM.

7 Temporal Data and Real-Time Algorithms (1) Distributed and parallel data. Many data sources operate in real time, producing data streams that can overwhelm data analysis pipelines. Moreover, there is often a desire to make decisions rapidly, perhaps also in real time. There is a need for further dialog between statistical and computational researchers. Statistical research has rarely considered constraints due to real-time decision-making in the development of data analysis algorithms, and computational research has rarely considered the computational complexity of algorithms for managing statistical risk.

8 Temporal Data and Real-Time Algorithms (2) Time imposes on tasks such as data acquisition, processing, representation, and inference. The initial phase of a temporal data analysis system is the acquisition stage. While in some cases the data are collected and analyzed in one location, many systems consist of a low-level distributed acquisition mechanism. The data from the distributed sources must generally be collected into one or more data analysis centers using a real-time, reliable data feeds management system. Such systems use logging to ensure that all data get delivered, triggers to ensure timely data delivery and ingestion, and intelligent scheduling for efficient processing.

9 DATA PROCESSING, REPRESENTATION, AND INFERENCE The next stage in time-aware data analysis includes building an abstract representation of the data and then using it for inference. Methods for abstract data representation include coding and sketching. To cope with the computational needs that real-time and long-range temporal queries impose, analytic tools for summarizing temporal data streams are a must. A common and very effective summarization tool is called sketching.

10 Data-stream algorithms When the input data rate exceeds the computing capabilities of online learning and prediction algorithms, one needs to resort to methods that provide approximate representations. Data-stream algorithms provide temporal tools for representing and processing input data that come at a very high rate. The high-rate input stresses the communication, storage, and computing infrastructure to the point that it is difficult, if not impossible, to transmit the entire input, compute complex functions over large portions of the input stream, and store and capture temporally the entire input stream. The fusion of stream approaches with efficient statistical inference for general models remains a major research challenge. This fusion poses significant challenges because state-of-the-art learning algorithms are not designed to cope with partial summaries and snapshots of temporal data.

11 Temporal Data and Real-Time Algorithms: CHALLENGES Design and implementation of new representation algorithms and methods for perpetually growing, non-stationary massive data, especially in conjunction with learning and modeling. Although sketching algorithms for streaming data naturally incorporate changes in the data streams, they do not necessarily give an easy and straightforward method for adjusting and updating models and inferences derived from these sketches over time. Current algorithms permit efficient model-building but do not efficiently change the models over time. Furthermore, there is not a natural way to identify or to detect model changes in a streaming setting, perhaps with limited data. The current algorithms for updating network metrics permit efficient calculation only for certain network structures. David P. Woodruff, Sketching as a Tool for Numerical Linear Algebra, November 18, [3]

12 Temporal Data and Real-Time Algorithms: CHALLENGES Streaming and sketching algorithms that leverage new architectures, such as flash memory and terascale storage devices. Distributed real-time acquisition, storage, and transmission of temporal data.

13 Large Scale Data Representation A major missing ingredient appears to be agreement on a notion of middleware, which would connect high-level analysis goals to implementations at the level of hardware and software platforms. The general goal of such middleware is to provide a notion of reuse, whereby a relatively small set of computational modules can be optimized and exploited in a wide variety of algorithms and analyses.

14 Building Models from Massive Data (1) The general goal of data analysis is to acquire knowledge from data. Statistical models provide a convenient framework for achieving this. Models make it possible to identify relationships between variables and to understand how variables, working on their own and together, influence an overall system. They also allow one to make predictions and assess their uncertainty. Statistical models are usually presented as a family of equations (mathematical formulas) that describe how some or all aspects of the data might have been generated. Typically these equations describe (conditional) probability distributions, which can often be separated into a systematic component and a noise component.

15 Building Models from Massive Data (2) Data-analytic models are rarely purely deterministic they typically include a component that allows for unexplained variation or noise. This noise is usually specified in terms of random variables, that is, variables whose values are not known but are generated from some probability distribution. Statistical modeling represents a powerful approach for understanding and analyzing data Modern datasets can be thought of as large random matrices. We do not make a sharp distinction between statistics and machine learning and believes that any attempt to do so is becoming increasingly difficult. The Frequentist View The Bayesian View Nonparametrics Loss Functions and Partially Specified Models

16 Building Models from Massive Data (3) DATA CLEANING. Real-world data are corrupted with noise. Such noise can be either systematic (i.e., having a bias) or random (stochastic). Measurement processes are inherently noisy, data can be recorded with error, and parts of the data may be missing. Data analysts build models for two basic reasons: to understand the past and to predict the future. Unsupervised Learning. One would like to understand how the data were generated, the relationships between variables, and any special structure that may exist in the data. Supervised learning. A more focused task is to build a prediction model, which allows one to predict the future value of a target variable as a function of the other variables at ones disposal, and/or at a future time.

17 Sampling and Massive Data: Random Sampling Sampling is the process of collecting some data when collecting it all or analyzing it all is unreasonable. Sampling may also be adaptive or sequential, so that the sampling rule changes according to a function of the observations taken so far. The goal is to over-sample regions with interesting data or more-variable data to ensure that the estimates in those regions are reliable. Randomly sampling a data stream may mean producing a set of observations such that, at any time, all observations that have occurred so far are equally likely to appear in the sample.

18 Sparse Signal Recovery In sparse signal recovery, a data vector x is multiplied by a matrix M, and only the resulting y = Mx is retained. If x has only a few large components, then the matrix M can have considerably fewer rows than columns, so the vector y of observations is much smaller than the original data set without any loss of information. This kind of data reduction and signal recovery is known as compressed sensing, and it arises in many applications, including analog-to-digital conversion, medical imaging, and hyperspectral imaging.

19 Sampling for Testing Rather Than Estimation We often focus on sampling for estimation, but it is also common to sample in order to test whether a set of factors and their interactions affect an outcome of interest. Hypothesis testing. Sequential testing and adaptive testing (e.g., multi-armed bandits) are also common in some areas, such as in medicine. Randomization and bootstrapped tests are nearly optimal in finite samples under weak conditions.

20 Data from Social Networks and Graphs Power grid can be modeled as a random graph. Sampling no matter how fine its resolution on graphs can never preserve all graph structure because sampling never preserves everything. Sampling designs that account for the network topology are needed.

21 Data from the Physical Sciences The emergence of very large mosaic cameras in astronomy has created large surveys of sky images that contain 500 million objects today and soon will have billions. As scientists query large databases, many of the questions they ask are about computing a statistical aggregate and its uncertainty. As each data point has its own errors, and statistical errors are often small compared to the known and unknown systematic uncertainties, using the whole data set to decrease statistical errors makes no sense.

22 Next Week David P. Woodruff, Sketching as a Tool for Numerical Linear Algebra, November 18, [3]

23 Thank you

24 N. R. Council, Frontiers in massive data analysis. The National Academies Press, A. Halevy, P. Norvig, and F. Pereira, The unreasonable effectiveness of data, Intelligent Systems, IEEE, vol. 24, no. 2, pp. 8 12, D. P. Woodruff, Sketching as a tool for numerical linear algebra, arxiv preprint arxiv: , 2014.

Introduction to Data Science

Introduction to Data Science UNIT I INTRODUCTION TO DATA SCIENCE Syllabus Introduction of Data Science Basic Data Analytics using R R Graphical User Interfaces Data Import and Export Attribute and Data Types Descriptive Statistics

More information

BUYING SERVER HARDWARE FOR A SCALABLE VIRTUAL INFRASTRUCTURE

BUYING SERVER HARDWARE FOR A SCALABLE VIRTUAL INFRASTRUCTURE E-Guide BUYING SERVER HARDWARE FOR A SCALABLE VIRTUAL INFRASTRUCTURE SearchServer Virtualization P art 1 of this series explores how trends in buying server hardware have been influenced by the scale-up

More information

An Introduction to Big Data Formats

An Introduction to Big Data Formats Introduction to Big Data Formats 1 An Introduction to Big Data Formats Understanding Avro, Parquet, and ORC WHITE PAPER Introduction to Big Data Formats 2 TABLE OF TABLE OF CONTENTS CONTENTS INTRODUCTION

More information

Survey of the Mathematics of Big Data

Survey of the Mathematics of Big Data Survey of the Mathematics of Big Data Issues with Big Data, Mathematics to the Rescue Philippe B. Laval KSU Fall 2015 Philippe B. Laval (KSU) Math & Big Data Fall 2015 1 / 28 Introduction We survey some

More information

Massive Scalability With InterSystems IRIS Data Platform

Massive Scalability With InterSystems IRIS Data Platform Massive Scalability With InterSystems IRIS Data Platform Introduction Faced with the enormous and ever-growing amounts of data being generated in the world today, software architects need to pay special

More information

Computational Statistics and Mathematics for Cyber Security

Computational Statistics and Mathematics for Cyber Security and Mathematics for Cyber Security David J. Marchette Sept, 0 Acknowledgment: This work funded in part by the NSWC In-House Laboratory Independent Research (ILIR) program. NSWCDD-PN--00 Topics NSWCDD-PN--00

More information

Strategic Briefing Paper Big Data

Strategic Briefing Paper Big Data Strategic Briefing Paper Big Data The promise of Big Data is improved competitiveness, reduced cost and minimized risk by taking better decisions. This requires affordable solution architectures which

More information

Modernize Your Infrastructure

Modernize Your Infrastructure Modernize Your Infrastructure The next industrial revolution 3 Dell - Internal Use - Confidential It s a journey. 4 Dell - Internal Use - Confidential Last 15 years IT-centric Systems of record Traditional

More information

Computer-based Tracking Protocols: Improving Communication between Databases

Computer-based Tracking Protocols: Improving Communication between Databases Computer-based Tracking Protocols: Improving Communication between Databases Amol Deshpande Database Group Department of Computer Science University of Maryland Overview Food tracking and traceability

More information

Modeling Plant Succession with Markov Matrices

Modeling Plant Succession with Markov Matrices Modeling Plant Succession with Markov Matrices 1 Modeling Plant Succession with Markov Matrices Concluding Paper Undergraduate Biology and Math Training Program New Jersey Institute of Technology Catherine

More information

Computational performance and scalability of large distributed enterprise-wide systems supporting engineering, manufacturing and business applications

Computational performance and scalability of large distributed enterprise-wide systems supporting engineering, manufacturing and business applications Computational performance and scalability of large distributed enterprise-wide systems supporting engineering, manufacturing and business applications Janusz S. Kowalik Mathematics and Computing Technology

More information

Pervasive PSQL Summit v10 Highlights Performance and analytics

Pervasive PSQL Summit v10 Highlights Performance and analytics Pervasive PSQL Summit v10 Highlights Performance and analytics A Monash Information Services Bulletin by Curt A. Monash, PhD. September, 2007 Sponsored by: Pervasive PSQL Version 10 Highlights Page 2 PSQL

More information

Computational and Statistical Tradeoffs in VoI-Driven Learning

Computational and Statistical Tradeoffs in VoI-Driven Learning ARO ARO MURI MURI on on Value-centered Theory for for Adaptive Learning, Inference, Tracking, and and Exploitation Computational and Statistical Tradefs in VoI-Driven Learning Co-PI Michael Jordan University

More information

The Definitive Guide to Preparing Your Data for Tableau

The Definitive Guide to Preparing Your Data for Tableau The Definitive Guide to Preparing Your Data for Tableau Speed Your Time to Visualization If you re like most data analysts today, creating rich visualizations of your data is a critical step in the analytic

More information

Record Linkage using Probabilistic Methods and Data Mining Techniques

Record Linkage using Probabilistic Methods and Data Mining Techniques Doi:10.5901/mjss.2017.v8n3p203 Abstract Record Linkage using Probabilistic Methods and Data Mining Techniques Ogerta Elezaj Faculty of Economy, University of Tirana Gloria Tuxhari Faculty of Economy, University

More information

Efficient, Scalable, and Provenance-Aware Management of Linked Data

Efficient, Scalable, and Provenance-Aware Management of Linked Data Efficient, Scalable, and Provenance-Aware Management of Linked Data Marcin Wylot 1 Motivation and objectives of the research The proliferation of heterogeneous Linked Data on the Web requires data management

More information

FMA901F: Machine Learning Lecture 3: Linear Models for Regression. Cristian Sminchisescu

FMA901F: Machine Learning Lecture 3: Linear Models for Regression. Cristian Sminchisescu FMA901F: Machine Learning Lecture 3: Linear Models for Regression Cristian Sminchisescu Machine Learning: Frequentist vs. Bayesian In the frequentist setting, we seek a fixed parameter (vector), with value(s)

More information

Databricks Delta: Bringing Unprecedented Reliability and Performance to Cloud Data Lakes

Databricks Delta: Bringing Unprecedented Reliability and Performance to Cloud Data Lakes Databricks Delta: Bringing Unprecedented Reliability and Performance to Cloud Data Lakes AN UNDER THE HOOD LOOK Databricks Delta, a component of the Databricks Unified Analytics Platform*, is a unified

More information

Technical Brief: Domain Risk Score Proactively uncover threats using DNS and data science

Technical Brief: Domain Risk Score Proactively uncover threats using DNS and data science Technical Brief: Domain Risk Score Proactively uncover threats using DNS and data science 310 Million + Current Domain Names 11 Billion+ Historical Domain Profiles 5 Million+ New Domain Profiles Daily

More information

Lab 9. Julia Janicki. Introduction

Lab 9. Julia Janicki. Introduction Lab 9 Julia Janicki Introduction My goal for this project is to map a general land cover in the area of Alexandria in Egypt using supervised classification, specifically the Maximum Likelihood and Support

More information

Active Archive and the State of the Industry

Active Archive and the State of the Industry Active Archive and the State of the Industry Taking Data Archiving to the Next Level Abstract This report describes the state of the active archive market. New Applications Fuel Digital Archive Market

More information

Enhanced Performance of Search Engine with Multitype Feature Co-Selection of Db-scan Clustering Algorithm

Enhanced Performance of Search Engine with Multitype Feature Co-Selection of Db-scan Clustering Algorithm Enhanced Performance of Search Engine with Multitype Feature Co-Selection of Db-scan Clustering Algorithm K.Parimala, Assistant Professor, MCA Department, NMS.S.Vellaichamy Nadar College, Madurai, Dr.V.Palanisamy,

More information

OPERATIONALIZING MACHINE LEARNING USING GPU ACCELERATED, IN-DATABASE ANALYTICS

OPERATIONALIZING MACHINE LEARNING USING GPU ACCELERATED, IN-DATABASE ANALYTICS OPERATIONALIZING MACHINE LEARNING USING GPU ACCELERATED, IN-DATABASE ANALYTICS 1 Why GPUs? A Tale of Numbers 100x Performance Increase Infrastructure Cost Savings Performance 100x gains over traditional

More information

Based on Big Data: Hype or Hallelujah? by Elena Baralis

Based on Big Data: Hype or Hallelujah? by Elena Baralis Based on Big Data: Hype or Hallelujah? by Elena Baralis http://dbdmg.polito.it/wordpress/wp-content/uploads/2010/12/bigdata_2015_2x.pdf 1 3 February 2010 Google detected flu outbreak two weeks ahead of

More information

High Performance Computing

High Performance Computing The Need for Parallelism High Performance Computing David McCaughan, HPC Analyst SHARCNET, University of Guelph dbm@sharcnet.ca Scientific investigation traditionally takes two forms theoretical empirical

More information

Ultra-low power wireless sensor networks: distributed signal processing and dynamic resources management

Ultra-low power wireless sensor networks: distributed signal processing and dynamic resources management Ultra-low power wireless sensor networks: distributed signal processing and dynamic resources management Candidate: Carlo Caione Tutor: Prof. Luca Benini Compressive Sensing The issue of data gathering

More information

CE4031 and CZ4031 Database System Principles

CE4031 and CZ4031 Database System Principles CE4031 and CZ4031 Database System Principles Academic AY1819 Semester 1 CE/CZ4031 Database System Principles s CE/CZ2001 Algorithms; CZ2007 Introduction to Databases CZ4033 Advanced Data Management (not

More information

How to Create, Deploy, & Operate Secure IoT Applications

How to Create, Deploy, & Operate Secure IoT Applications How to Create, Deploy, & Operate Secure IoT Applications TELIT WHITEPAPER INTRODUCTION As IoT deployments accelerate, an area of growing concern is security. The likelihood of billions of additional connections

More information

BACKUP TO THE FUTURE A SPICEWORKS SURVEY

BACKUP TO THE FUTURE A SPICEWORKS SURVEY BACKUP TO THE FUTURE A SPICEWORKS SURVEY 02 BACKUP TO THE FUTURE A SPICEWORKS SURVEY METHODOLOGY This research study was conducted by Spiceworks, the professional network for the IT industry, from a survey

More information

LOW LATENCY DATA DISTRIBUTION IN CAPITAL MARKETS: GETTING IT RIGHT

LOW LATENCY DATA DISTRIBUTION IN CAPITAL MARKETS: GETTING IT RIGHT LOW LATENCY DATA DISTRIBUTION IN CAPITAL MARKETS: GETTING IT RIGHT PATRICK KUSTER Head of Business Development, Enterprise Capabilities, Thomson Reuters +358 (40) 840 7788; patrick.kuster@thomsonreuters.com

More information

Hierarchy of knowledge BIG DATA 9/7/2017. Architecture

Hierarchy of knowledge BIG DATA 9/7/2017. Architecture BIG DATA Architecture Hierarchy of knowledge Data: Element (fact, figure, etc.) which is basic information that can be to be based on decisions, reasoning, research and which is treated by the human or

More information

Detection and Deletion of Outliers from Large Datasets

Detection and Deletion of Outliers from Large Datasets Detection and Deletion of Outliers from Large Datasets Nithya.Jayaprakash 1, Ms. Caroline Mary 2 M. tech Student, Dept of Computer Science, Mohandas College of Engineering and Technology, India 1 Assistant

More information

CE4031 and CZ4031 Database System Principles

CE4031 and CZ4031 Database System Principles CE431 and CZ431 Database System Principles Course CE/CZ431 Course Database System Principles CE/CZ21 Algorithms; CZ27 Introduction to Databases CZ433 Advanced Data Management (not offered currently) Lectures

More information

Analytics Fundamentals by Mark Peco

Analytics Fundamentals by Mark Peco Analytics Fundamentals by Mark Peco All rights reserved. Reproduction in whole or part prohibited except by written permission. Product and company names mentioned herein may be trademarks of their respective

More information

Video AI Alerts An Artificial Intelligence-Based Approach to Anomaly Detection and Root Cause Analysis for OTT Video Publishers

Video AI Alerts An Artificial Intelligence-Based Approach to Anomaly Detection and Root Cause Analysis for OTT Video Publishers Video AI Alerts An Artificial Intelligence-Based Approach to Anomaly Detection and Root Cause Analysis for OTT Video Publishers Live and on-demand programming delivered by over-the-top (OTT) will soon

More information

DISTRIBUTION STATEMENT A Approved for public release: distribution unlimited.

DISTRIBUTION STATEMENT A Approved for public release: distribution unlimited. AVIA Test Selection through Spatial Variance Bounding Method for Autonomy Under Test By Miles Thompson Senior Research Engineer Aerospace, Transportation, and Advanced Systems Lab DISTRIBUTION STATEMENT

More information

When, Where & Why to Use NoSQL?

When, Where & Why to Use NoSQL? When, Where & Why to Use NoSQL? 1 Big data is becoming a big challenge for enterprises. Many organizations have built environments for transactional data with Relational Database Management Systems (RDBMS),

More information

Data Model Considerations for Radar Systems

Data Model Considerations for Radar Systems WHITEPAPER Data Model Considerations for Radar Systems Executive Summary The market demands that today s radar systems be designed to keep up with a rapidly changing threat environment, adapt to new technologies,

More information

IP Video Network Gateway Solutions

IP Video Network Gateway Solutions IP Video Network Gateway Solutions INTRODUCTION The broadcast systems of today exist in two separate and largely disconnected worlds: a network-based world where audio/video information is stored and passed

More information

Causal Models for Scientific Discovery

Causal Models for Scientific Discovery Causal Models for Scientific Discovery Research Challenges and Opportunities David Jensen College of Information and Computer Sciences Computational Social Science Institute Center for Data Science University

More information

ECE 8201: Low-dimensional Signal Models for High-dimensional Data Analysis

ECE 8201: Low-dimensional Signal Models for High-dimensional Data Analysis ECE 8201: Low-dimensional Signal Models for High-dimensional Data Analysis Yuejie Chi Departments of ECE and BMI The Ohio State University September 24, 2015 Time, location, and office hours Time: Tue/Thu

More information

COMP 465 Special Topics: Data Mining

COMP 465 Special Topics: Data Mining COMP 465 Special Topics: Data Mining Introduction & Course Overview 1 Course Page & Class Schedule http://cs.rhodes.edu/welshc/comp465_s15/ What s there? Course info Course schedule Lecture media (slides,

More information

Topic 01. Software Engineering, Web Engineering, agile methodologies.

Topic 01. Software Engineering, Web Engineering, agile methodologies. Topic 01 Software Engineering, Web Engineering, agile methodologies. 1 What is Software Engineering? 2 1 Classic Software Engineering The IEEE definition: Software Engineering is the application of a disciplined,

More information

Why Machine Learning is More Likely to Cure Cancer Than to Stop Malware WHITE PAPER

Why Machine Learning is More Likely to Cure Cancer Than to Stop Malware WHITE PAPER Why Machine Learning is More Likely to Cure Cancer Than to Stop Malware WHITE PAPER Introduction Machine Learning (ML) is based around the idea machines can learn from data. ML techniques have been around

More information

Lecture 19. Lecturer: Aleksander Mądry Scribes: Chidambaram Annamalai and Carsten Moldenhauer

Lecture 19. Lecturer: Aleksander Mądry Scribes: Chidambaram Annamalai and Carsten Moldenhauer CS-621 Theory Gems November 21, 2012 Lecture 19 Lecturer: Aleksander Mądry Scribes: Chidambaram Annamalai and Carsten Moldenhauer 1 Introduction We continue our exploration of streaming algorithms. First,

More information

Monte Carlo modelling and applications to imaging

Monte Carlo modelling and applications to imaging Monte Carlo modelling and applications to imaging The Monte Carlo method is a method to obtain a result through repeated random sampling of the outcome of a system. One of the earliest applications, in

More information

Market Report. Scale-out 2.0: Simple, Scalable, Services- Oriented Storage. Scale-out Storage Meets the Enterprise. June 2010.

Market Report. Scale-out 2.0: Simple, Scalable, Services- Oriented Storage. Scale-out Storage Meets the Enterprise. June 2010. Market Report Scale-out 2.0: Simple, Scalable, Services- Oriented Storage Scale-out Storage Meets the Enterprise By Terri McClure June 2010 Market Report: Scale-out 2.0: Simple, Scalable, Services-Oriented

More information

Event: PASS SQL Saturday - DC 2018 Presenter: Jon Tupitza, CTO Architect

Event: PASS SQL Saturday - DC 2018 Presenter: Jon Tupitza, CTO Architect Event: PASS SQL Saturday - DC 2018 Presenter: Jon Tupitza, CTO Architect BEOP.CTO.TP4 Owner: OCTO Revision: 0001 Approved by: JAT Effective: 08/30/2018 Buchanan & Edwards Proprietary: Printed copies of

More information

Introduction to Data Mining

Introduction to Data Mining Introduction to Data Mining Lecture #1: Course Introduction U Kang Seoul National University U Kang 1 In This Lecture Motivation to study data mining Administrative information for this course U Kang 2

More information

A Framework for Securing Databases from Intrusion Threats

A Framework for Securing Databases from Intrusion Threats A Framework for Securing Databases from Intrusion Threats R. Prince Jeyaseelan James Department of Computer Applications, Valliammai Engineering College Affiliated to Anna University, Chennai, India Email:

More information

ECS289: Scalable Machine Learning

ECS289: Scalable Machine Learning ECS289: Scalable Machine Learning Cho-Jui Hsieh UC Davis Sept 22, 2016 Course Information Website: http://www.stat.ucdavis.edu/~chohsieh/teaching/ ECS289G_Fall2016/main.html My office: Mathematical Sciences

More information

1 Executive Overview The Benefits and Objectives of BPDM

1 Executive Overview The Benefits and Objectives of BPDM 1 Executive Overview The Benefits and Objectives of BPDM This is an excerpt from the Final Submission BPDM document posted to OMG members on November 13 th 2006. The full version of the specification will

More information

Automate Transform Analyze

Automate Transform Analyze Competitive Intelligence 2.0 Turning the Web s Big Data into Big Insights Automate Transform Analyze Introduction Today, the web continues to grow at a dizzying pace. There are more than 1 billion websites

More information

2.3 Algorithms Using Map-Reduce

2.3 Algorithms Using Map-Reduce 28 CHAPTER 2. MAP-REDUCE AND THE NEW SOFTWARE STACK one becomes available. The Master must also inform each Reduce task that the location of its input from that Map task has changed. Dealing with a failure

More information

Novel Lossy Compression Algorithms with Stacked Autoencoders

Novel Lossy Compression Algorithms with Stacked Autoencoders Novel Lossy Compression Algorithms with Stacked Autoencoders Anand Atreya and Daniel O Shea {aatreya, djoshea}@stanford.edu 11 December 2009 1. Introduction 1.1. Lossy compression Lossy compression is

More information

Horizontal Aggregations in SQL to Prepare Data Sets Using PIVOT Operator

Horizontal Aggregations in SQL to Prepare Data Sets Using PIVOT Operator Horizontal Aggregations in SQL to Prepare Data Sets Using PIVOT Operator R.Saravanan 1, J.Sivapriya 2, M.Shahidha 3 1 Assisstant Professor, Department of IT,SMVEC, Puducherry, India 2,3 UG student, Department

More information

Challenges in Ubiquitous Data Mining

Challenges in Ubiquitous Data Mining LIAAD-INESC Porto, University of Porto, Portugal jgama@fep.up.pt 1 2 Very-short-term Forecasting in Photovoltaic Systems 3 4 Problem Formulation: Network Data Model Querying Model Query = Q( n i=0 S i)

More information

Gradational conception in Cleanroom Software Development

Gradational conception in Cleanroom Software Development Gradational conception in Cleanroom Software Development Anshu Sharma 1 and Shilpa Sharma 2 1 DAV Institute of Engineering and Technology, Kabir Nagar, Jalandhar, India 2 Lovely Professional University,

More information

Introduction to Big-Data

Introduction to Big-Data Introduction to Big-Data Ms.N.D.Sonwane 1, Mr.S.P.Taley 2 1 Assistant Professor, Computer Science & Engineering, DBACER, Maharashtra, India 2 Assistant Professor, Information Technology, DBACER, Maharashtra,

More information

Measurements and Bits: Compressed Sensing meets Information Theory. Dror Baron ECE Department Rice University dsp.rice.edu/cs

Measurements and Bits: Compressed Sensing meets Information Theory. Dror Baron ECE Department Rice University dsp.rice.edu/cs Measurements and Bits: Compressed Sensing meets Information Theory Dror Baron ECE Department Rice University dsp.rice.edu/cs Sensing by Sampling Sample data at Nyquist rate Compress data using model (e.g.,

More information

III. CONCEPTS OF MODELLING II.

III. CONCEPTS OF MODELLING II. III. CONCEPTS OF MODELLING II. 5. THE MODELLING PROCEDURE 6. TYPES OF THE MODELS 7. SELECTION OF MODEL TYPE 8. SELECTION OF MODEL COMPLEXITY AND STRUCTURE 1 5. MODELLING PROCEDURE Three significant steps

More information

NetApp Clustered Data ONTAP 8.2 Storage QoS Date: June 2013 Author: Tony Palmer, Senior Lab Analyst

NetApp Clustered Data ONTAP 8.2 Storage QoS Date: June 2013 Author: Tony Palmer, Senior Lab Analyst ESG Lab Spotlight NetApp Clustered Data ONTAP 8.2 Storage QoS Date: June 2013 Author: Tony Palmer, Senior Lab Analyst Abstract: This ESG Lab Spotlight explores how NetApp Data ONTAP 8.2 Storage QoS can

More information

A BFS-BASED SIMILAR CONFERENCE RETRIEVAL FRAMEWORK

A BFS-BASED SIMILAR CONFERENCE RETRIEVAL FRAMEWORK A BFS-BASED SIMILAR CONFERENCE RETRIEVAL FRAMEWORK Qing Guo 1, 2 1 Nanyang Technological University, Singapore 2 SAP Innovation Center Network,Singapore ABSTRACT Literature review is part of scientific

More information

Contractual Approaches to Data Protection in Clinical Research Projects

Contractual Approaches to Data Protection in Clinical Research Projects Contractual Approaches to Data Protection in Clinical Research Projects EICAR, 24th Annual Conference Nürnberg, October 2016 Dr. jur. Marc Stauch Institute for Legal Informatics Leibniz Universität Hannover

More information

Predictive Indexing for Fast Search

Predictive Indexing for Fast Search Predictive Indexing for Fast Search Sharad Goel, John Langford and Alex Strehl Yahoo! Research, New York Modern Massive Data Sets (MMDS) June 25, 2008 Goel, Langford & Strehl (Yahoo! Research) Predictive

More information

Review on Managing RDF Graph Using MapReduce

Review on Managing RDF Graph Using MapReduce Review on Managing RDF Graph Using MapReduce 1 Hetal K. Makavana, 2 Prof. Ashutosh A. Abhangi 1 M.E. Computer Engineering, 2 Assistant Professor Noble Group of Institutions Junagadh, India Abstract solution

More information

Introduction to Algorithms

Introduction to Algorithms Lecture 1 Introduction to Algorithms 1.1 Overview The purpose of this lecture is to give a brief overview of the topic of Algorithms and the kind of thinking it involves: why we focus on the subjects that

More information

FOUNDATIONS OF A CROSS-DISCIPLINARY PEDAGOGY FOR BIG DATA *

FOUNDATIONS OF A CROSS-DISCIPLINARY PEDAGOGY FOR BIG DATA * FOUNDATIONS OF A CROSS-DISCIPLINARY PEDAGOGY FOR BIG DATA * Joshua Eckroth Stetson University DeLand, Florida 386-740-2519 jeckroth@stetson.edu ABSTRACT The increasing awareness of big data is transforming

More information

Course Outcome of M.E (ECE)

Course Outcome of M.E (ECE) Course Outcome of M.E (ECE) PEC108/109: EMBEDDED SYSTEMS DESIGN 1. Recognize the Embedded system and its programming, Embedded Systems on a Chip (SoC) and the use of VLSI designed circuits. 2. Identify

More information

High-Throughput Real-Time Network Flow Visualization

High-Throughput Real-Time Network Flow Visualization High-Throughput Real-Time Network Flow Visualization Daniel Best Research Scientist Information Analytics daniel.best@pnl.gov Douglas Love, Shawn Bohn, William Pike 1 Tools and a Pipeline to Provide Defense

More information

The Top Five Reasons to Deploy Software-Defined Networks and Network Functions Virtualization

The Top Five Reasons to Deploy Software-Defined Networks and Network Functions Virtualization The Top Five Reasons to Deploy Software-Defined Networks and Network Functions Virtualization May 2014 Prepared by: Zeus Kerravala The Top Five Reasons to Deploy Software-Defined Networks and Network Functions

More information

Uncertain Data Models

Uncertain Data Models Uncertain Data Models Christoph Koch EPFL Dan Olteanu University of Oxford SYNOMYMS data models for incomplete information, probabilistic data models, representation systems DEFINITION An uncertain data

More information

BIG DATA TESTING: A UNIFIED VIEW

BIG DATA TESTING: A UNIFIED VIEW http://core.ecu.edu/strg BIG DATA TESTING: A UNIFIED VIEW BY NAM THAI ECU, Computer Science Department, March 16, 2016 2/30 PRESENTATION CONTENT 1. Overview of Big Data A. 5 V s of Big Data B. Data generation

More information

Pouya Kousha Fall 2018 CSE 5194 Prof. DK Panda

Pouya Kousha Fall 2018 CSE 5194 Prof. DK Panda Pouya Kousha Fall 2018 CSE 5194 Prof. DK Panda 1 Observe novel applicability of DL techniques in Big Data Analytics. Applications of DL techniques for common Big Data Analytics problems. Semantic indexing

More information

Oracle and Tangosol Acquisition Announcement

Oracle and Tangosol Acquisition Announcement Oracle and Tangosol Acquisition Announcement March 23, 2007 The following is intended to outline our general product direction. It is intended for information purposes only, and may

More information

Data Preprocessing. Slides by: Shree Jaswal

Data Preprocessing. Slides by: Shree Jaswal Data Preprocessing Slides by: Shree Jaswal Topics to be covered Why Preprocessing? Data Cleaning; Data Integration; Data Reduction: Attribute subset selection, Histograms, Clustering and Sampling; Data

More information

Numerical computing. How computers store real numbers and the problems that result

Numerical computing. How computers store real numbers and the problems that result Numerical computing How computers store real numbers and the problems that result The scientific method Theory: Mathematical equations provide a description or model Experiment Inference from data Test

More information

STORAGE EFFICIENCY: MISSION ACCOMPLISHED WITH EMC ISILON

STORAGE EFFICIENCY: MISSION ACCOMPLISHED WITH EMC ISILON STORAGE EFFICIENCY: MISSION ACCOMPLISHED WITH EMC ISILON Mission-critical storage management for military and intelligence operations EMC PERSPECTIVE TABLE OF CONTENTS INTRODUCTION 3 THE PROBLEM 3 THE

More information

CloudSwyft Learning-as-a-Service Course Catalog 2018 (Individual LaaS Course Catalog List)

CloudSwyft Learning-as-a-Service Course Catalog 2018 (Individual LaaS Course Catalog List) CloudSwyft Learning-as-a-Service Course Catalog 2018 (Individual LaaS Course Catalog List) Microsoft Solution Latest Sl Area Refresh No. Course ID Run ID Course Name Mapping Date 1 AZURE202x 2 Microsoft

More information

Instructor: Dr. Mehmet Aktaş. Mining of Massive Datasets Jure Leskovec, Anand Rajaraman, Jeff Ullman Stanford University

Instructor: Dr. Mehmet Aktaş. Mining of Massive Datasets Jure Leskovec, Anand Rajaraman, Jeff Ullman Stanford University Instructor: Dr. Mehmet Aktaş Mining of Massive Datasets Jure Leskovec, Anand Rajaraman, Jeff Ullman Stanford University J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org

More information

Unit title: IT in Business: Advanced Databases (SCQF level 8)

Unit title: IT in Business: Advanced Databases (SCQF level 8) Higher National Unit Specification General information Unit code: F848 35 Superclass: CD Publication date: January 2017 Source: Scottish Qualifications Authority Version: 02 Unit purpose This unit is designed

More information

Big Data Analytics: Research Needs. Ali Ghassemian

Big Data Analytics: Research Needs. Ali Ghassemian Big Data Analytics: Research Needs Ali Ghassemian April 28, 2016 Plan DOE s Grid Modernization Initiative (GMI) represent a comprehensive effort to help shape the future of our nation s grid and solve

More information

Data Centres in the Virtual Observatory Age

Data Centres in the Virtual Observatory Age Data Centres in the Virtual Observatory Age David Schade Canadian Astronomy Data Centre A few things I ve learned in the past two days There exist serious efforts at Long-Term Data Preservation Alliance

More information

Harnessing Grid Resources to Enable the Dynamic Analysis of Large Astronomy Datasets

Harnessing Grid Resources to Enable the Dynamic Analysis of Large Astronomy Datasets Page 1 of 5 1 Year 1 Proposal Harnessing Grid Resources to Enable the Dynamic Analysis of Large Astronomy Datasets Year 1 Progress Report & Year 2 Proposal In order to setup the context for this progress

More information

Slides for Data Mining by I. H. Witten and E. Frank

Slides for Data Mining by I. H. Witten and E. Frank Slides for Data Mining by I. H. Witten and E. Frank 7 Engineering the input and output Attribute selection Scheme-independent, scheme-specific Attribute discretization Unsupervised, supervised, error-

More information

Popular SIEM vs aisiem

Popular SIEM vs aisiem Popular SIEM vs aisiem You cannot flip a page in any Cybersecurity magazine, or scroll through security blogging sites without a mention of Next Gen SIEM. You can understand why traditional SIEM vendors

More information

Compressed Sensing and Applications by using Dictionaries in Image Processing

Compressed Sensing and Applications by using Dictionaries in Image Processing Advances in Computational Sciences and Technology ISSN 0973-6107 Volume 10, Number 2 (2017) pp. 165-170 Research India Publications http://www.ripublication.com Compressed Sensing and Applications by using

More information

Best Practices for Deploying Web Services via Integration

Best Practices for Deploying Web Services via Integration Tactical Guidelines, M. Pezzini Research Note 23 September 2002 Best Practices for Deploying Web Services via Integration Web services can assemble application logic into coarsegrained business services.

More information

(Refer Slide Time 3:31)

(Refer Slide Time 3:31) Digital Circuits and Systems Prof. S. Srinivasan Department of Electrical Engineering Indian Institute of Technology Madras Lecture - 5 Logic Simplification In the last lecture we talked about logic functions

More information

Benefits of Programming Graphically in NI LabVIEW

Benefits of Programming Graphically in NI LabVIEW Benefits of Programming Graphically in NI LabVIEW Publish Date: Jun 14, 2013 0 Ratings 0.00 out of 5 Overview For more than 20 years, NI LabVIEW has been used by millions of engineers and scientists to

More information

Benefits of Programming Graphically in NI LabVIEW

Benefits of Programming Graphically in NI LabVIEW 1 of 8 12/24/2013 2:22 PM Benefits of Programming Graphically in NI LabVIEW Publish Date: Jun 14, 2013 0 Ratings 0.00 out of 5 Overview For more than 20 years, NI LabVIEW has been used by millions of engineers

More information

Today. Lecture 17: Reality Mining. Last time

Today. Lecture 17: Reality Mining. Last time Today We will introduce the idea of a relational database, discuss its underlying data model and present a slightly simplified view of how to access its information Lecture 17: As with all new technologies

More information

Business Benefits of Policy Based Data De-Duplication Data Footprint Reduction with Quality of Service (QoS) for Data Protection

Business Benefits of Policy Based Data De-Duplication Data Footprint Reduction with Quality of Service (QoS) for Data Protection Data Footprint Reduction with Quality of Service (QoS) for Data Protection By Greg Schulz Founder and Senior Analyst, the StorageIO Group Author The Green and Virtual Data Center (Auerbach) October 28th,

More information

Data Mining. Jeff M. Phillips. January 7, 2019 CS 5140 / CS 6140

Data Mining. Jeff M. Phillips. January 7, 2019 CS 5140 / CS 6140 Data Mining CS 5140 / CS 6140 Jeff M. Phillips January 7, 2019 What is Data Mining? What is Data Mining? Finding structure in data? Machine learning on large data? Unsupervised learning? Large scale computational

More information

ALIGNING CYBERSECURITY AND MISSION PLANNING WITH ADVANCED ANALYTICS AND HUMAN INSIGHT

ALIGNING CYBERSECURITY AND MISSION PLANNING WITH ADVANCED ANALYTICS AND HUMAN INSIGHT THOUGHT PIECE ALIGNING CYBERSECURITY AND MISSION PLANNING WITH ADVANCED ANALYTICS AND HUMAN INSIGHT Brad Stone Vice President Stone_Brad@bah.com Brian Hogbin Distinguished Technologist Hogbin_Brian@bah.com

More information

Big Data Analytics: What is Big Data? Stony Brook University CSE545, Fall 2016 the inaugural edition

Big Data Analytics: What is Big Data? Stony Brook University CSE545, Fall 2016 the inaugural edition Big Data Analytics: What is Big Data? Stony Brook University CSE545, Fall 2016 the inaugural edition What s the BIG deal?! 2011 2011 2008 2010 2012 What s the BIG deal?! (Gartner Hype Cycle) What s the

More information

Real-Time Insights from the Source

Real-Time Insights from the Source LATENCY LATENCY LATENCY Real-Time Insights from the Source This white paper provides an overview of edge computing, and how edge analytics will impact and improve the trucking industry. What Is Edge Computing?

More information

A Comparative Study of Data Mining Process Models (KDD, CRISP-DM and SEMMA)

A Comparative Study of Data Mining Process Models (KDD, CRISP-DM and SEMMA) International Journal of Innovation and Scientific Research ISSN 2351-8014 Vol. 12 No. 1 Nov. 2014, pp. 217-222 2014 Innovative Space of Scientific Research Journals http://www.ijisr.issr-journals.org/

More information

OLAP Introduction and Overview

OLAP Introduction and Overview 1 CHAPTER 1 OLAP Introduction and Overview What Is OLAP? 1 Data Storage and Access 1 Benefits of OLAP 2 What Is a Cube? 2 Understanding the Cube Structure 3 What Is SAS OLAP Server? 3 About Cube Metadata

More information

Texture Mapping using Surface Flattening via Multi-Dimensional Scaling

Texture Mapping using Surface Flattening via Multi-Dimensional Scaling Texture Mapping using Surface Flattening via Multi-Dimensional Scaling Gil Zigelman Ron Kimmel Department of Computer Science, Technion, Haifa 32000, Israel and Nahum Kiryati Department of Electrical Engineering

More information