Big Data Technology Ecosystem. Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara
|
|
- Morris Davis
- 6 years ago
- Views:
Transcription
1 Big Data Technology Ecosystem Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara
2 Agenda End-to-End Data Delivery Platform Ecosystem of Data Technologies Mapping an End-to-End Solution Case Studies Pentaho Key Capabilities Summary Q&A
3 End-to-End Data Delivery Platform Ingest Process Publish Report Data Agnostic Metadata Driven Ingestion Data Orchestration Native Hadoop Integration Scale Up & Scale Out Blend Unstructured Data Streamlined Data Refinery Data Virtualization Machine Learning Production Reporting Custom Dashboards Self-Service Dashboards Interactive Analysis Embedded Analytics
4 Delivering Insight Data Integration & Orchestration Custom Dashboards Self-Service Dashboards Ingest Process Publish Report Interactive Analysis Data Engineers Data Scientists Data Analyst Consumers Production Reporting
5 Big Data Ecosystem Relational Database Analytical Databases NoSQL Database HDFS Map Reduce SQL on Hadoop Distributed Search Message Streaming Event Stream Processing (ESP) Complex Event Processing (CEP)
6 Data Source Attributes Volume (Data Size) Small Medium Large Relational Database Analytical Databases NoSQL Database Variety (Data Type) Structured Semi-Structured Unstructured HDFS Map Reduce SQL on Hadoop Distributed Search Velocity (Processing) Batch Micro-Batch RT Streaming Message Streaming Event Stream Processing (ESP) Complex Event Processing (CEP) Latency (Reporting) Scheduled Prompted Interactive
7 Core Competency Relational Database Good Fit Not Optimal Not Recommended Relational Database MSFT SQL Server, Oracle, MySQL, PostGreSQL, IBM DB2 Volume (Data Size) Small Medium Large Operational databases for OLTP apps that require high transaction loads and user concurrency. Can scale up to data volumes but lack ability to easily scale-out for large data processing. Variety (Data Type) Structured Semi-Structured Unstructured Structured schema of tables containing rows and columns of data emphasizing integrity and consistency over speed and scale. Structured data accessed with the SQL query language. Velocity (Processing) Batch Micro-Batch RT Streaming Rigid schemas with batch-oriented ingestion and SQL query processing are not designed for continuous streaming data Latency (Reporting) Scheduled Prompted Interactive Optimized for frequent small CRUD queries (create, read, update, delete), not for analytic or interactive query workloads on large data
8 Core Competency Analytical Database Good Fit Not Optimal Not Recommended Analytical Database Columnar, In-Memory, MPP, OLAP Teradata, Oracle Exadata, IBM Netezza, EMC Greenplum, Vertica Volume (Data Size) Variety (Data Type) Velocity (Processing) Latency (Reporting) Small Medium Large Structured Semi-Structured Unstructured Batch Micro-Batch RT Streaming Scheduled Prompted Interactive Data warehouse/mart databases to support BI and advanced analytics workloads. MPP architecture gives ability to scale out to large data volumes at a financial cost. Structured schema of tables containing rows and columns of data offering improved speed and scalability over RDBMS but still limited to structured data. Rigid schemas with batch-oriented SQL queries are not designed for streaming applications. All four types (Columnar, In-Memory, MPP, OLAP) designed for improved query performance for analytic or interactive query workloads on large data.
9 Core Competency NoSQL Database Good Fit Not Optimal Not Recommended NoSQL Database MongoDB, HBase, Cassandra, MarkLogic, Couchbase Volume (Data Size) Variety (Data Type) Velocity (Processing) Latency (Reporting) Small Medium Large Structured Semi-Structured Unstructured Batch Micro-Batch RT Streaming Scheduled Prompted Interactive Good for web applications - less web app code to write, debug and maintain. Scale out - horizontal scaling w auto-sharding data to support millions of web app users. Compromise on consistency (ACID transactions) in favor of scale & up-time. Hierarchical, key-value or document design to capture all types of data in a single location. Schema-less design allows for rapid or continuous ingest at scale. Good storage option for high throughput, low latency requirements of streaming applications for real-time views of data. Seen as a key component to Lambda architecture. Low level query languages, lack of skills, lack SQL support makes NoSQL less appealing for reporting and analysis.
10 Core Competency HDFS MapReduce Good Fit Not Optimal Not Recommended HDFS Map Reduce Cloudera, Hortonworks, MapR, Pivotal, Amazon EMR, Hitachi HSP, MSFT HDInsights Volume (Data Size) Variety (Data Type) Velocity (Processing) Latency (Reporting) Small Medium Large Structured Semi-Structured Unstructured Batch Micro-Batch RT Streaming Scheduled Prompted Interactive Hadoop Distributed File System designed to distribute and replicate file blocks horizontally scaled across multiple commodity data nodes. MapReduce programming takes compute to the data for batch processing large data volumes. File system is schema-less allowing easy storage of any file type in multiple Hadoop file formats. HDFS and MapReduce designed for distributing batch processing workloads on large datasets, not for micro-batch or steaming use cases. MapReduce on HDFS lacks SQL support and report queries are slow and less appealing for reporting and analysis.
11 Core Competency SQL on Hadoop Good Fit Not Optimal Not Recommended SQL on Hadoop Batch-oriented, Interactive, and In-Memory Apache Hive, Apache Drill/Phoenix, Hortonworks Hive on Tez, Cloudera Impala, Pivotal HawQ, Spark SQL Volume (Data Size) Variety (Data Type) Velocity (Processing) Latency (Reporting) Small Medium Large Structured Semi-Structured Unstructured Batch Micro-Batch RT Streaming Scheduled Prompted Interactive SQL queries on a metadata layer (Hcatalog) in Hadoop. The queries are converted to MapReduce, Apache Tez, Impala MPP, and Spark and run on different storage formats such as HDFS and HBase. SQL was designed for structured data. Hadoop files may contain nested data, variable data, schema-less data. A SQL-on-Hadoop engine must be able to translate all these forms of data to flat relational data and optimize queries (Impala/Drill) SQL-on-Hadoop engines require smart and advanced workload managers for multiuser workloads designed for query processing not stream processing. Ad-hoc reporting, iterative OLAP, and data mining) in single-user and multi-user modes. For multi-user queries, Impala is on average 16.4x faster than Hive-on-Tez and 7.6x faster than Spark SQL with Tungsten, with an average response time of 12.8s compared to over 1.6 minutes or more.
12 Core Competency Distributed Search Good Fit Not Optimal Not Recommended Distributed Search ElasticSearch, Solr (based on Apache Lucene), Amazon CloudSearch Volume (Data Size) Variety (Data Type) Velocity (Processing) Latency (Reporting) Small Medium Large Structured Semi-Structured Unstructured Batch Micro-Batch RT Streaming Scheduled Prompted Interactive Search engines have to deal with large systems with millions of documents and are designed for index and search query processing at scale with clustering and distributed architecture. XML, CSV, RDBMS, Word, PDF,ActiveMQ, AWS SQS, DynamoDB (Amazon NoSQL), FileSystem, Git, JDBC, JMS, Kafka, LDAP, MongoDB, neo4j, RabbitMQ, Redis, and Twitter. ES scalable to very large clusters with near real-time search. The demands of real time web applications require search results in near real time as new content is generated by users. Some contention handling concurrent search + index requests. Both use key-value pair query language. Solr is much more oriented towards text search while Elasticsearch is often used for more advanced querying, filtering, and grouping. Good for interactive search queries but not interactive analytical reporting.
13 Core Competency Message Streaming Good Fit Not Optimal Not Recommended Message Streaming Kafka, JMS, AMQP Volume (Data Size) Variety (Data Type) Velocity (Processing) Latency (Reporting) Small Medium Large Structured Semi-Structured Unstructured Batch Micro-Batch RT Streaming Scheduled Prompted Interactive Kafka is an excellent low latency messaging platform that brokers massive message streams for parallel ingestion into Hadoop Data sources, such as the internet of things, sensors, clickstream, and transactional systems. Realtime streaming providing high throughput for both publishing and subscribing, with constant performance even with many terabytes of stored messages. Designed for streaming and can configure batch size for brokering micro batches of messages. Stream topics need to be processed by additional technology such as PDI, ESP, CEP, query processing engines for reporting.
14 Core Competency Event Stream Processing (ESP) Good Fit Not Optimal Not Recommended Message Streaming Apache Storm Volume (Data Size) Variety (Data Type) Velocity (Processing) Latency (Reporting) Small Medium Large Structured Semi-Structured Unstructured Batch Micro-Batch RT Streaming Scheduled Prompted Interactive Apache Storm is a distributed event-at-a-time stream processing system for processing large volumes in parallel with sub-second latency. Storm applications process 1 incoming event at a time as tuples of data; a tuple may can contain object of any type such as the internet of things, sensors, and transactional systems. Storm is extremely fast, with the ability to process over a million messages per second per node. Compromises on fault tolerance by offering at least once semantics in favor of speed. ESP provides the most recent processed data for all types of reporting. Example ESP Use Case: Stock market tickers showing stock performances with a Green up arrow or Red down arrow in real time.
15 Core Competency Complex Event Processing (CEP) Good Fit Not Optimal Not Recommended Message Streaming Spark, Flink Volume (Data Size) Variety (Data Type) Velocity (Processing) Latency (Reporting) Small Medium Large Structured Semi-Structured Unstructured Batch Micro-Batch RT Streaming Scheduled Prompted Interactive Spark and Flink are distributed micro-batch stream processing engines for processing large volumes of high-velocity data in parallel with a few seconds latency. Complex event processing for internet of things, sensors, and transactional systems. An aggregation-oriented CEP solution is focused on executing on-line algorithms as a response to event data entering the system. Detection-oriented CEP is focused on detecting combinations of events called events patterns or situations. Micro-batch processing engines with few seconds latency that is not as fast as Storm, but has better fault tolerance guaranteeing exactly once semantics for stateful computations. Great for machine learning computations. CEP provides the most recent processed data for all types of reporting. Example CEP use case: user sets up alert to the stock market saying "let me know if GOOG stocks went up by 10% and stayed up for 3 hours or more".
16 Big Data Ecosystem Relational Database Analytical Databases NoSQL Database HDFS Map Reduce SQL on Hadoop Distributed Search Message Streaming Event Stream Processing (ESP) Complex Event Processing (CEP)
17 Mapping A Solution
18 Core Competency Good Fit Not Optimal Not Recommended Matrix for Analytics Performance (MAP) Relational Database Analytical Database NoSQL Database Hadoop File System (HDFS MR) SQL on Hadoop Distributed Search Message Streaming Event Stream Processing (ESP) Complex Event Processing (CEP) Volume (Data Size) Variety (Data Type) Velocity (Processing) Latency (Reporting) Small Medium Large Structured Semi-Structured Unstructured Batch Micro-Batch RT Streaming Scheduled Prompted Interactive
19 Big Data Projects BIG DATA SOURCES PENTAHO DATA INTEGRATION HADOOP/ DATA LAKE PENTAHO DATA INTEGRATION ANALYTIC DATASETS LINE OF BUSINESS CENTRALIZED ANALYTICS AT SCALE SELF- SERVICE ANALYTICS ON- DEMAND DATAMART EMBEDDED ANALYTICS EXTRANET DEPLOYMENTS PDI ANALYTICS TRADITIONAL DATA PENTAHO DATA INTEGRATION DATA WAREHOUSE PENTAHO DATA INTEGRATION DATA MARTS
20 A Single Flow Data Engineering Data Prep Analytics Ingestion Processing Blending Data Delivery Data Discovery / Analysis Analysis & Dashboards Administration Security Lifecycle Management Data Provenance Dynamic Data Pipeline Monitoring Automation
21 Key Takeaways Data architecture modernization involves many technologies Understanding the ecosystem of data technologies Mapping an end-to-end solution Pentaho key capabilities
22
Stages of Data Processing
Data processing can be understood as the conversion of raw data into a meaningful and desired form. Basically, producing information that can be understood by the end user. So then, the question arises,
More informationOracle GoldenGate for Big Data
Oracle GoldenGate for Big Data The Oracle GoldenGate for Big Data 12c product streams transactional data into big data systems in real time, without impacting the performance of source systems. It streamlines
More informationSQT03 Big Data and Hadoop with Azure HDInsight Andrew Brust. Senior Director, Technical Product Marketing and Evangelism
Big Data and Hadoop with Azure HDInsight Andrew Brust Senior Director, Technical Product Marketing and Evangelism Datameer Level: Intermediate Meet Andrew Senior Director, Technical Product Marketing and
More informationBig Data Architect.
Big Data Architect www.austech.edu.au WHAT IS BIG DATA ARCHITECT? A big data architecture is designed to handle the ingestion, processing, and analysis of data that is too large or complex for traditional
More informationSpagoBI and Talend jointly support Big Data scenarios
SpagoBI and Talend jointly support Big Data scenarios Monica Franceschini - SpagoBI Architect SpagoBI Competency Center - Engineering Group Big-data Agenda Intro & definitions Layers Talend & SpagoBI SpagoBI
More informationMODERN BIG DATA DESIGN PATTERNS CASE DRIVEN DESINGS
MODERN BIG DATA DESIGN PATTERNS CASE DRIVEN DESINGS SUJEE MANIYAM FOUNDER / PRINCIPAL @ ELEPHANT SCALE www.elephantscale.com sujee@elephantscale.com HI, I M SUJEE MANIYAM Founder / Principal @ ElephantScale
More informationModern ETL Tools for Cloud and Big Data. Ken Beutler, Principal Product Manager, Progress Michael Rainey, Technical Advisor, Gluent Inc.
Modern ETL Tools for Cloud and Big Data Ken Beutler, Principal Product Manager, Progress Michael Rainey, Technical Advisor, Gluent Inc. Agenda Landscape Cloud ETL Tools Big Data ETL Tools Best Practices
More informationThings Every Oracle DBA Needs to Know about the Hadoop Ecosystem. Zohar Elkayam
Things Every Oracle DBA Needs to Know about the Hadoop Ecosystem Zohar Elkayam www.realdbamagic.com Twitter: @realmgic Who am I? Zohar Elkayam, CTO at Brillix Programmer, DBA, team leader, database trainer,
More informationBig Data with Hadoop Ecosystem
Diógenes Pires Big Data with Hadoop Ecosystem Hands-on (HBase, MySql and Hive + Power BI) Internet Live http://www.internetlivestats.com/ Introduction Business Intelligence Business Intelligence Process
More informationBIG DATA COURSE CONTENT
BIG DATA COURSE CONTENT [I] Get Started with Big Data Microsoft Professional Orientation: Big Data Duration: 12 hrs Course Content: Introduction Course Introduction Data Fundamentals Introduction to Data
More informationThe age of Big Data Big Data for Oracle Database Professionals
The age of Big Data Big Data for Oracle Database Professionals Oracle OpenWorld 2017 #OOW17 SessionID: SUN5698 Tom S. Reddy tom.reddy@datareddy.com About the Speaker COLLABORATE & OpenWorld Speaker IOUG
More informationSyncsort DMX-h. Simplifying Big Data Integration. Goals of the Modern Data Architecture SOLUTION SHEET
SOLUTION SHEET Syncsort DMX-h Simplifying Big Data Integration Goals of the Modern Data Architecture Data warehouses and mainframes are mainstays of traditional data architectures and still play a vital
More informationBig Data com Hadoop. VIII Sessão - SQL Bahia. Impala, Hive e Spark. Diógenes Pires 03/03/2018
Big Data com Hadoop Impala, Hive e Spark VIII Sessão - SQL Bahia 03/03/2018 Diógenes Pires Connect with PASS Sign up for a free membership today at: pass.org #sqlpass Internet Live http://www.internetlivestats.com/
More informationHadoop 2.x Core: YARN, Tez, and Spark. Hortonworks Inc All Rights Reserved
Hadoop 2.x Core: YARN, Tez, and Spark YARN Hadoop Machine Types top-of-rack switches core switch client machines have client-side software used to access a cluster to process data master nodes run Hadoop
More informationVOLTDB + HP VERTICA. page
VOLTDB + HP VERTICA ARCHITECTURE FOR FAST AND BIG DATA ARCHITECTURE FOR FAST + BIG DATA FAST DATA Fast Serve Analytics BIG DATA BI Reporting Fast Operational Database Streaming Analytics Columnar Analytics
More informationThe Hadoop Ecosystem. EECS 4415 Big Data Systems. Tilemachos Pechlivanoglou
The Hadoop Ecosystem EECS 4415 Big Data Systems Tilemachos Pechlivanoglou tipech@eecs.yorku.ca A lot of tools designed to work with Hadoop 2 HDFS, MapReduce Hadoop Distributed File System Core Hadoop component
More informationHadoop. Introduction / Overview
Hadoop Introduction / Overview Preface We will use these PowerPoint slides to guide us through our topic. Expect 15 minute segments of lecture Expect 1-4 hour lab segments Expect minimal pretty pictures
More informationBig Data on AWS. Big Data Agility and Performance Delivered in the Cloud. 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Big Data on AWS Big Data Agility and Performance Delivered in the Cloud 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Big Data Technologies and techniques for working productively
More informationThe Technology of the Business Data Lake. Appendix
The Technology of the Business Data Lake Appendix Pivotal data products Term Greenplum Database GemFire Pivotal HD Spring XD Pivotal Data Dispatch Pivotal Analytics Description A massively parallel platform
More informationSecurity and Performance advances with Oracle Big Data SQL
Security and Performance advances with Oracle Big Data SQL Jean-Pierre Dijcks Oracle Redwood Shores, CA, USA Key Words SQL, Oracle, Database, Analytics, Object Store, Files, Big Data, Big Data SQL, Hadoop,
More informationPerformance and Scalability Overview
Performance and Scalability Overview This guide provides an overview of some of the performance and scalability capabilities of the Pentaho Business Anlytics platform PENTAHO PERFORMANCE ENGINEERING TEAM
More informationmicrosoft
70-775.microsoft Number: 70-775 Passing Score: 800 Time Limit: 120 min Exam A QUESTION 1 Note: This question is part of a series of questions that present the same scenario. Each question in the series
More informationNew Approaches to Big Data Processing and Analytics
New Approaches to Big Data Processing and Analytics Contributing authors: David Floyer, David Vellante Original publication date: February 12, 2013 There are number of approaches to processing and analyzing
More informationFlash Storage Complementing a Data Lake for Real-Time Insight
Flash Storage Complementing a Data Lake for Real-Time Insight Dr. Sanhita Sarkar Global Director, Analytics Software Development August 7, 2018 Agenda 1 2 3 4 5 Delivering insight along the entire spectrum
More informationOverview. Prerequisites. Course Outline. Course Outline :: Apache Spark Development::
Title Duration : Apache Spark Development : 4 days Overview Spark is a fast and general cluster computing system for Big Data. It provides high-level APIs in Scala, Java, Python, and R, and an optimized
More informationBring Context To Your Machine Data With Hadoop, RDBMS & Splunk
Bring Context To Your Machine Data With Hadoop, RDBMS & Splunk Raanan Dagan and Rohit Pujari September 25, 2017 Washington, DC Forward-Looking Statements During the course of this presentation, we may
More informationInteractive SQL-on-Hadoop from Impala to Hive/Tez to Spark SQL to JethroData
Interactive SQL-on-Hadoop from Impala to Hive/Tez to Spark SQL to JethroData ` Ronen Ovadya, Ofir Manor, JethroData About JethroData Founded 2012 Raised funding from Pitango in 2013 Engineering in Israel,
More informationPřehled novinek v SQL Server 2016
Přehled novinek v SQL Server 2016 Martin Rys, BI Competency Leader martin.rys@adastragrp.com https://www.linkedin.com/in/martinrys 20.4.2016 1 BI Competency development 2 Trends, modern data warehousing
More informationTopics. Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples
Hadoop Introduction 1 Topics Big Data Analytics What is and Why Hadoop? Comparison to other technologies Hadoop architecture Hadoop ecosystem Hadoop usage examples 2 Big Data Analytics What is Big Data?
More informationEight Essential Checklists for Managing the Analytic Data Pipeline
Eight Essential Checklists for Managing the Analytic Data Pipeline Contents Introduction.... 3 Checklist 1: Data Connectivity.... 4 Checklist 2: Data Engineering.... 6 Checklist 3: Data Delivery.... 8
More informationMicrosoft Big Data and Hadoop
Microsoft Big Data and Hadoop Lara Rubbelke @sqlgal Cindy Gross @sqlcindy 2 The world of data is changing The 4Vs of Big Data http://nosql.mypopescu.com/post/9621746531/a-definition-of-big-data 3 Common
More informationConfiguring and Deploying Hadoop Cluster Deployment Templates
Configuring and Deploying Hadoop Cluster Deployment Templates This chapter contains the following sections: Hadoop Cluster Profile Templates, on page 1 Creating a Hadoop Cluster Profile Template, on page
More informationCONSOLIDATING RISK MANAGEMENT AND REGULATORY COMPLIANCE APPLICATIONS USING A UNIFIED DATA PLATFORM
CONSOLIDATING RISK MANAGEMENT AND REGULATORY COMPLIANCE APPLICATIONS USING A UNIFIED PLATFORM Executive Summary Financial institutions have implemented and continue to implement many disparate applications
More informationIan Choy. Technology Solutions Professional
Ian Choy Technology Solutions Professional XML KPIs SQL Server 2000 Management Studio Mirroring SQL Server 2005 Compression Policy-Based Mgmt Programmability SQL Server 2008 PowerPivot SharePoint Integration
More informationThe SMACK Stack: Spark*, Mesos*, Akka, Cassandra*, Kafka* Elizabeth K. Dublin Apache Kafka Meetup, 30 August 2017.
Dublin Apache Kafka Meetup, 30 August 2017 The SMACK Stack: Spark*, Mesos*, Akka, Cassandra*, Kafka* Elizabeth K. Joseph @pleia2 * ASF projects 1 Elizabeth K. Joseph, Developer Advocate Developer Advocate
More informationHadoop, Yarn and Beyond
Hadoop, Yarn and Beyond 1 B. R A M A M U R T H Y Overview We learned about Hadoop1.x or the core. Just like Java evolved, Java core, Java 1.X, Java 2.. So on, software and systems evolve, naturally.. Lets
More informationOracle Big Data Connectors
Oracle Big Data Connectors Oracle Big Data Connectors is a software suite that integrates processing in Apache Hadoop distributions with operations in Oracle Database. It enables the use of Hadoop to process
More informationData Lake Best Practices
Data Lake Best Practices Agenda Why Data Lake Key Components of a Data Lake Modern Data Architecture Some Best Practices Case Study Summary Takeaways What is a Data Lake? What, why etc. What is a data
More informationIT directors, CIO s, IT Managers, BI Managers, data warehousing professionals, data scientists, enterprise architects, data architects
Organised by: www.unicom.co.uk OVERVIEW This two day workshop is aimed at getting Data Scientists, Data Warehousing and BI professionals up to scratch on Big Data, Hadoop, other NoSQL DBMSs and Multi-Platform
More informationAccelerate Your Data Pipeline for Data Lake, Streaming and Cloud Architectures
WHITE PAPER : REPLICATE Accelerate Your Data Pipeline for Data Lake, Streaming and Cloud Architectures INTRODUCTION Analysis of a wide variety of data is becoming essential in nearly all industries to
More informationData Architectures in Azure for Analytics & Big Data
Data Architectures in for Analytics & Big Data October 20, 2018 Melissa Coates Solution Architect, BlueGranite Microsoft Data Platform MVP Blog: www.sqlchick.com Twitter: @sqlchick Data Architecture A
More informationBig Data. Big Data Analyst. Big Data Engineer. Big Data Architect
Big Data Big Data Analyst INTRODUCTION TO BIG DATA ANALYTICS ANALYTICS PROCESSING TECHNIQUES DATA TRANSFORMATION & BATCH PROCESSING REAL TIME (STREAM) DATA PROCESSING Big Data Engineer BIG DATA FOUNDATION
More informationSpotfire Advanced Data Services. Lunch & Learn Tuesday, 21 November 2017
Spotfire Advanced Data Services Lunch & Learn Tuesday, 21 November 2017 CONFIDENTIALITY The following information is confidential information of TIBCO Software Inc. Use, duplication, transmission, or republication
More informationThe Reality of Qlik and Big Data. Chris Larsen Q3 2016
The Reality of Qlik and Big Data Chris Larsen Q3 2016 Introduction Chris Larsen Sr Solutions Architect, Partner Engineering @Qlik Based in Lund, Sweden Primary Responsibility Advanced Analytics (and formerly
More informationHDInsight > Hadoop. October 12, 2017
HDInsight > Hadoop October 12, 2017 2 Introduction Mark Hudson >20 years mixing technology with data >10 years with CapTech Microsoft Certified IT Professional Business Intelligence Member of the Richmond
More informationAgenda. AWS Database Services Traditional vs AWS Data services model Amazon RDS Redshift DynamoDB ElastiCache
Databases on AWS 2017 Amazon Web Services, Inc. and its affiliates. All rights served. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon Web Services,
More informationIncrease Value from Big Data with Real-Time Data Integration and Streaming Analytics
Increase Value from Big Data with Real-Time Data Integration and Streaming Analytics Cy Erbay Senior Director Striim Executive Summary Striim is Uniquely Qualified to Solve the Challenges of Real-Time
More informationIBM Data Replication for Big Data
IBM Data Replication for Big Data Highlights Stream changes in realtime in Hadoop or Kafka data lakes or hubs Provide agility to data in data warehouses and data lakes Achieve minimum impact on source
More informationManaging IoT and Time Series Data with Amazon ElastiCache for Redis
Managing IoT and Time Series Data with ElastiCache for Redis Darin Briskman, ElastiCache Developer Outreach Michael Labib, Specialist Solutions Architect 2016, Web Services, Inc. or its Affiliates. All
More informationData 101 Which DB, When. Joe Yong Azure SQL Data Warehouse, Program Management Microsoft Corp.
Data 101 Which DB, When Joe Yong (joeyong@microsoft.com) Azure SQL Data Warehouse, Program Management Microsoft Corp. The world is changing AI increased by 300% in 2017 Data will grow to 44 ZB in 2020
More informationDatabases 2 (VU) ( / )
Databases 2 (VU) (706.711 / 707.030) MapReduce (Part 3) Mark Kröll ISDS, TU Graz Nov. 27, 2017 Mark Kröll (ISDS, TU Graz) MapReduce Nov. 27, 2017 1 / 42 Outline 1 Problems Suited for Map-Reduce 2 MapReduce:
More informationHitachi Vantara Overview Pentaho 8.0 and 8.1 Roadmap. Pedro Alves
Hitachi Vantara Overview Pentaho 8.0 and 8.1 Roadmap Pedro Alves Safe Harbor Statement The forward-looking statements contained in this document represent an outline of our current intended product direction.
More informationEmbedded Technosolutions
Hadoop Big Data An Important technology in IT Sector Hadoop - Big Data Oerie 90% of the worlds data was generated in the last few years. Due to the advent of new technologies, devices, and communication
More informationData Acquisition. The reference Big Data stack
Università degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Data Acquisition Corso di Sistemi e Architetture per Big Data A.A. 2016/17 Valeria Cardellini The reference
More informationStream Processing Platforms Storm, Spark,.. Batch Processing Platforms MapReduce, SparkSQL, BigQuery, Hive, Cypher,...
Data Ingestion ETL, Distcp, Kafka, OpenRefine, Query & Exploration SQL, Search, Cypher, Stream Processing Platforms Storm, Spark,.. Batch Processing Platforms MapReduce, SparkSQL, BigQuery, Hive, Cypher,...
More informationHadoop Overview. Lars George Director EMEA Services
Hadoop Overview Lars George Director EMEA Services 1 About Me Director EMEA Services @ Cloudera Consulting on Hadoop projects (everywhere) Apache Committer HBase and Whirr O Reilly Author HBase The Definitive
More informationSTATE OF MODERN APPLICATIONS IN THE CLOUD
STATE OF MODERN APPLICATIONS IN THE CLOUD 2017 Introduction The Rise of Modern Applications What is the Modern Application? Today s leading enterprises are striving to deliver high performance, highly
More informationBig Data Analytics using Apache Hadoop and Spark with Scala
Big Data Analytics using Apache Hadoop and Spark with Scala Training Highlights : 80% of the training is with Practical Demo (On Custom Cloudera and Ubuntu Machines) 20% Theory Portion will be important
More informationIntro to Big Data on AWS Igor Roiter Big Data Cloud Solution Architect
Intro to Big Data on AWS Igor Roiter Big Data Cloud Solution Architect Igor Roiter Big Data Cloud Solution Architect Working as a Data Specialist for the last 11 years 9 of them as a Consultant specializing
More informationAccelerate MySQL for Demanding OLAP and OLTP Use Cases with Apache Ignite. Peter Zaitsev, Denis Magda Santa Clara, California April 25th, 2017
Accelerate MySQL for Demanding OLAP and OLTP Use Cases with Apache Ignite Peter Zaitsev, Denis Magda Santa Clara, California April 25th, 2017 About the Presentation Problems Existing Solutions Denis Magda
More informationOracle Big Data SQL. Release 3.2. Rich SQL Processing on All Data
Oracle Big Data SQL Release 3.2 The unprecedented explosion in data that can be made useful to enterprises from the Internet of Things, to the social streams of global customer bases has created a tremendous
More informationMicrosoft Analytics Platform System (APS)
Microsoft Analytics Platform System (APS) The turnkey modern data warehouse appliance Matt Usher, Senior Program Manager @ Microsoft About.me @two_under Senior Program Manager 9 years at Microsoft Visual
More informationSOLUTION TRACK Finding the Needle in a Big Data Innovator & Problem Solver Cloudera
SOLUTION TRACK Finding the Needle in a Big Data Haystack @EvaAndreasson, Innovator & Problem Solver Cloudera Agenda Problem (Solving) Apache Solr + Apache Hadoop et al Real-world examples Q&A Problem Solving
More informationCloud Computing 2. CSCI 4850/5850 High-Performance Computing Spring 2018
Cloud Computing 2 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning
More informationBig Data Hadoop Stack
Big Data Hadoop Stack Lecture #1 Hadoop Beginnings What is Hadoop? Apache Hadoop is an open source software framework for storage and large scale processing of data-sets on clusters of commodity hardware
More informationApril Copyright 2013 Cloudera Inc. All rights reserved.
Hadoop Beyond Batch: Real-time Workloads, SQL-on- Hadoop, and the Virtual EDW Headline Goes Here Marcel Kornacker marcel@cloudera.com Speaker Name or Subhead Goes Here April 2014 Analytic Workloads on
More informationCERTIFICATE IN SOFTWARE DEVELOPMENT LIFE CYCLE IN BIG DATA AND BUSINESS INTELLIGENCE (SDLC-BD & BI)
CERTIFICATE IN SOFTWARE DEVELOPMENT LIFE CYCLE IN BIG DATA AND BUSINESS INTELLIGENCE (SDLC-BD & BI) The Certificate in Software Development Life Cycle in BIGDATA, Business Intelligence and Tableau program
More informationIntroduction to Big-Data
Introduction to Big-Data Ms.N.D.Sonwane 1, Mr.S.P.Taley 2 1 Assistant Professor, Computer Science & Engineering, DBACER, Maharashtra, India 2 Assistant Professor, Information Technology, DBACER, Maharashtra,
More informationBig Data Hadoop Course Content
Big Data Hadoop Course Content Topics covered in the training Introduction to Linux and Big Data Virtual Machine ( VM) Introduction/ Installation of VirtualBox and the Big Data VM Introduction to Linux
More information<Insert Picture Here> Introduction to Big Data Technology
Introduction to Big Data Technology The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into
More informationData Science and Open Source Software. Iraklis Varlamis Assistant Professor Harokopio University of Athens
Data Science and Open Source Software Iraklis Varlamis Assistant Professor Harokopio University of Athens varlamis@hua.gr What is data science? 2 Why data science is important? More data (volume, variety,...)
More informationHOW TO ACHIEVE REAL-TIME ANALYTICS ON A DATA LAKE USING GPUS. Mark Brooks - Principal System Kinetica May 09, 2017
HOW TO ACHIEVE REAL-TIME ANALYTICS ON A DATA LAKE USING GPUS Mark Brooks - Principal System Engineer @ Kinetica May 09, 2017 The Challenge: How to maintain analytic performance while dealing with: Larger
More informationA NoSQL Introduction for Relational Database Developers. Andrew Karcher Las Vegas SQL Saturday September 12th, 2015
A NoSQL Introduction for Relational Database Developers Andrew Karcher Las Vegas SQL Saturday September 12th, 2015 About Me http://www.andrewkarcher.com Twitter: @akarcher LinkedIn, Twitter Email: akarcher@gmail.com
More informationChallenges for Data Driven Systems
Challenges for Data Driven Systems Eiko Yoneki University of Cambridge Computer Laboratory Data Centric Systems and Networking Emergence of Big Data Shift of Communication Paradigm From end-to-end to data
More informationBig Data on AWS. Peter-Mark Verwoerd Solutions Architect
Big Data on AWS Peter-Mark Verwoerd Solutions Architect What to get out of this talk Non-technical: Big Data processing stages: ingest, store, process, visualize Hot vs. Cold data Low latency processing
More informationData Acquisition. The reference Big Data stack
Università degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Data Acquisition Corso di Sistemi e Architetture per Big Data A.A. 2017/18 Valeria Cardellini The reference
More informationHadoop Development Introduction
Hadoop Development Introduction What is Bigdata? Evolution of Bigdata Types of Data and their Significance Need for Bigdata Analytics Why Bigdata with Hadoop? History of Hadoop Why Hadoop is in demand
More informationIntegrating Oracle Databases with NoSQL Databases for Linux on IBM LinuxONE and z System Servers
Oracle zsig Conference IBM LinuxONE and z System Servers Integrating Oracle Databases with NoSQL Databases for Linux on IBM LinuxONE and z System Servers Sam Amsavelu Oracle on z Architect IBM Washington
More informationCloud Analytics and Business Intelligence on AWS
Cloud Analytics and Business Intelligence on AWS Enterprise Applications Virtual Desktops Sharing & Collaboration Platform Services Analytics Hadoop Real-time Streaming Data Machine Learning Data Warehouse
More informationMicrosoft Exam
Volume: 42 Questions Case Study: 1 Relecloud General Overview Relecloud is a social media company that processes hundreds of millions of social media posts per day and sells advertisements to several hundred
More informationDATA SCIENCE USING SPARK: AN INTRODUCTION
DATA SCIENCE USING SPARK: AN INTRODUCTION TOPICS COVERED Introduction to Spark Getting Started with Spark Programming in Spark Data Science with Spark What next? 2 DATA SCIENCE PROCESS Exploratory Data
More informationCmprssd Intrduction To
Cmprssd Intrduction To Hadoop, SQL-on-Hadoop, NoSQL Arseny.Chernov@Dell.com Singapore University of Technology & Design 2016-11-09 @arsenyspb Thank You For Inviting! My special kind regards to: Professor
More informationEvolving To The Big Data Warehouse
Evolving To The Big Data Warehouse Kevin Lancaster 1 Copyright Director, 2012, Oracle and/or its Engineered affiliates. All rights Insert Systems, Information Protection Policy Oracle Classification from
More informationTop Five Reasons for Data Warehouse Modernization Philip Russom
Top Five Reasons for Data Warehouse Modernization Philip Russom TDWI Research Director for Data Management May 28, 2014 Sponsor Speakers Philip Russom TDWI Research Director, Data Management Steve Sarsfield
More informationModern Data Warehouse The New Approach to Azure BI
Modern Data Warehouse The New Approach to Azure BI History On-Premise SQL Server Big Data Solutions Technical Barriers Modern Analytics Platform On-Premise SQL Server Big Data Solutions Modern Analytics
More informationHadoop An Overview. - Socrates CCDH
Hadoop An Overview - Socrates CCDH What is Big Data? Volume Not Gigabyte. Terabyte, Petabyte, Exabyte, Zettabyte - Due to handheld gadgets,and HD format images and videos - In total data, 90% of them collected
More informationPart 1: Indexes for Big Data
JethroData Making Interactive BI for Big Data a Reality Technical White Paper This white paper explains how JethroData can help you achieve a truly interactive interactive response time for BI on big data,
More informationDown the event-driven road: Experiences of integrating streaming into analytic data platforms
Down the event-driven road: Experiences of integrating streaming into analytic data platforms Dr. Dominik Benz, Head of Machine Learning Engineering, inovex GmbH Confluent Meetup Munich, 8.10.2018 Integrate
More informationBlended Learning Outline: Developer Training for Apache Spark and Hadoop (180404a)
Blended Learning Outline: Developer Training for Apache Spark and Hadoop (180404a) Cloudera s Developer Training for Apache Spark and Hadoop delivers the key concepts and expertise need to develop high-performance
More informationLambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL. May 2015
Lambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL May 2015 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved. Notices This document
More informationFluentd + MongoDB + Spark = Awesome Sauce
Fluentd + MongoDB + Spark = Awesome Sauce Nishant Sahay, Sr. Architect, Wipro Limited Bhavani Ananth, Tech Manager, Wipro Limited Your company logo here Wipro Open Source Practice: Vision & Mission Vision
More informationOracle NoSQL Database Enterprise Edition, Version 18.1
Oracle NoSQL Database Enterprise Edition, Version 18.1 Oracle NoSQL Database is a scalable, distributed NoSQL database, designed to provide highly reliable, flexible and available data management across
More informationModernizing Business Intelligence and Analytics
Modernizing Business Intelligence and Analytics Justin Erickson Senior Director, Product Management 1 Agenda What benefits can I achieve from modernizing my analytic DB? When and how do I migrate from
More informationData Lake Based Systems that Work
Data Lake Based Systems that Work There are many article and blogs about what works and what does not work when trying to build out a data lake and reporting system. At DesignMind, we have developed a
More informationUnderstanding the latent value in all content
Understanding the latent value in all content John F. Kennedy (JFK) November 22, 1963 INGEST ENRICH EXPLORE Cognitive skills Data in any format, any Azure store Search Annotations Data Cloud Intelligence
More informationSQL in the Hybrid World
SQL in the Hybrid World Tanel Poder a long time computer performance geek 1 Tanel Põder Intro: About me Oracle Database Performance geek (18+ years) Exadata Performance geek Linux Performance geek Hadoop
More informationData 101 Which DB, When Joe Yong Sr. Program Manager Microsoft Corp.
17-18 March, 2018 Beijing Data 101 Which DB, When Joe Yong Sr. Program Manager Microsoft Corp. The world is changing AI increased by 300% in 2017 Data will grow to 44 ZB in 2020 Today, 80% of organizations
More informationNew Technologies for Data Management
New Technologies for Data Management Chaitan Baru 2 2 Why new technologies? Big Data Characteristics: Volume, Velocity, Variety Began as a Volume problem E.g. Web crawls 1 spb-100 spb in a single cluster
More informationUnderstanding NoSQL Database Implementations
Understanding NoSQL Database Implementations Sadalage and Fowler, Chapters 7 11 Class 07: Understanding NoSQL Database Implementations 1 Foreword NoSQL is a broad and diverse collection of technologies.
More informationIOTA ARCHITECTURE: DATA VIRTUALIZATION AND PROCESSING MEDIUM DR. KONSTANTIN BOUDNIK DR. ALEXANDRE BOUDNIK
IOTA ARCHITECTURE: DATA VIRTUALIZATION AND PROCESSING MEDIUM DR. KONSTANTIN BOUDNIK DR. ALEXANDRE BOUDNIK DR. KONSTANTIN BOUDNIK DR.KONSTANTIN BOUDNIK EPAM SYSTEMS CHIEF TECHNOLOGIST BIGDATA, OPEN SOURCE
More information