Big Data & Hadoop Ecosystem. Diógenes Santos

Transcription

1 Big Data & Hadoop Ecosystem Diógenes Santos

2 Sponsors

3 Grupos Locais

4 SQL Saturday 2018 Mês Abril Maio Junho Agosto Local Joinville Belo Horizonte Caxias do Sul Brasília

5 Os sorteios vão ser realizados no final do evento.

6 Consultor BI Diógenes Santos Consultor Business Intelligence Capgemini Mestrado em Ciência da Computação, UFBA, em curso, pós-graduado especialista em Banco de dados com ênfase em alta disponibilidade, pós-graduado especialista em Governança de TI. Consultor de Business Intelligence/Big Data (Hadoop) e Professor Universitário. Nas horas vagas diogenespires.santos bolachadegoma@gmail.com /diógenes-pires-b375a172/ Mixologista/Barman; Churrasqueiro; Batman;

7 Definitions Business intelligence (BI) is a technology-driven process for analyzing data and presenting actionable information to help executives, managers and other corporate end users make informed business decisions. Big data is a term for data sets that are so large or complex that traditional data processing application software is inadequate to deal with them. Wikipedia

8 Introduction

9 Internet Live

10 Business Intelligence vs Big Data

11 Landscape

12 Landscape

13 Google File System (GFS or GoogleFS) Google File System (GFS or GoogleFS) is a proprietary distributed file system developed by Google to provide efficient, reliable access to data using large clusters of commodity hardware. A new version of Google File System code named Colossus was released in Wikipedia

14 Architecture

15 A Little Joke

16 Apache Hadoop The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models The project includes these modules: Apache Hadoop. Hadoop Common: The common utilities that support the other Hadoop modules. Hadoop Distributed File System (HDFS ): A distributed file system that provides highthroughput access to application data. Hadoop YARN: A framework for job scheduling and cluster resource management. Hadoop MapReduce: A YARN-based system for parallel processing of large data sets.

17 Others Hadoop Projects

18 Distributions

19 Hadoop 2.0

20 HDFS The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. Apache Hadoop.

21 HDFS Architecture

22 GFS vs HDFS PaaS Techniques File System

23 MapReduce A programming paradigm that allows for massive scalability across hundreds or thousands of servers in a Hadoop cluster. IBM

24 MapReduce Intuitive Explanation of "MapReduce"

25 Apache Hive The Apache Hive data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage using SQL. Structure can be projected onto data already in storage. A command line tool and JDBC driver are provided to connect users to Hive. Hive.org

26 Apache Architecture

27 Cloudera Impala Cloudera Impala provides fast, interactive SQL queries directly on your Apache Hadoop data stored in HDFS or HBase. In addition to using the same unified storage platform, Impala also uses the same metadata, SQL syntax (Hive SQL), ODBC driver, and user interface (Cloudera Impala query UI in Hue) as Apache Hive. Cloudera.

28 Impala Architecture

29 NoSQL NoSQL is a term used to describe high-performance non-relational databases. NoSQL databases use a variety of data models, including documents, graphs, key-values, and columnar data. Amazon.

30 Types of NoSQL NoSQL - Types of NoSQL Database - part 2

31 NoSQL No PAIN No GAIN #NoSQL no JOIN.

32 CAP Theorem NoSQL & HBase overview

33 New SQL NewSQL

34

35 Communication Channels facebook.com/bilivre BI Livre

36 OBRIGADO