Application monitoring with BELK. Nishant Sahay, Sr. Architect Bhavani Ananth, Architect

Size: px
Start display at page:

Download "Application monitoring with BELK. Nishant Sahay, Sr. Architect Bhavani Ananth, Architect"

Transcription

1 Application monitoring with BELK Nishant Sahay, Sr. Architect Bhavani Ananth, Architect

2 Why logs Business PoV Input Data Analytics User Interactions /Behavior End user Experience/ Improvements 2017 Wipro wipro.com confidential 2

3 Why logs Ops PoV Real time monitoring Root cause analysis Anomaly Detection and Predictive Monitoring Debugging Troubleshooting / Support

4 Challenges with log analytics No standard log formats Multiple logging frameworks Issue in normalizing and correlating logs from disparate sources Logs highly decentralized Scalability Issues Limited real time visualization capability 2017 Wipro wipro.com confidential 4

5 BELK to the rescue! 2017 Wipro wipro.com confidential 5

6 Beats Filebeat Metricbeat Packetbeat Heartbeat 2017 Wipro wipro.com confidential 6

7 Filebeat Log data shippers for log files Supports modules, multiline and selective filtering Guarantees at least once delivery Does not parse log lines 2017 Wipro wipro.com confidential 7

8 Metricbeat Collects metrics from OS Collects metrics from services running on the server Periodically pollsaggregates data into documents- pushes to ES Pluggable Architecture a module for each metric type. E.g. Apache module, MySQL module 2017 Wipro wipro.com confidential 8

9 Packetbeat Real-time network packet analyzer Supported protocols DNS, HTTP, MySQL, ICMP, AMQP, Cassandra, PostgreSQL, Redis, Thrift, MongoDB Can run on a dedicated server or in server where the application is hosted Useful in performance analysis; debugging and troubleshooting network and app issues 2017 Wipro wipro.com confidential 9

10 Logstash 2017 Wipro wipro.com confidential 10

11 Collect Parse Transform Store Logstash Input Filter Output File Syslog Redis Beats Grok Mutate Drop Clone geoip Elasticsear ch File Graphite Statsd 2017 Wipro wipro.com confidential 11

12 Elasticsearch 2017 Wipro wipro.com confidential 12

13 Elastic Search: Differentiators Faster search, indexing Ingest node, aggregations Painless scripting language Roll over and shrink API Clustering cross cluster search

14 Kibana 2017 Wipro wipro.com confidential 14

15 Powerful visualizations Basic Charts Line area, bar, heat maps, pie charts Data - Data table, Metric Time series Timelion and Time series visual builder Others Tag Cloud, Markdown Widget 2017 Wipro wipro.com confidential 15

16 BELK in a nutshell Logs Collect - Transform Search - Analyze Visualize - Manage 2017 Wipro wipro.com confidential 16

17 Live demo Wipro wipro.com confidential 17

18 HTTP TCP Demo 1 HTTP NGINX E commerce web MySQL Logstash & Beats Log monitoring foundation components Elasticsearch Kibana Message Queue Spark 2017 Wipro wipro.com confidential 18

19 Demo 1 Demo 1 App Stack components Spark ecosystem Streaming MLib Streaming Spark SQL + Dataframes Message Queue Spark Core API 2 R SQL Python Scala Java 2017 Wipro wipro.com confidential 19

20 Demo 1 Duration since midnight (min) Time Group Sales Amount Duration since midnight (min) Time Group Sales Amount Cluster Generate Model (K Means) Predict 2017 Wipro wipro.com confidential 20

21 Live demo Wipro wipro.com confidential 21

22 HTTP HTTP Demo 2 HTTP Gateway Service Customer Service Account Service Logstash & Beats Log monitoring foundation components Elasticsearch Kibana Zipkin 2017 Wipro wipro.com confidential 22

23 Deployment architectures 2017 Wipro wipro.com confidential 23

24 Deployment architecture 1 *** Wipro wipro.com confidential 24

25 Deployment architecture 2 *** Wipro wipro.com confidential 25

26 Deployment architecture 3 *** Wipro wipro.com confidential 26

27 Have a great day! Nishant & Bhavani 2017 Wipro wipro.com confidential 27

Monitor your containers with the Elastic Stack. Monica Sarbu

Monitor your containers with the Elastic Stack. Monica Sarbu Monitor your containers with the Elastic Stack Monica Sarbu Monica Sarbu Team lead, Beats team monica@elastic.co 3 Monitor your containers with the Elastic Stack Elastic Stack 5 Beats are lightweight shippers

More information

Monitor your infrastructure with the Elastic Beats. Monica Sarbu

Monitor your infrastructure with the Elastic Beats. Monica Sarbu Monitor your infrastructure with the Elastic Beats Monica Sarbu Monica Sarbu Team lead, Beats team Email: monica@elastic.co Twitter: 2 Monitor your servers Apache logs 3 Monitor your servers Apache logs

More information

Ingest. Aaron Mildenstein, Consulting Architect Tokyo Dec 14, 2017

Ingest. Aaron Mildenstein, Consulting Architect Tokyo Dec 14, 2017 Ingest Aaron Mildenstein, Consulting Architect Tokyo Dec 14, 2017 Data Ingestion The process of collecting and importing data for immediate use 2 ? Simple things should be simple. Shay Banon Elastic{ON}

More information

Ingest. David Pilato, Developer Evangelist Paris, 31 Janvier 2017

Ingest. David Pilato, Developer Evangelist Paris, 31 Janvier 2017 Ingest David Pilato, Developer Evangelist Paris, 31 Janvier 2017 Data Ingestion The process of collecting and importing data for immediate use in a datastore 2 ? Simple things should be simple. Shay Banon

More information

Fluentd + MongoDB + Spark = Awesome Sauce

Fluentd + MongoDB + Spark = Awesome Sauce Fluentd + MongoDB + Spark = Awesome Sauce Nishant Sahay, Sr. Architect, Wipro Limited Bhavani Ananth, Tech Manager, Wipro Limited Your company logo here Wipro Open Source Practice: Vision & Mission Vision

More information

Unifying logs and metrics data with Elastic Beats. Monica Sarbu Team lead, Elastic Beats

Unifying logs and metrics data with Elastic Beats. Monica Sarbu Team lead, Elastic Beats Unifying logs and metrics data with Elastic Beats Monica Sarbu Team lead, Elastic Beats # Who am I Team lead at Elastic Beats Software engineer Joined Elastic 1 year ago @monicasarbu http://github.com/monicasarbu

More information

BUILDING HA ELK STACK FOR DRUPAL

BUILDING HA ELK STACK FOR DRUPAL BUILDING STACK FOR DRUPAL Marji Cermak DevOps track, Experience level: Intermediate Marji Cermak Systems Engineer at @cermakm Scope of this presentation technical talk targeting sysadmins and systems savvy

More information

About the Tutorial. Audience. Prerequisites. Copyright and Disclaimer. Logstash

About the Tutorial. Audience. Prerequisites. Copyright and Disclaimer. Logstash About the Tutorial is an open-source, centralized, events and logging manager. It is a part of the ELK (ElasticSearch,, Kibana) stack. In this tutorial, we will understand the basics of, its features,

More information

Amazon Elasticsearch Service

Amazon Elasticsearch Service Amazon Elasticsearch Service Fully managed, reliable, and scalable Elasticsearch service. Have Your Frontend & Monitor It Too Scalable Log Analytics Inside a VPC Lab Instructions Contents Lab Overview...

More information

The webinar will start soon... Elasticsearch Performance Optimisation

The webinar will start soon... Elasticsearch Performance Optimisation The webinar will start soon... Performance Optimisation 1 whoami Alan Hardy Sr. Solutions Architect NEMEA 2 Webinar Housekeeping & Logistics Slides and recording will be available following the webinar

More information

Harvesting Logs and Events Using MetaCentrum Virtualization Services. Radoslav Bodó, Daniel Kouřil CESNET

Harvesting Logs and Events Using MetaCentrum Virtualization Services. Radoslav Bodó, Daniel Kouřil CESNET Harvesting Logs and Events Using MetaCentrum Virtualization Services Radoslav Bodó, Daniel Kouřil CESNET Campus network monitoring and security workshop Prague 2014 Agenda Introduction Collecting logs

More information

Log Analysis When CLI get's complex. ITNOG3 Octavio Melendres Network admin - Fastnet Spa

Log Analysis When CLI get's complex. ITNOG3 Octavio Melendres Network admin - Fastnet Spa Log Analysis When CLI get's complex ITNOG3 Octavio Melendres Network admin - Fastnet Spa Introduction Network engineer at Fastnet Spa from 2003 Fastnet Spa is an ISP from Marche Region located in Ancona

More information

Infrastructure at your Service. Elking your PostgreSQL Database Infrastructure

Infrastructure at your Service. Elking your PostgreSQL Database Infrastructure Infrastructure at your Service. About me Infrastructure at your Service. Arnaud Berbier Senior Consultant +41 79 128 91 45 arnaud.berbier@dbi-services.com Page 2 Agenda 1.Playground Infrastructure 2.Elastic

More information

Ruby in the Sky with Diamonds. August, 2014 Sao Paulo, Brazil

Ruby in the Sky with Diamonds. August, 2014 Sao Paulo, Brazil Ruby in the Sky with Diamonds August, 2014 Sao Paulo, Brazil JELASTIC PLATFORM AS INFRASTRUCTURE Jelastic provides enterprise cloud software that redefines the economics of cloud deployment and management.

More information

Amazon Search Services. Christoph Schmitter

Amazon Search Services. Christoph Schmitter Amazon Search Services Christoph Schmitter csc@amazon.de What we'll cover Overview of Amazon Search Services Understand the difference between Cloudsearch and Amazon ElasticSearch Service Q&A Amazon Search

More information

E l a s t i c s e a r c h F e a t u r e s. Contents

E l a s t i c s e a r c h F e a t u r e s. Contents Elasticsearch Features A n Overview Contents Introduction... 2 Location Based Search... 2 Search Social Media(Twitter) data from Elasticsearch... 4 Query Boosting in Elasticsearch... 4 Machine Learning

More information

The SMACK Stack: Spark*, Mesos*, Akka, Cassandra*, Kafka* Elizabeth K. Dublin Apache Kafka Meetup, 30 August 2017.

The SMACK Stack: Spark*, Mesos*, Akka, Cassandra*, Kafka* Elizabeth K. Dublin Apache Kafka Meetup, 30 August 2017. Dublin Apache Kafka Meetup, 30 August 2017 The SMACK Stack: Spark*, Mesos*, Akka, Cassandra*, Kafka* Elizabeth K. Joseph @pleia2 * ASF projects 1 Elizabeth K. Joseph, Developer Advocate Developer Advocate

More information

Creating a Recommender System. An Elasticsearch & Apache Spark approach

Creating a Recommender System. An Elasticsearch & Apache Spark approach Creating a Recommender System An Elasticsearch & Apache Spark approach My Profile SKILLS Álvaro Santos Andrés Big Data & Analytics Solution Architect in Ericsson with more than 12 years of experience focused

More information

Using DC/OS for Continuous Delivery

Using DC/OS for Continuous Delivery Using DC/OS for Continuous Delivery DevPulseCon 2017 Elizabeth K. Joseph, @pleia2 Mesosphere 1 Elizabeth K. Joseph, Developer Advocate, Mesosphere 15+ years working in open source communities 10+ years

More information

Big Data Architect.

Big Data Architect. Big Data Architect www.austech.edu.au WHAT IS BIG DATA ARCHITECT? A big data architecture is designed to handle the ingestion, processing, and analysis of data that is too large or complex for traditional

More information

End-to-End Security Analytics with the Elastic Stack. Samir Bennacer

End-to-End Security Analytics with the Elastic Stack. Samir Bennacer End-to-End Security Analytics with the Elastic Stack Samir Bennacer!1 !2 Attacks are inevitable Cybersecurity Maturity Curve Phase 1 Security Event Management Phase 2 Automation Phase 3 Proactive Analytics

More information

Building a Scalable Recommender System with Apache Spark, Apache Kafka and Elasticsearch

Building a Scalable Recommender System with Apache Spark, Apache Kafka and Elasticsearch Nick Pentreath Nov / 14 / 16 Building a Scalable Recommender System with Apache Spark, Apache Kafka and Elasticsearch About @MLnick Principal Engineer, IBM Apache Spark PMC Focused on machine learning

More information

Elasticsearch & ATLAS Data Management. European Organization for Nuclear Research (CERN)

Elasticsearch & ATLAS Data Management. European Organization for Nuclear Research (CERN) Elasticsearch & ATAS Data Management European Organization for Nuclear Research (CERN) ralph.vigne@cern.ch mario.lassnig@cern.ch ATAS Analytics Platform proposed eb. 2015; work in progress; correlate data

More information

Jupyter and Spark on Mesos: Best Practices. June 21 st, 2017

Jupyter and Spark on Mesos: Best Practices. June 21 st, 2017 Jupyter and Spark on Mesos: Best Practices June 2 st, 207 Agenda About me What is Spark & Jupyter Demo How Spark+Mesos+Jupyter work together Experience Q & A About me Graduated from EE @ Tsinghua Univ.

More information

Elasticsearch. Presented by: Steve Mayzak, Director of Systems Engineering Vince Marino, Account Exec

Elasticsearch. Presented by: Steve Mayzak, Director of Systems Engineering Vince Marino, Account Exec Elasticsearch Presented by: Steve Mayzak, Director of Systems Engineering Vince Marino, Account Exec What about Elasticsearch the Company?! Support 100s of Companies in Production environments Training

More information

The State Of Open Source Logging

The State Of Open Source Logging The State Of Open Source Logging Rashid Khan (@rashidkpc) Shay Banon (@kimchy) Rashid Khan Developer @ elasticsearch Operations guy Logging Nerd Kibana project IRC/Twitter: rashidkpc Logs suck. 3am What

More information

Ingest Node: (re)indexing and enriching documents within

Ingest Node: (re)indexing and enriching documents within Ingest Node: (re)indexing and enriching documents within Elasticsearch @lucacavanna # Agenda 1 Why ingest node? 2 How does it work? 3 Where can it be used? 2 # Why ingest node? # I just want to tail a

More information

ELK Stack Elasticsearch, Logstash, Kibana

ELK Stack Elasticsearch, Logstash, Kibana www.netways.de ELK Stack Elasticsearch, Logstash, Kibana Munich 19.10.2015 INTRODUCTION Bernd Erk CEO at NETWAYS GmbH Co-Founder Icinga @gethash info@netways.de NETWAYS GmbH Open Source Service Provider

More information

Big Data Hadoop Developer Course Content. Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours

Big Data Hadoop Developer Course Content. Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours Big Data Hadoop Developer Course Content Who is the target audience? Big Data Hadoop Developer - The Complete Course Course Duration: 45 Hours Complete beginners who want to learn Big Data Hadoop Professionals

More information

Using AWS to Build a Large Scale Dockerized Microservices Architecture. Dr. Oliver Wahlen moovel Group GmbH Frankfurt, 30.

Using AWS to Build a Large Scale Dockerized Microservices Architecture. Dr. Oliver Wahlen moovel Group GmbH Frankfurt, 30. Using AWS to Build a Large Scale Dockerized Microservices Architecture Dr. Oliver Wahlen moovel Group GmbH Frankfurt, 30. Juni 2016 The moovel Group GmbH Our vision is an ecosystem that simplifies mobility

More information

FUJITSU Software ServerView Cloud Monitoring Manager V1.1. Release Notes

FUJITSU Software ServerView Cloud Monitoring Manager V1.1. Release Notes FUJITSU Software ServerView Cloud Monitoring Manager V1.1 Release Notes J2UL-2170-01ENZ0(00) July 2016 Contents Contents About this Manual... 4 1 What's New?...6 1.1 Performance Improvements... 6 1.2

More information

Wrangling Logs with Logstash and ElasticSearch

Wrangling Logs with Logstash and ElasticSearch Wrangling Logs with Logstash and ElasticSearch Nate Jones & David Castro Media Temple OSCON 2012 Why are we here? Size Quantity Efficiency Access Locality Method Filtering Grokability Noise Structure Metrics

More information

Are you visualizing your logfiles? Bastian Widmer

Are you visualizing your logfiles? Bastian Widmer Are you visualizing your logfiles? Bastian Widmer / @dasrecht Visualizing Logfiles with ELK Stack Bastian Widmer / @dasrecht Hola Com estàs? Bastian Widmer @dasrecht / bastianwidmer.ch DrupalCI: Modernizing

More information

Overview. Prerequisites. Course Outline. Course Outline :: Apache Spark Development::

Overview. Prerequisites. Course Outline. Course Outline :: Apache Spark Development:: Title Duration : Apache Spark Development : 4 days Overview Spark is a fast and general cluster computing system for Big Data. It provides high-level APIs in Scala, Java, Python, and R, and an optimized

More information

LOG AGGREGATION. To better manage your Red Hat footprint. Miguel Pérez Colino Strategic Design Team - ISBU

LOG AGGREGATION. To better manage your Red Hat footprint. Miguel Pérez Colino Strategic Design Team - ISBU LOG AGGREGATION To better manage your Red Hat footprint Miguel Pérez Colino Strategic Design Team - ISBU 2017-05-03 @mmmmmmpc Agenda Managing your Red Hat footprint with Log Aggregation The Situation The

More information

利用 Mesos 打造高延展性 Container 環境. Frank, Microsoft MTC

利用 Mesos 打造高延展性 Container 環境. Frank, Microsoft MTC 利用 Mesos 打造高延展性 Container 環境 Frank, Microsoft MTC About Me Developer @ Yahoo! DevOps @ HTC Technical Architect @ MSFT Agenda About Docker Manage containers Apache Mesos Mesosphere DC/OS application = application

More information

Big Data Technology Ecosystem. Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara

Big Data Technology Ecosystem. Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara Big Data Technology Ecosystem Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara Agenda End-to-End Data Delivery Platform Ecosystem of Data Technologies Mapping an End-to-End Solution Case

More information

Table 1 The Elastic Stack use cases Use case Industry or vertical market Operational log analytics: Gain real-time operational insight, reduce Mean Ti

Table 1 The Elastic Stack use cases Use case Industry or vertical market Operational log analytics: Gain real-time operational insight, reduce Mean Ti Solution Overview Cisco UCS Integrated Infrastructure for Big Data with the Elastic Stack Cisco and Elastic deliver a powerful, scalable, and programmable IT operations and security analytics platform

More information

Securing the Elastic Stack

Securing the Elastic Stack Securing the Elastic Stack Jay Modi, Security Software Engineer Tim Vernum, Security Software Engineer Elastic March 1st, 2018 @jaymode2001 @TimVernum Authentication Who are you? 3 Built-in Users elastic

More information

Microservices log gathering, processing and storing

Microservices log gathering, processing and storing Microservices log gathering, processing and storing Siim-Toomas Marran Univeristy of Tartu J.Liivi 2 Tartu, Estonia siimtoom@ut.ee ABSTRACT The aim of this work is to investigate and implement one of the

More information

Blended Learning Outline: Developer Training for Apache Spark and Hadoop (180404a)

Blended Learning Outline: Developer Training for Apache Spark and Hadoop (180404a) Blended Learning Outline: Developer Training for Apache Spark and Hadoop (180404a) Cloudera s Developer Training for Apache Spark and Hadoop delivers the key concepts and expertise need to develop high-performance

More information

The Future of Real-Time in Spark

The Future of Real-Time in Spark The Future of Real-Time in Spark Reynold Xin @rxin Spark Summit, New York, Feb 18, 2016 Why Real-Time? Making decisions faster is valuable. Preventing credit card fraud Monitoring industrial machinery

More information

The Art of Container Monitoring. Derek Chen

The Art of Container Monitoring. Derek Chen The Art of Container Monitoring Derek Chen 2016.9.22 About me DevOps Engineer at Trend Micro Agile transformation Micro service and cloud service Docker integration Monitoring system development Automate

More information

Increase Value from Big Data with Real-Time Data Integration and Streaming Analytics

Increase Value from Big Data with Real-Time Data Integration and Streaming Analytics Increase Value from Big Data with Real-Time Data Integration and Streaming Analytics Cy Erbay Senior Director Striim Executive Summary Striim is Uniquely Qualified to Solve the Challenges of Real-Time

More information

Bring Context To Your Machine Data With Hadoop, RDBMS & Splunk

Bring Context To Your Machine Data With Hadoop, RDBMS & Splunk Bring Context To Your Machine Data With Hadoop, RDBMS & Splunk Raanan Dagan and Rohit Pujari September 25, 2017 Washington, DC Forward-Looking Statements During the course of this presentation, we may

More information

Search Engines and Time Series Databases

Search Engines and Time Series Databases Università degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Search Engines and Time Series Databases Corso di Sistemi e Architetture per Big Data A.A. 2017/18

More information

STATE OF MODERN APPLICATIONS IN THE CLOUD

STATE OF MODERN APPLICATIONS IN THE CLOUD STATE OF MODERN APPLICATIONS IN THE CLOUD 2017 Introduction The Rise of Modern Applications What is the Modern Application? Today s leading enterprises are striving to deliver high performance, highly

More information

Developing Enterprise Cloud Solutions with Azure

Developing Enterprise Cloud Solutions with Azure Developing Enterprise Cloud Solutions with Azure Java Focused 5 Day Course AUDIENCE FORMAT Developers and Software Architects Instructor-led with hands-on labs LEVEL 300 COURSE DESCRIPTION This course

More information

Analyze Bug Statistics using Kibana Dashboard and Get Voice Alerts

Analyze Bug Statistics using Kibana Dashboard and Get Voice Alerts Analyze Bug Statistics using Kibana Dashboard and Get Voice Alerts Kibana Dashboard Elast Alert Sensiple Notification System Abstract This white paper describes how Kibana Dashboard can be used to analyze

More information

São Paulo. August,

São Paulo. August, São Paulo August, 28 2018 Going Cloud Native with Cloud Foundry Luis Macedo Sr Platform Engineer, Pivotal @luis0macedo What is Cloud Native Platform? - A platform that delivers predictable deployment

More information

Specialist ICT Learning

Specialist ICT Learning Specialist ICT Learning APPLIED DATA SCIENCE AND BIG DATA ANALYTICS GTBD7 Course Description This intensive training course provides theoretical and technical aspects of Data Science and Business Analytics.

More information

Whooo s calling Whooo?

Whooo s calling Whooo? Whooo s calling Whooo? Jodi Spacek Hootsuite March 11, 2016 Part 1: Microservice Migration How we adjust to our ever-changing environment leading to reasons why microservice calls are hard to track Part

More information

Data Ingestion at Scale. Jeffrey Sica

Data Ingestion at Scale. Jeffrey Sica Data Ingestion at Scale Jeffrey Sica ARC-TS @jeefy Overview What is Data Ingestion? Concepts Use Cases GPS collection with mobile devices Collecting WiFi data from WAPs Sensor data from manufacturing machines

More information

Overview. SUSE OpenStack Cloud Monitoring

Overview. SUSE OpenStack Cloud Monitoring Overview SUSE OpenStack Cloud Monitoring Overview SUSE OpenStack Cloud Monitoring Publication Date: 08/04/2017 SUSE LLC 10 Canal Park Drive Suite 200 Cambridge MA 02141 USA https://www.suse.com/documentation

More information

BIG DATA COURSE CONTENT

BIG DATA COURSE CONTENT BIG DATA COURSE CONTENT [I] Get Started with Big Data Microsoft Professional Orientation: Big Data Duration: 12 hrs Course Content: Introduction Course Introduction Data Fundamentals Introduction to Data

More information

Unifying Big Data Workloads in Apache Spark

Unifying Big Data Workloads in Apache Spark Unifying Big Data Workloads in Apache Spark Hossein Falaki @mhfalaki Outline What s Apache Spark Why Unification Evolution of Unification Apache Spark + Databricks Q & A What s Apache Spark What is Apache

More information

Filebeat is able to do multiline while collecting logs from the container. you can use autodiscover to configure it in many ways

Filebeat is able to do multiline while collecting logs from the container. you can use autodiscover to configure it in many ways elasticguest8086 Since k8 by default is using the json log file driver which will not support call trace (multi-line log).. Any solution for it with ELK? like using logstash as a logging driver? Filebeat

More information

Fluentd. Open Source Data Collector. Eduardo Jan 23, 2016 Scale14x, Pasadena!

Fluentd. Open Source Data Collector. Eduardo Jan 23, 2016 Scale14x, Pasadena! Fluentd Open Source Data Collector Jan 23, 2016 Scale14x, Pasadena! Eduardo Silva eduardo@treasuredata.com @edsiper spread the word! #scale14x #fluentd @edsiper About Me Eduardo Silva Github & Twitter

More information

Big Data. Big Data Analyst. Big Data Engineer. Big Data Architect

Big Data. Big Data Analyst. Big Data Engineer. Big Data Architect Big Data Big Data Analyst INTRODUCTION TO BIG DATA ANALYTICS ANALYTICS PROCESSING TECHNIQUES DATA TRANSFORMATION & BATCH PROCESSING REAL TIME (STREAM) DATA PROCESSING Big Data Engineer BIG DATA FOUNDATION

More information

AWS 101. Patrick Pierson, IonChannel

AWS 101. Patrick Pierson, IonChannel AWS 101 Patrick Pierson, IonChannel What is AWS? Amazon Web Services (AWS) is a secure cloud services platform, offering compute power, database storage, content delivery and other functionality to help

More information

Search and Time Series Databases

Search and Time Series Databases Università degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Search and Time Series Databases Corso di Sistemi e Architetture per Big Data A.A. 2016/17 Valeria

More information

Monitoring for IT Services and WLCG. Alberto AIMAR CERN-IT for the MONIT Team

Monitoring for IT Services and WLCG. Alberto AIMAR CERN-IT for the MONIT Team Monitoring for IT Services and WLCG Alberto AIMAR CERN-IT for the MONIT Team 2 Outline Scope and Mandate Architecture and Data Flow Technologies and Usage WLCG Monitoring IT DC and Services Monitoring

More information

Cloud Computing & Visualization

Cloud Computing & Visualization Cloud Computing & Visualization Workflows Distributed Computation with Spark Data Warehousing with Redshift Visualization with Tableau #FIUSCIS School of Computing & Information Sciences, Florida International

More information

Go Faster: Containers, Platforms and the Path to Better Software Development (Including Live Demo)

Go Faster: Containers, Platforms and the Path to Better Software Development (Including Live Demo) RED HAT DAYS VANCOUVER Go Faster: Containers, Platforms and the Path to Better Software Development (Including Live Demo) Paul Armstrong Principal Solutions Architect Gerald Nunn Senior Middleware Solutions

More information

Java Architectures A New Hope. Eberhard Wolff

Java Architectures A New Hope. Eberhard Wolff Java Architectures A New Hope Eberhard Wolff http://ewolff.com What happens with a talk titled like this? Architecture of Enterprise Java Apps How can I implement a new feature??? ! ECommerce System

More information

Container 2.0. Container: check! But what about persistent data, big data or fast data?!

Container 2.0. Container: check! But what about persistent data, big data or fast data?! @unterstein @joerg_schad @dcos @jaxdevops Container 2.0 Container: check! But what about persistent data, big data or fast data?! 1 Jörg Schad Distributed Systems Engineer @joerg_schad Johannes Unterstein

More information

DISQUS. Continuous Deployment Everything. David

DISQUS. Continuous Deployment Everything. David DISQUS Continuous Deployment Everything David Cramer @zeeg Continuous Deployment Shipping new code as soon as it s ready (It s really just super awesome buildbots) Workflow Commit (master) Integration

More information

Introducing Jaeger 1.0

Introducing Jaeger 1.0 Introducing Jaeger 1.0 Yuri Shkuro (Uber Technologies) CNCF Webinar Series, Jan-16-2018 1 Agenda What is distributed tracing Jaeger in a HotROD Jaeger under the hood Jaeger v1.0 Roadmap Project governance,

More information

Powering Monitoring Analytics with ELK stack

Powering Monitoring Analytics with ELK stack Powering Monitoring Analytics with ELK stack Abdelkader Lahmadi, Frédéric Beck To cite this version: Abdelkader Lahmadi, Frédéric Beck. Powering Monitoring Analytics with ELK stack. 9th International Conference

More information

FUJITSU Software ServerView Cloud Monitoring Manager V1.0. Overview

FUJITSU Software ServerView Cloud Monitoring Manager V1.0. Overview FUJITSU Software ServerView Cloud Monitoring Manager V1.0 Overview J2UL-2073-01ENZ0(00) November 2015 Trademarks Copyright FUJITSU LIMITED 2015 LINUX is a registered trademark of Linus Torvalds. The OpenStack

More information

What to expect from the session Technical recap VMware Cloud on AWS {Sample} Integration use case Services introduction & solution designs Solution su

What to expect from the session Technical recap VMware Cloud on AWS {Sample} Integration use case Services introduction & solution designs Solution su LHC3376BES AWS Native Services Integration with VMware Cloud on AWS Technical Deep Dive Ian Massingham, Worldwide Lead, AWS Technical Evangelism Paul Bockelman, AWS Principal Solutions Architect (WWPS)

More information

Towards a Real- time Processing Pipeline: Running Apache Flink on AWS

Towards a Real- time Processing Pipeline: Running Apache Flink on AWS Towards a Real- time Processing Pipeline: Running Apache Flink on AWS Dr. Steffen Hausmann, Solutions Architect Michael Hanisch, Manager Solutions Architecture November 18 th, 2016 Stream Processing Challenges

More information

Scaling Pinterest. Marty Weiner Level 83 Interwebz Geek

Scaling Pinterest. Marty Weiner Level 83 Interwebz Geek Scaling Pinterest Marty Weiner Level 83 Interwebz Geek Evolution Growth March 2010 Page views per day RackSpace 1 small Web Engine 1 small MySQL DB 1 Engineer + 2 Founders Mar 2010 Jan 2011 Jan 2012 May

More information

API Connect. Arnauld Desprets - Technical Sale

API Connect. Arnauld Desprets - Technical Sale API Connect Arnauld Desprets - arnauld_desprets@fr.ibm.com Technical Sale 0 Agenda 1. API Understanding the space 2. API Connect 3. Sample implementations 4. Démonstration 1 sales introduction growth decline

More information

Ninja Level Infrastructure Monitoring. Defensive Approach to Security Monitoring and Automation

Ninja Level Infrastructure Monitoring. Defensive Approach to Security Monitoring and Automation Ninja Level Infrastructure Monitoring Defensive Approach to Security Monitoring and Automation 1 DEFCON 24 06 th August 2016, Saturday 10:00-14:00 Madhu Akula & Riyaz Walikar Appsecco.com 2 About Automation

More information

Mothra: A Large-Scale Data Processing Platform for Network Security Analysis

Mothra: A Large-Scale Data Processing Platform for Network Security Analysis Mothra: A Large-Scale Data Processing Platform for Network Security Analysis Tony Cebzanov Software Engineering Institute Carnegie Mellon University Pittsburgh, PA 15213 REV-03.18.2016.0 1 Agenda Introduction

More information

Implementing Microservices Tracing with Spring Cloud and Zipkin

Implementing Microservices Tracing with Spring Cloud and Zipkin Implementing Microservices Tracing with Spring Cloud and Zipkin Marcin Grzejszczak, @mgrzejszczak 1 2017 Pivotal About me Spring Cloud developer at Pivotal Working mostly on Spring Cloud Sleuth Spring

More information

Tungsten Replicator for Kafka, Elasticsearch, Cassandra

Tungsten Replicator for Kafka, Elasticsearch, Cassandra Tungsten Replicator for Kafka, Elasticsearch, Cassandra Topics In todays session Replicator Basics Filtering and Glue Kafka and Options Elasticsearch and Options Cassandra Future Direction 2 Asynchronous

More information

We are ready to serve Latest Testing Trends, Are you ready to learn?? New Batches Info

We are ready to serve Latest Testing Trends, Are you ready to learn?? New Batches Info We are ready to serve Latest Testing Trends, Are you ready to learn?? New Batches Info START DATE : TIMINGS : DURATION : TYPE OF BATCH : FEE : FACULTY NAME : LAB TIMINGS : PH NO: 9963799240, 040-40025423

More information

Unifying Events & Logs into the Cloud

Unifying Events & Logs into the Cloud Unifying Events & Logs into the Cloud October 05, 2015 CloudOpen/LinuxCon, Dublin Eduardo Silva eduardo@treasuredata.com @edsiper About Me Eduardo Silva Github & Twitter Personal Blog @edsiper http://edsiper.linuxchile.cl

More information

AWS Lambda + nodejs Hands-On Training

AWS Lambda + nodejs Hands-On Training AWS Lambda + nodejs Hands-On Training (4 Days) Course Description & High Level Contents AWS Lambda is changing the way that we build systems in the cloud. This new compute service in the cloud runs your

More information

UNDER THE HOOD. ROGER NUNN Principal Architect/EMEA Solution Manager 21/01/2015

UNDER THE HOOD. ROGER NUNN Principal Architect/EMEA Solution Manager 21/01/2015 UNDER THE HOOD 1 ROGER NUNN rnunn@redhat.com Principal Architect/EMEA Solution Manager 21/01/2015 TOPICS CONTEXT AVAILABILITY UNDER THE HOOD INTEGRATION 2 TOPICS CONTEXT AVAILABILITY UNDER THE HOOD INTEGRATION

More information

Personal Statement. Skillset I MongoDB / Cassandra / Redis / CouchDB. My name is Dale-Kurt Murray. I'm a Solutiof

Personal Statement. Skillset I MongoDB / Cassandra / Redis / CouchDB. My name is Dale-Kurt Murray. I'm a Solutiof My name is Dale-Kurt Murray. 'm a Solutiof +1 876 345 7375 Architect who loves new challenging probl :i "rite hello@dalekurtmurray.com which allows me to think outside of the box. visit www.dalekurtmurray.com

More information

Cloud Computing 3. CSCI 4850/5850 High-Performance Computing Spring 2018

Cloud Computing 3. CSCI 4850/5850 High-Performance Computing Spring 2018 Cloud Computing 3 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning

More information

VMware Cloud on AWS Technical Deck VMware, Inc.

VMware Cloud on AWS Technical Deck VMware, Inc. VMware Cloud on AWS Technical Deck # 2 Enterprise Adoption Driving Strong Growth of Public Cloud Infrastructure as a Service, According to IDC. Press release. IDC. July 14, 2016 3 Cloud Building Challenges

More information

Streamline AWS Security Incidents

Streamline AWS Security Incidents IMF 2018 Streamline AWS Security Incidents Asif Matadar @d1r4c #whoami Director of Endpoint Detection & Response (EDR), EMEA at Tanium Seasoned Incident Response professional with over 7 years experience

More information

Spotfire and Tableau Positioning. Summary

Spotfire and Tableau Positioning. Summary Licensed for distribution Summary So how do the products compare? In a nutshell Spotfire is the more sophisticated and better performing visual analytics platform, and this would be true of comparisons

More information

Accelerate MySQL for Demanding OLAP and OLTP Use Case with Apache Ignite December 7, 2016

Accelerate MySQL for Demanding OLAP and OLTP Use Case with Apache Ignite December 7, 2016 Accelerate MySQL for Demanding OLAP and OLTP Use Case with Apache Ignite December 7, 2016 Nikita Ivanov CTO and Co-Founder GridGain Systems Peter Zaitsev CEO and Co-Founder Percona About the Presentation

More information

Cloud Computing 2. CSCI 4850/5850 High-Performance Computing Spring 2018

Cloud Computing 2. CSCI 4850/5850 High-Performance Computing Spring 2018 Cloud Computing 2 CSCI 4850/5850 High-Performance Computing Spring 2018 Tae-Hyuk (Ted) Ahn Department of Computer Science Program of Bioinformatics and Computational Biology Saint Louis University Learning

More information

File system, 199 file trove-guestagent.conf, 40 flavor-create command, 108 flavor-related APIs list, 280 show details, 281 Flavors, 107

File system, 199 file trove-guestagent.conf, 40 flavor-create command, 108 flavor-related APIs list, 280 show details, 281 Flavors, 107 Index A Amazon AWS, 7, 10 Amazon RDS DBaaS solutions, 10 service vs. platform, 8 single-tenant DBaaS solution, 6 Amazon RedShift DBaaS solutions, 10 single-tenant DBaaS solution, 6 AMD-V, 17 AMQP server

More information

Luigi Build Data Pipelines of batch jobs. - Pramod Toraskar

Luigi Build Data Pipelines of batch jobs. - Pramod Toraskar Luigi Build Data Pipelines of batch jobs - Pramod Toraskar I am a Principal Solution Engineer & Pythonista with more than 8 years of work experience, Works for a Red Hat India an open source solutions

More information

Beyond 1001 Dedicated Data Service Instances

Beyond 1001 Dedicated Data Service Instances Beyond 1001 Dedicated Data Service Instances Introduction The Challenge Given: Application platform based on Cloud Foundry to serve thousands of apps Application Runtime Many platform users - who don

More information

App Service Overview. Rand Pagels Azure Technical Specialist - Application Development US Great Lakes Region

App Service Overview. Rand Pagels Azure Technical Specialist - Application Development US Great Lakes Region App Service Overview Quickly create powerful cloud apps using a fully-managed platform Rand Pagels Azure Technical Specialist - Application Development US Great Lakes Region Security & Management Platform

More information

jetnexus Virtual Load Balancer

jetnexus Virtual Load Balancer jetnexus Virtual Load Balancer Mitigate the Risk of Downtime and Optimise Application Delivery We were looking for a robust yet easy to use solution that would fit in with our virtualisation policy and

More information

jetnexus Virtual Load Balancer

jetnexus Virtual Load Balancer jetnexus Virtual Load Balancer Mitigate the Risk of Downtime and Optimise Application Delivery We were looking for a robust yet easy to use solution that would fit in with our virtualisation policy and

More information

Overview. : Cloudera Data Analyst Training. Course Outline :: Cloudera Data Analyst Training::

Overview. : Cloudera Data Analyst Training. Course Outline :: Cloudera Data Analyst Training:: Module Title Duration : Cloudera Data Analyst Training : 4 days Overview Take your knowledge to the next level Cloudera University s four-day data analyst training course will teach you to apply traditional

More information

Distributed CI: Scaling Jenkins on Mesos and Marathon. Roger Ignazio Puppet Labs, Inc. MesosCon 2015 Seattle, WA

Distributed CI: Scaling Jenkins on Mesos and Marathon. Roger Ignazio Puppet Labs, Inc. MesosCon 2015 Seattle, WA Distributed CI: Scaling Jenkins on Mesos and Marathon Roger Ignazio Puppet Labs, Inc. MesosCon 2015 Seattle, WA About Me Roger Ignazio QE Automation Engineer Puppet Labs, Inc. @rogerignazio Mesos In Action

More information

Accenture Cloud Platform Serverless Journey

Accenture Cloud Platform Serverless Journey ARC202 Accenture Cloud Platform Serverless Journey Tom Myers, Sr. Cloud Architect, Accenture Cloud Platform Matt Lancaster, Lightweight Architectures Global Lead November 29, 2016 2016, Amazon Web Services,

More information

Developing Cloud Based Log Analytic Services

Developing Cloud Based Log Analytic Services Selvan Arumugam Developing Cloud Based Log Analytic Services Helsinki Metropolia University of Applied Sciences Master Of Engineering Information Technology Master s Thesis 26 May 2017 Preface This graduate

More information

AUTOMATE CLOUD RECOVERY it s the only way to be sure Ashley Neely Sr. Solution Strategist 2017 Proprietary and Confidential

AUTOMATE CLOUD RECOVERY it s the only way to be sure Ashley Neely Sr. Solution Strategist 2017 Proprietary and Confidential LHC1951BE Let OVH Help You Automate your Cloud Recovery for when you are Nuked from Orbit: It s the only way to be sure. VMworld 2017 Content: Not for publication Ashley Neely Sr. Solution Strategist #VMworld

More information