Powerful Insights with Every Click FixStream Agentless Infrastructure Auto-Discovery for Modern IT Operations
The Challenge AIOps is a big shift from traditional ITOA platforms. ITOA was focused on data collection and unification for historical data analysis across domains to resolve problems with observational data. AIOps leverages big data and machine learning techniques to deliver proactive and predictive insights into problems, to recommend as well as automate remedial actions. This helps businesses to proactively plan and identify business impacting issues before they occur. One of the key challenges IT organizations face is the complexity associated with discovering the disparate entities in hybrid IT environment. Environments are quickly changing as digital service deployments adopt newer technologies in the domains of virtualization, hybrid cloud, containers, micro services, etc. The application environment is becoming very dynamic and distributed. Traditional discovery tools and mechanisms provided by legacy ITOA vendors lack the agility, reliability and completeness in keeping up with the changes that take place across a multi-vendor, multi-domain and multitechnology environment. Auto-discovery is becoming a fundamental requirement for today s IT Operations because troubleshooting, capacity planning, maintenance and effective management are dependent on it. 2
FixStream AIOps (Artificial Intelligence for IT Operations) platform delivers correlation, analytics and visualization across end-user business transactions, applications and infrastructure in hybrid IT environment, using innovative big data and machine learning technologies. The FixStream Solution Following are some of the key functionalities provided by the FixStream AIOps platform: Rapid auto-discovery at a rate of over 2000 devices in 30 minutes! Discovery of physical, virtual and logical entities of compute, network and storage across legacy, virtualization, hybrid cloud and containers by using intelligent agentless discovery methods Real-time application-centric view by auto-discovering application services and flows, and then mapping the dependencies with infrastructure entities Contextual application maps correlated with alerts, faults, tickets, log events ingested from different sources via FixStream Open API ingestion layer Time-series event correlation across end user transactions, application and infrastructure layer across multi-vendor and multi-domain environment to identify patterns for root cause analysis and remediation Rapid autodiscovery at a rate of over 2,000 devices in 30 minutes! FixStream s AIOps platform delivers significant ROI on the following critical operational use-cases: Automate root cause analysis and reduce MTTR Accelerate adoption of hybrid cloud technologies Optimize IT resources and reduce infrastructure cost Reduce compliance risk and audit cost 3
Key Characteristics of FixStream Auto-discovery Rapid discovery of thousands of devices in minutes Purely agentless with minimal overhead Ease in discovering unknowns It needs only two high level categories of input read-only user account (known as service account) and network boundaries of discovery in terms of IP address range, subnets, FQDN API based southbound and northbound data ingestion and exploration APIs for configuration, set-up, execution of discovery Automated application discovery, mapping for business context Automated multi-layer (business, application, infrastructure) and multi-domain (network, storage, compute) discovery with a normalized and semantic data model for correlation, mapping, analytics and visualization Concepts of maps (topology, application map) as a result of discovered entities for intuitive exploration of relationships between discovered entities Concepts of overlays for intelligent correlation of operational data such as performance metrics, faults, alerts 4
Agentless Auto-discovery Agentless discovery using smart data collectors FixStream auto-discovery is completely agentless. FixStream smart data collectors are lightweight, developed primarily in python and can be deployed in standard Linux VMs. Data collectors are intelligent to scan IP addresses in network subnets or user input boundaries to quickly learn about the infrastructure entities including vendor make, model, configuration data, topology data such as MAC address, dynamic table, interfaces, VLANs, routing information, etc. Data collectors can be flexibly deployed across different network subnets to provide unlimited scalability for large customer deployment. The data collectors can scale to collect millions of data points from disparate entities in hybrid IT environment. Data is collected from physical and virtual compute, network and storage entities. For a complete list of supported vendors, please refer to FixStream product datasheet. The discovery method uses techniques such as SNMP, SSH, CDP/LLDP, Powershell, WMI, and API based integration to collect relevant data. The Data Collector Module (DCM) has built-in device libraries with appropriate commands and metadata that allows the collection of all the required information without user input. This is where FixStream auto-discovery delivers much richer value to the customers over its competitors. It eliminates the operations overhead typically needed for discovery in a heterogeneous and complex, multi-vendor IT environment. Given the IP address range and appropriate service accounts, FixStream auto-discovery scans thru all the IP addresses in the range and identifies the vendor make/model of the device. It then picks up the corresponding command library to collect 5
subsequent data to deliver the capabilities. Out-of-the-box automated CI relationship discovery and CMDB update FixStream s auto-discovery delivers a complete inventory of the environment and an end-to-end Topology map that shows how the devices are connected including interfaces, vlan, MAC address, etc. FixStream provides connectors with ITSM vendors such as ServiceNow, and ingests the discovered information to enterprise CMDB. This enables out-of-the-box discovery of relationships between Configuration Items (CIs), and automatically updates the CMDB. FixStream also provides inventory information by device type and IP, highlighting IP and CI parameter duplication. 6
Data Collection and Normalization Architecture FixStream s platform architecture has two sub-components DCM and NCE (Normalization Correlation Engine) as represented in the following diagram. The DCM is responsible for data collection and normalization. NCE is the core engine of the platform where the data is processed for contextual correlation, analytics and visualization. FixStream s architecture leverages a Syntax to Semantic architectural approach to keep up with the volume, velocity and variety of challenges of data collection, processing and storage of millions of data points collected from a hybrid IT environment. Data from various compute, network and storage vendor entities are collected using vendor specific syntax and normalized for FixStream standard data model. Data is processed and stored by device type and device sub-type along with the metadata, relationship and associated health and performance attributes. 7
The following diagram shows FixStream semantic data model using device types and device sub-type concept. 8
The following diagram shows a sample topology map to show how network, compute and storage entities (physical, virtual and logical) are interconnected within hybrid IT. 9
Application Discovery and Dependency Mapping Dynamic and automatic discovery of application services, correlated with infrastructure FixStream application discovery and dependency mapping with infrastructure, is dynamic and automated. To discover the application environment and map the dependency to the underlying infrastructure automatically: 1. 2. 3. FixStream discovers the application services within the hosts dynamically and automatically. This is done using an algorithm that reads the process description within the host and using regex expressions, it appropriately gives it a service name and service role. Signature for custom application services can be easily added to the reference file within few hours. Once the application services are discovered, FixStream platform also discovers the unique flows between the services using different flow detection techniques. A unique flow is defined with source IP, source service, destination IP, destination service, destination port 10
Once the application services and corresponding flows are detected, FixStream dynamically discovers groups of interconnected flows by using its proprietary flow clustering algorithm and presents it to the user as a suggested application. The user simply has to provide a business application name and make necessary updates to the discovered flows and hosts. A complete end-toend application map is then automatically computed by extracting the topology for the application entities with its complete dependency with physical and virtual network, storage and compute entities. 11
Flexibility of Connection with New Data Sources FixStream s auto-discovery allows a very agile process to onboard new technologies. New vendors for network, storage and compute as well as other technologies for virtualization, cloud and containers can be easily on-boarded by creating the vendor specific discovery command libraries and developing the parser to parse the output to FixStream s standard normalized data model. Typically, new collectors are built in 1 to 3 months depending on the complexity. New device supports are added during FixStream major releases or ondemand to meet customer needs. Additionally, FixStream provides open southbound APIs (called UDM ) which allows ingestion of performance metrics, faults and performance alerts. These are then correlated with the associated device for overlay on inventory view, topology and application map view. Discovery of Emerging Technology Entities FixStream s auto-discovery supports virtualization, software defined data center, private cloud and public cloud technologies. Hybrid applications running across private data center and AWS or Azure cloud are discovered by using open APIs provided by AWS and Azure and hybrid application maps are created accordingly. The following cloud technologies are supported today: Virtualization Technologies - VMWare, Xen, Hyper-V, KVM Openstack AWS, Azure Docker New technologies are added in each FixStream release. 12
FixStream s auto-discovery strategy for cloud and containerized environment is to discover the most up to date information about the provisioned environment from the controllers such as vcenter, Openstack, Docker engine and subsequently discover additional information as required from the guest OS to build the topology map and application map. Telemetry information such as performance metrics, faults, alerts are discovered from the controller APIs as possible. Additionally, we plan to automate the application mapping process by integrating with application deployment templates such as CloudFormation, and HEAT. The following diagram shows a simple map of application services to underlying infrastructure for a hybrid application deployed across private IT data center and Azure public cloud in a VPC set up. 13
Multi-Layer and Multi-Domain Correlation Multi-layer correlation across Business Application and Infrastructure FixStream s value proposition is to enable multi-layer correlation across domains of hybrid IT. The three layers are business, application and infrastructure which are the critical building blocks for successful delivery of digital services. In order to enable this vision, FixStream provides open APIs to ingest business metrics from different BI sources. FixStream then correlates the business metrics to business processes and then further to the application and underlying infrastructure. It also provides event correlation across the stack in time series to enable playback of critical events, pattern identification and optimal root cause analysis of KPI threshold violation for business metrics caused by issues in hybrid IT stack. 14
Pre-requisites, Set up and Configuration FixStream software is packaged in two sub components and deployed using standard Chef based deployment scripts. The platform can be deployed and configured within as quick as 1 hour and discovery for 2000 devices can be successfully executed in 30 minutes making the discovery process extremely scalable, complete and accurate. The following pre-requisites need to be fulfilled prior to the discovery process to open the access required for discovery process to run successfully. The target devices within data should be reachable from FixStream Data Collector Module (DCM) on appropriate ports. Command level read-only access to the devices is required for application mapping and flow-to-path algorithm. The firewall should be off for internal datacenter network on windows servers and linux servers. For windows servers, discovery & other FixStream features, Administrative share on windows servers should be enabled. Specific Linux commands will be shared to create SUDO user with the required commands execution access. CDP / LLDP should be enabled on all the network devices. One service account on LDAP is required for LDAP integration with FixStream Service account should be able to browse through LDAP. This account will be used as FixStream administrator. SNMP Traps need to be configured on all devices in Data Center to send alerts to our Data Collector Module (DCM). 15
Security FixStream is architecturally secure by using various security standards and guidelines at each layer of the architecture. Data in motion is encrypted using AES 256-bit encryption standards. ElasticSearch which is the primary source of data, is secured using ElasticSearch x-pack security framework. Data stored in intermediate Kafka queues is AES 256 bit encrypted and sensitive data stored in ElasticSearch is AES 256 bit encrypted before storing in ElasticSearch. Access to the UI is secured via oauth 2.0 AAA standards using certificate-based SSL/TLS encrypted HTTPS communication protocol. FixStream provides role-based access to various features of the platform. The platform supports LDAP integration for authentication and authorization. An administrative user id is usually created exclusively for the discovery process to ensure discovery is done by an authorized user. 16
Summary Resources Quick, agentless and automatic discovery of hybrid infrastructure is the need of the hour for IT Operations. FixStream s AIOPs platform is agile and can be easily extended to suit your environment. It provides multi-layer (business, application, infrastructure) and multi-domain (network, storage, compute) correlation, enabling rapid root cause analysis of business impacting issues across the hybrid IT stack. Gartner, Cool Vendors in Enterprise Networking, 2017, 17 April 2017. The Gartner Cool Vendor Logo is a trademark and service mark of Gartner, Inc., and/or its affiliates, and is used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. FixStream 2001 Gateway Place, Suite 520W San Jose, CA 95110, USA www.fixstream.com 2017 FixStream All Rights Reserved. DS 03-20-17 Email: info@fixstream.com 17