Cisco IT Tetration Deployment, Part 1 of 2

Cisco IT ACI Deployment White Papers Cisco IT Tetration Deployment, Part 1 of 2 This is the fifth white paper in a series of case studies that explain how Cisco IT deployed ACI to deliver improved business performance. These in-depth case studies cover the Cisco IT ACI data center design, migration to ACI, the ACI NetApp storage area network deployment, compute at scale with AVS, UCS, KVM, and VMware, server load balancing, Tetration analytics (parts 1 and 2), and ACI automation. These white papers will enable field engineers and customer IT architects to assess the product, plan deployments, and exploit its application centric properties to flexibly deploy and manage robust highly scalable integrated data center and network resources. Contributors to this white paper from the Cisco IT include Benny Van De Voorde, Principal Engineer. Publication Date: October 25, 2017 2016 Cisco or its affiliates. All rights reserved. Page 1 of 30

Table of Contents CISCO IT TETRATION DEPLOYMENT WITH ACI, PART 1... 3 CISCO DATA CENTER SCALE... 4 CISCO TETRATION OVERVIEW... 5 AGENTS AND ASICS GATHER TELEMETRY DATA AND ENFORCE POLICY... 7 ANALYTICS... 10 AUTOMATION OF INTENT BASED POLICY CREATION... 11 CISCO IT TETRATION DEPLOYMENT... 13 AUTOMATED INVENTORY CATALOGUING WITH CUSTOM TAGGING... 14 TETRATION WITH ACI ADM CASE STUDY... 16 UNTANGLING APPLICATION DEPENDENCY... 18 CISCO IT HADOOP ON ACI... 19 ACI HADOOP APPLICATION PROFILE/EPG/CONTRACT POLICIES... 22 DASHBOARD, MONITORING, AND DATA PLATFORM... 23 CISCO IT TETRATION POLICY ENFORCEMENT DESIGN... 25 GENERATING A TAG/ATTRIBUTE BASED SECURITY POLICY... 26 POLICY PRECEDENCE... 27 LAYERED APPROACH TO DATA CENTER NETWORKING SECURITY... 28 BEST PRACTICES AND LESSONS LEARNED... 29 2017 Cisco or its affiliates. All rights reserved. Page 2 of 30

Cisco IT Tetration Deployment with ACI, Part 1 Cisco IT data center environment deploys thousands of applications that support the enterprise, its partners, and customers. Cisco ACI technology easily provides great value in automating operations of classical networking processes. Cisco ACI enables Cisco IT to use a common application-aware policy-based operating model across their entire physical and virtual environments. A critical requirement is to have insight into what applications are running, how they are composed, how they are inter-dependent upon infrastructure services and how to keep this information up to date as new versions of applications are deployed. Cisco Tetration TM solves these application dependency problems using machine learning and goes further in enforcing granular policies for segmentation, there by meeting security requirements. As Benny Van De Voorde, Cisco IT Principal Engineer, says, There is simply no other way to perform application dependency mapping and policy enforcement in large scale data centers as effectively. According to an IDC white paper, Cisco achieved a 70% reduction in staff time required to gain insight into application behavior. Staff Time Needed for Application Dependency Mapping, Tetration Versus Manual Approach The result is that Cisco IT can not only be more agile in delivering scalable high performance premise data center services but also more quickly and fully achieve the business intent of the organization. 2017 Cisco or its affiliates. All rights reserved. Page 3 of 30

With Cisco Tetration and Cisco ACI, Cisco IT can provide much higher value to the enterprise by cost effectively performing the functions at scale that were previously not feasible. This is the first of two white papers that show exactly how this is possible. The first paper covers how Cisco IT has used and plans to use the following Tetration capabilities: Automatically perform these critical tasks: o o o Dynamic real-time inventory generation / update with custom tags Application dependency mapping Application segmentation / zero-trust policy generation Enhanced security and access agility design based on deploying scopes, RBAC, ACI security policies along with other security mechanisms such as WAF, IDF, and encryption. Note: Cisco IT deployed Tetration Analytics v2.0 in August 2017, after using v1.0 since mid-2016. Today, Tetration Analytics v2.0 provides advanced security that analysts are identifying as the future direction of the industry. The second white paper will cover how Cisco IT is using the following Tetration capabilities: Enhanced security and access agility case study. Simulate policy for impact analysis Policy compliance audit Forensic analysis with replay of historical full flows Cisco Data Center Scale The Cisco IT organization operates multiple business application and engineering development data centers distributed around the world. 2017 Cisco or its affiliates. All rights reserved. Page 4 of 30

Cisco IT Worldwide Data Centers Cisco IT supports 141,000 employees (71,000 regular employees and 70,000 contractors) in 583 offices across more than 100 countries. The data centers occupy more than 269,000 sq. ft. of floor space and draw 30.1 MW of UPS power. More than 11,000 Cisco Unified Computing System (Cisco UCS ) blades are deployed with 92% of the servers in new data centers virtualized. The infrastructure for the core business data centers (DC) is big. For example, the Allen, Texas DC alone includes 856 network devices that support 2300 traditional and private-cloud applications, run 8000 virtual machines, including 1700 Cisco Unified Computing System (Cisco UCS ) blades and 710 bare metal servers, with 14.5PB of NAS storage and 12PB of SAN storage. Cisco is driven to migrate to ACI because, as its data centers grow, quick and agile application deployment becomes increasingly challenging. ACI enables Cisco IT to use a common application-aware policy-based operating model across their entire physical and virtual environments. Growing hybrid cloud deployments and growth in east-west traffic, including encapsulations such as Virtual Extensible LAN (VXLAN), pose increasing barriers to network visibility, both for ongoing operational efficiency and for network forensics. Application dependencies are a particularly vexing issue. It can be a time intensive struggle to identify application dependencies. Essentially, traditional data centers run in the dark, with little or no insight into how the various parts are moving and interacting. Cisco Tetration Overview Cisco Tetration is the only platform that uses machine-learning and other algorithmic 2017 Cisco or its affiliates. All rights reserved. Page 5 of 30

approaches to automate identifying application flows across data center, cloud, and hybrid deployments at a level of detail that can enable the application dependency mapping, security enforcement, and added business value that an enterprise like Cisco requires. Cisco Tetration Analytics The Tetration Analytics TM policy recommendation and enforcement engine can deliver fine-grained application segmentation, far better than today s micro-segmentation solutions and at greater scale. As Tom Edsall, Cisco SVP GM, says, What you get out of Tetration is a single application policy that incorporates multiple requirements, provides enforcement across heterogeneous infrastructure, and is monitored in real time. Cisco IT uses Tetration Analytics TM to identify exactly how applications consume data center resources and automatically generate secure application policies. Tetration derives deep telemetry from lightweight software agents that run on servers and built-in hardware agents in the Nexus 9K platform. 2017 Cisco or its affiliates. All rights reserved. Page 6 of 30

Cisco Tetration Analytics Key Characteristics It delivers real-time analytics to achieve actionable insights by searching billions of records in seconds. Tetration is capable of processing millions of flows per second with the capacity to retain and replay billions of flow records without aggregation. Agents and ASICs Gather Telemetry Data and Enforce Policy Cisco Tetration uses agents that can be deployed across heterogeneous environments, from public or private clouds, to virtual machines and bare metal servers, and from the network all the way to the endpoint. The following table lists the agents available in the Tetration v2.0 release. 2017 Cisco or its affiliates. All rights reserved. Page 7 of 30

Agents and ASICs Platforms Capability Deep Visibility Agents Microsoft Windows, Ubuntu, Linux, and CentOS servers Application dependency mapping and real-time full flow capture, including out of band agent capture via ERSPAN Enforcement Agents (bundled with Deep Visibility) Microsoft Windows, Ubuntu, Linux, and CentOS servers Network policy enforcement Universal Visibility Agents Older versions of Windows OS, Linux servers, Solaris, and AIX Application dependency mapping based on flow sampling, but no enforcement Nexus 9000 EX/FX Network ASICs Cisco Nexus 9000 EX / FX Series Switches ASIC support for application dependency mapping, real time flow capture, and policy enforcement When gathering flow telemetry, there are a number of technical considerations about the scale, efficiency and accuracy of the collection mechanism. Except for the universal visibility agent, Tetration agents use a full flow approach that observes every packet. As a result, Tetration can see and report all flows. The advantage of full flow based telemetry is that it provides full visibility into the traffic without depending on statistical approximations. Full flow is not full packet capture. Although visibility into all traffic on a network may seem like a good idea, it is not necessarily useful or allowed by regulators. Tetration full flow header metadata capture does not suffer from these limitations. The information required to analyze what is happening and to perform threat analysis is available to Tetration, overhead on the network is limited (1-3 percent), and overhead on a device CPU is close to zero. 2017 Cisco or its affiliates. All rights reserved. Page 8 of 30

Cisco IT Tetration Deployment Overhead Example The Cisco IT experience with Tetration shows that the overhead on the network is limited (1 percent in this example), and overhead on a device CPU is close to zero (0.35% in this example). Switch ASICs and Tetration agents use a full flow approach that observes every packet. Cisco Nexus series switches incorporate a new family of ASIC s that introduce a mechanism for packet and flow monitoring that avoids any CPU bottleneck or overhead. The dedicated FlowTable module built into the Cisco next-generation data center ASICs provides a full view of all packets and all flows. This module collects information on a per-packet basis, without any sampling and without introducing any negative latency or performance degradation. To accomplish this, the module pulls information from the pipeline without being in the traffic path. This complete view enables a broad range of telemetry-based network security measures and mitigates the risk of missed information in statistical analysis. Tetration takes metadata directly from various ASIC functions while the packet is processed. This approach helps ensure that no payload can leak to a collector. In addition to the traditional forwarding information, the FlowTable module collects other elements such as detailed IP and TCP flags and tunnel endpoint (TEP) IDs. The FlowTable module also introduces new capabilities such as the ability to detect anomalies in the packet flow such as inconsistent TCP flags. FlowTable tracks flow performance information such as the burst characteristics and latency of a flow. By providing this level of information, 2017 Cisco or its affiliates. All rights reserved. Page 9 of 30

FlowTable enables a better more complete view of a flow and its health. Unlike other options such as NetFlow, FlowTable is complete and bi-directional it identifies both the source and destination of a flow. Because no sampling is involved in this process, Tetration has complete visibility into the flow. Other options, such as NetFlow, provide summarized, aggregated data. Also, such uni-directional methods don t enable determining who or what initiated the flow. FlowTable allows us to see bi-directionally. To complement anomaly detection, FlowTable has an events mechanism. This configurable mechanism defines a set of parameters that represent an interesting packet. When a packet has these parameters, an event is triggered with the metadata that triggered the event (not just the accumulated flow information). This special capability gives FlowTable visibility into interesting events. In addition to FlowTable, all the usual Cisco NX-OS Software mirroring features are available, enabling a deep view of specific flows as needed. Whereas Tetration collects TCP/IP, TEP, and other flow information (such as burst characteristics, and latency) from Cisco switch ASICs, Tetration host agents rely on host IP table data and add host operating system process information, and metadata. There is no privacy risk with either the agents or ASICs because packet payloads are never exported. Within a flow, differential analysis can identify discrepancies between the data that a host agent provides and the data that an ASIC provides. Host agents can also provide visibility into VDI environments using software sensors on desktop virtual machines. ERSPAN based sensors can generate Tetration Telemetry, which allows customers to send a copy of the traffic using ERSPAN to out-of-band virtual machines with Tetration sensors that generate the telemetry. Beyond application dependency mapping, and automated policy generation and enforcement, these capabilities of the ASICs and agents all aid in satisfying compliance requirements, forensic analysis, and security incident detection/response with alerts triggered for defined types of events. Analytics In conjunction with its next-generation ASICs and software agents, Cisco developed the next-generation Tetration Analytics collection engine. ASIC and agent capabilities are critical, but the capabilities for processing, visualizing, and acting on the information are 2017 Cisco or its affiliates. All rights reserved. Page 10 of 30

just as important as the quality of the source information. Today, merchant silicon provides capabilities to original equipment manufacturers, but leaves the collector design to others. Cisco is the only vendor that provides an end-to-end solution from the network to the Cisco Tetration Analytics platform collector. Tetration Analytics provides deep visibility into the network, for all packets, and enables you to track the life of a flow, including historical replay, and the ability to run what-if scenarios. In addition, Tetration automatically generates and can enforce policies, and it can provide alerts when specific conditions are met that could represent security or performance risks. Automation of Intent Based Policy Creation Tetration Application Dependency Mapping (ADM) automatically detects application tiers and groups similar endpoints into clusters. Tetration learns the flows between endpoints, the processes running on them, and dynamically keeps this information up to date. Tetration Machine Learning ADM Automatically Groups Application Endpoints into Clusters Tetration maps the datacenter into clusters containing similar endpoints, and generates a policy that can be reviewed, analyzed, and enforced. 2017 Cisco or its affiliates. All rights reserved. Page 11 of 30

Flows should only be allowed as needed. Cisco Tetration Intent Based Automatic Policy Generation Tetration supports both whitelist and blacklist policies. ACI uses the whitelist model, which is the most secure, to add enhanced levels of security to applications that have stringent security requirements. Cisco IT deploys such applications using the whitelist model as soon as they host them in their ACI data centers. In practice, Cisco IT has found that there are in-between scenarios. Some applications are deployed with less strict whitelist security, with the expectation that over time Cisco IT will gradually enforce progressively stricter whitelist security policies. Cisco Tetration Whitelist Policy Generation Tetration can automatically generate an application whitelist policy that can be enforced directly from Tetration or through other segmentation techniques. In the case of Cisco IT, 2017 Cisco or its affiliates. All rights reserved. Page 12 of 30

after some post-processing, Tetration security policies are uploaded to the ACI fabric infrastructure. Cisco IT Tetration Deployment Today, the Tetration platform can be deployed on-premise in two form factors, and in the public cloud, such as Amazon Web Services today. Tetration is multi-tenant aware, which allows multiple customers to be hosted in a secure manner on a single TA cluster. Cisco Tetration Deployment Options Both the on premise and public cloud Tetration deployments can be used across both internal and external networks. Tetration includes cloud migration analysis, that allows customers to run hypothetical scenarios for cost associated with traffic volume if they move a specific application component to the public cloud. For Cisco IT, Tetration provides the benefits of big data but in a simple plug and play clustered appliance that is self-monitoring - you don t need big data expertise to operate or care for Tetration. The Tetration clustered servers and software are pre-packaged, optimized, and are easy to set up and simple to operate. Tetration clusters centrally manage secure automated deployment, upgrade, and configuration of its agents using a mutual certificate process: The Tetration cluster inserts a certificate in the installer. Code signed agents can only talk to their specific Tetration cluster. The current and target Cisco IT deployment uses the v2.0 Tetration on premise options, 2017 Cisco or its affiliates. All rights reserved. Page 13 of 30

as illustrated in the following figure. Cisco IT Tetration Deployments Cisco IT is installing additional Tetration agents on hosts that are in their roadmap for migration to ACI. The Cisco IT target is to have 3 Tetration Clusters deployed using over 30,000 agents. This is all managed by a small team that manage other systems at the same time. Automated Inventory Cataloguing with Custom Tags Cisco Tetration uses machine learning to offer inventory cataloguing with custom tags, network analysis, application dependency mapping, and security enforcement features that are possible only when paired with its full flow comprehensive data set. The custom tag annotation capability enables Cisco IT to visualize and define policies using consistent attributes across its environment Cisco Tetration Automated Inventory Cataloging Agent feed with custom tags discovers inventory based on all nodes observed on the network directly via agents/asics (including vcenter and AWS virtual machine attributes), or indirectly via a flow to or from an agent/asic, merges with uploaded inventory - for example, from a configuration management database - and custom 2017 Cisco or its affiliates. All rights reserved. Page 14 of 30

metadata tags (32 arbitrary tags). Inventory tracked in real time (updated every minute), along with historical trends. Inventory includes both internal and external hosts. An internal host is a host running a software agent or included in the Tetration collection rules. An external host is any other host with traffic observed on the network. Inventory access can be restricted by scope and RBAC rules. User uploaded tags with annotations for inventories enable observing the network in the known familiar terms of an enterprise. For example, Cisco IT used python scripts to upload a CSV file taken from a configuration management database containing IP address/device name items to Tetration. Inventory Cataloging with Custom Tagging Annotations In this example, Cisco IT used python scripts to upload to Tetration CSV tables containing categories of items that included subnets, descriptions, DNS servers, zones, ACI fabrics, tenants, application profiles, EPGs, and place in network. The result is that query tables display the results using labels Cisco IT uploaded to Tetration. As shown in the illustration below, this makes for a much easier to read and understand set of information. 2017 Cisco or its affiliates. All rights reserved. Page 15 of 30

Inventory Cataloging with Custom Tag Annotations Portion 1 of this illustration shows the columns Tetration provides. Portions 2 and 3 of this illustration show asterisks next to the column names which indicates that they are categories of information Cisco IT customized within Tetration. As you can see, the query result table uses the naming conventions of the Cisco IT data center. Moreover, custom inventory tag annotations provide additional identifiers for discovered endpoints. Inventory query filters can match many identifiers provided to Tetration. For example, an endpoint can have an identifier that specifies it is a production or non-production workload, or PCI or HIPPA, or its place in the network. An inventory query filter that finds all production workloads enables easily creating a policy that strictly enforces prevention of production workloads from communicating with non-production workloads. Tetration with ACI ADM Case Study In late 2014, before Tetration was available, Cisco IT began deploying ACI according to a design plan that phased in the full implementation of the ACI whitelist security model. 2017 Cisco or its affiliates. All rights reserved. Page 16 of 30

Cisco IT first moved applications to a basic ACI fabric deployment with allow-all contracts because manual analysis of application flows was difficult and because of the risk of missing flows. The entire existing security infrastructure outside the ACI fabric still applied to these phase 1 basic ACI fabric application flows. In phase 1 of the migration roadmap, applications that moved to ACI still benefited from the zero-trust environment due to the isolation ACI tenants, application profiles, and EPGs provide. Even in the allow-all mode of the phase 1 basic ACI fabric, communication could not jump from tenant to tenant, from application profile to application profile or from endpoint group to endpoint group without explicit permission Cisco IT granted. Starting in 2016, Cisco IT began using Tetration to migrate applications to its ACI zero-trust security environment using policies based on Tetration Analytics. These policies allow only what the applications need. The application migration process starts with an architecture review and proceeds to the specifications for a particular application. Cisco IT ACI Application Migration Process Flow A central activity in the migration process is defining the application EPGs and the contract requirements between EPGs. 2017 Cisco or its affiliates. All rights reserved. Page 17 of 30

Infrastructure Applications Untangling Application Dependency Working with the Cisco application developer owners, the security team, and the networking teams, the Cisco IT team assembles application tribal knowledge into a best effort definition of application dependencies. This information enables placing application workloads in the ACI fabric. While this information is significant, Cisco IT requires a more thorough process to assure that there are no gaps caused by insufficient visibility into the datacenter environment, especially for applications that have high security and high availability requirements. Untangling Application Dependency OS Servers Network Routing ACE Configuration DNS Info Application Groups Tetration Analytics Application Team Security Team EPGs & Contracts ACI Configuration Identify Tenant for EPGs & Contracts Tetration application dependency mapping enables validating the information that various Cisco enterprise stakeholders provide, identifying gaps in that information, and automatically grouping the application dependent system components into logical units that map into ACI application profile endpoint groups (EPGs), along with ACI security policies (contracts). With this, Cisco IT can then easily place the application profile (including its EPGs and contracts), in the suitable ACI tenant. 2017 Cisco or its affiliates. All rights reserved. Page 18 of 30

Cisco IT Hadoop on ACI Cisco Tetration machine learning grouped pre-aci Hadoop flows that Cisco IT labelled according to the naming conventions in the Cisco data center and the labels are color-coded. The following color-coded Tetration screen illustrates the various Hadoop EPG cluster flows. Cisco IT Hadoop Tetration Application Dependency Map (prior to migration to ACI) The Cisco Tetration screen arrays the Hadoop EPG clusters around the perimeter of the screen. The illustration includes the following color-coded types of flows: Green: Cisco IT foundational services, including LDAP, OAM, OCM, etc. Blue: Database Hadoop flows, including Platfora Yellow: Cisco enterprise internal Orange: DMZ/external flows Purple: Edge application flows 2017 Cisco or its affiliates. All rights reserved. Page 19 of 30

Cisco IT uses two routing contexts (VRFs) within the ACI fabric, one for DMZ/external and one for internal. This assures that there is complete isolation between the DMZ and internal security zones. The following illustration is an example of where the internal and DMZ Hadoop flows occur in the data center topology. Tetration Identified Rogue Hadoop Flows in the Internal Data Center and DMZ Contexts (VRFs) While the information Cisco application developer owners, the security team, and the networking teams provided the migration team was fairly comprehensive, there were several surprises that Tetration uncovered: Some Cisco internal data center flows were not known to any of the teams. Examples include flows to labs. These flows were not seen as problematic or security concerns. 2017 Cisco or its affiliates. All rights reserved. Page 20 of 30

Some DMZ/external flows were going to Amazon AWS that were not known to any of the teams. This was a surprise that was a security concern. Tetration confirmed all the Hadoop TCP/IP ports that the team had specified. This validation enabled specifying whitelist contract filters that would not cause problems by inadvertently blocking required ports. The figure below illustrates the ACI whitelist contract filter specifications Tetration identified for the Cisco migration of its Hadoop deployment to ACI. Tetration Flows Validate ACI Contract Filter Specifications Tetration is able to export ACI contract specifications in various formats, including XMP, JSON, and YAML. Cisco IT chose to incorporate the contract specifications into its standard YAML library that was then posted to ACI. Cisco IT used Tetration to verify the contract specifications and assembled the YAML contract code for various contracts that specify how to allow data flows between Hadoop EPGs. A portion of the clients-to-hadoop-cluster contract is listed below. 2017 Cisco or its affiliates. All rights reserved. Page 21 of 30

Portions of the Tetration Auto-Generated ACI Contract YAML Code Contract name: clients-to-hadoop-cluster scope: 'Private Network' #VRF subjects: -#Hadoop name: 'tcp-5181' isunidirectional: True filtersintoepg: - 'dst-tcp-5181-filter'... -#Web name: 'tcp-7221' isunidirectional: True filtersintoepg: - 'dst-tcp-7221-filter'... -#Drill name: 'tcp-31010' isunidirectional: True filtersintoepg: - 'dst-tcp-31010-filter'... -#RADIUS name: 'udp-1812' isunidirectional: True filtersintoepg: - 'dst-udp-1812-filter' ACI Hadoop Application Profile/EPG/Contract Policies After understanding the application dependencies, it was easy for Cisco IT to map the application to application profiles with their corresponding EPGs. Then, it was very simple to migrate all the Hadoop applications from the traditional network to the ACI fabric. The application owner and the Cisco security teams chose to enforce strict limits on communications between clients and the Hadoop cluster as well as between the Platfora 2017 Cisco or its affiliates. All rights reserved. Page 22 of 30

application and the Hadoop cluster. Communications between other Hadoop EPGs were set to allow-all, with the expectation that these settings would be reviewed in the future and revised accordingly. ACI Hadoop Application Policies Contracts are directional; they are provided, consumed, or both. The cisco-internal-extnet EPG provides the clients-to-hadoop-cluster contract. The hadoop-1-cluster EPG consumes the clients-to-hadoop-cluster contract. The filters in this contract specify which ports are open for inbound client connectivity that connects to the hadoop-1-cluster EPG. The clients-to-hadoop-cluster contract is reused for connectivity between the platfora-1-app and hadoop-1-cluster EPGs. Dashboard, Monitoring, and Data Platform The dashboard presents graphical views of Tetration data, which you can customize according to requirements for tasks such as monitoring, incident resolution, or forensics. The Tetration data platform enables running various logic within Tetration such as simple SQL queries to get filtered data to monitor network flows. The data platform also provides the capability to bring your own data streams into Tetration, using a framework that 2017 Cisco or its affiliates. All rights reserved. Page 23 of 30

integrates external data with Tetration applications to visualize the data in the Tetration GUI or send notifications to northbound systems. These two features can aid in quickly assessing actionable insights from Tetration. Cisco IT uses Tetration to monitor application performance and deviations. The Cisco IT Lightweight Application Environment (LAE) is the platform as a service (PaaS) environment that provides operating system, middleware, and system functions as services. Cisco IT monitors its LAE application for a variety of reasons, including proactively assuring service level agreements are met. LAE is deployed in an active/active mode across the Richardson Texas and Allen Texas data centers. Example of dashboard view of the Cisco IT LAE application traffic The Tetration dashboard shows the relative distribution of the load across both data centers. The normal case is for the workload to be distributed evenly across both data centers. If Cisco IT operations sees that one data center has a very low workload, they would suspect a problem exists that must be addressed before there is a disruption in the operation of the LAE application. 2017 Cisco or its affiliates. All rights reserved. Page 24 of 30

Example of dashboard view of the Cisco IT LAE application DNS requests Another example of a dashboard filter on the LAE application shows detailed DNS request information. Furthermore, Cisco IT used another query with specific filters that identify a WannaCry DNS attack. The results of user created routines that extract actionable data from Tetration automatically can be handed off to other systems such as monitors or for reporting, further investigation, or compliance audits. For example, application latency can be monitored vs. Smoothed Round Trip Time (SRTT) latency for various servers. In Tetration, you can specify if you want to see any network flow taking more SRTT and you can add multiple filters (for example, host names, port, protocol). A simple SQL query could be written to pull the filtered data from Tetration to monitor the network flow. Then, if the SRTT SLA value is over 90 ms, the Tetration open APIs enable using scripts that easily and automatically push an alert to a monitoring system. Cisco IT Tetration Policy Enforcement Design Cisco IT has developed the following design for deploying the Tetration policy enforcement capabilities. This topic will be more fully covered in a case study that will be published in the Cisco IT Tetration Deployment on ACI Part 2 white paper. 2017 Cisco or its affiliates. All rights reserved. Page 25 of 30

Generating a Tag/Attribute Based Security Policy Starting with v2.0, Tetration provides scope and Role Based Access Control (RBAC) access control. Scopes are hierarchically organized groups of assets/endpoints to which role abilities (read, write, execute, enforce, owner) rules and RBAC access control (including Active Directory) can be applied. Cisco IT has designed a tag/attribute based security model it will deploy in Tetration to enhance the security of its ACI data center operations. Cisco IT Tetration tag/attribute security model 1. Cisco IT uploaded custom inventory tag attributes to Tetration. One of the custom inventory tags Cisco IT uploaded to Tetration is ACI application network profile (ANP). 2. Now, they can use Tetration to create a filter that identifies a particular ACI application profile in its data center. 3. Based on that filter, they create a scope that includes those tagged items. 4. Finally, they establish ACI security policies with contracts and appropriate ACI filters. 2017 Cisco or its affiliates. All rights reserved. Page 26 of 30

Policy Precedence Using these scopes as building blocks, Cisco IT can easily enable the following access capabilities: Application owners have a level of autonomy to make application level changes quickly. Security and network teams control the global aspects of application inter-connection and shared services. Tetration flattens intent in a deterministic order, prioritizing higher authority users intent over application owners. Cisco IT Tetration policy hierarchy In this scenario, any consumer of the Cisco IT ACI resources must comply with the policies defined in section 1 where all the IT must-follow infrastructure services policies are covered. In section 2, IT defines a set of default policies for their owned shared services where application owners can choose to use these default policies. In section 3, the application owners can specify their own policies. 2017 Cisco or its affiliates. All rights reserved. Page 27 of 30

Layered Approach to Data Center Networking Security With these building blocks in place, Cisco IT designed a layered approach to data center security that provides greater agility and enhanced security to the Cisco enterprise. When it comes to data center networking security, Tetration gives Cisco IT the visibility of all the flows that need to happen within any portion of the data center. This visibility enables enforcing security in different ways according to whatever the security requirements might be. Cisco IT security requirements determine what they will enforce with any given technology. Cisco IT multi-layered data center networking security Using ACI contracts, Tetration scopes, RBAC, and mandated firewall rules, Cisco IT greatly enhances their security posture which already includes web application firewall (WAF), intrusion detection systems (IDS), and encryption (both standing and in transit). Tetration can provide granular tracking of policy changes which enhances compliance related notifications. With this foundation in place, Cisco IT can then use the scope and RBAC features of Tetration to enable individuals to have secure access to only the scope-defined portion of the data center according to the defined RBAC rules that are suitable for that person. 2017 Cisco or its affiliates. All rights reserved. Page 28 of 30

Cisco Tetration Whitelist Policy Deployment While the example of an auto-generated policy illustrated here is small, an actual Cisco IT Tetration auto-generated whitelist policy could have thousands of lines. Cisco IT takes that policy and deploys the relevant portions of it in multiple areas of its ACI data center infrastructure, such as ACI contracts enforced in the switches, firewall policies, and in the Tetration host agents that enforce the policy as well. For example, if there is an ACI EPG running in the DMZ VRF that needs to communicate with an EPG in the internal VRF, then it must go through a firewall. In addition Cisco IT will specify security requirements that the Tetration agent will enforce at the hosts. Best Practices and Lessons Learned Start off focused on application dependency mapping: Cisco IT found that Tetration machine learning effectively automates application dependency mapping, achieving a 70% reduction in staff time required to gain insight into application behavior. The inventory annotation feature makes it easy to recognize what Tetration finds. Modifying sample scripts from the ACI Toolkit, Cisco IT was able to upload custom tagging annotations based on known data sets such as those in their configuration management database. Collaborate with application owners, security teams and other stakeholders to integrate Tetration into the relevant processes and procedures the organization uses. Deploy with automation in mind: create standard and reusable queries, and build scripts that take advantage of the Tetration open APIs to automate tasks such uploading custom tagging annotations. 2017 Cisco or its affiliates. All rights reserved. Page 29 of 30

Security: Tetration is able to export ACI contract specifications in various formats, including XMP, JSON, and YAML. The Tetration generated contracts specify how data flows are allowed between EPGs. Cisco IT incorporates the contract specifications into its standard YAML library that they post to ACI. Applications with a high security requirement and deployed in ACI using strict whitelist policies. Other applications continue to run with the traditional data center security while Cisco IT uses Tetration to gain full insight into the application, and in the process gradually enforce stricter whitelist policies. The scope and RBAC features enable the creation of a multi-layered security model that provides enhanced white list security, along with more agile distributed role based access control. 2017 Cisco or its affiliates. All rights reserved. Page 30 of 30