Service Insertion with Cisco Application Centric Infrastructure

Transcription

1 Guide Service Insertion with Cisco Application Centric Infrastructure August Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 1 of 21

2 Contents Introduction... 3 Benefits... 3 Topology and Design Principles... 4 Connecting Endpoint Groups with a Service Graph... 4 Extension to Virtualized Servers... 5 Management Model... 5 Service Graphs, Functions, and Rendering... 6 Hardware and Software Support... 6 Cisco ACI Modeling of Service Insertion... 8 Service Graph Definition... 8 Concrete Devices and Logical Devices Logical Device Selector (or Context) Splitting Bridge Domains Configuration Steps Configurations in XML Format for Use with REST Calls Configuration of the Logical Device and Concrete Device Configuration of the Logical Device Context (Cluster Device Selector) Package Definition of the Metadevice and Connectors Service Graph Configuration Association of the Service Graph with a Contract Conclusion For More Information Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 2 of 21

3 Introduction Cisco Application Centric Infrastructure (ACI) technology provides the capability to insert Layer 4 through Layer 7 functions using an approach called a service graph. The industry normally refers to the capability to add Layer 4 through Layer 7 devices in the path between endpoints as service insertion. The Cisco ACI service graph technology can be considered a superset of service insertion. This document describes the service graph concept and how to design for service insertion with the service graph. As Figure 1 shows, Layer 4 through Layer 7 services can be physically located anywhere in the fabric, and they can be running as physical appliances or as virtual appliances. Figure 1. Cisco ACI Fabric with Layer 4 Through Layer 7 Services Benefits The main purpose of a data center fabric is to move traffic from physical and virtualized servers and forward it to its destination, and while doing so apply meaningful Layer 4 through Layer 7 services such as: Firewalls Load balancing Traffic inspection SSL offloading Application acceleration The main benefits of using a Cisco ACI fabric to provision Layer 4 through Layer 7 services are: Single point of provisioning through the GUI, the Representational State Transfer (REST) API, or Python scripts Powerful scripting and programming environment with a Python software development kit (SDK) 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 3 of 21

4 Capability to provision very complex topologies instantaneously Capability to add and remove workloads from the load balancers or firewall configurations without human intervention Capability to create a logical flow of functions instead of just a sequence of Layer 4 through Layer 7 devices Multitenancy (network slicing) on the fabric and on the service devices Capability to create portable configuration templates Intuitive and easy configuration process One of Cisco ACI s several innovations in the area of service insertion is that Cisco ACI allows you to concatenate functions offered by individual Layer 4 through Layer 7 devices instead of simply connecting discrete boxes in sequence. Topology and Design Principles Appliances don t need to be placed in any particular place in the fabric. They can run as physical appliances connected to any leaf, or as virtual appliances running on any virtualized server. Physical appliances can run with multiple virtual contexts as well. Cisco ACI can model this concept in the construction of the policy. Note: At the time of this writing, virtualized appliances can be deployed with VLAN as a transport between VMware ESX servers and leaf nodes and can be deployed only with VMware ESX as the hypervisor. Connecting Endpoint Groups with a Service Graph A service graph is a variation of the concept of a contract. In the Cisco ACI policy model, a contract connects two endpoint groups (EPGs). A contract can also offer functions such as traffic filtering, traffic load balancing, and SSL offloading. Cisco ACI locates the devices that provide such functions and inserts them into the path as defined by the service graph policy. As Figure 2 shows, a sequence of Layer 4 through Layer 7 functions can be used to connect two EPGs. Figure 2. Cisco ACI with Service Graphs 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 4 of 21

5 Extension to Virtualized Servers Virtual appliances are automatically inserted into the Cisco ACI fabric by the Cisco Application Policy Infrastructure Controller (APIC). Cisco ACI can locate the virtual network interface card (vnic) of the virtual firewalls and virtual load balancers and automatically connect them to the correct EPG. Management Model The user can define configurations on Cisco APIC in several ways (Figure 3). These configurations can include the definition of the service graph. Cisco APIC communicates with the load balancers and firewalls to allocate the necessary network path to create the desired service graph path. The user can define the service graph configuration using the following options: Easy-to-use GUI running on the same appliance that provides the controller function Representational State Transfer (REST) calls with intuitive XML or JavaScript Object Notation (JSON) formatted payloads that are sent to the Cisco APIC: these can be sent in many ways, using tools such as Google POSTMAN or Python scripts that send REST calls Custom-built GUI that sends REST calls Command-line interface (CLI) to navigate the object model from Cisco APIC Python scripts that use the associated Cisco ACI libraries Figure 3. Cisco APIC Provides the Capability to Configure Services with REST, Scripts, or a GUI 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 5 of 21

6 Service Graphs, Functions, and Rendering The concept of service graph is different from simply doing service insertion. A service graph is a concatenation of functions (and not of network devices). The service graph specifies that the path from one EPG to another EPG must pass through certain functions. Cisco APIC translates the definition of the service graph into a path through firewalls and load balancers, called rendering. As Figure 4 shows, Cisco APIC is aware of the pool of load balancers and firewalls (concrete devices) and can translate the user intentions expressed in the service graph by using the available pool of resources. Figure 4. Concept of a Service Graph Therefore, the service graph is more like a template, which can be ported to different data centers and rendered with locally available resources. The rendering involves allocation of the necessary bridge domains, configuration of IP addresses on the firewall and load balancer interfaces, creation of the VLAN on these devices to create the path for the functions, and performance of all the work necessary to make sure that the path between EPGs is the path defined in the service graph. Hardware and Software Support Cisco APIC communicates with the firewalls or load balancers to render the graph defined by the user. For Cisco ACI to be able to talk to firewalls or load balancers, it needs to speak to their APIs. The administrator needs to install plug-ins on Cisco APIC that enable this communication. A plug-in is referred to as a device package, and the vendor of a firewall and load balancer must provide it so that Cisco APIC can communicate with it. As shown in Figure 5, the device package includes a description of the device and lists the parameters it is exposing for Cisco APIC configuration and the scripts that allow Cisco ACI to talk to this device Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 6 of 21

7 Figure 5. Device Package Before performing any configuration based on service graphs, you need to install the plug-in on the Cisco APIC to enable communication between Cisco APIC and the device Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 7 of 21

8 Figure 6 illustrates how you can import the device package in Cisco APIC. Figure 6. Using the GUI to Import the Device Package Cisco ACI Modeling of Service Insertion This section describes how to define workload connectivity with services in Cisco ACI. Service Graph Definition The service graph is a sequence of functions. The administrator can define these functions either using XML format or using the GUI. The GUI allows you to choose the functions exported with the device package and to concatenate them. Figure 7 shows the list of functions that that Citrix NetScaler exports with the device package. The administrator can pick functions individually and stitch them together through the GUI, as shown in Figure 7. Note that the function or device that is inserted is a metadevice: that is, it is not a specific load balancer or firewall, but instead is simply a load balancer or firewall of a certain type. The association of the metadevice (for example, a function from a load balancer of type Citrix or F5 or from a firewall of type Cisco Adaptive Security Appliances [ASA]) with an actual device connected to the fabric is performed in the rendering stage. Figure 7. Using Citrix Functions to Create a Service Graph 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 8 of 21

9 The service graph also defines the virtual services and server pools that you want Cisco ACI to program on the load balancer or firewall when the graph is instantiated. Figure 8 shows the parameters that the administrator can add to the load balancer used in this example. Figure 8. Parameters to Be Configured on the Service Devices Upon Rendering Not all parameters need to be hard-coded IP addresses. You can also define parameters that are populated by the appearance of new endpoint in a particular EPG in the fabric. The graph is rendered when it is associated with a contract, as shown in Figure 9. Figure 9. Associating a Graph with a Contract When the graph is rendered, you will see configurations appear in the device that is part of the graph. For instance, in the case of an F5 BIG-IP load balancer, you may see Self IP appear in the interface and a server pool being programmed Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 9 of 21

10 To know whether the graph was rendered, you can check the GUI (Figure 10). The Faults tab shows you which configurations weren t applied and why.figure 10 shows the Graph Instances. If the service graph has been rendered it will be listed in the Graph Instances section of the GUI. Figure 10. Verifying That a Service Graph Has Been Rendered Concrete Devices and Logical Devices Service graphs are composed of abstract nodes, which are metadevices. Cisco APIC can translate the intention expressed by the user in the abstract graph into a sequence of concrete devices that are actually connected in the fabric. Firewalls and load balancers are never deployed as single devices. Instead, they normally are deployed as clusters of active-standby pairs. Cisco ACI provides an abstraction to represent these clusters. Cisco ACI calls this abstraction a device cluster or a logical device. The administrator must help Cisco ACI perform the mapping between the service graph and the clusters of firewalls and load balancers. The administrator also needs to tell Cisco ACI which pairs of concrete devices constitute a cluster. The GUI simplifies this process, guiding you through the steps to define each cluster of firewalls or load balancers. Figure 11 provides an example of the configuration of concrete and logical devices. The various fields in this figure refer to the management information about the cluster of devices. The virtual address is the management address to be used when the pair of firewalls or load balancers is operating in active-standby mode Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 10 of 21

11 Figure 11. Defining Logical Devices Through the GUI The GUI also asks you to configure logical interfaces. A logical interface defines a naming convention for the building block of the cluster and its mapping to the concrete device and to the metadevice. For example, the metadevice of an F5 load balancer defines an external and an internal interface. The cluster model in Cisco ACI defines two interfaces and lets you choose the name (logical interface [Lif]). Each interface maps to a metadevice interface and also to a physical (concrete) device interface. This process allows Cisco ACI to correctly render the graph. Figure 12 shows an example of a mapping of a logical interface to a concrete device interface. Figure 12. Mapping of Logical Interface to Concrete Device Interface 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 11 of 21

12 The interface naming process may seem complicated at first. Cisco ACI allows you to model concrete devices into clusters of devices and then to select these clusters of devices to render the service graph policy. In addition, the interfaces have different names on the service device itself. For instance, in the case of F5 the interfaces are numbered as 1.1, 1.2, etc. In the case of Cisco ASA, they are numbered Gig0/0, Gig0/2, etc. Cisco ACI allows you to reference these interfaces using the character _ as a replacement for the / and. characters. For example, F5 interfaces are referred to as 1_1 and 1_2, and Cisco ASA interfaces are referred to as Gig0_0 and Gig0_1. For help with the mapping, you can refer to Table 1. Table 1 also provides the correct naming convention from the XML object model. Table 1. Naming Conventions for Service Graph Building Blocks Metadevice Concrete Device On the Device Itself Cluster of Devices: LDev Abstract Node miflbl for the device and mconn for an mfunc vnscif1_1 (not an arbitrary name) to be mapped to vnic MAC address or to physical port on the switch For example, 1.1, which you can reference from Cisco APIC as 1_1 or gig0/0, which you can reference as Gig0_0 vnslif includes reference to mdev miflbl and to vnscif vnsabsfuncconn name = name vnsrsmconnatt indicates the type of connector by pointing to the metadevice Logical Device Selector (or Context) To help Cisco ACI render the service graph, you need to indicate which cluster of devices (logical devices) can be used for which purposes. This configuration is called the logical device context or cluster device selector. Figure 13 shows the fields that you can program in Cisco APIC for the mapping. The selector lets you indicate the name of the graph, the name the contract, the name of the node in the graph, and the device cluster that should be used to render this graph. Figure 13. Configuring the Cluster Selection Policy The device cluster selector also lets you indicate which interface should be associated with which bridge domain and the mapping of the connector in the graph with the logical interface (Figure 14) Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 12 of 21

13 Figure 14. Interface Mapping in the Device Cluster Selection Policy Splitting Bridge Domains For traffic to flow through service devices correctly, you need to make sure that bridge domains and Virtual Routing and Forwarding (VRF) instances are correctly provisioned. Cisco ACI categorizes service devices into two types: GoThrough devices: Devices operating in bridge mode; also called transparent devices GoTo devices: Devices operating in routed mode If the service node is a GoThrough device (Layer 2 device), these configurations are required: Split the bridge domain (and create an EPG shadow). Disable IP-based forwarding on the bridge domain. Enable MAC address proxy. Enable MAC address-based forwarding. Enable flood-and-learn semantics. If there is a routed hop (routed fabric, GoTo service, GoTo IP, and Layer 3 external connectivity domain router) between the two ends of the service chain, then the following configurations are required: Create a VRF split and a bridge domain split. Create shadow EPGs. IP-based forwarding is OK, unless the next bridge domain leads to a GoThrough service. Cisco ACI adds the static routes on the service device and on the VRF instances in the Cisco ACI fabric Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 13 of 21

14 Configuration Steps To configure a service graph, you should start with the following steps: Configure the management IP address for the pair of load balancers or firewalls. Connect the management interface to a management EPG that you configure, or use out-of-band connectivity. Configure the service devices in active-standby or active-active mode. Connect the devices to a leaf node Alternatively, install the virtual appliance on a virtualized server and make sure that the management vnic is connected to a network that can be reached by Cisco APIC. Make sure the device package is installed on Cisco APIC. Furthermore, you should split the bridge domain as needed and associate subnets with it to help ensure that the forwarding path is ready for the deployment. If a service device is dedicated to a tenant, you should perform the following configuration steps within the tenant context. If a service device is shared across multiple tenants, you should configure the following in the management (mgmt.) tenant and then export it to the desired tenant: Create a logical device. Create the associated concrete devices. If the concrete devices are virtual appliances, provide the name of the virtual machine, the vnic name (with the exact spelling and capitalization of the network adapter, with spaces: for instance, Network adapter 2), and the name of the VMware vcenter configuration (as defined in Cisco APIC). See Figure 15. Figure 15. Configuring a Virtual Appliance as a Concrete Device 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 14 of 21

15 Within a tenant context, you should then configure the following: Create a service graph. Create selection criteria (logical device context) to render the service graph. Associate the service graph with a contract. Configurations in XML Format for Use with REST Calls You can achieve the same configurations described in the previous section by using REST calls following the XML model. The configurations steps are as follows: Configure the logical device vnsldevvip. Configure the concrete device vnscdev under vnsldevvip. Create the logical interface vnslif vnsldevvip. Create the graph vnsabsgraph. Configure the logical device context vnsldevctx to define the way that the graph will be rendered. Attach the graph to a contract. Configuration of the Logical Device and Concrete Device The following example assumes that you have a cluster of two firewalls: Firewall 1 is active and Firewall 2 is standby. The configuration looks like this: <vnsldevvip name= my_cluster_of_fw > <vnscdev name= Firewall1 /> <vnscdev name= Firewall2 /> The definition of the concrete device looks like this: <vnscdev name= Firewall1 /> <vnscif name="gig0_0"> <vnsrscifpathatt port <vnscmgmt IP, port> The vnsrscifpathatt configuration points to a specific port on the fabric or to the name of the vnic. The vnslif configuration under LDevVip defines the association of the cluster interface with the interface of each concrete device. It also specifies the type of interface according to the package. For instance: <vnsrsmetaif tdn="uni/infra/mdev-f5-bigip-1.0/miflbl-internal"/> Furthermore, it defines the association with the interface on the device itself: <vnsrscifatt tdn="uni/tn-customer1/ldevvip-cluster-of-f5/cdev-f5- loadbalancer1/cif-1_2"/> This example shows the complete configuration for the concrete devices: <vnsldevvip name="cluster-of-f5"> <vnscdev name="f5-loadbalancer1" vcentername="vcsa" vmname="big-ip VE "> <vnscif name="1_1" vnicname="network adapter 2"> </vnscif> <vnscif name="1_2" vnicname="network adapter 3"> </vnscif> 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 15 of 21

16 <vnscmgmt name="f5mgmt host=" port="443 /> <vnsccred name="username" value="admin"/> <vnsccredsecret name="password" value="admin"/> </vnscdev> </vnsldevvip> This example shows the complete configuration for the logical device: <vnsldevvip name="cluster-of-f5"> <vnslif name="clusterintf1"> <vnsrsmetaif tdn="uni/infra/mdev-f5-bigip-1.0/miflbl-external"/> <vnsrscifatt tdn="uni/tn-customer1/ldevvip-cluster-of-f5/cdev-f5- loadbalancer1/cif-1_1"/> </vnslif> <vnslif name="clusterintf2"> <vnsrsmetaif tdn="uni/infra/mdev-f5-bigip-1.0/miflbl-internal"/> <vnsrscifatt tdn="uni/tn-customer1/ldevvip-cluster-of-f5/cdev-f5- loadbalancer1/cif-1_2"/> </vnslif> </vnsldevvip> Configuration of the Logical Device Context (Cluster Device Selector) The selection of each individual logical device is based on the definition of a context, which is a set of metatags to create a menu to be used later. The metatags are contract name, graph name, and node label. <vnsldevctx ctrctnameorlbl=<name of the contract> > graphnameorlbl=<name of the graph> nodenameorlbl=<name of the node in the graph, e.g. N1> This configuration also includes the subnets that the logical device needs to find to plug itself into the graph. The following configuration specifies that the cluster of load balancers called cluster-of-f5 can render the node F1 in the service graph slbgraph1. This configuration also specifies that the interface that has a label of inside-slb1 in the graph can be rendered by the cluster-of-f5 interface called lif-clusterintf2. <vnsldevctx ctrctnameorlbl="any" graphnameorlbl="slbgraph1" nodenameorlbl="fw1"> <vnsrsldevctxtoldev tdn="uni/tn-customer1/ldevvip-cluster-of-f5"/> <vnslifctx connnameorlbl= outside-slb1 > <vnsrslifctxtolif tdn="uni/tn-customer1/ldevvip-cluster-of-f5/lifclusterintf1"/> </vnslifctx> <vnslifctx connnameorlbl="inside-slb1"> <vnsrslifctxtolif tdn="uni/tn-customer1/ldevvip-cluster-of-f5/lifclusterintf2"/> </vnslifctx> </vnsldevctx> 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 16 of 21

17 Package Definition of the Metadevice and Connectors Figure 16 shows the naming of connectors on the metadevice. Figure 16. Metadevice The naming of the interfaces as defined by the device package is based on the concept of the metadevice and metainterface label. These labels are the labels you see when you insert a metadevice in the service graph: <vnsmiflbl name="external"/> <vnsmiflbl name="internal"/> <vnsmiflbl name="management"/> The metaconnections are defined as follows: <vnsmconn name="external" dir="input" enctype="vlan" notifications="endpoint"> <vnsrsinterface tdn="uni/infra/mdev-f5-bigip-1.0/miflbl-external"/> </vnsmconn> <vnsmconn name="internal" dir="output" enctype="vlan" notifications="endpoint"> <vnsrsinterface tdn="uni/infra/mdev-f5-bigip-1.0/miflbl-internal"/> </vnsmconn> Service Graph Configuration The following is a sample configuration of a service graph in XML format. The service graph is contained within an abstract container. The service graph container starts with AbsTermNodeProv-<name>/AbsTConn. The service graph container ends with AbsTermNodeCon-<name>/AbsTConn. The connectors are typically referred to as C1 and C2. Before looking at the complete configuration, you need to understand vnsabsnode and vnsabsconnection Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 17 of 21

18 Each node in the graph has a name and connectors, as shown in Figure 17. Figure 17. Abstract Node <vnsabsnode name = Node1" functype="gothrough" > <vnsabsfuncconn name = connector1" direction = "input"> </vnsabsfuncconn> <vnsabsfuncconn name = connector2" direction = "output"> </vnsabsfuncconn> The vnsabsconnection configuration creates an object with an arbitrary name chosen by the user. This object has the two entities that need to be stitched together to create a connection. This configuration attaches the node SLB1 to the provider side of the contract by using the only connector, AbsTConn. <vnsabsconnection name = <name> <vnsrsabsconnectionconns tdn="uni/tn-customer1/absgraphslbgraph1/abstermnodeprov-input1/abstconn" /> <vnsrsabsconnectionconns tdn="uni/tn-customer1/absgraph-slbgraph1/absnode- SLB1/AbsFConn-external" /> </vnsabsconnection> Figure 18 shows the relationship between the various elements of the service graph. Figure 18. Service Graph Example 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 18 of 21

19 <vnsabsgraph name = "slbgraph1 > <!-- This is the anchor to the Provider side of the contract --> <vnsabstermnodeprov name = "ConnectorToProvider"> <vnsabstermconn name = "ProviderInterface" /> </vnsabstermnodeprov> <!-- This is a generic node that can be rendered by an F5 load balancer --> <vnsabsnode name = "SLB1" functype="goto"> <vnsabsfuncconn name = "outside-slb1" direction = "input"> <vnsrsmconnatt tdn="uni/infra/mdev-f5-bigip-1.0/mfunc- VirtualServer/mConn-external" /> </vnsabsfuncconn> <vnsabsfuncconn name = "inside-slb1" direction = "output"> <vnsrsmconnatt tdn="uni/infra/mdev-f5-bigip-1.0/mfunc- VirtualServer/mConn-internal" /> </vnsabsfuncconn> <vnsrsnodetomfunc tdn="uni/infra/mdev-f5-bigip-1.0/mfunc-virtualserver"/> </vnsabsnode> <!-- This is the anchor to the Consumer side of the contract --> <vnsabstermnodecon name = "ConnectorToConsumer"> <vnsabstermconn name = "ConsumerInterface"/> </vnsabstermnodecon> <!-- Connection1 is just a name, you don't need to reference it later --> <vnsabsconnection name = "Connection1"> <vnsrsabsconnectionconns tdn="uni/tn-customer1/absgraphslbgraph1/abstermnodeprov-providerinterface/abstconn" /> <vnsrsabsconnectionconns tdn="uni/tn-customer1/absgraph-slbgraph1/absnode- SLB1/AbsFConn-outsideSLB1" /> </vnsabsconnection> <!-- Connection2 is just a name, you don't need to reference it later --> <vnsabsconnection name = "Connection2"> <vnsrsabsconnectionconns tdn="uni/tn-customer1/absgraph-slbgraph1/absnode- SLB1/AbsFConn-insideSLB1" /> <vnsrsabsconnectionconns tdn="uni/tn-customer1/absgraphslbgraph1/abstermnodecon-consumerinterface/abstconn" /> </vnsabsconnection> </vnsabsgraph> Association of the Service Graph with a Contract The following configuration shows how to associate a service graph with a contract. Figure 19 illustrates the configuration. <poluni> <fvtenant name= Customer1"> <vzbrcp name="webctrct"> <vzsubj name="http"> 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 19 of 21

20 <vzrssubjgraphatt graphname="webgraph"/> </vzsubj> </vzbrcp> </fvtenant> </poluni> Figure 19. Association of Service Graph with Contract Conclusion Cisco ACI provides an advanced data center networking methodology that abstracts networking constructs from application deployments. In addition, it provides a robust set of network telemetry, security, and Layer 4 through Layer 7 automation functions. The service graph is a concept that allows the definition of a sequence of functions such as SSL offloading, load balancing, and traffic filtering in a way that can be abstracted from the concrete implementation in a given data center. Cisco APIC communicates with the service devices to render the service graph by using the resources that are available in the fabric. These functions can be implemented using the GUI or programmatically in Python and can be automated using the REST API. For More Information For more information about Cisco ACI and service graphs, please refer to Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 20 of 21

21 Printed in USA C / Cisco and/or its affiliates. All rights reserved. This document is Cisco Public Information. Page 21 of 21