CNA1509BU Developer-Ready Infrastructure from VMware and Pivotal Merlin Glynn, VMware Ramiro Salas, Pivotal #VMworld #CNA1509BU
Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. #CNA1509BU CONFIDENTIAL 2
CNA1509BU Developer-Ready Infrastructure from VMware and Pivotal Merlin Glynn (Vmware) Ramiro Salas (Pivotal) #VMworld #CNA1509BU
Agenda 1 2 Pivotal Cloud Foundry 101 Why do my Developers want it? Site Reliability Engineering (SRE) The role of the Operator 3 Ops: PCF & vsphere 4 Ops: PCF Network & Security 5 Ops: PCF Monitoring & Logging 6 Ops: PKS & Developer Ready Infrastructure #CNA1509BU CONFIDENTIAL 4
Pivotal Cloud Foundry 101 Why do my Developers want It?
Pivotal Cloud Foundry 101 Developer `cf push` war Here is my source code Run it on the cloud for me I do not care how Root FS Build Pack Staging Drop war let PCF Routing PCF Routing PCF Routing A I URL Request: myapp.foo.com myapp.foo.com Availability Zone 1 Availability Zone 2 Availability Zone 3 PCF Elastic Runtime A I #CNA1509BU CONFIDENTIAL 6
Pivotal Cloud Foundry 101 How do the Containers run? Running Containers are called Application Instances (AIs) PCF Diego schedules & maintains the health of containers on special VMs called Diego Cells. GoRouters are another type of VM that balances & forwards requests to the correct Cell AIs VMworld 2017 URL Request: myapp.foo.com VM= GoRouter PCF Routing myapp.foo.com A I Availability Zone 1 PCF Elastic Runtime A I VM=Cell Content: Not for publication VM=Cell PCF Diego #CNA1509BU CONFIDENTIAL 7
Pivotal Cloud Foundry 101 Developer `cf scale myapp i 3` Self Service & Automated Scaling Diego Dynamically keeps apps Healthy in case of IaaS faults No Intervention from the Developer and no Tickets PCF also provides auto recovery when IaaS is repaired CF API VMworld 2017 PCF Routing PCF Routing PCF Routing Content: Not for publication A I A I A myapp.foo.com I Availability Zone 1 Availability Zone 2 Availability Zone 3 PCF Elastic Runtime A I vsphere #CNA1509BU CONFIDENTIAL 8
Pivotal Cloud Foundry 101 PCF Service Broker `cf create-service mysql` `cf bind-service mysql` Developer PCF Mysql PCF RabbitMQ PCF Spring Svcs Self Service to app services PCF Managed & External Binds creds to app (12 Factor) CF API DATA VMworld 2017 Content: Not for A I MSG BUS jdbc url & creds vcap env {} myapp.foo.com A I Netflix OSS publication User Provided Availability Zone 1 Availability Zone 2 Availability Zone 3 PCF Elastic Runtime #CNA1509BU CONFIDENTIAL 9
Pivotal Cloud Foundry 101 Operator `bosh deploy mypcf` Pivotal Cloud Foundry is a distributed system made up of many VMs Automated deployment and health management is provided by Pivotal Operations Manager & BOSH Enables API driven & repeatable deployment Ops Manager (OVA) BOSH PCF Routing PCF Routing PCF Routing A I A I Developer CF API A myapp.foo.com I Availability Zone 1 Availability Zone 2 Availability Zone 3 PCF Elastic Runtime A I vsphere #CNA1509BU CONFIDENTIAL 10
What Pivotal Cloud Foundry solves for Agility & Cost Savings Apps Team Manage All Old School Apps Data Runtime Middleware O/S Virtualization Servers Storage Networking Apps Manages IaaS + Containers Apps Data Runtime Middleware O/S Virtualization Servers Storage Networking Ops Cloud Native Apps Manages Apps Data Runtime Middleware O/S Virtualization Servers Storage Networking PCF Ops Cloud Native Maturity _Ops #CNA1509BU CONFIDENTIAL 11
Site Reliability Engineering (SRE) The role of the Operator
Key Developer Ready Infrastructure Personas App Lifecycle Cloud-Native App Services Automation Monitoring, Security & Logging Cloud-Native Physical Infrastructure VMworld 2017 Content: Not for Developer Operator IT Operator Innovation of Business Capability as Cloud native Apps Develop, Deploy, Scale, Monitor Apps publication SRE (Site Reliability Engineering) Deploy, Scale, Operate Physical Infrastructure is Operated Network & Security Control Policy is defined #CNA1509BU CONFIDENTIAL 13
Key Developer Ready Infrastructure Personas Cloud Native Applications at scale can & should be kept running by a 2 Pizza Team mentality (DevOps in Action) Operators Site Reliability Engineers (SRE) s Role Shift In most case the VI Admins (IT Ops), are becoming the Operators Developer Operator IT Operator Innovation of Business Capability as Cloud native Apps Develop, Deploy, Scale, Monitor Apps SRE (Site Reliability Engineering) Deploy, Scale, Operate Physical Infrastructure is Operated Network & Security Control Policy is defined #CNA1509BU CONFIDENTIAL 14
Site Reliability Engineers What do they Do? is Reliable Capacity Is planned for is Secured & Controlled is Auditable Developers are Agile as Code{} % of Time Spent Operator SRE (Site Reliability Engineering) Deploy, Scale, Operate They combine the knowledge & skills of the IT Ops guys with the mandate of providing ops processes as code {} Traditional Ops Coding Ops #CNA1509BU CONFIDENTIAL 15
Ops: PCF & vsphere
vsphere Fundamentals for PCF PCF is many services deployed as many VMs, called instances PCF distributes instances across availability zones (AZ) vsphere s & Resource Pools map to PCF AZs Operator PCF Org App App Developer PCF Space go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr OTHER OTHER INSTANCES INSTANCES OTHER OTHER INSTANCES INSTANCES OTHER OTHER INSTANCES INSTANCES Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 AZ1 AZ2 AZ3 Ops Manager (OVA) BOSH #CNA1509BU CONFIDENTIAL 17
vsphere Fundamentals for PCF VMotion Helps keep load balanced within clusters vsphere HA helps recover instances Operator PCF Org App App Developer PCF Space go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr OTHER OTHER INSTANCES INSTANCES OTHER OTHER INSTANCES INSTANCES OTHER OTHER INSTANCES INSTANCES Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 AZ1 AZ2 AZ3 Ops Manager (OVA) BOSH #CNA1509BU CONFIDENTIAL 18
vsphere Fundamentals for PCF vsan, VMFS, NAS Datastores are supported Storage SVMotion is NOT recommended Operator PCF Org App App Developer PCF Space go_rtr go_rtr go_rtr go_rtr go_rtr go_rtr OTHER OTHER INSTANCES INSTANCES OTHER OTHER INSTANCES INSTANCES OTHER OTHER INSTANCES INSTANCES Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 Cell_0 Cell_1 Cell_2 AZ1 AZ2 AZ3 Ops Manager (OVA) BOSH Data store Data store Data store 19 #CNA1509BU CONFIDENTIAL
vsphere Fundamentals for PCF Single PCF Availability Zone deployments are supported Limited availability Usually smaller scale deployments Operator PCF Org App App Developer go_rtr PCF Space Cell_0 Cell_1 Cell_2 AZ1 go_rtr OTHER OTHER INSTANCES INSTANCES VMworld 2017 Content: Not for publication Ops Manager (OVA) BOSH #CNA1509BU CONFIDENTIAL 20
vsphere Fundamentals for PCF Most instances do not deploy dynamically Operator needs to plan for scale based on Application Instances (AIs) https://pcfsizer.cfapps.io Operator 100 AIs AZ1 AZ2 AZ3 Ops Manager (OVA) BOSH #CNA1509BU CONFIDENTIAL 21
Ops: PCF Network & Security
Network Fundamentals for PCF NSX Logical Switch Primer NSX Logical Switches provide a single Layer 2 over Layer 3 Network. Operator can have a VLAN per Physical Fault Domain () Logical Switch presents as a simple single network to PCF VMworld 2017 Content: Not for publication NSX Logical Switch 192.168.10.0/24 172.16.101.1/24 172.16.102.1/24 172.16.103.1/24 VTEP VLAN VTEP VLAN VTEP VLAN #CNA1509BU CONFIDENTIAL 23
Network Fundamentals for PCF NSX Logical Switches give PCF a common Layer 2 Network across AZs Recommend 1 Switch per deployment, sometimes called a PCF tile 192.168.0.0/28 192.168.8.0/22 192.168.12.0/24 Infra Operations Manager BOSH ERT Elastic RunTime Deployment SVC1 AZ1 AZ2 AZ3 MySQL Deployment VTEP VLAN VTEP VLAN VTEP VLAN #CNA1509BU CONFIDENTIAL 24
Network Fundamentals for PCF NSX Distributed Logical Router (DLR) Allows for hundreds of connected Logical Switches NSX Edge has a 10 Interface limit Optimized East & West traffic Infra Operations Manager BOSH NSX Distributed Logical Router 192.168.0.0/28 192.168.8.0/22 192.168.12.0/24 ERT Elastic RunTime Deployment SVC1 AZ1 AZ2 AZ3 MySQL Deployment VTEP VLAN VTEP VLAN VTEP VLAN #CNA1509BU CONFIDENTIAL 25
Network Fundamentals for PCF PCF requires a production grade external load balancer NSX Edge provides: SNAT & DNAT SSL Term & Acceleration HTTP & TCP Load Balancing RCF 1918 on the Logical Switches = Repeatability Infra Operations Manager BOSH VTEP VLAN NSX Edge Routing Logical Switch NSX Distributed Logical Router 192.168.0.0/28 192.168.8.0/22 192.168.12.0/24 ERT Elastic RunTime Deployment VTEP VLAN PUBLIC VIPS INTERNAL NAT SVC1 AZ1 AZ2 AZ3 MySQL Deployment VTEP VLAN #CNA1509BU CONFIDENTIAL 26
Network Security & Controls Use DFW NSX Distributed Firewall (DFW) Single Policy Engine across all Logical Switches Use for Internal East/West and Egress Control Use Edge for Perimeter Ingress Control 192.168.0.0/28 192.168.8.0/22 192.168.12.0/24 Infra BOSH VTEP VLAN NSX Edge Routing Logical Switch NSX Distributed Logical Router ERT Elastic RunTime Deployment VTEP VLAN PUBLIC VIPS INTERNAL NAT SVC1 AZ1 AZ2 AZ3 MySQL Deployment VTEP VLAN #CNA1509BU CONFIDENTIAL 27
Network Security & Controls PCF Application Security Groups (ASG): Uses iptables in the Diego Cell Server Controls Egress only at the container source level Can control any IP address as the target Operator Declares in the PCF Org Operator cf create-security-group SECURITY-GROUP PATH-TO-RULES-FILE cf create-security-group dev-mssql mssql.json Prod Mssql 10.0.11.10 Prod Mssql 192.168.11.10 AppA AppB AppC PCF Space [ { "protocol": "tcp", "destination": "10.0.11.0/24", "ports": "1-65535" }, { "protocol": "udp", "destination": "10.0.11.0/24", "ports": "1-65535" } ] #CNA1509BU CONFIDENTIAL 28
Network Security & Controls PCF Container to Container Networking: Creates and Overlay (VXLAN) Controls ingress & egress between Ais (containers) Uses CNI Today Flannel Tomorrow NSX-T Developer can Declare in CI/CD cf allow-access SOURCE-APP DESTINATION-APP --protocol PROTOCOL --port PORT cf allow-access AppA Appc --protocol TCP --port 443 PCF Org Developer AppA AppB AppC PCF Space #CNA1509BU CONFIDENTIAL 29
Ops: PCF Monitoring & Logging
Monitoring & Logging Metrics & Logs will emit from many sources: PCF PCF Applications vsphere NSX Physical & Logical Operator MUST leverage ALL of them METRICS LOGS #CNA1509BU CONFIDENTIAL 31
Monitoring & Logging Developer Operator I need to keep my apps healthy I need self service to my Apps Log s Virtual Data Center I need to instrument my Apps (APM) I need to keep the healthy I need to plan capacity I need to watch & Alert on KPIs I need to audit & Report vrealize Operations (vrops) KPI Visualization & Alerting for: PCF vsphere NSX vrops #CNA1509BU CONFIDENTIAL 32
Monitoring & Logging Developer Operator I need to keep my apps healthy I need self service to my Apps Log s Virtual Data Center I need to instrument my Apps (APM) I need to keep the healthy I need to plan capacity I need to watch & Alert on KPIs I need to audit & Report vrealize Log Insight (vrli) Log Aggregation & Alerting for: PCF vsphere NSX vrops vrli vrni vrealize Network Insight (vrli) Network & Security Reporting for: Physical & Logical Networks #CNA1509BU CONFIDENTIAL 33
Monitoring & Logging Developer Operator I need to keep my apps healthy I need self service to my Apps Log s Virtual Data Center I need to instrument my Apps (APM) I need to keep the healthy I need to plan capacity I need to watch & Alert on KPIs I need to audit & Report vrealize Log Insight (vrli) Long term log Aggregation & Alerting for: Applications Running in PCF vrops vrli vrni #CNA1509BU CONFIDENTIAL 34
Monitoring & Logging Developer Operator I need to keep my apps healthy I need self service to my Apps Log s Virtual Data Center I need to instrument my Apps (APM) I need to keep the healthy I need to plan capacity I need to watch & Alert on KPIs I need to audit & Report Wavefront by VMware, Application Performance Monitoring (APM) for: Applications Running in PCF vrops vrli Wavefront vrni #CNA1509BU CONFIDENTIAL 35
Wrapping It up w/ PKS and DRI
Kubernetes 101 Developer `kubectl apply f myapp.yml` Docker Registry Master etcd Service: nodeport kube-proxy kube-proxy POD URL Request: myapp.foo.com/k8siscool or Worker K8s Load Balancer distribution POD Worker #CNA1509BU CONFIDENTIAL 37
Kubo What is KUBO? Kubernetes Powered By BOSH Can Be deployed independent of PCF Can Deploy & Manage Multiple K8S s Day 1 Day 2 Operator BOSH Master Master etcd etcd etcd kube-proxy POD Worker Worker K8s service Load Balancer kube-proxy POD Worker Worker #CNA1509BU CONFIDENTIAL 38
Analytics Logging Operations Automation Monitoring Security Container Registry etcd VMworld 2017 vsphere master VMware PKS worker Kubernetes on BOSH (Kubo) NSX etcd BOSH (PKS) master worker vsan GCP Service Broker Content: Not for publication #CNA1509BU CONFIDENTIAL 39
PCF + PKS PCF + PKS Integrated PCF Service Broker Developer `cf create service mykubo` I need a K8S cluster VMworld 2017 Content: Not for BOSH Why Kubernetes + PCF App Packaging Need Only Need Specific hardware and image stack (example GPU) Complex multi VM Data services with perisistent diskj like ELK not a good fit for PCF Elastic Runtime staging publication #CNA1509BU CONFIDENTIAL 40
PCF + PKS PCF + PKS Integrated PCF Service Broker Integrated PCF Routing Developer `http://myk8sapp.io` I need to route to my K8S Service VMworld 2017 Content: Not for publication BOSH #CNA1509BU CONFIDENTIAL 41
Developer Ready Infrastructure Solves for DevOps Reqs Operator Automation Day 2 Operations Control Automation Developer Application Services Application Services or Container Services Application Logging & Monitoring Self Service Container Services Compute Network Storage App Monitoring App Logging Logging Monitoring #CNA1509BU CONFIDENTIAL 42
Developer Ready Infrastructure Solves for DevOps Reqs Operator Automation Day 2 Operations Control BOSH Developer Pivotal Cloud Foundry PCF Application Services or Container Services Application Logging & Monitoring Self Service PKS BOSH powered Kubernetes vsphere NSX vsan Wavefront vrli (Dev) vrli (Ops) vrops vrni #CNA1509BU CONFIDENTIAL 43
Developer Ready Infrastructure @ VMworld VMworld US Key Focus Description CNA1509BU DRI Developer-Ready Infrastructure from VMware & Pivotal CNA1612BU CNA2006BU PCF & Kubo DRI Use Cases: Deploying real-world workloads on Kubernetes and Pivotal Cloud Foundry Deep Dive: Architecting Container Services with VMware and Pivotal Developer Ready Infrastructure CNA2080BU Kubo Deep Dive: How to Deploy and Operationalize Kubernetes CNA3429BU CNA3430BU MGT2871BU Kubo PCF PCF & vrops, vrli Basics of Kubernetes on BOSH: Run Production-grade Kubernetes on the SDDC Your Enterprise Cloud-Native App : An Introduction to Pivotal Cloud Foundry Bridging the Operations Gap Between the Software-Defined Data Center and Pivotal CF for VMware Deployments NET1523BU PCF & NSX Integrating NSX and Cloud Foundry PAR4411PU DRI Emerging Technologies with VMware and Pivotal - presented jointly by VMware, Pivotal and Special Guest Speakers from Cognizant and WWT #CNA1509BU CONFIDENTIAL 44
Thank You & Any Questions