Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Transcription

1 SER2980BU vcenter Server 6.5 Deep Dive and Troubleshooting Pawan Naidu Prateek Jain VMworld 2017 Content: Not for publication #VMworld #SER2980BU

2 Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined. VMworld 2017 Content: Not for publication CONFIDENTIAL 2

3 Agenda 1 Why High Availability for vcenter Server 2 Platform Services Controller High Availability 3 vcenter High Availability aka VCHA 4 Troubleshooting vcenter High Availability Issues 5 Troubleshooting Certificates Issues VMworld 2017 Content: Not for publication #SER2980BU CONFIDENTIAL 4

4 Why High Availability is Important for vcenter Server Backup Orchestrator vrops VRA Loginsight Horizon View VMworld 2017 Content: Not for publication NSX SRM VASA VCD #SER2980BU CONFIDENTIAL 4

5 Platform Services Controller High Availability VMworld 2017 Content: Not for publication

6 VMware Platform Services Controller (PSC) Platform Services Platform Controller Services Controller Platform Services Controller Platform Platform Services Services Controller Controller Apart from Single Sign On functionality PSC got several new components introduced in vcenter 6.0 which continued in version 6.5 Component Manager (CM) Service Control Agent (SCA) VMware Certificate Authority (VMCA) VMware Endpoint Certificate Store (VECS) VMware Authentication Framework Demon (VMAFD) VMworld 2017 Content: Not for publication VMware HTTP Reverse Proxy (RHTTP) VMware License Service Global Permissions 9 #SER2980BU CONFIDENTIAL 7

7 Loadbalancer Overview External Platform Services Controller Deployed in a Single Site Traffic Flow Functionality VMworld 2017 Content: Not for External Platform Services Controller publication SSO Site A 8

8 Platform Services Controller High Availability Update Config and Endpoint Run SSOConfig and LsEndpoint Certificate Validation 05 Confirm if we get the same Certificate on LB and on PSC nodes 06 PSC HA 01 Installation PSC nodes MachineSSL Generate and replace MachineSSL Validation VMworld 2017 Content: Not for publication MachineSSL has to be validated on all the PSC nodes Loadbalancer 04 Prerequisites & Configuration #SER2980BU CONFIDENTIAL 9

9 Platform Services Controller High Availability PSC HA VMworld 2017 Content: Not for Single Site Node 1 Node 2 01 Installation Install and configure PSC nodes in a single SSO domain and Site publication #SER2980BU CONFIDENTIAL 10

10 Platform Services Controller High Availability PSC HA PSC HA VMworld MachineSSL Generate and replace machine SSL for all the PSC nodes behind the LB KB: FQDN and Common Name MachineSSL configured in SSL Passthrough Mode only Content: Not for publication #SER2980BU CONFIDENTIAL 11

11 Platform Services Controller High Availability /var/log/vmware/vmacd/certificate-manager.log 03 Validation At this stage we must validate the properties of the MachineSSL on both the PSC nodes T06:57:17.195Z INFO certificate-manager [u' MACHINE_CERT'] T06:57:17.195Z INFO certificate-manager Running command : ['/usr/lib/vmware-vmafd/bin/vecs-cli', 'entry', 'getcert', '-- text', '--store', 'MACHINE_SSL_CERT', '--alias', ' MACHINE_CERT'] T06:57:17.210Z INFO certificate-manager Output : Certificate: Data: PSC HA Version: 3 (0x2) Serial Number: 79:00:00:00:03:7a:4b:03:66:55:32:c2:e4:00:00:00:00:00:03 Signature Algorithm: sha256withrsaencryption Issuer: DC=local, DC=virtualworld, CN=virtualworld-AD-CA Validity Not Before: Jul 25 06:18: GMT Not After : Jul 25 06:18: GMT Subject: C=IN, ST=KA, L=Bangalore, O=VMware, OU=GSS, CN=psc-lb.virtualworld.local VMworld 2017 Content: Not for publication X509v3 Subject Alternative Name: DNS:psc1.virtualworld.local, DNS:psc2.virtualworld.local, DNS:psc-lb.virtualworld.local #SER2980BU CONFIDENTIAL 12

12 Platform Services Controller High Availability PSC HA VMworld 2017 Content: Not for 04 Loadbalancer Prerequisites NSX Manager configured and registered to an existing vcenter Server At least one Edge deployed to configure Edge Services Gateway Follow KB : publication #SER2980BU CONFIDENTIAL 13

13 Platform Services Controller High Availability [ /usr/lib/vmware-vmafd/bin ]#./vecs-cli entry list --store MACHINE_SSL_CERT Number of entries in store : 1 Alias : MACHINE_CERT Entry type : Private Key Certificate : -----BEGIN CERTIFICATE----- MIIF2TCCBMGgAwIBAgITeQAAAAN6SwNmVTLC5AAAAA AAAzANBgkqhkiG9w0BAQsFADBSMRUwEwYKCZImiZPy LGQBGRYFbG9jYWwxHDAaBgoJkiaJk/IsZAEZFgx2aXJ0d PSC HA WFsd29ybGQxGzAZBgNVBAMTEnZpcnR1YWx3b3JsZC1 BRC1DQTAeFw0 8MxTExreHvNBzeja2hRK+Rj7JBvuSGn9tLYFF8Mi9sDu3sb 0pL0hCxH4ljSgDFNM4LTuv3PqeWBIw9Z53171qYMM/V+7 VyVOenKUQqTgaIBsGC9D1pFluufpq6yQyhcn+o5ZMkV6Yj CRtpS9a9XK271vUycv/hR3GJ+lrjehGJaKINcZqLqHKwxGn 5VMaIoWQ== -----END CERTIFICATE Certificate Validation Validating MachineSSL vcenter Server Appliance - /usr/lib/vmwarevmafd/bin/vecs-cli entry list --store root@psc1 [ ~ ]# openssl s_client -connect psc-lb.virtualworld.local:443 MACHINE_SSL_CERT text Number of entries in store : 1 Alias : MACHINE_CERT Validate if the loadbalancer is presenting the Entry type : Private Key same certificate Certificate : -----BEGIN CERTIFICATE----- openssl s_client -connect psclb.virtualworld.local:443 MIIF2TCCBMGgAwIBAgITeQAAAAN6SwNmVTLC5AAAAA AAAzANBgkqhkiG9w0BAQsFADBSMRUwEwYKCZImiZPy LGQBGRYFbG9jYWwxHDAaBgoJkiaJk/IsZAEZFgx2aXJ0d WFsd29ybGQxGzAZBgNVBAMTEnZpcnR1YWx3b3JsZC1 BRC1DQTAeFw0 8MxTExreHvNBzeja2hRK+Rj7JBvuSGn9tLYFF8Mi9sDu3sb 0pL0hCxH4ljSgDFNM4LTuv3PqeWBIw9Z53171qYMM/V+7 VyVOenKUQqTgaIBsGC9D1pFluufpq6yQyhcn+o5ZMkV6Yj CRtpS9a9XK271vUycv/hR3GJ+lrjehGJaKINcZqLqHKwxGn 5VMaIoWQ== -----END CERTIFICATE----- VMworld 2017 Content: Not for publication #SER2980BU CONFIDENTIAL 14

14 Platform Services Controller High Availability PSC HA [ /usr/lib/vmware-sso/bin ]# python updatessoconfig.py --lbfqdn=psc-lb.virtualworld.local VMworld 2017 Content: Not for [ /usr/lib/vmware-sso/bin ]# python UpdateLsEndpoint.py --lbfqdn=psc-ha.virtualworld.local -- password=******* 06 Update Config and Endpoint updatessoconfig.py updatelsendpoint.py publication #SER2980BU CONFIDENTIAL 15

15 Platform Services Controller High Availability Update Config and Endpoint Run SSOConfig and LsEndpoint Certificate Validation 05 Confirm if we get the same Certificate on LB and on PSC nodes 06 PSC HA 01 Installation PSC nodes MachineSSL Generate and replace MachineSSL Validation VMworld 2017 Content: Not for publication MachineSSL has to be validated on all the PSC nodes Loadbalancer 04 Prerequisites & Configuration #SER2980BU CONFIDENTIAL 16

16 vcenter High Availability VMworld 2017 Content: Not for publication

17 Comparing vcenter High Availability Easy to deploy No shared storage required No reliance on physical mode RDMs Managed via the vsphere Web Client Failover across vsphere Clusters vcenter High Availability VMworld 2017 Content: Not for Cross DC failover w/o storage replication vsphere HA Windows Server Failover Clustering publication #SER2980BU CONFIDENTIAL 18

18 vcenter Server High Availability 01 VMworld 2017 Content: Not for Available only with vcenter Server Appliance Consists of 3 nodes i.e. Active, Passive and Witness Node publication Basic Mode and Advanced mode deployment options available Enhanced Linked mode not available with Embedded PSC vcenter HA deployment 05 Supports Both Embedded and External PSC deployments #SER2980BU CONFIDENTIAL 19

19 vcenter High Availability Deployment Modes 1 VMworld Basic (Wizard-driven) vcenter Server Appliance virtual machine running in the same vcenter vcenter in same SSO domain Automated deployment Content: Not for publication Advanced (Manual Configuration) Active/Passive and Witness nodes are running on vcenter servers in different SSO domain Manual Cloning and Manually Configuring 2 nd network interface Manual DRS Rules Configuration #SER2980BU CONFIDENTIAL 20

20 Cluster Modes of Operation VMworld 2017 Content: Not for publication #SER2980BU CONFIDENTIAL 21

21 Validating VCHA Configuration VMworld 2017 Content: Not for publication #SER2980BU CONFIDENTIAL 22

22 Troubleshooting vcenter High Availability Issues VMworld 2017 Content: Not for publication

23 vcenter HA Failure Scenarios #1 Active Passive VMworld 2017 Content: Not for Passive node promotes to Active and serves client requests Active Node failure triggers a failover & Cluster Degraded publication Users would see Failover in Progress on vsphere UI during failover Witness #SER2980BU CONFIDENTIAL 24

24 vcenter HA Failure Scenarios #2 Active VMworld 2017 Witness Passive Active node continues to serves client requests No Failover on Passive Node failure & Cluster Degraded Content: Not for publication Cluster Quorum intact as Witness and Active nodes are available #SER2980BU CONFIDENTIAL 25

25 vcenter HA Failure Scenarios #3 Active VMworld 2017 Witness Passive Active node continues to serves client requests No Failover on Witness Node failure & Cluster Degraded Content: Not for publication Cluster Quorum intact as Passive and Active nodes are available #SER2980BU CONFIDENTIAL 26

26 vcenter HA Failure Scenarios #4 Active VMworld 2017 Passive Active node would stop serving client requests VCHA cannot tolerate multiple node failures Content: Not for publication Resolve node issues to achieve quorum and ensure cluster stability Witness #SER2980BU CONFIDENTIAL 27

27 Upgrading Nodes in a VCHA Cluster Identify the current roles of deployed VCHA nodes Identify Nodes Maintenance Place VCHA Cluster in maintenance mode Mount Patch ISO on all 3 nodes Patch ISO Patch the witness Node, Command - softwarepackages install --iso -- accepteulas Patching Witness Node Patching Passive Node Patch the Passive node Command - softwarepackages install --iso -- accepteulas Initiate Failover to change node roles Initiate Failover Patch the last node VMworld 2017 Content: Not for publication Patch the last node, Command - softwarepackages install --iso -- accepteulas #SER2980BU CONFIDENTIAL 28

28 Troubleshooting Certificates Issues VMworld 2017 Content: Not for publication

29 Replacing SSL Certificates on a VCHA Cluster 1 2 Best Practices for replacing Certificates Replace Machine SSL Certificate on the vcenter Server Appliance Node prior to configure VCHA Running VCHA Setup VMworld 2017 Content: Not for publication or distribution Removal of VCHA is required, before replacing Machine SSL on vcenter Server

30 Replacing Machine SSL Certificates on PSC HA [ T18:37:52.924Z] [INFO ] http-bio-9090-exec-9 com.vmware.vise.vim.security.sso.impl.ssocmlocatorimpl Fetching SSO endpoints from CM [ T18:37:52.995Z] [ERROR] http-bio-9090-exec-9 com.vmware.vise.vim.security.sso.impl.ssoserviceimpl Error when fetching sts root certificates java.util.concurrent.executionexception: java.security.cert.certificateexception: Could not parse certificate: java.io.ioexception: java.lang.illegalargumentexcep tion: Input byte array has incorrect ending byte at 2004 at java.util.concurrent.futuretask.report(futuretask.java:122) at java.util.concurrent.futuretask.get(futuretask.java:206) at com.vmware.vise.vim.security.sso.impl.ssoserviceimpl.getstsconfig(ssoserviceimpl.java:351) at com.vmware.vise.vim.security.sso.impl.ssoserviceimpl.getstscertificates(ssoserviceimpl.java:422) VMworld 2017 Content: Not for publication #SER2980BU CONFIDENTIAL 31

31 VMworld 2017 Content: Not for publication #SER2980BU CONFIDENTIAL 32

32 VMworld 2017 Content: Not for publication

33 VMworld 2017 Content: Not for publication