[Update 2016-01-07]: VMware has released vCenter Server 6.0 U1b, which claims to solve the ‘Server certificate chain not verified’ problem (according to the Release Notes). It is not clear whether updating a PSC and vCenter Server will solve any existing problem or if it only prevents the problem from appearing when changing PSC/vCS certificates.
Background: I was in a project for a customer where we built a self-service portal based on vSphere 6.0 (vCSA with external PSC) along with vRA and NSX. When we tried to register the NSX Manager to the vCenter SSO lookup service (to be able to use the accounts and identity sources in SSO), we got an error message:
NSX Management Service operation failed.(Initialization of Admin Registration Service Provider failed. Root Cause: Error occurred while registration of lookup service, com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified)
We tried troubleshooting the PSC/SSO SSL certificate and its chain, but couldn’t find anything wrong with it. We had replaced its machine certificate with a customer CA issued one, and when connecting to it using a web browser, we got the green SSL icon and no error messages.
After looking for another answer we eventually found the following article: vCenter Server or Platform Services Controller certificate validation error for external VMware Solutions in vSphere 6.0 (2109074). It described not only our error message for NSX Manager, but also similar problems from multiple VMware solutions when connecting to vCenter Server or SSO:
- vSphere Replication: Unable to obtain SSL certificate: The vCenter Server vCenter_FQDN is not correctly registered in LookupService
- vCenter Site Recovery Manager: SRM server with GUID GUID of vCenter not paired.
Failed to connect to vCenter Server at vCenter_FQDN:443/sdk. Reason:
com.vmware.vim.vmomi.core.exception CertificateValidationException: Server certificate chain not verified.
VMware NSX for vSphere (NSX-v): NSX Management Service operation failed.(Initialization of Admin Registration Service Provider failed. Root Cause: Error occurred while registration of lookup service, com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified)
VMware Integrated OpenStack: Connection failed!
Please check whether the server has enabled SSO from management server log at:/installer.log.In the VMware Integrated OpenStack installer.log file, you see entries similar to:[2015-04-10 14:49:18,848 main ERROR com.vmware.vim.install.impl.AdminServiceAccess] com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified
[2015-04-10 14:49:18,849 main DEBUG com.vmware.vim.install.impl.AdminServiceAccess]
com.vmware.vim.sso.admin.exception.CertificateValidationException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified
VMware vCenter Support Assistant: Something failed. Try Again.
com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified
Server certificate chain not verified
peer not authenticated
VMware Customer Experience Improvement Program
The vSphere Web Client reports: Error occurred while processing request. Check vSphere WebClient logs for details.
The vsphere_client_virgo.log reports an error similar to:
[2015-10-07T13:08:41.001Z] [ERROR] http-bio-9090-exec-3 70000101 100009 200004 com.vmware.vsphere.client.ceip.impl.CeipServiceImpl Error occurred in showNotification. com.vmware.vim.binding.vmodl.fault.SystemError: Internal server error.
The fix for the problem is already included as a script named ls_update_certs.py in the vCSA, but it first requires extracting some thumbprints and certificates from the PSC lookupservice /mob interface and doing some cutting and pasting. It’s all described in one of the following KB articles:
- vCenter Server certificate validation error for external solutions in environments with Embedded Platform Services Controller (2121689) (for environments with embedded PSC)
- vCenter Server or Platform Services Controller certificate validation error messages for external solutions in environments with a External Platform Services Controller (2121701) (for environments with external PSC)
So it seems a lot of customers should be having these problems, or perhaps it only affects some of them? If all, then surely VMware would have fixed this in vSphere 6.0 Update 1, right? Well, I tested this on two other customers who had also upgraded to vSphere 6 or installed vSphere 6.0 U1 from scratch, and they both had the same problem. The common denominator is that they all had replaced the SSL certificates for the vCenter Server and their PSC (if external).
The bug seems to be that the VMCA/VECS/CertTool doesn’t properly update the metadata of the SSL certificates when replacing them. This leads to the external solutions (NSX, SRM etc.) not being able to verify that the presented vCenter/PSC SSL certificate is “okay”, which it would have done by comparing its thumbprint to the thumbprint that’s viewable in the /mob, also known as ‘sslTrust Anchors’.
There is still no solution for this announced from VMware, and the symtom KB article (2109074) is now even mentioned in multiple general KB articles such as Replacing default certificates with CA signed SSL certificates in vSphere 6.0 (2111219) and Important information before upgrading to vSphere 6.0 Update 1 (2131738)
Anyone else seeing this bug? The easiest way to verify that you have it is to deploy vCenter Support Assistant and try to connect it to your vCenter Server.