This blog post covers the VMware vSphere aspect of the CPU security vulnerabilities known as Meltdown (aka CVE-2017-5754) and Spectre (aka CVE-2017-5753 and CVE-2017-5715).
The vulnerabilities allow an attacker to (among other things) read memory areas from other processes or even other virtual machines running on the same physical CPU. This is a serious issue, since it can be used to retrieve not only sensitive data, but also passwords, encryption keys and other things that can be used in further attacks. At the time of writing, there is no way of detecting when memory has been read using this vulnerability.
VMware has initially issued a patch in VMSA-2018-0002 which
protects against mitigates Spectre attacks against ESXi 6.5 and 6.0 itself (not the guests). This patch is now also included in VMSA-2018-0004 (see below), so there is no need to install it separately. ESXi itself is not vulnerable to Meltdown, since it doesn’t allow untrusted user mode code to run, so currently no such patch is needed.
The newest patches needed for mitigation of Meltdown/Spectre all the way up to the Guest OS can be found at VMSA-2018-0004.3 They are called Hypervisor-Assisted Guest Mitigation for branch target injection, and patch ESXi to allow for the guest operating systems in the virtual machines to mitigate the ‘Branch target injection’ vulnerability in CVE-2017-5715. The Guest OSs will need to be patched individually for this to work (see info further below).
There are several steps necessary to deploy and activate these patches:
- Upgrade vCenter Server(s) to the patched version (6.5 U1g, 6.0 U3e or 5.5 U3h) – This patches the vCenter appliance itself, as well as enables the new EVC modes for exposing the new CPU instructions (since the cluster hosts will have a discrepancy during the patching/testing process).
- Install the corresponding ESXi patches using VUM as usual – This will also upgrade the physical server’s CPU microcode to enable the guest mitigation, unless this has already been done using the server vendor’s BIOS/firmware upgrade or similar mechanism.
The steps above can be performed without any disruption to the VMs. However, as usual we recommend that the patches are initially tested on a limited number of ESXi hosts per cluster, to make sure they all come back up and don’t cause any problems. Keep an eye on your logs using Log Insight, and look for discrepancies in log volume and/or error messages (there is a feature in Log Insight for displaying differences in logs between time periods)
Now comes the slightly more time consuming part, since it will require rebooting each VM at least once:
- Install the applicable security patches for your guest operating system. A good list is available here. Try to sync the reboot (if one is necessary) with step 3 below.
- Make sure all your VMs are upgraded to virtual hardware version 9 or newer. If you need to upgrading the virtual hardware, aim for version 11 or higher, since it includes new CPU features that can reduce the performance impact of these patches.
- Power cycle the VM (cold boot).
The steps above are described in more detail at https://kb.vmware.com/s/article/52085.
After completion, use William Lam’s excellent script that checks that the above steps have all been completed successfully on VMs/clusters/environments. It checks both the vHW version and the presence of the new CPU features that the CPU microcode update and new EVC mode enables. You can find it at https://www.virtuallyghetto.com/2018/01/verify-hypervisor-assisted-guest-mitigation-spectre-patches-using-powercli.html
Make sure you also patch any remaining VMware software/appliances, according to https://kb.vmware.com/s/article/52264
Finally, once you’ve patched your entire environment, I’m sorry to tell you that this is probably not the last we’ve heard about Spectre-related vulnerabilities and attacks. It might unfortunately haunt us for quite some time, hence the name.
[Previous update 2018-01-08]:
The VMware KB article https://kb.vmware.com/s/article/52245 gives us some additional information. In summary:
- There is no expected performance degradation from applying the VMware Hypervisor-specific mitigation patches (VMSA-2018-0002) or the VMware Hypervisor-assisted Guest OS mitigation patches (more info on these below)
- The VMware Hypervisor-assisted Guest OS patches will patch the physical CPU microcode, unless a server vendor firmware update has already done so. These microcode patches are not expected to cause any performance degradation either.
- VMware appliances (such as vCenter Server, NSX Manager, vROps Manager etc) might be susceptible to the vulnerabilities. There is a separate KB article listing which are affected or not at https://kb.vmware.com/s/article/52264