[Update 2018-03-20]: Now the new Intel microcode patches seem stable enough, and have been released in an updated VMware ESXi patch release. Go to https://kb.vmware.com/s/article/52085 and follow the instructions on how to upgrade vCenter Server, ESXi hosts, VM hardware and VM Guest OS.
The blog text below will be updated with current status later, but for now just follow the KB article above and then use the script at https://www.virtuallyghetto.com/2018/01/verify-hypervisor-assisted-guest-mitigation-spectre-patches-using-powercli.html to check that the mitigation works all the way up to the Guest OS.
[Update 2018-01-13]: VMware is now recommending that we don’t install the CPU microcode patches included in VMSA-2018-0004. This is after Intel has found issues in these microcode patches that “can occur when the speculative execution control is actually used within a virtual machine by a patched OS”.
If you have not already installed the VMSA-2018-0004 patches (ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG), don’t do it until further information becomes available from Intel and VMware. The patches have been pulled back from the VMware online repository, but might still be present locally if already downloaded.
Do install the VMSA-2018-0002 patches, though, since these mitigate ESXi itself from Spectre attacks.
If you have already installed the VMSA-2018-0004 patches or installed the CPU microcode patches using the server vendor’s BIOS/firmware upgrades AND your CPU is highlighted in yellow in the table in https://kb.vmware.com/s/article/52345, follow the instructions in that KB article to deactivate the mitigation features.
(A PowerCLI script to automate the deactivation of the mitigation features is being worked on by the always brilliant William Lam (@lamw): https://twitter.com/lamw/status/952546254172577792)
For physical servers, check with your server vendor for information regarding which BIOS/firmware/CPU/microcode patches are okay to install, and which ones need to be rolled back or deactivated for now.
This blog post covers the VMware vSphere aspect of the CPU security vulnerabilities known as Meltdown (aka CVE-2017-5754) and Spectre (aka CVE-2017-5753, CVE-2017-5715).
The vulnerabilities allow an attacker to read memory areas from other processes or even virtual machines running on the same physical CPU. This is a serious issue, since it can be used to retrieve not only sensitive data, but also passwords, encryption keys and other things that can be used in further attacks. (Stay tuned for upcoming blog post on how to protect against this by using physical separation)
VMware has issued a patch in VMSA-2018-0002 which
protects against mitigates Spectre attacks against ESXi 6.5 and 6.0.
[Update]: This patch is included in / superseded by VMSA-2018-0004 (see below), so there is no need to install it separately.
ESXi is not vulnerable to Meltdown, since it doesn’t allow untrusted user mode code to run, so currently no such patch is needed.
This post will be updated with information as it’s made available from VMware.
[Update 2018-01-09]: VMware has now released the second wave of patches for vSphere (ESXi + vCenter Server), named VMSA-2018-0004.
These patches, called Hypervisor-Assisted Guest Mitigation for branch target injection allow for the guest operating systems in the virtual machines to mitigate the ‘Branch target injection’ vulnerability in CVE-2017-5715. They also include the previous Hypervisor-Specific Remediation patches from VMSA-2018-0002.
There are several steps necessary to deploy and activate these patches:
- Upgrade vCenter Server(s) to the latest version – This patches the vCenter appliance itself, as well as enables the new EVC modes (mentioned further down).
- Install the new ESXi patches using VUM as usual – This will also upgrade the physical server’s CPU microcode to enable the guest mitigation, unless this has already been done using the server vendor’s BIOS/firmware upgrade or similar mechanism. Note that ESXi 6.0 and 6.5 have two separate patches each, one for the hypervisor and one for the microcode. ESXi 5.5. has both in one patch.
The steps above can be performed without any disruption to the VMs. However, as usual we recommend that the patches are initially tested on a limited number of ESXi hosts per cluster, to make sure they all come back up and don’t cause any problems. Keep an eye on your logs using Log Insight, and look for discrepancies in log volume and/or error messages (there is a feature in Log Insight for displaying differences in logs between time periods)
Now comes the slightly more time consuming part, since it will require rebooting each VM at least once:
- Install the applicable security patches for your guest operating system. A good list is available (as mentioned above) here. Try to sync the reboot (if one is necessary) with step 3 below.
- Make sure all your VMs are running on virtual hardware version 9 or newer. If you need to upgrading the virtual hardware, aim for version 11 or higher, since it includes new CPU features that can reduce the performance impact of these patches.
- Power cycle the VM (cold boot).
The steps above are described in more detail at https://kb.vmware.com/s/article/52085 ,
which also has information on how to check in the VMs’ vmware.log files that the steps above have been completely successful.
[Update]: William Lam has written an excellent script that checks that the above steps have all been completed successfully on VMs/clusters/environments. It checks both the vHW version and the presence of the new CPU features that the CPU microcode update and new EVC mode enables. You can find it at https://www.virtuallyghetto.com/2018/01/verify-hypervisor-assisted-guest-mitigation-spectre-patches-using-powercli.html
You can also check the status of your patched hosts using the excellent script at https://virtualcornerstone.com/2018/01/08/validating-compliance-of-vmsa-2018-0002-and-bios-update/ and you can check only your Windows VMs using the script at https://www.powershellgallery.com/packages/SpeculationControl/1.0.3
Make sure you also patch any remaining VMware software/appliances, according to https://kb.vmware.com/s/article/52264
Finally, once you’ve patched your entire environment, I’m sorry to tell you that this is probably not the last we’ve heard about Spectre-related vulnerabilities and attacks. It might unfortunately haunt us for quite some time, hence the name.
[Previous update 2018-01-08]:
The VMware KB article https://kb.vmware.com/s/article/52245 gives us some additional information. In summary:
- There is no expected performance degradation from applying the VMware Hypervisor-specific mitigation patches (VMSA-2018-0002) or the VMware Hypervisor-assisted Guest OS mitigation patches (more info on these below)
- The VMware Hypervisor-assisted Guest OS patches will patch the physical CPU microcode, unless a server vendor firmware update has already done so. These microcode patches are not expected to cause any performance degradation either.
- VMware appliances (such as vCenter Server, NSX Manager, vROps Manager etc) might be susceptible to the vulnerabilities. There is a separate KB article listing which are affected or not at https://kb.vmware.com/s/article/52264