Google Project Zero (Spectre and Meltdown) patch and flash info for VMware vSphere (ESXi/VCSA) on Intel Xeon D Supermicro SuperServers
This story continues to develop. Please expect that the information in the article below may be updated at any time, especially these first few days. I suggest that you to revisit and refresh. For example, it appears that No image profile found with name 'ESXi-6.5.0-20180104001-standard' has been pulled, see updates here for details.
Disclosure:
I'm a vSAN Systems Engineer (I started at VMware one year ago today!), but this article isn't official VMware documentation. Please refer to vmware.com, and to each of the vendor's sites listed here, for their latest official information on this issue.
Disclaimer:
It's your responsibility to back up first, and to proceed with updates at your own risk, as stated in the detailed disclaimer below every TinkerTry article.
SUMMARY
The net of my detailed article:
is that most of the vendors states that this potential issue hasn't been exploited in the wild yet, as far as we know. Snooping like this isn't the sort of thing that gets logged though. The other take-home is that patching is a multi-step process. I sifted through these sites to come up with an action plan for my two Xeon D Bundles in my home lab, where a thorough remediation requires me to patch my:
- VCSA
VMware vCenter Server 6.5 Update 1e
If you use VCSA 6.5.x, you should update it first, since it's recommended to upgrade your VCSA before your ESXi hosts. This VCSA includes patches for CVE-2017-5715, see
VMware vCenter Server 6.5 Update 1e Release Notes. - VMware ESXi hypervisor(s)
VMware ESXi 6.5, Patch Release ESXi-6.5.0-20180104001-standard (52200)
6.5.0 Update 1 (Build 7526125)
ESXi release notes in KB52236 and KB52200. - CPU microcode (Flash BIOS)
BIOS 1.3 (coming soon). - VMs
Automatic or manual updates.
As for item 4 above, the integrated update mechanisms in most OSs have already handled this automatically, such as Microsoft Jan 3 2018 KB4056892 for Windows 10 version 1709 that hasn't been perfect, see also Microsoft yanks buggy Windows Meltdown/Spectre patches for AMD computers posted just yesterday at Computerworld.
In a home lab environment, it's harder to see how this patching exercise is nearly as urgent as it would be for IT Professionals scrambling to patch their production environments. That said, it sure could be a good "rehearsal" for IT Pros who need to do this at work! It seems the urgency is highest for cloud providers, where any known (albeit theoretical) data leakage possibility must be eliminated as quickly and carefully as possible.
Unfortunately, unless you have an HA cluster where only VM reboots affect availability, you'll be having considerable down time here, as reboots of the server itself are required for the other 2 remediation steps.
As for when BIOS upgrades will be available industry-wide, here's what Intel says earlier today:
In early December we began distributing Intel firmware updates to our OEM partners. For Intel CPUs introduced in the past five years, we expect to issue updates for more than 90 percent of them within a week, and the remainder by the end of January.
explored by me in much further detail here. Those are some pretty bold time frame estimates, but it is reassuring that the last time a BIOS patches by OEMs was needed urgently, the OEMs stepped up about as fast as promised, see also Many Intel CPUs contain Management Engine vulnerability to remote execution, here's sample outputs of the new INTEL-SA-00086 Detection Tool, which didn't affect Xeon D, but did affect my Dell Precision 5520 Core i7.
XEON D REMEDIATION INSTRUCTIONS
Step 1) PATCH your VCSA Appliance
- If you have VCSA 6.x, you should update that to 6.5U1e too, see VCSA release notes here, then follow along with the easy VCSA update method I documented step-by-step here.
Step 2) PATCH your ESXi 6.5 Hypervisor(s)
- After backing up your ESXi itself, use VUM (VMware Update Manager), or follow along with the ESXCLI method detailed in my Update 6.5 Update 1 Patch 02 article, but using this revised command below, but be sure to read the whole article to guide you on how you re-add (optional) Intel X557 10GbE support right after this update:
esxcli software profile install -p ESXi-6.5.0-20180104001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xmlThis installs
ESXi650-201801402-BG, aka VMware ESXi 6.5, Patch Release ESXi650-201801001 (52236), and requires a reboot.How to easily update your VMware Hypervisor from 6.5.x to 6.5 Update 1 Build 7526125
Step 3) FLASH your Xeon D BIOS
- Unfortunately, we're just not there yet for Supermicro Xeon D, I will update this article when BIOS 1.3 becomes available. For now, 1.2c is the latest release for the popular X10SDV in the Mini-ITX form-factor. Details (with video) of the Supermicro BIOS update procedure are available:
- Supermicro Xeon D SuperServer BIOS 1.2c / IPMI 3.58 released
Oct 21 2017 at TinkerTry - Alternatively, you may prefer Michael White's alternative method detailed here.
- Supermicro has pulled together all BIOS updates on the "X10 Generation Listing" tab of their detailed information page:
Step 4) UPDATE your VMs
- Fire up those dormant VMs, check to be sure automatic updates are turned on, then manually kick off an update check and reboot when prompted. VMware has step-by-step details here, or use William Lam's more automated approach, see Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI.
- "Ensure that your VMs are using Virtual Hardware Version 9 or higher" (VMware KB 52085).
- You may wish to verify the steps actually worked, see SpeculationControlSettings details below.
- You may also wish to verify with @lamw's PowerShell script
VerifyESXiMicrocodePatch.ps1from Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI.
Jan 10 2018 Update
The article above was updated to make it clearer that VCSA should be upgraded first, and that it includes patches for Spectre/Meltdown.
References for AMD's description of the issue were added, and all references were alphabetized in the original BACKSTORY/CHRONOLOGY section above that has now been moved off to the new article
The reference links were also moved off the the new article:
- Meltdown and Spectre side channel attack mitigation information from CPU, Server, and Software vendors
Jan 10 2018
Jan 11 2018 Update
Today, I received this heads-up tweet, then I noticed that my article didn't actually refer to Bulletin ID
ESXi650-201801402-BG
anywhere, which could make it a lot harder for folks to find this article when doing searches. This is now fixed.
- VMware ESXi 6.5, Patch Release ESXi650-201801001 (52236)
Document Id
52236
Details
Release Date: January 9, 2018NOTE: This ESXi patch provides part of the hypervisor-assisted guest remediation of CVE-2017-5715 for guest operating systems. For important details on this remediation, see VMware Security Advisory VMSA-2018-0004.
Download Filename:
ESXi650-201801001.zip
Build:
7526125
Bulletins
Bulletin ID Category Severity Knowledge Base Article
ESXi650-201801402-BG
Bugfix
Important
52198
ESXi650-201801401-BG Bugfix Important 52199
Now let's take a closer look at what's going on here. It appears that Get-SpeculationControlSettings seems works in a Windows VM! See screenshots below.
I plan to test and confirm all this later on today, in my 2 Xeon D lab. I intentionally left one host patched to ESXi650-201801402-BG already (as shown in my video), and the other has not been patched.
Thank you, Faranoosh!
Read all about Microsoft's Get-SpeculationControlSettings here:
- Understanding Get-SpeculationControlSettings PowerShell script output
Last Updated Jan 10 2018Summary
To help customers verify that protections have been enabled, Microsoft has published a PowerShell script that customers can run on their systems. This topic explains how to run the script and what the output means.More information
Install and run the script by running the following commands:
...
The output of this PowerShell script will resemble the following. Enabled protections appear in the output as “True.”PS C:\> Get-SpeculationControlSettings
Speculation control settings for CVE-2017-5715 [branch target injection]
Hardware support for branch target injection mitigation is present: False
Windows OS support for branch target injection mitigation is present: True
Windows OS support for branch target injection mitigation is enabled: False
Windows OS support for branch target injection mitigation is disabled by system policy: True
Windows OS support for branch target injection mitigation is disabled by absence of hardware support: True
Speculation control settings for CVE-2017-5754 [rogue data cache load]
Hardware requires kernel VA shadowing: True
Windows OS support for kernel VA shadow is present: False
Windows OS support for kernel VA shadow is enabled: False
Windows OS support for PCID optimization is enabled: False
BTIHardwarePresent : False
BTIWindowsSupportPresent : True
BTIWindowsSupportEnabled : False
BTIDisabledBySystemPolicy : True
BTIDisabledByNoHardwareSupport : True
KVAShadowRequired : True
KVAShadowWindowsSupportPresent : False
KVAShadowWindowsSupportEnabled : False
KVAShadowPcidEnabled : False
...
Let's have a closer look at @cdninmel's screenshots from his Xeon D tweet he's referring us to:
Jan 11 2018 Update
Added William's new article to step 4 above.
Jan 13 2018 Update
Jan 14 2018 Update
Update!
It currently appears this patch has been pulled off of the VMware update servers, see details at TinkerTry here.
This also means that folks trying the update featured above will now find that the update command fails:
[root@xd-1541-5028d:~] esxcli software profile install -p ESXi-6.5.0-20180104001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
[NoMatchError]
No image profile found with name 'ESXi-6.5.0-20180104001-standard'
id = ESXi-6.5.0-20180104001-standard
Please refer to the log file for more details.
[root@xd-1541-5028d:~] [NoMatchError]
-sh: [NoMatchError]: not found
[root@xd-1541-5028d:~] No image profile found with name 'ESXi-6.5.0-20180104001-standard'We will have to wait for more information from VMware, to see what happens next. If you're not in production with your Xeon D and it's just a home lab, it would seem best to simply hold-off for a while, until things settle down, ideally waiting until the new BIOS 1.3 comes out too. At that point, I'm hoping to produce a video that shows me demonstrating the remediation steps, and the verification script in a VM, in the form of updates I'll be adding below this article!
See also:
- Automating Intel Sighting remediation using PowerCLI (SSH not required)
Jan 14 2018 by William Lam at virtuallGhetto
See also at TinkerTry
See also
- Meltdown and Spectre
Vulnerabilities in modern computers leak passwords and sensitive data.
Jan 03 2018
- Spectre Vulnerability – How to Patch VMware ESXi
Jan 05 2018 by VIRTUADMIN at VIRTUABYTES
- Meltdown and Spectre: Microsoft products
Jan 07 2018 by Andrea Mauro at vInfrastructure Blog
- Verify Hypervisor-Assisted Guest Mitigation (Spectre) patches using PowerCLI
Jan 11 2018 by William Lam at virtuallyGhetto...
The script will ensure VMs are at least vHW9+ (as older vHW are not applicable) and that one of the three new CPU features are available as outlined by the KB.
...
- Recent, related William Lam tweets @lamw:
Jan 04 2018
Jan 06 2018
Jan 09 2018
Update log
As seen in my video, here's the full contents of my ssh session, as I completed my Xeon D-1541 upgrade from
Version: 6.5.0 Update 1 (Build 7388607)
to:
Version: 6.5.0 Update 1 (Build 7526125)
login as: root
Using keyboard-interactive authentication.
Password:
The time and date of this login have been sent to the system logs.
WARNING:
All commands run on the ESXi shell are logged and may be included in
support bundles. Do not provide passwords directly on the command line.
Most tools can prompt for secrets or accept them from standard input.
VMware offers supported, powerful system administration tools. Please
see www.vmware.com/go/sysadmintools for details.
The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
[root@xd-1541-5028d:~] esxcli software profile install -p ESXi-6.5.0-20180104001-standard -d https://hostupdate.vmware.com/software/VUM/PR
ODUCTION/main/vmw-depot-index.xml
[Exception]
You attempted to install an image profile which would have resulted in the removal of VIBs ['INT_bootbank_intel-nvme_1.2.1.15-1OEM.650.0. 0.4598673']. If this is not what you intended, you may use the esxcli software profile update command to preserve the VIBs above. If this is what you intended, please use the --ok-to-remove option to explicitly allow the removal.
Please refer to the log file for more details.
[root@xd-1541-5028d:~] esxcli software profile install -p ESXi-6.5.0-20180104001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml --ok-to-remove
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed: VMW_bootbank_net-igb_5.0.5.1.1-5vmw.650.0.0.4564106, VMW_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.650.0.0.4564106, VMware_bootbank_cpu-microcode_6.5.0-1.38.7526125, VMware_bootbank_esx-base_6.5.0-1.38.7526125, VMware_bootbank_esx-tboot_6.5.0-1.38.7526125, VMware_bootbank_vsan_6.5.0-1.38.7395176, VMware_bootbank_vsanhealth_6.5.0-1.38.7395177
VIBs Removed: INT_bootbank_intel-nvme_1.2.1.15-1OEM.650.0.0.4598673, INT_bootbank_net-igb_5.3.3-1OEM.600.0.0.2494585, INT_bootbank_net-ixgbe_4.5.3-1OEM.600.0.0.2494585, VMware_bootbank_cpu-microcode_6.5.0-0.0.4564106, VMware_bootbank_esx-base_6.5.0-1.36.7388607, VMware_bootbank_esx-tboot_6.5.0-1.36.7388607, VMware_bootbank_vsan_6.5.0-1.36.7388608, VMware_bootbank_vsanhealth_6.5.0-1.36.7388609
VIBs Skipped: VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.650.0.0.4564106, VMW_bootbank_ata-pata-amd_0.3.10-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.650.0.0.4564106, VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-via_0.3.3-2vmw.650.0.0.4564106, VMW_bootbank_block-cciss_3.6.14-10vmw.650.0.0.4564106, VMW_bootbank_char-random_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.650.0.14.5146846, VMW_bootbank_elxnet_11.1.91.0-1vmw.650.0.0.4564106, VMW_bootbank_hid-hid_1.0-3vmw.650.0.0.4564106, VMW_bootbank_i40en_1.3.1-5vmw.650.1.26.5969303, VMW_bootbank_igbn_0.1.0.0-15vmw.650.1.36.7388607, VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-devintf_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ixgben_1.4.1-2vmw.650.1.26.5969303, VMW_bootbank_lpfc_11.1.0.6-1vmw.650.0.0.4564106, VMW_bootbank_lsi-mr3_6.910.18.00-1vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt2_20.00.01.00-3vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt3_12.00.02.00-11vmw.650.0.0.4564106, VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.650.0.0.4564106, VMW_bootbank_misc-drivers_6.5.0-1.36.7388607, VMW_bootbank_mtip32xx-native_3.9.5-1vmw.650.0.0.4564106, VMW_bootbank_ne1000_0.8.0-16vmw.650.1.26.5969303, VMW_bootbank_nenic_1.0.0.2-1vmw.650.0.0.4564106, VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.650.0.0.4564106, VMW_bootbank_net-bnx2x_1.78.80.v60.12-1vmw.650.0.0.4564106, VMW_bootbank_net-cdc-ether_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.650.0.0.4564106, VMW_bootbank_net-e1000_8.0.3.1-5vmw.650.0.0.4564106, VMW_bootbank_net-e1000e_3.2.2.1-2vmw.650.0.0.4564106, VMW_bootbank_net-enic_2.1.2.38-2vmw.650.0.0.4564106, VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.650.0.0.4564106, VMW_bootbank_net-forcedeth_0.61-2vmw.650.0.0.4564106, VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-nx-nic_5.0.621-5vmw.650.0.0.4564106, VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.650.0.0.4564106, VMW_bootbank_net-usbnet_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.650.0.0.4564106, VMW_bootbank_nhpsa_2.0.6-3vmw.650.0.0.4564106, VMW_bootbank_nmlx4-core_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-en_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-rdma_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx5-core_4.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_ntg3_4.1.3.0-1vmw.650.1.36.7388607, VMW_bootbank_nvme_1.2.0.32-5vmw.650.1.36.7388607, VMW_bootbank_nvmxnet3_2.0.0.23-1vmw.650.1.36.7388607, VMW_bootbank_ohci-usb-ohci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_pvscsi_0.1-1vmw.650.1.26.5969303, VMW_bootbank_qedentv_2.0.3.29-1vmw.650.0.0.4564106, VMW_bootbank_qfle3_1.0.2.7-1vmw.650.0.0.4564106, VMW_bootbank_qflge_1.1.0.3-1vmw.650.0.0.4564106, VMW_bootbank_qlnativefc_2.1.50.0-1vmw.650.1.26.5969303, VMW_bootbank_sata-ahci_3.0-26vmw.650.1.26.5969303, VMW_bootbank_sata-ata-piix_2.12-10vmw.650.0.0.4564106, VMW_bootbank_sata-sata-nv_3.5-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-promise_2.12-3vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil24_1.1-1vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil_2.3-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-svw_2.3-3vmw.650.0.0.4564106, VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.650.0.0.4564106, VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.650.0.0.4564106, VMW_bootbank_scsi-aic79xx_3.1-5vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.650.0.0.4564106, VMW_bootbank_scsi-hpsa_6.0.0.84-1vmw.650.0.0.4564106, VMW_bootbank_scsi-ips_7.12.05-4vmw.650.0.0.4564106, VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.650.0.0.4564106, VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.650.0.0.4564106, VMW_bootbank_scsi-mpt2sas_19.00.00.00-1vmw.650.0.0.4564106, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.650.0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-3-0_6.5.0-0.0.4564106, VMW_bootbank_uhci-usb-uhci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usbcore-usb_1.0-3vmw.650.1.26.5969303, VMW_bootbank_vmkata_0.1-1vmw.650.1.36.7388607, VMW_bootbank_vmkplexer-vmkplexer_6.5.0-0.0.4564106, VMW_bootbank_vmkusb_0.1-1vmw.650.1.36.7388607, VMW_bootbank_vmw-ahci_1.0.0-39vmw.650.1.26.5969303, VMW_bootbank_xhci-xhci_1.0-3vmw.650.0.0.4564106, VMware_bootbank_emulex-esx-elxnetcli_11.1.28.0-0.0.4564106, VMware_bootbank_esx-dvfilter-generic-fastpath_6.5.0-1.36.7388607, VMware_bootbank_esx-ui_1.23.0-6506686, VMware_bootbank_esx-xserver_6.5.0-0.23.5969300, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-5vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-10vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-7vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-8vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-6vmw.650.1.26.5969303, VMware_bootbank_native-misc-drivers_6.5.0-0.0.4564106, VMware_bootbank_rste_2.0.2.0088-4vmw.650.0.0.4564106, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.10-1.26.5969303, VMware_locker_tools-light_6.5.0-1.33.7273056
[root@xd-1541-5028d:~] reboot
