Google Project Zero (Spectre and Meltdown) patch and flash info for VMware vSphere (ESXi/VCSA) on Intel Xeon D Supermicro SuperServers

Posted by Paul Braren on Jan 9 2018 (updated on Jan 14 2018) in
  • CPU
  • HomeLab
  • HomeServer
  • ESXi
  • VMware
  • This story continues to develop. Please expect that the information in the article below may be updated at any time, especially these first few days. I suggest that you to revisit and refresh. For example, it appears that No image profile found with name 'ESXi-6.5.0-20180104001-standard' has been pulled, see updates here for details.

    Disclosure:
    I'm a vSAN Systems Engineer (I started at VMware one year ago today!), but this article isn't official VMware documentation. Please refer to vmware.com, and to each of the vendor's sites listed here, for their latest official information on this issue.

    Disclaimer:
    It's your responsibility to back up first, and to proceed with updates at your own risk, as stated in the detailed disclaimer below every TinkerTry article.


    SUMMARY

    meltdown-and-spectre-info

    The net of my detailed article:

    is that most of the vendors states that this potential issue hasn't been exploited in the wild yet, as far as we know. Snooping like this isn't the sort of thing that gets logged though. The other take-home is that patching is a multi-step process. I sifted through these sites to come up with an action plan for my two Xeon D Bundles in my home lab, where a thorough remediation requires me to patch my:

    1. VCSA
      VMware vCenter Server 6.5 Update 1e
      If you use VCSA 6.5.x, you should update it first, since it's recommended to upgrade your VCSA before your ESXi hosts. This VCSA includes patches for CVE-2017-5715, see
      VMware vCenter Server 6.5 Update 1e Release Notes.
    2. VMware ESXi hypervisor(s)
      VMware ESXi 6.5, Patch Release ESXi-6.5.0-20180104001-standard (52200)
      6.5.0 Update 1 (Build 7526125)
      ESXi release notes in KB52236 and KB52200.
    3. CPU microcode (Flash BIOS)
      BIOS 1.3 (coming soon).
    4. VMs
      Automatic or manual updates.

    As for item 4 above, the integrated update mechanisms in most OSs have already handled this automatically, such as Microsoft Jan 3 2018 KB4056892 for Windows 10 version 1709 that hasn't been perfect, see also Microsoft yanks buggy Windows Meltdown/Spectre patches for AMD computers posted just yesterday at Computerworld.

    In a home lab environment, it's harder to see how this patching exercise is nearly as urgent as it would be for IT Professionals scrambling to patch their production environments. That said, it sure could be a good "rehearsal" for IT Pros who need to do this at work! It seems the urgency is highest for cloud providers, where any known (albeit theoretical) data leakage possibility must be eliminated as quickly and carefully as possible.

    Unfortunately, unless you have an HA cluster where only VM reboots affect availability, you'll be having considerable down time here, as reboots of the server itself are required for the other 2 remediation steps.

    As for when BIOS upgrades will be available industry-wide, here's what Intel says earlier today:

    In early December we began distributing Intel firmware updates to our OEM partners. For Intel CPUs introduced in the past five years, we expect to issue updates for more than 90 percent of them within a week, and the remainder by the end of January.

    explored by me in much further detail here. Those are some pretty bold time frame estimates, but it is reassuring that the last time a BIOS patches by OEMs was needed urgently, the OEMs stepped up about as fast as promised, see also Many Intel CPUs contain Management Engine vulnerability to remote execution, here's sample outputs of the new INTEL-SA-00086 Detection Tool, which didn't affect Xeon D, but did affect my Dell Precision 5520 Core i7.


    XEON D REMEDIATION INSTRUCTIONS

    Step 1) PATCH your VCSA Appliance

    • If you have VCSA 6.x, you should update that to 6.5U1e too, see VCSA release notes here, then follow along with the easy VCSA update method I documented step-by-step here.

    Step 2) PATCH your ESXi 6.5 Hypervisor(s)

    • After backing up your ESXi itself, use VUM (VMware Update Manager), or follow along with the ESXCLI method detailed in my Update 6.5 Update 1 Patch 02 article, but using this revised command below, but be sure to read the whole article to guide you on how you re-add (optional) Intel X557 10GbE support right after this update:
      esxcli software profile install -p ESXi-6.5.0-20180104001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

      This installs ESXi650-201801402-BG, aka VMware ESXi 6.5, Patch Release ESXi650-201801001 (52236), and requires a reboot.

      How to easily update your VMware Hypervisor from 6.5.x to 6.5 Update 1 Build 7526125

    Step 3) FLASH your Xeon D BIOS

    950541075185315840
    • Unfortunately, we're just not there yet for Supermicro Xeon D, I will update this article when BIOS 1.3 becomes available. For now, 1.2c is the latest release for the popular X10SDV in the Mini-ITX form-factor. Details (with video) of the Supermicro BIOS update procedure are available:
    • Supermicro Xeon D SuperServer BIOS 1.2c / IPMI 3.58 released
      Oct 21 2017 at TinkerTry
    • Alternatively, you may prefer Michael White's alternative method detailed here.

    Step 4) UPDATE your VMs


    Jan 10 2018 Update

    The article above was updated to make it clearer that VCSA should be upgraded first, and that it includes patches for Spectre/Meltdown.

    References for AMD's description of the issue were added, and all references were alphabetized in the original BACKSTORY/CHRONOLOGY section above that has now been moved off to the new article

    The reference links were also moved off the the new article:

    meltdown-and-spectre-info

    Jan 11 2018 Update

    951415426696675328

    Today, I received this heads-up tweet, then I noticed that my article didn't actually refer to Bulletin ID
    ESXi650-201801402-BG
    anywhere, which could make it a lot harder for folks to find this article when doing searches. This is now fixed.

    52236-cropped-faded
    • VMware ESXi 6.5, Patch Release ESXi650-201801001 (52236)

      Document Id
      52236
      Details
      Release Date: January 9, 2018

      NOTE: This ESXi patch provides part of the hypervisor-assisted guest remediation of CVE-2017-5715 for guest operating systems. For important details on this remediation, see VMware Security Advisory VMSA-2018-0004.

      Download Filename:

      ESXi650-201801001.zip

      Build:

      7526125

      ...

      Bulletins

      Bulletin ID Category Severity Knowledge Base Article
      ESXi650-201801402-BG
      Bugfix
      Important
      52198
      ESXi650-201801401-BG Bugfix Important 52199

    Now let's take a closer look at what's going on here. It appears that Get-SpeculationControlSettings seems works in a Windows VM! See screenshots below.

    I plan to test and confirm all this later on today, in my 2 Xeon D lab. I intentionally left one host patched to ESXi650-201801402-BG already (as shown in my video), and the other has not been patched.

    Thank you, Faranoosh!

    Read all about Microsoft's Get-SpeculationControlSettings here:

    understanding-the-output-of-get-speculationcontrolsettings-powershell-cropped
    • Understanding Get-SpeculationControlSettings PowerShell script output
      Last Updated Jan 10 2018

      Summary
      To help customers verify that protections have been enabled, Microsoft has published a PowerShell script that customers can run on their systems. This topic explains how to run the script and what the output means.

      More information
      Install and run the script by running the following commands:
      ...
      The output of this PowerShell script will resemble the following. Enabled protections appear in the output as “True.”

      PS C:\> Get-SpeculationControlSettings
      Speculation control settings for CVE-2017-5715 [branch target injection]
      Hardware support for branch target injection mitigation is present: False
      Windows OS support for branch target injection mitigation is present: True
      Windows OS support for branch target injection mitigation is enabled: False
      Windows OS support for branch target injection mitigation is disabled by system policy: True
      Windows OS support for branch target injection mitigation is disabled by absence of hardware support: True
      Speculation control settings for CVE-2017-5754 [rogue data cache load]
      Hardware requires kernel VA shadowing: True
      Windows OS support for kernel VA shadow is present: False
      Windows OS support for kernel VA shadow is enabled: False
      Windows OS support for PCID optimization is enabled: False
      BTIHardwarePresent : False
      BTIWindowsSupportPresent : True
      BTIWindowsSupportEnabled : False
      BTIDisabledBySystemPolicy : True
      BTIDisabledByNoHardwareSupport : True
      KVAShadowRequired : True
      KVAShadowWindowsSupportPresent : False
      KVAShadowWindowsSupportEnabled : False
      KVAShadowPcidEnabled : False
      ...

    Let's have a closer look at @cdninmel's screenshots from his Xeon D tweet he's referring us to:

    cdninmel-pic2
    BEFORE ESXi650-201801402-BG - See Faranoosh @cdninmel using the PowerShell script Get-SpeculationControlSettings in a VM on Xeon D
    cdninmel-pic1
    AFTER ESXi650-201801402-BG - See Faranoosh @cdninmel using the PowerShell script Get-SpeculationControlSettings in a VM on Xeon D

    Jan 11 2018 Update

    Added William's new article to step 4 above.


    Jan 13 2018 Update

    952100908594216960
    952100908594216960-pic
    Faranoosh @cdninmel testing William Lam's script on Xeon D.

    Jan 14 2018 Update

    Update!
    It currently appears this patch has been pulled off of the VMware update servers, see details at TinkerTry here.

    This also means that folks trying the update featured above will now find that the update command fails:

    [root@xd-1541-5028d:~] esxcli software profile install -p ESXi-6.5.0-20180104001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
     [NoMatchError]
     No image profile found with name 'ESXi-6.5.0-20180104001-standard'
             id = ESXi-6.5.0-20180104001-standard
     Please refer to the log file for more details.
    [root@xd-1541-5028d:~] [NoMatchError]
    -sh: [NoMatchError]: not found
    [root@xd-1541-5028d:~]  No image profile found with name 'ESXi-6.5.0-20180104001-standard'

    We will have to wait for more information from VMware, to see what happens next. If you're not in production with your Xeon D and it's just a home lab, it would seem best to simply hold-off for a while, until things settle down, ideally waiting until the new BIOS 1.3 comes out too. At that point, I'm hoping to produce a video that shows me demonstrating the remediation steps, and the verification script in a VM, in the form of updates I'll be adding below this article!

    See also:


    See also at TinkerTry

    metool
    fix-xeon-d-inaccurate-cim-data-default-in-vsphere65

    See also

    spectreattack

    spectre-vulnerability-patch-vmware-esxi

    meltdown-spectre-microsoft-products

    verify-hypervisor-assisted-guest-mitigation-spectre-patches-using-powercli

    • Recent, related William Lam tweets @lamw:

      Jan 04 2018

      948926455760674821

      Jan 06 2018

      949662333038559232

      Jan 09 2018

      950810761818816512

    Update log

    As seen in my video, here's the full contents of my ssh session, as I completed my Xeon D-1541 upgrade from
    Version: 6.5.0 Update 1 (Build 7388607)
    to:
    Version: 6.5.0 Update 1 (Build 7526125)

    login as: root
    Using keyboard-interactive authentication.
    Password:
    The time and date of this login have been sent to the system logs.
    
    WARNING:
       All commands run on the ESXi shell are logged and may be included in
       support bundles. Do not provide passwords directly on the command line.
       Most tools can prompt for secrets or accept them from standard input.
    
    VMware offers supported, powerful system administration tools.  Please
    see www.vmware.com/go/sysadmintools for details.
    
    The ESXi Shell can be disabled by an administrative user. See the
    vSphere Security documentation for more information.
    [root@xd-1541-5028d:~] esxcli software profile install -p ESXi-6.5.0-20180104001-standard -d https://hostupdate.vmware.com/software/VUM/PR
    ODUCTION/main/vmw-depot-index.xml
     [Exception]
     You attempted to install an image profile which would have resulted in the removal of VIBs ['INT_bootbank_intel-nvme_1.2.1.15-1OEM.650.0.                                                   0.4598673']. If this is not what you intended, you may use the esxcli software profile update command to preserve the VIBs above. If this                                                    is what you intended, please use the --ok-to-remove option to explicitly allow the removal.
     Please refer to the log file for more details.
    [root@xd-1541-5028d:~] esxcli software profile install -p ESXi-6.5.0-20180104001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml --ok-to-remove
    Installation Result
       Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
       Reboot Required: true
       VIBs Installed: VMW_bootbank_net-igb_5.0.5.1.1-5vmw.650.0.0.4564106, VMW_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.650.0.0.4564106, VMware_bootbank_cpu-microcode_6.5.0-1.38.7526125, VMware_bootbank_esx-base_6.5.0-1.38.7526125, VMware_bootbank_esx-tboot_6.5.0-1.38.7526125, VMware_bootbank_vsan_6.5.0-1.38.7395176, VMware_bootbank_vsanhealth_6.5.0-1.38.7395177
       VIBs Removed: INT_bootbank_intel-nvme_1.2.1.15-1OEM.650.0.0.4598673, INT_bootbank_net-igb_5.3.3-1OEM.600.0.0.2494585, INT_bootbank_net-ixgbe_4.5.3-1OEM.600.0.0.2494585, VMware_bootbank_cpu-microcode_6.5.0-0.0.4564106, VMware_bootbank_esx-base_6.5.0-1.36.7388607, VMware_bootbank_esx-tboot_6.5.0-1.36.7388607, VMware_bootbank_vsan_6.5.0-1.36.7388608, VMware_bootbank_vsanhealth_6.5.0-1.36.7388609
       VIBs Skipped: VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.650.0.0.4564106, VMW_bootbank_ata-pata-amd_0.3.10-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.650.0.0.4564106, VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-via_0.3.3-2vmw.650.0.0.4564106, VMW_bootbank_block-cciss_3.6.14-10vmw.650.0.0.4564106, VMW_bootbank_char-random_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.650.0.14.5146846, VMW_bootbank_elxnet_11.1.91.0-1vmw.650.0.0.4564106, VMW_bootbank_hid-hid_1.0-3vmw.650.0.0.4564106, VMW_bootbank_i40en_1.3.1-5vmw.650.1.26.5969303, VMW_bootbank_igbn_0.1.0.0-15vmw.650.1.36.7388607, VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-devintf_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ixgben_1.4.1-2vmw.650.1.26.5969303, VMW_bootbank_lpfc_11.1.0.6-1vmw.650.0.0.4564106, VMW_bootbank_lsi-mr3_6.910.18.00-1vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt2_20.00.01.00-3vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt3_12.00.02.00-11vmw.650.0.0.4564106, VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.650.0.0.4564106, VMW_bootbank_misc-drivers_6.5.0-1.36.7388607, VMW_bootbank_mtip32xx-native_3.9.5-1vmw.650.0.0.4564106, VMW_bootbank_ne1000_0.8.0-16vmw.650.1.26.5969303, VMW_bootbank_nenic_1.0.0.2-1vmw.650.0.0.4564106, VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.650.0.0.4564106, VMW_bootbank_net-bnx2x_1.78.80.v60.12-1vmw.650.0.0.4564106, VMW_bootbank_net-cdc-ether_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.650.0.0.4564106, VMW_bootbank_net-e1000_8.0.3.1-5vmw.650.0.0.4564106, VMW_bootbank_net-e1000e_3.2.2.1-2vmw.650.0.0.4564106, VMW_bootbank_net-enic_2.1.2.38-2vmw.650.0.0.4564106, VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.650.0.0.4564106, VMW_bootbank_net-forcedeth_0.61-2vmw.650.0.0.4564106, VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-nx-nic_5.0.621-5vmw.650.0.0.4564106, VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.650.0.0.4564106, VMW_bootbank_net-usbnet_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.650.0.0.4564106, VMW_bootbank_nhpsa_2.0.6-3vmw.650.0.0.4564106, VMW_bootbank_nmlx4-core_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-en_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-rdma_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx5-core_4.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_ntg3_4.1.3.0-1vmw.650.1.36.7388607, VMW_bootbank_nvme_1.2.0.32-5vmw.650.1.36.7388607, VMW_bootbank_nvmxnet3_2.0.0.23-1vmw.650.1.36.7388607, VMW_bootbank_ohci-usb-ohci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_pvscsi_0.1-1vmw.650.1.26.5969303, VMW_bootbank_qedentv_2.0.3.29-1vmw.650.0.0.4564106, VMW_bootbank_qfle3_1.0.2.7-1vmw.650.0.0.4564106, VMW_bootbank_qflge_1.1.0.3-1vmw.650.0.0.4564106, VMW_bootbank_qlnativefc_2.1.50.0-1vmw.650.1.26.5969303, VMW_bootbank_sata-ahci_3.0-26vmw.650.1.26.5969303, VMW_bootbank_sata-ata-piix_2.12-10vmw.650.0.0.4564106, VMW_bootbank_sata-sata-nv_3.5-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-promise_2.12-3vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil24_1.1-1vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil_2.3-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-svw_2.3-3vmw.650.0.0.4564106, VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.650.0.0.4564106, VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.650.0.0.4564106, VMW_bootbank_scsi-aic79xx_3.1-5vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.650.0.0.4564106, VMW_bootbank_scsi-hpsa_6.0.0.84-1vmw.650.0.0.4564106, VMW_bootbank_scsi-ips_7.12.05-4vmw.650.0.0.4564106, VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.650.0.0.4564106, VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.650.0.0.4564106, VMW_bootbank_scsi-mpt2sas_19.00.00.00-1vmw.650.0.0.4564106, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.650.0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-3-0_6.5.0-0.0.4564106, VMW_bootbank_uhci-usb-uhci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usbcore-usb_1.0-3vmw.650.1.26.5969303, VMW_bootbank_vmkata_0.1-1vmw.650.1.36.7388607, VMW_bootbank_vmkplexer-vmkplexer_6.5.0-0.0.4564106, VMW_bootbank_vmkusb_0.1-1vmw.650.1.36.7388607, VMW_bootbank_vmw-ahci_1.0.0-39vmw.650.1.26.5969303, VMW_bootbank_xhci-xhci_1.0-3vmw.650.0.0.4564106, VMware_bootbank_emulex-esx-elxnetcli_11.1.28.0-0.0.4564106, VMware_bootbank_esx-dvfilter-generic-fastpath_6.5.0-1.36.7388607, VMware_bootbank_esx-ui_1.23.0-6506686, VMware_bootbank_esx-xserver_6.5.0-0.23.5969300, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-5vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-10vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-7vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-8vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-6vmw.650.1.26.5969303, VMware_bootbank_native-misc-drivers_6.5.0-0.0.4564106, VMware_bootbank_rste_2.0.2.0088-4vmw.650.0.0.4564106, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.10-1.26.5969303, VMware_locker_tools-light_6.5.0-1.33.7273056
    [root@xd-1541-5028d:~] reboot