How to easily update your VMware Hypervisor from 6.x to 6.5 Update 1 Patch Release ESXi650-201801001 (ESXi Build 7388607) for Meltdown/Spectre remediation (patch has been pulled)

Posted by Paul Braren on Jan 13 2018 (updated on Mar 31 2018) in
  • ESXi
  • Virtualization
  • HowTo
  • HomeLab

    • *Jan 14 2018 Update - This patch has been pulled, details here.

    Important Update - On Mar 20 2018, VMware VMSA-2018-0004.3 announced that CVE-2017-5715 (Spectre-2) mitigation is now included in the latest patch that you should be using instead of the older patch featured in the original article below. You'll find the newer article that features an even easier update method here:

    Article below as it originally appeared.

    3262786803
    The comment above is still relevant, as I'm admittedly this is just one more-universal way to upgrade ESXi, avoiding the need to download the ISO separately. Booting from a new ISO has the advantage of checking for CPU compatibility before installing, the method below does not. All upgrades come with risks, including the possibility of losing your network connections. Proceed at your own risk, and always back up first.

    Meltdown and Spectre are looming large this year, and this article is in direct response to all that, see also at TinkerTry

    Warning!
    But don't rush things. Xeon D owners, and a few other Xeon E5 and E7 owners, will want to read this entire article before patching anything! There's a surprise twist to this story at the end!

    Update

    After backing up, and reading the below warnings and updates, here's the updated line to issue in an SSH session as root:

    esxcli software profile install -p ESXi-6.5.0-20180104001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

    That's the only line you need to change, otherwise just follow along with this detailed article, as-is:

    easy-upgrade-to-esxi-65u1-patch02

    If you've already performed this patch and rebooted, various UIs will show your ESXi version, depending upon where you look:

    • 6.5.0 Update 1 (Build 7526125)
    • Version: 6.5.0 Update 1 (Build 7526125)
    • (Updated) ESXi-6.5.0-20171204001-standard (VMware, Inc.)
    • VMware ESXi 6.5.0 build-7526125 | VMware ESXi 6.5.0 Update 1
      (output from ESXCLI vmware -vl as root`)
    • 6.5.0 #1 SMP Release build-7526125 Jan 7 2018 19:27:01 x86_64 x86_64 x86_64 ESXi
      (output from ESXCLI from uname -a as root)

    VMware ESXI 6.5 Build 7388607

    Read both of these KB articles for details on what this patch fixes:

    Warning!
    If you have an Intel Xeon D system, earlier today, VMware published a new article today on Jan 13 2018. You'll want to read this before you proceed, especially the sentenced at the end that I bolded:

    52345
    Xeon D is of particular interest to TinkerTry readers.
    • Intel Sightings in ESXi Bundled Microcode Patches for VMSA-2018-0004 (52345)

      Document Id
      52345
      Purpose
      Although VMware strongly recommends that customers obtain microcode patches through their hardware vendor, as an aid to customers, VMware also included the initial microcode patches in ESXi650-201801402-BG, ESXi600-201801402-BG, and ESXi550-201801401-BG. Intel has notified VMware of recent sightings that may affect some of the initial microcode patches that provide the speculative execution control mechanism for a number of Intel Haswell and Broadwell processors. The issue can occur when the speculative execution control is actually used within a virtual machine by a patched OS. At this point, it has been recommended that VMware remove exposure of the speculative-execution mechanism to virtual machines on ESXi hosts using the affected Intel processors until Intel provides new microcode at a later date.
      Resolution
      For ESXi hosts that have not yet applied one of the following patches ESXi650-201801402-BG, ESXi600-201801402-BG, or ESXi550-201801401-BG, VMware recommends not doing so at this time. It is recommended to apply the patches listed in VMSA-2018-0002 instead.
      ...

    Ok, there you go, saved you! Having a look at VMSA-2018-0002, it's telling you to go with ESXi650-201712101-SG, which is part of the ESXi-6.5.0-20171201001s-standard version my December article fully documents installing! That's right, for now, if you're not already on Build 7388607, you should get there, simply follow along with this recent TinkerTry article:

    easy-upgrade-to-esxi-65u1-patch02

    Jan 14 2018 Update

    Update!
    It currently appears this patch has been pulled off of the VMware update servers, see details at TinkerTry here.

    This also means that folks trying the ESXi-6.5.0-20180104001-standard update anyway will now find that it fails:

    [root@xd-1541-5028d:~] esxcli software profile install -p ESXi-6.5.0-20180104001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
 [NoMatchError]
 No image profile found with name 'ESXi-6.5.0-20180104001-standard'
         id = ESXi-6.5.0-20180104001-standard
 Please refer to the log file for more details.

    Here's a screenshot:

    no-image-profile-found-on-Jan-14-2018-at-0906-for-20180104001

    We will have to wait for more information from VMware, to see what happens next. If you're not in production with your Xeon D and it's just a home lab, it would seem best to simply hold-off for a while, until things settle down, ideally waiting until the new BIOS 1.3 comes out too. At that point, I'm hoping to produce a video that shows me demonstrating the remediation steps, and the verification script in a VM, in the form of updates I'll be adding below this article:

    google-project-zero-spectre-and-meltdown-patch-and-flash-for-vmware-esxi-on-supermicro-xeon-d

    Jan 16 2018 Update

    Changed title from:
    How to easily update your VMware Hypervisor from 6.x to 6.5 Update 1 Patch Release ESXi650-201801001 (ESXi Build 7388607) for Meltdown/Spectre remediation (unless you have certain CPUs including Xeon D)
    to:
    How to easily update your VMware Hypervisor from 6.x to 6.5 Update 1 Patch Release ESXi650-201801001 (ESXi Build 7388607) for Meltdown/Spectre remediation (patch has been pulled)

    See also at TinkerTry

    See also

    Upgrade Log

    Below, I've pasted the full text of my upgrade that I later found out I shouldn't have done. It will help you see what drivers are touched. Just use the horizontal scroll bar or shift + mousewheel to look around, and Ctrl+F to Find stuff quickly:

    As seen in my video, here's the full contents of my ssh session, as I completed my Xeon D-1541 upgrade from
    Version: 6.5.0 Update 1 (Build 7388607)
    to:
    Version: 6.5.0 Update 1 (Build 7526125)

    login as: root
Using keyboard-interactive authentication.
Password:
The time and date of this login have been sent to the system logs.

WARNING:
   All commands run on the ESXi shell are logged and may be included in
   support bundles. Do not provide passwords directly on the command line.
   Most tools can prompt for secrets or accept them from standard input.

VMware offers supported, powerful system administration tools.  Please
see www.vmware.com/go/sysadmintools for details.

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
[root@xd-1541-5028d:~] esxcli software profile install -p ESXi-6.5.0-20180104001-standard -d https://hostupdate.vmware.com/software/VUM/PR
ODUCTION/main/vmw-depot-index.xml
 [Exception]
 You attempted to install an image profile which would have resulted in the removal of VIBs ['INT_bootbank_intel-nvme_1.2.1.15-1OEM.650.0.                                                   0.4598673']. If this is not what you intended, you may use the esxcli software profile update command to preserve the VIBs above. If this                                                    is what you intended, please use the --ok-to-remove option to explicitly allow the removal.
 Please refer to the log file for more details.
[root@xd-1541-5028d:~] esxcli software profile install -p ESXi-6.5.0-20180104001-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml --ok-to-remove
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: VMW_bootbank_net-igb_5.0.5.1.1-5vmw.650.0.0.4564106, VMW_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.650.0.0.4564106, VMware_bootbank_cpu-microcode_6.5.0-1.38.7526125, VMware_bootbank_esx-base_6.5.0-1.38.7526125, VMware_bootbank_esx-tboot_6.5.0-1.38.7526125, VMware_bootbank_vsan_6.5.0-1.38.7395176, VMware_bootbank_vsanhealth_6.5.0-1.38.7395177
   VIBs Removed: INT_bootbank_intel-nvme_1.2.1.15-1OEM.650.0.0.4598673, INT_bootbank_net-igb_5.3.3-1OEM.600.0.0.2494585, INT_bootbank_net-ixgbe_4.5.3-1OEM.600.0.0.2494585, VMware_bootbank_cpu-microcode_6.5.0-0.0.4564106, VMware_bootbank_esx-base_6.5.0-1.36.7388607, VMware_bootbank_esx-tboot_6.5.0-1.36.7388607, VMware_bootbank_vsan_6.5.0-1.36.7388608, VMware_bootbank_vsanhealth_6.5.0-1.36.7388609
   VIBs Skipped: VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.650.0.0.4564106, VMW_bootbank_ata-pata-amd_0.3.10-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.650.0.0.4564106, VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-via_0.3.3-2vmw.650.0.0.4564106, VMW_bootbank_block-cciss_3.6.14-10vmw.650.0.0.4564106, VMW_bootbank_char-random_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.650.0.14.5146846, VMW_bootbank_elxnet_11.1.91.0-1vmw.650.0.0.4564106, VMW_bootbank_hid-hid_1.0-3vmw.650.0.0.4564106, VMW_bootbank_i40en_1.3.1-5vmw.650.1.26.5969303, VMW_bootbank_igbn_0.1.0.0-15vmw.650.1.36.7388607, VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-devintf_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ixgben_1.4.1-2vmw.650.1.26.5969303, VMW_bootbank_lpfc_11.1.0.6-1vmw.650.0.0.4564106, VMW_bootbank_lsi-mr3_6.910.18.00-1vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt2_20.00.01.00-3vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt3_12.00.02.00-11vmw.650.0.0.4564106, VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.650.0.0.4564106, VMW_bootbank_misc-drivers_6.5.0-1.36.7388607, VMW_bootbank_mtip32xx-native_3.9.5-1vmw.650.0.0.4564106, VMW_bootbank_ne1000_0.8.0-16vmw.650.1.26.5969303, VMW_bootbank_nenic_1.0.0.2-1vmw.650.0.0.4564106, VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.650.0.0.4564106, VMW_bootbank_net-bnx2x_1.78.80.v60.12-1vmw.650.0.0.4564106, VMW_bootbank_net-cdc-ether_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.650.0.0.4564106, VMW_bootbank_net-e1000_8.0.3.1-5vmw.650.0.0.4564106, VMW_bootbank_net-e1000e_3.2.2.1-2vmw.650.0.0.4564106, VMW_bootbank_net-enic_2.1.2.38-2vmw.650.0.0.4564106, VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.650.0.0.4564106, VMW_bootbank_net-forcedeth_0.61-2vmw.650.0.0.4564106, VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-nx-nic_5.0.621-5vmw.650.0.0.4564106, VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.650.0.0.4564106, VMW_bootbank_net-usbnet_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.650.0.0.4564106, VMW_bootbank_nhpsa_2.0.6-3vmw.650.0.0.4564106, VMW_bootbank_nmlx4-core_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-en_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-rdma_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx5-core_4.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_ntg3_4.1.3.0-1vmw.650.1.36.7388607, VMW_bootbank_nvme_1.2.0.32-5vmw.650.1.36.7388607, VMW_bootbank_nvmxnet3_2.0.0.23-1vmw.650.1.36.7388607, VMW_bootbank_ohci-usb-ohci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_pvscsi_0.1-1vmw.650.1.26.5969303, VMW_bootbank_qedentv_2.0.3.29-1vmw.650.0.0.4564106, VMW_bootbank_qfle3_1.0.2.7-1vmw.650.0.0.4564106, VMW_bootbank_qflge_1.1.0.3-1vmw.650.0.0.4564106, VMW_bootbank_qlnativefc_2.1.50.0-1vmw.650.1.26.5969303, VMW_bootbank_sata-ahci_3.0-26vmw.650.1.26.5969303, VMW_bootbank_sata-ata-piix_2.12-10vmw.650.0.0.4564106, VMW_bootbank_sata-sata-nv_3.5-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-promise_2.12-3vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil24_1.1-1vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil_2.3-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-svw_2.3-3vmw.650.0.0.4564106, VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.650.0.0.4564106, VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.650.0.0.4564106, VMW_bootbank_scsi-aic79xx_3.1-5vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.650.0.0.4564106, VMW_bootbank_scsi-hpsa_6.0.0.84-1vmw.650.0.0.4564106, VMW_bootbank_scsi-ips_7.12.05-4vmw.650.0.0.4564106, VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.650.0.0.4564106, VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.650.0.0.4564106, VMW_bootbank_scsi-mpt2sas_19.00.00.00-1vmw.650.0.0.4564106, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.650.0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-3-0_6.5.0-0.0.4564106, VMW_bootbank_uhci-usb-uhci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usbcore-usb_1.0-3vmw.650.1.26.5969303, VMW_bootbank_vmkata_0.1-1vmw.650.1.36.7388607, VMW_bootbank_vmkplexer-vmkplexer_6.5.0-0.0.4564106, VMW_bootbank_vmkusb_0.1-1vmw.650.1.36.7388607, VMW_bootbank_vmw-ahci_1.0.0-39vmw.650.1.26.5969303, VMW_bootbank_xhci-xhci_1.0-3vmw.650.0.0.4564106, VMware_bootbank_emulex-esx-elxnetcli_11.1.28.0-0.0.4564106, VMware_bootbank_esx-dvfilter-generic-fastpath_6.5.0-1.36.7388607, VMware_bootbank_esx-ui_1.23.0-6506686, VMware_bootbank_esx-xserver_6.5.0-0.23.5969300, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-5vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-10vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-7vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-8vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-6vmw.650.1.26.5969303, VMware_bootbank_native-misc-drivers_6.5.0-0.0.4564106, VMware_bootbank_rste_2.0.2.0088-4vmw.650.0.0.4564106, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.10-1.26.5969303, VMware_locker_tools-light_6.5.0-1.33.7273056
[root@xd-1541-5028d:~] reboot

    TinkerTry - PCs, EVs, home tech, efficiency and more, including virtualization. My opinions here, not my employer's.

    Paul Braren, MCSE in 1998, VCP #2681 in 2005, vExpert 2014 to 2024, Veeam Vanguard Emeritus, 2015 to 2019.

    Support TinkerTry

    PayPal | Patreon | VMware Store

    After 6 successful years testing then shipping well over 1,000 Xeon D Bundles, Wiredzone had to stop selling them in mid-2021 due to cost, supply, and logistics challenges. So far, Xeon D-1700/2700 (Ice Lake D) was a minor refresh for 2023, with Xeon D-1800/2800 (Granite Rapids D) refresh looking a little better for 2024. More cores, but still just PCIe Gen4 and DDR4. Looking ahead, I'm glad it's Pat Gelsinger at Intel's helm. I'm also grateful to have had the honor of working at VMware when Pat was the CIO there. I'll leave it at that, given the whole Broadcom thing.

    Easily update VMware ESXi at TinkerTry.com/easy-update-to-latest-esxi