How to easily update your VMware Hypervisor from ESXi 6.5 to 6.5.0a Build 5224934 Advisory ID VMSA-2017-0006 - Critical Severity

Posted by Paul Braren on Mar 31 2017 (updated on Mar 31 2018) in
  • ESXi
  • Virtualization
  • HowTo
  • HomeLab

    • Important Update - On Mar 20 2018, VMware VMSA-2018-0004.3 announced that CVE-2017-5715 (Spectre-2) mitigation is now included in the latest patch that you should be using instead of the older patch featured in the original article below. You'll find the newer article that features an even easier update method here:

    Article below as it originally appeared.

    VMware-Security-and-Compliance-Blog

    Let's keep this recent story in perspective, some excerpts below, but you should read the whole article:
    The Security Landscape: Pwn2Own 2017

    ...two teams succeeded in demonstrating arbitrary host code execution on VMware Workstation. Today, VMware is releasing updated versions of VMware vSphere ESXi, VMware Fusion, and VMware Workstation to address these vulnerabilities. VMSA-2017-0006 contains details on impacted versions and the releases which contain fixes.
    No active exploitation
    VMware is not aware of any active exploitation of the vulnerabilities revealed in this competition. Though the vulnerabilities seem to apply to all VMware virtual platforms (ESXi, Fusion, and Workstation), demonstration exploit code appears to exist only for VMware Workstation for Windows.
    ...
    VMware also recommends examining the vSphere Hardening Guide and
    vSphere Security Guide. Among the recommendations in the guides is to remove unnecessary virtual hardware.
    ...
    Customers should consider the need to update for a full mitigation, the absence of active exploitation, the pace at which updates can safely be deployed, and any other risk mitigations (like IDS applications) which may protect their environments. At this point VMware’s recommendation is that customers expedite updating, though need not take emergency measures like taking environments offline.

    ESXi 6.5.0a, Build 5224934 released 2017-03-28

    VMware Security Advisory

    ESXI 6.5.0a (most recent-release of ESXi in ISO format as of Mar 31 2017)

    VMware Release Notes for ESXi 6.5.0a | 02 FEB 2017 | ISO Build 4887370

    Warning:

    1. vCenter/VCSA 6.x should be upgraded to 6.5.0a before upgrading your host(s) to ESXi 6.5.0a Build 5224934, see:
      How to easily update your VMware vCenter Server Appliance from VCSA 6.5 to 6.5.0a
      Feb 07 2017
    2. I have only tested this method when upgrading from 6.5.0a Build 4887370 to Build 5224934, your experience from earlier 6.x versions may vary.
    3. This is not official VMware documentation, it's merely a convenient upgrade technique that may help in lab tests, it's up to you to adhere to the backup-first advice detailed below.

    Why esxcli?

    If you're in production, beware, this just came out 3 days ago. This article is for the lab, where you may want to give this critical patch a try. All the background story on how this easy ESXCLI upgrade method came about was covered in my earlier articles about updating 6.0 U2 and 6.5.

    Benefits

    1. No new license needed to go from 6.0.x or 6.5.0a Build 4887370 l to 6.5.0a Build 5224934
    2. Users of the free hypervisor and folks who can't download the GA Offline bundle now have a path forward as well, without needing to read TinkerTry's My VMware's "You either are not entitled or do not have permissions to download this product." error, and what to do about it.

    Prerequisites

    Once you've completed ALL of the following preparation steps:

    1. upgraded to VCSA 6.5.0a
    2. ensured your ESXi 6.5.x host has a working internet connection
    3. reviewed the release notes
    4. reviewed this VMSA-2017-0006 patch
    5. reviewed How to easily update your VMware Hypervisor to ESXi 6.0 Update 2 for the full back story that includes some warnings about potential gotchas/driver issues
    6. backed up the ESXi 6.5.x you've already got, using something like one of the home-lab-friendly techniques such as using USB Image Tools, as detailed by Florian Grehl here

    you can now continue with this simple approach to upgrading your lab environment. Unsupported, at your own risk, see the full disclaimer at below left.

    What's nice about this ESXCLI upgrade method is that its super simple and you don't have to worry about requesting a trial to be able to log into My VMware to download your ESXi 6.5.0a ISO:

    Name: VMware-VMvisor-Installer-201701001-4887370.x86_64.iso
    Release Date: 2017-02-02
    Build Number: 4887370

    Upgrade

    Download and upgrade to 6.5.0a plus the VMSA-2017-0006 update using the patch directly from the VMware Online Depot

    The entire process including reboot is usually well under 10 minutes. Triple-clicking on a line of code below highlights the whole thing, to right-click, copy into your clipboard:

    1. Open an SSH session (eg. PuTTY) to your ESXi 6.0.x server
      (if you forgot to enable SSH, here's how)
    2. Turn on maintenance mode, or ensure you've set your ESXi host to automatically gracefully shutdown all VMs upon host reboot, or shutdown all the VMs gracefully that you care about, including VCSA.
    3. Firewall allow outbound http requests - Paste the one line below into into your SSH session, then press enter: 
      esxcli network firewall ruleset set -e true -r httpClient

      More details about the firewall here.

    4. Pull down ESXi Image Profile using https and run patch script - Paste the line below into into your SSH session, then hit enter and wait while nothing seems to happen, taking somewhere between roughly 3 to 10 minutes before the completion screen (sample below) appears: 
      esxcli software profile install -p ESXi-6.5.0-20170304101-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

      Wait time depending mostly on the the speed of the ESXi's connection to the internet, and a little on the speed of the storage media that ESXi is installed on.

    5. Firewall disallow outbound http requests - Paste the line below into into your SSH session: 
      esxcli network firewall ruleset set -e false -r httpClient
    6. If you turned on maintenance mode earlier, remember to turn maintenance mode off.
    7. If you normally leave SSH access off, go ahead and disable it now.
    8. Type reboot and hit return (to restart your ESXi server), or use your favorite ESXi UI to restart the host.
    9. After the reboot is done, it would be a good idea to test login using ESXi host client, pointing your browser to the IP or hostname of your just-graded server, to be sure everthing seems to be working right.

    You're done!

    Special thanks to VMware ESXi Patch Tracker by Andreas Peetz at the VMware Front Experience Blog. This upgrade test was performed on a TinkerTry'd VMware HCL system. Yes, on both the very popular 8 core and the rather special 12 core version of the beloved Supermicro SuperServer SYS-5028D-TN4T system.

    esxcli-update-successful-6.5.0.a-Build-5224934-VMSA-2017-0006-TinkerTry-Mar-31-2017
    Here's how my upgrade from 6.5.0a Build 4887370 to 6.5.0a Build 4887370 looked, right after the 1 minute download/patch.
    DCUI-showing-6.5.0a-Build-5224529-TinkerTry-2017-Mar-31
    Yep, it worked! This is called the DCUI, using Supermicro's iKVM HTML5 UI to show you what my console looked like after the patch & reboot.
    ESXi-host-client-6.5.0a-Build-5224529-full-length-TinkerTry-2017-Mar-31
    ESXi Host client view of Build 5224529.

    That's it! When the reboot is complete, you'll see for yourself that you now have the latest ESXi, Build 5224529, as pictured above. Now you have more spare time to read more TinkerTry articles!

    Potential gotchas

    1. Depending upon your ESXi firewall configuration, if the above command results in a network related error such as:
      'NoneType' object has no attribute 'close'
      then you skipped the firewall configuration step above, try again!

    2. Notice that the command recommended you use when clicking on the ESXi-6.5.0-20170304101-standard link at VMware ESXi Patch Tracker: 
      esxcli software profile update -p ESXi-6.5.0-20170304101-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

      doesn't work, says:
      Message: Host is not changed.
      but simply changing from update to install worked for me, but your results may vary. See also the interesting comment below.

      host-is-not-changed
      Using the update parameter doesn't work, as seen above, but using install does.

    Potential SATA and Realtek NIC gotcha

    1. If you find some of your SATA/AHCI datastores disappear from view after this upgrade, worry not, the VMFS datastores are still there, you just can't see them. This article should still save you:
      For ESXi 6.0, those ESXi 5.1 VIBs for ASMedia SATA ports and Realtek NICs still seem to be working (but unsupported)
      Mar 04 2015

    Closing Thoughts

    Alternatively, you could have used VMware Update Manager on a Windows system or VM, but for one-off upgrades typical in a small home lab, pasting these 3 lines of code is pretty darn easy.

    Looking ahead, since VUM is now built into VCSA 6.5, this will add another way to do future upgrades and patches, even in a small home lab environment.

    Video

    How to easily update your VMware vCenter Server Appliance from VCSA 6.5 to 6.5.0a Build 5224934

    Apr 01 2017 Update

    For ESXi 6.0 users who don't wish to move to 6.5, there are also patches for this issue, but those are not the focus of this article. See:

    See also at TinkerTry

    See also

    Upgrade Log

    Below, I've pasted the full text of my upgrade, helps you see what drivers were touched, use the horizonal scroll bar or shift + mousewheel to look around, Ctrl+F works as needed too:

    [root@xd-1541-5028d:~] esxcli software profile install -p ESXi-6.5.0-20170304101-standard -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.650.0.14.5146846, VMW_bootbank_ixgben_1.0.0.0-9vmw.650.0.14.5146846, VMW_bootbank_misc-drivers_6.5.0-0.14.5146846, VMW_bootbank_ne1000_0.8.0-11vmw.650.0.14.5146846, VMW_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.650.0.0.4564106, VMW_bootbank_vmkusb_0.1-1vmw.650.0.14.5146846, VMW_bootbank_vmw-ahci_1.0.0-34vmw.650.0.14.5146846, VMware_bootbank_esx-base_6.5.0-0.15.5224529, VMware_bootbank_esx-ui_1.15.0-5069532, VMware_bootbank_vsan_6.5.0-0.15.5224529, VMware_bootbank_vsanhealth_6.5.0-0.15.5224529
   VIBs Removed: INT_bootbank_net-ixgbe_4.4.1-1OEM.600.0.0.2159203, VMW_bootbank_ehci-ehci-hcd_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ixgben_1.0.0.0-8vmw.650.0.0.4564106, VMW_bootbank_misc-drivers_6.5.0-0.0.4564106, VMW_bootbank_ne1000_0.8.0-9vmw.650.0.0.4564106, VMW_bootbank_vmkusb_0.1-1vmw.650.0.0.4564106, VMW_bootbank_vmw-ahci_1.0.0-32vmw.650.0.0.4564106, VMware_bootbank_esx-base_6.5.0-0.9.4887370, VMware_bootbank_esx-ui_1.8.0-4516221, VMware_bootbank_vsan_6.5.0-0.9.4887370, VMware_bootbank_vsanhealth_6.5.0-0.9.4887370
   VIBs Skipped: VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.650.0.0.4564106, VMW_bootbank_ata-pata-amd_0.3.10-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.650.0.0.4564106, VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-via_0.3.3-2vmw.650.0.0.4564106, VMW_bootbank_block-cciss_3.6.14-10vmw.650.0.0.4564106, VMW_bootbank_char-random_1.0-3vmw.650.0.0.4564106, VMW_bootbank_elxnet_11.1.91.0-1vmw.650.0.0.4564106, VMW_bootbank_hid-hid_1.0-3vmw.650.0.0.4564106, VMW_bootbank_i40en_1.1.0-1vmw.650.0.0.4564106, VMW_bootbank_igbn_0.1.0.0-12vmw.650.0.0.4564106, VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-devintf_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.650.0.0.4564106, VMW_bootbank_lpfc_11.1.0.6-1vmw.650.0.0.4564106, VMW_bootbank_lsi-mr3_6.910.18.00-1vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt2_20.00.01.00-3vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt3_12.00.02.00-11vmw.650.0.0.4564106, VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.650.0.0.4564106, VMW_bootbank_mtip32xx-native_3.9.5-1vmw.650.0.0.4564106, VMW_bootbank_nenic_1.0.0.2-1vmw.650.0.0.4564106, VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.650.0.0.4564106, VMW_bootbank_net-bnx2x_1.78.80.v60.12-1vmw.650.0.0.4564106, VMW_bootbank_net-cdc-ether_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.650.0.0.4564106, VMW_bootbank_net-e1000_8.0.3.1-5vmw.650.0.0.4564106, VMW_bootbank_net-e1000e_3.2.2.1-2vmw.650.0.0.4564106, VMW_bootbank_net-enic_2.1.2.38-2vmw.650.0.0.4564106, VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.650.0.0.4564106, VMW_bootbank_net-forcedeth_0.61-2vmw.650.0.0.4564106, VMW_bootbank_net-igb_5.0.5.1.1-5vmw.650.0.0.4564106, VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-nx-nic_5.0.621-5vmw.650.0.0.4564106, VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.650.0.0.4564106, VMW_bootbank_net-usbnet_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.650.0.0.4564106, VMW_bootbank_nhpsa_2.0.6-3vmw.650.0.0.4564106, VMW_bootbank_nmlx4-core_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-en_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-rdma_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx5-core_4.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_ntg3_4.1.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nvme_1.2.0.32-2vmw.650.0.0.4564106, VMW_bootbank_nvmxnet3_2.0.0.22-1vmw.650.0.0.4564106, VMW_bootbank_ohci-usb-ohci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_pvscsi_0.1-1vmw.650.0.0.4564106, VMW_bootbank_qedentv_2.0.3.29-1vmw.650.0.0.4564106, VMW_bootbank_qfle3_1.0.2.7-1vmw.650.0.0.4564106, VMW_bootbank_qflge_1.1.0.3-1vmw.650.0.0.4564106, VMW_bootbank_qlnativefc_2.1.30.0-11vmw.650.0.0.4564106, VMW_bootbank_sata-ahci_3.0-22vmw.650.0.0.4564106, VMW_bootbank_sata-ata-piix_2.12-10vmw.650.0.0.4564106, VMW_bootbank_sata-sata-nv_3.5-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-promise_2.12-3vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil24_1.1-1vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil_2.3-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-svw_2.3-3vmw.650.0.0.4564106, VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.650.0.0.4564106, VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.650.0.0.4564106, VMW_bootbank_scsi-aic79xx_3.1-5vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.650.0.0.4564106, VMW_bootbank_scsi-hpsa_6.0.0.84-1vmw.650.0.0.4564106, VMW_bootbank_scsi-ips_7.12.05-4vmw.650.0.0.4564106, VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.650.0.0.4564106, VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.650.0.0.4564106, VMW_bootbank_scsi-mpt2sas_19.00.00.00-1vmw.650.0.0.4564106, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.650.0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-3-0_6.5.0-0.0.4564106, VMW_bootbank_uhci-usb-uhci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usbcore-usb_1.0-3vmw.650.0.0.4564106, VMW_bootbank_vmkata_0.1-1vmw.650.0.0.4564106, VMW_bootbank_vmkplexer-vmkplexer_6.5.0-0.0.4564106, VMW_bootbank_xhci-xhci_1.0-3vmw.650.0.0.4564106, VMware_bootbank_cpu-microcode_6.5.0-0.0.4564106, VMware_bootbank_emulex-esx-elxnetcli_11.1.28.0-0.0.4564106, VMware_bootbank_esx-dvfilter-generic-fastpath_6.5.0-0.0.4564106, VMware_bootbank_esx-tboot_6.5.0-0.0.4564106, VMware_bootbank_esx-xserver_6.5.0-0.0.4564106, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-3vmw.650.0.0.4564106, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-7vmw.650.0.0.4564106, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-6vmw.650.0.0.4564106, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-7vmw.650.0.0.4564106, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-5vmw.650.0.0.4564106, VMware_bootbank_native-misc-drivers_6.5.0-0.0.4564106, VMware_bootbank_rste_2.0.2.0088-4vmw.650.0.0.4564106, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.10-0.0.4564106, VMware_locker_tools-light_6.5.0-0.0.4564106
[root@xd-1541-5028d:~] esxcli network firewall ruleset set -e false -r httpClient
[root@xd-1541-5028d:~] reboot

    TinkerTry - PCs, EVs, home tech, efficiency and more, including virtualization. My opinions here, not my employer's.

    Paul Braren, MCSE in 1998, VCP #2681 in 2005, vExpert 2014 to 2024, Veeam Vanguard Emeritus, 2015 to 2019.

    Support TinkerTry

    PayPal | Patreon | VMware Store

    After 6 successful years testing then shipping well over 1,000 Xeon D Bundles, Wiredzone had to stop selling them in mid-2021 due to cost, supply, and logistics challenges. So far, Xeon D-1700/2700 (Ice Lake D) was a minor refresh for 2023, with Xeon D-1800/2800 (Granite Rapids D) refresh looking a little better for 2024. More cores, but still just PCIe Gen4 and DDR4. Looking ahead, I'm glad it's Pat Gelsinger at Intel's helm. I'm also grateful to have had the honor of working at VMware when Pat was the CIO there. I'll leave it at that, given the whole Broadcom thing.

    Easily update VMware ESXi at TinkerTry.com/easy-update-to-latest-esxi