How to easily update your VMware Hypervisor from 6.x to 6.5 Update 1 Patch Release ESXi-6.5.0-20180304001-standard (ESXi Build 7967591) with Spectre mitigation

Posted by Paul Braren on Mar 20 2018 (updated on Mar 31 2018) in
  • ESXi
  • Virtualization
  • HowTo
  • HomeLab

  • As covered earlier at TinkerTry, patching for the branch target injection vulnerability known as Spectre-2 requires you to patch your:

    Supermicro-SuperServer-Bundle-2-Xeon-D-1567-patched-ESXi-and-BIOS-2018-03-24--TinkerTry
    My own home lab tests of these patches are going well so far.
    1. VCSA/vCenter
    2. ESXi server(s) (this article)
    3. Each VM operating system
    4. Each server('s) BIOS/firmware
    3262786803

    The comment above is still relevant, as this ESXCLI method is just a more-universal way to upgrade ESXi, a one-liner that side-steps the preferred VUM method for those without VCSA and/or without a My VMware account to download the patch separately. This method doesn't have quite as easy a way to revert if things go wrong. All hypervisor upgrades come with risks, including the slight possibility of losing your network connections, so proceed at your own risk only after reading the entire article, and after backing up your hypervisor first.

    Disclaimer/Disclosure - I cannot feasibly provide support for your upgrade, especially given the variety of unsupported hardware out there, see full disclaimer at below-left. This article is focused mostly on small home labs, was voluntarily authored, and not associated with my employment at VMware. It is not official documentation. I work in the storage division, separate from the group developing and supporting the hypervisor.

    Meltdown and Spectre are looming large this year, and this post is one of many that are in direct response to that potential threat, see also at TinkerTry:

    meltdown-and-spectre-info

    google-project-zero-spectre-and-meltdown-patch-and-flash-for-vmware-esxi-on-supermicro-xeon-d

    Yes, this ESXi patch is the latest that mitigates Branch Target Injection, released today, March 20 2018. If you patch your system right away, you are by definition at the bleeding edge. You may want to wait and see how this upgrade goes for others before you jump in yourself. Since the procedure below is based on a reader's input and various refinements to the previous set of similar articles, any and all feedback through comments below would be greatly appreciated.

    Warning!
    Don't rush things. At a minimum, even for a home lab, you'll want to read this entire article before patching anything! Special thanks go out to VCDX 194 Matt Kozloski, whose invaluable feedback made this article much better than my previous set of ESXi update articles.

    Step 1 - do your homework

    VMware ESXI 6.5 Update 1 Build 7967591

    Read all three of these KB articles below for details on what this patch fixes. I have some brief excerpts below each link, to encourage you to read each of the source kb articles in their entirety. Much of this information came from My VMware under Product Patches, where I searched on ESXi 6.5.0.

    • Hypervisor-Assisted Guest Mitigation for Branch Target injection (52085)

      ...
      Purpose
      Recent microcode updates by Intel and AMD provide hardware support for branch target injection mitigation (Spectre v2). In order to use this new hardware feature within virtual machines, Hypervisor-Assisted Guest Mitigation must be enabled.

      This document will focus on Hypervisor-Assisted Guest Mitigation as it pertains to vSphere. Please review KB52245: VMware Response to Speculative Execution security issues, CVE-2017-5753, CVE-2017-5715, CVE-2017-5754 (aka Spectre and Meltdown) for a complete view on VMware’s response to these issues.

      See VMware Security Advisory VMSA-2018-0004.3 for the VMware provided patches related to this KB.

      Resolution
      Patching the VMware vSphere hypervisor and updating the CPU Microcode (which the vSphere patches will do for the processors described in the below table will allow guest operating systems to use hardware support for branch target mitigation.

      To enable hardware support for branch target mitigation in vSphere, apply these steps, in the order shown:

      Note: Ensure vCenter Server is updated first, for more information, see the vMotion and EVC Information section.

      1. Upgrade to one of the following versions of vCenter 5.5 – 6.5:
        6.5 U1g: Release Notes.
        6.0 U3e: Release Notes.
        5.5 U3h: Release Notes.
        Important: Please review the release notes for vCenter as there are new items listed in the ‘known issues’ section.
      2. Apply both of the following ESXi patches. Note: these can both be applied at once so that only 1 reboot of the host is required:
        • ESXi 6.5: ESXi650-201803401-BG* and ESXi650-201803402-BG**
        • ESXi 6.0: ESXi600-201803401-BG* and ESXi600-201803402-BG**
        • ESXi 5.5: ESXi550-201803401-BG* and ESXi550-201803402-BG**

      * These ESXi patches provide the framework to allow guest OSes to utilize the new speculative-execution control mechanisms. These patches do not contain microcode.

      ** These ESXi patches apply the microcode updates listed in the Table below. These patches do not contain the aforementioned framework.
      ...

    • VMware ESXi 6.5, Patch Release ESXi650-201803401-BG - Updates esx-base, esx-tboot, vsan, and vsanhealth VIBs (52460)

      Release date: March 20, 2018
      Download Filename:
      VIBs Included:

      • VMware_bootbank_esx-base_6.5.0-1.41.7967591
      • VMware_bootbank_esx-tboot_6.5.0-1.41.7967591
      • VMware_bootbank_vsan_6.5.0-1.41.7547709
      • VMware_bootbank_vsanhealth_6.5.0-1.41.7547710

      Summaries and Symptoms
      This patch updates the esx-base, esx-tboot, vsan and vsanhealth VIBs to resolve the following issue:

      This ESXi patch provides part of the hypervisor-assisted guest mitigation of CVE-2017-5715 for guest operating systems. For important details on this mitigation, see VMware Security Advisory VMSA-2018-0004.3.
      ...

    • VMware ESXi 6.5, Patch Release ESXi650-201803402-BG - Updates cpu-microcode VIB (52461)

      Release date: March 20, 2018
      ...
      VIBs Included:

      • VMware_bootbank_cpu-microcode_6.5.0-1.41.7967591

      NOTE: If you have added the workaround step mentioned in KB 52345, the workaround will not be removed automatically if you apply the cpu-microcode VIB alone. You must also apply the VIBs in bulletin ESXi650-201803401-BG to remove the workaround.

      Solution
      Summaries and Symptoms
      This patch updates the cpu-microcode VIB to resolve the following issue:

      • This ESXi patch provides part of the hypervisor-assisted guest mitigation of CVE-2017-5715 for guest operating systems. For important details on this mitigation, see VMware Security Advisory VMSA-2018-0004.3.
        ...

    Step 2 - Follow Prerequisites

    Once you've completed ALL of the following preparation steps:

    1. upgraded to the latest VCSA, which is currently 6.5 U1g, see 6.5 Update 1g Release Notes, along with the easy instructions at How to easily update your VMware vCenter Server Appliance from 6.5.x to 6.5 Update 1f (VCSA 6.5 U1f) for Meltdown/Spectre-1 mitigation
    2. I tend to put my modern systems BIOs setting to UEFI mode (instead of Dual), see details here, as a bit of future proofing. You can read Mike Foley's warnings in Secure Boot for ESXi 6.5 – Hypervisor Assurance

      ...
      Possible upgrade issues
      UEFI secure boot requires that the original VIB signatures are persisted. Older versions of ESXi do not persist the signatures, but the upgrade process updates the VIB signatures.

      If your host was upgraded using the ESXCLI command then your bootloader wasn’t upgraded and doesn’t persist the signatures. When you enable Secure Boot after the upgrade, an error occurs. You can’t use Secure Boot on these installations and will have to re-install from scratch to gain that support.
      ...

      usbit01
    3. Backed up the ESXi 6.5.x hypervisor you've already installed and configured, for easy roll-back in case things go wrong. If it's on USB or SD, it's best to clone to a new USB drive and boot from it, to be sure your "backup" is good. You can use something like one of the home-lab-friendly and super easy methods such as USB Image Tools under Windows, as detailed by Florian Grehl here.
      I've occasionally had some trouble with that tool, so an alternative you may want to try (that I'm still testing) is EaseUS free backup software instead, note the blue button below is a direct download link, preventing you from having to provide an email address.
      EaseUS-free-download-cloning
      Direct Download - EaseUS free backup software for Windows allows you to easily clone USB drives.

      If you don't wish to do either, at least follow this VMware KB article:
      How to back up ESXi host configuration (2042141)

    4. Ensured your ESXi 6.5.x host has a working internet connection.
    5. Review the ESXi 6.5 release notes too.
    6. Read this entire article, yes, even the entire set of prerequisites above.

    Step 3 - Perform Upgrade

    Step-by-Step Instructions

    Download and upgrade to VMware ESXI 6.5 Update 1 Build 7967591 using the patch bundle that comes directly from the VMware Online Depot

    The entire process including reboot is usually well under 10 minutes, and many of the steps below are optional, making it appear more difficult than it is. Triple-clicking on a line of code below highlights the whole thing with a carriage return, so you can then right-click and copy it into your clipboard, which gets executed immediately upon pasting into your SSH session. If you want to edit the line before it's executed, manually swipe your mouse across each line of code with no trailing spaces at the end.

    1. Open an SSH session (eg. PuTTY) to your ESXi 6.x server
      (if you forgot to enable SSH, here's how)

    2. OPTIONAL - Turn on Maintenance Mode - Or you can just be sure to manually shutdown all the VMs gracefully that you care about, including VCSA. These instructions are geared to a home lab without High Availability enabled. This is also a good time to ensure you've also set ESXi host to automatically gracefully shutdown all VMs upon host reboot, or if you don't use vCenter or VCSA, use this Host Client method.

    3. OPTIONAL - Reboot (Pro Tip courtesy of VCDX 194 Matt Kozloski) - Consider rebooting your ESXi server and maybe even a hard power cycle before updating. Matt explains:

      if people are running on SD cards or USB sticks and they haven't rebooted the server in a LONG time to patch/update, I would strongly recommend doing a reboot of the server before applying any updates. I've seen, more than once, the SD card or the controller goes into some funky state and as ESXi is running largely in memory, it can comes up half patched or not patched at all. A [cold] reboot before update helps with that (again, if a server has been running for a long period of time - like a year+ - since it was rebooted last). Cold (remove the power cables) can be important, if the SD card or USB stick is actually running on an embedded controller like iLO or iDRAC.

    4. OPTIONAL - Firewall allow outbound http requests - This command is likely not needed if you're upgrading from 6.5.x, and is here in case you get an error about https access. I'm trying to make these instructions applicable to the broadest set of readers. Paste the one line below into into your SSH session, then press enter:

      esxcli network firewall ruleset set -e true -r httpClient

      More details about the firewall here.

    5. OPTIONAL - See a list of all available ESXi profiles - VMware's Upgrade or Update a Host with Image Profiles documentation tells you how this command was formed. Paste the one line below into into your SSH session, then press enter:

      esxcli software sources profile list --depot=https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml

      You can cut-and-paste the output from the above command into a spreadsheet if you'd like, so you can then sort it, making it apparent which profile is the most recent.

    6. Dry Run - Taking this extra step will help you be sure of what is about to happen, before it actually happens.
      Here's the simple command to cut-and-paste into your SSH session:

      esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-6.5.0-20180304001-standard --dry-run

      If you see some VIBs that are going to be removed that you need, you'll need to be fully prepared to manually re-install them after the actual upgrade below. If it's a network VIB that is used for your ESXi service console, you'll want to be extra careful to re-install that same VIB before rebooting your just-patched host(s). Don't just assume some later VIB version will work fine with your hardware, use what you know works, and carefully double-check the VMware Compatibility Guide for the recommended version.

    7. ACTUAL RUN - This is it, the all-in-one download and patch command, assuming your ESXi host has internet access. This will pull down the ESXi Image Profile using https, then it will run the patch script.
      When you paste this line into your SSH session and hit enter, you'll need to be patient, as nothing seems to happen at first. It will take somewhere between roughly 3 to 10 minutes before the completion screen (sample below) appears:

      esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-6.5.0-20180304001-standard
    8. Firewall disallow outbound http requests - To return your firewall to how it was before (optional) step 4 above, simply copy and paste the following:

      esxcli network firewall ruleset set -e false -r httpClient
    9. Attention Xeon D-1500 Owners - See 3 lines you may want to paste in before rebooting, details below, then return to the next step when done.

    10. Reboot - This is needed for the new hypervisor version to be loaded upon restart. You may want to watch the DCUI (local console) as it boots, to see if any errors show up.

      reboot
    11. OPTIONAL - If you turned on Maintenance Mode in step 3 above, you'll need to turn it off.

    12. You're Done! - You may want to be continue with checking whether everything is working correctly after your systems is back up again, but you are done with the update itself. You can also watch DCUI during the boot if you'd like, to see if you spot any warnings.

    13. Test things out - Log in with ESXi Host Client (pointing your browser directly at your IP address or ESXi servername), and be sure everything seems to function fine. You may need to re-map USB devices to VMs that use USB, and you may need to re-map VT-d (passthrough) devices to VMs that use passthrough devices like GPUs.

    14. You're Really Done! - If you're happy that everything seems to be working well, that's a wrap, but keep that backup, just in case you notice something odd later on.

    Version Confirmation

    Now that you've updated and rebooted, various UIs will show your ESXi version, depending upon where you look.

    Host Client:

    • Version: 6.5.0 Update 1 (Build 7967591)
    • Image profile: (Updated) ESXi-6.5.0-20180304001-standard (VMware, Inc.)

    vSphere Web Client (Flash):

    • Hypervisor: VMware ESXi, 6.5.0, 7967591
    • Image Profile: (Updated) ESXi-6.5.0-20180304001-standard

    vSphere Client (HTML5):

    • Hypervisor: VMware ESXi, 6.5.0, 7967591
    • Image Profile: (Updated) ESXi-6.5.0-20180304001-standard

    SSH session to updated ESXi host:

    vmware -vl
    • VMware ESXi 6.5.0 build-7967591 | VMware ESXi 6.5.0 Update 1
    uname -a
    • VMkernel xd-1567-5028d.lab.local 6.5.0 #1 SMP Release build-7967591 Mar 7 2018 23:06:14 x86_64 x86_64 x86_64 ESXi

    Notes for Xeon D Owners

    2018-03-20_20-03-38
    • There is now BIOS 1.3 out for Spectre mitigation.
    • The default ESXi 6.5 install works great with your integrated Intel SATA3 AHCI ports, but there are better drivers for your I350 1GbE ports, and a driver needed for your two X552/X557 10GbE ports, also handy as extra 1GbE connections. There's also a fix for odd RPM and temperature readings. No problem, all 3 items easily remedied, and the Xeon D-1500 is on the VMware HCL.
    • All of the optional ESXCLI commands below are really only needed for Xeon D-1500 if you haven't already done these steps to your prior ESXi host before the update.
    • Future reference - The not-yet-shipping Xeon D-2100 is said to work with the drivers that are included with ESXi 6.5 Update 1, a good sign. But that doesn't necessarily mean it's fully supported at those versions. I'll soon have my hands on a loaner Supermicro SuperServer SYS-300-9D to find out for sure.
    1. OPTIONAL - Xeon D 12 or 16 core - If your system uses the Xeon D-1557, Xeon D-1567, or Xeon D-1587, you may find the VMware ESXi 6.0 igbn 1.4.1 NIC Driver for Intel Ethernet Controllers 82580,I210,I350 and I354 family performs better for the service console on either ETH0 or ETH1 instead of the included-with 6.5U1 VMware inbox driver for the I-350 called
      VMW_bootbank_net-igb_5.0.5.1.1-5vmw.650.0.0.4564106. Here's how you can determine your driver version to verify, but running the command anyway is generally harmless, since it will skip the upgrade attempt if the same exact driver is already present. This driver can help prevent ethernet enumeration (reversal) issues, where the service console can jump over to the second I350 port after updating. You simply copy-and-paste the following one-liner:

      esxcli software vib install -v https://cdn.tinkertry.com/files/igbn-1.4.1-1OEM.600.0.0.2768847.x86_64.vib --no-sig-check

      before proceeding, or just download the VIB yourself, then follow the install instructions in the readme.

    2. OPTIONAL - Xeon D with 10GbE - If your system includes two 10GbE Intel X552/X557 RJ45 or SFP+ NICs ports, they can be used for 1GbE or 10GbE speeds, but you'll need to have the newer 4.5.3 10GbE Intel driver VIB, rather than older VMW_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.650.0.0.4564106 release that came with ESXi 6.5. Here's how you can determine your driver version to verify, but running the command anyway is generally harmless, since it will skip the upgrade attempt if the same exact driver is already present. To update, simply copy and paste the following one-liner easy fix:

      esxcli software vib install -v https://cdn.tinkertry.com/files/net-ixgbe_4.5.3-1OEM.600.0.0.2494585.vib --no-sig-check

      with the details and fully supported download method described in detail here before proceeding.

    3. OPTIONAL - Xeon D with inaccurate RPM and Temperature readings in Health Status Monitoring
      The reasons for these commands are explained in detail here, note, this fix shouldn't be necessary for future major releases of ESXi, with this bug already reported:
      esxcli system wbem set --ws-man false
      esxcli system wbem set --enable true

    Video

    Step-by-step video showing me upgrading a Xeon D in my home lab is coming soon.

    REFERENCE

    You should wind up with the same results after this upgrade as folks who upgrade by downloading the full ESXi 6.5 U1 ISO / creating bootable media from that ISO / booting from that media (or mounting the ISO over IPMI/iLO/iDRAC/IMM/iKMV) and booting from it:

    File size: 332.63 MB
    File type: iso
    Name: VMware-VMvisor-Installer-6.5.0.update01-5969303.x86_64.iso
    Release Date: 2017-07-27
    Build Number: 5969303

    installing it, rebooting, patching per instructions below, and rebooting again.


    Mar 21 2018 Update

    kelser-cropped-resized

    This article has some updates to suggest an extra reboot before upgrading, based on some excellent new feedback I received this morning. Thank you, Matt Kosloski, my Connecticut neighbor at Kelser!


    Mar 24 2018 Update

    Title changed from:
    How to easily update your VMware Hypervisor from 6.x to 6.5 Update 1 Patch Release ESXi650-201803001 (ESXi Build 7967591) for hypervisor-assisted guest mitigation for branch target injection
    to:
    How to easily update your VMware Hypervisor from 6.x to 6.5 Update 1 Patch Release ESXi650-201803001 (ESXi Build 7967591) with Spectre mitigation


    Mar 28 2018 Update

    I found mistakes in the procedure to be sure you have the right X557 driver, now corrected, thanks to Shawn Mix's comment.


    Mar 31 2018 Update

    To be more technical precise, I have changed the title changed from:
    How to easily update your VMware Hypervisor from 6.x to 6.5 Update 1 Patch Release ESXi650-201803001 (ESXi Build 7967591) for hypervisor-assisted guest mitigation for branch target injection
    to:
    How to easily update your VMware Hypervisor from 6.x to 6.5 Update 1 Patch Release ESXi-6.5.0-20180304001-standard (ESXi Build 7967591) with Spectre mitigation


    See also at TinkerTry

    easy-upgrade-to-vcsa-65u1g

    supermicro-superservers-vcg-updated-to-65u1

    supermicro-sys-e300-9d-superserver-is-the-only-xeon-d-2100-for-home-labs

    meltdown-and-spectre-info

    superservers


    See also

    VMSA-2018-0004

    Hypervisor-Assisted Guest Remediation

    • VMSA-2018-0004.3
      VMware vSphere, Workstation and Fusion updates add Hypervisor-Assisted Guest Remediation for speculative execution issue

      Jan 09 2018 at VMware Security Advisories, updated Mar 03 2018

      ...
      1. Summary
      VMware vSphere, Workstation and Fusion updates add Hypervisor-Assisted Guest Remediation for speculative execution issue.

      The mitigations in this advisory are categorized as Hypervisor-Assisted Guest Mitigations described by VMware Knowledge Base article 52245.

      2. Relevant Products
      VMware vCenter Server (VC)
      VMware vSphere ESXi (ESXi)
      VMware Workstation Pro / Player (Workstation)
      VMware Fusion Pro / Fusion (Fusion)

      3. Problem Description
      New speculative-execution control mechanism for Virtual Machines

      Updates of vCenter Server, ESXi, Workstation and Fusion virtualize the new speculative-execution control mechanism for Virtual Machines (VMs). As a result, a patched Guest Operating System (Guest OS) can remediate the Branch Target Injection issue (CVE-2017-5715). This issue may allow for information disclosure between processes within the VM.
      ...
      Column 5 of the following table lists the action required to remediate the vulnerability in each release, if a solution is available.

      VMSA-2018-0004-table-excerpt-resized--TinkerTry
      First portion of the table, click the image to visit the source article.

    Here's the 2 patches for VCSA and ESXi 6.5 that the table above points to, hyperlinked for you:

    • VCSA 6.5 U1g available here, and Release Notes.
      Name: VMware-VCSA-all-6.5.0-8024368.iso Release Date: 2018-03-20 Build Number: 8024368
    • ESXi 6.5: ESXi650-201803401-BG KB52460 and ESXi650-201803402-BG KB52461 are both seen here, with ESXi Build Number 7967591 KB52456.
    ESXi-6.5.0

    GUID-E51C5DB6-F28E-42E8-ACA4-0EBDD11DF55D

    Upgrade Log

    Below, I've pasted the full text of my update. It will help you see what drivers are touched. Just use the horizontal scroll bar or shift + mousewheel to look around, and Ctrl+F to Find stuff quickly:

    As also seen in my video of my previous upgrade, here's the full contents of my ssh session, as I completed my Xeon D-1541 upgrade from
    Version: 6.5.0 Update 1 (Build 7388607)
    to:
    Version: 6.5.0 Update 1 (Build 7967591)

    login as: root
    Using keyboard-interactive authentication.
    Password:
    The time and date of this login have been sent to the system logs.
    
    WARNING:
       All commands run on the ESXi shell are logged and may be included in
       support bundles. Do not provide passwords directly on the command line.
       Most tools can prompt for secrets or accept them from standard input.
    
    VMware offers supported, powerful system administration tools.  Please
    see www.vmware.com/go/sysadmintools for details.
    
    The ESXi Shell can be disabled by an administrative user. See the
    vSphere Security documentation for more information.
    [root@xd-1567-5028d:~] esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-6.5.0-20180304001-standard --dry-run
    Update Result
       Message: Dryrun only, host not changed. The following installers will be applied: [BootBankInstaller]
       Reboot Required: true
       VIBs Installed: VMware_bootbank_cpu-microcode_6.5.0-1.41.7967591, VMware_bootbank_esx-base_6.5.0-1.41.7967591, VMware_bootbank_esx-tboot_6.5.0-1.41.7967591, VMware_bootbank_vsan_6.5.0-1.41.7547709, VMware_bootbank_vsanhealth_6.5.0-1.41.7547710
       VIBs Removed: VMware_bootbank_cpu-microcode_6.5.0-0.0.4564106, VMware_bootbank_esx-base_6.5.0-1.36.7388607, VMware_bootbank_esx-tboot_6.5.0-1.36.7388607, VMware_bootbank_vsan_6.5.0-1.36.7388608, VMware_bootbank_vsanhealth_6.5.0-1.36.7388609
       VIBs Skipped: VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.650.0.0.4564106, VMW_bootbank_ata-pata-amd_0.3.10-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.650.0.0.4564106, VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-via_0.3.3-2vmw.650.0.0.4564106, VMW_bootbank_block-cciss_3.6.14-10vmw.650.0.0.4564106, VMW_bootbank_char-random_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.650.0.14.5146846, VMW_bootbank_elxnet_11.1.91.0-1vmw.650.0.0.4564106, VMW_bootbank_hid-hid_1.0-3vmw.650.0.0.4564106, VMW_bootbank_i40en_1.3.1-5vmw.650.1.26.5969303, VMW_bootbank_igbn_0.1.0.0-15vmw.650.1.36.7388607, VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-devintf_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ixgben_1.4.1-2vmw.650.1.26.5969303, VMW_bootbank_lpfc_11.1.0.6-1vmw.650.0.0.4564106, VMW_bootbank_lsi-mr3_6.910.18.00-1vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt2_20.00.01.00-3vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt3_12.00.02.00-11vmw.650.0.0.4564106, VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.650.0.0.4564106, VMW_bootbank_misc-drivers_6.5.0-1.36.7388607, VMW_bootbank_mtip32xx-native_3.9.5-1vmw.650.0.0.4564106, VMW_bootbank_ne1000_0.8.0-16vmw.650.1.26.5969303, VMW_bootbank_nenic_1.0.0.2-1vmw.650.0.0.4564106, VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.650.0.0.4564106, VMW_bootbank_net-bnx2x_1.78.80.v60.12-1vmw.650.0.0.4564106, VMW_bootbank_net-cdc-ether_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.650.0.0.4564106, VMW_bootbank_net-e1000_8.0.3.1-5vmw.650.0.0.4564106, VMW_bootbank_net-e1000e_3.2.2.1-2vmw.650.0.0.4564106, VMW_bootbank_net-enic_2.1.2.38-2vmw.650.0.0.4564106, VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.650.0.0.4564106, VMW_bootbank_net-forcedeth_0.61-2vmw.650.0.0.4564106, VMW_bootbank_net-igb_5.0.5.1.1-5vmw.650.0.0.4564106, VMW_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.650.0.0.4564106, VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-nx-nic_5.0.621-5vmw.650.0.0.4564106, VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.650.0.0.4564106, VMW_bootbank_net-usbnet_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.650.0.0.4564106, VMW_bootbank_nhpsa_2.0.6-3vmw.650.0.0.4564106, VMW_bootbank_nmlx4-core_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-en_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-rdma_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx5-core_4.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_ntg3_4.1.3.0-1vmw.650.1.36.7388607, VMW_bootbank_nvme_1.2.0.32-5vmw.650.1.36.7388607, VMW_bootbank_nvmxnet3_2.0.0.23-1vmw.650.1.36.7388607, VMW_bootbank_ohci-usb-ohci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_pvscsi_0.1-1vmw.650.1.26.5969303, VMW_bootbank_qedentv_2.0.3.29-1vmw.650.0.0.4564106, VMW_bootbank_qfle3_1.0.2.7-1vmw.650.0.0.4564106, VMW_bootbank_qflge_1.1.0.3-1vmw.650.0.0.4564106, VMW_bootbank_qlnativefc_2.1.50.0-1vmw.650.1.26.5969303, VMW_bootbank_sata-ahci_3.0-26vmw.650.1.26.5969303, VMW_bootbank_sata-ata-piix_2.12-10vmw.650.0.0.4564106, VMW_bootbank_sata-sata-nv_3.5-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-promise_2.12-3vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil24_1.1-1vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil_2.3-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-svw_2.3-3vmw.650.0.0.4564106, VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.650.0.0.4564106, VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.650.0.0.4564106, VMW_bootbank_scsi-aic79xx_3.1-5vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.650.0.0.4564106, VMW_bootbank_scsi-hpsa_6.0.0.84-1vmw.650.0.0.4564106, VMW_bootbank_scsi-ips_7.12.05-4vmw.650.0.0.4564106, VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.650.0.0.4564106, VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.650.0.0.4564106, VMW_bootbank_scsi-mpt2sas_19.00.00.00-1vmw.650.0.0.4564106, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.650.0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-3-0_6.5.0-0.0.4564106, VMW_bootbank_uhci-usb-uhci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usbcore-usb_1.0-3vmw.650.1.26.5969303, VMW_bootbank_vmkata_0.1-1vmw.650.1.36.7388607, VMW_bootbank_vmkplexer-vmkplexer_6.5.0-0.0.4564106, VMW_bootbank_vmkusb_0.1-1vmw.650.1.36.7388607, VMW_bootbank_vmw-ahci_1.0.0-39vmw.650.1.26.5969303, VMW_bootbank_xhci-xhci_1.0-3vmw.650.0.0.4564106, VMware_bootbank_emulex-esx-elxnetcli_11.1.28.0-0.0.4564106, VMware_bootbank_esx-dvfilter-generic-fastpath_6.5.0-1.36.7388607, VMware_bootbank_esx-ui_1.23.0-6506686, VMware_bootbank_esx-xserver_6.5.0-0.23.5969300, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-5vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-10vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-7vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-8vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-6vmw.650.1.26.5969303, VMware_bootbank_native-misc-drivers_6.5.0-0.0.4564106, VMware_bootbank_rste_2.0.2.0088-4vmw.650.0.0.4564106, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.10-1.26.5969303, VMware_locker_tools-light_6.5.0-1.33.7273056
    [root@xd-1567-5028d:~] esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-6.5.0-20180304001-standard
    Update Result
       Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
       Reboot Required: true
       VIBs Installed: VMware_bootbank_cpu-microcode_6.5.0-1.41.7967591, VMware_bootbank_esx-base_6.5.0-1.41.7967591, VMware_bootbank_esx-tboot_6.5.0-1.41.7967591, VMware_bootbank_vsan_6.5.0-1.41.7547709, VMware_bootbank_vsanhealth_6.5.0-1.41.7547710
       VIBs Removed: VMware_bootbank_cpu-microcode_6.5.0-0.0.4564106, VMware_bootbank_esx-base_6.5.0-1.36.7388607, VMware_bootbank_esx-tboot_6.5.0-1.36.7388607, VMware_bootbank_vsan_6.5.0-1.36.7388608, VMware_bootbank_vsanhealth_6.5.0-1.36.7388609
       VIBs Skipped: VMW_bootbank_ata-libata-92_3.00.9.2-16vmw.650.0.0.4564106, VMW_bootbank_ata-pata-amd_0.3.10-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-atiixp_0.4.6-4vmw.650.0.0.4564106, VMW_bootbank_ata-pata-cmd64x_0.2.5-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-pdc2027x_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-serverworks_0.4.3-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-sil680_0.4.8-3vmw.650.0.0.4564106, VMW_bootbank_ata-pata-via_0.3.3-2vmw.650.0.0.4564106, VMW_bootbank_block-cciss_3.6.14-10vmw.650.0.0.4564106, VMW_bootbank_char-random_1.0-3vmw.650.0.0.4564106, VMW_bootbank_ehci-ehci-hcd_1.0-4vmw.650.0.14.5146846, VMW_bootbank_elxnet_11.1.91.0-1vmw.650.0.0.4564106, VMW_bootbank_hid-hid_1.0-3vmw.650.0.0.4564106, VMW_bootbank_i40en_1.3.1-5vmw.650.1.26.5969303, VMW_bootbank_igbn_0.1.0.0-15vmw.650.1.36.7388607, VMW_bootbank_ima-qla4xxx_2.02.18-1vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-devintf_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.650.0.0.4564106, VMW_bootbank_ixgben_1.4.1-2vmw.650.1.26.5969303, VMW_bootbank_lpfc_11.1.0.6-1vmw.650.0.0.4564106, VMW_bootbank_lsi-mr3_6.910.18.00-1vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt2_20.00.01.00-3vmw.650.0.0.4564106, VMW_bootbank_lsi-msgpt3_12.00.02.00-11vmw.650.0.0.4564106, VMW_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.650.0.0.4564106, VMW_bootbank_misc-drivers_6.5.0-1.36.7388607, VMW_bootbank_mtip32xx-native_3.9.5-1vmw.650.0.0.4564106, VMW_bootbank_ne1000_0.8.0-16vmw.650.1.26.5969303, VMW_bootbank_nenic_1.0.0.2-1vmw.650.0.0.4564106, VMW_bootbank_net-bnx2_2.2.4f.v60.10-2vmw.650.0.0.4564106, VMW_bootbank_net-bnx2x_1.78.80.v60.12-1vmw.650.0.0.4564106, VMW_bootbank_net-cdc-ether_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-cnic_1.78.76.v60.13-2vmw.650.0.0.4564106, VMW_bootbank_net-e1000_8.0.3.1-5vmw.650.0.0.4564106, VMW_bootbank_net-e1000e_3.2.2.1-2vmw.650.0.0.4564106, VMW_bootbank_net-enic_2.1.2.38-2vmw.650.0.0.4564106, VMW_bootbank_net-fcoe_1.0.29.9.3-7vmw.650.0.0.4564106, VMW_bootbank_net-forcedeth_0.61-2vmw.650.0.0.4564106, VMW_bootbank_net-igb_5.0.5.1.1-5vmw.650.0.0.4564106, VMW_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.650.0.0.4564106, VMW_bootbank_net-libfcoe-92_1.0.24.9.4-8vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-core_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-mlx4-en_1.9.7.0-1vmw.650.0.0.4564106, VMW_bootbank_net-nx-nic_5.0.621-5vmw.650.0.0.4564106, VMW_bootbank_net-tg3_3.131d.v60.4-2vmw.650.0.0.4564106, VMW_bootbank_net-usbnet_1.0-3vmw.650.0.0.4564106, VMW_bootbank_net-vmxnet3_1.1.3.0-3vmw.650.0.0.4564106, VMW_bootbank_nhpsa_2.0.6-3vmw.650.0.0.4564106, VMW_bootbank_nmlx4-core_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-en_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx4-rdma_3.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_nmlx5-core_4.16.0.0-1vmw.650.0.0.4564106, VMW_bootbank_ntg3_4.1.3.0-1vmw.650.1.36.7388607, VMW_bootbank_nvme_1.2.0.32-5vmw.650.1.36.7388607, VMW_bootbank_nvmxnet3_2.0.0.23-1vmw.650.1.36.7388607, VMW_bootbank_ohci-usb-ohci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_pvscsi_0.1-1vmw.650.1.26.5969303, VMW_bootbank_qedentv_2.0.3.29-1vmw.650.0.0.4564106, VMW_bootbank_qfle3_1.0.2.7-1vmw.650.0.0.4564106, VMW_bootbank_qflge_1.1.0.3-1vmw.650.0.0.4564106, VMW_bootbank_qlnativefc_2.1.50.0-1vmw.650.1.26.5969303, VMW_bootbank_sata-ahci_3.0-26vmw.650.1.26.5969303, VMW_bootbank_sata-ata-piix_2.12-10vmw.650.0.0.4564106, VMW_bootbank_sata-sata-nv_3.5-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-promise_2.12-3vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil24_1.1-1vmw.650.0.0.4564106, VMW_bootbank_sata-sata-sil_2.3-4vmw.650.0.0.4564106, VMW_bootbank_sata-sata-svw_2.3-3vmw.650.0.0.4564106, VMW_bootbank_scsi-aacraid_1.1.5.1-9vmw.650.0.0.4564106, VMW_bootbank_scsi-adp94xx_1.0.8.12-6vmw.650.0.0.4564106, VMW_bootbank_scsi-aic79xx_3.1-5vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.650.0.0.4564106, VMW_bootbank_scsi-fnic_1.5.0.45-3vmw.650.0.0.4564106, VMW_bootbank_scsi-hpsa_6.0.0.84-1vmw.650.0.0.4564106, VMW_bootbank_scsi-ips_7.12.05-4vmw.650.0.0.4564106, VMW_bootbank_scsi-iscsi-linux-92_1.0.0.2-3vmw.650.0.0.4564106, VMW_bootbank_scsi-libfc-92_1.0.40.9.3-5vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-mbox_2.20.5.1-6vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.650.0.0.4564106, VMW_bootbank_scsi-megaraid2_2.00.4-9vmw.650.0.0.4564106, VMW_bootbank_scsi-mpt2sas_19.00.00.00-1vmw.650.0.0.4564106, VMW_bootbank_scsi-mptsas_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-mptspi_4.23.01.00-10vmw.650.0.0.4564106, VMW_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.650.0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-iscsi-linux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libata-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfc-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-libfcoe-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-1-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-2-0_6.5.0-0.0.4564106, VMW_bootbank_shim-vmklinux-9-2-3-0_6.5.0-0.0.4564106, VMW_bootbank_uhci-usb-uhci_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usb-storage-usb-storage_1.0-3vmw.650.0.0.4564106, VMW_bootbank_usbcore-usb_1.0-3vmw.650.1.26.5969303, VMW_bootbank_vmkata_0.1-1vmw.650.1.36.7388607, VMW_bootbank_vmkplexer-vmkplexer_6.5.0-0.0.4564106, VMW_bootbank_vmkusb_0.1-1vmw.650.1.36.7388607, VMW_bootbank_vmw-ahci_1.0.0-39vmw.650.1.26.5969303, VMW_bootbank_xhci-xhci_1.0-3vmw.650.0.0.4564106, VMware_bootbank_emulex-esx-elxnetcli_11.1.28.0-0.0.4564106, VMware_bootbank_esx-dvfilter-generic-fastpath_6.5.0-1.36.7388607, VMware_bootbank_esx-ui_1.23.0-6506686, VMware_bootbank_esx-xserver_6.5.0-0.23.5969300, VMware_bootbank_lsu-hp-hpsa-plugin_2.0.0-5vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-10vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-7vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-8vmw.650.1.26.5969303, VMware_bootbank_lsu-lsi-mpt2sas-plugin_2.0.0-6vmw.650.1.26.5969303, VMware_bootbank_native-misc-drivers_6.5.0-0.0.4564106, VMware_bootbank_rste_2.0.2.0088-4vmw.650.0.0.4564106, VMware_bootbank_vmware-esx-esxcli-nvme-plugin_1.2.0.10-1.26.5969303, VMware_locker_tools-light_6.5.0-1.33.7273056
    [root@xd-1567-5028d:~] reboot