My VMware vSphere 6.7 Update 3 Upgrade Experience

Posted by Paul Braren on Aug 21 2019 (updated on Sep 19 2019) in
  • ESXi
  • HowTo
  • HomeLab
  • Virtualization
  • This vSphere update was officially announced 6.7 Update 3 last week, and yesterday, it became available to download. For folks with only VMUG Advantage EVALExperience with its (currently) older 6.7 Update 2
    VMware-VMvisor-Installer-6.7.0.update02-13006603.x86_64.iso
    they can instead use ESXCLI to upgrade which downloads and installs with one command, and doesn't require my.vmware.com credentials or software trials. This is the method of upgrading that I tested first, but not with doing things in the correct order, which was to upgrade my VCSA first!

    My VCSA 6.7 Update 3 Upgrade

    I encountered one issue with my VCSA upgrade using the built-in Update feature. I got the following error:
    Error in method invocation Timeout happens while sending message to microservice
    and until I restarted my VCSA appliance, it wouldn't let me actually proceed with the "Stage and install" upgrade. Video footage of that issue is lost (hotel room with poor internet), but I'm glad the work-around was a simple restart. Unfortunately, I didn't have a time or a mechanism to report this potential bug to get to root cause, but at least I'm documenting it here in case you also bump into this issue. Google searching for that error didn't get me any useful hits, so it appears to be a rare issue.

    My ESXi 6.7 Update 3 Upgrade

    I used ESXCLI on my test system and it went smoothly, and I grabbed a transcript of that SSH session so you can see each and every VIB that was updated on my Supermicro SYS-5028D-TN4T SuperServer Bundle.

    2019-08-21_9-19-24
    Note the new VCSA login page. Dump Adobe Flash, HTML5 UI has 100% feature parity since 6.7 Update 1!

    At least preliminarily, this upgrade seems to function just fine on the Xeon D, but I'll need some more days of testing, along with an upgrade of my primary workstation when I'm back home, to be much more comfortable with that assertion. In other words, proceed with upgrades at your own risk, and backing up your ESXi first, especially in a home lab where you likely have no official VMware support mechanism. Rolling back to when things worked might be your only option should things go wrong for you.

    Share your upgrade experience

    Please feel free to let us know how your upgrade goes by dropping a comment below, particularly if you're on a supported hardware such as a SuperServer Bundle, and try to mention what upgrade method you used.

    Video

    This video is a bit of a mess, with poor hotel room/laptop audio, and a rather unpolished flow. You're basically looking at me doing my first upgrade test, and working through a VAMI upgrade issue and a browser caching issue, and it's nearly unedited.

    TinkerTry YouTube Channel - Aug 20 2019 - Updating to VMware vSphere 6.7 Update 3 - VCSA first using VAMI, then ESXi Xeon D host using ESXCLI

    Aug 21 2019 10:10pm Update

    Flush browser cache after VCSA 6.7U3 upgrade

    Reports are in that others had browser caching issues after their upgrade to VCSA 6.7U3, see Matt Webb's tweet:

    1164347065566023680

    New 10G ixgben driver behavior on Xeon D

    I got pinged by a fellow vExpert that there may be a change/problem with the behavior of the X557 10G ports on Xeon D motherboards, using 6.7U3's default (included/inbox) driver. Here's the excerpt from the 6.7u3 release notes:

    ESXi670-201908210-UG

    Patch Category Enhancement
    Patch Severity Important
    Host Reboot Required Yes
    Virtual Machine Migration or Shutdown Required Yes
    Affected Hardware N/A
    Affected Software N/A
    VIBs Included
    VMW_bootbank_ixgben_1.7.1.16-1vmw.670.3.73.14320388
    PRs Fixed 2315652
    CVE numbers N/A
    This patch updates the ixgben VIB to resolve the following issue:

    • The ixgben driver adds queue pairing to optimize CPU efficiency.

    Testing/monitoring 6.7U3 on Xeon D now...

    2019-08-21_22-13-54
    ESXi Host Client Physical NICs view. Note I've set link speeds manually, vmnic2 on a 10G switch, vmnic3 on a 1G switch. Automatic negotiation didn't work, further testing needed.
    2019-08-21_22-07-00
    vSphere Client Physical adapters view. Note I've set link speeds manually, vmnic2 on a 10G switch, vmnic3 on a 1G switch. Automatic negotiation didn't work, further testing needed.

    Aug 22 2019 Update

    The person first mentioning this 10G issue on Xeon D was TinkerTry reader Jorge de la Cruz (Veeam, jorgedelacruz.uk, @jorgedlcruz), thank you Jorge, I really appreciate it, and so do my readers!

    My two X557 interfaces stayed up overnight, one at 10G (on a 10G switch) and one at 1G (on a 1G switch), but I haven't really tested them beyond the physical link layer yet, seen pictured above.

    For others reading this, please don't assume somebody else already reported things you run into. If you run into something with 6.7U3, you really should report it as a VMware Service Request. But if you don't have VMware support, dropping a comment below relevant articles like this one really helps everybody help each other out. Thank you!


    Sep 01 2019 Update

    My two X557 interfaces stayed up over the past 10 days, one at 10G (on a 10G switch) and one at 1G (on a 1G switch). This is good.

    I have also now upgraded two Xeon D-1500 series systems to 6.7U3, and two Xeon D-2100 systems to 6.7U3, without incident. This is also good. That said, others didn't fare as well, with the usual assortment of issues that sometimes get in peoples way reported in some of the comments below this article.

    I used ESXCLI for the first host, as described above back on Aug 21. I used VMware Update Manager to upgrade the other 3 hosts. Since I don't have DRS enabled on my cluster, I made sure sure to manually shut down all VMs before the upgrade on each node so that it could put them into Maintenance Mode for me.

    Using Update Manager to upgrade VMware Tools only seemed to succeed with VMs that were powered off.


    Sep 09 2019 Update

    In the 8th day since I updated my 8 core host to ESXi 6.7U3, my two X557 interfaces stayed up, one at 10G (on a 10G switch) and one at 1G (on a 1G switch). This is good.

    kb2970

    Today, I spotted that Veeam Backup & Replication 9.5 Update 4b is ready, check out the Release Notes for Veeam Backup & Replication 9.5 Update 4b:

    VMware vSphere 6.7 U3 readiness based on the pre-release build. The official support will still require the full regression testing performed on the generally available (GA) build. UPDATE: vSphere 6.7 U3 GA is officially supported with Update 4b.


    Sep 18 2019 Update

    For my primary workstation, which is a Windows 10 1903 VM, I'm using:
    Compatibility: ESXi 6.7 Update 2 and later (VM version 15)

    win10multiboot-VM-USB-mappings-67U3-2019-09-19--TinkerTry
    My Windows 10 VM's Host USB Device connections.

    My own testing in my home lab is going well, other than an issue with audio dropping, audio via a USB audio adapter assigned to my primary Windows 10 VM "workstation," which is different than other problems I've had with audio back in 2018. I've also had some sporadic network lag, and a host disconnection. Not sure of the root cause, but I went ahead ahead with installing VMware Tools 10.3.10.12406962 that comes with 6.7U3 (using the built-in wizard that offers to upgrade my VM) to see if that makes any difference with my audio and networking. We'll see.

    Worth noting this interesting comment from TinkerTry commenter esmith69 though:

    Note that there is a known issue with 6.7 Update 3 and the Dell OpenManage VIB. Dell is apparently telling people to simply not put U3 in place yet. The symptoms I experienced on my R630 hosts were vMotion timeouts, VM power event timeouts, etc, although general VM performance was OK.

    The workaround for now is to remove the OpenManage VIB from the command line on each host: esxcli software vib remove -n OpenManage. I guess you could also try rolling back the hosts to U2 but that wasn't an option for me.

    VMware support is aware, although as of yesterday the release notes for Update 3 still didn't include any kind of warning

    See also this warning:

    • Alert: Possible Issues with vSphere 6.7U3
      Sep 05 2019 by JefNew1213 at spiceworks

      This is an unofficial relay of information from a VMware TAM Advisory Customer Alert.

      Issue: vSphere 6.7 U3 Issues Reported in the Field

      Synopsis: We have been receiving feedback from the field with customers running into issues when upgrading to vSphere 6.7 U3 (VC and ESXi). Multiple reports from GSS are identifying issues with vSAN, vMotion, and general slowness (communication) from both vCenter and ESXi.


    Sep 19 2019 Update

    It seems my intermittent network and audio issues may be behind me, even though it's only been about 12 hours since the VMware Tools upgrades. Continuing to monitor. USB connections screenshot added to Sep 18th update above.


    See also at TinkerTry

    easy-update-to-latest-esxi

    vsphere-67u3-download

    vsphere-67u3-announced

    See also

    vsphere