My VMware vSphere 6.7 Update 3 Upgrade Experience

Posted by Paul Braren on Aug 21 2019 (updated on Sep 29 2019) in

This vSphere update was officially announced 6.7 Update 3 last week, and yesterday, it became available to download. For folks with only VMUG Advantage EVALExperience with its (currently) older 6.7 Update 2
VMware-VMvisor-Installer-6.7.0.update02-13006603.x86_64.iso
they can instead use ESXCLI to upgrade which downloads and installs with one command, and doesn't require my.vmware.com credentials or software trials. This is the method of upgrading that I tested first, but not with doing things in the correct order, which was to upgrade my VCSA first!

My VCSA 6.7 Update 3 Upgrade

I encountered one issue with my VCSA upgrade using the built-in Update feature. I got the following error:
Error in method invocation Timeout happens while sending message to microservice
and until I restarted my VCSA appliance, it wouldn't let me actually proceed with the "Stage and install" upgrade. Video footage of that issue is lost (hotel room with poor internet), but I'm glad the work-around was a simple restart. Unfortunately, I didn't have a time or a mechanism to report this potential bug to get to root cause, but at least I'm documenting it here in case you also bump into this issue. Google searching for that error didn't get me any useful hits, so it appears to be a rare issue.

My ESXi 6.7 Update 3 Upgrade

I used ESXCLI on my test system and it went smoothly, and I grabbed a transcript of that SSH session so you can see each and every VIB that was updated on my Supermicro SYS-5028D-TN4T SuperServer Bundle.

2019-08-21_9-19-24 — Note the new VCSA login page. Dump Adobe Flash, HTML5 UI has 100% feature parity since 6.7 Update 1!

At least preliminarily, this upgrade seems to function just fine on the Xeon D, but I'll need some more days of testing, along with an upgrade of my primary workstation when I'm back home, to be much more comfortable with that assertion. In other words, proceed with upgrades at your own risk, and backing up your ESXi first, especially in a home lab where you likely have no official VMware support mechanism. Rolling back to when things worked might be your only option should things go wrong for you.

Please feel free to let us know how your upgrade goes by dropping a comment below, particularly if you're on a supported hardware such as a SuperServer Bundle, and try to mention what upgrade method you used.

Video

This video is a bit of a mess, with poor hotel room/laptop audio, and a rather unpolished flow. You're basically looking at me doing my first upgrade test, and working through a VAMI upgrade issue and a browser caching issue, and it's nearly unedited.

TinkerTry YouTube Channel - Aug 20 2019 - Updating to VMware vSphere 6.7 Update 3 - VCSA first using VAMI, then ESXi Xeon D host using ESXCLI

Aug 21 2019 10:10pm Update

Flush browser cache after VCSA 6.7U3 upgrade

Reports are in that others had browser caching issues after their upgrade to VCSA 6.7U3, see Matt Webb's tweet:

New 10G ixgben driver behavior on Xeon D

I got pinged by a fellow vExpert that there may be a change/problem with the behavior of the X557 10G ports on Xeon D motherboards, using 6.7U3's default (included/inbox) driver. Here's the excerpt from the 6.7u3 release notes:

ESXi670-201908210-UG

Patch Category Enhancement
Patch Severity Important
Host Reboot Required Yes
Virtual Machine Migration or Shutdown Required Yes
Affected Hardware N/A
Affected Software N/A
VIBs Included
VMW_bootbank_ixgben_1.7.1.16-1vmw.670.3.73.14320388
PRs Fixed 2315652
CVE numbers N/A
This patch updates the ixgben VIB to resolve the following issue:

The ixgben driver adds queue pairing to optimize CPU efficiency.

Testing/monitoring 6.7U3 on Xeon D now...

Aug 22 2019 Update

The person first mentioning this 10G issue on Xeon D was TinkerTry reader Jorge de la Cruz (Veeam, jorgedelacruz.uk, @jorgedlcruz), thank you Jorge, I really appreciate it, and so do my readers!

My two X557 interfaces stayed up overnight, one at 10G (on a 10G switch) and one at 1G (on a 1G switch), but I haven't really tested them beyond the physical link layer yet, seen pictured above.

For others reading this, please don't assume somebody else already reported things you run into. If you run into something with 6.7U3, you really should report it as a VMware Service Request. But if you don't have VMware support, dropping a comment below relevant articles like this one really helps everybody help each other out. Thank you!

Sep 01 2019 Update

My two X557 interfaces stayed up over the past 10 days, one at 10G (on a 10G switch) and one at 1G (on a 1G switch). This is good.

I have also now upgraded two Xeon D-1500 series systems to 6.7U3, and two Xeon D-2100 systems to 6.7U3, without incident. This is also good. That said, others didn't fare as well, with the usual assortment of issues that sometimes get in peoples way reported in some of the comments below this article.

I used ESXCLI for the first host, as described above back on Aug 21. I used VMware Update Manager to upgrade the other 3 hosts. Since I don't have DRS enabled on my cluster, I made sure sure to manually shut down all VMs before the upgrade on each node so that it could put them into Maintenance Mode for me.

Using Update Manager to upgrade VMware Tools only seemed to succeed with VMs that were powered off.

Sep 09 2019 Update

In the 8th day since I updated my 8 core host to ESXi 6.7U3, my two X557 interfaces stayed up, one at 10G (on a 10G switch) and one at 1G (on a 1G switch). This is good.

Today, I spotted that Veeam Backup & Replication 9.5 Update 4b is ready, check out the Release Notes for Veeam Backup & Replication 9.5 Update 4b:

VMware vSphere 6.7 U3 readiness based on the pre-release build. The official support will still require the full regression testing performed on the generally available (GA) build. UPDATE: vSphere 6.7 U3 GA is officially supported with Update 4b.

Sep 18 2019 Update

For my primary workstation, which is a Windows 10 1903 VM, I'm using:
Compatibility: ESXi 6.7 Update 2 and later (VM version 15)

win10multiboot-VM-USB-mappings-67U3-2019-09-19--TinkerTry — My Windows 10 VM's Host USB Device connections.

My own testing in my home lab is going well, other than an issue with audio dropping, audio via a USB audio adapter assigned to my primary Windows 10 VM "workstation," which is different than other problems I've had with audio back in 2018. I've also had some sporadic network lag, and a host disconnection. Not sure of the root cause, but I went ahead ahead with installing VMware Tools 10.3.10.12406962 that comes with 6.7U3 (using the built-in wizard that offers to upgrade my VM) to see if that makes any difference with my audio and networking. We'll see.

Worth noting this interesting comment from TinkerTry commenter esmith69 though:

Note that there is a known issue with 6.7 Update 3 and the Dell OpenManage VIB. Dell is apparently telling people to simply not put U3 in place yet. The symptoms I experienced on my R630 hosts were vMotion timeouts, VM power event timeouts, etc, although general VM performance was OK.

The workaround for now is to remove the OpenManage VIB from the command line on each host: esxcli software vib remove -n OpenManage. I guess you could also try rolling back the hosts to U2 but that wasn't an option for me.

VMware support is aware, although as of yesterday the release notes for Update 3 still didn't include any kind of warning

Sep 19 2019 Update

It seems my intermittent network and audio issues may be behind me, even though it's only been about 12 hours since the VMware Tools upgrades. Continuing to monitor. USB connections screenshot added to Sep 18th update above.

Sep 29 2019 Update

My network problems persist, but may be a little less frequent. They may have nothing to do with Update 3 itself, it's too early to say, it could just be the local laptop I'm connecting to, with some recent driver updates.

Note that Release Notes for Veeam Backup & Replication 9.5 Update 4b says the following:

VMware vSphere 6.5 U3 readiness based on the pre-release build. The official support will still require the full regression testing performed on the generally available (GA) build. UPDATE: vSphere 6.5 U3 GA is officially supported with Update 4b.

Also spotted this KB 74607:

Excessive Hardware health alarms being triggered for "Sensor -1 type" on ESXi hosts running vSphere 6.7 U3 (74607)
Last Updated: 9/11/2019

Purpose
This article provides information and workaround for the hardware health alarms getting triggered (for sensor -1 type) on ESX hosts running vSphere 6.7U3 build-14320388.
...

The article goes on to explain it's not serious, and there is a workaround.