VMware vCenter Server 7.0.0d resolves issue with 7.0.0c high CPU usage in VCSA VM

Posted by Paul Braren on Aug 26 2020 in
  • ESXi
  • HowTo
  • Virtualization
  • vSphere7
    • IMG_4580--TinkerTry.PNG
    You can probably figure out that I upgraded from 7.0.0b to 7.0.0c around August 20th, and gave it 4 vCPUs. Note the daily spikes, presumably backups, I haven't yet verified. Also note that this screenshot is vSphere Client on iOS Safari, with no certificate errors.
    vSphere-Mobile-Performance-of-VCSA-7.0.0c-to-7.0.0d--TinkerTry-2020-08-26
    VMware Fling called vSphere Mobile sure comes in handy for a quick look at performance over the last month, it's pretty easy to spot when VCSA 7.0.0c was running.

    I received some interesting comments from Brian S., in response to my DC VMUG announcement - Washington DC VMUG Presentation on Aug 20 2020 - TinkerTry home lab’s VMware vSphere 7.0 upgrade challenges, successes, and lessons learned.. He tipped me off to an issue I had started to notice in my home lab, so I added some words of caution to my recent upgrade to 7.0c video description, giving suitable warnings about possible high CPU utilization. Brian also referred me to this VMware VMTN post:

    VMTN

    with the temporary workaround nicely documented by LaurensvanDujin back on Aug 03 2020, using VAMI to stop the Workload Control Plane service. This action immediately lowers the CPU back to normal levels, but also prevents the host from being able to enter into Maintenance Mode. Also, this workaround doesn't persist through VCSA reboots, so your VCSA CPU use will just spike again after a restart of VCSA.

    If you're on a host like an Intel NUC with just 4 cores, you could see 100% CPU utilization. That's a significant bug, not the first one related to CPUs this year, and it's unfortunate this made its way past QA testing. Gladly, we have a community looking out for one-another, read onward.

    Today, Brian S.'s new reply tipped me off that the new VCSA version 7.0.0.10700 is now out, aka 7.0.0d, which is also now appended to the same VMTN post. Thank you Brian S.!

    By the way, while I don't know Brian personally, in case you're looking for a consultant, Brian S. is currently seeking new opportunities.

    VMware vCenter Server 7.0.0d Release Notes

    vsphere-vcenter-server-700d-release-notes

    Release Notes:

    ...
    Resolved Issues
    The resolved issues are grouped as follows.

    Miscellaneous Issues
    Miscellaneous Issues

    • After an upgrade to vCenter Server 7.0.0c, you see high CPU usage

    After you upgrade your vCenter Server system to vCenter Server 7.0.0c, CPU usage continuously stays high. On a single core, CPU usage might spike up to 100% for hours. The Workload Control Plane service causes the issue, even if you do not have Workload Management enabled in your environment.

    This issue is resolved in this release.
    ...

    Strangely, searching for 7.0.0d on Google comes up nearly empty at the moment, but here's the download page for the ISO:

    Name: VMware-VCSA-all-7.0.0-16749653.iso
    Release Date: 2020-08-25
    Build Number: 16749653

    or just use VAMI's built-in Update feature:

    VAMIpull
    Using VAMI to go from 7.0.0c to 7.0.0d is this simple, if your VCSA has an internet connection.

    Aug 26 2020 Update - 08:29am Eastern

    I have just completed my VCSA 7.0.0d update process, it went smoothly, and my VCSA is now showing normal CPU utilization levels. I may even be able to return to 2 vCPUs assigned to this VM. I've just published a video of the CPU problem followed by a straight-forward Update to 7.0.0d. I also moved from 4 vCPU down to 2 vCPU, and from 12GB of RAM to 10GB of RAM, and 7.0.0d seems to be running quite well with those VM settings in my home lab with 4 hosts / 32 cores, and few dozen VMs that are mostly powered off.

    Aug 26 2020 Update - 10:00pm Eastern

    I tried moving from 4v CPUs down to 2, and 12GB of RAM down to 10GB which required a reboot. Afterward, I got these warnings when logging back in with vSphere Client. Even though the steady-state RAM usage seems to hover around only 4GB of RAM, I'm happy to simply get rid of the error by giving it back the 12GB it apparently wants. I've deleted the 7.0.0c snapshot, and am ready to move on to other projects now.

    VCSA-7.0.0d-warning-about-Memory-Exhaustion-on-vcsa--TinkerTry
    Seems 10GB just isn't enough, moved it back to 12GB, and all is well again.

    Video

    TinkerTry - Aug 26 2020 - Using VAMI Update to VMware vCenter Server 7.0.0d resolves issue w/ 7.0.0c high CPU usage in VCSA VM
    TinkerTry - Aug 19 2020 - Successful and simple VAMI Update from VMware vSphere 7 0.0b to 7.0.0c

    See also at TinkerTry

    All vSphere 7 articles.

    All vSphere 7 videos.

    See also

    strange-thermal-issue-after-update-to-esxi-7-0b
    • Strange thermal Issue after Update to ESXi 7.0b
      Jul 2020 by Michael Schroeder at ELASTICSKY

      ...
      After I’ve monitored one host with the new BIOS for a while and nothing strange happened, I decided to flash all of my E300-9D host with the new BIOS. Then I fired up the vSAN cluster and applied latest ESXi patches. These patches used to cause thermal issues in combination with the old BIOS 1.3 but now with 1.3a CPU temperature remaind at a normal level.
      ...

    id1459517032
    • vSphere Mobile Client
      App Store Preview by VMware, Inc., for iPhone or iPad, seen also here at VMware Flings including Android version. It's only tested with 6.5, but the July 23 2020 v1.13.2 version seems to work fine with VCSA 7.0.0d.

    vcsa_700d_available_to_fix_high_cpu

    vmware-vcenter-7-0-0c-high-cpu-usage

    TinkerTry - PCs, EVs, home tech, efficiency and more, including virtualization. My opinions here, not my employer's.

    Paul Braren, MCSE in 1998, VCP #2681 in 2005, vExpert 2014 to 2024, Veeam Vanguard Emeritus, 2015 to 2019.

    Support TinkerTry

    PayPal | Patreon | VMware Store

    After 6 successful years testing then shipping well over 1,000 Xeon D Bundles, Wiredzone had to stop selling them in mid-2021 due to cost, supply, and logistics challenges. So far, Xeon D-1700/2700 (Ice Lake D) was a minor refresh for 2023, with Xeon D-1800/2800 (Granite Rapids D) refresh looking a little better for 2024. More cores, but still just PCIe Gen4 and DDR4. Looking ahead, I'm glad it's Pat Gelsinger at Intel's helm. I'm also grateful to have had the honor of working at VMware when Pat was the CIO there. I'll leave it at that, given the whole Broadcom thing.

    Easily update VMware ESXi at TinkerTry.com/easy-update-to-latest-esxi