How to easily update your VMware vCenter Server Appliance to VCSA 6.5

Posted by Paul Braren on Nov 20 2016 (updated on Mar 31 2018) in
  • ESXi
  • Virtualization
  • HowTo
  • HomeLab
    • 8 Comments

    Important Update - On Mar 20 2018, VMware VMSA-2018-0004.3 announced that CVE-2017-5715 (Spectre-2) mitigation is now included in the latest patch that you should be using instead of the older patch featured in the original article below. You'll find the newer article here:

    Article below as it originally appeared.

    vCenter Server Appliance 6.5 Release Notes 15 NOV 2016 | Build 4602587

    Warning - vCenter/VCSA 6.x should be upgraded to 6.5 before upgrading your host(s) to ESXi 6.5!

    Different approaches

    The TinkerTry'd method that works reliably in my home lab just isn't ready yet, I'll update this article as time permits. A serious bug, where I'm unable to get a fresh-installed VCSA 6.5 to survive a simple reboot, showing blank inventory in vSphere Web Client and vSphere Client. This is slowing me down a bit, and was already reported here:

    The gist of the new slick upgrade method is that Windows users simply double-click on the downloaded file named:
    VMware-VCSA-all-6.5.0-4602587.iso
    to mount it as an ISO, then you launch
    E:\vcsa-ui-installer\win32\installer.exe
    to kick off the wizard style Upgrade routine that looks like the image below.

    2016-11-28_10-51-29

    You can see the complete process with screenshots at

    I can say that it's not quite as simple as it was when we moved from 6.0 to 6.0U2, jump below to the end of the story below to learn why that is. Or have a little fun first.

    Here's how we used to do this, when moving up to just a minor point-release:

    software-packages install --url --acceptEulas

    That method doesn't seem to realize there's major new upgrade to 6.5 out there for the asking:

    software-packages-command-doesnt-work

    How about a little creative alteration to the command, adding the --url parameter that allows me to specify the remote repository location at VMware? I've not found anybody else trying this (with good reason, keep reading). Why not give it a go, to see what happens, to learn how VMware has anticipated this sort of silliness:

    software-packages install --url https://vapp-updates.vmware.com/vai-catalog/valm/vmw/8d167796-34d5-4899-be0a-6daade4005a3/6.5.0.5100.latest/ --acceptEulas

    Nice try, but no go, as it runs the pre-stage.py script, errors out with:
    Repository does not contain updates for this product.
    as shown below:

    software-packages-command-with-url-doesnt-work

    You can read all about how it was done so easily before vSphere 6.5, in my original post:

    easy-upgrade-to-vcsa-60u2
    SolarWinds-VM-Monitor-doesnt-know-Photon-OS

    So that's enough tinkering, time to move onward.

    It's more of a migrate than an update

    Learn more about Photon OS at VMware:

    Photon OS™ is a minimal Linux container host, optimized to run on VMware platforms.

    upgrade-vcsa-55-60-to-vcsa65
    Will I get Photon OS when I upgrade my VCSA 5.5/6.0 to VCSA 6.5? by William Lam

    and check out the Photon OS wiki at github, and William Lam's Will I get Photon OS when I upgrade my VCSA 5.5/6.0 to VCSA 6.5?, that explains:

    VCSA upgrades are "Migration" based upgrades and has been since the very first release of the VCSA in vSphere 5.0.

    So we're being moved from SuSE/SLES to Photon OS for this major upgrade, I just haven't tested it all out yet. For now, as always, be sure to back up your VCSA first (NAKIVO or Veeam or many other options), then follow along at:

    JonKensey
    Photon-version-1-0-splash-screen

    Nov 29 2016 Update

    This bug, where I'm unable to get a fresh-installed VCSSA 6.5 to survive a simple reboot, is still an issue. What happens is that it installs fine and works great, until VCSA is restarted, along with the ESXi host. All I see is a blank inventory in vSphere Web Client, and in (HTML5) vSphere Client. This is slowing me down a bit, and was already reported here:

    When pointing my browser to the vcsa appliance like this:
    https://vcsa
    below you'll see the error I sometimes get:

    503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007feeac069480] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

    Dec 21 2016 Update

    I've resolved this VCSA issue, reporting exactly how in the same community post. Yay! This article is now complete.

    See also at TinkerTry

    See also

    Emad-Younis-blog
    JonKensey
    VMware vSphere 6.5 taskbar shortcuts can make your Chrome browser UIs look like native Windows apps!

    All Comments on This Article (8)

    Paul Braren | TinkerTry.com

    I'm not seeing a screenshot, but in general, if you ssh into your VCSA, then type
    shell
    then type
    nslookup vmware.com
    if you get an answer like this:
    Server: 10.10.1.1
    Address: 10.10.1.1#53

    Non-authoritative answer:
    Name: vmware.com
    Address: 107.154.106.19
    Name: vmware.com
    Address: 107.154.105.19
    your update from the internet directly to your VCSA appliance should work. Instead, you are receiving some sort of DNS errors.

    My VCSA is DHCP fed a lease, and told to use my router 10.10.1.1 for DNS, You may be configured quite differently, or hard coded. You can fix your VCSA DNS settings through VAMI, subtituting the name or IP of your VCSA for mine which is named VCSA, visit here:
    https://vcsa.lab.local:5480/#/appliance/networking?locale=en
    logging as root, then click on the Manage button, typing in your DNS or something like Google DNS at 8.8.8.8, then try to follow the update steps in this article again.

    https://uploads.disquscdn.com/images/a8c3b34abd9be48feb1a496240862977fcdff8bea1c2238037823ca0dfd35960.png

    Wes

    Another failed VCSA 6.0U2 >> VCSA 6.5 migration here. I had done all the reading and was aware of the issues some were having, but having a small environment with DNS and a nearly perfect 3 year track record of VMware upgrades, I decided to try anyway.

    It appears that 6.5 has installed and migrated, but no, the reboot issue. The main reason I was rebooting in the first place is because during the migration, I could not select any environment smaller than Medium. Previously we were on Tiny. The 8 vCPUs on 8 sockets and 24GB of RAM from the Medium deployment size was killing my vcsa/hmtl5 fling/vma/Nakivo ESXi host, which is a 4C/8T Ivy Xeon with 32GB. So i tried to cut the virtual hardware down to 4 CPUs and 16GB of RAM for the new, Medium-sized Photon OS VCSA 6.5. That's when the reboot issues started.

    I tried LCWSteve's Group Policy "Log on as a batch job" instructions. Easy enough, but had no affect. I probably wouldn't leave a system name blank even if it told me to, so the field that Paul references was filled out (I count Active Directory DNS as dynamic, and I know that's not really DDNS as we normally think of it). I figured I had all the research-based answers to the problems, so after unsuccessfully trying to find second instances of vSphere and SSO in VCSA log files (there was only one instance, so nothing to unregister), and completely unable to fix the {https://:443/sdk} plus empty inventory issue, I started over.

    I actually fired back up the 6.0U2 VCSA (vcsa6) and moved the first migration attempt (vcsa65) to another datastore, and started migrating again (vcsa-65). Ultimately, this had the same result, but it must have hammered the 480GB Intel SSD 730 (which is dedicated to VCSA and nothing else) because from migration attempt #2 forward, I couldn't get more than 1 MB/s of write speed on the SSD. Deploying the appliance the 2nd time around took about 4 hours, with the 81% step (formatting storage) taking at least 2 hours. I couldn't get network transfers greater than 17 Mb/s when deploying from the same workstation that earlier was maxing out GbE.

    The result with the second VCSA 6.5 migration was Initially successful, as well, once it finished overnight. Knowing the impending reboot issue, I backed up everything with Nakivo. When things went awry later as I knew they would, I had backups of every appliance.

    However, with vSphere down, it's hard to get Nakivo to do anything useful. I found Vladan's excellent post, http://www.vladan.fr/how-to-recover-a-vcenter-vm-if-vcenter-is-down-with-nakivo/ very helpful, and it's true - I would not have thought of recovering a VM with that method. Make a temporary Nakivo install, connect it to the existing repo and transporter, and pull the VM backups "through" the temporary Nakivo install from the main install. Once setup, this works really well, by the way. Problem is, after recovering the VM, it has to be powered on! Oh no! So back to square on and a whole lot of effort, config, and copying for nothing.

    I decided to scrap the upgrade. I have 2 hosts and maybe 11 VM's, so I deployed a FRESH VCSA 6.5, having used the old C# client to disassociate the hosts from vCenter. Once I went through the whole AD-join and SSO process again, my small vSphere is back. And what happened the first night it was back? The host apparently froze - hard lock, no PSOD, nothing. Frozen at unresponsive, black DCUI screen. Even SuperMicro IPMI couldn't power reset the machine. Only a Power Off and Power On command would do it. (For the record, that would be instability event #1 for either host in its lifetime, and I'm still reluctant to blame it on hardware, although watching very closely.)

    Paul Braren | TinkerTry.com

    Very interesting report, thanks for detailing it, will forward this along to a VMware contact. Not saying I can somehow help with any kind of free support, just saying that I may be able to get this informal report over to somebody at VMware who might know the right people. They can add your story as another data point/field report, in addition to my own experience here:
    https://www.youtube.com/watch?v=gjls1PcxiWY
    that I've since resolved:
    https://communities.vmware.com/thread/547345?sr=stream

    T Technik

    I also had some "fun" with the vCSA 6.5 upgrade. I was mainly running against two problems:

    1) Source vCSA has to have SSH enabled (not sure if I overlooked it in the vm documents), but I had it disabled before and then the whole procedure failed.

    2) I was hitting the same issue already descibed by Paul and others, that the vCSA web or backend part was not coming up again after restart. I also remember .. that I was playing with that system name (where it states something about DDNS).
    Sadly .. it was still somehow not working well (maybe due to other things being wrong).. so I did a fresh install and it seems to be fine right now.

    TomasF

    Hi Paul,
    Thanks for forwarding it and getting back to me. I wish I did a better job of noting down the timeline and issues I ran into. But yes I'm confident my DNS was working properly. I say "was" because the issues came up very late in the migration process, after the source server had been shutdown, and the script had registered the appliance in AD with the same name, effectively breaking the trust between the old Windows server (domain member) and AD. This of course results in a number of things that need to be sorted out after restoring the original vCenter server. I'll double check DNS before I try again, but the output from the migration part of the script that you run on the source system, outputs what one would expect in terms of fqdn etc.
    If I decide to give it another try I'll be sure to post to the Community Forum. I did search while I was battling with it, but didn't find any issues similar to mine at the time.
    Thanks again :)

    Paul Braren | TinkerTry.com

    Thank you for your feedback, it's always good to find out what folks are actually running into out there. Please don't be offended when I ask a question, and I know this is a long shot, given that your description seems to indicate you get pretty far along in the process.

    Is your DNS set up correctly, with FQDN and reverse configured?
    https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2115142
    https://communities.vmware.com/thread/516895?start=0
    https://pubs.vmware.com/vsphere-65/index.jsp?topic=%2Fcom.vmware.vsphere.vcsa.doc%2FGUID-56C3BA9A-234E-4D81-A4BC-E2A37892A854.html

    I appreciate your candid feedback, and I hear you. I will forward it along to my VMware contacts, who will (hopefully ;-) appreciate such honesty.

    I would also recommend you consider posting to the VMware Community Forum:
    https://communities.vmware.com/community/vmtn/vsphere/content

    TomasF

    I hope upgrading from an existing appliance is easier than what I've encountered this weekend. I decided to try to upgrade my home lab, but was unable to migrate my Windows install of 6.0 to appliance, or to do an inplace upgrade of the Windows install. For the migration I ran into a lot of issues, but it seems it finally stranded completely on starting the inventory service at the very end of the process.
    I then gave up and decided to continue using the windows installation instead, but that upgrade also fails at the very end when it fails to start the lifecycle manager API service. I guess I'll have to wait for an update.
    The sad thing is this is what I've come to expect from VMware of late :(

    Mahmood

    Can't wait for it :)