How to easily install VMware ESXi 6.7 on an Intel Xeon D Supermicro SuperServer
This is a basic overview document, highlighting the steps that I walk you right through in the accompanying video below. The video features my live recording done on the very day vSphere 6.7 was announced and released, so it's a bit rough around the edges and completely unrehearsed. All that aside, I'm delighted to say that it went well. Like, really well. The Xeon D-1500's included Intel I350 1GbE RJ45 network connections seems to work fine, and for the first time ever, the Xeon D-1500's Intel X552/X557 10GbE RJ45 network connections seem to work fine as well. Even better, as I was promised months ago, the fix for the inaccurate RPM and temperature readings has been incorporated too, no more kludgey "fix"!
Following along, you will learn some stuff even if you've done this before, as I do my usual voice-over, and side-step any dependencies on crummy Java for the Out of Band Management / iKVM install using just HTML5 and a network share.
The install and initial configuration procedure for ESXi has always been quite straight forward, but this detailed step-by-step will be extra helpful if you've never installed ESXi recently, and/or it's your your first time using the HTML5 iKVM and the vSphere Host Client, which is also HTML5.
Cautiously optimistic, but by far the easiest install yet!
SuperServer Bundles owners aren't likely to be asking me questions about how to install ESXi 6.7 on this system because it's now become quite simple, and this is great! The days of fiddling with CDs and DVDs burners are well behind us, let's just hope the X557 woes that some 12 core Xeon D owners experience, myself included, are also behind us. Only time, and a whole lot more Tinkering will tell! The next system I'll be test upgrading is my expermental Xeon D-1567 SuperServer Workstation that I use as my primary system, used to write hundreds of articles and videos, like this one, and to run vSphere, simultaneously.
You should keep in mind that this is just day one, release day. It could be a while before we see what network driver & firmware combination is recommended for this system, and which solid VM backup solutions offers full compatibility. So this is really just an initial test. I'll add updates below this article.
Prerequisites
- a workstation or laptop with a browser and an OS, such as Linux or Windows
- a Supermicro SuperServer with an Intel Xeon D-1500 processor, such as the Xeon D-1540 based SYS-5028D-TN4T Bundle 2 featured in the video below
- the ISO file you'll be using to install from, already downloaded from VMware, as explained here, with the one file you need,
VMware-VMvisor-Installer-6.7.0-8169922.x86_64.iso
as featured in the video below - a USB thumb drive to install to, such as the SanDisk Ultra Fit featured in the video below, included with any SuperServer Bundle
- at least one available hard drive or SSD, for VMware's VMFS filesystem where your VMs will live, such as an insanely fast Samsung 950 M.2 NVMe SSD
Stuff that's not required
since you have that nifty IPMI management feature, you don't need
- a monitor attached directly to the server
- a keyboard attached directly to the server
- a mouse attached directly to the server, see also
Little secret those new to virtualization often miss - ESXi 6.0 continues to be mostly headless, just as it was for all prior VMware hypervisor releases
Step-by-step
- use your browser to connect to your SuperServer's IPMI management features using your browser (the IP address that you point your browser to is displayed if you temporarily connect a VGA monitor, shortly after power on)
- launch the Remote Console feature, aka, iKVM, but use the HTML5 iKVM and follow along at my detailed article, How to map a network share to boot from ISO, moving Supermicro Java iKVM to browser-only HTML5 iKVM.
- enter BIOS Configuration and configure it for UEFI, as described in detail here, note, if you bought a SuperServer Bundle, you can skip this step, since you have a turn-key solution that's ready for ESXi as-is
- click the Power Control to restart the server
- press F11 to invoke Boot Menu
- from the Please select boot device menu, select UEFI: SanDisk then press Enter
- proceed with a normal ESX installation
- from the Select a Disk to Install or Upgrade menu, choose
SanDisk Ultra Fit (mpx.vmhba32:C0:T0:L0) (or similar) - when prompted to Remove the installation disc before rebooting, click the iKVM Virtual Media / Virtual Storage menu option, then click the Plug Out button then click OK
- you may now hit Enter to Reboot
- once the newly installed ESXi is booted off USB, it will show you the hostname or IP address that you can point your browser to, then type root and the password you configured during install
- now that you're authenticated, click on the Open the VMware Host Client seen at the top left of the VMware ESXi Welcome page, which launches the new HTML5 interface of VMware's future, all the basic features needed to format a datastore and create your first VMs are there for you
The rest of the video goes through me setting up the clock/NTP.
VCSA install and configuration is more involved, and will need to wait for another day.
Step-by-step Video
Apr 18 2018 Update
Note that the popular Supermicro SuperServer SYS-5028D-TN4T is on the VMware Compatibility Guide for 6.7 already, very nice! A lot faster than it took to get 6.5 listed ;), which is a good thing, a very good thing.
See also an interesting conversation here at TinkerTry, talking about CBT and backup products like Veeam and NAKIVO vSphere 6.7 readiness.
May 09 2018 Update
I managed to not mention that the awesomeness of not having to use Oracle Java or Adobe Flash for any aspect of bringing vSphere 6.7 up in your home lab. This is something we've all been waiting for, and the news gets even better soon!
- Fully Featured HTML5-based vSphere Client Coming in Fall 2018
by Himanshu Singh at VMware vSphere BlogThe vSphere Client is a tool you use every day to manage your virtual infrastructure. You’ve told us what you want to see improved and that you want progress to be prioritized. We’ve been listening and acting on your feedback and we’d like to share the progress that’s been made, as well as our plans moving forward.
Many of you have been asking when will the vSphere Client become fully featured. We are targeting to deliver to you a fully featured version of the vSphere Client later this year, in Fall 2018.
See also at TinkerTry
- How to easily update your VMware ESXi Hypervisor to the latest version with one ESXCLI command
Aug 14 2018
- VMware vSphere 6.7 featuring vSAN 6.7 released!
Apr 17 2018
All Comments on This Article (24)
Pretty useful article. I have just installed Vmware in my home lab (Supermicro SYS-E300-8D) today, using the instructions from it. Thanks
No, I had to go back to 6.5 to get the link working again, and even then I had to install the newer driver. I never received a response on here from either of my posts about this issue.
Jeff, or anyone else with the Intel D-1500 SFP+ nics? Did you get them working under ESXi 6.7?
I was about to purchase a X10SDV-7TP4F from wiredzone.
I've installed this driver on my X10SDV-7TP4F and I still can't get either of the SFP+ links to come up, connected via DAC to Netgear XS708Ev2, which was working under ESXi 6.0, with the additional drivers
Maybe an issue related to the specified boot order in the BIOS: are you booting from USB, and is that device listed first? Easier if BIOS is set to UEFI mode rather than DUAL, since the USB device will then show only once.
I tried this procedure on a E200-9B (not in HCL) and have had mostly good success. There are a couple of differences, but my main issue is upon boot after installation, there system always seems to only go to a UEFI shell instead of fully booting in ESXi. Any hints or help are appreciated.
Greetings from seat 33F in Dulles. I have reported this thread, and am looking into this, it could just be old web page Supermicro forgot to update. I agree that his is frustrating. Also feel free to reach me via email at https://TinkerTry.com/contact#email directly is you have details you’d like to share, including where you bought the system(s) and what config (I’ll try to help best effort regardless), and Supermicro SM#s and/or VMware SR#s.
Fantastic, thank you for sharing that great news , so glad to hear that!
I have some great news! In order to find a solution for the X552 networking issue, I was looking for new drivers on VMware compatibility pages. I just found a version 1.7.1 which seems to be new and decided to test it. Luckly, it seems that problem is fixed on this driver version. Here are the links:
Compatibility:
https://www.vmware.com/resources/compatibility/search.php?deviceCategory=io&productid=41982&deviceCategory=io&details=1&partner=46&releases=369&keyword=X552&deviceTypes=6&page=1&display_interval=500&sortColumn=Partner&sortOrder=Asc&b=1525983800129
List of drivers for Device ID 0x15ad (Onboard 10GE on my Supermicro Xeon D-1521 has this DID. I just checked on ESXi):
https://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=39968&deviceCategory=io&details=1&partner=46&releases=369&keyword=X552&deviceTypes=6&page=1&display_interval=500&sortColumn=Partner&sortOrder=Asc
Downloaded and tested ixgben version 1.7.1
https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESX67-INTEL-NIC-IXGBEN-171&productId=742
Updated my VMNIC Speed/Duplex config to Auto again:
esxcli network nic set --auto -n vmnic0
esxcli network nic set --auto -n vmnic1
Rebooted and now seems to be working fine. I would be glad if someone may test and comment as well.
Cheers, Bruno
Official answer: Supermicro cannot support my request because Supermicro didn't tested/validated this VMware ESXi version.
Supported OS are listed here:
https://www.supermicro.com/products/motherboard/Xeon/D/X10SDV-4C-TLN2F.cfm (OS Compatibility)
https://www.supermicro.com/support/resources/OS/Broadwell-DE.cfm
So, I guess the subjacent message is:
"If you want to get Supermicro Support, you should be running a very old Vmware ESXi 6.0 version."
It is ridiculous, so I guess VMware will say they won't support specific Motherboards, as they only have entire Systems from Supermicro (not individual motherboards).
So, only way to make them solve this would be if you could replicate the problem with your Supermicro tower server.
Another hipcup I found this week was that: If I disconnect the ethernet cable and reconnect again (when system is already powered up), nic will not come up (even with port speed set to Gigabit), unless I go to ESXcli and administratively shutdown and enable the port again.
So, I would need a help from someone with an actual Supermicro Server to replicate the problem and raise the case with VMware. (I guess Supermicro will refuse anyway as ESXi is "not supported" on their end).
Thanks, bruno
Curious what happened, did you call back, and/or share the validation that it very much is supported? I did report your story to Supermicro, but it will likely take a while for support to be up to speed on this recent release.
Oh dear, don't despair, and feel free to share this article with them ;)
"Supermicro SuperServer Xeon D-1500 Bundle mini-tower and 1U rack mount are already on the VMware Compatibility Guide for ESXi 6.7"
https://TinkerTry.com/supermicro-superservers-vcg-updated-to-6-7
and meanwhile, I'll share this disappointing/odd feedback to my contact at Wiredzone too. He's been working with Supermicro for 20 years (anniversary today), and knows his way around a lot better than I.
Well, so far, this is the official answer on my Supermicro Support case: "Hi,
We are sorry but we cannot support you on this request because Supermicro didn't tested/validated this VMware ESXi version. "
---
I'm insisting they should work close to VMware to fix this. But, considering the first answer, I'm not very optimistic about it.
Bruno, thank you! I have also contacted Wiredzone with details of how big these Intel X557 issues have become, as they discuss next-steps with Supermicro (& Intel). A single picture that speaks a thousand words, with 7 of 8 comments about X557 issues, right along the right-edge of all 960 articles on my site right now. https://uploads.disquscdn.com/images/b69e8794c5b52305d9871896de680bbf427e60b7b5acffa92975b16e5bfe9890.png
Thank you, just to mention that I raised a case with Supermicro to report the bug. I will keep you posted once there is a formal answer from them.
Cheers, Bruno
I've added your great info to my X557 article here:
https://TinkerTry.com/how-to-work-around-intermittent-intel-x557-network-outages-on-12-core-xeon-d#may-06-2018-update
Bruno, my X557 ports are connected to my Netgear 708XT 10GbE switch, but I'll be trying those ports out on 1GbE to see if I get similar behavior. I can't thank you enough for reporting this, and I'll go ahead and append this info to the bottom of my article:
https://TinkerTry.com/how-to-work-around-intermittent-intel-x557-network-outages-on-12-core-xeon-d
and reporting this to Supermicro. Even better if you're willing to also report it too:
https://www.supermicro.com/24Hour/24hour.cfm
You may also want to see what Alessandro said here:
https://TinkerTry.com/easy-update-to-esxi-67#comment-3887952362
I agree, this is concerning, and I’m glad you found workarounds...
I think it would be worth investigating if the symptom is confirmed to be affecting all on-board 10GE Intel X552/X557 interfaces (Supermicro Intel Xeon D-1500 series) running on Gigabit speed. Even though interface is connected (and switch establishes the link at Gigabit), the driver don't recognise it until we administratively shut down the nic and bring it up again. If speed is manually set, the problem does not appear.
I see your vmnic2 and 3 are Intel X557, just like mine. Are they connected to 10GE ports or 1000Base-T switches? If you connect them to a Gigabit port, you can prove if the above hypothesis is valid.
I was on BIOS 1.3 already. Actually, I managed to make it work, on ESX CLI, I did:
esxcli network nic down -n vmnic0
esxcli network nic up -n vmnic0
But, it didn't survive to reboot. Every reboot, I had to do the same. Also installed latest driver ixgbe 4.5.3. But still not working.
To fix the issue, solution was to set the speed manually for both NICs. Finally fixed.
esxcli network nic set --speed 1000 --duplex full -n vmnic0
esxcli network nic set --speed 1000 --duplex full -n vmnic1
If support gives you push back, reference this:
https://tinkertry.com/supermicro-superservers-vcg-updated-to-6-7
and Supermicro can’t realistically support ESXi install, can you roll back to backup of 6.5, or install ESXi 6.5 first, then do a upgrade again via ESXCLI
https://tinkertry.com/easy-update-to-esxi-67
but with 4.5.3
https://tinkertry.com/how-to-install-intel-x552-vib-on-esxi-6-on-superserver-5028d-tn4t
installed for X557 first before reboot this time?
A longshot, but worth trying a new BIOS 1.3?
https://tinkertry.com/supermicro-superserver-bios-13-and-ipmi-358-released
I’ll need to add warnings to the article for Xeon D-1521 (which I don’t have) and thank you for reporting this, please also contact Supermicro support.
Nice post. Although, I have an issue with ESXi 6.7 on SuperMicro X10SDV-4C-TLN2F. NICs are recognized, but they are not getting link status updates or link is not going up. This is Xeon D-1521 which only has 2x 10GE network interfaces. Therefore, major issue as I don't have other means of network connectivity.
I've been waiting for ESXi native support for this card, but in fact, after implementing 6.7, now the network link status are not detected. (keep saying Disconnected).
I'm using Gigabit links although these are 10GE interfaces (it works and is supported).
Did you test the 10GE interfaces? Do they work in your case?
Paul Braren | TinkerTry.com
That is great to hear, thank you for your readership, and for taking the time to leave a comment that helps provide others with reassurance.