VMware ESXi 6.5 runs well on Xeon D Supermicro ServerServers, here's what you need to know
The install of ESXi 6.5 is just as easy as the install of all ESXi 6.0.x versions before it. Given the 7 year product life design of this powerful and efficient Intel SoC (System on a chip) targetted at virtualization workloads, I expected no less. I have a video below that goes through the simple install steps with voice-over. I used my world's-first Supermicro Xeon D-1567, the SuperServer SYS-5028D-TN4T 12 Core Bundle 2, pre-configured by Wiredzone with my well-documented ESXi-ready BIOS settings. Later on, confirmed the install/configure experience was identical on the Xeon D-1541.
Actually, any Supermicro Xeon D systems/X10SDV motherboards will be highly likely to enjoy the same easy install and initial configuration experience, especially now that BIOS 1.1c and IMPI 3.46 is available across all the RJ45 10GBase-T equipped SuperServers. Why am I confident that installing and running ESXi 6.5 will be identical on all Supermicro variants? Because I spent months with loaner (admittedly noisier) 1U systems, and every experience with them was identical from a software install and configuration perspective. They're basically different chassis built around the same motherboards that vary only in core count and cooling. See detailed comparison tables that include dB and watt measurements, including the Skull Canyon Intel NUC, and the wider Flex ATX SYS-E300-8D with SFP+:
A new install of Windows 10 Professional Anniversary Edition was used to create the videos below, using these two two vSphere 6.5 ISO files. Java was needed for the browser, so that Supermicro iKVM's Remote Console could mount the ESXi installer ISO, but (hopefully) soon, that won't be necessary. VMware Remote Console 9.0 (VMRC) was used for controlling the 3 ESXi 6.5 hosted VMs.
Networking
VMware hasn't changed
The two 1GbE ports work right off the bat, using drivers built into ESXi 5.5/6.0/6.5.
Now let's discuss the Intel X552/X557 10GBase-T VIB. Notice that VMware ESXi 6.0U2 and ESXi 6.5 both seem to ship with the same old pre-Xeon-D ixgbe driver 3.7.13.7.14, with iov hinting at SR-IOV support:
VWM_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.600.0.0.2494585
VWM_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.650.0.0.4564106
Yes, the last part of each line seems to indicate the ESXi version it shipped with.
Of course you'll want to add the newer 10GbE driver to activate those 2 Intel X552/X557 10GbE interfaces, which can of course be used for 1GbE speeds too. Additional watts only start being burned once you physically cable those ports, starting at roughly 0.75 watts per port, depending upon cable length. It's a simple fix with a one-liner option for labs, demonstrated here, and on video below.
Seems things get a bit tricker for Xeon D-1521 owners, don't yet have documentation on that exact fix. One upgrader who already had the 10GbE driver loaded reported this error:
No NIC with MAC Address "xx:xx:xxx:xX:x:x" found
don't know much about the circumstances, investigation ongoing.
NVMe Storage
No issues with the built-in NVMe driver, as I went about testing these 3 drives with fresh VMFS 6 formatting, as seen in the videos below:
- Samsung 950 PRO 512GB M.2 NVMe SSD
- Samsung SM951 128GB M.2 NVMe SSD
- Samsung 850 EVO 256GB SATA3 SSD
Yep, those speeds look excellent. Compared to native speeds under Windows/NTFS, running under ESXi/VMFS does take a hit on speeds at 4K sizes, but no more or less than 6.0U2 did. NVMe really shines here, as multiple VMs accessing SATA3 drives can bog down. Remember, I used the recommended BIOS settings, not the ~5% faster benchmark optimized BIOS settings.
Health
During beta tests, this was an area where issues showed up, with the new HTML5 UIs falling behind. A more thorough testing of health monitoring under ESXi 6.5 is still underway in my busy home lab, stay tuned. Meanwhile, know that Supermicro plans to get all their Xeon D gear onto the VMware 6.5 HCL in 1Q2017, to join the SYS-5028D-TN4T that's been on the 6.0.x HCL since February.
Fresh install versus upgrade
There is a potential issue with SATA3 speeds that upgraders to 6.5 might experience, likely due to AHCI VIBs/drivers that are carried over. There's an easy fix.
When VMware's online 6.5 bundle is available, I plan to create a 6.5 version of this super-popular article:
- How to easily update your VMware Hypervisor to ESXi 6.0 Update 2
so consider subscribing to TinkerTry Weekly to get notified, or follow me @paulbraren.
Videos
Apr 24 2017 Update
There's a new refinement to the ESXi install process on Xeon D, see:
and there's a newer 4.5.1 version of the 10GbE driver, see
- How to download and install the Intel Xeon D 10GbE X552/X557 driver/VIB for VMware ESXi 6.x
Apr 19 2017 updated
See also at TinkerTry
See also
- Homelab - Will ESXi 6.5 run on Intel NUC?
Nov 17 2016 by Florian Grehl at Virten.net
All Comments on This Article (43)
Great post Paul; love the channel; content and no-BS approach!
Sorry I got so behind on replies. Are you on a Xeon D Bundle? Do you still need guidance?
Basically, you can install fresh into the VM on the passed through NVMe device. Or you can just remove the ESXi media (USB), and mount the ISO using iKVM (or physical CDROM) with the BIOS in UEFI mode, putting Windows right onto that NVMe SSD. Now boot ESXi, turn on passthrough, and follow along with what my article says, skipping the re-install part. It should boot right up in that empty VM, more details on (first draft like) article at:
https://TinkerTry.com/how-to-configure-vmdirectpath-pass-through-of-nvme-on-esxi-6-5-update-1
Hi Paul.
I had hoped to use the RDM "translation" to pass through the Windows 10 Pro install that was already on the NVMe SSD so that I *wouldn't* have to install the OS _after_ I had configured it for pass-through in vSphere 6.5.
I had reviewed that exact article prior to my decision to re-install Windows 10 Pro on my M.2 NVMe, after configuring it for pass-through on vSphere 6.5 u1.
I may have mis-understood the instructions in step #9 though.
When you say:
"finish the VM creation wizard, bring up a VM console view, then power up the VM and be ready to quickly press any key to boot from the bootable ISO, *thus beginning your normal installation from your bootable ISO right onto your NVMe device*..."
I took that to mean that I'd have to re-install Windows 10 Pro from within vSphere.
Did I miss something there?
As always, "Thank you" for the continued encouragement and investigative articles on Tinkertry.com!!
~vRickD
So awesome that I got to meet you at VMworld 2017 US last week!
https://twitter.com/paulbraren/status/903631365555531776
I think this article, with a horribly long title, will help:
https://TinkerTry.com/how-to-configure-vmdirectpath-pass-through-of-nvme-on-esxi-6-5-update-1
the nifty thing is that you don't need to type fancy RDM commands, it's kind of awesome. Let us know how it goes!
Hi Paul. I've gotten my "Remote mobile Data Center" (lol) up and running with VSphere 6.5 and VCSA and it's been a blast so far! I do have one question though: I bought a Samsung EVO 250Gb NVMe M.2 SSD and have directly installed Windows 10 Pro on it (vs through VSphere ESXi) and am wondering if you know of any technical "walk throughs" for how to pass through an already-populated NVMe M.2 SSD via RDM?
Looking forward to continuing my VMWare eduation in this community!
~Rick
So glad I brought you some joy, always great to hear! More ideas to come, stay tuned. Hope you find what you need to get going!
Immensely helpful guide and included details, Paul! It's refreshing to be a part of such an open and technologically diverse community @Tinkertry.com - I'll be pouring over the information you provided, over the coming days and will hopefully have my lab up and running by the end of this week (I'm on a VERY slow satellite ISP so pulling down packages can be a little delayed).
Thanks again!
VRickD
Congrats, and thank you! You are joining hundreds of other happy owners before you!
So, licensing, not fun, but I'm trying to help with articles like this work-around (sign up for a new account, get a 60 day trial):
https://TinkerTry.com/my-vmwares-you-either-are-not-entitled-or-do-not-have-permissions-to-download-this-product-error-and-what-to-do-about-it
If you wind up happy, then $200 gets you many perks that include getting you out of 60 time bomb jail for 365 days by licensing your already installed ESXi 6.5 and the fun part, VCSA (for vMotion etc.) see:
https://TinkerTry.com/evalexperience
discounted to $180 if purchased at local VMUGs:
https://twitter.com/paulbraren/status/837374379663163393
but the code here is >6 months old, so you'll want to immediate patch to the latest versions that have fixed many of this major release came out, just follow along here:
ESXi 6.5.0.a Build 5224934:
https://TinkerTry.com/easy-upgrade-to-esxi-650a-build-5224934-security-advisory-vmsa-2017-0006
VCSA 6.5.0b:
https://TinkerTry.com/easy-upgrade-to-vcsa-650a
Or, if using production workloads, you should go all in for 1 year of "vSphere Essentials Kit for Academic Users" for $336, and about $399.96 for 3 years of free upgrades to new releases, see terms & conditions for eligibility:
http://store.vmware.com/store/vmware/html/pbPage.academicstore_terms
and this referral link for exact pricing for your location:
http://fave.co/2nzlgBl
Hi Paul (and everyone)!
I'm finally taking the plunge into Virtualization on an Enterprise-scale and have been reading through ALOT of posts on TinkerTry.com.
I've been in IT Security for the past 17 years but before that, was a SysAdmin both in the military (10 years) and in civilian life (10 years).
I've already purchased the SuperMicro 5028D Tn4T bundle #2 and am going to be building it out as my home Virtualization/Security Certification lab -- it's supposed to be on it's way from WiredZone and arrive by 4/12/17 -- I'm "salivating" (LOL).
It's been awhile since I tried to download any VMWare ESX images from VMware.com and when I try to use the credentials I've used since VMWare Workstation was at v6, the VMware site says I don't have the correct creds to be able to download the ISOs you ref'd here (URL https://tinkertry.com/downloadvsphere65#download).
Any help you (or anyone else) could recommend to a vNoob would be most-appreciated.
Thanks,
vRickD
Got it, thanks! Just updated this table FYI:
https://TinkerTry.com/compare#table
(RAM prices rose)
It was 6.5 that I had the issue with. I haven't updated to 6.5a
I really appreciate your taking the time to document that. This is actually the 2nd such report I've had, but first on this exact system, for a fresh install rather than upgrade. Was that 6.5, or 6.5.0a?
In case you didn't spot it already, see also my documentation of this issue at:
https://tinkertry.com/supermicro-xeond-superserver-slow-sata-driver-fix-for-esxi65
which I will update as needed.
Here's the simple 6.5.0.a procedures, found first at TinkerTry by the way:
https://tinkertry.com/easy-upgrade-to-vcsa-650a
https://tinkertry.com/easy-upgrade-to-esxi-650a
Have fun working/playing with your new gear!
Paul I had the SATA3 speed issues on a fresh install of 6.5 on the mini tower. As soon as I disabled the AHCI driver, all works as expected. I was surprised since this was a fresh install but just wanted to share.
Looks like VMware vCenter Server and ESXi 6.5.0a were released yesterday/today. Wonder if this fixes any of the IPv6 or HA GUI issues.
Time to update!
Me too!
:)
Thanks Paul, issue reported via the smiley face!
Looking forward to any updates to vSphere 6.5 which will hopefully fix any bugs/issues, especially with IPv6!
Nice, thanks for sharing the great news! So, as for that discrepancy, can you use the little yellow smiley face seen at top right of the (HMTL5) vSphere Client? that is likely the best way to directly give developers feedback on your findings, having just listened to a podcast all about this just yesterday actually, with updates coming soon, here's the section that talks about how updates will arrive:
VMware Communities Roundtable: #371 vSphere 6.5 Web Client w/Dennis Lu
https://overcast.fm/+I_ltXrbo/16:45
https://uploads.disquscdn.com/images/d3dc053db38154a605af2b1ecbe85d1e044709c4d0ca0c3b931d425002317a6b.png
Upgrade complete! Phew!! Upgraded VMware tools, hardware version of the VMs, the hosts, vCenter and have formatted all datastores with VMFS6 and so far so good! The only exception the problem I have with the ESXi hosts is with AD authentication but other than that it's looking promising. Can't wait for the HTML5 GUI to be feature complete.
Oh there was one thing I thought odd. I have HA enabled and in the HTML5 GUI it shows the Repsonse to restart VMs, shutdown etc but in the Flash client it shows the response as Disabled?? Odd discrepancy. All VMs are saying they are HA protected so I'm not sure why the GUIs differ with this.
Further update since my last post ;-)
Upgraded second host in cluster and all seemed to go well. Rebooted and things were looking good (no disconnects of the host from vCenter) but...then I tried to vMotion a VM from host1 to host2 and it failed saying there was no vMotion networking in place.
Don't know how this happened but during/after the upgrade it removed my vMotion vmkernel adapters and, annoyingly, even though they weren't present in the web client or CLI I couldn't recreate them as vSphere thought they still existed. Luckily the fix was easy: Remove host from VDS and re-add it and I was back in business and could recreate the vMotion vmkernel adapters..
The only issue I have now is both my upgraded hosts were a member of my AD domain and that authentication doesn't work anymore. When I try to remove either host from the domain the operation times out so I'm still looking into that one.
So glad to meet you, and thanks for reaching out to me at
https://TinkerTry.com/contact too!
I am glad that there still doesn't appear to be any issues with Integrated Intel I-350 chipset 1GbE networking on Xeon D, only with the IPv6 implementation, presmably qauite fixable with future software updates.
Thats ok, thought I'd mention it as I have been struggling with this all week. Previous upgrades have gone fine but not this one unfortunately. It's just so odd as IPv6 worked/s great in 6.0U2 but not in 6.5.
I'm wondering if these issues/bugs will be fixed in the next release/update.
Anyways, I completed the upgrade to 6.5 and have removed all IPv6 addresses from my vCenter and hosts. Fingers crossed.
Glad you solved the problem, sorry I wasn't able to help, with my career now more focused on VMware now, I'll hopefully be better able to report such findings appropriately. For now, still in the "new guy" phase...I too have experienced IPv6 issues with ESXi...
I think I had found the source of my issues with vSphere 6.5! I *think* IPv6 is broken/incomplete in vSphere 6.5. My ESXi 6.5 host kept saying in the events/logs that it was disconnecting every minute or 2 from vCenter 6.5. As soon as I removed the IPv6 address from the host ALL my issues went away. Also, whenever I reboot my vCSA 6.5 it loses my static IPv6 address. You also can't set a static IPv6 address ONLY on an ESXi 6.5 host.
I've been running a dual stack IPv4/6 network for a year now and vSphere 6.0 has been setup with IPv4/6 addresses and has worked really well for this time with 6.0U2.
But not 6.5. I think there are issues with vSphere 6.5 and IPv6. It's broken/buggy?
vCenter upgraded to 6.5 successfully but I haven't upgraded the hosts yet which I will be doing tomorrow or on the weekend. I want vCSA 6.5 to run a few days before proceeding. I unregistered the VM with the USB UPS in it before upgrading and completely turned off HA/DRS. I also removed my IPv6 address from the vCSA 6 appliance. In addition to all this, and I know this sounds daft, I copied the vCSA 6.5 ISO onto the server I upgraded from rather than mounting the ISO over the network (I'm sure this is mentioned in the release notes).
I'm kind of nervous about upgrading the hosts but am hoping that doing the above helps so that my host(s) don't keep disconnecting. I've now used PowerCLI to backup my ESXi config so heres hoping...
Wow, let us know how your ReFS adventure goes!
See also:
https://twitter.com/paulbraren/status/770742990792822784
https://twitter.com/paulbraren/status/770803736897728512
Even scrogged my laptop once when horsing around with ReFS, self-inflicted pain:
https://twitter.com/paulbraren/status/770803736897728512
Thanks for the update.
Look forward to hearing your findings.
My current project is to get v6.5 fully operational using Workstation 12.
The reason behind this is we are getting lots of individuals signing up for our VMware "Official" courses.
The plan is to have them using their own swimming pool (home lab), instead of us providing access to our vLab environment, that we call: "The COCOON".
I'm going to give the upgrade another try! I got my Veeam backups working again, had to do with me removing my IPv6 IP from the vCSA so thats all good now. Took the opportunity to setup Veeam with ReFS and tidy my backup jobs so it all worked out in the end with regards to this.
Heh, I'm feeling better now that I have rolled back and am back on 6.0U2 ;-)
So I think the 1Gb "connectivity issues" may be a red herring but I'm not 100% sure at this stage.
Having posted this on forums and read up about it, it sounds like I need to unregister my one VM that has the USB UPS connected to it BEFORE upgrading to vCSA 6.5. Apparently there is a nasty bug in 6.5 regarding the USB bug. I'm using a Cyberpower UPS currently.
Anyhow, I'm going to give the upgrade another whirl with my UPS VM unregistered (the one with the USB device in it) and see what happens.
Watch this space!
(Hope your new job is going well Paul!)
Hi Paul,
Thank your for acknowledgment/awareness.
I did not want to make it sound like an advertisement, but did want it to be known I've been around the block a time or two, or ....
btw: I like this/your site.
It reminds me of old-school scientist/engineers communities.
fyi: I can say old-school. Here's my linkedin link, which is the ONLY social network I use, and that even very limited. www.linkedin.com/in/petestulginskis
Our college link is: www.cccti.edu/vmware
I noticed your subsequent post regarding: "1GbE connectivity".
It is rare but I have had one other supermicro box that when I upgraded to v6.0, I had issue with the 1G nics.
Being the age of that server it gave me the reason(?) to acquire new servers.
Again with SuperMicro.
Last year, about this same time, I noticed the SM-5028D-TN4T, and purchased one.
Time/Money kept me from building my home-lab around this box.
When v6.5 was released, and being that VMware will be offering v6.5 courses I used that as my justification. I'm sure you understand!
Would be glad to establish further conversations/collaborations.
I do want to acknowledge that I have seen very few with VCP2 certifications. WOW!
Dang, this is my first report of 1GbE connectivity issues, and an unhappy unending makes it worse. Do you have some info on what makes UPS a likely culprit? Which UPS? Sorry, I'm the curious sort, if you've moved onward (well, backward), and are not in the mood, I completely understand...
Love this post, thank you! I feel the same way, having spent so many hours with the product, and surfacing little (and big) bugs. At least now I can more effectively (and affordably) report the things I find.
Would love to hear more about what your academy does, feel free to share a URL or two (I'll moderate/approve). Thanks again!
I agree v6.5 has a few issues that I expect to see an update v6.5.1 soon!
I also understand about the long Days/Nights wondering if it is worth while.
However, I just will not let technology win!
I am the director of a VMware Academy for a college in NC.
I started our academy 10yrs ago.
We are one of a few that support our own lab env for these courses.
We offer our course Live onLine and I spend endless time making sure everything is working.
Sorry, I was not much help, but sometimes it good to know that what you are doing does work!
Thanks for the post but my issues had to do with the 1Gb connections. Anyway, I have since rolled back to 6.0 as it was a complete disaster in the end. After my upgrade of vCenter to 6.5 it worked for exactly one day and then after a day I logged into it and the web interface "died" on me (completely empty with some error about the sdk). A reboot didn't help and my ESXi 6.5 host kept disconnecting every 3 - 4 minutes from vCSA 6.5..
I have a VM with a USB UPS passed through to it which *MAY* be the cause of the issue but I'm not 100% sure yet. Apparently there is a bug with having USB devices attached to it.
So many frustrating hours later I rolled back to vCSA and rebuilt my host to ESXi 6.0 again. Not impressed ;-(
I'm not even sure how to proceed with the upgrade now as I did read up about it so thought I was prepared but apparently not.
And the cherry on the top is my Veeam isn't working either now after the rollaback to vCSA 6...argh!
I have an identical setup which has run without any issue for over a month.
Did you add the vib for the intel 10G nics?
That was the only modification I had to do with the esx65-4564106, install.
The intel X552/557 vib I used was: net-ixgbe_4.4.1-1OEM.600.0.0.2159203.
This was discussed under the Networking area of this post.
btw: Great Post!!
So I upgraded my vCenter to 6.5 yesterday (which was a nightmare!) and this morning I thought, now comes the easy part, I'll upgrade my first (of two) hosts to ESXi 6.5. The ESXi upgrade was eacy and there were no errors but after rebooting all the issues started.
My upgraded host keeps disconnecting from vCenter now. During the upgrade it changed all my NICs to 1000Mb/Full Duplex when they were set to Auto Negotiate. I changed them all back to Auto Negotiate but the upgraded hosts keeps disconnecting from vCenter.
I also noticed that the onboard NIC driver for the 1Gb ports on the Supermicro 5028D-TN4T changed from "igb" in ESXi 6.0 to "igbn" in ESXi 6.5.
Is anyone else having this issue? My ESXi 6.0 host has never ever disconnected from vCenter (nor did the upgraded host when it was on ESXi 6.0) so I am baffled why this is happening. Do I need to change the driver? The upgraded host seems to get disconnected every few minutes which makes it unusable.
Thanks!
Sorry Chris, Intel RST never was supported by VMware ESXi, and is unlikely to be supported in the future, see details at:
https://tinkertry.com/superserverworkstation#comment-3087864568
Hello and thanks for all your great posts. Is RSTe RAID 10 (SATA) working with Firmware 1.0c and ESXi 6.5, or still necessary with extra RAID controller? Thank you so much!
I hear you, and I've forwarded such comments to Supermicro. Whenever it does make it onto the 6.5 HCL, you can be certain I'll let everybody at TinkerTry know!
Paul Braren | TinkerTry.com
I really appreciate the feedback Bobob, thank you!