VMware ESXi 6.5 runs well on Xeon D Supermicro ServerServers, here's what you need to know
The install of ESXi 6.5 is just as easy as the install of all ESXi 6.0.x versions before it. Given the 7 year product life design of this powerful and efficient Intel SoC (System on a chip) targetted at virtualization workloads, I expected no less. I have a video below that goes through the simple install steps with voice-over. I used my world's-first Supermicro Xeon D-1567, the SuperServer SYS-5028D-TN4T 12 Core Bundle 2, pre-configured by Wiredzone with my well-documented ESXi-ready BIOS settings. Later on, confirmed the install/configure experience was identical on the Xeon D-1541.
Actually, any Supermicro Xeon D systems/X10SDV motherboards will be highly likely to enjoy the same easy install and initial configuration experience, especially now that BIOS 1.1c and IMPI 3.46 is available across all the RJ45 10GBase-T equipped SuperServers. Why am I confident that installing and running ESXi 6.5 will be identical on all Supermicro variants? Because I spent months with loaner (admittedly noisier) 1U systems, and every experience with them was identical from a software install and configuration perspective. They're basically different chassis built around the same motherboards that vary only in core count and cooling. See detailed comparison tables that include dB and watt measurements, including the Skull Canyon Intel NUC, and the wider Flex ATX SYS-E300-8D with SFP+:
A new install of Windows 10 Professional Anniversary Edition was used to create the videos below, using these two two vSphere 6.5 ISO files. Java was needed for the browser, so that Supermicro iKVM's Remote Console could mount the ESXi installer ISO, but (hopefully) soon, that won't be necessary. VMware Remote Console 9.0 (VMRC) was used for controlling the 3 ESXi 6.5 hosted VMs.
Networking
VMware hasn't changed
The two 1GbE ports work right off the bat, using drivers built into ESXi 5.5/6.0/6.5.
Now let's discuss the Intel X552/X557 10GBase-T VIB. Notice that VMware ESXi 6.0U2 and ESXi 6.5 both seem to ship with the same old pre-Xeon-D ixgbe driver 3.7.13.7.14, with iov hinting at SR-IOV support:
VWM_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.600.0.0.2494585
VWM_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.650.0.0.4564106
Yes, the last part of each line seems to indicate the ESXi version it shipped with.
Of course you'll want to add the newer 10GbE driver to activate those 2 Intel X552/X557 10GbE interfaces, which can of course be used for 1GbE speeds too. Additional watts only start being burned once you physically cable those ports, starting at roughly 0.75 watts per port, depending upon cable length. It's a simple fix with a one-liner option for labs, demonstrated here, and on video below.
Seems things get a bit tricker for Xeon D-1521 owners, don't yet have documentation on that exact fix. One upgrader who already had the 10GbE driver loaded reported this error:
No NIC with MAC Address "xx:xx:xxx:xX:x:x" found
don't know much about the circumstances, investigation ongoing.
NVMe Storage
No issues with the built-in NVMe driver, as I went about testing these 3 drives with fresh VMFS 6 formatting, as seen in the videos below:
- Samsung 950 PRO 512GB M.2 NVMe SSD
- Samsung SM951 128GB M.2 NVMe SSD
- Samsung 850 EVO 256GB SATA3 SSD
Yep, those speeds look excellent. Compared to native speeds under Windows/NTFS, running under ESXi/VMFS does take a hit on speeds at 4K sizes, but no more or less than 6.0U2 did. NVMe really shines here, as multiple VMs accessing SATA3 drives can bog down. Remember, I used the recommended BIOS settings, not the ~5% faster benchmark optimized BIOS settings.
Health
During beta tests, this was an area where issues showed up, with the new HTML5 UIs falling behind. A more thorough testing of health monitoring under ESXi 6.5 is still underway in my busy home lab, stay tuned. Meanwhile, know that Supermicro plans to get all their Xeon D gear onto the VMware 6.5 HCL in 1Q2017, to join the SYS-5028D-TN4T that's been on the 6.0.x HCL since February.
Fresh install versus upgrade
There is a potential issue with SATA3 speeds that upgraders to 6.5 might experience, likely due to AHCI VIBs/drivers that are carried over. There's an easy fix.
When VMware's online 6.5 bundle is available, I plan to create a 6.5 version of this super-popular article:
- How to easily update your VMware Hypervisor to ESXi 6.0 Update 2
so consider subscribing to TinkerTry Weekly to get notified, or follow me @paulbraren.
Videos
Apr 24 2017 Update
There's a new refinement to the ESXi install process on Xeon D, see:
and there's a newer 4.5.1 version of the 10GbE driver, see
- How to download and install the Intel Xeon D 10GbE X552/X557 driver/VIB for VMware ESXi 6.x
Apr 19 2017 updated
See also at TinkerTry
See also
- Homelab - Will ESXi 6.5 run on Intel NUC?
Nov 17 2016 by Florian Grehl at Virten.net
