VMware ESXi 6.5 runs well on Xeon D Supermicro ServerServers, here's what you need to know

Posted by Paul Braren on Nov 18 2016 (updated on Apr 26 2017) in
  • Virtualization
  • ESXi
  • Network
  • HowTo
  • HomeLab
  • HomeServer
  • tinkertry-superserver-stack-cropped

    The install of ESXi 6.5 is just as easy as the install of all ESXi 6.0.x versions before it. Given the 7 year product life design of this powerful and efficient Intel SoC (System on a chip) targetted at virtualization workloads, I expected no less. I have a video below that goes through the simple install steps with voice-over. I used my world's-first Supermicro Xeon D-1567, the SuperServer SYS-5028D-TN4T 12 Core Bundle 2, pre-configured by Wiredzone with my well-documented ESXi-ready BIOS settings. Later on, confirmed the install/configure experience was identical on the Xeon D-1541.

    Actually, any Supermicro Xeon D systems/X10SDV motherboards will be highly likely to enjoy the same easy install and initial configuration experience, especially now that BIOS 1.1c and IMPI 3.46 is available across all the RJ45 10GBase-T equipped SuperServers. Why am I confident that installing and running ESXi 6.5 will be identical on all Supermicro variants? Because I spent months with loaner (admittedly noisier) 1U systems, and every experience with them was identical from a software install and configuration perspective. They're basically different chassis built around the same motherboards that vary only in core count and cooling. See detailed comparison tables that include dB and watt measurements, including the Skull Canyon Intel NUC, and the wider Flex ATX SYS-E300-8D with SFP+:


    A new install of Windows 10 Professional Anniversary Edition was used to create the videos below, using these two two vSphere 6.5 ISO files. Java was needed for the browser, so that Supermicro iKVM's Remote Console could mount the ESXi installer ISO, but (hopefully) soon, that won't be necessary. VMware Remote Console 9.0 (VMRC) was used for controlling the 3 ESXi 6.5 hosted VMs.



    VMware hasn't changed

    The two 1GbE ports work right off the bat, using drivers built into ESXi 5.5/6.0/6.5.

    Now let's discuss the Intel X552/X557 10GBase-T VIB. Notice that VMware ESXi 6.0U2 and ESXi 6.5 both seem to ship with the same old pre-Xeon-D ixgbe driver, with iov hinting at SR-IOV support:
    Yes, the last part of each line seems to indicate the ESXi version it shipped with.

    Of course you'll want to add the newer 10GbE driver to activate those 2 Intel X552/X557 10GbE interfaces, which can of course be used for 1GbE speeds too. Additional watts only start being burned once you physically cable those ports, starting at roughly 0.75 watts per port, depending upon cable length. It's a simple fix with a one-liner option for labs, demonstrated here, and on video below.

    Seems things get a bit tricker for Xeon D-1521 owners, don't yet have documentation on that exact fix. One upgrader who already had the 10GbE driver loaded reported this error:
    No NIC with MAC Address "xx:xx:xxx:xX:x:x" found
    don't know much about the circumstances, investigation ongoing.

    NVMe Storage

    Click twice to zoom way in. Samsung 950 PRO/SM951/850EVO tested.

    No issues with the built-in NVMe driver, as I went about testing these 3 drives with fresh VMFS 6 formatting, as seen in the videos below:

    1. Samsung 950 PRO 512GB M.2 NVMe SSD
    2. Samsung SM951 128GB M.2 NVMe SSD
    3. Samsung 850 EVO 256GB SATA3 SSD

    Yep, those speeds look excellent. Compared to native speeds under Windows/NTFS, running under ESXi/VMFS does take a hit on speeds at 4K sizes, but no more or less than 6.0U2 did. NVMe really shines here, as multiple VMs accessing SATA3 drives can bog down. Remember, I used the recommended BIOS settings, not the ~5% faster benchmark optimized BIOS settings.


    During beta tests, this was an area where issues showed up, with the new HTML5 UIs falling behind. A more thorough testing of health monitoring under ESXi 6.5 is still underway in my busy home lab, stay tuned. Meanwhile, know that Supermicro plans to get all their Xeon D gear onto the VMware 6.5 HCL in 1Q2017, to join the SYS-5028D-TN4T that's been on the 6.0.x HCL since February.

    Fresh install versus upgrade

    There is a potential issue with SATA3 speeds that upgraders to 6.5 might experience, likely due to AHCI VIBs/drivers that are carried over. There's an easy fix.

    When VMware's online 6.5 bundle is available, I plan to create a 6.5 version of this super-popular article:


    First look at easy install of VMware ESXi 6.5, preparing for vSphere 6.5/VCSA
    How to add 10GBase-T driver VIB to VMware ESXi 6.5 for Supermicro Xeon D SuperServers
    Failed attempt to replicate possible SM951 speed issues under new ESXi 6.5 install failed
    Failed attempt to replicate possible SATA3/AHCI speed issues under new ESXi 6.5 install
    VMware vSphere 6.5 taskbar shortcuts can make your Chrome browser UIs look like native Windows apps!
    Happy TinkerTry'd Bundle Supermicro SuperServer owners share their stories, featuring Intel Xeon D

    Apr 24 2017 Update

    There's a new refinement to the ESXi install process on Xeon D, see:

    and there's a newer 4.5.1 version of the 10GbE driver, see

    See also at TinkerTry

    See also