VMware ESXi 6.5 runs well on Xeon D Supermicro ServerServers, here's what you need to know

Posted by Paul Braren on Nov 18 2016 (updated on Nov 22 2016) in
  • Virtualization
  • ESXi
  • Network
  • HowTo
  • HomeLab
  • HomeServer
  • This article is in draft mode, and the information throughout is changing rapidly these first few days.

    tinkertry-superserver-stack-cropped

    The install of ESXi 6.5 is just as easy as the install of all ESXi 6.0.x versions before it. Given the 7 year product life design of this powerful and efficient Intel SoC (System on a chip) targetted at virtualization workloads, I expected no less. I have a video below that goes through the simple install steps with voice-over. I used my world's-first Supermicro Xeon D-1567, the SuperServer SYS-5028D-TN4T 12 Core Bundle 2, pre-configured by Wiredzone with my well-documented ESXi-ready BIOS settings. Later on, confirmed the install/configure experience was identical on the Xeon D-1541.

    Actually, any Supermicro Xeon D systems/X10SDV motherboards will be highly likely to enjoy the same easy install and initial configuration experience, especially now that BIOS 1.1c and IMPI 3.46 is available across all the RJ45 10GBase-T equipped SuperServers. Why am I confident that installing and running ESXi 6.5 will be identical on all Supermicro variants? Because I spent months with loaner (admittedly noisier) 1U systems, and every experience with them was identical from a software install and configuration perspective. They're basically different chassis built around the same motherboards that vary only in core count and cooling. See detailed comparison tables that include dB and watt measurements, including the Skull Canyon Intel NUC, and the wider Flex ATX SYS-E300-8D with SFP+:

    intel-xeon-d-superserver-sys-e200-8d-and-sys-e300-8d-and-sys-5028d-tn4t-and-sys-5018d-fn4t-with-intel-nuc6i7kyk-700x406

    A new install of Windows 10 Professional Anniversary Edition was used to create the videos below, using these two two vSphere 6.5 ISO files. Java was needed for the browser, so that Supermicro iKVM's Remote Console could mount the ESXi installer ISO, but (hopefully) soon, that won't be necessary. VMware Remote Console 9.0 (VMRC) was used for controlling the 3 ESXi 6.5 hosted VMs.

    Networking

    VMware-DCUI-shows-Intel-X557-loaded-properly-at-TinkerTry

    VMware hasn't changed

    The two 1GbE ports work right off the bat, using drivers built into ESXi 5.5/6.0/6.5.

    Now let's discuss the Intel X552/X557 10GBase-T VIB. Notice that VMware ESXi 6.0U2 and ESXi 6.5 both seem to ship with the same old pre-Xeon-D ixgbe driver 3.7.13.7.14, with iov hinting at SR-IOV support:
    VWM_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.600.0.0.2494585
    VWM_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.650.0.0.4564106
    Yes, the last part of each line seems to indicate the ESXi version it shipped with.

    Of course you'll want to add the newer 10GbE driver to activate those 2 Intel X552/X557 10GbE interfaces, which can of course be used for 1GbE speeds too. Additional watts only start being burned once you physically cable those ports, starting at roughly 0.75 watts per port, depending upon cable length. It's a simple fix with a one-liner option for labs, demonstrated here, and on video below.

    Seems things get a bit tricker for Xeon D-1521 owners, don't yet have documentation on that exact fix. One upgrader who already had the 10GbE driver loaded reported this error:
    No NIC with MAC Address "xx:xx:xxx:xX:x:x" found
    don't know much about the circumstances, investigation ongoing.

    NVMe Storage

    TinkerTry-tests-Samsung-950PRO-SM951-850EVO
    Click twice to zoom way in. Samsung 950 PRO/SM951/850EVO tested.

    No issues with the built-in NVMe driver, as I went about testing these 3 drives with fresh VMFS 6 formatting, as seen in the videos below:

    1. Samsung 950 PRO 512GB M.2 NVMe SSD
    2. Samsung SM951 128GB M.2 NVMe SSD
    3. Samsung 850 EVO 256GB SATA3 SSD

    Yep, those speeds look excellent. Compared to native speeds under Windows/NTFS, running under ESXi/VMFS does take a hit on speeds at 4K sizes, but no more or less than 6.0U2 did. NVMe really shines here, as multiple VMs accessing SATA3 drives can bog down. Remember, I used the recommended BIOS settings, not the ~5% faster benchmark optimized BIOS settings.

    Health

    During beta tests, this was an area where issues showed up, with the new HTML5 UIs falling behind. A more thorough testing of health monitoring under ESXi 6.5 is still underway in my busy home lab, stay tuned. Meanwhile, know that Supermicro plans to get all their Xeon D gear onto the VMware 6.5 HCL in 1Q2017, to join the SYS-5028D-TN4T that's been on the 6.0.x HCL since February.

    Fresh install versus upgrade

    There is a potential issue with SATA3 speeds that upgraders to 6.5 might experience, likely due to AHCI VIBs/drivers that are carried over. There's an easy fix.

    When VMware's online 6.5 bundle is available, I plan to create a 6.5 version of this super-popular article:

    Videos

    First look at easy install of VMware ESXi 6.5, preparing for vSphere 6.5/VCSA
    How to add 10GBase-T driver VIB to VMware ESXi 6.5 for Supermicro Xeon D SuperServers
    Failed attempt to replicate possible SM951 speed issues under new ESXi 6.5 install failed
    Failed attempt to replicate possible SATA3/AHCI speed issues under new ESXi 6.5 install
    VMware vSphere 6.5 taskbar shortcuts can make your Chrome browser UIs look like native Windows apps!
    Happy TinkerTry'd Bundle Supermicro SuperServer owners share their stories, featuring Intel Xeon D

    See also at TinkerTry


    See also