First Look - SR-IOV support in Intel Xeon D-1541

Posted by Paul Braren on Feb 11 2016 (updated on Mar 13 2016) in
  • Virtualization
  • ESXi
  • Network
  • HomeLab
  • HomeServer

    • [Jump down below the article to see updates.]
    It's early days folks. The Intel Xeon D-1541 isn't yet widely available. While I know that ASRock, and Gigabyte are offering Xeon D-1540/1541 solutions, I'm currently using a loaner system, the Supermicro SuperServer SYS-5028D-TN4T, with the usual X10SDV-TLN4F inside. Well, same familiar SKUs/part numbers. But as we also discovered together last week, this little mini-tower now has a new PCB (Printed Circuit Board) Rev. 2.00 design that should support either DDR3 and DDR4 RAM, see all the Intel ARK's differences between the Xeon D-1540 and Xeon D-1541, and my article with pictures of the new PCB. These are all minor differences, with a 5% boost in speeds not exactly a reason for most folks to experience buyer's regret for early adopters (June 2015) like me.

    support-server
    Supermicro SuperServer SYS-5028D-TN4T*

    BIOS 1.0c - the return of SR-IOV

    You may recall discussions about BIOS 1.0a having the SR-IOV feature in the original Xeon D-1540 that was announced in May 2015, but didn't become generally available until late June 2015. This includes my discussion with TinkerTry visitor davewolfs that featured this first screenshot in the series of 1.0a/1.0b/1.0c screenshots below:

    BIOS1.0a_SR-IOV
    BEFORE -PCB Rev. 1.02 with BIOS 1.0a, viewed with Java iKVM Viewer 1.69.24 from Windows 8.1
    BIOS1.0b_SR-IOV_missing
    AFTER - PCB Rev. 1.02 with BIOS 1.0b, viewed with Java iKVM Viewer 1.69.24 from Windows 10
    BIOS1.0c_SR-IOV_is_back
    AFTER - PCB Rev. 1.02 with BIOS 1.0c, viewed with Java iKVM Viewer 1.69.26 from Windows 10

    Some controversy surrounding failed SR-IOV support in Xeon D-1540

    It wasn't long after BIOS 1.0b arrived that folks noticed the disappearance of the SR-IOV BIOS feature. Patrick Kennedy of STH did his usual fantastic job helping not just his forum members, but the home server community at large, getting right to the bottom of what was really going on, seen at STH (ServeTheHome):

    Alas, we learned SR-IOV was never coming to the Xeon D-1540. You see, even though it briefly made an appearance in 1.0a, it never worked properly anyway, so it was appropriately removed from 1.0b. A bit of a PR black-eye, since the initial Intel publicity touted SR-IOV capability for the Xeon D-1500, so we deal with it, and move on. Why? Because in so many other respects, this SoC (System on a Chip) is an outstanding choice for virtualization, blessed with 8 cores/16 threads, up to 128GB of RAM, 1 1GbE management interface, 2 1GbE Intel I-350 ports, and 2 10GbE Intel X552 ports. Dang! Still loving it, after 7 pretty intense months of VMware/Windows/Red Hat/SuSE.

    Advantages of using SR-IOV

    Looking to step back, and get a deeper understanding of SR-IOV?

    • Intel's PCI-SIG SR-IOV Primer
    • A way to share a single physical NIC port with many VMs efficiently, using the optimized driver for that NIC instead of going through a virtualized VMware NIC

    Drawbacks to using SR-IOV (default is [Disabled] in BIOS)

    Keeping perspective here, there are many home lab enthusiasts very pleased with these little mini-towers, myself included. This is a somewhat niche feature that most folks would not implement in a small home lab environment.

    The drawbacks include:

    • cannot hot-add any resources to a booted SR-IOV-enabled VM, NICs included

    • cannot snapshot that VM

    • cannot vMotion VMs where you've enabled SR-IOV, much like you can't vMotion VMs with PCI/VT-d passthrough/pass-through/passthru, such as my OpenVPN isolated WAN project or USB 3.0 PCIe card project, or RDM drive mappings to get a drive seen directly by the underlying OS for things like data recover, see TinkerTry's Using RDM mappings to pass a drive through to an ESXi VM, featuring Windows 10 and VMFS Recovery

    • Here's how the VMware vSphere Web Client warns you:

      Note: Some virtual machine operations are unavailable when SR-IOV passthrough devices are present. You cannot suspend, migrate with vMotion, or take or restore snapshots of such virtual machines.

    Today, I'm revisiting trying SR-IOV on those 2 10GbE X552 ports. This interface is the one that many folks wishing to put their 5028D-TN4T into a production role really wanted to see implemented and working. Many potential buyers made this functional SR-IOV requirement known to me via comments and email, and in their pre-sales inquiries to Supermicro reseller Wiredzone.

    Then, two big steps forward

    from an official VMware support perspective.

    1) Supermicro VMware test matrix

    culminating in this big news from just 2 days ago, inclusion in the

    2) VMware Compatibility Guide (HCL)

    Why am I just testing SR-IOV on the 10GbE interface? Well, with 10Gb, you have more bandwidth to share that NIC port with your VMs, and because I couldn't get it implemented on the 1GbE Intel I-350 interface, at least not in this pre-production PBC Rev. 2.00 mobo with this pre-production BIOS 1.0c.

    What does the end-result look like, once you enable SR-IOV for those two physical 10GbE ports? Basically, your VMs on that SR-IOV enabled host will no longer see a E1000E or VMXNET 3 ethernet adapter type, instead, they'll "see" the underlying actual Intel X552 gear. In Windows, the exact device enumerated says:

    Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T

    So, why the uncertaintly that I have everything working right? Well, I got right to the end, and couldn't get the Windows X552 driver to initialize properly, at least on my initial first-tests. I might be rebuilding my ESXi from scratch next time, doing the following steps that are outlined in the:

    VMware vSphere 6.0 Documentation Center

    Single Root I/O Virtualization (SR-IOV)

    Advanced User's Guide to Enabling SR-IOV

    This is an overview of the steps:

    It's only available as Intel PRO Set bundle download PROWinx64.exe, yet, PROSet doesn't support the X552, as stated in the readme.txt. So I used 7-zip to unpack the EXE to:
    C:\Users\user\Downloads\PROWinx64
    then pointed the driver update wizard to that path, to automatically install this new release, but still some hangs and other strangeness, that work continues.

    I will be making updates below this post with my discoveries, as soon as I get a chance to re-test. Keep in mind that the fact that I failed to get the 10GbE driver enabled on my first shot means very little. I need a clean slate to restart my tests, and a good hour or two of focused attention to wrap this test up and document the steps. As a busy IT worker, this happens late nights.

    Closing thoughts

    Is SR-IOV important to home lab enthusiasts? Likely not, as I tend to like my vMotioning and snapshot abilities, and I'm highly unlikely to notice any potential for any improvements in the kinds of stuff I'd be doing, including possibly testing out vSAN. But that's just me.

    Please realize I'm not likely to produce polished videos or screenshots about SR-IOPV at this time, since everything I'm doing is on a pre-release motherboard with pre-release BIOS level 1.0c. I'm also told that a single BIOS that unifies Xeon D-1540 owners and Xeon D-1541 owners is coming in about a month or so. Let's call that 1.0d, and that's where I'll try focus my efforts, revisiting my BIOS upgrade procedures and BIOS settings recommendations, and another quick look at SR-IOV, just to be sure it's still working. I will also be revisiting IPMI settings, since IPMI is new as well in the PCB Rev. 2.00, see much more detail on that at First look at the new Intel Xeon D-1541 Supermicro SuperServer SYS-5028D-TN4T that includes this video:

    Video - Deep-dive into the differences between BIOS 1.0b and 1.0c

    Supermicro Xeon D-1540/1541 BIOS 1.0b/1.0c BIOS differences. Boot-from-NVMe, CPU-Z, ATTO tests.
    Edit_virtual_hardware_11_settings_for_sr-iov
    Yeah, it's Windows 10. Next time, I'll try Windows Server 2012 R2 and/or Windows Server 2016 Technical Preview 4.


    Feb 11 2016 Update

    update will appear here, later tonight

    Good news!

    Once the 10GbE NICs were live on my 1GbE network with DHCP leases fed right out, suddenly the latest Intel driver driver installed nicely, and came up just fine. So behold, I present to you the first Xeon D-1541 screenshot that I know of with SR-IOV working, anywhere:

    TinkerTry-Xeon-D-1541-SR-IOV-demonstration-sharing-one-Intel-X552
    Click the image above to enlarge, click again for 1:1 pixel-perfect viewing, then slide the 2560x1440 image around. Two Windows 10 VMs with SR-IOV active, using the same physical Intel X552 network port. Both are using Intel X552 driver 1.1.215.0 from 11/11/2015.

    I hope to have time to do Windows Server 2016 Technical Preview / Hyper-V testing next. Interested? I have no way to know, unless you leave a comment below, thank you!

    Feb 13 2016 Update

    vSphereWebClientSR-IOVx32
    vSphere Web Client - I gave SR-IOV 32 VFs [Virtual functions], and it shows.
    vSphereClientSR-IOVx32view
    vSphere Client - I gave SR-IOV 32 VFs [Virtual functions], and it shows.

    Feb 18 2016 Update

    Mar 09 2016 Update

    Once Supermicro has BIOS 1.1 released, I'll try to revisit this article and see how things go. Currently, 1.1 is not yet available for download.

    Mar 13 2016 Update

    BIOS 1.1 has arrived, and SR-IOV still works just fine, see:

    See also at TinkerTry

    See also

    Disclosure

    TinkerTry.com, LLC is an independent site, has no sponsored posts, and all ads are run through 3rd party BuySellAds. All equipment and software is purchased for long-term productive use, and any rare exceptions are noted.

    TinkerTry's relationship with Wiredzone is similar to the Amazon Associates program, where a very modest commission is earned from each referral sale from TinkerTry's SuperServer order page. I chose this trusted authorized reseller for its low cost and customer service, and a mutual desire to help folks worldwide, including a new way to reduce EU shipping costs. Why? Such commissions help reduce TinkerTry's reliance on advertisers, while building a community around the Xeon D-1500 chipset that strikes a great balance between efficiency and capability.

    I personally traveled to Wiredzone near Miami FL to see the assembly room first-hand, and to Supermicro HQ in San Jose CA to share ideas and give direct product feedback.

    I'm a full time IT Pro for the past 23 years. I've worked with IBM, HP, Dell, and Lenovo servers for hands-on implementation work across the US. Working from home lately, I'm quite enjoying finally owning a lower-cost Supermicro solution that I can recommend to IT Pro colleagues, knowing it will "just work." That's right, no tinkering required.

    TinkerTry - PCs, EVs, home tech, efficiency and more, including virtualization. My opinions here, not my employer's.

    Paul Braren, MCSE in 1998, VCP #2681 in 2005, vExpert 2014 to 2024, Veeam Vanguard Emeritus, 2015 to 2019.

    Support TinkerTry

    PayPal | Patreon | VMware Store

    After 6 successful years testing then shipping well over 1,000 Xeon D Bundles, Wiredzone had to stop selling them in mid-2021 due to cost, supply, and logistics challenges. So far, Xeon D-1700/2700 (Ice Lake D) was a minor refresh for 2023, with Xeon D-1800/2800 (Granite Rapids D) refresh looking a little better for 2024. More cores, but still just PCIe Gen4 and DDR4. Looking ahead, I'm glad it's Pat Gelsinger at Intel's helm. I'm also grateful to have had the honor of working at VMware when Pat was the CIO there. I'll leave it at that, given the whole Broadcom thing.

    Easily update VMware ESXi at TinkerTry.com/easy-update-to-latest-esxi