[Jump down below the article to see updates.]
It's early days folks. The Intel Xeon D-1541 isn't yet widely available. While I know that ASRock, and Gigabyte are offering Xeon D-1540/1541 solutions, I'm currently using a loaner system, the Supermicro SuperServer SYS-5028D-TN4T, with the usual X10SDV-TLN4F inside. Well, same familiar SKUs/part numbers. But as we also discovered together last week, this little mini-tower now has a new PCB (Printed Circuit Board) Rev. 2.00 design that should support either DDR3 and DDR4 RAM, see all the Intel ARK's differences between the Xeon D-1540 and Xeon D-1541, and my article with pictures of the new PCB. These are all minor differences, with a 5% boost in speeds not exactly a reason for most folks to experience buyer's regret for early adopters (June 2015) like me.
You may recall discussions about BIOS 1.0a having the SR-IOV feature in the original Xeon D-1540 that was announced in May 2015, but didn't become generally available until late June 2015. This includes my discussion with TinkerTry visitor davewolfs that featured this first screenshot in the series of 1.0a/1.0b/1.0c screenshots below:
It wasn't long after BIOS 1.0b arrived that folks noticed the disappearance of the SR-IOV BIOS feature. Patrick Kennedy of STH did his usual fantastic job helping not just his forum members, but the home server community at large, getting right to the bottom of what was really going on, seen at STH (ServeTheHome):
- The Broadwell-DE Intel Xeon D-1500 series and SR-IOV
- forum thread Intel Xeon D-1500 Series Discussion
- forum thread SM 5028D-TN4T X10SDV-TLN4F X557 sr-iov Does Not Work, i350 Work
Alas, we learned SR-IOV was never coming to the Xeon D-1540. You see, even though it briefly made an appearance in 1.0a, it never worked properly anyway, so it was appropriately removed from 1.0b. A bit of a PR black-eye, since the initial Intel publicity touted SR-IOV capability for the Xeon D-1500, so we deal with it, and move on. Why? Because in so many other respects, this SoC (System on a Chip) is an outstanding choice for virtualization, blessed with 8 cores/16 threads, up to 128GB of RAM, 1 1GbE management interface, 2 1GbE Intel I-350 ports, and 2 10GbE Intel X552 ports. Dang! Still loving it, after 7 pretty intense months of VMware/Windows/Red Hat/SuSE.
Looking to step back, and get a deeper understanding of SR-IOV?
- Intel's PCI-SIG SR-IOV Primer
- A way to share a single physical NIC port with many VMs efficiently, using the optimized driver for that NIC instead of going through a virtualized VMware NIC
Keeping perspective here, there are many home lab enthusiasts very pleased with these little mini-towers, myself included. This is a somewhat niche feature that most folks would not implement in a small home lab environment.
The drawbacks include:
cannot hot-add any resources to a booted SR-IOV-enabled VM, NICs included
cannot snapshot that VM
cannot vMotion VMs where you've enabled SR-IOV, much like you can't vMotion VMs with PCI/VT-d passthrough/pass-through/passthru, such as my OpenVPN isolated WAN project or USB 3.0 PCIe card project, or RDM drive mappings to get a drive seen directly by the underlying OS for things like data recover, see TinkerTry's Using RDM mappings to pass a drive through to an ESXi VM, featuring Windows 10 and VMFS Recovery
- Here's how the VMware vSphere Web Client warns you:
Note: Some virtual machine operations are unavailable when SR-IOV passthrough devices are present. You cannot suspend, migrate with vMotion, or take or restore snapshots of such virtual machines.
Today, I'm revisiting trying SR-IOV on those 2 10GbE X552 ports. This interface is the one that many folks wishing to put their 5028D-TN4T into a production role really wanted to see implemented and working. Many potential buyers made this functional SR-IOV requirement known to me via comments and email, and in their pre-sales inquiries to Supermicro reseller Wiredzone.
from an official VMware support perspective.
1) Supermicro VMware test matrix
- Supermicro SuperServer SYS-5028D-TN4T X10SDV-TLN4F Xeon D-1540 motherboard is compatible with VMware ESXi 5.5 and ESXi 6.0
culminating in this big news from just 2 days ago, inclusion in the
2) VMware Compatibility Guide (HCL)
- Supermicro SuperServer SYS-5028D-TN4T mini-tower now ready for business as the first Intel Xeon D-1500 in the VMware Compatibility Guide
This appears to be the very first serious home lab system in a practical mini-tower form factor that's made it onto the VMware Compatibility Guide. That's big news.
Why am I just testing SR-IOV on the 10GbE interface? Well, with 10Gb, you have more bandwidth to share that NIC port with your VMs, and because I couldn't get it implemented on the 1GbE Intel I-350 interface, at least not in this pre-production PBC Rev. 2.00 mobo with this pre-production BIOS 1.0c.
What does the end-result look like, once you enable SR-IOV for those two physical 10GbE ports? Basically, your VMs on that SR-IOV enabled host will no longer see a E1000E or VMXNET 3 ethernet adapter type, instead, they'll "see" the underlying actual Intel X552 gear. In Windows, the exact device enumerated says:
Intel(R) Ethernet Connection X552/X557-AT 10GBASE-T
So, why the uncertaintly that I have everything working right? Well, I got right to the end, and couldn't get the Windows X552 driver to initialize properly, at least on my initial first-tests. I might be rebuilding my ESXi from scratch next time, doing the following steps that are outlined in the:
This is an overview of the steps:
- clean-new install ESXi 6.0U1b, using iKVM to mount VMware-VMvisor-Installer-201601001-3380124.x86_64.iso
- install target will be the SanDisk Ultra Fit 32USB USB drive
- add 10GbE drivers to ESXi, using TinkerTry's How to download and install the Intel Xeon D-1540 10GbE X552/X557 driver/VIB for VMware ESXi 6 on Supermicro 5028D-TN4T/X10SDV-TLN4F
- install VCSA appliance VMware-VCSA-all-6.0.0-3040890.iso
- install a fresh Windows Server 2012 R2 VM from en_windows_server_2012_r2_with_update_x64_dvd_6052708.iso and let Windows Update do its thing, rebooting as many times as needed to finish up
- use vSphere Web Client to enable SR-IOV on the physical 10GbE ports
- assign SR-IOV to a powered-off VM by reserving all memory for the VM, then adding an SR-IOV NIC to the VM
- install Intel X552 drivers in the VM, seeing if Intel's newer 20.4.1 2/5/2015 driver does better than the older version featured in my Installing Windows 10 on the Supermicro SuperServer 5028D-TN4T article, buried on Supermicro's ftp site:
Intel 10G LAN Windows driver (D-1500_SoC) Version 3.10.162 JUN 12 2015
It's only available as Intel PRO Set bundle download PROWinx64.exe, yet, PROSet doesn't support the X552, as stated in the readme.txt. So I used 7-zip to unpack the EXE to:
then pointed the driver update wizard to that path, to automatically install this new release, but still some hangs and other strangeness, that work continues.
I will be making updates below this post with my discoveries, as soon as I get a chance to re-test. Keep in mind that the fact that I failed to get the 10GbE driver enabled on my first shot means very little. I need a clean slate to restart my tests, and a good hour or two of focused attention to wrap this test up and document the steps. As a busy IT worker, this happens late nights.
Is SR-IOV important to home lab enthusiasts? Likely not, as I tend to like my vMotioning and snapshot abilities, and I'm highly unlikely to notice any potential for any improvements in the kinds of stuff I'd be doing, including possibly testing out vSAN. But that's just me.
Please realize I'm not likely to produce polished videos or screenshots about SR-IOPV at this time, since everything I'm doing is on a pre-release motherboard with pre-release BIOS level 1.0c. I'm also told that a single BIOS that unifies Xeon D-1540 owners and Xeon D-1541 owners is coming in about a month or so. Let's call that 1.0d, and that's where I'll try focus my efforts, revisiting my BIOS upgrade procedures and BIOS settings recommendations, and another quick look at SR-IOV, just to be sure it's still working. I will also be revisiting IPMI settings, since IPMI is new as well in the PCB Rev. 2.00, see much more detail on that at First look at the new Intel Xeon D-1541 Supermicro SuperServer SYS-5028D-TN4T that includes this video:
update will appear here, later tonight
Once the 10GbE NICs were live on my 1GbE network with DHCP leases fed right out, suddenly the latest Intel driver driver installed nicely, and came up just fine. So behold, I present to you the first Xeon D-1541 screenshot that I know of with SR-IOV working, anywhere:
I hope to have time to do Windows Server 2016 Technical Preview / Hyper-V testing next. Interested? I have no way to know, unless you leave a comment below, thank you!
- Intel Xeon D-1541 Benchmarks – the minor speed jump
Feb 12 2016 by Patrick Kennedy at STH
Once Supermicro has BIOS 1.1 released, I'll try to revisit this article and see how things go. Currently, 1.1 is not yet available for download.
BIOS 1.1 has arrived, and SR-IOV still works just fine, see:
- Upgrade Supermicro Xeon D-1540/1541 from BIOS 1.0b/1.0c to 1.1 for identical features, including SR-IOV for 10GbE!
- How to configure ESXi 5.0 for USB 3.0 passthrough to a Windows VM
Nov 21 2011
TinkerTry.com, LLC is an independent site, has no sponsored posts, and all ads are run through 3rd party BuySellAds. All equipment and software is purchased for long-term productive use, and any rare exceptions are noted.
TinkerTry's relationship with Wiredzone is similar to the Amazon Associates program, where a very modest commission is earned from each referral sale from TinkerTry's SuperServer order page. I chose this trusted authorized reseller for its low cost and customer service, and a mutual desire to help folks worldwide, including a new way to reduce EU shipping costs. Why? Such commissions help reduce TinkerTry's reliance on advertisers, while building a community around the Xeon D-1500 chipset that strikes a great balance between efficiency and capability.
I personally traveled to Wiredzone near Miami FL to see the assembly room first-hand, and to Supermicro HQ in San Jose CA to share ideas and give direct product feedback.
I'm a full time IT Pro for the past 23 years. I've worked with IBM, HP, Dell, and Lenovo servers for hands-on implementation work across the US. Working from home lately, I'm quite enjoying finally owning a lower-cost Supermicro solution that I can recommend to IT Pro colleagues, knowing it will "just work." That's right, no tinkering required.