Researching the latest Skylake Core i5 Intel NUC's suitability for VMware virtualization and vSAN

Posted by Paul Braren on Feb 11 2016 (updated on Mar 5 2016) in

I have a wonderful article to share with those of you that are home-lab-curious, and/or VMware-curious. Specifically, an interest in the petite and increasingly powerful darling of the affordable home lab universe. There's a great new read ready for you, by virtualization blogger Florian Grehl:

I am a System Engineer specializing in cloud infrastructure design, virtualization and networking living in Germany near Hamburg.

VMware Homeserver – ESXi on 6th Gen Intel NUC

If your focus is on self-education and running a few VMs for various fun and education projects, this looks appealing. Perhaps you're trying to find a way to afford your own triple-node vSAN, then these little guys should do nicely for you as well, read onward for details. I freely admit, I have not tested a NUC first-hand, and am likely seen as rather biased given my extensive experience with the Supermicro lately, the only real server I currently own, on my multi-year power saving adventure. See also my full disclaimer below. Despite all that, I know this article will reveal at least some aspects of shopping for an efficient, quiet, and cool-running home lab that you may not have considered, especially if you're thinking of a cluster and/or vSAN. Yeah, I do enjoy researching the heck out of things before I lay down my hard-earned $, why not share my bookmarks. These are kits in the sense that you generally install your own memory and storage, then they're all ready to boot and use.

I've read many fun articles about Intel NUCs for years, such as this gem by none-other than Stephen Foskett:

Adding a Second Ethernet Port to an Intel NUC via Mini PCIe

I'm always interested in what's out there, and the creative cooling approaches folks come up with to work around the manufacturer's intended use case.

Now let's talk about support, should you feel you might need some for your virtualization endeavors.

VMware Compatibility Guide - System Search

vmware.com/resources/compatibility/search.php

Unfortunately, the Intel NUC is not on this list, nor are any other similar form factor systems such as the Gigabyte BRIX or even the latest Skylake models, the Core i5 NUC6i5SYH and NUC6i5SYK models. All this means is that you won't be calling VMware or Intel for support with any VMware related issues you might encounter. If your a DIYer type, you might not care, so let's talk next about...

Community Support

The support of your peers is very important, and seems to be very strong for the beloved NUC, just Google search for NUC6i5SYH esxi 6.

Things to know about the NUC, as an informed shopper

Note that there are some limitations, 32GB maximum memory, and some NVMe speed limitation on the Skylake model's M.2 slot, discussed here, with a direct link to page 45 of the manual:

Supports M.2 SSD SATA drives
- Maximum bandwidth is approximately 540 MB/s
Supports M.2 SSD PCIe drives (PCIe x1, x2, and x4)
- Using PCIe x4 M.2 SSD maximum bandwidth is approximately 1600 MB/s

For the Supermicro SuperServer, at full fan speeds you can do sustained benchmarking of the tiny M.2 Samsung 950 PRO NVMe SSD, without thermal throttling kicking in. I got slightly beyond the stated Samsung specs, obtaining a remarkable 2,598 MB/sec reads and 1,576 MB/sec writes. What does this mean? On full speed NVMe, an entire 18GB Windows 10 VM can be deployed from a template in 18 seconds. Wow, that's barely long enough for this tiny chip (about the size of a stick of gum) to heat up. As the only user of my VMware vSphere 6.0 datacenter-in-a-box, most of the time that little M.2 chip is idle, doing next to nothing and running cool. What SSD role is not so idle? Read onward...

vSAN

Known to be much tougher on flash storage, there's vSAN.

As far as I know, there's only one mini-tower efficient form-factor server with 10GbE that's on the VMware HCL here as of yesterday, the Supermicro SuperServer SYS-5028D-TN4T. There are some larger SFP-equipped Xeon D-1500 systems arriving from ASRock, and Gigabyte, but if RJ45s with affordable CAT6a or CAT7 cabling is more your home lab thing, the Supermicro fits the bill. Speaking of bills, Xeon D-1541 is admittedly considerably more expensive than the NUC. But remember, we're talking about 4 times the:

CPU threads (4 versus 16, with 24 and 36 variants arriving soon)
memory (32GB max versus 128GB max)
NIC ports (1 versus 4)
Network bandwidth (1Gb versus 22Gb total)

But that's just the VMware HCL. What about the vSAN list? Essentially RAID stretched over IP, so it really needs things like NVMe and 10GbE to shine. As I mentioned, vSAN is also known to be tough on SSDs, see also a recent presentation by Peter Keilty of VMware:

VMware Virtual SAN 6.2 Features Deduplication, Compression, and RAID5 or RAID6 over 10GbE

So that leads us to the next HCL, the more exclusive and more expensive list...

VMware Compatibility Guide - vSAN

www.vmware.com/go/virtualsan-hcl

Darn, now we're getting pretty pricey, with all sorts of enterprise-priced SSDs, PCIe SSDs, Ultra DIMMs, and NVMe SSDs. Yeah, it doesn't include consumer stuff like the widely-heralded Samsung 950 PRO, see also World's fastest consumer SSD - Samsung 950 PRO M.2 NVMe benchmark results.

Consider also the Intel 750 Series PCIe NVMe, if sustained abuse from vSAN is planned and you'd start to worry about temperatures. But it's still a consumer product. See also VMware ESXi 6.0 Windows 10 VM speed comparison between Intel 750 Series NVMe SSD and Samsung 950 PRO M.2 NVMe SSD.

Or step up all the way to enterprise-ready variants like the Intel SSD DC P3608 Series.

Remember, you have two tiers to equip yourself with, see also schematic below. Whether you have spindles you want to add a flash caching layer to, or you plan to go all-in an All Flash vSAN 6.2. Both types are pictured below.

VSANTiers — Remember, you have two tiers to equip yourself with, whether you have spindles you want to add a flash caching layer to, or you go with All Flash vSAN 6.2, pictured below.

If you're thinking vSAN, and you want fully supported pre-tested solutions, well, that's a whole 'nother world I won't claim to have experience tinkering in. Using admittedly unsupported gear and 2 loaner SuperServers I briefly had my hands on, here was my first look:

Supermicro SuperServer 5028D-TN4T vSAN dream is alive! Home lab video overview features Intel SSDs and Samsung M.2 SSDs

I just don't have enough experience with vSAN yet to know much of anything for sure, including how to determine how well counting TBW (Terabytes Written) from within ESXi will work for consumer focused SSD. I also don't even own a 10GbE switch, see closely related discussion of this [H]ard|Forum thread.

Instead of a three node cluster, what about two instead? See [2 Node Virtual SAN (ROBO vSAN) Deployment](2 Node Virtual SAN (ROBO vSAN) Deployment).
Maybe a SuperServer + Intel NUC combo starts to make some financial sense for just "kicking the tires" with vSAN, especially if the hefty performance compromise the 1GbE interface on the NUC is tolerable.

intel-nuc-vsan-datastore-300x194 — "VMware vSAN on Intel NUC – Mobile Cluster for $2000"

How about unsupported vSAN for NUC? No problem! It would seem Florian has also been on top of that since way back in Dec. 2014:

VMware vSAN on Intel NUC – Mobile Cluster for $2000

Go ahead and geek out on the Intel® NUC Products page and poke around, then dive back into the aforementioned article by Florian Grehl:

VMware Homeserver – ESXi on 6th Gen Intel NUC

Closing thoughts

There is no single correct or best solution for everybody. I realize this. This article just highlights a few approaches that might help you in your hardware research for gear that works well with VMware software.

It’s rather early days for vSAN, especially at home, where it’s likely to be a pretty pricey proposition for a while yet, in hardware, and don't forget, in software/licensing, with EVALExperience helping out greatly.

Mar 05 2016 Update

William Lam has jumped over from Apple Mac Mini to Intel NUC!

vSAN 6.2 (vSphere 6.0 Update 2) homelab on 6th Gen Intel NUC

Disclosure

TinkerTry.com, LLC is an independent site, has no sponsored posts, and all ads are run through 3rd party BuySellAds. All equipment and software is purchased for long-term productive use, and any rare exceptions are noted.

TinkerTry's relationship with Wiredzone is similar to the Amazon Associates program, where a very modest commission is earned from each referral sale from TinkerTry's SuperServer order page. I chose this trusted authorized reseller for its low cost and customer service, and a mutual desire to help folks worldwide, including a new way to reduce EU shipping costs. Why? Such commissions help reduce TinkerTry's reliance on advertisers, while building a community around the Xeon D-1500 chipset that strikes a great balance between efficiency and capability.

I personally traveled to Wiredzone near Miami FL to see the assembly room first-hand, and to Supermicro HQ in San Jose CA to share ideas and give direct product feedback.

I'm a full time IT Pro for the past 23 years. I've worked with IBM, HP, Dell, and Lenovo servers for hands-on implementation work across the US. Working from home lately, I'm quite enjoying finally owning a lower-cost Supermicro solution that I can recommend to IT Pro colleagues, knowing it will "just work." That's right, no tinkering required.