TinkerTry's Xeon D cluster for VMworld 2016, man with a vSAN demo plan!
Fun new time-lapse video now available, in the updates section below the original article.
The idea here is not what you should do in a home lab, it's really just determining what's become possible. Let's hold-off on concluding much of anything until I get this All-Flash vSAN rig fully operational, hopefully in time for my attendance at VMworld 2016 Las Vegas. See also my article about how vSAN essential ties your SSDs together as one storage pool, using 10GbE interconnected NVMe as an extremely-fast caching layer:
I can say that I was giddy with how well this crazy year+ home vSAN dream is finally coming together. Heck, not just the recently reduced price on suitable Xeon D servers, but details, like those tiny 10GbE CAT6a cables. It all just has me so super eager to share this build, and it's an amazing opportunity for me to try to answer questions about how core counts and cache matter to virtualization performance.
Don't get me wrong, I'm well aware that even entry level 8 port 10GbE switches cost nearly $1000, but at least one model became quiet enough this month to consider, and doesn't use more watts than all the servers combined. Fingers crossed that those 10GbE switch prices drop soon.
Now that I (temporarily) have the parts, I can begin to fire these suckers up! Configure the BIOS, install operating systems, re-test watts and noise. Finally, I know!
Oh, and I'm not sure where I'll be able to demonstrate this rig at VMworld. Still working on that. Any ideas?
I'm out of words for today, but that's OK, this is my mic-drop moment, this one photo speaks for itself.
Parts list for VMworld 2016 demonstration
Top to bottom, then left to right:
-
(1) Supermicro SuperServer SYS-5028D-TN4T (8 Core Xeon D-1541) from Wiredzone.
-
(1) Cisco SG112-24 unmanaged 24 port gigabit switch from Amazon, also available on Newegg, soon on Wiredzone.
-
(1) Netgear XS708T managed 8 port 10GbE switch from Amazon,
also available on Newegg, soon on Wiredzone. -
(1) Supermicro SuperServer SYS-E300-8D (4 Core Xeon D-1518)
on loan from Supermicro via Wiredzone. -
(1) Supermicro SuperServer SYS-E200-8D (6 Core Xeon D-1528)
on loan from Supermicro via Wiredzone. -
(2) 10-Pack, SlimRun™ Cat6A Ethernet Network Patch Cable, 1ft Black
from Monoprice, Product # 15149, with Monoprice's list of lengths and colors here. - (1) 10-Pack, SlimRun™ Cat6A Ethernet Network Patch Cable, 1ft Blue
from Monoprice Product # 15150.
Parts list - inside
-
(4) 32GB DDR4 memory modules per Xeon D system, with a mix of 2400 (loaned) and 2133 (owned) speeds, see also discussion here.
-
(3) M510DC 2.5" SATA3 SSD 960GB,
on loan with one for each system, available at Amazon and Newegg. -
(1) Supermicro RSC-RR1U-E8 Riser Card available at Amazon and Newegg, backordered on Wiredzone
Supermicro Support hotline admitted they haven't actually tested this riser card with the SYS-E300-8D yet. - (3) Micron NVMe, models TBD, on loan with one for each system.
Unfortunately, I just learned these crical parts aren't arriving until mid-August, so this evening-and-weekend VMworld prep is going to be a challenge!
Parts list - not pictured
-
(1) Supermicro SuperServer SYS-5018D-FN45 (8 Core Xeon D-1541)
on loan from Supermicro via Wiredzone, just arrived on Aug 5. - (2) Dell C6Y7M Force10 Passive Copper Network Cable SFP-10GE-PASS-0.5M SFP+ to SFP+ available at Amazon.
The Supermicro CBL-0347L 1M 10GbE SFP+ to SFP+ pull release type 30AWG adapters are tough to find in stock, check Amazon, Newegg, and Wiredzone.
Saving money, there's the SFP+ AOC-E10GSFPSR copper patch cable assembly, confirmed by Supermicro Support to be suitable for use with the SYS-E300-8D. Affordable 3rd party 0.5 meter SFP+ to SFP+ 10GbE copper parts likely work fine as well, such as the 0.5m SFP+ Cable - Amphenol 10GbE SFP+ Direct Attach Copper Cable (1.6 ft) available at Cables On Demand. I went with the Dell C6Y7M mostly because of its short length. Doesn't hurt that it happens to be somewhat color-coordinated.
Network details
Given how many network ports we're talking about here, it's rather important to stay organized!
DHCP reservations and DNS courtesy of my newly beloved Ubiquiti EdgeRouter Lite from Amazon, also available on Newegg, and coming to Wiredzone soon. No write-up or review yet, but for now know that things are going well after 3 weeks of HEAVY testing. Once I publish a deployment guide, it will be easier for anybody with decent consumer router configuration skills to follow along.
AUG 03 2016 Update
Yeah, we've all heard how important it is for the SSDs one uses for vSAN to be on the vSAN portion of the VMware Compatiblity Guide, but nothing like a heaping dose of reality for me to keep things real, see S's comments from below this article:
Thank you, S, your story sharing is always greatly appreciated, and you may have just saved me a great deal of trouble, given how little time I have before VMworld to get something working. Let's hope Micron (or somebody else) pulls through!
Keep in mind that the arrangement tinkering is still underway. With rubber feet now on all the components, sliding things around this isn't an issue. With modest weights for each device, piling them up like this isn't a problem either, even if it's not exactly how the products were designed to (temporarily) be used.
I may want to go with this sort of arrangement pictured at right, with all the networking resting on a surface behind the 3 systems. The mini tower overhang out back looks a bit strange, but any arrangement will be a compromise.
FYI, I have some 3' Monoprice SlimRun cable lengths on the way that will allow me much more re-arrangement flexibility, stay tuned. Wiredzone also now plans to carry their line of slender cabling too, see search results for 1' to 14' lengths at Monoprice, in 5 or 10 packs of black, blue, grey, or white.
Time Lapse Video
AUG 04 2016
I received an email blast from Ubiquiti yesterday, having signed up for forum access. The EdgeSwitch 16 XG sure got my attention:
The ES-16-XG features sixteen independent 10 Gbps switching ports:
12 SFP+ ports and 4 RJ45 10GBASE-T ports. It delivers maximum
performance and low latency, providing up to 160 Gbps of total
non-blocking throughput.
Especially once I noticed the $599 pre-order price at B&H. Apparently, I'm not alone:
Too bad it's not out yet. For this project, not really enough copper RJ45 10GbE ports, but still very intriguing...
Received the 1.5m Dell SFP+ to SFP+ cable arrived today, install was super easy, testing to follow.
AUG 05 2016
Just arrived! To be added to the pictured-above cluster, likely with just 3 of the 4 systems participating in the planned vSAN:
- (1) Supermicro SuperServer SYS-5018D-FN45 (8 Core Xeon D-1541)
on loan from Supermicro via Wiredzone.
Reactions
See also at TinkerTry
- Supermicro SYS-E300-8D/SYS-E200-8D/SYS-5028D-TN4T and Intel NUC NUC6i7KYK - decibel and watt comparison
JUL 31 2016
Hints about what I'm doing for WiFi in this article, with my eero testing (in bridged mode with wired backhauls to Ubiquiti router) going rather well:
- Intel's first M.2 NVMe SSD might be available by late 2016, around the time Intel Micron 3D XPoint arrives
FEB 26 2016
