TinkerTry's Xeon D cluster for VMworld 2016, man with a VSAN demo plan!

Posted by Paul Braren on Aug 2 2016 (updated on Aug 31 2016) in
  • Virtualization
  • ESXi
  • HomeLab
  • HomeServer
  • Storage
  • Presentation
  • Fun new time-lapse video now available, in the updates section below the original article.

    The idea here is not what you should do in a home lab, it's really just determining what's become possible. Let's hold-off on concluding much of anything until I get this All-Flash VSAN rig fully operational, hopefully in time for my attendance at VMworld 2016 Las Vegas. See also my article about how VSAN essential ties your SSDs together as one storage pool, using 10GbE interconnected NVMe as an extremely-fast caching layer:

    I can say that I was giddy with how well this crazy year+ home VSAN dream is finally coming together. Heck, not just the recently reduced price on suitable Xeon D servers, but details, like those tiny 10GbE CAT6a cables. It all just has me so super eager to share this build, and it's an amazing opportunity for me to try to answer questions about how core counts and cache matter to virtualization performance.

    Don't get me wrong, I'm well aware that even entry level 8 port 10GbE switches cost nearly $1000, but at least one model became quiet enough this month to consider, and doesn't use more watts than all the servers combined. Fingers crossed that those 10GbE switch prices drop soon.

    Now that I (temporarily) have the parts, I can begin to fire these suckers up! Configure the BIOS, install operating systems, re-test watts and noise. Finally, I know!

    Oh, and I'm not sure where I'll be able to demonstrate this rig at VMworld. Still working on that. Any ideas?

    I'm out of words for today, but that's OK, this is my mic-drop moment, this one photo speaks for itself.

    Click once to zoom, again to zoom further, or use the fullscreen toggle at top-right for maximum effect.

    Parts list for VMworld 2016 demonstration

    Top to bottom, then left to right:

    Parts list - inside

    • (4) 32GB DDR4 memory modules per Xeon D system, with a mix of 2400 (loaned) and 2133 (owned) speeds, see also discussion here.

    • (3) M510DC 2.5" SATA3 SSD 960GB,
      on loan with one for each system, available at Amazon and Newegg.

    • (1) Supermicro RSC-RR1U-E8 Riser Card available at Amazon and Newegg, backordered on Wiredzone
      Supermicro Support hotline admitted they haven't actually tested this riser card with the SYS-E300-8D yet.

    • (3) Micron NVMe, models TBD, on loan with one for each system.
      Unfortunately, I just learned these crical parts aren't arriving until mid-August, so this evening-and-weekend VMworld prep is going to be a challenge!

    Parts list - not pictured

    The Supermicro CBL-0347L 1M 10GbE SFP+ to SFP+ pull release type 30AWG adapters are tough to find in stock, check Amazon, Newegg, and Wiredzone.

    DHCP reservations rev7.

    Saving money, there's the SFP+ AOC-E10GSFPSR copper patch cable assembly, confirmed by Supermicro Support to be suitable for use with the SYS-E300-8D. Affordable 3rd party 0.5 meter SFP+ to SFP+ 10GbE copper parts likely work fine as well, such as the 0.5m SFP+ Cable - Amphenol 10GbE SFP+ Direct Attach Copper Cable (1.6 ft) available at Cables On Demand. I went with the Dell C6Y7M mostly because of its short length. Doesn't hurt that it happens to be somewhat color-coordinated.

    Network details

    Given how many network ports we're talking about here, it's rather important to stay organized!

    DHCP reservations and DNS courtesy of my newly beloved Ubiquiti EdgeRouter Lite from Amazon, also available on Newegg, and coming to Wiredzone soon. No write-up or review yet, but for now know that things are going well after 3 weeks of HEAVY testing. Once I publish a deployment guide, it will be easier for anybody with decent consumer router configuration skills to follow along.

    Ubiquiti EdgeRouter Lite, handles Gigabit internet, under $100 USD.

    AUG 03 2016 Update

    Yeah, we've all heard how important it is for the SSDs one uses for VSAN to be on the vSAN portion of the VMware Compatiblity Guide, but nothing like a heaping dose of reality for me to keep things real, see S's comments from below this article:


    Thank you, S, your story sharing is always greatly appreciated, and you may have just saved me a great deal of trouble, given how little time I have before VMworld to get something working. Let's hope Micron (or somebody else) pulls through!


    Keep in mind that the arrangement tinkering is still underway. With rubber feet now on all the components, sliding things around this isn't an issue. With modest weights for each device, piling them up like this isn't a problem either, even if it's not exactly how the products were designed to (temporarily) be used.

    I may want to go with this sort of arrangement pictured at right, with all the networking resting on a surface behind the 3 systems. The mini tower overhang out back looks a bit strange, but any arrangement will be a compromise.

    FYI, I have some 3' Monoprice SlimRun cable lengths on the way that will allow me much more re-arrangement flexibility, stay tuned. Wiredzone also now plans to carry their line of slender cabling too, see search results for 1' to 14' lengths at Monoprice, in 5 or 10 packs of black, blue, grey, or white.

    Time Lapse Video

    There's a balance here between minimizing how many square feet of space this rig will take up in tight booth or hotel room space. You'll see time-lapse exposure-locked footage of my trying out different arrangements. A little like playing Tower of Hanoi, only way more fun. Published AUG 02, 2016.

    AUG 04 2016

    I received an email blast from Ubiquiti yesterday, having signed up for forum access. The EdgeSwitch 16 XG sure got my attention:

    The ES-16-XG features sixteen independent 10 Gbps switching ports:
    12 SFP+ ports and 4 RJ45 10GBASE-T ports. It delivers maximum
    performance and low latency, providing up to 160 Gbps of total
    non-blocking throughput.

    Especially once I noticed the $599 pre-order price at B&H. Apparently, I'm not alone:


    Too bad it's not out yet. For this project, not really enough copper RJ45 10GbE ports, but still very intriguing...

    Received the 1.5m Dell SFP+ to SFP+ cable arrived today, install was super easy, testing to follow.

    AUG 05 2016

    Just arrived! To be added to the pictured-above cluster, likely with just 3 of the 4 systems participating in the planned VSAN:



    See also at TinkerTry

    Hints about what I'm doing for WiFi in this article, with my eero testing (in bridged mode with wired backhauls to Ubiquiti router) going rather well: