Superguide: Home virtualization server enthusiasts' colorful variety of ESXi whiteboxes

Posted by Paul Braren on Jan 5 2014 (updated on Jan 14 2016) in
  • ESXi
  • Featured
  • HomeLab
  • Superguides
  • Virtualization
  • (I've not been updating this article since May 2015, because I've been focusing my efforts on a big step up from typical of Intel NUC / Shuttle PCs, moving to 128GB of RAM with the Xeon D-1540, details here)


    Let's start with the shopping tools. If you're after which mobo works with which CPU, then pcpartspicker.com and the virtualized computing forum are excellent resources. But if you're after solutions that work well with VMware ESXi 5.5, the going gets a little bit tougher. This guide will help you as you begin to shop around for options. While Dell, HP, and IBM all make plenty of server class gear, the initial cost of the hardware, and the ongoing cost of the electricity and cooling load often means fully redundant gear is overkill. Many home lab builders would rather find an (unsupported) consumer class system they can leave running 24x7, staying under ~125 watts total consumption. Lately, Supermicro and Tyan motherboards with Xeon processors are coming down in price as well, even when loaded up with ECC DIMMs, such as the gear over at ServeTheHome.com.

    There are companies out there that make pre-assembled solutions that claim to run quietly and efficiently, such as UK's Brontastor. But what if you really want to dive in and assemble your own system, learning a lot along the way? That's what I did back in 2011, with Z68 Sandy Bridge vZilla build.

    Over the past 2+ years, many TinkerTry visitors asked me about newer ESXi home builds out there that folks have blogged about in detail. Of course, I keep an eye on things. I've come across quite a few articles, collected below for your convenience.

    What's my overall impression of how Haswell has been going for ESXi builds lately? Folks seem happy overall, with fairly low watt burn, even with the capable GPU that goes largely unused in most home labs.

    But honestly, things haven't changed that much since Sandy Bridge, with most of the focus on mobile CPU development. The biggest limitation of Haswell is the 32GB maximum chipset support. It's not that we're waiting for a higher density DIMM for those 4 slots, it's worse than that. The 32GB limit is in Intel's Haswell specs. And some folks report early adopter pains, such as pcdoc's series of Haswell stories, but those sorts of issues are true . See also VMware ESXi 5.5 Free Hypervisor breaks past 32GB RAM restriction and Should I try to upgrade my Intel Core i7 2600 (Sandybridge) past 32GB of RAM for juggling a lot of VMs, or plan to rebuild on Core i7 4770 (Haswell)?

    I typically have about 6-8 VMs running 24x7, and those are using up about 90% of my 32GB available. I realize the X79 chipset offered 64GB compatibility. But it's getting too long in the tooth for me to bother with this upgrade to my Sandy Bridge system. Let's hope that the wait isn't too long before there's an affordable option for home virtualization enthusiasts to get past 32GB of RAM, whether that be a Core i7, Xeon, or even AMD processor. So far, it's not looking like the next chipsets, Haswell-E and Broadwell, will have support for >32GB of RAM either, but at least M.2 support is likely (for faster than SATA SSDs). Time will tell. See also related articles about M.2 and Timeline of USB, SATA, and Thunderbolt speeds.

    This article is really about collecting just some of the apparent successes already out there, intended to help you be more informed if you're researching your own white-box build in early 2014.

    Please remember the fairly narrow focus for this particular post:

    • Haswell or recent Xeons that work well with ESXi 5.5
    • well suited for 24x7 use, defined loosely as quiet and economical, using less than <125 watts for the system + internal storage
    • less than roughly $2000 USD
    • internal fast storage suitable for day-to-day use (assuming you have the appropriate licensing)
    • alternatively, for an approach that more closely approximates a datacenter's LAN topology in your home, see Mark Gabryjelski's build in the Ivy Bridge section below
    • my own focus happens to be more on simplicity of initial set up, which for my lab means:
      • no separate DNS
      • no managed network switch
      • no SQL Server installation

    such as my vZilla build outlined in this simple-as-possible guide with video that is generic enough to work for most ESXi compatible hardware. There will be a refresh of those instructions soon, based on the latest 5.5.0b version of the vCenter appliance, things are even simpler now, with no more hosts file tweaks needed on the vCenter appliance, since SSO configuration now works without those tweaks.

    If you also have a successful home lab build you'd like me to consider adding to this list, please drop a comment below!


    )

    Haswell ASRock Z87 Extreme6 LGA 1150 build

    by TomasF in 2013
    TinkerTry.com visitor comment # 1188600891 below

    Haswell ESXi Build – Low Power & Low Noise

    by Steve Smith @SteveActual on Aug 26 2013
    vdobbs.com/2013/08/26/low-power-low-noise-haswell-esxi-host

    Haswell ESXi Whitebox

    by +Vladan Seget @vladan on Jun 22 2013
    vladan.fr/haswell-esxi-whitebox

    Haswell low power whitebox for ESXi and Hyper-V

    by +Ivo Beerens @ibeerens on Jun 06 2013
    ivobeerens.nl/2013/06/25/haswell-low-power-whitebox-for-esxi-and-hyper-v

    Lab Designs

    by +Chris Wahl @ChrisWahl on Jul 23 2013
    wahlnetwork.com/lab-designs

    Shuttle PC - vSphere 5.5 White Box Gotchas

    by Tom Fojta @fojta on Mar 16 2014
    fojta.wordpress.com/2014/03/16/shuttle-pc-vsphere-5-5-white-box-gotchas

    The New Haswell Fueled ESXi 5.5 Home Lab Build

    by +Chris Wahl @ChrisWahl on Dec 02 2013
    wahlnetwork.com/2013/12/02/new-haswell-fueled-esxi-5-5-home-lab-build

    Three Example Home Lab Storage Designs using SSDs and Spinning Disk

    by +Chris Wahl @ChrisWahl on Oct 14 2013
    wahlnetwork.com/2013/10/14/three-example-home-lab-storage-designs-using-ssds-spinning-disk/#more-8512


    )

    ASRock Z77 Extreme4-M Motherboard works nicely with ESXi 5.0

    by +Paul Braren @tinkererguy on Aug 22 2012
    TinkerTry.com/asrock-z77-extreme4-m/#comment-1059010467

    How to Build A Home Lab for $2,000 - The Presentation

    by +Mark Gabryjelski @MarkGabbs on Jan 13 2014
    YouTube.com/channel/UCNwtoV_iLvZy1fTcVl5UnUg
    With Mark's consent, I put together his build under a single Amazon (TinkerTry Affiliate) Wishlist, just click the Amazon logo at right, with a screenshot below it showing exactly how the shopping cart looks configured with 32GB of RAM and 4 1TB drives.

    Running ESXi 5.0 & 5.1 on 2012 Mac Mini 6.2

    by +William Lam @lamw on Dec 04 2012
    virtuallyghetto.com/2012/12/running-esxi-50-51-on-2012-mac-mini-62.html

    The Shuttle XH61V Ultra Mini vSphere Lab

    by +Ryan Birk @ryanbirk on Dec 8 2013
    ryanbirk.com/the-shuttle-xh61v-ultra-mini-vsphere-lab

    The Updated Home Lab 2.0

    by +Brian Graf @vTagion on Mar 06 2013
    vTagion.com/updated-home-lab-2-0

    The vTesseract Lab

    by +Josh Atwell @Josh_Atwell on Aug 10 2013
    vtesseract.com/post/57929678657/the-vtesseract-lab

    VMware ESXI Low Power Home Server Lab Build

    by +Matt Hill @matthilluk on Mar 02 2013
    matthill.eu/projects/vmware-esxi-low-power-home-lab


    )

    Drezilla – Allan Dresner’s Core i7 X79 Home Storage and Virtualization Server, your thoughts?

    by +Paul Braren @tinkererguy on Apr 05 2012
    TinkerTry.com/drezilla

    Intel X79 whitebox for vSphere 5 and Hyper-V 3

    by +Ivo Beerens @ibeerens on Jun 06 2013
    ivobeerens.nl/2013/06/25/haswell-low-power-whitebox-for-esxi-and-hyper-v


    )

    Baby Dragon Home Lab

    by +Eric Shanks @eric_shanks on Apr 2013
    theithollow.com/2013/04/new-baby-dragon-home-lab


    )

    Building ESXi 5 Whitebox Home Lab Servers

    by +Chris Wahl @ChrisWahl on Mar 13 2012
    wahlnetwork.com/2012/03/13/building-esxi-5-whitebox-home-lab-servers

    Haswell ESXi Whitebox

    by +Vladan Seget @vladan on Jun 22 2013
    vladan.fr/haswell-esxi-whitebox

    Presented at Security BSides Boston on May 18 2013, “Build Your Own VMWare ESXi and Microsoft Hyper-V lab at Home, Using Affordable and Efficient Hardware”

    by +Paul Braren @tinkererguy on May 18 2013
    TinkerTry.com/security-bsides-boston-presentation-may-18-2013

    Superguide: “vZilla” Z68/Core i7/LSI 9265-8i – built for virtualization, storage, and speed

    by +Paul Braren @tinkererguy on Jul 12 2011
    TinkerTry.com/vzilla


    )

    8-Core, 32GB RAM, 360GB Flash, 2TB, Dual-NIC Home Lab Part List

    by +Jonathan Frappier @jfrappier on Jan 17 2014
    virtxpert.com/8-core-32gb-ram-360gb-flash-2tb-dual-nic-home-lab-part-list

    ESXi Home Lab Specs

    by +Don Fountain in early 2013
    thehomeserverblog.com/category/esxi-whitebox-builds
    where Don also goes on to discuss 64GB on Intel processors in the comments.

    ESXi 5 on MicroServer?

    by +John Stutsman @JohnStutsman on Jan 30 2012
    homeservershow.com/forums/index.php?/topic/3490-esxi-5-on-microserver

    Home Lab (HP MicroServer N40L)

    by +Manfred Hofer @Fred_vBrain on unknown date
    vbrain.info/?page_id=230

    Quad-Core, 32GB RAM, 240GB Flash, 2TB, Dual-NIC Home Lab Part List

    by +Jonathan Frappier on Jan 17 2014
    virtxpert.com/quad-core-32gb-ram-240gb-flash-2tb-dual-nic-home-lab-part-list


    Jul 07 2014 Update

    Thanks to Twitter commenter Pat Richards @patrichards, who points out Atom in this tweet, with the Intel® Atom™ Processor C2750 with 8 cores burning just 20 watts, maximum:
    ark.intel.com/products/codename/54859/Avoton

    Intel® Atom™ Processor C2000 product Family (codenamed Avoton and Rangeley) is the next generation System-On-Chip (SoC) built on Intel’s 22-nanometer process technology.

    Atom (aka Avoton/Rangely) server SoC (System on a Chip) supports VT-d

    A1SRi-2558F

    ESXi and Atom Avoton/Rangeley new SoCs
    damiendebin.net, Mar 03 2014

    Here's a motherboard/CPU combos with the C2558, the Supermicro A1SRi-2558F, see also Amazon listing:
    Supermicro Mini ITX DDR3 1600 NA Motherboards MBD-A1SRi-2558F-O

    See also Supermicro A1SAi-2750F Review – Intel Avoton Platform.

    Amazon commenter Rads confirms this Supermicro Atom C2750 works with ESXi 5.5 just fine. Note that Supermicro hasn't yet stated any of their Atom based servers support ESXi 5.5 yet, at Supermicro compatibility chart.

    See also ASRock C2750D4I. Note that nobody seems to be saying they've successfully tested (and afforded) 64GB yet, examples here and here.

    See also Supermicro A1SAi-2750F vs ASRock C2750D4I.


    Nov 05 2014 Update

    William Lam is on a roll with Apple stuff lately, all three of these are an awesome read:
    A killer custom Apple Mac Mini setup running vSAN
    How to install ESXi 5.5 Patch03 on the new Mac Pro 6,1?
    Want to run ESXi on an Apple MacBook Pro, MacBook Air & iMac? #YesYouCan!


    Mar 12 2015 Update

    SMCI_X10SDV-TLN4F_Angled
    Image of X10SDV-TLN4F courtesy of Supermicro

    Big week in the home server world, with the announcement of the very promising Broadwell-DE, aka, Xeon D. Read ALL about it at this linkfest:
    Broadwell-DE announced – Intel Xeon D-1520/1540 crams incredible virtualization lab specs into tiny mITX size

    Things are changing!