(I've not been updating this article since May 2015, because I've been focusing my efforts on a big step up from typical of Intel NUC / Shuttle PCs, moving to 128GB of RAM with the Xeon D-1540, details here)
Let's start with the shopping tools. If you're after which mobo works with which CPU, then pcpartspicker.com and the virtualized computing forum are excellent resources. But if you're after solutions that work well with VMware ESXi 5.5, the going gets a little bit tougher. This guide will help you as you begin to shop around for options. While Dell, HP, and IBM all make plenty of server class gear, the initial cost of the hardware, and the ongoing cost of the electricity and cooling load often means fully redundant gear is overkill. Many home lab builders would rather find an (unsupported) consumer class system they can leave running 24x7, staying under ~125 watts total consumption. Lately, Supermicro and Tyan motherboards with Xeon processors are coming down in price as well, even when loaded up with ECC DIMMs, such as the gear over at ServeTheHome.com.
There are companies out there that make pre-assembled solutions that claim to run quietly and efficiently, such as UK's Brontastor. But what if you really want to dive in and assemble your own system, learning a lot along the way? That's what I did back in 2011, with Z68 Sandy Bridge vZilla build.
Over the past 2+ years, many TinkerTry visitors asked me about newer ESXi home builds out there that folks have blogged about in detail. Of course, I keep an eye on things. I've come across quite a few articles, collected below for your convenience.
What's my overall impression of how Haswell has been going for ESXi builds lately? Folks seem happy overall, with fairly low watt burn, even with the capable GPU that goes largely unused in most home labs.
But honestly, things haven't changed that much since Sandy Bridge, with most of the focus on mobile CPU development. The biggest limitation of Haswell is the 32GB maximum chipset support. It's not that we're waiting for a higher density DIMM for those 4 slots, it's worse than that. The 32GB limit is in Intel's Haswell specs. And some folks report early adopter pains, such as pcdoc's series of Haswell stories, but those sorts of issues are true . See also VMware ESXi 5.5 Free Hypervisor breaks past 32GB RAM restriction and Should I try to upgrade my Intel Core i7 2600 (Sandybridge) past 32GB of RAM for juggling a lot of VMs, or plan to rebuild on Core i7 4770 (Haswell)?
I typically have about 6-8 VMs running 24x7, and those are using up about 90% of my 32GB available. I realize the X79 chipset offered 64GB compatibility. But it's getting too long in the tooth for me to bother with this upgrade to my Sandy Bridge system. Let's hope that the wait isn't too long before there's an affordable option for home virtualization enthusiasts to get past 32GB of RAM, whether that be a Core i7, Xeon, or even AMD processor. So far, it's not looking like the next chipsets, Haswell-E and Broadwell, will have support for >32GB of RAM either, but at least M.2 support is likely (for faster than SATA SSDs). Time will tell. See also related articles about M.2 and Timeline of USB, SATA, and Thunderbolt speeds.
This article is really about collecting just some of the apparent successes already out there, intended to help you be more informed if you're researching your own white-box build in early 2014.
Please remember the fairly narrow focus for this particular post:
- Haswell or recent Xeons that work well with ESXi 5.5
- well suited for 24x7 use, defined loosely as quiet and economical, using less than <125 watts for the system + internal storage
- less than roughly $2000 USD
- internal fast storage suitable for day-to-day use (assuming you have the appropriate licensing)
- alternatively, for an approach that more closely approximates a datacenter's LAN topology in your home, see Mark Gabryjelski's build in the Ivy Bridge section below
- my own focus happens to be more on simplicity of initial set up, which for my lab means:
- no separate DNS
- no managed network switch
- no SQL Server installation
such as my vZilla build outlined in this simple-as-possible guide with video that is generic enough to work for most ESXi compatible hardware. There will be a refresh of those instructions soon, based on the latest 5.5.0b version of the vCenter appliance, things are even simpler now, with no more hosts file tweaks needed on the vCenter appliance, since SSO configuration now works without those tweaks.
If you also have a successful home lab build you'd like me to consider adding to this list, please drop a comment below!
by Steve Smith @SteveActual on Aug 26 2013
by Tom Fojta @fojta on Mar 16 2014
by +Paul Braren @tinkererguy on Aug 22 2012
by +Mark Gabryjelski @MarkGabbs on Jan 13 2014
With Mark's consent, I put together his build under a single Amazon (TinkerTry Affiliate) Wishlist, just click the Amazon logo at right, with a screenshot below it showing exactly how the shopping cart looks configured with 32GB of RAM and 4 1TB drives.
by +Ryan Birk @ryanbirk on Dec 8 2013
by +Josh Atwell @Josh_Atwell on Aug 10 2013
by +Matt Hill @matthilluk on Mar 02 2013
Presented at Security BSides Boston on May 18 2013, “Build Your Own VMWare ESXi and Microsoft Hyper-V lab at Home, Using Affordable and Efficient Hardware”
by +Don Fountain in early 2013
where Don also goes on to discuss 64GB on Intel processors in the comments.
Thanks to Twitter commenter Pat Richards @patrichards, who points out Atom in this tweet, with the Intel® Atom™ Processor C2750 with 8 cores burning just 20 watts, maximum:
Intel® Atom™ Processor C2000 product Family (codenamed Avoton and Rangeley) is the next generation System-On-Chip (SoC) built on Intel’s 22-nanometer process technology.
Here's a motherboard/CPU combos with the C2558, the Supermicro A1SRi-2558F, see also Amazon listing:
Supermicro Mini ITX DDR3 1600 NA Motherboards MBD-A1SRi-2558F-O
Amazon commenter Rads confirms this Supermicro Atom C2750 works with ESXi 5.5 just fine. Note that Supermicro hasn't yet stated any of their Atom based servers support ESXi 5.5 yet, at Supermicro compatibility chart.
William Lam is on a roll with Apple stuff lately, all three of these are an awesome read:
A killer custom Apple Mac Mini setup running vSAN
How to install ESXi 5.5 Patch03 on the new Mac Pro 6,1?
Want to run ESXi on an Apple MacBook Pro, MacBook Air & iMac? #YesYouCan!
Big week in the home server world, with the announcement of the very promising Broadwell-DE, aka, Xeon D. Read ALL about it at this linkfest:
Broadwell-DE announced – Intel Xeon D-1520/1540 crams incredible virtualization lab specs into tiny mITX size
Things are changing!