Haswell Z97 limited to 32GB memory max, an unfortunate reality for virtualization enthusiasts

I decided to open a web page feedback dialogue with Intel about Z97 memory limits earlier this week, on behalf of the home virtualization lab enthusiast community. I wanted to have confirmation before publishing this article, to be reasonably sure that Intel's site didn't have some sort of errors. Turns out there were some errors of omission, which I like to think my inquiry has helped address, but the apparent 32GB limit is no mistake:

From: [Intel employee] On Behalf Of ark_support Sent: Monday, June 02, 2014 5:57 PM To: Paul Braren Cc: ark_support Subject: RE: ARK.intel.com - Assistance Form #[111508] Paul, I just got the confirmation. There are no CPU/Z97 combinations that support 64MB Max Memory. I don’t know if it’s possible that there will be in the future or not. The Z97 supports Haswell CPUs and all the CPUs we have that presently support 64MB are all Ivy Bridge and below (the High End processors).

On June 3rd, an update from the same very helpful Intel employee:

I believe all of this information can be found through ark.intel.com or intel.com, so none of it is protected information. I’m working on getting the Z97 compatibilities in place and that will tell everyone which CPUs are compatible with the Z97 and that none of them support 64MB max memory.

*It's pretty safe to assume that 32GB was meant, not the 32MB that was typed.

[The following section was added just hours after the original article was published on June 7 2014]

See also the amazingly detailed, technical info on 32GB limits by multiple time TinkerTry commenter Tom on June 7 2014 here:

Broadwell may support 64GB (4x16GB DIMMs) but we don't know for certain yet. The new Intel Atom families (example: C2750 family) support 16GB UDIMMs for 64GB total but we're waiting for intelligentmemory.com to make the UDIMMs available. The only caveat is these Atom CPUs don't support VT-d (for all-in-one ESXi + SAN/NAS setups) in this generation.

This article explains the IMC limitations anandtech.com/show/7742/im-intelligent-memory-to-release-16gb-unregistered-ddr3-modules

and back in October 2013, read his full original post, here's just one of the paragraphs:

Haswell (and other mainstream) CPUs won't handle these DIMMs. 32GB is the limit due to the IMC. This is a market segmentation decision by Intel and I don't see it being lifted any time soon. This way people with large memory requirements either go towards X79 chipset-based boards and Intel enthusiast-class CPUs (Core i7 38xx/39xx or Core i7 48xx/49xx) which allow up to 64GB of non-ECC RAM. If >32GB RAM and ECC is required then a Xeon chipset (Intel C60x) and Xeon CPUs.

If you've tried to put together an affordable and powerful home virtualization server for yourself that is also efficient enough to be left running 24x7, you'll quickly notice that you can't easily find CPU and motherboard combinations that support anything greater than 32GB of memory, maximum. These days, even the free VMware ESXi 5.5 Hypervisor can handle well beyond that. Yep, 1TB actually. The 32GB of practicle memory barrier is evident in the list at the TinkerTry Superguide: Home virtualization server enthusiasts’ colorful variety of ESXi whiteboxes, so if you really like messing around with beefy VMs (such as SQL Server) that have a large working set of memory use, RAM is very likely going to be your first bottleneck felt, depitected at right. Dont' get me wrong, the Z97's heralding the arrival of the coveted M.2 interface really intrigues me, as I've mentioned way back in August of 2013. Finally, a way to get SSD performance that blows past SATA3 constraints. You can already browse Amazon here for the various Z97 motherboards already shipping. The dual M.2 equipped ASRock Z97 Extreme (pictured above) seems to be the current pick of the crop, featured at The SSD Review. But still, it's just 32GB of memory. Hardly justification to spend well over a thousand for me to upgrade my beloved Sandybridge Z68-based vZilla, which has been at 32GB for years. In the remainder of this article, we'll dive in together and take a look at the unfolding Intel chipset story. Today, and into the future. Assuming the use of a Core i7 (consumer) processor.

)

Recently, I noticed the wave of Z97 motherboard announcements roll in, just in case there was actually a way to get past 32GB. The many Z97s seen at [H]ard|OCP, for example. Nope, nothing beyond 32 GB spotted. So I dove in a bit deeper, and wanted to verify this limitation on a Haswell wiki page:

Native support for dual-channel DDR3 memory, with up to 32 GB of RAM on LGA 1150 variants

that doesn't look good. So time to head over to the usual Intel Product Information ark.intel.com site:

Intel® Z97 Chipset (Intel® DH82Z97 PCH) - see "Memory Specifications"

Memory Specifications # of DIMMs per channel 2

and dove in to inspect the maximum memory listings for all CPUs that'd fit go with that Z97 chipset:

4th Gen Intel® Core™ i7 Processors

and it's still a max of 32GB. Sigh. I was briefly thinking that you'd need to go with a bit older Intel Haswell-based E3-1xxx v3-series Xeons from May 2013, but then again, maybe not, seems those are stuck at 32GB of RAM too, see Maximum Memory of the Haswell Intel Xeon E3-1200 V3 Series. Stepping up to E5 series Xeons puts you on a path toward server class mobos and ECC DIMMs, which usually brings you into a whole different price category, in terms of the cost to purchase, then the cost to run (electricity, cooling). Most of the Xeon E5 family isn't exactly affordable, with the lowest TDP model tipping 130 watts. Ouch. Not for me, I want to go forward reducing electricity use, not backward by using more. So if you're really itching to use commodity, affordable, off-the-shelf modern hardware for your efficient 24x7 home lab in 2014, the outlook isn't good right now. Not unless you want to build on X79 or Z77 (from 2011), with 8 DIMM slots. You can see I've been talking about 32GB and 64GB for a while right here on TinkerTry. And you can bet I'll tell you about it once I do get to 64GB or beyond, affordably!

)

It's successor, Broadwell, has now been delayed until near the holidays for the mobile version:

Intel CEO 'Guarantees' Broadwell Chips by This Holiday Season reported on May 19 at MacRumors.

"I can guarantee for holiday, and not at the last second of holiday," Krzanich said in an interview. "Back to school - that's a tight one. Back to school you have to really have it on-shelf in July, August. That's going to be tough."
Worse news for the desktop variants, according to WCCF Tech:

Intel Roadmap Confirms Devil’s Canyon Delay To Q3 2014 – Haswell-E In September, Broadwell in Late Q1 2015.

Sources suggest that Broadwell for desktops is pushed to May – June 2015 from its original launch period of Q1, the same year. While Intel is pushing Broadwell for mobility devices in Holiday of 2014 (Q4 2014), the desktop lineup would be further delayed. Broadwell would be the first 14nm consumer chip and would be headed to mobility platforms first in Q4 2014 followed by desktop chips in Q1 / Q2 2015.

Skylake

Further into the future, it's Skylake, demonstrated at Computex in Taiwan last week, slated for late 2015 release. On June 4th 2014, Hassan Mujtaba posted this on WCCF Tech:

[Intel Skylake Processors To Launch in 2H 2015 – Compatible With LGA 1151 Socket and Z170 Chipset, Will Feature DDR3 / DDR4 Memory Support](Intel Skylake Processors To Launch in 2H 2015 – Compatible With LGA 1151 Socket and Z170 Chipset, Will Feature DDR3 / DDR4 Memory Support)

While Intel launched their 22nm Haswell processors last year (Tock), they were supposed to introduce Broadwell early this year however due to issues in 14nm production, the launch was halted till late 2014. Hence, this year, we have only seen an updated Haswell lineup in the form of the Haswell Refresh along with the 9-Series chipset platform. The 9-Series chipset was prepared specifically for Broadwell but due to launch delays, Intel tried into convincing their desktop users with new Refresh and the Devil’s Canyon processors.

DDR4

What about the expected arrival of DDR4 memory? It sure looks appealing, with faster speeds and lower energy costs. But according to Computerworld back in November of 2013:

DDR4 memory may not find way into PCs, tablets until 2015

Today, Intel's site:

Intel® Z97 Chipset (Intel® DH82Z97 PCH) - see "Memory Specifications"

Memory Specifications # of DIMMs per channel 2
Eventually, with Skylake:

Skylake gets both DDR3 and DDR4 support + more

Finally, there's this Swedish June 3 2014 article with an MSI DDR4 X99 motherboard pictured, indicating first in world USB 3.1, and:

...using DDR4 memory standard. This should result in lower power consumption, higher bandwidth and the ability to use memory modules of 32GB or more. Initially comes four channels at up to 2133 MHz.

See also

CPU Category

TinkerTry

Superguide: Timeline of USB, SATA, and Thunderbolt speeds

TinkerTry

Superguide: Home virtualization server enthusiasts’ colorful variety of ESXi whiteboxes

TinkerTry

Intel 9-Series Chipsets

bit-tech, May 11 2014

Intel 2014 Enthusiast Processors: Haswell-E To Feature 8 Cores, X99, DDR4

AnandTech, March 19 2014

Intel Haswell-E Core i7-5960X, Core i7-5930K, Core i7-5820K Specifications Unveiled – Flagship 8 Core To Boost Up To 3.3 GHz

WCCF Tech, May 24 2014

Jul 07 2014 Update

Thanks to Twitter commenter Pat Richards @patrichards, who points out Atom in this tweet, with the Intel® Atom™ Processor C2750 with 8 cores burning just 20 watts, maximum:
ark.intel.com/products/codename/54859/Avoton

Intel® Atom™ Processor C2000 product Family (codenamed Avoton and Rangeley) is the next generation System-On-Chip (SoC) built on Intel’s 22-nanometer process technology.

Atom (aka Avoton/Rangely) server SoC (System on a Chip) supports VT-d

ESXi and Atom Avoton/Rangeley new SoCs
damiendebin.net, Mar 03 2014

Here's a motherboard/CPU combos with the C2558, the Supermicro A1SRi-2558F, see also Amazon listing:
Supermicro Mini ITX DDR3 1600 NA Motherboards MBD-A1SRi-2558F-O

See also Supermicro A1SAi-2750F Review – Intel Avoton Platform.

Amazon commenter
Rads confirms this Supermicro Atom C2750 works with ESXi 5.5 just fine. Note that Supermicro hasn't yet stated any of their Atom based servers support ESXi 5.5 yet, at Supermicro compatibility chart.

See also ASRock C2750D4I. Note that nobody seems to be saying they've successfully tested (and afforded) 64GB yet, examples here and here.

See also Supermicro A1SAi-2750F vs ASRock C2750D4I.

Aug 10 2014 Update

We now know the Haswell-E enthusiast chipset will be called the X99S.

World's First Windows® 8.1-Certified Intel® X99 Chipset Based Motherboard : ASRock X99 Extreme4

asrock.com/news/index.asp?id=2299
by ASRock on Aug 01 2014

MSI’s Next Haswell-E Teaser: X99S Gaming 9 AC_

anandtech.com/show/8349/msis-next-haswell-e-teaser-x99s-gaming-9-ac
by Ian Cutress on Aug 07 2014

This Week in Computer Hardware 279 - It's a NAS and a HTPC (Aug 08 2014)

twit.tv/show/this-week-in-computer-hardware/279
by Ryan Shrout and Patrick Norton on August 08 2014
or jump to just the right spot in the podcast:
pca.st/arde

Haswell Z97 limited to 32GB memory max, an unfortunate reality for virtualization enthusiasts

TinkerTry.com/haswell-z97-limited-to-32gb-memory-max-an-unfortunate-reality-for-virtualization-enthusiasts
by Paul Braren on June 07 2014