Superguide: Z68 Sandybridge Motherboard VT-d Test Matrix: Which Mobo/CPU combo works with VMware ESXi 4.1U1 VMDirectPath feature?

I’ve tested 4 brands of Z68 motherboards out, and the results are in, certain ASRock and MSI Z68 Motherboards can do VMDirectPath!

I tested these 4 mid price point boards myself while building vZilla and each cost somewhere between $170-$230.

As of July 14, 2011, using the BIOS version listed in this detailed test matrix spreadsheet (including BIOS levels etc), click TinkerTry's VT-d Compatibility Matrix. You'll see I captured the informal tests I did, to make sure I could use all the virtualization features while leveraging the latest Intel Core i7 on the Z68 chipset.  ON the ASRock, I used the PCI slot nearest the CPU labeled "“PCI Express 2.0 x16 Slot (PCIE2, Blue)” for my LSI 9260-8i RAID adapter, fyi.

This sort of compatibility testing is apparently not well documented anywhere else on the web that I could find back in April 2011, especially since these Z68 chipsets are geared more toward general workstation use and gaming.

But I know I’m not the only one that was curious about this:
communities.vmware.com/thread/303243
hardforum.com/showthread.php?p=1036672188
www.vm-help.com/esx40i/esx40_vmdirectpath_whitebox_HCL.php

Here’s more background information:
www.vmware.com/files/pdf/techpaper/vsp_4_vmdirectpath_host.pdf
kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1010789

Short version of the long story, to keep in mind as you choose your mobo+CPU combo:

• Gamers will want the Core i7 2600K on any Z68 board that suits their needs
• Virtualizers may want the Core i7 2600 on the ASRock Fatal1ty Z68 Professional Gen3, ASRock Z68 Extreme4, or MSI Z68A-GD80 boards

You now have reasonable assurance that you too will also be able to pin a particular PCI device or USB device to one virtual machine, with no driver support in ESXi needed (you install the device driver in the Virtual Machine).

Please post your experiences or opinions below.

*RAID Errata:  if you also wish to add-on a RAID adapter like the LSI 9260-8i, you’ll find the ASRock board gets along with that particular RAID adapter better than the MSI board.  This is because these types of consumer boards have only 8MB of BIOS RAM, unlike the server chipsets, which are compatibility tested for RAID controllers.   Looking forward to PCI adapters with full UEFI support, someday, to avoid all that.

The way I got into the initial “Web Bios” RAID configuration, which you bring up with the Ctrl+H keystroke at boot time, was to temporarily disable BIOS features (that take up RAM).  This included USB 3.0, boot-from-LAN, and planar storage controllers (Marvel and Intel).  Once I finished with configuring the RAID array, I put the BIOS back to the the way I had it, and the RAID adapter continues to work fine, and is seen by ESXi natively (it’s on the VMware HCL).

JUL 20 2011 Update

Added NIC and RAID compatibility tests to the spreadsheet/images

DEC 18 2011 Update

I have since finalized on an LSI 9265-8i RAID adapter coupled with the  ASRock Fatal1ty Z68 Professional Gen3 motherboard, see also:
TinkerTry.com/rationaleforvzilla
TinkerTry.com/vzillastoragereasoning

*I also need to add that using this technique to get into WebBIOS is fairly likely to work for any brand of Z68 motherboard, something I discovered months after writing the above article:
TinkerTry.com/lsi-knowledgebase-article-points-tinkertry-method-configure-lsi-raid-z68-motherboard

Finally, ESXi 5.0 has been released, and works great with the Z68 Fatal1ty and the LSI 9265-8i, with native driver support in the recent ESXi 5.0 driver rollup!

MAY 04 2013 Update

ESXi 5.1 seems to have worse support for VMDirectPath than 5.0 did, based on my own tests with my HighPoint RocketU 1144A USB 3.0 PCI-E card, let's hope the upcoming 5.5/6.0 release reverses that trend! I may get a chance to test ESXi 5.1 Update 1 soon as well.

JUL 03 2015 Update

Didn't edit the above article, just adding this brief comment, as Tinkererguy in July of 2012, I was referenced on Wikipedia, see List of IOMMU-supporting hardware.