Superguide: “vZilla” Z68/Core i7/LSI 9265-8i - built for virtualization, storage, and speed

Posted by Paul Braren on Jul 13 2011 (updated on Jul 24 2014) in
  • CPU
  • HomeLab
  • HomeServer
  • Motherboard
  • Storage
  • Superguides
  • UPS
  • Virtualization
  • FractalCaseAsrockz68-10

    Jul 12 2011 Original Post: (see also updates, appended at the end)

    Create your own fast and resilient VMware whitebox, for both "production" and "lab" use.

    I'm happy to finally be publishing this summary on July 12, 2011...the day vSphere 5 was announced. Back in April, my goals for "vZilla" were straight forward, or so I thought. I naively thought the preparatory research would be simple. But I quickly realized I was pretty much alone in what I was attempting, and Googling Z68 AND ESX turned up proverbial crickets... So I began posting some thoughts over here, at one of my favorite sites, The Home Server Show Forums.

    I set out to build a modern, Z68 motherboard based VMware ESXi system to run 24x7, one that is both very fast, and relatively efficient, given I plan to leave this box running 24x7. A roughly $2K budget was set, with the $8000 works of art/masterpieces seen here are admittedly far out of my range. I also wanted to avoid complex hackery required to get ESXi (4.1U1 or later) to install, with mass storage devices supported for storage pools, either natively (best), or via VMDirectPath / VT-d feature (VM assigned a particular PCI device).  And this server could also become a full-time NAS for myself as well, perhaps using storage appliances with NFS or iSCSI. This server would give me much higher capacity at a more reasonable price than most NAS I've looked at, but this home-brew could have a lot more utility and speed. Ideally, I also wanted Intel RST (Rapid Storage Technology - SSD caching) to work, in passthru mode, knowing VMware HCL doesn't support any newer Intel ICH RAID stuff, and RST is actually software caching by the host OS, even less likely to get VMware support (no software RAID solutions supported by ESX/ESXi that I know of). I also thought that if passthru mode didn't work out, given how complex the mobo's built in RST is, I'm happy with at least seeing Z68's Marvell and/or Intel AHCI attached drives to be usable by VMware ESXi for the storage pool, perhaps using drive pooling of some variety some day. But for the primary storage pool, I required a modern RAID controller, one that would allows >2TB drives, support dynamic volume growth to allow me flexibility with online upgrades in the future, and have SSD caching for the potential for read speed enhancements.

    With plans to also duplicate key data offsite automatically, my shopping emphasis was a bit more toward raw speed, having been bitten with slow, low-end NAS devices before. For my fastest storage needs, I'll go with an SSD anyway, for my planned 25 user VM running "Windows Storage Server 2008 R2 Essentials," for example, but with the backup pool using the redundant/resilient RAID array. Well, the good news is, that my vision has become reality. But not without a great deal of heartache (a bad motherboard), pain (bricked RAID adapter, that I later resurrected), and suffering (for a week, I wasn't sure I'd get any combination of mobo/RAID working well, having encountered numerous random issues with nasty boot errors). So, after 4 motherboards, 2 RAID controllers, and numerous drive configurations, I've settled on the combination RAID controller/mobo that works, and works well. I went through trying ASUS, MSI, Gigabyte, and finally, ASRock motherboards, not because I like restocking fees, but because the only board that really worked with the LSI controller, and VMDirectPath, also turned out be the last board I tried. I get to use all special VMware features I wanted that normally only Xeon based server motherboards handle, like VMDirectPath (VT-d), which may allow me to pass USB 3.0 devices straight through to my storage server virtual machine, could be handy for rapid copy to external USB3 enclosure for offsite encrypted storage. And with vSphere, I'll get nested VMs, potentially even handling Hyper-V learning/testing! Drum roll.. Yes, the secret sauce for this "vZilla" beast of a personal pet project (aka "VMwareZilla" or "vSphereZilla"), to run VMware ESXi with a Home Server backup VM left running full-time, is the ASRock Z68...

    Here's the major components:


    ASRock Fatal1ty Z68 Professional Gen3


    Intel Core i7-2600 3.4GHz LGA1155


    LSI 9265-8i RAID controller (with hardware encryption enabled)

    USB 3 for passthru

    HighPoint RU1144A PCI-Express 2.0 x4 USB 3.0 Controller Card


    G.SKILL Ripjaws X Series 32GB (4 x 8GB) DDR3 SDRAM F3-10666CL9Q-32GBXL


    Fractal Design Define XL Titanium Grey ATX Full Tower Silent PC Computer Case


    7 2TB drives (I already owned)
    1 Corsair Force Series GT CSSD-F120GBGT-BK 2.5" 120GB SATA III Internal SSD

    Yes, this big case has 10 internal 3.5" drive bays with trays.  The SSD for hardware RAID caching was chosen for speed, price, and built-in automatic garbage collection, since RAID and ESXi don't support TRIM. So, now do I have your attention? I plan to beef up this new site with more build details, photos, RAID level test matrix spreadsheets, and a complete parts lists in the very near future, please come back soon. The long saga of how it all went down these past 2 months is quite drawn out, but will lead to lots of content, and I learned a LOT along the way that I look forward to sharing, including how important the 8GB BIOS memory is (for RAID adapters to work with BIOS, since they're not UEFI yet), the importance of testing RAID levels and tweaking the settings (dramatically affecting throughput), and the importance of researching UPS types that work with the new efficient power supplies. And now that VMware has announced that vSphere 5 will have >2TB volume support, I'm now well equipped to create a large RAID array pool up front, one that I don't have to split into individual small 2TB RAID arrays, so VMware can see the storage properly, ready to be divvied up among many VMs I'm planning, starting with my PC backups by Windows Home Server 2011/Windows Storage Server 2008 R2 Essentials. More fun yet to come!


    Here's a new Fractal Design USB 3.0 upgrade kit to upgrade the built in ports on this tower case, nice!

    And a really detailed Fractal Design XL review with many photos.

    Here's how the install process looks.

    Here's a more comprehensive parts list, hoping to get a better/prettier table built soon (any suggestions for tools to make that easier?)

    Italics indicates future planned purchase

    Plan Part # Company Description Qty. Unit Price  Site RAID related
     N82E16816118160 LSI LSI MegaRAID Internal SAS 9265-8i 6Gb/s Dual Core ROC w/ 1GB cache memory RAID Controller Card, Kit 1  $664.99  $664.99
     Y  TBD LSI MegaRAID CacheCade Pro 2.0 Software Pack and FastPath Combination for 9265 (1Q2012) 1  $299.99  TBD  $299.99
     N82E16819115071 Intel Intel Core i7-2600 Sandy Bridge 3.4GHz (3.8GHz Turbo Boost) 4 x 256KB L2 Cache 8MB L3 Cache LGA 1155 95W Quad-Core Desktop Processor BX80623I72600 1  $299.99
     N82E16820231487  G.SKILL G.SKILL Ripjaws X Series 32GB (4 x 8GB) DDR3 SDRAM 1333 (PC3 10666) F3-10666CL9Q-32GBXL 1 $269.99
     N82E16813157263 ASRock ASRock Z68 PROFESSIONAL GEN3 LGA 1155 Intel Z68 HDMI SATA 6Gb/s USB 3.0 ATX Intel Motherboard 1  $254.99
     12594475 LSI MEGARAID BBU09 Controller (Battery Backup Unit) 1  $178.84  pcconnection  $178.84
     N82E16811352006 Fractal Design Define XL Titanium Grey ATX Full Tower Silent PC Computer Case 1  $149.99
     Y  N82E16820233191 CORSAIR Corsair Force Series GT CSSD-F120GBGT-BK 2.5" 120GB SATA III Internal Solid State Drive (SSD) 1  $149.99 TBD (waiting for low price)
     N82E16842102131 CyberPower CP850PFCLCD UPS 850VA / 510W PFC compatible Pure sine wave 1  $129.99
     N82E16816115104 HighPoint RU1144A PCI-Express 2.0 x4 USB 3.0 Controller Card 1  $89.99
     N82E16816215222 iStarUSA BPU-126-SA 1 x 5.25" to 6 x 2.5" SATA 6.0 Gb/s Hot-Swap Cage 1   $89.99
     N82E16817371016 Antec BP550 Plus 550W Continuous Power ATX12V V2.2 80 PLUS Certified Modular Active PFC Power Supply 1  $74.99
     N82E16835181015 CORSAIR CWCH60 Hydro Series H60 High Performance Liquid CPU Cooler 1  $73.74
     N82E16827106358 LITE-ON CD/DVD Burner(Slot-load) 8X DVD+R 8X DVD+RW 6X DVD+R DL 8X DVD-R 6X DVD-RW 8X DVD-ROM 24X CD-R 24X CD-RW 24X CD-ROM SATA Model DC-8A2SH - OEM 1  $48.99
    various Intel Intel Pro 1000 PT Dual Port Server Adapter (eg., IBM 39Y6128, IBM/Dell/Intel variants of same chipset widely available on refurbished or used cards) 1 $40.00
     11857721 LSI Remote Battery Kit for LSIIBBU06 LSIIBBU07 1  $34.40  pcconnection $34.40
     N82E16817998078 SYBA SY-MRA55003 Enclosure for 2.5" SATA HDD and Slim Optical Drive, Fit in 5.25" Bay, 2x USB Por 1  $31.99
     N82E16835233037 XIGMATEK Cooling System XLF XLF-F1455 140mm White LED Black Case Fan PSU Molex Adapter/extender included 2  $25.98
     N82E16817998048 StarTech SATADOCK525 5.25in Bay Mounted 2.5in and 3.5in SATA Hard Drive Dock 1  $19.99
     N82E16812200161 StarTech 4in SATA Power Y Splitter Adapter Cable 2  $19.98
     ULT40101 Ultra Molex To Sata Adapter 3  $14.97
    B000I5KSNQ Antec SpotCool System Cooler 1  $12.99
     FLT-6020-18 Cables Unlimited Slim-line SATA Drive Cable to SATA and 4 p-inch Molex Power 18" 1  $11.18
    FD-AC-USB3-DEFINEXL Fractal Design USB 3.0 upgrade for Define XL  Product code: FD-AC-USB3-DEFINEXL 1 $9.99
     N82E16812200855 StarTech 6.02" 6in TX3 Fan Power Splitter Cable Model TX3SPLITTER 1  $9.99
     N82E16812196309 Cables To Go 6" One 5-1/4in to Two 5-1/4in Internal Power Y-Cable Model 03166 1 $7.99
    N82E16820191269 Lexar Lexar JumpDrive FireFly 2GB USB 2.0 Flash Drive Model LJDFF2GBASBNA 1 $7.99
    vZilla Grand TOTAL  approx $3,000  Portion that is RAID Related  approx $1,178

    Aug 25 2011 Update: Now running ESXi 5.0!
    Motherboard chosen: ASRock Fatal1ty Z68 Professional Gen3
    (eventual "Ivy Bridge" compatibility, PCI Express 3.0, VMDirectPath support, and lots of SATA ports)

    Oct 01 2011 Update:  Originally titled:
    "vZilla is born!" Z68/Core i7/LSI9265-8i running VMware ESXi
    This is a whitebox/all-in-one lab box, a virtualization 24×7 lab really, but does "production" backup duties at night.  It meets all my system backups, and my VMware ESXi and Microsoft Hyper-V self-training needs.  It inspired me to create this blog, to share things I've learned along the long and winding road to getting this thing built.

    Nov 14 2011 Update: Finalized parts list completed, planning a vZilla grows up blog post soon!  I need to also make a new set of photos, to reflect the slight changes in hardware, mostly pertaining to the drive bays at the front which don't really affect the system build. Scroll up to see the detailed build list.

    Dec 11 2011 Update: I should probably note that the below table doesn’t indicate the price of any 3.5″ drives for actual storage, which I already owned, re-used from previous projects. Otherwise, this list is a complete parts list, to build an identical or similar machine, should you wish to do so.

    Jan 09 2012 Update: vZilla completed!

    Feb 09 2012 Update: DIMM change
    4 x 4GB = 16GB of Corsair DIMMs eren't on ASRock's memory table
    4 x 8GB = 32GB of G.SKILL memory added, more appropriate for ESXi, currently $269 USD , on G.SKILL's site listing for compatiblity with ASRock here, for sale on here, and discussed at length here:

    Aug 20 2012 Update:
    Still going strong well over a year later, very pleased with the overall experience of owning this mobo/CPU combo, heavily abusing this system with ESXi 5.0 (now with Windows 8 backups and Windows Server 2012 betas). No compelling reason to try to jump to Ivybridge (too much cost for too little gain). See also:

    With 10,000 unique visitors to this particular blog post, and over 24,000 page views, is among my most popular posts ever. Daily views are actually growing even now, after well over a year. And >95% of the system components are still available.

    Jul 24 2014 Update:
    This is system is still serving my needs very well. There is still nothing affordable and new out there that'll allow >32GB of RAM, so I'll just stick with this system as-is, for a while yet.