Good RAID Controllers with SSD caching and ESX support

Here's the original title of this article:
"RAID controllers with >2TB drive support, 6Gb/sec (SATA3) support, and various levels of SSD caching: results of my taking both the LSI 9260-8I and the Adaptec 2405Q out for a spin"

Kind of a mouthful, eh?  Hopefully my new title is both more concise and intelligible. The title is the easy part. This is going to be a rough/deep ride through the quagmire of new RAID technology intricacies.  Hang in there, this epic journey is not for the faint of heart!  Making a reasonable cost, large storage array that also performs well is a challenge.  Ideally, I'd like to serve up some speedy NAS style workload (NFS, iSCSI, CIFS, etc.) to all my lab machines.

So, with this post, I'm finally beginning to write up some detailed information about my recent tests of the Adaptec 2405Q with maxCache, and the LSI 9260-8i RAID controller with CacheCade.  Both are SSD read caching technologies.  I guess you could say I caught the bug when I was using a Seagate Momentus hybrid drive this past winter, thinking, wouldn't it be nice if my RAID arrays performed more like my affordable and large hybrid drive?  And was operating system agnostic, that is, didn't require Windows-only software to run?

My posts about this RAID shopping began way back in May 2011, over at this spot on the extremely helpful homeservershow.com/forums.  Since ESX 4.1 Update 1 doesn't have Intel Z68 motherboard SRT (Smart Response Technology) support with SSD caching, I'm seeking alternatives to speed up the performance of all my VMs.

I've always found testing RAID performance to be complex and time consuming process, especially when you sprinkle in my desire to try out the latest and greatest techniques of making a RAID array perform more like a hybrid hard drive.  Think of a cross between SSD and a traditional spinning platter RAID arrays, taking the best of both worlds and mushing them together.  In a perfect world, this would get you speed, capacity, and resistance to data loss from inevitable drive failures.

Notice, I stated I want my choice to be VMware ESX compatible, which really limits choices even further, as seen over at the VMware Hardware Compatibility Guide (which doesn't yet have ESX  5.0 info).  I don't wish to dedicate this storage chunk to just one system or virtual machine, and with ESX 5.0's support for >2TB volumes, I'd like to manage this big lump of roughly 10TB of storage with ease. I know I'll change my mind many times on how to carve things up over the next 4 years of expected life of this system.

And just to make the picture even muddier, I'm also very interested in the upcoming "Host Cache Configuration" setting in ESXi 5.0, but that appears to be no substitute for true caching of a VM's "hotspots."  Playing with that software cache will have to wait, and there's very little info out there yet anywhere Googling, and of course, vSphere 5.0 (with ESXi 5.0) isn't even shipping yet.

So, back to the here and now.

August 9th 2011, a bunch of new firmware/drivers/MegaRAID software releases hit the LSI site, requiring a new set of RAID5 speed testing for me.  I've been on the April firmware, and am  not getting consistent results in Windows or ESX, so I'll hold off until I retest before I publish more.  Meanwhile, here's the most interesting pieces of that recent news from LSI I just spotted today:

LSI is claiming read/write SSD caching of RAID arrays, finally! Even better, I own the card that is eligible for this free upgrade for CacheCade 1.1 owners like me (I paid $226 for 1.1 in April, and already placed my order for 2.0).  Here's a little diagram to give the gist of what I'm talking about here, and a whole bunch of background links to enjoy/ponder:

LSI MegaRAID CacheCade gets write caching capability
LSI MegaRAID® CacheCade™ Software 2.0 Video Overview
MegaRAID CacheCade Pro 2.0 Software
LSI MegaRAID CacheCade Pro 2.0 Caching Software Technical Brief
MegaRAID® CacheCade Pro 2.0 Software FAQ

The SSD Review:
LSI MegaRAID CacheCade Pro 2.0 Review – Enter Write Caching
LSI MegaRAID CacheCade Pro 2.0 Review – Basic Concepts and Application
LSI MegaRAID CacheCade Pro 2.0 Review – Real World Results and Conclusion

At first, I was excited at the idea of CacheCade 2.0, since I already have a 9260-8i.  But LSI claims to be the first vendor to offer this, hmm, I'm used to being on the bleeding edge, but not necessarily with my treasured storage, at least until I have 2 or 3 copies of everything critical anyway.

So I dove in deeper to all those links, and noticed that for the new read/write caching that this CacheCade Pro 2.0 offers, there's this BIG gotcha here:

Cached write data protected by non-volatile CacheCade cache pools (RAID 0, 1, 10) and data availability protected by RAID data redundancy

RAID 5 in upcoming release (9265 Series only)[/box]

So that means you may need a lot of SSDs, so I may need to re-visit exactly how LSI's flavor of RAID10 caching works.  But I'm finding no new MegaRAID instruction manual yet, that'd be too easy. The issue with multiple SSDs is cost of course, my RAID5 config in my vZilla build has 7 (2TB) drives and 1 (96GB) SSD drive.  Time for even more research.

I will make updates right back here in this post, when I learn something significant, or have some speed tests of the 9260-8i with the new LSI August firmware/drivers.

Footnote:  I no longer own the Adaptec 2405Q, 4 ports was simply too restrictive, and port multipliers were not compatible with it.  Yes, I realize there are other higher end Adaptec controllers out there too, always eager to hear what others with similar goals have found, please use Disqus comments below...

Update Sep 09 2011 :  Nice new article spotted here:
LSI MegaRAID CacheCade Pro 2.0 Review – Test Bench and Protocol