Playing with LSI CacheCade Pro 2.0: The OCZ Vertex 4 256GB SSD (VTX4-25SAT3-256G) (Caution!)

*Be sure to read about important updates to this article at the bottom, basically admitting this may not be the best choice of drive for this application

Finally, the drum-roll moment, will CacheCade Pro 2.0's Read and Write caching of my LSI 9265-8i's RAID5 work well?  Will my RAID5 array become the homebrew SAN and NAS of my dreams, far exceeding most typical under $1,000 USD NAS appliances?

Am I taking a risk by marrying an enterprise RAID adapter with such a "consumerish" SSD, with potentially immature Indilinx firmware on Marvell Silicon?
www.2fit.anandtech.com/show/5741/ocz-confirms-octane-and-vertex-4-use-marvell-based-silicon

Sure, I'm taking a risk, but my data is backed up, and I'd like to see what kind of performance I get with this latest generation of SSDs, the current speed champion in the consumer space. Well, the 512GB model beats it, but at a huge $ penatly.

I would suspect LSI is going to eventually come out and list the pricey and no longer new Intel 510/520 series of drives as CacheCade Pro 2.0 compatible, a brand more typical for "enterprise" use (see also item #4 here):
TinkerTry.com/lsi-faq-cachecade-pro-2-0-coming-soon-meanwhile-summary-open-questions 
update, see page 25-28 for a somewhat out of date list (older LSI firmware tested, older SSDs):
www.lsi.com/downloads/Public/RAID%20Controllers/RAID%20Controllers%20Common%20Files/MegaRAID_Value_Feature_Interop_List_SAS2.pdf

You may want to  check out this closely related article from friend welchworks, who has a 9260-4i with SAS Expander, and CacheCade Pro 2.0 on his RAID10 with multiple OCZ Agility drives for caching:
homeservershow.com/forums/index.php?/topic/3791-playing-with-cachecade-20

and the really detailed LSI articles he refers to:
www.lsi.com/downloads/Public/Direct%20Assets/LSI/Benchmark_Tips_Jan_2010_v2_0.pdf
www.lsi.com/downloads/Public/Direct%20Assets/LSI/6Gbs_LSI_LRb.pdf

and the MegaRAID Storage Manager Manual:
<www.lsi.com/downloads/Public/MegaRAID%20Common%20Files/51530-00_RevF.pdf>

Want to also see the other published Vertex 4 benchmarks? Here's quite a list:
www.ocztechnologyforum.com/forum/showthread.php?100402-Vertex-4-performance-and-analysis 

My twist is not about trying to be a source of official benchmarks, it's really more about testing the SSD in a wide variety of ESXi use-case scenarios (not sure I'll have the patience to try all of these):

• as a physical NTFS drive, running under Windows 7 x64 SP1
  TESTS COMPLETE
• as a CacheCade Pro 2.0 SSD read/write cache for my RAID5 that holds my ESXi 5.0 thin provisioned VMs
  TESTS COMPLETE, not yet published
• as a motherboard's Intel AHCI-connected, VMFS formatted datastore, in ESXi for thin provisioned VMs
• as a motherboard's Intel AHCI-connected, , RDM mapped to a Windows VM, where it's NTFS formatted (SATA passthrough)
• as an LSI-9265-8i attached single drive RAID0, VMFS formatted datastore, in ESXi for thin provisioned VMs
• as an LSI-9265-8i attached single drive RAID0, RDM mapped to a Windows VM, where it's NTFS formatted (SATA passthrough)

My story is just starting to unfold here, come back and visit to watch progress. I have many more test scenarios left to try, including a little Iometer action.

May 2 2012 7:11pm:
Not sure if I'll stick with using this drive for my SSD cache in my RAID5 array, or elsewhere, so I gotta kick the tires first, no?

Finally made my purchase from Amazon:
www.amazon.com/OCZ-Industrys-Highest-Performance-2-5-Inch/dp/B007RARHSO/ref=sr_1_2?s=electronics&ie=UTF8&qid=1335886598&sr=1-2

Here's the brief unboxing video (iPhone 4 on a trip works quite well for quick 720p video creation, sent right to YouTube, as-is):

May 2 2012 11:00pm:
Initial round of tests completed, with a lot of video to edit soon.  Meanwhile, I am nearly ready to post preliminary results that show how this drive performed under Windows 7 x64 SP1, formatted as a secondary NTFS drive. I'll also give a look at the performance when used as SSD Cache for my RAID5, using a VM under ESXi 5.0 (VMFS filesystem).

May 3 2012 12:15am:
Using a 1.6GB test file (Windows 2008 ISO), takes 15 seconds to copy over the network, that's right, for the first time ever, I'm seeing >90% utilization of my gigabit network when doing file transfers from an SSD laptop to a VM, 100MB/sec, this is good, this is fun, this not your typical RAID5!

First round of results from just the ATTO and AS SSD benchmarks run under Windows 7 x64 SP1 are ready, seen below. To be clear, these graphics below do not show the RAID5 at all, they merely test out the base performance of the Vertex 4 itself.

Notice that the 6Gbps Marvell/ASMedia ports on the ASRock Z68 Fatal1ty motherboard, results listed first,  don't come close to the Intel 6Gbps ports, listed second.
Also notice that if 4K write speed is what really matters for SSDs, then this drive is impressive, and may get better with future firmware (1.3 is still current):

Marvell/ASMedia 6Gbps connected OCZ Vertex 4, formatted NTFS as drive F: under Windows 7 x64 SP1, using 2GB Total Length

Intel 6Gbps connected OCZ Vertex 4, formatted NTFS as drive F: under Windows 7 x64 SP1, using 2GB Total Length

Marvell/ASMedia 6Gbps connected OCZ Vertex 4, formatted NTFS as drive F: under Windows 7 x64 SP1

Intel 6Gbps connected OCZ Vertex 4, formatted NTFS as drive F: under Windows 7 x64 SP1

Please come back soon, for more results of testing the RAID5 from within ESXi 5.0, and full videos of many of the actual testing procedures.

Here's my first ESXi 5.0 test, a Windows 7 VM on VMFS filesystem on 5 x 1.5TB Seagate 7200.11 drives RAID5, with 256GB Vertex 4 configured as CacheCade 2.0, next test will be identical, with CacheCade 2.0 turned off:

After a few more tests, I'm able to tentatively put some observations out there:

1 CPU versus 2 CPU cores given to the VM doesn't change benchmark results (CPU utilization never goes above 60%)
2GB versus 4GB of RAM given to the VM doesn't change benchmark results (Memory utilization of ATTO never goes above 10MB)
Thick Provisioning give about 10% better overall performance than Thin Provisioning, but the benefits of Thin Provisioning (no wasted (expensive) storage) outweigh this minor performance cost
Running ATTO 2x in a row shows no significant improvement in speed, so I don't bother

Sep. 15 2012 Update:
I've had a scrape with my array dropping in the middle of some ATTO tests, back when the LSI had a previous firmware, with the OCZ Vertex at firmware 1.4. Yeah, it would seem choosing a drive not on the compatibility table has it's risks, not a huge surprise. What I don't know is whether the incident (where the array disappeared from ESXi 5.0 visibility) was because of faulty firmware, or fault behavior of the OCZ Vertex 4, and while I've been working with LSI Support on this (sharing logs and videos with LSI and a lot of info.), with frustratingly little progress made. Once I know more about what may have really happened in this incident, and where I may have contributed/gone wrong, I'll go ahead and publish full details of the incident.

So after talking it all over with a Samsung engineer at the VMworld 2012 booth late last month, I've moved forward, stopping bothering to look back. I'm now full-time on ESXi 5.1, and LSI 9265-8i August 2012 firmware .1776 , with a single Samsung 830 256GB  HDD at firmware CXM03B1Q Release Date: 2012-01-19, which is still the latest available:
www.samsung.com/us/support/owners/product/MZ-7PC256N/AM#firmware

Still working to get benchmark-in-VM numbers near these OCZ levels (but struggling with getting Read+Write caching going with .1776 firmware at the moment), so please stand by for further updates.

I also still haven't worked out of LSI is near "blessing" the consumer grade (affordable) 256GB SSD MZ-7PC256N SSD for CacheCade 2.0, but pushing for that, and hope to have news to break soon. Also asking why ASUS and Gigabyte mobos get tested, but no ASRock yet. Meanwhile, here's LSI's latest 9/11/2012 published compatibility matrix:

Compatibility Report for MegaRAID Value and Feature 6Gb/s SAS Controllers

This represents basic compatibility and is only a guide to hardware which is
Known to function with the listed MegaRAID SATA+SAS controllers.
Updated 09/11/2012
Controllers ♦ SAS/SATA Enclosures ♦ Motherboards ♦ Hard Disk Drives ♦ Solid State Drives ♦ Cables