Intel Optane M.2 16GB/32GB consumer NVMe SSDs for Windows caching are not that fast as VMware VMFS or NTFS, save up for bigger PCIe versions

Posted by Paul Braren on Jun 11 2017 (updated on Sep 4 2017) in
  • ESXi
  • Virtualization
  • Storage

    • Here are the Technical Specifications for the tested 32GB module.

    • Intel® Optane™ Memory Series 32GB M.2 80mm

      Performance
      Sequential Read (up to) 1350 MB/s
      Sequential Write (up to) 290 MB/s
      Power - Active 3.5 Watts
      Power - Idle 1 Watt
      Endurance Rating (Lifetime Writes) 182.5 TB
      Enhanced Power Loss Data Protection No

    This is essentially the speeds I was seeing in the tests under ESXi 6.5.0d, as described below. Note that these particular consumer M.2 modules are unlikely to be on the VMware HCL/VCG.

    Intel Optane is available now at:

    20-167-427-V08-cropped
    Newegg - Intel Optane M.2 2280-S3-B-M 32GB PCIe 3.0 x2 with NVMe Memory Module MEMPEK1W032GAXT

    Backstory

    +IsKkhWVs
    How Intel Optane / Micron QuantX / 3D XPoint actually works

    There has been months of build-up surrounding the Optane product launch, with a consumer-friendly use case that I outlined here:

    Later on, the PCIe Optane cards clearly intended for the datacenter emerged, with 3 different use cases. They'll likely soon be followed up with a more general purpose consumer version. This PCIe form factor is not yet widely-available for testing, initially announced in capacities up to 800GB. All Optane products feature this very new 3D XPoint based storage/memory. See also:

    "Introducing 3D XPoint™ Technology" by Micron.

    Test Plans

    This weekend, it was finally time time to tinker with my just-arrived Intel Optane M.2 drive as plain old storage. Well, not just any storage, but presumably very fast NVMe storage. Just how fast was the first question.

    I should be very clear here, Optane M.2 modules only come in 16GB and 32GB sizes right now. Given this tiny size, it's even clearer that they're only intended to accelerate (cache) slower mechanical drives very new Optane ready laptops and desktops. Those tests have shown impressive results for accelerating boot times and applications. But these tests weren't comparing performance of Optane to drives like the new Samsung 960 EVO or PRO M.2 NVMe SSDs, when used as a normal NVMe storage device. I was curious. How about I compare the performance with my 1TB 960 EVO drive.

    Given Intel's consumer focus here, I knew full well that I was highly unlikely to find drivers or firmware for VMware ESXi. Heck, these M.2 modules are so new that I wasn't even sure a new firmware for them would be available, and whether those update tools would be Windows-only.

    All that aside, it didn't take me long to spot the driver VIB for ESXi 6.5 over at vmware.com, which could speed things up. This had happened before, greatly accelerating NVMe speeds when I tested the Intel 750 Series under ESXi 6.0, way back in 2015. I know, long shot.

    VMware ESXi 6.5 NVMe Driver for Intel

    2017-06-11_20-37-45

    Unsupported and Unintended Use

    I just wanted to get my hands on a relatively affordable bit of 3D XPoint that is shipping now for under $100. Basic testing would hopefully help me get a taste of drive speeds. Having great luck with a variety of M.2 NVMe devices, install was expected to go smoothly on my SuperServer's PCIe slot using any adapter card, or right into my motherboard's M.2 slot, since both options benefit from full PCIe 3.0 x 4 speeds.

    Sad Trombone Moment

    Turns out I got the same low performance numbers with this Optane when using a PCIe adapter, or when installed on the motherboard. These results were considerably lower than my many positive experiences with my Samsung 950 PRO, 960 EVO, and 960 PRO M.2 NVMe drives, also formatted as VMFS. Especially slow were the write speeds. Darn. Was hoping 3D XPoint would be able to shine even in such an affordable and tiny capacity, but my hopes are so far unrealized, at least with the drivers I tested so far under ESXi. There are so many potential reasons for this, but it's highly unlikely that either VMware or Intel are particularly interested in optimizing their drivers for this unintended use case.

    TinkerTry-speed-comparison-Samsung-960EVO-v-Optane-32GB-M2-cropped
    Samsung 960 EVO at left, Intel Optane M.2 at right. When using either the VMware NVMe driver or the Intel NVMe driver, speed were essentially the same for Optane.

    Test Hardware and Software

    For my first tests seen in the video below, you'll see the 32GB drive was installed in my Supermicro SuperServer Xeon D-1541 Bundle 2 system that's at IPMI 3.52 and BIOS 1.2 set to my Recommended BIOS settings, running VMware ESXi easily updated to 6.5.0d, with Windows 10 Creators Update running ATTO Disk Benchmark 2.47.

    Test Steps Taken in the Video Below

    863217430629421057

    Here's an outline of what you'll see me do, in the video below, recorded live as I went, with voice-over:

    • format 100% of the available 32GB of space as VMFS 6.81
    • used vSphere Client (HTML5) to deploy two Windows 10 VMs from my "Golden Master" Template that uses VMware Paravirtual instead of the default LSI Logic SAS, with essentially the same speeds for both SCSI types here, placing one VM on the Samsung 960 EVO, the other on the Intel Optane 32GB M.2
    • booted both VMs
    • ran ATTO Disk Benchmark on both VMs on the same host concurrently
    • ran ATTO Disk Benchmark on each VM sequentially, disappointing Optane speeds
    • added Intel NVMe VIB, rebooted
    • ran ATTO Disk Benchmark on each VM sequentially, same disappointing Optane speeds
    • deactivated VMware NVMe VIB, rebooted
    • ran ATTO Disk Benchmark on the Optane VM, the 960 EVO VM vanished due to missing VMware NVMe driver, same disappointing Optane speeds

    Closing Thoughts

    Remember, these drives are meant mostly for fast read speeds, for cache. This article is just a first look, not the end of the story, and tinkering will continue. I should also test these Optane M.2 SSDs under Windows 10 formatted as NTFS, just to see what kind of impact virtualization is having. I'd also like to test with VMware vSAN 6.6, even though it's completely unsupported. Backup first, and see what happens. But I don't have 4 hosts to really do vSAN right though, only 2. So I'll have to get a bit creative, that's a work in progress.

    Remember, these little 16GB and 32GB M.2 modules aren't the same as the 800GB DC P4800X, a much faster device that proclaimed full vSAN support on day 0:

    • vSAN Got a 2.5x Performance Increase: Thank You Intel Optane!
      Mar 19 2017 by Michael Haag

      For vSAN, the results demonstrate that the Intel Optane NVMe SSDs provide an extremely high-performance caching device for write-intensive workloads. Customers can see immediate benefits for applications like VDI persistent storage use cases and next-generation applications such as Big Data, video streaming and real time streaming analytics.**

      An All-Flash vSAN system comprised of Optane-based NVMe cache devices delivers a very scalable and performant HCI solution for next-gen apps in the modern data center.

    Eventually getting my hands on a P4800X at some later date would certainly be interesting!

    Wrapping up with a set of NVMe relate esxcli commands that you might find helpful. Many of these were used in the video below, inspired by Anthony Spiteri's similar work with replacing AHCI drivers.

    How to install the Intel NVMe Optane driver

    1. first, use WinSCP to push the intel-nvme-1.2.1.15-1OEM.650.0.0.4598673.x86_64.vib file from within the downloaded zip file to the ESXi host's /tmp folder

    2. second, install the VIB using esxcli from an SSH session to the host, logged in as root

      esxcli software vib install -v /tmp/intel-nvme-1.2.1.15-1OEM.650.0.0.4598673.x86_64.vib
    3. reboot
    install-intel-nvme-vib
    How to install the Intel NVMe Optane driver

    How to disable the VMware NVMe driver

    esxcli system module set --enabled=false --module="nvme"

    How to Remove VIBs from a Host

    esxcli software vib remove --vibname=intel-nvme
    remove-intel-nvme-vib-by-TinkerTry
    Intel NVMe VIB just removed, host needs to be rebooted.

    NVMe Driver Command Reference

    How to verify which NVMe driver(s) is/are installed

    esxcli software vib list | grep nvme

    How to see also active drivers

    esxcli system module list | more

    How to make changes active

    reboot

    How to re-enable the VMware NVMe driver

    esxcli system module set --enabled=true --module="nvme"
reboot

    Summary of the two driver VIBs I worked with

    Here were looking at the relevant subset of active drivers that are discussed in the video, when both were set to active. I learned that the first driver listed by VMware is named nvme and is used for the Samsung 960 EVO SSD, and the second driver by Intel named intel-nvme is used by the Intel Optane M.2 SSD.

    esxcli system module list | more

Name                           Is Loaded  Is Enabled
-----------------------------  ---------  ----------
nvme                                true        true
intel-nvme                          true        true

    Video

    Intel Optane M.2 consumer SSDs for caching might not work well for tiny VMware VMFS datastores
    B06XSXX3NS
    Intel Optane Memory Module 32 GB PCIe M.2 80mm MEMPEK1W032GAXT at Amazon, 32GB.

    Intel Optane is available now at:

    Jun 13 2017 Update

    Here are the Technical Specifications for the tested 32GB module.

    • Intel® Optane™ Memory Series 32GB M.2 80mm

      Performance
      Sequential Read (up to) 1350 MB/s
      Sequential Write (up to) 290 MB/s
      Power - Active 3.5 Watts
      Power - Idle 1 Watt
      Endurance Rating (Lifetime Writes) 182.5 TB
      Enhanced Power Loss Data Protection No

    This is essentially the speeds I was seeing in the tests above, with write speeds actually slower than what SATA3 2.5" SSDs can do. Also notice the lack of Enhanced Power Loss Data Protection, so another reason these are not recommended for use for caching for vSAN. That is quite an endurance rating compared to any consumer Samsung M.2 NVMe SSD.

    Jun 14 2017 Update

    Given I'm already getting near native speed, the priority of retesting with NTFS on this M.2 Optane module is not a priority for me personally, with other folks already benchmarking them, here at TweakTown for example.

    Jun 18 2017 Update

    See Evan's feedback in the comments below. I've also now measured the VM at various queue depths, from the lowest ATTO allows of 2, to the highest of 10. Admittedly, I have not directly measured latency. This article continues to be a work in progress, as I determine whether I keep this drive or not.

    2017-06-18_11-32-43
    ATTO Disk Benchmark 2.47 at Queue Depths of 2, 3, 4, 6, and 10. Click twice to zoom all the way in, the slide the image left to right.

    Jul 25 2017 Update

    Saving your money for the DC P4800X, or the rumored P900 consumer version still seems to be a good strategy, since these capacities are still rather small, even in a RAID0, where the performance at low Queue Depths does become more intriguing:

    • Intel Optane in RAID 0 - World's Fastest System Disk
      Jun 28 2017 by Jon Coulter at TweakTown

      As we explained and demonstrated throughout the entirety of this review, 4K random read at QD1-2 has the largest influence on system performance in an OS environment. This is exactly where Optane has changed the game forever. Optane Memory delivers random read performance that is between three and ten times better than the best flash has to offer.

    Sep 04 2017 Update

    Fantastic work by Florian Grehl! All kinds of details, what an analysis, simply incredible!

    • Using the first 3D Xpoint based Intel Optane SSD with ESXi
      Aug 31 2017 by Florian Grehl

      With 32GB, it doesn't make sense to buy them for anything else than their intended use case: Cache device to enhance SSD/HDD Performance. If you want to use Optane technology as VM Datastore, wait a couple of months when devices with a higher capacity are available.

    See also at TinkerTry

    All Optane mentions at TinkerTry, dating all the way back to Feb 26 2016!

    discussing-ssd-storage-tech-with-allyn-malventano

    See also

    client-1
    • Intel Optane Memory 32GB Review - Faster Than Lightning
      Apr 24 2017 by Allyn Malventano at PC Perspective

      Random performance results are quite interesting. I've highlighted and labeled read performance here as it is more relevant to the discussion. Typical NAND-based SSDs have higher low QD write performance as they can cache incoming requests and let the NAND handle them after acknowledgment from the host, but their reads are still limited to the response time of the NAND flash memory. This is why we see the HDD + Optane random reads (blue bars) score greater than 2x higher than the Samsung 850 and even 960 EVO SSDs added for comparison.

    TinkerTry - PCs, EVs, home tech, efficiency and more, including virtualization. My opinions here, not my employer's.

    Paul Braren, MCSE in 1998, VCP #2681 in 2005, vExpert 2014 to 2024, Veeam Vanguard Emeritus, 2015 to 2019.

    Support TinkerTry

    PayPal | Patreon | VMware Store

    After 6 successful years testing then shipping well over 1,000 Xeon D Bundles, Wiredzone had to stop selling them in mid-2021 due to cost, supply, and logistics challenges. So far, Xeon D-1700/2700 (Ice Lake D) was a minor refresh for 2023, with Xeon D-1800/2800 (Granite Rapids D) refresh looking a little better for 2024. More cores, but still just PCIe Gen4 and DDR4. Looking ahead, I'm glad it's Pat Gelsinger at Intel's helm. I'm also grateful to have had the honor of working at VMware when Pat was the CIO there. I'll leave it at that, given the whole Broadcom thing.

    Easily update VMware ESXi at TinkerTry.com/easy-update-to-latest-esxi