Intel Optane SSD 900P Series is the latest in 3D XPoint NVMe hotness but it is not ideal for most home virtualization lab enthusiasts

Posted by Paul Braren on Oct 31 2017 (updated on Dec 28 2017) in
  • Storage
  • Virtualization
  • Disclosure: The U.2 version of the 900P 280GB was the only model available anywhere, so I purchased one to test. Intel later provided me a loaner AIC version of the 900P as well, in the 280GB capacity. There were no stipulations, I continue to candidly share my experiences. Unfortnately, things did not turn out well for using this 900P drives in my VMware environment, see the many reasons why detailed below.

    Intel Optane SSD 900P Series AIC and U.2 Announced

    Intel's October 27 2017 press release:

    • Blazing-Fast Gaming with Intel’s First Client Optane SSD

      Intel today announced the launch of the Intel® Optane™ SSD 900P Series, the first SSD for desktop PC and workstation users built on Intel® Optane™ technology.
      ...
      The Intel® Optane™ SSD 900P Series is ideal for the most demanding storage workloads, including 3D rendering, complex simulations, fast game load times and more. Up to 22 times more endurance than other drives also gives the heaviest users peace of mind.

    Shop

    • Newegg
    • Amazon? Probably best to choose only what's available on Amazon Prime.
      Nov 01 Update - Note, at 4:10pm eastern, Newegg auto-notifier informed me that the 280GB PCIe version might be available for pre-order, but by the time I checked later that evening, pre-orders were no longer being accepted, for any of Newegg's 900P versions.
      Nov 03 Update - Only the U.2 model with the U.2 connection became available, so I ordered it
      Nov 19 Update - Inventory at Newegg looking better, Amazon doesn't seem to have Prime orders in stock, not sure how I feel about the many 3rd party sources.

    Introduction

    This was a great week for home virtualization enthusiasts! A more affordable PCIe NVMe and U.2 drive is arriving soon, featuring the spectacular specs of the enterprisey P4800X that you saw here at TinkerTry first:

    hands-on-intel-optane-dc-p4800x-series-3d-xpoint-nvme-ssd

    What's the big deal here with the 900P then? Well, it's pretty much the same as a P4800, but at a greatly reduced price of $599.00 retail at Newegg for 480GB of Optane. Yes, still a hefty premium over SATA3 SSDs, but still less than half the price of the 375GB P4800X. This great-for-enterprise-workloads Optane P4800X might as well been called the Unobtainium P4800X, given the realities of most lab budgets. The 900P promised to bring most of that goodness down to earth, just be sure you have a PCIe 3.0 x4 lanes slot available to obtain top speeds.

    Specifications say it all

    Intel ARK for Intel Optane SSD 900P

    Intel-Optane-SSD-900P-Series-specs-highlighted-by-TinkerTry

    This sucker is begging to be driven hard, see values highlighted values. yes, those aren't ms (milliseconds), those are µs (microseconds). What does that mean?

    intel-nvme-esxi-6-driver-install

    Well, 10 µs is 0.01 ms. Holy guacamole, that's half the latency of the Intel Solid State Drive 750 Series of 3 years ago.

    Let's go further. How about the Endurance Ratings? How's 8.76 PBW (Petabytes Written) sound? Well, that converts to 8760 TBW (Terabytes Written). Compared with the 750 Series that has a 127 TBW (Terabytes Written) rating, that’s about a 68 times the write endurance.

    Compare 750 Series 400GB with 900P 480GB on Intel Ark

    What if we stick a DataCenter product in the mix, the Intel SSD DC P4600 Series 1.6TB. Not a fair comparison, but fun nonetheless:

    Various capacities, all run at the same speed

    See also the main Intel Optane SSD 900P Series landing page.

    Finally, a fun look at how involved proper testing is done in the Intel Optane Solid State Drive 900P Series Evaluation Guide.

    Press Coverage

    Now you know why the positive press is pouring in over this new little hotness in NVMe SSD, here's a select few:

    PC-Perspective-Logo

     

    intel-optane-900p-low-price-nvme-might-perfect-fit-home-lab
    • Intel Optane 900P Low Price NVMe might Be a Perfect Fit For a Home Lab
      Oct 30 2017 by Vladan Seget at ESX virtualization.

      If you remember, a few months back VMware announced a support for latest Optane drives from Intel with 3D XPoint memory, and the solution was the fastest one tested, on vSAN. This has had of course a price (I saw a $1,791.00 on Amazon), which is quite pricey for a 375GB PCIe NVMe card. Not something a homelabber can afford, right?

     

    the-intel-optane-ssd-900p-review
    • The Intel Optane SSD 900P 280GB Review
      Oct 27 2017 by Billy Tallis at AnandTech

      However this year's revolution from Intel will be very hard for the competition to match anytime soon. All of Intel's previous record-setting SSDs have relied on the drive's controller to stand out from the crowd. This time, Intel's advantage comes from the storage medium: its 3D XPoint memory technology, a new nonvolatile memory that offers much higher performance than flash memory.

     

    VMware Compatibility Guide

    VMware Compatibility Guide - I/O Device Search for Optane only shows the two P4800X drives, AIC (Add in Card) and U.2 (hot swap NVMe in a 2.5" enclosure).

    925340805484687360

    I have reached out for comment on whether this 900P will ever appear on the VCG (VMware Compatibility Guide), but likely the answer will be no. Even less likely to ever be on vSAN VCG. This is really a consumer drive here, and VMware's vSAN testing is on the supported gear for the enterprise, for very good reasons.

    Power Loss Protection in a consumer NVMe drive

    Here's the thing. The 900P has PLP (Power Loss Protection), sometimes referred to as Supercapacitors. (See Nov 29 2017 correction below, it turns out the 900P does NOT have PLP) PLP was actually just discussed on the Virtually Speaking Podcast, right at this spot. Yes, scary stories just in time for Halloween! PLP capability could mean that the 900P is the best somewhat-affordable choice as a caching layer for vSAN lab tests. Completely unsupported by VMware mind you, but a lot less likely to need support in the first place. Consumer drives don't usually make sure uncommitted writes make it out of the buffer and into the non-volatile NAND, but this 3D XPoint sure does! Nobody wants to hear storage array and corruption in the same sentence, no matter how backed up you are.

    vSAN?

    Let's be real clear here. In no way am I recommending that such consumer drives be used for vSAN, especially when it comes to production workloads where having a support contract with completely supported enterprise rated components is essential.

    There is no telling whether using the 900P for vSAN is a good idea for a home lab, with hands-on testing certainly needed. C'mon now, the press release even says "Blazing-Fast Gaming."

    So, as a thought experiment, let's just say you were to format a 900P as VMFS 6 using VMware vSphere / ESXi 6.5 Update 1. You then use this datastore for nested ESXi 6.5U1 instances, creating a 4 or even 6 node vSAN. Alternatively, maybe you get even more "creative" and try NVMe pass through of each NVMe device you have, which isn't hard at all anymore, see my recent How to configure VMware ESXi 6.5.x for VMDirectPath I/O pass-through of any NVMe SSD, such as Windows Server 2016 installed directly on an Intel Optane P4800X booted in an EFI VM. Got your attention yet? Maybe you've got yourself one very fast vSAN caching tier in your lab tests for your other more affordable SSDs. Maybe...

    These 900P SSDs aren't in stock or even available for pre-order yet

    So where's that leave TinkerTry? Well, for now, I can't know how well this works or performs until I can get my hands on one of these. Please know that I sure am trying to get one, as is nearly every other blogger out there.

    Closing thoughts

    Decent capacities in M.2 sizes are a long way off

    Even more versatile would be a 900P in the M.2 form factor, like a much better version of the original M.2 Optane module, see:

    optane-m2-consumer-ssd-first-look

    What I'm getting at here is that you can stuff 4 M.2 drives into a single PCIe 3.0 x 16 slot, and pass each NVMe device through to a VM independently. That would be a much more practical way to handle cache testing scenarios such as vSAN POCs.

    Thermal dissipation

    NSeSwNDT26g

    As for cosmetics and market differentiation? Nice job on the distinctive backplate Intel, good move. Smart looking, and flow-through practicality, demonstrating your apparent mastery of AIC cooling that was so evident in my FLIR thermal imaging tests of the similarly designed P4800X that easily handled my no air flow tests. I strongly suspect the 900P will perform very similarly, can't wait to get my hands on one to try it.

    Sorry, these are just renders, not quite like taking close-ups of a production sample, such as I did with the loaner P4800X here. Still, you'll get a pretty good idea of how the stickered GA product will look. After clicking on image, don't forget to go full screen with the icon at top right. Enjoy!

    Intel Optane SSD 900P Series AIC - Front
    Intel Optane SSD 900P Series AIC - Front
    Intel Optane SSD 900P Series AIC - Right Angle
    Intel Optane SSD 900P Series AIC - Right Angle
    Intel Optane SSD 900P Series AIC - Left Angle
    Intel Optane SSD 900P Series AIC - Left Angle
    Intel Optane SSD 900P Series AIC - Left Angle Tilt
    Intel Optane SSD 900P Series AIC - Left Angle Tilt
    Intel Optane SSD 900P Series AIC - Hero
    Intel Optane SSD 900P Series AIC - Hero
    Intel Optane SSD 900P Series AIC - Rear
    Intel Optane SSD 900P Series AIC - Rear
    Intel Optane SSD 900P Series Box - Generic
    Intel Optane SSD 900P Series Box - Generic
    Intel Optane SSD 900P Series U.2 - Front
    Intel Optane SSD 900P Series U.2 - Front
    Intel Optane SSD 900P Series U.2 - Hero
    Intel Optane SSD 900P Series U.2 - Hero
    Intel Optane SSD 900P Series U.2 - Right Angle
    Intel Optane SSD 900P Series U.2 - Right Angle
    Intel Optane SSD 900P Series U.2 - Left Angle
    Intel Optane SSD 900P Series U.2 - Left Angle
    Intel Optane SSD 900P Series U.2 - Left Angle Tilt
    Intel Optane SSD 900P Series U.2 - Left Angle Tilt
    Intel Optane SSD 900P Series U.2 - Rear
    Intel Optane SSD 900P Series U.2 - Rear
    Intel Optane SSD 900P Series U.2 Box Contents MiniSAS Cable
    Intel Optane SSD 900P Series U.2 Box Contents MiniSAS Cable
    Intel Optane SSD 900P Series U.2 Box Contents M.2 Cable
    Intel Optane SSD 900P Series U.2 Box Contents M.2 Cable

    Video

    "Intel Optane SSD 900P Series U.2 SSD unboxing and installing in Xeon D Supermicro SuperServer [4K]" published Nov 03 2017.
    "Installing the Supermicro AOC-SLG3-2E4 U.2 PCIe adapter with the Intel Optane 900P [4K]" published Nov 14 2017.
    "Intel Optane SSD 900P U.2 formatted VMware VMFS 6 by ESXi 6.5U1 for Win10 VM boot tests" published Nov 19 2017.
    "TinkerTry'd Supermicro SuperServer 5028D-TN4T Assembly and Windows 10 OEM build procedure" queued to the spot where I give you a close look at the easy 2.5" drive installation. Installing U.2 drives in either of those internal drive bays works fine, with adequate clearance in either location, published Aug 19 2015.
    "TinkerTry at VMworld 2015 - Intel discussion about SSDs and VSAN, with Ken LeTourneau" queued to the spot where Ken shows us the U.2 form factor, published Sep 11 2015.
    "VMworld 2017 US - at Intel, Juan Fernandez shows NVMe SSDs, Ruler & Optane P4800X in PCIe & U.2" published Sep 25 2017.

    Nov 02 2017 Update

    my-tinkertry-d-xeon-d-bundle-2-supermicro-superserver-bundle-2-of-joy
    All Supermicro SuperServer SYS-5028D-TN4T systems include 2 internal 2.5" SATA drive bays that should fit 2.5" U.2 drives, with drive bracket screw hole spacing that appears identical.

    In the original article above, I neglected to mention that you should look closely at that photo gallery, specifically, the last photo. See the caption? It's the Intel Optane SSD 900P Series U.2 Box Contents M.2 Cable that has my attention for folks that own the SYS-5028D-TN4T system. While host swap U.2 drives, which look like extra thick 2.5" laptop drives, wouldn't normally do much good in this SATA3 system. But remember, there's 2 bays in there for 2.5" drives that get screwed into a bracket, and both have extra space for taller drives. As for the 1U height systems like the SYS-E300-8D and SYS-E200-8D, I strongly suspect there just isn't near enough clearance in there, where the 7mm SSDs I tried on my loaner 1U systems barely fit, seen here and here.

    Special thanks to a TinkerTry reader who wrote in on November 1st, bringing this oversight to my attention.

    Internally mounted U.2 thermals

    What about airflow? Admittedly, the 4 hot-swap SATA drive bays get the airflow, but there isn't much air across the two internal drive bays. That said, even with no airflow, the similar P4800X did fine, even with quite a bit of stress, as already mentioned above, and seen here. So I suspect the aluminum heat sinks integrated into the U.2 900P design will be sufficient, with Intel learning from years of making U.2 drives for the enterprise, see also my visit to Intel at VMworld 2015 featured here. That said, let's not forget that U.2 dives are typically mounted where there's active airflow in a noisy 2U server. Such airflow is not easily possible with this particular system.

    Intel U.2 to M.2 Adapter Cable (18" length confirmed by Intel)

    Installation should be this simple:

    1. remove either of the 2.5" drive brackets by removing the 2 chassis screws.
    2. secure the U.2 900P to the bracket using 4 2.5" screws that you'll apparently need to provide.
    3. reinstall the bracket into the SuperServer again with the same 2 chassis screws you removed earlier.
    4. running the included U.2 to M.2 cable down to the motherboard's M.2 slot that runs at full PCIe 3.0 x4 lane speeds.

    That's it! Now you've got an additional 900P U.2 drive added to your mini-tower system, leaving your precious PCIe slot open for a good cause, such as:

    How could I fit two 900P drives into my SYS-5028D-TN4T mini-tower?

    AOC-SLG3-2E4
    Supermicro OC-SLG3-2E4 AIC.
    1. one U.2 900P in the side internal 2.5" bay, connected to the motherboard's one M.2 slot using the included cable
    2. one PCIe 900P in the one PCIe slot

    How could I fit three 900P drives into my SYS-5028D-TN4T mini-tower?

    You'd need the extra expense of an SFF NVMe adapter card, such as the Supermicro OC-SLG3-2E4 (see User's Guide that's available at Wiredzone, and two U.2 P900 SSDs that include the SFF 8643 cable, used to connect to the OC-SLG3-2E4 in the PCIe slot, and a third U.2 P900 with the included M.2 cabled, used to connect to the motherboard's M.2 slot.

    Reference Material

    25736_image2
    Physical Installation Instructions for Intel® Optane™ SSD 900P Series
    • Physical Installation Instructions for Intel® Optane™ SSD 900P Series
      Oct 27 2017 by Intel Support, excerpt:

      Installation steps for the Intel® Optane™ SSD 900P Series 2.5-inch U.2 drive

      1. Mount the SSD in the chassis using screws to hold the SSD securely
      2. Connect the cable to the U.2 SSD connector (SFF 8639)
      3. Depending on the cable that came in your box at purchase, either connect the:
        M.2 card in the PCIe supported M.2 motherboard connector
        SFF 8643 to your PCIe supported motherboard connector
      4. Connect SATA power to the connector on the cable
    • What’s in the Box for Intel® Optane™ SSD 900P Series
      Oct 27 2017 by Intel, excerpt:

      Article ID: 000025739
      The Intel® Optane™ SSD 900P Series is available in three different retail kits.
      Note - This type of solid-state drive does not receive data from SATA / AHCI connectors.

      Intel® Optane™ SSD 900P Series Retail Box for PCIe* U.2 2.5” with SFF 8643 Cable
      Intel® Optane™ SSD 900P Series (U.2 2.5” form factor)
      U.2 to SFF 8643 Adapter Cable
      Warranty Booklet

      Intel® Optane™ SSD 900P Series Retail Box for PCIe* U.2 2.5” with M.2 Cable
      Intel® Optane™ SSD 900P Series (U.2 2.5” form factor)
      U.2 to M.2 Adapter Cable
      Warranty Booklet

      Intel® Optane™ SSD 900P Series Retail Box for PCIe* Add-In Card
      Intel® Optane™ SSD 900P Series (add-in card form factor)
      Half Height Bracket
      Warranty Booklet


    Nov 03 2017 Update

    It's here! I now have my "Intel® Optane™ SSD 900P Series Retail Box for PCIe* U.2 2.5” with SFF 8643 Cable." I don't yet have the U.2 Adapter, such as the OC-SLG3-2E4, so I can't test it yet, but I did testing mounting it in the SYS-5028D-TN4T chassis, and it went well. Unboxing video, recorded in 4K, is being produced now, hopefully ready to share by Nov 4th.

    Added 900P unboxing video above.

    Intel-900P-mounted-in-SYS-5028D-TN4T-top-internal-25-drive-bay-wide-view--TinkerTry-Bundle-2
    Intel 900P mounted in Supermicro SuperServer SYS-5028D-TN4T top internal 2.5" drive bay
    Intel-900P-mounted-in-SYS-5028D-TN4T-top-internal-25-drive-bay--TinkerTry
    Intel 900P mounted in Supermicro SuperServer SYS-5028D-TN4T top internal 2.5" drive bay

    Nov 14 2017 Update

    Supermicro-AOC-SLG3-2E4-installed-close-up-TinkerTry
    Supermicro AOC-SLG3-2E4 REV: 1.02

    It's here! My Supermicro AOC-SLG3-2E4 - Supermicro NVMe PCIe Host Bus Adapter - 2 internal NVMe ports 6.4GB/s Gen-3 and were both home tonight, and got a chance to unbox and install the PCIe card, on camera of course. Kind of a cool moment when I quickly realize that the same exact BIOS tweaks that were needed for my earlier P4800X tests were needed for the 900P as well.

    Of course, I learned the hard way again that I'd get a BIOS POST hang that simply says A9 as soon as the AOC-SLG3-2E4 is installed in the PCIe slot with a 900P attached, just like I saw with the very similar P4800X.

    Supermicro-AOC-SLG3-2E4-installed-shows-U2-cable-routing
    easy cable routing of the Intel "U.2 to SFF 8643 Adapter Cable" is seen, no cable ties used

    Remove the AOC-SLG3-2E4, and the boot hang went away. Not sure what would have happened had I simply removed the U.2 connection cable from the drive.

    I used my own exact BIOS settings fix for this hang, right at this spot in the original P4800X article. Short version is be sure that:

    • SLOT7 PCI-E 3.0 X16 Bifurcation is set to default x16
    • SLOT7 PCI-E 3.0 X16 OPROM is changed from the default of Legacy to EFI

    and now I was good to go, boots up and works just fine. I have a lot of basic changes to attend to next, including:

    • upping the number of Paravirtual SCSI adapters and vCPUs, as outlined by Intel here
    • revisiting my BIOS settings, as Intel describes here
    • revisiting my NVMe driver, note that there are no ESXi downloads for the 900P, only for the similar P4800X, this isn't at all surprising, given the consumer focus of the 900P.
    • check on the 900P firmware, by booting over to Windows and trying out Intel's utility
    • check out performance on Windows 10 + NTFS using the 2.0.0.1024 NVMe driver, from:
      NVMe* Drivers for Intel Optane SSD 900P Series.
    • view the Supermicro AOC-SLG3-2E4 PCIe card and the 900P U.2 drive itself with my FLIR thermal imaging camera while under extended load

    That's just for starters. Once the PCIe version of the 900P arrives, and/or the U.2 to M.2 cable, I'll have some more tests up my sleeve to share with you, stay tuned!

    New video now available above, "Installing the Supermicro AOC-SLG3-2E4 U.2 PCIe adapter with the Intel Optane 900P [4K]".

    Video production underway of initial testing under ESXi 6.5U1, to be published right here soon. Done! Added to the video gallery above.


    Nov 19 2017 Update

    Intel has provided me with one of each of the following:
    • Qty 1: Intel U.2 to M.2 Adapter Cable
    • Qty 1: Intel Optane SSD 900P 280GB (U.2)

    which enabled me to proceed with testing that both 900P drives work properly in the SYS-5028D-TN4T Supermicro SuperServer system, concurrently.

    First, I removed the Supermicro AOC-SLG3-2E4 PCIe adapter card. I then wired up my 900P U.2 drive to the motherboard's M.2 slot using the Intel U.2 to M.2 cable. Finally, I was able to use the PCIe slot for the loaner 900P AIC (Add-In Card), as pictured below.

    Problems were unearthed that slowed my progress a bit:

    1. BIOS tweak needed to avoid BIOS hang at A9 error, setting EFI for both the M.2 and PCIe slots, recommended BIOS settings article updated
    2. if both M.2 and PCIe are occupied by Optane, I can't get back into the BIOS without removing the PCIe card (upcoming video I'll publish shows me figuring this out)
    3. coincidentally timed bump in the road explained here.

    Testing continues to go fairly well overall, more videos and additions to this article to come.

    Intel-Optane-900P-U2-and-M2-in-Supermicro-SuperServer-SYS-5028D-TN4T--TinkerTry
    Intel-Optane-900P-in-Supermicro-SuperServer-SYS-5028D-TN4T--TinkerTry
    Intel-Optane-900P-U2-and-M2-in-Supermicro-SuperServer-SYS-5028D-TN4T-with-boxes--TinkerTry

    Next up will be booting over to Windows to check in on firmware levels, and getting my video published of the 900P AIC unboxing and install, along with my VMFS 6 formatting and initial testing of the new 900P AIC under VMware ESXi 6.5U1.


    Nov 20 2017 Update

    932450566978129921

    As expected, the stated absolute throughput values of 2500Mbps reads and 2000 MBps writes the can be exceeded slightly by other M.2 SSDs like the Samsung 960 PRO, and my initial testing slightly exceeding these values when using simple ATTO Disk Benchmark tests in a Windows 10 VM that's running on ESXi 6.5U1, with the Optane 900P formatted VMFS 6.

    There's more to the story here. Don't forget the latency, which I haven't really tested at all yet. There's also the lack of any need for typical SSD housekeeping like garbage collection, so performance is consistent, all the time. Don't forget the ability to fill this sucker up to 99% without worrying about performance degradation, not something that can be said about older SSDs where performance tanked once you accidentally filled the SSD accidentally, which isn't exactly hard to do quite by accident in these days of thin provisioned VMs.

    As Allyn Malventano says on PC Perspective Podcast episode 470, at this spot:

    From a benchmarking standpoint, Optane is a dream to test, you don't have to
    worry about pre-conditioning...the performance is always stable, it doesn't matter what you do to it.

    In the context of his VROC tests, Allyn goes on to discuss comparative performance at this spot:

    I compared the 4 drive Optane array to a 4 drive 960 PRO 512 gig drive array, so 2 terabytes worth of Samsung's fastest SSD that they make...how high does the throughput go?...Looking at random access, now you would think that the 960 PROs would be pretty quick, but the Optane array, even though it's 1/20th of the capacity...just walks over it in random access...how high are the IOPs when you're at low queue depths, because that's, you know, where you'll actually be.

    Power Loss Protection.

    According to the Intel spec sheets, the Intel® Optane™ SSD 900P Series doesn't offer what Intel calls Enhanced Power Loss Data Protection.

    According to the Intel spec sheets, the Intel® Optane™ SSD DC P4800X Series does offer what Intel calls Enhanced Power Loss Data Protection.

    Is this a concern, especially if you're considering using the 900P in a completely unsupported use-case, such as in the vSAN caching layer? This warrants further investigation.

    Meanwhile, see what chuckula says at phoronix here:

    Absolutely none of Intel's Optane parts (including the $3000+ enterprise drives) include large capacitors for power loss prevention.

    BUT: EVERY Optane product on the market now including the consumer drive Phoronix tested and even the cheap 16GB cache drives provides power loss protection.

    That's because none of these drives use a RAM cache at all and the data are stored fast enough that PLP is an inherent feature of the drive that doesn't need extra circuitry.


    Nov 21 2017 Update

    With the impending holiday that I quite enjoy, with family and friends all around, I wanted to get as much as I could documented for you, given I'll be falling behind on the follow-along-with-me video production:

    Intel-900P-Firmware-E2010325--2017-11-20-TinkerTry

    Nov 22 2017 Update

    Today, I downloaded Intel SSD Data Center Tool 3.0.4, which is the latest version for VMware. Then I followed the installation instructions in the filename Intel_SSD_DCT_Install_Guide_3_x.pdf included in the downloaded DataCenterTool_3.0.4_ESXi.zip file, and unfortunately, it doesn't seem to pick up that two 900P drives are installed in my system with the 1.3.2.4 NVMe Driver. This thwarted my efforts to see the NVMe firmware levels, and my usual procedure here doesn't work either:

    I also tweeted some ATTO Disk Benchmark early tests, but Twitter blurs screenshots pretty badly. Here's the full resolution pic.

    TinkerTry-ATTO-Diskbench-in-Win10-VMs-in-ESXi-65U1-on-Xeon-D-1541
    Click again to really zoom in.

    Tomorrow, I'll be eating turkey. See you right back here soon!


    Nov 29 2017 Update

    I heard back from an Intel spokesman with a definitive response on the whole question about Power Loss Protection:

    As an enterprise part, the Intel® Optane™ SSD DC P4800X offers multiple data protection features that the Intel® Optane™ SSD 900P does not, including DIF data integrity checking, circuit checks on the power loss system and ECRC. The DC P4800X also offers a higher MTBF/AFR rating.


    Dec 12 2017 Update

    I'm working with VMware support lately, as it seems like there's a chance that my 2 recent PSODs might both be NVMe driver related. There's also some small chance it's also related to the old KB 2146388 PSOD, and it definitely not KB 2151749 (10G connection loss), despite the "#PF Exception 14" similarity.

    Here's the details of one of one of my PSODs, note that it actually mentions NVMe, and my BIOS 1.2c:

    =========================================
    
    PSOD =
    
    2017-11-30T17:26:14.926Z cpu13:65974)@BlueScreen: #PF Exception 14 in world 65974:intel-nvme-c IP 0x0 addr 0x0
    PTEs:0x1e3dd1027;0x1925ce027;0xbfffffffff001;
    2017-11-30T17:26:14.926Z cpu13:65974)Code start: 0x41801d000000 VMK uptime: 0:15:42:28.981
    2017-11-30T17:26:14.929Z cpu13:65974)base fs=0x0 gs=0x418043400000 Kgs=0x0
    2017-11-30T17:26:14.929Z cpu13:65974)vmkernel             0x0 .data 0x0 .bss 0x0
    
    BIOS =
    
    Vendor: "American Megatrends Inc."
    Version: "1.2c"
    Date: "09/19/2017"
    Version: "Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz"
    
    =========================================

    Up until recently, I had never experienced the PSOD in KB 2146388 myself, and it had seemingly been resolved with Supermicro's release of BIOS upgrade 1.1c way back in October of 2016.

    This could well be the presence of the 2 900P Optane drives (notice NVMe mention in the first line of the PSOD above) and/or the Intel 1.3.2.4 NVMe Driver VIB, active investigation is still underway. My VMware Support SR# 17648400312 and details of my PSODs have been reported to both Supermicro and Intel.


    Dec 23 2017 Update

    Well, it's not looking so good for folks ever looking for 900P Optane with VMware ESXi for a variety of reasons that include:

    1. No power loss protection
      This important detail, confirmed by an Intel spokesperson above, means the 900P will never be well-suited for vSAN. It also assures that it has no chance of ever gaining official support from VMware since it's a consumer drive that won't appear on the VCG, never mind the more stringent vSAN VCG. It will have an inherently higher chance of data corruption for those who choose to use it for vSAN anyway.
    2. No support under VMware ESXi - PSODs experienced when using the P4800X's VIB
      While Intel's 1.3.2.4 NVMe VIB that's intended for the P4800X may sort of work, I'm getting PSODs. These PSODs seem to be not just happening on Xeon D systems. According to the VMware technician who had a look at my logs under VMWare SR# 17648400312:

      I have checked the new set of logs and can see the post reboot, there are still command '0x93' repeatedly failing on these drives...Either the drives or the driver still seem to be not looking normal in the current state.

      2017-12-22T22:27:12.865Z cpu1:65997)ScsiDeviceIO: 2962: Cmd(0x439501175840) 0x93, CmdSN 0x5e7 from world 68537 to dev "t10.NVMe____INTEL_SSDPED1D280GA_PHMB73600129280CGN__00000001" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x0 0x0.
      2017-11-30T17:26:14.836Z cpu13:65974)NMP: nmp_ThrottleLogForDevice:3617: Cmd 0x28 (0x439511235040, 67280) to dev "t10.NVMe____INTEL_SSDPE21D280GA_PHM2739000EC280AGN__00000001" on path "vmhba2:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Invalid sense data: 0x0 0$
    3. I still get BIOS hangs at A9
      Whenever the 900P is installed and I try to get into BIOS settings using the Del key, and I hear this might be occurring on non-Xeon D motherboards too.
    4. No warranty
      Not surprising, but still disappointing. I've recently learned from an Intel spokesperson that it appears VMware vSphere is outside of the supported and warrantied uses, seen in the warranty here, notably this clause:

      “Additionally, the Product will not be subject to this Limited Warranty if used in: (i) any compute, networking or storage system that supports workloads or data needs of more than one concurrent user or one or more remote client device concurrently; (ii) any server, networking or storage system that is capable of supporting more than one CPU per device; or (iii) any device that is designed, marketed or sold to support or be incorporated into systems covered in clauses (i) or (ii).”

    It's understandable that they'd state this, but given the much higher endurance of the 900P versus all their other consumer SSDs, it's still disappointing they're going with the same older boilerplate warranty statement. What if multiple users are using a Workstation, or a VMware Workstation is used to run multiple VMs that remote users access on a Windows 10 system?

    Then there's the question of how Intel would know what kind of workload the drive was subject to during its lifecycle. Presumably inferred from some of of their S.M.A.R.T. data? More investigation is needed. Obvious ethical issues aside, Intel 900P shoppers would still need to ask themselves whether they'd really like to find out how Intel enforces this, the hard way.


    Dec 27 2017 Update

    I regret to have to change the title of this article, from:

    • Intel Optane SSD 900P Series is the latest in 3D XPoint NVMe hotness, should be great for home virtualization lab enthusiasts
      to

    • Intel Optane SSD 900P Series is the latest in 3D XPoint NVMe hotness but it is not ideal for most home virtualization lab enthusiasts

    I try to change titles as infrequently as possible, and am not exactly happy when I essentially retract what I had hoped the 900P would be. It's just very hard to recommend something that is 2x the cost per GB of so many other good options. Only high stress enterprise workloads are currently likely to see the benefits of Optane, along with a handful of vSAN home lab enthusiasts with deep pockets.

    Please don't sweat this too much, you have lots of great options. The Samsung 960 EVO and Samsung 960 PRO continue to be excellent performers when formatted as VMFS, and the 960 PRO is available in up 2TB capacity. The 960s both now have mature firmware already loaded on them, and work out of the box with VMware ESXi's included NVMe drivers.

    Consumer drives like the 960 EVO and 960 PRO never claimed to be on the VMware HCL (for VMware ESXi support), nor on the Supermicro Xeon D systems' aging Tested M.2 List that features mostly enterprise devices, with higher write endurance than any home lab enthusiasts is ever likely to encounter, and higher prices than any home lab enthusiast is likely to justify. It also strangely has a lot of SATA3 AHCI devices which I would stay away from. The focus of Supermicro's limited test budgets seem to be mostly on production workload testing, on M.2 SSDs available when the motherboard first shipped. The focus is on on basic interoperability testing, and appropriate warranties. Keeping their M.2 list up-to-date doesn't appear to be one of Supermicro's priorities, but I've certainly been encouraging them to include a lot more M.2 NVMe SSDs on that list.

    The performance of Samsung 960 series of SSDs is very similar to the Intel 900P with typical VM workloads in most uses. Admittedly a bit of conjecture here, since my own 900P drives were unstable under load on my Xeon D-1541 system (PSODs), so it's a little hard to informally benchmark. Also note that my loaner, engineering-sample P4800X drive had firmware and speed issues of its own, and was been returned to Intel months ago. But even synthetic benchmarks posted elsewhere don't seem to show some huge benefit to read and write speeds at the important 4K size, which greatly affects the speed of VMs. Overall, the 960 Pro series enjoy considerably higher max throughputs for reads and writes, albeit at much higher latencies.

    The "gumstick" M.2 form-factor is much more convenient than HHHL AIC (PCIe cards), and I will continue to be comfortable using them in my home lab. I just don't expect any official support from Supermicro or Samsung or VMware, ever, and I'll keep making sure I automatically back up all VMs that matter to me.

    I've been hammering away at my Samsung 950 PRO formatted VMFS5 then VMFS6 for over 2 years now, with only roughly 26 TBW so far. No issues have been noted, despite some nasty abuse like abrupt system power-offs. Similar success for my year-old Samsung 960 EVO and Samsung 960 PRO drives, see also:

    929202430683774977

    Why am I not talking much about the affordable consumer Intel M.2 NVMe drives like the 600P here at TinkerTry? Because they're relatively new to Intel, their performance just isn't very competitive with Samsung quite yet, and they still don't seem to offer enterprise-warranty M.2 NVMe devices, just PCIe.

    Also worth noting that no significant rumors have surface lately about solid dates for Intel or Samsung's next-gen M.2 NVMe devices, but keep your eyes peeled. I currently don't have any special advance knowledge of what Intel plans to release next.

    Closing, upbeat thoughts

    For enterprises with heavy duty multi-user workloads and high IOPs 24x7, the P4800X will shine, with much improved write endurance, and consistent performance with low latency.

    For those with deep pockets and a thirst for experimentation like myself, the 900P will continue to be of considerable interest, and not just for the gamers it's marketed toward. I will probably still tinker with my 900P from time-to-time, especially as new firmware and device drivers arrive. Who knows, maybe my particular PSODs will even go away. Since I don't really know the root cause, I won't be able to get any official support: is it the driver, the firmware, an 900P-on-ESXi incompatibility, or a Xeon D incompatibility (unlikely, remember, the BIOS hang at A9 is annoying, but doesn't just happen on Xeon D). Maybe it's just an early adopter thing, and a future BIOS release addresses it. I've certainly encouraged Supermicro to look into this.

    Hopefully I'll get enough time and maybe even a budget to have a system to leave my 900P running, where not being able to reliably get into my BIOS isn't as big of an issue. Meanwhile, others like @brad_lay seem to be doing quite well with their 900P experiments, as pictured here. Consider it a peek at the future, where vSAN can only become more and more optimized for all that goodness that 3D XPoint has to offer, something VMware has publicly stated numerous times. The initial P4800X and 900P 3D XPoint drives are just the start of exciting times ahead, looking forward to M.2 NVMe packaging too. Given the current watt burn of Optane drives, that kind of density may mean the wait will be long, especially for capacities anywhere near the 2TB Samsung 960 PRO M.2 NVMe SSD.


    See also at TinkerTry

    All P4800x articles.

    All NVMe articles.

    intel-optane-publications-oct-2017

    ruler

    pcie-to-m2-nvme-accessories-overview

    See also

    • Intel Optane SSD 900P Review (480GB) – Understanding Disruptive Technology
      intel-optane-ssd-900p-review-480gb-understanding-disruptive-technology

      Nov 12 2017 by Les Tokar at The SSD Review

      The Intel Optane SSD 900P, and specifically Intel 3D XPoint memory changes this, in that, this memory can add, delete or replace information at the most basic level without requiring the large movement of data to accommodate such. This results in significant performance and endurance increase and is very much considered disruptive as it has the potential to change the storage industry as we see it today.

    • The Intel Optane SSD 900P 280GB Review
      the-intel-optane-ssd-900p-review-full

      Oct 27 2017 by Billy Tallis at AnandTech

      Who is the Optane SSD 900P for?
      With a price per GB a little over twice that of the the fastest flash-based consumer SSDs, the Optane SSD 900P is an exclusive high-end product. For most desktop usage, drives like the 960 PRO are already fast enough to make storage no longer a severe bottleneck. The most noticeable delays due to storage performance on a 960 PRO are when moving around large files, and the Optane SSD doesn't offer any significant improvement to sequential transfer speeds. Random writes can be a challenge for flash-based SSDs, but volatile write caches and SLC caches allow them to handle short bursts with very high performance.

    Here's a really thorough look at the subtleties of M.2 NVMe benchmark testing, and an overview of why NVMe is better than SATA in very measurable ways.

    • Benchmarking Samsung NVMe SSD 960 EVO M.2
      benchmarking-samsung-nvme-ssd-960-evo-m-2

      Mar 24 2017 by Taras Shved at StarWind

      Everyone knows that, currently, the SSDs are one of the best storage devices that allow you to upgrade your architecture and significantly accelerate the performance of the computer. SSD accelerates the loading speed of your PC, applications opening and files searching speed, and generally increases the performance of your system. Despite the fact that solid-state drives are more expensive than standard hard drives, the performance improvement can hardly be overlooked.

    • How 3D XPoint Phase-Change Memory Works
      pc-perspective-logo-small

      Jun 02 2017 by Allyn Malventano at PC Perspective

      While we were initially told at the XPoint announcement event Q&A that the technology was not phase change based, there is overwhelming evidence to the contrary, and it is likely that Intel did not want to let the cat out of the bag too early.