Intel Optane SSD 900P Series is the latest in 3D XPoint NVMe hotness but it is not ideal for most home virtualization lab enthusiasts

Disclosure: The U.2 version of the 900P 280GB was the only model available anywhere, so I purchased one to test. Intel later provided me a loaner AIC version of the 900P as well, in the 280GB capacity. There were no stipulations, I continue to candidly share my experiences. Unfortnately, things did not turn out well for using this 900P drives in my VMware environment, see the many reasons why detailed below.

Intel Optane SSD 900P Series AIC and U.2 Announced

Intel's October 27 2017 press release:

• Blazing-Fast Gaming with Intel’s First Client Optane SSD

  Intel today announced the launch of the Intel® Optane™ SSD 900P Series, the first SSD for desktop PC and workstation users built on Intel® Optane™ technology.
  ...
  The Intel® Optane™ SSD 900P Series is ideal for the most demanding storage workloads, including 3D rendering, complex simulations, fast game load times and more. Up to 22 times more endurance than other drives also gives the heaviest users peace of mind.

Shop

• Newegg
• Amazon? Probably best to choose only what's available on Amazon Prime.
  Nov 01 Update - Note, at 4:10pm eastern, Newegg auto-notifier informed me that the 280GB PCIe version might be available for pre-order, but by the time I checked later that evening, pre-orders were no longer being accepted, for any of Newegg's 900P versions.
  Nov 03 Update - Only the U.2 model with the U.2 connection became available, so I ordered it
  Nov 19 Update - Inventory at Newegg looking better, Amazon doesn't seem to have Prime orders in stock, not sure how I feel about the many 3rd party sources.

Introduction

This was a great week for home virtualization enthusiasts! A more affordable PCIe NVMe and U.2 drive is arriving soon, featuring the spectacular specs of the enterprisey P4800X that you saw here at TinkerTry first:

• World's First Close Look at Intel Optane SSD DC P4800X Series - PCIe NVMe arrives with 375GB of 3D XPoint!
  Aug 11 2017
   
   
   

What's the big deal here with the 900P then? Well, it's pretty much the same as a P4800, but at a greatly reduced price of $599.00 retail at Newegg for 480GB of Optane. Yes, still a hefty premium over SATA3 SSDs, but still less than half the price of the 375GB P4800X. This great-for-enterprise-workloads Optane P4800X might as well been called the Unobtainium P4800X, given the realities of most lab budgets. The 900P promised to bring most of that goodness down to earth, just be sure you have a PCIe 3.0 x4 lanes slot available to obtain top speeds.

Specifications say it all

Intel ARK for Intel Optane SSD 900P

This sucker is begging to be driven hard, see values highlighted values. yes, those aren't ms (milliseconds), those are µs (microseconds). What does that mean?

Well, 10 µs is 0.01 ms. Holy guacamole, that's half the latency of the Intel Solid State Drive 750 Series of 3 years ago.

Let's go further. How about the Endurance Ratings? How's 8.76 PBW (Petabytes Written) sound? Well, that converts to 8760 TBW (Terabytes Written). Compared with the 750 Series that has a 127 TBW (Terabytes Written) rating, that’s about a 68 times the write endurance.

Compare 750 Series 400GB with 900P 480GB on Intel Ark

• ark.intel.com/compare/86740,123626

What if we stick a DataCenter product in the mix, the Intel SSD DC P4600 Series 1.6TB. Not a fair comparison, but fun nonetheless:

• ark.intel.com/compare/86740,123626,97005

Various capacities, all run at the same speed

See also the main Intel Optane SSD 900P Series landing page.

Finally, a fun look at how involved proper testing is done in the Intel Optane Solid State Drive 900P Series Evaluation Guide.

Press Coverage

Now you know why the positive press is pouring in over this new little hotness in NVMe SSD, here's a select few:

• Intel Optane SSD 900P 480GB and 280GB NVMe HHHL SSD Review - Lots of 3D XPoint!
  Oct 27 2017 by Allyn Malventano at PC Perspective

  Optane’s strongest benefits are the very performance traits that do not effectively scale with additional drives added to an array. The preferred route is to just get a larger single SSD with more 3D XPoint memory installed on it, and we have that very thing today (and in two separate capacities)!

• Intel Optane 900P Low Price NVMe might Be a Perfect Fit For a Home Lab
  Oct 30 2017 by Vladan Seget at ESX virtualization.

  If you remember, a few months back VMware announced a support for latest Optane drives from Intel with 3D XPoint memory, and the solution was the fastest one tested, on vSAN. This has had of course a price (I saw a $1,791.00 on Amazon), which is quite pricey for a 375GB PCIe NVMe card. Not something a homelabber can afford, right?

• The Intel Optane SSD 900P 280GB Review
  Oct 27 2017 by Billy Tallis at AnandTech

  However this year's revolution from Intel will be very hard for the competition to match anytime soon. All of Intel's previous record-setting SSDs have relied on the drive's controller to stand out from the crowd. This time, Intel's advantage comes from the storage medium: its 3D XPoint memory technology, a new nonvolatile memory that offers much higher performance than flash memory.

VMware Compatibility Guide

VMware Compatibility Guide - I/O Device Search for Optane only shows the two P4800X drives, AIC (Add in Card) and U.2 (hot swap NVMe in a 2.5" enclosure).

I have reached out for comment on whether this 900P will ever appear on the VCG (VMware Compatibility Guide), but likely the answer will be no. Even less likely to ever be on vSAN VCG. This is really a consumer drive here, and VMware's vSAN testing is on the supported gear for the enterprise, for very good reasons.

Power Loss Protection in a consumer NVMe drive

Here's the thing. The 900P has PLP (Power Loss Protection), sometimes referred to as Supercapacitors. (See Nov 29 2017 correction below, it turns out the 900P does NOT have PLP) PLP was actually just discussed on the Virtually Speaking Podcast, right at this spot. Yes, scary stories just in time for Halloween! PLP capability could mean that the 900P is the best somewhat-affordable choice as a caching layer for vSAN lab tests. Completely unsupported by VMware mind you, but a lot less likely to need support in the first place. Consumer drives don't usually make sure uncommitted writes make it out of the buffer and into the non-volatile NAND, but this 3D XPoint sure does! Nobody wants to hear storage array and corruption in the same sentence, no matter how backed up you are.

vSAN?

Let's be real clear here. In no way am I recommending that such consumer drives be used for vSAN, especially when it comes to production workloads where having a support contract with completely supported enterprise rated components is essential.

There is no telling whether using the 900P for vSAN is a good idea for a home lab, with hands-on testing certainly needed. C'mon now, the press release even says "Blazing-Fast Gaming."

So, as a thought experiment, let's just say you were to format a 900P as VMFS 6 using VMware vSphere / ESXi 6.5 Update 1. You then use this datastore for nested ESXi 6.5U1 instances, creating a 4 or even 6 node vSAN. Alternatively, maybe you get even more "creative" and try NVMe pass through of each NVMe device you have, which isn't hard at all anymore, see my recent How to configure VMware ESXi 6.5.x for VMDirectPath I/O pass-through of any NVMe SSD, such as Windows Server 2016 installed directly on an Intel Optane P4800X booted in an EFI VM. Got your attention yet? Maybe you've got yourself one very fast vSAN caching tier in your lab tests for your other more affordable SSDs. Maybe...

These 900P SSDs aren't in stock or even available for pre-order yet

So where's that leave TinkerTry? Well, for now, I can't know how well this works or performs until I can get my hands on one of these. Please know that I sure am trying to get one, as is nearly every other blogger out there.

Closing thoughts

Decent capacities in M.2 sizes are a long way off

Even more versatile would be a 900P in the M.2 form factor, like a much better version of the original M.2 Optane module, see:

• Intel Optane M.2 16GB/32GB consumer NVMe SSDs for Windows caching are not that fast as VMware VMFS or NTFS, save up for bigger PCIe versions
  Jun 11 2017

What I'm getting at here is that you can stuff 4 M.2 drives into a single PCIe 3.0 x 16 slot, and pass each NVMe device through to a VM independently. That would be a much more practical way to handle cache testing scenarios such as vSAN POCs.

Thermal dissipation

As for cosmetics and market differentiation? Nice job on the distinctive backplate Intel, good move. Smart looking, and flow-through practicality, demonstrating your apparent mastery of AIC cooling that was so evident in my FLIR thermal imaging tests of the similarly designed P4800X that easily handled my no air flow tests. I strongly suspect the 900P will perform very similarly, can't wait to get my hands on one to try it.

Image Gallery

Sorry, these are just renders, not quite like taking close-ups of a production sample, such as I did with the loaner P4800X here. Still, you'll get a pretty good idea of how the stickered GA product will look. After clicking on image, don't forget to go full screen with the icon at top right. Enjoy!

Video

Nov 02 2017 Update

In the original article above, I neglected to mention that you should look closely at that photo gallery, specifically, the last photo. See the caption? It's the Intel Optane SSD 900P Series U.2 Box Contents M.2 Cable that has my attention for folks that own the SYS-5028D-TN4T system. While host swap U.2 drives, which look like extra thick 2.5" laptop drives, wouldn't normally do much good in this SATA3 system. But remember, there's 2 bays in there for 2.5" drives that get screwed into a bracket, and both have extra space for taller drives. As for the 1U height systems like the SYS-E300-8D and SYS-E200-8D, I strongly suspect there just isn't near enough clearance in there, where the 7mm SSDs I tried on my loaner 1U systems barely fit, seen here and here.

Special thanks to a TinkerTry reader who wrote in on November 1st, bringing this oversight to my attention.

Internally mounted U.2 thermals

What about airflow? Admittedly, the 4 hot-swap SATA drive bays get the airflow, but there isn't much air across the two internal drive bays. That said, even with no airflow, the similar P4800X did fine, even with quite a bit of stress, as already mentioned above, and seen here. So I suspect the aluminum heat sinks integrated into the U.2 900P design will be sufficient, with Intel learning from years of making U.2 drives for the enterprise, see also my visit to Intel at VMworld 2015 featured here. That said, let's not forget that U.2 dives are typically mounted where there's active airflow in a noisy 2U server. Such airflow is not easily possible with this particular system.

Intel U.2 to M.2 Adapter Cable (18" length confirmed by Intel)

This cable can now be ordered separately at Intel.
Installation should be this simple:

1. remove either of the 2.5" drive brackets by removing the 2 chassis screws.
2. secure the U.2 900P to the bracket using 4 2.5" screws that you'll apparently need to provide.
3. reinstall the bracket into the SuperServer again with the same 2 chassis screws you removed earlier.
4. running the included U.2 to M.2 cable down to the motherboard's M.2 slot that runs at full PCIe 3.0 x4 lane speeds.

That's it! Now you've got an additional 900P U.2 drive added to your mini-tower system, leaving your precious PCIe slot open for a good cause, such as:

• another P900, the HHHL PCIe form-factor version.
• a GPU.
• a 4 way full speed M.2 PCIe card that leverages this system's 4x4x4x4 bifurcation feature that's easily enabled in the BIOS.

How could I fit two 900P drives into my SYS-5028D-TN4T mini-tower?

1. one U.2 900P in the side internal 2.5" bay, connected to the motherboard's one M.2 slot using the included cable
2. one PCIe 900P in the one PCIe slot

How could I fit three 900P drives into my SYS-5028D-TN4T mini-tower?

You'd need the extra expense of an SFF NVMe adapter card, such as the Supermicro OC-SLG3-2E4 (see User's Guide that's available at Wiredzone, and two U.2 P900 SSDs that include the SFF 8643 cable, used to connect to the OC-SLG3-2E4 in the PCIe slot, and a third U.2 P900 with the included M.2 cabled, used to connect to the motherboard's M.2 slot.

Reference Material

• Physical Installation Instructions for Intel® Optane™ SSD 900P Series
  Oct 27 2017 by Intel Support, excerpt:

  Installation steps for the Intel® Optane™ SSD 900P Series 2.5-inch U.2 drive

  1. Mount the SSD in the chassis using screws to hold the SSD securely
  2. Connect the cable to the U.2 SSD connector (SFF 8639)
  3. Depending on the cable that came in your box at purchase, either connect the:
     M.2 card in the PCIe supported M.2 motherboard connector
     SFF 8643 to your PCIe supported motherboard connector
  4. Connect SATA power to the connector on the cable

• What’s in the Box for Intel® Optane™ SSD 900P Series
  Oct 27 2017 by Intel, excerpt:

  Article ID: 000025739
  The Intel® Optane™ SSD 900P Series is available in three different retail kits.
  Note - This type of solid-state drive does not receive data from SATA / AHCI connectors.

  Intel® Optane™ SSD 900P Series Retail Box for PCIe* U.2 2.5” with SFF 8643 Cable
  Intel® Optane™ SSD 900P Series (U.2 2.5” form factor)
  U.2 to SFF 8643 Adapter Cable
  Warranty Booklet

  Intel® Optane™ SSD 900P Series Retail Box for PCIe* U.2 2.5” with M.2 Cable
  Intel® Optane™ SSD 900P Series (U.2 2.5” form factor)
  U.2 to M.2 Adapter Cable
  Warranty Booklet

  Intel® Optane™ SSD 900P Series Retail Box for PCIe* Add-In Card
  Intel® Optane™ SSD 900P Series (add-in card form factor)
  Half Height Bracket
  Warranty Booklet

Nov 03 2017 Update

It's here! I now have my "Intel® Optane™ SSD 900P Series Retail Box for PCIe* U.2 2.5” with SFF 8643 Cable." I don't yet have the U.2 Adapter, such as the OC-SLG3-2E4, so I can't test it yet, but I did testing mounting it in the SYS-5028D-TN4T chassis, and it went well. Unboxing video, recorded in 4K, is being produced now, hopefully ready to share by Nov 4th.

Added 900P unboxing video above.

Nov 14 2017 Update

It's here! My Supermicro AOC-SLG3-2E4 - Supermicro NVMe PCIe Host Bus Adapter - 2 internal NVMe ports 6.4GB/s Gen-3 and were both home tonight, and got a chance to unbox and install the PCIe card, on camera of course. Kind of a cool moment when I quickly realize that the same exact BIOS tweaks that were needed for my earlier P4800X tests were needed for the 900P as well.

Of course, I learned the hard way again that I'd get a BIOS POST hang that simply says A9 as soon as the AOC-SLG3-2E4 is installed in the PCIe slot with a 900P attached, just like I saw with the very similar P4800X.

Remove the AOC-SLG3-2E4, and the boot hang went away. Not sure what would have happened had I simply removed the U.2 connection cable from the drive.

I used my own exact BIOS settings fix for this hang, right at this spot in the original P4800X article. Short version is be sure that:

• SLOT7 PCI-E 3.0 X16 Bifurcation is set to default x16
• SLOT7 PCI-E 3.0 X16 OPROM is changed from the default of Legacy to EFI

and now I was good to go, boots up and works just fine. I have a lot of basic changes to attend to next, including:

• upping the number of Paravirtual SCSI adapters and vCPUs, as outlined by Intel here
• revisiting my BIOS settings, as Intel describes here
• revisiting my NVMe driver, note that there are no ESXi downloads for the 900P, only for the similar P4800X, this isn't at all surprising, given the consumer focus of the 900P.
• check on the 900P firmware, by booting over to Windows and trying out Intel's utility
• check out performance on Windows 10 + NTFS using the 2.0.0.1024 NVMe driver, from:
  NVMe* Drivers for Intel Optane SSD 900P Series.
• view the Supermicro AOC-SLG3-2E4 PCIe card and the 900P U.2 drive itself with my FLIR thermal imaging camera while under extended load

That's just for starters. Once the PCIe version of the 900P arrives, and/or the U.2 to M.2 cable, I'll have some more tests up my sleeve to share with you, stay tuned!

New video now available above, "Installing the Supermicro AOC-SLG3-2E4 U.2 PCIe adapter with the Intel Optane 900P [4K]".

Video production underway of initial testing under ESXi 6.5U1, to be published right here soon. Done! Added to the video gallery above.

Nov 19 2017 Update

Intel has provided me with one of each of the following:
• Qty 1: Intel U.2 to M.2 Adapter Cable
• Qty 1: Intel Optane SSD 900P 280GB (U.2)

which enabled me to proceed with testing that both 900P drives work properly in the SYS-5028D-TN4T Supermicro SuperServer system, concurrently.

First, I removed the Supermicro AOC-SLG3-2E4 PCIe adapter card. I then wired up my 900P U.2 drive to the motherboard's M.2 slot using the Intel U.2 to M.2 cable. Finally, I was able to use the PCIe slot for the loaner 900P AIC (Add-In Card), as pictured below.

Problems were unearthed that slowed my progress a bit:

1. BIOS tweak needed to avoid BIOS hang at A9 error, setting EFI for both the M.2 and PCIe slots, recommended BIOS settings article updated
2. if both M.2 and PCIe are occupied by Optane, I can't get back into the BIOS without removing the PCIe card (upcoming video I'll publish shows me figuring this out)
3. coincidentally timed bump in the road explained here.

Testing continues to go fairly well overall, more videos and additions to this article to come.

Next up will be booting over to Windows to check in on firmware levels, and getting my video published of the 900P AIC unboxing and install, along with my VMFS 6 formatting and initial testing of the new 900P AIC under VMware ESXi 6.5U1.

Nov 20 2017 Update

As expected, the stated absolute throughput values of 2500Mbps reads and 2000 MBps writes the can be exceeded slightly by other M.2 SSDs like the Samsung 960 PRO, and my initial testing slightly exceeding these values when using simple ATTO Disk Benchmark tests in a Windows 10 VM that's running on ESXi 6.5U1, with the Optane 900P formatted VMFS 6.

There's more to the story here. Don't forget the latency, which I haven't really tested at all yet. There's also the lack of any need for typical SSD housekeeping like garbage collection, so performance is consistent, all the time. Don't forget the ability to fill this sucker up to 99% without worrying about performance degradation, not something that can be said about older SSDs where performance tanked once you accidentally filled the SSD accidentally, which isn't exactly hard to do quite by accident in these days of thin provisioned VMs.

As Allyn Malventano says on PC Perspective Podcast episode 470, at this spot:

From a benchmarking standpoint, Optane is a dream to test, you don't have to
worry about pre-conditioning...the performance is always stable, it doesn't matter what you do to it.

In the context of his VROC tests, Allyn goes on to discuss comparative performance at this spot:

I compared the 4 drive Optane array to a 4 drive 960 PRO 512 gig drive array, so 2 terabytes worth of Samsung's fastest SSD that they make...how high does the throughput go?...Looking at random access, now you would think that the 960 PROs would be pretty quick, but the Optane array, even though it's 1/20th of the capacity...just walks over it in random access...how high are the IOPs when you're at low queue depths, because that's, you know, where you'll actually be.

Power Loss Protection.

According to the Intel spec sheets, the Intel® Optane™ SSD 900P Series doesn't offer what Intel calls Enhanced Power Loss Data Protection.

According to the Intel spec sheets, the Intel® Optane™ SSD DC P4800X Series does offer what Intel calls Enhanced Power Loss Data Protection.

Is this a concern, especially if you're considering using the 900P in a completely unsupported use-case, such as in the vSAN caching layer? This warrants further investigation.

Meanwhile, see what chuckula says at phoronix here:

Absolutely none of Intel's Optane parts (including the $3000+ enterprise drives) include large capacitors for power loss prevention.

BUT: EVERY Optane product on the market now including the consumer drive Phoronix tested and even the cheap 16GB cache drives provides power loss protection.

That's because none of these drives use a RAM cache at all and the data are stored fast enough that PLP is an inherent feature of the drive that doesn't need extra circuitry.

Nov 21 2017 Update

With the impending holiday that I quite enjoy, with family and friends all around, I wanted to get as much as I could documented for you, given I'll be falling behind on the follow-along-with-me video production:

• I updated my ESXi 6.5U1 to Intel's 1.3.2.4 version of NVMe driver, from:
  Download VMware ESXi 6.5 intel-nvme 1.3.2.4 NVMe Driver for Intel(R) DC Series NVM Express Solid-State Drives
  my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI65-INTEL-INTEL-NVME-1324&productId=614

• I dual booted over to Windows 10, installed the latest toolbox from
  Download Intel® Solid State Drive Toolbox
  downloadcenter.intel.com/download/27330/Intel-Solid-State-Drive-Toolbox
  only to discover that both of my 280GB 900P Optane SSDs (U.2 and AIC) are already at the latest firmware level E2010325.

• I am finding performance using ATTO Disk Benchmark to be as expected

• the BIOS A9 hangs continue to be a problem only when I want to get in to the BIOS to change something, work-around is to pull the Optane drives out, make my changes, then put them back in

• I do not yet have an answer from Intel to help get clarity on Enhanced Power Loss Data Protection on the 900P versus the P4800X Optane parts

Nov 22 2017 Update

Today, I downloaded Intel SSD Data Center Tool 3.0.4, which is the latest version for VMware. Then I followed the installation instructions in the filename Intel_SSD_DCT_Install_Guide_3_x.pdf included in the downloaded DataCenterTool_3.0.4_ESXi.zip file, and unfortunately, it doesn't seem to pick up that two 900P drives are installed in my system with the 1.3.2.4 NVMe Driver. This thwarted my efforts to see the NVMe firmware levels, and my usual procedure here doesn't work either:

• How to find NVMe SSD firmware versions in a VMware ESXi 6.5 Server

  

I also tweeted some ATTO Disk Benchmark early tests, but Twitter blurs screenshots pretty badly. Here's the full resolution pic.

Tomorrow, I'll be eating turkey. See you right back here soon!

Nov 29 2017 Update

I heard back from an Intel spokesman with a definitive response on the whole question about Power Loss Protection:

As an enterprise part, the Intel® Optane™ SSD DC P4800X offers multiple data protection features that the Intel® Optane™ SSD 900P does not, including DIF data integrity checking, circuit checks on the power loss system and ECRC. The DC P4800X also offers a higher MTBF/AFR rating.

Dec 12 2017 Update

I'm working with VMware support lately, as it seems like there's a chance that my 2 recent PSODs might both be NVMe driver related. There's also some small chance it's also related to the old KB 2146388 PSOD, and it definitely not KB 2151749 (10G connection loss), despite the "#PF Exception 14" similarity.

Here's the details of one of one of my PSODs, note that it actually mentions NVMe, and my BIOS 1.2c:

=========================================

PSOD =

2017-11-30T17:26:14.926Z cpu13:65974)@BlueScreen: #PF Exception 14 in world 65974:intel-nvme-c IP 0x0 addr 0x0
PTEs:0x1e3dd1027;0x1925ce027;0xbfffffffff001;
2017-11-30T17:26:14.926Z cpu13:65974)Code start: 0x41801d000000 VMK uptime: 0:15:42:28.981
2017-11-30T17:26:14.929Z cpu13:65974)base fs=0x0 gs=0x418043400000 Kgs=0x0
2017-11-30T17:26:14.929Z cpu13:65974)vmkernel             0x0 .data 0x0 .bss 0x0

BIOS =

Vendor: "American Megatrends Inc."
Version: "1.2c"
Date: "09/19/2017"
Version: "Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz"

=========================================

Up until recently, I had never experienced the PSOD in KB 2146388 myself, and it had seemingly been resolved with Supermicro's release of BIOS upgrade 1.1c way back in October of 2016.

This could well be the presence of the 2 900P Optane drives (notice NVMe mention in the first line of the PSOD above) and/or the Intel 1.3.2.4 NVMe Driver VIB, active investigation is still underway. My VMware Support SR# 17648400312 and details of my PSODs have been reported to both Supermicro and Intel.

Dec 23 2017 Update

Well, it's not looking so good for folks ever looking for 900P Optane with VMware ESXi for a variety of reasons that include:

1. No power loss protection
   This important detail, confirmed by an Intel spokesperson above, means the 900P will never be well-suited for vSAN. It also assures that it has no chance of ever gaining official support from VMware since it's a consumer drive that won't appear on the VCG, never mind the more stringent vSAN VCG. It will have an inherently higher chance of data corruption for those who choose to use it for vSAN anyway.
2. No support under VMware ESXi - PSODs experienced when using the P4800X's VIB
   While Intel's 1.3.2.4 NVMe VIB that's intended for the P4800X may sort of work, I'm getting PSODs. These PSODs seem to be not just happening on Xeon D systems. According to the VMware technician who had a look at my logs under VMWare SR# 17648400312:

   I have checked the new set of logs and can see the post reboot, there are still command '0x93' repeatedly failing on these drives...Either the drives or the driver still seem to be not looking normal in the current state.

   2017-12-22T22:27:12.865Z cpu1:65997)ScsiDeviceIO: 2962: Cmd(0x439501175840) 0x93, CmdSN 0x5e7 from world 68537 to dev "t10.NVMe____INTEL_SSDPED1D280GA_PHMB73600129280CGN__00000001" failed H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x0 0x0.
   2017-11-30T17:26:14.836Z cpu13:65974)NMP: nmp_ThrottleLogForDevice:3617: Cmd 0x28 (0x439511235040, 67280) to dev "t10.NVMe____INTEL_SSDPE21D280GA_PHM2739000EC280AGN__00000001" on path "vmhba2:C0:T0:L0" Failed: H:0x2 D:0x0 P:0x0 Invalid sense data: 0x0 0$

3. I still get BIOS hangs at A9
   Whenever the 900P is installed and I try to get into BIOS settings using the Del key, and I hear this might be occurring on non-Xeon D motherboards too.
4. No warranty
   Not surprising, but still disappointing. I've recently learned from an Intel spokesperson that it appears VMware vSphere is outside of the supported and warrantied uses, seen in the warranty here, notably this clause:

   “Additionally, the Product will not be subject to this Limited Warranty if used in: (i) any compute, networking or storage system that supports workloads or data needs of more than one concurrent user or one or more remote client device concurrently; (ii) any server, networking or storage system that is capable of supporting more than one CPU per device; or (iii) any device that is designed, marketed or sold to support or be incorporated into systems covered in clauses (i) or (ii).”

It's understandable that they'd state this, but given the much higher endurance of the 900P versus all their other consumer SSDs, it's still disappointing they're going with the same older boilerplate warranty statement. What if multiple users are using a Workstation, or a VMware Workstation is used to run multiple VMs that remote users access on a Windows 10 system?

Then there's the question of how Intel would know what kind of workload the drive was subject to during its lifecycle. Presumably inferred from some of of their S.M.A.R.T. data? More investigation is needed. Obvious ethical issues aside, Intel 900P shoppers would still need to ask themselves whether they'd really like to find out how Intel enforces this, the hard way.

Dec 27 2017 Update

I regret to have to change the title of this article, from:

• Intel Optane SSD 900P Series is the latest in 3D XPoint NVMe hotness, should be great for home virtualization lab enthusiasts
  to

• Intel Optane SSD 900P Series is the latest in 3D XPoint NVMe hotness but it is not ideal for most home virtualization lab enthusiasts

I try to change titles as infrequently as possible, and am not exactly happy when I essentially retract what I had hoped the 900P would be. It's just very hard to recommend something that is 2x the cost per GB of so many other good options. Only high stress enterprise workloads are currently likely to see the benefits of Optane, along with a handful of vSAN home lab enthusiasts with deep pockets.

Please don't sweat this too much, you have lots of great options. The Samsung 960 EVO and Samsung 960 PRO continue to be excellent performers when formatted as VMFS, and the 960 PRO is available in up 2TB capacity. The 960s both now have mature firmware already loaded on them, and work out of the box with VMware ESXi's included NVMe drivers.

Consumer drives like the 960 EVO and 960 PRO never claimed to be on the VMware HCL (for VMware ESXi support), nor on the Supermicro Xeon D systems' aging Tested M.2 List that features mostly enterprise devices, with higher write endurance than any home lab enthusiasts is ever likely to encounter, and higher prices than any home lab enthusiast is likely to justify. It also strangely has a lot of SATA3 AHCI devices which I would stay away from. The focus of Supermicro's limited test budgets seem to be mostly on production workload testing, on M.2 SSDs available when the motherboard first shipped. The focus is on on basic interoperability testing, and appropriate warranties. Keeping their M.2 list up-to-date doesn't appear to be one of Supermicro's priorities, but I've certainly been encouraging them to include a lot more M.2 NVMe SSDs on that list.

The performance of Samsung 960 series of SSDs is very similar to the Intel 900P with typical VM workloads in most uses. Admittedly a bit of conjecture here, since my own 900P drives were unstable under load on my Xeon D-1541 system (PSODs), so it's a little hard to informally benchmark. Also note that my loaner, engineering-sample P4800X drive had firmware and speed issues of its own, and was been returned to Intel months ago. But even synthetic benchmarks posted elsewhere don't seem to show some huge benefit to read and write speeds at the important 4K size, which greatly affects the speed of VMs. Overall, the 960 Pro series enjoy considerably higher max throughputs for reads and writes, albeit at much higher latencies.

The "gumstick" M.2 form-factor is much more convenient than HHHL AIC (PCIe cards), and I will continue to be comfortable using them in my home lab. I just don't expect any official support from Supermicro or Samsung or VMware, ever, and I'll keep making sure I automatically back up all VMs that matter to me.

I've been hammering away at my Samsung 950 PRO formatted VMFS5 then VMFS6 for over 2 years now, with only roughly 26 TBW so far. No issues have been noted, despite some nasty abuse like abrupt system power-offs. Similar success for my year-old Samsung 960 EVO and Samsung 960 PRO drives, see also:

• Where to buy your Samsung 960 EVO or PRO M.2 NVMe SSDs, featuring the latest ordering and availability info
  Nov 30 2016

Why am I not talking much about the affordable consumer Intel M.2 NVMe drives like the 600P here at TinkerTry? Because they're relatively new to Intel, their performance just isn't very competitive with Samsung quite yet, and they still don't seem to offer enterprise-warranty M.2 NVMe devices, just PCIe.

Also worth noting that no significant rumors have surface lately about solid dates for Intel or Samsung's next-gen M.2 NVMe devices, but keep your eyes peeled. I currently don't have any special advance knowledge of what Intel plans to release next.

Closing, upbeat thoughts

For enterprises with heavy duty multi-user workloads and high IOPs 24x7, the P4800X will shine, with much improved write endurance, and consistent performance with low latency.

For those with deep pockets and a thirst for experimentation like myself, the 900P will continue to be of considerable interest, and not just for the gamers it's marketed toward. I will probably still tinker with my 900P from time-to-time, especially as new firmware and device drivers arrive. Who knows, maybe my particular PSODs will even go away. Since I don't really know the root cause, I won't be able to get any official support: is it the driver, the firmware, an 900P-on-ESXi incompatibility, or a Xeon D incompatibility (unlikely, remember, the BIOS hang at A9 is annoying, but doesn't just happen on Xeon D). Maybe it's just an early adopter thing, and a future BIOS release addresses it. I've certainly encouraged Supermicro to look into this.

Hopefully I'll get enough time and maybe even a budget to have a system to leave my 900P running, where not being able to reliably get into my BIOS isn't as big of an issue. Meanwhile, others like @brad_lay seem to be doing quite well with their 900P experiments, as pictured here. Consider it a peek at the future, where vSAN can only become more and more optimized for all that goodness that 3D XPoint has to offer, something VMware has publicly stated numerous times. The initial P4800X and 900P 3D XPoint drives are just the start of exciting times ahead, looking forward to M.2 NVMe packaging too. Given the current watt burn of Optane drives, that kind of density may mean the wait will be long, especially for capacities anywhere near the 2TB Samsung 960 PRO M.2 NVMe SSD.

Apr 29 2018 Update

• U.2 to M.2 SSD cable - Replacement U.2 to M.2 Cable for PCIe NVMe supporting Intel® Solid State Drives

  U.2 to M.2 cable for data and power delivery to U.2/2.5” PCIe devices
  Supports drives that require 12V source. Requires external PSU SATA power connector.
  18” for flexibility of mounting U.2/2.5” drives in drive bays.
  Shielded for support of reference clock.

See also at TinkerTry

All P4800x articles.

All NVMe articles.

• Intel announcements include details about benchmarking Optane P4800X in VMware ESXi, new 750GB capacity, and 3D XPoint fab expansion
  Nov 13 2017

• Intel's new Ruler form factor allows up to 1PB of NVMe in 1U, along with a new 2.5" 3.8TB SATA SSD and Dual Port SSDs for Data Centers
  Aug 15 2017
   
   

• M.2 expansion for your NVMe SSDs - EZDIY adds, Angelbird Wings adds and cools, Amfeltec Squid adds, cools, and quadruples
  Dec 06 2016

See also

• Samsung SZ985 Z-NAND SSD - Upcoming Competition for Intel's P4800X?

  

  Nov 20 2017 by Allyn Malventano at PC Perspective

  While the SZ985 runs at ~1/3rd the latency of Samsung's own NAND SSDs, it has roughly double the latency of the P4800X. For the moment that is actually not as bad as it seems as it takes a fair amount of platform optimization to see the full performance benefits of optane, and operating slightly higher on the latency spectrum helps negate the negative impacts of incorrectly optimized platforms.

  

  Image Source: ExtremeTech

• Intel Optane SSD 900P Review (480GB) – Understanding Disruptive Technology

  

  Nov 12 2017 by Les Tokar at The SSD Review

  The Intel Optane SSD 900P, and specifically Intel 3D XPoint memory changes this, in that, this memory can add, delete or replace information at the most basic level without requiring the large movement of data to accommodate such. This results in significant performance and endurance increase and is very much considered disruptive as it has the potential to change the storage industry as we see it today.

• The Intel Optane SSD 900P 280GB Review

  

  Oct 27 2017 by Billy Tallis at AnandTech

  Who is the Optane SSD 900P for?
  With a price per GB a little over twice that of the the fastest flash-based consumer SSDs, the Optane SSD 900P is an exclusive high-end product. For most desktop usage, drives like the 960 PRO are already fast enough to make storage no longer a severe bottleneck. The most noticeable delays due to storage performance on a 960 PRO are when moving around large files, and the Optane SSD doesn't offer any significant improvement to sequential transfer speeds. Random writes can be a challenge for flash-based SSDs, but volatile write caches and SLC caches allow them to handle short bursts with very high performance.

• New Optane disks appear on web shops' lists
  280GB and 480GB beasts on PCIE 4.0, apparently

  

  Oct 25 2017 by Team Register at The Register

  SSDPE21D280GASM, a 2.5-inch drive with 280GB capacity and U.2 or M.2 connections
  SSDPE21D280GASX, as above, but with PCIE 4.0 connector
  SSDPED1D280GASX, a 280GB add-in card for PCIE 4.0
  SSDPED1D480GASX, a 480GB add-in card, again for PCIE 4.0

Here's a really thorough look at the subtleties of M.2 NVMe benchmark testing, and an overview of why NVMe is better than SATA in very measurable ways.

• Benchmarking Samsung NVMe SSD 960 EVO M.2

  

  Mar 24 2017 by Taras Shved at StarWind

  Everyone knows that, currently, the SSDs are one of the best storage devices that allow you to upgrade your architecture and significantly accelerate the performance of the computer. SSD accelerates the loading speed of your PC, applications opening and files searching speed, and generally increases the performance of your system. Despite the fact that solid-state drives are more expensive than standard hard drives, the performance improvement can hardly be overlooked.

• How 3D XPoint Phase-Change Memory Works

  

  Jun 02 2017 by Allyn Malventano at PC Perspective

  While we were initially told at the XPoint announcement event Q&A that the technology was not phase change based, there is overwhelming evidence to the contrary, and it is likely that Intel did not want to let the cat out of the bag too early.

• Supermicro AOC-SLG3-2E4R and Supermicro AOC-SLG3-2E4 Differences

  

  May 30 2015 by Patrick Kennedy at STH

  The Supermicro AOC-SLG3-2E4R – $149 (Amazon or Ebay but I got from ebay) is a lower cost part with no PLX chip onboard. It also therefore consumes less power but has a much tighter compatibility matrix.