World's First Close Look at Intel Optane SSD DC P4800X Series - PCIe NVMe arrives with 375GB of 3D XPoint!
Disclosure
This SSD is on a temporary loan from Intel, and it appears to be an engineering sample, in the 375GB size that is planned for product launch. This loaner was made with no formal expectations or stipulations, just a brief chance to TinkerTry this datacenter technology in my own VMware vSphere home lab, in line with this site's tagline - TinkerTry IT @ home. Efficient virtualization, storage, backup, and more. The loaner P4800X was returned to Intel in October 2017.
Completely New Type of Storage
Such an opportunity doesn't come along often. If you've followed TinkerTry for a while, you'll know that I'm always happy to get my hands on something new, such as the world's first Intel Xeon D-1541 and Xeon D-1567 SuperServers, and the very first SYS-E300-8D and SYS-E200-8D SuperServers. I was even lucky enough to get a look at the early days of PCIe NVMe heralded by the Intel 750 Series back in 2015. I'm very glad to have this chance to tinker with new tech, again!
While many of us have been enjoying the benefits of NAND-based flash memory for nearly a decade now, the limits of what NAND can handle in terms of write speed and endurance have been increasingly obvious lately. See also:
3D XPoint
I've been writing about 3D XPoint, which Intel dubs Optane, since February of 2016. And now, I finally have it in hand. So tantalizing! My recent article title sums its capabilities up as succinctly as I could:
Plans
- I can't wait to make some rough measurements of power use in watts, at idle and under load, using with my Ubiquiti mPower Pro power strip. Not as fancy or precise as PC Perspective's measurement rig, but a good start.
- I also plan check out temperatures using my FLIR ONE Thermal Imaging camera.
- I want to have a look at the driver install under VMware and under Windows Server 2016.
- I hope to look at general VMFS performance characteristics when used with VMware ESXi 6.5 Update 1.
- I plan to update this article when I have those measurements.
Enterprise versus Consumer
Note that there is likely to be a more consumer focused version of this product coming soon, featuring both a lower price and an appropriate warranty policy. You may recall that Intel's DC S3700 Series (DC for DataCenter) was followed up by the Intel 750 Series aimed at enthusiasts/consumers.
Warning
It appears some BIOS updates and/or UEFI settings changes may be needed to (hopefully) allow this Optane drive to become "visible" to VMware ESXi in my Xeon D server. I've loaded up the Intel NVMe driver labeled for use with the P4800X, but I'm only able to check the firmware, not VMFS format the drive.
Note, Supermicro makes no claims this particular drive has been tested for compatibility at a hardware or software level. I can't make any promises, but I also don't give up easily. Stay tuned to TinkerTry for more updates as they happen.
Meanwhile, you'll find a short unboxing video, pictures, and lot more information about 3D XPoint in the set of articles listed below. If time permits, I hope to also try a nested environment of ESXi 6.5 Update 1 running vSAN 6.6.1, we'll see.
Shop
If you're in possession of a new server that has been fully tested for P4800X compatibility, such as the new Intel Xeon Scalable Processor family of systems, you can be assured of compatibility, and can now pre-order the P4800X on Amazon, with ship dates currently showing August 12-17. I'm a bit skeptical though, as that listing also says "Date first available at Amazon.com" as May 19 2017. I am not seeing other major sites taking pre-orders just yet.
Sep 01 2017 Update - It seems mouser.com has them in stock, see details below.
Photos
Video (4K)
Aug 11 2017 Evening Update
I'm beginning to make some progress by skipping VMware testing for now, and cutting over to Windows Server 2016. BIOS is in UEFI mode, at these Recommended Settings, as mentioned in my Nov 05 2015 article about booting from NVMe.
Aug 12 2017 Update
I managed to get past the BIOS configuration issues that caused POST to hang on A9, will document details later. This allowed me to do a quick test of the P4800X under Windows Server 2016, with more testing to follow.
Aug 13 2017 Update
Things are going well. I now have some informal ATTO Disk Benchmarks to share.
All these informative windows were brought up after the initial ATTO Disk Benchmark, subsequent tests showed consistently impressive results, whether the network adapters were disabled or not. I had to wait for all Windows Updates to finish, and made sure CPU was at or below 1% utilization before each run. More tools and tests results coming soon.
Below are some informal looks at performance, with nowhere near the complexity and stringent documentation required of proper, official benchmarks, as detailed by Intel here, for example. These are consumer tools run on an enterprise drive, with things like HCIBench and Iometer really more appropriate for such a product, which are a little more error prone and time consuming.
Tests with shipping product may differ from what is seen here.
Test Hardware and Software Configuration
- Supermicro SuperServer SYS-5028D-TN4T Bundle 2 featuring Xeon D-1541
Bundle 2 as configured here, with 128GB of RAM and 8 CPU cores - BIOS 1.2a and IPM 3.58, downloads here
- BIOS configured per Recommended Settings here, UEFI mode
- Windows Server 2016 Standard Desktop Experience, download per instructions here, this filename:
14393.0.160715-1616.RS1_RELEASE_SERVER_EVAL_X64FRE_EN-US.ISO - Intel Optane SSD DC P4800X Series - PCIe NVMe 375GB, with the firmware that was already loaded on this engineering sample (no firmware update tools are available that I could find)
- default GPT/NFTS filesystem used
Test A - Windows Server 2016 natively installed and booted
Boot from NVMe configured, I wrote more about this back in November of 2015, when it was less common, and a bit harder to do.
Test B - Windows Server 2016 natively installed, but booted as VM
Yes, I used VT-d passthrough of the same Windows Server 2016 on P4800X SSD for this test, worked beautifully, here's the steps:
- VMware ESXi 6.5 Update 1 installed on 32GB USB, downloaded from my links here
- configured hypervisor to passthrough the P4800X NVMe PCI device and rebooted
- VM created with 100GB of RAM, 14 vCPUs, EFI BIOS type, no SCSI controllers
- attached P4800X NVMe PCI device to that VM, booted
- installed VMware Tools 10.1.7.5541682, rebooted
Video
Aug 14 2017 Update
The subtle but big news above is about my first success ever in getting NVMe passthrough working so darn easily, with no need to resort to RDM mapping trickery. This is awesome, and seems to have arrived with July's mature release of ESXi 6.5 called Update 1.
There is so much left to test, including Intel SSD Data Center Tool, along with any newer than VMware VIB (driver) bundles I can find that include this DC P4800X SSD. It appears Version 1.2.1.15 Release Date 2017-04-14 just isn't working. I really want to know how well the P4800X handles multiple VMs concurrently, on one VMFS datastore. My current use of passthrough is an extremely limiting use-case, benefitting only one of my many VMs. Any passthrough prevents the fun stuff in VMware vSphere, such as vMotion, and hot add/removes of devices from the VM, and nearly instant snapshots that allow easy time-warp (roll-back) right after messing around with the VM itself.
BIOS Changes needed on Supermicro
These screenshots are from my SYS-5028D-TN4T Xeon D-1541 system with BIOS 1.2a. The problem is that you can't get back into the BIOS configuration again, not without temporarily removing the P4800X first. I've added this to my list of known-issues with BIOS 1.2a here.
Aug 15 2017 Update
If you're curious what the details of the P4800X look like from ESXi, I installed a fresh copy of ESXi 6.5 Update 1, and here's what is "seen" when using the commands I detail at How to find NVMe SSD firmware versions in a VMware ESXi 6.5 Server:
[root@xd-1541-5028d:~] esxcli nvme device get -A vmhba1 | egrep "Serial Number|Model Number|Firmware Revision"
Serial Number: FUKS70660064375AGN
Model Number: INTEL SSDPED1K375GA
Firmware Revision: E2010211
[root@xd-1541-5028d:~] esxcli nvme device get -A vmhba1
Controller Identify Info:
PCIVID: 0x8086
PCISSVID: 0x8086
Serial Number: FUKS70660064375AGN
Model Number: INTEL SSDPED1K375GA
Firmware Revision: E2010211
Recommended Arbitration Burst: 0
IEEE OUI Identifier: 5cd2e4
Controller Associated with an SR-IOV Virtual Function: false
Controller Associated with a PCI Function: true
NVM Subsystem May Contain Two or More Controllers: false
NVM Subsystem Contains Only One Controller: true
NVM Subsystem May Contain Two or More PCIe Ports: false
NVM Subsystem Contains Only One PCIe Port: true
Max Data Transfer Size: 5
Controller ID: 0
Version: 0.0
RTD3 Resume Latency: 0 us
RTD3 Entry Latency: 0 us
Optional Namespace Attribute Changed Event Support: false
Namespace Management and Attachment Support: false
Firmware Activate and Download Support: true
Format NVM Support: true
Security Send and Receive Support: false
Abort Command Limit: 3
Async Event Request Limit: 3
Firmware Activate Without Reset Support: false
Firmware Slot Number: 1
The First Slot Is Read-only: false
Command Effects Log Page Support: true
SMART/Health Information Log Page per Namespace Support: false
Error Log Page Entries: 63
Number of Power States Support: 0
Format of Admin Vendor Specific Commands Is Same: false
Format of Admin Vendor Specific Commands Is Vendor Specific: true
Autonomous Power State Transitions Support: false
Warning Composite Temperature Threshold: 0
Critical Composite Temperature Threshold: 0
Max Time for Firmware Activation: 0 * 100ms
Host Memory Buffer Preferred Size: 0 * 4KB
Host Memory Buffer Min Size: 0 * 4KB
Total NVM Capacity: 0x0
Unallocated NVM Capacity: 0x0
Access Size: 0 * 512B
Total Size: 0 * 128KB
Authentication Method: 0
Number of RPMB Units: 0
Max Submission Queue Entry Size: 64 Bytes
Required Submission Queue Entry Size: 64 Bytes
Max Completion Queue Entry Size: 16 Bytes
Required Completion Queue Entry Size: 16 Bytes
Number of Namespaces: 1
Reservation Support: false
Save/Select Field in Set/Get Feature Support: false
Write Zeroes Command Support: false
Dataset Management Command Support: true
Write Uncorrectable Command Support: true
Compare Command Support: false
Fused Operation Support: false
Cryptographic Erase as Part of Secure Erase Support: true
Cryptographic Erase and User Data Erase to All Namespaces: false
Cryptographic Erase and User Data Erase to One Particular Namespace: true
Format Operation to All Namespaces: false
Format Opertaion to One Particular Namespace: true
Volatile Write Cache Is Present: false
Atomic Write Unit Normal: 0 Logical Blocks
Atomic Write Unit Power Fail: 0 Logical Blocks
Format of All NVM Vendor Specific Commands Is Same: false
Format of All NVM Vendor Specific Commands Is Vendor Specific: true
Atomic Compare and Write Unit: 0
SGL Length Able to Larger than Data Amount: false
SGL Length Shall Be Equal to Data Amount: true
Byte Aligned Contiguous Physical Buffer of Metadata Support: false
SGL Bit Bucket Descriptor Support: false
SGL for NVM Command Set Support: false
[root@xd-1541-5028d:~]Aug 20 2017 Update
CrystalDiskMark
I now have CrystalDiskMark 5.2.2 x64 results to share.
AS SSD Benchmark
I also have an attempt at getting full results for AS SSD Benchmark 1.9.586.35387 running to share.
New firmware
I've been provided with a new firmware to test out, hoping for better results when trying to VMFS format the P4800X, will keep you posted, right here in this article.
TinkerTry Shout Out during PC Perspective Podcast
Finally, I'm very happy to share this TinkerTry shout-out from Allyn Malventano at PC Perspective:
...I've talked with Paul over at TinkerTry a couple of times...and he managed to get his hands on Optane P4800X and I'm kind of jealous, cuz we don't have one, none of the review sites have one. However, Paul works for VMware, and you know VMware is kind of high on the priority list of where INtel is going to ship some of the first engineering samples of this product, for them to be able to code their tools and make the tools work properly with it...one of the big ideas with Optane is for it to be able to accelerate enterprise applications...a large one of which being ESX[i] Server for VMware...Paul got to have his, get his little grubby hands on one of these things, play around and install it on the system...he didn't do extensive testing because I think he's kind of leaving that for the reviewers which we already did a review of this, we just didn't have it in hand, cool to see what it looks like.
Aug 22 2017 Update
Ongoing issues with getting the firmware updated have inhibited progress on testing this as a VMFS and/or vSAN datastore.
Aug 23 2017 Update
I did get the nvme-cli installed and working:
sudo nvme list
using Ubuntu 17 on my Xeon D-1541 system. This also meant I was able to update my firmware to an unreleased test version, but I was unable to finish the required format procedure afterward. Unfortunately, no progress on visibility of this device under ESXi 6.5U1 to report.
Sep 01 2017 Update
Here's Intel's P4800X Datasheet.
FEATURE / SPECIFICATION
Capacity - 375GBForm Factor - Add-in-Card (AIC); Half-height, Low-profile
Interface - PCIe 3.0 x4, NVMe
Latency (typical) R/W - <10μs
Quality of Service (QoS): 99.999% 4KB Random, Queue Dept 1, Read/Write <60/100 μs
4KB Random, Queue Depth 16, R/W: <150/200 μsThroughput: 4K Random, Queue Depth 16, Read/Write: up to 550/500k IOPS
4KB Random, Queue Depth 16, Mixed 70/30 read/Write: up to 500k IOPSEndurance: 30 Drive Writes per day (JESD219 workload)
12.3 Petabytes Written (PBW)
Unlike Amazon, where the ship date seems to keep getting changed to the future, here's a site that says they have 105 of the P4800X in stock as of Sep 01 2017, with pricing and availability subject to change, of course.
Sep 18 2017 Update
Sep 21 2017 Update
Sep 24 2017 Update
Interested in learning about the Intel SSD Data Center Tool? Watch me download version 3.0.7 right here:
- Downloads for Intel® SSD Data Center Tool
downloadcenter.intel.com/product/87278/Intel-SSD-Data-Center-Tool
then install and use it, to do an nvme format.
Next, I replace the NVMe driver VIB that VMware ESXi 6.5U1 is using:
- Download VMware ESXi 6.5 intel-nvme 1.2.1.15 NVMe Driver for Intel(R) Solid-State Drive DC P3700, P3600, P3500, P3520, P4500, D3600, and P4800X NVM Express SSDs
my.vmware.com/de/web/vmware/details?downloadGroup=DT-ESXI65-INTEL-INTEL-NVME-12115&productId=614
Sep 25 2017 Update
I have ATTO Disk Benchmark results to share.
Oct 09 2017 Update
Yesterday, I was able to finally resolve my inability to VMFS format the P4800X. The full story is detailed at:
Nov 09 2017 Update
Intel announcements moved to shiny new article:
See also at TinkerTry
- Intel Xeon Scalable Processors (Purley) Unveiled - Intel and VMware touting roughly 2.5x the performance and 4x as many VMs versus the Xeon of 4 years ago
Jul 11 2017Today was a big announcement day for both Intel and VMware. But before I leave you wondering any longer, yes, those big claims assume Intel Optane / 3D XPoint Inside.
- Samsung 960 EVO vs 950 PRO M.2 NVMe SSD - FLIR thermal video of VMware vSphere 6.5 Windows 10 VM cloning
Jan 05 2017
- VMworld 2015 - The video interviews
Sep 09 2015Intel's Ken LeTourneau and I discuss SSDs and the vSAN qualification process.
See also
- Intel Optane SSD DC P4800X 375GB Review - Enterprise 3D XPoint
Apr 20 2017 by Allyn Malventano at PC PerspectiveXPoint. Optane. QuantX. We've been hearing these terms thrown around for two years now. A form of 3D stackable non-volatile memory that promised 10x the density of DRAM and 1000x the speed and endurance of NAND. These were bold statements, and over the following months, we would see them misunderstood and misconstrued by many in the industry...Fortunately cooler heads prevailed as Jim Handy and other industry analysts helped explain that a 1000x improvement at the die level does not translate to the same improvement at the device level...
- How 3D XPoint Phase-Change Memory Works
Jun 02 2017 by Allyn Malventano at PC Perspective
