World's First Close Look at Intel Optane SSD DC P4800X Series - PCIe NVMe arrives with 375GB of 3D XPoint!
This SSD is on a temporary loan from Intel, and it appears to be an engineering sample, in the 375GB size that is planned for product launch. This loaner was made with no formal expectations or stipulations, just a brief chance to TinkerTry this datacenter technology in my own VMware vSphere home lab. Remember this site's tagline - TinkerTry IT @ home. Efficient virtualization, storage, backup, and more.
Such an opportunity doesn't come along often. If you've followed TinkerTry for a while, you'll know that I'm always happy to get my hands on something new, such as the world's first Intel Xeon D-1541 and Xeon D-1567 SuperServers, and the very first SYS-E300-8D and SYS-E200-8D SuperServers. I was even lucky enough to get a look at the early days of PCIe NVMe heralded by the Intel 750 Series back in 2015. I'm very glad to have this chance to tinker with new tech, again!
While many of us have been enjoying the benefits of NAND-based flash memory for nearly a decade now, the limits of what NAND can handle in terms of write speed and endurance have been increasingly obvious lately. See also:
- Paul Braren and Allyn Malventano discuss NAND SSD's rough start, M.2 NVMe, and how 3D XPoint will get even faster
I've been writing about 3D XPoint, which Intel dubs Optane, since February of 2016. And now, I finally have it in hand. So tantalizing! My recent article title sums its capabilities up as succinctly as I could:
- Intel Optane SSD DC P4800X Series can be ultra fast vSAN cache, storage, or RHEL/SLES 8x memory extender using Intel Memory Drive Technology
- I can't wait to make some rough measurements of power use in watts, at idle and under load, using with my Ubiquiti mPower Pro power strip. Not as fancy or precise as PC Perspective's measurement rig, but a good start.
- I also plan check out temperatures using my FLIR ONE Thermal Imaging camera.
- I want to have a look at the driver install under VMware and under Windows Server 2016.
- I hope to look at general VMFS performance characteristics when used with VMware ESXi 6.5 Update 1.
- I plan to update this article when I have those measurements.
Note that there is likely to be a more consumer focused version of this product coming soon, featuring both a lower price and an appropriate warranty policy. You may recall that Intel's DC S3700 Series (DC for DataCenter) was followed up by the Intel 750 Series aimed at enthusiasts/consumers.
It appears some BIOS updates and/or UEFI settings changes may be needed to (hopefully) allow this Optane drive to become "visible" to VMware ESXi in my Xeon D server. I've loaded up the Intel NVMe driver labeled for use with the P4800X, but I'm only able to check the firmware, not VMFS format the drive.
Note, Supermicro makes no claims this particular drive has been tested for compatibility at a hardware or software level. I can't make any promises, but I also don't give up easily. Stay tuned to TinkerTry for more updates as they happen.
Meanwhile, you'll find a short unboxing video, pictures, and lot more information about 3D XPoint in the set of articles listed below. If time permits, I hope to also try a nested environment of ESXi 6.5 Update 1 running vSAN 6.6.1, we'll see.
If you're in possession of a new server that has been fully tested for P4800X compatibility, such as the new Intel Xeon Scalable Processor family of systems, you can be assured of compatibility, and can now pre-order the P4800X on Amazon, with ship dates currently showing August 12-17. I'm a bit skeptical though, as that listing also says "Date first available at Amazon.com" as May 19 2017. I am not seeing other major sites taking pre-orders just yet.
Sep 01 2017 Update - It seems mouser.com has them in stock, see details below.
I'm beginning to make some progress by skipping VMware testing for now, and cutting over to Windows Server 2016. BIOS is in UEFI mode, at these Recommended Settings, as mentioned in my Nov 05 2015 article about booting from NVMe.
I managed to get past the BIOS configuration issues that caused POST to hang on A9, will document details later. This allowed me to do a quick test of the P4800X under Windows Server 2016, with more testing to follow.
Things are going well. I now have some informal ATTO Disk Benchmarks to share.
All these informative windows were brought up after the initial ATTO Disk Benchmark, subsequent tests showed consistently impressive results, whether the network adapters were disabled or not. I had to wait for all Windows Updates to finish, and made sure CPU was at or below 1% utilization before each run. More tools and tests results coming soon.
Below are some informal looks at performance, with nowhere near the complexity and stringent documentation required of proper, official benchmarks, as detailed by Intel here, for example. These are consumer tools run on an enterprise drive, with things like HCIBench and Iometer really more appropriate for such a product, which are a little more error prone and time consuming.
Tests with shipping product may differ from what is seen here.
- Supermicro SuperServer SYS-5028D-TN4T Bundle 2 featuring Xeon D-1541
Bundle 2 as configured here, with 128GB of RAM and 8 CPU cores
- BIOS 1.2a and IPM 3.58, downloads here
- BIOS configured per Recommended Settings here, UEFI mode
- Windows Server 2016 Standard Desktop Experience, download per instructions here, this filename:
- Intel Optane SSD DC P4800X Series - PCIe NVMe 375GB, with the firmware that was already loaded on this engineering sample (no firmware update tools are available that I could find)
- default GPT/NFTS filesystem used
Test A - Windows Server 2016 natively installed and booted
Boot from NVMe configured, I wrote more about this back in November of 2015, when it was less common, and a bit harder to do.
Test B - Windows Server 2016 natively installed, but booted as VM
Yes, I used VT-d passthrough of the same Windows Server 2016 on P4800X SSD for this test, worked beautifully, here's the steps:
- VMware ESXi 6.5 Update 1 installed on 32GB USB, downloaded from my links here
- configured hypervisor to passthrough the P4800X NVMe PCI device and rebooted
- VM created with 100GB of RAM, 14 vCPUs, EFI BIOS type, no SCSI controllers
- attached P4800X NVMe PCI device to that VM, booted
- installed VMware Tools 10.1.7.5541682, rebooted
The subtle but big news above is about my first success ever in getting NVMe passthrough working so darn easily, with no need to resort to RDM mapping trickery. This is awesome, and seems to have arrived with July's mature release of ESXi 6.5 called Update 1.
There is so much left to test, including Intel SSD Data Center Tool, along with any newer than VMware VIB (driver) bundles I can find that include this DC P4800X SSD. It appears Version 18.104.22.168 Release Date 2017-04-14 just isn't working. I really want to know how well the P4800X handles multiple VMs concurrently, on one VMFS datastore. My current use of passthrough is an extremely limiting use-case, benefitting only one of my many VMs. Any passthrough prevents the fun stuff in VMware vSphere, such as vMotion, and hot add/removes of devices from the VM, and nearly instant snapshots that allow easy time-warp (roll-back) right after messing around with the VM itself.
BIOS Changes needed on Supermicro
These screenshots are from my SYS-5028D-TN4T Xeon D-1541 system with BIOS 1.2a. The problem is that you can't get back into the BIOS configuration again, not without temporarily removing the P4800X first. I've added this to my list of known-issues with BIOS 1.2a here.
If you're curious what the details of the P4800X look like from ESXi, I installed a fresh copy of ESXi 6.5 Update 1, and here's what is "seen" when using the commands I detail at How to find NVMe SSD firmware versions in a VMware ESXi 6.5 Server:
[root@xd-1541-5028d:~] esxcli nvme device get -A vmhba1 | egrep "Serial Number|Model Number|Firmware Revision" Serial Number: FUKS70660064375AGN Model Number: INTEL SSDPED1K375GA Firmware Revision: E2010211 [root@xd-1541-5028d:~] esxcli nvme device get -A vmhba1 Controller Identify Info: PCIVID: 0x8086 PCISSVID: 0x8086 Serial Number: FUKS70660064375AGN Model Number: INTEL SSDPED1K375GA Firmware Revision: E2010211 Recommended Arbitration Burst: 0 IEEE OUI Identifier: 5cd2e4 Controller Associated with an SR-IOV Virtual Function: false Controller Associated with a PCI Function: true NVM Subsystem May Contain Two or More Controllers: false NVM Subsystem Contains Only One Controller: true NVM Subsystem May Contain Two or More PCIe Ports: false NVM Subsystem Contains Only One PCIe Port: true Max Data Transfer Size: 5 Controller ID: 0 Version: 0.0 RTD3 Resume Latency: 0 us RTD3 Entry Latency: 0 us Optional Namespace Attribute Changed Event Support: false Namespace Management and Attachment Support: false Firmware Activate and Download Support: true Format NVM Support: true Security Send and Receive Support: false Abort Command Limit: 3 Async Event Request Limit: 3 Firmware Activate Without Reset Support: false Firmware Slot Number: 1 The First Slot Is Read-only: false Command Effects Log Page Support: true SMART/Health Information Log Page per Namespace Support: false Error Log Page Entries: 63 Number of Power States Support: 0 Format of Admin Vendor Specific Commands Is Same: false Format of Admin Vendor Specific Commands Is Vendor Specific: true Autonomous Power State Transitions Support: false Warning Composite Temperature Threshold: 0 Critical Composite Temperature Threshold: 0 Max Time for Firmware Activation: 0 * 100ms Host Memory Buffer Preferred Size: 0 * 4KB Host Memory Buffer Min Size: 0 * 4KB Total NVM Capacity: 0x0 Unallocated NVM Capacity: 0x0 Access Size: 0 * 512B Total Size: 0 * 128KB Authentication Method: 0 Number of RPMB Units: 0 Max Submission Queue Entry Size: 64 Bytes Required Submission Queue Entry Size: 64 Bytes Max Completion Queue Entry Size: 16 Bytes Required Completion Queue Entry Size: 16 Bytes Number of Namespaces: 1 Reservation Support: false Save/Select Field in Set/Get Feature Support: false Write Zeroes Command Support: false Dataset Management Command Support: true Write Uncorrectable Command Support: true Compare Command Support: false Fused Operation Support: false Cryptographic Erase as Part of Secure Erase Support: true Cryptographic Erase and User Data Erase to All Namespaces: false Cryptographic Erase and User Data Erase to One Particular Namespace: true Format Operation to All Namespaces: false Format Opertaion to One Particular Namespace: true Volatile Write Cache Is Present: false Atomic Write Unit Normal: 0 Logical Blocks Atomic Write Unit Power Fail: 0 Logical Blocks Format of All NVM Vendor Specific Commands Is Same: false Format of All NVM Vendor Specific Commands Is Vendor Specific: true Atomic Compare and Write Unit: 0 SGL Length Able to Larger than Data Amount: false SGL Length Shall Be Equal to Data Amount: true Byte Aligned Contiguous Physical Buffer of Metadata Support: false SGL Bit Bucket Descriptor Support: false SGL for NVM Command Set Support: false [root@xd-1541-5028d:~]
I now have CrystalDiskMark 5.2.2 x64 results to share.
AS SSD Benchmark
I also have an attempt at getting full results for AS SSD Benchmark 1.9.586.35387 running to share.
I've been provided with a new firmware to test out, hoping for better results when trying to VMFS format the P4800X, will keep you posted, right here in this article.
TinkerTry Shout Out during PC Perspective Podcast
Finally, I'm very happy to share this TinkerTry shout-out from Allyn Malventano at PC Perspective:
...I've talked with Paul over at TinkerTry a couple of times...and he managed to get his hands on Optane P4800X and I'm kind of jealous, cuz we don't have one, none of the review sites have one. However, Paul works for VMware, and you know VMware is kind of high on the priority list of where INtel is going to ship some of the first engineering samples of this product, for them to be able to code their tools and make the tools work properly with it...one of the big ideas with Optane is for it to be able to accelerate enterprise applications...a large one of which being ESX[i] Server for VMware...Paul got to have his, get his little grubby hands on one of these things, play around and install it on the system...he didn't do extensive testing because I think he's kind of leaving that for the reviewers which we already did a review of this, we just didn't have it in hand, cool to see what it looks like.
Ongoing issues with getting the firmware updated have inhibited progress on testing this as a VMFS and/or vSAN datastore.
I did get the nvme-cli installed and working:
sudo nvme list
using Ubuntu 17 on my Xeon D-1541 system. This also meant I was able to update my firmware to an unreleased test version, but I was unable to finish the required format procedure afterward. Unfortunately, no progress on visibility of this device under ESXi 6.5U1 to report.
Here's Intel's P4800X Datasheet.
FEATURE / SPECIFICATION
Capacity - 375GB
Form Factor - Add-in-Card (AIC); Half-height, Low-profile
Interface - PCIe 3.0 x4, NVMe
Latency (typical) R/W - <10μs
Quality of Service (QoS): 99.999% 4KB Random, Queue Dept 1, Read/Write <60/100 μs
4KB Random, Queue Depth 16, R/W: <150/200 μs
Throughput: 4K Random, Queue Depth 16, Read/Write: up to 550/500k IOPS
4KB Random, Queue Depth 16, Mixed 70/30 read/Write: up to 500k IOPS
Endurance: 30 Drive Writes per day (JESD219 workload)
12.3 Petabytes Written (PBW)
Unlike Amazon, where the ship date seems to keep getting changed to the future, here's a site that says they have 105 of the P4800X in stock as of Sep 01 2017, with pricing and availability subject to change, of course.
Interested in learning about the Intel SSD Data Center Tool? Watch me download version 3.0.7 right here:
- Downloads for Intel® SSD Data Center Tool
then install and use it, to do an nvme format.
Next, I replace the NVMe driver VIB that VMware ESXi 6.5U1 is using:
- Download VMware ESXi 6.5 intel-nvme 22.214.171.124 NVMe Driver for Intel(R) Solid-State Drive DC P3700, P3600, P3500, P3520, P4500, D3600, and P4800X NVM Express SSDs
I have ATTO Disk Benchmark results to share.
Yesterday, I was able to finally resolve my inability to VMFS format the P4800X. The full story is detailed at:
- Change Intel NVMe SSDs with variable sector size to 512e to allow VMFS or vSAN, since 4Kn drives aren't supported on VMware ESXi 6.5/vSAN 6.6.1 yet
- Intel Xeon Scalable Processors (Purley) Unveiled - Intel and VMware touting roughly 2.5x the performance and 4x as many VMs versus the Xeon of 4 years ago
Jul 11 2017
Today was a big announcement day for both Intel and VMware. But before I leave you wondering any longer, yes, those big claims assume Intel Optane / 3D XPoint Inside.
- Intel Optane SSD DC P4800X Series can be ultra fast vSAN cache, storage, or RHEL/SLES 8x memory extender using Intel Memory Drive Technology
Mar 21 2017
- Samsung 960 EVO vs 950 PRO M.2 NVMe SSD - FLIR thermal video of VMware vSphere 6.5 Windows 10 VM cloning
Jan 05 2017
- VMworld 2015 - The video interviews
Sep 09 2015
Intel's Ken LeTourneau and I discuss SSDs and the vSAN qualification process.
- Intel Optane SSD DC P4800X 375GB Review - Enterprise 3D XPoint
Apr 20 2017 by Allyn Malventano at PC Perspective
XPoint. Optane. QuantX. We've been hearing these terms thrown around for two years now. A form of 3D stackable non-volatile memory that promised 10x the density of DRAM and 1000x the speed and endurance of NAND. These were bold statements, and over the following months, we would see them misunderstood and misconstrued by many in the industry...Fortunately cooler heads prevailed as Jim Handy and other industry analysts helped explain that a 1000x improvement at the die level does not translate to the same improvement at the device level...
- How 3D XPoint Phase-Change Memory Works
Jun 02 2017 by Allyn Malventano at PC Perspective