World's First Close Look at Intel Optane SSD DC P4800X Series - PCIe NVMe arrives with 375GB of 3D XPoint!

Posted by Paul Braren on Aug 11 2017 (updated on Aug 22 2017) in
  • Storage
  • HomeLab
  • HomeServer
  • ESXi
  • Virtualization
  • Disclosure

    This SSD is on a temporary loan from Intel, and it appears to be an engineering sample, in the 375GB size that is planned for product launch. This loaner was made with no formal expectations or stipulations, just a brief chance to TinkerTry this datacenter technology in my own VMware vSphere home lab. Remember this site's tagline - TinkerTry IT @ home. Efficient virtualization, storage, backup, and more.


    Completely New Type of Storage

    Such an opportunity doesn't come along often. If you've followed TinkerTry for a while, you'll know that I'm always happy to get my hands on something new, such as the world's first Intel Xeon D-1541 and Xeon D-1567 SuperServers, and the very first SYS-E300-8D and SYS-E200-8D SuperServers. I was even lucky enough to get a look at the early days of PCIe NVMe heralded by the Intel 750 Series back in 2015. I'm very glad to have this chance to tinker with new tech, again!

    While many of us have been enjoying the benefits of NAND-based flash memory for nearly a decade now, the limits of what NAND can handle in terms of write speed and endurance have been increasingly obvious lately. See also:

    3D XPoint

    I've been writing about 3D XPoint, which Intel dubs Optane, since February of 2016. And now, I finally have it in hand. So tantalizing! My recent article title sums its capabilities up as succinctly as I could:

    Plans

    • I can't wait to make some rough measurements of power use in watts, at idle and under load, using with my Ubiquiti mPower Pro power strip. Not as fancy or precise as PC Perspective's measurement rig, but a good start.
    • I also plan check out temperatures using my FLIR ONE Thermal Imaging camera.
    • I want to have a look at the driver install under VMware and under Windows Server 2016.
    • I hope to look at general VMFS performance characteristics when used with VMware ESXi 6.5 Update 1.
    • I plan to update this article when I have those measurements.

    Enterprise versus Consumer

    Note that there is likely to be a more consumer focused version of this product coming soon, featuring both a lower price and an appropriate warranty policy. You may recall that Intel's DC S3700 Series (DC for DataCenter) was followed up by the Intel 750 Series aimed at enthusiasts/consumers.

    Warning

    It appears some BIOS updates and/or UEFI settings changes may be needed to (hopefully) allow this Optane drive to become "visible" to VMware ESXi in my Xeon D server. I've loaded up the Intel NVMe driver labeled for use with the P4800X, but I'm only able to check the firmware, not VMFS format the drive.

    Note, Supermicro makes no claims this particular drive has been tested for compatibility at a hardware or software level. I can't make any promises, but I also don't give up easily. Stay tuned to TinkerTry for more updates as they happen.

    Meanwhile, you'll find a short unboxing video, pictures, and lot more information about 3D XPoint in the set of articles listed below. If time permits, I hope to also try a nested environment of ESXi 6.5 Update 1 running vSAN 6.6.1, we'll see.

    Shop

    B071VRNXVM
    Available for pre-order on Amazon.

    If you're in possession of a new server that has been fully tested for P4800X compatibility, such as the new Intel Xeon Scalable Processor family of systems, you can be assured of compatibility, and can now pre-order the P4800X on Amazon, with ship dates currently showing August 12-17. I'm a bit skeptical though, as that listing also says "Date first available at Amazon.com" as May 19 2017. I am not seeing other major sites taking pre-orders just yet.

    Photos

    Intel-Optane-DC-P4800X-Series-SSD-in-oem-box-view-by-TinkerTry-Aug-06-2017.JPG
    Intel-Optane-DC-P4800X-Series-SSD-in-close-up-top-view-by-TinkerTry-Aug-06-2017.JPG
    Intel-Optane-DC-P4800X-Series-SSD-top-view-by-TinkerTry-Aug-06-2017.JPG
    Intel-Optane-DC-P4800X-Series-SSD-back-view-by-TinkerTry-Aug-06-2017.JPG
    Intel-Optane-DC-P4800X-Series-SSD-in-close-up-view1-by-TinkerTry-Aug-06-2017.JPG
    Intel-Optane-DC-P4800X-Series-SSD-in-close-up-view2-by-TinkerTry-Aug-06-2017.JPG
    Intel-Optane-DC-P4800X-Series-SSD-LED-view-by-TinkerTry-Aug-06-2017.JPG
    Intel-Optane-Inside-TinkerTry-Supermicro-SuperServer-Bundle-2-Aug-06-2017.JPG

    Video (4K)

    World's First Close Look at Intel Optane SSD DC P4800X Series.

    Aug 11 2017 Evening Update

    I'm beginning to make some progress by skipping VMware testing for now, and cutting over to Windows Server 2016. BIOS is in UEFI mode, at these Recommended Settings, as mentioned in my Nov 05 2015 article about booting from NVMe.

    Windows-Server-2016-with-Desktop-Experience-being-installed-on-Intel-Optane-DC-P4800X-TinkerTry-Aug-11-2017
    Windows-Server-2016-with-Desktop-Experience-default-install-first-boot-on-Intel-Optane-DC-P4800X-TinkerTry-Aug-11-2017

    Aug 12 2017 Update

    I managed to get past the BIOS configuration issues that caused POST to hang on A9, will document details later. This allowed me to do a quick test of the P4800X under Windows Server 2016, with more testing to follow.


    Aug 13 2017 Update

    Windows_Server_2016_with_Desktop_Experience_option_iKVM_screenshot_by_TinkerTry_Aug_11_2017

    Things are going well. I now have some informal ATTO Disk Benchmarks to share.

    All these informative windows were brought up after the initial ATTO Disk Benchmark, subsequent tests showed consistently impressive results, whether the network adapters were disabled or not. I had to wait for all Windows Updates to finish, and made sure CPU was at or below 1% utilization before each run. More tools and tests results coming soon.

    Below are some informal looks at performance, with nowhere near the complexity and stringent documentation required of proper, official benchmarks, as detailed by Intel here, for example. These are consumer tools run on an enterprise drive, with things like HCIBench and Iometer really more appropriate for such a product, which are a little more error prone and time consuming.

    Tests with shipping product may differ from what is seen here.

    Test Hardware and Software Configuration

    • Supermicro SuperServer SYS-5028D-TN4T Bundle 2 featuring Xeon D-1541
      Bundle 2 as configured here, with 128GB of RAM and 8 CPU cores
    • BIOS 1.2a and IPM 3.58, downloads here
    • BIOS configured per Recommended Settings here, UEFI mode
    • Windows Server 2016 Standard Desktop Experience, download per instructions here, this filename: 14393.0.160715-1616.RS1_RELEASE_SERVER_EVAL_X64FRE_EN-US.ISO
    • Intel Optane SSD DC P4800X Series - PCIe NVMe 375GB, with the firmware that was already loaded on this engineering sample (no firmware update tools are available that I could find)
    • default GPT/NFTS filesystem used

    Test A - Windows Server 2016 natively installed and booted

    Boot from NVMe configured, I wrote more about this back in November of 2015, when it was less common, and a bit harder to do.

    Test B - Windows Server 2016 natively installed, but booted as VM

    Yes, I used VT-d passthrough of the same Windows Server 2016 on P4800X SSD for this test, worked beautifully, here's the steps:

    • VMware ESXi 6.5 Update 1 installed on 32GB USB, downloaded from my links here
    • configured hypervisor to passthrough the P4800X NVMe PCI device and rebooted
    • VM created with 100GB of RAM, 14 vCPUs, EFI BIOS type, no SCSI controllers
    • attached P4800X NVMe PCI device to that VM, booted
    • installed VMware Tools 10.1.7.5541682, rebooted
    ATTO_Disk_Benchmark_Optane_P4800X_Windows_Server_2016_initial_tests_by_TinkerTry_Aug-12-2017
    Test A - Windows Server 2016 installed on Optane P4800X. BIOS set to UEFI type.
    ATTO_Disk_Benchmark_Optane_P4800X_Windows_Server_2016_drive_VT-d_NVMe_passthrough_to_a_VMware_ESXi_65U1_VM_by_TinkerTry_Aug-13-2017
    Test B - Windows Server 2016 installed on Optane P4800X, passed through to a VM running under ESXi 6.5 Update 1.

    Video

    Xeon D-1541 reboot test featuring Windows Server 2016 on Intel Optane SSD DC P4800X Series NVMe SSD

    Aug 14 2017 Update

    The subtle but big news above is about my first success ever in getting NVMe passthrough working so darn easily, with no need to resort to RDM mapping trickery. This is awesome, and seems to have arrived with July's mature release of ESXi 6.5 called Update 1.

    There is so much left to test, including Intel SSD Data Center Tool, along with any newer than VMware VIB (driver) bundles I can find that include this DC P4800X SSD. It appears Version 1.2.1.15 Release Date 2017-04-14 just isn't working. I really want to know how well the P4800X handles multiple VMs concurrently, on one VMFS datastore. My current use of passthrough is an extremely limiting use-case, benefitting only one of my many VMs. Any passthrough prevents the fun stuff in VMware vSphere, such as vMotion, and hot add/removes of devices from the VM, and nearly instant snapshots that allow easy time-warp (roll-back) right after messing around with the VM itself.

    BIOS Changes needed on Supermicro

    These screenshots are from my SYS-5028D-TN4T Xeon D-1541 system with BIOS 1.2a. The problem is that you can't get back into the BIOS configuration again, not without temporarily removing the P4800X first. I've added this to my list of known-issues with BIOS 1.2a here.

    Press_Del-by-TinkerTry
    When powering on the sytem, get ready to press "Del" at the right moment.
    On_Main_Tab_Right_Arrow_to_Advanced_Tab-by-TinkerTry
    On the Main tab, right-arrow to the Advanced tab.
    On_Advanced_Tab_Down_Arrow_to_PCIe_Configuration-by-TinkerTry
    On Advanced, down-arrow to PCIe Configuration
    On_Advanced_Tab_Change_SLOT7_PCI-E_from_Legacy-by-TinkerTry
    Change SLOT7 PCI-E from Legacy to EFI.
    On_Advanced_Tab_Change_SLOT7_PCI-E_to_EFI-by-TinkerTry
    Change SLOT7 PCI-E from Legacy to EFI.
    Press_F4_to_Save_and_Exit-by-TinkerTry
    Press "F4" to Save Changes and Exit, which kicks off a reboot, your P4800X should now be available as bootable storage for whatever modern OS you choose.

    Aug 15 2017 Update

    If you're curious what the details of the P4800X look like from ESXi, I installed a fresh copy of ESXi 6.5 Update 1, and here's what is "seen" when using the commands I detail at How to find NVMe SSD firmware versions in a VMware ESXi 6.5 Server:

    [root@xd-1541-5028d:~] esxcli nvme device get -A vmhba1 | egrep "Serial Number|Model Number|Firmware Revision"
       Serial Number: FUKS70660064375AGN
       Model Number: INTEL SSDPED1K375GA
       Firmware Revision: E2010211
    [root@xd-1541-5028d:~] esxcli nvme device get -A vmhba1
    Controller Identify Info:
       PCIVID: 0x8086
       PCISSVID: 0x8086
       Serial Number: FUKS70660064375AGN
       Model Number: INTEL SSDPED1K375GA
       Firmware Revision: E2010211
       Recommended Arbitration Burst: 0
       IEEE OUI Identifier: 5cd2e4
       Controller Associated with an SR-IOV Virtual Function: false
       Controller Associated with a PCI Function: true
       NVM Subsystem May Contain Two or More Controllers: false
       NVM Subsystem Contains Only One Controller: true
       NVM Subsystem May Contain Two or More PCIe Ports: false
       NVM Subsystem Contains Only One PCIe Port: true
       Max Data Transfer Size: 5
       Controller ID: 0
       Version: 0.0
       RTD3 Resume Latency: 0 us
       RTD3 Entry Latency: 0 us
       Optional Namespace Attribute Changed Event Support: false
       Namespace Management and Attachment Support: false
       Firmware Activate and Download Support: true
       Format NVM Support: true
       Security Send and Receive Support: false
       Abort Command Limit: 3
       Async Event Request Limit: 3
       Firmware Activate Without Reset Support: false
       Firmware Slot Number: 1
       The First Slot Is Read-only: false
       Command Effects Log Page Support: true
       SMART/Health Information Log Page per Namespace Support: false
       Error Log Page Entries: 63
       Number of Power States Support: 0
       Format of Admin Vendor Specific Commands Is Same: false
       Format of Admin Vendor Specific Commands Is Vendor Specific: true
       Autonomous Power State Transitions Support: false
       Warning Composite Temperature Threshold: 0
       Critical Composite Temperature Threshold: 0
       Max Time for Firmware Activation: 0 * 100ms
       Host Memory Buffer Preferred Size: 0 * 4KB
       Host Memory Buffer Min Size: 0 * 4KB
       Total NVM Capacity: 0x0
       Unallocated NVM Capacity: 0x0
       Access Size: 0 * 512B
       Total Size: 0 * 128KB
       Authentication Method: 0
       Number of RPMB Units: 0
       Max Submission Queue Entry Size: 64 Bytes
       Required Submission Queue Entry Size: 64 Bytes
       Max Completion Queue Entry Size: 16 Bytes
       Required Completion Queue Entry Size: 16 Bytes
       Number of Namespaces: 1
       Reservation Support: false
       Save/Select Field in Set/Get Feature Support: false
       Write Zeroes Command Support: false
       Dataset Management Command Support: true
       Write Uncorrectable Command Support: true
       Compare Command Support: false
       Fused Operation Support: false
       Cryptographic Erase as Part of Secure Erase Support: true
       Cryptographic Erase and User Data Erase to All Namespaces: false
       Cryptographic Erase and User Data Erase to One Particular Namespace: true
       Format Operation to All Namespaces: false
       Format Opertaion to One Particular Namespace: true
       Volatile Write Cache Is Present: false
       Atomic Write Unit Normal: 0 Logical Blocks
       Atomic Write Unit Power Fail: 0 Logical Blocks
       Format of All NVM Vendor Specific Commands Is Same: false
       Format of All NVM Vendor Specific Commands Is Vendor Specific: true
       Atomic Compare and Write Unit: 0
       SGL Length Able to Larger than Data Amount: false
       SGL Length Shall Be Equal to Data Amount: true
       Byte Aligned Contiguous Physical Buffer of Metadata Support: false
       SGL Bit Bucket Descriptor Support: false
       SGL for NVM Command Set Support: false
    [root@xd-1541-5028d:~]

    Aug 20 2017 Update

    CrystalDiskMark

    I now have CrystalDiskMark 5.2.2 x64 results to share.

    CrystalDiskMark_5_2_2_x64_Optane_P4800X_results_on_Xeon_D_1541_TinkerTry-Aug-13-2017
    CrystalDiskMark 5.2.2 x64 results for the P4800X, when using Windows Server 2016 Standard Desktop Experience installed on the same Xeon D-1541 test system.

    AS SSD Benchmark

    I also have an attempt at getting full results for AS SSD Benchmark 1.9.586.35387 running to share.

    AS_SSD_Benchmark_1.9.586.35387_test_issue_TinkerTry_Aug-13-2017
    Issue with getting AS SSD Benchmark 1.9.586.35387 to complete its tests on Windows Server 2016, same issue on subsequent tests too.

    New firmware

    I've been provided with a new firmware to test out, hoping for better results when trying to VMFS format the P4800X, will keep you posted, right here in this article.

    TinkerTry Shout Out during PC Perspective Podcast

    Finally, I'm very happy to share this TinkerTry shout-out from Allyn Malventano at PC Perspective:

    IsL-gVxk-notes
    Click the image to hear the audio, with playback controls.

    ...I've talked with Paul over at TinkerTry a couple of times...and he managed to get his hands on Optane P4800X and I'm kind of jealous, cuz we don't have one, none of the review sites have one. However, Paul works for VMware, and you know VMware is kind of high on the priority list of where INtel is going to ship some of the first engineering samples of this product, for them to be able to code their tools and make the tools work properly with it...one of the big ideas with Optane is for it to be able to accelerate enterprise applications...a large one of which being ESX[i] Server for VMware...Paul got to have his, get his little grubby hands on one of these things, play around and install it on the system...he didn't do extensive testing because I think he's kind of leaving that for the reviewers which we already did a review of this, we just didn't have it in hand, cool to see what it looks like.

    TinkerTry-Gets-Real-Look-Intel-Optane-SSD-DC-P4800X-cropped
    Queued to the right spot, "PC Perspective Podcast #463 - 08/17/17"

    Aug 22 2017 Update

    Ongoing issues with getting the firmware updated have inhibited progress on testing this as a VMFS and/or vSAN datastore.


    See also at TinkerTry

    check-nvme-ssd-firmware

     
     
     
     

    intel-xeon-scalable-processors-arrive-touting-2-point-5-x-performance-over-xeon-of-4-years-ago

     

    optane_usages4-700x345

     
     
     

    first-look-samsung-960-evo-m2-nvme-temps

     
     

    2017-08-11_1-42-25

    See also

    PC-Perspective-logo-small
    • Intel Optane SSD DC P4800X 375GB Review - Enterprise 3D XPoint
      Apr 20 2017 by Allyn Malventano at PC Perspective

      XPoint. Optane. QuantX. We've been hearing these terms thrown around for two years now. A form of 3D stackable non-volatile memory that promised 10x the density of DRAM and 1000x the speed and endurance of NAND. These were bold statements, and over the following months, we would see them misunderstood and misconstrued by many in the industry...Fortunately cooler heads prevailed as Jim Handy and other industry analysts helped explain that a 1000x improvement at the die level does not translate to the same improvement at the device level...

     

    PC-Perspective-logo-small

     
     
     

    2017-08-11_1-33-31