How to use Intel isdct to change NVMe SSD variable sector size before using as VMFS or vSAN, note 4Kn (native) drives aren't supported by VMware yet

Posted by Paul Braren on Oct 8 2017 (updated on Oct 15 2017) in

This article underwent significant changes since first published, as it was quickly found to contain conclusions that were incorrect. It was based on just one very early engineering sample I obtained directly from Intel that turned out not to be a reflection of the now generally available P4800X ownership experience. All inadvertently misleading information has been removed. Intel Optane P4800X drives normally ship already low-level-formatted at 512 Bytes Sector Size, ready for sysadmins to OS format as VMFS, vSAN, NTFS, ext3, etc.

If you happen to have an Intel Optane SSD DC P4800X Series PCIe NVMe SSD, currently featuring a capacity of 375 GBs of 3D XPoint goodness, then lucky you! I happen to have an early loaner engineering sample that I've been detailing all my experiences with here:

World's First Close Look at Intel Optane SSD DC P4800X Series - PCIe NVMe arrives with 375GB of 3D XPoint!
Sep 25 2017

This article is applicable to all of Intel's Enterprise SSDs that feature Variable Sector Size, and I've outlined the procedure below, in the step-by-step instructions.

This article isn't about benchmarking and reasons for changing sector sizes, it's about how to change Intel NVMe SSD sector sizes. While my testing was done on Intel Xeon D, this device is really intended for enterprise class servers like the Intel Xeon Scalable Processors (Purley). It's likely most home lab enthusiasts will wait for Intel to release a consumer variant of this 3D XPoint drive, at a lower cost per gigabyte. This article will still apply, since any Intel NVMe SSD features variable sector sizes.

Intel-Optane-DC-P4800X-Series-SSD-top-view-by-TinkerTry-Aug-06-2017.JPG — Note the label on my loaner engineering sample, it shows "FW: TBD." I upgraded to E2010320.

When first tinkering with the P4800, I found is that I could install Windows 10 or Windows Server 2016 on the drive directly and boot right up from it just fine, so NTFS was easy, as was boot-from-NVMe, once I had moved to BIOS 1.2b on my SuperServer. I could even do a little magic trick and boot this same Windows instance inside of a VM with minimal work, see details at:

How to configure VMware ESXi 6.5.x for VMDirectPath I/O pass-through of any NVMe SSD, such as Windows Server 2016 installed directly on an Intel Optane P4800X booted in an EFI VM
Aug 24 2017

VMware VCG / HCL

VMware Compatibility Guide, aka, Hardware Compatibility List.
For reference, you can see all drivers that are recommended for the newest E2010320 firmware over at the VMware Compatibility Guide, aka VMware HCL here. You'll see both the PCIe and U.2 versions listed (which are functionally identical devices).

When using 4096 KB, Note how ATTO behaves

I found it interesting that ATTO Disk Benchmark coughed up an error when running on Windows installed on the P4800X:

ATTO-Disk-Benchmark-Bench32-Starting-transfer-size-is-less-than-drive's-sector-size-It-has-been-adjusted — ATTO Disk Benchmark - Bench32 - Starting transfer size is less than drive's sector size. It has been adjusted.

I had never seen this error before, in nearly 2 decades of using this portable executable. This mysterious error stuck in the back of my mind for weeks. I don't like mysteries.

In a fun little twist, I happened to bump into a guy from ATTO very recently, when I was in Buffalo NY for two VMUG presentations on vSAN. That's near where ATTO is headquartered. He had heard me go off the rails for a little bit as I rationalized my need for NVMe speeds on all my servers and my laptop, along with a need to interconnect them all with 10GbE. Yes, we had lots to talk about! I then shared my unfolding story with him, letting him know I felt I was getting close to resolution. That's because...

Tips

...an esteemed VMware colleague had just tipped me off that the contents of these articles, explaining why and how to changing the P4800X sector size at will:

FAQ: Support statement for 512e and 4K Native drives for VMware vSphere and vSAN (2091600)
kb.vmware.com/kb/2091600

Do current GA versions of vSphere and VSAN support 4K Native drives?
No. 4K Native drives are not supported in current GA releases of vSphere and vSAN.
Change Variable Sector Size on NVM Express Drives
intel.com/content/www/us/en/support/memory-and-storage/data-center-ssds/000016238.html

Last Reviewed: 02-Oct-2017
Article ID: 000016238
Some Data Center NVM Express (NMVe) SSDs use Variable Sector Size technology as an alternative to Physical Sector Size, which is used by SATA SSDs.

The NVMe drives support 512, 520, 528, 4096, 4104, 4160, and 4224 bytes Variable Sector Size. The default value is 512 bytes.

Note - Changing the Variable Sector Size requires you to use the NVMe format command. The command erases all data on the selected device.

I theorized that this P4800X would also work fine if I simply reverted back to what the factory default is, 512 bytes, and side-step that ATTO error. Here's exactly how I did it.

Step-by-step Sector Size Change from 4096 KB to 512 bytes

Warning:
These instructions are followed at your own risk!

This low-level hardware reformatting will destroy all data that might be on your P4800X drive.
Back up your data first, then proceed with caution, using a UPS to protect from power failure during the ~8 minute reformatting, to reduce the risk of bricking your expensive device.

Install Windows 10, Windows Server 2016, or Linux on a PC with an available PCI 3.0 slot, using any SSD or HDD that is not the Optane P4800X you're about to reformat.
Download and install the Intel® SSD Data Center Tool, available for either Windows or Linux. I used Windows 10 installed on a SATA SSD, with the P4800X in a PCIe slot on my Supermicro SuperServe Bundle 2 system running BIOS 1.2b and IPMI 3.58.
Issue the following command to determine what sector size your P4800X is at:
```
isdct show -a -intelssd
```
You'll see mine was at 4096 KB, so I needed to change this to 512 bytes.
Issue the following command to make that change
```
isdct start –intelssd 0 -NVMeFormat LBAformat=0 SecureEraseSetting=0 ProtectionInformation=0 MetaDataSettings=0
```
If you get an error about existing partitions, you'll need to nuke them before you're allowed to proceed. Using diskpart to get this done is pretty straight-forward, explained by Aomei here.
Wait about 8 minutes for the formatting to complete, you'll get confirmation it's done,
Status : NVMeFormat successful.
Reboot the system into ESXi 6.x, and you can now add your P4800X as a vSAN drive, or as a VMFS datastore, where the Add Storage wizard will offer to format your P4800X for you. Choose VMFS 6 instead of 5, to learn why, see also KB 2091600:

VMFS 6.0 is required for 512e drive support starting with vSphere 6.5.

along with A comparison of VMFS5 & VMFS6 in vSphere 6.5 and Migrating VMFS 5 datastore to VMFS 6 datastore - KB2147824.

That's it, you're done!
If for some reason you want to go back to 4096 KB sector size NVMe formatting, here's how.

Step-by-step Sector Size Change from 512 bytes to 4096 KB

Issue the following command to change from 512 Bytes to 4096 KB.
This differs from above instructions only in the LBAformat= parameter.
```
isdct start –intelssd 0 -NVMeFormat LBAformat=3 SecureEraseSetting=0 ProtectionInformation=0 MetaDataSettings=0
```