Intel 750 Series NVMe PCIe SSD supported by ESXi 6.x out-of-the-box, install Intel's VIB for full speed

Posted by Paul Braren on Sep 29 2015 (updated on Nov 25 2015) in
  • Storage
  • Virtualization
  • HomeServer
  • HomeLab
  • Windows
    • NVMeStack

    Source: Intel

    B00UHJJQAY
    Here's an inside view of the 750 Series, with the heat sink/airflow channel removed, as pictured on Amazon. Click to shop.

    If you use ESXi 6.0 Update 1, and you're lucky enough to possess an Intel 750 NVMe, you're still going to want to use the latest Intel NVMe driver install procedure (with video) below.

    First, a special thank you. This post, and the recent 750 First Look, were all made possible by an Intel 750 Series 400GB NVMe on temporary loan to me from none other than Trond Eirik Haavarstein @xenappblog! Luck would have it we both needed each other's help recently, and I couldn't be more glad to pitch in. More exciting test results coming soon, some may be featured here at TinkerTry, and at Trond's wonderful xenappblog.com:

    xenappblog

    Even though vSphere 6 includes NVMe support out-of-the-box, even after updating to the recent ESXi 6.0 Update 1, the speed of the 750 Series still suffers when using that bundled driver.

    Here's the improved 1.0e.1.1 driver that's currently available on both VMware's site and Intel's site, with no login required at the Intel site.

    Download 1.0e.1.1 from VMware (My VMware login required)

    Download 1.0e.1.1 from Intel (no login required)

    Here's the same file available on Intel's site, no login required:

    Sep 30 2015 update
    At the Intel download site, the way the driver is labeled as 1.2.0.1002 (the version of the other OSs) is misleading. Both VMware and Intel sites currently offer the same exact tiny (58 KB) zip file, I even did a fc (file compare). Also, the mismatched dates are simply because Intel published this same driver on their site later.

    Note the tiny asterisk Intel uses at their download site:

    Intel_750_driver_download_screenshot_highlighted_asterisk

    I hadn't spotted this asterisk right away, and assumed this was a 5.5 driver, given the 5.5.0 in the filename. A more careful read cleared this up. I've pasted that crucial text below, with the parts that were most relevant to my own tests and video) bolded.

    *For installation under VMware® ESXi, download the VMware® Read Me file found under Release Notes.
    This Windows* driver is required for the system to recognize an Intel® Solid-State Drive Data Center P3700, P3600 and P3500 Series or Intel SSD 750 Series drive.

    This VMware* driver is required for the system to recognize an Intel® Solid-State Drive Data Center P3700, P3600 and P3500 Series.

    This driver package contains:
    Version 1.0e.1.1 for VMware* ESXi 5.5, ESXi 5.5 U1, ESXi 5.5 U2, ESXi 6.0

    To find the latest firmware available for Intel Solid State Drives please check this link.
    NOTE: If you purchased your Intel SSD from an OEM, your firmware version may have a different naming. Please contact your local OEM representative for latest firmware revision.

    This download is valid for the product(s) listed below.
    Intel® SSD 750 Series
    Intel® SSD 750 Series (1.2TB, 1/2 Height PCIe 3.0, 20nm, MLC)
    Intel® SSD 750 Series (1.2TB, 2.5in PCIe 3.0, 20nm, MLC)
    Intel® SSD 750 Series (400GB, 1/2 Height PCIe 3.0, 20nm, MLC)
    Intel® SSD 750 Series (400GB, 2.5in PCIe 3.0, 20nm, MLC)
    Intel® SSD 750 Series (800GB, 1/2 Height PCIe 3.0 x4, 20nm, MLC)
    Intel® SSD 750 Series (800GB, 2.5in PCIe 3.0 x4, 20nm, MLC)

    Install procedure

    1. Download from one of the direct links above
    2. Extract VMW-ESX-5.5.0-intel-nvme-1.0e.1.1-2791094.zip
    3. From the root of the extracted folder, upload the intel-nvme-1.0e.1.1-1OEM.550.0.0.1391871.x86_64.vib to the ESXi 6.x host's /tmp directory using WinSCP
    4. Put the ESXi host into Maintenance Mode
    5. PuTTY to the ESXi host
    6. Paste Issue the following one liner command:
      esxcli software vib install -v /tmp/intel-nvme-1.0e.1.1-1OEM.550.0.0.1391871.x86_64.vib
    7. Reboot the ESXi host
      750-vib-install

    Install procedure video

    Here's exactly how to install the driver, and the leap in ATTO speeds seen afterward.

    Here's the results!

    VMware ESXi 6.0 Update 1 running on a Supermicro SuperServer 5028D-TN4T. Windows 10 VM with VMware tools, default VM install (2GB RAM, 2 vCPU), with a 32GB thick provisioned virtual drive that's on the Intel 750's VMFS datastore.

    Intel_750_VM_default_ESXi_60_driver_versus_Intel_driver
    Before, and After! [Windows 10 VM under ESXi 6.0U1.]
    Intel_750_ATTO_results_on_Windows_10_non_VM
    Here's the same drive running on the same SuperServer, but Windows 10 running on the bare-metal.

    Oct 22 2015 Update

    Interesting bit of information about the Microsoft NVMe driver, over at FUTUREMARK:

    Short answer: because NVMe drivers behave differently than SATA/AHCI drivers. PCMark 8 benchmarks all storage devices the same way and NVMe is not any different.
    Long answer: New NVMe storage devices run on a completely new software stack. Traditional SATA drivers are not used and instead there's a new driver in Windows that can be either implemented by the device manufacturer or it can be the standard NVMe driver provided by Microsoft. What is different from old software stack, is that there's a command called FUA (force unit access) that is now implemented in Microsoft's driver so that all write operations are enforced to be written not only on a cache but on the non-volatile media. This is different from how AHCI was implemented and will lead to reduced performance. The benefit of the implementation with Microsoft NVMe driver is that it keeps your data safe in case of power outage or system crash.
    The storage performance (and as a side result also PCMark 8 Storage score) may be improved by installing device drivers provided by the device manufacturer. For example, Intel provides drivers for Intel 750 SSD that improve the performance. Another option is to tweak storage device policies in Windows. Futuremark does not recommend modifying Windows cache policies on a production system since it can lead to data loss during a power failure or a system crash.
    PCMark 8 never optimises or modifies the PC being benchmarked. Testing is done on the system as it is in order to reflect the real application performance an end user would observe when using the PC. Applications do not detect the storage device and change their behaviour according to device and neither should a benchmark do.

    See also at TinkerTry

    See also

    TinkerTry - PCs, EVs, home tech, efficiency and more, including virtualization. My opinions here, not my employer's.

    Paul Braren, MCSE in 1998, VCP #2681 in 2005, vExpert 2014 to 2024, Veeam Vanguard Emeritus, 2015 to 2019.

    Support TinkerTry

    PayPal | Patreon | VMware Store

    Easily update VMware ESXi at TinkerTry.com/easy-update-to-latest-esxi

    After 6 successful years testing then shipping well over 1,000 Xeon D Bundles, Wiredzone had to stop selling them in mid-2021 due to cost, supply, and logistics challenges. The Xeon D-1700/2700 (Ice Lake D) was a minor refresh for 2023, with Xeon D-1800/2800 (Granite Rapids D) refresh slightly better in 2024, and hopefully Xeon 6 (Granite Rapids-D) much better in 2025 featuring PCIe Gen5, MCRDIMMs, and 100GbE networking, wow! I'm glad it's Pat Gelsinger at Intel's helm in these challenging times, and grateful to have had the honor of working at VMware when he was the CIO there. I'll leave it at that, given the whole Broadcom thing.