VMware ESXi 6.0 Windows 10 VM speed comparison between Intel 750 Series NVMe SSD and Samsung 950 PRO M.2 NVMe SSD

Posted by Paul Braren on Jan 3 2016 in
  • Storage Virtualization ESXi HomeServer HomeLab
    • 6 Comments
    NVMeStack

    Source: Intel

    What happens when you take two identical Xeon servers with PCIe 3.0 x 4

    B01639694M
    Available now on Amazon in 256GB and 512GB sizes, 1TB not expected until early 2016.

    ...and bless each of them with magical NVMe powers

    1) install an Intel SSD 750 Series 400 GB NVMe PCIe card in the one on the left
    (related articles here)

    2) install a Samsung 950 PRO M.2 NVMe M.2 SSD into the one on the right
    (related articles here)

    IMG_8440edited.JPG
    Multiple capacities available on Amazon.

    This motherboard and chipsets supports the full speeds NVMe is capable of, given the PCIe 3.0 x 4 lanes. Note that the latest Skylake Intel NUC Kit NUC6i3SYH has PCIe 3.0 x4, but caps out at 1600 MB/sec throughput for NVMe devices, as Intel's manual page 45 confirms.

    ...then top it all off with a fresh copy of VMware ESXi 6.0 Update 1?

    3) format those two brands of delicious NVMe as VMFS5 as they become an ESXi datastore, then build a nice default Windows 10 VM one one, then clone it to the other

    Here's what happens, a fun video...

    The video appears below, showing the two systems booting each booting a Windows 10 VM simultaneously.

    Interesting to note, with the Intel 750, you need to install a VIB (driver) for decent speeds, but the Samung 950 "just works" and exhibits good speeds using just ESXi 6.x's built in NVMe driver.

    that also features some ATTO Disk Benchmark action...

    For these tests (which were consistent across off camera runs), I went with the highest system fan speed to avoid any thermal throttling during even the most extended abuse, with the BIOS configured the same way I did my Windows 10 ATTO benchmarks.

    ATTO 750 Series 400GB versus Samsung 950 PRO 512GB
    Intel 750 at left, Samsung 950 at right. ATTO Disk Benchmark results seen in each Windows 10 VM, as seen in the video. Click twice to zoom in fully.

    ...and finishes up with a typical use case demonstration....

    This strong ending gives you a real sense of how glorious it is to be able to deploy a new Windows 10 VM from Template (a full clone) in a mere 18 seconds. Now that's what I'm talking about!

    Win 10 VM on an Intel 750 Series and a Samsung 950 PRO, NVMe SSDs fly with VMware ESXi!

    ...topped off with some Benchmark results.

    Samsung Specifications state:
    2,500 MB/sec reads and 1,500 MB/sec writes.
    According to ATTO, my highest numbers from 3 runs, under Windows 10 natively:
    2,598 MB/sec reads and 1,576 MB/sec writes.

    Now let's look at the numbers for ATTO run inside a VM:
    2,578 MB/sec reads and 1,568 MB/sec writes.

    TinkerTry-Tested-Samsung_950_PRO_ATTO_on_SYS-5028D-TN4T_Run1of3
    Windows 10 installed on SuperServer directly, with ATTO Disk Benchmark results. Click on the image, then left/right arrow to compare each of the ATTO results with one another.
    TinkerTry-Tested-Samsung_950_PRO_ATTO_on_SYS-5028D-TN4T_ESXi-VM-cropped
    Windows 10 VM installed on SuperServer, ATTO Disk Benchmark results. This screenshot is taken from the video, which wasn't tall enough a window to get an unobstructed view of the results.
    TinkerTry-Tested-Samsung_840_EVO_mSATA_1TB_ATTO_on_SYS-5028D-TN4T
    For comparison sake, here's a more typical SATA3-based Samsung 840 EVO mSATA 1TB SSD drive, which hits a pretty obvious cap as it nears the 6 Gbps max of the interface.

    Closing thoughts and observations.

    Despite the diminished speeds at the very important 4.0K size seen on the 4th row of the graphs, these results are still pretty darn good overall. It would appear that both the Samsung Windows 10 drivers and the VMware ESXi 6.0 NVMe drivers are fairly mature, even in this brave and wonderful new world of NVMe where the speeds that blow away any SATA3 device out there. Some of my many NVMe articles were published rather long ago. It's about time for the future of storage to arrive, and it was worth the wait and cost. Let's hope the cost delta between a SATA3 and NVMe SSDs continues to fall throughout 2016, as more and more folks find themselves with a little M.2 interface on their motherboard, or a M.2 to PCIe adapter.

    See also at TinkerTry

    See also

    All Comments on This Article (6)

    Paul Braren | TinkerTry.com

    Good points, sorry about that, still hope you found some value to this article.
    You may find this article helpful as well, with some ATTO results:
    https://tinkertry.com/first-look-intel-750-nvme-storage-speeds-on-superserver
    I was admittedly a bit rushed and no longer have the 750 (or this equipment), unfortunately.

    Guest

    "Samsung Specifications state:
    2,500 MB/sec reads and 1,500 MB/sec writes.
    According to ATTO, my highest numbers from 3 runs, under Windows 10 natively:
    2,598 MB/sec reads and 1,576 MB/sec writes.
    Now let's look at the numbers for ATTO run inside a VM:
    2,578 MB/sec reads and 1,568 MB/sec writes."

    Where's intel 750? What's the point to put just one product and not other?

    Paul Braren | TinkerTry.com

    Great question!

    See also some more practical measurements (I should have also done the same from the SATA3-hindered Samsung 850 PRO):
    How does copying over 200GB from one Samsung 950 PRO NVMe SSD to another in under 3 minutes sound? (42 copies of a 4.4GB Linux distro)
    https://TinkerTry.com/superserver-with-two-samsung-950-pro-file-copy-tests

    Copying about 50GB from a Samsung 950 PRO NVMe SSD back to itself is slower than copying to another 950 PRO over 10GbE
    https://TinkerTry.com/samsung-950-pro-local-superserver-file-copy-slower-than-over-10gbe

    I think Allyn states it best in this article:
    http://www.pcper.com/reviews/Storage/Samsung-950-PRO-256GB-and-512GB-M2-NVMe-PCIe-SSD-Review/Latency-Distribution-and-Lat

    "October 22, 2015 | 01:48 PM - Posted by Allyn Malventano

    If you are moving from SATA you would notice a difference similar (or better) to running a 4x RAID-0 of good performance SATA SSDs. Boot times vary wildly based on each manufacturers NVMe implementation in their respective UEFI BIOS.

    We don't focus on boot times because the length of time booting a system can vary due to so many other factors, and that systems spend a small fraction of their total uptime in the boot process anyway."

    Naysan

    Hi Paul, the performance of the 950 Pro (and the Intel 750) are clearly crazy fast from the benchmarks above, but will the speed jump be so dramatic between say the 850 Pro (SATA3) and 950 Pro (NVMe) in the same tests - ie VM boot time, Application launches, VM clones)?

    Paul Braren | TinkerTry.com

    Why thank you, glad you had the Chuzpah to leave a comment!
    (on average, only 1 in 2,000 TinkerTry visitors take the time to do so, they really help other readers)

    Chuzpah

    Awesome, this was exactly what I was looking for, thanks for the great post and video!