Copying about 50GB from a Samsung 950 PRO NVMe SSD back to itself is slower than copying to another 950 PRO over 10GbE

Posted by Paul Braren on Nov 18 2015 in

Yesterday, I said:

Next torture test? Abuse of that 10GbE connection to the loaner Supermicro SuperServer right next to this one...

Well, that's what I went and done. Since the moment the idea entered my brain, until the test was done, I couldn't get the unanswered question out of my head. I just had to try it out. It's not like I can find anybody else out there who had done this, which only served to increase my interest in obtaining an answer.

Setup

First, you got your Samsung 950 PRO NVMe M.2 drive, 512GB size, specs here. Next you got another identical 950 PRO (borrowed), installed in another identical Supermicro SuperServer SYS-5028D-TN4T. NVMe is able to shine here, with the PCIe 3.0 x 4 lane M.2 socket in each system, paired with a 10GbE connection between the two. No jumbo frames, just install the Intel X552 Windows 10 driver and go. No pricey 10GbE switch required, a simple 10GbE capable 1' length of CAT6A cable will do.

b01639694m — Available now on Amazon in 256GB and 512GB sizes, 1TB not expected until early next year. I tested with the faster 512GB model, one that I owned, the other that I borrowed.

Finally, you got yourself a fresh build of Windows 10, and a BIOS set up like I used for benchmarking, explained here.

Question

Now you ask yourself, how long does it take to copy about 50GB of data, consisting of 17 copies of this 4.4GB openSUSE-12.3-DVD-x86_64.iso?

Answer

The answer, it turns out, depends highly on whether you're doing a local copy, where the same Samsung 950 is doing both reads and writes. Or whether you're doing it over the 10GbE network connection. Guess which wins?

Hands-down winner is the network copy. Yep, plain old SMB network sharing in Windows 10 can really hum along, given a fast enough network pipe, and fast enough SSD. Geez, my 10GbE is a bottleneck, when it comes to my Samsung 950 PRO? I'm ok with that ;-)

Analysis

Let's think for a minute together here. Yeah, just a minute, I can't wait to get the video out, and hear your feedback with comments right here below the article. So I'll just put my "draft" thoughts out there. Yesterday, reading and writing to the same drive seemed to fill a write buffer, then keep speeds down around 700MB/sec. Today, the same test over ethernet, I get back to 1200MB/sec, pretty near the 1576MB/sec of my best ATTO Disk Bench result. And very nearly identical to the 1250MB/sec I get when copying from one Samsung 950 PRO to another when inside the same system.

from_d_to_d_about_700_MBps — Copying 205GB from D: to C:

So it would seem more like the drop-off I noticed is just the way this drive and controller and firmware handles concurrent reads and writes that are sustained. I can't know for sure, this is just a conversation starter. I can't dwell on this, because in some sense it doesn't matter. This is simply the best damn SSD experience I've ever had, at work, or at home.

Enjoy the video, hopefully it's half as fun to watch as it was to record.

Copying from a 950 PRO NVMe SSD back to itself is slower than copying to another 950 PRO over 10GbE.

10GbE-seen-active — That's a CAT6a cable, with auto-crossover happening between the two 10GbE interfaces. The other cables are just to the IPMI management ports on the Supermicro system, making creating this video possible by showing you what's going on without using RDP, eliminating that "other" network traffic.

Source: Intel

Thank you

This article and video, and my articles about the Intel 750 Series NVMe SSD, wouldn't have been possible without the temporary loan of a second Samsung 950 PRO and a 128GB equipped second Supermicro SuperServer Bundle 2 by Trond Eirik Haavarstein aka Eric @xenappblog of xenappblog.com

You might even spot a familiar name among the presenters at xenappblog's upcoming virtual expo. Be sure to sign up early, because hundreds typically attend his events ;-)

Disclosure

This is not a sponsored post. I had been planning to do this testing for months now. The 2nd 950 PRO that I borrowed for this test was identical to the first, also from Amazon. My systems and my SSDs were all purchased, I just didn't have a 2nd identical system with a 2nd 950 PRO of my own to do these tests. If you purchase of this this same exact same SuperServer, it uses a Wiredzone affiliate link, and the Amazon's Samsung 950 PRO is also an affiliate link, helping quality articles like this be possible.