How to boot Windows 10 from NVMe based PCIe storage, featuring Samsung 950 PRO M.2 SSD in a Supermicro SYS-5028D-TN4T

Posted by Paul Braren on Nov 5 2015 (updated on Jul 18 2017) in

How to boot from NVMe (Non-Volatile Memory express). Why is that a thing to learn? Well, consider that it's an entirely new storage technology. The issue is not so much about the tiny M.2 2280 form factor, or PCIe or U.2 connections. It's everything to do with leaving legacy SATA/AHCI behind, which I've been wanting to do since well before my April post. This is not your slower Surface Book or Macbook air NVMe. This 950 PRO is faster than both of those, much faster. Especially for writes. Current the fastest consumer SSD in the world, and my initial tests sure seem to confirm this, partially because it's paired with a SuperServer that allows it to shine, with 4 lanes of PCIe 3.0 goodness.

B01639694M — Available now on Amazon in 256GB and 512GB sizes, 1TB not expected until early next year.

This is a very exciting day for me, the arrival of the final piece of my home lab storage strategy. And it seems to exceeed the high expectations of 2500MB/s reads and 1500MB/s writes. See also:

950 Pro review: Samsung’s first PCIe M.2 NVMe SSD is an absolute monster
Oct 22 2015 by Orestis Bastounis

You have some BIOS tweaks to do even if you use a PCIe card, like the recently tested Intel 750 Series PCIe NVMe. The YouTube video's commenter Dan L sums up the 750's intended use-case nicely:

Intel 750 is considered enterprise drive whereas Samsung 950 Pro is designed for client users. The reason why I didn't buy Intel 750 is because it mainly focus on higher queue depths such as data center, web server, and file server. In addition, it draws maximum power of 22watts and boot time was incredibly slow, though Intel released the new firmware update but boot time was still slower than Samsung SM951 and even Samsung 950 Pro.

Terminology

(above image from page 5 of Booting from an NVMe PCIe Intel Solid-State Drive - Technology Brief)

Requirements

Whatever NVMe drive you buy, you'll still likely need to meet all of the following requirements, if you plan to also use it as a boot device:

a newer system, such as the Xeon D-1540 or Skylake chipsets (2015 or later)
modern OSs have the NVMe driver built-in, like Windows 8.1/10 or Windows Server 2016 (the Microsoft NVMe driver is generally slower, replace it with vendor provided driver once Windows is installed)
a UEFI BIOS that supports boot from NVMe (not something you can assume)
(on this system's X10SDV-TLN4F motherboard, BIOS 1.0a or 1.0b both allow boot from NVMe)
an M.2 PCI-E x4 slot (as in 4 lanes, key to obtaining the maximum speeds)
BIOS in UEFI boot mode (nice bonus is that it allows your bootable device to be larger than 2TB, since Windows will use GPT rather than legacy MBR)
some mobos also require CSM (Compatibility Support Mode) be turned on, read ASRock example here, this Supermicro mobo has CSM Enabled by default
a willingness to do any required BIOS upgrades and/or BIOS configuration changes (varies by motherboard manufacturer)

Sounds a little like a hassle. The below video will demonstrate that it's really not, at least with this system, using Windows 10 that has built-in NVMe support.
Feb 8 2016 Update - With the benefit of hindsight, I've since learned that BIOS 1.0a may have needed the tweaks outlined below, but for BIOS 1.0b, all that's really required is to be sure you have the BIOS in UEFI mode when doing the install to NVMe, which lays down a GPT (not MBR) partition. The steps needed for other motherboard vendor BIOS configuration may be quite different.

What's the benefit of going through a bit of extra effort to be able to boot from NVMe? The convenience of one C: drive in Windows combined with an increased speed and reduced latency versus traditional 2.5" SATA SSDs, illustrated below.

Source: Intel

Step by step prep for Windows 10 boot from NVMe

From Download Windows 10, use the Download Tool 64-bit link to get MediaCreationToolx64.exe
Insert a USB 3.0 flash drive such as the SanDisk Ultra Fit and create the bootable install media
insert this flash drive into one of the blue rear USB 3.0 ports of the Supermicro SYS-5028D-TN4T
The default is Legacy, change these [and BIOS boot mode] to EFI
power up
press Del to get into BIOS
set BIOS to defaults, reboot
set BIOS to UEFI mode
go to SATA Configuration
change all SSDs you have from Hard Disk Drive to Solid State Drive
press ESC, then select PCIe/PCI/PnP Configuration
select M.2 PCI-E 3.0 X4, change from default [Legacy] to [EFI]
select SLOT 7 PCI-E 3.0 X16, change from default [Legacy] to [EFI]
optionally, for better speed, select Onboard Video Option ROM, change from default [Legacy] to [EFI], this allows you to do the next step
press ESC, select Security, select CSM Support, change from default [Legacy] to [EFI]
press ESC, select Boot Settings, Boot Mode Select, change from [Dual] default to [UEFI]
press ESC, Save Changes and Reset
if you have no data on any drives, your system will likely automatically boot from the USB drive (if not, press F11 at boot time to manually select USB UEFI)
perform a normal Windows 10 installation, you'll have a GPT drive type, and if you clone it someday to a >2TB drive, it will still be bootable
for this drive, the speed is slower with Windows 10's Microsoft NVMe driver, from Samsung's SSD site, download and install the Samsung NVMe Driver Installer, then reboot when prompted

That's it, you now have an incredibly fast boot time, and an amazing performer. Just a normal Windows 10 PC, only much faster.

Video

Thorough step-by-step look at the above steps, and more, including some basic ATTO, Crystalmark, and AS SSD benchmarks.

Brief unboxing, followed by simple installation into a Supermicro SYS-5028D-TN4T.

Benchmark Test Results

~~work-in-progress...~~

Nov 07 2015 Update - Benchmark Results are in! See:

World's fastest consumer SSD - Samsung 950 PRO M.2 NVMe benchmark results

Everything below is a peek at some early attempts to create a baseline of expected performance, before I head over to VMware ESXi 6.0 to perform these same tests again, but with Windows 10 then "living" as a VM on a VMFS datastore on this 950 PRO.

Nov 05 2015 Update

Thanks to YouTube commenter Dan L yet again! He's mentioned above, and added another tip, mentioning BIOS's CSM setting. Being a tinkerer, I headed back into the BIOS, and turned CSM off. What do you know, performance increased! Not a big change, but noticable, as you look over this image carefully. The above procedure has now been tweaked accordingly, and the video has been annotated.

Dan L goes on to give more excellent advice:

Keep in mind that some PCIe slots are connected directly to PCH (Platform Controller Hub) which runs in PCIe Gen2 Lanes instead of PCIe Gen3 lanes. So I encourage people to refer to user's manual as well as UEFI BIOS to check which PCIe slots are directly connected to PCIe Gen3 lanes.

Nov 06 2015 Update

Well that was unexpected! Turns out the 950's built in thermal protection was likely the reason the speeds dropped toward the end of these very first ATTO Disk Benchmark runs I did yesterday.

Silly me, after recording the installation of the M.2 drive, I had failed to put the PC's cover back on. Yep, this server is running far from me in my basement. Given months of experience with all the other components and drives in this system barely going up a degree with the lid off, even under heavy stress tests. So usually heat is a non-issue with this system, at all.

Now let's think about the tiny surface area of M.2 for moment. If it's using using up to 7 watts when under heavy load, well, that heat has to go somewhere. Given the small surface area, well, it's no surprise that airflow certainly matters. You'll even see that on Nov 4th in my older 950-related post, where I already had the nifty thermal video displayed there, so it didn't take me long to figure out. I'll publish my "tinkering" video with the 2nd M.2 in the PCIe slot soon, and revisit whether CSM really matters. This is just preliminary findings, as I actually like finding out stuff from accidents. It's called tinkering. And it's certainly nice to see even faster speeds now.

950-PRO-ATTO-Results-CSM-off-pc-cover-off-5028D-TN4T-2015-11-05-TinkerTry — Nov 05 2015 PC cover OFF.

950-PRO-ATTO-Results-CSM-off-pc-cover-on-5028D-TN4T-2015-11-06-TinkerTry — Nov 06 2015 PC cover ON.

All the basic benchmarks and observations will belong in a new post, here at TinkerTry. Stay tuned!

Meanwhile, here's a convenient copy of the thermal tests:

Nov 07 2015 Update

Turning the fan higher and leaving the lid on resolves any minor issues with thermal protection kicking in. At the expense of fan noise, of course. That D: drive on the Lycom PCI adapter held up a little longer (ran a little cooler) than the motherboard M.2 slot does, when put under heavy stress. Seems unlikely that you'd have an issue with it in normal use.

950-PRO-C-and-D-Lycom-adapter-performanced-comparison — Here's a first run of ATTO test, with BIOS default system fan speed, and cover back on the stock SuperServer 5028D-TN4T.

Feb 03 2016 Update

I've since learned that 1.0a really needs all those BIOS tweaks above, but BIOS 1.0b does not. In fact, other than turning on UEFI mode in the BIOS, the rest of the tuning is optional. Boot from NVMe will "just work"! See also:

I also plan to do some thermal testing of my various SSDs, see also my newly-published video:

I'm just warning up, more videos featuring thermal imaging to come!

It looks like you have JavaScript disabled. Click here to view the video above.

Video and Audio sped-up 4X, showing Samsung SSD 950 PRO M.2 NVMe during initial power-up and boot. You'll even hear the BIOS speaker beep at the end of POST.

This article and video, and my articles about the Intel 750 Series NVMe SSD, wouldn't have been possible without the temporary loan of a second Samsung 950 PRO and a second Supermicro SuperServer Bundle 2 by Trond Eirik Haavarstein aka Eric @xenappblog of xenappblog.com

You might even spot a familiar name among the presenters at xenappblog's upcoming virtual expo. Be sure to sign up early, because hundreds typically attend his events ;-)

Apr 23 2016 Update

Note that the 950 PRO can work on both BIOS and UEFI systems, details below, here's an excerpt:

Samsung says this is backwards compatible with Legacy BIOS systems so long your system can initialize IDE devices (like most chipsets), it can use this SSD. Your ONLY requirement is a free PCIe x4 slot and a modern OS with the appropriate NVMe driver. - SkOrPn