How to boot Windows 10 from NVMe based PCIe storage, featuring Samsung 950 PRO M.2 SSD in a Supermicro SYS-5028D-TN4T
How to boot from NVMe (Non-Volatile Memory express). Why is that a thing to learn? Well, consider that it's an entirely new storage technology. The issue is not so much about the tiny M.2 2280 form factor, or PCIe or U.2 connections. It's everything to do with leaving legacy SATA/AHCI behind, which I've been wanting to do since well before my April post. This is not your slower Surface Book or Macbook air NVMe. This 950 PRO is faster than both of those, much faster. Especially for writes. Current the fastest consumer SSD in the world, and my initial tests sure seem to confirm this, partially because it's paired with a SuperServer that allows it to shine, with 4 lanes of PCIe 3.0 goodness.
- 950 Pro review: Samsung’s first PCIe M.2 NVMe SSD is an absolute monster
Oct 22 2015 by Orestis Bastounis
Intel 750 is considered enterprise drive whereas Samsung 950 Pro is designed for client users. The reason why I didn't buy Intel 750 is because it mainly focus on higher queue depths such as data center, web server, and file server. In addition, it draws maximum power of 22watts and boot time was incredibly slow, though Intel released the new firmware update but boot time was still slower than Samsung SM951 and even Samsung 950 Pro.
(above image from page 5 of Booting from an NVMe PCIe Intel Solid-State Drive - Technology Brief)
Whatever NVMe drive you buy, you'll still likely need to meet all of the following requirements, if you plan to also use it as a boot device:
- a newer system, such as the Xeon D-1540 or Skylake chipsets (2015 or later)
- modern OSs have the NVMe driver built-in, like Windows 8.1/10 or Windows Server 2016 (the Microsoft NVMe driver is generally slower, replace it with vendor provided driver once Windows is installed)
- a UEFI BIOS that supports boot from NVMe (not something you can assume)
(on this system's X10SDV-TLN4F motherboard, BIOS 1.0a or 1.0b both allow boot from NVMe)
- an M.2 PCI-E x4 slot (as in 4 lanes, key to obtaining the maximum speeds)
- BIOS in UEFI boot mode (nice bonus is that it allows your bootable device to be larger than 2TB, since Windows will use GPT rather than legacy MBR)
- some mobos also require CSM (Compatibility Support Mode) be turned on, read ASRock example here, this Supermicro mobo has CSM Enabled by default
- a willingness to do any required BIOS upgrades and/or BIOS configuration changes (varies by motherboard manufacturer)
Sounds a little like a hassle. The below video will demonstrate that it's really not, at least with this system, using Windows 10 that has built-in NVMe support.
Feb 8 2016 Update - With the benefit of hindsight, I've since learned that BIOS 1.0a may have needed the tweaks outlined below, but for BIOS 1.0b, all that's really required is to be sure you have the BIOS in UEFI mode when doing the install to NVMe, which lays down a GPT (not MBR) partition. The steps needed for other motherboard vendor BIOS configuration may be quite different.
What's the benefit of going through a bit of extra effort to be able to boot from NVMe? The convenience of one C: drive in Windows combined with an increased speed and reduced latency versus traditional 2.5" SATA SSDs, illustrated below.
- From Download Windows 10, use the Download Tool 64-bit link to get MediaCreationToolx64.exe
- Insert a USB 3.0 flash drive such as the SanDisk Ultra Fit and create the bootable install media
- insert this flash drive into one of the blue rear USB 3.0 ports of the Supermicro SYS-5028D-TN4T
- power up
- press Del to get into BIOS
- set BIOS to defaults, reboot
- set BIOS to UEFI mode
- go to SATA Configuration
- change all SSDs you have from Hard Disk Drive to Solid State Drive
- press ESC, then select PCIe/PCI/PnP Configuration
- select M.2 PCI-E 3.0 X4, change from default [Legacy] to [EFI]
- select SLOT 7 PCI-E 3.0 X16, change from default [Legacy] to [EFI]
- optionally, for better speed, select Onboard Video Option ROM, change from default [Legacy] to [EFI], this allows you to do the next step
- press ESC, select Security, select CSM Support, change from default [Legacy] to [EFI]
- press ESC, select Boot Settings, Boot Mode Select, change from [Dual] default to [UEFI]
- press ESC, Save Changes and Reset
- if you have no data on any drives, your system will likely automatically boot from the USB drive (if not, press F11 at boot time to manually select USB UEFI)
- perform a normal Windows 10 installation, you'll have a GPT drive type, and if you clone it someday to a >2TB drive, it will still be bootable
- for this drive, the speed is slower with Windows 10's Microsoft NVMe driver, from Samsung's SSD site, download and install the Samsung NVMe Driver Installer, then reboot when prompted
That's it, you now have an incredibly fast boot time, and an amazing performer. Just a normal Windows 10 PC, only much faster.
Nov 07 2015 Update - Benchmark Results are in! See:
Everything below is a peek at some early attempts to create a baseline of expected performance, before I head over to VMware ESXi 6.0 to perform these same tests again, but with Windows 10 then "living" as a VM on a VMFS datastore on this 950 PRO.
Thanks to YouTube commenter Dan L yet again! He's mentioned above, and added another tip, mentioning BIOS's CSM setting. Being a tinkerer, I headed back into the BIOS, and turned CSM off. What do you know, performance increased! Not a big change, but noticable, as you look over this image carefully. The above procedure has now been tweaked accordingly, and the video has been annotated.
Dan L goes on to give more excellent advice:
Keep in mind that some PCIe slots are connected directly to PCH (Platform Controller Hub) which runs in PCIe Gen2 Lanes instead of PCIe Gen3 lanes. So I encourage people to refer to user's manual as well as UEFI BIOS to check which PCIe slots are directly connected to PCIe Gen3 lanes.
Well that was unexpected! Turns out the 950's built in thermal protection was likely the reason the speeds dropped toward the end of these very first ATTO Disk Benchmark runs I did yesterday.
Silly me, after recording the installation of the M.2 drive, I had failed to put the PC's cover back on. Yep, this server is running far from me in my basement. Given months of experience with all the other components and drives in this system barely going up a degree with the lid off, even under heavy stress tests. So usually heat is a non-issue with this system, at all.
Now let's think about the tiny surface area of M.2 for moment. If it's using using up to 7 watts when under heavy load, well, that heat has to go somewhere. Given the small surface area, well, it's no surprise that airflow certainly matters. You'll even see that on Nov 4th in my older 950-related post, where I already had the nifty thermal video displayed there, so it didn't take me long to figure out. I'll publish my "tinkering" video with the 2nd M.2 in the PCIe slot soon, and revisit whether CSM really matters. This is just preliminary findings, as I actually like finding out stuff from accidents. It's called tinkering. And it's certainly nice to see even faster speeds now.
All the basic benchmarks and observations will belong in a new post, here at TinkerTry. Stay tuned!
Meanwhile, here's a convenient copy of the thermal tests:
Turning the fan higher and leaving the lid on resolves any minor issues with thermal protection kicking in. At the expense of fan noise, of course. That D: drive on the Lycom PCI adapter held up a little longer (ran a little cooler) than the motherboard M.2 slot does, when put under heavy stress. Seems unlikely that you'd have an issue with it in normal use.
I've since learned that 1.0a really needs all those BIOS tweaks above, but BIOS 1.0b does not. In fact, other than turning on UEFI mode in the BIOS, the rest of the tuning is optional. Boot from NVMe will "just work"! See also:
- New Windows 10 NVMe driver 220.127.116.11 released for Samsung SSD 950 PRO M.2
I also plan to do some thermal testing of my various SSDs, see also my newly-published video:
This article and video, and my articles about the Intel 750 Series NVMe SSD, wouldn't have been possible without the temporary loan of a second Samsung 950 PRO and a second Supermicro SuperServer Bundle 2 by Trond Eirik Haavarstein aka Eric @xenappblog of xenappblog.com
Note that the 950 PRO can work on both BIOS and UEFI systems, details below, here's an excerpt:
Samsung says this is backwards compatible with Legacy BIOS systems so long your system can initialize IDE devices (like most chipsets), it can use this SSD. Your ONLY requirement is a free PCIe x4 slot and a modern OS with the appropriate NVMe driver. - SkOrPn
This article provides information on the PCIe SSD's which were initially available on Dell servers, but are now being offered on other systems. We will take you through what a PCIe SSD is, does and how it does it. We will also cover what's needed to use one of these drives as your boot drive with a Windows Operating System installed and go over any other relevant information that would be useful to know when dealing with this type of device.
Note : Best Practice for a Windows operating system Install is to only have the desired boot drive powered during installation.
- Intel 750 Series NVMe PCIe SSD supported by ESXi 6.x out-of-the-box, install Intel's VIB for full speed
Sep 29 2015
Superguide: SuperServer home virtualization lab storage tiers, platinum through bronze, how many efficient drives fit inside this tiny chassis?
Sep 25 2015
Here's where I talk about my plans to use the 950 PRO for both VMware ESXi 6 read cache and VMFS datastore duties, for my favorite VMs.
- Detailed assembly and configuration instructions for Supermicro SuperServer Workstation Windows 10 Pro bundle
Aug 19 2015
Booting from an NVMe PCIe Intel Solid-State Drive - Technology Brief
Apr 2015 by Intel
Samsung 950 PRO 256GB and 512GB M.2 NVMe PCIe SSD Review
Oct 22 2015 by Allyn Malventano at PC Perspective
Samsung 950 Pro M.2 NVME SSD Review (256/512GB) – The NVMe Effect
Oct 22 2015 by Les Tokar at The SSD Review
- The Samsung 950 Pro PCIe SSD Review (256GB and 512GB)
Oct 22 2015 by Billy Tallis at AnandTech