10GbE speeds

Posted by Paul Braren on Jul 30 2017 (updated on Oct 21 2017) in

Amazon - Promise SL3T1GL Thunderbolt 3 to Base-T Ethernet Bus Powered Adapter
Dell - Dell part #A9623557
eBay - All items with SL3T1GL in the description
Newegg - Promise Thunderbolt 3 to Base-T Ethernet Bus Powered Adapter (SL3T1GL)

Backstory

It's been about a decade since 1GbE (1 Gigabit) networking became prevalent, for desktops and laptops alike, even in home networks. In recent years, thinner lighter laptops did away with DVD first, then RJ45 connections for wired Ethernet next. If you are lucky enough to have a premium workstation or laptop equipped with Thunderbolt 3 over USB-C, then you have some incredible options available to you now, see also:

Road-warrior productivity boosters for laptops blessed with USB-C but no Ethernet or VGA
May 15 2017 by Paul Braren at TinkerTry

Turns out you can now also have full 10GbE on any Thunderbolt 3 equipped system. Promise has been at this connection to SAN game for a long time, featuring a variety of slightly older Thunderbolt 2 SANLink 2 Series and SANLink3 F2 series that allow for 10GbE or FC (up to 16Gb!) attachment of your system.

This article's only focus is on a new SANLink3 Product that fits my need to interconnect my M.2 NVMe equipped laptop to my Xeon-D 10GbE equipped servers, with all three systems featuring M.2 NVMe storage.

My use case for the SANLink3 T1

This is the product I've been patiently waiting for, having already been running 10GbE in my home network when my (world's-first 12 core) second Xeon-D SuperServer Bundle 2 arrived in October of 2016. The speed boost of vMotion between my 2 cluster nodes that have no shared storage was huge, each system only has local VMFS datastores. What if I could just as quickly and easily move huge 4K video files that I sometimes record to and from my laptop? See also some Xeon D multi-threaded processor performance measurements I recorded here. And what about huge VMs?

Here's the thing. I (experimentally) run one Xeon D SuperServer about 20 feet from my triple monitors, with a Windows 10 VM used as my primary workstation/datacenter upstairs. My second Xeon D SuperServer is in my basement. Working as a customer facing SE at VMware, I'm fortunate to also have a Dell Precision 5510 laptop upstairs as well. Wouldn't it be great if all 3 were interconnected, at full 10GbE speeds? This is when I'm glad I'm now equipped with RJ45 10GbE rather than SFP+, as it allows me to easily string CAT6a or CAT7 cabling through a PVC pipe I rammed from my home's basement to my attic, well over a decade ago.

Possibilities for 2.5/5.0/10GbE over RJ45 include:

Desktops / Servers

Give your Thunderbolt 3 over USB-C equipped PC 10GbE network capabilities that performs about as well as taking up a PCIe slot would, including small systems like the Intel NUC NUC6I7KYK that has no PCIe slots available, see also William Lam's excellent ESXi on the new Intel NUC Skull Canyon article. Note, it seems Xeon server-class systems don't currently feature Thunderbolt 3/USB-C connections, including the newly announced Purley, with the possible exception of the pricey, upcoming iMac Pro. Typically datacenter gear would use internal storage and PCIe slots anyway, so adding consumer friendly (but easily unplugged) USB-C likely isn't a priority. Read more at TinkerTry, Intel Xeon Scalable Processors (Purley) Unveiled - Intel and VMware touting roughly 2.5x the performance and 4x as many VMs versus the Xeon of 4 years ago.

Laptops

B06XN6XWD7 — "Dell 3GMVT Thunderbolt Dock TB16, 240W, black" at Amazon.

Give your Thunderbolt 3 over USB-C equipped PC 10GbE network capabilities that performs about as well as taking up a PCIe slot would.
It is possible to both charge the laptop, and have have triple displays and 10GbE on premium, workstation-caliber laptops, all through just one Thunderbolt 3/USB-C cable. I've personally tested this using a Dell Precision 5510 laptop and a Dell TB16 240 watt dock. This dock can work with other laptops, but it's clear from Dell's documentation that there is a real focus on ensuring the combination of BIOS updates and features mean the ownership experience may be smoother for Dell laptop users, see also Dell Thunderbolt Dock TB16 Driver Installation Guide. This reminds me of early USB 3.0 days, with some roughness around the edges that have been smoothed out with later Dell Precision 5510 BIOS updates, with Dell XPS13/XPS15 likely enjoying much of the same attention.
IMPORTANT NOTE - The TB16 has three video display outputs AND two Thunderbolt 3 USB-C connections: one goes to my laptop, the other to this Promise 10GbE adapter, this is crucial, since most Thunderbolt 3 docks only offer a 2nd USB-C port for charging only

Research

SANLink3 T1 product page

NBase-T speeds Full Duplex 1, 2.5, 5 or 10Gb/s
Single Thunderbolt™ 3 port and single NBaseT port
No Bulky external power supplies or cords
Compact, cool and extremely portable
0.5m 40Gb/s passive Thunderbolt™ 3 cable included
Linux and Windows driver download page
Amazon reviews

My TinkerTry'ing, so far...

I recorded 4K video of the unboxing, install, and basic driver install and testing, seen below. The BOM (Bill Of Materials) for all the equipment used in the video is featured here, including the popular Monoprice SlimRun Cat6a 10GbE cables. Important to note that they're only available in up to 14 feet lengths, so I may be testing out this 100' Cat7 flat cable by Jadaol instead, for my upstairs to basement runs. Any such lengths might need to be Plenum cable, check with your local building codes.

Device-Manager-Promise-SANLink3 — in my Device Manager, I disabled the Dell dock's Realtek USB 1GbE device, using the Promise Technology SANLink3 T1 instead

Promise-Driver-4.4.405.152 — Windows Driver Date 4/24/2016 Version 4.4.405.152

2017-07-31_0-29-20 — Promise driver's Advanced Settings. I didn't change any of them, and don't use Jumbo frames.

Here's the basics of what's featured in the video:

unboxing
connection to ASUS XG-U2008 Unmanaged 2-Port 10G and 8-Port Gigabit Switch
connection to VMware ESXi 6.5U1 running a Windows 10 VM on a 10GbE vSwitch
5GB Windows 10 ISO file copy over a Windows network share, from the Windows 10 laptop (driver v4.4.405.152) to the Windows 10 VM (VIB 4.5.1)

The dawn of 2.5/5.0/10GbE

Also known as 2.5 Gbps / 5.0 Gbps / 10 Gbps, various motherboards and PCIe cards and consumer-priced unmanaged switches are expected to become much more prevalent soon, and it's about time. We've been waiting for lower-priced options, catch a glimpse for yourself here:

ASUS XG-C100C NIC - 10 Gigabit Ethernet for the Masses
Jul 12 2017 by Ken Addison at PC Perspective

Given Aquantia is one of the only OEMs doing these new 2.5 and 5.0 speeds, I wouldn't be surprised if the Promise has an Aquantia AQC107 chipset inside. This also means that for now, the driver focus seems to be on Windows 10 and Mac, not datacenter OSs like Windows Server, Linux, or VMware vSphere.

I don't plan or need to have 2.5 or 5.0 speeds in my home's network, so the lack of compatibility that my current set of servers and switches isn't an issue for me personally. That's right, my Xeon D servers, and the 2 10GbE switches I currently own, won't negotiate future 2.5 or 5.0 speeds, simply falling back to 1GbE speeds.

I do hope we can all benefit from lower prices for all greater-than-1GbE technologies soon, something Mike Foley actually mentioned in passing when we were presenting at user group recently. It could also make home vSAN much more viable/obtainable, something I'm very interested in personally.

The future of datacenters is 25/40/50/100 GbE

What a mess, too many speeds, explained here, and well beyond the scope of this article.

Plans

I plan to continue to use my modest ASUS XG-U2008 when I travel with my SuperServer, doing a variety of live demonstrations at various most-local user group events. It's not a good choice for vSAN, and my Netgear XS708T might not be the best choice either, but the Netgear XS708T is what I'm now using to wire up my home with 10GbE goodness, full time. It doesn't have enough 1GbE ports for my needs, so I've also re-purposed my basic Cisco SG112-24, and attached it to my Netgear XS708T with a single CAT6a cable. It'd be better if it was SFP+ equipped for that uplink, but sure beats it gathering dust ever since the much cooler looking (and working) ASUS XG-U2008 replaced it for my public demos.

Conclusions - Expensive, but there's currently nothing else like it

You won't wonder why you'd want 10GbE in your home if you watch the video below, and if you ever produce content (or VMs) that consist of very large files. The amazing speeds of M.2 NVMe SSDs, and the even better speeds that NAND's eventual replacement by 3D XPoint (Intel Optane / Micron QuantX) will offer, signal the acute need for better networking, even in a home "lab" that power users like me really use for production.

Video

Promise SANLink3 T1 NBaseT Adapter blesses your Thunderbolt 3/USB-C PC with 1.0/2.5/5.0/10GbE speeds

Photo Gallery

Compatibility

Promise Technology published their SANLink3 T1 Compatibility List here.

Aug 01 2017

In the original article above, I failed to mention a bit of bad rep that copper 10GbE has seen for high power use. For each of my 2 Xeon D systems, I get about 0.6 watts of additional watt burn for each Intel X552/X557 10GbE RJ45 port that I plug in, using short (1 meter) cable lengths. I'll need to re-measure with longer (30 meter) lengths.

I also failed to mention cable types for copper (versus SFP/SFP+). One of the big goals of the new 2.5 and 5.0GbE speeds is a way to get standard CAT5 / CAT5e cabled homes a bit of a speed boost. With 10GbE, you'll likely require a re-wire:

CAT5e (wiki page) for short runs (<25 feet / 7.6 meters) may work for 10GbE speeds, but technically, CAT5e isn't certified for 10GbE at any length
CAT6 (wiki page) for 10GbE at up to 180 feet / 55 meters length
CAT6A (wiki page) for 10GbE at up to 328 feet / 100 meters length
CAT7 (wiki page) for 10GbE at up to 328 feet / 100 meters length
CAT8 (article) 10Gbps+

CAT8 is expected to support bandwidths of up to 2 GHz (2000 MHz) for up to 30 meters of cabling and can support 25Gbps / 40Gbps speeds. CAT8 cables will look similar to lower category cables and will most probably still be terminated in RJ45 connections.

More sources:
10 Gigabit Ethernet wiki featuring 10GBASE-T:

Class E channel using category 6, Class Ea channel using 6a or 7 twisted pair
avforums

D-Link 10 Gigabit Ethernet Performance and Speed:

10Gbps Ethernet is no longer restricted to fiber optic media only. Recent advancements in chip design have made 10GBASE-T (10Gbps over copper) more economical and easier to deploy than ever before. Direct Attach Copper (DAC) is also a popular choice, especially for shorter distance, latency sensitive applications.

D-Link-gbe-overview — From D-Link "Canada's 10 Gigabit Ethernet" page.

Aug 01 2017 Update 2

B018TTSUNM — "Cat 7 Ethernet Cable 100 ft Black – Jadaol" on Amazon.

It's alive! I have a fully operational 10GbE network now, with both 10GbE interfaces of my upstairs and downstairs Xeon D systems home-run CAT7 cabled to my Netgear XS708T switch. Of course, my Dell Precision 5510 laptop is also now home-run CAT7 cabled as well. The 3 cables barely fit, and the flat design made all the difference, as there's little chance I could have pulled the cables through the PVC had they been a conventional round design. I tested speeds with these cables before installing them, of course.

B00009363G — "MSA Safety Works 817663 Multi-Purpose Respirator" on Amazon.

This bit of home rewiring is something I've wanted for a long time, but choosing to enter the attic on a 90°F / 32°C day might not have been the best idea. I put my iPhone 7 Plus in a plastic snack bag to protect it from sweat when mowing or attic climbing, but that plastic also meant the phone got extra hot during an extended FaceTime session, as pictured below. That's right, being able to see and communicate to the person at the remote end of the PVC pipe that the cables are getting pulled through really helps, since flat cables do tend to need a second set of hands to detangle the cables while feeding them upward. I used my MSA Safety Works 817663 Multi-Purpose Respirator to protect my lungs while carefully maneuvering around the attic. I wished to avoid the "inconvenience" of me accidentally putting my foot through the ceiling again, and my wife would agree.