How to download and install the Intel Xeon D 10GbE X552/X557 driver/VIB for VMware ESXi 6.x, works with the X540 PCIe card too

Posted by Paul Braren on Nov 2 2015 (updated on Jul 12 2018) in
  • Virtualization
  • ESXi
  • Network
  • HowTo
  • HomeLab
  • HomeServer
  • It's gotten very easy to install this VIB, with the easy method right up top, featuring the latest well tested Jul 19 2017 4.5.3 driver, with a full description of how I developed and tested an easy 4.5.3 install procedure below. For the original fully supported but more time consuming method for production environments, jump below for details.

    Fast, easy, and effective 10GbE VIB installation method (unsupported)

    Same command works for upgrades too!

    This super fast download-and-install one liner takes under a minute to apply, partly because the VIB is only 187KB! This particular screenshot actually shows how it's the same command for an upgrade. Your 'Removed:' section may differ. This is an unaltered copy of the Intel VIB
    net-ixgbe_4.5.3-1OEM.600.0.0.2494585.vib

    Warning/Disclaimer:
    I've temporarily hosted this shortcut method at tinkertry.com, for convenience. There is no warranty implied, and there is no support given for anybody who finds they encounters issues. If you are at all concerned about 3rd party VIB source, your connection is https encrypted, and the MD5SUM to help detect data corruption is:
    3c69bc962e0571a9e265b13f405d0451
    You can also do an FC on this hosted file versus the VMware file. Of course, those concerned can simply use the fully VMware supported method below, which requires a few more steps, including authentication from "My VMware" just to get the VIB.

    Expert Short Instructions

    For non-production test labs.

    1. backup your ESXi, this will fully protect you from troubleshooting, especially if you have no VMware support, with peace-of-mind, since you'll have the option to roll-back your ESXi should your VIB install not go well
    2. shutdown all your VMs, or go into maintenance mode first
    3. login to your ESXi SSH shell, and paste in the following command:
      esxcli software vib install -v https://cdn.tinkertry.com/files/net-ixgbe_4.5.3-1OEM.600.0.0.2494585.vib --no-sig-check
    4. Now you can reboot through the web GUI, or via SSH with this command:
      reboot

      That's it! You can see some sample pictures of how things should look in ESXi after you've made the upgrade and now have this new driver loaded.

    4.5.3-installed-TinkerTry-Nov-01-2017
    Easy 10GbE install method used, note the use of the no-sig-check option, that's why the long method detailed below is recommended for production workloads, since it starts with logging into My VMware and downloading the zip directly from VMware.
    vSphere-Client-view-of-x552-installed
    vSphere Client view of installed Intel X552/X557 10GbE.
    vSphere-Web-Client-view-of-x552-installed
    vSphere Web Client view of installed Intel X552/X557 10GbE.

    Installation Instructions

    X552/X557 ESXi VIB on ESXi 6.x instructions, based on Release_Notes_ixgbe-4.5.3.txt.

    Before you Begin!

    Make sure you're ready to reboot soon, with VMs already shut down, or get into maintenance mode. That's because the end of this intruction set has you rebooting!

    Begin

    1. Open an SSH session (PuTTY) to your ESXi 6.0 (or ESXi 6.0 Update 1) server
      (if you forgot to enable SSH, here's how)
    2. Put your system into maintenance mode, or, shut down all your VMs now and ensure you've set your ESXi host to automatically gracefully shutdown all VMs upon host reboot.
    3. Unzip the downloaded ixgbe-4.5.3-2494585-6925533.zip file, the VIB you want is named net-ixgbe_4.5.3-1OEM.600.0.0.2494585.vib and is found at the top level of the unzipped folder.
    4. Upload net-ixgbe_4.5.3-1OEM.600.0.0.2494585.vib to one of your datastores using vSphere Web Client Datastore browser.
    5. Open an SSH session (PuTTY) to your ESXi 6.x host/server, my example below is at ESXi 6.0 Update 1a
      (if you forgot to enable SSH, here's how).
    6. Paste the following commands into your PuTTY session, one line at a time.
      (you will need to specify the full path to where you put your VIB, it will differ from what is seen as Intel_750_NVMe_400GB in my example screenshots, but using tab autocomplete, it's pretty easy to construct this command in your environment, on the fly).
      esxcli software vib install -v /vmfs/volumes/[datastore]/net-ixgbe_4.5.3-1OEM.600.0.0.2494585.vib
    7. Delete the VIB from the datastore.
    8. Type "reboot" and hit return.

    That's it!

    10gbe-vib-install-and-cleanup-successful
    This is my successful install screenshot on ESXi 6.0U2, click twice to zoom in fully. It should look very similar for ESXi 6.5.x, and the newer 4.5.3 on ESXi 6.x
    10GbE_active_on_5028d-tn4t_on_esxi6
    Now that's how it's supposed to look!

    Video

    How to add Intel X552/X557 10GBase-T driver VIB to VMware ESXi 6.x on Supermicro Xeon D SuperServers

    Original Install Method

    Kept for archival reasons, you can skip down to the many article updates below.

    It's here, at last! The long wait since June for this 10GbE VIB ended when JBBERLIN77777 replied to my reports of a VMware X552/X557 driver download site with invalid links. He announced his finding of the download link, over in the huge VMware Communities thread:

    Were's still not sure how he found that URL, but we're all glad it works! Can you tell I'm happy, especially now that I'll have 2 SuperServers directly connected over 10GbE using Cat6a cabling, and each will have a Samsung 950 PRO M.2 NVMe 512GB drive, on November 3rd. In other words, a way to actually push the speeds of that wire. Woo hoo, what timing, this is a great week! No, I don't own a 10GbE switch, and this second SuperServer 5028D-TN4T system is just a loaner from a very gracious Wiredzone customer. I only have it for another few weeks.

    Download

    Here's the driver family download page:

    March 17 2017 Update - new VIB 4.5.1 is released. Here's the Release_Notes_ixgbe-4.5.1.txt and the actual driver download page for the new VIB, requires My VMware login (sign-up is free):

    • VMware ESXi 6.0 ixgbe 4.5.1 NIC Driver for Intel Ethernet Controllers 82599, x540, x550, and x552

      File size: 851.5 KB
      File type: zip
      Download Now Name: ixgbe-4.5.1-2494585-5197765.zip
      Release Date: 2017-03-17
      Build Number: 5197765
      VMware ESXi 6.0 ixgbe 4.5.1 NIC Driver for Intel Ethernet Controllers 82599, x540, x550, and x552
      The ESXi 6.0 driver package, also compatible with ESXi 6.5, includes version 4.5.1 of the Intel ixgbe driver. It supports the products based on the Intel 82599, x540, x550, and x552 10 Gigabit Ethernet Controllers. For detailed information and ESX hardware compatibility, please check the I/O Hardware Compatibility Guide Web application.

      MD5SUM: db909c779fd141796fb97f1e04a4f304
      SHA1SUM: 1f071979d0f2ee0265117986271a37f68dd29043
      SHA256SUM: 192e16d6c37106404826ca944f943d653a49c8c31ba20745abf2ca81fa209d87

    This VIB works fine with 6.0, as we are reassured that's OK at the Download VMware vSphere site:

    > Important Note for Certified I/O drivers
    > ESXi 6.0 supports I/O drivers built and certified on ESXi 5.5. The VMware Compatibility Guide lists both ESXi 5.5-based and ESXi 6.0-based drivers as supported with ESXi 6.0. See Knowledge Base article 2111492.

    I do not know how that URL was found, and there seems to be no 6.0 version of that download site, but if one does show up, it's pretty likely to turn out to be the same exact VIB file inside.

    Nov 03 2015 9:00pm EST Update - Yep, they're all the same VIB. VMware has now fixed the URLs on the existing download site, and they use the same link for all the 5.5 and 6.0 versions.

    Here's the actual driver download, the page you get the VIB:
    - VMware ESXi 5.5 ixgbe 4.1.1.1 NIC Driver for Intel(R) 82599, X540 and X552/X557-AT 10 Gigabit Ethernet Controllers family
    my.vmware.com/group/vmware/details?downloadGroup=DT-ESXI55-INTEL-IXGBE-4111&productId=353


    Nov 11 2015 Update

    Turns out SR-IOV likely won't be enabled after all, on this X552 in the Xeon D-1540, but likely will in the Xeon D-1541, details at TinkerTry here.


    See also this great question in this TinkerTry commenter Kent Winsor:

    Hi Paul, can the 10 GB NICS on the Supermicro SuperServer also operate at 1 GB speed under ESXi 6.0? Thank you.

    Yep, no problem at all, auto speed negotiation on by default. So in a sense you have 4 1GbE ports, and one 1GbE dedicated to management.

    All 4 ports handle auto-crossover as well, should you use direct attachment to another system, without a network switch in the middle. Just use normal CAT5e for the 1GbE ports, and normal CAT6a or CAT7 cables for those 10GbE ports, no cross-over cabling required. Easy under Windows, but issues with getting this to work under VMware.

    auto-crossover-cat6a-cabling


    Mar 10 2016 Update

    SR-IOV is alive! (for Xeon D-1541 with BIOS 1.0c)


    May 18 2016 Update

    New driver version! Intel's network product page has since been updated to include the very new ixgbe version 4.4.1 driver. I have updated the instructions above accordingly. It is reported to work fine with a D-Link DXS-1210-TC 10GbE switch, and with ESXi 6.0 Update 2.


    Nov 18 2016 Update

    VMware ESXi 6.5 came out on Nov 15 2016, recorded a video of the simple install method using the unchanged one-liner install, and added it to the article above. Firewall commands instructions are optional for most configurations, so I've left them only in the longer instruction set above.

    Also worth noting that there are issues with trying to force data rates with this driver, so direct connect (auto-crossover RJ45 ports allow any normal CAT6a or CAT7 cable) gets tricky, discussed here.


    Dec 16 2016 Update

    VMware ESXi 6.5 / vSAN 6.5 now officially announced support for direct-connect.


    Apr 14 2017 Update

    4.5.1 driver came out here, here's the details:

    VMware ESXi 6.0 ixgbe 4.5.1 NIC Driver for Intel Ethernet Controllers 82599, x540, x550, and x552

    File size: 851.5 KB
    File type: zip
    Download Now Name: ixgbe-4.5.1-2494585-5197765.zip
    Release Date: 2017-03-17
    Build Number: 5197765

    Article above updated accordingly.

    Here's the contents of the Release_Notes_ixgbe-4.5.1.txt

    Copyright(c) 2013 - 2017 Intel Corporation

    This release includes the ixgbe VMware ESX Driver for Intel(R) Ethernet Controllers 82599, x540, x550, and x552

    Driver version: 4.5.1

    Supported ESXi release: 6.0
    Compatible ESX versions: 6.5

    Dependencies: None
    New hardware supported:

    • Add new devices support for specific OEMs

    Additional configuration options supported by the driver: None

    Bug fixes:

    b1208972122: Fix PSOD when CNA = 8 and VMDQ < 4
    b1208935559: Network device not showing "Cable Type" information for DA connection on X520-DA2
    b1208939075: Allow to force link speed using esxcli or vSphere Web Client

    Features: None

    Known Issues: None


    Apr 19 2017 Update

    If you've gone to ESXi 6.5.0d Build 5310538 and found your 10GbE NICs have disappeared, don't worry, just re-install them. It's all explained right here.


    Jun 09 2017 Update

    Turns out the exact chip in the Xeon D series is actually the X557, which is pictured in the schematics and descriptions Intel created here:
    Datasheet - Volume 4 of 4: Intel Xeon Processor D-1500 Product Family LAN Controller

    10GBASE-KR interface to:
    -- 10 Gb SFI PHY (Inphi* CS4227)
    -- 10GBASE-T PHY (Intel X557-AT / X557-AT2 (2-port)

    X557-AT2

    See also:

    Great tool from Microsoft called NTttcp Utility:
    NTttcp Utility: Profile and Measure Windows Networking Performance
    as featured in this excellent video:

    "10 Gigabit Networking - I wanna go fast. Really, really fast." at Linus Tech Tips.

    Aug 01 2017 Update

    4.5.2 is out, download and release notes here. Read those release notes, it would seem not everybody will want or need this particular update, which seems geared for FCoE customers, explained here in KB 2149835 as well.


    Aug 10 2017 Update

    This same driver works fine with the Intel X540 PCIe (AIC) cards as well! I tested 4.5.2 on my old Core i7 vZilla build, and there were no issues, 10000 speeds acheived.s


    Oct 29 2017 Update

    Testing of version 4.5.2 has gone very well, so the article has been updated to replace the 4.5.1 recommendation with 4.5.2.

    These instructions are still required for those using fresh or upgraded installs of 6.5, see:

    This new release is also known as ESXi 6.5U1EP04.


    Oct 31 2017 Update

    Testing of version 4.5.3 has begun contents of above article will be updated once I'm comfortable that this driver works well on Intel Xeon D-1541 and Xeon D-1567.

    While this new Intel driver shows as October 23 2017, the download just became available today. Here's how I located the driver.

    Step 1 - VMware Compatibility Guide

    If we do a VMware Compatibility Guide - I/O Device Search of the VMware Compatibility Guide site for:

    • Product Release Version: ESXi 6.5 U1
    • Brand Name: Intel
    • Keyword: X552
    • I/O Device Type: Network

    Yeah, I know, Xeon D has the X552/X557 family of 10GbE, but the actual device is X557, which yields strange results, so I went with X552 instead.

    You'll see we get several results that show two relevant lines with support for VMware ESXi releases 5.0 through 6.5 U1:

    Each URL is slightly different, but they all land at the same page, so it doesn't matter which link you pick (I chose the last one).

    Now you're a the driver portion of the VMware Compatibility Guide...

    Step 2 - VMware Compatibility Guide Drivers

    Let's choose to expand just the Relese - ESXi 6.5 U1, Device Driver(s) ixgbe version 4.5.3 section, where I find the actual download driver link in the section labeled Footnotes:

    Footnotes : Download driver from https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESX60-INTEL-NIC-IXGBE-453&productId=491

    which brings me to the driver download and release notes download page, I downloaded both...

    Step 3 - My VMware Downloads

    Download VMware ESXi 6.0 ixgbe 4.5.3 NIC Driver for Intel Ethernet Controllers 82599,x520,x540,x550,and x552 family

    VMware ESXi 6.0 ixgbe 4.5.3 NIC Driver for Intel Ethernet Controllers 82599,x520,x540,x550,and x552
    Download Now
    my.vmware.com/group/vmware/details?downloadGroup=DT-ESX60-INTEL-NIC-IXGBE-453&productId=491#

    File size: 859.3 KB
    File type: zip
    Name: ixgbe-4.5.3-2494585-6925533.zip
    Release Date: 2017-10-23
    Build Number: 6925533

    Release_Notes_ixgbe-4.5.3.txt
    Download Now
    my.vmware.com/group/vmware/details?downloadGroup=DT-ESX60-INTEL-NIC-IXGBE-453&productId=491#

    File size: 4.8 KB
    File type: txt
    Name: Release_Notes_ixgbe-4.5.3.txt
    Release Date: 2017-10-23

    Step 4 - Read Release_Notes_ixgbe-4.5.3.txt

    Read all of the release notes, every line, there's some strange stuff in there about FCoE, and these 3 Bug Fixes:

    Bug Fixes:

    • Fixed non-working network stack when increasing rx ring buffer size for more than 4 ports
    • Fixed X540 initialization failure following a PSOD
    • Fixed VF VLAN communication issue after disabling and re-enabling a PF

    file

    Now I have the file I want:
    /ixgbe-4.5.3-2494585-6925533/net-ixgbe_4.5.3-1OEM.600.0.0.2494585.vib
    all ready for my testing, so I can soon call it TinkerTry'd. You may want to push a copy to your intranet host that is accessible via a URL. I've put a copy on my external host, as a convenience, use-at-your-own-risk for home labs. You can verify the checksum if you'd like, 3 types of checksums provided above.

    After backing up the ESXi host, and after shutting down all VMs (as explained in detail in the full article above), I'm ready for the simple install two-liner, pasted in to my ssh session to my intended ESXi server:

    esxcli software vib install -v https://cdn.tinkertry.com/files/net-ixgbe_4.5.3-1OEM.600.0.0.2494585.vib --no-sig-check
    reboot
    4.5.3-installed-TinkerTry-Nov-01-2017

    this installs the driver the easy way, no My VMware login neeed.
    After the reboot, I issued this command

    esxcli network nic list

    to determine which nice name to use from column one, then:

    esxcli software vib list | grep gb

    to see which driver is loaded for that nic, which should include the
    4.5.3-1OEM.600.0.0.2494585
    portion of the bundle called
    INT_bootbank_net-ixgbe_4.5.3-1OEM.600.0.0.2494585.vib

    Now I'm done with the install, testing speeds and function over time can now commence, using my home's 10GbE network.


    Nov 15 2017 Update

    As see on page 3 of Intel's document called:

    You'll find the various products this driver works on:

    Product Codes: EZX557-AT, EZX557-AT2, and EZX557-AT4

    Also listed by Device ID, which can be found at various places in the vSphere GUIs:

    Table 1-2 Device ID
    X557 Device Vendor ID Device ID
    Intel® Ethernet Connection X557-AT (Single 19x19mm) 8086 0xB4A3
    Intel® Ethernet Connection X557-AT2 (Dual 19x19mm) 8086 0xB4C3
    Intel® Ethernet Connection X557-AT4 (Quad 25x25mm) 8086 0xB4B3


    Dec 13 2017 Update

    Testing of 4.5.3 has gone well, all references to 4.5.2 in the body of installation instructions have been updated to 4.5.3.


    Dec 24 2017 Update

    I've received some reports of some 12 core Xeon D systems having network outages as often as twice per day, which I've occasionally experienced in my home lab as well. The link layer LEDs go off on the attached 10G switch, regardless of the switch brand, cabling, or VIB used. I plan to be detailing this story in a separate post, stay tuned. Meanwhile, there is a workaround that really does works, albeit only temporarily:

    1. gracefully shut down your 12 core (or greater) ESXi system
    2. unplug the power cord for at least 15 second
    3. plug the power cord back in
    4. power up and boot ESXi

    A proper fix is available by contacting Supermicro directly, but I'm coordinating with Supermicro on this to hopefully see a fix incorporated right into future BIOS releases.


    Feb 17 2018 Update

    After reading through the 4.5.3 release notes, it would seem some folks may want to try the older 4.5.2 or 4.5.1 versions out. To make this easier, I'm publishing the handy one-liner commands to do so here:

    4.5.2 - for non-production test labs.

    1. backup your ESXi
    2. shutdown all your VMs, or go into maintenance mode first
    3. login to your ESXi SSH shell, and paste in the following command:
      esxcli software vib install -v https://cdn.tinkertry.com/files/net-ixgbe_4.5.2-1OEM.600.0.0.2494585.vib --no-sig-check
    4. Now you can reboot through the web GUI, or via SSH with this command:
      reboot

    4.5.1 - for non-production test labs.

    1. backup your ESXi
    2. shutdown all your VMs, or go into maintenance mode first
    3. login to your ESXi SSH shell, and paste in the following command:
      esxcli software vib install -v https://cdn.tinkertry.com/files/net-ixgbe_4.5.1-1OEM.600.0.0.2494585.vib --no-sig-check
    4. Now you can reboot through the web GUI, or via SSH with this command:
      reboot

    Jul 11 2018

    ESXi 6.7

    To test the ixben 1.7.1 driver, review this comment, then see the driver download page, with the VMW-ESX-6.7.0-ixgben-1.7.1-8349137.zip download and Release_Notes_ixgben-1.7.1.txt release notes.

    Alternatively, you can use the following command to directly install it to your internet connected ESXi host, in an SSH session:

    esxcli software vib install -v https://cdn.tinkertry.com/files/INT_bootbank_ixgben_1.7.1-1OEM.670.0.0.7535516.vib

    I have not completed my own testing of this quite yet.

    ESXi 6.5

    To test the ixben 1.6.5 driver, see the driver download page, with the VMW-ESX-6.0.0-ixgben-1.6.5-7146128.zip download and Release_Notes_ixgben-1.6.5.txt (release notes).

    Alternatively, you can use the following command to directly install it to your internet connected ESXi host, in an SSH session:

    esxcli software vib install -v https://cdn.tinkertry.com/files/INT_bootbank_ixgben_1.6.5-1OEM.600.0.0.2768847.vib

    You should see something like this when it's done

    Installation Result
       Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
       Reboot Required: true
       VIBs Installed: INT_bootbank_ixgben_1.6.5-1OEM.600.0.0.2768847
       VIBs Removed: VMW_bootbank_ixgben_1.4.1-2vmw.650.1.26.5969303
       VIBs Skipped:
    1dot6dot5-x557-driver

    but you'll need to disable the native driver before it will work, explained here.

    For reference, you can remove un-used VIBs. Here's how to remove 4.5.3, for example:

    [root@xd-1567-5028d:~] esxcli software vib remove --vibname=ixgbe
     [NoMatchError]
     No VIB matching VIB search specification 'ixgbe'.
     Please refer to the log file for more details.
    [root@xd-1567-5028d:~] esxcli software vib remove -n net-ixgbe
    Removal Result
       Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
       Reboot Required: true
       VIBs Installed:
       VIBs Removed: INT_bootbank_net-ixgbe_4.5.3-1OEM.600.0.0.2494585
       VIBs Skipped:
    [root@xd-1567-5028d:~]

    Source.


    See also

    2015-11-03_12-48-24