Supermicro SuperServer Xeon D-1500 BIOS 1.1b released, IPMI 3.31 released, heralding the arrival of HTML5 iKVM

Posted by Paul Braren on Aug 8 2016 (updated on Nov 3 2016) in
  • HomeLab
  • HomeServer
    • 23 Comments

    AUG 15 2016 I'm all set now, networking is fine on the SYS-E200-8D when using BIOS 1.1b and IPMI 3.26. Had to turn off the system for longer after both upgrades were complete for it to suddenly work just fine. Testing of Intel/VMware's suspect 4.4.1 10GbE VIBs is now starting, now that I have a 10GbE switch...

    AUG 13 2016 FYI, BIOS 1.1b does seem to work fine with Windows Server 2016 TP5 networking on the 5018D-FN4T, but on the SYS-E200-8D, BIOS 1.1.b seems to prevent Windows and ESXi networking from working at all. Luckily, my BIOS 1.1a direct download link is still available. I'm testing the back-level procedure on the E200 now...

    AUG 11 2016 - IMPORTANT UPDATE - For VMware ESXi 6.0U2 users, there may be a network issue with BIOS 1.1b, I strongly advice you hold-off on upgrading until this has been resolved. See also Michael K's comment below, and the new video where I demonstrate a workaround for the issue with network enumeration (re-ordering), while acknowledging that I still have 10GbE testing to do.

    AUG 10 2016 - I've now upgraded my systems, and found that the FlexATX SYS-E300-8D accepts the same 3.31 IPMI code, but has a different BIOS than the rest of the SuperServer Mini ITX Xeon D-1500 family. Article below updated accordingly, with video of these 4 system upgrades now added.

    Here's the current Supermicro Xeon D-1500 system line-up in form factors suited for home labs:

    superservers-ready-for-bios-upgrades.JPG
    I'll take one of each, please! First step, update the firmware. Wiredzone does this for Bundle customers.

    TinkerTry'd Supermicro Xeon D-1500 Systems.

    If you click on the BIOS and IPMI links on each system and motherboard page above, you'll get mixed results today:

    SYS-5028D-TN4T
    Mini Tower    		 SYS-5018D-FN4T
    1U    		 SYS-E200-8D
    Mini 1U    		 SYS-E300-8D
    Mini 1U
    BIOS 1.1b 1.1b 1.1b 1.0a
    IPMI 3.26 3.26 3.26 3.31

    What I don't know is whether Supermicro is done with testing across the Xeon D-1500 family of products and has just falled a little behind on cleaning up the web site, or whether they simply haven't yet finished testing. Or maybe the product line's BIOS and IPMI levels won't be unified until the next major release, say BIOS 1.2. Note that the SYS-E300-8D is slightly different animal, with a bigger Flex ATX versus Mini ITX design, and SFP+ instead of RJ45 10GbE. Whatever the reason(s) for the current discrepencies, I'm boldly going ahead with updating to the following levels on all 4 systems very soon:

    Download for Xeon D Supermicro SuperServers

    These have been tested by me with all 4 Mini-ITX Xeon D systems pictured above, but as far as I know, not yet by Supermicro.

    Download for FlexATX SYS-E300-8D

    I'll be peforming the upgrade using this method for the BIOS, and this method for IPMI.

    Disclaimer

    Warning
    If you decide to proceed with this upgrade anyway, and it's not listed on your product page, you are proceeding at your own risk, and you may have issues obtaining support from Supermicro, despite their public tweet about this:

    736316284304949249

    I'm in temporary possession of these 4 form factors through mid-October, as I attempt an all-flash vSAN, prepare for VMworld 2016 Las Vegas, and compare the systems with one-another. Comparisons will be difficult with differing code levels, so I'm going with these BIOS settings, and the default IPMI settings that automatically get set from the upgrade process. Based on some reader feedback, I may also need to look into some alterations of these settings, we'll see.

    Video

    Supermicro SuperServer Xeon D-1500 BIOS 1.1b and IPMI 3.31 Upgradeapolooza. Recorded on August 9, 2016.

    Release Notes

    Looking for release notes? For Supermicro, release notes can be a bit tough to come by. I managed to have a copy emailed right to me today, from a TinkerTry reader. These are not confirmed to be official release notes, and aren't found anywhere on Supermicro's site. They are provided to you on an as-is, FYI basis only, with only formatting tweaks made for legibility/mark-down compatibilty:

    BIOS 1.1b Release Notes

    1. Update SMBIOS information according to SMBIOS template ver 0.7
    2. Update RC 2.2
    3. SMBIOS test case 320 failed.
    4. ASPM support setup item doesn't work properly.
    5. Incorrect MAC information shown in IPMI WebUI when all onboard LAN disabled
    6. Update SPS to 03.00.03.025

    BIOS 1.0 through 1.1a Release Notes

    X10SDV-(TLN4)F release note.

    CPU Microcode | Option ROM/EFI driver | Intel RC/TXT/ME

    CPUID FAMILY | Aspeed VBIOS : 0.98.03 | RC Package :2.1.0
    50661 Broadwell DE V1 :F1000008 | Aspeed Graphics Driver : 0.98.03 | SPS : 03.00.03.023
    50662 Broadwell DE V1 :0000000F | Intel Giga PXE : 1.5.13 | BIOS ACM : 20150917 (1.2.2)
    50663 Broadwell DE V2 :0700000A | Intel iSCSI(i350) : 2.8.16 | SINIT ACM : 20150917 (1.2.2)
    50664 Broadwell DE Y0 :0F000009 | Intel Giga UEFI driver : 6.7.01 |
    | Intel RSTe SATA UEFI driver : 4.3.0.1018 |
    | Intel RSTe SATA RAID Option ROM : 4.3.0.1018 |
    | Intel Boot Agent XE : 2.3.45 [V1]|
    | Intel 10G UEFI Driver : 4.9.09 [V1]|

    BIOS feature | Support

    OOB Flash BIOS | Y
    OOB Update Setting | Y
    OOB Change SMBIOS | Y
    InBand Flash BIOS | Y
    InBand Update Setting | Y
    InBand Change SMBIOS | Y
    InBand SMI E7h supported | Y

    BIOS : X10SDVF6.505 Release Date : 05/05/2016
    BIOS revision: 1.1a BIOS Engineer : Wenhung Huang
    IPMI revision: 333
    BIOS Checksum: D063B22Bh
    BIOS Change:

    1. Restore the RTC date to build date when CMOS is cleared.
      Description : None.
      Validation :

      1. Clear CMOS.
      2. Check if the system date has been restored to BIOS build date.

    2. Fix Samsung NVMe device cannot install EFI OS.
      Description : Cannot install EFI OS with Samsung NVMe device.
      Validation : Install EFI OS and make sure boot to OS successfully.

    3. Add IPMI VLAN support
      Description :
      Validation :

      • Check if IPMI > BMC Network Configuration > VLAN can work properly.

    4. System always stop with "Press F1 to Resume..." message when "Wait For "F1" If Error".
      Description :
      Validation :

      • Connect HDD with SMART Error.
      • Check if system stops with message "Press F1 to Resume..."
      • Disable "Wait For "F1" If Error"
      • Check if system keep POST after prompted the error.

    5. Update TCG 2.0 module label08
      Description : Integrate from Grantley #3991 for other bug fixed.
      Validation :

      • Check if Windows 10 can enable the Bitlock properly.
      • Check if TXT function can enable properly.

    6. SRIOV cannot be enabled correctly in Windows 2012 HyperV
      Description : Expose ASPM to fully support SR-IOV
      Validation :

      • Check if SR-IOV can be enabled properly in Windows 2012 HyperV

    7. Fixed system reboot when enter some LSI card (AOC-S3108L-H8iR) setup screen problem.
      Description : system reboot when enter some LSI card (AOC-S3108L-H8iR) setup screen.
      Validation : Plugged AOC-S3108L-H8iR into PCIe slot and check if be able to enter OPROM setup screen.

    8. Update RC 2.1.0 and microcode M1050663_0700000A, M1050664_0F000009
      Description : Intel RC update.
      Validation : None.

    9. D-1587 CPU string is incorrect.
      Description : Production SoC only.
      Validation :
      • Check if the CPU string is correct in BIOS Setup and Windows system information.

    BIOS : X10SDVF6.302 Release Date : 03/02/2016
    BIOS revision: 1.1 BIOS Engineer : Wenhung Huang
    IPMI revision: 2.26
    BIOS Checksum: 6E2A4E57h
    BIOS Change:

    1. Update ACM 1.2.2
      Description : None.
      Validation :

      • Use production version SoC (The MSR 0xC80 bit0 should be "0")
      • Check if TXT "getsec -l sen" can execute without problem.

    2. Fixed system hang when trigger SERR/PERR.
      Description : System hang when trigger SERR/PERR.
      Validation :

      1. Enable "PCI PERR/SERR Support" and "PCI-Ex Error Enable" in Setup.
      2. Make sure system doesn't hang when trigger SERR and PERR.
      3. Check if SERR and PERR event log exist in Setup and IPMI Web.

    3. Adjust detection M.2 device function.
      Description : Modify conditional expression to detect M.2 device.
      Validation : Add M.2 device and make sure it can be detect in boot order.

    4. Support display BMC IP when available.
      Description : None.
      Validation :

      1. Connect the ethernet to BMC
      2. Check if BMC IP prompted on screen.

    5. Update microcode M1050664_0F000006, M1050662_0000000F, M1050663_07000009
      Description : None.
      Validation : None

    6. Update RC 2.0.0
      Description : None.
      Validation :

      1. Perform S4/C burn stress test on both V1/V2 SoC.

    7. Add Setup item to skip BIOS to IPMI command during POST.
      Description : Embedded specific function per PM request
      Validation :

      1. Check if the Setup item works properly.

    8. Update SPS kit to 03.00.03.020
      Description : None
      Validation : None

    9. There's no SERR/PERR after boot into windows.
      Description : None.
      Validation :
      1. Plug ASC-29320LPE.
      2. Boot to Windows
      3. Write 01/04 to the register offset 0x43 of ASC-2932 controller.
      4. Check if there's SERR/PERR log present at IPMI event log/Setup event log.

    BIOS : X10SDVF6.107 Release Date : 01/07/2016
    BIOS revision: 1.0c BIOS Engineer : Wenhung Huang
    IPMI revision: 2.26
    BIOS Checksum: D0146D7Ch
    BIOS Change:

    1. Update TCG 2.0 module.
      Description : To support both TPM 1.2/2.0
      Validation :
      a. Check if "Advanced > Trust Computing" exists when TPM 1.2 attached.
      b. Check if TPM state can be changed by BIOS Setup
      c. Check if OS "TPM Administrator" can change the TPM State properly.
      d. Check if BitLocker can work properly.
      e. Check if "Advanced > Trust Computing" exists when TPM 2.0 attached.
      f. check step (b) ~ (d) with TPM 2.0.

    2. Fix system will hang 0x92 after modify some LAN items.
      Description : System will hang 0x92 when set onboard LAN OPROM to EFI and network stack to enabled.
      Validation : Check system is boot up normally after change some LAN items.

    3. Always expose SR-IOV Setup item
      Description : The item is hidden since the onboard 10G sighting in V1 stepping but it's still support by onboard 1G and add-on card.
      Validation :

      1. Boot to Setup.
      2. Check if "Advanced > PCIe/PCI/PnP Configuration > SR-IOV Support" exists.

    4. Update RC 1.9.5
      Description : RC for V2 stepping support.
      Validation : N/A

    5. Update microcode m1050663_07000005,m1050664_0f000004
      Description : CPU microcode for V2 stepping support.
      Validation : N/A

    6. Update BIOS/SINIT ACM.
      Description : TXT ACM for V2 stepping support.
      Validation :

      1. Enable TXT support.
      2. Check if "getsec -l SEN" can execute properly.

    7. Remove prefix "6:" from ME version.
      Description : None
      Validation : Check if Setup ME version shown properly.

    8. Add second entry of SMBIOS type 39 to support redundant power.
      Description : None
      Validation :

      1. Check if there are two SMBIOS type 39 entries.
      2. Check if the power supply information correct.
    9. System hang at POST 0x00 during C burn with V1 SoC
      Description : None
      Validation :
      1. Perform C burn test for both V1 and V2 MBD

    BIOS : X10SDVF5.909 Release Date : 09/09/2015
    BIOS revision: 1.0b BIOS Engineer : Wenhung Huang
    IPMI revision: 1.93
    BIOS Checksum: 67BE9EDDh
    BIOS Change:

    1. There's no HDD information in Setup when set SATA to RAID mode.
      Description: Bug fixed.
      Validation:
      Boot Setup and set SATA mode to RAID
      Save and exit.
      Boot to Setup and check if the SATA HDD information shown.

    2. Above 4G decoding support.
      Description: Function implement.
      Validation:
      Boot to Setup.
      Set Advance > PCIe/PCI/PnP Configuration > Above 4G Decoding to Enabled.
      Attached add-on card with above 4G decode.
      Boot to OS and check the add-on card resource arrangement.

    3. M.2 NVMe support.
      Description: Function implement.
      Validation:
      Connect M.2 NVMe.
      Check if BIOS can recognize and boot to the device.

    4. Add SATA Frozen Setup item.
      Description: Function implement.
      Validation:
      Boot to Setup.
      Check if Advance>SATA Configuration>SATA Frozen present and default disabled.
      Boot to Linux
      Use "hdparm" to dump HDD information and HDD should not be frozen.
      Reset the system and boot to Setup.
      Set SATA Frozen to enable then save and exit.
      Boot to Linux
      Check if HDD is frozen.

    5. Hide the SR-IOV Setup item for V1 stepping.
      Description: Patch for Intel sighting #b287152: "Single Root- IO Virtualization feature in not enabled on the integrated 10G LAN controller"
      Validation:
      Boot to Setup
      Check if the CPU Version in Advance > CPU Configuration
      Check the "SR-IOV" item should not present at Advance > PCIe/PCI/PnP Configuration if CPU is D-1520/D-1540.

    6. System hang at POST screen 0x00 with X10SDV-8C-TLN4F
      Description: Rollback RC to 1.00..
      Validation:
      Run Cburn test/S4 stress test with X10SDV-8C-TLN4F.

    7. Force TPM enable after cleared by AMISCE.
      Description: Function implement.
      Validation:
      Use AMIBCP to enable the Advance > Trusted Computing > Enable TPM After cleared.
      Boot to shell
      Use AMISCE to change the Pending Operation to TPM Clear
      Check the TPM state should be Enabled after status change complete.
      Manual change the state and the state should change correct after reset.

    8. SCELNX support.
      Description: Function implement.
      Validation:
      Check if BIOS setting/password can export and import by SCELXN.

    9. Add second SMBIOS type 39 entry to support redundant power.
      Description: Function implement.
      Validation:
      1. Check if there are two SMBIOS type 39 entries present.
      2. Check if the power supply information correct.

    BIOS : X10SDVF5.527 Release Date : 05/27/2015
    BIOS revision: 1.0a BIOS Engineer : Wenhung Huang
    IPMI revision: 1.93
    BIOS Checksum: 2BC76942
    BIOS Change:

    1. SMLink 1 function failed.
      Description: BMC failed to issue command to ME through SMLink 1.
      Validation: Check command "Get ME Timestamp" - "ipmitool -H -U ADMIN -P ADMIN -b 0 -t 0x2c raw 0xa 0x48" works.

    2. Update TXT module to v1.0.2
      Description: Module update.
      Validation: Perform Intel TXT provision test.

    3. SATA RAID mode support
      Description: Integrated SATA RAID option/UEFI driver.
      Validation: Check if SATA RAID function works properly.

    4. Force to recovery mode if FV_MAIN is crash.
      Description: [BugFixed] system doesn't enter recovery when FV_MAIN is crash.
      Validation: 1. Use AFUDOS to update BIOS and remove AC power while programming FV_MAIN.

      1. Check if System enter Recovery automatically.

    5. Performance enhancement for FFS check.
      Description: POST time reduce.
      Validation: Check if system stops at POST code 0x4F over 10 second.

    6. Issue#61771 iSCSI Attempt missing after reboot
      Description: Bug fixed.
      Validation: 1. Add two iSCSI attempt.

      1. Save and restart system
      2. Check if the two attempt exist.

    7. SUM jumperless support.
      Description: Bug fixed.
      Validation: Check if SUM can update the FDT with jumperless.

    8. 10G PXE/UEFI driver support.
      Description: Driver update.
      Validation: Check if 10G PXE works properly in both Legacy/UEFI.

    BIOS : X10SDVF5.420 Release Date : 04/20/2015
    BIOS revision: 1.0 BIOS Engineer : Wenhung Huang
    IPMI revision: 1.93
    BIOS Checksum: 4743B80C
    BIOS Change:

    1. Initial version.

    See also at TinkerTry

    All Comments on This Article (23)

    william adams

    Apparently its reproducible. Shutdown the CVM virtual machine and then running ethtool eth0 and ethtool eth1 it will crash the box. Fun fun fun!

    I have posted this in the Nutanix CE forums as well:
    http://next.nutanix.com/t5/Discussion-Forum/Ugly-crash-quot-Dead-loop-on-virtual-device-br0-fix-it-urgently/m-p/12467#U12467

    William

    william adams

    I had a failure of a node again this morning where it lost link state. I was rebooting the server a bunch of times to see if it would eventually link up and all of a sudden before I issued the shutdown command, I ran ethtool eth0 and saw no link as expected and then the server crashed with an ugly error "Dead loop on virtual device br0, fix it urgently!" and the box locked up. Strange. Again this is Nutanix CE. I have a feeling there is something up with the driver because link state is lost on both NICs at the same time.

    Paul Braren | TinkerTry.com

    There are no BIOS settings that change the order that NICs are presented to the OS. I've reported the issue, and am hoping they get it right with BIOS 1.2 (or whatever the next release is).

    Paul Braren | TinkerTry.com

    See also my new comment here:
    https://TinkerTry.com/close-look-at-netgear-xs708t-100nes-including-decibels#comment-2857219495

    Paul Braren | TinkerTry.com

    Now that I have my SYS-E200-8D non-functional networking issue squarely behind me (see update above https://TinkerTry.com/supermicro-superserver-xeon-d-bios-1-1b-released), I can start some 10GbE tests under VMware ESXi 6.0U2 this week, to replicate the issues you're seeing, while continuing my dialogue with Supermicro about this.

    william adams

    No, it appears to be an issue with the actual Intel driver itself. Randomly doesn't link up, then magically decides to get a physical link and propagate the link state. I have the issue when connected to the same 10Gbe netgear switch you have as well as a Cisco 4900. Its a pain in the butt. :(

    Paul Braren | TinkerTry.com

    Wtih BIOS 1.1b, it would seem that the X552/X557 ports gets seen by ESXi first, instead of after the I-350 1GbE ports. Perhaps this re-ordering messed up your Acropolis?
    https://youtu.be/MdoupXFKUvU

    Paul Braren | TinkerTry.com

    I've also now published a brief video showing how I quickly determine why I can't get in to do ESXi management, and how I fixed the problem (a merely workaround, I hope BIOS 1.2 or whatever the next release does things right, like 1.1a did).
    https://youtu.be/MdoupXFKUvU

    Paul Braren | TinkerTry.com

    Here's the new way it enumerates the network devices it finds:
    after the crazy easy instructions to install the 4.4.1 VIB were followed weeks ago:
    https://TinkerTry.com/how-to-install-intel-x552-vib-on-esxi-6-on-superserver-5028d-tn4t#expert-short-instructions

    Paul Braren | TinkerTry.com

    I have finally gotten home, and quickly found an issue with the way ethernet devices are enumerated. Suddenly, my existing install of VMware ESXi 6.0U2, that worked fine on BIOS 1.1a, now put the 10GbE interfaces first in the list, since I made the move to BIOS 1.1b yesterday. Ouch! Easily remedied by using iKVM to get to the DCUI and change which NIC is used, but yeah, that's a pretty nasty problem. I've also engaged Supermicro on the matter.

    Tonight, I have just one active on my 10GbE switch, working tomorrow on getting a second host going, for 10GbE testing between them.

    FYI, I'm not using the Cisco switch that is apparently known to have issues with Xeon D's X552/X557 https://communities.intel.com/message/412677#412677
    I'm using this Netgear instead (for now) https://TinkerTry.com/close-look-at-netgear-xs708t-100nes-including-decibels.

    I'll update the article(s) and Disqus conversations when I know more, and I promptly warned people to refrain from BIOS 1.1b early this morning: https://TinkerTry.com/supermicro-superserver-xeon-d-bios-1-1b-released

    Pardon and William, I can't thank you enough for bringing this to my attention, and I'm confident we'll work this out with Supermicro.

    Paul Braren | TinkerTry.com

    Cisco 10 GbE switch owners, see also https://communities.intel.com/message/412677#412677

    william adams

    I am using Nutanix CE which uses CentOS 7.1 core and I have had this NIC link issue since yesterday when the NIC simply quit linking to the switch. Not sure if its the same reason as the issue in ESXi or not. Recompiling and updating ixgbe drivers hasn't helped.

    Paul Braren | TinkerTry.com

    Long shot, but try hooking th 10GbE ports up to a DHCP LAN and see if they then come alive.

    Paul Braren | TinkerTry.com

    I've added words of warning to the top of the article above:
    https://TinkerTry.com/supermicro-superserver-xeon-d-bios-1-1b-released
    thank you Pardon and Michael for bringing this to my attention!

    I'm also forwarding this thread to a Supermicro contact, but won't be able to start my own recreate until tonight. If either of you can also contact the SuperServer hotline before me at https://www.supermicro.com/24Hour/24hour.cfm that would be appreciated.

    Pardon Pierre-E

    I don't know if it's the same error, I can see my two X552 Intel 10G NICs but ESXi shows them as disconnected.
    I will try to connect my computer directly to ESXi and add screenshots.
    Do you know if supermicro support can help here?
    Regards,

    Pardon Pierre-E

    I'm using ESXi 6.0U2 yes.
    I will post screebnshots of my problem ASAP.
    Thank for your help.

    Michael K

    Your boot order has probably been deleted because all settings are reset after a bios upgrade.


    Regarding your second problem:
    I had a quite similar outcome on one of my servers. ESXi wouldn't display my NIC even though the correct driver was installed. I then did a BIOS upgrade from 1.0b to 1.1b, but it still didn't work. Eventually, I tracked it down to the NIC driver which wasn't loading because of the error "EEPROM checksum not valid." Reflashing didn't fix it- so unfortunately I'm stuck there now.


    You can check for the same error in /var/log/vmkernel.log

    Pardon Pierre-E

    6.0U2 yes.

    I will post some screenshots here:

    https://communities.vmware.com/message/2613806#2613806
    Do you know Paul if there are BIOS settings for NICs which can cause the issue?
    I'm using the X552 10G NIC for the ESXi management.
    Regards

    Paul Braren | TinkerTry.com

    All 4 systems now testing out fine at IPMI 3.31, see:
    https://TinkerTry.com/supermicro-superserver-xeon-d-bios-1-1b-released

    Paul Braren | TinkerTry.com

    I will try to see if I can recreate this behavior as soon as possible. Are you on ESXi 6.0U2, or some later build?

    Pardon Pierre-E

    I have upgraded the bios of my SYS-5028D-TN4T yesterday but there is a problem:

    First: the hard disque boot order has been deleted

    Second and the most problematic: ESXi is unable to acquire his networking. I have a http://0.0.0.0 displayed. I tried to set the IP manually without success.

    The problem is NIC are shown disconnected...

    Does anyone have the same issue?

    Thank you guys for your help.

    Regards,

    Pierre-Emmanuel

    Paul Braren | TinkerTry.com

    Yes, the SYS-5028D-TN4T and SYS-5018D-FN4T are both Xeon D-1540/1541, and their product pages do list BIOS 1.1b, but don't yet list IPMI 3.31. That said, 3.31 does already work on the SYS-5028D-TN4T, see https://TinkerTry.com/supermicro-superserver-5028d-tn4t-testing-bios-1-1-a-and-ipmi-3-3-1

    Mahmood

    Good,
    Is there any new updates for Xeon D-1541 Models ?