Easy fix for Supermicro and other Xeon D systems experiencing SATA3/AHCI slowdown on ESXi 6.5

Posted by Paul Braren on Nov 16 2016 (updated on Nov 18 2016) in
  • Virtualization
  • ESXi
  • HomeLab
  • Storage
  • Article is a work in progress, revisit/refresh for updates. So far, I've been unable to replicate his findings on a new ESXi 6.5 install, seeing normal SATA3 speeds. Others in different circumstances may still run into this slowdown, reasons why not known yet. Details of the ongoing investigation will be added below. So far, it appears mostly folks upgrading to 6.5 encounter this issue, but there is now a report of a fresh install experiencing this too, and Intel Avoton C2758, and an older Intel NUC. Anthony's quick fix seems to resolve all situations. Please leave a comment and share your experiences with us. Tbhe more details, the better!


    Before-fix-only-one-port-shown
    There's the symptom, only one SATA3/AHCI port seen by ESXi 6.5's slow default driver. C'mon VMware/Intel, really?

    Yesterday, VMware released ESXi 6.0 Build Number: 4564106. If that's what you're considering, and you:

    • use 3.5 or 2.5" drives
    • are planning to install or upgrade to ESXi 6.5 (released just yesterday) onto your Xeon D system, including Supermicro SuperServers/Wiredzone Bundles)
      this article is for you!
    featured

    The Fix

    It's all detailed in this wonderfully technical article by fresh Veeam recruit Anthony Spiteri:

    Yep, show your support by reading the source article, he did the work:

    NOTE: The M.2 NVMe 950 PRO by Samsung is apparently not affected, tested it myself. There are questions about the Samsung SM951, so I'm working on verifying that little M.2 gumstick SSD next.

    798227540875689984

    Highly likely this fix applies to other Xeon D systems as well. Have a look at the (early days/growing) ESXi 6.5 VMware Compatibility Guide for Xeon D here, and the much larger 6.0U2 list here. Of course, it's expected that all Xeon D systems that Supermicro makes will be on the 6.0U2 (and now 6.5) soon, as explained by Supermicro during my visit to their VMworld 2016 US booth.

    Yep, that simple, a one-line fix/reboot, and you're back to full speed ahead.

    Like Anthony, I'm proud to be also be a Veeam Vanguard, an amazing group that regularly gives back to the broader virtualization community. I really appreciate his helping out his fellow Xeon D owners like this!

    I frankly don't recall coming across any of this strangeness during my fairly extensive beta testing. Kinda strange that VMware would do this to their partners, I'm sure there was a reason?

    Once validated/reproduced, I'll work on getting this reported to VMware in some official way without having to open a $299 ticket. Ideally, squaring this away before their next 6.5.x release, see also VMware's colorful history, as told by homelaber Florian Grehl.

    virten.net's-image-vsphere65-days-since-last-release
    Days since prior VMware release

    Next up, testing out the 10GbE support for the Intel X552/X557 10GbE chipset that's on these systems. Tried to get that baked right into 6.5, and likely future releases will have it included. Meanwhile, also a very quick fix, with no reboot required.


    Nov 17 2016 Update

    The systems under test are the following:

    • SYS-5028D-TN4T 12 core Xeon D-1567
    • SYS-5028D-TN4T 08 core Xeon D-1541

    Each system was configured identically, as follows:

    • BIOS 1.1c
    • IPMI 3.46
    • fresh install of ESXi 6.5 (not upgrade) on 32GB USB
    • freshly VMFS 6 formatted SSDs datastore, seen below
    • Windows 10 Pro Anniversary Edition UEFI thin provisioned VM was cloned from one SSD to the other
    • Pictured at left, NVMe, at right, SATA3 SSD. Click to view larger, again to view native resolution, using arrow keys to easily compare.
    Test results for speed tests of NVMe versus SATA3 on Xeon D-1567
    SYS-5028D-TN4T 12 core Xeon D-1567
    Test results for speed tests of NVMe versus SATA3 on Xeon D-1541
    SYS-5028D-TN4T 08 core Xeon D-1541
    VMFS-Confirmed
    native-vmware-device-driver-listed
    Both the Intel Lynx Point Series Chipset driver and the native VMware device driver were loaded, but unlike Anthony's experience, this didn't seem to cause any performance issues on my systems.

    Video

    Here I demonstrate my first attempt to replicate Anthony's findings.

    Attempt to replicate possible SATA3 speed issues under ESXi 6.5

    Nov 17 2016 Update 2

    Preliminarily, it appears that this issue affects folks who upgraded from ESXi 6.0U2 to 6.5. So we might be all set here, folks seeing the slow-down do Anthony's fix, then they're all set.

    799469380421570560

    See also at TinkerTry


    See also