Supermicro SuperServer Workstation revisited with ESXi 6.5 - replacing vmkusb VIB with ESXi 6.0 version restores Windows 10 VM's USB sound

Posted by Paul Braren on Jun 19 2017 (updated on Jun 24 2017) in
  • ESXi
  • Virtualization
  • HowTo
  • HomeLab
  • vSphere-Client-view-of-VM-hardware
    That's a lot of USB devices mapped to my Windows 10 Creators Update VM.

    Back in March of 2017, I needed to go back to using my SuperServer Workstation as my daily driver Windows 10 workstation, in large part because of excessive video render times for 4K video production workflow. This mean it was time to revisit how well this VM worked with the latest ESXi 6.5.0d, determining whether all features and functions behaved as they did back in the ESXi 6.0 days, when I first wrote this article:

    This SuperServer Workstation hybrid combo is a niche build, and not a turn-key solution. Yes, I admit that Bundle 2 is much more popular for good reasons: it's sold without Windows 10 and without a GPU, a better choice for the vast majority of use cases. This is especially true for virtualization enthusiasts, who generally don't need or want a watt-burning GPU. But this article is about those who very much do want a compact GPU that takes up only one slot for their "workstation" VM, running right off the PCIe bus, using no more than 70 watts max. The "workstation" comes from the use of an attached keyboard, mouse, and monitor, so your vSphere 6.5 datacenter also becomes your Windows 10 workstation, see also How to locate your triple monitor (up to 2K) PC 20 feet away for less noise and more joy.

    With so many months having gone by since these SuperServer Workstations began shipping, it's high time I revisit this little screamer, and let you know what discoveries I made when moving from vSphere 6.0 to vSphere 6.5, and when determining whether any suitable replacements for the VisionTek AMD 7750 GPU card had arrived, with the promise of quiter operation.

    AMD Radeon Pro WX 4100 - Fail

    B01MQHEAYE

    Notably, I failed to get the newer and more powerful and quieter AMD Radeon Pro WX 4100 Workstation Graphics Card working properly for passthrough, when using the latest system BIOS 1.2, as explained here. Yellow bangs through the Display adapter listed in Windows Device Manager, and/or occasional PSODs. Maybe some manual tweaks to the .VMX file will do the trick someday, if somebody figures out what those tweaks are.

    NVIDIA K1200 - Fail

    B00UPHAT2C

    As for NVIDIA, well, the spiffy looking and quiet sounding K1200 was a no-go as well, exhibiting the dreaded yellow bangs through Device Manager. Yes, NVIDA still doesn't want you to use anything but the proper NVIDIA Grid product line for vGPUs carved up among many of your VMs, explained in their video. NVIDIA has historically not been interested in allowing easy VMware ESXi passthrough for either their Quadro (workstation) or GeForce (gaming) product lines.

    VisionTek AMD 7750 - Still a win!

    The noise of the always-on fan can be reduced by creatively using a fan speed reducer from nearly any Noctua cooling fan, installing it inline. Noctua calls it a Low-Noise Adaptor (L.N.A.), sold alone as the NA-SRC10 3-Pin Low-Noise Adaptors. Clumsy to get attached firmly, but very effective once in place, with my GPU still managing to stay cool to the touch even after long benchmarks like FurMark that doled out heavy abuse.

    B009ZQ5HW6

    I'm much happier using my Xeon D for daily use when compared to any laptop, even that lovely work-issued Core i7 Dell Precision 5510 with 1TB NVMe, worth around $3000 total. Why? Because it's still just a mobile CPU, the 4 core 8MB cache i7-6820HQ. Compare that with my 16 core 18MB cache Xeon D-1567 in my SuperServer Bundle 1, as detailed in this Intel ARK comparison table. These specs really do matter. Routine content creation tasks like Camtasia 9 video renders use all those cores and now take me much less time. See also my 4K video render measurements with various core counts here:

    4K-video-render-stressing-Xeon-D-cores-by-TinkerTry

    Backstory on the new USB issues noticed in 6.5.x

    It would seem that VMware ESXi doesn't officially support ISOC (isochronous) for passthrough, used for video and audio devices, see:

    If you find that you have one or both of these issues:

    1. attaching the USB 2.0 or the USB 3.0 controller or both controllers to your ESXi 6.5.x VM doesn't allow proper function of attached USB devices, using vSphere Client to attach the controller then the host's USB sound device to the VM, on Windows Device Manager, the display continually shows sound devices appearing and reappearing
    2. your Anywhere USB/2 device doesn't maintain connection to your keyboard and mouse properly, requiring you to RDP in remotely to get to Windows Device Manager, System devices, then right click on Network Attached USB Enumerator to disable then re-enable it

    good news, the workaround is easy to implement, and has kept things working well for me for over a month now. Flawless skip-less sound in Spotify in my VM for example, even when the physical server is under load.

    Just follow along with the one-line esxcli command to disable the vmkusb driver, as documented here:

    The sound workaround

    1. Open an SSH session (eg. PuTTY) to your ESXi 6.5.x server
      (if you forgot to enable SSH, here's how)
    2. Turn on maintenance mode, or ensure you've set your ESXi host to automatically gracefully shutdown all VMs upon host reboot, or shutdown all the VMs gracefully that you care about, including VCSA.
    3. Disable the vmkusb driver - Paste the one line below into your SSH session, then press enter:
      esxcli system module set -m=vmkusb -e=FALSE
    4. Reboot your ESXi host - Paste the one line below into into your SSH session, then press enter:
      reboot

    Keep in mind you may need to redo this fix after any future upgrades to later ESXi versions.

    Here's my current observations, fresh off about 2 months of heavy use.

    The good

    2017-06-13_13-46-08
    Premium CPU in Dell Precision 5510 versus Xeon D-1567 in SuperServer Bundle 2 12 Core.
    1. snappy UI, easy triple monitor support
    2. CPU speed and multitasking abilities are impressive
    3. extreme versatility compared to any laptop
    4. many drive bays for your storage needs, and a fast M.2 slot for exceptional M.2 NVMe storage performance when used as an ESXi datastore for your Windows 10 VM
    5. video render times using Camtasia 9 are greatly reduced over any laptop
    6. having great performance with assigning 20 vCPUs to this powerhouse VM I use for dozens of hours per week
    7. sound quality of my USB to Digital Coaxial and headphone jack adapter is great
    8. I've discovered that turning VGA to offboard in the BIOS, for hand-off of video from onboard VGA port to offboard GPU card, isn't necessary for stable VM operation, I may want to revisit the build procedure Wiredzone follows when prepping these systems for shipment

    The noteworthy gotchas

    Yes, full disclosure here, this is not a VMware supported way to run a VM, we already knew this. Only certain USB devices are support, and really only products like NVIDIA GRID are properly supported for use as vGPUs carved up across your most important VMs. But this is a home lab, where pushing technology forward with what's possible on budget can be fun, especially if somebody has figured out the bumps in the road before you.

    1. you need to have full daily backups of your precious VM you're using as your workstation, free and easy options include NAKIVO Backup & Replication 7 and Veeam Agent for Microsoft Windows v2.
    2. approximately every 5th reboot of the Windows 10 VM that is also my Windows 10 triple-monitor workstation, I encounter an issue with the VisionTek AMD 7750 GPU card not being passed through at all for mysterious reasons, requiring me to reboot the SuperServer's ESXi itself, using vSphere Client on another network attached PC
    3. on my attached triple monitors, I can't easily view BIOS screen and early Windows boot issues, such as BSODs, requiring me to use VMRC on another system for problem determination
    4. currently I've disabled AMD's sound over DisplayPort and HDMI, to avoid nuisance default sound reassignment to one of those devices, since I don't use my monitors for sound
    5. occasionally, 2-3x per-constant-use-hour, my mouse seems to drop some packets randomly for about 1/3 of a second, no big deal, and this isn't associated with any CPU or disk IO load
    6. can't snapshot or vMotion the VM, the same old restrictions ESXi VMs have for RDM users
    7. adding USB 3.0 devices a little clumsier than simply plugging in, need to take steps to map it to the VM as well, sometimes the VM needs to be shut down for this to work, gladly these mappings persist through reboots of VM or ESXi host, this guy has an easier way, see Running a virtual gaming rig using a Xeon D server, a GFX 750Ti, PCI passthrough and a Windows 10 VM, but it's USB 2.0 only, and only tested on ESXi 6.0
    8. can't sync iPhone with iTunes via a physical USB 3.0 connection attached to the host/server, mapped to the Win 10 VM using the ESXi UI (Apple device seen in Device Manager, but not in iTunes)
      B01LY3Y9PH
    9. avoid RDM mappings of the C: drive for this UEFI VM for much more robust booting, I'm quite happy now with thick provisioned 1.7TB virtual drive that lives on my VMFS 6.81 formatted Samsung 960 PRO 2TB M.2 NVMe SSD
    10. if you turn SR-IOV on or off in the BIOS, you'll need to reconfigure passthrough in your ESXi host, reboot, then re-add the PCI devices back to your VM settings, so they'll show up again in Device Manager as the expected 'AMD Radeon HD 7700 Series' video device and the 'AMD High Definition Audio Device', and for me, I just right-click disable the audio device, as I don't use my monitors speakers

    Screenshots

    This are the latest AMD Radeon Settings, showing I'm using AMD Radeon Software Crimson Edition 17.1.1 Driver for Windows® 10 64-bit on my Windows 10 Creators Update Workstation VM.

    AMD-RADEON-SETTINGS-Overview-for-Visiontek-7750-in-Win-10-VM-under-ESXi-65-on-Supermicro-SuperServer-Bundle-1-by-TinkerTry
    AMD-RADEON-SETTINGS-Software-for-Visiontek-7750-in-Win-10-VM-under-ESXi-65-on-Supermicro-SuperServer-Bundle-1-by-TinkerTry
    AMD-RADEON-SETTINGS-Hardware-for-Visiontek-7750-in-Win-10-VM-under-ESXi-65-on-Supermicro-SuperServer-Bundle-1-by-TinkerTry

    Jun 20 2017 Update

    Excellent comments left by Vic T below:

    Nice write-up, as always. Your site has always given me inspiration and keeping me up-to-date with what's possible in my home lab - a small-chassis x10sri-f on xeon e5 L v4, and a sys-e200-8d "frankensteined" with a 60mm fan to reduce the stock fan noise.

    Just wanted to share what I have with regard to GPU and USB3 isochronous, FWIW.

    On GPU, I managed to find an older Grid K2 card which has 2 GPUs on board - I passed through one of the GPUs to a VM for demanding tasks, and the other GPU can still accelerate other VMs vSGA (via VMware tools' 3d acceleration via Xorg on host) for lower requirements with the added advantage of being vMotion-able. The Grid K2 requires good cooling, so I ended up having to add a few more fans and so far the noise has been bearable. As opposed to the newer Grid, the K2 doesn't require the newer Nvidia software licensing which can get very expensive.

    On the USB, I've tried 3 USB-to-IP devices (yeah, part of work eval to passthrough USB-to-serial console and Rainbow tokenkeys): Digi's AnywhereUSB, SEH's UTN2500 and the Silex DS600. The AnywhereUSB is USB2.0 only and doesn't support isochronous and had driver issues. So far I've been having good results with SEH and Silex, both support isochronous and managed to run a USB-based DVD drive successfully.


    Jun 24 2017 Update

    Testing and using the Silex DS-600 Gigabit USB 3.0 High Throughput Device Server is going well so far, stay tuned for a separate article about it.

    The DS-600 is available on Amazon, with a funny typo in the product name, "Gigbie USB 3.0 Device Server."


    See also at TinkerTry

    tinkertry-front-view-of-home-datacenter-at-66-watts

    See also