GPU Passthrough? (VMDirectPath/VT-d/PCI Passthru)

Posted by Paul Braren on Jan 27 2012 (updated on Nov 2 2012) in
  • GPU
  • Virtualization
  • gpu-pass-future-test

    You may recall this article, from July of 2011:
    Z68 Sandybridge Motherboard VT-d Test Matrix: Which Mobo/CPU combo works with VMware ESXi 4.1U1 VMDirectPath feature?

    In the past few weeks, I've now received numerous requests for testing GPUs (graphics cards) with vZilla. Yeah, gZilla is my family's gaming rig, but the completed vZilla is for virtualization and daily backups.

    So why would I want to try this?  Well, what about video rendering acceleration? I also have a fast RAID array on vZilla, so this is starting to seem worth a shot.

    I tried GPU passthrough for all of 10 minutes last summer and quickly gave up, but that was ESX 4.1, see related saga told here.

    So it would make sense for me to test this again, with newer cards. Especially since I seem to be having success with passing PCI devices through to VMs, on my ASRock Fatal1ty Z68 Professional Gen3 based system. I took the time to document this in the HCL of the whitebox world at VMDirectPath HCL forum.

    I remind you, of course, none of this is supported by VMware in any way, this is merely experimental. Naturally, their enterprise focus means PCI passthrough is really meant for PCI I/O devices, such as RAID adapters or NICs, etc.

    The thought with the GPU is to look into using CUDA, NVIDIA's GPU leveraging rendering acceleration, for some of my own video crunching. If the GPU in makes this considerably faster, then that could be interesting. Well, that is assuming I get it working at all, and if the GPU doesn't waste a lot of watts when not in use. I'm also curious if the speed gains are similar to the CPU/GPU combo the Z68 Virtu technology offers that probably can't work in a VM, since Virtu uses a hybrid of video embedded in the CPU and the PCI GPU.

    I don't claim to have reseached this much yet, but a quick look around, and I see that GPU-in-a-VM stories out there are a mixed bag, seen here and here.

    All those potential gotchas aside, the stars are beginning to align.

    I may have access to a hand-me-down 1 to 2 year old NVIDIA graphics card(s) in a month or two, that won't overtax my modest (efficient) power supply.

    I do have a video editing/rendering system (left in standby 95% of the time) that I'd like to sunset at some point. My focus will not be on complex benchmarking, but instead, on video rendering speeds I can see in a VM with GPU, compared to the 4 year old system. The "real world" test for me is how long it takes to takes to convert a 24GB AVI file (2 hours of Hi8 video already transferred over FireWire) into MPEG4.

    So yes, I will give it a shot, and let you all know how it all goes.

    Thank you for your interest, and your patience!


    Sep. 15 2012 Update:
    I did finally get an NVIDIA 8800GTS inserted in the bottom and middle PCI-e slots, with no luck. ESXi 5.1 wouldn't recognize them for passthru. So sorry for the disappointment, and I just don't have spare GPUs laying around to test. I will say the card only burned an additional 12 watts being inserted (but not being used), but it offered me no function, so I removed it. There are others who have had some luck, watch this dialogue, for example.

    Nov. 02 2012 Update:
    In light of recent issues folks are having with VMDirectPath, when moving from ESXi 5.0 and 5.1, perhaps VMware is more interested in the vGPU approach?

    Hmm, these sorts of cards start at $750 USD: