Wow, is that a 62TB drive in my home lab?

Posted by Paul Braren on Sep 7 2013 in
  • ESXi
  • Hyper-V
  • Storage
  • Virtualization
    • 9 Comments

    Remember this post, from Dec 16 2011?

    Practical ways to deal with VMware ESXi 5.0′s 2TB virtual disk size limitation by Paul Braren on Dec 16 2011

    Well, 2 Terabytes isn't cool, you know what's cool? 62 Terabytes.

    With my new ESXi 5.5 GA build, passing through some or all of my 4 port USB 3.0 PCI card ports is working out quite nicely (passthrough/VMDirectPath/VT-D). But VT-d has some drawbacks, and it's become less necessary for my current configuration. Why? My affordable external Mediasonic HFR2-SU3S2 RAID enclosure can now simply be formatted as one big 5.6TB VMFS datastore, then VM data dropped right on there, for less crucial stuff, such as my daily backups. Yes, that is a ginormous 62TB drive you're seeing there. Admittedly, it's virtual, and thinly provisioned. I can't actually exceed my physical 5.6TB datastore limit. But I can pretend I can. I configured this using eSATA instead of USB 3.0 passthrough, with no RDM mappings needed. Simple, elegant, nice! And able to handle storage vMotion, should I actually upgrade the physical drive capacity at some point in the future. No resizing of drives, Windows just sees it as 62TB of NTFS.

    Next stop, testing a VM with a modern OS, like Windows Server 2012 R2 Essentials (Preview). This time, it'll start its virtual life with virtual hardware version 10, and a UEFI virtual BIOS. Then let's see if I can boot this massive 62TB GPT C: drive. A bad practice for so many reasons. But still, a lot of fun.

    What else is next, for TinkerTry? Feverishly working on:

    Any of this sound interesting to you? Then please considering helping TinkerTry continue grow, with 350 articles so far. Simply follow TinkerTry, and below each article, you'll find the super-simple share buttons and _leave a message _section, no sign-up required. Thank you for your constructive feedback, which helps me continue to improve. Thank you for stopping by!

    carving-up-62TB-is-quite-simple-these-days

    See also:

    4TB Seagate Faceoff - DX vs DM by JimmyJoe Feb 23 2013
    The Case for Larger Than 2TB Virtual Disks and The Gotcha with VMFS by Michael Webster Sep 17 2012
    Hyper-V Scalability in Windows Server 2012 by Microsoft Oct 3, 2012, updated Jun 24 2013, note that the Hyper-V role in Windows Server 2012 supports 64TB virtual disks

    Sep 20 2013 Update:
    Video of ESXi 5.5 Build133180 (which may well be the final RTM code) now available, demonstrating the creation of a UEFI BIOS VM, and Windows 8.1 installed on that VM's 62TB virtual drive. The physical 128GB SSD, formatted VMFS, holds the roughly 10GB of files, but the VM thinks it has 62TB of GPT type of disk, NTFS formatted as one giant C: drive.

    Sep 30 2013 Update:
    From the ESXi 5.5 Release Notes.

    Configuring virtual Flash Read Cache for VMDKs larger than 16TB results in an error
    Virtual Flash Read Cache does not support virtual machine disks larger than 16TB. Attempts to configure such disks will fail.

    Workaround: None

    All Comments on This Article (9)

    Paul Braren | TinkerTry.com

    I have not run into this particular issue, and I confess, I haven't used RDMs much at all these past 2 years. I hope somebody else chimes in, but might be best to also post on VMware forums too.
    https://communities.vmware.com/community/vmtn/vsphere/content?filterID=contentstatus[published]~objecttype~objecttype[thread]
    Thank you for visiting TinkerTry, and for leaving this comment/story! One guy who might have some ideas, if you are still stuck:
    https://twitter.com/paulbraren/status/1074502905405849600

    Adrian M

    I realize this is a rather old thread, but I couldn't find a relevant more recent thread to post this on instead.

    I'm using VMWare ESXi v6.5 at home, and I've created a virtual file server using Win2k16. One of the decisions I made early on was to use physical Raw Disk Mapping (RDM), so that in the event of something going bad, the physical disks would be native NTFS, and I could just mount on a regular windows server and read the data.

    Everything works fine, however, I've run into a problem lately that has me stumped. I have several disks mapped (using RDM) to my virtual file server. One of the disks happens to be 8TB in size. Windows sees it as being 8TB in size if I look at it in windows disk manager. If I right click and look at Properties in Windows Explorer, it also correctly shows the size at 8TB, with 2TB filled and about 6TB empty. When I try to copy a file to it, I get an error saying that there is not enough disk space. I've tried to see if I've done something wrong in how I've mapped the disks to the virtual machine and so far I can't seem to find anything wrong.

    I wanted to see if any of you folks had any issues as such using RDM in VMWare. I'm not entirely sure if this is a VMWare issue or a Windows issue, so any help in isolating the problem will also be of tremendous help.

    RFS42

    Further thought. In the research I've done since this post I'm getting the impression attaching/mounting and unmounting/detatching eSATA drives with VMWare datastores isn't a really good idea?

    RFS42

    Paul, Love your site and email. Question about your Mediasonic HFR2-SU3S2 RAID. We need an external ESXi 5.5 compatible, great, fast solution for physically moving TBs of data between 2 sites for replication seeding and to use for air-gapped offline, tertiary VM backups. We've been using the ReadyNAS Pro 4 with NFS but it's slow. With ESXi 5.5 and the right controller can you plug/unplug an eSata raid enclosure like this? Withouting rebooting ESXi? Suggestions from anyone on SATA controllers and adding an eSATA port to a 2U SuperMicro server?

    Paul Braren | TinkerTry.com

    Hyper-V and Red Hat Enterprise Virtualization will do >2TB virtual drives too:
    http://www.zdnet.com/blog/virtualization/hypervisor-comparisons-its-harder-than-you-think/4738

    and yes, I had other ideas in the past that worked as well, but those were additional (affordable) hardware, eSATA or USB 3.0:
    http://TinkerTry.com/beyond2tbvmdisksizemaximum/

    Richard Cross

    Paul, thanks for the reply. Is there another hypervisor that supports this function? I don't want to upgrade the hardware at this time and I really don't want to add another storage server. Any other ideas?
    Thanks again!

    Paul Braren | TinkerTry.com

    Without VT-d, one alternative would be the somewhat more awkward RDM mappings:
    http://TinkerTry.com/rdm4sata

    Richard Cross

    Here's a question that I am having a hard time getting an answer
    to...the SuperMicro X7DVL-E motherboard that I am using does not support
    VT-d. With that said, is there any way that I can add an internal
    drive just to use for backups? I currently have a RAID5 array for VM's
    and preferably shared data on an LSI card and an extra 2TB drive for
    backups. I would like to have a normal file system on them (partitions
    if possible), not VMFS, so that I can access the shared data from
    another system if it is removed from the system. From what I can tell,
    it seems that any local drives that exist are formatted using VMFS. Is
    there any way to have a "shared data drive" and backup drive on this
    system without adding a NAS box or something similar to my network?
    Thanks!
    Rich

    Paul Braren | TinkerTry.com

    Got the RTM of Windows Server 2012 R2 Essentials Sep 9, and installed it in a UEFI BIOS ESXi 5.5 VM, with a 62TB (thin provisioned) C: drive, and it worked just fine!