vSphere 6.5 Core Storage white paper - one home virtualization lab enthusiast's perspective

Posted by Paul Braren on Dec 7 2016 in
  • Storage
  • HomeLab
  • HomeServer

    • Let's start with Cormac Hogan's recent post:

    • Announcing vSphere 6.5 Core Storage white paper
      Nov 28 2016 by Cormac Hogan at VMware | Blogs > Virtual Blocks

      I’m delighted to announce the availability of a new vSphere 6.5 core storage white paper. The paper covers new features such as VMFS-6 enhancements, policy driven Storage I/O Control, policy driven VM Encryption, NFS and iSCSI improvements and of course new limit increases in vSphere 6.5...You can download the paper from our storagehub site by clicking here.

    Actually, that last link fails:

    vSphere-6-5-Storage-cropped
    View "vSphere 6.5 Storage" document, in PDF format.

    Ok, now let's strip that URL down even shorter to explore a little more:

    *Let's explore this one today, Cormac's new baby:

    The navigation along the left edge works well enough, but you may prefer the EXPORT TO PDF button.

    Now that you've got the document of interest, it's time for me to share the tantalizing bits within that really grabbed my attention, along with my commentary to give a bit of backstory before each quoted-in-green section of his white paper.

    Perhaps you've seen my mention of my first speed tests of my home lab's 10TB HGST He10 drive that I use for backups. Yep, this SATA3 drive infused with Helium gas features that new 512e / 4K Native Advanced Format, see also KB 2091600:

    So with those giant capacities in mind, and my predilection for using RDMs when needed, let's have a read of what VMware has to say about these HDDs.

    PDF page 4, or here on storagehub:

    Storage Limit Improvements

    512e Advanced Format Device Support

    The storage industry is hitting capacity limits with 512N (native) sector size used currently in rotating storage media. To address this issue, the storage industry has proposed new Advanced Format (AF) drives which use a 4K native sector size. These AF drives allows disk drive vendors to build high capacity drives which also provide better performance, efficient space utilization and improved reliability and error correction capability.
    Given that legacy applications and operating systems may not be able to support 4KN drives, the storage industry has proposed an intermediate step to support legacy applications by providing 4K sector size drives in 512 emulation (512e) mode. These drives will have a physical sector size of 4K but the logical sector size of 512 bytes and are called 512e drives. These drives are now supported on vSphere 6.5 for VMFS and RDM (Raw Device Mappings).

    Descriptor Format Sense

    ESXi version 6.5 introduces Descriptor Format Sense support throughout its Pluggable Storage Architecture (PSA). This enables better support for larger disk drives and third party plugins, which must use Descriptor Format Sense to report media errors and other check conditions referring to Logical Block Addresses (LBA).
    Support for Descriptor Format Sense was also a necessary step for pass-through Raw Device Mapping (RDM) device support. This enables a more correct pass-through mechanism.

    I've witnessed a loose cable induced PSODs first-hand, followed by heroic (and successful) efforts by VMware support to manually recovered that scrogged VMFS. So anything that says journaling might be improved gets my attention, have a look!

    PDF page 5, or here on storagehub:

    VMFS-6

    VMFS-6 is the new filesystem version that is included with the vSphere 6.5 release. In this section, some of the new features and characteristics of this new file system are explored.

    Sector Readiness

    As part of future-proofing, all metadata on VMFS-6 is aligned on 4KB blocks. This means that VMFS-6
    is ready to fully support the new, larger capacity, 4KN sector disk drives when vSphere supports them.

    File System Resource Management

    File Block Format

    ...enhancements should result in much faster file creation times. This is especially true with swap
    file creation so long as the swap file can be created with all LFBs. Swap files are always thickly
    provisioned.

    Journaling

    ...
    These new mechanism was introduced to address journal issues on previous versions of VMFS, due to the use of regular files blocks as journal blocks and vice-versa. Tracking journal blocks separately in a new resource file reduces the risk of issues arising due to journal blocks being interpreted as regular file blocks.

    Parallelism/Concurrency Improvements

    ...
    There are also benefits to Thin provisioning operations. Previous versions of VMFS only allowed one transaction at a time per host on a given filesystem. VMFS-6 supports multiple concurrent transactions at a time per host on a given filesystem. This results in improved IOPS for multi-threaded workloads on thin files.

    Yep, I've got a 10TB drive, I thought it seemed to take on the VMFS 6 format rather quickly, now I know why! You'll also see I've had a thing about 2TB barriers for many years, glad to hear that last barrier has been cleared.

    PDF page 7 or here on storagehub:

    New VMFS-6 Features in action

    VMFS Creation

    Using these new enhancements, the initialization and creation of new VMFS datastore has been significantly improved in ESXi 6.5. For a 32 TB volume, VMFS creation time was halved.

    Upgrading from previous versions of VMFS to VMFS-6

    Datastore filesystem upgrade from VMFS-5 (or previous versions) to VMFS-6 is not supported. Customers upgrading from older versions of vSphere to 6.5 release should continue to use VMFS-5 datastores (or older version) until they create new VMFS-6 datastores.
    Since there is no direct in-place upgrade of filesystem supported, customers should use Virtual Machine migration techniques such as Storage vMotion to move VMs from the old datastore to the new VMFS-6 datastore.

    Hot Extend for VMDKs > 2TB

    Prior to ESXi 6.5, thin virtual disks could only be extended if their size was below 2TB when the VM was powered on. If the size of a VMDK was 2TB or larger, or the expand operation caused it to exceed 2TB, the hot extend operation would fail. This required administrators to typically shut down the virtual machine to expand it beyond 2TB. The behavior has been changed in vSphere 6.5 and hot extend no longer has this limitation.
    It is important to note that this does not require VMFS-6 or Virtual Machine HW version 13 to work. VMFS-5 will also support this functionality as long as ESXi is version at 6.5.

    I frequently move huge VMs around in my home lab, including my Window Server 2012 Essentials VM that is also the file share for my extended family's PCs running daily scheduled backups from their Veeam Endpoint Backup FREE, with VPN for connectivity. This means a data drive with a virtual disk drive size between 2TB and 8TB! Things get tricky if I thin provision them, check out this crazy TinkerTry article/title:

    I had thought this was a rather niche topic. I just went and looked anyway, and to my amazement, it turns out 10,000 people have read that year-old article, and growing! I think it's fair to guess that these space reclamations innovations are long awaited improvements. They'll mostly likely have the greatest positive impact on enterprise customers who manage storage arrays that live on SANs. Yes, I'm thinking about the plight of many of my valued customers at my day job too, this is good!

    PDF page 10, or here on storagehub:

    UNMAP

    VAAI UNMAP was introduced in vSphere 5.0 to allow the ESXi host to inform the backing storage that files or VMs had be moved or deleted from a Thin Provisioned VMFS datastore. This allowed the backing storage to reclaim the freed blocks. There was no way of doing this previously, resulting in many customers with a considerable amount of stranded space on their Thin Provisioned VMFS datastores.
    In vSphere 6.0, additional improvements to UNMAP were introduced which facilitated the reclaiming of stranded space from within a Guest OS. Effectively, if you have deleted files within a Guest OS, and your VM is thinly provisioned, you can tell the backing storage that you are no longer using these blocks. This allows the backing storage to reclaim this capacity for other uses.

    Introducing Automated UNMAP Space Reclamation

    In vSphere 6.5, there is now an automated UNMAP crawler mechanism for reclaiming dead or stranded space on VMFS datastores. This is a big change from previous versions of vSphere, where UNMAP was run manually. Now UNMAP will run continuously in the background. The vSphere UI can be used...

    CxTvg1HW8AAqO80

    Soon after TinkerTry was born, I posted this article about my first home lab experiment, with the lofty goal of using (expensive/optional/future) encryption capabilities my LSI 9265-8i RAID controller claimed. Well, that never really worked out. Here we are, years later, getting closer to something we can use.

    Have a look at my recent conversation with VMware's Mike Foley, where we discuss his recent podcast appearance where he details what's needed to implement VM encryption. Hmm, perhaps this is not as far fetched for a home lab as you might think!

    PDF of page 17, or here on storagehub:

    vSphere VM Encryption

    Introduction to vSphere VM Encryption

    vSphere 6.5 introduces a new VM encryption mechanism. This encryption mechanism is implemented in the hypervisor, making vSphere VM encryption agnostic to the Guest OS. This not only encrypts the VMDK, but it also encrypts the VM Home directory contents, e.g. VMX file, metadata files, etc.

    VM Encryption Limitations and Considerations

    There are some limitations and considerations associated with vSphere VM Encryption in vSphere 6.5, some of which are highlighted here.
    Since VMware does not own the KMS, customers are urged to engage with the KMS provider for conversations around backup, DR, recovery, etc. of their KMS. It is critical that some plan is in place to retrieve the encryption keys in the event of a failure, or you may render your VMs unusable.
    Administrators should not encrypt their vCenter server. This will lead to a chicken-and-egg situation where you need vCenter to boot so it can get the KEK from the KMS to unencrypt its files, but it will not be able to boot as its files are encrypted. Remember, vCenter server does not manage encryption.
    It is only a client of the KMS.

    Ok, this one is easy, "only" 693 NVMe mentions at TinkerTry so far, with proven speed for home VMFS datastores. Consider investing your hard-earned money on something like a Samsung 960 M.2 "gumstick" NVMe SSD instead of a "legacy" high-queue-depth RAID adapter intended for spinning drives. Check out those queue depths seen below, and in my new article:

    4-NVMe-devices-in-my-SuperServer
    Yep, that's 4 M.2 drives in my Supermicro SuperServer Bundle 2, in one slot! I could have actually had a 5th NVMe drive in the motherboard's M.2 slot, but I ran out.

    Want to learn and test vSAN, which loves NVMe, but can be rather cost prohibitive to do right? How does a Virtual NVMe Device sound? Now did I got your attention? Seriously, did I? After reading these next sections, please consider leaving your comments below, it really does make everything more fun for all of us!

    PDF page 19, or here on storagehub:

    New Virtual Storage Hardware

    ...

    Virtual NVMe Device

    Virtual NVMe device is a new virtual storage HBA which is designed for providing lower IO overhead and scalable IOs for all flash SAN/vSAN storages. NVMe devices are increasingly becoming the primary storage interface for flash-based storages due to its well-designed, low overhead storage protocol and its ability to support scalable IO on multi-core processors. As recent Linux and Windows guest OSes provide much lower I/O stack overhead by skipping SCSI I/O stacks, and leveraging multiple queues with NVMe devices, virtual NVMe device allows VMs to take advantage of such inguest IO stack improvements. Virtual NVMe device provides 30-50% lower CPU cost per I/O and 30-80% higher IOPS compared to virtual SATA device on local PCIe SSD devices.

    Probably time for me to repair my VMworld-2016-travel-weary, my modest Synology NAS that I turned into a decent RAID0 iSCSI performer. Could be interesting to see what this NFS 4.1 might mean for some NAS owners. Got some more reading to do on that one.

    PDF page 20, or here on storage hub:

    NFS 4.1

    Hardware Acceleration/VAAI-NAS Improvements

    One of the major NFS 4.1 client enhancements in vSphere 6.5 is to introduce support for hardware acceleration. In other words, certain operations may now be offloaded to the storage array. This comes in the form of a plugin to the ESXi host that is developed/provided by the storage array partner. Refer to your NAS storage array vendor for further information.

    So now that we've enjoyed this little journey through my mindset, as I read Cormac's work, what do you think of vSphere 6.5's storage advancements? Let us know, leave a comment below!

    See also at TinkerTry

    TinkerTry'd-two-node-xeon-d-home-datacenter-cluster-November-2016.JPG
    Supermicro SuperServer Bundle 2 - Xeon D-1541 and Xeon D-1567 two node vSphere 6.5 cluster, with an ASUS 10G switch, and a Ubiquiti EdgeRouter Lite doing DNS/DHCP reservations, for easy VCSA deployment.

    Google search for boot from nvme:

    Doing home vSAN is a challenge, see my updates on this here.

    Ways to cram even more M.2 NVMe goodness into my TinkerTry'd SuperServer Bundle 2 Xeon D home datacenter:

    See also

    Check out this little ROBO-friendly new supported feature that I know home lab folks will be interested to hear more about (I'm working on it):

    TinkerTry - PCs, EVs, home tech, efficiency and more, including virtualization. My opinions here, not my employer's.

    Paul Braren, MCSE in 1998, VCP #2681 in 2005, vExpert 2014 to 2024, Veeam Vanguard Emeritus, 2015 to 2019.

    Support TinkerTry

    PayPal | Patreon | VMware Store

    After 6 successful years testing then shipping well over 1,000 Xeon D Bundles, Wiredzone had to stop selling them in mid-2021 due to cost, supply, and logistics challenges. So far, Xeon D-1700/2700 (Ice Lake D) was a minor refresh for 2023, with Xeon D-1800/2800 (Granite Rapids D) refresh looking a little better for 2024. More cores, but still just PCIe Gen4 and DDR4. Looking ahead, I'm glad it's Pat Gelsinger at Intel's helm. I'm also grateful to have had the honor of working at VMware when Pat was the CIO there. I'll leave it at that, given the whole Broadcom thing.

    Easily update VMware ESXi at TinkerTry.com/easy-update-to-latest-esxi