Best parts of VMware's ESXi 5.5 free hypervisor rely on vCenter, which isn't free. Uh oh?
I tend to stick with technical articles and how to guides. But there's something else I've been worrying about for months now. How will VMware be handling the move toward vSphere 5.5 and eventually 6.0, where the vSphere Client will be entirely gone, completely replaced by the vSphere Web Client. That means you will have to have a VCSA (vCenter Server Appliance) to attach to, making VCSA is no longer just optional. This VCSA is largely pre-configured, a great match for the home lab, and finally a decent performer (on SSD). But it needs a license to go longer than 60 days. So, where's this leave the free hypervisor user, who abides by EULAs and license restrictions, legitimately avoiding the time bombs?
We can already see the inevitable progression toward Windows 32 bit client oblivion in the current 5.5 release. You only need the client to get the VCSA quickly deployed. After that, all the good stuff is only gotten from the vSphere Web Client. This is a rude awaking for many who may not have tried ESXi before. Examples of this good stuff include VMs with greater than 2TB drives, mapping USB 3.0 ports to VMs, better GPU handling, and vSphere Flash Read Cache. These compelling features are locked to hardware version 10 VMs, which cannot be managed from the vSphere Client, and cannot be run on ESXi 5.1. Hmm. See the issue?
A VMware Community post went up a few hours ago, that really highlights the frustration and uncertainty at play here:
jgkurzSep 24 2013 7:03 PM
The good news: I upgraded to 5.5 and now my USB 3.0 adapter works. I connected it to my Win7 x64 VM and so far it's functioning as expected. My adapter is a NEC based chipset marketed as the StarTech 2 Port PCI Express SuperSpeed USB 3.0 Card Adapter Model PEXUSB3S2
The bad news: I am a home user and have uncovered an unfortunate issue with the free version of ESXi 5.5. vSphere client no longer supports editing VM settings of a version 10 machine. You must use vSphere Web Client which requires vCenter. I downloaded a trial copy of vCenter Appliance and vSphere Enterprise Plus so that I could make the required pass-thru edit from vSphere Web Client.
I'm not sure how I'm going to administrate my host after my eval licenses expire.. : (
Good question, isn't it?
Many of us have legitimate access to 365 day corporate use trials, or full blown vSphere license keys, but I realize most folks don't. Despite Signs of Life for the VMware VMTN Subscription Movement (Chris Wahl, Aug 29 2013), right now, we have no VMTN. And vSphere 5.5 has arrived. With Microsoft recently pulling back on TechNet, it'll also be interesting to see how VMware reacts, given a good numbers of consultant types that can't afford MSDN may wind up losing their easy access to Hyper-V.
Licensing of vSphere has never been simple, and 5.5 is no exception. See the 402 hits for "licensing" in the vCenter online manual here, and pictured at right.
This sort of thing really is a deterrent for folks that otherwise might be interested in ESXi, like literally everybody I met at the MSDN/TechNet heavy home server enthusiast crowd I spent the past weekend with.
This article is just meant to start to openly discuss the elephant in the room, posing the tough questions weighing on many IT Pro's minds these past 2 days of "playing" with the generally available vSphere 5.5 (ESXi + vCenter).
1) How are you handling the vCenter requirement?
, preserving all your VMs, side-stepping those 60 day time bombs, even though the only thing spent in this scenario is your valuable time? (up to you to follow your EULAs)
I honestly don't have answers, or any insight about where things are headed for the already entirely unsupported home virtualization crew. But I do know that I'm very curious to hear what you all think, my valued TinkerTry visitors. Don't worry, I don't give out homework often!
Closely related threads:
communities.vmware.com/thread/457843
communities.vmware.com/message/2292129#2292129
TinkerTry.com/vmware-esxi-5-5-free-hypervisor-breaks-past-32gb-restrictions/#comment-1023174996
Sep 26 2013 Update
In my original post above, I didn't call out, why VMware? We each have our own answers to that one. For me personally, it's simple really. ESXi is at a vast majority of my customer's sites, and as far as I know, it's the only bare metal hypervisor that lets me test all the other big names in hypervisors. Testing by nesting, on just one Core i7 system I affectionately call vZilla. Invaluable for self training purposes, and very relevant to my career in IT. See also:
Which hypervisor for your home lab, VMware ESXi, Microsoft Hyper-V, Linux KVM, or Citrix XenServer? by Paul Braren on Mon Sep 02 2013.
VMware ESXi 5.1 can run Microsoft Hyper-V Server 2012 VMs, nice! by Paul Braren on Mon Sep 24 2012, You heard all about guestOS = "winhyperv" here first!
Sep 27 2013 Update
Back on Sep 25, Josh R noted VMware Workstation 10 based workarounds in the comments below, see also ESXi 5.5 Free Hypervisor: Will Home Labs Survive? published Sep 26 2013, where VINTAGEDON summarizes:
You can read more concerning this in VMWare’s support article, Connecting to a remote ESX / ESXi / vCenter Server from VMware Workstation
As it stands, unless something changes, what you end up with is a free product that can only be managed by a paid, licensed product. At least this is how it stands at the moment.
Oct 01 2013 Update
Very relevant, spot on post by Vladan yesterday, you'll want to read the whole article,
Realtek 8169 NIC in ESXi 5.5 not detected by default – install a VIB by Vladan Seget on Sep 30 2013, where he says:
VMware did strip down many drivers from the ESXi 5.5 ISO, and additionally, they do not provide the Offline bundle for ESXi 5.5 free version.
VMware provides full list of hardware which is “deprecated” in the ESXi 5.5. Here is VMware KB listing all unsupported NICs – Devices deprecated and unsupported in ESXi 5.5.
Hopefully this post help someone who was in the same situation. Good option is to go for Intel based NICs which in most cases are supported. The best option is to check the VMware HCL before buying a NIC, still.
Dec 07 2013 Update
Good news and bad news.
The good news is, in under 2 hours, I got my ESXi 5.5 lab rebuilt, as a sort of exercise, to be sure my instructions are still accurate. It included me injecting Realtek NIC support into the ISO, using the Andreas Peetz ESXi-Customizer, which I explained in my very popular post here. I then went on to fix ASMedia drivers for full access to all SATA3 ports on my mobo, also care of Andreas. Finally, I got the health of my LSI-9265-8i seen again, by injecting a VIB. None of this was that hard, but I will write all that up separately, especially for the folks out there who invested in a vZilla-like build. I've already recorded a long video of all of this, but it'll take days of edits to get it all into a consumable, sensible form to publish.
The bad news is this somewhat worrisome post, by the same Andreas Peetz:
The Good and the Bad of the new Native Driver Architecture in ESXi 5.5 Oct 30 2013
I recently became aware of that with vSphere 5.5 VMware introduced a new Native Driver Architecture for ESXi. William Lam has written an excellent blog post describing the motivation behind and the benefits of this new architecture. I will shortly summarize it here, but also add some concerns about the way it is introduced, because - in the future - it might severely limit the ability to run ESXi on white box and commodity hardware.
Dec 17 2013 Update
I'm happy to announce my handy new post that'll help you inject those driver VIBs right up front, for a nice clean install experience.
Use ESXi-Customer GUI to inject multiple driver VIBs into your ESXi installer ISO Dec 16 2013
Mar 07 2015 Update
Newly announced EVALExperience might make things much better for the home lab soon, see more about a possible way to license for 1 year at a time, for $200, described here at TinkerTry:
VMTN reincarnated! EVALExperience is to VMware Professionals what MSDN/TechNet is to Microsoft Professionals. Jan 15 2015
Also, the C# vSphere Client isn't gone in the vSphere 6.0 betas I've been using. It's actually become more capable (before it eventually gets eliminated?), with a key feature added back in 5.5:
Using vSphere 5.5U2 Client to edit the settings of virtual machines of version 10 or higher
Mar 11 2016 Update
The ESXi Host Client seems to be the HTML future of the vSphere product line:
- Faster HTML5 future for VMware vSphere Web Client? ESXi Embedded Host Client Fling flies! (30 second install instructions)
Aug 26 2015 by Paul B raren at TinkerTry
but we still have that awkward issue of getting VCSA installed in the first place. Then again, perhaps marrying the two together might be an indication of more to come:
- How to bootstrap the VCSA using the ESXi Embedded Host Client?
Dec 22 2015 by William Lam at virtuallyGhetto
All Comments on This Article (80)
A Question:
Would purchasing the VMware vSphere Essentials Kit ($560) which includes :
vSphere Hypervisor (ESXi)
vCenter Server Essentials
vCenter Operations Manager Foundation
Mean you get a license for vCenter and can use the Web Client?
Thanks,
dc
Thank you for reading my post.
Rich
I have not experienced that issue (in home lab, just using root login for the appliance), I'm sorry, I just don't know. I don't even have a guess really. Probably will have better luck in the VMware community forums.
VSphere Web Client 5.5
I have a VMware ESXi host with a VCVA Vcenter Appliance and connect using the vSphere Web Client. When using the vSphere Web Client 5.5 to connect using the default administrator @vsphere.local user account I see datacenter, hosts and VMs. When trying to add ESX Admins to log in to vmWare vCenter Server as Administrator and login, I am able to authentic but no longer see the Hosts or Virtual Machines. It indicates Empty. ANy idea. Testing the authenication succeeds.
Nah, not a PSOD, of course. I'd seen it only once -- after that very update which broke something and was fixed within days.
Not even the appliance, I think I saw only the vpxd (Server Service) stop once, had to restart manually other times. Must be timeout issue. See, many times the cause seemed to be my local Flash addon (Mozilla FF 26) asking whether it could use some more local hdd space despite VCSA being in the whitelist, besides flash settings default to allow using local HDD. Quite annoying.
So sorry about that Basil, too much snow blowing, not enough sleeping and paying attention (got your thread mixed with another one about drive recognition).
Quite strange that you're having crashes with just 2 hosts, but your configuration is so different than mine that I can't really make an education guess as to what is going wrong. Thanks for taking the time to type the details up though, as it's quite possible it'll help somebody out there.
Not to take too much more of your time, but curious what kind of crash you're getting, is it PSOD
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1004250
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006796
or just the vCenter appliance crashing?
Wait a second ) I never said it wasn't seeing the drives.
Config 1: 3GHz core i3 with 8 GB RAM, sata HDD, workstation 10 (2 cores and 6 GB RAM given to the appliance).
Config 2: 2xXeon5110@1,6GHz, 4 GB RAM (saddens me to no end) sata HDD, ESXi 5.5 (all available resources given to the appliance).
Now, I have a 3rd option, and it worked a lil' bit better, allowed me to add 2 hosts and crashed afterwards. 2xXeon5405@2GHz, 6GB RAM, iSCSI disk made of 4 HDDs in RAID10 on a NAS4Free box over 1G link. It worked alongside with two 2k3 machines, and, as I said, allowed me to add 2 hosts to a newly created datacenter before crashing.
As of today, I've incidentally updated the host from the 5.5.0, 1331820 to 1474528 and shut down both 2k3 servers. Now we're talking. Stable and fairly smoth. Will have to do. I needed the thing for Vmware Education Labs, couldn't even imagine you'd need the best in class hardware to manage one datacenter with 2 hosts, duh.
Sorry about that, SSD or spinning drive, doesn't matter, somehow you're saying it's just not seeing those drives. Can you share more about your configuration? Are you definitely in AHCI mode in the BIOS, and are the drives recognized fine by other OSs or by the BIOS itself?
It's not a SSD, it's a 7200 HDD. Yes, in the easiest of configurations.
Wow, it would seem likely there is something else going on here. Any chance the SATA port the SSD is attached to is set in the BIOS to be in IDE mode instead of AHCI? I realize I"m grasping at straws here, just hard to know why your performance and stability is so poor. Also, is your vCenter appliance at default easy configuration, or includes AD/LDAP/etc? 4GB is perfectly legit for small labs, and is even mentioned in the vCenter doc
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2005086
so yeah, you've got me at a loss for words...
/* This vCSA is largely pre-configured, a great match for the home lab, and finally a decent performer (on SSD) */
Can't afford a SSD for simply trying this out, and on a 2*7200RPM HDD raid1 and 6 GB RAM the thing won't even show a bit of a stable behaviour. I may get to log in, and then it crashes while adding a second host to my datastore. Past reboot, services start for like 15 minutes, none of the perf counters above 50%.
Tried both vCSA and VCS on 2012R2. No bloody luck.
Hard to argue this point, hoping somebody will step in with some insight/ideas, other than my politely pointing out that a vSphere 5.5 home lab can be successfully rebuilt every 60 days in around an hour, as crazy as that sounds, with all fancy features enabled (and the VMs themselves left in place, merely re-imported):
TinkerTry.com/installvsphere55
TinkerTry.com/which-hypervisor-for-your-home-lab
As far as strict adherence to licensing, that's a different matter. I'm merely saying what can be done practically. It's still best if you can get a 365 day trial or even an NFR copy, of course, if all your needs are non-production.
Overall, it seems that Microsoft Hyper-V has it's own hoops. Here's a strange one, https://www.veeam.com/blog/rdp-rsat-free-hyper-v-server-2012-r2-remote-desktop-appliance-linux.html
Yes, it's all complicated, and frankly, the less fun side of blogging about all this. For your needs, doing billable work replicating customer environments (which I used to do at various customer-site labs, levering their fully licensed environments), yeah, it does seem that VMware is leaning toward moving toward charging for that full-featured web interface for getting the latest and greatest features out of ESXi 5.5.
I'm a professional IT guy, selling vmware solutions. My point here is, that I do most work in a home lab for testing and trying to reproduce problems we're facing at the customers. I usually got 2 kinds of VMs: Infrastructure (licensed) and Testing stuff. Infrastructure means DHCP services, DNS, some mailserver to get notifications and so on... Testing machines are all kinds of buggers. From *nix to Win/OSX.
Honestly, I cannot afford those 500 bucks for an essentials license just for that web interface. In the end it all adds up to the lab costs if you want to run it all in a legal manner. Sum it up: 1500 per server hardware, add licenses for various OS, ...
Correct me if I am wrong, but, the point of visualization was reducing costs, was it?
Wow, so rare somebody comes back to make a "confessional" post like this, and I thank you so much for taking the time to do so. Very insightful, very helpful.
Yes, I also care a lot about solid USB 2.0 (and even USB 3.0):
TinkerTry.com/esxi-5-1-sort-of-supports-usb-3-0
TinkerTry.com/usb3passthru
before I jumped into vZilla back in 2011. Consumer level support doesn't exactly matter as much in the enterprise, but in the home lab environment, it certainly matters, making 15 minute projects like this so valuable, and simple:
TinkerTry.com/configure-automated-shutdown-homelab-datacenter-15-minutes
Just to throw an additional idea out there. At an additional costs, there could be Hyper-V work arounds to do USB mapping through software, discussed around 34 minutes in to the recent somewhat related podcast about Hyper-V role of Windows 8:
homeservershow.com/byob-episode-137.html
Here's the product link ($149):
fabulatech.com/usb-over-network-purchase.html
fabulatech.com/usbnet-case-studies/dongle.html
So...turns out I was wrong; I've been tinkering with MS Hyper-v 2012 R2 on my HP N54L ... what a pain!,
- it is _not_ more user-friendly than VMWare, unless you consider regularly Googling Powershell commands user-friendly!
- VMWare ESXi was _so_ much easier to use, even with the vCenter appliance changes.
- I had to install a new network card, as Hyper-V doesn't support the HP's integrated NIC.
- I've now discovered (didn't think to check!) Hyper-V has no USB pass-through, so my WHS / USB security camera project is dead.
I'm considering going back to bare-metal install on WHS2011 on the N54L, and perhaps getting a new (more powerful) box in future for VM tinkering
The discussion is not really about what the limitations are now but where things are headed. You can hold out on v8 for now but for how long? Are they effectively dropping free ESXi if they make it unusable?
The real problem is that the existing licencing models are massively inflexible and the step between using the free version or paid is too big. For my usage licences would cost 5x the hardware cost (inc power/maint) that I would be saving. Essentials licences that would call for multiple resource hogging vservers and introduce SPOFs with no added functionality are not really a viable option either.
vmware are too busy looking greedily at the big boys to realise that they could be ubiquitous if only there were cheaper versions that had cut down features rather than cut out features.
I really question the use of a thin client model without a very HA ESXi setup. The resilience of many independant desktop machines should not be overlooked.
Can you give readers some sense of the set up time it took for you? Or, wording that differently, now that you've been through it, how long would it take you to bring such a lab up, all over again?
Thank you for weighing in here, as you can see from other's comments, you are not alone in your sentiment. I really appreciate your taking the time to add to the dialogue.
Just as a counterpoint:
For those of us with mostly VMware customers for our jobs, not staying up on VMware isn't really an option though. That reality is also common.
Not all doom and gloom:
At least there's still the 60 days, and rebuild (while preserving VMs) can take just an hour or so, in a simple lab environment:
TinkerTry.com/installvsphere55
I also have used the free Hyper-V Server Core edition for bare metal. You need Windows Pro or better to manage it with the free tools, but it's very suitable as a free alternative to the vSphere free hypervisor. Add to that the free StarWind SAN solution that sits atop Windows Server or Server Core and you've got a great test solution that runs well on a variety of hardware. My old PowerEdge SC1430's do very well with it.
I was hoping the next free ESXi version was going to be less restrictive and more incentivizing for IT admins to use at home and/or in labs, but it seems that's not the case with 5.5 if not quite the opposite. Three years ago I never thought MS would come close to topeling VMware's marketshare. Two years ago I turned my head and said maybe. Now, I've tossed VMware aside for bare metal and have started using Windows Datacenter 180 day trial as my host using Hyper-V. So sad to say that. This was clearly VMware's market to lose. So sad.
Help4yourpc, great to see you've helped jaxun and likely many others like jaxun out there with this great tip. Much appreciated!
http://www.techsoup.org
Great point! I had thought about that when creating the video, but apparently never actually wrote about that. Your taking the time to write in is very much appreciated, not just by me, but by others reading this article. Thank you!
An excellent tip. I have purchased many Windows client and server OS licenses through TechSoup, as well as Office products. TechSoup is definitely a vaulable resource for this kind of thing. I only wish they'd work with VMWare to get us NPs deeper discounts on their enterprise products.
Being that VMWare does not have an arrangement with TechSoup, I will be going the Hyper-V route.
I just want to know, what comes first, the chicken or the egg? Fast forward to the time the vsphere client is gone. You install ESXi 6+, how would you even initially get into it to build a vcenter server or appliance assuming you have nothing in place now. I think getting rid of vsphere client is a huge mistake in and of itself anyway. I really like the web client as an "option" but they should keep the vsphere as well. It's much easy to work on things in bulk and a ton faster
Regarding USB 3.0 support I can't determine if USB 3.0 is supported under ESXi 5.5 with non VT-d CPU like the one in HP Microserver G1610T.
ESXi can't see the attached USB 3.0 hard drive connected on USB 3.0 on board controler.
Is it an issue with the HP onboard USB 3.0 controller support by ESXi or a question of ESXi support of USB 3.0 without passthrough ?
In case the issue is coming from HP controller, does any NEC controller will work on ESXi without Passthrough support CPU ?
Thanks,
Pierre
Chris, thank you for sharing your perspective on this.
Figuring you've found the HP Microserver area of this forum helpful too:
http://homeservershow.com/forums/index.php?app=core&module=search&do=search&fromMainBar=1
I'm hoping we can affordably get beyond 32GB of RAM soon, which would help a bit.
But yeah, I hear you, I cannot imagine running any of this on non-SSD at this point, vCenter appliance, or vSphere Web Client.
Just some added observations I just made:
I've also noticed that once the vCenter appliance has been booted for a while, it uses about 1.5GB of RAM to manage a dozen or so VMs on one ESXi host, and about 170Mhz of CPU time when used for light admin tasks. See also picture below. That doesn't mean it'd be "happy" with just 2GB assigned to the VM (I've given it 4GB). And I have a lot more VMs that I just haven't gotten around to adding to my inventory yet...
having tinkered with VMWare ESXi 5.1 first on my HP Microserver (and enjoying the experience), the 5.5 experience has left me feeling disappointed. As a home user I don't need a thin, slow, resource intensive Web Client (another VM) running.
I'm currently wavering between Hyper-V R2 and XenServer, both have thick clients.
My gut says Hyper-V is going to be more home-user friendly, and I can live with being tied into Microsoft (we've 2 laptops and a desktop on Windows 7, 8.1); although Xen open source is used by Oracle VM Server, and othershttp://en.wikipedia.org/wiki/Xen#Commercial_versions_of_Xen .. and Linux-based, which is the future (Steam OS), right?
maybe I should save up for another Microserver :)
I've bent the ear of every VMware employee I've met lately (numerous user group meetings this year, VMworld last year with a bunch of VMware employees eating lunch on a patio with me, etc). Most seem to agree with pretty much what you've stated here, but are also too low in rank to do much of anything about it.
This is a poor strategy move by VMware given the position of Mcrosoft pulling Technet, the capitalisation of such a decision by VMware is clear to see with HyperV and Technet, so it makes little business sense. Citrix on the other hand appear to be more tuned in turning XenServer into a completely open offering; so I really don't understand VMware's strategy and yes I am one of you guys who have heavily invested in a VMware based home lab.
Thanks Sawyer, for contributing!
FYI, I have also noticed that you can choose the older v8 hardware even using the vSphere Web Client, seen here:
http://youtu.be/79XmsaNRQeo?t=1m57s
and pictured below, and more details in a big table VMware wrote up here:
http://pubs.vmware.com/vsphere-55/index.jsp?lang=en&single=true&topic=/com.vmware.vsphere.vm_admin.doc/GUID-789C3913-1053-4850-A0F0-E29C3D32B6DA.html
If you build a VM using the client it makes them v8 so you can manage them with the client. You can rt-click the VM though and 'upgrade hardware' to change it to v10 though if you'd like, but then you loose the ability to edit it using the client.
If you build the VM with the vSphere, it will create the hardware in v10
Yes, the drawbacks are mostly longer term, like perhaps issues with running whatever comes after Windows 8.1 / Windows Server 2012 R2, for example, or future Linux variants (less likely a problem).
Back to now, it's possible that 5.1 won't nest the version of Hyper-V from 2012 R2. I'll be revisiting Hyper-V, this time using 5.5, since many folks have only one machine to juggle hypervisors, see also:
http://TinkerTry.com/esxi-5-1-can-run-hyper-v-server-2012
http://TinkerTry.com/which-hypervisor-for-your-home-lab
Booting from UEFI BIOS VMs that have >2TB drive sizes is another immediate 5.5 benefit, as is use of >2TB drive sizes in VMs.
http://TinkerTry.com/is-that-a-62tb-drive-in-my-home-lab
For me, and my Windows Server 2012 R2 Essentials plans, this was key (and is in use on my Windows Server 2012 Essentials VM now, backing up 12 PCs daily overnight effortlessly, with full bare metal restore capabilities).
But you're right, not everybody is looking to do those sorts of things, so staying put is an option.
If passthrough is your thing, ESXi 5.0 was better than 5.1 actually. I'm publishing an article on using passthrough for USB 3.0 today actually, stay tuned (it works very well).
Jim, and Avram, I think you've hit on a key point. There are people posting here about sticking with 5.1 or even downgrading to 5.0 in order to avoid the restrictions of the vSphere Client under ESX 5.5; but if you don't upgrade the hardware versions of your VMs to v10 there are no limitations of vSphere Client, are there? (someone correct me if I'm wrong).
So if, like me, you want USB pass-through that was broken in 5.1, and if your hardware is supported properly by 5.5, there are no disadvantages of 5.5 over what you already had in 5.1... you just have to remember not to upgrade the hardware version of existing VMs (and the vSphere client gives you plenty of warnings if you try).
I guess the key question is what are we missing by not being able to create and manage v10 VMs (like a previous poster I only see options to create v4, v7 or v8 machines).
Here's a link to hardware features for different VM versions:
http://pubs.vmware.com/vsphere-55/topic/com.vmware.vsphere.vm_admin.doc/GUID-789C3913-1053-4850-A0F0-E29C3D32B6DA.html
Of course, people wanting or needing to evaluate the latest and greatest features available from 5.5 are going to miss out, but for me at least there are benefits to be had from 5.5 without any immediate drawbacks as far as know.
Here's an FYI, an addendum I added to the article above:
Sep 26 2013 Update:
In my original post above, I didn’t call out, why VMware? We each have our own answers to that one. For me personally, it’s simple really. ESXi is at a vast majority of my customer’s sites, and as far as I know, it’s the only bare metal hypervisor that lets me test all the other big names in hypervisors. Testing by nesting, on just one Core i7 system I affectionately call vZilla. Invaluable for self training purposes, and very relevant to my career in IT.
On my next build, I might, if I can find the price close enough to commodity mobos to make it reasonable (in 2011, wasn't a viable choice with my budget, which is similar to most folks' budgets). This site is all about pushing the envelope on the lower end. I know it's easier on the server class gear, and I do appreciate all opinions. See also servethehome.com
Server 2012 Hyper-V does. Giving it a run now, after hitting the wall on our free ESXi hosts. Much easier to set up than I though it would.
Thanks Avram, you make valid points. For me, it isn’t about “free,” but moreover, “accessibility.” I realize that Paul’s site here is focused on “home labs,” and indeed I
used to be in that boat…placing servers at my house, hosting e-mail for the
extended family, etc. I’ve since simplified my life and pushed that stuff back out, and get by with a large, beefy PC with lots of storage, to house my immediate family’s data and media, and for me to play games and “tinker” with things in Workstation.
But back to my point…for my day job we have a smallish 7-host ESX 4.1 cluster in our primary data center, and that infrastructure is getting “long in the tooth,” and needs to be upgraded soon. I work primarily out of a small satellite office with only six people, and we have two servers at that location – an ESXi (now 5.5) host, and a file server which is also a DC for some geographic redundancy for that primary site. The ESXi host is mostly about “lab” stuff, where I have virtual versions of servers that are physically deployed at thousands of remote sites. I like to use it to “tinker” with new things, like Server 2012 [R2] and Windows 8.1, since I can’t run those on ESX 4.x. I’d like to know what “special” VM features are available in vhw version 10, but I currently don’t know, because it would take extra time and complexity to set up a vCenter Server, just so that I can run the web client. And it isn’t about “laziness” but rather “time and resources,” – like many others I’m sure, I’m in an organization that expects a lot from a little staff. Instead of a quick path to seeing some new features, which would potentially ignite a real interest and cause me to dedicate time and resources to it, and money, there’s an arbitrary barrier there. Similarly, the arbitrary barrier of not being able to [by default] run the vSphere client on the DC, just because some nannys at VMware decided to force some best practices on us, when there is no real technical reason behind the decision.
Best practices or not, I’m pretty sure the vSphere client isn’t going to push a DC “over the edge,” and running the client on my DC equals a lot of convenience for me.
This is just my two-cent rant – thanks for listening.
P.S. Perhaps I'll get a wild hair, and really push the envelope by attempting to install vCenter Server on my DC :-)
VMware is moving backwards for us homelab users. With the removal of support for some devices and no ability to edit a version 10 or higher VM, looks like I'm staying put at ESXi 5.1 for the time being.
Gonna look at other free hypervisors like KVM or Xen.
Avram, thank you for an excellent counterpoint, very helpful perspective. Much appreciated.
I guess I need (want?) to play Devil's Advocate here. So.... the free version limiting people is a large complaint? Don't get me wrong, I'd like more free in my box of cereal too. But if you're using the free version you don't get things like vMotion, right click cloning, customization templates, etc. If you're only using one host or spinning up VM's only every so often, I guess that's not a big deal. But an Essentials Kit for 6 sockets, 3 hosts and vCenter Server Essentials, is what, $600? If you're running infrastructure on vSphere, and $600 is out of budget...... Also, not-for-profit tends to get discounted product from most vendors (I'm not an NFP, so I don't know if VMware does or not)
As for the memory requirements for vCenter - 7GB? My home lab has a C6100 with 96GB RAM picked up for $800. The extra 96GB cost me $280. If you're running 10 VM's? This is < $100/VM. Granted, it's not "free" but it's not expensive either. It runs considerably more than 10 VM's as well. Granted, if you're looking for a single host box that is limited to 32GB....
You don't 'lose' the vSphere Client in 5.5. You lose the ability to EDIT VM Version 10 hardware. You can still start/stop/reboot/etc them. If you do NOT upgrade to VMX-10 hardware, you're just fine as well - there's a reason they ship it with VM Hardware v7. Absolute worst case, you can still edit the *.VMX file by hand.
vSphere client on a DC? Why?
Don't get me wrong. I have a home lab. Cheaper the better. But there's a lot of good practices being ignored, and just general 'cheapness' from a lot of what I see commented on here. Sure, not everyone should have to upgrade everything for the latest goodies. But it is _free_ right? Good god, that's a lot of complaining for no cost of entry.
I think people want to be able to passthrough usb ports and stuff. Unless I'm mistaken, Hyper-V doesn't support that.
Man, this is a frustrating development. I run the IT show for a medium size non-profit, and migrated our VM infrastructure to ESXi 5.1 last year. LOVE IT! I am setting up a proof of concept test box to shift our desktop architecture to thin client via ThinLinc (not going through the box by box upgrade on our Windows 7 machines like I am having to with XP). I *just* downloaded ESXi 5.5 and was looking for some insight about RAID support.
Boy is this an eye opening article. I am not sure which patch fixed the SSH issue, but losing the vSphere client feature is a show stopper, so I can't see going forward with 5.5. I guess I will wait and see how things play out for a few months, and try to get a budget for Essentials before I am ready to go beta on this thin client project. And putting money into Essentials only makes sense if the webclient can enable the same functions as the fat client.
I would really like to keep with ESXi, but VMWare is certainly complicating things. Yes, it's free, but at the cost of my sanity!
They also dropped support of the vSphere 5.5 fat client being installed on a domain controller, although artificially. There is a workaround: https://communities.vmware.com/message/2292358
Check out Supermicro Motherboards, just rolled my first one and I will never use another brand, affordable, and have IPMI which I have come to find invaluable. And everything Tom said below...
Thank you for weighing in vl, and I did watch the Archipel video overview. Always good to know what's out there!
Thank you for your contribution, and your frustration is palpable. This probably won't cheer you up, but there's a good list of stuff they dropped from ESXi 5.5 now out, the deprecated list KB 2056935:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2056935
Hi Paul, I am the one who posted the "Good news and Bad News" part of your article. Although 5.5 seems like a solid release, I have decided to roll back to 5.0U1 so I have passthrough capability and can still manage my ESXi host without vCenter. This was VERY painful because I had to use the standalone converter to migrate to a temporary 5.0u1 host. I did this so I could downgrade machine level from ver 10 to ver 8. The final straw was that my 3ware RAID card was no longer supported in 5.5. Using trial licenses I vMotioned all my recently migrated VM's now on machine level ver 8 to my rebuilt and permanent 5.0U1 host. All is well but this was a major hassle and for now it seems I will be infinitely stuck on 5.0U1, or until a free version of vCenter is released. I'm not hopeful.
I think the cost effective solutions at least for home users,
would be to move to a truly free hyper-visor like XenServer or Xen or KVM
all of them a free,
if you need the support than get a support for XenServer from Cisco.
other than that and openSuse setup with Xen or KVM
will get you a pretty good VM server where you can mange most of the things from basic GUI built into the OS.(Yast--VM Manager)
if you need better support install Archipel. a very nice, and comprehensive Web Base management client.
I mean for business with a budget VMWare is great, but for a home use where budget is non-existent free is always better :-)
That sure seems to be what the description is telling us, but I'm not 100% sure at this time that the license key it gives you would apply to the vCenter Appliance, although it's highly likely, which would mean yet, it allows the web client. By the way, I should have a video up of the full install and configure process for a 5.5 based home datacenter, later tonight, stay tuned!
Yeah, I hint at that a bit in my related post
http://TinkerTry.com/vmware-workstation-10-or-vmware-player-6-plus
and your URLs are quite handy references, thank you!
I'm finding that the bare metal hypervisor still greatly outperforms even the latest VMware Workstation, and makes a far more efficient use of memory, with useful features like auto start and unattended, graceful shutdown during power outages:
http://TinkerTry.com/configure-automated-shutdown-homelab-datacenter-15-minutes
and the ability to easily move VMs from machine to machine or from drive to drive, even while running. Yeah, I know, these are far more than the average home user's needs.
Different needs for different folks, indeed.
Thank you, "Reader" for chiming in, I really appreciate your stopping by!
I'm sorry. I should have read my post before I hit the arrow. Correction: Will the vCenter Server Essentials version allow me to install the web client and manage my ESXi 5.5 host?
So, the most cost effective solution would be to purchase the basic VMware vSphere Essentials Kit ($560) which includes vCenter Server Essentials? Am I correct? Will the vCenter Server Essentials version allow me to run the web client on ESXi 5.5?
As simple Home Lab solution you can always use the (free) vmware 6 player :
https://www.vmware.com/support/player60/doc/player-60-release-notes.html
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1014006
This is : 64Gb Physical RAM
4TB Disk size
16 cores
More than enough for the average home user.
Just an FYI, a very practical use of USB 2.0, with no VT-d needed, since my UPS's USB 2.0 monitoring cable is just mapped to a VM at will:
http://TinkerTry.com/configure-automated-shutdown-homelab-datacenter-15-minutes
That's not to say having true hypervisor recognition of USB 3.0 ports on a motherboard wouldn't be nice, for at-will assignment to VMs, that's really another issue, one that you noted VMware Workstation does handle.
Thank you for the careful and thoughtful thoughts on this difficult topic. It's always good to see things from other people's point of view.
See also:
http://TinkerTry.com/usb3speed
http://TinkerTry.com/usb-3-0-thumb-drive-speeds-thumb-drives-vary-widely
http://TinkerTry.com/usb3passthru
This is incredible work you've done here Tom, and know that within a day or two Google will have indexed your words as well (they've added Disqus crawling), This is a good thing. A very good thing.
I cannot tell you how much I appreciate all you've put into this post, and I know that all my site readers will learn a lot from it, myself included.
See also:
http://TinkerTry.com/vmware-esxi-5-5-free-hypervisor-breaks-past-32gb-restrictions
http://TinkerTry.com/vmdirectpath
I've been using esxi for years and since 5.1 came out, I've been struggling with the usual passthrough issues (trying to pass through USB 2 via vmdirectpath). This seems to be working in esxi 5.5 but for how long? With the cost (and ridiculous memory footprint) of the vcenter VM (to manage under 10 VM's btw), I am seriously considering trying out a headless ubuntu server install with VMWare Workstation 10. This way I can (hopefully) even gain access to my USB 3.0 onboard ports which are currently unusable under esxi. As long as performance doesn't suffer, this might be the better option for those of us who mostly use our vmware installs as media servers, homeautomation VM's, etc that need to interface with a lot of devices. esxi always seems two steps behind on hardware support too. None of my new motherboard's sensors show up in the client and an onboard NIC is unusable. Whereas, everything seems to just work on ubuntu. But until I try migrating, I won't know for sure if this'll be a viable alternative.
Oops, formatting on the previous comment is bad. Use this one instead...
This isn't as much of a motherboard issue as it is an Intel issue. Case in point, I have a Supermicro X9SRL-F configured with a Xeon E5-1660 and 8x16GB (128GB) of RAM. It works because the CPU's memory controller (IMC) can handle that many DIMMs. In fact, the IMC can handle up to 64GB LRDIMMs so I could put in 512GB of RAM on this single socket board and Xeon E5-26xx dual socket boards can take up to a terabyte of RAM.
Haswell (and other mainstream) CPUs won't handle these DIMMs. 32GB is the limit due to the IMC. This is a market segmentation decision by Intel and I don't see it being lifted any time soon. This way people with large memory requirements either go towards X79 chipset-based boards and Intel enthusiast-class CPUs (Core i7 38xx/39xx or Core i7 48xx/49xx) which allow up to 64GB of non-ECC RAM. If >32GB RAM and ECC is required then a Xeon chipset (Intel C60x) and Xeon CPUs.
There are cheaper options. The newly release 4-core/8-thread Intel 4820K costs ~US$330. It supports VT-d and up to 64GB of non-ECC RAM. There are some consumer motherboards that support VT-d (but have problems) and support 64GB of RAM on a X79 chipset. The problem is the need for 8 DIMM slots because Intel's non-Xeon CPUs that support greater than 32GB of RAM require the use of more than 4 DIMM slots (8 typically). This leaves us with the cheapest motherboard I could find: ASRock X79 Extreme6 for ~US$220. I would recommend contacting ASRock support to make sure VT-d works [1] *and* 64GB is supported with a 4820K CPU. It should be but CONFIRM it.
Honestly, I'm in two minds about this recommendation. On one hand it seems like ASRock is really taking VT-d seriously. They have a mainstream Haswell Xeon E3-12xx v3 capable mini-ITX motherboard coming out with ECC and VT-d support. [2] A couple of these (depending on price) could make up for the 16GB ECC RAM limit with density. ASRock are one of the few motherboard manufacturers that allow the use of VT-d on Z87 chipset motherboards when coupled with a Haswell Core CPU that supports VT-d. Then on the other side, I don't have a lot of experience with their boards to be able to give a solid recommendation.
If a homelab needs ECC RAM (virtualized or bare-metal FreeNAS with ZFS) then a single Supermicro mATX X10xxx motherboard (supporting VT-d and 32GB of ECC RAM) with a Xeon E3-12xx v3 CPU would be a perfect addition to non-ECC 64GB Asrock-based machine(s).
If a single do-it-all homelab is needed with VT-d and >32GB ECC RAM then it'll have to be a Xeon E5-16xx v2 CPU (the E5-1620v2 is about the same price as the 4820K) with a Supermicro X9SRL-F (~US$90 more than the ASRock motherboard above).
Another option might be Intel's Aveton-based Atom CPU that has 8 cores and support for up to 64GB of ECC RAM but the only two motherboard I've seen are the Supermicro A1SAi-2750F and ASRock C2750D4I. I haven't seen 16GB ECC SODIMMs yet so I'm not sure how any of the above two options will support >32GB of RAM at the moment. I also find the cost of these Atom CPUs to be a bit prohibitive.
I hope this helps your search.
[1] This thead says VT-d works: http://www.overclock.net/t/1338063/vt-d-compatible-motherboards
[2] ASRock E3C226D2I: http://www.asrock.com/server/overview.asp?Model=E3C226D2I
I completely agree. I don't want to be required to run vCenter in my home lab. Even the appliance, which is much lighter than a full Win2k8 install, seems like a waste of resources to me. Why do I need to run something that uses 4GB+ of RAM just to manage my server? I don't need all of the power of vCenter (logs, ect), just the ability to do basic management tasks. I also don't want to have to purchase (or pirate...) both an ESXi license and a vCenter license to avoid having to reinstall every 60 days...
Also interesting that there is no 5.5 version of this key document yet:
"Services bundled with vCenter Server Appliance (2002531)"
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=2002531&sliceId=1&docTypeID=DT_KB_1_1&dialogID=46588437&stateId=1%200%2046592192
Yes, it looks like they missed that warning in the documentation for those of us upgrading Windows based VCenter deployments.
I'm sure VMware will release a patch soon.
Good point, the vCenter Client code is still needed for this situation you wrote in about (thank you!)
Also note that the vCenter 5.5 documentation
http://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc%2FGUID-25FCBA87-5D2F-4CB6-85D7-88899B4AC174.html&resultof=%22server%22%20%22appliance%22%20%22applianc%22%20%22prerequisites%22%20%22prerequisit%22%20
talk about the virtual hardware. It's a strange sort of transition that we're in. We're kind of in limbo, or perhaps purgatory? ;-)
For convenience, here's the section of the doc I'm talking about.
"Version 5.5 of the vCenter Server Appliance is deployed with virtual hardware version 7, which supports eight virtual CPUs per virtual machine in ESXi. Depending on the hosts that you will manage with the vCenter Server Appliance, you might want to upgrade the ESXi hosts and update the hardware version of the vCenter Server Appliance to support more virtual CPUs:■ESXi 4.x supports up to virtual hardware version 7 with up to 8 virtual CPUs per virtual machine.■ESXi 5.0.x supports up to virtual hardware version 8 with up to 32 virtual CPUs per virtual machine.■ESXi 5.1.x supports up to virtual hardware version 9 with up to 64 virtual CPUs per virtual machine.
Caution
If you update the vCenter Server appliance to hardware version 10, you cannot edit the virtual machine settings for the appliance using the vSphere Client. This might cause difficulties in managing the vCenter Server Appliance, because you cannot use the vSphere Web Client to connect directly to the host on which the vCenter Server Appliance resides to manage it. Do not upgrade the vCenter Server Appliance to hardware version 10."
Yeah, not sure why you're not seeing version 10, I've had that working just fine, see also the graphic at the top of the site where this comment sits:
http://TinkerTry.com/vmware-vsphere-5-5-announced-with-general-availability-likely-the-week-of-september-9-2013/#comment-1033120721
Hit this last night, unfortunately I needed to increase the memory to our VCenter VM, but couldn't because the VM needed to be powered off to do so which took down the web client.
So even if you have purchased VCenter, it can still hit you.
I'd be happier if VMware didn't let their marketing folks create gotchas like this. I understand they need to make money, but jeez, give us a light web client for the free hypervisor so we're more likely to kick the tires for advanced stuff, and potentially buy it for our day jobs. This kind of thing motivates me to take another look at Hyper-V, whether it's inferior or not.
Just curious. I've built two VMs with Virtual Center Appliance 5.5 now and I see that the highest hardware version you can use is 8. Why doesn't it let me build VMs in version 10, and why do we need version 10? I guess I have some Googling to do...
Jim, you're comment cracks me up, and I thank you for that. So glad Josh took a moment to drop that comment, so you could do the most convoluted gyrations, and wind up happy ;-)
Wow, great tip. I upgraded a 5.1 host to 5.5, updated the virtual hardware in my two VMs, and then got the "web client denial" when going to edit the virtual hardware. So I remoted in to my home PC which has Workstation 10 on it, VPN'd to the office, connected Workstation to my 5.5 host, and then edited the hardware on my VM. Awesome, and stupid at the same time...
Managing ESXi 5.5 Autostart VMs from VMware Workstation 10.
It does appear to be rather restrictive, as far as what can be done from the Workstation 10 UI.
See screenshot "Managing ESXi 5.5 basics from VMware Workstation 10" below:
Interesting thought Josh. More about the recently released VMware Workstation 10 here:
http://TinkerTry.com/vmware-workstation-10-or-vmware-player-6-plus
and details about connecting to a remote ESX / ESXi / vCenter Server from VMware Workstation (2005822) here:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2005822
You can also manage this using VMWare workstation. I know it still requires a license but might be a work around for some.
Good point. It seems our next big bottlneck is finding an affordable (non-server-class) motherboard that can handle >32GB of RAM. Not so much with latest Haswell. See also:
http://TinkerTry.com/vmware-esxi-5-5-free-hypervisor-breaks-past-32gb-restrictions
http://TinkerTry.com/32gb-boundary
Paul Braren | TinkerTry.com
I have not tried that version myself, but the answer sure seems to be yes, based on what I'm reading here:
http://www.vmware.com/products/vsphere/compare
When it says "Server virtualization and consolidation with centralized management" that would sure look to be the vCenter/Web Client combo.