VMUG Advantage EVALExperience includes latest VMware vRealize Log Insight 4.5 syslog server appliance for easy vSphere, vSAN, IoT, and networking gear log file analysis
My 10GbE network has fallen to 1GbE and can't get up
I was working with Netgear ProSUPPORT Services for Business Users on an issue I was having with one of my 2 Xeon D SuperServers this past summer. Specifically, my XS708T 10GbE switch was staying connected at 10GbE for one Xeon D server, but on the other Xeon D server, speeds would mysteriously drop to 1GbE after some random time interval. The amount of time ranged from about 2 to 20 hours. This misbehavior really put a cramp in my speedy NVMe to NVMe vMotions! The only fix was to shutdown and unplug power, then power back up again. This was rather disruptive inelegant "fix," and it quickly became quite annoying. The cause wasn't apparent, the problem was intermittent and almost random, and there was no obvious way to cause it to fail faster so I could get to root cause quicker.
This issue lingered for weeks. I really didn't want to climb into my attic to replace the 100' CAT7 cabling I had just proudly installed and tweeted to the world about, as a just-in-case last-ditch I'll-try-anything approach. Instead, I wanted to science this, to get some data that'd help me solve this problem smartly, to stop with all this guessing nonsense.
I was asked "what syslog server are your running," imagine my embarrassment
Netgear L1 support recommended I share my logs from my syslog server with them. I thought to myself, doh! I didn't have a syslog server right now, oh no, how embarrassing! I recalled that I knew of just the syslog server I had always wanted to try and that could be mighty handy right about now. So I told the representative that I'd be using VMware vRealize Log Insight, and I'd get back to him with the syslog export data later that night. So I set forth on making a plan to make it so.
First, I noted that I'd need to make sure my 10GbE switch was NTP time synced, just like the rest of my vSphere cluster already was. I'd then "point" my XS708T switch to the vRealize IP address, and cranked up the switch log output to 11, also known as the
Debug level. This new syslog server would allow the switch logs to be collected and could be used to alert me to whenever the 10GbE connection speed changed again in the future. This logging and alerting might even help me see if we could spot any correlation with any vSphere related events. This was just the occasion I'd been waiting for, to give vRealize Log Insight a try, to see if I could use it as one tool in getting to the bottom of this particular networking issue, and other surprises down the road.
So it was was time to log in to VMUG Advantage EVALExperience:
to see what vRealize Log Insight version they had available for current subscribers like myself. Never mind that I happen to work at VMware now, what I really wanted to know what everybody else who also uses VMUG Advantage EVALExperience can also use in their non-production environment, for free! Read all about EVALExperience here.
At the time of my initial testing in August 2017, all I found at EVALExperience was the older vRealize Log Insight 4.0. This was disappointing. So I Googled around for a bit, and soon found 4.3 in my my.vmware.com account, so I installed 4.3. Of course, I took a moment to "roll camera," recording everything I did right as I went about testing it in my home lab, just in case this first-timer's experience would be useful to share someday. Of course, I recorded an impromptu voice-over as I went along, seen in the video below.
All my problems have been solved, here's how it works
Well, at least logged. Here's the gist of the simple install and configure process. Once you've downloaded and deployed the (SuSE-based) OVA "appliance" file, you point your browser to the assigned IP, then connect it to your vSphere infrastructure by providing it with VCSA 6.5U1 credentials.
Here's the really slick part. It then logged in to VCSA on my behalf, and quickly went about automatically configuring syslogging from all of my ESXi 6.5U1 servers. How cool is that? Easy as pie, as I demonstrated at this exact spot in the video below. No need to dive into each host with clumsy manual tweaks, which is an important part of bringing up a home lab for those like me using devices like SD or USB for ESXi itself. This syslog auto-configuration also side-steps the need to follow Configuring syslog on ESXi - KB 2003322, and is good news for even the laziest of home lab sysadmins, who really don't mind some simplicity on the home front once in a while.
Later on, on October 14th actually, I discovered that 4.3 had actually been released back in June, oops! But I also found that the features were nearly identical so the install procedure was still the same, and upgrading to 4.5 later on was no big deal at all. Thus, I decided to add the simple
.PAK upgrade footage to this same video, also just published. The timing worked out great, because today EVALExperience upgraded their download to the very latest version 4.5! Note, the same bits are also available for full (purchased) license holders at My VMware here.
I don't have to choose
There are many other solutions for syslogging. Does this mean I've made up my mind, as for my solution for my home lab? Nope. Apparently, I can forward all events to another syslog server, such as the popular, feature-rich, and free SexiLog. Nice! Admittedly, I've not yet tried this myself.
I've been using various syslog products since the 90s. Hey you bashful former Linksys router owners out there. Remember that Kiwi Syslog Server, and that free WallWatcher? Now you can step-up to far great functionality, for your vSphere, and for all your other networked gear that features syslogging.
What about my network problem?
Yes, vRealize helped me reliably provide meaningful logs to Netgear by using its native export function. Basically, before I could get escalated to L2 Support, I had to follow their script, and get them the requested syslogs, which I was able to do. For my strange bug, in the end, using the Intel X557 4.4.2 driver VIB and the latest Netgear firmware helped alleviate the problem, but resolving it was hardware and the complete story is yet to be told in a future blog post here at TinkerTry. Subscribe to get notified automatically!
Ready for vSAN
For folks running vSAN, great, vRealize Log Insight has you covered. Read the excerpt about the included content pack for vSAN below. In my role as a VMware vSAN System Engineer, the more I know about any products that have anything to do with vSAN, the better. What better way than to leave vRealize Log Insight running in my home lab, ready for anything I throw at?
This is not official VMware documentation, and nobody asked me to write this post.
Step-by-Step vRealize Log Insight Install/Configure/Alert/Upgrade
Here's what the action packed full length video below offers you a look at:
- vRealize Log Insight Appliance install using the vSphere Client (HTML5)
- Configure connection with vSphere 6.5 (VCSA 6.5U1/ESXi 6.5U1)
- Alerts via email, I used a Gmail account
- Search syslog, create filters, and configure alerts
- Configure 10GbE Netgear XS708T 10GbE switch logging to vRealize for debug
- Update from 4.3 to 4.5 via upgrade .PAK file
Don't forget to set it to auto-start at boot, and to back it up
- Set this appliance to start with your hosts, see Automating the process of starting and stopping virtual machines on VMware ESX\ESXi - KB 850
- Configure daily automated backups using something like Veeam Backup & Replication or NAKIVO VMware Backup. Both of these links go to their respective NFR code request forms, to request free access to the code for use in your non-production environment. There are many other backup solutions such as Vembu VMBackup that I haven't yet tested.
vRealize Log Insight Home Page
Oct 22 2017 Update
Luck would have it that I bumped into Steve Flanders at the Boston VMUG UserCon this past Thursday. That's right, none other than @smflanders, VMware vRealize Log Insight Staff Architect & Senior Manager, note the familiar Twitter logo in his pinned tweet below, featuring all sorts of great references:
Oct 24 2017 Update
I added my Ubiquiti EdgeRouter Lite to the growing list of devices I set to log output to my vRealize Log Insight syslog server, somehow forgot to tell you that.
Sep 21 2018 Update
vRealize Log Insight 4.7 is now available. For those who already have an earlier version of the appliance installed, just download the .pak from https://my.vmware.com/web/vmware/info/slug/infrastructure_operations_management/vmware_vrealize_log_insight/4_7 then apply it using simple method at https://docs.vmware.com/en/vRealize-Log-Insight/4.7/com.vmware.log-insight.administration.doc/GUID-C89E69DF-5950-432B-B35F-F9DFE28855A6.html
See also at TinkerTry
In this story, you'll see syslogging figuring prominently as a tool for 10G networking problem determination/root cause analysis.
- Temporary workaround to recover from intermittent Intel X552/X557 10GbE network outages on 12 and 16 core Xeon D, hoping for a public firmware update fix
Dec 26 2017
- VMUG Advantage just added the latest VMware Workstation 14 Pro and Fusion 10 Pro to EVALExperience
Oct 12 2017
- Promise SANLink3 T1 NBaseT Adapter blesses your Thunderbolt 3/USB-C desktop or laptop with 1.0/2.5/5.0/10GbE speeds
Jul 30 2017
- VMUG Advantage just added the latest NSX 6.3.1 and All Flash vSAN 6.6 bits to EVALExperience, great for vSphere home labs, just $200 a year!
May 01 2017
- Netgear XS708T managed 8 port 10GbE switch unboxed, noise and power draw measured - suited for Xeon D cluster
Jul 31 2016
New cybersecurity best practices are emerging for IoT
Applying tactics commonly used to secure laptops and servers is not enough for Internet of Things devices and apps.
Jul 27 2017 by Bill Siwicki at HealthcareITNews
vRealize LogInsight 4: The best syslog solution you didn’t know you had
Feb 18 2017 by Jon Kensy at JonKensy.com
- Using vRealize Log Insight Content Pack for vSAN for better visibility
Feb 16 2017 by Pete Koehler at VMware Blogs:
Many perceive log data as nothing more than an insurance policy for when something goes wrong. This perception implies that if everything is functioning as expected, then there is little value to log data. This misconception overlooks one of the key benefits log data can provide for you and your environment. When used correctly, with the right tools, log data can provide context and understanding to changing conditions in the data center. Alarm mechanisms, while important, often indicate just a state or condition, and can be transient, only remaining visible during the period of time the alarm threshold is met. Log data tells a much more detailed story, and does so over time. Log analytics can complement other forms of information gathering, such as performance graphs, and alarms. Capitalizing on the intelligence buried in log data is a superb opportunity to manage a data center in a smarter way.
I had the honor of sitting in on Ryan Johnson presenting and proctoring a Cloud Foundation Hands On Lab in Cincinnati earlier this year, as I was there to do the vSAN Hands On Lab anyway! Here's his very-relevant article:
- VMware Validated Design for Software-Defined Data Center 4.1 is Now Generally Available
Aug 22 2017 by Ryan Johnson at VMware Blogs:
A quick note on management packs and content packs, too. While there have been some minor version updates to these, it’s worth mentioning that vRealize Operations 6.6.1 now includes the management packs for vSAN, vRealize Automation, and vRealize Business for Cloud in the product deployment and they not longer have to be installed post-deployment. Similarly, vRealize Log Insight now includes the content pack for vSAN in the product deployment. Lastly, we’ve added the Content Pack for Linux to the BOM to pull log data from the virtual appliances directly into vRealize Log Insight.
Marketplace - Content Packs | vRealize Log Insight
On Oct 22 2017, I grabbed the entire Log Insight Content Pack Marketplace text, for future reference:
Log Insight Content Pack Marketplace Apache - HTTP Server Version: 1.0 Author: VMware, Inc. Apache - Tomcat Version: 1.0 Author: VMware, Inc. Apache - CLF Version: 1.2 Author: VMware, Inc. Arista - EOS Version: 1.0 Author: Arista Networks, Inc. BigSwitchNetworks - BCF Version: 1.0 Author: Big Switch Networks Brocade - SAN & IP Networks Version: 3.2 Author: Brocade Cisco - ASA Version: 1.5 Author: VMware, Inc. Cisco - Nexus Version: 2.1 Author: VMware, Inc. Cisco - UCS Version: 1.5 Author: Cisco Systems, Inc. DataGravity - Discovery Array Version: 1.0 Author: DG Labs Dell EMC - VMAX Version: 1.0 Author: Dell EMC Dell - iDRAC Version: 1.1 Author: VMware, Inc. Dell Networking Version: 1.0 Author: Dell Inc. Dell EMC OS10 Networking Version: 1.0 Author: Dell Technologies EMC - VMAX Version: 3.0 Author: EMC Corporation EMC - VNX Unified Version: 1.0 Author: VMware, Inc. EMC-XtremIO Version: 1.5 Author: VMware Inc. ExtraHop - Wire Data Version: 1.0 Author: ExtraHop Networks Extreme Networks - Purview Analytics Version: 1.0 Author: Extreme Networks F5 - BIG-IP Version: 1.0 Author: VMware, Inc. General Version: 2.5 Author: VMware, Inc. Installed HAProxy Version: 1.0 Author: VMware, Inc. Hitachi - Server Version: 1.0 Author: Hitachi, Ltd Hitachi - Storage Version: 1.0 Author: Hitachi, Ltd. HP - Servers Version: 1.0 Author: Blue Medora HP - StoreFront Analytics Version: 1.0 Author: Hewlett-Packard HP - TippingPoint Version: 1.0 Author: Vater Operations GmbH INFINIDAT - InfiniBox Version: 1.0 Author: INFINIDAT Infoblox - DDI Version: 1.5 Author: Infoblox Juniper - Security Version: 1.0 Author: Juniper Networks, Inc Kaminario-K2 Version: 1.0 Author: Kaminario Lenovo – Networking CNOS Version: 2.0 Author: Lenovo Lenovo - Networking Version: 1.2 Author: Lenovo Lenovo – xClarity Version: 1.1 Author: Lenovo Linux Version: 1.0 Author: VMware, Inc. Microsoft - Exchange Version: 3.2 Author: VMware, Inc. Microsoft - .NET CLR Version: 3.0 Author: VMware, Inc. Microsoft - Active Directory Version: 3.2 Author: VMware, Inc. Microsoft - IIS Version: 3.0 Author: VMware, Inc. Microsoft - SharePoint Version: 3.0 Author: VMware, Inc. Microsoft - SQL Server Version: 3.1 Author: VMware, Inc. Microsoft - Windows Version: 3.5 Author: VMware, Inc. MongoDB - Database Version: 1.0 Author: Blue Medora NetApp - Data ONTAP Version: 1.0 Author: Blue Medora Nginx Version: 1.0 Author: VMware, Inc. Nimble Storage Version: 1.0 Author: Blue Medora Nutanix - Clusters Version: 1.1 Author: Blue Medora OpenStack Version: 1.0 Author: VMware, Inc. Oracle - Database Version: 1.0 Author: Blue Medora Oracle - JRE Version: 1.2 Author: VMware, Inc. Palo Alto Networks - PAN-OS Version: 1.1 Author: Palo Alto Networks, Inc. Pivotal - Cloud Foundry Version: 1.0 Author: Blue Medora Puppet Enterprise Version: 1.0 Author: VMware, Inc. Pure Storage - FlashArray Version: 2.5 Author: Pure Storage SolarWinds - NPM Version: 1.0 Author: Blue Medora Synology - DSM Version: 2.0 Author: VMware, Inc. Veeam Backup & Replication Version: 1.0 Author: Veeam Software AG VMware - EVO SDDC Suite Version: 1.0 Author: VMware, Inc. VMware - Horizon View Version: 3.3 Author: VMware, Inc. VMware Identity Manager Version: 1.0 Author: VMware Inc. VMware - NSX-T Version: 2.0 Author: VMware, Inc. VMware - NSX-vSphere Version: 3.7 Author: VMware Inc. VMware - Orchestrator - 7.0.1+ Version: 2.0 Author: VMware, Inc. VMware - Orchestrator Version: 1.1 Author: VMware, Inc. VMware - SRM Version: 1.5 Author: VMware Inc. VMware - vCAC 6.0 Version: 1.0 Author: VMware, Inc. VMware - vCloud Director Version: 8.8 Author: VMware VMware - vCNS Version: 1.0 Author: VMware, Inc. VMware - vC Ops 5.x Version: 1.0 Author: VMware, Inc. VMware - vRA 7 Version: 1.5 Author: VMware, Inc. VMware - vRA 6.1+ Version: 1.1 Author: VMware, Inc. VMware - vRops 6.x Version: 1.7 Author: VMware, Inc. Installed VMware - VSAN Version: 2.0 Author: VMware Inc. Installed VMware - vSphere Version: 1.0 Author: VMware, Inc. Installed