VMUG Advantage EVALExperience includes latest VMware vRealize Log Insight 4.5 syslog server appliance for easy vSphere, vSAN, IoT, and networking gear log file analysis

Posted by Paul Braren on Oct 16 2017 (updated on Sep 21 2018) in

My 10GbE network has fallen to 1GbE and can't get up

I was working with Netgear ProSUPPORT Services for Business Users on an issue I was having with one of my 2 Xeon D SuperServers this past summer. Specifically, my XS708T 10GbE switch was staying connected at 10GbE for one Xeon D server, but on the other Xeon D server, speeds would mysteriously drop to 1GbE after some random time interval. The amount of time ranged from about 2 to 20 hours. This misbehavior really put a cramp in my speedy NVMe to NVMe vMotions! The only fix was to shutdown and unplug power, then power back up again. This was rather disruptive inelegant "fix," and it quickly became quite annoying. The cause wasn't apparent, the problem was intermittent and almost random, and there was no obvious way to cause it to fail faster so I could get to root cause quicker.

This issue lingered for weeks. I really didn't want to climb into my attic to replace the 100' CAT7 cabling I had just proudly installed and tweeted to the world about, as a just-in-case last-ditch I'll-try-anything approach. Instead, I wanted to science this, to get some data that'd help me solve this problem smartly, to stop with all this guessing nonsense.

I was asked "what syslog server are your running," imagine my embarrassment

Netgear L1 support recommended I share my logs from my syslog server with them. I thought to myself, doh! I didn't have a syslog server right now, oh no, how embarrassing! I recalled that I knew of just the syslog server I had always wanted to try and that could be mighty handy right about now. So I told the representative that I'd be using VMware vRealize Log Insight, and I'd get back to him with the syslog export data later that night. So I set forth on making a plan to make it so.

First, I noted that I'd need to make sure my 10GbE switch was NTP time synced, just like the rest of my vSphere cluster already was. I'd then "point" my XS708T switch to the vRealize IP address, and cranked up the switch log output to 11, also known as the Debug level. This new syslog server would allow the switch logs to be collected and could be used to alert me to whenever the 10GbE connection speed changed again in the future. This logging and alerting might even help me see if we could spot any correlation with any vSphere related events. This was just the occasion I'd been waiting for, to give vRealize Log Insight a try, to see if I could use it as one tool in getting to the bottom of this particular networking issue, and other surprises down the road.

So it was was time to log in to VMUG Advantage EVALExperience:
vmugadvantage.onthehub.com

to see what vRealize Log Insight version they had available for current subscribers like myself. Never mind that I happen to work at VMware now, what I really wanted to know what everybody else who also uses VMUG Advantage EVALExperience can also use in their non-production environment, for free! Read all about EVALExperience here.

Momentary sadness

At the time of my initial testing in August 2017, all I found at EVALExperience was the older vRealize Log Insight 4.0. This was disappointing. So I Googled around for a bit, and soon found 4.3 in my my.vmware.com account, so I installed 4.3. Of course, I took a moment to "roll camera," recording everything I did right as I went about testing it in my home lab, just in case this first-timer's experience would be useful to share someday. Of course, I recorded an impromptu voice-over as I went along, seen in the video below.

All my problems have been solved, here's how it works

vRealizeOrder — On Fri Oct 13th, I notified VMUG Advantage that the OVA for 4.5 was missing, and by Mon Oct 16, they fixed it!

Well, at least logged. Here's the gist of the simple install and configure process. Once you've downloaded and deployed the (SuSE-based) OVA "appliance" file, you point your browser to the assigned IP, then connect it to your vSphere infrastructure by providing it with VCSA 6.5U1 credentials.

Here's the really slick part. It then logged in to VCSA on my behalf, and quickly went about automatically configuring syslogging from all of my ESXi 6.5U1 servers. How cool is that? Easy as pie, as I demonstrated at this exact spot in the video below. No need to dive into each host with clumsy manual tweaks, which is an important part of bringing up a home lab for those like me using devices like SD or USB for ESXi itself. This syslog auto-configuration also side-steps the need to follow Configuring syslog on ESXi - KB 2003322, and is good news for even the laziest of home lab sysadmins, who really don't mind some simplicity on the home front once in a while.

Later on, on October 14th actually, I discovered that 4.3 had actually been released back in June, oops! But I also found that the features were nearly identical so the install procedure was still the same, and upgrading to 4.5 later on was no big deal at all. Thus, I decided to add the simple .PAK upgrade footage to this same video, also just published. The timing worked out great, because today EVALExperience upgraded their download to the very latest version 4.5! Note, the same bits are also available for full (purchased) license holders at My VMware here.

I don't have to choose

Event-Forwarding — Click to view the Event Forwarding panel, to forward events to another syslog server too.

There are many other solutions for syslogging. Does this mean I've made up my mind, as for my solution for my home lab? Nope. Apparently, I can forward all events to another syslog server, such as the popular, feature-rich, and free SexiLog. Nice! Admittedly, I've not yet tried this myself.

I've been using various syslog products since the 90s. Hey you bashful former Linksys router owners out there. Remember that Kiwi Syslog Server, and that free WallWatcher? Now you can step-up to far great functionality, for your vSphere, and for all your other networked gear that features syslogging.

What about my network problem?

Yes, vRealize helped me reliably provide meaningful logs to Netgear by using its native export function. Basically, before I could get escalated to L2 Support, I had to follow their script, and get them the requested syslogs, which I was able to do. For my strange bug, in the end, using the Intel X557 4.4.2 driver VIB and the latest Netgear firmware helped alleviate the problem, but resolving it was hardware and the complete story is yet to be told in a future blog post here at TinkerTry. Subscribe to get notified automatically!

Ready for vSAN

For folks running vSAN, great, vRealize Log Insight has you covered. Read the excerpt about the included content pack for vSAN below. In my role as a VMware vSAN System Engineer, the more I know about any products that have anything to do with vSAN, the better. What better way than to leave vRealize Log Insight running in my home lab, ready for anything I throw at?

This is not official VMware documentation, and nobody asked me to write this post.

Step-by-Step vRealize Log Insight Install/Configure/Alert/Upgrade

Here's what the action packed full length video below offers you a look at:

vRealize Log Insight Appliance install using the vSphere Client (HTML5)
Configure connection with vSphere 6.5 (VCSA 6.5U1/ESXi 6.5U1)
Alerts via email, I used a Gmail account
Search syslog, create filters, and configure alerts
Configure 10GbE Netgear XS708T 10GbE switch logging to vRealize for debug
Update from 4.3 to 4.5 via upgrade .PAK file

Don't forget to set it to auto-start at boot, and to back it up

Set this appliance to start with your hosts, see Automating the process of starting and stopping virtual machines on VMware ESX\ESXi - KB 850
Configure daily automated backups using something like Veeam Backup & Replication or NAKIVO VMware Backup. Both of these links go to their respective NFR code request forms, to request free access to the code for use in your non-production environment. There are many other backup solutions such as Vembu VMBackup that I haven't yet tested.

Video

VMware vRealize Log Insight installation/upgrade/alerting configuration for Netgear XS708T syslogs

vRealize Log Insight Home Page

vRealize Log Insight: Intelligent Log Management & Analytics
Click to visit the vRealize Log Insight homepage.

vRealize-Log-Insight-System-Monitor — System Monitor view.

Oct 22 2017 Update

VMware's Steve Flanders presenting at the Boston VMUG UserCon Oct 19 2017.

Luck would have it that I bumped into Steve Flanders at the Boston VMUG UserCon this past Thursday. That's right, none other than @smflanders, VMware vRealize Log Insight Staff Architect & Senior Manager, note the familiar Twitter logo in his pinned tweet below, featuring all sorts of great references:

Oct 24 2017 Update

I added my Ubiquiti EdgeRouter Lite to the growing list of devices I set to log output to my vRealize Log Insight syslog server, somehow forgot to tell you that.

Sep 21 2018 Update

‪vRealize Log Insight 4.7 is now available. For those who already have an earlier version of the appliance installed, just download the .pak from https://my.vmware.com/web/vmware/info/slug/infrastructure_operations_management/vmware_vrealize_log_insight/4_7 then apply it using simple method at https://docs.vmware.com/en/vRealize-Log-Insight/4.7/com.vmware.log-insight.administration.doc/GUID-C89E69DF-5950-432B-B35F-F9DFE28855A6.html‬

New cybersecurity best practices are emerging for IoT
Applying tactics commonly used to secure laptops and servers is not enough for Internet of Things devices and apps.
Jul 27 2017 by Bill Siwicki at HealthcareITNews
vRealize LogInsight 4: The best syslog solution you didn’t know you had
Feb 18 2017 by Jon Kensy at JonKensy.com
Using vRealize Log Insight Content Pack for vSAN for better visibility
Feb 16 2017 by Pete Koehler at VMware Blogs:

Many perceive log data as nothing more than an insurance policy for when something goes wrong. This perception implies that if everything is functioning as expected, then there is little value to log data. This misconception overlooks one of the key benefits log data can provide for you and your environment. When used correctly, with the right tools, log data can provide context and understanding to changing conditions in the data center. Alarm mechanisms, while important, often indicate just a state or condition, and can be transient, only remaining visible during the period of time the alarm threshold is met. Log data tells a much more detailed story, and does so over time. Log analytics can complement other forms of information gathering, such as performance graphs, and alarms. Capitalizing on the intelligence buried in log data is a superb opportunity to manage a data center in a smarter way.

I had the honor of sitting in on Ryan Johnson presenting and proctoring a Cloud Foundation Hands On Lab in Cincinnati earlier this year, as I was there to do the vSAN Hands On Lab anyway! Here's his very-relevant article:

VMware Validated Design for Software-Defined Data Center 4.1 is Now Generally Available
Aug 22 2017 by Ryan Johnson at VMware Blogs:

A quick note on management packs and content packs, too. While there have been some minor version updates to these, it’s worth mentioning that vRealize Operations 6.6.1 now includes the management packs for vSAN, vRealize Automation, and vRealize Business for Cloud in the product deployment and they not longer have to be installed post-deployment. Similarly, vRealize Log Insight now includes the content pack for vSAN in the product deployment. Lastly, we’ve added the Content Pack for Linux to the BOM to pull log data from the virtual appliances directly into vRealize Log Insight.

Marketplace - Content Packs | vRealize Log Insight

Log-Insight-Content-Pack-Marketplace-Oct-22-2017 — Log Insight Content Pack Marketplace, as it appeared on Oct 22 2017. Note that vRops 6.x, vSAN, and vSphere are correctly showing as already installed.

On Oct 22 2017, I grabbed the entire Log Insight Content Pack Marketplace text, for future reference:

Log Insight Content Pack Marketplace

Apache - HTTP Server
Version:    1.0
Author: VMware, Inc.

Apache - Tomcat
Version:    1.0
Author: VMware, Inc.

Apache - CLF
Version:    1.2
Author: VMware, Inc.

Arista - EOS
Version:    1.0
Author: Arista Networks, Inc.

BigSwitchNetworks - BCF
Version:    1.0
Author: Big Switch Networks

Brocade - SAN & IP Networks
Version:    3.2
Author: Brocade

Cisco - ASA
Version:    1.5
Author: VMware, Inc.

Cisco - Nexus
Version:    2.1
Author: VMware, Inc.

Cisco - UCS
Version:    1.5
Author: Cisco Systems, Inc.

DataGravity - Discovery Array
Version:    1.0
Author: DG Labs

Dell EMC - VMAX
Version:    1.0
Author: Dell EMC

Dell - iDRAC
Version:    1.1
Author: VMware, Inc.

Dell Networking
Version:    1.0
Author: Dell Inc.

Dell EMC OS10 Networking
Version:    1.0
Author: Dell Technologies

EMC - VMAX
Version:    3.0
Author: EMC Corporation

EMC - VNX Unified
Version:    1.0
Author: VMware, Inc.

EMC-XtremIO
Version:    1.5
Author: VMware Inc.

ExtraHop - Wire Data
Version:    1.0
Author: ExtraHop Networks

Extreme Networks - Purview Analytics
Version:    1.0
Author: Extreme Networks

F5 - BIG-IP
Version:    1.0
Author: VMware, Inc.

General
Version:    2.5
Author: VMware, Inc.
Installed

HAProxy
Version:    1.0
Author: VMware, Inc.

Hitachi - Server
Version:    1.0
Author: Hitachi, Ltd

Hitachi - Storage
Version:    1.0
Author: Hitachi, Ltd.

HP - Servers
Version:    1.0
Author: Blue Medora

HP - StoreFront Analytics
Version:    1.0
Author: Hewlett-Packard

HP - TippingPoint
Version:    1.0
Author: Vater Operations GmbH

INFINIDAT - InfiniBox
Version:    1.0
Author: INFINIDAT

Infoblox - DDI
Version:    1.5
Author: Infoblox

Juniper - Security
Version:    1.0
Author: Juniper Networks, Inc

Kaminario-K2
Version:    1.0
Author: Kaminario

Lenovo – Networking CNOS
Version:    2.0
Author: Lenovo

Lenovo - Networking
Version:    1.2
Author: Lenovo

Lenovo – xClarity
Version:    1.1
Author: Lenovo

Linux
Version:    1.0
Author: VMware, Inc.

Microsoft - Exchange
Version:    3.2
Author: VMware, Inc.

Microsoft - .NET CLR
Version:    3.0
Author: VMware, Inc.

Microsoft - Active Directory
Version:    3.2
Author: VMware, Inc.

Microsoft - IIS
Version:    3.0
Author: VMware, Inc.

Microsoft - SharePoint
Version:    3.0
Author: VMware, Inc.

Microsoft - SQL Server
Version:    3.1
Author: VMware, Inc.

Microsoft - Windows
Version:    3.5
Author: VMware, Inc.

MongoDB - Database
Version:    1.0
Author: Blue Medora

NetApp - Data ONTAP
Version:    1.0
Author: Blue Medora

Nginx
Version:    1.0
Author: VMware, Inc.

Nimble Storage
Version:    1.0
Author: Blue Medora

Nutanix - Clusters
Version:    1.1
Author: Blue Medora

OpenStack
Version:    1.0
Author: VMware, Inc.

Oracle - Database
Version:    1.0
Author: Blue Medora

Oracle - JRE
Version:    1.2
Author: VMware, Inc.

Palo Alto Networks - PAN-OS
Version:    1.1
Author: Palo Alto Networks, Inc.

Pivotal - Cloud Foundry
Version:    1.0
Author: Blue Medora

Puppet Enterprise
Version:    1.0
Author: VMware, Inc.

Pure Storage - FlashArray
Version:    2.5
Author: Pure Storage

SolarWinds - NPM
Version:    1.0
Author: Blue Medora

Synology - DSM
Version:    2.0
Author: VMware, Inc.

Veeam Backup & Replication
Version:    1.0
Author: Veeam Software AG

VMware - EVO SDDC Suite
Version:    1.0
Author: VMware, Inc.

VMware - Horizon View
Version:    3.3
Author: VMware, Inc.

VMware Identity Manager
Version:    1.0
Author: VMware Inc.

VMware - NSX-T
Version:    2.0
Author: VMware, Inc.

VMware - NSX-vSphere
Version:    3.7
Author: VMware Inc.

VMware - Orchestrator - 7.0.1+
Version:    2.0
Author: VMware, Inc.

VMware - Orchestrator
Version:    1.1
Author: VMware, Inc.

VMware - SRM
Version:    1.5
Author: VMware Inc.

VMware - vCAC 6.0
Version:    1.0
Author: VMware, Inc.

VMware - vCloud Director
Version:    8.8
Author: VMware

VMware - vCNS
Version:    1.0
Author: VMware, Inc.

VMware - vC Ops 5.x
Version:    1.0
Author: VMware, Inc.

VMware - vRA 7
Version:    1.5
Author: VMware, Inc.

VMware - vRA 6.1+
Version:    1.1
Author: VMware, Inc.

VMware - vRops 6.x
Version:    1.7
Author: VMware, Inc.
Installed

VMware - VSAN
Version:    2.0
Author: VMware Inc.
Installed

VMware - vSphere
Version:    1.0
Author: VMware, Inc.
Installed