VMUG Advantage EVALExperience includes latest VMware vRealize Log Insight 4.5 syslog server appliance for easy vSphere, vSAN, IoT, and networking gear log file analysis

Posted by Paul Braren on Oct 16 2017 (updated on Oct 24 2017) in
  • HowTo
  • Network
  • Review
  • Virtualization
  • 2017-10-16_20-51-59
    Main Dashboard looks pretty cool, click the image to zoom in and have a look around.

    Boo hoo, my 10GbE network falls to 1GbE and can't get up

    I was working with Netgear ProSUPPORT Services for Business Users on an issue I was having with one of my 2 Xeon D SuperServers this past summer. Specifically, my XS708T 10GbE switch was staying connected at 10GbE for one Xeon D server, but on the other Xeon D server, speeds would mysteriously drop to 1GbE after some random time interval. The amount of time ranged from about 2 to 20 hours. This misbehavior really put a cramp in my speedy NVMe to NVMe vMotions! The only fix was to shutdown and unplug power, then power back up again. This was rather disruptive inelegant "fix," and it quickly became quite annoying. The cause wasn't apparent, the problem was intermittent and almost random, and there was no obvious way to cause it to fail faster so I could get to root cause quicker.

    This issue lingered for weeks. I really didn't want to climb into my attic to replace the 100' CAT7 cabling I had just proudly installed and tweeted to the world about, as a just-in-case last-ditch I'll-try-anything approach. Instead, I wanted to science this, to get some data that'd help me solve this problem smartly, to stop with all this guessing nonsense.

    Netgear-XS708T-Device-View-by-TinkerTry

    I was asked "what syslog server are your running," imagine my embarrassment

    Netgear L1 support recommended I share my logs from my syslog server with them. I thought to myself, doh! I didn't have a syslog server right now, oh no, how embarrassing! I recalled that I knew of just the syslog server I had always wanted to try and that could be mighty handy right about now. So I told the representative that I'd be using VMware vRealize Log Insight, and I'd get back to him with the syslog export data later that night. So I set forth on making a plan to make it so.

    First, I noted that I'd need to make sure my 10GbE switch was NTP time synced, just like the rest of my vSphere cluster already was. I'd then "point" my XS708T switch to the vRealize IP address, and cranked up the switch log output to 11, also known as the Debug level. This new syslog server would allow the switch logs to be collected and could be used to alert me to whenever the 10GbE connection speed changed again in the future. This logging and alerting might even help me see if we could spot any correlation with any vSphere related events. This was just the occasion I'd been waiting for, to give vRealize Log Insight a try, to see if I could use it as one tool in getting to the bottom of this particular networking issue, and other surprises down the road.

    So it was was time to log in to VMUG Advantage EVALExperience:
    vmugadvantage.onthehub.com

    to see what vRealize Log Insight version they had available for current subscribers like myself. Never mind that I happen to work at VMware now, what I really wanted to know what everybody else who also uses VMUG Advantage EVALExperience can also use in their non-production environment, for free! Read all about EVALExperience here.

    Momentary sadness

    At the time of my initial testing in August 2017, all I found at EVALExperience was the older vRealize Log Insight 4.0. This was disappointing. So I Googled around for a bit, and soon found 4.3 in my my.vmware.com account, so I installed 4.3. Of course, I took a moment to "roll camera," recording everything I did right as I went about testing it in my home lab, just in case this first-timer's experience would be useful to share someday. Of course, I recorded an impromptu voice-over as I went along, seen in the video below.

    All my problems have been solved, here's how it works

    vRealizeOrder
    On Fri Oct 13th, I notified VMUG Advantage that the OVA for 4.5 was missing, and by Mon Oct 16, they fixed it!

    Well, at least logged. Here's the gist of the simple install and configure process. Once you've downloaded and deployed the (SuSE-based) OVA "appliance" file, you point your browser to the assigned IP, then connect it to your vSphere infrastructure by providing it with VCSA 6.5U1 credentials.

    Here's the really slick part. It then logged in to VCSA on my behalf, and quickly went about automatically configuring syslogging from all of my ESXi 6.5U1 servers. How cool is that? Easy as pie, as I demonstrated at this exact spot in the video below. No need to dive into each host with clumsy manual tweaks, which is an important part of bringing up a home lab for those like me using devices like SD or USB for ESXi itself. This syslog auto-configuration also side-steps the need to follow Configuring syslog on ESXi - KB 2003322, and is good news for even the laziest of home lab sysadmins, who really don't mind some simplicity on the home front once in a while.

    Later on, on October 14th actually, I discovered that 4.3 had actually been released back in June, oops! But I also found that the features were nearly identical so the install procedure was still the same, and upgrading to 4.5 later on was no big deal at all. Thus, I decided to add the simple .PAK upgrade footage to this same video, also just published. The timing worked out great, because today EVALExperience upgraded their download to the very latest version 4.5! Note, the same bits are also available for full (purchased) license holders at My VMware here.

    I don't have to choose

    Event-Forwarding
    Click to view the Event Forwarding panel, to forward events to another syslog server too.

    There are many other solutions for syslogging. Does this mean I've made up my mind, as for my solution for my home lab? Nope. Apparently, I can forward all events to another syslog server, such as the popular, feature-rich, and free SexiLog. Nice! Admittedly, I've not yet tried this myself.

    I've been using various syslog products since the 90s. Hey you bashful former Linksys router owners out there. Remember that Kiwi Syslog Server, and that free WallWatcher? Now you can step-up to far great functionality, for your vSphere, and for all your other networked gear that features syslogging.

    What about my network problem?

    Yes, vRealize helped me reliably provide meaningful logs to Netgear by using its native export function. Basically, before I could get escalated to L2 Support, I had to follow their script, and get them the requested syslogs, which I was able to do. For my strange bug, in the end, using the Intel X557 4.4.2 driver VIB and the latest Netgear firmware helped alleviate the problem, but resolving it was hardware and the complete story is yet to be told in a future blog post here at TinkerTry. Subscribe to get notified automatically!

    Ready for vSAN

    For folks running vSAN, great, vRealize Log Insight has you covered. Read the excerpt about the included content pack for vSAN below. In my role as a VMware vSAN System Engineer, the more I know about any products that have anything to do with vSAN, the better. What better way than to leave vRealize Log Insight running in my home lab, ready for anything I throw at?

    Disclaimer!

    This is not official VMware documentation, and nobody asked me to write this post.

    Step-by-Step vRealize Log Insight Install/Configure/Alert/Upgrade

    Here's what the action packed full length video below offers you a look at:

    1. vRealize Log Insight Appliance install using the vSphere Client (HTML5)
    2. Configure connection with vSphere 6.5 (VCSA 6.5U1/ESXi 6.5U1)
    3. Alerts via email, I used a Gmail account
    4. Search syslog, create filters, and configure alerts
    5. Configure 10GbE Netgear XS708T 10GbE switch logging to vRealize for debug
    6. Update from 4.3 to 4.5 via upgrade .PAK file

    Don't forget to set it to auto-start at boot, and to back it up

    Video

    VMware vRealize Log Insight installation/upgrade/alerting configuration for Netgear XS708T syslogs

    vRealize Log Insight Home Page

    2017-10-16_21-10-05
    Here's VMware vRealize Log Insight showing my vSphere General Problems view.
    vRealize-Log-Insight-System-Monitor
    System Monitor view.

    Oct 22 2017 Update

    Steve-Flanders-Boston-VMUG-Oct-19-2017.JPG
    VMware's Steve Flanders presenting at the Boston VMUG UserCon Oct 19 2017.

    Luck would have it that I bumped into Steve Flanders at the Boston VMUG UserCon this past Thursday. That's right, none other than @smflanders, VMware vRealize Log Insight Staff Architect & Senior Manager, note the familiar Twitter logo in his pinned tweet below, featuring all sorts of great references:

    653696685554724866
    921110777750155265

    Oct 24 2017 Update

    I added my Ubiquiti EdgeRouter Lite to the growing list of devices I set to log output to my vRealize Log Insight syslog server, somehow forgot to tell you that.

    2017-10-24_7-47-00
    Ubiquiti EdgeRouter Lite sending "System Log" to vRealize Log Insight syslog server.

    See also at TinkerTry

    vmug-advantage-evalexperience-updated-to-include-latest-vmware-workstation-14-and-fusion-10
    vrealize-automation-enterprise-is-now-on-vmug-advantage-evalexperience
    promise-sanlink3-t1-adapter-gives-thunderbolt-3-usb-pc-10gbe
    latest-nsx-and-all-flash-vsan-added-to-vmug-advantage-evalexperience
    evalexperience-2016
    close-look-at-netgear-xs708t-100nes-including-decibels

    See also

    I had the honor of sitting in on Ryan Johnson presenting and proctoring a Cloud Foundation Hands On Lab in Cincinnati earlier this year, as I was there to do the vSAN Hands On Lab anyway! Here's his very-relevant article:

    • VMware Validated Design for Software-Defined Data Center 4.1 is Now Generally Available
      Aug 22 2017 by Ryan Johnson at VMware Blogs:

      A quick note on management packs and content packs, too. While there have been some minor version updates to these, it’s worth mentioning that vRealize Operations 6.6.1 now includes the management packs for vSAN, vRealize Automation, and vRealize Business for Cloud in the product deployment and they not longer have to be installed post-deployment. Similarly, vRealize Log Insight now includes the content pack for vSAN in the product deployment. Lastly, we’ve added the Content Pack for Linux to the BOM to pull log data from the virtual appliances directly into vRealize Log Insight.

    Marketplace - Content Packs | vRealize Log Insight

    Log-Insight-Content-Pack-Marketplace-Oct-22-2017
    Log Insight Content Pack Marketplace, as it appeared on Oct 22 2017. Note that vRops 6.x, vSAN, and vSphere are correctly showing as already installed.

    On Oct 22 2017, I grabbed the entire Log Insight Content Pack Marketplace text, for future reference:

    Log Insight Content Pack Marketplace
    
    Apache - HTTP Server
    Version:    1.0
    Author: VMware, Inc.
    
    Apache - Tomcat
    Version:    1.0
    Author: VMware, Inc.
    
    Apache - CLF
    Version:    1.2
    Author: VMware, Inc.
    
    Arista - EOS
    Version:    1.0
    Author: Arista Networks, Inc.
    
    BigSwitchNetworks - BCF
    Version:    1.0
    Author: Big Switch Networks
    
    Brocade - SAN & IP Networks
    Version:    3.2
    Author: Brocade
    
    Cisco - ASA
    Version:    1.5
    Author: VMware, Inc.
    
    Cisco - Nexus
    Version:    2.1
    Author: VMware, Inc.
    
    Cisco - UCS
    Version:    1.5
    Author: Cisco Systems, Inc.
    
    DataGravity - Discovery Array
    Version:    1.0
    Author: DG Labs
    
    Dell EMC - VMAX
    Version:    1.0
    Author: Dell EMC
    
    Dell - iDRAC
    Version:    1.1
    Author: VMware, Inc.
    
    Dell Networking
    Version:    1.0
    Author: Dell Inc.
    
    Dell EMC OS10 Networking
    Version:    1.0
    Author: Dell Technologies
    
    EMC - VMAX
    Version:    3.0
    Author: EMC Corporation
    
    EMC - VNX Unified
    Version:    1.0
    Author: VMware, Inc.
    
    EMC-XtremIO
    Version:    1.5
    Author: VMware Inc.
    
    ExtraHop - Wire Data
    Version:    1.0
    Author: ExtraHop Networks
    
    Extreme Networks - Purview Analytics
    Version:    1.0
    Author: Extreme Networks
    
    F5 - BIG-IP
    Version:    1.0
    Author: VMware, Inc.
    
    General
    Version:    2.5
    Author: VMware, Inc.
    Installed
    
    HAProxy
    Version:    1.0
    Author: VMware, Inc.
    
    Hitachi - Server
    Version:    1.0
    Author: Hitachi, Ltd
    
    Hitachi - Storage
    Version:    1.0
    Author: Hitachi, Ltd.
    
    HP - Servers
    Version:    1.0
    Author: Blue Medora
    
    HP - StoreFront Analytics
    Version:    1.0
    Author: Hewlett-Packard
    
    HP - TippingPoint
    Version:    1.0
    Author: Vater Operations GmbH
    
    INFINIDAT - InfiniBox
    Version:    1.0
    Author: INFINIDAT
    
    Infoblox - DDI
    Version:    1.5
    Author: Infoblox
    
    Juniper - Security
    Version:    1.0
    Author: Juniper Networks, Inc
    
    Kaminario-K2
    Version:    1.0
    Author: Kaminario
    
    Lenovo – Networking CNOS
    Version:    2.0
    Author: Lenovo
    
    Lenovo - Networking
    Version:    1.2
    Author: Lenovo
    
    Lenovo – xClarity
    Version:    1.1
    Author: Lenovo
    
    Linux
    Version:    1.0
    Author: VMware, Inc.
    
    Microsoft - Exchange
    Version:    3.2
    Author: VMware, Inc.
    
    Microsoft - .NET CLR
    Version:    3.0
    Author: VMware, Inc.
    
    Microsoft - Active Directory
    Version:    3.2
    Author: VMware, Inc.
    
    Microsoft - IIS
    Version:    3.0
    Author: VMware, Inc.
    
    Microsoft - SharePoint
    Version:    3.0
    Author: VMware, Inc.
    
    Microsoft - SQL Server
    Version:    3.1
    Author: VMware, Inc.
    
    Microsoft - Windows
    Version:    3.5
    Author: VMware, Inc.
    
    MongoDB - Database
    Version:    1.0
    Author: Blue Medora
    
    NetApp - Data ONTAP
    Version:    1.0
    Author: Blue Medora
    
    Nginx
    Version:    1.0
    Author: VMware, Inc.
    
    Nimble Storage
    Version:    1.0
    Author: Blue Medora
    
    Nutanix - Clusters
    Version:    1.1
    Author: Blue Medora
    
    OpenStack
    Version:    1.0
    Author: VMware, Inc.
    
    Oracle - Database
    Version:    1.0
    Author: Blue Medora
    
    Oracle - JRE
    Version:    1.2
    Author: VMware, Inc.
    
    Palo Alto Networks - PAN-OS
    Version:    1.1
    Author: Palo Alto Networks, Inc.
    
    Pivotal - Cloud Foundry
    Version:    1.0
    Author: Blue Medora
    
    Puppet Enterprise
    Version:    1.0
    Author: VMware, Inc.
    
    Pure Storage - FlashArray
    Version:    2.5
    Author: Pure Storage
    
    SolarWinds - NPM
    Version:    1.0
    Author: Blue Medora
    
    Synology - DSM
    Version:    2.0
    Author: VMware, Inc.
    
    Veeam Backup & Replication
    Version:    1.0
    Author: Veeam Software AG
    
    VMware - EVO SDDC Suite
    Version:    1.0
    Author: VMware, Inc.
    
    VMware - Horizon View
    Version:    3.3
    Author: VMware, Inc.
    
    VMware Identity Manager
    Version:    1.0
    Author: VMware Inc.
    
    VMware - NSX-T
    Version:    2.0
    Author: VMware, Inc.
    
    VMware - NSX-vSphere
    Version:    3.7
    Author: VMware Inc.
    
    VMware - Orchestrator - 7.0.1+
    Version:    2.0
    Author: VMware, Inc.
    
    VMware - Orchestrator
    Version:    1.1
    Author: VMware, Inc.
    
    VMware - SRM
    Version:    1.5
    Author: VMware Inc.
    
    VMware - vCAC 6.0
    Version:    1.0
    Author: VMware, Inc.
    
    VMware - vCloud Director
    Version:    8.8
    Author: VMware
    
    VMware - vCNS
    Version:    1.0
    Author: VMware, Inc.
    
    VMware - vC Ops 5.x
    Version:    1.0
    Author: VMware, Inc.
    
    VMware - vRA 7
    Version:    1.5
    Author: VMware, Inc.
    
    VMware - vRA 6.1+
    Version:    1.1
    Author: VMware, Inc.
    
    VMware - vRops 6.x
    Version:    1.7
    Author: VMware, Inc.
    Installed
    
    VMware - VSAN
    Version:    2.0
    Author: VMware Inc.
    Installed
    
    VMware - vSphere
    Version:    1.0
    Author: VMware, Inc.
    Installed