In the first part, we looked at proactively finding hosts with excessive number of events. This first part will demonstrate how to dig deeper into the events that we find.
We’ll start where we left off, which is in the filtered Interactive Analysis view:
Let’s say that the event type at the bottom is something we’re interested in digging a little deeper into. What LI can help us do is to figure out a lot more info about this event, some of which would have been impossible to do using traditional log reading:
- What times of the day does this event occur?
- Does this event occur on other hosts than the one we’re looking at now, and if so, at what times?
- Does this event occur for other disk devices than the one we’re looking at now (in the case of disk events)?
Number 1 we already have in the graph view above, as long as we have it set to Time Series.
Number 2 we can see by clicking the left hand side of event type and creating a filter for the event in question
Then clearing the filter for hostname and setting the graph to group by hostname (sometimes already by default)
Now we can see that all hosts have this particular event, but the top host (..esx-05..) is the one that has the most occurrences of it.
Let’s click on one of the blue boxes on the top of the graph, to filter out only the first host. This also means that we are setting a time span, so we also need to set the time range back to Latest 24 hours.
Now, let’s also group the graph by disk device, which will cover question 3 in the list of questions that we want answered:
Above we can see that there is one particular device that is generating most of the events, so we can drill down even deeper on that device, group by status codes etc, in the same manner that we just did.
To conclude: By using these filters, views, groups and graphs and twisting and turning the log data, we can cover many different troubleshooting scenarios. I’ve used it for figuring out many things that would have been difficult or impossible using traditional log reading methods, and I hope you will too!
As a teaser for coming LI blog posts, we can also look at the distribution of device IDs by simply clicking on the device ID in the Event Type tab view and selecting View Graph:
This and many more tricks will be covered in future LI blog posts!