How Storage Insights Can Enhance Your Support Experience


An Introduction

The first week in May IBM announced IBM Storage Insights.    As of 11 June, Storage Insights has these key items:
  • IBM Blue Diamond support
  • Worldwide support for opening tickets
  • Custom dashboards
  • New dashboard table view
  • Clients can now specify whether IBM Storage Support can collect support logs.  This is done on a per-device basis.
You can get a complete list of the new features here:  Storage Insights New Features
There are some other new features such as new capacity views on the Storage Insights Dashboard.    With these new features, especially support for IBM Blue Diamond customers, Storage Insights is an increasingly important and valuable troubleshooting tool.  My team here is seeing more and more customers that are using Storage Insights.   I thought I would discuss the potential benefits of Storage Insights as a troubleshooting tool.  

Some Background

The problems my team fixes can be categorized as either:
  1. Root-cause analysis (RCA) - meaning a problem happened at some point in the past, and the customer wants to know why it happened
  2. Ongoing issue - the problem is happening now (or happens repeatedly, also called intermittent)
The above two types of problems can further be broken down into partially working or completely broken.  Of the two, partially working can be much more difficult to troubleshoot, especially if it's an intermittent issue and not constant.  As an example, some years ago my van had a misfire on one of its cylinders, but we didn't know which one.  Of course it never occurred when my mechanic was driving it.  It finally took several hours at the dealer with the dealer hooking the car up to a test rig to record the failure and identify the misfire.  Had the problem been a completely broken spark plug wire instead of partial, it would have been much easier to identify.
You can imagine the difficulty of attempting to root-cause a problem that happened hours or days ago  on a large and busy SAN if the problem is/was not severe enough to cause the switches to record any time-stamped errors or other indicators of problems.   As an example  I'm confident you've been in slow-moving traffic where the cause of the problem isn't readily apparent.   The analogy isn't perfect, but suppose the traffic cameras in your city were configured to only start recording when traffic is moving less than 30 mph for 2 minutes and/or generate an alert back to the traffic center.   They do record the number of cars passing by and the number of cars exiting and entering the freeway at each ramp but they don't timestamp these numbers.  They only timestamp the video and/or alerts.  Now further suppose you were stuck in traffic last week that was moving at 32 mph.  Since it didn't meet meet the threshold, the cameras never recorded anything  and no alerts were sent.  You could collect the statistics on the number of cars counted by the cameras but without anything recorded from last week  it would be extremely difficult to provide an explanation as to why traffic was slow, since you can't reconcile the count of cars to any specific point in time.  If traffic had been completely stopped, the cameras would have started recording and then you'd be able to see the car fire, or accident, or whatever the cause of the problem was last week.  The same limitations exist for ongoing issues.  If traffic is moving slowly but not completely stopped, then identifying the cause of the slow traffic can be difficult.  

How Storage Insights Can Help Provide Root-Cause

Storage Insights has the potential to provide an explanation for these partially working root-cause investigations by regularly sampling the performance statistics and providing timestamps on this data.  If we had something like Storage Insights regularly sampling statistics from our traffic cameras, we could go back and analyze these for the time period where you were sitting in traffic.  We might find a certain exit ramp from the freeway was congested at the time of the problem.    We could take this information and correlate that with other data to try and determine why the ramp was backed up.  We might find a concert was going on at a venue near the ramp, or some other event that caused an increase in traffic to that ramp.  

How Storage Insights Can Decrease Problem Resolution Time

For a problem that is happening now, Storage Insights can help provide resolution more quickly than without it.  Going back to our traffic example, suppose there is an accident or some other problem on a surface street that an exit ramp connects to.  Traffic eventually will back up onto the exit ramp and then onto the freeway.  Without Storage Insights, you'd have to look at each of your traffic cameras in turn and trying to figure out where the congestion starts.   With Storage Insights, since it's collecting the statistics you can filter them to find out which of your exit ramps is the congested ramp.


Comments

Post a Comment

Popular posts from this blog

Image
Troubleshooting CRC Errors On Fibre-channel Fabrics There is no "Easy Button" for troubleshooting CRC errors. It is an iterative process. You make a change, you monitor your fabric, and if necessary you make more changes until the issues are resolved. I frequently have customers who want it to be a one step process. It can be, but usually takes multiple steps. Having said that, before we can fix them, we need to know what CRC errors are and why they occur. What Are CRC Errors? When Do they Occur? The simple answer is that CRC errors are damaged frames. The more complicated answer is that before a fibre-channel frame is sent, some math is done. The answer is added to the frame footer. When the receiver gets the frame, the receiver repeats the math. If the receiver gets a different answer then what's recorded in the frame, then the frame was changed in flight. This is a CRC error. The only time these happen is if the physical plant - cabling, SFPs is somehow
Read more

Troubleshooting Slow Drain Devices on Broadcom Switches

Image
  Slow drain devices are one of the more common problems on storage networks.  They can occur for a variety of reasons.  For a refresher on how they can affect your storage network you should watch  this video .    In this blog post I will go through the basic steps to troubleshoot a slow drain device on a Broadcom fabric.    I will be using command line output from switches.  The CLI format lends itself better to a blog post more readily than screen shots from a GUI, and the commands are consistent across different versions of FOS.    SANnav is a huge change from Brocade or IBM Network Advisor and the screens would look quite different between the two. The first command we will be using is porterrshow.    The above output has been truncated for the ports we are interested in. The counters of interest are in the c3timeout column.  You can see that there are 2 sub-columns, 'tx and 'rx'.   'tx' means the switch is trying to send frames to the device attached to that p
Read more

Spectrum Virtualize NPIV and Host Connectivity

Image
 A while ago I wrote  this post  as an introduction to the Spectrum Virtualize NPIV feature.  In this follow-up post I thought I would focus more on host connectivity and the effects of NPIV.    You can watch a quick review of the NPIV feature in this IBM Systems Rockstar video:      NPIV has 3 modes: 1.  Disabled - this mode means that hosts cannot connect to the virtual World Wide Port Names  (WWPNs) on the Spectrum Virtualize cluster, regardless of the fabric zoning 2.  Transitional - this mode means hosts can connect to either the physical or virtual WWPNs on the cluster.  If a host is zoned to both, it will connect to both.   Transitional mode is meant to only be used while you are migrating to NPIV mode and rezoning your hosts to the virtual WWPNs.  It is not meant to be used permanently or even long-term.   3.  Enabled - this means hosts can only connect to the virtual WWPNs.  If they are zoned to the physical WWPNs the connection will be listed as 'blocked' in the devic
Read more