Fabric Zoning for the IBM Spectrum Virtualize and FlashSystem NPIV Feature

Zoning Basics

Before I talk about some zoning best-practices, I should explain two different types of zoning and how they work.  There are two types of zoning:  WWPN Zoning and Switch-Port Zoning

World-wide Port Name (WWPN) Zoning

WWPN zoning is also called "soft" zoning and is based off the WWPN that is assigned to a specific port on a fibre-channel adapter.  The WWPN serves a similar function as a MAC address does on an ethernet adapter.  WWPN-based zoning uses the WWPN of devices logged into the fabric to determine which device can connect to which other devices.  Most fabrics are zoned using WWPN zoning.  It is more flexible than switch-port zoning - a device can be plugged in anywhere on the SAN (with some caveats beyond the scope of this blog post) and the device can connect to the other devices it is zoned to.    It has one distinct advantage over Switch-Port based zoning, which is that zoning can always be specified on a single WWPN level.  

Switch-Port Zoning

 Switch-Port Zoning is also called "hard" zoning.  Zoning is done based on switch ports.  When switch port zoning is used all WWPNs logged into the fabric through the switch ports in the same zone are allowed to communicate.  

Problems with Switch Port Zoning and the Spectrum Virtualize/FlashSystem NPIV Feature

If you are unfamiliar with the NPIV feature in Spectrum Virtualize or the  FlashSystems storage products you can read about it here.

Switch Port zoning works without issue until you have multiple devices logging into the same switch port such as an NPV device (VMWare, VIO, access gateway, etc) or you enable the NPIV feature on Spectrum Virtualize.   To illustrate, consider we have this simple diagram:


In the above diagram, we have an NPV device on the left with 3 virtual WWPNs (vWWPN) logged into the fabric on switch Port 1.  The NPV device could be a hypervisor, or it could be an access gateway.   We have an SVC Node with NPIV enabled on the right.  This blog post uses SVC as an example but applies to all of the Storwize and FlashSystem products that support NPIV.  The SVC node port has physical WWPN and a vWWPN logged into the fabric on switch Port 2.  If we were to use Switch-Port zoning, the solid line in the next diagram is the zone as created by the SAN administrator.  The dashed line is the effective zone.  All the devices in the effective zone can communicate with each other.  For our example, the host with  vWWPN2 is not defined as a host on the SVC cluster and has no vdisks assigned to it.


This zoning can work but has the potential to cause problems.  The first problem is  vWWPN2 should not be zoned to the SVC,  but there is no way to prevent it from connecting to both the SVC physical and virtual WWPNs,  since all host WWPNs logged into the fabric on Port 1 can communicate with all node WWPNs logged into the fabric on port 2.

Another problem as long as the I/O Group the SVC Node is in is configured for transitional mode, then the host WWPNs can connect to both the physical and vWWPNs on the node, thus increasing the path count of devices attached to the cluster.

 For most of the Storwize and FlashSystem products, the maximum is 512 connections per port per node.  This includes nodes, controllers, and hosts but you should check the configuration limits documentation for your specific product.  Using Switch-Port based zoning you are at increased risk of hitting the maximum number of fibre-channel connections on that port.   If you hit the limit, then the cluster will not allow new host, controller or cluster node connections on that port.  This connection count problem is made worse by vWWPN2 connecting to the SVC node port even though it is not defined as a host.  You can see how having a large number of vWWPNs logged in to switch Port 1 causes the connection count to the SVC node port to rise rapidly.   Running out of available connections to the  cluster is the most common problem we see with using switch-port zoning and NPIV. 

The third potential problem is you have far less control over balancing fibre-channel connections across node ports.   In the above diagram, switch port 1 has 3 NPV devices connected, so there are 6 logins to the SVC node port.   Another switch port might have a higher number.  I have seen as many as 48 devices connected via NPV to a single switch port.   For that situation, I would have 96 connections to each  cluster node port with switch-port zoning and NPIV in transitional mode on the cluster.  That is almost 1/4 of the available connections to each node port that switch port is zoned to.   With switch-port zoning I do not have the ability to balance the host WWPNs across the cluster ports.

Lastly,  vWWPN1 vWWPN2 and vWWPN3 can all communicate with each other.  This violates the general best practice recommendation of isolating hosts to their own zone where possible.   You could implement smart zoning, but that complicates your environment and is an additional step to administer and troubleshoot. 

WWPN-Based Zoning

Taking our first diagram and implementing WWPN based zoning gives us the below diagram for both the implemented and effective zoning.



You can see that  vWWPN1 and vWWPN3 are each zoned separately to only the vWWWPN on the SVC node.  They are not zoned to the physical WWPN, and vWWPN2 is not zoned at all.  This has a few advantages over switch-port zoning.  First, it reduces our connection count to the node port by four.  We would have 6 connections if we used Switch-Port based zoning - two connections for each host vWWPN.  With WWPN zoning we have two host vWPPNs connecting only to the Node vWWPN.  Second, you can balance the host connections across the SVC node ports.  Third, if the I/O group is in transitional mode, zoning the hosts exclusively to the node vWWPN minimizes disruption when the NPIV mode is changed to enabled.  

In Summary

While it is supported, it is recommended that you do not use the NPIV feature with Switch-Port zoning.  If you choose to do so, be aware of the potential problems that can impact the health of your SAN and potentially prevent new hosts from connecting to your Spectrum Virtualize, SVC or FlashSystems cluster.  




Comments

  1. fincherjcApril 21, 2020 at 3:07 PM

    Also keep in mind, the first thing we will do when you open a support case is tell you to not use port base zoning.

    ReplyDelete

Post a Comment

Popular posts from this blog

Image
Troubleshooting CRC Errors On Fibre-channel Fabrics There is no "Easy Button" for troubleshooting CRC errors. It is an iterative process. You make a change, you monitor your fabric, and if necessary you make more changes until the issues are resolved. I frequently have customers who want it to be a one step process. It can be, but usually takes multiple steps. Having said that, before we can fix them, we need to know what CRC errors are and why they occur. What Are CRC Errors? When Do they Occur? The simple answer is that CRC errors are damaged frames. The more complicated answer is that before a fibre-channel frame is sent, some math is done. The answer is added to the frame footer. When the receiver gets the frame, the receiver repeats the math. If the receiver gets a different answer then what's recorded in the frame, then the frame was changed in flight. This is a CRC error. The only time these happen is if the physical plant - cabling, SFPs is somehow
Read more

Troubleshooting Slow Drain Devices on Broadcom Switches

Image
  Slow drain devices are one of the more common problems on storage networks.  They can occur for a variety of reasons.  For a refresher on how they can affect your storage network you should watch  this video .    In this blog post I will go through the basic steps to troubleshoot a slow drain device on a Broadcom fabric.    I will be using command line output from switches.  The CLI format lends itself better to a blog post more readily than screen shots from a GUI, and the commands are consistent across different versions of FOS.    SANnav is a huge change from Brocade or IBM Network Advisor and the screens would look quite different between the two. The first command we will be using is porterrshow.    The above output has been truncated for the ports we are interested in. The counters of interest are in the c3timeout column.  You can see that there are 2 sub-columns, 'tx and 'rx'.   'tx' means the switch is trying to send frames to the device attached to that p
Read more

Spectrum Virtualize NPIV and Host Connectivity

Image
 A while ago I wrote  this post  as an introduction to the Spectrum Virtualize NPIV feature.  In this follow-up post I thought I would focus more on host connectivity and the effects of NPIV.    You can watch a quick review of the NPIV feature in this IBM Systems Rockstar video:      NPIV has 3 modes: 1.  Disabled - this mode means that hosts cannot connect to the virtual World Wide Port Names  (WWPNs) on the Spectrum Virtualize cluster, regardless of the fabric zoning 2.  Transitional - this mode means hosts can connect to either the physical or virtual WWPNs on the cluster.  If a host is zoned to both, it will connect to both.   Transitional mode is meant to only be used while you are migrating to NPIV mode and rezoning your hosts to the virtual WWPNs.  It is not meant to be used permanently or even long-term.   3.  Enabled - this means hosts can only connect to the virtual WWPNs.  If they are zoned to the physical WWPNs the connection will be listed as 'blocked' in the devic
Read more