CCIE Data Center Storage Networking Fibre Channel Switching Configuration Nexus 5500UP FC Initialization Allocate interfaces as type FC slot 1 port 25 32 type fc FC must start at top of module and count down Requires config save & reload Enable feature fcoe FC (and FCoE) features are not activated until FC_FEATURES_PKG is checked out
Fibre Channel Interfaces Like Ethernet, FC Auto Negotiation is on by default Port Type Trunk Mode Speed In cases where negotiation fails, static config is switchport mode [F FL E SD NP ] switchport trunk mode [on off auto] switchport speed [1000 2000 4000 8000] Verified as show interface [brief] Fibre Channel Expansion Ports E_port - Expansion Port Inter Switch Link (ISL) On certain MDS line cards, E cannot be autonegotiated E_port requires dedicated rate-mode switchport rate-mode dedicated Similar to how port groupings work on Nexus 7K All ports on Nexus 5500 are dedicated rate mode
FC Trunking Expansion Ports TE_port Trunking Expansion Extended ISL, analogous to an 802.1Q Trunk Trunk negotiation is on for E ports trunk protocol enable show trunk protocol Manually configured as combination of switchport mode [e auto] switchport trunk mode [on auto] There is no switchport mode te command Like Ethernet trunks, all VSANs are allowed switchport trunk allowed vsan Verified as show interface fc1/1 [trunk vsan] SAN Port Channels Used to aggregate the bandwidth of physical links Ethernet PCs and SAN PCs use the same number space Created with link level channel-group 1 New members added with link level channel-group 1 force Port Channeling Protocol (PCP) enabled with PC link level channel mode active interface Port-Channel in MDS interface SAN-Port-Channel in Nexus Verified as show [san-]port-channel summary
Virtual SANs (VSANs) Use to segment Fabric Services into logical SAN Islands FLOGI, FC Domain, FC Name Server, Zoning, Aliases, etc. Configured as vsan database vsan 10 vsan 10 interface fc1/1 VSANs and EISLs All VSANs allowed on trunk ports by default Remotely un-configured VSANs automatically pruned off of trunks Similar to Ethernet Bridge Assurance To avoid merging fabrics, VSANs should be pruned manually switchport trunk allowed vsan FC Fabric Services Fibre Channel networks are connection oriented Nodes must first register with the control plane of the fabric before sending any traffic FC Fabric Services include FC Domain Principle Switch Election & assignment of Domain IDs FSPF Fabric Routing between Domain IDs FLOGI End station registers with the fabric and are assigned FCID FC Name Services / Directory Services Distributes FCID to pwwn/wwpn mappings through Fabric Zoning Controls which Initiators can talk to which Targets
FC Domain Principle Switch assigns Domain IDs to Fabric PS Election Based on lowest priority and SWWN Modified as fcdomain priority 1 vsan 1 Domain ID selection Dynamic from PS by default Static as fcdomain domain 0x51 [preferred static] Preferred requests Domain ID but will join without the assignment Static requests Domain ID and isolates itself without the assignment FC Domain FC Domain process restart Graceful as fcdomain restart vsan 1 Forced as fcdomain restart disruptive vsan 1 Not recommended Verified as show fcdomain [vsan] show fcdomain domain-list [vsan]
Fabric Shortest Path First Used to build an SPT through the fabric Similar logic to OSPF & IS-IS Domain-ID is the Node ID in the SPT Occurs automatically on a per-vsan basis Can be modified as Global fspf config vsan 1 Interface fspf [cost dead-interval hello-interval passive retransmit-interval] Verified as show fspf [database interface] [vsan] Fabric Login (FLOGI) All Initiators and Targets must FLOGI before sending any data into the Fabric Verified as show flogi database No configuration needed No FLOGI indicates a basic link-level negotiation problem
FC Name Services FCNS (AKA Directory Services) keeps a mapping of FCIDs to pwwns/wwpns Analogous to the IP ARP Cache End devices register with the FCNS after FLOGI show flogi database only shows local Initiators & Targets show fcns database shows everyone in the Fabric If Node did FLOGI but is not in everyone s FCNS, indicates the Fabric is broken E.g. VSAN is isolated, EISL allowed list is wrong, etc. Zoning Controls which Initiators can talk to which Targets Analogous to an ACL in the LAN/WAN world Zoning is required, not optional Default zone policy is to deny Can be changed to permit as zone default-zone permit vsan 1 system default zone
Soft vs. Hard Zoning Soft Zoning Initiator registers with FCNS to get Zoning Zoning enforced in the control plane but not data plane Initiator could manually mount the wrong Target Hard Zoning Initiator registers with FCNS to get Zoning Zoning enforced in the control plane and the data plane Initiator cannot manually mount the wrong Target NX-OS / SAN-OS runs Hard Zoning by default Zone vs. Zoneset Zone is used to create a mapping between pwwns/wwpns, FCIDs, Aliases, Interface, Domain-ID, etc. Zones are grouped together in a Zoneset I.e. Zoneset is the ACL, Zone is the ACE Zoneset is applied to the VSAN and then activated Makes the Full Zoneset become the Active Zoneset Zoneset must be re-activated after each change
Full vs. Active Zoneset Only one Zoneset per VSAN can be Active in the Fabric at a time Same logic as one ACL per interface per direction Full Zoneset is the one in the configuration Active Zoneset is the one being enforced in the Fabric By default only the Active Zoneset is advertised, not the Full Zoneset Can result in misconfigured or Isolated Fabric Zoning Configuration & Verification show zone status vsan 1 Display zone mode and default action (permit or deny) show zone Display full zone info show zone active Display the currently active zones show zoneset Display full zoneset info show zoneset active Display the currently active zoneset
Zoning Configuration & Verification clear zone database vsan 1 Deletes the local full zone but not the active one zoneset distribute full vsan 1 In global config enables full distribution when new E ports come up zoneset distribute vsan 1 In exec mode force the distribution of the full zoneset FC Aliases Zoning based on pwwn/wwpn is error-prone Zoning errors can be catastrophic to the Fabric FC Aliases give user-friendly names to WWNs, FCIDs, etc. Analogous to DNS in IP Configured as fcalias name Can be advertised through Zoneset distribution zoneset distribute vsan 1
Basic vs. Enhanced Zoning By default the Full Zoneset is local and the Active Zoneset is Fabric-wide Order of operations errors can corrupt the Active Zoneset Analogous to VTP deleting all your Ethernet VLANs Enhanced Zoning prevents this by locking the Fabric Ensures that people don t accidentally overwrite each other Using Enhanced Zoning Admin logs into any switch in the Fabric and starts to configure Zoning Lock is advertised to all switches in the Fabric Other admins cannot edit Zoneset until lock is released Lock is released by committing the Zoneset Configured as zone mode enhanced vsan system default zone mode enhanced
FC Device Aliases FC Aliases are locally significant Can be distributed through manual Zoneset distribution Still prone to becoming unsynchronized through the Fabric Device Aliases solve this problem Device Aliases serve the same purpose as FC Aliases Bind a pwwn/wwpn to a user-friendly name Difference is that the binding is advertised to the Fabric Using Device Aliases Device Aliases are advertised like Enhanced Zoning Device Alias session is created and lock is advertised to the Fabric Changes are made and committed Aliases are advertised through CFS and lock is removed Configured as device-alias database
Q&A