Configuring Nexus vPC – Part 2 – Parameters

This is a continuation of the previous blog “Configuring Nexus vPC – Part 1“. In this blog I’m taking a look at some of the best practice vPC parameters.

Assuming basic vPC peer-link is established, let’s take a look at the default configuration:

show run vpc all
vpc domain 1
 peer-switch
 role priority 2
 system-priority 1000
 peer-keepalive destination 10.10.99.1 source 10.10.99.2 udp-port 3200 vrf PEER
-KEEPALIVE interval 1000 timeout 5 precedence 6 hold-timeout 3
 delay restore 30
 no dual-active exclude interface-vlan
 graceful consistency-check
 auto-recovery reload-delay 240
 delay restore interface-vlan 10
 no fabricpath switch-id
 no fabricpath multicast load-balance
 port-channel limit

We didn’t configure these bolded commands, so what are they and what else should I configure? Let’s take a look at what’s already configured:

delay restore 30

This is enabled by default, but it’s good to know. This command tells the failed vPC peer to delay bringing up it’s member ports for the configured time (30 seconds by default, 3600 max), which allows for graceful convergence of layer 3 routing protocols.

Say you have a router connected north to the Nexus 7K.  If you bring up the vPC member links connected south of the Nexus 7K before routing has had time to converge, you may run into a scenario where traffic will traverse the peer-link and try to leave a member port on the peer.  When that happens, the vPC loop prevention mechanism will Gandalf the packet, Thou shall not pass!

A similar command (also enabled by default, set to 10 seconds) – delay restore interface-vlan <1-3600 sec > – can be used to delay the SVI bring-up timing upon vPC peer device failure and recovery.

no dual-active exclude interface-vlan

The only time you want to enable this is if you need the SVIs on both Nexus 7Ks to remain up even if there is a vPC peer-link failure. Some use cases for this are orphan ports connected to only 1 Nexus 7K, or for backhaul routing purposes. Otherwise, leave it off.

graceful consistency-check

With vPC graceful type-1 check capability, only member ports on the secondary vPC peer device are brought down. vPC member ports on the primary vPC peer device remain up and process all traffic coming to/from the access device. This is verbatim from Cisco documentation.

auto-recovery reload-delay 240

This is important to understand, play close attention.  Auto-recovery provides a backup mechanism in case of a vPC peer-link failure followed by a vPC primary peer device failure (bad scenario). When the vPC peer-link goes down, the secondary peer shuts down all of its vPC member ports. If this is a situation where the primary device actually goes down, then after 3 consecutive keepalive timeouts, the secondary vPC peer will assume the role of primary and re-enable it’s vPC member ports.

vPC auto-recovery reload-delay is used in situations where both vPC peers reload and only one comes online (worse scenario). If only one vPC peer comes online, all member ports will remain suspended until the other peer comes online. Auto-recover reload-delay allows the online vPC peer to assume the primary role and bring up all local vPC ports after the reload-delay timer expires.

no fabricpath switch-id
no fabricpath multicast load-balance

I will touch on the fabricpath knobs in a future post.

Now that we know what is already configured, what else should we configure, and why?

peer-switch

The vPC Peer-switch feature allows a pair of vPC peer devices to appear as a single STP root in the Layer 2 topology (they have the same bridge ID). vPC peer-switch must be configured on both vPC peer devices to become operational, and both devices must have an identical STP configuration. This simplifies Spanning-tree configuration, and shortens vPC convergence time by almost 3 seconds. There are no reasons I can think of why you wouldn’t configure this.

Something else not listed under the vpc domain, but instead on the vPC peer-link, is Bridge Assurance. This is enabled when you configure the vpc peer-link, placing the port-channel in spanning-tree port type “network”. Bridge Assurance is a bi-directional layer 2 loop avoidance mechanism in a Spanning-tree environment, by which BPDUs are periodically sent to all operational ports (including backup and alternate ports). If a port stops receiving BPDUs, it is placed in a blocking state until BPDUs are received again. This is great in a pure Spanning-tree environment, but can cause issues in a vPC environment when enabled on vPC member ports with peer-switch enabled. This is due to a possible timing issue during a vPC peer failure and recovery. You will certainly see the error “BA_Inconsistent” if you run into this.  I recommend using spanning port type “network” on links other than the vPC peer-links with responsible caution.

peer-gateway

vPC peer-gateway is a gateway MAC localization mechanism, resulting in both vPC peers acting as the active gateway for packets addressed to the other peer device gateway MAC. It keeps the forwarding of traffic local to the vPC peer device and avoids use of the peer-link (by not bridging the traffic to the other vPC peer device).

I’m going to show a demonstration below, and show you some hsrp on NX-OS along the way. 🙂

Let’s configure Interface VLANs on the Earth Nexus 7Ks:

EARTH-CORE-7K-1:

feature interface-vlan
feature hsrp
interface vlan 10
 desc VLAN10-SERVERS
 ip add 10.10.10.2 255.255.255.0
 hsrp version 2
 hsrp 10
 preempt delay minimum 180
 prioriy 150
 timers 1 3
 ip 10.10.10.1
 no shut

EARTH-CORE-7K-1# show hsrp 
Vlan10 - Group 10 (HSRP-V2) (IPv4)
 Local state is Active, priority 150 (Cfged 150), may preempt
 Forwarding threshold(for vPC), lower: 1 upper: 150 
 Preemption Delay (Seconds) Minimum:180 
 Hellotime 1 sec, holdtime 3 sec
 Next hello sent in 0.312000 sec(s)
 Virtual IP address is 10.10.10.1 (Cfged)
 Active router is local
 Standby router is 10.10.10.3 , priority 130 expires in 0.580000 sec(s)
 Authentication text "cisco"
 Virtual mac address is 0000.0c9f.f00a (Default MAC)
 2 state changes, last state change 00:01:57
 IP redundancy name is hsrp-Vlan10-10 (default)

EARTH-CORE-7K-2

feature interface-vlan
feature hsrp
interface Vlan10
 description VLAN10-SERVERS
 no shutdown
 no ip redirects
 ip address 10.10.10.3/24
 hsrp version 2
 hsrp 10 
 preempt delay minimum 180 
 priority 130
 timers 1 3
 ip 10.10.10.1

EARTH-CORE-7K-2(config-if-hsrp)# sh hsrp
Vlan10 - Group 10 (HSRP-V2) (IPv4)
 Local state is Standby, priority 130 (Cfged 130), may preempt
 Forwarding threshold(for vPC), lower: 1 upper: 130 
 Preemption Delay (Seconds) Minimum:180 
 Hellotime 1 sec, holdtime 3 sec
 Next hello sent in 0.365000 sec(s)
 Virtual IP address is 10.10.10.1 (Cfged)
 Active router is 10.10.10.2, priority 150 expires in 1.932000 sec(s)
 Standby router is local 
 Authentication text "cisco"
 Virtual mac address is 0000.0c9f.f00a (Default MAC)
 1 state changes, last state change 00:02:16
 IP redundancy name is hsrp-Vlan10-10 (default)

Verify the MAC address of SVI 10 on 7K-1 and 7K-2

EARTH-CORE-7K-1# sh int vlan 10 | i SVI
 Hardware is EtherSVI, address is e8ed.f339.4f43

EARTH-CORE-7K-2# sh int vlan 10 | i SVI
 Hardware is EtherSVI, address is e8ed.f339.4e43

Now take a look at the CAM table. Notice that each device has the G flag set for VLAN 10 for it’s own MAC address, but not for it’s peer’s MAC address.

EARTH-CORE-7K-1# sh mac add
Legend: 
 * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
 age - seconds since last seen,+ - primary entry using vPC Peer-Link,
 (T) - True, (F) - False
 VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
G 10 0000.0c9f.f00a static - F F sup-eth1(R)
* 10 e8ed.f339.4e43 static - F F vPC Peer-Link
G - e8ed.f339.4f43 static - F F sup-eth1(R)
G 10 e8ed.f339.4f43 static - F F sup-eth1(R)
G 900 e8ed.f339.4f43 static - F F sup-eth1(R)
G 999 e8ed.f339.4f43 static - F F sup-eth1(R)
* 900 001c.0e1a.f141 dynamic 0 F F Eth4/47
* 999 e8ed.f339.4e43 dynamic 0 F F Eth4/48

EARTH-CORE-7K-2# show mac address-table 
Legend: 
 * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
 age - seconds since last seen,+ - primary entry using vPC Peer-Link,
 (T) - True, (F) - False
 VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
G 10 0000.0c9f.f00a static - F F vPC Peer-Link(R)
G - e8ed.f339.4e43 static - F F sup-eth1(R)
G 10 e8ed.f339.4e43 static - F F sup-eth1(R)
G 901 e8ed.f339.4e43 static - F F sup-eth1(R)
G 999 e8ed.f339.4e43 static - F F sup-eth1(R)
* 10 e8ed.f339.4f43 static - F F vPC Peer-Link
* 901 001c.0e1a.f172 dynamic 0 F F Eth4/47
* 999 e8ed.f339.4f43 dynamic 0 F F Eth4/48

Wouldn’t it be nice if each device would act as the gateway for each other? This happens already via VMACs, but not for devices that do not ARP for a gateway, such as NAS’s or load-balancers. Let’s configure peer-gateway and revisit the CAM table.

NOTE: You will get a warning message when enabling peer-gateway about traffic loss, but I can assure you that vPC did not go down and I received no traffic loss when enabling this feature.  See below…

EARTH-CORE-7K-1(config)# vpc domain 1 
EARTH-CORE-7K-1(config-vpc-domain)# peer-gateway 
This peer-gateway config may cause traffic loss. Do you want to continue (y/n)? [n] y
EARTH-CORE-7K-1(config-vpc-domain)# 2014 Mar 1 17:57:06 EARTH-CORE-7K-1 %ICMPV6-3-BAD_INFO_RECEIVED: icmpv6 [21344] Invalid major message type 1 received in icmpv6_main()

vPC is still up

EARTH-CORE-7K-1(config-vpc-domain)# show vpc
Legend:
 (*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1 
Peer status : peer adjacency formed ok 
vPC keep-alive status : peer is alive 
Configuration consistency status : success 
Per-vlan consistency status : success 
Type-2 consistency status : success 
vPC role : primary 
Number of vPCs configured : 0 
Peer Gateway : Enabled
Peer gateway excluded VLANs : -
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled (timeout = 240 seconds)
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans 
-- ---- ------ --------------------------------------------------
1 Po1 up 10-11

Now look at the CAM table, this 7K is already acting as gateway for it’s peer’s MAC address.

EARTH-CORE-7K-1# show mac address-table 
Legend: 
 * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
 age - seconds since last seen,+ - primary entry using vPC Peer-Link,
 (T) - True, (F) - False
 VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
G 10 0000.0c9f.f00a static - F F sup-eth1(R)
G 10 e8ed.f339.4e43 static - F F vPC Peer-Link(R)
G - e8ed.f339.4f43 static - F F sup-eth1(R)
G 10 e8ed.f339.4f43 static - F F sup-eth1(R)
G 900 e8ed.f339.4f43 static - F F sup-eth1(R)
G 999 e8ed.f339.4f43 static - F F sup-eth1(R)
* 900 001c.0e1a.f141 dynamic 0 F F Eth4/47
* 999 e8ed.f339.4e43 dynamic 0 F F Eth4/48

Configure the other side, and verify it now has the G flag set for the peer’s MAC.

EARTH-CORE-7K-2(config)# vpc domain 1 
EARTH-CORE-7K-2(config-vpc-domain)# peer-gateway 
This peer-gateway config may cause traffic loss. Do you want to continue (y/n)? [n] y
EARTH-CORE-7K-2(config-vpc-domain)# 2014 Mar 1 17:56:39 EARTH-CORE-7K-2 %ICMPV6-3-BAD_INFO_RECEIVED: icmpv6 [6144] Invalid major message type 1 received in icmpv6_main()

EARTH-CORE-7K-2# show mac address-table 
Legend: 
 * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
 age - seconds since last seen,+ - primary entry using vPC Peer-Link,
 (T) - True, (F) - False
 VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
G 10 0000.0c9f.f00a static - F F vPC Peer-Link(R)
G - e8ed.f339.4e43 static - F F sup-eth1(R)
G 10 e8ed.f339.4e43 static - F F sup-eth1(R)
G 901 e8ed.f339.4e43 static - F F sup-eth1(R)
G 999 e8ed.f339.4e43 static - F F sup-eth1(R)
G 10 e8ed.f339.4f43 static - F F vPC Peer-Link(R)
* 901 001c.0e1a.f172 dynamic 0 F F Eth4/47
* 999 e8ed.f339.4f43 dynamic 0 F F Eth4/48

Ok, phew! Almost done. Now, what does ip arp synchronize do? vPC ARP Sync improves convergence time for Layer 3 flows (North to South traffic). When vPC peer-link fails and then recovers, vPC ARP Sync performs an ARP bulk synchronization over Cisco Fabric Services (CFS) from the vPC primary peer device to the vPC secondary peer device. Enable with:

ip arp synchronize

Wouldn’t it be great if Cisco just enabled all this by default? Well, we’re one step closer to that. You can enable all of these features with one handy command:

EARTH-CORE-7K-2(config-vpc-domain)# mode ?
 auto Enable peer-gateway, auto-recovery, fabricpath multicast load balance
 (if available), IP ARP Sync and IPv6 nd Sync

I’ll be looking to post more on vPC in the future, looking at particular topologies and the limitations with Layer 3, firewalls, and so on.