Cisco Nexus 1000v useful show commands

Using the topology view below, refer to useful show commands and the output.

YGNDC-1000V-ACC-01# show module

Mod  Ports  Module-Type                       Model               Status

---  -----  --------------------------------  ------------------  ------------

1    0      Virtual Supervisor Module         Nexus1000V          active *

2    0      Virtual Supervisor Module         Nexus1000V          ha-standby

3    1022   Virtual Ethernet Module           NA                  ok

4    1022   Virtual Ethernet Module           NA                  ok

Mod  Sw                  Hw     

---  ------------------  ------------------------------------------------ 

1    5.2(1)SV3(1.15)     0.0                                             

2    5.2(1)SV3(1.15)     0.0                                              

3    5.2(1)SV3(1.15)     VMware ESXi 6.0.0 Releasebuild-3620759 (6.0)    

4    5.2(1)SV3(1.15)     VMware ESXi 6.0.0 Releasebuild-3620759 (6.0)    

Mod  Server-IP        Server-UUID                           Server-Name

---  ---------------  ------------------------------------  --------------------

1    10.55.26.7       NA                                    NA

2    10.55.26.7       NA                                    NA

3    10.55.26.244     c4c014c0-d421-11e5-a0f5-ab9f62f8bc24  BOTVMWESXP1041

4    10.55.26.249     2e734260-d43c-11e5-aadc-959f62f8bc24  BOTVMWESXP1043

* this terminal session

YGNDC-1000V-ACC-01# sh vtracker module-view pnic

--------------------------------------------------------------------------------

Mod  EthIf     Adapter  Mac-Address    Driver    DriverVer           FwVer          

               Description                                                     

--------------------------------------------------------------------------------

3    Eth3/2    vmnic1   0050.56d2.fcbf                                              

               Emulex Corporation Emulex OneConnect OCe15100 NIC

              

3    Eth3/3    vmnic2   0050.56d2.fd46                                              

               Emulex Corporation Emulex OneConnect OCe15100 NIC

              

3    Eth3/4    vmnic3   0050.56d2.fd47                                              

               Emulex Corporation Emulex OneConnect OCe15100 NIC

              

4    Eth4/2    vmnic1   0050.56d2.fc8d                                              

               Emulex Corporation Emulex OneConnect OCe15100 NIC

              

4    Eth4/3    vmnic2   0050.56d2.fd20                                              

               Emulex Corporation Emulex OneConnect OCe15100 NIC

              

4    Eth4/4    vmnic3   0050.56d2.fd21                                              

               Emulex Corporation Emulex OneConnect OCe15100 NIC

              

--------------------------------------------------------------------------------

 

To view what VM are in each VLAN

-------------------------------------

To view each VM info hosted on specified VEM module

----------------------------------------------------------

To view the IP adderss of the VM, its Veth info and the module it's on

In older versions, the output is like below. 

To know where the VM (using IP address) is located on which ESX host, use this command and find out the module. Then use “show interface virtual” or “show module vem <num>” or “show module”.

------------------------------------------------------------------------------

To view previous vmotion events

To view which Veth resides on which host

To view which module or Eth port is on which host,

 

To view each vethernet port statistics 

To view how port-profiles are being used and what ports are using

Cisco Nexus 1000v Installation Steps and Logs

Steps to install VSM/VEM

1. Import OVA
2. can use same vlan for management, control and packet. only management vlan in used in Layer3 mode.
3. use manual install to go through step by step wizard for config when n1kv is online
4. use vrf management to test connectivity
5. deploy second vsm using secondary option.
6. after second vsm reload and online, primary vsm detect it and HA is established. (use terminal monitor on primary vsm to see events)
7. To connect to vCenter, https to vsm management IP and download cisco_nexus_1000v_extension.xml file
8. Go to vCenter --> manage plug in and install this plugin.
9. create vlans for servers, vsm mgmt, etc.
10. configure port-profile for uplinks (trunks, po, etc); l3 control
11. configure port-profile for vm - vlans (access)
12. create a vmkernel interface for vem ..to communicate with vsm (or use the same mgmt interface as Esxi vmnic0)
13. install vib on esxi host --> sftp the file (need to check which version of vib is compatible with esxi)  ~#esxcli software vib install -v /tmp/cross_cisco-vem....vib 
14. Check vem status on ESXi host  - vem status -v (to verify)

Connection to vCenter from N1000v can be done after step 8. Refer to the logs below.

ESX6_N1KV(config-svs-conn)# svs connection VS6_VC

ESX6_N1KV(config-svs-conn)# protocol vmware-vim

ESX6_N1KV(config-svs-conn)#  remote ip address 10.55.23.16 port 80

ESX6_N1KV(config-svs-conn)# vmware dvs datacenter-name YGN

ESX6_N1KV(config-svs-conn)# connect

2016 Jun 16 20:55:51 ESX6_N1KV vms[2713]: %VMS-5-CONN_CONNECT: Connection 'YGN_VC6' connected to the vCenter Server.

2016 Jun 16 20:55:52 ESX6_N1KV vms[2713]: %VMS-5-DVS_CREATE: dvswitch 'ESX6_N1KV' created on the vCenter Server.

Note: Command execution in progress..please wait

2016 Jun 16 20:55:54 ESX6_N1KV msp[2710]: %MSP-5-DOMAIN_CFG_SYNC_DONE: Domain config successfully pushed to the management server.

ESX6_N1KV(config-svs-conn)# 2016 Jun 16 20:55:54 ESX6_N1KV vshd[12445]: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by root on vsh.12445

2016 Jun 16 20:55:54 ESX6_N1KV vshd[12449]: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by root on vsh.12449

2016 Jun 16 20:55:54 ESX6_N1KV last message repeated 1 time

2016 Jun 16 20:55:54 ESX6_N1KV vshd[12445]: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by root on vsh.12445

2016 Jun 16 20:55:54 ESX6_N1KV vms[2713]: %VMS-5-DVPG_CREATE: created port-group 'Unused_Or_Quarantine_Uplink' on the vCenter Server.

2016 Jun 16 20:55:54 ESX6_N1KV vshd[12449]: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by root on vsh.12449

2016 Jun 16 20:55:54 ESX6_N1KV vshd[12445]: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by root on vsh.12445

2016 Jun 16 20:55:54 ESX6_N1KV vshd[12449]: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by root on vsh.12449

2016 Jun 16 20:55:54 ESX6_N1KV vshd[12445]: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by root on vsh.12445

2016 Jun 16 20:55:54 ESX6_N1KV vshd[12449]: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by root on vsh.12449

2016 Jun 16 20:55:54 ESX6_N1KV vshd[12445]: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by root on vsh.12445

2016 Jun 16 20:55:54 ESX6_N1KV vms[2713]: %VMS-5-DVPG_CREATE: created port-group 'Unused_Or_Quarantine_Veth' on the vCenter Server.

2016 Jun 16 20:55:56 ESX6_N1KV vms[2713]: %VMS-5-VMS_PPM_SYNC_TIME_EXP: Attempt #0 to resync Port-Profile Manager to local vCenter Server cache due to sync timer expiration

2016 Jun 16 20:55:56 ESX6_N1KV msp[2710]: %MSP-5-DOMAIN_CFG_SYNC_DONE: Domain config successfully pushed to the management server.

2016 Jun 16 20:56:01 ESX6_N1KV vms[2713]: %VMS-5-VMS_PPM_SYNC_TIME_EXP: Attempt #0 to resync Port-Profile Manager to local vCenter Server cache due to sync timer expiration

2016 Jun 16 20:56:01 ESX6_N1KV msp[2710]: %MSP-5-DOMAIN_CFG_SYNC_DONE: Domain config successfully pushed to the management server.

2016 Jun 16 20:56:06 ESX6_N1KV vms[2713]: %VMS-5-VMS_PPM_SYNC_TIME_EXP: Attempt #0 to resync Port-Profile Manager to local vCenter Server cache due to sync timer expiration

2016 Jun 16 20:56:06 ESX6_N1KV msp[2710]: %MSP-5-DOMAIN_CFG_SYNC_DONE: Domain config successfully pushed to the management server.

2016 Jun 16 20:56:11 ESX6_N1KV vms[2713]: %VMS-5-VMS_PPM_SYNC_TIME_EXP: Attempt #0 to resync Port-Profile Manager to local vCenter Server cache due to sync timer expiration

2016 Jun 16 20:56:11 ESX6_N1KV msp[2710]: %MSP-5-DOMAIN_CFG_SYNC_DONE: Domain config successfully pushed to the management server.

ESX6_N1KV(config-svs-conn)#

During VSM migration to 1000v VEM and ESXi mgmt. interface joining 1000v

YGNDC-1000V-ACC-01# sh module

Mod  Ports  Module-Type                       Model               Status

---  -----  --------------------------------  ------------------  ------------

2    0      Virtual Supervisor Module         Nexus1000V          active *

Mod  Sw                  Hw     

---  ------------------  ------------------------------------------------ 

2    5.2(1)SV3(1.15)     0.0                                             

Mod  Server-IP        Server-UUID                           Server-Name

---  ---------------  ------------------------------------  --------------------

2    10.55.23.254     NA                                    NA

* this terminal session

YGNDC-1000V-ACC-01#

2016 Jun 22 15:33:34 YGNDC-1000V-ACC-01 vem_mgr[2528]: %VEM_MGR-2-VEM_MGR_DETECTED: Host BOTVMWESXP1041 detected as module 3

2016 Jun 22 15:33:34 YGNDC-1000V-ACC-01 vns_agent[3294]: %VNS_AGENT-2-VNSA_LIC_NO_ADVANCED_LIC: VSM does not have Advanced licenses. May not be able to use VSG services. Please install Advanced licenses.

2016 Jun 22 15:33:34 YGNDC-1000V-ACC-01 vem_mgr[2528]: %VEM_MGR-2-MOD_ONLINE: Module 3 is online

2016 Jun 22 15:33:36 YGNDC-1000V-ACC-01 redun_mgr[2493]: %REDUN_MGR-2-AC_AC_DETECTED: Active-Active VSM detected.

2016 Jun 22 15:33:37 YGNDC-1000V-ACC-01 platform[2256]: %PLATFORM-2-MOD_DETECT: Module 1 detected (Serial number ) Module-Type Virtual Supervisor Module Model

2016 Jun 22 15:33:41 YGNDC-1000V-ACC-01 redun_mgr[2493]: %REDUN_MGR-2-AC_AC_ACTION: Primary VSM will be rebooted.

YGNDC-1000V-ACC-01#

YGNDC-1000V-ACC-01# sh module

2016 Jun 22 15:33:59 YGNDC-1000V-ACC-01 platform[2256]: %PLATFORM-2-MOD_REMOVE: Module 1 removed (Serial number)

Mod  Ports  Module-Type                       Model               Status

---  -----  --------------------------------  ------------------  ------------

2    0      Virtual Supervisor Module         Nexus1000V          active *

3    1022   Virtual Ethernet Module           NA                  ok

Mod  Sw                  Hw     

---  ------------------  ------------------------------------------------ 

2    5.2(1)SV3(1.15)     0.0                                              

3    5.2(1)SV3(1.15)     VMware ESXi 6.0.0 Releasebuild-3620759 (6.0)    

Mod  Server-IP        Server-UUID                           Server-Name

---  ---------------  ------------------------------------  --------------------

2    10.55.23.254     NA                                    NA

3    10.55.26.244     c4c014c0-d421-11e5-a0f5-ab9f62f8bc24  BOTVMWESXP1041

* this terminal session

YGNDC-1000V-ACC-01#

2016 Jun 22 15:34:06 YGNDC-1000V-ACC-01 platform[2256]: %PLATFORM-2-MOD_DETECT: Module 1 detected (Serial number ) Module-Type Virtual Supervisor Module Model

YGNDC-1000V-ACC-01#

YGNDC-1000V-ACC-01# sh module

Mod  Ports  Module-Type                       Model               Status

---  -----  --------------------------------  ------------------  ------------

1    0      Virtual Supervisor Module                             powered-up

2    0      Virtual Supervisor Module         Nexus1000V          active *

3    1022   Virtual Ethernet Module           NA                  ok

Mod  Sw                  Hw     

---  ------------------  ------------------------------------------------ 

2    5.2(1)SV3(1.15)     0.0                                              

3    5.2(1)SV3(1.15)     VMware ESXi 6.0.0 Releasebuild-3620759 (6.0)    

Mod  Server-IP        Server-UUID                           Server-Name

---  ---------------  ------------------------------------  --------------------

2    10.55.23.254     NA                                    NA

3    10.55.26.244     c4c014c0-d421-11e5-a0f5-ab9f62f8bc24  botvmwesxp1041.oml.com

* this terminal session

YGNDC-1000V-ACC-01#

  

Cisco Nexus 1100 Series Virtual Services Appliances

The management VLAN is used for management of the Cisco Nexus 1100 Series VSA.
The control VLAN is a Layer 2 interface used for communication between the redundant Cisco Nexus 1100 Series appliances. This interface handles low-level control packets such as heartbeats as well as any configuration data that needs to be exchanged between the Cisco Nexus 1100 Series appliances.

Network Connectivity Options

The interfaces on the Cisco Nexus 1100 Series can be connected to the network in five ways.

Network Connection Option 1 uses the two LOM interfaces to carry all traffic types: management, control, packet, and data. In this configuration, each uplink connects to two different upstream switches to provide redundancy. Option 1 is preferred in cases in which customers are not using a Cisco NAM and therefore have little or no data traffic traversing the uplinks. It is commonly used when the Cisco Nexus 1100 Series is used only for VSMs; the simplest configuration and lowest risk of misconfiguration. The LOM ports are active-standby only and cannot be part of a PortChannel or virtual PortChannel (vPC).




Option 2 uses the two LOM interfaces to carry management, control, and packet traffic. The other four interfaces on the PCI card carry only data traffic. Option 2 is well suited for customers who are deploying a Cisco NAM in the Cisco Nexus 1100 Series. Option 2 provides the most dedicated bandwidth for Cisco NAM traffic. The 4-port network interface card (NIC) adapter does support PortChannel and vPC capabilities and can provide added bandwidth utilization and redundancy.








Option 3 uses the two LOM interfaces for management traffic only, and it uses the four interfaces on the PCI card to carry control, packet, and data traffic. It's well suited for customers who are deploying a Cisco NAM or VSG in the Cisco Nexus 1100 Series but require a separate management network. Recommended for most deployments because it provides the flexibility to handle both currently supported and future VSBs.




Option 4 uses the two LOM interfaces for management traffic, two of the four PCI interfaces for control and packet traffic, and the other two PCI interfaces for data traffic. Well suited for customers who want to use the Cisco NAM but require separate data and control networks.




Option 5 Users can now more flexibly deploy their VSBs on the Cisco Nexus 1100 Series. With this option, you do not need to specify which ports allow which types of traffic (management, control, or data traffic). One of the main advantages of this option is that you can define a VSB to use a particular interface. This flexible option is an excellent option for users who want more control over the design of the VSBs for optimized flexibility and redundancy.

Deployment Considerations

Typically, the Cisco Nexus 1100 Series is best deployed at the aggregation layer of the network so that it can host a larger set of servers. Deploying the Cisco Nexus 1100 Series on the Cisco Nexus 2000 Series provides a large pool of servers supported on a single point of management for those servers. Because the Cisco Nexus 1100 Series uses 1 Gigabit Ethernet interfaces to connect to the network, a fabric extender provides an optimal connectivity solution.

In the uplink type 1 topology, all traffic (management, control, and VSB data traffic) is switched out at an effective bandwidth of 1 Gbps. Both LOM ports on the Cisco Nexus 1100 Series, Ethernet interfaces 1 and 2, are teamed to form an active-standby pair. This uplink type is simplistic and does not require any PortChannel or LACP configuration on the upstream switches.

In the uplink type 2 topology below, management and control traffic is switched out of the first two Ethernet interfaces. Ethernet interfaces 1 and 2 are forwarding as an active-standby pair, just as in uplink type 1. However, VSB data traffic is carried out of Ethernet interfaces 3 through 6.

Uplink type 3 is physically identical to uplink type 2 because it uses all the Ethernet interfaces available. The difference is in the way that the traffic is carried across these interfaces. In this topology, management traffic is switched out of the first two Ethernet interfaces. Ethernet interfaces 1 and 2 are forwarding as an active-standby pair, just as in the other uplink types. However, both control and VSB data traffic is carried out of Ethernet interfaces 3 through 6.

In uplink type 4, The VSMs residing on the Cisco Nexus 1100 Series VSA and the hosts that are managed by the VSMs can be connected over Layer 2 or 3 as explained in the previous sections.


Uplink type 5 is for the flexible network option and can be a combination of any of the other uplink types.

Cisco Nexus 1000V Series Backup and Restore Procedures

Here are the high-level steps for the VSM installed on the Cisco Nexus 1100 Series VSA.

Backup Procedure
1. Shut down the secondary or standby VSM VSB.
2. Export that VSB to remote storage.
3. Back up the running configuration of the Cisco Nexus 1000V Series VSA to a remote server or site.
a. Copy the running configuration often or whenever network the configuration has changed.
4. Power back on the secondary or standby VSM.
Restore Procedure
1. Completely remove the Cisco Nexus 1000V Series VSB if it is still on the Cisco Nexus 1100 Series VSA.
2. Create a new Cisco Nexus 1000V Series VSB.
a. Import a backup Cisco Nexus 1000V Series instance to the new VSB.
b. Verify that the Cisco Nexus 1000V Series instance is operational.
3. Restore the backup network configuration as the running configuration.
a. Verify that the port profiles and configurations are correct.
b. Verify that the virtual machines are connected to the appropriate port profiles.
c. Create a backup configuration of the running configuration after the environment has stabilized.

Cisco Nexus 1000v virtual switch

A software-based switch that spans multiple hosts running VMware ESX or ESXi.

VMware Networking

Each virtual machine has one or more vNICs(virtual NIC). These vNICs are connected to a virtual switch to provide network connectivity to the virtual machine. The guest OS sees the vNICs as pNICs.

Hosts running VMware ESX have a virtual management port called vswif, sometimes called as service console interface. It is used for communication with VMware vCenter Server, to manage the device directly with the VMware vSphere client, or to use Secure Shell (SSH) to log in to the host’s command-line interface (CLI). VMware ESXi hosts do not use vswif interfaces because the hosts lack a service console OS.

Each host also has one or more virtual ports called virtual machine kernel NICs (vmknic) used by VMware ESX for Small Computer Systems Interface over IP (iSCSI) and Network File System (NFS) access, as well as by VMware vMotion. On a VMware ESXi system, a vmknic is also used for communication with VMware vCenter Server.

The pNICs on a VMware ESX host, called virtual machine NICs (VMNIC), are used as uplinks to the physical network infrastructure.

Each vNIC is connected to a standard vSwitch or vDS through a port group. Each port group belongs to a specific vSwitch or vDS and specifies a VLAN or set of VLANs that a VMNIC, vswif, or vmknic will use.

===

Nexus 1000v Components

VSM – Virtual Supervisor Module

VEM – Virtual Ethernet Module  //linecards

Features – vTracker (shown in cisco device), VCPlugin (shows network info in Vmware)

The VSM is a virtual appliance that can be installed independent of the VEM: that is, the VSM can run on a VMware ESX server that does not have the VEM installed. The VEM is installed on each VMware ESX server to provide packet-forwarding capability.

VMware’s management hierarchy is divided into two main elements: a data center and a cluster. A data center contains all components of a VMware deployment, including hosts, virtual machines, and network switches, including the Cisco Nexus 1000V Series.

Within a VMware data center, the user can create one or more clusters. A cluster is a group of hosts and virtual machines that forms a pool of CPU and memory resources. A virtual machine in a cluster can be run on or migrated to any host in the cluster. Hosts and virtual machines do not need to be part of a cluster; they can exist on their own within the data center.

Port profiles create a virtual boundary between server and network administrators. Port profiles are network policies that are defined by the network administrator and exported to VMware vCenter Server. Within VMware vCenter Server, port profiles appear as VMware port groups in the same locations as traditional VMware port groups would. Port profiles are also used to configure the pNICs in a server; known as uplink port profiles, are assigned to the pNICs as part of the installation of the VEM on a VMware ESX host.

The Management VLAN is used for system login, configuration. and corresponds to the mgmt0 interface. The management interface appears as the mgmt0 port on a Cisco switch. Although management interface is not used to exchange data between the VSM and VEM, it is used to establish and maintain the connection between the VSM and VMware vCenter Server. When the software virtual switch (SVS) domain mode for control communication between the VSM and VEM is set to Layer 3 mode, the management interface can also be used for control traffic. Control interface handles low-level control packets such as heartbeats as well as any configuration and programming data that needs to be exchanged between the VSM and VEM. Because of the nature of the traffic carried over the control interface, it is of most importance in the Cisco Nexus 1000V Series solution. Packet interface is used to carry packets that need to be processed by the VSM; mainly used for Cisco Discovery Protocol and Internet Group Management Protocol (IGMP) control packets.

You can use the same VLAN for control, packet, and management, but if needed for flexibility, you can use separate VLANs.

switch# show switch edition //to show what version essential or advanced license for 256 sockets is free. License per VEM are sticky.

VEM are running on headless mode. If VSM goes down, data continue to flow but you can’t make changes. Each vSphere host gets a VEM. 

VSM

Deploy VSM in pairs just like in regular switches. 1 HA pair (or a VSM if standalone) can manage 64 VEM. So, it’s like virtual 66 slot chassis.

VSM heartbeats between Pri and Sec VSM are layer 2.

VSM control modes – L2 mode or L3 mode (more flexible; uses udp-4785) //VSM controlling VEM

VSM mgmt0 is default interface for L3

VSM connects to vmware vCenter using SSL connection.

Primary and secondary VSM must be in the same L2 domain. Latency should be 5-10ms max.  Usually makes sense to use Control0 interface for L3. Backup your config early and often.

VSM and VEM version can be 1 version difference but try to match.

#show module uptime

#show module vem

#show module vem counters

#show license usage

  

Backing up the VSM – running-config alone isn’t enough to restore. Clone the VSM and save it. To clone, shut down the standby VSM, and clone it as VSM doesn’t have vmware tools installed.

 

VEM

VEM is installed on each VMWare ESX host as a kernel component.

L3 control requires a VMKernel NIC(a virtual NIC on ESX host created by Vmware admin). Recommended using the ESXi management vmkernel nic as VMware does not have vrf concept – requires you migrate the management interface to VEM. –doesn’t require static routes on ESXi hosts.

VEM installation – VMware update manager (VUM). Or install vib file on ESXi     

Port Types

Nexus 1000V Series supports multiple switch-port types for internal and external connectivity: virtual Ethernet (vEth), Ethernet (Eth), and PortChannel (Po). The most common port type in a Cisco Nexus 1000V Series environment is the vEth interface which represents the switch port connected to a virtual machine’s vNIC or connectivity to specialized interface types such as the vswif or vmknic interface. A vEth interface’s name does not indicate the module with which the port is associated; a vEth interface is notated like this: vEthY. This unique notation is designed to work transparently with VMware VMotion, keeping the interface name the same regardless of the location of the associated virtual machine.

The second characteristic that makes a vEth interface unique is its transient nature. A given vEth interface appears or disappears based on the status of the virtual machine connected to it. The mapping of a virtual machine’s vNIC to a vEth interface is static. When a new virtual machine is created, a vEth interface is also created for each of the virtual machine’s vNICs. The vEth interfaces will persist as long as the virtual machine exists.

The switch contains two interface types related to the VMNICs (pNICs) in a VMware ESX host. An Ethernet, or Eth, interface is the Cisco Nexus 1000V Series’ representation of a VMNIC. A PortChannel is an aggregation of multiple Eth interfaces on the same VEM.

How Nexus1000v Operates 

Unlike physical switches with a centralized forwarding engine, each VEM maintains a separate forwarding/MAC address table. There is no synchronization between forwarding tables on different VEMs. In addition, there is no concept of forwarding from a port on one VEM to a port on another VEM. Packets destined for a device not local to a VEM are forwarded to the external network, which in turn may forward the packets to a different VEM. Static entries are automatically generated for virtual machines running on the VEM; these entries do not time out. For devices not running on the VEM, the VEM can learn a MAC address dynamically, through the pNICs in the server.

Cisco Nexus 1000V Series does not run Spanning Tree Protocol. Because the Cisco Nexus 1000V Series does not participate in Spanning Tree Protocol, it does not respond to BPDU packets, nor does it generate them. BPDU packets that are received by Cisco Nexus 1000V Series Switches are dropped. N1kv prevents loops between the VEMs and the first-hop access switches without the use of Spanning Tree Protocol. PortFast is configured per interface and should be enabled on interfaces connected to a VEM, along with BPDU guard and BPDU filtering. Filtering BPDUs at the physical switch port will enhance VEM performance by avoiding unnecessary processing at the VEM uplink interfaces.

VEM to VSM Communication 

The VSM can communicate with the VEM over the Layer 2 or Layer 3 network. Layer 3 is the recommended mode for control and packet communication between the VSM and the VEM. The VEM uses vSwitch data provided by VMware vCenter Server to configure the control interfaces for VSM-to-VEM control and packet communication. The VEM then applies the correct uplink port profile to the control interfaces to establish communication with the VSM. In layer 3 mode, you can specify whether to use the VSM management interface or use the dedicated Control0 interface for VSM-to-VEM control traffic. The port profile configured for Layer 3 (VSM-to-VEM) communication on the VSM needs to have capability l3control enabled. This process requires configuration of a VMware vmkernel interface on each VMware ESX host.

Layer 3 connectivity between VSM/VEM uses vmkernel ports. The VEM will use any vmkernel port in a port-profile tagged with “capability l3control” parameter.

Port Profiles

Live policy changes – changes to active port profiles are applied to each switch port that is using the profile.

Virtual Ethernet profiles  - A vEth profile can be applied on virtual machines and on VMware virtual interfaces such as the VMware management, vMotion, or vmkernel iSCSI interface.

Ethernet or Uplink Profiles - To define a port profile for use on pNICs, the network administrator applies the ethernet keyword to the profile. When this option is used, the port profile is available only to the server administrator to apply to pNICs in a VMware ESX server.

System VLANs – are defined by an optional parameter that can be added in a port profile. Interfaces that use the system port profile and that are members of one of the system VLANs defined are automatically enabled and forwarded when VMware ESX starts, even if the VEM does not have communication with the VSM. This behavior enables the use of critical host functions if the VMware ESX host starts and cannot communicate with the VSM.

VLAN consistency - Multiple VEMs require a physical Ethernet switch for inter-VEM connectivity. Each VEM needs consistent connectivity to all VLANs that are defined for the Cisco Nexus 1000V Series. Thus, any VLAN that is defined for the Cisco Nexus 1000V Series must also be defined for all upstream switches connected to each VEM. Each VLAN should be trunked to each VEM using IEEE 802.1q trunking.

#show module
#show interface brief

In the running config, there is “vem 3” , “vem 4” – a new line is added everytime a new host is added. “interface port-channel” configuration was dynamically created and added. “interface VethernetX” configuration was dynamically created and added.

System Vlans enable interface connectivity before an interface is programmed. System Vlans address the issue à VEM (line card) needs to be programmed but it needs working network (back plane) for this to happen. VEM will load system port-profiles and pass traffic even if VSM is not up.

Port Channels

3 modes à LACP, VPC –MAC Pinning and VPC-Host Mode CDP/Manual.

For LACP, on the VEM, use vemcmd show lacp

On the VSM and Upstream switch, use show port-channel summary, show lacp counters/neighbor

VPC-MAC Pinning works with any upstream switch. Allows for pinning of veths (VM) to specific links. Use it when upstream switch does not support LACP or using blade servers. Most blade servers cannot do LACP down to their servers.

When you assign the vmware physical NIC to the port-profile, that NIC has a number like vmnic0, vmnic1. And that vmnic becomes service group id on n1000kv. That gives us the ability to pin the traffic from VM based on the service group id number. If all esx host are the same, it is good but if they are not, those vmnic numbers could be different. To solve this, cisco has Relative mode à to assign the service group id based on the order which you add the nic to n1kv. 

Switch Stack

All stack members must run the same Cisco IOS software image to ensure compatibility among stack members.
From the stack master, you configure:
  • System-level (global) features that apply to all stack members
  • Interface-level features for each stack member
The stack master contains the saved and running configuration files for the switch stack. The configuration files include the system-level settings for the switch stack and the interface-level settings for each stack member. Each stack member has a current copy of these files for back-up purposes.

Switch Stack Membership
A standalone switch is a switch stack with one stack member that also operates as the stack master.
Adding powered-on switches (merging) causes the stack masters of the merging switch stacks to elect a stack master from among themselves. The reelected stack master retains its role and configuration as do its stack members.
Make sure that you power off the switches that you add to or remove from the switch stack.
After adding or removing stack members, make sure that the switch stack is operating at full bandwidth(40 Gb/s). Press the Mode button on a stack member until the Stack mode LED is on. The last two right port LEDs on all switches in the stack should be green.

A new, out-of-the-box switch (one that has not joined a switch stack or has not been manually assigned a stack member number) ships with a default stack member number of 1. When it joins a switch stack, its default stack member number changes to the lowest available member number in the stack.

A higher priority value for a stack member increases the probability of it being elected stack master and retaining its stack member number. The priority value can be 1 to 15. The default priority value is 1. Use "show switch". We recommend assigning the highest priority value to the switch that you prefer to be the stack master. To change the priority value for a stack member, use the switch stack-member-number priority new priority-value global configuration command. The new priority value takes effect immediately but does not affect the current stack master.

The stack master is elected or reelected based on one of these factors and in the order listed:
1. The switch that is currently the stack master.
2. The switch with the highest stack member priority value.
3. The switch that has the configuration file.
4. The switch with the lowest MAC address.
After a new stack master is elected and the previous stack master becomes available, the previous stack master does not resume its role as stack master.

Switch Stack Configuration Files
The stack master has the saved and running configuration files for the switch stack. All stack members periodically receive synchronized copies of the configuration files from the stack master.

A new, out-of-box switch joining a switch stack uses the system-level settings of that switch stack. If a switch is moved to a different switch stack, that switch loses its saved configuration file and uses the system-level configuration of the new switch stack.
  • If an interface-specific configuration does not exist for that member number, the stack member uses its default interface-specific configuration.
  • If an interface-specific configuration exists for that member number, the stack member uses the interface-specific configuration associated with that member number.
If you replace a failed member with an identical model, the replacement member automatically uses the same interface-specific configuration as the failed switch. must have the same stack member number as the failed switch.

Offline configuration to Provision a Stack Member
The configuration that you create on the switch stack is called the provisioned configuration. The switch that is added to the switch stack and that receives this configuration is called the provisioned switch. You manually create the provisioned configuration through the switch stack-member-number provision type global configuration command.

When you configure the interfaces associated with a provisioned switch (for example, as part of a VLAN), the switch stack accepts the configuration, and the information appears in the running configuration. The interface associated with the provisioned switch does not appear in the display of the specific feature(show vlan). The switch stack retains the provisioned configuration in the running configuration whether or not the provisioned switch is part of the stack.

If you remove a provisioned switch from the switch stack, the configuration associated with the removed stack member remains in the running configuration as provisioned information. (use the no switch stack-member-number provision global configuration command.) You can display the stack protocol version by using the show platform stack manager all privileged EXEC command.


Stack Protocl Version
A switch with the same Cisco IOS software version as the stack master can immediately join the switch stack. If an incompatibility exists, the fully functional stack members generate a system message that describes the cause of the incompatibility on the specific stack members. Switches with different major version numbers are incompatible and cannot exist in the same switch stack. Switches with the same major version number but with a different minor version number are considered partially compatible. When connected to a switch stack, a partially compatible switch enters version-mismatch (VM) mode and cannot join the stack as a fully functioning member.

If the auto-upgrade feature is enabled on the new switch, the stack master automatically upgrades the new switch with the same software image running on a compatible stack member.

Connectivity to Stack Member
To debug a specific stack member, you can access it from the stack master by using the session stack-member-number privileged EXEC command. Only the show and debug commands are available in a CLI session to a specific stack member.
The switch stack is managed through a single IP address. The IP address is a system-level setting and is not specific to the stack master or to any other stack member. Stack members retain their IP addresses when you remove them from a switch stack (so, change the IP addresses of any switches that you remove from the switch stack to avoid IP conflict.)

Setting the Stack Port Speed to 10 Gbps
In a mixed stack of Catalyst 2960-X and 2960-S switches, you must set the stack port speed to 10 Gbps. This task is required in a mixed stack configuration and must be run on a Catalyst 2960-X switch in the switch stack before you add a 2960-S switch to the stack. Otherwise, the switches will not stack.