Networking configuration for ESX or ESXi Part 3

Shares

Today, third part, this time ESX(i) host has 10 pNIC’s (1Gbps) on Standard Switches (vSS)

Scenario #1 – 10 NIC’s (1Gbps – 2 x quad port adapters and 2 on-board ports) – standard Switch for each type of traffic

In  scenario I have to design network for 5 different type of traffic. Each of the traffic has different vLAN ID which will help to utilize all NIC’s for more than one traffic, optimize pNIC utilization and have network secured.

    1. mgmt – VLANID  10
    2. vMotion – vLANID 20
    3. VM network – vLANID 30
    4. VM Backup – vLANID 40
    5. DMZ – vLANID 50

When you don’t have Enterprise Plus vSphere license the only way to configure virtual networking is vSS. In a diagram below, mgmt (Service Console or vmk port) and vMotion were placed on common vSwitch0 with active passive approach (in vSphere 4 vMotion can use only on vmnic), Active and Stand by state is set in a portgroups. On physical ports, where both pNICs are connected two vLANs must be trunked (vLAN 10 and 20) cause we need both network available on each port, such as  in case of failover – traffic from both networks will carry over one port.

[box type=”warning”] Make sure that connection between physical switches are configured to carry all VMware specific traffic.[/box]

Other networks, have their own dedicated vSwitch’es, each vSwitch has at least 2 NICs connected to two physical switches and all vmnics are in Active state (see table below for details). Below configuration follows virtual networking best practices in terms of:

  • hardware redundancy – 2 physical switches, at least two pNIC per vSwitch,
  • failover – each virtual network has at least two vmnics available
  • security – separate vLAN for each traffic (e.g vMotion is not encrypted), vSwitch security options set to Reject
  • capacity – each network has preserve bandwidth capacity (sending traffic over separate physical NIC)

ESX ESXi networking configuration for 10 nics

vSwitch settings (applicable for all vSwitches)

  • Promiscuous mode – Reject
  • MAC address changes – Reject
  • Forget Transmits – Reject
  • Load balancing = route based on the originating virtual port ID (default)
  • Network failover detection – link status only
  • Notify switches – Yes
  • Failback – No
vmnic location vSwitch portgroup state vLANID pSwitch
vmnic0 on board vswitch3 backup VM active  30 Switch1
vmnic1 on board vswitch3 backup VM active  30 Switch2
vmnic2 quad NIC 1 vSwtich0 mgmt/vMotion active in mgmt passive in vMotion 10, 20 Switch1
vmnic3 quad NIC 1 vSwitch1 DMZ active  40 Switch1
vmnic4 quad NIC 1 vswitch2 VM network active  50 Switch1
vmnic5 quad NIC 1 vswitch2 VM network active  50 Switch1
vmnic6 quad NIC 2 vSwtich0 mgmt/vMotion active in vMotion passive in mgmt 10, 20 Switch2
vmnic7 quad NIC 2 vSwitch1 DMZ active  40 Switch2
vmnic8 quad NIC 2 vswitch2 VM network active  50 Switch2
vmnic9 quad NIC 2 vswitch2 VM network active  50 Switch2

If you have questions regarding particular case scenario, put question in comments and I will be glad to help you

Next post, further this week, will describes scenario with 10 pNIC but using vSS together with vDS (mixed virtual networking configuration approach)

 

UPDATE:

Network configuration 10 x 1Gbps for vSphere 5.1

 

10x1Gbps vSphere 5.1 - vDS

10x1Gbps vSphere 5.1 – vDS

Above is my recommended network configuration for vSphere 5.1 with Enterprise Plus license. As you know one of  the cooles new features in vSphere 5.1 is backup possibility of the Virtual Distributed Switches. In case you lost vCenter Database and there is no way to restore it you can easily restore vDS config into new DB – awesome. No risk of loosing network after vCenter DB lost and all network types including mgmt vMotion can run on single Virtual Distributed Switch. All vLAN has to be trunk on all physical switch ports.

Network configuration 10 x 1Gbps for vSphere 5.x and vSphere 4.x

 

10x1Gbps vSphere 5.x and vSphere 4.X - mixed - vSS and vDS

10x1Gbps vSphere 5.x and vSphere 4.X – mixed – vSS and vDS

My recommended network configuration for vSphere 5.X and vSphere 4.X with Enterprise Plus license. In above config vMotion and mgmt run on Virtual Standard Switch and Active/Passive vmnic configuration, where Storage, VM and FT traffic utilize Virtual Distributed Switch. The reason of heaving mgmt traffic on vSS is, in case of vCenter database lost you wont loose possibility to change ESXi/ESX host networking (N/A on vSphere 5.1 and above).

 

[box type=”info”] See links below for different networking configuration

ESX and ESXi networking configuration for 4 NICs on standard and distributed switches

ESX and ESXi networking configuration for 6 NICs on standard and distributed switches

ESX and ESXi networking configuration for 10 NICs on standard and distibuted switches

ESX and ESXi networking configuration for 4 x10 Gbps NICs on standard and distributed switches

ESX and ESXi networking configuration for 2 x 10 Gbps NICs on standard and distributed switches[/box]

0 0 votes
Article Rating

Artur Krzywdzinski

Artur is Consulting Architect at Nutanix. He has been using, designing and deploying VMware based solutions since 2005 and Microsoft since 2012. He specialize in designing and implementing private and hybrid cloud solution based on VMware and Microsoft software stacks, datacenter migrations and transformation, disaster avoidance. Artur holds VMware Certified Design Expert certification (VCDX #077).

Subscribe
Notify of
guest
19 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
sanderdaems

What about the “DMZ” network, if you need a fully NEN certified DMZ network? In that situation you need a separate PCI NIC adapter (separate bus) to split the network traffic, in case you combine 1 dual/quart port adapter with LAN/DMZ connections it’s “possible” to sniff the traffic/packets. To configure this redundant you need to add a second physical NIC adapter in the host to connect the physical DMZ network switches. If you don’t need a NEN certified DMZ and you mean “DMZ” as a different subnet with VLAN ID.. why don’t you add the two DMZ network adapters to… Read more »

StanJ

Hey Artur. I have also one proposal. As you also marked the topic as vSphere 5 related (upgrade expected) I would recommend you to move also one NIC port from the VM network to the Mgmt/vMotion group. This decision ofc depends on number of expected VMs (related to expected nr of vmotion migrations). You will have in this case environment more prepared for vSphere 5 from vmotion perspective and 3 NICs should be standardly enough for Prod traffic. Hi Sander, I’m also not familiar with NEN security certification. Can you briefly describe it or dirrect us to some documentation? Also… Read more »

Gopinath

hello I have 2 pswitches Cisco 6500 series as core switches, and i have vsphere 5 with enterprise plus license. I have one esxi host with 4 PNICS, the 2 core switches are interconnected via etherchannel trunk and we are not using stacking and stack cable. Scenario-1 in the esxi host, i have created 1 vswitch and 4 pnics are attached to it. 2 pnics are connected to the pswitch1 and other 2 pnics are connected to the pswitch2. the vswitch teaming policy is selected as (Load balancing = route based on the originating virtual port ID (default)). 5 virtual… Read more »

Gopinath

Great Help !!! Thanks for your timely help.

I am designing Vsphere 5, with HP 3PAR and HP C7000 blade center for a bank. The network consultant told there will be a packet drop in this design, so i confused.

Thanks for your advice and quick response.

Gopinath

One more doubt,

With same scenario of above, if there is 2 Top of rack switches connected between ESXi hosts and core switch,

I will connect the esx hosts pnics to the top of rack switches and the top of rack switches to the core switch in mesh topology.

1- In this case, i believe there wont be any issue, just like the previous post ?

2- Do I need to enable the STP and portfast in all switches (top of rack and core switch)?

thanks
Gopi

Ken

Thank you for this informative post. Just one quick question, I thought we can not have two active connection to to pswitch from one vswitch, like the one on vm backup. I tried this without luck.

Can you explain in couple of sentences please.

Thank you in advance

Ben

Hi, I am wondering why for vSwitch0 and vSwitch1, the vmnic(s) are not being used in a running number way?
As in, vSwitch0 – vmnic2 and vmnic 3, while vSwitch1 – vmnic6 and vmnic7.

Could you please enlight me on this part? Thank you!

Harry

How will be network configuration with 10 NICs (1Gbps) with Enterprise Plus license

Chuck

Lets say in the VM LAN I had 3 port groups would I want to use the Active/Standby approach or just leave all vmnics as active??

19
0
Would love your thoughts, please comment.x
()
x

FOR FREE. Download Nutanix port diagrams

Join our mailing list to receive an email with instructions on how to download 19 port diagrams in MS Visio format.

NOTE: if you do not get an email within 1h, check your SPAM filters

You have Successfully Subscribed!

Don't miss any posts. Subscribe To Our Newsletter!!

Join our mailing list to receive the latest news and updates from www.vmwaremine.com.

You have Successfully Subscribed!

Pin It on Pinterest