Troubleshooting VVF Networking

0:00Hello and welcome.

0:01My name is Keith Barker, and in this set of videos, you and I get to focus our attention

0:06on troubleshooting networking inside of vSphere.

0:09This includes distributed switches and standard switches, and boy, there's a lot

0:13that can go wrong.

0:15So I'm super excited to have you join me in this set of videos and in this skill.

0:19So join me in the next video as we take a big picture look at the wonderful world of

0:23networking inside of vSphere.

0:01We're going to use the topology that we deployed with the VCF installer in the previous skill.

0:06So as part of that, we have four ESX hosts, ESXA, B, C, and D.

0:11And based on the deployment, how those VMs, those ESXA hosts were originally configured,

0:16they were all set up with a single VMNIC, which is the name of the uplink to the physical world.

0:22And so they have VMNIC 0 by default because they only had one.

0:27However, VMNIC 0 has a few problems.

0:29There's no fault tolerance on the network.

0:32So one thing I'd like to do to all four of our hosts here is to go ahead and add some additional VMNICs.

0:38So as a game plan, when we go in, we'll go ahead and add three VMNICs.

0:43So we'll have VMNIC 0, the one that's already there, plus VMNIC 1 and 2 and 3 for a total of four VMNICs.

0:51Also, as part of the deployment, the VCF installer removed completely the standard switch on all the hosts.

0:59So on one of the hosts, I'd like to go ahead and create a standard switch.

1:03And then what we can do is we can have the standard switch have a couple of uplinks,

1:07and also the distributed switch have a couple of uplinks.

1:10And when we get to the distributed switch part, we'll need to have that on all of the hosts,

1:14A, B, C, and D, the additional adapters, if we want to have uplinks associated with the distributed switch.

1:20And with working with switching, especially if we're changing things around,

1:23that always, in my experience, always introduces the opportunity for problems.

1:27And here's a few problems we can have.

1:29We can have VLAN tagging and trunking problems.

1:32So effectively, if this VM is in a port group, let's say that port group is associated with VLAN 20,

1:39when the VM sends their frames in and before they are sent out on the VMNIC associated with the virtual switch,

1:45where that port group is, the host is going to add a 802.1Q tag of 20 on there,

1:51so that when the receiving physical switch receives that, it knows exactly which VLAN that frame belongs to.

1:57And so if there's a misconfiguration between the port group up here and or the switch down here

2:03regarding tagging and trunking, that will cause a problem.

2:07Because the port down here on the switch, the physical switch, would need to be set for trunking

2:10to be able to interpret and understand that tag in the first place.

2:14And along those lines, we could also have an issue with link aggregation.

2:17And with link aggregation, we take a couple of links and logically treat them like one big pipe.

2:22Now, there's different ways of doing that.

2:24And so we need to make sure that if we're doing that up here at the host,

2:28we're also using that same method down here at the physical switch.

2:31So one common protocol is link aggregation control protocol, or we can just manually set up the lag.

2:37But if there's misconfigurations between the host and the physical switch, that could cause a problem as well.

2:43And I'll put that on the general category there of tagging and trunking issues.

2:47Another challenge that can come up is the nick teaming and failover.

2:51So with nick teaming and failover, there's lots of options here we can set up.

2:54We'll go through those in the interface.

2:56And if they're not set up correctly, that could cause a problem for delivering the traffic.

3:00So we'll take a look at that and the options there and also what to do as far as troubleshooting goes

3:05if it's not working.

3:07And this is especially true if when we're deploying a brand new environment where it's not going well,

3:12I'd like to give you some strategies on, first of all, get it working,

3:15and then piece by piece, add on the additional components.

3:19Now, another big gotcha that often requires troubleshooting is migration.

3:24And here I'm not talking about migrating a VM, for example, with a running VM with vMotion over to another host,

3:29but rather here in this context of troubleshooting networking,

3:32migrating VM kernel adapters and or VMs from one port group to another.

3:37And let me give you a foreshadowing of why that's so risky.

3:41So this is box A and this is box B.

3:44Let's say box A is using this path and box B is using this path,

3:47and we're migrating a VM kernel adapter from here to here,

3:51and we're also migrating a VM from here to here.

3:54Because there's a little moment there where it's doing the transitioning and other devices in the network

3:58with things like MAC addresses and MAC learning on the physical network side,

4:02there could be a problem.

4:04So I'd also like to walk you through examples of how you can baby step the actual conversion over,

4:09for example, from one port group on one switch to a different port group on a different switch.

4:13And part of it involves not using the same interface.

4:16So maybe over here we have vmnic0 and over here we have vmnic1.

4:21So we can set that up first, and then we can do the migration, for example,

4:25one at a time, the VM kernel adapter.

4:27And then we can go ahead and migrate over the VMs.

4:31So then when it's all done and we're no longer using switch A, for example,

4:34we can go ahead and take vmnic0 and add that into the other switch,

4:38so we can have vmnic0 and 1 here, and then go back and set up our teaming.

4:42But most of the time, if you just, say, move everything over at once,

4:45there's going to be a problem.

4:46So we'll troubleshoot that together in this set of videos.

4:49Another challenge that can come up is the MTU.

4:52The maximum transmission unit on Ethernet is generally 1,500 bytes.

4:56However, if we're doing network storage or vMotion,

5:00it's very likely we're going to have a much bigger MTU, like 9,000,

5:04which would fall into the category of a jumbo frame.

5:07And so if we want to be able to move 9,000 byte frames around the network,

5:12we need to make sure everybody in the path supports it,

5:14which would include the switch that we're using here at the host

5:18and also the physical switch and any other switches

5:21that traffic would be going through.

5:22If we're going to use jumbo frames,

5:24they all need to support the MTU that we want to use.

5:28So I'll walk through some techniques to troubleshoot and verify that as well.

5:32So with that in mind,

5:33let's keep this party moving by adding those three additional uplinks

5:38to our ESX hosts A, B, C, and D.

5:41So I'll see you in the next video for exactly that.

0:00So in this video, we're going to add VMNIC 3 Uplink 3 physical network interface cards

0:04or virtualized network interface cards to host A, B, C, and also D. Also, as a fun fact,

0:12the actual deployment that we did in the previous skill, it ran for about two hours last night.

0:17And when I came back in the morning, it was done. And that's the network that deployed

0:21network is what we're going to be working with in this skill.

0:24So also about VCF installer right here, we can go ahead and click on view details for

0:29the previous deployment. So it completed everything. Now, the reason I completed it was because

0:34it did a whole bunch of those validations early on that we troubleshot and work through.

0:38So once everything was placed, it's just completely finished. Also, I'm going to download the

0:42JSON spec right now. So it'll time and date stamp that for me. And that way I know I had

0:46the latest working configurations. Also right here, it shows me I can log into the VCF operations

0:53UI by clicking there. And also if I want to see the passwords, I can click here on review

0:57passwords. And this has the passwords for everything that was deployed. So I saved that.

1:04So for example, VCF operations, the login is admin, and that is the password. So I'm

1:11using just the credentials that it set up for me on the new appliances it deployed,

1:16including VCF ops. All right, I click on cancel there. So for vCenter, we can click this link

1:21right here, it would take us to the URL for vCenter for the login, then we can use the

1:26password specified right there. So the administrator is administrator at vSphere.local. And that's

1:31the password. So I already have a tab open for that. And that is right here. So I'm logged

1:37on as administrator at vSphere.local. In the host and clusters view, I've got four ESX

1:43hosts, fantastic. And as far as what it deployed, it looked like it deployed DRS on the cluster.

1:50So it deployed HA on the cluster, all through the VCF installer. So here is our CPU memory

1:57and storage capacity and usage for the cluster. So there's our vCenter right there. And there's

2:03three VMs that are currently running as part of VCF management services. The lab license

2:08appliance is running. And this is VCF ops right there, all running. I want to add some

2:13network interface cards to each of these hosts. However, they are currently supporting VMs.

2:19So if we click on VMs here, we have the cluster selected and make the font a little bit bigger.

2:23Fantastic. So here in the cluster, I've got six VMs that are running and they are distributed

2:29across our hosts, A, B, C, and D. So if we want to add network adapters, what we want

2:34to do is go ahead and take a host and let's start with C here. So we'll click on C and

2:39then right click and go to maintenance mode and enter maintenance mode. And it'll cause

2:45a vMotion evacuation of the VM there. So we'll click here on enter maintenance mode

2:51and okay. And this little VM lab ops 01 is in the process or will be shortly being migrated

2:58over to another host. So ESXC can be put in maintenance mode. Then once it's in maintenance

3:03mode, we'll go ahead and power it off and then add the adapters. And I'm going to do

3:08these one at a time because HA is enabled on the cluster and it's going to get upset

3:12if I try to take two or more offline at the same time. All right. So ESXIC has no more

3:18VMs running. So we'll right click on it. It's currently in maintenance mode and we'll go

3:22to power and we'll shut it down by clicking on shut down. So for the reason I'll say adding

3:27Nick, I'm going to put Nick's there and in the background, that'll be VM Nick's. And

3:31then we'll click on okay. And that is on its way down. Now, one of the benefits of doing

3:37this in a nested environment is that we can take a look at the status here and very easily

3:42add a network adapter to ESXIC once it's fully off. So the dark background here represents

3:49the control network ESX5 effectively. That is the control plane for all the nested VMs.

3:56So there's ESXIC. It's powered off. We'll right click on it. We'll go to edit settings

4:02and we want to add three more network adapters. So to do that for ESXIC, we'll click add new

4:07device from the pick list. We'll grab the very bottom here and network adapter and repeat

4:11that two more times. So add new device, network adapter, add new device, network adapter.

4:17And I want all of them going to on the host computer ESX5. I want them all going to the

4:23trunk port group, which leads to a trunked VM Nick on its own going out to the physical

4:29network. So that's all in place. So we'll click on okay. And now what we'll do is we'll

4:34right click, go to power and we'll power back on ESXIC. So back here in our vSphere environment,

4:41this is our vSphere environment in the lab. We have a alarm saying, Hey, ESXIC, we lost

4:45power. Yep. I'll go to actions and reset that screen and it should be back really shortly.

4:51So once it comes back, we'll go ahead and take it out of maintenance mode and then we'll

4:55repeat that process on ESX A and B and D. All right. So after three or four minutes

5:00there and a refresh is now back in maintenance mode. So we'll right click on it from the

5:06menu. We'll click on maintenance mode and then from the sub menu, exit maintenance mode.

5:10If on ESXIC, we go to configure and let's take a look at the networking section here

5:16and go to physical adapters. It now has four physical adapters, VM Nick zero, one, two

5:22and three. And the current one here, VM Nick zero is still associated with that distributed

5:26switch. So I'm going to take a moment right now and repeat that process for host A, B

5:33and D one at a time by doing the maintenance mode, which will evacuate any running VMs

5:38to the other hosts and then power it down, add the network cards, then bring it back

5:43up and take it out of maintenance mode. All right. And welcome back. So I just finished

5:48ESXID is just rebooting. So let's just verify the four VM Nick's we now have on all four.

5:54So starting with ESX dash a, we'll go to configure and we'll get onto networking and physical

5:59adapters and it has VM Nick zero, one, two, three. The first one is already associated

6:05with their distributed switch and it should be a similar case for host B, which it is

6:10and host C. And then we'll wait just for a moment or two until ESXID reconnects. And

6:16once it does, we'll take it out of maintenance mode and also verify that it also has the

6:19four network interface cards. Let's go take a look. There's the four network interface

6:23cards and we'll right click and take that out of maintenance mode by clicking on maintenance

6:27mode and saying exit maintenance mode. And once it's up and running in the background,

6:33DRS will very likely migrate one or more VMs back over to ESXD. And if it does, we'll

6:40see them right here on this list. And while we're waiting for that, let's also chat about

6:45a few things to expect when working with VM Nick's, especially for adding VM Nick's. It

6:50may not appear inside of the networking configuration for that VM. So currently I'm at the VMs,

6:57but if we went back to configure the networking and physical adapters, sometimes it won't

7:01show up right away. Now, in our case it did. So if you have added new adapters and they

7:06aren't showing up, you can go ahead and click refresh right there. Hopefully they'll refresh

7:10and show you those. And because we did have it powered down to add the VM Nick's when

7:14we rebooted it and it came back, we shouldn't need the refresh because we're looking at

7:19fresh information regarding that host. Also, the VM Nick numbering may not be what you

7:23expect, but a good idea is to just document what the actual Mac address is associated

7:27with those. And that way if you need to trace that or look at that from a switch perspective

7:31on a physical switch or with a packet capture, you can know exactly what the Mac address

7:35is associated with an interface and that can help in the troubleshooting process as well.

7:40And there's also some great options at the CLI for validating and verifying VM Nick's.

7:45Let me show you those right now. So here at the host, we can use this command ESX CLI

7:50network Nick list, and that will give us a list of our VM Nick's. So there's VM Nick

7:55zero, one, two, and three. There's their Mac addresses over here. Here's the speed. Here's

7:59the link status. And there's an older legacy command that still works. That'll give us

8:03that similar information. It's EX config dash Nick's space dash L press enter. Again, that

8:09gives us a similar information, including the names of the VM Nick's as well as their

8:13Mac addresses associated with them. So now that we have on each of our hosts for VM Nick's

8:18zero, one, two, and three in the next video, let's go ahead and put those additional three

8:24Nick's to work. So I'll see you in the next video for exactly that.

0:00In this video, I'd like to walk you through using these additional VMNICs.

0:03And so I do want to create a standard switch for the host here.

0:07So we're in the host and clusters view, configure.

0:09Let's go to how about let's pick on D, that'll work.

0:12So host D, if we go down to virtual switches,

0:16all it has is the distributed switch, no other standard switch,

0:20because the deployment for BCF removed that standard switch

0:24and put everything in the distributed switch with this associated port groups.

0:28So if we look at the networking view here and we click on the distributed switch

0:32and then go to configure, there's a beautiful view here under topology

0:36that will show us the details for that switch.

0:38So we currently have VMNIC zero and all four hosts

0:41that are being used by the distributed switch.

0:43And then we have four port groups.

0:45So we have one that's containing all the VM kernel adapters.

0:48And then we have one for NFS and the kernel adapters there.

0:52Then we have one for vMotion and the kernel adapters there.

0:55And then we have a management port group as well.

0:57So this one here is not using any tagging.

1:00This is VLAN 20.

1:02And this one also is not using any tagging.

1:04So the only one we have tagging at the moment is this port group right here

1:08doing a VLAN 20.

1:09So before we get into the distributed switch,

1:12let's go ahead and do a standard switch and we'll pick on host D.

1:15So here in host and clusters view host D, the way we do this

1:19to modify our work with the networking on a standard switch

1:23just locally on ESXD would be to right click.

1:26And then from here, click on add networking.

1:30Very similar to how we do it in the interface for the host client.

1:33There's an add networking option there as well.

1:35So we'll click on add networking and we're going to add a new physical adapter.

1:38So we'll go ahead and go here.

1:40Add new physical adapter.

1:41Click on next.

1:43Now it's asking where do you want to apply this?

1:45So if I wanted to add it to the distributed switch, I wouldn't do it here.

1:47I would want to do the distributed switch, adding a link

1:50over in the switching view in vSphere client.

1:53So I want to create a new standard switch

1:56and just for grins, I'm going to set the M2 to 9000 and then click on next.

2:01And now it's asking me, OK, here are three unclaimed VMNICs.

2:06One, two and three.

2:07And let's go ahead and take VMNIC and we'll click move down to the active adapter.

2:12And when you're doing new network interface cards and such,

2:15I would recommend adding one, testing it, and then you can set up

2:19the other ones and specify the teaming order and so forth.

2:22So for the moment, it's going to use VMNIC one here.

2:25And click on next and then click on finish.

2:28So now on ESXD, it's got this new standard switch, vSwitch zero,

2:33and it has one uplink associated with it.

2:36That's VMNIC one.

2:37So we want to add port groups here to the switch.

2:39We click on add networking and then we can add a port group.

2:42Click on next.

2:43We'll select our standard switch.

2:44Click on next.

2:45So I'm going to call it VLAN 23 port group.

2:48And then for the actual VLAN, I'll have VLAN 23.

2:51I've got a DHCP server out there on my physical network.

2:54That way we can test it as well.

2:56So click on next and then finish.

2:59And bada bing, bada boom.

3:00This should refresh here in a moment.

3:02And there's a port group so we could apply or attach virtual machines

3:06to this port group for the network interface card.

3:08And effectively, they would be on VLAN 23 on the physical network.

3:12Now, another way of testing that VLAN 23 connectivity works

3:15is we could also just create a VM kernel adapter

3:18and associate that with VLAN 23.

3:20And that would be another way of doing it.

3:21So let's do that as a test.

3:22We'll click on add networking

3:24and we want to add a VM kernel adapter.

3:26Click on next.

3:27And I want to go ahead and choose a existing standard switch.

3:32So we'll choose our switch.

3:34Switch zero. Click on next.

3:35And then we'll call this test V23.

3:39And then for the VLAN, we'll go and choose 23.

3:42If it wasn't already there, we can just type in 23.

3:44It'll get the MT from the switch and be in the default TCP stack.

3:48And I don't need any additional features here.

3:50So click on next.

3:51And then I'm going to say obtain IPv4 settings automatically.

3:55So that way we'll know very quickly if it works or not,

3:57because if we click on finish here, if it shows an IP address

4:02associated with that VM kernel adapter,

4:04that means it made it to the VLAN 23 network and the DHCP services worked.

4:10So I'm going to click on refresh here.

4:11And sure enough, look at that 23.1.2.223.

4:15And the default gateway on that 23 network is 23.1.2.1.

4:20So if we wanted to test that, we're still there at an SSH session for ESXD host D.

4:24We could do a ping to 23.1.2.1 and that works.

4:28Or we could use VMK ping and 23.1.2.1.

4:32And VMK ping has the additional flexibility

4:34because we can actually specify which protocol stack to use.

4:38So if we have a separate, for example, vMotion protocol stack,

4:41we can go ahead and specify vMotion and we can also specify

4:44which source interface to use.

4:46So ping in this case and VMK ping both are going to work great.

4:49Just to verify basic connectivity.

4:51So now we know that that works.

4:52Let's go ahead and add some redundancy for the standard switch.

4:56So to add redundancy, we'd go here to switch zero, click add networking

5:00and we're going to add another physical adapter.

5:02So go ahead and select that, click on next.

5:04We're going to add that to our existing vSwitch zero, click on next.

5:08And then we'll grab vMNIC two and click on move down

5:11and add it to our active adapters.

5:14So now at this point, once we click on next and then click on finish.

5:18So now it has the two vMNICs.

5:20So if we go back to the properties of the switch, we click on edit for vSwitch zero.

5:24If we go to teaming and failover, we can now choose, for example,

5:27you know what, I want these both to be active.

5:29And if so, how is it going to decide how to load balance?

5:33So we'll have a separate video in the set on load balancing.

5:35But I just want to show you that that option is here for a standard switch

5:39as well as a distributed switch.

5:41So I'm going to go ahead and say for this example, I'm going to move that down

5:44and say use vMNIC one.

5:45And I'm also going to specify use the explicit failover order.

5:48That way I'll use vMNIC one.

5:51And if vMNIC one fails, loses link, it'll go ahead and start using vMNIC two.

5:55And then we'll click on OK.

5:57So now if we click on vMNIC three here, it should show us

5:59that it's going to be using vMNIC one because that's the order.

6:03I have that set up as the primary adapter and the standby is vMNIC two.

6:08And the same thing here for this port group called VLAN 23.

6:11Also, if we want to set up a port group that did trunking.

6:14So maybe we have something connected to our standard switch

6:16that is going to be sending in 802.1Q tagged frames.

6:20What we want to do is create a port group to support that.

6:23Let me show you that as well.

6:24We'll click on add networking and I'm going to add a port group.

6:27We'll click on next.

6:28We'll add that to our standard switch.

6:30We'll click on next.

6:31And I'm going to call this trunk.

6:33And I'm calling it that because I'm going to use this all here.

6:36And that's how you specify

6:37that you want this logical port group to support incoming 802.1Q tags.

6:43So we'll click on next and finish.

6:45So here's our port group called trunk.

6:47And here for this port group, VLAN 23 PG for VLAN 23.

6:51If we connect devices to it like PCs, printers, et cetera,

6:54and they send their ethernet frames in the switch, this virtual switch

6:58is going to add that tag of 23 before sending it over the uplink.

7:02And for this trunk, if we connect devices to it,

7:05it's simply able to accept incoming 802.1Q tags from whatever device

7:10is sending those in, and it can continue to propagate those

7:13down the uplink to the physical world.

7:15So all of this virtual switching, because the standard virtual switch

7:18is only in the brain here of ESXD.

7:21So now that we've taken a look at it, we did some testing.

7:23I'm going to click on the three dots for virtual switch zero.

7:26Click on remove. And yes.

7:29And if we get an error, we'll work through it together.

7:32And it's done.

7:34Wow. No errors.

7:35So let's also go back to the VM kernel adapters.

7:38It also got rid of our VM kernel adapter

7:41that was associated with that virtual switch.

7:44But we still have our four uplinks and only the first one's tied up.

7:47So now that we have ESXA, A, B, C and D,

7:50all with these four uplinks with three of them available.

7:53Let's go ahead and add some redundancy into our distributed switch.

7:57So to do that, we'll go to the switching view up here.

8:00We'll select our distributed switch right here.

8:02One of the views I love is going to configure and the topology

8:05to kind of give you a nice, beautiful view of the port groups and so forth.

8:09So to add a new uplink to a distributed switch,

8:11we'd go to the distributed switch here in the switching view,

8:14right click on the distributed switch and go to settings

8:18and then click on edit settings.

8:19And then here we click on uplinks.

8:21And here we can simply click on add and add.

8:24Let's add a couple.

8:25So here's a placeholder called uplink one.

8:28I'm going to go ahead and call this VM zero.

8:30And that way, when I see it, I'll remember which VMNIC goes there.

8:33And I'm going to call this one VMNIC one and I'm going to call this one VMNIC two

8:38because all the hosts are the same.

8:40That way, I'm going to make sure I allocate the proper adapters.

8:43So the first one's already in place.

8:44So I'll click on OK.

8:46Now, even though we made that change.

8:47Oh, look at that more task.

8:48It didn't like me renaming the existing uplink.

8:52It was already in use.

8:52All right. Let's go back to the inventory view

8:55and we'll say thank you for that little error.

8:57And we'll right click on that distributed switch.

9:00And then we'll click on settings and we'll click on edit settings and uplinks.

9:04So I won't rename that. It's already in use.

9:06But I'll call this next one VMNIC one

9:09and I'll add one more and I'll call it VMNIC two.

9:12And it didn't like me changing the original name there

9:14because it's already associated with the hosts.

9:16So I'll just add those two extras and we'll click on OK.

9:20All right. No complaints there.

9:21So now if we go to configure and topology,

9:24now we have four uplinks, one from each host,

9:28but we don't have the additional uplinks

9:30in the placeholder of VMNIC one and VMNIC two.

9:32So we need to add those.

9:33So we'll right click on the distributed switch

9:36and we'll go to add and manage hosts.

9:38And then from here, we're going to click on manage host networking.

9:41So we'll click on next.

9:43We'll select all four hosts and click on next.

9:46And then we're going to do here.

9:47I want to go ahead and assign VMNIC one in the position of VMNIC one

9:52and then VMNIC two in the position of VMNIC two.

9:55And boom. So that's going to be VMNIC zero there

9:58that's already assigned in the placeholder for uplink one.

10:00So we'll click on next.

10:02I'm not going to migrate anything over.

10:04I'm just adding uplinks at this time.

10:06So one step at a time, click on next and then next

10:10and finish and bada bing bada boom here in a moment.

10:13So there's VMNIC zero, one and two.

10:17So now at this point that we have multiple VMNICs to play with

10:20associated with a distributed switch, we can then go in and start

10:23configuring how we want the load balancing to happen.

10:26So let's do that part in the next video, and I'll see you there in just a moment.

0:00So now that all four hosts, let's go back and just confirm here, if we go to the host and clusters view

0:06and we go to ESXIA and go to configure and physical adapters, there's four adapters there

0:11and it's also showing us now that the first three adapters are associated with that distributed

0:15switch. And that's going to be the case for B and C and D. If we go to the switching view here,

0:21we go to our distributed switch and click on configure and topology, that will also confirm

0:26it there as far as which VMNICs are being used as part of the distributed switch. However, what's

0:31not configured for us is perhaps the ideal method for load balancing and that's done on a per port

0:39group basis. So let's go ahead and start with this port group right there, which is used for ESX

0:46management. So we can right click on it and then go to edit settings for that port group. And from

0:50here, we can go ahead and specify the teaming and failover. So by default, we have one active uplink

0:56that's the uplink labeled uplink one. And then the unused uplinks are VMNIC one and two. So let's

1:02talk about our options here. So if we wanted all three to be active uplinks, let's go ahead and do

1:07that. Let's just go ahead and highlight VMNIC one and two with them both highlighted. We'll click on

1:12move up and move up. And now they're all three active and we'll click on okay. Now before I do

1:18that, we also can control how that load balancing will happen because we have three uplinks. They're

1:23all 10 gig. So which one is it going to use as it sends frames down to the physical network down

1:29through the VMNICs? And the answer is based on the load balancing question up here. So we have route

1:33based on IP hash, route based on source MAC hash, route based on originating virtual port, or use

1:40the explicit failover order if you have failover setup, meaning like one active link and standby

1:44links or route based on physical NIC load. And here's where I'm going to give you the device

1:49regarding troubleshooting. If you're putting in a brand new deployment and let's say you have three

1:54uplinks and you're choosing one of these load balancing options and it's not working, what I

2:00would strongly recommend is go ahead and use one uplink. Work on the basic connectivity, making

2:06sure it works, and then you can introduce additional VMNICs as active uplinks. Or if you want to have

2:11just a primary uplink and a backup, it would look something like this. You can go ahead and grab the

2:16two you're not using, click on move down, and then grab the one you want as a standby so it's highlighted

2:21by itself, and click on move up, and there it is. So in this case here we could say I want to go ahead and

2:26use explicit failover order, and that way I would just use uplink one. Unless it failed, there's a loss

2:31of link, and then we'd go over to this uplink right here as a standby. So just remember that this load

2:37balancing is part of the outbound traffic from this virtual switch on this port group. So this is not

2:43controlling what the physical switch we're connected to, how it's sending data up to us. And where it

2:49gets a little tricky is that if we do have two active links, I'll go ahead and move that up, we'd

2:54want to be sure that the switch at the other side is set up in a compatible manner to support that.

3:00So when in doubt, go to one uplink, one active uplink, get things working, and then with the person in

3:07charge of your switch, whether it's a Cisco switch, or Juniper switch, or an HP switch, or some other

3:12flavor, get with them to make sure that the settings you have are compatible with what they have on the

3:17physical switch. There's another really cool option I should show you right now, and that is if you want

3:22to change this for multiple port groups, what you can do is go up to the switch itself, right click,

3:29and then click on distributed port group, and look at this, I love this feature, manage distributed

3:33port groups. And that way you can say, you know what, I want to go ahead and change the teaming

3:38and failover for a bunch of port groups at the same time. So we'll click on next, and then I'm

3:43going to go ahead and choose all of them. Now sometimes you might want to use like one link

3:47as a primary for like NFS, and a different link as the primary for management or vMotion, and then

3:53have the other one as a backup, so you can mix and match. However this example, I'm going to set them

3:57all the same way, so I'm going to go ahead and click on next, and then for the uplinks, I'm going to go

4:02ahead and I'm going to take these two, I'll move them down to standby. So these will be to standby,

4:09and also I'm going to use the explicit failover order. So if uplink one fails, you'll start using

4:16vMNIC one, and if vMNIC one fails, and start using vMNIC two. So we have some fault tolerance there.

4:21And in my lab on the back end, this is all a 10 gig link, so we have lots of bandwidth for everything,

4:26even without using multiple links. So we'll click on next, and finish, and we just applied that to

4:32all four of these port groups. So if we went to this first port group, and went to configure and

4:36policies, here is the active and standby links, and same thing for this port group, and the same

4:43thing for this port group as shown right there, and the same thing for that port group as shown

4:47right there. We could also go to the switch, and go to configure and topology, and then we could click

4:52for example on a vmkernel adapter, and it would show us which of the links it's going to use,

4:57because that's the primary link, which in our case is vmnic zero on all four hosts.

5:02And if vmnic zero goes away, it would then go ahead and start using vmnic one. And then also

5:07if vmnic one also lost link, it would start using vmnic two. And I intentionally didn't use vmnic

5:13three yet, because in the next video, I'd like to leverage that additional vmnic, and walk you

5:19through troubleshooting of migration, whether it's vmkernel adapters, or vms, and moving them between

5:25different switches, which are associated with different uplinks. So we'll do that in the next

5:29video, and I'll see you there in just a moment.

0:00For this demonstration about troubleshooting and working with migrating VMs and VM kernel

0:05adapters, let's go grab a critical system.

0:08So if we go to this port group, and there is vCenter right there, that is currently

0:14connected to our distributed switch, which is using the primary uplink of vmnic0.

0:19So if we're doing migration from one switch to another, let's create another distributed

0:22switch.

0:23And so to do that, we'll just right click here in our data center when the networking

0:27view.

0:28We'll right click on the data center.

0:29We'll click on distributed switch and select new distributed switch.

0:33And we'll call this VDS test and click on next.

0:36So it's asking us what's the minimum version of the ESXi host are going to be used with

0:41this distributed switch.

0:42It's all 9.1.

0:43So I'll click on next.

0:45And then the number of uplinks.

0:46I only need one uplink.

0:47I only have one left.

0:49And then do we want to create a default port group?

0:51I'll say no.

0:52I'll come back to that in just a minute.

0:53And we'll click on next and then click on finish.

0:56So now we have this new distributed switch.

0:59But I want to go ahead and also set the MTU larger than normal.

1:03So at the switch level, we'll right click, click on settings and click on edit settings.

1:08And then for the advanced, let's set it to 9000.

1:12And then for the uplinks, we're going to go ahead and we're going to be using vmnic3.

1:16So I'm going to label this uplink before any hosts are using it.

1:19I'm going to call it vmnic3.

1:21That way in the heat of battle, when I have to assign vmnic3 to that uplink, I'll know

1:25exactly which uplink to use.

1:27Then also over here, let's go ahead and do LLDB for both send and receive for discovery

1:34and then general.

1:35That looks great.

1:36So we'll click on OK.

1:37All right.

1:38So now we have this beautiful distributed switch with vmnic3 as a placeholder there,

1:42but we have no ESXi hosts that are using it yet.

1:45So let's go ahead and onboard some.

1:47So we'll right click on distributed switch and we'll go to add and manage hosts.

1:51We'll click on add host there and click on next.

1:54We'll grab everybody and then click on next.

1:57And then as far as the mapping, we're going to take vmnic3 on all of those four hosts

2:02and map it to the label vmnic3 on the distributed switch and then click on next.

2:07We're not going to do any migration and that's a big temptation.

2:10It's better to do it in steps.

2:12So right now we're not going to do any kind of migration here because we're just onboarding

2:17the host to use this switch.

2:19So if we tried to onboard the host and migrate vmkernel adapters and migrate VMs, there's

2:24such a great opportunity for failure that I don't even attempt to do it anymore.

2:28I'll do it step by step.

2:29So we're not migrating any vmkernel adapters on the hosts.

2:34So I'll leave those all non-selected.

2:36Click on next.

2:37Not migrating any VMs yet.

2:39So we'll click on next.

2:41And here's a summary.

2:42Four hosts added, one adapter from each for a total of four adapters being added to this

2:46distributed switch and click on finish.

2:49So that should show up over here in just a moment as it updates.

2:53And there it is.

2:54For the uplinks, we have no port groups yet, but we have four uplinks, one from ESX A,

2:59B, C, and D. So next let's create a port group that we attach VMs to.

3:04So to do that, we simply right click over here on the distributed switch, go to distributed

3:08port group, and then click on new distributed port group.

3:11I'm going to call this V23 and click on next.

3:15And then we're going to associate it with VLAN 23.

3:18So I'll put those details in there.

3:20And I also want to look at the other parameters as well.

3:22So I'm going to click on the checkbox for advanced, click on next.

3:25And for this port group, we're going to go ahead and use those defaults.

3:28Mac learning, you can turn on or off.

3:29We'll click on next.

3:30I'm not going to do any traffic shaping for inbound or outbound traffic.

3:34Click on next.

3:35Teaming and failover.

3:36So for fun here, I'm going to use explicit failover order, although I don't have anything

3:40to failover to because we used up all our uplinks.

3:43Click on next and NetFlow is disabled and not blocking ports and ready to go.

3:48Click on finish.

3:50So boom.

3:51Now we have this port group V23 that's associated with VLAN 23.

3:56Now one interesting note is that if we went back to the hosts, hosts and clusters view,

4:00went to ESXIA and we went to virtual switches here, but just be aware that in the host view

4:06here for the distributed switch, if they don't have some reason to show the actual port group,

4:10for example, it's not being used, it's not going to show up here.

4:13So as a test, let's go ahead and add some VM kernel adapters.

4:18So we're going to go back here to the networking view and here's our distributed switch.

4:24And for this port group called V23, I'm going to right click and I'm going to add VM kernel

4:28adapters.

4:29The easiest way is the right here, instead of going back to the host and adding one and

4:33the second one, just add VM kernel adapters.

4:36So I'm going to add a VM kernel adapter to all four hosts.

4:39Click on next.

4:40So it's going to get the MTU from the switch.

4:42I reset that to 9,000.

4:43So that's great.

4:44And if we wanted this to be used for some specific purpose, we could add on that service

4:48down here.

4:49And if it's one of these five, we could actually put it in a separate protocol stack just for

4:53that function.

4:54So I'll use the default for now.

4:56We'll click on next.

4:57And I'm going to select obtain an IP address automatically.

5:00And I have a DHCP server already down there on the 23 network and we'll click on next.

5:05So here's our summary and we'll click on finish.

5:08And it is now in the process of creating VM kernel adapters on all four of our hosts.

5:13So if we go back to the distributed switch, go to configure and topology, there they are

5:18right there.

5:20And I'll click on refresh.

5:23Maybe I don't have DHCP services set up for VLAN 23.

5:27Oh, there I do.

5:29Okay.

5:30So three of them have it.

5:31And the last one here will also get it, I believe.

5:34Yeah, there we go.

5:36Just took a moment for them to do the discover offer request acknowledgement process with

5:41my DHCP server.

5:42All right.

5:43So now that we've tested and verified, we have contact with the physical world who's

5:46issuing out these IP addresses.

5:49Let's go ahead and do a migration.

5:52Now I don't want to migrate anybody over to V23.

5:54But what I do want to do is migrate a device, in fact, our vCenter, I'm going to make it

5:58real.

5:59Let's migrate our vCenter over to a port group on this switch.

6:03So let's create a port group appropriate for that.

6:05So I'll go ahead and go to distributed port group, new distributed port group.

6:09And we're going to call this 192-VDS-PG.

6:15So effectively to me, that's going to remind me that this is not going to do 802.1Q tagging.

6:20And it's going to be basically a straight shot to the physical network, which is my

6:24192 network.

6:25So I'm going to name it 192-VDS-PG, click on next.

6:29And then for the VLAN, I'm going to say none.

6:31And then click on customize default policies, click on next.

6:35Those are fine.

6:36That's fine here.

6:39That's fine.

6:40And fine and fine.

6:42So I guess I didn't need to look at those.

6:43All right.

6:44So here's the new port group.

6:45Again, if we went to a host view on this, and we looked at it, it's showing the V23

6:50because that's where the VM kernel ports are.

6:52However, because ESX host A does not have a VM or a VM kernel adapter in that new port

6:58group that we just created, for that reason, it's not showing up here from the perspective

7:03of the actual hosts.

7:04But if we go to the networking view, it's right there.

7:07So we'll go to configure and topology, and it's right there.

7:11So let's do a migration.

7:12First of all, let's do a migration first of a VM kernel adapter, and let's pick on ESX-C.

7:19So currently, you've got a configure and VM kernel adapters.

7:23It has this adapter of .63, which is VM kernel zero, that is the main management interface

7:30that it's using to communicate with vCenter.

7:33So I propose we migrate that and put it over on the other switch, that one we just made.

7:39So to do that, we'd go back to the switching view, and then right click on BDS test, and

7:44then click here on add and manage hosts.

7:46And then we want to go ahead and manage host networking, click on next.

7:51And we want to go ahead and take C, I think we said, and click on next.

7:55And I'm not adding any new network adapters, click on next.

7:58But what I do want to do is take just the VM kernel adapter for host C, and also I can

8:04do adapters by host.

8:05Now because we only chose C, if we chose a whole bunch of hosts, this is how you could

8:09individually do it.

8:10So I'm going to stay here, just to confirm it's only happening to host C. So here on

8:14host C, I'm going to take this VMK zero, which is currently associated with that source port

8:19group.

8:20And I'm going to click on assign port group, and assign it to the port group called 192VDSPG.

8:26And click on assign, and click on next.

8:28I'm not going to migrate any VMs at this point, because I'm doing it one step at a time.

8:32We'll click on next.

8:33So it's saying one host is going to be updated, and one VM kernel adapter is going to be moved

8:38from the old switch it was on to the new switch.

8:42So we'll click on finish.

8:43And let's take a look.

8:45We'll go to the distributed switch, click on configure, click on topology, and here

8:51for this new 192VDSPG, we have the VMK zero, 192.168.1.63, now associated with that new

8:59port group.

9:00Now they both lead to the same place, to the same physical network.

9:03And so vCenter can still talk to it.

9:05And that's an example of a successful migration of a VM kernel adapter.

9:10Now what's more scary to me, is moving a vCenter over to a different network where

9:18it has different uplinks, because there's a lot that can go wrong.

9:21And so here's the recipe for disaster, creating a new distributed switch, and onboarding your

9:27host and at the same time in that same wizard sweep, also migrating your VM kernel adapter

9:32and also migrating your vCenter adapter on the VM, all at the same time.

9:38There's just so much that can go wrong, and that's why I've done them in three separate

9:41steps.

9:42Create the distributed switch in the port group, onboard the host, assign the links,

9:47and then migrate VM kernel adapters, and then separately, go ahead and migrate your VMs.

9:52So let's go confirm our vCenter.

9:55So I'm going to go to the VM and templates view.

9:57There it is right there.

9:59And it's IP address is .65, yeah, that's our vCenter that we're using right now that we're

10:03logged on to.

10:04So one way of moving it over, we need to right click and go to edit settings.

10:08And then we go down to the settings here for the network adapter, click on browse, and

10:13we can choose that new port group.

10:15Or a better way, especially if you're doing lots of them, is to go back to the switching

10:18view and then right click the current distributed port group that VM is a member of and click

10:24migrate VMs to another network.

10:27So then we can choose the target network.

10:29So that's the port group right there for that new switch, we'll click on next.

10:33And now we can go ahead and specify the VMs.

10:35Oh, I must have grabbed the wrong one.

10:37There's no VMs here.

10:38Let's go back.

10:39And let me go ahead and go cancel.

10:41And let's just go take a look real quick.

10:43So I'm going to click on this port group, go to VMs, and that one.

10:47And that one.

10:48Oh, it's right there.

10:49All right.

10:50So there it is right there, labbc01.

10:53So we're going to do is we're going to right click on that port group and click on migrate

10:58VMs to another network.

10:59It's going to ask us where do you want to send them.

11:01So we'll send them to the 192 VDSPG, click on next.

11:05And now it's asking, which VMs do you want to send over and do just the vCenter right

11:09there and click on next.

11:12So for me, now this is a very nervous moment because anytime you move vCenter from one

11:16switch to another and there's different uplinks, the physical network could have a problem

11:20as well.

11:21So we may lose some connectivity for a moment.

11:22We're avoiding the troubleshooting by doing most of it in order, creating the switch,

11:26the port group, bringing on the host, migrating VM kernel adapters, and then separately migrating

11:32the VMs.

11:33And one of the secrets of that is having enough uplinks, even temporarily enough uplinks.

11:36So you can go ahead and move stuff over and have both switches active.

11:40And then once you're done with one switch and you no longer need it, you can delete

11:43it.

11:44So I'll go ahead and click on finish here.

11:45And let's take a look at recent tasks, updated network configuration for labbc01.

11:53We're still connected.

11:54That's good news.

11:55So if I click on refresh here, we are still connected to vCenter.

11:59So that was successful.

12:01And I'm going to move it back because I want it in that same port group where VCF installer

12:04put it.

12:05So we could go ahead to this port group right here.

12:08So if we select that port group and go to VMs, there's that little VM.

12:10So we could go ahead and right click on that port group and then migrate VMs to another

12:15network or we could go to details for that VM right there.

12:19So we'll click on the VM, right click on labbc01, and then click here on edit settings.

12:25And then for the network adapter here, click on browse and put it back on that management

12:29port group right there.

12:30Click on okay and okay.

12:32That's yet another way to do it.

12:33However, if you have to do it with lots of VMs, it makes sense to do it from the networking

12:38view and do them in bulk.

12:40Also I'd like to migrate my VM kernel adapter back over.

12:44So to migrate host C's VM kernel adapter zero back over to this switch, we'll right click

12:50on that distributor switch.

12:51We'll go back to add and manage hosts, manage host networking, click on next.

12:56And then we're going to focus on ESXIC.

12:58So we'll select that one only, click on next.

13:01And then I'm not adding or removing any uplinks, we'll click on next.

13:04And then we have the VM kernel adapters.

13:06We want VMK0, which is currently on our test switch.

13:09We want to go ahead and assign that to the proper port group on this switch.

13:14And that's the one that ended in ESX management, which is this one right here.

13:18And we'll click on assign and then click on next, then we'll click on next and finish.

13:25And in the background, that VM kernel adapter has been moved off of the VDS test and the

13:32port group there over back to this distributor switch to this port group right there.

13:38And we can verify that by going back to the host view.

13:41And so there's A and B, and now that matches also C, which just had its VM kernel adapter

13:46zero migrated back.

13:48Now in this example we did, it didn't cause a problem or crash and we didn't lose connectivity.

13:53And here's why.

13:55It's because we did it in step by step.

13:57So here's the original switch A and here's our target switch.

14:00And then we have our port group here and our port group here and a link here and a link here.

14:04When people try to migrate, for example, the uplink from one switch to another, and at

14:10the same time do a migration of the VM kernel adapter, at the same time migrate the VMs,

14:15that's what causes the problems.

14:17So we want to make sure we have at least one uplink in place on both switches, create the

14:22switches first, then make sure we have the port group, then do the migrations one at

14:26a time.

14:27First the VM kernel adapter, if that's what we want to do, great, great, great, verify

14:30that works.

14:31And then migrate the VMs, verify that works.

14:33And then for this other switch, if we're no longer using it, we can go ahead and delete

14:37that switch and reclaim that uplink and put it over here on the second switch.

14:42So that way we can have fault tolerance and teaming here on switch B, but during the migration,

14:48we need to have at least one connection, one uplink in place.

14:51And again, as I mentioned, do it step-by-step.

14:54And by doing it step-by-step, we can avoid the common failures that often come up when

14:58we're doing migrations in the world of vSphere networking.

0:00In this video I'd like to chat with you about VLAN and trunking issues that can often come

0:05up. Because at the end of our VMNICs we are connected to either a layer 2 or a multi-layer

0:12switch. So let's imagine ESXIA has one of its VMNICs coming down to this physical switch

0:17right here to this port. And most of the time on these switches we're going to configure

0:21the switch port to be a trunking port. So if there's an 802.1Q tag that comes in,

0:26it'll respect that and then forward that to all the other ports associated with that

0:31same VLAN. So let's make that one of our options. Now on the virtual switch, let's say this is the

0:36virtual switch inside host A. If it has a port group that's set up for example for VLAN 23

0:42and there's a device connected to it, when the switch forwards it down to the actual physical

0:47switch, because it's in a port group associated with VLAN 23, it's going to have an 802.1Q tag

0:53of 23. And that's what the switch receives and then it can forward that based on all the other

0:58layer 2 ports in that same VLAN, including trunks that may go to other switches. Another reason we

1:03could have an incoming tag would be we have a port that is set up to support all VLANs. So let's

1:09imagine this port right here is set up to support all VLANs. For example, if we create a new port

1:15group, so we'll go to our switch, we'll click on distributed port group, click on new, give it a

1:20name. And then here, when it talks about the VLAN type, if we said VLAN trunking, effectively it's

1:260 through 4094, meaning this will take any incoming frames that include 802.1Q tags and allow those to

1:34be forwarded down the uplinks to the physical environment. So I'm going to cancel that. I don't

1:38need to create that. I just wanted to show it to you. So here that port group would be supporting

1:42trunking. And then here, if we had another port group that wasn't set up for any VLAN and we have

1:47a device connected to it, that'd be an example of traffic that would be coming into the switch

1:52with no tag. And in a Cisco environment, and that's a very popular vendor on the internet

1:57and in many businesses today, if there's no tag that comes in on the frame, this trunk port is

2:03going to assume it belongs to what's called the native VLAN. And most of the time in a Cisco

2:06environment, that is VLAN one. It doesn't have to be, but it usually is. So let me redraw those

2:12scenarios. So the first one, we have a port group for VLAN 23 and clients connected to that port

2:18group and the traffic that sent down the switch injects or adds VLAN 23 as it goes down to the

2:23trunk port on the switch. The next one is a port group with no VLAN associated with it. There's

2:27not going to be any 802.1Q tags associated with it. And the trunk port that receives that is going

2:32to assume that's on the native VLAN. Also in this scenario, if we had some device that was

2:38including tags, like a router or a firewall or a server, if it included tags, that would

2:43not be propagated. Those tags would not be propagated as it goes down to the physical

2:47switch. However, if we have another port group and it's set up as a trunking port group,

2:52but that means that we have a device that's including, for example, a server or a firewall

2:55or some other device that's injecting an 802.1Q tag on its own, a port group set up as a trunk

3:01will forward whatever tag was there and let it continue. So let's say here, forward tag. And

3:07if there's some device that's connected and it doesn't send a tag or no 802.1Q tag with it,

3:11you will simply forward that as an untagged frame. So we'll simply forward with or without a tag.

3:17Whatever came in is what gets to go forward it down to the switch. And that concept applies to

3:21port groups that are set for VMs. It's also applicable to port groups associated with VM

3:26kernel adapters. So we have a VM kernel adapter here as part of a port group associated with VLAN23.

3:31It's going to include and add an 802.1Q tag of 23. And so the key here in troubleshooting is to

3:37be aware of these possibilities and also what the receiving switch is going to do. It's also

3:43important to make sure that the receiving switch is correctly configured. So if this is a Cisco

3:48catalyst switch, we'd want to configure that to support 802.1Q trunking. And that way, as frames

3:53come in, it can handle and process those tags correctly. So for fun, let me share with you

3:58a single packet that I captured a few moments ago, just as a demonstration. So when this frame

4:04came into the switch, I set up a monitor session to forward the traffic so I could analyze it.

4:10And this is a Cisco catalyst switch where I did the capturing. So this is the source and

4:14destination layer two addresses. And then inside that Ethernet header, it specifies based on the

4:20Ether type, what the next thing is, the next protocol. And so 8100 in hex represents that

4:26the next payload is an 802.1Q header. I like to think of it like a layer two and a half because

4:32this is the layer two header, the Ethernet header. And this is also effectively layer two, adding the

4:37actual tag. So if we then look at the 802.1Q header here, it's showing right here, that's the

4:42key element ID 23. That's the VLAN identifier. And then it goes up to layer three, the networking

4:47information. This is from dot one back to 23.1.2.209. And the payload was an ICMP message.

4:55And it was a ping reply. But the key here is I want you to be aware that the tags are important

5:00to be aware of the 802.1Q tags, because if our switching environment isn't correctly set up to

5:05support it, or if we have our port groups incorrectly configured, for example, trying to

5:10support some device that's including tags like a firewall or something that has virtualized

5:15interfaces, if it's including tags, but it's on a port group assigned to VLAN 23, those individual

5:20tags are not going to be propagated through. But what will be added is the VLAN 23 tag

5:25from the switch, because that port group is including that VLAN tag. And that's why on my

5:30host ESXi5 host that's hosting our virtualized environment, I have a port group called trunk,

5:36and it is set up to go ahead and do forwarding. And that's what each of the network adapters on the

5:42ESX VMs are connected to. And my port group on ESXi5 is willing to forward those tags through

5:49down to the physical network. So here on ESXi5, I'm looking at the standard switch,

5:54and I've got a port group called trunk, and it's set up as a trunk. So the VLAN ID of 4095,

6:00what that means is it's allowing the incoming tags to be received, and it's going to simply

6:05propagate them as they go through. And off of the switch, this one uplink of VMNIC 0,

6:11which is a 10 gig port, it's going to a physical Cisco Catalyst switch, which is also in my rack.

6:17And that port on the Cisco Catalyst switch is a trunk port. So if frames come in and they're

6:22tagged, it knows how to deal with those. If frames come in and they're not tagged,

6:26it'll assume they belong to VLAN1, which in a Cisco environment is referred to as the

6:31native VLAN by default on a trunked interface. So as an example of creating a trunk port,

6:37let's go ahead and do it. So let's create a new port group. So on our distributed switch,

6:42we'll say new distributed port group. So we'll select that. We'll call it trunking and click on

6:47next. And then for the VLAN, we'll say VLAN trunking. So here's showing 0 through 4094,

6:54which effectively means it's allowing the 802.1Q tags to be passed through. So we can connect

7:00devices that are including tags to this new port group, and those tags will be forwarded down to

7:05the physical network over the uplinks. So the key takeaway here is that we need to have the

7:09vSphere side set up correctly. And also for the uplinks that go to the physical switches,

7:14we want the physical switches to be configured correctly also. So if we created, for example,

7:18VLAN20 here, and we start sending out VLAN20 tags, the switch does to the physical network.

7:24If that VLAN doesn't exist over on the physical network, or if the uplink were connected to a

7:29port that's not set up as a trunk, those are two great examples of why that traffic from anybody

7:34coming in off this port group would not work in the physical network. And that's because the

7:39physical site also has to be set up to make it work. So as an example of that, here's my top

7:44of rack switch. This is a Cisco Catalyst switch. And if you do a show interface trunk here and

7:49press enter. So I've got a lot of trunk ports, but this one right here, TE111, that's for 10 gig,

7:541 slash 1 slash 1, it's supporting all these VLANs. And this is the port that leads up to

7:59ESXi 5, and it's 10 gig port, which is then supporting the virtualized lab environment.

8:05So the end result here is that the physical environment needs to have the correct trunking

8:08set up, and it needs to have those VLANs. So for example, if VLAN20 didn't exist here on this

8:15switch, that'd be the end of the road for any frames that came in tagged as VLAN20. So now

8:20that we've taken a look at the world of VLAN tagging and trunking issues in the world of vSphere

8:25as they connect to the physical environment, in the next video, I'd like to turn our attention

8:29to MTU and Jumbo frame support. So I'll see you in the next video in just a moment.

0:00In this video, we're going to chat about MTU, Maximum Transmission Unit, and Jumbo Frame Support.

0:05And the general gist for a jumbo frame is something that is greater than the traditional 1500 byte

0:11maximum, which is the default standard for Ethernet networks. So let's first chat about

0:16why devices would even want to use an MTU for frames bigger than 1500 bytes. The answer is

0:22less overhead. So if we're sending, you know, five or six gigs of data, and we have to chop

0:27it up into teeny little 1500 byte compartments, that's a lot of individual packets. Each packet

0:32has their own overhead. So if we can send a larger frame of data, that means less individual frames

0:38because we're putting more in each one, and that equates to less overhead. So we're often going to

0:42use the jumbo frame, something larger than 1500, when we're accessing storage. So let's say there's

0:48our NFS, or if we're doing vMotion between, for example, ESX-A and ESX-B, we want to use jumbo

0:54frames to make it more efficient. So we've got a few challenges. Number one, we have physical

0:58switches here. Let's call this switch A and switch B and switch C that's carrying all the traffic,

1:04or it could be one switch. Hopefully we have a couple for some fault tolerance. And then they've

1:08got trunks between them as well, that can forward. And that's also where this NFS is plugged into

1:14one of our switches. And so let's just take a look at the path from host A all the way to,

1:20how about this NFS device? Let's say that's at 192.168.1.34, which it is actually. So for

1:29host A to get to the storage, I'll go ahead and label it down here, NFS. That traffic is logically

1:34going through virtualized switch, and then it would go out the uplink and hit the physical switch.

1:41And that physical switch may have to go to another physical switch over a trunk. So these are all

1:45ports. And let's say the NFS is connected on an access port right there. And it's addresses dot

1:5134. So as far as IP addressing goes, the host to have an IP address on the 192.168.1 network. So

1:58logically they're on the same layer three IP network, but we're going through multiple devices

2:03to get there. So if we want to use a frame size of let's say 8,000 bytes for the payload,

2:09we need to make sure that this BDS supports it, and this switch supports it, and this switch

2:15supports it, and also this NFS supports it, the appliance itself. And if we have any of these

2:21devices that are not set up to support the bigger frames, that's the end of the route.

2:26It's not going to work when they start trying to use frames that are greater than 1500. So let

2:31me walk you through an example of where we would go to check for these values. Let's first start

2:36at the distributed switch here that host A is using. So here's our distributed switch. So at

2:41the switch itself, we could right click on that switch, go to settings for the distributed switch,

2:45click on edit settings. And here under advanced, it's going to show us the MTU. So that's where

2:51we configure the MTU at the switch. And then the distributed port groups, they're going to inherit

2:56that by default. So it's not set to something bigger than the default. You're not going to

3:01be able to send frames bigger than the default. And then once we connect, let's go ahead and go

3:05back to configure and topology. Once we connect down to the physical environment, we need to make

3:09sure our switches are supporting the bigger frames as well. So in my environment, everything leads

3:14down to this physical switch. So here, if we do a show system MTU and press enter here, it's showing

3:20us that we are supporting larger than normal frame sizes. So the jumbo MTU size is 9198.

3:27The virtual switch is capping at 9,000 because that's how it's currently configured. But at the

3:32end of the day, it means that the switch, this physical switch is not getting in the way of being

3:36able to forward. Here at the interface is a Synology NFS appliance or Synology appliance configured to

3:42support NFS. We've got a control panel, and then we've got a network and network interface. So here's

3:48the interface we're connecting on. If we double click that to open it up, here's showing that jumbo

3:53frames are enabled and the MTU is 9,000. Meaning this appliance, the NFS device can also support

3:59that larger frame size. Now, one thing I want to share with you is this, that an IP network,

4:04the IP protocol, IP version four is built to be fragmented. So if we do try to send a larger frame,

4:11there's a good possibility that intermediate devices may fragment that for us, or the two

4:16devices realizing they can't use a bigger frame may negotiate down to a smaller size. However,

4:23that's going to deeply impact our throughput. So if we want to use jumbo frames to reduce overhead

4:28and have better throughput, again, just make sure that the virtual switches support it and the

4:33physical switches support it and your end devices support it. And there's one more thing I want to

4:37chat with you about that's super important to understand. And that is the concept of fragmentation.

4:41Because if we do a simple test, this is my workstation computer here, and we do a ping,

4:46and I'm going to do a dash L for length. And let's go ahead and send a payload of how about,

4:51how about 20,000. Fantastic. And we'll go to my default gateway at 192.168.1.1, press enter. Yeah,

5:00working. Look at that. So we're seeing this huge payload. How is that possible? My local computer

5:06is fragmenting that traffic. So to confirm that, let me bring up a protocol analyzer for you.

5:12So here's the protocol analyzer is capturing everything on my local network interface card.

5:16And let me do a filter display filter for just ICMP. And let's do that ping again. So back at

5:22the command prompt, we'll send that 20,000 byte payload, press enter. And away it goes. All right,

5:30and I will go ahead and stop the capture. And let's just grab one of those packets. So I currently have

5:34a filter just for ICMP. But what it's not showing is for all the fragments. So let's take this

5:39request right there, packet 516, and open it up. And I'm going to just grab the identifier for this

5:46IP packet. So each packet has a fairly unique identifier, there's some crossover occasionally,

5:51but I'm going to take it and drag it up as a display filter and say selected. And that's

5:55why it's only going to show us packets with that exact ID. So all these packets here 509,

6:02all the way through 516. That's the effect of saying that ping request. And this is just one

6:07of those ping requests. All of these others are fragments. So the workstation, my computer

6:12chopped it all up into sizes. And if we look at the size, let me go ahead and add a column for that.

6:19So let me bring in the length column right there. So there's the length. And you'll notice there's

6:241514. So at 1500 bytes is the actual maximum MTU. But then there's some overhead due to the

6:31headers involved at layer two and layer three. But I wanted to point out that we aren't really

6:36sending a jumbo frame of 20,000 bytes, we are doing some fragmentation. So what does that mean

6:41for us here in vSphere? Well, we definitely want to verify that jumbo frames are being supported.

6:46So we can go to the CLI. In fact, right here, if you go to a host, and we go to configure,

6:51and then we go to the VM kernel adapters right there. And then we drill into one of them. So

6:56for example, VMK zero has an MTU of 1500. But VMK one has an MTU of 9000. VMK two has a MTU of 9000.

7:05And VMK three has an MTU of 9000. And they got those based on where those VM kernel adapters

7:11were originally created, what switches they were connected to. And that's what allows these VM

7:15kernel adapters to send frames up to that size. So if we were here at the command line to host a,

7:20and we shoot this command ESX CLI network IP interface list and press enter, that's going to

7:25show us all of our VM kernel adapters, along with what I look at here is the MTU. So there's VMK one,

7:32there's VMK three's MTU, there's VMK two's MTU. And at the top is VMK zero, which has an MTU of

7:401500. Another way of seeing those MTUs is with an older command, but it still works ESX config dash

7:47VMK nick dash L, press enter. And that shows us the VM kernel adapters, the IP addresses. And if

7:54we scroll over here far enough, it's also going to show us the MTU. So they're the MTU and also

8:00the protocol stack they belong to. So all of these, except for the vMotion one are part of

8:05the default TCP stack. So I love that command. It's a great one. So there's our VM kernel

8:10adapters. There's the actual IP addresses associated with them. So let's do some testing.

8:14So if we do a VMK ping NFS, NFS is simply a DNS resolve name that goes to the NAS appliance that

8:21we've been using the NFS device at 192.168.1.34. And so we can verify that real quick with NS look

8:30and press enter. And sure enough, it maps to that IP address. So if we do a ping to NFS,

8:35that works. But the question is, which interface did they use to source that? Did it use VMK zero

8:41or did it use VMK one? Because they are both on that same subnet from a routing perspective.

8:46They're both directly connected. So that's a good question to ask. And that's why we could

8:51also use a tool like VMK ping, where we can control which interface is used. So if we type

8:57in VMK ping and then dash capital I, in fact, we just do a press enter there, and it'll show us

9:01all of the options, including an option for which interface to use, whether or not to allow

9:07fragmentation or not. And here we can also specify the stack if we're not using the default TCP stack,

9:14for example, like vMotion. So if we do a VMK ping dash I, and let me also bring up the list of

9:20interfaces again. So there we go. So we did a VMK dash I, we said we want to source it from

9:25interface VMK zero, and then ping to NFS. That's working. Let's also source it from VMNIC one. So

9:32I'll hit the up arrow key and change that zero to a one. And that's also working. So then let's go

9:38ahead and try a bigger frame size. So the first one again, VMK ping, interface VMK zero, and let's go

9:44ahead and do a dash S for size. And let's do 5,000 bytes and press enter. And that's working.

9:51And let's go ahead and do it with VMK one. And that's working. But the challenge here is that

9:58if we go back and look at those adapters, the VMK zero has an MTU of 1500, and we just sent a larger

10:05frame. So what really happened to the background is that when we sent that from VMK zero, who has

10:10an MTU of 1500, just like my PC did earlier, the host actually did the fragmentation and sent many

10:17little packets to go ahead and get the effective full amount sent. So it's not actually sending

10:23jumbo frames when we source it here from VMK zero because of the MTU of 15. But it kind of looks

10:30like it made it. So one additional option I like to include would be the option that says do not

10:36fragment. So if we scroll up here, that is this option right here. So if we include that option,

10:41it'll tell the host, hey, don't fragment it. And it's not just for the host. So it's going to be

10:46included in the IP header. So when a device in the path sees it, for example, a router that may have

10:51to fragment it, it won't fragment it. It'll simply stop and say, you know what, can't do it. And it

10:55might send an ICMP message back saying, hey, I killed the packet. But the point here is that

11:00the dash D says don't fragment. That's how we can really verify that we have the MTU size that we

11:05think we do. So let's do the test here again from VMK zero, but this time we'll add the flag

11:10of dash D for don't fragment. And we'll press enter. And now we have a message. This is basically

11:16the host itself saying, oh, can't do it. I was going to fragment it, but I can't because you

11:20said don't fragment. However, we did that same command, but we sourced it from VMK one, which

11:26has an MTU of 9,000. It won't be a problem. So let's go ahead and put VMK one there. The do not

11:33fragment bit is set and the dash S of 5,000 says, please use the payload of 5,000 to that target,

11:39press enter. And that one works no problem because the host can support it. The switching in the

11:44middle can support it. And that NFS appliance has a jumbo size also set up in the configuration.

11:50So it also supports it. So I wanted to point that out also as a confirming method about whether or

11:57not we can actually send jumbo frames, or does it just look like we're sending jumbo frames?

12:02And that same trick works right here as well. So if we did a ping, this ping earlier worked

12:07because my PC was fragmenting it. But if we add the dash F, which on a Windows computer is how

12:13you tell ping not to fragment. So we'll add a dash F that adds that do not fragment bit to on,

12:18we'll press enter. And now the local computer saying, oh, I can't fragment that packet. So

12:23I'm not going to go ahead and send it. So packet needs to be fragmented, but it can't be done.

12:27So we set a smaller amount. So let's go ahead and send a normal size packet. How about something

12:32like 1,400? And then we also capture this as well. So I've got a new capture that is running.

12:39Let's go ahead and send that. Let's press enter there. So it's working, but I also want to show

12:43you what that looks like in the packet capture. So let's go ahead and let me go ahead and stop

12:48the capture and do a filter for ICMP first. That'll narrow the field. And these are my

12:53request and replies right here and open up that packet. So here in the layer three header,

12:57under the flags for this IP packet, there is a one set or it's on for the don't fragment bit.

13:04So if there's some device who needs or wants to normally do fragmentation, like a router

13:08that has a smaller MTU on one of his paths, when it sees that bit, it'll say, oh, I can't do it.

13:13And as a result, it won't fragment it. So if the router is going to drop the packet,

13:18because it can't fragment it, it needed to, it'll send an ICMP message back to the source

13:22indicating what happened. Here's the original packet. It was too big. The don't fragment bit

13:28was set. And so as a result, I couldn't forward it. So to get around all that, the best option

13:33is to use that dash F here on a Windows computer or in VMK ping the dash D so that we are telling

13:41the device itself and anybody in the routed path to not fragment. And that way we can verify

13:47we have jumbo frame support from end to end.

0:01For our customer scenario, let's imagine that there's a new NAS that they're going to place online.

0:05They've got DNS for it. It's NAS-33.

0:09It resolves correctly here from ESX-A to the IP address of 192.168.1.33.

0:16And there's even a nice little VMK ping sourcing it from interface VMK1.

0:22That seems to be successful based on the replies. Fantastic.

0:25So what I'd like you to do is take a look at this screen.

0:28And then let's take a look at this next screen, which is showing the actual VM kernel adapters 0, 1, 3, and 2 here on host A.

0:36And it's also showing the MTUs associated with them.

0:39And then if we take a look at the next screen, here's where the problems start to rear their ugly heads.

0:44When a VMK ping is done, sourced from interface VMK0 with a size of 5000 bytes going to that same target, it works.

0:53However, when they do VMK ping again and they source it from VMK1 with that same size of 5000, it fails.

1:00I'd like you to take a look at these three screenshots that I've included as part of the skill and determine what do you think the problem is?

1:09Where does the problem lie based on the results of these three screenshots?

1:14So go ahead and pause me now. Give that some thought.

1:17Feel free to take your time with it. And when you're ready, click on resume and we'll walk through this together.

1:24So let's go ahead and first of all, let's replicate what they experienced in the screenshots.

1:29So we'll go ahead and this VMK ping from interface 0. In fact, let's take a look at the interfaces.

1:35So one way of doing that is the ESXCFG-VMKNIC-L, press enter.

1:40And that has the VM kernel adapters, IPv4 addresses, and also the MTUs.

1:46VMK0 has an MTU of 1500 and the other three have an MTU of 9000.

1:52So if we ping that NAS server that we want to bring online and use with a source address of VMK0 and we'll press enter and that works.

2:01That means the host is actually doing the segmentation and sending smaller than the 5000 bytes.

2:07So it's sending overall 5000, but it's doing it in multiple packets due to it doing fragmentation.

2:13So we do that same command from VMK1 and press enter and VMK1 has an MTU of 9000 and press enter.

2:21That is failing. So a couple of things that we might want to do, I'm going to do a control C.

2:27So I have to wait for that. Let's go ahead and do that first ping again.

2:30But let's add the D flag, which says don't fragment. It says the don't fragment bit.

2:36And also the host itself is going to say, well, I'm also not going to fragment this.

2:40So press enter. So the first one worked, the pings worked without the do not fragment bit set.

2:45And that gives the false impression that we're supporting large frames when in reality it's chunking them.

2:51And so the other question is, what about sending it from VMK1?

2:56And if that fails and let's also include the do not fragment there.

3:00And that way we can see whether or not the host itself is trying to fragment.

3:04So, oh, I put the D there in front of the S. Size 5000 and then a D, fair enough.

3:11And then press enter. So here it's actually setting up that payload with the do not fragment.

3:16So it's not the problem here, which leads me to believe it's either with the switch, the physical switch,

3:22or that destination that can't support that size.

3:26However, if they've been using jumbo frames with other NAS appliances, for example,

3:30the other NAS appliance that we used in this lab was at 192.168.1.34.

3:36That does support, that end device does support jumbo frames.

3:39So we press enter here. If that works, it means we have end-to-end between the ESX host

3:44and VM kernel adapter number one and that NAS device.

3:48And it also implies that the other device at NAS 33 is probably not configured to support those bigger frames.

3:57So it's a great idea in troubleshooting to always verify with the tools you have

4:02that what you think is happening is actually happening.

4:04So if we went to our top of rack switch here and we did a ping over to 192.168.1.33.

4:13So that's the one way I think doesn't support the larger frame size.

4:16Let's also go ahead and go to 34, which I think it does support the larger frame size.

4:20Then we can go ahead and use the size command here and send a larger frame.

4:24So let's go ahead and do a size 8000 and that's working to the .34.

4:29Let's go ahead and try that to .33 and press enter.

4:32Yeah, sure enough. So that confirms for us that is the target, the final destination at .33

4:39that doesn't support the larger frame sizes.

4:41And that's why also it's a good idea to test it from a couple angles again,

4:45just to verify that what we think is happening, for example, jumbo frames is actually happening or it's not.

4:51And in this case, for .34, no problem. But for .33, it's not supporting jumbo frames.

4:57So here is that NAS device. So at the .34 address, it's supporting jumbo frames.

5:02And for LAN1, it is not. The MTU is 1500, just to confirm that, yeah,

5:07it's the end device in this example that isn't supporting the jumbo frames.

5:11So thanks for joining me in this journey through the wonderful world of network troubleshooting in vSphere.

5:18And I look forward to seeing you, my friend, in another set of videos very, very soon.

5:22Until then, I hope this has been informative and I'd like to thank you for viewing.