FGT Troubleshooting Methodologies

0:00Hello and welcome. My name is Keith Barker. And of course, we would love it if everything just

0:05worked as expected, as it's supposed to, every single time. But the reality is,

0:10in the real world that we live in, that's not always the case. So in this set of videos,

0:14I'd like to walk you through some troubleshooting methodologies as they apply to networks in general

0:18and also specifically to 48 firewalls. So I'm glad that you've joined me for this journey,

0:23and I'll see you, my friend, in the very next video.

0:00Regarding troubleshooting methodologies, before we start troubleshooting a network,

0:05if we have one, and it's not always possible because we get called in as a contractor or

0:08whatever, but if possible, we'd want to have a baseline of what normal looks like. I'll put

0:14that in quotes. For example, this user here on the network can get out to the public internet

0:19or they can communicate with servers over at the branch side or vice versa,

0:23which applications are they using, what's the normal latency involved with those applications,

0:28and so forth. So there's a lot of tools when the network is functioning that we can use to get a

0:32baseline. Things like using NetFlow to identify what type of traffic is flowing over our networks.

0:37We can have packet captures that identify the flows of traffic regarding a certain session

0:42that we have as reference. We may have baselines that include some timestamps regarding how long

0:46things are normally taking. We'd also want to have some documentation regarding IP addressing

0:50schemes and so forth and what's being used, and also documentation on how the firewall,

0:55the switches, and the routers are all configured as well. In addition to these tools here, we can

0:59also use tools at the FortiGates to help identify and build a baseline of what quote unquote normal

1:04looks like. And again, this is ideal before we have a problem or issue on the network.

1:08And in the old days, I mean, going back 20 or 30 years, we were really big on memorizing a lot of

1:14commands, but there's so much documentation available. It really makes sense to be aware

1:19of the big picture, how things are working, and then we can drill down and get the commands that

1:22we need based on the situation. So this is the document, the FortiOS 765 Admin Guide. And as time

1:29marches forward, it'll probably be updated for newer versions. And this section is called Establish

1:33a Baseline. And in this version of the document, it starts here at page 4575. So they have some

1:39commands here that we could use on the FortiGate to help get information from that FortiGate

1:43firewall, including details such as the version in use, which would be good as part of our

1:47documentation, how much memory it has, because as time marches forward, we may have some FortiGate

1:51firewalls, like I have a few in the lab here, that only have around two gigs of RAM. And as a

1:56result, they cannot support proxy-based policies. So if we were going to do an upgrade on the network,

2:01for example, we'd want to make sure that our hardware supports it. Other commands here can

2:04reveal to us the routing table on the firewall and other details as well. So as far as the baseline

2:09is concerned, I would encourage you with whatever documentation is available, have that ready and

2:14available as we begin our troubleshooting process. So let me clean up this ink a little bit and let's

2:19talk about some troubleshooting methodologies. And even though we are talking about FortiGates here,

2:23this is pretty much applicable to pretty much any vendor as far as the methodologies here.

2:28And when we're doing troubleshooting, we have documentation also, we're going to the topology,

2:32that's super helpful, especially for me anyway, as I look at a network, if I have a topology,

2:36I first of all, want to verify that it's accurate. We can do some basic testing and then use that as

2:41we do the troubleshooting. So in this topology, we have a couple of computers here representing

2:45client devices, a Linux device, a Windows device. We have a VLAN10, I presume here from this logical

2:50interface on the firewall using the 10.10.0 subnet. Then I've got firewall2 connected to

2:55firewall1 over the 10.12 network. Looks like they're using VLAN interfaces here, VLAN12 there

3:01and VLAN12 there. So based on this diagram, VLAN12 is supporting the IP addressing space at layer 3

3:07of 10.12.0. And then firewall1 is connected to the internet via, looks like a couple of service

3:12providers here and here. So having a topology diagram of some type is going to be helpful

3:17in troubleshooting. I remember many, many years ago, I was in site at a customer location and

3:23they had this great big, you know, poster on the wall that was all plotted out. And back in those

3:27days it was done with a plotter. Anyway, it was their network topology. And I thought, wow, this

3:32is really cool. So I started looking through and doing a trace and it wasn't accurate. It was made

3:36like two or three years ago and they were so proud of it, they left it on the wall, but it wasn't

3:40accurate. So another concept regarding troubleshooting is, you know, believe people,

3:46but also verify every step of the way. As far as methodologies go, I like to think about

3:50troubleshooting as a circle rather than a checklist like this, like do this, do this, this, this. I

3:57like to think of it like a circle and not a linear checklist. Also when we do our troubleshooting,

4:02it should be hypothesis-based. For example, instead of just reacting like, oh no, reboot it.

4:08We want to, you know, take some time, even before we start looking at the configs and the details

4:12to take some time and have some theories about what is possibly causing the problem. What is

4:17wrong? And if we think it's a firewall, for example, this one here, or this one here,

4:21or one of these over here, an acronym that can be very helpful in remembering what to look at

4:25is PPP. Not the point to point protocol, which also is the acronym of PPP, but PPP, the first P

4:31being policy. Is there some type of policy on the firewall that's preventing that traffic

4:35from the user from going through the firewall? Or is it the path? Maybe the routing table is

4:39mucked up. So if the router doesn't believe that it should forward traffic in a certain direction,

4:44it won't even try considering the policies associated with those interfaces involved,

4:49or is it having a problem with processing? Now, another acronym that I like to use when

4:53I'm troubleshooting firewalls, and this is pretty much for any vendor, is RAP. Like,

4:58don't give those firewalls a bad rap. And the RAP reminds me, when I think of it,

5:02the R is for routing, because if the routing isn't working, the firewall's not going to be

5:05forwarding traffic. The A is for address translation, or as his friends call it,

5:10NAT or a subset of PAT. And the P is for permissions. So the routing here is about the

5:14path. The permissions is about the policy. And because it comes up so darn much, I also add the

5:19separate element here just for the address translation. So if we have clients that can't

5:23get traffic through the firewall, assuming it's even getting to the firewall, I use RAP almost

5:28every single time. And most of the time, it's one of those elements that's causing the problem.

5:33So let me clear up some of this ink, and let's chat about a methodology that we can use

5:37consistently that can help us identify the problem and solve it. So starting off,

5:42we want to define the problem. And I'm going to add on here precisely. Another way of thinking

5:46about that is identifying the symptom that is happening. So if a user, for example, here

5:52or here, can't get to this or that or the other, we want to make sure we can replicate that

5:57whenever possible to identify exactly what isn't working. The other good question also to ask is,

6:02when's the last time this did work? I remember many years ago working with a president's office,

6:07not the president of the country, but president of a company. And I was working with what I thought

6:12was his full-time secretary, and she couldn't do X, Y, or Z. I think it was printing at the time.

6:17She couldn't print. And so I spent a lot of time, unnecessary time, with trying to troubleshoot

6:23printing. When it finally occurred to me, she was a temp. She was brought in as a temp. She hadn't

6:28logged on. She hadn't got the authentication she needed to get access to the resources.

6:32And as a result, she had never printed before. So another part of that would be, when's the last

6:37time this did work? Identify the problem and also identify or determine when's the last time this

6:43worked. And whenever we can, we want to be able to replicate on demand that symptom or that problem.

6:48All right. Number two is to classify the failure type. For example, is it just one application

6:54that these users can't get to? Or is it just one computer on that subnet? All the other computers

6:59have no problem whatsoever. What is the domain, the fault domain, where it's happening? Also,

7:04in troubleshooting networks, I like to go back to the TCP protocol stack, which we know and love

7:09as these layers right here. That's how we refer to it these days. So if we're at a computer and

7:14this computer can't get to a certain website or a certain internal server, some other resource,

7:19we can start from a bottom-up approach and say, okay, great. Can we ping our default gateway?

7:23Or can we ping, for example, another device beyond that default gateway? So over here,

7:27the default gateway is this interface address here on firewall two, which is 10.10.0.52. Or

7:33can this user ping by IP address some resource out on the public internet? And if they can,

7:38that means that we have IP connectivity, at least at some level, out to the public internet. So

7:43we can rule out the data link layer and the network layer because we have reachability,

7:47but perhaps we have some name resolution that's happening or not happening as it needs to,

7:51or perhaps we have a DNS cache that's been poisoned. So again, we want to classify the

7:54failure type, the scope of it, identify the fault domain. Is it one PC, one application,

8:00or is it complete blackout for the entire subnet? And that will help us in refining

8:05our troubleshooting. Now, step three would be to prove where the traffic is flowing or where it's

8:10not flowing. And here's an example. Let's imagine that a user says, I can't get to the internet,

8:14which is a pretty wide statement. So here's an example. Let's imagine this user right here

8:19calls the service desk and says, I can't get to the internet, or I can't get to this server,

8:22or I can't get to something else. What we could do is just verify by using some basic connectivity

8:27tests like trace and ping and so forth, that we have connectivity through the firewall.

8:32So if this computer, for example, has a messed up internal routing table,

8:37maybe it's not even trying to forward the traffic to its default gateway as it tries to get that

8:41traffic off the local subnet. And we can test that in many ways. We can look at logs here

8:45at the firewall regarding traffic flowing through, making sure we're doing logging,

8:49and then look at those logs. Or if we're using 4D Analyzer or 4D Manager, we could do it from

8:53a central management position. We could also do captures on the local computer, or if we have a

8:58tap set up, or we're capturing our span port set up, or we're capturing network traffic and analyzing

9:03it there, we could also verify whether or not that traffic is actually going from the PC over to

9:08the firewall. But at the end of the day, we want to prove whether or not that traffic is flowing or

9:13trying to flow to or through this firewall. Or if the client is trying to get to the internet and

9:17it's not working, we'd also want to verify traffic going to and through Firewall 1 here in this

9:22topology. Another important step is to validate with data what our assumptions are. So if we're

9:27assuming, oh, it's a problem here at the firewall, Firewall 2 or Firewall 1, it's a problem here at

9:31the PC, we want to get enough data to validate that that's actually what's happening. So let's

9:36identify that we think it's a misconfigured firewall rule at Firewall 2. And that's what

9:42we think the problem is. Now, before we make changes in a company, we want to have change

9:47control involved to make sure we're just not making nilly-willy changes on our infrastructure

9:52devices. So the next step is once we've identified we think it's a problem here, we also might want

9:57to identify, okay, when did this change? And it was working before, now it's not, what happened?

10:02So all that's a good idea as well. But when we make a change, again, with approval, we'd want

10:07to change one variable at a time or one component at a time. And I'm going to label that as part,

10:12just one part. So if we think it's a firewall rule, first of all, get the change control approval,

10:18make that change, do the testing again. And then if that didn't work, what we want to do is revert

10:23back. Because what we don't want to do is we don't want to leave droppings regarding all the

10:28changes that we're making. Because if we're making small changes and we leave those in place,

10:32it's very likely that that's going to cause other problems in the near future, or if not

10:37immediately. So our next logical step, if we change one part, we want to verify whether or not that

10:42corrected the problem. And then we want to do a couple of things. Number one, we want to go ahead

10:46and stop. Don't continue on because network's like, oh, I'm in here, I might as well tweak this

10:51or tweak that. Stop, and then also update documentation. And unless it's a physical

10:57problem or some component went bad, like a switch or a cable came loose or some other issue,

11:03if it's not physical in nature, most of the time, if it stopped working, it's due to a change.

11:08Something happened, like a software update that wasn't fully tested or somebody added some code.

11:14And another piece here that I want to point out is that in a production environment,

11:18the key word is production and security is really important. So as we're making changes,

11:23here's what we don't want to do. We don't want to add a new rule at the top of our policy that says,

11:28hey, permit everything and no inspection. Because if that solves the problem, it's also opening up

11:34a hole in the wall, big enough to drive a truck through where we've now just nixed our security.

11:40So I'm going to label it as no blanket permit. Now we may have this one device here with a specific

11:44IP address. And for testing purposes, maybe we make a specific rule. We add it up in the rule

11:49base high enough so it hits first, just for that one device's IP address, just for testing. Because

11:55unfortunately, if we start putting in blanket permits for this and that and the other,

11:59and things start working, once again, because we're in production, it's really tough to go back

12:04and fine tune that without causing other problems. And although it may be necessary to jump into the

12:10CLI and do a bunch of diagnose commands with debug, that's usually not the very first place

12:16we're going to go. So I'm going to have some cautions over here. I'm going to say debug,

12:20and on a four gate, it's diagnosed, debug, et cetera. We want to not do that as our first

12:24approach, our first step. We also want to make sure we're treating our stateful firewalls as

12:29stateful firewalls and not just as a typical router. Because the firewalls, if traffic is

12:33going through the firewall to go out to the internet, for example, the reply path is dynamically

12:39allowed back based on the initial flow of traffic being allowed. And also, as we're looking at

12:43diagnosed debug, a lot of those flows are going to be offloaded to processors on the 40 gates.

12:50So they may not show up in the output of a debug. Another caution I want to remind us of is making

12:55changes in general in a production network. Because sometimes we might have a lot of pressure

13:00right down on us to make it work. And so if we're making changes, including blanket permits and

13:04things like that, and they start working, even though we have resolved the quote unquote problem

13:09we were having, we've also just now opened ourselves up to a bunch of security risk.

13:14Because in my experience, most outages and issues that happen on computer networks,

13:18they are misunderstandings, not bugs. Another important item to be aware of regarding changes,

13:24if somebody asks us, and we're going to have to with our change control, if somebody asks us,

13:28okay, why is this change going to solve that problem? We want to be able to explain that.

13:33So if we haven't thought through it enough, or been able to verify that this is what we need to

13:37do, and this is why we want to spend a little bit more time in rethinking it. So here's what we get

13:41to do in this set of videos. We're going to take this methodology and we'll go through some examples

13:45of it using this topology. And the first step is of course, to define and confirm what that problem

13:51is or what the problem is on the network. So we'll do that in the next video. And I'll see you there

13:56in just a moment.

0:01So one of our first steps is to define the problem.

0:03And I would also add to that, define the problem before we ever look at or touch at the network devices.

0:10So if we're working with a customer, for example, either here or here,

0:14and we have access at the CLI and or the GUI to the FortiGates

0:17or whatever management tool we're using to manage those FortiGates,

0:19we want to define the problem before we start looking at the configs.

0:23Because if we have an ambiguous or missing problem definition,

0:27we're going to waste a lot of time in troubleshooting.

0:29So here are some things we'd want to get based on that fact-finding.

0:33We'd want to learn what exactly is failing.

0:35And that may also require, as we interact with users and other people,

0:39bringing it down to the level that's appropriate for our audience.

0:42For example, if we have a secretary or somebody that's not familiar with, you know,

0:46DNS and all those other protocols that are being used, as they go out to the Internet,

0:49they just may say, well, the network is down or the Internet's down or I can't get to the Internet.

0:54So we might want to clarify, you know, what they're attempting to do.

0:58And again, also asking, when's the last time that this worked?

1:01Another great question to clarify is, who is this affecting?

1:05So if there's a couple people on the team and this is person one here and this is person two here,

1:09we want to confirm is, you know, all the people in that same location on the same subnet, for example,

1:14are they all having the same problem?

1:16And then another great question is to ask, how often is it having a problem?

1:20Is it always broken or is it intermittent?

1:23And if they say it's intermittent, that's a little tougher

1:25because it's harder to verify consistently the problem.

1:28But if it's always, you know, the problem is always there, that's great news because we can clarify that,

1:34confirm that, recreate the symptom, and then start our troubleshooting.

1:37So as we're defining the problem, let me add some cautions as well.

1:40Number one, if we have a false positive where we, you know, think something's an issue,

1:45but it really isn't, that's going to take our troubleshooting in a misdirection.

1:49For example, maybe we're not allowing, based on policy, ping requests from clients out to the public internet.

1:56And as a result, they try a ping or we try a ping from here and it's not successful

2:00and I think, oh, there's a problem with IP connectivity when it really could just be a policy.

2:04Or we could be doing a trace from these devices.

2:08And that also can cause problems because trace is not done the same.

2:11Another challenge is the scope.

2:13Now, if we don't correctly identify the scope regarding who it's affecting,

2:16once again, we could be troubleshooting in the wrong direction

2:19because if the entire subnet is having a problem,

2:21it's very likely going to be something that connects those devices to the rest of the network.

2:25It could be a switch here or it could be the firewall here or the firewall here

2:28if they're all trying to get up to the internet and they can't.

2:30And that's a much different situation than just one device that can't access some resource.

2:35So for our scenario, let's imagine that we have these two devices here.

2:38I don't know their addresses. We'll look at those in just a moment.

2:41They're both on the 10.10.0 subnet.

2:43I think this guy's at .102 and the Linux device is at 1.0.1.

2:47We can verify that here in a moment.

2:49And let's imagine we get a call from the Windows user

2:52who indicated that they can't get to any of their traditional websites on the internet.

2:56Now, we say traditional. It's like, what does that mean?

2:58You might want to narrow that scope and say, okay, which website, as an example, can you not get to?

3:04And then we want to replicate that and verify that.

3:06So with this topology in mind, let's go to this PC right here.

3:10And let's just validate the problem that the user is talking about.

3:13So here's this Windows client.

3:15And let's go and bring up a command prompt.

3:17And let's do an ipconfig.

3:20So there's this IP address, 10.10.0.102 with a 24-bit mask, default gateway is 10.10.0.52.

3:27And that's the network interface it's using as part of the topology.

3:31All right, let me minimize that.

3:32And let's open up a browser.

3:34So here at the browser, it looks like it's trying to load a whole bunch of different tabs.

3:39Let's go ahead and close some of those.

3:41And let's just go to fast.com as an example.

3:44As I mentioned, that's a critical app.

3:46And sure enough, loading, loading, loading.

3:48And it is not working.

3:50All right.

3:51So we've now defined that this is one of the issues in measurable terms that the client is experiencing.

3:56They cannot get to the Internet.

3:58Let me go ahead and close that.

3:59So regarding exactly what's failing, we confirmed that it's Internet access to fast.com.

4:05And we have other websites that also are not working.

4:07But it would be nice to just do one at a time.

4:09By focusing on just one site at a time, we're keeping our symptoms identified and narrow in scope.

4:15So we can solve for that.

4:16And then we can take a look at the other websites if there's other websites as well.

4:19As far as who is affecting, let's see whether or not it's affecting this Linux device, which is a .101.

4:25So here is that Linux device.

4:26Let's open up a command prompt.

4:28And let's do an ifconfig on this Linux device.

4:31And its network interface we're using for the lab is this one right here with the IP address of 10.10.0.101.

4:37If we do a route, there we go.

4:39So its default gateway is 10.10.0.52.

4:42That looks good.

4:43But let's also just bring up a browser and verify whether or not this client can get out to fast.com.

4:49Again, the same kind of a test, same website.

4:51So we'll go to fast.com.

4:53Press Enter.

4:55And sure enough, that is working.

4:57So fast.com is provided by Netflix to simulate Netflix traffic.

5:01So that means that this client, this device, is able to access this website while the other one isn't.

5:07So as far as who is this affecting, it's just the client .102 here on the 10.10.0 subnet.

5:14And as far as always or intermittent, I can just tell you it's always.

5:19That's not because I put the bug in there or put the problem in there, or maybe it is.

5:23Anyway, it's always that case.

5:24This client just cannot get to any websites, including fast.com, while the Linux device has no problem.

5:30So the benefits of defining the problem and doing it exactly, as far as what's failing and the scope,

5:36is going to help us as we continue moving forward with the troubleshooting of this problem.

5:41So now that we have clearly defined these symptoms, the problem that the customer is having,

5:45we've also identified the scope.

5:47It's just this one PC.

5:48In the next video, we'll take the next logical step.

5:51And I'll see you in the next video for that next step.

0:00So now that we've identified the problem, that is this device right here on the 10.10.0 network at

0:05dot one zero two, and this user cannot access fast.com. We're using that as an example of a

0:12resource that normally is accessible and used by this user that they can't get to. And in conjunction

0:17with that, we'd also want to ask the user when's the last time it did work, which is a really

0:21important question to ask. And now we're going to do the next step is to classify the failure.

0:25So here are some of the questions we want to confirm as we do our troubleshooting regarding

0:30the traffic from this user that's trying to go to fast.com. In this example, is that traffic

0:36reaching the FortiGate? Because if for whatever reason that traffic never leaves the network

0:40adapter on this computer and makes it over to the FortiGate, it's not a firewall problem.

0:44Number two, if it is reaching the FortiGate, is that traffic being routed correctly? And I'm

0:50going to put that in as routing tables on the FortiGate because if the packet shows up at the

0:55firewall and the firewall says, oh, based on this destination address, I'm going to route it this

0:59way or that way or the other way. If it's not going in this direction, let me go ahead and put

1:03this in green. If it's not going this direction here, leading over to firewall one, that traffic

1:08is not going to make it out to the public internet. So this firewall didn't have, for example, a

1:12default route or didn't have a route out to the public internet using firewall one as the next

1:16hop here, that would be a problem. And then the third question, assuming it is showing up at the

1:20firewall and we have routing in place, is the firewall denying it or implicitly denying it and

1:26not forwarding that traffic? So I'm going to write that out as being denied at the firewall. And that

1:31could be based on any of the policies or security profiles associated with policies that either

1:36firewall two or firewall one have in place. So let's go ahead and answer these questions. Now,

1:41if we have access to this device, or if the user is willing to work with us, we could issue some

1:46commands here, but we could also issue commands here at firewall two or firewall one, or we could

1:51use GUI interfaces to communicate with those devices, or we could use Forta Manager to look

1:57at the routing information and do tests from these devices as well. So let's start over here at the

2:01client because I have a remote desktop to this client. Let's start here and ask the question,

2:06is the traffic from this device reaching the 48? So here at that Windows client, let's go and bring

2:11up a command prompt. So this is our IP address. There's our default gateway of 10.10.0.52, which

2:17I believe is the FortiGate. We'll confirm that here in a little bit. Let's also go ahead

2:21and just do a ping to our default gateway. That just means we have connectivity between us,

2:26the Windows client and the default gateway. So we'll do a ping to 10.10.0.52. And that is working.

2:34We also on this Windows computer could do an ARP-A and look at the ARP cache. And that's showing us

2:39here on this interface that this is the MAC address associated with the 10.10.0.52. So if we

2:44looked at the firewall, we could confirm that this MAC address A and E5 is associated with the

2:49interface on the firewall that has the IP address 10.10.0.52. And down here below with the IP address

2:55of 1.3, I'm using that for remote access into this Windows computer. And there's no default gateway

3:01set up off of that interface. So all the internet-based traffic is going to be coming from

3:05the 10.10.0.102 address off this lab interface. And if we use the upper key a couple of times

3:10and do an IP config again, we currently have 10.10.0.52 as our default gateway. Fantastic.

3:17So from a routing perspective from the client, it has firewall 2 as its default gateway. That

3:21looks good. Let's look at the routing tables on firewall 2 and firewall 1 to make sure that their

3:26routing tables are both appropriate for forwarding traffic from 10.10.0 going out to the public

3:31internet and also being able to forward traffic back to that subnet 10.10.0. Because firewall 2

3:36is directly connected, but firewall 1 is going to need to have a route that says how to get to

3:40the 10.10.0 network. So let's take a look at the routing tables. So here is a CLI. I've got an SSH

3:46session over to firewall 2. I'm using secure CRT for that. And here at firewall 2, let's take a

3:51look at the routing table. So we use a get router info, routing table, question mark, and let's go

3:57ahead and ask for all the complete routing tables being used by firewall 2. So here we have a default

4:03route. Fantastic. It's being learned via OSPF. Fantastic. The next hop is going to be 10.12.0.51,

4:11which based on our topology is the next interface over at firewall 1. We can also verify with an

4:16execute ping that we can reach 10.12.0.51. And sure enough, we can. Fantastic. And this also shows in

4:23the routing table that we are directly connected to the 10.10.0 network. Fantastic. So the routing

4:30table on firewall 2 looks great. Let's take a look at firewall 1. So here's firewall 1. Let me go

4:35ahead and copy that command. So I'll go back. I'll do a get router info, routing table all,

4:40go back to firewall 1, paste that in and press enter. So on firewall 1, we also have a default

4:45route. And the next hop is 23.1.2.1. Let's take a look at our topology. So in our topology, 23.1.2.1

4:54is this service provider A. So it looks like we've got connectivity out through service provider A.

4:59We have a route, a default route using service provider A. And let's also verify we can ping

5:04our service provider. So we'll do an execute ping at 23.1.2.1. And that looks good. Also here on

5:11firewall 1, let's take a look at the route back to 10.10.0. Because if this firewall is sending

5:16traffic back to our client or on the 10.10.0 network, it needs to have a route. So it has an

5:20OSPF on route. Great, great, great. And the next hop is 10.12.0.52, which is an interface over on

5:27firewall 2, which looks great. So another thing we could do here is we could try a ping or even

5:33a trace from this Windows computer or from firewall 2 to get to the 23.1.2.1. But because

5:39this Linux device has no problem getting out to the public internet, it's very likely not a routing

5:45issue. It's going to be something else. So just to verify that from this Windows computer, let's do a

5:49quick ping out to 23.1.2.1 just to verify we have connectivity. And we can also do a trace as well.

5:55So back here at PC3, the Windows computer, if we do a ping to 23.1.2.1, that looks good. Also,

6:01it's showing a TTL of 253. And that's for the responses that came back, which implies that

6:07ISP is a couple hops away. Going through firewall 2 and firewall 1, we can also do a trace.

6:12So on a Windows computer, it's called trace RT-D. Don't bother doing name resolution.

6:18And let's go ahead and go to 23.1.2.1. And that will also confirm the path we're using. So our

6:24first hop was 10.10.0.52, which is firewall 2. Next hop was 10.12.0.51, which is firewall 1.

6:32And then the last hop here, the last device was the service provider itself at 23.1.2.1.

6:37So as we went through our checklist here, the traffic is reaching the FortiGate.

6:41The routing tables on the FortiGate are correct based on what we looked at. And now the question

6:46is, is that traffic being denied at a firewall? And for that test, let's go ahead and go back

6:50to fast.com again or attempt that. So that is currently not working. And as far as matching

6:56on that, we could look at the logs here on firewall 2 and firewall 1. We could also do a

7:00packet capture right here on the computer itself. Or if we have a span port set up, we could do a

7:05packet capture off the network to see whether or not the traffic that's trying to go to fast.com

7:10is going there or not. And again, this is before we're even looking at the policies here on

7:14firewall 2 and firewall 1. So let's go over to this computer here. I've got a remote desktop

7:18open to it. Let's go ahead and run Wireshark right here and capture the traffic as it's going in and

7:24out of the network interface card. And then we'll attempt to go to fast.com and look at the results.

7:29All right. So on this computer, I'm going to go ahead and minimize the command prompt there.

7:34Let's launch Wireshark. There's a shortcut here on the desktop for it. And our network adapter is

7:40this one right here, the VLAN10 interface. I'll go ahead and start a capture. Okay. So I see

7:45maybe what the problem is, but now we have this capture running. Let me minimize that.

7:49And let's go ahead and open up a browser. So currently this browser has a bunch of default

7:55tabs trying to load. Let me close them just one at a time here. And let's open up a new tab and

8:00let's try to go to fast.com. That's our test that we're using to verify whether or not it works.

8:06Sure enough, fast.com's DNS address could not be... Oh, look at that. It's telling us that

8:11the name resolution from fast.com didn't happen on this computer. So if we go back to the command

8:16prompt, here we can type in nslookup for name resolution and try to resolve fast.com. And

8:24survey says, yeah, DNS request timed out. Here's the DNS server we're trying to use. It's trying

8:31again and again. I'll go ahead and do a control C. Also because we're capturing that, let's take a

8:36look. So here's our Wireshark. I'll go ahead and stop the capture and let's take a look at DNS.

8:42So I'm going to put a display filter just for DNS up in the top left there. And look at that. So

8:47it's asking about a lot, including fast.com. So here at the command prompt, we do the ifconfig

8:54and then we do a slash display DNS. It says, Keith, you're not on Linux. Okay. So let's hit

8:58the up arrow key and do an ipconfig display DNS. And here's what it knows about. Effectively,

9:03it has no quad A records, no IPv6. It knows about itself. That's the loop address for IPv4.

9:09And then it also has this one here, but it has no other names, including fast.com that's resolved,

9:15which would imply that this client is not correctly doing DNS name resolution.

9:20So if we go back to our capture, I'm going to stop it. And we do a display filter for just DNS

9:25up in the top left there. And if we scroll down, it has a ton of standard queries and there is

9:30fast.com right there. However, if we go in and say, you know, instead of looking for a query,

9:36go ahead and I'm going to just click and drag on response, drag that up to the display filter

9:40and say selected. And I'm going to say response equals true and press center. Here's the only

9:46response that we got. And it came from 8, 8, 8, 8, 8, coming back to 10, 10, 0, 1, 0, 2.

9:51And there's a response for google.com with the appropriate IP address associated with google.com,

9:56but none of the name resolution worked. So if we go back to the command prompt in an ipconfig

9:59dash all or slash all, either way works with the current version of the windows.

10:03Currently it's using a DNS server of 6, 7, 8, 3. However, based on our NS lookup that we just did,

10:09that's not the right DNS server, or at least it's not responding with DNS answers back to the

10:14client. So if we use the PPP that we talked about earlier, or the RAP, which I personally like the

10:20most, as far as routing goes, that was okay. Uh, address translation. Hmm. How do we verify that?

10:27You know, we could do if it's just DNS issue, uh, we could go ahead and reach a device based

10:32on its IP address. That would confirm that address translation is all working. So let's

10:36try that real quick as well. So back at the client, let's do a trace route. So in windows

10:40is T R A C E R T dash D for don't bother doing reverse lookups because name resolution is not

10:45working anyway. And let's go out to 8, 8, 8, 8, and press enter. And I'll do a control C that's

10:50enough. So here's showing going through firewall two and then firewall one, and then the service

10:55provider a or ISP a, and then other hops in the path towards 8, 8, 8, 8. So as far as reaching the

11:01public internet or getting off of our local network, that is working. So this implies a

11:06few things. Number one, IP connectivity wise, we have access out to the public internet

11:11to also because it's just the name resolution is failing. It's probably just a name resolution

11:16issue local here on the windows PC. And again, we can see that with IP config slash or dash all,

11:22and there's the DNS server it's trying to use. And with NS lookup, we confirmed it can't do name

11:27resolution. So as far as permissions go, the ICMP traffic that was used by trace route on this

11:32windows computer that was allowed out address translation worked because the replies came back.

11:38So as far as what this likely is, it's likely the wrong DNS server that this single client is

11:44attempting to use. Now, another element that I also want to chat about is that if we have a user

11:49that has some issue getting up to the internet or some resource, and they claim is the firewall.

11:54Another really important tool that we have available to us is to look at the firewall

11:58and to identify existing flows to show that, Hey, this flow of traffic is going through the

12:03firewall. It's something else outside of the firewall itself. So even though we have narrowed

12:08this down in this scenario, that is very likely a DNS issue on this computer. I'd also like to

12:13walk you through examples of confirming traffic flow for sessions going through a firewall,

12:18and we get to do that in the next video. So I'll see you there in just a moment.

0:00In this video, I'd like to walk you through a couple examples of being able to prove traffic

0:04flow through the firewall. So one of the benefits of looking at these sessions is that if the

0:09traffic is being allowed through, and then the reply traffic eventually comes back via the

0:13stateful database, via the stateful nature of the firewall, that effectively shows that the traffic

0:19is going through the firewall, that the firewall is not denying it. Now we could look at the logs

0:23and look for denies, but a lot of the implicit denies at the end of the FortiGate policies

0:27don't have logging enabled by default. So if we're looking for just the deny statement,

0:31we may not find it because of the implicit deny that is perhaps matched. However,

0:36if we show sessions from this address going out to the servers and so forth in the firewall,

0:41that can help verify that traffic is actually being forwarded correctly by the firewall.

0:45So a few questions asked. Number one, does a session exist for when a client is trying to

0:50go out to a resource? Number two, is that traffic, is that session being offloaded?

0:54In many vendors implementation, they call it the slow path versus the fast path,

0:57where the initial packets flow and they get processed by the central CPU. And then for

1:02remaining packets in that session, they're offloaded to a dedicated processor for that

1:06purpose. And that's also a good question to ask because if we are doing debugs, which we'll do

1:11in the next video, and we're looking at specific traffic flows, we might not see all that traffic

1:16if it's being offloaded to another processor. Another question to ask is, is the traffic

1:21asymmetric? Meaning, is it coming in through one path? In this topology, there's only one path in

1:26and out from this PC going through firewall two to firewall one after the public internet. But

1:31if we had another firewall or another routing device, we could have traffic that's going out

1:34one path and coming in another. For example, we have a router here that's also connected to the

1:3910.10.0 network. And the reply traffic, if it's coming back in this path or this direction,

1:44that could cause a problem as well, because the firewall two is not seeing the reply paths

1:48coming back in. Also, this could indicate that we have a routing issue if that wasn't our intent.

1:53And normally, if traffic is going out through firewall two, we want the reply traffic to come

1:57back in to the client via firewall two. So from this client, let's generate a ping stream that we

2:02can actually look at as far as session flow. And then we'll take a look at a couple different

2:06options for looking at the session information. So back at our Windows client here in Wireshark,

2:12and fantastic, we have traffic showing there. I'll minimize that. Let's go back to the command prompt

2:16and let's do a ping slash question mark. And I want to go ahead and do a dash T. That'll just

2:22keep on pinging. So we'll do a ping dash T and let's go to 8888. That's a Google DNS server.

2:28And that's working. All right. So that's running. Now, as far as validating and verifying session

2:33information, there's a few ways of doing it. Number one, we could use the GUI. That would

2:36certainly do it. Number two, we could use the CLI. So let's take a look at both those examples

2:41of verifying session information. So let's start off on the GUI here. So this is firewall one.

2:45So here on firewall one, we'll go to FortiView sessions. And sure enough, here we have some pings.

2:49So it's showing here that we have some pings going on that are coming from 10.10.0.102. And

2:55then we have a whole bunch of other DNS requests that have gone out and continue to go out from

3:00that client. So that is one way to validate that we have traffic flowing through the FortiGate.

3:05Another option would be go down to log and report and go to forward traffic. And again,

3:09this is on firewall one that we're doing this. And here we could say, you know what? I want to

3:13go ahead and filter or anything that's not and accept. And that should show us the denies. And

3:18sure enough, we have a bunch of denies, but it's all from 10.12.0.52. It's not from our clients on

3:25the 10.10.0 subnet. So based on this happening, I also probably want to look into why that's

3:30happening. But one troubleshooting step at a time, we're focused on the client on the 10.10.0 network.

3:35And currently with the source addresses here, we're not denying on that source address of 10.10

3:40anything. It's all 10.12. So now that we've taken a look at the FortiView sessions here from the

3:44GUI, and I just want to confirm I can still see the pink, sure enough, right there, and it's

3:48continuing to grow. Fantastic. Let's also take a look at the CLI, how we could verify a session.

3:54So instead of just looking at every single possible session that may be going through the firewall,

3:59this is firewall one, we could do a diagnosis session filter, and then put in the details

4:05regarding the interfaces involved, the ports involved, et cetera, et cetera, et cetera. Or

4:10if you just want to press enter here, it's going to show you that currently based on the filter,

4:15it's any VDOM, any source interface, any destination interface, et cetera, et cetera,

4:19et cetera. So in my small environment, I could do a diagnosis session list, press enter. This is

4:24pages and pages and pages. So for example, right here, we have our client at 10.10.0.1.0.2

4:31with a source port of 62,000 and change, trying to go out to 6783. And because it's in the session

4:36table, that traffic was forwarded out there to the well-known port for DNS of 53. And so once again,

4:42this is another option we can use to help verify traffic is actually flowing through the firewall.

4:47So if we wanted to filter the output a little bit, what we could do is add, for example,

4:51the protocol. So I'll add in PROTO space, question mark, and I'm going to go ahead and use 01. So

4:57protocol number one is ICMP, we'll press enter. And now if we do the command diagnose session list

5:02and press enter, it'll just show us the ICMP-based sessions. And here's the information regarding that

5:06session traffic from 10.10.0.1.0.2, that's our Windows client, going out to 8888 with those

5:12continuous ping requests. And back at the GUI, we can do similar treatment. We can just right-click,

5:16for example, on the ping here, filter by application, exact match of ping, and that would

5:21show us that information. So effectively we're seeing here that ping session, that ping conversation

5:26is happening between our Windows client and the server at 8888, the Google DNS server. Also,

5:31if we scroll here to the right, it indicates that that session has not been offloaded to an NPU.

5:36So if we remove the filter and show all the sessions and then go to the right,

5:40we can notice that some of the traffic has been offloaded to the NPU. For example,

5:44this traffic right here in this session, which is coming from 10.12.0.52, another device in our

5:49topology, going out to this address, and that's our firewall at 10.12.0.52, that's actually firewall

5:54two, that's interacting with FortiGuard. So whether you want to look at the sessions here,

5:58or you want to use the CLI, whichever is most convenient or most handy,

6:02these are two different ways of looking at the session information on the FortiGate itself.

0:00In this video, I'd like to chat about using Diagnose Debug.

0:02And the core theme of using debug is to use it with intent

0:06and not just for, you know, thousands of lines of content to be thrown at a screen

0:11where at that point it just becomes noise.

0:13So out here with intent.

0:15For example, we want to verify whether or not DNS is working

0:18or we want to verify certain connectivity is working

0:21from a client to another IP address.

0:22And with this protocol or that protocol or this interface or that interface,

0:25we're using debug to confirm what we think is happening on the network.

0:29So as far as some of the thought process involved here,

0:32we might want to ask the question, what am I trying to confirm?

0:36And also, what filters should I be using?

0:38Because if we're just trying to troubleshoot this device here

0:41with the address of .102 on the 10.10.0 network,

0:44we want to put filters in place regarding that specific IP address

0:48and perhaps specific protocols as well.

0:51We could also throw in, you know, the interfaces involved and so forth.

0:54And that way we're just focusing with the debug output

0:56on just the traffic that's of concern to us

0:59or that we want to have questions answered about.

1:01So if something is not flowing through the firewall as it should be,

1:04and we know that routing is okay,

1:06it could be, you know, part of the policy

1:08and or security profiles associated with that policy.

1:10So let's make a road trip over to firewall one.

1:13And let's take a look at a few examples regarding diag debug.

1:16Now, actually, before we do that, I still have a capture running.

1:19Let me go ahead and stop the pings.

1:21And we have a capture that has about 4,600 packets in it.

1:25So we'll go ahead and stop that.

1:27And I'll go ahead and minimize that.

1:28We can come back to that later if we want to.

1:30All right, so let's head over to the CLI for firewall one.

1:33So here is firewall one.

1:34Now, as far as running diag debug and other options similar to that,

1:38we can actually use the documentation for it.

1:40We don't have to memorize every possible command.

1:42And again, we're focusing in on specific commands

1:46trying to validate certain conditions on the network.

1:48So one of the things I've done is I have a bunch of buttons here

1:51I set up earlier today in preparation for this.

1:53And these here just are going to reset the debugs.

1:57So diagnose, debug, disable, trace stop, filter clear,

2:02also turning off the debug flow, show function name,

2:05and also the iProbe information, and then resetting.

2:07So that way, we don't have anything running.

2:10So let me share with you a few debugs I have currently set up.

2:14So these are in this button I've set up earlier today.

2:16And diagnose debug flow filter address 10.10.0.1.0.2.

2:21Now, the more often you use the diagnose debug commands

2:25on a FortiGate firewall, the more comfortable you'll be with them.

2:28However, there is online documentation

2:31that we can use to look up those commands

2:33and filter down just to what we want to look at.

2:35So this says, I want to look at flow information

2:38for just the IP address 10.10.0.1.0.2.

2:41That could be source or destination.

2:43However, we could set up a filter looking for that

2:45just as a source or as a destination or both.

2:48But that's what I currently have here.

2:50I'm also setting up the filter to look at protocol one,

2:53which is ICMP, which is used by Ping.

2:56I've also enabled this feature here, the show function name.

2:59And that way, as part of the output,

3:00it's going to show us additional information

3:02about what's being done.

3:03I've also included the trace messages about iProbe.

3:06Effectively, those are going to give us additional details

3:09about the flows that are being looked at.

3:11In this case, flows involving 10.10.0.1.0.2 with ICMP.

3:14I've also enabled timestamps, which are convenient,

3:17especially if we're looking at or comparing

3:18a couple of different flows

3:19on a couple of different devices.

3:21This is also presuming we have time synchronization in place

3:24so that our firewalls have identical

3:26or nearly identical times

3:28all the way down to like the millisecond.

3:30And I've also set up Diagnose Debug Flow Trace Start 10

3:34so that it's not going to actually do it beyond 10.

3:37That's not a bad idea either.

3:38And then to turn on is Diagnose Debug Enabled.

3:41Now, why isn't the screen flooding with events?

3:44And there's a couple of reasons for that.

3:45Number one, it's not going to flood forever

3:47because I limited it at 10,

3:49meaning 10 packets effectively

3:51that are going to be reported on.

3:52And because of the additional information here,

3:54we could have multiple pieces of information

3:56on each of those 10.

3:57And then also, even though I've turned it on with Enable,

3:59I haven't actually sent any pings yet

4:01or anything else that's using ICMP from 10.10.0.1.0.2.

4:05So let's head over.

4:06In fact, you know what?

4:07I've got a window right here to my Windows client.

4:10So here we know an IP config

4:11just to confirm its address information.

4:13So it's at 10.10.0.1.0.2

4:15using firewall2 as its default gateway.

4:17So let's go ahead and do a ping out to...

4:20In fact, let's do a trace.

4:22Trace RT-D to 8.8.8.8 and press Enter.

4:27So on a Windows computer, trace uses ICMP.

4:30So I'm going to Control-C that.

4:31Let me also do a ping to 8.8.8.8.

4:33And then let me go ahead and do a ping to 8.8.8.8

4:36with a dash T for a continuous ping.

4:39All right.

4:39So it's going to stop at the console

4:42once it has 10 that it's reported on

4:44because we set the limit of 10.

4:46So let's go back to our console for firewall1.

4:49And let me go ahead.

4:50And you notice how it's not growing anymore.

4:52So here it has a lot of output,

4:54but it's not going on indefinitely

4:56because we told it to stop at 10.

4:58So if we go to the top here,

4:59we have trace ID scrolling up here.

5:02We have trace ID 284.

5:03We have multiple entries and multiple reports

5:06on each of those IDs.

5:08So if we need to get in the nitty gritty

5:10regarding which policies were checked

5:12and which were matched and so forth,

5:14it's all going to be right here with this debug.

5:16So for the very first line here,

5:18there's our timestamp.

5:19If we scroll to the right,

5:20this firewall received a packet

5:22and it's protocol number one ICMP

5:24coming from 10.10.0.1.0.2

5:26going to 8.8.8.8.

5:28The ingress or inbound interface

5:29it received that on was v12.

5:31That's the name of the interface on firewall1.

5:34It's a type eight code zero,

5:35which is an ICMP echo request.

5:37And as we continue down,

5:38it has more details about

5:39what the firewall was going through in his mind

5:41regarding address translation and forwarding.

5:44And if we scroll to the right,

5:46it went through a bunch of checks

5:47regarding firewall policy,

5:49which includes the security profiles

5:51associated with that policy.

5:52And if we scroll down,

5:53oh, this is interesting right here.

5:54Check this out.

5:55TTL is exceeded, drop the packet.

5:58And that's because that's how trace route works.

6:01It uses the TTL initially set to one,

6:04sends out three probes,

6:05and then it changes it to two,

6:07sends out three probes and change it to three.

6:09And that way we're getting feedback

6:11from the devices that are dropping those packets.

6:13So at this point, based on this packet,

6:15the firewall said, I need to drop this packet.

6:17So that's the end of this packet.

6:18It got dropped here at the firewall.

6:20And then for the next packet,

6:21because the PC sent three of them out,

6:23this one also, which has the ID of 285,

6:26it should also be dropped.

6:28Let's go ahead and go down here to 285 to the end,

6:31it got dropped.

6:32And this should happen at least one more time.

6:34So if we do the end of 286 and go right there,

6:37it was dropped as well.

6:38Now the next one is going to have a higher TTL.

6:40And as a result, it shouldn't be dropped by the firewall.

6:43So if we go to 287 here,

6:44which is another ping request from 10.10.0.1.0.2

6:48out to 8.8.8.8,

6:48and we go down to the end of 287 here,

6:51it actually made it and SourceNet was used.

6:54So the address 10.10.0.1.0.2

6:58got swapped over to 23.1.2.51

7:01as a source address before being forwarded.

7:03And then the next packet, 288,

7:05if we scroll down,

7:06looked like it was attempting

7:07or thinking about offloading that to the NPU,

7:09but that didn't happen.

7:10Then we have the next packet of 290.

7:12If we scroll down,

7:13the end of 290,

7:14SourceNet again,

7:15and it continues.

7:16So some of the mistakes often with debug in general

7:20on any platform

7:21is just to turn it on

7:22and have it show you everything.

7:24And at that point, again,

7:25it just becomes noise.

7:26So we want to make sure

7:26we're using the filters in place

7:28to narrow down the scope

7:30into the protocol or the devices

7:32that we want to look at.

7:33And as an example of not putting in

7:36appropriate filters as part of our debugs,

7:38let me share this little script

7:39I have here with you.

7:40So I made a button here and secure CRT.

7:42We'll go to edit button

7:44so you can see the commands.

7:44I'll zoom in here

7:45and setting up a filter effectively

7:47with no filters.

7:48So it's allowing every IP address,

7:50every port, every protocol, etc.

7:52And then for this demo,

7:53I've also limited it at 100,000.

7:56So it won't show us more than 100,000.

7:58So let me go ahead and click on OK there.

8:00And let me launch that.

8:01So in the background,

8:03what we have going on

8:04is that ping request is still going on,

8:05but even if we turn that off,

8:06we have the Windows client.

8:08We'll do a control C.

8:09Let's go back to the console.

8:10You'll notice here,

8:11it's just going and going and going

8:13because there's other traffic

8:15happening on the network.

8:16And so as part of our debug without filtering,

8:18this is just a lot of noise.

8:20We'll do a control C there.

8:21And that's just pages and pages

8:22and pages and pages of output.

8:25So at this point,

8:25it's like trying to find a needle in a haystack.

8:27So the moral of the story

8:28is put the appropriate filters in place

8:31to help confirm

8:32what specifically you're looking for.

8:34Think of it like doing a debug with intention,

8:36not just to get a lot of output

8:38and say, well, look at all that,

8:38but rather to focus and narrow the scope.

8:41Also, another tool that since we're here,

8:42I want to share with you as well

8:44is we could go ahead

8:45and use this option right here

8:46of diagnose sniffer packet.

8:48And then we could specify in this case,

8:50VLAN 12 interface.

8:52And then I specified in quotes

8:53the actual host address I'm looking for.

8:55So I'm going to launch it for the .102.

8:58That's our Windows device.

8:59So here it's showing us the interface is V12

9:01and I didn't put a filter just for ICMP.

9:03So this is actually showing us

9:05all those DNS requests

9:07that that client is making,

9:09trying to go to this DNS server,

9:10which either is incorrect

9:13or isn't responding correctly

9:15with the answers the client needs.

9:16So let me leave this running

9:18and let's go back to our client

9:19and solve its problem,

9:20which is an incorrect DNS address.

9:22So let's go back to our client

9:23and we'll go to the configuration

9:24for networking on this Windows device

9:26and we'll right click on VLAN 10 interface,

9:28go to properties

9:29and we'll tell it that,

9:30hey, we want to go ahead

9:32and use the correct DNS server.

9:34So currently it has 6783

9:36and we use a Google DNS server of 8888.

9:38So I'll put that in there,

9:39click on OK, click on OK.

9:41And now what we'd want to do

9:43is also validate that this solution works.

9:45So before we open up a browser,

9:48let's go back to the CLI on Firewall 1.

9:50So here at Firewall 1, check it out.

9:52Oh yeah, it's going like crazy.

9:54So 443 is HTTPS services.

9:58And so if we scroll up here,

9:59we should be going out with DNS requests

10:01that are going out to the correct DNS server.

10:04In fact, here we have a whole bunch right here.

10:05Oh, look at that.

10:07This is not using traditional DNS.

10:08It's using some secure flavor,

10:10either DNS over HTTPS or DNS over TLS.

10:15In that case, it was encrypted

10:16and that's based on the client's browser

10:18and how they interact with DNS.

10:19But now we have a lot of conversations

10:22happening between this client

10:23and the rest of the world.

10:25And that's being shown here

10:26based on this output.

10:28And again, the command was to do this

10:30was diagnose sniffer packet.

10:32I told Firewall 1,

10:33I want to pay attention to the V12 interface

10:35and the IP address of 10.10.0.1.0.2,

10:38either a source or destination.

10:40So now with the DNS issue resolved,

10:42we go back and we try an NS lookup.

10:44Let's go ahead and bring that up

10:45from our history here.

10:46There we go.

10:47NS lookup for fast.com.

10:48We now have that name resolution

10:50that's working.

10:51Fantastic.

10:51And if we did an IP config slash display DNS,

10:54it would show us our DNS cache,

10:56which is populated with a lot of entries.

10:58And one of these entries equates to fast.com.

11:01So we did an NS lookup for fast.com again.

11:04Press enter.

11:04It's resolving to 23.222.162.95.

11:09And as a result,

11:09if we open up a browser

11:10and we try to go to fast.com again,

11:12so we'll click on a reload here.

11:14It should now work because DNS is now working.

11:16So again, now that we've resolved the DNS problem

11:18that this one local computer

11:20was having on its own,

11:21it wasn't affecting the rest of the network.

11:22Once that's corrected,

11:23we've now verified at the CLI for Firewall 1

11:26and also the functionality here at the client

11:29that internet connectivity to fast.com works.

11:31And also now that we've corrected our verified fast.com,

11:34if we open up a browser,

11:36we should be able to go to other websites

11:37that are allowed by policy.

11:39So there's Keith Barker page.

11:41There's Palo Alto.

11:42There's Fortinet.

11:43There's Cisco.

11:44And there's CBT Nuggets.

11:46Fantastic.

11:46Looks like everything's working.

0:00In this video, I'd like to chat about knowing when enough is enough and when is a good time

0:04to stop. And in a production environment, it just doesn't mean stop and go home. It means stop

0:09and then escalate. For example, if we are the network team and then we have a different team

0:15that's working on and supporting the clients, as far as fixing that client, which we think

0:19is a DNS related issue, we'd want to escalate that to that team. Or if we simply can't sort

0:25it out, we can't figure it out, we'd escalate it to the next level or the next group of

0:30individuals at our company who are there or who can be there through contracting to come in and

0:35sort it out. And it doesn't mean failure. It means, hey, we're trying another approach.

0:39So being able to recognize when further troubleshooting is not going to be helpful

0:44or not going to be productive is a great time to go ahead and escalate. Now, some of the decisions

0:48as far as when to stop and escalate would involve questions such as, do I have enough evidence,

0:53including what we already diagnosed? So maybe we've run several commands at the firewalls,

0:58including commands that will show us the routing table and the session information.

1:03Maybe we have screenshots of the GUI. We want to make sure that we have enough information

1:07to help document what we've discovered so far as we escalate it. And that's going to save them,

1:12whoever we escalated to, time and effort because they won't have to redo the same steps we did.

1:16Now, even if we escalate it and we provide the data that we have, a good escalation team is

1:22also going to do quick validations that what you have told them is accurate. And that also goes for

1:27us. If we have individuals that have escalated tickets to us that we need to solve, we want to

1:32also make sure that whatever they've told us, we do quick validation checks just as if they were an

1:37end user where we're going to believe them, trust them, but also verify that what they're saying is

1:43happening is happening and do it in a polite and courteous way. Also, as part of this, it's really

1:48important to make a nice clean trail of what you have tried and documented. And that's not just for

1:52yourself. It's also for the benefit of whoever escalating this problem or this case too. And

1:57as we do that documentation regarding what we checked, what we looked at, what's existing,

2:02we can also use that to compare against our other documentation for a configuration of a firewall.

2:07Maybe we have some baselines in place regarding what should be there and what is there. And if

2:12we've documented what currently is in place, and it's different than what we show in the documentation

2:16of what should be there, we can include that information as we escalate. I would also say

2:21that another great time to escalate is if we are working with a system, especially an infrastructure

2:26device like a firewall or a switch, and when we're configuring something or looking at the details,

2:31we see that a change has been made because every change in a corporate environment,

2:36every change needs to go through change control, which is going to verify what is going to be

2:41changed, has been tested, what's the change window, what's the rollback process in case it doesn't go

2:46well. So if there are changes that have been made to a firewall and they haven't gone through the

2:50change control process, that's also another great reason to escalate that so that the powers that be

2:56can actually get with the individuals who did it and correct that and make sure that future behavior

3:01doesn't repeat the past. Meaning we want all changes on all infrastructure devices to go

3:05through appropriate change control. So now that we've gone through a troubleshooting methodology

3:10with this device having a problem getting out to the public internet, in the next video, I'd like

3:15to create another scenario for you as a validation section so you can go ahead and look at the

3:20evidence of what's currently in place and then make deductions on what you think the problem may be

3:26based on the evidence provided. So I'll see you in the next video in just a moment.

0:00In this validation section here of this set of videos, I'd like us to go ahead and from

0:05a few screenshots that I will provide, I'd like you to, based on that data, make an assumption

0:10or a determination of what the likely problem is. And so every time I'm troubleshooting

0:16network firewalls, I'm always using RAP. Is there routing in place? Is address translation

0:22working as it should be? And are the permissions in place? So let me tell you about the symptoms

0:26that our customers are experiencing. Both the Windows device and the Linux device are

0:30both complaining that they cannot access any web servers on the public internet whatsoever.

0:37So let me share with you the screenshots I have for you as far as the troubleshooting

0:42exercise goes. So this is the Linux client. They have a browser opened up to fast.com

0:47and it says, hmm, we're having trouble finding that site. All right. And that is the Linux

0:52device. If we go to our next device, it's the Windows computer. It looks like it's running

0:57Chrome and it says no internet. And try this, try that, try the other. It also says DNS

1:02probe finished, no internet. But when we left off this device, we had corrected the DNS

1:08issue locally on that system and that is still in place, correctly in place. But however,

1:13now in this new scenario, both PCs, meaning this Windows computer and this Linux device

1:18are both having problems reaching fast.com. So let's go ahead and do some more screenshots

1:24here. This is a screenshot of the PC. This is the Windows PC pinging 23.1.2.51, which

1:32is the IP address over on firewall one. Let me show you where that is. So here on firewall

1:36one, this V23 interface, all the IP addresses on firewall one and in 51. So the V23 interface

1:44has the address of 23.1.2.51. And if we go back to our ping, that ping is successful.

1:50So that part's working. Next, let's take a look at the routing table. And this is on

1:55firewall two, as shown by the prompt here. And this is showing us that we have a default

1:59route. We have some directly connected networks and a whole bunch of OSPF learned routes.

2:04And then on the next screen, and I have all these images as part of the skill as well.

2:07So you can take a look at them. This is firewall one and it's routing table. So it's got a

2:11static default route. It's got some OSPF learned routes. It has some directly connected

2:14routes. And let me share with you one additional screenshot and that is the execute trace route.

2:20This is from firewall two, trying to go out to the public internet of 8.8.8.8. And this

2:24trace route says asterisk, asterisk, asterisk, meaning either the devices in the path don't

2:30want to respond and they timed out or we simply are getting no response. Either way, it would

2:36create these little asterisks. And so I just let it go for four and then I cut it off.

2:40So with these screenshots in mind, in conjunction with this topology, I'd like you to make some

2:44deductions on what you think the problem is regarding these two devices, not being able

2:51to get out to fast.com. So go ahead and pause me right now, take a minute or two, or as

2:56long as you'd like to look through the screenshots, make your best deductions. And then when you're

3:01ready, click on resume and we'll analyze them together.

3:07And welcome back. So one of the first things we'd want to do is verify that the issues

3:11that the customers are having are real. So let's go to the windows computer as an example.

3:16Here it is. Let's go ahead and open up a browser to fast.com and sure enough, it is not working.

3:24And let's go to a command prompt and let's do a ping to firewall one and that's working.

3:29That means we have connectivity to firewall one and firewall one has connectivity back

3:33to us. So based on the topology, we have a path between this device and firewall one

3:39because we're pinging over to firewall one to this IP address on this interface and we're

3:42getting a response back. So with this wrap acronym, let's take a look at the routing

3:46tables for firewall two and firewall one. Now we know they both have access to the 1010

3:50zero on the routing tables because the ping was responded to and firewall two is directly

3:55connected there. But let's also confirm the default routes because firewall two, when

4:00it tried to do a trace route out to the public internet, that was having a problem. So let's

4:03look at the routing tables on both firewall two and firewall one and make sure they both

4:08have a route that's correct for the public internet. So here's firewall to do a get router

4:13info routing dash table all and here is an OSPF learn route for the default route and

4:20the next top is 10.12.0.51. If we didn't execute ping over to 10.12.0.51. And that works great.

4:29All right. So we have a default route. That's fine. Let's go ahead and take a look at our

4:32old one. Uh, the routing tables are on the screen, but let me go ahead and issue the

4:35command again, get router info routing dash table all. And we have a default route. Uh,

4:41we have a OSPF learn route back to the 1010 network, but look at this for the default

4:45route. It's a static route. And that static route says that the next top should be 10.12.0.52.

4:53And that's not in the direction of our service provider. Let's take a look at the topology.

4:57So the default route here of 0.0.0.0 should use this IP address of service provider, a

5:0323.1.2.1 as its next top. At least that's what we had previously. And what it's showing

5:09is is a static route that says the next top for the default route should go back to 10.12.0.52,

5:15which is firewall two. So that basically means that if we have traffic that this client is

5:22trying to send out to the public internet, it's going to be looping right here because

5:26the default route on firewall two says send it this way. And the static route here on

5:30firewall one for the default route says send it back this way. So it's just going to loop

5:33and loop and loop until we get to a TTL of one. And then the firewall that gets that

5:36with a TTL of one is going to say, ah, TTL is one. I'm going to go ahead and kill it.

5:40And that'll be the end of it. The problem is that the default route that firewall one's

5:44using is pointing back to 10.12.0.52. And as a result, that traffic trying to go out

5:49to the internet, we used to be sent back to firewall two, who would then based on his

5:53default route, let's take a look at it real quick. We'd forward it back to firewall one,

5:57who would then forward it back to firewall two, who would forward it back to firewall one and

6:01back and forth and back and forth until the TTL was exceeded. But at the end of the day,

6:05our problem is right here. This default route is pointing in the wrong direction. So let's

6:10solve that and then test again. So if you go back to firewall one and go to the dashboard for

6:15network, here's our static and dynamic routing widget, just to confirm that the default route

6:20is pointing back to 10.12.0.52, which is firewall two. And over here, it's showing

6:25us this as static routes. So let's go ahead and change that by going to network and static routes

6:29and I'm just going to remove it. So I'll right click on it and go to delete. Okay. Now in a

6:35production environment, it might be safer to just disable it. In any event, it's gone. So let's go

6:40back to dashboard and network and go to our static and dynamic routing widget. And now it's showing

6:45the correct next hop for the default route. Let's also verify that at the CLI. So here at the CLI,

6:50I'll do an up arrow key. And now we have our OSPF learned default route going in the right direction.

6:56So now let's go back to our client and test again. So here's our client. Let's go ahead and do a

7:00ping. Yep. That's now working. And let's open up a browser. And now it's working. Initially,

7:05it was a routing issue on the firewall one that had the next hop for the default route pointing

7:10back to here. So now that we have the correct default route going out to our service provider,

7:14now we have connectivity from this device and we should also have connectivity from our Linux

7:18device. Let's verify that as well. So here at the Linux device, let me go ahead and close the

7:23browser. And let's open up the browser once again, nice and fresh. And let's go to fast.com. And that

7:29looks like it is working, which is fantastic. So thanks for joining me in this set of videos

7:33regarding some troubleshooting methodologies, including troubleshooting stateful firewalls,

7:38next generation firewalls from Fortinet, AKA the FortiGates. And I look forward to seeing you,

7:42my friend, in another set of videos very, very soon. Until then, I hope this has been informative,

7:47and I'd like to thank you for viewing.