Logging Concepts and Architecture

0:00It's time to talk about logs. That's right. We're not going to go out there with our lumberjack

0:04axe and chop down trees and make logs. We're talking about system logs. That's right. So we

0:09got Windows logs, Linux logs, all your network devices create logs. There's logs everywhere.

0:15So let's jump in and talk about logs. Let's get ourselves ready to understand why we need them,

0:21where they come from, what formats they're in, all these wonderful things. So let's get started.

0:26So the first question is, why do we need logs? That is a great question. So our systems all

0:34create logs, but why do we need them? There is definitely a need. And that's because logs

0:42tell the story about what happened. So without logs, we don't really know what happened. Okay.

0:50And to think about this, if there's an incident and you begin investigating this incident,

0:56take your magnifying glass and you go in there, what are you going to be looking at?

0:59Well, I'll tell you the logs because the logs, they tell a story about what happened. So here's

1:06a scenario. Let's say a user over here, uh, report suspicious logging activity at about 2 15 PM.

1:15Okay. And the windows authentication logs, we've got our windows logs here. They show this

1:22happening. All right. But there's a Linux application. We'll put an application down here

1:28running on a server. And it shows that this suspicious logging activity here

1:33also tried to access the application, but this shows up at 8 16 PM. So what's the discrepancy

1:43with these times here? Because if an attacker logs in with this suspicious activity, and then

1:50we see the login and windows, but the application is, well, six hours later, do you think that evil

1:57bad guy or, uh, just hung around for six hours, waiting to try to access the system's application?

2:03Of course not. And this can make it very difficult to do an investigation. So we're going to talk

2:13about this here, time discrepancy and how to solve that so that you have your logs all using

2:20the same time source. So they match up. Okay. So that is something very important. So let's go

2:26ahead and, and talk more about some laws. Where do these logs come from? Well, they come from all

2:31your operating systems. That's for sure. So you got your windows, you got your Linux. So, uh,

2:38here we go. I'm going to try to draw a little penguin here. And there we go. Okay. Give them

2:42a flappy arm here. There you go. There's our penguin. And then of course, uh, well, that's my

2:49apple. So there you go. Of course, it comes from your Mac and your Linux and windows. They all

2:52create logs, but also all of your network devices can create logs too. So you're talking about

3:00firewall, IPS, routers, switches, access points, all these things create logs. And the thing is

3:10even applications create logs. So all your apps as well. And this is wonderful because logs tell

3:18the story of what happened. All right. Now time is very important. And we kind of talked about a

3:25scenario here a minute ago where our windows authentication logged at 2 15 PM. And it was

3:33kind of questionable. There were some weird activities, some strangeness, but our application

3:38server over here said it happened at, I think, would we say 8 16 PM? So we had this time discrepancy

3:46there. So the thing is we need to be using something called network time protocol. Okay.

3:56And network time protocol known as NTP uses UDP port one 23. All right. Just so you know, that's

4:04kind of important because you've got to make sure that your firewalls aren't blocking it. If you're

4:08using it now, why do we use NTP? Well, on all of our systems here, no matter if it's windows, Mac,

4:14Linux, doesn't matter. You configure an NTP source and probably two of them, actually, maybe three.

4:21And what this is, that source is you're pointing to an NTP server and you want all of your devices,

4:28this includes your routers and your firewalls and your switches and access points and IPS,

4:35all of your things, all of them need to point to the same NTP source. That will ensure that they

4:42all have the same time they're synchronized. So like the, the spy movie synchronize your watches.

4:47Well, that's what we're doing. We're synchronizing our watches or clocks on all of our systems so

4:53that all the logs are going to have the same timestamp. So they're going to be able to be

4:58analyzed as one set of logs, because as you have these logs come in, if we had something at 2.15

5:05and then 8.16 and you're trying to investigate them, you know, they're connected somehow,

5:10but this time really is a pain because now I have to go look at these logs and realize

5:15that they're six hours behind. So I have to continually to remember minus six hours. Well,

5:21when you're looking at five or six or 10 system logs, that's just a real pain. Let me tell you,

5:27it makes life hard. We don't want that. So NTP is how we fix that and make sure all of our logs

5:35are synchronized time-wise and all of our systems. All right. Super. And that is very important.

5:40Now log retention, something else. So how long is the question? Should you retain your logs?

5:50That's the big question. That's right. So what is the answer then? Well, here's the answer.

5:56Ready? This is like industry secret knowledge. Okay. I'm going to let you in on this.

6:02The secret is it depends. Was that a big letdown? I hope not. It depends. Okay. And that's because

6:11this is going to be based on your organizational policies. All organizations are going to have

6:15policies saying this is how long we retain logs. Okay. And it might be that some logs are retained

6:21longer than other logs. It just depends. But here's the thing. You need to understand that

6:26log retention is a thing. So why is this necessary? Why are we talking about it? Other than

6:33I need to know that, yeah, log retention is a thing. All right. Let me clear a little space

6:38here and let's talk about this. So we know that the policies are going to guide what we're doing

6:44with our log retention. Now, here's why it's a thing. On your systems, logs are only kept there

6:51for so long. So it may be basically, it could be based on time, how long you want to keep them,

6:59like 30 days or something. Or it could be space. So maybe you're going to allow for 500 megabytes

7:06of log space. Well, when you fill this space up or you reach that time limit, what's going to

7:13happen is, by default, generally, they start overwriting themselves. All right. Now, what's

7:19the problem with that? Well, if today is, we'll say, April 10th, and we realize, hey, something's

7:30not right. Something happened. There was an incident or something. So you start tracing back.

7:35Well, if you only keep so much log, let's say 30 days, if you keep 30 days worth of logs, what

7:40happens if that incident occurred 35 days ago? Well, guess what? You're not going to get the

7:49full story. You're going to get some after effects, but you don't really know what happened

7:54at the beginning of that incident. This is why log retention is important. Well, it's also

8:01important for compliance and some things like that. But for now, let's just focus on this and

8:06security and incidents. Okay. So it's very important that we understand log retention and

8:12that we need to keep it for a certain amount of time so that we can go back and look at what

8:17happened X number of days or weeks ago. So when it comes to log retention, though, let me make a

8:24little more room over here, right over here. There's something that you need to understand,

8:30because when you're retaining logs, well, space isn't free. That's what I'm getting at. And your

8:38logs, oftentimes what we'll do is you might actually send those logs off of the system

8:46to a log repository. And maybe this over here keeps the logs for 90 days. That way you have

8:55a separate copy of your logs, which is always good. And you can go back 90 days. And oftentimes

9:02this is known as like warm storage. And sometimes they even get shipped off from there to what's

9:07called cold storage. And it might be kept there for, you know, seven years, who knows. But the

9:14thing is, this is expensive here or well, somewhat relatively, but down here, cold storage is usually

9:22pretty cheap. However, it takes time to get access to this data. This data, you can just jump in and

9:29analyze as all, you know, till the sun goes down, till the cobs come home, all that. But this, you

9:34usually have to request access to it, and it takes a couple days. So there you go. That's your log retention.

0:00Okay, let's talk about a SIEM now, or SIEM. Some call it SIEM, some call it SIEM. Security

0:06Information and Event Management. So what this is, we talked here about a log repository. Well,

0:13this is a type of log repository. It is a SIEM. You send your firewall logs, your IPS logs,

0:21all your systems logs, application logs, and this has intelligence built into the platform.

0:28So you could have a general log repo server and all it does is, it's just a storage space for

0:34your logs. That's all it does. You can go in and search the logs, but that's it. Okay. Well,

0:40what a SIEM does, or SIEM, it has intelligence built into it. So it analyzes. So yeah, it does

0:47the analyzing of all of your log data and it's looking for abnormalities and malicious activity.

0:56And then it can sound the alarm over here, letting you know something's going on,

1:01so you can investigate. And that's what a SIEM is. Now I will say SIEMs or SIEMs, they are not

1:07meant for long term storage of your logs. Okay. If you want to keep long-term storage,

1:15you're not going to do it in your SIEM. And that's because of how much data gets into this database

1:19here and it can get bloated and get really slow. So it's not meant for long-term storage. We use

1:25a log repository for that. So continuing with our SIEM, it uses something called threat

1:30intelligence. And it's a feed that you sign up for and it can feed into your SIEM and it provides

1:36information like IP reputation, domain reputation, attack signatures, email reputation. And

1:41reputation is, is this a malicious IP or not? That kind of information. Is this email address or

1:48domain name used recently in attacks? Okay. So it's basically, it's making it smarter, keeping it

1:58up to date with the current activities that are going on in the security realm.

2:04Now, something else that SIEMs do is called normalization and correlation. So let's talk

2:10about normalization first. So here we go. So normalization, what this is, is the process of

2:19organizing, structuring, and standardizing of data. So really what it's doing is trying to reduce

2:25duplicate information and make searching easier. Let me give you an example of what this means.

2:30So if you've got logs coming in from various systems and devices, well, let's say the field

2:36names, because in logs, you have field names. Think of it like a spreadsheet. And one of the

2:41field names is going to be, let's say SRC underscore IP, and that equals a source IP address. But

2:49another system might call this source underscore IP and another one, origin underscore IP or

2:59origin client or source client. So they can have all these different field names, but guess what?

3:04It's the same thing. And what normalization does, it does away with all those names. Now they're

3:11still there under the hood, but it replaces it with one field name. So you can look through all

3:17your various system logs as if they were one big log. And that normalization really helps to make

3:25searching logs much easier because the next thing is correlation. So what are we doing in these logs?

3:31Well, we're looking at linking separate isolated security events from the logs and various alerts

3:37across different systems and time windows to form a single meaningful attack narrative. So really,

3:42what are we doing? We're telling the story of what happened. That's what correlation is. It says,

3:46oh, the firewall saw this source IP over here. Okay. We've got the IP address. We've got the

3:54server. Okay. We saw it come in. Oh, and I saw it then it shows up in this windows server where it

3:58went to, uh, it tried to log in and then maybe it went over to this web server over here and it

4:03tried to access something. So what we're doing is putting together the storyboard of what happened.

4:08That's what data correlation is. Okay. Super. Moving on. Okay. So why do we need SIM or syslog

4:17other than for log storage and analysis? Okay. I'll tell you why, because let's say we've got

4:23this bad guy down here and he breaks into one of our systems over here. Guess what is most likely

4:29going to do as part of his attack. I'll tell you, delete the logs. That's right. So your logs gone,

4:38poof vanished. So, uh, there's a problem. You realize there's an incident. You log in. There's

4:44no logs to look at. You don't know what happened. Here's the thing. If you are using syslog or maybe

4:51you install some type of agent to ship logs, it's a shipping agent. What happens is you're sending

4:58those logs to a log repo like syslog is or to a SIM. That means you have a separate copy of your

5:08logs. So during an attack, if these logs get erased, you still have a copy over here. That

5:15is another reason to use some type of log repo or SIM. All right. And as part of log ingestion and

5:23shipping, okay, which we just kind of looked at, there's various ways to do this, but you're going

5:29to be shipping logs from like your firewalls, from your various servers and operating systems

5:34into a SIM. I'm going to put up here or syslog or some type of log repository. Okay. Let's take a

5:41look at those options. Here we go. Here's some various options. So you could put an endpoint

5:46agent. So oftentimes you'll install an agent on an OS and it will do the shipping for you. It'll

5:51ship your logs. You just tell it which logs and of course the destination right there. Okay.

5:59And then it'll do that. Now syslog, which uses TCP and UDP 514, by default it uses UDP,

6:06but you can configure to use TCP. But what that does, and generally we do this with our

6:11firewalls and our network devices, we use syslog a lot. And what it does is you basically just point

6:16it to a destination and of course tell it a, what we call a logging level. And we're going to get

6:22into that. And then it will ship those logs to that destination. You can also use Windows Event

6:27Forwarding. You can use APIs or you could use file-based collectors. These are all options to

6:32get your logs to a log repository. Okay. Now let's talk about data log levels. Now each system out

6:42there is going to have some type of logging level, and then we're going to see this because

6:47we're going to go through and look at this. Don't worry. But it means, you know, how much to log.

6:53And I'm not talking about space necessarily. I'm saying how detailed do you want those logs? Do you

7:02want every single little thing that happens on the system log? Or do you want just normal things like

7:08your authentication, maybe a file access, maybe permission changes? Let's see. What else? New user

7:18accounts created, new permissions assigned, those kinds of things. You might want these, but you

7:26don't really care about the other little things. Okay. So you have to figure that out, and you'll

7:31need to configure it as such. Okay. Now when it comes to syslog, and a lot of systems will use this,

7:38they use 0 through 7, a numbering system. Now Windows doesn't do this for its Windows logs,

7:44but a lot of other systems do. Now the thing is, you'll need to decide that. So let's look at

7:50these logging levels here. Okay. Here we go. 0 is known as emergency, meaning the system is unusable

7:57really bad. But then all the way at the bottom, we have debug, which records every little thing.

8:06And I do mean every little thing. So here, your log storage is going to be huge. You're going to

8:12have a lot of data. Here, you're going to have a tiny amount of data. And then, of course, everywhere

8:17in between. So then you've got alert, and critical, error, warning, notice, and informational. And here

8:23are the definitions. So I would spend a couple minutes just looking at this. Okay. It's pretty

8:28important you understand how these numbers and words correlate to each other. Now the thing is,

8:34when you're looking at this, if you choose the option, let me change colors here, down here,

8:40you're going to have so much data, you're just going to be overwhelmed. So we don't normally do

8:45that. Now during troubleshooting or something, you'll enable that debugging. Okay. But not normally.

8:50Okay. We don't run like that every day. So it might be that we turn this level on, our warning.

8:57Because if you have a bunch of excess data, it can cause what we call false positives. And a false

9:03positive simply means it's not true. Okay. That's what it means. Not true. So it's not really a

9:09problem. It'll raise the flag, say, hey, we've got a problem over here. But you know what? There's

9:13really not a problem. And this can cause what we call alert fatigue. Get that curly around there.

9:21Okay. Alert fatigue. That's right. Okay. Okay. We're going to wrap this up now with a logging

9:27lifecycle. So basically we start by generating logs. Then we use agents and services to collect

9:34those logs. And then we're going to transport and ship them. They're going to be ingested

9:39at our log repo. They're going to be stored and retained based on our retention settings.

9:43If we're using a SIEM or SIEM, analyzing correlation along with alert and investigation.

9:49And then we're going to be archiving for our retention policies or disposal of the logs.

9:55So that's it, folks. That's logging in a nutshell. In the next nugget, we're going to jump into

10:00Windows and start taking a look at the Windows logging options. See you there shortly.

0:00All right. Welcome back folks. Now it's time to work on windows logging.

0:03So I've got a lab down here.

0:04So what you'll do is just go ahead and launch the lab if you would and give it

0:08probably about five, six minutes to fire up as it starts the virtual machine and

0:12such. Then you can follow along with me if you'd like to. Let's jump over here.

0:16So in the lab, we have a password. Let me move this out of the way.

0:18We have a password file over here. It's got your passwords in it.

0:22And basically this is the password. Now these should be automatically logged in,

0:25but if they time out and log out, there is your password right there. Okay.

0:30We've got two hosts in here. What? VMs really? Server Nug.

0:33This is a domain controller in windows. And we have an Ubuntu host as well.

0:36So we're starting off here in windows. So let me go ahead and maximize that.

0:41Yeah, that didn't work too well. Uh, so there we go. We'll just leave with this.

0:46Okay. So here we go. What are you doing? We're looking at logging.

0:49So where does windows send logs? Well, they're stored on here,

0:53but how do you access them? Well, use the event viewer.

0:57So event, give this a second. It's a little slow getting started.

1:01And there's our event viewer.

1:02So we'll just click on that and let this open up and we're gonna take a look at

1:07the logs that are available in here. Now I will tell you right now out of the box,

1:12windows does not log enough information. Okay. Uh,

1:16we need more info and we're going to see how to turn that on here momentarily.

1:20So it's adding the snap in.

1:21We're going to get our event viewer up so we can take a look at the logs in

1:25windows. There we go. Let's go ahead and maximize.

1:28This is slide this over a little bit. Okay.

1:31So here we have our logs, windows logs, and you've got application,

1:36security setup system and forwarded events,

1:40but that's not all.

1:41If you come under applications and service logs and give that a second,

1:46it's thinking there because there's a lot in there. Let me tell you folks.

1:50Okay. Uh, what I had the problem here.

1:53I don't want to let you know where I clicked on this and it just gave me an hour

1:56glass and it just sat there. So if it does that,

1:59go ahead and close it out and reopen event viewer.

2:02It just didn't start right or something. Uh, just to let you know. Okay.

2:05So under here you see we have active directory, web services, DFS replication,

2:09directory services, DNS server logs there under Microsoft.

2:14You've got all these and look, I mean, if we go under windows, look at this thing,

2:17it's just a huge list of all of these logs.

2:22So there's a ton in there. So what I want to show you though, is these up here,

2:26these are our primary logs, like your security and stuff.

2:28So let's take a look at one of these.

2:29I'm going to open up an event over here and let's talk about what we see in here.

2:33So you've got a security ID saying it's a system, a,

2:37an account name that's involved in this event.

2:41You have account domain,

2:42you have a log on ID here and a log on type.

2:47Okay. And you don't need to memorize these or anything, but over here,

2:50event ID, this is 4634.

2:53That's how windows identifies various types of events and event ID.

2:57And so you can see this log information in here. So there's a lot.

3:01You can come over here. You can clear the log, which attackers like to do.

3:05You can filter the log.

3:07You can create a custom searches like custom views over here.

3:12And this is the logs. Now, if you right click on this and go down to

3:17properties right here, you'll see right here,

3:23this is the maximum log size.

3:25Now what's it going to do when that maximum log size is reached.

3:29We talked about this and this is set to overwrite events as needed.

3:34All right. So that's an overwrite or do you want it to archive the log? Okay.

3:38Or do not overwrite events, clear logs manually.

3:41You have those options there and there's a clear log button. All right.

3:45So that's how you can set your log retention.

3:48You can also do it through policies like group policy and such. Okay.

3:52So that's where you go to view your logs.

3:54I'm going to minimize this because now what I'm do is I'm going to open up group

3:59policy. So go down here. I'm going to type group policy management,

4:05because I want to show you where you can turn on additional logging.

4:08So let's just go into the default domain policy here.

4:13Say, okay, we're going to edit this, right? Click on it. Once it loads up here,

4:18there we go and go to edit and it's going to bring up the settings. Okay, super.

4:22So let that populate.

4:26I'm going to move that down just a little bit. Okay.

4:29So let's take a look at this.

4:30If I go under policies and then I go down to windows settings and then down to

4:36security settings,

4:39and I need to slide this over so you can see what's happening here.

4:42A little more. There we go. What we have here.

4:47Then we go down to local policies right here. Let me expand that.

4:52You have audit policy.

4:55Now in our group policy here, Microsoft doesn't call it logging.

5:00They refer to it as auditing. So if you go to audit policy,

5:04it says auditing. So if you turn auditing on, it means it's going to log it.

5:08So right here we have a bunch of not defined.

5:10So you'll want to come through and define these.

5:12Now it does log when somebody logs on and logs off,

5:15it logs the authentication process, but there's some things it doesn't log.

5:19All right. So let me do this. Let's see here. These real quick.

5:24So you have audit account log on events. So if you double click on it,

5:28you can define it and say success and failure or success or failure.

5:32Just depends on what you want there. I'm going to cancel that.

5:36But there's audit log on events, object access, audit policy changes,

5:41all these things. So what I'm going to do real quick,

5:43I'm going to minimize both of these and I'm going to right click here and create

5:48a new folder. There we go. And I'm going to call it test. Excellent.

5:53Now I'm going to go in here and give it just a minute here.

5:57Okay.

6:01I'm going to create a new text file and there we go.

6:05Text document. And I'm going to say doc one. Why not?

6:08And I'm going to put some information in it.

6:12Just like that. Okay. File save and close.

6:16Now let's go back to our event viewer.

6:20I'm in security. I'm going to right click and I'm going to refresh. Okay.

6:26So what we have here is log on, log off, special log on, log off.

6:29It doesn't show anything where I was working with creating a file or accessing

6:33the file. Well,

6:34that's because you have to turn that type of logging on and there's a couple

6:37different places that you have to do it. So let's go back here to our settings.

6:40So what we're going to do

6:44is we're going to select object access, audit object access,

6:48and I want both success and failure. And I'm going to say, okay,

6:53now we see it has changed over here.

6:55Now you can do this for all of these different things to make sure that they

6:59meet your organization's needs. Now, if we scroll down a little further,

7:03there's also advanced audit policy configurations.

7:07Let's expand this. There we go. These are audit policies.

7:11So here's one for account log on. So if I select this,

7:16audit credential validation, Kerberos authentication service.

7:19So you can get deeper in there. There's directory service access,

7:23log on, log off activity,

7:26object access, policy changes, privileged use all these things.

7:31So this is where you come in and you really configure what you want windows to

7:35log. Okay. That's where this is done. Then you push it out via group policy.

7:39Now, if it's just a single host and it's on a domain,

7:42you can go to your system policies and you'll see these same thing in there or

7:46something very similar.

7:48So now that we've turned our auditing on for object access,

7:53which if I go back here to audit policy,

7:56you'll see we turn that on for our object access. Well,

8:00since this is using group policy, I'm going to close this.

8:02I need to open a command prompt here. There we go.

8:06Cause I need to refresh our policy.

8:08So what I'm gonna do is GP update force and that's just going to force the

8:12system to update the policy that I just changed. All right.

8:16And that way we'll be able to show you something here in just a minute with our

8:20object access. Okay. That took about 45 seconds or so. And there it goes.

8:25It's done. All right, super. So I'm gonna close that.

8:27Now I'm going to go back to my security logs.

8:32Let's see. Is that it? There we go.

8:35I'm going to open up the event viewer again.

8:37I'm going to do a quick refresh on security.

8:41Okay. Refresh. Okay. Ooh, here we go.

8:45Here's some object access stuff happening now. Okay.

8:49And if you look down here, this is the object name.

8:52So it tells you what's accessing it, who's accessing it.

8:55So what I'm gonna do is just minimize this for now.

8:58And here's what we're going to do. I'm going to,

9:00cause I actually have to go back to the test

9:05folder. So let me close that out. Uh,

9:09let's see. This is Active Directory User Computers. I don't need that.

9:12I'm going to right click on this and go to properties because you also have to

9:15turn object access on, uh, on your files individually, or well,

9:20your directories. Okay. And it'll flow down from there.

9:23So what you do is go to security, go to advanced.

9:26Then you go to auditing right there. And you're going to say,

9:29add select a principal. We're going to say everyone. There we go.

9:34Okay. And I'm going to say full control.

9:37So anytime they do any of this stuff with that file and all sub files,

9:41because by default, let's see. Yeah. Right here.

9:44It says only apply these audit settings to objects and or containers within this

9:49container. All right. And that's fine. There we go.

9:53Say, okay. And okay. Again. And okay. Again.

9:56Now auditing is turned on here.

9:58So if I go into test and I open doc and I close it and I open doc

10:03and I close it and I open doc and I add some data to it,

10:07just like that file save and close. We'll close that.

10:12Okay. Now let's go back to our logs. There we go.

10:15Let's do a quick refresh on security. There we go.

10:19And file system. Okay. Well, this is different. Let me do another refresh.

10:24It does take it just a couple seconds. Okay. There we go. Okay.

10:28So we see that. And what I'm looking at is right down here,

10:32looking to see what was accessed. So there we go. Uh,

10:37systems, more stuff like that. Oh, that's log off file system.

10:42Uh, there, Oh, here we go. Now that's windows explorer. It's me. Explore,

10:46explore. Uh, let's see.

10:52Let me do another refresh and now it's showing up right here.

10:56There we go. So this is the administrator. That's me, uh,

11:00accessing the desktop test directory. There it goes.

11:04And there's a test again, test again, test there's my doc one where I accessed it.

11:09Okay.

11:10So you'll see you can turn on lots of auditing features or logging features in

11:15windows.

11:16And if you want to enable object access and record when people log files,

11:21you not only have to turn it on in the background. So it logs it.

11:24You have to go to individual directories and it is inherited.

11:28So it does go down the tree, but you need to turn it on there as well.

11:32So that is windows logging in a nutshell. Next up,

11:35we're going to jump over to our Linux or Ubuntu host over here and take a look

11:39at logging in Linux. See you there shortly.

0:00All right, here we are on our Ubuntu host and I want to start off with a little disclaimer

0:05saying different Linux distributions work differently. Okay, and we're going to be

0:08focusing on Ubuntu here, which comes from Debian. It's a distribution from Debian.

0:15All right, so just as a little disclaimer. Okay, so within Ubuntu we have two different logging

0:21systems. The first one is systemd-journald. Okay, and it collects and stores journal events. We use

0:29the journalctl command. So let's bring that up here. Let's open our terminal here. So we access

0:36that journald logs from systemd via the journalctl command. Okay, and so if I hit enter,

0:47it's going to print out a bunch of information in this journalctl command. Let me actually maximize

0:51this. There we go. It's going to put out things related to kernel messages, startup processes,

0:57system services, apps, all those things. Okay, and you can kind of just scroll through here and

1:02see all this information. There's a lot of information here. Hit q to stop. Okay, and then

1:07we're going to clear the screen. So let's take a look at different commands. So journalctl is the

1:13base command. You can do dash r to list it in reverse. So you're going to see the most recent

1:21activity. All right, so things like this. Okay, I hit q again and clear. Get us back up to the top.

1:30There we go. Okay, I can also use, instead of minus r here, we can use dash n 20, and this is

1:42going to show you the last 20 log entries. All right, so this will give us the last 20. There we go.

1:47Okay, and we'll clear again. We could also use, let's go here, dash f, and what this does is it

1:57creates a stream. So I'll hit enter, and it's going to, as things happen on the system, it's just going

2:04to show the events in real time. So I'm going to go over here and click on these Firefox and watch

2:10what happens. If I click it, there you go. It's starting to do things now, right? We've got new

2:14logs. So this allows you to kind of view a live stream of the logs. All right, I'm going to hit

2:22control c. Go ahead and stop that. Clear it. We'll try to spell clear properly. Turn my caps lock off.

2:29There we go. Oh goodness. Okay, now you can also use grep, and if you're not familiar with grep,

2:35it's a Linux search term. So you could do journalctl pipe. You're passing that whole journal

2:43log over to search. So I could then search for something like Firefox. Okay, and this is only

2:50going to give me the log entries that include Firefox, and you'll see there's a lot of them.

2:55So let's stop that. All right, control c. There we go, and clear. So that is another way you can

3:04search through those. Okay, so that was, let me type this out, so you can see systemd-journald.

3:12Okay, so that's one logging system on Linux. Now another logging system on Linux is rsyslog,

3:21and rsyslog is used to send data into files, and most of them are located in the var log directory.

3:28So if I was to do an ls var log, you're going to see a bunch of log files in here.

3:35There's alternative support, auth log, boot log, bootstrap, btm, dmessage, cloud, dpick,

3:41kg. You know, there's a ton of them in here. syslog, ubuntu. There's a bunch of log files.

3:47Well, that's because it is actually shipping those logs via syslog into these files. All right,

3:55so let's say I wanted to search my auth log for failed logins. What I would do is start grep,

4:01and I would do dash i, and then put failed in quotes, and do var log auth dot log, and I can

4:11search that log for the word failed, and there we go. We see there are some entries there.

4:16So that's one way that you can look through those. Now, yes, this is why something like a sim

4:23is so helpful, right? It brings all our logs into one place, lets us easily view them.

4:28So let's take a look at the seven most common Linux log files. I'm going to bring this in

4:33so we can take a look at this. So here we have var log, and all these are var log. We see

4:39we've got your syslog. That's just general system and services activity. Auth log dealing with your

4:44authentication. Kern log dealing with kernels and hardware and firewall messages. Boot,

4:50it's events generated during the boot and service startup failures. Your audit log

4:55is detailed security events collected by the audit d process. History dot log,

5:01packages that are installed, that's your software that you install, upgrade, or remove through the

5:06apt package manager. And then ufw dealing with allowed or blocked events in the Ubuntu firewall.

5:14So these are just seven of the most common. I'll include this below the nugget here. Okay,

5:19so now that we know that, we've got log files in here. Okay, we use journalctl to search journal

5:25logs. We can search through these logs. Okay, we know where they're at. But let's now look at log

5:32settings within journald. We're going to go back to journald. So here's what we're going to do.

5:36I'm going to take a look at something real quick. I'm going to do lsetc systemd, just like that.

5:43And let's see what's in here. I want to see there is a journal conf right here. See this file. So

5:51let's take a look at that file. So here's what we're going to do. This is for our journal settings.

5:56So let's do this. Let's do a nano, which is a text editor. And let's do etsy systemd journal

6:06dot conf and enter. So here are the settings for our journal d. So if we scroll down in here,

6:12and this is really kind of a master file of the settings. You actually have to go through and

6:16create a folder and put settings in there. But I just want to show you the settings. That's really

6:21what we're getting into. I'm not teaching you how to configure all your Linux logs.

6:25But what we have here are various settings. They're all commented out, which means they're

6:28not being used. But let's just talk about a couple of these real quick. Like storage here,

6:33it's just set to auto. So it should then try to save the logs and be persistent through storage.

6:40You can use compression. There's lots of rate limiting information in here.

6:47Let's see what else is in here. Let's scroll on down here. Here we go. Max file retention.

6:54Max file second one month. So right here, it's going to save those logs for one month. Also

7:02looking for max level store. Max level store right here. So this is set to this is your zero through

7:08seven. This is debug. So that's seven. We definitely wouldn't want to debug that log every

7:13little thing. Then there's also a system max use in here. And let's see, where is that? That should

7:20be up here somewhere. System max use right here. System max use. And what this does, this is going

7:27to define how much space can be used for your journal delogging. Okay, so these are just

7:33settings. So you'll see in here, this is kind of what it looks like. So let's go ahead and

7:39close out of here. Yes. Yes. Close terminal. Okay. Let's open a new terminal and start fresh as we

7:47start looking at our syslog. Now this is different. So what we're gonna do is do ls etsy rsyslog.d

7:56Enter. And we have here 20 ufw.conf, 21 cloudinit.conf, and 50 default.conf. Now what these are

8:05is it goes in order and it looks at these. These are logging information. So here, this is the

8:09configuration for firewall logging. This is the cloud initiator logging. And this is your default

8:14settings in here. Okay, so let's take a look at those default settings. So we're going to do is

8:19etsy rsyslog.d slash 50. Tab that out. Default.conf. Enter. And here you go.

8:29So, oh, I counted that. I should have nano'd it. Let me hit up arrow and

8:33press A. And let's get rid of that. And let's go to nano. So we can see this a little better.

8:40Okay, here we go. So here we can see the auth and priv. Those are going to the var log auth.

8:47And then here we have additional going to var log system. Your kernel right here going to var log

8:54kern. Any mail going to var log mail. And you see these others like cron. If you just edit this,

9:00see I don't have, it's not writable. It's a read only at this point. But if you go and edit,

9:06you can then enable these things. And these are the settings here. And so these are where we can

9:12define where to log to. Now, if you wanted to create retention policies and stuff like that,

9:18that requires actually creating a retention policy file for each of the various log files.

9:26Okay. So you can do that as well. So this is kind of the basics of Linux logging. So we got to see

9:32some Windows, some Linux. And next up, we're going to take a look at syslog. So I'll see you there

9:36shortly.

0:00All right, welcome back folks. Now that we've got our Windows and our Linux logging out of the way,

0:04let's take a few minutes to look at syslogs. And that's because generally your network devices

0:09are going to be using syslog to ship the data most of the time. But Linux also uses syslog.

0:16Okay, so what syslog is, it is basically it's a standardized shipping method. Okay,

0:26there we go. Method or really it's a protocol. I should put that in. Protocol with a specific

0:32format. So it has a format. Okay, and that's what we're looking at here. So let's start here and go

0:38through one of these. So again, remember with syslog you have zero through seven. Zero meaning

0:44emergency. I'll change colors. Emergency, right, that's our zero. And let's go back to blue over

0:50here or let's go green or debug. That was our seven. Okay, so those are our numbers there. So

0:57let's look here at one of these log entries. These are coming from a Cisco ASA and 165. Let's see

1:04here. This 165, it's a priority field and it's calculated using the facility and severity.

1:13Okay, Bob, you're throwing new words at me. What are we talking about here? Well, severity

1:17is this number here. Okay, that's the severity. The facility, I'll put that out here. Facility,

1:24there we go. What that is, that's where the log came from. So where, get my E in there. It

1:31came from, and generally this is a type of service on the system. So it might be from the kernel. So

1:39it'd be like Kern or it might be from user or from mail or daemon of some sort. That's what we're

1:47talking about, the type of service it's coming from. All right, so it's the combination of those

1:51two that you get your priority over there. All right, so let me get rid of this here. All right,

1:57let's continue here. So then we have a timestamp. Hey, you got to have a timestamp, right? So we

2:02know exactly what time this happened. All right, so that's our timestamp. Then we have here edge

2:09ASA. That's the host name that it came from. So that's your host that it came from. All right,

2:15and then we see ASA-5-111010. Well, this means it was generated by a Cisco ASA. We've got our ASA

2:23there. The five, that's the severity of our logging, which is, if we were to look back at our

2:30chart, it's notice. There we go. That's that level. And then we have 111010. That is a Cisco

2:37message ID. Just like in Windows, we have an event ID, and in Linux, it has an event ID or code.

2:42Basically, that's what that is. All right, and then after that, well, that's the message. This

2:48over here in the brackets all the way down to here, that's just your message. That is what was

2:53sent. Okay, so we have a severity here. So we have a priority. We have a timestamp. Let me change

3:00colors here. Timestamp, a host, and then right here, this is key. This is going to tell us our

3:06logging level and the event ID there. All right, and then, of course, you have your message. Oh,

3:11and then down here, just a little notice about the facility and also our severity.

3:18Okay, so now that we see what this looks like, let's take a look at how to configure it on an

3:25ASA. Now, you don't need to know this, okay? This is not a Cisco course. I'm just giving you

3:31what it looks like, just so you have an idea. Don't try to memorize this. Don't worry about

3:35that. I just want to show you what it looks like, okay? Because we went through the Windows settings

3:41and the Linux settings. I want to show you a network device setting. So here you go.

3:45So you enable logging. Logging enabled. That makes sense. Logging timestamp. You're telling

3:50it to add a timestamp to your log entries. Then we're saying to save this to our little buffer,

3:55okay? And it's going to save at the five level, which was our notice. But not just that. It's

4:02everything below. So we have zero, one, two, three, four, five, six, and seven. So if you're

4:08saying log at this level, it includes all of the lower levels as well. So there you go. Logging

4:16buffer size, we're just saying how much space to give to the log. And then here we're telling it

4:20to wrap. So once it reaches a full buffer, to go ahead and save it to a different place in memory

4:29before overwriting, so that we keep a copy, okay? And then here you're just telling it how much to

4:34save. And the thing about Flash is, oftentimes in network devices, Flash is persistent. So it

4:42persists through a reboot and it's not lost. Okay. Then, well, what about shipping my logs off? Well,

4:48here you go. Logging trap notice. So we're saying basically a trap is when we send data. That is

4:56known as a trap. So basically once a certain event has occurred, go ahead and do this. So here what

5:02we're saying is send five and below to our logging host. And then down here, we're defining our

5:08logging host. This is the interface. This is the IP. And then we're saying use UDP 514, which is

5:16syslog. Okay. And we're just saying where to send it. So this is a glimpse as to what you would do

5:21to configure a device. Now there's GUIs and stuff as well. I just want to kind of give you a little

5:26insight, a little behind the scenes as to what it looks like. Now, the thing is you really do need

5:30to understand the syslog formatting. You want to know that so that you're able to identify maybe

5:37the host name or the event ID or the syslog value there. Pretty important. Okay. All right. So guess

5:46what, folks? That wraps up this skill. Well, yeah, it sure does. Except for, that's right, our

5:53validation challenge and it's coming up. I hope this has been informative for you and I'd like to thank you for viewing.