Design an Azure Logging Solution

0:00[AUDIO LOGO]

0:05Welcome to the AZ-305 certification exam,

0:09designing Azure solutions.

0:11This is a major part to a cloud architects suite of tools.

0:17What are we really talking about here

0:19before we actually dig into the stuff

0:20that this particular content is about?

0:22When we talk about design solutions and design

0:26certification exams, what we're really

0:28trying to figure out here is how well do you

0:31know the individual products and their features

0:34within the Azure ecosystem and how do they all

0:38fit together as puzzle pieces to build a full fledged end

0:43to end solution.

0:44That's what the certification exam is all about.

0:46It's not about implementation.

0:49It's not about where do you go to click this button, to make

0:52this thing happen.

0:53And it's not about what PowerShell commandlets

0:56do you actually use to implement this given solution.

0:59No, instead.

1:00It's in your particular situation,

1:03you should use this particular feature in conjunction

1:07with that particular feature to arrive

1:10at this full fledged solution.

1:12This keeps in mind things like high availability, disaster

1:16recovery, even scalability, how quickly can we spin up or tear

1:21down new resources.

1:23Beyond that, it also keeps in mind cost.

1:25Cost is a major factor when it comes to using cloud resources.

1:29If you're on Twitter at all, you no doubt

1:31see a lot of Cloud Billing memes flying all over the place

1:36because it's very easy to rack up a huge bill

1:39if you don't understand how billing and the features

1:42work within them.

1:43Now this particular set of videos

1:45is focused in, first, on designing a logging

1:49and monitoring solution.

1:50Now I know that out of the entire scope of all

1:53of the things that we're going to cover in this set of videos,

1:56logging, monitoring, even governance and access are not

2:01the most exciting things on the exam.

2:04But here's why I would argue that they

2:06may be the most important.

2:08It's because designing a solution has an end.

2:12It has a finite endpoint once it's designed.

2:15Implementing a solution has an end.

2:18It has a finite endpoint.

2:20It's once we've clicked the buttons

2:21and now the resource is spun up and working.

2:23But monitoring, logging, and governing a solution

2:28is something that is ongoing for a very long part,

2:31for a very long time.

2:33And giving our employees a resource

2:37that they can go to to monitor and watch and troubleshoot

2:41our resource after it's deployed,

2:42that's something that could actually

2:44have a huge impact on people's entire career.

2:47That could be their entire job is to keep the resources up

2:50and running once they've been deployed.

2:52The monitoring and logging solution

2:54is a major part of that.

2:56Now for the most part we focus in on Azure monitor

2:59in this set of videos.

3:00But there are other ways that we can get insights

3:05into what our applications are doing,

3:07depending on the resources that we actually deploy.

3:09So without further ado, what we're

3:11going to do in this set of videos

3:13is focus in on the monitoring and logging solutions, what

3:17are the options available to us, and why would we

3:20configure them in certain ways.

3:22Let's go.

0:00[AUDIO LOGO]

0:06The first thing we're going to do is talk about Azure monitor.

0:09And in particular, we're going to talk about

0:11where does Azure monitor get its data

0:14and what can we do with that data?

0:16Really Azure monitor collects two types of data.

0:20The first one is actual log records themselves.

0:23The second one is metrics, like streaming

0:27statistics or telemetry on the resources that we build.

0:30So what I'm going to do is I'm going to jump into the Azure

0:33portal and we'll explore Azure monitor to get a better

0:36feel about where this data comes from

0:39and what can we do about it.

0:40Let's go.

0:42So when it comes to monitoring your resources that you have

0:46deployed in Azure, you are no doubt

0:49going to begin your journey right here with Azure monitor.

0:53Shows up just as monitor from the Home page.

0:56If you don't see it on your Home page,

0:58simply search for monitor.

1:00And it looks like this.

1:01It's got a little gauge on it.

1:03And if you give it a click, it takes you

1:04in where you can see there are a whole lot of ways

1:09that you can leverage Azure monitor

1:11in your entire environment.

1:13Now insights are there entirely own separate thing.

1:18We've got its own dedicated video coming up to it.

1:21In fact, Insights are not something

1:24that are deployed by default, there

1:25are things that you actually have

1:27to go in and turn on and sometimes

1:30even configure on their own.

1:32Instead, what happens mostly by default with Azure monitor

1:37whenever you deploy your Azure resources,

1:40you have these two items right here that I'm circling,

1:45metrics and logs.

1:47Now here's what I want you to think about these.

1:49What are the two different things?

1:52What are the two differences between metrics?

1:54Metrics, if I were thinking just in a basic Windows viewer here,

1:59when I think of metrics, I am thinking

2:01of something like the actual.

2:03When you bring up the actual Task Manager, let me go ahead

2:06and bring in a task right here, Task Manager.

2:08And I go to the Performance tab.

2:12Let's bring it over here and I go to the Performance tab.

2:15This is what metrics is.

2:17Well, of course my computer is going

2:18to freeze as it comes on there, it goes right there.

2:20This is metrics.

2:21Metrics are near real time live streaming data

2:27of what's going on with the various components or guts

2:31within your environment.

2:33Whenever you deploy things, like anything that would consume,

2:37any type of compute resource in Azure,

2:41that's a virtual machine, that's a function app,

2:43that's even a logic app.

2:45Things where you actually monitor app services--

2:50app service plans.

2:51That's basically extracting away the virtualization.

2:54Even containers have their own metrics

2:57that we need to keep up with.

2:58Whenever we actually start to think about the types of things

3:02that consume--

3:03compute consume resources and we need

3:05to make sure they're behaving appropriately and understanding

3:10the impacts that they could have on our billing

3:12performance of the applications, we

3:15go to metrics because metrics are the live streaming

3:18items here.

3:19One of the cool things about metrics

3:21also is the fact that we can very quickly hook in metrics

3:26with alerting systems here.

3:28So if we actually need to be proactively monitoring

3:31and getting alerted to when our metrics fall

3:34outside of a range, this is when we can configure alerts.

3:39And Azure monitor serves as a landing zone

3:43for all of the metrics in our entire ecosystem.

3:47This is not just one resource that we

3:49can monitor from within metrics, we can monitor all of it.

3:52Similarly, we have logs.

3:54Now what is the difference between logs and metrics?

3:57Well, if I were to say if this were metrics where we're

4:00actually live streaming my CPU here,

4:03then logs would be equivalent to my Event Viewer.

4:07So if I actually bring up the Event Viewer right here

4:10and I look at things like my application logs in my Windows

4:13logs, it's going to launch up.

4:14Here you see all of these.

4:15Every single time something happened in my environment,

4:20it wrote a log to my Event Viewer.

4:22This is what logs are all about.

4:25And both of these, by default land in Azure

4:29monitor, so long as we configure them or turn them on.

4:32And for the most part, they are turned on by default,

4:35it's a really important thing to wrap your head around.

4:39The easiest example to use is spinning up a virtual machine.

4:44When it comes to a virtual machines metrics,

4:47the virtual machine deploys with something

4:50called a diagnostic extension.

4:53This is an extension that gets installed

4:56on our virtual machines.

4:58This allows the extension to grab metrics data, basically

5:02statistics like we just saw coming out of my Task Manager

5:05and stream it into Azure monitor.

5:08When it comes to logs, logs are slightly different.

5:11There is a monitor logs agent that

5:16gets installed on a virtual machine that

5:19streams the log data in right here into the actual logging

5:24environment.

5:25Now again, we did talk about something

5:27up above that I just glazed over.

5:29There is something called Insights

5:31that is its own separate agent that we could install.

5:34And it'll install, it will actually

5:36stream Insights into the mix.

5:38So if metrics are great for actually looking

5:41at live streaming telemetry about how

5:44my resources are performing, what are logs good for?

5:47Well, with logs, we're actually logging

5:50all of the events that happen on our resources.

5:53Within Azure, we even have the ability

5:56to create our own applications.

5:58It's actually the point of the cloud after all, isn't it?

6:01We write our own applications and let the cloud hosting.

6:05Well, with that, we can create our own logging solutions too

6:08and we can tell our applications log your data

6:12to the actual Azure monitor logs,

6:14if that's what we want to do.

6:16Right here, this is the big win.

6:19Once it's logged into the actual logs environment,

6:23we can write our own complex queries

6:26to filter out or grab log events as they happen

6:31or if we ever need to go back historically and find out

6:35specific information that happened in the past.

6:38And we can even leverage those specific queries to trigger

6:43alerts, that's right.

6:44Metrics and logs can both be used to trigger alerts.

6:48That way you can be proactively notified whenever issues

6:52come up in the system itself.

6:54And that's another major, major win

6:57that you may not have realized actually

6:58existed within the Azure offerings themselves.

7:02So if we were to break down what are the all resources that

7:06actually log to Azure monitor.

7:08You actually want to break them down at the actual Azure

7:12hierarchy level.

7:14For instance, we may have an Azure tenant.

7:19Tenant level data.

7:20Tenant changes themselves can be logged to Azure monitor.

7:24Within that we may have different kinds

7:27of subscriptions.

7:28That's right, subscription level data

7:31can be actually monitored and logged

7:33into the environment themselves.

7:35We also have-- as part of both of these,

7:38we actually have Azure AD, Active Directory information

7:42and changes can be logged into the actual Azure monitor

7:46information itself.

7:48We of course also have operating system information.

7:52This is both for IaaS and PaaS information, whenever we deploy

7:59things like virtual machines.

8:01We've already talked a little bit about that.

8:02We know that they're going to install

8:04different kinds of agents, whether we want to do metrics,

8:07logs, and insights.

8:09Of course, PaaS, this could be things like again Azure app

8:12services.

8:13There's still an operating system under the hood.

8:16We've just abstracted our need to manage it away.

8:20We can still get the metrics from that underlying operating

8:23system and stream them and log data into the hood as well.

8:28That's a big win.

8:30So then from there, we can actually

8:32look into the individual resources

8:34that we deployed underneath the operating system.

8:37If we were actually deploying an App Service, a logic app,

8:40a function app, something like that,

8:41we could log and monitor our metrics

8:44from our resources themselves.

8:47And then beyond that, we actually

8:49have the application itself that we can monitor via metrics

8:54and logs.

8:55Again, that's something that I just hinted on.

8:57When you have the ability to write your own code,

9:00you then have the ability to tell

9:01it to log to actual Azure resources themselves.

9:05And of course that comes with using

9:07just the plain old Azure SDK.

9:10So if you're developing an, say .NET,

9:13you can use the Nugget packages for Azure monitoring.

9:16And just tell it to log straight to the Azure monitor

9:19application.

9:19And the SDK just does it for you automatically.

9:22Of course, if you're using other frameworks or languages

9:25like Python or Go, those SDK exists as well too.

9:29All you have to do is just add that particular module

9:32or that library into your SDK or into your application.

9:35And there you go, you're monitoring and logging.

9:38So when it comes to actually using Azure monitor,

9:41we've got something called insights

9:43which we haven't really talked about,

9:45which gives you deeper understanding of what's

9:47going on.

9:48We have the ability to view our metrics and our logs right

9:54here.

9:54That's one way that we can visualize our data.

9:56Workbooks are another way that we

9:58can create interactive reports to actually visualize our data.

10:03And we'll take it even a step further

10:04and say that we can actually push this out

10:07into Power BI dashboards if that's what we want to do.

10:10It's worth pointing that out there.

10:12We also have the ability to use alerts

10:15to respond to events that happen within our metrics or our logs.

10:21And I'm also going to push this out a little bit

10:23and say, we can even configure auto scale events in the event

10:28that a certain threshold of metrics or logs

10:31have been triggered as well.

10:33So we can dig into this.

10:34We can actually analyze what's happening.

10:36We can visualize what's happening

10:38and we can automate even the response

10:41of what's happening within our Azure environment.

10:44So this is understanding at a very high level what

10:48Azure monitor is all about.

10:49It does a lot of really cool things for us.

10:52In the next few videos, we're going

10:53to dig deeper into what else we can do with the Azure

10:57monitoring resources.

10:58I hope this has been informative for you

11:00and I'd like to thank you for viewing.

0:00[AUDIO LOGO]

0:06Next up, we're going to focus in on Azure Log Analytics

0:09workspaces.

0:10This is a very commonly deployed resource.

0:13Whenever we want to collect things like logs

0:16from any of our resources, lots of times

0:18you see this being deployed with virtual machines,

0:22alongside virtual machines.

0:23Because as virtual machines, servers create logs,

0:27we need to put them someplace.

0:29That's what a Log Analytics workspace does.

0:31The very important concept to wrap your head

0:33around is that the Log Analytics workspace

0:37is a resource within Azure.

0:39And with that comes governance, like we

0:42do with other Azure resources.

0:44So let's unpack what that really means in this video

0:47and explore the Analytics workspaces a little bit more.

0:50Let's go.

0:51Now we're going to focus in on the Log Analytics workspace.

0:55Whenever we actually use Azure monitor to collect logs,

1:00a Log Analytics workspace is a resource

1:03that gets deployed with it.

1:05And that is what we're currently using.

1:07Now what you need to think about a Log Analytics workspace

1:10really is it's basically Azure table storage.

1:15We're taking resources that can generate

1:18logs and streaming those logs into storage that we can then

1:23query after the fact.

1:26Now the fact that this is a resource, very important thing

1:30means probably the biggest aspect

1:33of the Log Analytics workspace is we can govern access to it.

1:38There's really two aspects to the whole thing that make a Log

1:42Analytics workspace special.

1:44And that is the fact that we have

1:46first of all, a boundary of who can access it

1:50as well as a geographic location for the log data itself.

1:55We're not restricted to just having one Log Analytics

1:59workspace.

2:00We're going to have multiple Log Analytics workspaces.

2:03So that way, if we really needed separation of logs

2:07where they're stored for whatever reason,

2:10whether it's latency, whether it's

2:11internal policy, whether it's access controls, whatever

2:15the case may be, we can actually deploy the resources

2:18that do exactly that.

2:20But the big one really like I just said,

2:22it all comes down to governing access to this.

2:26We may have our global admins.

2:29They need access to all of the data that's

2:32contained within Log Analytics.

2:35But then, we may have a group of virtual machine admins.

2:40And we have virtual machines that we spin up within Azure

2:45and they stream their telemetry, their log data

2:49straight into the Log Analytics workspace.

2:52Outside of that, we may have Azure app services admins

2:58and they do what you think the name is.

3:00They administer the App Service resources

3:04that get deployed within Azure.

3:06So I'll deploy--

3:07I'm just going to put AS right here for App Services.

3:11And they generate their own logs and stream data

3:14in to the actual environment itself.

3:18The cool thing about this is even though this one Log

3:21Analytics workspace contains my virtual machine logs and my App

3:27Service logs, I can now say, OK.

3:30App Service admins are only allowed

3:33to access the logs that were generated by the App Service

3:37resources themselves.

3:39And virtual machine admins are only

3:43allowed to access the logs by the virtual machines

3:47themselves.

3:47They call this the resource context.

3:52And that's why it's called resource context logging

3:55or resource context controls.

3:59The cool thing about it is now we can actually

4:01govern who has access to which logs within a Log Analytics

4:06workspace.

4:08Now when it comes to actual deployment of the Log Analytics

4:11workspace, like I said, this is really

4:13table storage which really means we're really just looking

4:16at a storage account under the hood that we're working with.

4:21We can deploy a Log Analytics workspace in conjunction

4:24with a premium blob.

4:27This is basically Azure data lake Gen 2.

4:30If we have very large amounts of data

4:33that need to be accessed and processed very quickly,

4:36premium blob storage will get the job done.

4:39Otherwise, we could deploy a generic storage account,

4:42which gives us access to hot, cold, or archive storage tiers.

4:48That way, depending on the types of logs that we're storing,

4:51we could determine if these need to be hot

4:53logs, cold logs, or even stored in archive storage.

4:56Beyond that, we can set immutability settings.

5:00Basically meaning if we have legal holds in place,

5:04we can put legal holds.

5:06I got to fix my pin there, legal holds on our archive storage

5:12as well as set time based retention policies for our Log

5:17Analytics workspaces too.

5:18So a lot of the features that come with storage accounts

5:21are actually applicable here to Log Analytics workspaces.

5:25Now like I said, data that gets stored in a Log Analytics

5:29workspace is placed into tables.

5:32In fact, if you want to see the tables under the Settings

5:34section, now in the Preview we can actually

5:37see the tables that could generate it.

5:39Now as we add more resources into the Log Analytics

5:43workspace that are generating their own logs,

5:46they're going to create their own tables.

5:48You can see here though these are Azure tables that

5:50are stored here, aren't they?

5:52So we can see things like VM connections,

5:54VM processes, VM compute.

5:56And you would think about these just like Windows event logs.

6:00Windows event viewers that would be storing compute information,

6:03connection information, process information.

6:06In the actual Event Viewer, they would be stored here

6:08in the actual table storage.

6:10You can also see under the workspace data sources.

6:13If I click on Virtual Machines here,

6:15well, this is where those virtual machines

6:17would be listed If we actually ended up having them.

6:19But we don't have them right now because this is just

6:22an empty landing zone as it stands.

6:25So what are the limitations to these tables or the storage

6:28accounts?

6:29Well, there's not a lot of limitations to it.

6:32By default when we deploy these, the data

6:35is stored on a cluster.

6:38If we ingest more than, greater than 4 terabytes

6:43of data, here's the big kicker right here, per day

6:47then we need to deploy our own cluster

6:50to actually host our own data.

6:51Otherwise, Azure itself is going to be

6:54able to physically store this data in the dedicated

6:57workspace on its own.

6:59So when it comes to an actual design

7:01for deployment of a workspace, what are the big considerations

7:04to keep in mind?

7:05Well, let's say I deploy my virtual machines in the South

7:10Central US data center.

7:12Not a big deal, just the standard deployment there.

7:15If I deployed my Log Analytics workspace in

7:21say the UK, what could be the problem using here?

7:25Well, of course, we've got a latency issue

7:27that could take place just by default.

7:29Even though latency between Microsoft data centers

7:32is very quickly.

7:33But even the bigger issue here is we

7:35could incur outbound data charges.

7:40That's a big issue here.

7:42Ideally, we want our Log Analytics workspaces

7:45as close to the resources that they're

7:48monitoring as humanly possible.

7:50That way we have the lowest latency as well

7:53as the outbound data charges are suppressed that way.

7:56Some of the issues that could come up

7:59is if we're actually deploying applications

8:03within governments or geographic regions that actually have

8:08data sovereignty or compliance issues, that

8:11would dictate exactly where log data needed to be stored.

8:14That could be a challenge for you

8:16that you may want to keep in mind.

8:18But the whole idea is to avoid these outbound data transfer

8:21charges.

8:21So that way we keep the workspaces in the same region

8:25that the Azure resources that are being managed are.

8:28That's the big thing that we want to keep in mind.

8:31So when it comes to that, when it comes to the actual resource

8:34deployment, we want to keep it close to the resource itself.

8:37But then it comes down to--

8:39what are the actual permissions of who can access this?

8:44At first we have a workspace context.

8:49This means that for each individual user or groups

8:54of users, they would get their own workspace.

8:57So in the previous example, I talked about my VM admins.

9:01Would it make sense for me to actually deploy

9:03a workspace that's just dedicated to virtual machines?

9:06Maybe it depends on your particular environment

9:09and how your internal policies in governance are set up.

9:12It's a very clean way to do it.

9:14All virtual machines go to the virtual machine workspace.

9:18All Azure App Services go to the App Service workspace.

9:21Kubernetes clusters go to the Kubernetes workspace.

9:25Makes sense.

9:25But then we also talked about the possibility

9:28of there being a resource context,

9:31meaning we can have groups of different resources

9:35all log to the same Log Analytics workspace,

9:38again maybe that's the cost, maybe that's

9:41to be within our compliance of our own internal policy,

9:45our government policy.

9:46And then we set up permissions or access controls, basically

9:51our back-access controls, for each of the groups

9:55to have permissions only to the data

9:57that they actually need to have access to.

10:00So this is getting to know the power of the Log Analytics

10:03workspace.

10:03This is a space where we can actually

10:05dig into the individual logs that have been logged by each

10:09of our different resources.

10:11Virtual machines, storage account, system center,

10:14applications themselves, even automation

10:17can all log here within the workspace.

10:20We can put this in any region that we really desire.

10:24And we can govern access to the individual logs

10:28based on the generating resource, that's

10:30the resource context.

10:32I hope this has been informative for you

10:33and I'd like to thank you for viewing.

0:00[AUDIO LOGO]

0:06As your Insights is a catch all term,

0:09that's not a one resource that you deploy.

0:12Really what happens is when you deploy resources or even

0:17applications straight to Azure themselves,

0:19you have the ability to collect additional information

0:23and telemetry from the resources or applications

0:27that you just deployed.

0:29Insights grab that telemetry and put it somewhere, streams it

0:33somewhere so that you could unpack it

0:35and digest it a little bit more.

0:37Now like I said, it's not just Azure Insights,

0:41it can be things like virtual machine Insights or Application

0:45Insights or even something like function app Insights.

0:48That's what we're going to take a look at.

0:50This video is what are the different Insights

0:53that you can deploy within Azure and when might you use them.

0:57Let's go.

0:58Now honestly we're going to start shifting to the more fun

1:02things to talk about when it comes to our monitoring

1:05solution.

1:06I'm back here on Azure Monitor.

1:08And you can see that I've gone ahead

1:10and I've jumped into the Workbooks.

1:13What are workbooks?

1:14Workbooks are predefined queries that come off the shelf.

1:19Now it's worth pointing out, you can custom

1:22create your own workbooks if that's what you want to do.

1:25But for people who are like, hey, I

1:28need a quick solution that can leverage my logs and my metrics

1:32and just give me a quick visual as to how my infrastructure is

1:38performing, workbooks are a great win for you.

1:41Now behind the scenes, I've gone ahead

1:43and I've deployed a very basic virtual machine.

1:46Let me jump over here and show you real quick,

1:48it's called monvm for monitor virtual machine.

1:52And then I jump down here into the monitoring section.

1:56And under Insights, I went ahead and I enabled Insights.

2:00When you first click on this, it's disabled by default,

2:03so I enabled it.

2:04Then under logs, I did the exact same thing.

2:07I enabled the logs as well.

2:09So I just want to click the Enable Click to confirm

2:12and let those get started.

2:14It takes 5 to 10 minutes for the agents

2:17to be installed in the virtual machine and for data

2:19to start streaming in.

2:21But with that being done, now I can

2:23start to leverage some of my other more important things

2:27like actually looking into the Insights as well

2:30as the workbooks themselves.

2:33So let's take a careful look at what workbooks

2:35can bring to the table.

2:36Of course, we have the Quickstart

2:38which is just an empty workbook where we can actually

2:41run queries and visualize the queries.

2:43But if I scroll down here under the Virtual Machine section,

2:47there are four workbooks that we can very quickly click on

2:51by default. We can look at things like Key Metrics.

2:54Then we've also got a performance workbook

2:56which tells you the basics, CPU, memory, disk, and network.

3:00We've got the performance counters

3:02which actually leverages the metrics collection itself.

3:05And then we even have an availability.

3:08You could almost think about these like dashboard.

3:10Let's click on the Performance one here for a second.

3:13And if we were collecting a lot of data in here,

3:17you would see, let's go ahead and change the subscription

3:19to pay as we go.

3:21And we'll choose the workspace that I have.

3:23This one is my EUS workspace.

3:25And then click off that.

3:28And it loads my man VM up right here on the screen.

3:32So as you can see, when I enabled Insights and logs,

3:35of course, it immediately began logging data

3:38to my Log Analytics workspace.

3:40Well, with Workbooks, we can now use this pre-built item

3:44to query the data that's in my Log Analytics workspace

3:48and generate a quick report for me to actually see.

3:51We even have a little visual here that increments over time.

3:55You see I've got two little bars here

3:57at the very end because this resource is relatively new.

4:01So we can see some really interesting information

4:04right here within my virtual machine.

4:07Now this counter that we're looking at

4:09is my available megabytes here in RAM.

4:12But if I wanted to look at things

4:13like what is my logical disk, my free space,

4:15my reads and writes, my network utilization, my CPU heartbeat,

4:19I can click this dropdown right here.

4:21And you can see it immediately changes this like so.

4:25We can also see the aggregators that are used here.

4:28So if I want to look at this basically

4:30as a statistical distribution, I can choose that

4:33as well as we can do things like what is the average as well

4:37or the distribution that we're looking for in the trend.

4:40We also see this top 100 machines

4:42by default. If I just wanted to pick on the top 10,

4:44we would actually be able to drill down

4:46into this in chart form.

4:48That's the beauty of the top 10 machines.

4:51So when we have top 100 machines,

4:53it's going to be in list form, very tabular.

4:55When we look at the top 10 machines,

4:57we would be looking at an individual line chart here

5:00but it would have 10 lines on it,

5:02each color representing one of the 10 machines

5:05that we're looking at.

5:07So Workbooks, great win when it comes

5:09to using something like that.

5:11Availability, same drill here.

5:13This is going to be a really important one that you're

5:15going to want to look at.

5:16You're going to want to look at what

5:17is the uptime and the available hours

5:19and the total hours that it would be available for.

5:22Now in my case, the data is very skewed at the very beginning.

5:26Says this machine has been available for 72 hours,

5:29or has been online for 72 hours but only available for one.

5:33That's not true.

5:33I just spun this machine up about 30 minutes ago.

5:36So the data is a little wonky at first

5:38and it's going to need some time to find its feet.

5:41But there we go.

5:42Why does it say 72 hours?

5:44Because the time range here is set to the last 72 days.

5:47So since this machine has only been up

5:49for one hour of the last three days,

5:52that's why it throws off the metrics right there.

5:55Nonetheless, you can see, wow, that would

5:57be a very useful item.

5:59Now that's just for virtual machines.

6:01As you can see here, we have some other things

6:04that we can monitor within workbooks.

6:06Synapse.

6:07Well, what is Synapse?

6:08Synapse is a SQL server.

6:11Effectively, that's a very, very generous generalization

6:16for all of the things that Synapse can do.

6:18But for the most part, you, the infrastructure minister,

6:22leveraged Synapse apps to serve as a OLAP heavy SQL Data

6:28Warehouse.

6:29So I'm going to put DW right there.

6:31It does, as you can see right there,

6:33also allow you to leverage on Apache Spark cluster

6:37for processing data and querying data

6:40if that's what you want to do.

6:41And in this particular case, it comes

6:43with a couple of workbooks for actually

6:45monitoring your spark cluster if that's what you want to do.

6:48But the primary use case for Synapse, I would argue

6:52is to actually leverage the OLAP SQL Data Warehouse.

6:56Now we also have the ability to look at container clusters.

6:59We can actually dig into our hosted applications themselves.

7:04This is where we write our own code

7:06and stream it into the Log Analytics workspace.

7:09So from there, we can actually monitor our own data.

7:12We can even look into Log Analytics workspaces metrics

7:15here.

7:16We can actually look at our own workspace usage reports

7:19as well as our Log Analytics health, the agent health

7:23status.

7:24Now of course, like I said, we can create our own workbooks

7:27if that's what we want to do or we can even borrow templates

7:31from other users who have created their own templates

7:34in the entire environment.

7:35Really good stuff to look into, you

7:37see things like Azure Cache, Key Vault, storage accounts,

7:41virtual desktops.

7:43There's a lot of other templates that you can build on top of.

7:47And I'd really encourage you to dig

7:49into that when it comes to working with the Workbooks.

7:53But I did say in addition to the workbooks,

7:56we have another monitoring tool.

7:58If I jump back to the Overview, we have this thing here

8:01called Insights.

8:02And Insights are honestly, one of my favorite tools

8:05that we have.

8:06Insights combine metrics with logs

8:11in order to actually dig deeper and ask bigger questions

8:15about the architecture of your applications.

8:19How is your application, your virtual machines,

8:22your infrastructure, your resources performing

8:25and how do they communicate with each other?

8:28How's the communication between resources

8:31interacting and working or having issues here.

8:35With Insights, we can monitor applications, containers,

8:40network Insights, and virtual machine Insights.

8:43But we can also look into resource groups.

8:47We can actually get Insights into resource groups

8:49themselves.

8:50We can look into Azure Cache.

8:53If you've worked with Redis cache before,

8:55it's a very similar thing.

8:57We can work with Cosmos DB, a globally distributed document

9:02database, very similar to MongoDB.

9:05We have a Key Vault Insight where we can actually

9:09get reports on what our Key Vault

9:12requests have looked like.

9:14Who are they coming from?

9:15Who are we giving our credentials or our secrets

9:19out to in the first place?

9:20What are the performance issues that we've had,

9:22any latency between an application

9:25connecting to the Key Vault and getting the response back.

9:29Then we even have Storage Insights.

9:31Again, this is a unified report where

9:34we look at the performance of our underlying storage account.

9:38What is the capacity of our storage account?

9:40How available has our storage been

9:42and how has it been interacting with other Azure resources who

9:46are trying to access that particular storage account?

9:50As your app insights or Azure insights really

9:53hit home for me, if you actually click,

9:55let me show you how I do that.

9:57I did that with--

9:57I'll say, if you go to the Insights,

9:58you can click View All Insights here.

10:00And you'll actually see basically

10:02a list of all of the things that I just listed off here

10:04in addition to things like hyperconverged infrastructure,

10:07SQL databases, Service Bus, and so on.

10:10Application Insights are really the first time where it really

10:15drilled home for me, how powerful

10:18this particular resource is.

10:20Because with applications insights again,

10:23it was just an SDK that you dropped into your application

10:27and turned on, and that was effectively it.

10:29From there, the SDK or really the Application Insights

10:34package itself starts monitoring all

10:37of the requests that actually take place on the application.

10:40So if I had a website and people start off with my Home page,

10:45let's call it home.html.

10:47And then from there they navigate to their Cart page.

10:52And then from there they navigate to Checkout page.

10:56And then from there they navigate back

10:58to a specific product page.

11:01Maybe they clicked on it because they got there

11:03from an ad that popped up in the sidebar or something like that.

11:07As your app Insights keeps track of all of this navigation

11:12information within the application itself,

11:15it also keeps track of things like what

11:18are the geographies that my end users are coming from.

11:21Am I getting most of my data or most of my requests

11:24coming from the UK, the United States, Australia,

11:27wherever they may be?

11:29It keeps track of not only the standard metrics

11:33that we keep track of as well as the logs themselves,

11:36but it then digs deeper into finding things

11:38like geolocation and the paths that people are

11:41taking through the application.

11:43And it generates this in a live dashboard for you

11:47to actually see what's happening in your application.

11:51It's a fascinating thing to check out.

11:54Now like I said, I already deployed my virtual machine

11:57Insights.

11:58So if I go to Virtual Machines, I can then go to Analyze data.

12:02And from here, I can choose my subscription, my resource,

12:06and then pick the virtual machines

12:08that I want to analyze.

12:10So here we go.

12:11I'm looking at things like the performance

12:13of my virtual machines via workbooks.

12:16But this is all done through my insights portal.

12:20So now, I can look at my CPU utilization

12:23for my virtual machine over a given trend

12:26of time based on the 95th granularity

12:31of the actual utilization.

12:34Same thing with memory utilization

12:36on this virtual machine, byte set and so on.

12:38And if I had multiple virtual machines,

12:41they would all be listed right here,

12:43just in different colors of line so

12:44that we can see how virtual machines are performing when

12:48compared to each other.

12:50Now, the map is what I was talking

12:52about where we're looking at how processes

12:54within the virtual machine connect to each other, as well

12:57as geographies where clients are connecting

13:01into my virtual machine in the first place.

13:03Now, this map feature is only available on specific operating

13:08systems.

13:09It's not available on the Windows Server operating system

13:12that I specifically picked for this virtual machine.

13:14So just know that is a thing.

13:16If I click on it, you'll see.

13:17It'll fuss at me and say, I don't have any virtual machines

13:20that are available for this.

13:21So if you wanted to leverage this virtual or this map

13:24feature, make sure you're selecting a compatible image

13:28to get it going.

13:29So this is understanding Workbooks as well as Insights

13:32to dig deeper and visualize what's

13:35really happening within the resources

13:38that you deploy in Azure.

13:40I hope this has been informative for you

13:41and I'd like to thank you for viewing.

0:00[AUDIO LOGO]

0:06The Azure Data Explorer does exactly what

0:08the name indicates it would do.

0:10It explores data.

0:12This offering has really transformed a lot, even

0:16in recent history.

0:17I know I started working with Azure five

0:19or six years ago at this point.

0:21And at the time, Azure Data Explorer

0:23was just a really great way to explore the contents

0:27of your storage account.

0:28Now it really gives us a robust way

0:30to unpack all sorts of data that could exist

0:33within our entire environment regardless

0:36of where that data comes from.

0:37Great use case here to use this for monitoring our resources,

0:42whether by metrics or our logs yet again.

0:45So in this video, we're going to take a look at how Azure Data

0:47Explorer could be part of the design for your monitoring

0:52and logging solutions.

0:53Let's go.

0:54So Azure Data Explorer-- if I actually

0:56bring it onto the screen here, anybody

0:59with an Azure tenant an Azure Active Directory tenant

1:03can get to it.

1:03It's dataexplorer.azure.com.

1:06It'll ask you to sign in with your credentials,

1:09and this is where you'll land.

1:12Now, again, Data Explorer has really evolved quite a lot.

1:16I actually remember using something called Storage

1:19Explorer and thinking like, oh, well, this

1:21is a great starting point, as Storage Explorer does what

1:24you think it's going to do.

1:25It's going to explore your data.

1:27But with Data Explorer, it takes accessing the data

1:31and ratchets it up all the way.

1:35With Azure Data Explorer, now what we're doing

1:38is we are analyzing big data in real-time.

1:45Where does this come into play when it comes to monitoring?

1:48Well, first of all, if we have a lot of resources in Azure

1:52that just a lot of data very, very quickly,

1:56that's a major point that we want to keep up with.

1:59If we have a lot of logs, we have a lot of insights.

2:04If we have a lot of metrics that we want to keep up with,

2:07that's a start.

2:09This oftentimes goes hand in hand

2:12with monitoring data that comes in from Azure Active

2:15Directory or a SIEM solution like Azure Sentinel.

2:21Sentinel, its entire point, its entire purpose

2:24is to grab all of this data that comes in

2:27from all of our resources, both in the cloud and on-premises,

2:32and process it very quickly to look for security anomalies.

2:39That's the idea here is to track out those anomalies.

2:44That's A-N-O-M-A-L-Y. There we go.

2:49Data Explorer really shines when it

2:52comes to grabbing all sorts of data,

2:55whether structured, semi-structured,

2:58or unstructured data crunching it very quickly,

3:03and then spitting it through some machine learning or AI

3:08to help you find not only anomalies

3:11but also prediction forecasts of where it's going to go.

3:15When you want to use Data Explorer,

3:18these would be your use cases.

3:20The first one is you need interactivity.

3:24It needs to be interactive analytics.

3:27This is going to be a major part where

3:28we're going to be needing to interact with

3:30to find aggregations or correlations in our data

3:34or detect anomalies.

3:36That's what interactive analytics are all about.

3:39We also have a lot of Vs. The velocity of our data,

3:45the growth rate of our data, or the rate at which the data

3:50moves and the rate at which we need to monitor it is very,

3:54very critical and sensitive.

3:55If we need to monitor data in real-time,

3:58that is a very big reason why we would

4:00need to use Data Explorer.

4:02But also, the sheer volume of data that we have

4:06is tremendous.

4:08Then lastly, the variety or the different kinds

4:12of data sources, as well as whether

4:14or not the data is structured in the first place, is a big deal.

4:19But then maybe we haven't even had the time

4:21to ingest and model the data yet, basically meaning we

4:25haven't had our ETL solutions run

4:28to pull the data from their sources.

4:30Instead, we want to analyze raw data as it lives

4:35row-level raw data on the fly.

4:39That's the big issue here.

4:40This is another great use case for Data Explorer.

4:44Multiple querying happening at the same time.

4:48We call this querying with concurrency.

4:52So what we're saying here is maybe

4:54we have multiple users analyzing the same underlying

4:58data at the same time.

5:00It's a tremendous amount of data and just processing one query

5:03alone takes a tremendous amount of compute resources.

5:07This is where Data Explorer would really shine.

5:10Now it even gets more complicated than that.

5:13And that's why I am going to bring some Microsoft

5:16documentation onto the screen because they have a great flow

5:19diagram as to when you would actually want

5:22to use the Azure Data Explorer.

5:25It says Start Here at the very top.

5:27And you go through a basic flow where

5:29you answer a bunch of yes and no questions.

5:31And ultimately, what these questions

5:34are trying to do are say, is there

5:36a better, more targeted fit for what

5:39you're trying to accomplish that you should

5:41use a different type of tool like a SQL database or a Cosmos

5:46Database?

5:46Should you use Spark or Azure Batch?

5:49Should you use a Synapse Data Warehouse or a SQL pool?

5:53Otherwise, it all kind of takes you

5:55down here to Azure Data Explorer is a good operation here.

6:00There's is a good option.

6:02So where as your Data Explorer really

6:05shines at the end of the day, is does provide you

6:08better flexibility when it comes to working

6:11with tremendous amounts of data.

6:13Also enabling near real-time analysis, including dashboards.

6:19It's a big, big deal.

6:21With these dashboards, we can do things

6:23like pattern detection or time series analysis.

6:28And when you think time series analysis,

6:30you need to be thinking logs because every time your log

6:34runs, it's logging what is the date and time

6:37that this exact log was issued, and that's

6:39where the time series analysis really comes into play.

6:42This does come with role-based access control

6:46so we can limit who can query what types of data.

6:49We then have things like integration

6:53directly into machine learning, including training machine

6:57learning models, which is a major part of using machine

7:01learning.

7:02You have to train the model to learn what's

7:04normal in the first place.

7:05And that way, you can then detect anomalies.

7:10So that means that this does integrate with Azure's machine

7:14learning workspace, the machine learning tools, as well

7:18as Databricks, which is a very common data crunching solution.

7:23Very, very useful in the olden days of Azure.

7:26But now, recently, I've started using Azure ML,

7:28and I've really enjoyed my time working with that as well,

7:31for what it's worth.

7:32So you can also see where this really comes in handy when

7:36you're using the same solution like Sentinel,

7:39that's S-I-E-M for security monitoring.

7:43So when you're using these security solutions

7:46like Sentinel in conjunction with big data streaming

7:49and machine learning, it can detect security anomalies

7:52even quicker, logging them and then

7:55visualizing them in dashboards.

7:58This is, again, for really large data sets,

8:00but it is an incredibly powerful tool

8:03and one that you should design for if you're ingesting just

8:07a tremendous amount of data.

8:10So if you're in a very large enterprise

8:12and you're collecting logs from all of your network devices,

8:16whether they're switches or routers or firewalls,

8:19or intrusion detection and prevention devices.

8:22Then you've got all of your security servers, then

8:26the virtual machines, the databases, then

8:29client desktop machines.

8:31Then IoT devices.

8:33All of this data is going to be coming in

8:35and logged through things like Kafka or Event

8:40Hub itself and leveraging applications like Logstash

8:46to actually be doing the collection of the logs

8:49and the ingestion of the logs itself.

8:52They make their way into Azure, where they eventually

8:55live in Azure data lake before they're

8:58ingested or transformed and moved even further

9:00down if they need to.

9:02Azure Data Explorer is a piece of this puzzle--

9:06I'm going to put Azure DE for Azure Data Explorer here--

9:09is a piece of this puzzle for grabbing

9:12the data at various points, analyzing it, and detecting

9:16eye issues in near real-time.

9:19So understand if you have a tremendous amount of data

9:22for a large enterprise coming in,

9:24Azure Data Explorer should be part of your monitoring

9:27solution.

9:28I hope this has been informative for you,

9:30and I'd like to thank you for viewing.

0:00[AUDIO LOGO]

0:06So there you have it, designing a monitoring and logging

0:09solution coming to you via video format.

0:13I really like this content because after you've

0:17deployed an application before and then had

0:20to troubleshoot it, you may be thinking to yourself,

0:22oh, man, I wish I had spent a little bit more

0:25time tuning up that monitoring and logging solution.

0:29That way I could have maybe gotten

0:30to the crux of my actual resources

0:34or my problems a little bit sooner.

0:35Beyond that, it does help you get insights

0:38into how your application is currently performing.

0:41You can get ahead of things that way by saying,

0:43you know what, this application probably

0:45needs to be tuned a little bit.

0:47Maybe we need to scale up my resources a little bit,

0:50or maybe we need to scale them down

0:51because we overprovision the resources in the first place.

0:55This is a really important factor

0:56yet again, because like I said at the very beginning

0:59of this set of videos, you can spend a finite amount of time

1:03designing and implementing a solution,

1:05but that solution could be up and running significantly

1:09longer than the design and implementation took.

1:12And that's where the monitoring and logging comes into play.

1:16That's where we keep the resources going.

1:18And really that's where you spend

1:21the bulk of your time, that's where our staff will

1:23spend the bulk of their time.

1:24And that's why I can't stress this enough,

1:26even though it's easy for your eyes

1:29to gloss over when it comes to logging and monitoring,

1:32this is actually a really, really important topic,

1:35arguably one of the most important topics and one

1:38of the most cost effective topics, one

1:40of the those cost effective wins that you can get.

1:42So thank you for joining me in this content on monitoring

1:46and logging as we progress through the AC 305.

1:49I hope this has been informative for you

1:51and I'd like to thank you for viewing.