Deploy a Kubernetes Cluster using Kubeadm

0:11Hi, guys.

0:12My name is Trevor Sullivan.

0:13And welcome to another CBT Nuggets skill.

0:16In this skill, we are going to be exploring

0:18how you can set up your own self-managed cluster

0:23with Kubernetes.

0:24And we're going to be using a tool known as kubeadm,

0:28or kubeadmin, depending on what your preference is

0:31in pronunciation, in order to set up that cluster.

0:35Now, before we get too deep into kubeadm specifically,

0:38I wanted to provide a overview of the landscape of utilities

0:43that you can use to set up a Kubernetes cluster.

0:46And this will give you a better understanding of where exactly

0:49kubeadm fits into the picture.

0:52Now, there's really two primary different mechanisms

0:55that you can use to deploy a Kubernetes

0:58cluster if you need it.

0:59You can do something that is self-managed,

1:02as with all the options right down here

1:04in the bottom section.

1:05Or you could use a major cloud vendor solution in order

1:09to deploy a Kubernetes cluster.

1:11Most of the major cloud vendors out there,

1:13like Microsoft Azure and Amazon Web Services or Google Cloud,

1:18Digital Ocean, and Linode all offer

1:22managed Kubernetes services.

1:24In Microsoft Azure, they call it Azure Kubernetes Service,

1:27or AKS.

1:28And in AWS, they call it EKS, or Elastic Kubernetes Service.

1:36They have a much longer name, which

1:37is a little bit more formal.

1:38I think it's Elastic Container Service for Kubernetes.

1:42But that's just more of a formality.

1:44Most people know it as EKS.

1:46Google Cloud has their GKE engine

1:48that allows you to spin up a cluster.

1:50And then Digital Ocean and Linode

1:52both have their own managed Kubernetes offerings as well,

1:55which really fundamentally operate similarly

1:58to the big three cloud vendors.

2:01Now, what's really nice about these services

2:03is that it makes it very easy to spin up a cluster.

2:06So you don't really have to know anything about Kubernetes

2:09in order to use these managed services.

2:11As long as you have an account with those vendors,

2:14you can spin up a Kubernetes cluster

2:16and then immediately start using it.

2:18Also, they take care of doing things

2:20like handling automated upgrades of the cluster.

2:23So if you need to increment your Kubernetes version when

2:27a new release becomes available, then you

2:29can use that cloud vendor's upgrade path in order

2:32to manage that upgrade and offload some of that process

2:36off to the cloud vendor.

2:38So you as an administrator don't have to do as much work

2:41to upgrade the cluster and then verify that everything's

2:44been upgraded correctly because that's

2:46part of the value that's offered by a managed Kubernetes

2:50service.

2:51Also, the cloud vendors such as Azure, AWS, Google Cloud,

2:55et cetera offer built-in monitoring and logging

2:58of the Kubernetes master nodes.

3:00And this is really helpful because you

3:02don't need to worry about, as an administrator,

3:04going and hunting down log files on the file system.

3:07That can be a little bit of a pain

3:09if you are having to go out and troubleshoot a cluster.

3:12And so by using built-in monitoring and logging

3:15solutions-- such as in AWS, they call it Amazon CloudWatch--

3:18you can benefit from having a central place to go look

3:22for logs about your applications rather than having

3:25to go log into different virtual machines, all

3:27the different master nodes in your cluster, for example,

3:30and then hunt down those files on the file system

3:32and make sure that you're looking at the right data.

3:34So that's a value-add with the major cloud

3:38vendors in terms of offering a managed Kubernetes service.

3:41We also have the ability to do scaled compute.

3:44So because we're not self-hosting the compute nodes

3:47that are part of our Kubernetes cluster,

3:49we're able to benefit from the scalability,

3:52or what's known as the elasticity of the cloud.

3:56And what that essentially just means

3:57is that the cloud has free compute capacity that's

4:01inside of their data centers.

4:02And when you are ready to consume that compute capacity,

4:06you can simply provision it into your account

4:09and basically take it out of their pool of compute capacity,

4:13and then pay for the provisioned amount of compute capacity

4:16that you have.

4:17So if your application needs to scale up very rapidly,

4:20then it's really nice to be using a cloud vendor

4:23because then you can simply add more virtual machines, more

4:25worker nodes, to your Kubernetes cluster.

4:28And you don't have to worry about purchasing hardware

4:31and installing operating systems,

4:33configuring low-level networking,

4:36installing device drivers, and just all

4:38of the complexities that come along with the traditional IT

4:42infrastructure configuration where

4:43you're responsible for that physical layer on up.

4:47Now, there are lots of other options

4:50that you have in the Kubernetes ecosystem

4:52to deploy a Kubernetes node.

4:55And some of those options have the names in their logos here.

4:58Some of the logos can be a little bit weird.

5:00Like this one down here in the bottom left corner,

5:03that's actually K0S.

5:05You've also got this option right here.

5:07This is the logo for, I believe it's minikube.

5:09And minikube is basically a command line tool

5:12that you can use to rapidly spin up a Kubernetes cluster.

5:15It gives you the ability to spin one

5:16up inside of a virtual machine or inside of a container

5:19or directly on a bare metal machine

5:25that you might have in your environment as well.

5:27K3s is actually really cool.

5:29I like K3s, and I've toyed around

5:30with that in the past a little bit.

5:32It's a very lightweight distribution

5:34that comes packaged up as just a single binary.

5:36And so this makes it very easy to install because really

5:40all you have to do is copy this local binary to your system,

5:43and then you can immediately start

5:45calling the different commands inside of K3s.

5:48It also makes it very easy to set up the initial cluster

5:52and then run kubectl commands in order

5:55to administer the cluster.

5:57You've also got options like Rancher.

5:59So Rancher, you can think of like an orchestrator

6:01of orchestrators.

6:02So it allows you to rapidly deploy Kubernetes clusters.

6:05They also have their own distribution

6:06of Kubernetes as well.

6:08I forget what the name of that is,

6:10but they do offer a distribution of Kubernetes

6:12that you can simply spin up yourself.

6:15Also, "kind" is known as Kubernetes in Docker.

6:18And if you'd like to run a Kubernetes

6:19cluster as an array of Docker containers,

6:22you can do that as well.

6:23It's a really convenient option if you're

6:25looking to isolate that cluster and then very easily be

6:28able to tear it down and then rebuild it

6:31again so that you can very easily do rapid development

6:34on your environment.

6:35And you don't have to worry about spinning up

6:37an entirely new operating system every time to make sure

6:40that you're starting from a clean slate.

6:41By working with containers, you're

6:43actually helping to ensure that every time you spin it up,

6:46it's going to start with a clean slate.

6:49One of the easiest ways to spin up a Kubernetes environment

6:52as well is Docker Desktop.

6:54And so if you're looking for just the most straightforward

6:57option to create a Kubernetes cluster,

6:59you'll probably just want to install Docker Desktop

7:01on your dev system.

7:03Docker Desktop runs on Linux, Mac, and Windows.

7:06And you can actually run Kubernetes

7:08directly inside of Docker Desktop on the Windows

7:11platform, for example.

7:12There's a little systray icon.

7:13You can just go down into that systray

7:15and then indicate that you want to go into Kubernetes mode,

7:19at which point, Docker Desktop will

7:21allow you to run kubectl commands against a cluster

7:24that it sets up itself.

7:26Also, Ubuntu-- or Canonical, I should say,

7:29the makers of Ubuntu Linux, have a distribution of Kubernetes

7:33known as MicroK8s.

7:35And depending on which distribution you choose here,

7:38you're going to be looking at different options that

7:40are available.

7:41Some distributions might have things

7:43like support for Kubeflow or support

7:45for GPU compute inside of containers on your Kubernetes

7:49cluster.

7:50Other options might include authentication options that

7:53are available, and also things like logging systems

7:57that are built in.

7:57Like, you might have a Fluentd agent

7:59that's built in to automate the obtaining of logs

8:03so you can troubleshoot that cluster.

8:05So depending on which distribution of Kubernetes

8:07you choose, you'll have a different kind of combination

8:09of options there.

8:10But if you're looking for something really easy

8:12to get started, definitely recommend Docker Desktop.

8:15If you're looking for just having a raw interface to run

8:18Kubernetes in containers, you probably want to look at kind.

8:20If you're looking for something that's

8:22very simple and lightweight, K3s definitely

8:24takes the cake since it's just a single binary distribution.

8:27But we're actually going to be focusing on kubeadm here,

8:30which is the official tool from the Kubernetes group

8:34that is basically administering Kubernetes itself as a project.

8:37That's their official tool to set up a Kubernetes cluster,

8:41and so that's the one that we're going to be exploring.

8:43And in particular, we're going to be using kubeadm

8:46on an Amazon Web Services EC2 instance in order

8:50to provide the virtual machine compute and virtual networking

8:53platform in order to create a cluster.

8:56So in any case, let's go ahead and jump in to our next video,

8:59where we'll actually take a look at the steps

9:01to install the kubeadm utility.

9:02And then we'll take a look at some of the dependencies

9:05that you're going to need in order to create a Kubernetes

9:07cluster, things like your container runtime,

9:11as well as your container network interface as well.

9:14In this particular skill, we're going

9:15to be using an open-source project

9:17out there known as Calico for our container networking.

9:20So let's go ahead and jump into our next video

9:22and start taking a look at kubeadm itself.

9:25I hope this has been informative for you,

9:27and I'd like to thank you for viewing.

0:11Hi, guys.

0:11And welcome back.

0:12Let's go ahead and take a look at the documentation for how

0:16to install kubeadm. And then we'll

0:18spin up an Amazon Web Services EC2 instance or Elastic Compute

0:22Cloud, which is basically just a virtual machine running

0:24Ubuntu Linux.

0:26And then we'll install kubeadm on that Linux virtual machine

0:30in order to give us the ability to then initialize

0:33a new Kubernetes cluster.

0:35So in order to take a look at the documentation,

0:38we're going to head over to the kubernetes.io website

0:42under Docs.

0:43And they've got some really excellent documentation

0:46here that talks about how to bootstrap

0:49a cluster with kubeadm. And then that's

0:52broken down into a bunch of different subtopics,

0:55such as how to install kubeadm itself,

0:57how to troubleshoot any problems that you might come across

1:00with kubeadm. And then, of course,

1:02how to create a cluster once you've got kubeadm

1:04itself installed, how do I initialize a Kubernetes

1:07cluster, and things like that.

1:09So let's head up for starters up to Installing

1:12kubeadm here and just explore what we need.

1:16So for starters, we're going to need a compatible Linux host.

1:19Now, I often prefer to use Ubuntu Linux.

1:22However, there are plenty of other

1:24supported distributions of Linux out there

1:26that you could use as well.

1:27You could use SUSE Linux.

1:29In fact, SUSE acquired Rancher back in late 2020, I believe.

1:33And so they're definitely a good supporter

1:36of the Kubernetes project.

1:38You could also use just Debian Linux as well,

1:41or Red Hat Linux, which is a very population distribution.

1:44So there's a bunch of different ones out there.

1:46I'm going to be using Ubuntu, which is actually

1:49based on Debian Linux.

1:50So that is definitely compatible.

1:52You will want to make sure that you

1:54have a couple of gigs of RAM available on each

1:56of your virtual machines as well.

1:57And this is important for especially the master

2:00nodes in your Kubernetes cluster because you're

2:02going to want to make sure that the master nodes have

2:04enough space to run all of the different components,

2:07like the Kubernetes Controller Manager, the Kubernetes

2:10Scheduler, as well as the API server that kubectl

2:13communicates with in order to administer your Kubernetes

2:16cluster.

2:17And if you don't have enough RAM to run

2:19all of those different components,

2:21then you may run into things like system crashes.

2:23You might run into just odd behavior from your Kubernetes

2:26cluster.

2:27And you can avoid that by ensuring that your master

2:30nodes have plenty of RAM.

2:32Now, as far as the worker nodes go,

2:34any worker nodes that you join to your cluster you're

2:36going to want to make sure have enough

2:37RAM as well because those worker nodes are

2:40the nodes in your cluster that are actually going to be

2:42running your different pods and containers inside of those pod

2:46specs.

2:46And so if you don't have enough RAM to run

2:49your different application components,

2:50like maybe you're running database servers

2:52like MySQL inside of containers, maybe

2:54you're running web servers, maybe

2:55you're running some middleware servers in there, if you don't

2:58have enough RAM to run your containers,

3:00then you're not going to be able to reliably run

3:02your application, as well as scale that application up

3:05when you actually need it.

3:08You also want to make sure you have enough CPU capacity

3:10as well just so that your application components aren't

3:12fighting for CPU utilization.

3:15If you're running very CPU heavy processes,

3:18maybe you're building a website like YouTube,

3:20for example, where they're doing very heavy video processing,

3:23which is very CPU intensive work, if you are building

3:26a service like that, then your focus

3:28on having enough CPU and memory capacity for that matter

3:31is going to be essential.

3:33Also, you're going to want to make sure

3:35that you have network connectivity

3:36between the machines in the cluster.

3:38So this is really important, especially

3:40if you are dealing with things like multicloud scenarios,

3:43where maybe you've got some virtual machines running

3:45in Microsoft Azure, maybe you've got

3:47some virtual machines running in Amazon Web

3:49Services and other cloud vendors.

3:51In order to ensure that your cluster is functioning

3:53properly, you're going to want to make sure

3:55that you have routing between your different networks

3:58in different cloud vendors.

4:00Or if you're doing something even

4:01just as simple as multi-region setups in something

4:05like Amazon VPC, where you've got two different VPCs that

4:08exist in different regions, then you

4:09want to make sure that you have routing setup

4:11between those VPCs so that your worker nodes can communicate

4:15with the master nodes, and so that the master nodes

4:18themselves can communicate with each other on TCP 6443.

4:25And of course, your load balancers and ingress

4:27controllers and all of your pods are

4:29going to need to make sure that they have network connectivity

4:32as well.

4:33Also, this is kind of something I would take for granted,

4:38I should say, but oftentimes, you

4:40might not be thinking about it.

4:42And you do want to make sure that you have a unique hostname

4:44and MAC address for all of the nodes in your cluster just

4:47to ensure that all of those nodes

4:49are uniquely identified within the cluster itself.

4:53Also, disabling swap is very important in order

4:56for the kubelets to work properly.

4:58And the kubelet is basically the component of software

5:02that you use as part of a Kubernetes cluster

5:04in order to register all of the nodes in your cluster

5:07with the Kubernetes master node.

5:10So that ensures that your worker nodes

5:12are going to be able to communicate with the API server

5:16and receive any instructions from the scheduler

5:19through the API server that they need to process,

5:22like starting up pods or deleting pods or creating

5:26services, that kind of thing.

5:29All right, so let's go ahead and start by spinning up

5:33a virtual machine.

5:34And then we're going to go ahead and basically just verify

5:38that we meet all of the dependencies

5:40in order to install kubeadm and get our cluster up and running.

5:44So I'm going to head over to the AWS Management Console here.

5:47And what we're going to do for starters

5:49is just provision a new virtual private cloud.

5:53And this is basically the network environment

5:54that's going to host our cluster and allow all of our nodes

5:58to communicate with each other.

6:00Of course, we're going to start simple.

6:02We're just going to have a single node for starters.

6:03But if you did have additional nodes, you of course

6:05want to deploy them into the same VPC.

6:07Or if you're doing a more complex network setup,

6:10then you'd want to make sure that you have that routing

6:12all set up ahead of time.

6:14Now, on the Amazon VPC Dashboard here, you

6:17can just go to the Launch VPC Wizard.

6:19And you can provision a VPC with both public and private

6:22subnets.

6:23And so we're going to go ahead and do that for now.

6:25And then you want to make sure that you just

6:26choose a private CIDR block for your cluster.

6:28So I'll do something kind of random like 10.77/16.

6:33And then I'll call this k8s just so I

6:36can find this VPC later on.

6:38And then for our public subnet's CIDR block,

6:41we're going to do something like 10.77.10/24.

6:47And then we'll do, for our private subnet,

6:49we'll do 10.77.110.0/24.

6:56So those are just a couple of different CIDR

6:58blocks for public and private.

7:00And we're also going to need to make sure

7:02that we allocate an elastic IP for our NAT gateway.

7:06So I'm going to head back to Services and go over to VPC.

7:10And we'll head down to Elastic IPs.

7:14Actually, I think that's over in EC2.

7:17So let's go over to--

7:19oh, here it is, Elastic IPs.

7:21Some of those things only appear in EC2 Console.

7:23But let's go to Allocate Elastic IP.

7:26We'll just do our region here, and do Allocate.

7:29And then I'll just go ahead and give it a name like k8s.

7:33And let's come back here to our VPC Console.

7:36And now we should be able to choose that elastic IP.

7:40And we'll go ahead and hit Create VPC.

7:43So we'll give that a second to create.

7:45And then once our VPC has been created,

7:47we'll go ahead and spin up a virtual machine

7:49over in EC2 that is going to be a member of this VPC.

7:54All right, so our VPC has been created here.

7:56So we'll go ahead and head over to the EC2 console here,

8:00and we'll create a new virtual machine with Ubuntu Linux.

8:04So I'm going to go to Launch Instances here.

8:07And then I'll find Ubuntu 20.04 Focal Fossa right here.

8:13So I'm going to use that image.

8:15And then we're going to choose just a relatively small

8:18EC2 instance type.

8:20Of course, we want to make sure it's got enough vCPU and RAM.

8:23So I'm going to choose a t3.medium here

8:24because that has 2 virtual CPUs and 4 gigs of RAM.

8:28And then on our Instance Details here, I'm

8:31going to go ahead and choose my VPC.

8:33So let's choose our k8s VPC.

8:36Choose our public subnet because we

8:38want to remotely log in to that virtual machine with SSH.

8:41And then we'll enable auto-assign public IP

8:44so that we can remotely access that virtual machine.

8:48All right, let's go ahead and head to the Storage screen

8:50next.

8:51And for storage, I'm going to make sure

8:53that my master nodes have, let's say, 30 gigabytes of storage.

8:58And then we'll go ahead and do Review and Launch.

9:01And then we'll hit Launch.

9:02And I'm going to go ahead and create a new key pair here.

9:05And I'll just call this k8scluster,

9:08hit Download Key Pair to download the private key

9:12for the cluster.

9:14And then we'll go ahead and hit Launch Instance.

9:17All right, so while that's launching,

9:18we need to make sure that we have access

9:20to log into this EC2 instance.

9:22So I'm going to head back over to EC2 here.

9:25And we're going to find the virtual machine that we just

9:29spun up, which is this t3.medium right here.

9:31And then I'm going to go over to security

9:33and find the security group.

9:35And we need to make sure that we have inbound SSH access.

9:39And it looks like the automatically generated

9:41security group already has a rule that

9:44allows inbound access from anywhere online to TCP port 22.

9:49So we'll be able to log into this virtual machine

9:51here in just a second.

9:53So let me call this k8smaster just

9:56to give it a friendly name in our console here.

9:59And now to log into it, I'll head over to the Details tab

10:02and find the public IPv4 address,

10:04and just click on this button to copy it to my clipboard.

10:07And then I'm going to go ahead and fire up a new terminal.

10:11And we're going to go ahead and just run ssh.

10:14And then we'll do ubuntu@ IP address.

10:17And then I also need to specify my private key, or my what's

10:21known as the identity file.

10:22So I'll do -i and then go into my Download folder

10:27and then look for the k8sscluster key.

10:31And we'll go ahead and run that.

10:34And we'll confirm our host key.

10:35Hit yes.

10:37And now we have our virtual machine here.

10:39So this is the virtual machine that we just spun up.

10:41So I'm going to go ahead and do a standard sudo apt update

10:44and sudo apt upgrade just to make sure

10:47that all of our base operating system packages are up to date.

10:51And then we'll go ahead and disable swap

10:54as one of the first steps that we needed to do.

10:57Awesome.

10:58Now that our packages are up to date, let's go ahead

11:00and check to make sure that swap is disabled.

11:03So I'll do sudo vim /etc/fstab.

11:06And it looks like there is no swap actually configured

11:09on this Focal Fossa image that AWS provides.

11:12But you do just want to make sure that in fstab

11:15that you don't have a swap partition there.

11:18Also, if I exit out of vim and I do a cat /proc/swaps,

11:24then that will show you any swap partitions that are configured

11:27on your system here.

11:28And you can see that I have empty output.

11:31I'm also going to go ahead and change the hostname of my node

11:34here.

11:35So by default, the Amazon Web Services platform

11:38assigns this IP address as its name.

11:40But I'm just going to go ahead and do sudo vim /etc/hostname.

11:45And we'll go ahead and just delete that line

11:47with a dd command.

11:49And then I'll do insert, i for insert.

11:51And then we'll go ahead and call it k8smaster01.

11:56All right, now that I've renamed our node,

11:58let's go ahead and do a sudo reboot.

12:01And we'll let our system reboot, and we should

12:03have it named something else.

12:05So now that we've got this node up and running,

12:07we can go ahead and proceed with installing

12:09kubeadm. I hope this has been informative for you,

12:12and I'd like to thank you for viewing.

0:11Hey, guys, and welcome back.

0:13Now that we've got our virtual machine set up

0:15and we've ensured swap is disabled

0:18and we've renamed our node here, let's go ahead

0:20and take a look at what the next steps are in order

0:22to get kubeadm installed on our system.

0:25So I'm going to head back over to our kubeadm documentation

0:29here.

0:29And we've already ensured that we have a unique MAC address.

0:35Of course, we're only dealing with one node

0:37here, so we don't really need to worry about that.

0:39And MAC addresses are pretty much always unique.

0:41Not something you really need to check,

0:43but they do provide the commands that you

0:45can do to check that, as well as the product UUID on a Linux

0:49system.

0:50So the other thing we need to do here is to configure IP tables.

0:54And they provide the steps to do that right here.

0:57It's basically going to load a kernel module called

1:00br_netfilter, and ensure that these bridging options are

1:05enabled in our configuration for that module.

1:09So let's go ahead and copy this to our clipboard.

1:13And then we'll switch over to our terminal and paste that in.

1:16And then once our IP tables is configured here,

1:19we can go ahead and move on.

1:22Of course, there are a variety of different network ports

1:24that you'll need to ensure are open between nodes.

1:27And in this particular case, we're

1:29only dealing with a single node at the moment.

1:31So we're not going to be doing any internode configuration.

1:34However, you will want to ensure that your control plane nodes

1:38have the following ports open.

1:39You've, of course, got 6443 for your Kubernetes API server.

1:43That's what kubectl is talking to in order

1:46to manage the cluster.

1:47For etcd, you have ports 2379 and 2380

1:51for the etcd server API.

1:54You've also got the Kubelet API at 10250.

1:57And then 10259 for kube-scheduler and 10257

2:01to for the controller-manager.

2:05So if we were running multiple nodes,

2:07then we'd want to go back to our security group in Amazon VPC

2:11here.

2:12And we would want to ensure that we have rules set up to allow

2:16all of those different ports.

2:18So if we, for example, wanted to access,

2:21let's say, kubectl across port 6443,

2:24we'd go ahead and do a new security group rule here

2:27for TCP 6443.

2:30And then we'd specify either the CIDR blocks that we

2:32want to allow to access it--

2:34so we could do something like 10.77/16,

2:37and that would allow anything from our VPC here--

2:41or we could go ahead and choose the security groups that we

2:45want to allow to access.

2:47So if we created a virtual machine that

2:48had the default security group, we

2:52could go ahead and just assign that.

2:53And then that would allow anybody

2:55who has that default security group to communicate

2:58with port 6443.

3:00But because we're just dealing with a single node for now,

3:02I'm just going to go ahead and cancel that.

3:04And so SSH will be the only port that we

3:06are allowing inbound access to at the moment.

3:10Now, they do provide a telnet command

3:12that you can use in order to validate connectivity.

3:14But at this point, we don't have any services that are actually

3:17listening on TCP 6443.

3:20So you'd actually have to install the cluster node first

3:23and get the API server up and running.

3:25And then that would be listening on 6443,

3:28and then you could use this telnet command

3:30to validate that you actually do have connectivity

3:33to that port on a particular node.

3:36Now, one of the other things you'll need to install

3:38is what's known as a container runtime.

3:41And the container runtime is essentially

3:43what Kubernetes is going to use to deploy pods

3:47into your cluster.

3:48Now, you do have several different options

3:50that are available.

3:51You can use the Docker engine, you

3:53can use the containerd project, or you can use the container

3:57runtime interface here as well.

3:59I'm going to be installing Docker.

4:01And we're going to use that as the container runtime.

4:04So essentially, when Kubernetes wants

4:06to spin up any containers, it will actually

4:08be using the Docker API in order to do that.

4:11So what we're going to do for now

4:13is just go ahead and head back over to our virtual machine.

4:16And we'll do a sudo apt get install docker.io --yes.

4:23And then once you've got the Docker engine installed,

4:25you'll want to go ahead and just make sure that your Ubuntu user

4:28account is able to communicate with the daemon just

4:31for verification purposes.

4:33So in order to do that, we need to actually add our user

4:36to the Docker group.

4:38So the installation process creates

4:40the group called docker, and anybody

4:42who's in the docker group will have access

4:44to manage the Docker daemon.

4:46But by default, if we try to run a docker command,

4:49you can see we don't have access to it.

4:50So we'll do a sudo usermod --groups docker --append,

4:57and then the username that we want

4:58to append the docker group to.

5:00And that's ubuntu.

5:02And so we'll go ahead and run that command.

5:04And we'll just run exit.

5:05And then we'll ssh back into that virtual machine

5:09so that we can continue working on it.

5:11So let's do groups now.

5:13And we should in our groups list see docker.

5:16Sure enough, there it is.

5:17So now if I run docker ps, you can

5:19see I'm able to communicate with the Docker daemon.

5:22You could also run docker version or docker info.

5:25Both of those will verify that you

5:27have connectivity to the server component through the Docker

5:31socket as well.

5:33So once you've got your container runtime installed,

5:35let's go ahead and see what's next.

5:37So now we need to actually install kubeadm itself.

5:41And that's the utility we use to initialize the cluster

5:43and join nodes to the cluster.

5:46And then we also have the kubelet.

5:47And the kubelet is that component

5:49that registers our nodes with the cluster.

5:52And then we've finally got kubectl,

5:54which is the command-line utility that we

5:55use to communicate with the API server

5:58once that's been installed.

6:01So there are a few different steps

6:02that we need to run through in order

6:04to install these components.

6:06First of all, we need to install these components right here.

6:09So we'll just paste that in.

6:12And then once the ca-certificates and curl

6:14and everything is installed, we can download the Google Cloud

6:18public signing key.

6:19This will allow us to talk to the Google Cloud package

6:22repositories from our Linux VM here.

6:26And then finally, we'll add the apt repository for Kubernetes

6:30in particular, which uses that GPG key.

6:33And we'll paste that in.

6:35And then once that repository has been configured,

6:38we now need to update our apt cache.

6:40So we'll do a sudo apt update.

6:43And then once our package cache is installed,

6:45we can then install all of these components

6:48with a single apt install command.

6:50So let's do apt get install on kubelet to kubeadm and kubectl.

6:55And as you can see, those components

6:57are getting installed.

6:58And it'll show you the version numbers

7:00of each of those that's getting installed.

7:02So it looks like we're going to be using Kubernetes version

7:051.23.

7:06That's the latest as of this recording.

7:09So now the other thing you'll want to do

7:11is use apt-mark here in order to hold those packages

7:14at their current version.

7:16And so basically, we're going to pin

7:18version 1.23 of kubelet, kubeeadm, and kubectl

7:23on this particular node.

7:25And so that's basically just setting

7:26a flag indicating that if a new version of these packages

7:30comes out, we do not want those to be upgraded unless we

7:34explicitly unhold them and then actually perform the upgrade

7:39ourselves once we're ready to do that.

7:42So that helps just ensure consistency

7:44on the version of Kubernetes that's going

7:46to run on your cluster here.

7:48All right, so now we are just about at the point

7:51where we can go ahead and use kubeadm

7:54to initialize our cluster.

7:56And we'll do that in the next video.

7:58I hope this has been informative for you,

7:59and I'd like to thank you for viewing.

0:00[MUSIC PLAYING]

0:11Hey, guys, and welcome back.

0:13The one other thing that you'll see at the end

0:15here in the documentation for installing kubeadm

0:18is called configuring a cgroup driver.

0:20Now on Ubuntu Linux, we're actually

0:22going to be using the systemd driver here,

0:26but it does actually talk about configuring a cgroup driver.

0:29This is actually the document that I want here,

0:31kind of talks about how to configure

0:34either the systemd driver or the cgroupfs driver.

0:38But in this particular case, they

0:39recommend that if you're using kubeadm,

0:42then you should be using systemd.

0:44And I believe that's actually the default.

0:46And you could pass that into your configuration

0:48file for kubeadm, but I'm just going

0:50to go ahead and let it use the default value for that.

0:54So now we're ready to go ahead and go over

0:56to creating a cluster with kubeadm. And so we're

1:00going to use kubeadm to initialize a brand

1:04new Kubernetes cluster, and then we'll

1:05validate that the cluster is actually functioning

1:08and can deploy pods and things like that.

1:11So, of course, we've already gone through the process

1:13of installing kubeadm itself.

1:15And so we've gone through most of these preliminary steps that

1:18are kind of duplicated from the other documentation here.

1:22So at this point, what we want to do

1:23is actually initialize the control plane node

1:27or what's known as a master node in kubernetes.

1:29And one of the things that you can do as a preliminary step

1:33is to download the necessary container images.

1:36Of course, kubeadm does that itself as well.

1:40So you don't have to do that.

1:41But if you want to speed up the kubeadm command itself,

1:44you could go ahead and pre-download those container

1:47images if you'd like.

1:49So we'll open up this document here called running kubeadm

1:52without an internet connection.

1:54And this will show me how to list

1:57the images that are required for kubeadm,

1:59as well as the pull command that will allow Kubernetes

2:03to, or rather, kubeadm itself to pull those images

2:07and cache them locally.

2:09So let's head over to our cluster here.

2:13I'm going to head over to my cluster node,

2:14and then I'll just paste in these commands.

2:19And it doesn't look like it's copied that correctly.

2:22So let me just do kubeadm, and then

2:24we'll take a look at the contexts that are available

2:27here, and we'll do config, we'll do kubeadm config,

2:32and then we'll do --help to take a look at the subcontexts

2:35there.

2:36And then we've got the images context here.

2:39We will do images, oops, let me do images --help.

2:44And then, of course, I have the list command here.

2:46So we'll do list, and these are the images

2:50that Kubernetes is going to require.

2:52We've got etcd, which is the backing key value store

2:55for the Kubernetes cluster.

2:57So that it can share state amongst all the master nodes.

3:00Then we've also got all of our core components,

3:02like our API server that kubectl talks to.

3:05We've got the controller manager that's

3:07responsible for working with any deployments

3:12and determining what needs to happen

3:13in terms of spinning up pods.

3:15Then we've got the kube-scheduler

3:16that actually schedules pods onto nodes.

3:19We've got kube-proxy, pause, and then

3:21core-dns, which is an essential component for Kubernetes

3:24as well to handle DNS resolution of pods and services.

3:28So let's go ahead and do an up arrow,

3:30and then we'll change list to pull and pre-cache

3:33all of those images.

3:34And then while that's running, we'll

3:36go ahead and take a look at the next steps

3:38here back under creating a cluster documentation.

3:41Now, for initializing your first master node,

3:44there are several different options that are available.

3:47If you are going to be spinning up

3:50multiple nodes in your cluster, multiple master nodes

3:53for high availability purposes, then you

3:56can specify a parameter here called control-plane-endpoint

3:59and set the shared endpoint that you'd

4:02like to use for all of those control plane nodes.

4:05And then you can put a load balancer in front of that,

4:08and load balance your traffic across all

4:10of your different nodes.

4:12Now in this case, I'm just going to have a single node.

4:14So I'm not going to go ahead and specify that option,

4:17but that is something that you'd want

4:18to do if you are going to have high availability

4:21for your kubeadm cluster.

4:24Also you are going to need to install a pod networking

4:27add-on, and you are going to need

4:30to set the pod-network-cidr, and that's

4:34going to be the cluster network that we're

4:36going to use for pods to obtain IP addresses from the cluster

4:41once they are deployed.

4:42So you want to be thinking about what

4:44CIDR block do I want to use in my network

4:47in order to assign pods?

4:49And of course, you don't want that to overlap

4:51with any other important CIDR blocks.

4:53So for example over here in our VPC console,

4:57I think we chose to use 10.77/16.

5:01Yep, that's our CIDR block there.

5:03So we wouldn't want to use 10.77/16 as our CIDR block

5:08for our pod network because then we could run into potential IP

5:12conflicts, and that could cause our cluster to just fail.

5:16So we probably don't want to use that.

5:18But you could choose something else like, I don't know,

5:2010.88/16 or 192.168.

5:24something.

5:25It's kind of up to you what you'd

5:27like to use for your CIDR block there.

5:29There's a couple of other important options on kubeadm

5:32here as well.

5:33Item number 3 here is referring to the container runtime

5:38interface that you're going to be using with your Kubernetes

5:41cluster.

5:41Now in the earlier video, we actually

5:44installed a Docker as our container runtime interface.

5:47So our Kubernetes cluster is going to be using Docker.

5:50However, since version 1.14, kubeadm actually

5:54will automatically detect your container runtime

5:57based on the domain sockets that are accessible.

6:00So if the Docker socket is accessible,

6:03then Kubernetes will automatically

6:05detect that the Docker socket is available,

6:07and it will configure the Docker as the container runtime

6:11interface for your cluster.

6:13So we don't have to specify this,

6:15but if you did want to override that to a specific socket,

6:18you could use the cri-socket parameter on kubeadm init.

6:24And then finally, if your virtual machine

6:26has multiple network interfaces, then

6:29you can specify which network interface

6:31you want to use for your Kubernetes cluster here.

6:35But in our case, we only have a single network interface

6:38on our virtual machine, and we can kind of

6:40verify that by going back to our command line here.

6:43And then if we use the IP command

6:45to take a look at IP links, that's

6:49the command we can use to check out our network interfaces.

6:52And of course, we have our standard loopback interface,

6:54which would not make sense to use for a cluster.

6:56We've also got our Docker network interface here as well,

6:59which is not a physical interface.

7:01It's basically just a virtual interface

7:03that Docker uses for its bridge networking.

7:05And then that leaves us with by process of elimination

7:09just ens5 here, which is our actual VPC network interface

7:14or elastic network interface that is

7:16attached to our EC2 instance.

7:19So we don't need to specify that,

7:20but if we did have multiple interfaces,

7:22then we could specify which one we

7:24want to use for the cluster in the kubeadm init command.

7:29So the only option that I'm really

7:31going to be using here to initialize our cluster

7:33is pod network CIDR.

7:34So let's go ahead and run kubeadm init,

7:38and then we'll pass in that option,

7:41and that will initialize our cluster here.

7:43So let's come back to our terminal,

7:45and we'll run kubeadm. And then we'll do init.

7:49Take a look at the help with the --help parameter here.

7:53And as you can see, we've just got a whole bunch

7:55of different flags here.

7:56And I'm just going to be specifying

7:59the pod-network-cidr, which is basically

8:02just a string value with the CIDR block for our pods.

8:06And so I'll do kubeadm init --pod-network-cidr.

8:12And then I'll set that to let's do 10.88.0.0/16.

8:18All right, so of course, I need to run that as superuser.

8:21So I'll run sudo in front of it.

8:24And that should give us access to run it now.

8:26And so now kubeadm is going to go

8:28through a whole bunch of steps to set up certificates for all

8:31of the different services here, and it's going to spin up a cd,

8:35it's going to spin up the kube-proxy, the API server,

8:38and all of those components that actually

8:40create the cluster here.

8:43And as you can see, it's going to spin up

8:45things like kube-scheduler, kube-controller-manager,

8:47kube-apiserver, and pretty much everything

8:50that our cluster needs to run.

8:52I believe it also spins up coredns here as well.

8:55And so once this command is done executing,

8:58we should have a cluster up and running.

9:01However, we're going to have kind of a crash looping

9:05process occurring, and we'll take a look

9:07at how to resolve that.

9:09All right, so it seems like our kubelet isn't healthy here.

9:12And so that's kind of causing the kubeadm init

9:17process to hang a bit here.

9:19And so we need to kind of figure out

9:21why exactly the kubelet is failing.

9:25So what we can do is use journalctl to explore our logs

9:32in our operating system here.

9:34However, I need to specifically find the kubelet component

9:37because we're dealing with lots and lots of different services

9:40here.

9:41So if I do journalctl --help, there's actually

9:45a parameter here called -t or --identifier,

9:48and you can set the identifier to kubelet and find

9:52just the relevant logs for the kubelet component.

9:56So let's do a journalctl --identifier,

10:00and then set that to kubelet.

10:02And I'm going to head all the way to the bottom here,

10:06you see what our latest logs look like.

10:09So as you can see, we've got some logs right over here.

10:12And if I scroll over just a little bit,

10:14you can see it says failed to run kubelet,

10:17and then in the more detailed message here,

10:19it says that there's a misconfiguration,

10:22and that the kubelet cgroup driver systemd is

10:25different from the doker cgroup driver, which

10:28is set to cgroupfs.

10:30All right, so that is actually an indicator that we do

10:33need to manually configure our cgroup driver

10:37on the Docker engine, which is our container runtime over

10:41to systemd as well.

10:43So I'm going to hit Q to quit out of journalctl.

10:46And now I'm going to run docker info here just to confirm.

10:50And sure enough, the cgroup driver

10:52that's being used by default when we installed the Docker

10:56package on our operating system here is sure enough,

10:59it's actually cgroupfs.

11:01So what I need to do is to reconfigure the Docker engine

11:04to use systemd instead of cgroupfs.

11:08So how do we do that?

11:09Well, first of all, we need to figure out where

11:11the unit file for Docker is.

11:13I'm going to do systemctl status on docker.service.

11:19And it looks like our unit file for the docker engine is under

11:23lib/systemd/syst em/docker.service.

11:27So I'm going to do sudo vim etc, sorry,

11:31/lib/systemd/sys tem/docker.service.

11:37All right now, inside of this systemd unit file,

11:40we basically need to reconfigure docker

11:41to run with the systemd cgroup driver instead.

11:47So I'm going to come down here under exec start,

11:49because this is the command line that's

11:51going to be used to spin up the Docker daemon.

11:54I'm going to go to the end of the line here

11:57and we'll add in the exec option and set

12:01it to native.cgroupdriver equals systemd.

12:06And so that should configure the Docker engine

12:09to use systemd as the cgroup driver instead.

12:11So I'll hit Escape and then :wq to save and quit that file.

12:15And then we'll do a sudo systemctl

12:18restart on the docker.service unit file.

12:23And we, of course, need to run systemctl daemon reload here,

12:27because we reloaded or we changed the unit file.

12:30We'll do sudo systemctl reload on the engine here.

12:36And then we'll go ahead and do sudo systemctl restart

12:40docker.service once more.

12:42All right, so that time it restarted,

12:44and let's just make sure that it's actually running.

12:47So I'm going to use my status command here again.

12:49And sure enough, it looks like the Docker engine

12:52is back up and running.

12:53And now if I do docker info again,

12:56let's just confirm that the cgroup driver has changed.

12:58And sure enough, you can see right here cgroup driver is now

13:01set to systemd.

13:03Now if this happens, you'll want to make sure

13:05that you actually reset kubeadm, and then

13:08re-initialize your cluster.

13:09So I'm going to go ahead and clean out our configuration

13:12by doing sudo kubeadm. And then there

13:16is a command called reset.

13:18And go ahead and just confirm that you

13:20do want to reset your cluster back to scratch.

13:23And this command will basically go and stop

13:25all those containers for Kubernetes

13:27that were spun up when you attempted to initialize it

13:30the first time.

13:31And it will go ahead and reset a bunch of stuff.

13:34And then the other thing that we need to do

13:36is to delete our kube/config file because that's

13:38going to get regenerated.

13:40So I'm just going to go ahead and delete that kube/config

13:42from our directory.

13:43And then we'll go ahead and try to re-initialize

13:46our cluster with the same command that we used before.

13:50And hopefully, everything will work fine this time.

13:53So let's give that a minute to run and see what we get back.

13:57OK, so that works pretty quickly there.

13:59And you can see we get much nicer output here.

14:01We're not getting any errors anymore.

14:03Plus it installs things like coredns, as well as kube-proxy

14:07here, which would have gotten skipped

14:09in the previous initialization, since we had

14:11that error with the kubelet.

14:13All right, so now let's go ahead and run these commands here.

14:16And this is going to take our config file in admin.conf

14:20and copy it to our home directory.

14:23So I'm going to go ahead and just run these commands here.

14:26And now we should be able to use the kubectl command line

14:30utility to interact with our clusters.

14:32So one of the most basic things you can do

14:34is just run kubectl version.

14:36And if you get an error down here under a server version,

14:40it will basically tell you that you can't connect

14:43for one reason or another, maybe the API server is not running

14:46or maybe your credentials file is incorrect or something.

14:49But if you do get back a client version and a server version

14:53here without any error messages, then you

14:55are successfully connected to your Kubernetes cluster.

14:59So let's do kubectl, and then we'll do get namespace.

15:04And this will just list out the default name spaces

15:07in our cluster here.

15:08So we've got default kube-node-lease,

15:10we've got kube-public, and kube-system.

15:13So let's take a look at kube default actually for starters.

15:17So I'll say get all from namespace default.

15:21And that just has our cluster IP in there for Kubernetes.

15:25And then if we take a look at kube-system,

15:27this is where the bulk of the pods

15:29are pertaining to the actual Kubernetes cluster.

15:32So let's explore get all on kube-system.

15:35And so this is where you can see we've got things

15:38like coredns, we've got etcd, that's our key value store,

15:41we've got our API server up and running, our controller

15:44manager, proxy, and scheduler as well.

15:48So everything looks to be up and running just fine here

15:50with one exception, which you might see these two

15:54coredns pods are actually zero out of one ready,

15:58and they're just sitting here in a pending state.

16:01And if you take a look at our deployments

16:04down here for coredns, you'll see

16:06that the deployment requested that there be two pods running.

16:09However, we have zero out of two that are actually ready to go.

16:14And so what's happening here is that coredns is actually

16:19failing.

16:19And that's because we haven't installed a container network

16:22interface yet.

16:23So coredns will sit here in a pending state

16:26until you actually install that CNI into your cluster.

16:30So we'll go ahead and do that in the next video.

16:33I hope this has been informative for you,

16:35and I'd like to thank you for viewing.

0:11Hey, guys.

0:11And welcome back.

0:12Let's go ahead and resolve this issue

0:14with CoreDNS not starting up in our cluster here.

0:17And we are going to do that by, of course, installing our CNI.

0:22And so we're going to head over to the documentation here.

0:27And if you take a look at the documentation,

0:31somewhere right in here it does tell you

0:33that CoreDNS will not start.

0:36Let me just do a quick search for CoreDNS here.

0:38There's a lot of text in this documentation.

0:41And sure enough, under Installing a Pod network

0:44add-on, installing a CNI, a Container Network Interface,

0:47it says that Cluster DNS (CoreDNS)

0:50will not start up before a network is installed.

0:54So we need to install a Container Network

0:56Interface in order for CoreDNS to spin up.

1:00Now, there's a bunch of different supported Container

1:03Network Interfaces out there.

1:05However, there is one in particular

1:07that I like, that I've tested out,

1:10that works well, which is called Project Calico.

1:13So if you just head out and search for Project Calico,

1:16this is an open-source project from a company called Tigera.

1:20And Project Calico provides your Container Network Interface,

1:24as well as network policy capabilities as well.

1:28So this is a really cool open-source project that

1:31is available out on GitHub.

1:33So I am going to go ahead and go back and just

1:36add on GitHub to our search here.

1:38And once we get over to GitHub here,

1:41this should take us over to the project page here.

1:45And then we are going to look for how

1:47to get started running Calico.

1:49So they've got a link to a quick start

1:51guide on their documentation that

1:54describes how to get Calico up and running on your Kubernetes

1:57cluster.

1:58And they've got a few different options.

1:59So if you're using a managed public cloud

2:01offering, like EKS or GKE or AKS,

2:06they've got directions for those.

2:08They've also got self-managed public cloud options

2:11here as well.

2:12But in this particular case, I'm actually

2:14going to do self-managed on-premises,

2:16and choose that option.

2:19And then we're just going to scroll down a little bit here

2:22and look for how to install Calico with the Kubernetes API

2:26datastore of 50 nodes or less.

2:29And so they've got a really simple command

2:31to just download a YAML configuration file

2:34for Calico here.

2:36So we're going to download that.

2:37And then we are going to just run kubectl apply on

2:41that YAML file, and that should get projectcalico, the CNI, up

2:45and running inside of our environment.

2:48And you can see what features we get here.

2:50We get network policy, IP address management-- of course,

2:54the CNI itself--

2:56the overlay routing capabilities, as well as

2:59the data store integration here.

3:01So let's run our curl command here.

3:04For starters, just copy that to my clipboard.

3:08And download that file, and then we can explore it.

3:10So we'll just cat that file here.

3:12And this is a really, really large file.

3:14But it basically contains all of the necessary components

3:18in order for Calico to get up and running.

3:20You can see we have a deployment there.

3:22And we've also got-- let's see.

3:24What else?

3:24If we keep scrolling up, we've also got a rolling update

3:30strategy here for a DaemonSet.

3:32So we've got a DaemonSet that's running a calico-node right

3:37here.

3:37And we've also got a ClusterRoleBinding here

3:41and a bunch of other resources.

3:42But I won't bore you with all the details on that.

3:45And then let's go ahead and do a kubectl apply.

3:48So we'll just copy this command here.

3:50I could just type it out, but I'm just

3:52feeling lazy right now.

3:53And that's going to go ahead and apply the entire YAML file

3:57to our network here.

3:59And that should get projectcalico up and running.

4:02And if we come back and run kubectl

4:05get all on our kube-system namespace

4:08here, once projectcalico gets fully up and running

4:12we should see coredns eventually start up here.

4:15So right now, coredns is just kind of crashing and restarting

4:19on its own.

4:20But once Calico is up and running and initialized here,

4:23you'll see that, sure enough, coredns is up and running.

4:27And it looks like calico-node is up and running as well.

4:30And calico-kube-controllers is still in the creation process.

4:34But in a few more seconds, you'll

4:36see that that comes up running as well.

4:38So now we've got a fully functioning cluster here.

4:41We've got coredns running.

4:42We've also got projectcalico running as our Container

4:45Network Interface.

4:46And so we could go ahead and run a test pod.

4:50So I'm going to use an nginx test pod as an example

4:53just to make sure that we can spin something up.

4:55So for starters, I'm going to create a namespace.

4:58So I'll run kubectl.

4:59Let's say create namespace.

5:02And I'll give that a name.

5:04Let's call it something like trevor.

5:07So now we've got a new namespace.

5:08So if I do kubectl get ns, you can

5:11see we've got this trevor namespace.

5:13So let's go ahead and do a kubectl run.

5:16And we'll take a look at the help for that.

5:19There's a whole bunch of options here.

5:21But we could do run pod with the name nginx.

5:26And then we use the image nginx off of the Docker Hub.

5:30And that should be a pretty good example.

5:32So I'll actually just copy that to my clipboard

5:34and then paste it in.

5:35And I also want to tack on the namespace option

5:38so that the pod gets run inside of our trevor namespace.

5:44So we'll do that.

5:45And sure enough, it created that pod.

5:47So now we need to verify that it's actually running.

5:50So I'll do a kubectl get all for the namespace trevor.

5:56And as you can see, that pod is pending.

5:58So I'm going to do a kubectl.

6:01Let's say describe.

6:04We'll do describe pod in namespace trevor.

6:09And so this gives us some more details on it.

6:11And as you can see, we get this message

6:13saying that 0 of 1 nodes are available.

6:16And it actually tells us the specific reason

6:19that no nodes are available, because the only node that we

6:22have in our cluster has something known as a [garbled].

6:25And that [garbled] is basically specifying

6:27that the node is a master node.

6:31And so because this is a master node,

6:33the kube-scheduler is refusing to schedule this pod

6:36that we've just run on this node and so therefore it's

6:40just failing because there's no other nodes in the cluster that

6:43can satisfy this pod's needs.

6:46So what we can do, if we switch back to our documentation here,

6:51they actually provide a command that we

6:54can use to untaint the node with that master annotation.

6:59So I'm just going to run this kubectl [garbled] nodes command,

7:02and that will remove--

7:04you can see we've got a minus sign here at the end.

7:06That will remove the master [garbled] from our node,

7:10and it'll allow the kube-scheduler

7:12to schedule our pod to run on our one

7:15and only node in our cluster.

7:17All right, so now the k8smaster01 node

7:21has been untainted.

7:22And if we come back up here and run our describe command,

7:26you can see that in a matter of seconds,

7:28the scheduler says OK, I'm now going to schedule

7:31you to run on this node.

7:33And it pulls the image from Docker Hub

7:36and then it starts the pod with the container nginx

7:40inside of it.

7:40So now if we do a kubectl get all on namespace trevor,

7:44you can see that that pod is actually running.

7:47So our Container Network Interface

7:49seems to be working just fine.

7:51You can see that our pod was able to get an IP address

7:54on our pod CIDR block here, which is 10.88/16.

7:58And we are good to go to continue working

8:01with our Kubernetes cluster.

8:02So that's how we can use kubeadm to spin up a Kubernetes cluster

8:07on your own self-managed node.

8:10I hope this has been informative for you,

8:11and I'd like to thank you for viewing.