Describe Types of Core Data Workloads

0:06We are collecting a lot of data these days.

0:09There's no way around it.

0:10It doesn't matter what industry you're in, what business you're

0:12involved with.

0:13If you are in IT, there is data collection going on.

0:16Everyone's walking around with a little data collection device

0:19right in their pockets.

0:20And we all have data collection going on with IoT,

0:23our refrigerators and our cars.

0:25And even our light bulbs now are data collection devices.

0:27So there's a lot of data.

0:28And we have to deal with it.

0:29And it's no longer really feasible

0:31to deal with all this data instead of our own data center.

0:33There's too much cost associated with developing and building

0:36out a big, robust data center to store this data.

0:40So increasingly, many of us are choosing

0:42to migrate and move our data processing and our data storage

0:45into the cloud.

0:46Now, when we are talking about data storage in the cloud,

0:49there's some things that we need to consider.

0:50We need to consider our data storage models.

0:52And we need to consider our data processing models.

0:55And if we're going to ingest all this data from all

0:56these different places, well, we need

0:58to understand what's the appropriate way

1:00to handle all that data.

1:02So in this video series, I want to talk a little bit

1:04about that, some of the different standard data models

1:06that you might run into when you're deploying

1:08to the cloud, as well as some of the different standard data

1:11processing or ingestion models that you'll run into.

1:14I'll finish up by talking about a few of the different services

1:16that some popular cloud offerings have

1:18that allow us to handle the data the way

1:20that we want to handle it.

1:21So stick around.

1:22This should be interesting.

0:00[MUSIC PLAYING]

0:05I mean, look, there's probably as many different ways

0:08to store data in a computer system

0:09as there are people on the planet.

0:11Everyone, if they had to sit down and think about it,

0:13would likely come up with a different way.

0:15So certainly, there are a lot of options

0:17when it comes to data storage.

0:18But by and large, the technology community

0:20has settled on two very broad buckets

0:23in which our data storage technology, our databases,

0:27rely on in order to architect and store the data.

0:30And those two buckets are what we call NoSQL data, which

0:33is a little more loosey-goosey unformed data,

0:35and what I want to talk about here,

0:36relational data stored in relational databases.

0:39Now, relational data is this concept

0:42that was developed in the 1970s, and it's

0:44all about how we can logically structure the data

0:47storage on our computer systems so that we

0:50can rely on the database engine software that's already

0:53written and compiled and running in order

0:55to handle that storage for us.

0:56And we can simply ask questions of that database,

0:59typically using something called SQL, Structured Query Language,

1:02and get back results quickly and efficiently,

1:05while also being comfortable and confident that that data is

1:07being stored in an efficient way,

1:09so that everything's hunky-dory, and we're really happy,

1:11and we're really good with how our data is being

1:14stored in our computer systems.

1:17So relational data-- this is one of the more traditional ways

1:20of storing data.

1:21It's one of the more traditional database systems

1:23that you will run into.

1:24And it's really about storing data in a relatively fixed

1:27structure, and one that super emphasizes

1:30efficiency, data integrity, and something

1:33I call transactionality.

1:35So efficiency is efficiency in storage space

1:38to make sure that we don't take up

1:39a whole bunch of extra storage space inside of our computer

1:42systems, we don't duplicate a bunch of data,

1:44we don't have data that's too large or too many indexes.

1:48Data integrity is what we talk about ensuring that the data

1:51lines up, that it matches.

1:52If we have a table that lists a bunch of user IDs in it,

1:55for example, well, data integrity

1:57ensures that those user IDs are actually

1:59represented in a user table.

2:00Those user IDs tie back to a real user.

2:03We don't want to insert a fake user ID or a user

2:05ID that doesn't exist.

2:06That would make our data not make sense,

2:08so that is data integrity.

2:09And transactionality, what I call transactionality,

2:12is the fact that relational data is very good at handling

2:15transactional data.

2:17Think about entries in a bank account or a ledger.

2:20There are just one after the other.

2:22They add or they build on the previous entry itself.

2:25Relational data is very good for that type of data

2:27and that type of data system.

2:29Now, you've likely heard of relational databases.

2:31They are pretty much the most common and most popular

2:34databases out there.

2:35They have been for a long time.

2:36We're talking about Microsoft's SQL server, Oracle, Postgres,

2:40MySQL.

2:41These are all relational database systems.

2:43And one of the things that you'll notice in almost all

2:45of their names are those three letters, SQL,

2:48which I mentioned before, stands for Structured Query Language.

2:54And we'll take a look at an example of some SQL in a second

2:57here, but I just want to talk about this a little bit.

2:59Structured query language is a different standard

3:03that was developed alongside of relational databases,

3:06but certainly not limited or contained to them in any way.

3:09And it's a standardized way to insert data into our databases,

3:13retrieve data from our databases,

3:15and even modify and manage the actual databases themselves.

3:18I have a whole course here at CBT Nuggets

3:20on SQL, both inside of my course on Microsoft Access,

3:24as well as on some courses on the Oracle database system.

3:27So if you want to learn more about SQL,

3:29you want to know more about it, check out those courses.

3:31I really dive deep into it.

3:32I'm just going to touch on it here.

3:33So hopefully, we can understand a little bit

3:35about how it works.

3:37In practice, I think relational databases are very

3:39easy to sort of comprehend.

3:41It's very intuitive.

3:42When you look at a database that is stored in a relational data

3:45system, you're going to look at collections of data called

3:48tables, and those tables are going

3:50to be filled with rows of data, and that data

3:52is split into fields.

3:54Now, this is similar to-- well, the analogy that's often made

3:57is a spreadsheet.

3:58So think about a spreadsheet full of, say,

4:00customers like I have up here.

4:02You've got this worksheet, which is analogous to a table.

4:04This is like a table of data.

4:06And it's split up into rows.

4:07They are numbered down the left-hand side.

4:09And each of those rows is split into separate individual cells,

4:13or fields that we call them, inside of relational database.

4:16And the key here is that each little field contains this sort

4:19of little nugget of data in it.

4:20And it's all related to one another.

4:22All of the data that's on row 5, for instance, right here,

4:25is related to the same customer.

4:27It's their phone number.

4:28It's their email address.

4:28It's their last name.

4:29It's their first name.

4:30But the columns are also related to each other.

4:32All of the data that's in column D here, as an example,

4:35is email addresses for all the different customers.

4:37So we have this relationship inside

4:39of our table between the different rows and fields,

4:42so that we can reference specific individual data

4:45points inside of this.

4:46And this really is what it looks like when you work

4:48with a relational database.

4:49For instance, here is an example of an Access database.

4:52I have a whole course here on Microsoft Access.

4:54It's a great starter relational database

4:55if you want to learn about working with databases.

4:58This is an excellent place to start.

4:59But here, I'm looking at the customer's table.

5:01And look, it's laid out exactly the same.

5:03It looks just like a spreadsheet.

5:04We have our different rows.

5:06We have our different columns.

5:07In the inside of the individual cells, or again fields,

5:10as they're called here, we have the individual values.

5:13So what makes it a relational database?

5:15Well, there's a couple of specific properties

5:18I think you find universal to relational databases,

5:21regardless of whether it's MySQL or Postgres or SQL Server.

5:24Whatever you're using, you're going

5:26to run into these features.

5:27The first thing is your data is what we call atomic.

5:30This is kind of a database term, a little bit

5:32of database vocabulary.

5:33And atomic data just means that the cell or the field

5:36contains the smallest unit of data

5:38that you would ever reasonably or practically want

5:41to look at or collect.

5:43So oftentimes, how that works in practice

5:45is a cell does not have a whole customer name or whole user

5:47name or whole, I don't know, employee name.

5:50But rather, you have a first name cell and a last name cell.

5:52You split those up because sometimes you

5:54only want the last names.

5:55Sometimes you only want the first names.

5:57If you need the whole name, you can retrieve them both,

5:59and then add them together using your software-- connect them

6:02up using software.

6:03But the fields are atomic.

6:04It's got, well, the first name.

6:05That's the smallest value I would ever want.

6:08And there's some kind of wiggle room here.

6:10There would be people who point out, hey,

6:12a letter is smaller than a name.

6:13Why don't you put each individual letter

6:15in its own cell?

6:16And that's because, typically speaking,

6:17you don't need to select or grab an individual letter

6:19of a person's name.

6:20I very rarely need just one letter of a person's name.

6:24On the few times that I do, I can select out the entire name,

6:27and then just use some code again

6:29to strip out the individual letter I need.

6:31So there's always a little bit of sort of wiggle room

6:34there as you figure out what exactly

6:35makes sense for your system and how

6:37you're going to use your data.

6:38But by and large, you can count on your data being atomic

6:42within those cells, and that is really a key property

6:45of relational data.

6:46Also, a key property is that each row is unique.

6:48For instance, we were just looking

6:50at those rows of customer data.

6:52You would not want to have two different customers listed.

6:54Or rather, you would not want to have the same customer listed

6:57twice inside of that table.

6:58That could cause some real confusion.

7:00Why?

7:00Well, you might end up changing the information on one row

7:03and not on the other.

7:03Now you have two rows that represent the same customer

7:06with different data.

7:07Which one's right?

7:08That's how you enforce integrity.

7:09And one of the ways that you enforce data integrity--

7:11by having unique rows, by having data and information only

7:14in one place ever at one time, you

7:16can help to ensure that you don't get conflicting data.

7:19You don't get ambiguous data.

7:20The data exists in one place and one place only,

7:22so the system can enforce integrity on that data.

7:25And then lastly, like I talked about,

7:26SQL is the language that is really

7:29typically used to interact with relational databases.

7:31I mean, it doesn't have to be.

7:32But 99.9% of the time SQL, Structured Query Language,

7:36is in fact, the system or the language

7:39that we use when we want to interact with these databases.

7:41If you're wondering what SQL looks like in practice,

7:44well, it looks a little something like this.

7:47Here, you can see a simple SQL statement,

7:49and it's almost read like English.

7:51I'm saying select first name, last name,

7:53and phone from customers where last name is Like F*.

7:56You can almost read it out like a sentence.

7:58What this is saying is, I want to grab the first name value,

8:01the last name value, and the phone number value from

8:04the customer's table, but only when the last name value begins

8:08with the letter F. That's what this little Like F* is.

8:10That's saying, hey, anyone whose last name

8:12begins with the letter F, pull that data out,

8:14return that data to me.

8:15And inside of whatever system I'm using,

8:18this would allow me to get those records out

8:20of that database really, really easily.

8:23And again, because SQL's kind of standardized,

8:25this statement, this query right here,

8:27can be applied to a lot of different databases.

8:29It helps make your data and your query language a little bit

8:32separated, a little bit unique from one another.

8:34So all of this together is relational data.

8:36And it's a lot we just covered.

8:37I realize that.

8:38If you want to know more about it,

8:39check out those courses I recommended

8:40at the beginning of this video.

8:42Here's what your main takeaway should be.

8:43Here's what you should understand

8:45about relational databases.

8:46They store the data, the information,

8:48in a relatively fixed, rigid, enforced structure that

8:51helps to maintain data integrity and efficiency,

8:55and it helps to support the SQL query language so that you

8:57can structure your queries against that database

9:00with relative ease.

9:01I hope this has been informative for you,

9:02and I'd like to thank you for viewing.

0:00[MUSIC PLAYING]

0:05As I mentioned before, there's a lot of different ways

0:08that we can imagine storing data inside of a computer system,

0:10but THAT doesn't mean that all of those methods

0:12are going to be, well, a good idea.

0:14They're not all equal, and they're not all

0:16going to serve our needs.

0:17So it's probably unsurprising that we have kind of naturally

0:20fallen into a few different standardized data storage

0:23models over the years.

0:25One of the first and earliest data storage models

0:27that was developed is something called the relational database.

0:30That was in the 1970s, and it's still widely popular

0:32and widely used today.

0:33You may often hear referred to as an SQL database.

0:36And I talk a lot about that in my video on relational data

0:39models.

0:40But with the rise of the internet

0:41and the world wide web in the '90s and early 2000s,

0:44we started to develop a need for a different type of data

0:46storage model, something that has come

0:48to be known as the NoSQL model.

0:51And the idea was that, well, not all of our data

0:54fits the model of a relational data model.

0:56Sometimes the data that we get is not rigid.

0:58It's not structured, particularly when

1:00we're collecting it from the wide diaspora of the internet.

1:03Data can be coming from any different place,

1:05from any different system, from a different language,

1:07or any different geographic location.

1:09So we need to be able to be a little bit more

1:11flexible, a little more loosey-goosey

1:13and handle some different things that go on or might

1:15happen with the data.

1:16We also need to deal with a lot of data.

1:18It's no longer the case that we have small really constrained

1:21portions of data.

1:22We are collecting a ton of data.

1:24And so we need to be able to maybe split our database up

1:26over multiple servers.

1:27And traditional relational databases

1:29don't deal well with being split up over multiple servers.

1:32They expect to be located on one single server, one

1:34single computer handling all of that workload.

1:37So enter the NoSQL model, and the NoSQL model

1:41is really about valuing speed and flexibility

1:44over consistency and efficiency.

1:46So consistency and efficiency are two things

1:48that I would generally consider to be features

1:51of a relational data model.

1:53When we talk about NoSQL, we're really

1:54talking about speed and flexibility.

1:57You're going to hear the word ad-hoc a lot.

1:59And there's a lot of popular NoSQL databases

2:01that are around and available now.

2:03You may have heard of some of them, MongoDB or Cassandra.

2:06Redis is a very popular in-memory caching system

2:08as well as a key-value pair database, a NoSQL database.

2:12DynamoDB is Amazon's AWS cloud-based NoSQL database

2:17service.

2:17And in relation to that, CosmosDB

2:19is the one from Azure, their cloud-based NoSQL

2:22model that's served up on the Azure platform.

2:24So these are just some of the different databases.

2:26There's a lot of different NoSQL databases

2:28that you might run into.

2:29And what makes NoSQL important, again,

2:32is its speed and its flexibility,

2:33its ability to really deal with a lot of different stuff.

2:36Now, NoSQL was a phrase that was originally

2:39developed to mean no SQL at all, but really today, most people

2:43will indicate that it means not only SQL.

2:45This is really the terminology that's been adopted.

2:48And that's because usually you're

2:49not going to have an SQL interface to NoSQL databases,

2:52but you still can.

2:53People are very familiar and comfortable with SQL.

2:56So a lot of NoSQL databases, while they

2:58don't store in a relational model,

3:00will still use the relational SQL language or some variation

3:04of it to allow you to interface with that database

3:06and put data in or query data out.

3:08So you still run into some SQL.

3:10Not only SQL is what the NoSQL typically stands for.

3:14And NoSQL databases like to handle ad-hoc data ingestion

3:18from a lot of diverse sources.

3:21Think about IoT, right?

3:22We have all these different devices

3:24that might be scattered all over a warehouse that

3:25might be scattered all over the country

3:27or even all over the world, and they're

3:29pulling in all of this data.

3:30And a lot of things can happen.

3:31They might miss data values.

3:32They might report the wrong data.

3:34The data may not quite fit the standard definition of the data

3:37that you expected inside of your data schema.

3:40NoSQL databases are ready to handle that.

3:42They're ready to say, OK, I can take

3:44on a lot of this different data, and I'm

3:45going to be able to be flexible enough

3:47to know that even if it doesn't fit what I'm expecting,

3:49I can still store it, capture it in some way that

3:51can be understood and read later down the line by future systems

3:55so that it can either be discarded or cleaned

3:57up or otherwise addressed in the way

3:59that it needs to be addressed.

4:01NoSQL databases also take advantage

4:03of public cloud offerings to really

4:05fully realize their potential.

4:07I think this is very, very important

4:08because one of the things that we lose with NoSQL databases

4:11is efficiency.

4:12That means we need more computing resources,

4:15more memory, more CPU power.

4:17Well, that stuff is expensive, or at least, it

4:19was expensive until the rise of public cloud offerings.

4:22The ability to simply pay for what

4:24we need to use to take advantage of a big, already

4:27installed managed data center like Microsoft Azure

4:31to run our NoSQL databases means that we can really

4:34get a lot of the advantages of NoSQL

4:35and minimize the impact of that loss of efficiency.

4:40Now, the relational model is fairly straightforward

4:42and fairly understood.

4:43NoSQL is kind of a catch-all bucket for a lot of things that

4:46are not the relational model.

4:47So there's-- well, again, a million and one different ways

4:50that you might store data and call it NoSQL,

4:53but by and large, we've settled into a few different

4:55standardized data models.

4:57The really popular ones that you're going to run into

4:59are going to be something called key-value pair NoSQL data

5:02models.

5:02And this is just like it sounds like where

5:04you have a key and a value paired up together.

5:07So I can look up inside of my NoSQL database,

5:09and I can say, hey, what is the value of the name?

5:11The database will shout back to me, oh, that value is Ben.

5:14That's a key-value pair.

5:15You see key-value pairs a lot in memory caching.

5:18Redis Cache is a key-value pair database.

5:20You also see it in things like configuration files

5:23where you just have a text file with a list

5:25of different configuration variables

5:27and what each of their value is.

5:28That's a key-value pair database.

5:31The other common NoSQL data model that you'll run into

5:33is something called a document database,

5:35and a document database is kind of like a collection

5:37of key-value pairs that are all bundled up.

5:39So you'll see here, I've got name Ben,

5:41but that key-value pair is actually

5:43part of a larger entity or a larger object

5:46that we call a document.

5:47And those key-value pairs may not just

5:49be simple standard values.

5:51For instance, here I have a key called order items,

5:54and its value is actually another document with a series

5:56of key-value pairs.

5:58So in a document database, you may well

5:59have nested documents or nested lists,

6:02things of that nature, more complex data structures.

6:04A great example of a document database is JSON.

6:07Have you ever worked with JSON data?

6:09That's basically a document database.

6:10Another good example is an HTML document.

6:13You may have heard of the DOM, the D-O-M.

6:14That stands for Document Object Model.

6:16That's a way to look at your HTML document

6:18and treat it like a NoSQL data document.

6:23So some other less common NoSQL data models but still very,

6:26very important and you'll want to understand them

6:27are going to be things like a graph database

6:29where you take those documents and you actually

6:31create relationships between them

6:32so that you can understand how those different objects, how

6:35those different documents relate to one another, column

6:37databases, time series databases,

6:39object storage, which is how you store binary blob objects.

6:42Think about things like video files or media images,

6:46large binary files that don't have a lot of discrete data

6:49but are rather large big chunks of objects

6:51that you just need to store somewhere efficiently.

6:54So all of this comes together to form

6:56the backbone of what we call the NoSQL data model, not only SQL.

7:00And these are the ad-hoc, fast, less efficient, but more

7:04responsive data models that are really frequently used

7:07for a lot of data operations inside of the web and inside

7:10of the cloud these days.

7:11The most common data NoSQL data model

7:13that you've interacted with is the search engine.

7:16Every time you interact with a search engine on the internet,

7:18that data is stored in the back end

7:20inside of a NoSQL structure.

7:22That is not inside of a relational database.

7:24If it were inside of a relational database,

7:26you would never be able to get the responsiveness.

7:28You would never get the speed out of those results

7:30that you see.

7:31So the NoSQL model sacrifices some things.

7:34Those databases take up a lot of space.

7:36They are not especially efficient with space usage.

7:38But in return, you get very fast search results.

7:41So that is the NoSQL data model.

7:43I hope this has been informative for you,

7:44and I'd like to thank you for viewing.

0:00[MUSIC PLAYING]

0:05So we've talked about relational data.

0:07And we've talked about NoSQL data.

0:09And the question now is, well, which one do I use?

0:11I don't know.

0:12The answer to that is going to depend on your workload

0:14and your requirements.

0:15I think you understand that the different pros

0:17and cons to these two different models

0:19can apply in different scenarios.

0:20And I would venture to guess that in most modestly sized

0:23organizations, you're definitely going to run into the need

0:25to use both of them for different types of data

0:28that you might be collecting.

0:29For example, maybe your customer order data

0:31is being stored in a transactional MySQL database,

0:34whereas feedback, customer feedback, coming in

0:36over a web form, might be getting

0:38dumped into a NoSQL database.

0:40So you'll have these different database models

0:42used in different systems and different scenarios,

0:44depending on the type of data that you're receiving

0:46and how you want to store and manage that data.

0:49Now, one important thing I think to keep in mind

0:51is we're talking about this in the context of public cloud

0:54providers because these cloud providers have really opened

0:57the floodgates for NoSQL data.

0:59One of the number one limiting factors

1:01for adopting a NoSQL model in the past

1:03has been the cost for setting up the computing infrastructure

1:06to support it, right?

1:07It's not especially efficient with our computing use.

1:09That's why we need a lot of power behind it.

1:12But public cloud providers take a lot of that cost

1:14off our shoulders.

1:15By offering us the ability to pay for what we want to use

1:17on their service, we can simply spin up these

1:21Platform-as-a-Service NoSQL services and only pay

1:24for the amount that we need to use without worrying about

1:26the upfront cost of setting up these large data centers.

1:29So how do we compare and contrast

1:31relational data versus NoSQL?

1:33Let's just get right down into it

1:34and start talking about how these things are the same

1:37and how they're different.

1:38And I'll start by talking about core storage.

1:42In relational and NoSQL data, the core storage is actually,

1:45I would argue, exactly the same.

1:47It's some discrete data value.

1:49This is good news.

1:49Regardless of which data model you choose, at the end

1:52of the day, you're trying to do the same thing.

1:54And that is store some individual discrete value

1:57of data.

1:57Maybe it's an order date or a customer number or a latitude

2:01and longitude.

2:02Whatever it is, that's what you're trying to do.

2:04You want to store it, save it somewhere inside

2:06of your computer system with the ability

2:07to relatively easily retrieve it when you need to retrieve

2:11it to include it on a report or output it

2:13in some other fashion.

2:15So both of these different data models

2:16ultimately boil down to the same thing-- storing data.

2:20Oftentimes in a relational model,

2:21we call that core data value a field, whereas in NoSQL,

2:25it's frequently referred to as a value.

2:27But certainly, that vocabulary can change.

2:30It's at that point where these two models start

2:32to diverge or differentiate.

2:34Relational databases store the structure, the data structure,

2:37instead of tables.

2:38Those fields are gathered up into rows that are

2:40collected inside of a table.

2:42And a table is a collection of rows

2:43that are similar to one another, say, customers, right?

2:46Your customers table will have data values, fields,

2:49for all the different information

2:50that you collect about a customer.

2:52And all of those different customers

2:53will be on individual rows that are grouped collectively

2:57inside of the customers table.

2:58Your orders will be in a different table

3:00that will have all the different data values about orders.

3:03In a NoSQL model, you're looking more

3:05at things like, well, maybe keys or entities or documents,

3:08right?

3:09You can have these different values collected up

3:11and organized and structured in some different ad hoc ways

3:14and some different vocabulary that's used.

3:16So you can have something that looks a lot like a table.

3:19A document is oftentimes very similar to a relational table.

3:22And they can be somewhat compared to one another.

3:25But they're not the same thing.

3:26So we have a lot of divergence here.

3:28And one of the big ways that they diverge

3:30is in schema enforcement.

3:32On a relational database, this is strong.

3:34On our NoSQL database, really, it's very weak or, in fact,

3:37nonexistent.

3:38What do we mean by schema enforcement?

3:41With a relational data, our tables

3:43and our rows and our fields are very strongly defined.

3:46We will indicate for a given field what type of data

3:49it has to store, how big or how long that data can be,

3:53whether it's a date or a numeric field or a text field.

3:55There's going to be a lot of rules about the type of data

3:57that can go into that field.

3:59And the reason we do that is because if some data comes

4:01along that does not meet that requirement, well, the database

4:04system itself, the relational database system itself,

4:07will kick that data back and say, hey, that data is no good.

4:10It will bubble up an error, so to speak.

4:12And then in our overwriting system, whatever

4:14is collecting that data, we can then

4:16choose how we want to actually handle that error

4:18and what we need to do when that data comes in bad.

4:20Typically, that's not the case with a NoSQL database.

4:22With a NoSQL database, we can handle bad data values

4:26that come in because, well, the schema is not

4:28necessarily enforced.

4:29If someone types in some letters in a field that's meant

4:31to store the phone number, we can probably

4:33just take that in and store it inside

4:35of that particular entity.

4:36That particular value can have numbers in it--

4:38or letters in it, rather, even though every other value

4:40for that entity has numbers in it because it's a phone number

4:43field.

4:44Same thing with the actual specific values themselves.

4:47What happens if your database does not

4:49have a specific field defined for middle initial or middle

4:52name and yet you get some data from a customer that

4:55has their middle name included?

4:56Well, in a relational database, there's nothing you can do.

4:59Your choices are to either stick that middle name

5:01in with the first name or in with the last name,

5:03or to just throw it out entirely and not store it.

5:05With a NoSQL database, you can choose

5:07to let that data value get inserted into that entity

5:10or into that document-- and only that document.

5:13It doesn't matter that you've got 500 customer documents

5:15in your table and not a single one of them except for this one

5:18has that middle initial.

5:20It's allowed.

5:21You can have that kind of flexibility,

5:22that loosey goosey answer to the data that's coming in.

5:26So a lot of times, for a lot of people,

5:28that sounds a lot more intuitive, a lot better.

5:29You're like, hey, I don't want to lose data

5:31if weird data comes in.

5:32I want to collect it.

5:32And that's a good reason, a good choice,

5:34for choosing NoSQL databases.

5:37But that also means that any data validation that you really

5:40rely on, you probably have to develop an overlay on top

5:43of the NoSQL database yourself.

5:45It's another piece of software that you have to write.

5:46It's more code that you have to maintain.

5:48It's more work that you have to do.

5:49As opposed to the relational database,

5:51if you know that you really want to enforce these strong rules

5:54on your data, the relational database

5:55is already doing that for you.

5:57Once you set up your database and establish

5:59that those are the rules for the data,

6:01errors will get bubbled up to you when the data does not

6:04meet those requirements.

6:05So there's pros and cons to both of these.

6:07But that's how the schema enforcement

6:09differs between relational and NoSQL databases.

6:12Relationships in a relational database,

6:14we always have relationships, specifically

6:16between our tables, and usually something called normalization.

6:20I cover normalization in another skill in another video.

6:23I would look that up here at CBT Nuggets

6:25and watch that if you're curious.

6:26Normalization is very cool.

6:28But it's really the rules about how

6:29the tables should relate to one another inside

6:31of a relational database.

6:33Inside of a NoSQL database, well, sometimes you

6:35have relationships, but not often, right?

6:37A key value pair doesn't really have any relationships in it

6:40whatsoever.

6:41A document database, yeah, there might

6:42be some relationships between the two different documents.

6:45But they're probably, like we said before,

6:47because the schema is not enforced, not going

6:49to be very strongly enforced.

6:50You may have them, you may not.

6:51You can, again, force relationships inside

6:54of NoSQL database.

6:55But you've got to write your own software.

6:57You've got to write your own layer to do that.

6:59With a relational database, those relationships

7:01are enforced for you by the actual database engine itself.

7:04Finally, ease of use--

7:06I would consider most relational databases

7:07to be moderate to expert ease of use systems.

7:10There's not really an easy way to get up

7:12and running on a relational database.

7:14They to be fairly robust, complex,

7:16enterprise-level database systems.

7:18There are some smaller, easier to use databases,

7:20such as Microsoft Access.

7:22But even that has a bit of a learning curve

7:24as you come to understand how it should be used.

7:26NoSQL databases, on the other hand, cover the whole gamut.

7:29There are some NoSQL databases that

7:31are so easy to use and set up and understand

7:32that it can be one of the first database systems

7:34that you ever work with.

7:36But you can also have very robust, very expansive,

7:40difficult to use NoSQL systems.

7:41Think about Cassandra or HBase.

7:43These require a lot of technical knowledge

7:45to use appropriately and use correctly.

7:47So you can really get any kind of level of complexity

7:50that you want out of your NoSQL databases.

7:53Again, the takeaway here, you need

7:55to look at your different data collection, your data storage

7:57operations, and decide which of these is right for you.

8:00And be sure that you are paying attention

8:02to opportunities to mix and match when it makes sense.

8:04You don't need to store all of your data inside

8:06of a relational database.

8:07You don't need to store all of your data instead of a NoSQL

8:09database.

8:10Your data is going to fit different models

8:12at different points in time, both within your organization,

8:14as well as in the life cycle of that data.

8:16So pay attention to that.

8:17And make sure that you make the right choice.

8:19I hope this has been informative for you.

8:20And I'd like to thank you for viewing.

0:00[CHIMING MUSIC]

0:05It's a common complaint-- at least here in the United

0:07States, it's a common complaint--

0:09about how slow our financial institutions are.

0:11You get a deposit made into your bank account,

0:14and it takes two or three business days

0:15before that's actually reflected inside

0:17of your bank account-- which is crazy to most of us, right?

0:20We live in this digital, connected age.

0:21We expect things to happen instantaneously.

0:25Well, I was fortunate enough-- or perhaps, unfortunate enough,

0:27depending on how you look at it-- to actually

0:29have worked for a bank.

0:30And I got to see a little bit about how the sausage was made.

0:32And the reason that it takes so long

0:35for something like your deposits to show up in your bank account

0:38is because banks still rely heavily

0:41on what we call batch data processing.

0:44So batch data processing is an older form of data processing,

0:48but is still widely used.

0:49And the key to what batch data processing is

0:52lies right there inside of the name.

0:54Batch data processing is all about processing data

0:57in a batch-- in a big lump sum, right?

0:59Think about if you--

1:00I don't know, you make cookies in a batch.

1:02What do you do?

1:03Well, you make a whole bunch of dough,

1:04and then you put all the little cookies on the trays.

1:06And then, if you happen to have like a cool industrial oven,

1:09you put all of those cookies inside of your big oven

1:11at once.

1:12And you cook them all at once.

1:13A bakery would do this, for instance.

1:14So that they have nice, fresh cookies

1:16available for all of their customers, all day long.

1:19They cook a big batch of cookies.

1:21Well, batch data is the same thing.

1:22You are cooking, or processing, a big batch of data.

1:26And it's important to understand how

1:27to use batch data because, with the overwhelming prevalence

1:31of the cloud and big data, batch is, again,

1:34a really relevant model for processing data.

1:37It's really important to know how to do this.

1:40So first of all, when and how is batch data processing used?

1:42Well, the number one way you're going to see it

1:44is when you have to interface with older legacy systems.

1:47Like that example I just described at the bank,

1:49they still rely on the mainframe a lot.

1:51The mainframe is still a popular choice of system

1:53because it's so reliable, and because it handles such

1:55a large volume of data well.

1:57They don't really have a good way

1:59to stream or to do real-time data processing

2:02on the mainframe.

2:03That's not what the mainframe is built for.

2:05The mainframe is built to gather up all of its data

2:07into one big file--

2:08one really big file--

2:10and then turn it over and process it all at once.

2:13And then maybe wait 12 hours for data

2:15to all pile up again, and do another big processing--

2:18process another batch of data.

2:20So if you're going to be interacting

2:21with any sort of older legacy systems

2:23you may very well run into the need

2:25to process your data in a batch format.

2:28Batch processing is also great for high volumes of data.

2:32Instantaneous, streaming data processing is great--

2:34Processing all the data as it comes in, that's fine.

2:37But, look, if you've got a lot of users,

2:38if you have a lot of data, if you've got a ton of stuff

2:40coming in, you may not have the computing power

2:42available to handle all that data in real time,

2:45as it comes in.

2:46You may need to lump it up into batches

2:49and process it inside of batch windows, which

2:51I'll talk about in a second.

2:52So high volume in data can often indicate the need

2:55to look at batch data processing.

2:57You'll also run into batch data processing

2:59when you work on something called Extract, Transform,

3:01and Load.

3:02I'm not going to dig into this here,

3:04but ETL processes, as they're called,

3:07are all about taking data out of one database,

3:09performing some operation on it, and then loading it

3:11into another database.

3:12And it is almost, by definition, a batch data processing

3:15process.

3:16ETL is pretty much always using batch data processing.

3:18That's simply how it's laid out, and how it's designed,

3:21and how it works.

3:21And it's still a very common operation

3:23that a lot us do in a lot of systems, these days.

3:26And batch data processing is great for things

3:28like media manipulation.

3:29Maybe you need to add captioning-- closed

3:32captioning-- to a bunch of video files.

3:33Or you need to change your crop, or otherwise

3:36adjust a bunch of images.

3:38When CGI is made for movies, they

3:40do all batch data processing.

3:42They make the adjustments to their individual frames--

3:44their individual scenes.

3:45They load those up into the cloud,

3:47and then they let them batch process and render each

3:50of those frames over a period of time during their batch window.

3:53And then the next day, or two days later,

3:55they look at what they got out.

3:56They look at how their changes are reflected inside

3:58of those frames.

3:59So media manipulation is a really common,

4:01frequent occurrence for batch data processing, as well.

4:04Now, as far as how you actually do

4:05batch data processing-- well, I think

4:06it's fairly straightforward.

4:07I think you probably understand it pretty well.

4:09You've got to load your data up into a batch, right?

4:11You've got to sit there, and wait,

4:13and let your data pile up.

4:14And you need to think about things like your batch window.

4:17The batch window is the window of time

4:20when it's appropriate to run your batch processing.

4:22So a lot of times, your batch window

4:24is going to be like, overnight.

4:25You say, oh you know what?

4:26No one's really using our system from midnight till 6:00 AM,

4:29so we will run our batch processes at that time--

4:32during a period of lower usage of our system.

4:34That's our batch window.

4:35And maybe you're an international company,

4:37and your batch window's at a different time.

4:39But otherwise, you need to somehow identify

4:41what your batch window is.

4:43You've also got your batch size-- how big of a batch

4:45do you want to process?

4:46It's all well and great to say, hey,

4:47I've got a nice 6-hour window, here,

4:49where I can process my batch.

4:50But if your batch size is so large,

4:52that it's going to take eight hours to process,

4:54well, now you've got a new problem.

4:56So you've got to focus on your batch window and your batch

4:58size, which in turn feeds into the processing time.

5:00You also want to think about scheduling.

5:02That's, of course, very important

5:03for your batch processes as well.

5:05Are you going on a fixed schedule?

5:07Are you always processing at midnight every day?

5:10Or are you waiting until your batch reaches a certain size

5:14before you process it?

5:15So once it hits a certain threshold,

5:16now you know you need to process,

5:18and you perform that operation dynamically,

5:20depending on when that data reaches that particular point.

5:23Now, of course, batch processing does

5:25introduce its own challenges, its own questions, of course--

5:28the number one being that sometimes you

5:29will have stale data in your downstream systems.

5:32So if data is piling up into a batch,

5:34and it's not going to get processed until midnight

5:36tonight, all the systems that rely on--

5:38or they're expecting those changes

5:39to be processed and input into them, are going to be stale.

5:42They're going to be out of date until that batch

5:44process is actually run.

5:45For instance, the bank--

5:46They were only running their batch once a night at midnight.

5:49That meant that the system would run its batch,

5:52send the data over to system B. Then

5:53system B would wait and not run its batch until that night.

5:56Hence, it would take two days for that deposit

5:58to be reflected inside of the bank account.

6:00It took time for each of these batch processes to occur.

6:03And in the meantime, the data downstream was stale.

6:06You also run into resource utilization.

6:09It can be expensive to process large volumes of data.

6:12I talked about CGI for Hollywood.

6:15That's a really expensive thing to do.

6:16It takes a lot of computing power

6:18to process those huge batches of CGI that they build.

6:21Obviously, they have the money to do it.

6:23Not everyone does, so you need to consider

6:25what the cost and the resource utilization implications

6:27are going to be if you batch process data.

6:30Otherwise, you still run into batch processing

6:32quite a bit in a lot of different ways,

6:33like I talked about before.

6:35So it's important to understand how it works.

6:37And it's important to recognize when

6:38it's appropriate to use it.

6:39I hope this has been informative for you.

6:41And I'd like to thank you for viewing.

0:00[MUSIC PLAYING]

0:06If batch data processing is the old-school way

0:09of processing data, then the new greatness is streaming data.

0:15Our interconnected world means that streaming data processing

0:18is a lot more viable, and a much more relevant option than it

0:21was in the past.

0:22I mean, after all, we didn't always

0:23have computers and devices that were interconnected

0:26on Wi-Fi and the internet 24/7.

0:29It used to be that devices were relatively standalone

0:31and relatively isolated from one another,

0:33and while there was some network communication,

0:35that communication was slow.

0:37It might be intermittent.

0:38Maybe you even had to dial in and connect to the network

0:41only for a temporary amount of time.

0:42So piling up data, waiting, and then processing it

0:45in a batch format made sense.

0:47Doesn't make sense anymore.

0:48Why not?

0:49Now we can process data in real time.

0:51We can stream it to where it needs to go as it comes in,

0:54and we can reduce the lag in processing data and getting

0:57the answers and the questions and the results out

0:59of that data by processing the stream of data, as opposed

1:02to waiting for it to come up in batch.

1:04I previously used the example of a batch cooking.

1:08Well, a stream is more like handling the drips of water

1:10as they come out of the faucet.

1:11You don't wait for the drips to fill up the sink before you

1:14do something about it.

1:15You're going to handle those drips as they come out,

1:17as they drop out.

1:18Think of stream data as the drip, drip,

1:20drip of data coming into your system.

1:23Now, it's all well and good that data streaming is a cool thing

1:27that we can now handle, but it does

1:28come with some additional complications.

1:30I would argue that handling streaming

1:31data is quite a bit more complicated

1:33than handling batch data.

1:35Those complications come in a lot of different forms.

1:37A lot of different reasons to be a little more cautious

1:40and a little more proactive about your structure

1:42and your rules around streaming data.

1:44For instance, streaming data is going

1:46to be dynamic and continuous.

1:47You might be getting it from a lot of different places,

1:50a lot of different sources.

1:51You have to be always ready to handle,

1:53to accept the incoming stream.

1:55The continuous part is particularly important.

1:57What happens if your receiving system--

1:58your processing system--

2:00is down for some reason?

2:01What happens to the data that's being generated

2:03and is expected to be processed during that time?

2:06Does it get queued up and wait somewhere,

2:07or does it get lost in the ether?

2:09Do you care, or do you not care?

2:10So that's a really important part.

2:11And of course, because it's dynamic,

2:13you don't necessarily always know

2:15what the structure or the format of that data is going to be.

2:17I mean, you may well have control over all these devices

2:20that are providing your data, but it's

2:22coming in in real time.

2:24If you're not doing any preprocessing or any assurances

2:26to assure that it meets the structures that you exactly

2:29expect, it might be a little more dynamic.

2:31Might be a little different, and a little more flexible

2:33than you're used to dealing with in a structured batch data file

2:36format.

2:37And that, of course, is the same thing

2:39that we're talking about here, when we talk

2:40about more diverse sources.

2:41Typically, streaming data is coming not just

2:43from a single device.

2:44It's coming in from lots of devices.

2:46Think IoT-- the Internet of Things.

2:48With IoT devices, you might have dozens, or hundreds, or even

2:51thousands of little sensors or devices

2:53out there all producing data at the same time, as well

2:56as lots of other different things-- people's phones.

2:59You might have a website that's building streaming data.

3:01All these different sources, logs, social feeds--

3:05anything you can imagine might be producing streaming data,

3:08which means your processing needs to handle

3:10all these different sources.

3:11There are a lot of different types

3:12of data that might communicate in different ways,

3:14and might structure their data in different ways.

3:16You've also got to deal with and understand the lag

3:19that happens between data processing

3:20and what I'll call data finalization.

3:23And what I mean is with a batch data feed,

3:25well, you know when you have all the data,

3:27because you get all the data at a certain scheduled time,

3:30or when it reaches a certain point.

3:31That is the entire data set.

3:33As you're streaming data and processing data in real time,

3:36you don't necessarily know when you've

3:38gotten to the end of the data.

3:39Just because there's a small gap,

3:41or data wasn't sent over a short period of time,

3:43maybe there's still additional data to come.

3:45It just hasn't shown up yet.

3:46So it's a little more blurry.

3:47It's a little more difficult to identify

3:49where that demarcation line is.

3:51When are you done getting data and you need

3:54to consider it finalized?

3:55When is the end of the day of data,

3:57or the end of the group of data that you need to process,

4:00and does it even have an end?

4:01Maybe it doesn't.

4:01Maybe you're just constantly continually streaming data

4:04and analyzing it like you would with, say, logs.

4:07Streaming data is also very latency sensitive.

4:10And this is important.

4:11We get a lot of help from this data point with public cloud

4:13providers.

4:13They offer a lot more resources.

4:15They are much quicker and faster, and more resilient

4:18than our own data centers.

4:19But the simple fact of the matter

4:20is if are streaming data, you need

4:22to be very concerned about the latency of your responsiveness

4:24of your systems, because data is continually coming in.

4:26You're not waiting for a batch to be finished.

4:29You were just getting the data as it comes,

4:31and you've got to process it quickly, and move it along,

4:33and do what you need to do.

4:34So any sort of latency or intermittent breakups

4:37can be potentially very dangerous for your data stream.

4:40Now, typically when we talk about data streaming,

4:43we really look at it in two buckets--

4:45two processes or models for handling streaming data--

4:48and that's going to be a push queue or a pull queue.

4:50A push queue is kind of like that faucet

4:52that I just talked about.

4:53The systems, or the devices that are generating

4:56the data are pushing it out as they get it, right on time--

4:59just bink, bink, bink.

5:00They're constantly throwing that data out there

5:02to the processing system.

5:04This would be something that's happening over a network

5:06communication, maybe a web server as the processing

5:08system.

5:09And it's sitting there with a public endpoint.

5:11And all of your devices are sending their data

5:13to that public endpoint as they receive it in real time,

5:15just getting thrown or pushed to the processing system.

5:19The other way-- the pull queue-- relies

5:21on an intermediary system, where the data is stored temporarily.

5:25So the data is put into the queue-- into the pull queue--

5:28and the processing system then, on its own schedule,

5:30on its own terms, pulls those data pieces off of that queue

5:33one by one.

5:34This is something like the pub/sub model

5:36that you may have heard of.

5:37It's a little more complicated, but it

5:39gets you a little more robust and a little more predictable

5:41data processing options out of it.

5:43So when you're talking about setting up a streaming

5:45processing system, you want to understand

5:47if you're going to be pushing or pulling,

5:48and what systems you're going to rely on to do those things.

5:51So that's a little bit about how we process data in a stream,

5:54as opposed to in a batch.

5:55I hope this has been informative for you,

5:57and I'd like to thank you for viewing.

0:00[MUSIC PLAYING]

0:05When we're talking about all of this stuff,

0:07it probably makes sense to get down

0:09to a little bit of the practical matter of it

0:11and how does this apply to our relationship

0:13with data in the cloud.

0:15Well, when you roll out your cloud services,

0:17you need to start to think about what services on which

0:20cloud platform or which cloud provider you are actually

0:22going to take advantage of.

0:24And all the things that we've talked about here--

0:25relational databases and streaming versus batch data

0:28and NoSQL--

0:29it all ties into those decisions.

0:30And now I'm going to help you make that decision.

0:33So let's set a few baseline ground rules about the cloud

0:36first.

0:36I want to talk about three different ways

0:38that cloud providers offer their services to us.

0:40Now, not everything falls neatly into these buckets,

0:43but they are relatively good measures or metrics

0:46for how you can look at the different cloud services

0:48that you're going to run into.

0:50First, there's infrastructure as a service.

0:53I think of Infrastructure as a Service, or IaaS,

0:56as kind of the baseline, beginners'

0:58all-purpose introduction to the cloud.

1:00With infrastructure as a service,

1:02the cloud provider, whether it's Google or Azure or Amazon

1:05or whomever you're using, controls

1:07all of the physical components of what you are deploying.

1:11They are in charge of the building

1:12that the data center's in.

1:13They're in charge of the data center itself.

1:15All the networking equipment, all

1:16of the computers-- they install all

1:18of those bare-metal computers.

1:19They are in charge of almost no software.

1:21The only software that they're in charge of

1:23is the hypervisor that's running the VMs.

1:25At that point, it's handed off to us

1:27as the consumers of the cloud to install whatever operating

1:30system we want in these virtual machines, whatever

1:32patches we want installed in those operating

1:34systems, whichever versions of whichever

1:36software that we want.

1:37We call this infrastructure as a service.

1:40Now next in line after we go up a level

1:43or so from infrastructure as a service

1:44is what they call Platform as a Service, PaaS.

1:47And in the platform as a service model,

1:50the cloud provider takes over control

1:52of not just the underlying hardware,

1:54but also the underlying platform that's

1:56running on that hardware.

1:57So they are going to control the operating system that's

1:59installed on those virtual machines.

2:01They are going to control the memory management, the CPUs,

2:04how much patching levels or what security levels are

2:07on those operating systems.

2:08So there are some things that they will expose to us

2:11as the users of that platform.

2:12So we can make some choices from dropdown menus

2:15or using command-line interface options.

2:17To adjust some of those features,

2:19we can choose from a limited list of features.

2:21But really, the cloud provider's in charge of all of that.

2:24We are just spinning up instances of that platform

2:27in order to deploy our code.

2:28What we are deploying out there is our own customized code,

2:31usually something like JavaScript or Node or maybe

2:35Python.

2:36And those services are going to host that code

2:38and execute that code for us.

2:40But we don't have control over the underlying platform itself.

2:43Then the top level, the most abstract level,

2:45is what we call Software as a Service.

2:47This is where us as users--

2:48we just take advantage of fully functional, fully running

2:51software.

2:51We don't really have to do much configuration, if any at all.

2:54Think about the way you might use something like Hotmail

2:57or Outlook.com, right?

2:58You just sign up for account, and you've

2:59got a full-fledged email client available for you to use.

3:02That is a software as a service.

3:04Office 365 in the cloud or Google Docs, right,

3:08all of these different Google applications

3:09that you might use in the cloud--

3:11these are all software as a service options.

3:13Now, there's one additional one that I do want to mention here.

3:16And that comes a little bit in between the software

3:18and the platform as a service.

3:20It's kind of a little bit of both.

3:21And that is the database as a service.

3:24And you can probably guess why I think that's important.

3:27A lot of the database tools that you look at in the cloud

3:29are actually a database as a service.

3:31They're more like the platform as a service

3:33and software as a service.

3:34But really, they share some of the features

3:36of both of those things.

3:38So let's talk about what some of these options

3:40actually are when we're looking inside of the cloud.

3:43To begin with that exact database as a service level

3:46that I was talking about, you'll find relational and NoSQL

3:48database options available to you.

3:50And I'm just talking about three very popular cloud providers

3:53here--

3:53Azure from Microsoft, AWS from Amazon,

3:56and Google from, well, Google.

3:57And they all offer something-- if you

3:59are using a different cloud provider,

4:02they probably have something in this realm

4:03as well, at least the larger ones do.

4:05And this is going to be where you simply

4:07spin up an instance or multiple instances of a specific server.

4:11If you go to use Azure SQL DB, you

4:13will be spinning up a relational SQL server database.

4:16Azure will control everything that underlies that platform.

4:19You don't have any control over the platform,

4:21the bare-metal hardware.

4:22You are spinning up individual Azure SQL databases.

4:25So I call this database as a service

4:27because the management happens at the database level.

4:30You spin up individual database instances.

4:32There are also NoSQL options available on all three

4:36of these platforms.

4:37On AWS, it's DynamoDB.

4:38On Google, its Datastore, which is being renamed to Firestore.

4:41And on Azure, it's something called Cosmos DB.

4:44These are all database-as-a-service NoSQL

4:47databases.

4:48So that's likely what you're going to run into.

4:50I think these are the most common database deployment

4:51options.

4:52Inside of the cloud, these are the ones

4:53that are going to be easiest for you to get your head around

4:56and understand when you want to move to cloud-based database

4:59providers.

5:00But there's other options.

5:01For instance, there is a software as a service storage

5:04model on all of these.

5:05Azure calls it Azure Storage, AWS calls it AWS S3,

5:09and Google calls it Google Cloud Storage.

5:11These are largely NoSQL providers.

5:14They are not relational database provider engines.

5:16They are NoSQL engines.

5:18Although they are fairly unique and fairly typical,

5:20they're somewhat object- or entity-focused.

5:23But they're really very interesting

5:24software-as-a-service NoSQL options,

5:26really great for storing large blob or binary data files.

5:30If you need to store a bunch of images or media

5:32files somewhere, these are great options,

5:34things that you might use.

5:36Azure your has one interesting, I

5:37think, somewhat unique service that's basically

5:40a relational platform as a service called the Azure

5:42Managed SQL Instance.

5:44This is relational.

5:45It is an SQL server database.

5:47It's not quite a database as a service because you do have

5:50some control, some managerial control over the actual

5:53instance itself, but not as much as you have

5:55in an infrastructure-as-a-service

5:57option.

5:58And then lastly, we do have that infrastructure as a service

6:00itself.

6:01And this is-- well, this is virtual machines.

6:03This is when you can simply spin up

6:05whatever operating system and whatever database software

6:07you want.

6:08That's why I've listed it as either relational or NoSQL

6:10because, really, you can choose any database

6:12software that you want.

6:13You have complete control over what software

6:15is running on those machines.

6:16So the infrastructure-as-a-service

6:17level is maybe where you migrate initially your on-premise

6:21database applications to before you deploy them or before you

6:24migrate them over to one of these more abstract,

6:26more heavily managed models.

6:28So all this together makes up some of the different cloud

6:32data store options available to us, both relational and NoSQL.

6:36There are also going to be options about data ingestion

6:39that will impact whether or not you

6:40choose to use streaming data options versus batch data

6:44options.

6:45Now, unfortunately, when we talk about stream processing

6:48versus batch processing, it's not

6:49nearly as straightforward and simple because, well, there's

6:52a lot of different options that play

6:53a lot of different roles in the workflow.

6:55Whereas relational versus NoSQL is really

6:58about which particular tool you're choosing to use,

7:01processing data is about an entire workflow.

7:03And there can be a lot of options.

7:05Take this example from the Azure documentation

7:07that I just pulled up.

7:08Notice all of the different tools

7:10that are available to us to use when we're batch processing.

7:13We're talking about data lake store, maybe Cosmos DB,

7:15maybe not.

7:16There's things like Apache Spark available to us

7:18inside of the Azure cloud.

7:19And from there, we might go into a HBase or SQL Data Warehouse,

7:22so a lot of different things that can

7:24be used at different levels.

7:26There's a lot of different examples

7:27available for both batch processing and stream

7:29processing.

7:30And unfortunately, it looks like this all over the place.

7:33If you look at, say, the documentation for AWS,

7:35it's just as complicated.

7:37The thing to remember is that each of these tools

7:39has their use cases and has their purpose.

7:41And if you're going to engage with them,

7:43you want to make sure that you read the documentation

7:44and understand that they are the appropriate solution

7:47for the problem that you are facing.

7:48So understand what challenges you have,

7:50understand whether or not you're using stream or batch

7:53processing, and how you need that data

7:55to be handled and processed on what schedule,

7:57and with what constraints.

7:59So this covers a little bit about how

8:00we can leverage these technologies

8:02inside of the cloud, just an introduction to the concepts.

8:04Obviously, we have a lot more to talk about in future skills.

8:07I hope this has been informative for you.

8:08And I'd like to thank you for viewing.