AWS Certified Solutions Architect Professional | AWS SAP | Full FREE AWS Course | Part 2

welcome back everyone this is mike gibbs from go cloud architects and we're so thankful to have you here for the second day of our live certified solution architecture so a professional boot camp um we're excited to have you as part of this aws certified solution architect professional boot camp and we're thrilled to have you here this morning some of you joined our live stream where we talked about cloud architect questions and this morning we released a video that i'm going to post a link to this chat very quickly the reason this video is important is in this video we tell you the steps of literally everything that you need to know to start your cloud to get your first cloud architect job and all the things that are necessary so since we're doing computing training i want to make sure we give you all whatever is the necessary guidance to really build a great cloud computing career i made a new video today on how to become a cloud architect and i've been doing this for ever i've been helping people get their first jobs and these are the critical skills for that so um i wanted to make sure that we provided that again it's free it's a video we released this morning i strongly strongly strongly encourage you to watch this i think it'll do a world of good for your cloud architect careers now that we've talked about that you know we're going to get back into the cloud architect career training and part of that today is going to be storage so today we're going to talk about storage um on the cloud specifically storage on the aws cloud now we're going to cover what you need to know for certification but because me i train architects and i train people to get hired as opposed to just passing certification exams i'm going to try and give you a lot more of what's necessary to get hired when we do our architectural training program so any kind of questions you have feel free to leave them in the chat box we're going to answer as many as we can the format for today's live boot camp is 20 minutes question of teaching 10 minutes to questions throughout the day if you've got friends or anyone that's desires to learn about this kind of training please invite them and if you can help support our channel by liking or leaving a comment on the video um we can appreciate that so let's get involved in the training and let's begin with uh storage on the cloud so the first thing i'd like to discuss is what is storage long before we talk about tuning it we always have to start with what it is how it works and why to use it so let's talk about storage today storage is where an organization keeps the data that's it but when we talk about storage i want to break it down into volatile storage non-volatile storage because storage is so important and it's so critical to your design i want to make sure we discuss this so let's begin with the concept of storage and let's talk about the types of storage but before we do this i want to talk about volatile and non-volatile storage volatile storage is something that goes away upon a reboot so think of the dram in your computer but also think of the storage that's attached to your computing instances in the cloud that is volatile storage you terminated instance and literally all and everything that you have is completely lost volatile storage use cases fine but you can understand how it works and why to use and when not to now non-volatile storage by comparison is storage that actually survives a reboot so non-volatile storage is something that survives a reboot so something like an ebs volume which we'll talk about is going to be non-volatile storage the hard drive in your computer non-volatile storage so when we consider storage we have to understand what it is how it works and why to use it because it's cloud architects we're systems designers and you can't design what you don't understand so we're going to go really deep in talking about storage technologies today so let's talk about block storage first because they're basically we're going to be dealing with block storage object storage file storage but let's we want you to understand all of the networking because if you don't understand the storage you can't design the storage so block storage is a type of storage area network technology so think big giant boxes these big giant raid arrays with hundreds of hard drives connected to the back of these servers typically speaking with fiber channel cards so block storage big storage and radar technology now what's interesting about block storage because first it's network attached storage so the network will be your performance limitation when you're dealing with block storage but block storage takes your data and it breaks it down into these little mini pieces called blocks the good news about black storage which is why it's so fast and efficient for compared to things like object storage is that block storage can be placed anywhere in the environment that makes sense so you can place it where it's fastest you can place it where it's most efficient when you're dealing with block storage you're dealing with a highly performance oriented type of storage our network at least the best you get but because these data your data gets braced in the blocks and the box can be placed anywhere in your network it effectively decouples your storage environment from your server so when you think about a cloud environment when you've got an ec2 instance for example and it's attaching to block storage that storage could be anywhere so understand where it's stored could be anywhere but you're attaching it to the compute instance which is nothing more than a virtual machine and you're making this virtual machine attached to the storage and it looks and feels like it's local storage it's not local storage it's network stores and if you don't have enough network performance your block storage performance is going to be terrible because it's related to network storage so understand that block storage is network storage but block storage is scalable it decouples your storage environment from your host and it's very simple and it's very elegant now block storage is non-volatile no the last thing i want to talk about box storage is it's very versatile you can store anything you desire in block storage operating systems frequently changing files block storages is widely useful for a tremendous tremendous number of different things wonderful so now let's talk about another kind of storage and we'll get into the specifics of ebs volumes and s3 in real depth but that's not good enough if you're an architect you need to be a system designer you just can't design things that you don't understand so we're going to talk about how things work long before we talk about the aws specific configuration thing so please be patient it's coming the next type of storage environment is something called object storage what is object storage object starts is another type of storage area network technology just like block storage but it's different see the thing with object storage is you take your data and you break it down into objects with block storage you take your data and break it down on the blocks but object storage is broken down into objects now this object thing is really cool because when you're using object storage you're basically breaking it down to an object and the object's going to have its own unique id but it's not just a unique id each object has metadata now metadata is basically data about data and because when you're dealing with object starts and you're getting metadata about each piece you can do some really cool things for example you can create a data lake which we'll talk about later in the process which is basically integrating your databases and your storage and creating an environment where you can search and query and do these things and you can do that because of the metadata components of it so object storage is really cool and it's that metadata that makes it so special you can run sql queries on it you can integrate into environments now let's talk about the problems with object storage first and foremost it's not too fast so it's relatively low performance compared to other storage environments but object storage is absolutely terrible for files that change frequently why is object storage so good for for certain things by so bad for other things see by nature and object storage every time you modify a file it creates a new version and every time you create a new version this creates more space more space is it filling up your raider razer if you're on the cloud is going to get more expensive so when you're in this environment you've got to really think about it from a perspective of how do you maximize the performance of your systems object storage is great for things like software distribution creation the data lakes files that don't change like static website hosting not useful for files to change a lot so you gotta understand that in the context of your storage and how you're using it the last kind of storage i want to talk about before we go into the specifics of each of the services that are available for you from a solution architect profession perspective is first file store the storage now what is file storage in your computer you've got a hard drive on the hard drive you put your photos your videos your stuff that's file storage on your computer now if you've ever worked in a windows environment where you map a drive you've got a nut like a drive d or erx or whatever from your network that's also file storage it's just network file storage which is on windows it uses technology called server message block protocol you can you could also create a linux server and create an smb or samba share which would integrate for windows for windows file storage now let's also say you're using a unix environment they also have file storage either the hard drive in the system or you can do network file storage so if you've got 30 or 40 linux machines or unix machines and they need to access the same data no big deal they're going to pull it from a file storage and when you're using linux you're going to create something called a network file system which was invented by sun microsystems a long long long time ago and that's how linux systems and unix systems access file storage so at this point we covered the types of storage environments that exist but now we're going to get into the fun now we're going to start talking about some aws storage environments and we're going to start with our absolute favorite because it's so widely used for backups and recovery and all kinds of cool things is s3 so now we're going to go into s3 so what is aws s3 object storage on aws why did i spend some time talking about object storage if you don't run out of object storages you will not understand what to use which service at the time so s3 or amazon supply storage is just object storage on the cloud now object storage in aws is integrated into so many environments website hosting database backups system backups machine images software def distribution data lakes it's used for so many things so we're going to spend a lot of time talking about amazon simple storage one thing to note this is relatively high availability storage i mean not super high availability but it's relatively high availability meaning 99.99 of the time which is about 52 minutes of lack of downtime per year or lack of access which is still pretty high but it's relatively low in the ground scheme of thing now what's really special about s3 is it's durability so while availability refers to having it available when you need it durability revolves into is it going to be there so for example with aws s3 and 49's availability you won't be able to access your data roughly for 52 minutes a year which is a lot but you know for most organizations can tolerate that but your data is not going anywhere even if you can't access it for an hour once per year your data is not going anywhere it's 11 9s available that means 99.9999999999999 so percent of durable meaning it's not going anywhere they back it up and they've got such redundancy that your data is very safe on aws so before we really spend a lot of time talking about s3 i told you that it's object storage so what is object storage used for so we're going to go architecture first and then we're going to go to the technical leads we start architecture first because architecture is designed and we train architects so that's really we're going to focus on so let's start with the use cases backup and archival of an organization's data one of the primary uses for s3 it's fantastic for static website hosting and we'll be talking about cloudfront and static website hosting but if you wanted to just host a website basically there's a couple files and they don't change s3 is a great cheap way to do it fantastic distribution of content large videos that are going to be downloaded not modified software media wonderful wonderful wonderful use case disaster recovery planet take what you have here move it somewhere else absolutely fantastic and it also gets highly highly highly capable when we start talking about big data analytics and data lakes and big data so the data scientists can have a lot of fun with that as well so when we start looking at s3 let's talk a little bit of it from a technical perspective when you create your s3 data you organize it in something called a bucket and each bucket is really just a container for your files and the way these containers actually work is each bucket is basically giving a full dns namespace which is really cool so you can access these buckets very easily from the from the dns names that come with it now these bucket names can be up to 63 characters include letters numbers hyphens periods and i want you to understand that when you can use a delimiter and we'll talk about that but you can use forward slashes and backslashes with s3 what makes it look and feel like regular storage to the user now remember object storage is not regular storage it's very special object stewards is more like a database than a storage area environment and that basically you've got this big flat storage regardless of what it looks like in buckets and then you've got a database pointer to the storage so think of object storage it's kind of like a database storing things so you can't access things in s3 and make them look like folders when you do that by adding into limiter and we'll walk through that through the process it's really just a pointer when you get this dns name it's a pointer for the database to access your data it's a little bit different than traditional storage because of that if you're going to put your things in s3 which is a great place to store your files that don't change a lot fantastic you have to secure your data and how do you secure your data first and foremost you can create a bucket policy which is extremely granular way to do this so on your policy you're going to create basically an in policy which basically says who's allowed to access what fantastic fantastic fantastic way to set these environments up who can do what basically you create a bucket policy and on these bucket policies um you create something based on im now of course if you wanted to you could actually use basically an access control list like windows would always recommend um you can do that but that's basically going to be read write or full control just like your traditional unix you or windows permissions you've got much much much more granular access by creating something called a bucket policy so just understand that's how you're going to secure it we'll talk more about encrypting your data but this is accessing your database's encryption encrypting of your data is basically what happens if the hard drive is stolen keep that secure or what happens but this is related to how do you secure your systems by not allowing people into them now when we're talking about s3 we thankfully have the ability to use the right version of s3 or object storage that's cheaper for us and aws gives you the opportunity to do that in something called a storage tier and basically a storage tier is pay for performance levels or availability levels or access levels that you need and that's all this tiering is with aws pay for what you need so what kind of storage classes do they have they have s3 standard which we'll talk about we'll talk about s3 infrequent access standard we'll talk about s3 and frequent access one zone then we'll talk about s3 intelligent tiering which is pretty interesting and then we'll talk about s3 glacier so we're going to start with traditional s3 what is it it's your standard high performance high availability high durability object storage on the cloud it's common it's used for frequently accessed data it's great storage and that's your typical use case for this kind of thing standard high availability storage you've got data that's coming into your object storage environment it's going to be accessed a whole lot you want to put it here this is your environment to basically put it here your traditional perfect storage for most of your data and that's s3 standard now what if you need access to your data when you need it you need durable storage but you don't need to access your data that much but you'd like to pay less that's where we get into this concept of s3 and frequent access basically it's much cheaper than the standard s3 you put your stuff in this s3 infrequent access and you pay to access your data so this is really good for infrequently accessed data because for example if you have to pay to retrieve your data and you retrieve your data a lot it's going to be cheaper to use standard s3 than infrequent access so what most organizations do is they have their data we're going to talk a lot more about how to set up a policy in a minute organizations typically may need their data for 30 60 90 days every day and then maybe after that they need it for a period of six months infrequently and then after that they never need it so we're going to talk about setting up a policy where you keep your data where you need it access and then move it to lower your cost that's definitely part of the process and we'll get there so s3 and frequent access high durability high availability but it's only there but you have to pay to use it so not for frequently access files now there's a reduced capability s3 in frequent access we don't recommend this because it's reduced availability but see what this is is basically with s3 typically your data is spread across two availability zones which is really great because that means if you lose one the other one's still safe and protected in this case you're only in one availability zone so your data is much less protected so there's a choice to do this if you need if you have infrequent access data and you care less about it this is a good place to put it because it's cheap but this is only good for data that has less availability requirements now let's talk about moving it around a little bit let's talk about the concept of s3 intelligent hearing this is a very cool service and it happens pretty much it's an automated service and what happens is aws will monitor your data the size of your data the frequency of access and it will migrate to the appropriate type of storage for you based upon what's going to be most efficient in cost in this case cost optimization is managed by aws you never want your cost in general managed by the people that are selling you things having said that there are times that you actually need to this is a pretty good service if you're not completely sure what you're using the data and this will be good for most customers for organizations that have more sophistication on their data and what they're doing and how to use it of course they're going to be better by manually moving things but and building a strategy but this is a fantastic way for most people now the last type of object stewards that we're going to talk about is glacier and glacier which is s3 glacier is basically where you take your data that you almost never need it's really good because the cost of this is really low it is an amazing place to store your data and you can get access to your data when you're by paying for it when you need it but you have to understand that if you need your data it's not immediately available generally about three to five hours waiting so glacier is a great place it's cheap it's a perfect place to store your long-term data but you don't access that often very important remember you don't access that often you have to pay to do it and it could take you three to five hours now you can pay an expedited retrieval fee and if you do that that's totally fine but understand when you do these kind of things it has to make sense so let's look at some life cycle policies and what it actually looks like so when it comes to setting up your data remember i said you've got the s3 for frequent access data and your s3 and frequently accessed for data that you need when you need it but it's not as frequently accessed and you've got deep archival purposes like glacier this is what we're talking about this is how you can create a life cycle policy so a life cycle policy will enable you to place your data in an environment that's going to be most efficient what i mean most efficient exactly what you need when you need it and nothing else so let's take this life cycle policy that i created it's a random policy but it's useful for a lot of organizations in this case let's say the organization has determined that their data is highly valid and they need it every day for 30 days and then after 30 days they need a little bit but they're trying to save costs and then after and that those 30 days passed they never use it again but they want to keep it either archival data training data or anything else so here's what they do we set up a life cycle policy that says store on s3 standard move migrate to is s3 improvement access after 30 days migrate today to uh glacier that'll give you the cheapest access to your training and then once it's in glacier we can't even set up a policy to delete after a period of time this is how you optimize your cost savings it's a fantastic way to do things so now remember i told you that object storage by nature every time you do something it creates a new version that's the way object storage works now because that would get expensive and literally users that would work on frequently accessed files could spend so much money on storage aws has done lots of things to basically make object storage not look and feel like object storage by deleting old versions of objects so for example with the way ada bill is s3 works which is different than traditional object storage even though it's object storage is if i work on a word document and i save it and then i open the same document i save it 10 minutes later and i have 50 people that are all working on the same document there's going to be 50 new versions aws knows that can fill up your hard drives and your storage environment fast so basically what they do is they delete all your old versions on their end but if you want to keep a record and an archive of all your versions you just enable s3 versioning and that enables object start and s3 to work the way it's supposed to be the way it was designed is object storage and typically you can see what we've done in this graphic you put a document into a bucket versioning is enabled and every time you modify the document it creates a new version version 147 148 149 150 all the way through to 154 so we're going to do that now you're dealing with your data and an organization's data is one of its most critical assets and you have to make sure that you protect these assets and this kind of stuff is really important so when you're protecting your assets you want to make sure that people can't delete things what happens if somebody was inside your business and they basically deleted everything you're doing they could put you out of business so generally speaking when we're dealing with storage what we recommend is something in s3 something called multi-factor authentication delete and the way this works is let's say i go to delete something instantly i get this challenge that's sent to me that asks for a one-time password or a code now i could be using like a google authentication application an rsa key id there's lots of applications that can generate an mf multi-factor authentication challenge so i go to delete i get a challenge i enter the special password i have the one time token password i put it in if it's appropriate the object is deleted but it's giving you a safety check why is this so good it is so easy to accidentally delete something and something like this gives you the perfect confirmation and it means if somebody hacks into your system and they try and delete your bit something to hurt your business no big deal guess what we're still in a fantastic position because they can't delete it because it's going to send you the text so these are really good things and this is why we are such big fans of multi-factor authentication delete now we're going to talk about some more things in s3 and when we talk about these things really what we're going to talk about is s3 is not like traditional storage i told you it's organized much more like a database flat flat flat storage pointer pointing to your data so what if you wanted to create something that looks and feels like uh like storage and what i mean by that is i just pasted a link um something that could look like storage and another link of something that could look like storage and we look at this look at that it looks just like unix file storage identical so let's say i created a bucket like a bucket policy that would do these kind of things and i created these little delimiters or splat slashes and forward slashes i can make object storage look and feel like traditional storage even though it's not to kind of keep that stuff in mind that s3 is flat but you can do a lot with it now there's a lot more i want to talk about with regards to s3 but we've covered so much in such a short period of time that i want to see if any of you guys have any questions on this content right now does anybody have any questions on on this things that we've covered with s3 we don't we want to spend upwards of 10 minutes no more talking about s3 um and then we're going to go back to the concept if any of you guys have any questions ask them now and if not we're going to go back to the training i just want to make sure nobody's lost i want to give you guys the best training experience i wanted to look and feel as much like a classroom as possible oh by the way i don't expect you to be able to click on the links that i just posted to the chat um they were just sample links that i just made to just show you how it would work um so first question king things what frequently do enterprise use rss could you not use the abbreviation rss and telling me exactly what you mean yes or so on that link should not be working it's just a sample that i was trying to show you king thanks if you want to tell me what you mean by rrs i'll uh i'll i'll definitely answer it if not i'm going to get back to the content since uh i don't see too many questions which is great i just always want to make sure that you know we don't get you stuck i hate to see you thinking that there's a gap in your knowledge i want to answer it i don't want anything that we do free paid or anywhere in between to put you guys in a position where you're basically not you're lost reduce redundancy storage i've not used reduced redundancy storage in any environment i think reducing your redundancy is a pretty dangerous thing to do so i've not worked within any architecture but to be fair typically when i do some systems i'm dealing with the world's largest companies in the world so they would never do anything like this but it's not very common sir ken is it possible to create folders or new areas in your sd object storage as far as i know you you can't create folders you can create a delimiter which makes it look and feel like a folder so that's what you're doing okay so great great great questions coming in now what would you do if your company is requiring you to do more than just design an architect role well if you want to be an architect you take an architect role if you want to be an engineer you take an engineering role and to uh i can i'm happy to answer that but you know there are jobs that most organizations that hire architects hire architects most organizations that hire engineers higher engineers the only time where there's even a crossover in these things is when it's a tiny organization and they don't really know what a cloud architect is and i actually have a video coming out on this soon cloud architects design that's it cloud engineers build that's it you got to make it for that and if you want guys one am happy happy happy to talk about cloud architects versus cloud engineers anytime we train cloud architects in our organization and because we train cloud architects that's our main thing i'll put a link to our program as well as a 20 coupon code if you're interested in learning architecture certification is 10 percent of the process architecture lessons or are 90 of the process architecture soft skills emotional intelligence those kind of things um critically important so i pop that in there for you guys i'm derek you're more than happy more than happy to provide any kind of training and help i can for the cloud architect community it has two more questions and i'm going to go right back um to the training so um abri can you store video recordings in s3 that's exactly what it's designed for you put a video in there you send a link to it and people download it perfect now if you're going to edit the video on s3 you would never use s3 because every time you would edit it would create new versions and it would be a problem but storage of videos distributions of videos fantastic s3 it's perfect for that so it's designed for sapiro great great question what is meant by automation on aws automation is meant by can you can make one thing do something so if you had an automation engineer someone in the sysops department and they wanted to patch the systems automatically by a python script they wrote that's automation if you basically have a create a policy where as soon as you get a cloud watch alarm that it can go trigger something else through some lambda functions to change something that's also automation we're going to cover that um supercharged that is a really really really great question okay can you store closed circuit tv recordings yes meaning you take your recordings and then you archive them on object storage but you wouldn't be recording to object standards it would have to be recruiting that you're transferring to object storage after the fact next question what level of aws knowledge and certification will be helpful to progress your care as a technical program manager i have a couple people that i'm working with i've got about five of them right now i get my people to the certified solution architect professional level because that's basically the minimum level of knowledge that's necessary for these kind of roles and i work on their soft skills emotional intelligence and those kind of things my students do really well so certified solution architect professional knowledge minimum knowledge of the network knowledge of the data center just like a network engineer just like a network architect and just like a cloud architect because what you're going to be doing as a technical program manager you're going to be doing 90 lift and shift you're going to be going from the cloud to the data i mean the data center to the cloud and sometimes vice versa so you have to understand how all these technologies work so that's why you need to understand the cloud and the network for everyone if you want to be a cloud architect you must understand the network and you must understand the data center because that's what you're designing the cloud is nothing more than a virtualized network or a virtualized data center now let's get back and we'll do some more discussions on s3 three so let's talk about encrypting your data in s3 if you've got data and you don't want to if the hard drives are stolen it could cause damage to your business this is not a good situation and since it's not a good situation um we need to make sure that we do what's really good for the situation so let's look at this we're going to encrypt your data and it's highly recommended that you encrypt your data when you store it what is encryption encryption technologies enable you to do the following it protects your data so that if you don't have the decryption key you won't be able to see it so it's super super important to encrypt your data to keep it safe so that's why these things are so important and yes we do include network and data center training in all of our courses because it's so critical but when you encrypt your data it makes it safe because you can't read the data if it's not encrypted so let's talk about ways that you can actually encrypt your data on s3 now you can use what's called the customer managed keys with aws key management system or sse dash kms so custom management key customer managed keys with the key management system is sse kms now this is really cool because it's a complete key management solution basically the way this works the user manages the master creep and the key management system monitors the data key and the key management system is really great and then it's going to provide an audit trail of how who and when the data was accessed so this is a very automated key management system it's very easy to do you pick the master key and aws manages your data keys great way to secure your data next option is sse-s3 now this is the easiest option but you're giving up autonomy in this option aws is going to manage your keys which means they are going to do all of your key management so all you need to do is use the system they're going to manage your keys they're going to rotate your keys and what's even cooler every object is managed with the unique encryption key why wouldn't everybody use this certain environments certain regulatory environments just mandate that you manage your own encryption queues like i said architects design they meet with customers they find out their business their legal but technical and regulatory requirements a lot of organizations are regulated and you can't use this ssc-s3 so you have to understand that this is why architecture training is different than certification training now let's talk about customer provided keys some organizations are so regulated that they have to manage everything when it comes to encryption keys so this is the customer provided keys or sse-c so and that gives you complete autonomy over your keys and you manage all the keys to the customer so we covered the encryption options so let's realistically talk about tuning s3 for optimal use so we're going to talk in this section we're going to talk about pre-signed urls multi-part uploads range gets and cross-region replication these are the things we're going to be covering in this section those those types of things so let's begin with pre-signed urls everything that you put in s3 is private which is great unless you make it public so you need to enable access super important so you could provide iam access for anybody who want to access your systems which works highly secure this way but you know be cooler if you could just generate a pre-signed url and send it to someone so the simplest way to share files that are stored on there with people that are outside of your environment is to generate something called a pre-signed url basically what you're doing is you're creating a link for the person to download using your encryption key and your stuff so it's going to provide temporary and secure access to your systems so when you make a pre-signed url what you're dealing with is an expiration time because if i made a link and i posted it on the internet i don't want people to speeching up for the next 20 years and using it i want to give it for a certain period of time give access to the person that needs it i want that link expiring and gone so let's talk about what would make you expired and gone you create this link and you which will have an expiration period of time so how do you make it if you make it with your im instance profile you can keep that link active for up to six hours based upon the way you sign it now if you want something longer you're going to use the aws token service the aws token service will buy you up to 36 hours why because it's using a one-time password which is in the form of a token it's much more secure than a password that people can just be clicking on and downloading forever now let's say for example that you wanted to create it from your i am user and create an imuser well that link needs to be along uh last longer because you know if i'm sending it to jade or chris or joe then and they're already authenticated users they have more access so it's i can give them a lot more time access and then if it's a temporary token or whatever the token expires so look at it this way instance profile six hours token service of the day and a half i am user up to seven days temporary token whenever the token expires so that's realistically the way these things are going to kind of work now let's talk really quickly about in ways you can actually improve your performance and we're going to talk about something called multi-part upload so let's first talk about what is multi-part uploads and let's talk about the challenges without using multi-part uploads as it stands let's say you wanted to upload a file or a file system to the cloud which is great here's what's so important if i send a 5 gigabit file or a 10 gigabit or 100 gigabit file which is nothing to the internet what happens if half of my file gets there and then there's some network injection then the rest of it doesn't get there i've got to start over again this is crazy internet performance is not guaranteed when we talked about direct connections yesterday versus vpns we talked about internet performance being best effort with zero guarantees whatsoever so zero guarantees whatsoever so if you're gonna use a multi-part upload and especially if it's coming over the internet too many things can go wrong and even over the network things can go wrong so anytime you're building a file working with files that you're transferring and uploading that are larger than 100 megabytes it is recommended that you use something called multi-part upload what is a multi-part upload it's quite simply does you take a file use what's called the multi-part upload it splits your file onto multiple pieces sends little pieces may end it's going to look for the pieces and it's going to say receive the whole file reconstitute it in one file and store it in rs3 so it stops the problem of what happens if there's network congestion or a problem in your systems tick file file gets broken down into multiple pieces it gets reconstituted into file and placed in s3 this is a highly elegant solution best way to transfer your information on a network especially if there's potential to lose data very smartly designed by aws here now let's talk about um taking your data from one region to another so with aws you have the opportunity to do the following let's say you wanted more availability and more durability potato you could take your data in the us like uh and the east coast of the us and you could then copy it to another region like the west coast or to asia or to europe or anywhere you wanted to be and you could do that in real time with something called cross region replication so what is cross region replication cross-region replication is when you take your data from one region and copy it to another reason and they synchronize real-time why would you do this well a couple of reasons let's talk about the architectural reasons of why you do these things let's say you've got users in asia and users in america and they're all coming to this static website basket so let's say that's what's realistically going on um that it's coming from a static website so the way aws bills you is stuff comes in and they charge you anytime you're transferring data between regions and things like this so let's say you've got a five gigabyte website which is a lot for a website but let's pretend it was five gigabytes and let's say you were serving 150 gigabytes of data or a terabyte of data every day out of your website that'd be a lot of interregion transfer charges like an incredible amount by comparison if you just copied your five gig bug bucket from the us to asia then it would all be local in asia and it would all be local in your u.s consumers and that will reduce the cost so crosstalk some replication it's basically think disaster recovery i think reducing your performance reducing your inner region transfer cost remember with aws they don't just charge you to build a link they charge you to use it too so this is one of those kind of things that you should think about so at this point we've covered s3 now we're going to talk a little bit about instant storage before we get to ebs and i want to talk about what that is what's good about instant storage what's better wellness and storage but why on the cloud computing environment it's not very useful for most things when you uh sorry just pulled water on me when you set up uh when you set up your system the compute instance which is nothing more than a virtual machine so if anybody on an interview ever asks you what is ec2 you tell them a virtual machine i have seen 50 of the people not be able to communicate that in a job interview and literally lose that on the job because they didn't understand the underlying technologies virtualization and an ec2 instance is nothing more than a virtual machine if any of you guys are curious on virtualization i can do some live teaching on virtualization i'm happy to set up a live stream configure virtual machines we can build some containers we can do it all live one can do it together and if any of you guys desire that comment virtualization training and uh in in the in the description below and if i get at least 100 users that desire this i will do some free live virtualization training virtualization training is critical skills for cloud architects and i am always always always happy to provide free trainings for the cloud architecture community but i need to know you guys want it so if you guys want it comment virtualization training and i'll take care of it so when we talk about instant storage let's really talk about what these things are let's look at the architecture of your systems inside of aws look at it this way most organizations at this point that need servers and do a lot of virtualization have standard on these new amd epic servers with these 64 car cpus and a couple of terabytes around and that's what they're sticking on almost all the servers amazon uses them i use them everybody uses them now because of the highest performing servers at about a third of the cost of the price of the intel system they are amazing so when you get one of your if you go to aws and you're looking at one of their server racks you see these systems and these systems are these big vp 128 core more servers a couple terabytes around and they have storage on them and the storage they actually have is basically a whole bunch of nvme drives in a raid environment and they get eight 10 million iops on one of these systems in fact on my systems that my students use uh there were there they have a million plus iops because you can buy a million iops drive literally speaking for a hundred bucks a best draw now this is the iops that's available on these ver on these servers at aws is incredible think about 10 million iops on these virtualization servers now when you use instant storage remember it's probably a hundred to a thousand times faster than the fastest evs volume which we're going to talk about next abs volumes are low performance compared to instant storage critically low pathetically low but ebs storage survives a reboot the instant storage that you actually have is really really really critical because that instant because that stuff is so critical um you know for speed but it it loses everything so could you imagine storing all of your stuff in an instance then it crashes you reboot your instance then proof you lose everything your company's been working on forever so for those kind of things uh realistically speaking that's where you're not going to use instant storage devil's bane i'm going to give you a couple answers to that in the next few minutes to learn how you can do these things because i can and will make recommendations so instant storage super high performance storage super super high performance storage but it goes away upon reboot so understand if you're using the storage like a bare metal server you can install the operating system you would get a hundred to a thousand times better performance than you ever could in an ebs file don't worry i'm important right now we're going to talk about how you maximize evf file performance in a minute so i'm going to answer two or three questions that i've seen popped up in the last two minutes and then we are going to go we're going to go right back to the training first question that i saw was uh definitely why him you need to know this devil's bone give us a budget build please can you use a spare laptop no so to learn virtualization devil's bone which you are going to need is a 16 core xeon server with a minimum of 128 gigs of ram and 2 to 3 drives in our right zero environment because you're going to be taking the system and you're going to be bet literally chopping it up into 50 to 100 virtual machines you're going to do to do what we do with our students you're going to set up active directory you're going to build your firewalls you're going to build virtual machines you're going to build containers all in the same system you're going to need you're going to nfs file servers and and sandbag servers you need a bunch of computers to map to it then you need to basically set up an active directory thing for your im then you need to build the cloud and our students all build a cloud in our program that's the kind of training that employers care about that's the kind of stuff that they want to see on the resume and that's how you get it good it's at least a thousand dollars for a server that can do that kind of training because of that we've done the following we used to tell people just by servers and it's the right thing to do um but what we did is we created our training program and i don't like to talk about our training too much and inside of our training program we give you 16 weeks unlimited access to our servers you build your own clouds you do all these things and our training program including the 250 hours of training cost less than it would cost you to get your servers if you are in a developing nation some people have asked us what do you do for servers and for that i created the cloud architect experience program and that is a program that is basically just server access and again i did that to make it a whole heck of a lot cheaper than what other people could do to buy the servers on their own realistically speaking we are a company of the cloud architect career development program that's how we take in people that's how we go get you guys jobs that's what we do we are so effective at what we do and i've been doing this for 20 years that i received a call from one of the directors at aws the other day he called me and he said thank you michael for all you're doing for the cloud community community said i watch your videos i love them he said thank you so much for training soft skills emotional intelligence presentation skills questions skills how to integrate interrogate people how to do the right things he said it's impossible for me to hire people for cloud architect positions he said your training and what you're doing in every one of your videos is so close to what i need you said i want your people i've already introduced them to one of my people yesterday and he's immediately going on for the next stages of interviews aws feels so strongly about needing cloud architects versus the versus you know people they can just type a bunch of commands in the soft skills and emotional intelligence they've chosen to come to my students team called this friday and all the students that are going to be on this team call on friday are going to get to speak with one of the directors who's looking for 12 people immediately from my student base why does it matter so much if you train with us we love it if you train on your own we love it make sure under all circumstances regardless of what you're doing you're training soft skills emotional intelligence empathy enthusiasm and everything that's necessary to get your job we don't make our training methods a secret you can literally see everything we do we have our free webinars which we do twice per week where we literally teach you how to get a job we tell you all the inner workings of our system you can do it with us you can do it on your own we just want to help you get that cloud architect job that's what we care about i made a video today on how to become a cloud architect it is completely free you can watch that it's super important content these are the videos that have caused all these people to start reaching out to us saying we need your graduate so this is what's required to do it train with us we love it if you don't train with us learn presentation skills jerry weissman's presenting to win awesome emotional intelligence dan on goldman's training is fantastic make sure you do it make sure you do good interview trying all of it it matters more i'm gonna say this again it matters more than your ability to configure things from a command line it's just not critical skills for a cloud architect whereas communication is well i am um i focus on architecture because i train architects if i was going to train engineers i would focus on engineering skills architects do not configure but an architect for 25 years no i haven't configured anything in the last 20 some years but i will tell you that i've met with the ceos of all of the world's largest companies all of them i've helped design most of the largest internet service providers in the world i've helped design most of the largest bank systems and i've helped design most of the world's largest healthcare organizations as well and i haven't touched a configuration command in two decades why because i'm an architect there are positions and we have a video coming out soon um where they put engineering and architecture roles that is not an engineering role as an archit that is not an architectural role it's an engineering role it's so pervasive of people not knowing what a cloud architect is that they just published it in cio magazine what is a cloud architect they were that clear about publishing it because it's gotten so bad um it's so so so bad in the difference so and i'm going to answer one more question i don't like to talk about our stuff for too long and i want to go back to training with teacher sex you want to know the difference between solutions architect training and the cloudex cloud experience branding well you can go to my you uh you can go to my training page you can look at it here's the difference solutions architect training is could mean anything it could mean an aws certification there is no concept of what is what a solutions architect is in the industry dennis use it interior decorators use it i've seen everybody use it um construction people cloud architect training trains you to work on any cloud anywhere anytime solutions architect training specifically typically is how do you work for one vendor and it's a much more limited role cloud architect training can enable you to work anywhere and typically pays two to three times better than solution architect training and cert because it's a job that's bigger more global and helps you work in the right variant environments so so a cache raz which scenario can we use instant storage instead of block storage in the cloud environment you can use it for nothing for the most part unless you buy a dedicated host and if you could remove the hypervisor it would be great then you get true high performance skills cloud storage speeds are so bad i mean just so so so bad um that do we have to do really incredible things to try and make desk performance okay on the cloud we're going to talk about that very soon did i miss anything else um and if you if you want to know the difference you can go to the website that i have everything that i have and everything that i absolutely list is available on my training dot go cloud architects and just to let you know um derek the reason we created these programs is ninety percent of the people don't know what a cloud architect is and then they get the sysops associate the devops associate all these other things they spend a year of their time losing money and in the end they've got all these unrelated certifications on their resume and a hiring manager looks like me and says you want to be a cloud architect or an infrastructure architect what's this devops such as sysup do you even know what you want to be and then we often will pass on even talking to the person because the certifications are so unrelated we don't know what they know we assume they don't know anything look good look strong look focused focus on exactly what you need to know and you'll be good to go the last question someone asked me is is a cloud architect or a cloud infrastructure architect what it what it is is it someone that basically like a network architect someone that knows everything and how the system's designed plus heavy soft skills heavy business skills the ability to do roi modeling the ability to communicate with an executive the community to elicit the correct information the ability to design things the ability to present it to the board the c-level executives the engineers and the architects that's the job so i hope i answered your question let's get back into some cool tech stuff because we're here to do tech training and i'm always happy to talk about what excuse me what we do i love doing it i'm also always happy to talk about anything but let's get back to the tech we're having a tech party today but tomorrow morning if any of you guys desire and i will send out the new training program i am going to do a free webinar and now some people that are on this call can actually chime in if they've learned anything from my webinars we are going to do a free webinar and we're going to tell you everything that you need to do to get your first cloud architecture every step everything that you need to know not only that we'll take you off mute we will uh we'll ask questions we will do anything anything anything we can to get hired i'm going to do something i don't normally do because we've got a smaller group over here how can you contact us and ask questions i'm going to give you my phone number you can text me and after you text me i'll text you back and we can arrange like a 15-minute discussion where we can speak so text me when you want and i will try and i'll have some for my team where me reach out to you we'll have a time to speak so we can guide you to get you to your goals last year last question is leticia um do we get any do you get an ex um yeah we give you certificates and we do everything that we need to do our program is typically about 16 weeks the career development program and that's about our average time from the time we take somebody in to the time they get their first caller architect job in most cases i will come back and answer some more questions but i want to get back to the tech again you have my phone number you have the links to our programs and we would love love love to speak with you so now we're going to talk about the next type of storage in the aws environment and that's block storage now block storage is what you're going to be using for pretty much everything in the cloud so let's go back and let's talk about some block storage we're going to have some fun with us so block storage is where you're going to store your things in the cloud there so versus object storage which we'll talk about in a minute damian will if we get enough we'll definitely happily do some virtualization training so let's walk through block storage lock storage as i mentioned was a type of storage area network technology that takes your data breaks it down into blocks which is pretty darn cool and it's acts like a virtual hard drive and this is what you're going to use to store things because it does not go away with instance reboot you reboot your systems you terminate your systems anything that's stored on block storage will still be there whereas with instant storage you know it's not going to be there so realistically speaking this is what you have to use it's going to act like a hard drive and it's going to look and feel like a hard drive to the system that you're using where does this get really cool because it looks and feels like it's a local hard drive it scales well and we'll talk about how great ebs is but these evs volumes will literally scale to any size you need for so for any of you guys that were infrastructure architects like me we're building the network and we're doing our capacity planning and after we're doing the networking and our capacity planning we run out of space then we've got to come up with a new architecture a new architecture new architecture this is where the cloud shines the ebs volumes will scale up in size as needed so these are really great so ebs volumes are considered to be mission critical use they're nine they're five knives available meaning 99.999 percent of the time which means that volume is going to be available upwards of all but 5 minutes and 5.25 minutes per year which is pretty good availability now ebs is designed for high transaction workloads it's the best you're going to get on the cloud and we'll talk about the best you're going to get on the cloud but it's the best you're going to get on the cloud and then we're going to compare cloud performance storage options with instant storage options or what you could actually get in the data center and why i'm going to cover that even though it's outside of traditional aws training because if you're an architect you need to know you need to be able to architect around the poor performance of the cloud you need to understand what it exists in the data center so we're going to spend a lot of time on that but understand at least this is optimized for high performance high transaction workloads lots of performance options your ebs biomes are in your availability zone they're local to your availability zone in case you see that on the test and they're automatically backed up to another availability zone in case you see that on the test so the good news is the way these systems really back themselves up is absolutely incredible for those of you that have been involved in the tech environment like me for long periods of time here's what i can tell you about a backup so let's say you've got your ec2 instance and your ec2 instance has an ebs biome attached to it which is where it stores data it gets backed up via snapshot the snapshot is a 100 identical copy of the entire drive i want you to think about how elegant and awesome that is it is an identical copy literally a pure identical copy of everything on the system so let's take this you compare it to a tape backup solution tape backup you back up a folder this that and the other thing when you need to restore it you stick in your if you need to restore it traditionally you install your operating system your applications and then you restore your stuff from the tape that's traditional backup not here this is awesome takes your volume backs up your volume to a snapshot all you need to do then if something happens with your system you take the snapshot you create which is a machine image and you launch a new machine image and everything is up and running just like that fast quick great access this is again one of the really cool benefits of the cloud i will call out things of the cloud which it does terribly and i will talk about things that it does well this is a wonderful wonderful elegant perfect solution ebs backup so now let's talking about let's start talking about picking the right kinds of volumes and types so when you're going to be dealing with ebs you're going to choose a volume type and the volume type that you choose is going to be based upon your performance requirements in terms of latency and throughput and we're going to cover the difference as well as input output operations per second which means latency as well as throughput so let's define the terms latency is how long it takes to write to access your system read write read write as fast as possible latency is how long it takes nanoseconds for example how fast is this time to pull or store data to your hard drive that is latency latency is determined by the number of input and output operations you can do per second so the higher your ops the lower your latency i'm going to put this into context performance again i want you to understand good and bad of the cloud and all the trade-offs the fastest i mean the truly fastest violin you could actually get for ebs is 64 000 iops 64 000 iops is the fastest volume you could possibly purchase from aws a 100 hard drive from best buy has a million iops which means that 100 hard drive is 15 times faster than the best drive you can actually get in the cloud 100 15 times faster than something that costs you far far far more than 100 per month on the clock i understand that's the constraint now we're going to talk about throughput throughput is the amount of stuff you can move what do i mean by stuff i like the term split stuff so let's say you had two vehicles going at 75 miles an hour down the motorway or highway whatever term you'd like to use in your country two vehicles are driving it's oh let's say two vehicles are driving at 100 kilometers per hour 62 miles an hour and you've got a porsche and you've got a tractor trailer now the porsche's throughput would be all the stuff you can put in the backseat of the car the trunk the boot whichever you call them in your country the tractor trailers throughput would be all the amount of stuff that you could stick in the entire truck controller which has more throughput the tractor trailer which would have lower latency latency you could look at how fast the car can go from zero to 60 the porsche the tractor trail is going to be slow generally speaking throughput and latency are related but not in aws they are not related so we're gonna they they're based on the way they run their architecture so i'm gonna teach you the kind of drive that you need for the appropriate circumstances the best best best performing you can actual performance you can get in an ebs file is something called ebs provision iops this is where you're going to tell them your necessary input output operations for a second and these ebs volumes are based on high performance ssd storage and they have the lowest latency you can actually buy and they say they're designed for workflows to require high io but you know it's the best you're going to get on the cloud it's good for large databases it's good for apps that need lower latency storage throughput on these biomes is is okay it's limited to about a thousand megabit a second where do you guys think that thousand megabit per second is it's about the max speed that you can actually get on a 10 gigabit ethernet interface so since this is network storage and that's the fastest you can get on the 10 gigabit ethernet ethernet interface that's about your speed that you're getting with a pre-owned pos volume so highest performance option on the cloud lowest latency highest throughput now the next file we're going to talk about is as follows we're going to talk about ebs general purpose ssd binds gp2 now this is relatively good for traditional storage general purpose storage and it's going to give you a good balance of price and performance it's going to be fantastic for boot volumes why are they good for boot volumes because the stuff stays there after instance reboot and that is really cool your things are going to be there whether you're using them or not using them it's going to stay past an instant reboot so good balance of price and performance it's generic ssds it's not nvme ssds that they're traditionally talking about at the speed good volume high iops but lower than the provisioned iops throughput very very mediocre these things have throughput of 250 megabits per second which is relatively slow 100 nvme drive at best buy is 3500 megabits per second a 100 standard ssd drive is 560 megabit a second traditional magnetic hard drive is about 180 megabits per second if it's 7200 rpm low throughput drive but it's great for transactional workloads where would you use these general purpose ssd volumes they make a great boot volume for example on your instance they're also relatively good for a dev and a test environment where you need some good relatively lower latency performance you don't need to want to pay the for the provision iops because these things are a lot cheaper price performance balance relatively good option not good throughput relatively good latency so that's when your use case you need the best use the provision for your databases and things for your dev test environment um use gp2 instances for system booths the gp2 and gp2 volumes or general purposes are fantastic now let's talk about throughput optimized hard drives so if you're looking for lower cost magnetic storage you got to move a lot of data going back to the tractor trailer example versus versus the porsche and you don't need low latency lots of data you don't need to worry about latency what you're going to use is throughput optimized drives they're called st1 what is it it's magnetic storage and a raid array this is low iops because it's magnetic so it's going to be super slow in terms of disk read write but this has relatively good throughput it's got 500 megabit a second which is good throughput for normal circumstances this is great this is what you'd expect to see with standard ssd speeds and you're getting it with magnetic storage at a low price fantastic really good for frequently accessed workloads really good for throughput intensive workloads like moving large video files and things like that the high throughput excellent there's a lot of lead data to store and low latency is not required this is your choice if latency is required not your choice if you've got huge log files this is your choice if you've got to do a lot of sequential raise and write this is your choice not your choice for low latency environments now the next thing that we're going to talk about is ebs cold sc1 lowest cost low iops low throughput at 250 megabit a second basically this is like the equivalent glacier use it for stuff that you don't need very frequently now the next thing we're going to talk about is i'm going to go through raid and then we're going to start talking about we'll take a break and then we'll talk about nfs efs and those kind of things but first let's talk about raid raid or or the redundant array of inexpensive disks is the way that you can improve disk size and disk performance in the most cost effective manner what is raid raid is when you combine the speeding capacity from a bunch of hard drives for performance by speed and we'll talk about the various raid levels when you're using raid you might take 10 20 100 drives combine them together for the speed and the capacity but when you mount it to the computer they're not going to know it's 10 or 100 drives it's only going to think it's one drive and it's going to be really really fast so because of that rate is a great way and organizations use ray to improve the speed the performance and the redundancy of their environment or we're going to walk you through the four raid options that are typically used in the cloud computing environment um the one is not used in the cloud computing environment but it's used in everybody else including probably by aws in their data center and we're going to talk about that rate is a means to combine the speed and performance from a bunch of drives so let's talk about the rate options first we're going to talk about raid 0. when you need this performance raid 0 is the best you've got what is raid 0 you take a bunch of hard drives you take your data drive one drive two drive three drive four drive one or two drive three drive flare drive one drive two drive free drop four all the way through all the volumes by doing that you get four times this or you get it if you've got four hard drives you're gonna get four times the speed fast fast fast no fault tolerance if you're taking a file and you're splitting your file across four drives and one of the four drive cells you lose everything because you don't have the file raid 0 is super fast but it's got zero redundancy lose any disk in the raid 0 array bye bye data so when you're using raid 0 and it's going to look like this you're going to take your data drive drive you see black one business drive one block two goes to drive two block three goes to drive one block four goes to drive two one vice versa lots of speed lots of performance but hard drive two fails everything goes so when you're using raid 0. think back up back up back up back up and back up some more because i promise you using raid 0 you will run into a drive failure it's just a matter when your systems will go down and it will be a horrible experience if you don't have a good things so we'll talk about what the types of raid to use and when and why and how raid 0 is the only way let's say this again some flavor of raid 0 is the only way to get acceptable performance in the cloud i'm going to repeat this again some flavor of raid 0 is necessary to get relatively good performance on the cloud why is that 10 million iops is what i'm getting from my servers that my students work on 64 000 is the max you can possibly get in an ebs file if i needed 10 million iops but i don't but if i needed a million iops i'd have to put 15 it pay iops raid abs volumes in raid 0 just to get to the level of performance that i could get with 100 disk drive from best buy like i said know this translate cloud know the weakness of the cloud if you need high performance disk access on the cloud you're not going to have it you're gonna have to do the best you can which is raid 0. and raid 0 is no foul tolerance so we're going to talk about raid 10 later but you have to understand raid 0 to understand raid 10. so enterprises don't use raid 0 because it's too risky the only time you'll ever see raid 0 used in the production environment as a rule you'll see it on the hard drives for video gaming computers you'll see it for video editors like when i do video editing and i need extreme speed i put a couple drives in raid 0 i use it as temporary storage to work on my files then poop they get off of it but that's the only time you would ever use that you can use it with ebs volumes and it's a little less risky than it would be on an environment other environments because you can do a lot of backups of ebs volumes quite easily and they're perfect backups and these aren't exactly real hard drives i kind of like logical hard drives so just understand radio on the cloud is less bad than it would be in the enterprise but it's still too risky as a rule so now let's talk about raid one so if raid zero striping is across as many hard drives let's talk about raid one raid one is called mirroring what is raid one you take two drives drive one drive two everything goes on drive one copy to drive two in real time drive one fails guess what you got another one so raid one or disk marion provides a real-time backup everything is backed up all in real time here's the thing it's slow why is it slow because you're backing up a 250 megabit hard drive per second that's your max speed it's going to be 250 and then it's going to be the other one 250 and you might even lose some performance due to the backup but you know highly redundant lose one drive and it works if it was rate zero you had two drives you'd have double the speed so this is expensive it doesn't improve capacity its performance isn't great but it is highly highly highly redundant so what does this look like let's go back to our chart that i drew look black one black one block two black two black three block three complete identical mirror image of everything that we need to do that is raid one now next we're going to talk about what every enterprise in the entire world does everybody does it you'll probably see it on aws servers they use some flavor of raid 5 or 50 or whatever what is it raid 5 gives you an amazing balance of speed and redundancy most common rate environment deployed in enterprises throughout the world everyone uses this and raid 5 basically takes your data and it provides a combination of speed performance and redundant basically every enterprise is using it but it's not recommended by aws and we're going to after we teach you about this we'll tell you why it's not recommended by aws the way this typically works is let's say you have four drives you take your data from it'll be block a you'll take data the first block will go to the first drive the second block go to the second drive the third block will go to the third drive and then there's this stuff called parity which we have listed as power to a that's data basically if we lose one of the hard drives to fix it backup data so then we have block two block two power to data block three and what we're doing as you can see is we're cycling through so that every hard drive we have if we have four environments we'll have three quarters of the data and one quarter of the data necessary to reconstitute the information so this raid five is pretty cool we've got hard drives if one of them fails all we do we pull out the drive pop out a new drive we typically push our rebuild button and it copies the priority data from all that to from those other drives to the drive and poop you're up and running and all your data is restored and everything works and you get speed everybody across the world uses raid 5 or raid 50 or some flavor this best and highest performance highest redundancy to be used for everyone why don't we use it on the cloud there is some latency overhead from ready to go this party data it adds latency in an enterprise environment dealing with regular hard drives not a problem the latency it has is minimal and the redundancy is worth it the speed throughput is incredible why is it not good in the cloud if all you have is 64 000 iops which is horrible i mean horrible and that's the max you can get we can't afford additional latency from raid 5. that's why aws does not recommend this in your data center raid 5 is perfect but not in the cloud because you only have 64 000 iops max which is terrible i mean literally terrible it basically is the kind of performance that you would get 10 years ago on a computer system so terrible terrible terrible actually 10 years ago you'd still get better this performance on your on your computer it's just terrible so when you have to use ebs and that's your only choice in the cloud raid 5 is not a good option because your iops is already low but that's block storage is what you're going to have to use in the cloud computing environment why am i still talking cloud computing why do i still love cloud computing even though i can tell you and do a better job in the network in the data center auto scaling agility scaling up and down on demand these things are so amazing that it makes it worthwhile because of that the cloud is terrific but you have to understand the trade-off as an architect you need to say can't bring this to the cloud keep it the data center can bring this to the cloud oh i need to bring this to the cloud we need a lot more disk performance and how are we going to do it so how are we going to get acceptable disk performance in the cloud and not have all of our systems go down if we lose a hard drive i'm going to tell you that's called raid 10. so if you can afford it and this gets real real real expensive but if you can afford it raid 10 is a combination of raid 0 and read 1. so basically the way this works is you're going to have 2 raid 0 you're going to raid 0 rig and a backup rate 0 right so in the primary raid 0 array you're going to get the speed the capacity the performance you desire and in the backup array you're going to get a complete and total backup so raid 0 mirrored to another raid 0 way high speed high redundancy is becoming incredibly popular on the cloud because it's your only option for good performance but it's expensive why is it expensive because you need double the number of drives so if you need 15 pio ps volumes to get to the equivalent of my 100 best buy drive then you need 15 to back it up so you might need 30 times so that it can get expensive really fast so high high high performance phenomenal in every way shape or form but understand just understand that when you're dealing with raid it helps your performance so this is what it's going to actually look like raid 10. we've got a raid 0 environment you can see we're striping striping striping and then we're backing it up to another one so ray 10 is a combination of raid 1 plus right 0. what are your options raid zero is basically stripe data data data data data data data data data data data super fast raid one copy data from drive one to drive two so you can see um let's say so drive one to drive two raid five date data from the first drive data the second drug data third dive parity data and vice versa raid 10 raid 0 backed up by raid 0. now you know the types of raid any quick questions on raid before we move into efx and other types of file storage or piops volumes or anything like that i want to make sure you guys really understand you're doing good you're learning good i want a real classroom like experience for you guys i know there's approximately a 30-second delay between the time i ask questions and the time you guys see it so let's make sure we kind of cover those concepts i'm waiting to see if any of you guys do this while i'm waiting i've received a message from my partners in the organization and they told me that i should include a link to our career development program so i'm going to do so and then we're going to start answering questions so keep the rest of my team happy by doing this the question is why is it different than ray one okay derek if you're asking why raid 10 is different than raid 1 that is an exceptionally good question raid 1 basically takes one hard drive and mirrors it to another hard drive which means if you've got hard drive a that's 10 terabytes hard drive b has 10 terabytes there is no improvement in capacity raid 0 if you've got 10 10 terabyte drives you get to combine the speed and performance of all 10. so you write to the first drive second drive third drive fourth drive fifth drive six drive seven start hr nine drive and tenth drive so you get ten times the capacity because you're using all ten think of it this way you know let's say let's say you had ten drives in a raid zero environment or let's say you have ten friends and you wanna push a car how much faster could your 10 people push a car than a single person push a car what you're doing is you're doing effectively parallel processing for your hard drives and that's why raid 0 was so fast the problem is with all this parallel processing with raid 0 if any one of them fails any one of the 10 drives or your 10 friends would let's say you have 10 friends that are passing a car and one of them falls on the ground in raid 0 the car stops moving that's the difference derek houston did i make that clear for you um please let me know in the comments section if i did great if not i will draw some charts and do anything i need to to help you get better knowledge while we're waiting for derek does anybody else have any more questions because i want to make sure we give you guys a really great training experience okay i'm gonna wait a few more seconds because there's a 30 second delay okay you got it i'm so happy devil's bane with raid 10 is one set of d i don't i don't know what you mean by rw um but uh raid 10 is simply raid zero that bat makes an identical cop up back up to another raid zero volume cup so double the so speed copy everything speed and then copy it to someone else um identical copy whatever's in the first array gets copied to the second array um so no i want so so double bound um all the drives in the single raid zero array um are read write and all the drives in the backup array are read write so um that is just that's really all it is it's just a primary and a backup did i answer your question there i'm going to make sure i answered your question before going back to content or anything like that and wait a few more seconds and then we're going to start talking about efs and then file systems okay so um we'll we'll if if need be towards the end we're gonna in the end we'll have a lot of questions and answer sections and if anybody needs we'll find a time to go back to things but sounds like you basically got it which i'm happy about so let's talk about efs so when you're dealing with a cloud computing environment or any type of network environment what you're actually dealing with is file sharing now when you're using linux or windows systems they use different sharing protocols every linux system and unix system in the entire world for that escort when they share files they use something called the network file system which was invented by sun micro systems about 20 30 years ago it's a little old file system and in the cloud if you're dealing with the nx and linux systems you need nfs because that's how unix and listen seems so amazon offers you a a brand their brown branded version of their nfs and they call it an efs or elastic file system basically speaking it's nfs the network file system that's existed for the last 30 plus years now when you're dealing with it it is what you would consider to be postx compatible storage what i mean by postx compatible storage is it works with old legacy system so efs will work with anything that nfs would work with because it's really much the same thing but it's the amazon branded version of efs there's two versions like anything else in aws there is uh your standard which is your normal best performance and then there's your infrequent access which is low cost stores for files that are on access let's look at your two ways that you could do this you could conceivably set up a linux server on a virtual machine on a giant um ebs volume to it or a raid array of ebs volumes and do an nfs share directly off of that thing or what you could do is you could purchase efs which is just a managed service version of nfs for you to store your data so just think of efs as nfs nothing more nothing else pretty simple like i mentioned it's post-excallable it's as high a throughput as you're going to get on the cloud it's as high iops as you can get on the cloud which means lower the lowest latency now this is really elegant these efs volumes scale on demand so again for people like me that have been working around for a long long long period of time working in tech their capacity planning and changing servers out and adding storage constantly you don't have to worry about this here this is where the cloud shines that agility that it just makes it so beautiful what does it look like you use this managed storage service efs which is basically nfs like i said you could just basically set up if you wanted you could do it yourself on the cloud too you set up efs and then all your unix systems mount to it and they use it as shared file storage there you go that's the way the shared file storage works now let's talk about what do you do with windows clients well a couple of options you could easily do the following you could set up a linux instance match them ebs volume and set up a samba share that's what we do in the data center because very few of us actually use real windows servers why do we not use windows servers they're expensive there's licensing fees and linux servers with sound by shares is much cheaper and more reliable as a rule so most of us basically use uh linux samba but you can also use windows servers and if you have heavy duty windows workloads in your environment meaning sql type things um heavy virtualization environment if and when that happens then guess what you need windows file servers like for sql microsoft sql exchange you know those kind of things so aws gives you the option to buy these hosted high availability windows file servers and basically they just use the server message block cloud calls and they're serv their hosted windows servers that's it and that way you can do any of your file systems active directories quotas literally anything that you can do anything that you could do in a traditional windows environment you can do on these because they're just windows servers so amazon fsx if you see it on exam is servers for windows and that's realistically it so now let's talk about getting your data to aws you've got to get your data there somehow now there's a lot of ways that you can get your data to aws you can transfer it over the network and if you've got enough time and enough network bandwidth this is going to be an exceptionally good solution but what if you don't what if you don't have enough network but if you don't have enough bandwidth which if you don't have enough performance then you're going to need something different first we're going to talk about getting it over on the network and then we're going to talk about the other options to get your data over the network you're going to use something like a storage gateway what is a storage gateway let's talk about it when you're migrating your data center to the cloud or you're doing a hybrid environment you've got to get your data there somehow because what good is that everything you have if you don't have your data your data is everything so what do you do you basically set up a storage gateway a storage gateway is a virtual machine which we're going to call appliance that you put in your data center and typically speaking you're going to put it on a vmware esxi server and you're going to mount your local computers to the device that is going to copy stuff to the cloud and it is a really great way to deal with a hybrid environment and it's also a great way for disaster recovery so you put this virtual machine in your data center you mount it as if it's a regular server and then asynchronously copy stuff to the cloud it's kind of an elegant solution so when you're dealing with this you've got a couple different kinds you've got a file gateway and you've got these volume great waves and stored mode and cache mode and we're going to talk about all of it and we'll talk about tape gateways and backup so let's look first at the storage gateway it's an appliance like i said a virtual machine and it acts like a file server in this machine you can connect to with your windows host via smb or the server message block protocol you can connect your linux systems to it via nff just like anything else server sits in a virtual machine put in your data center and you mount these drives and when you put your data on these drives and you mount it like a server it gets copied to aws s3 so it becomes an object in the ns3 for you and when your things are stored as objects on s3 guess what they're already there they're encrypted already there and then once they're on s3 you can do whatever you need with them so architecturally this is what a storage gateway looks like basically you've got your servers sitting in your data centers they cut they mount via smb for windows or nfs for linux it takes your direct connection to your vpn connection and it copies your data over to s3 and from there you can run any lifecycle policies you want in your own data but it's a great great great simple elegant way to back up your data and get your data to the cloud so now let's talk about volume gateways restored mode in cache mode we're first going to talk about stored mode and then we're going to move over to cache mode so in a volume gateway on stored mode it assumes that 90 of your data is sitting in your data center and this is realistically speaking the same kind of appliance you're going to mount the appliance and it's going to come move your data over to the data center asynchronously and it'll all be backed up as an ebs snapshot on s3 so the when your hosts are using this they're going to mount this device via something called the iscsi protocol which is basically scuzzy um over an internet protocol so that's all that really is and what this is going to look like is as follows in this particular environment your systems access your users access the systems they the application servers mount to the storage gateway virtual machines that you have copies it to the aws cloud as a snapshot backing up your stuff constantly non-stop what an elegant solution it's really terrific now let's say most of your data is actually stored in the cloud now this changes things if your data is stored in the cloud and hosts cannot use object storage as if it's real storage it's just different object storage is very different than block storage or file storage so host can't really map to object storage and use it as a real drive but with a biome gateway they can so let's say you're an organization and you keep your data in the data center in the data center versus your your environment so if your data is in the data center i'm sorry in the cloud and you're working out of your data center how do you pull the information from your data center the cloud that's where this volume gateway cache mode works and it is really elegant so if most of your data is in the cloud we're going to use the storage gateway in the reverse mode what we're going to do is we're going to basically use the storage gateway to pull information from the cloud and then our hosts are going to mount it via iscsi and it's going to make s3 or object storage in the cloud look and feel just like local storage or file storage so this is going to kind of remove the object storage feel out of the object storage in order to do that to help all of you um and be pull your data from the cloud to make it look this way so when you're dealing with this all your data is going to be encrypted with server-side encryption and your data from your data center is going to be local to your cloud so what does this look like in this biome gateway cached environment look at it this way pretend your data is sitting in s3 and your volume gateway is in your data center your volume gateway has mapped to this thing now here's the way it works let's see you've got a user that pulls information a user that requests some information which is actually stored on s3 but the user doesn't know that the user with iscsi makes a connection to the gateway and then the gateway provides the information the gateway if it doesn't have the information it goes to the cloud pulls it from the cloud and goes back to the storage gateway and then sends the information to the user now the second person that requests that information is going to go to the gateway and it's going to say i have it it's going to send it to the user hmm have you ever seen anything like this before avayam gateway basically is a content delivery network and a caching system for your stuff that's stored in the cloud and so so it's the combination between using a caching environment to improve the uh efficiency of what's stored on the cloud and at the same time it's doing that it's actually making object storage and the cloud feel like local storage so i'll explain that relatively well now very simple very elegant but of course if you copy stuff to your data it'll also copy it to the cloud so the volume gateway in cash mode is predominantly for your data that's stored in the cloud but it does allow bi-directional communication and storage back and forth keep your data synchronized keep your data accessed elegant elegant elegant wonderful situation on the on the cloud so now let's talk about tape gateways so for those of you like me that have been around for a long long period of time and we're around for a long period of time you know we work on tapes what did we do with tapes we basically backed up our system to these huge capacity tapes we changed tapes periodically when the tapes around the capacity and then we store these tips somewhere secure often off-site in a fireproof waterproof container to make sure that if anything happened we can restore our data tape gateways or tape storage is still pretty common in the enterprises that have large amounts of data because there's a tape back up it's a very relatively low cost way to back up your data so what did aws do they took away to work with your current tape backup appliances and they created this virtual tape environment so for organizations that have these big tape storage backup environments they're going to take the backup they're going to basically copy it to the cloud and instead of using a tape it's going to create a virtual tape and it's going to copy it to the cloud so it's going to connect to the tapes we have iscsi just like everything else and then it will take the data and we'll back it up to the cloud and on s3 and then once it's on s3 you can move it to glacier or deep glacier it's a relatively excellent way to truly truly truly get your data to the cloud in a low-cost environment here's what it's going to look like so you take your data you're backing up your data you basically have your tape gateway and it's going to copy it to virtual tapes on s3 and then once it's on s3 you can do anything you want with it that's pretty much the way the tape gateways work simple elegant solution so now we've got a couple more very few things before we get into computing things to cover we're going to cover the snowball the snowmobile and the import export service then we're going to stop for a few more minutes of questions and then we'll start talking about computing on the cloud which is the thing i love talking about so first thing we're going to discuss is the snowball let's say you've got a lot of stuff to get to the cloud and you don't have a lot of network availability about and you don't have a lot of time if you've got enough time enough network connections you can move any amount of data you desire anywhere anytime but if you don't you're not in a good position to do these things because of that aws and this is a very smart option came up with something called the snowball what is a snowball it's a highly ruggedized container the highly ruggedized container is filled with lots of hard drives so aws has this ruggedized hard drive high performance computing system they ship it to you you can get either a 50 gig version or an 8 gig version now mind you there's overhead from formatting so the 50 gig version is about 42 terabytes that's usable 40 to i think it's about 40 terabytes it's usable the 80 terabyte version has 72 usable terabytes of options it's got a 10 gigabit networking card so what do you do you plug it into your system of the tangled link you copy your stuff and then you move it to aws so here's how it works you request to snowball from aws you copy your stuff on it you call aws they should get shipped back to them they take your data on s3 and then when it's on s3 you do whatever you need to great great great way to get your data to aws in a fast environment next option for getting your data to aws is something called the snowmobile now let's say you've got a big bank with tons of data and they decided that they're going to all go to the cloud by tomorrow because of some reason this is where the snowboard comes from so if you've got petabyte after petabyte after petabyte of data and you have no time you call aws and they ship you a shipping container which is basically on a truck or trailer with a hard drive after a hard drive after servers it's a big huge data center on wheels you plug it into your systems you copy your data onto the snowmobile which is a basically a shipping container and then the snowmobile is driven back on the tractor trailer to aws and the tim take up a whole tractor trailer and then it's copied to s3 by the aws personnel and then once your date is on s3 you can do anything you want with it that is the concept of the snowmobile the last transport service to get your data to the cloud is something called the aws import expert service and let's talk about what that is basically the import extra service is like a mini snowball basically it's a rental hard drive from aws you copy your stuff to the hard drive and you ship to aws that's it aws after it pulls the information off of your hard drives sticks it into to your environment and poof everything works and you're in great shape that is the way these environments work now in any of these environments the snowball the snowmobile and import export you must encrypt your data and it happens automatically on these devices why is encrypting your data so important when you're shipping it if your hard drive gets lost you don't want anybody to be able to use your data if your hard drive is stolen you don't want anybody accessing your systems that's why it is super important so now you know the way these services work last thing that i want to cover is something that just in case it pops up on an exam i want you to know there's this concept of amazon work docs what is amazon work docs it's a fully managed secure content creation service and storage so think google drive or dropbox it enables collaboration on creative projects document sharing cheap and affordable and they've got clients for windows and mac os and it meets all the compliance standards hipaa pci dss but i hope you so think of it as a as basically a google drive or a or what do you call it um or at dropbox so now let's uh talk about what we covered in this section we covered s3 ebs efs raid we talked about file systems for windows we talked about storage gateways we talked about the snowmobile mobile the snowball the import exosport service and workdocs before we get to computing which we're going to do um let's take it from here and what we can actually do is take some questions and answers give you guys a break we've been working pretty hard for uh about an hour and 40 minutes so if you have any questions let's try and address your questions does anybody have any questions and while we do i would like to invite all of you while you guys are typing some questions because there's a delay i would like to invite all of you to our webinar tomorrow morning it's going to be completely free on this webinar we will teach you everything that you need to learn to get your first cloud architect job certification is about 10 of the process and that's why we give it to you for free but there's about 90 more that you need to learn and we'd like you to learn them so we teach it all completely free and we have that at 9 00 am to eastern time tomorrow it's normally a different time but it'll be 9 a.m tomorrow and we'd love to see you there so let's go through and see if there were any questions over here that were previously typed and if anybody had any questions please ask them now so so the first question i see is from a reef is nfs the same as ebs no nfs this network file system is just like elastic file system with aws elastic black storage is something different so if you if you'd like us to clarify what ebs is again we are more than happy to do so but i want to make sure we understood your question correctly nfs is almost the same as efs efs is the amazon branded version of nfs with some minor tweaks does anybody have any more questions are you all having fun or at least as much fun as you possibly can if you can let me know in the comments section below or in the comment section or the chat box i want to make sure you guys are all learning and having fun if you are please say fun or cloud architect actually type cloud architect would be a perfect thing for you to type if you're having fun feedback is pretty important to us we want to make sure you guys are all happy okay great to hear alexander you're enjoying it thank you for letting us know by using the word clock architect are the rest of you guys having fun or as much fun as possible okay derek we will go back to the vyam gateway cash mode i'll do that right now but i'd like to know if you guys are happy if you guys are happy please type the term cloud architect in the chat box okay awesome now while i go back and find my slide i'm going to go back to the volume gateway in cash mode okay and then i'll ask some questions take it seriously what's the difference between this free course and the paid one take it seriously that is a really really really great question um as this is certification training certification training basically teaches the name of the services how they work and why you would use them on our cloud architect training we teach you everything that's necessary to become a color architect this training that we're doing today is 10 of the process the career development program is a 16-week program and in that cloud architect program that because you're asking the question in this 16-week program we cover exactly everything necessary to be hired we teach you how to speak like a card architect we teach you how to write like a cloud architect we teach how to interview like a cloud architect we teach you how to design like a cloud architect we have you build your own cloud like a cloud architect we have you literally do the full job of a cloud architect we have 250 hours of training in this program uh we have a hundred hours of live training and in this live training we do all architecture designs uh i'll do a free architecture design class on youtube because everybody asked me to do so but we do about 100 hours of live on demand design with our students our students by the end of this are really solid we teach soft skills emotional intelligence literally everything that's necessary to do the job as a cloud architect so this training covers 10 of it the 90 the rest of what we do is very important in the video that i released today i was very serious if you want to become a cloud architect certification is 10 only ten percent the rest of it is soft skills presentation skills writing skills training communication skills training presentation skills training learning how to deliver the different message for the ceo versus the cio versus the cto versus the cfo it involves how to ask the right questions to make a design how to present your information to the to the different audiences what does a cio care about um literally all of it it's all that important literally every piece of desk components is there like i said amazon is sending one of the directors to my class on friday because we're the only people that teach architecture training and not just configuration training so that's how different and how important it is i will leave a link to that as well as the coupon to make it a lot cheaper some of my students are actually here just because they like extra training and i'm sure they could let you know if they're having a good experience in the program i know bison did he's actually taken my training and even passed the exam but he's still here because whether we do whatever we do we try and take really really good care of our students when they come with us before they get hired after they get hired we love our students they're very important to us so i'll drop a link to the program and the coupon code and then i'm going to get to that cash gateway because someone asked me a question on that and i want to make sure we cover every technical question every technical question we can because i want everyone to get the maximum experience from this free training program so let's go back someone asked me about a volume k gateway in cash mode so let me go back to describing this in this volume gateway in cash mode what you can see is the cash mode is really about making your your s3 look and feel like traditional storage we want s3 to look and feel like traditional storage why is this so important hosts can't use object storage like regular storage because it's different kinds of storage but what if we wanted them to so the way this volume gateway works is as follows in your data center you put a virtual machine you can see the virtual machine where it's called gateway vm and it's got the little red things where it says volume storage a blood buffer and what will happen is when a user requests information you can see the users in the uh upper left hand corner from their computer the computer then goes to the storage gateway and it pulls the information from s3 to the storage gateway and then it sends it back to the user now if a second user goes to the storage gateway and requests that same information it's actually going to be cached so it doesn't have to go back to s3 to pull it back second user is going to go to the storage gateway that's going to get their information so that is the way this cash mode works um before i move on to a piece question did i answer your question there because i want to make sure that we answered your question and if you guys are all having fun if you don't mind leaving a like it really helps signal the algorithms that we're doing a good job some more people can see our free content we want to provide the best free training in the world and if you know what our free training is you can only imagine what our pay training is because it's 100 times better but we still want our free training to be better than any other paid option training options on the market because we love this cloud architect community it's so important to us i've been an architect know for 25 years architecture is so important to me it's near and dear to my heart and i want to bring in the next generation of architects and i want to make all of you wildly successful so two three four years from now thousands and thousands and thousands of you all tell me what great jobs you have actually i'd like my students tell me this you know within 12 weeks 16 weeks in the program but i'd like to hear literally from all of you a year from now two years from now about your fantastic cloud computing careers and that's why we do so much of this for free we love you guys we love the cloud architect community so a b you ask the question on what exactly is throughput so let's go back to the throughput throughput is the amount of stuff and the amount of data that you can literally move at a period of time so latency is how fast you can read an access throughput is how much you moved throughput is measured in megabits per second latency is measured in iops a throughput of a drive that can move 3500 megabits per second means it can move 3 500 per megabit per second so throughput is how fast you can move data from multiple locations by comparison iops is related to latency or how fast you can access the disk so think of latency as how fast it takes for a car to go from zero to sixty let's take two vehicles side by side the latency is how fast it goes from zero to sixty but latency is gonna be measuring is the correlated if you've got a race car and you've got a tractor trailer and there and you take them to a quarter mile finish line the race car can get there faster because it's got lower latency now let's say you had to move um a lot of data what do i mean by a lot of data let's say the throughput is the amount of stuff you could put in the vehicles how much could you put in the race car not a lot how much could you put in the big giant truck a whole lot so throughput is related to the max amount of information that you can move in a period of time where latency or iops is how fast you can access the drive i'm going to go back to training but before i go back to the training i want to make sure you understood the concept between throughput and latency could you let me know if you did okay takes about 30 seconds from the time i ask a question for you tommy it takes for you guys to hear it so um we'll go through that and at this point there's no more questions we're going to start talking about computing okay let's start talking about competing on the aws platform so [Music] when you're working in a cloud the cloud is nothing more i mean literally nothing more than a virtualized network and a virtualized data center that's it virtual network virtual data center that's all your cloud is and when you understand that you're going to be in a much much much better position to uh design your systems so that's realistically speaking let's look at what we're talking about when you're dealing with uh when you're dealing with competing in the cloud you're really going to be dealing with virtual machines or containers that's your only options you're not going to be really dealing with bare metal servers there are a few ways that you can get them but typically speaking in the cloud environment they already have hypervisors on them and because of that you're going to be dealing with virtual machines this is important everything you do in architecture is based upon your performance requirement so if you have a virtual machine in the data center that has 64 cores and a terabyte of ram and it's running at peak efficiency and peak memory perfect memory memorialization and everything's right it's optimal capacity what do you think you need in the cloud if you need 64 cars and two terabytes a day dave sir your vm in the cloud needs to be 64 quarters and two terabytes of dram that's it so you're going to size your virtual machine in terms of virtual cpus the same way you would do it with any other virtual machine anywhere in the world any time map your cpu occurs from your current i go to the data center and you're good now we'll talk about auto scaling ways to make this more elegant but as a rule if it's optimally sized in the data center lift and shift it identically to the cloud in the same way shape or form it is and you're going to be in great shape so still just think of your current systems cpu memory storage network performance whatever you're doing in the data center is exactly what you're going to need on the cloud no different it's all going to be the same kind of concept so let's work on that so let's look at the type of compute instances now this is not an exclusive list but aws has a bunch of compute instances that they use for a variety of reasons they've got some that are designed to be their arm based for workloads they've got some that are compute optimized they've got some that are designed for gp workloads these are things for machine learning and things that you might be using or rendering they've got some some i3s that are designed for high-speed storage for like warehousing databases they've got some general purpose instances do you need to remember all these the answer is no good to brush us among them for the exam if you're going to be an architect and we're training architects you're going to check every single time the chart in aws they try to gcp the chartered house you're going to do it every time to determine the optimal compute platform why because these things change constantly and there's no way for you to make mistakes you're always going to verify with the right one pick what a virtual machine based on your cpgs dram gpu requirements and you're going to be in good shape so let's talk about virtually these machines on aws they're called ec2 or elastic compute instances when somebody asks you on a job interview what is an ec2 instance your answer needs to be it is a virtual machine which is used by the which is the primary computing platform used in the aws cloud if you say a compute instance that's it the interview's over if you say compute instance it means that you don't know what it is and you've heard the term compute instance and it's not going to impress the hiring manager you need to tell the hiring manager this is what it is it's a virtual machine we're hiring managers to ask these open-ended questions we want to know if you passed an exam or if we understand the concept it's a virtual machine so ec2 instances predominantly are for linux and for windows now some software developers need to compile code on a mac and because of that aws came up with these concepts of mac os virtual machines what they really are is just a mac mini um hosted on the aws cloud and when you're dealing with virtual machines or ec2 instances whatever you want to call them you can either create your own virtual machine or you can use pre-built marginal machines these pre-built master machines come in the form of an amazon machine image and basically what happens you basically take the image you need a storage environment instant and basically in an instance type and then you pop it and you launch it works great so what is an ami or an amazon machine image it's just a file that basically includes everything necessary to boot a virtual machine so for those of us that have worked with vmdk files and vmware and ami is basically the same thing it's just basically a file that you can turn into a virtual machine in launcher instance fantastic environment now like i said you can make them you can use previous ones or you can buy them you could make them by starting an instance with a based image instance and then customize it you can use a pre-built one from aws or with everything is optimized or what you can do is you can buy one so when we talk about getting into security we're going to spend a lot of time on security when we're talking about firewalls for example if you need something good meaning something stronger than laugh because you're an enterprise you need something sophisticated you're going to basically have to do this you're gonna have to buy a virtual machine and in the virtual machine you're basically gonna have to buy a firewall from the marketplace but you can you can't put a physical firewall in the cloud it's not like you can walk in the door and say i need your access to the rack and bring your screwdrivers in and screw your firewall into the rack and then plug it in you know it's aws data that's just not going to happen so when you need these kind of things you go to what's called the marketplace and you buy a pre-built virtual machine that's got all your firewall or idea systems all installed that's it you can buy your mis like from the marketplace you can buy them with special software or software licenses you can buy security appliances this is the way it works so machinima is just a pre-built file that basically is like a vmdk disk pro what's part of an ami all the components it's the operating system the launch permissions and what you call block device mapping that attaches specifies the storage volumes you can copy an ami from one region to another region which is a great environment you can make an image or a snapshot of recovery great way to do disaster recovery look at it this way take your server make a machine image of it store it somewhere else something happens to our server launch the new to launch a new server from the scene image and guess what everything is going to be identical to your old system with one exception files will be the same access will be the same 100 of everything is going to be same but it's going to come up with a new ip address so it's going to have a new dns name so you may have to remap those things but otherwise it will be an identical copy of the machine with the exception of a new ip address because it's going to be a new instance now when you're dealing with these kind of things these ami's you could do it a couple ways you could basically launch your ami just like you installed a nuke operating system you launch your new ami basically you've got your ami it comes out and then you basically you know do a pseudo-scout update for like an ubuntu system or a deviant system or you could do a yum update if it's a red hat or a send task on a system or what have you you can do that and you can get your systems up and running but what if you wanted to do it in a more automated way what if you wanted the systems to patch themselves as soon as they're built maybe launch an apache web server you can do that of course you can what you do is you create a bootscrap script and this is a simple simple simple shell script even architects like me that don't do any coding in our lives i can do this in about a minute and a half basically it's basically a list of the commands that you want to do on linux paste it in the space and poof your systems come up they patch themselves they do absolutely wonderful so this is a really great way to basically set up your systems bootscraps guests are a way you can do it now like anything else you've got lots of ways we can buy this stuff purchase the right program the right way to purchase this and your move to the cloud is going to be successful low cost and wonderful do it the wrong way and it will be an absolute disaster so this is really critical stuff for us so when you buy your instance or you rent your instance pick the right choice and the way you can do this is basically you can buy something that's on demand which is buy it as you need it you pay by the second this is perfect for auto scaling we're going to talk a lot more of it then what you can do is you can buy something called a reserved instance and if you know that you need a server of this capacity for three years long or one year long this is awesome because you're going to get a very substantial price savings because you're committing absurd things to amazon and by committing to them they can make better investments so basically you're a lower risk customer so they can charge you a lot less and then the last option is a scheduled reserve instance which is if you know you're going to need it for a certain point of the day so let's talk about this again if you have unknown capacity and you have no idea what your needs are you're going to use auto scaling and you're going to use on demand because it works perfectly perfectly perfectly if by comparison you know your demands and you know that you're going to need your systems for a long period of time purchase a discounted plan what is the discounted plan the discounted plan is basically reserving some capacity for one year to three years if you're guaranteed to need it and you give amazon a guaranteed purchase your price will go down this is wonderful so there's your instance purchasing options now there's a couple more now there's the scheduled reserve which i mentioned before but that's like if you need it every saturday and sunday to do a batch job but not the rest of the week that's what you could do now there's two more systems that we can talk about the first thing that we can do is something called the spot instance what is a spot instance the spot instances aws has a tremendous amount of unused capacity at all times and if you want to do need some compute capacity on the cheap really inexpensively you can buy what's you can temporarily bid in an auction-like manner to aws and you can get the cheapest thing that they have available that's just called the spot instance and a spot intensive is basically a bid just like an ebay environment or auction-like environment on unused capacity and if your bid is accepted you get sold and use capacity now this is really great because you get really really cheap access to high performance computing wonderful here's a problem if the price of the service goes above the price that you bid you're cut off literally they'll shut your systems down in a moment's notice so if you've got anything critical it's not going out of spot instance could you imagine running a hospital and having the or an investment bank having it shut down so nobody would do this what is this used for if you've got a batch job that can tolerate starts and stops and you need the cheapest computing platform known to man it's great so what does it look like when you're truly optimizing your cost combination of reserve instances for what you know you need to reserve auto scaling which means on demand which has capacity as needed so you don't need to over purchase spot instances for bat jobs the combination of all three just understand in google they call them preemptable instances at amazon they call them spot instances when you use one of these things they can shut you down at a moment's notice so don't put anything critical on one of these things just not a good idea so while we're talking about this let's talk about what's called a dedicated host if you desire you can purchase a dedicated host from aws which basically means you're buying a server and it's dedicated to you which means on your system you'll have system level information you'll be able to see the cpu cores the memory utilization you'll be able to manage the hypervisor your own virtual machines if you need a host and you've got say something that needs a license that needs to be attached to a specific host this is your option as a dedicated host again not cheap because you're purchasing the entire system but you know very good in security gives you lots of control and that's what you call a dedicated host now when you get these things your servers and your virtual machines are going to be placed and spread throughout your environment so there's the tenancy options which is the concept of a shared tendency a dedicated instance and a dedicated host so let's talk about what these things are shared tenancy by default is you come up with 30 virtual machines probably spread across 30 servers at aws and on these servers it's not just your data it's a lot of people's data so that's what we mean by means share tennessee your virtual machines are on same servers as other people's virtual machines but you're logically separated now let's say you purchase a dedicated instance think of a dedicated instance as um effectively your own vmware esxi server where it's not because you're using the nitro hypervisor but it's just like that it's like a server that you have where you're still where basically you can create you can create any of the virtual machines you need on your server that's a dedicated instance versus dedicated host think of it as a bare metal server and if you need actual access to the server that's realistically what we're doing so let's talk about securing ec2 access and we're going to talk about how to do that this is a pretty uh relatively easy easy thing to do there's lots of ways that you can actually do this so let's talk about securing ec2 access so if you've got these virtual machines you've got to protect them right so a real security posture is going to include firewalls ids systems ips systems ddos protection access lists on the routers to protect your subjects and host-based firewalls and as such what is a security group a security group is effectively a host-based firewall when you have your ec2 instances or virtual machines you want to protect them against bad data so what do you do use what's called the security group the security group is basically a stateful host based firewall that's placed on the instance and it only allows the traffic in that you like so by default it's good to deny all traffic just like all firewalls so you're gonna have to allow anyone exactly what you want now you only allow in what you want you all need to deny because it's implicit so you're going to specify the source and destination address if appropriate the protocol tcp udp icmp and it's going to look like any access control list or firewall rules so for example i can allow port 80 and port 443 coming in from any source or i can allow the host 10.1.1.1 24 into the subnet this is what you're going to do and this is a fantastic way to visually do this and we're going to walk i'll tell you what i mean by stateful again because that's a very common interview question and most people get it wrong getting this question wrong is something that can tank in interviewing really fast if you're looking for a cloud architect position i want to put a little bit of cloud architect training while we're doing certification training because they're so different so you can see in this diagram we've got our compute instances and we have this concept of a security group and because we've got this instance in the concept of a security group it's going to keep bad stuff from coming in now the security group is stateful what do i mean by stateful this is really really important stateful means it tracks the connection so in a real firewall let's say i have a firewall in my house and i do i've got a very extensive expensive firewall sitting in my home i spent a lifetime at cisco so they sent me a lot of their gear to test and evaluate some stuff so i've got this firewall when i want to go to the internet i go through my firewall to the internet and the firewall says mike gibbs from his computer which is 192 168.2.29 sourced an address going to www.cisco.com allow mike gives his return traffic because i saw my gibbs sent the request i saw my gibbs went to this web server and sent a sin so when the ack and the sin comes back from the web server which is part of the tcp ip protocol setup package i know full well it's allowed why do i know what's allowed because i watch mike's traffic out and i allow it back so stateful means that it pays attention to the connection and it sees where the connection came from because stateful knows and because stateful has stayed you only have to apply these things in one direction so stateful does not mean you only apply in one direction do not say that in an interview i've literally lost about 800 cloud architect potential cloud architects just for not knowing stateful because if you don't understand faithful you don't understand firewalls and if you don't understand file rules you can't understand securities so if someone ever asks you in a cloud architect interview what does stateful mean it means something is tracking the connection something understands because something goes through and the file maintains knowledge of it and allows the return traffic that's why it only needs to be applied in one direction it doesn't mean that it is applied in one direction it means that state the state is known that's why it works with only applied in one direction so let's talk about some more tuning of ec2 instances how about putting an address on an instance no ip address no communication on the network so this is really really important stuff an ec2 instance can have multiple network interfaces just like any server can have multiple network cards of course on any server you've got multiple network cards guess what you need multiple ip addresses each network card has to be on a different subnet so you're going to need an ip address and a difference on that now when these ec2 instances come up they're assigned a name by what's called ec2 dns which is dns for ec2 instances only and if you need a system to be reachable from the internet you can put a public ip address on it but you know but otherwise you're going to be using a private ip address because it's not going to be original from the internet so it's going to be much more secure now when instances come up they're automatically assigned an ipv6 address of course you could disable that if you want and every interface meaning every network card needs to be on a different subnet and it needs to be a unique address space because if you have two instances in the same address they can't talk to each other just remember that how can you access and manage your systems well the following ways you can do it from the ec2 console from the management console you can ssh or secure shell onto your machines if you're using windows you can use this remote desktop protocol for windows systems not exactly the most secure way to access the system with remote desktop particle but that's the best you can do with windows and you can do some management be the software development kit think api so what did we talk about in this section we talked about ec2 we talked about aw we talked about amis we talked about purchasing options for your systems we talked about securing your instances and we talked about ec2 addressing and then accessing and managing your ec2 instances we've covered a lot so far so let's take some questions uh some questions okay so bellwinder occur how is shared tenancy instance different than any regular ac2 instance all regular ec2 instances are shared tendency it's only when you want more control or something better that you're not going to be using shared tenancy so share tenancy and that's a great question is the default that actually comes for free when you're actually dealing with shared tenancy who has another ques so oh sir khan asked a great question and latency is different than network latency sister can there are a lot of kinds of latency that we're talking about when we're dealing with a computing environment it could be latency for the hard drives latency for the network latency for the amount of time it takes to get per packet to be processed by the cpu of a router last frequently that would be contributing to network latency even though it's not linked lanes there's lots of latency but we were referring to iops we were referring to latency completely related to hard drive access speed who else has a question stagio i have no idea what you mean by sold or sold latency there's latency related to uh to how fast you can access your hard drive there's latency related to network access speeds i've never heard the term soul soul latency so i don't know what you're referring to anyone else have a question how do we manage vms and aws um well sir can it's a great question in vcenter um it's very simple and very elegant um in aws basically speaking the way you manage them is your ssh into them um you can convert it into an ami or you can use the management console or you can use the api but that's pretty much it there's no simple elegant way to manage your systems on the cloud as well as you would with vmware esxi unless you're getting into terraform and there's infrastructure as code stuff that comes up and comes down so but realistically speaking it's more of an ssh kind of environment that you're you're dealing with oh i said i'm sorry somebody from my team uh translated your question on feijoa so when it comes to different environments as it stands normally um what you're talking about is your unshared tendency meaning your stuff and everybody else's stuff is all on the same system that's why it's shared tenancy now you can purchase a dedicated instance which is basically a dedicated server running the aws hypervisors which means you can create all of your ec2 instances on your own server and it won't have other people's servers on it so a dedicated instance is basically when you buy a server from aws that enables you to put all your vms on there but it's a server with a hypervisor installed and it's designed specifically for ec2 instances meaning vms by comparison a dedicated host is basically just the server blank so if you're given that server you have complete and total access to the server the cpu sockets the memory everything so basically look at it this way if in your data center you had a real server where you just installed windows or linux and then you installed your applications that's it excuse me a dedicated host in your data center you have a vmware esxi server for example and you put all your virtual machines on that that's a dedicated instance where a shared tendency is you just create a compute platform um anywhere in the environment and that's realistically speaking what you need to do so i'm going to give you guys some architecture training and some free architecture trips right now when you speak to executives don't use abbreviations when you write do not use abbreviations there are a million and one abbreviations that all mean the same thing so i don't know what nfv is um i could give you some financial terms that equal to nfv i could give you some medical terms that mean nfv but nf and and no i don't know what you but i don't know what nfa is so if you actually type it out do this and whenever you write an architecture document or you present to executives today we're going to talk about the elast compute cloud which is a virtual virtual machine which aws calls ec2 meaning elastic compute always define a term before you talk about it um critical critical critical reskill required for a cloud architect is that level of communication so i'm giving you that guidance network for okay network function virtualization okay is that that is another type of nfv but is that what you actually meant because there's a whole lot of nfv's that are there so abby i need to know your definition of nfv although it could be network function virtualization but it could be a whole lot of other things literally there's about 150 terms that used on nfv so i want to make sure we get the right one sir khan does while we wait for you to answer that does aws provide ssl or ipx vp service vpn service and you buy an ecm okay wait a second so these are all different things so does aws provide ssl or ipxvp of service when you buy ec2 instances of course not um vpn services are related to your access of the network when you when you connect your data center to the network you need connectivity it could be a direct connection or it could be a vpn and you're going to have to order that it's a completely separate service network access is basically the plumbing to get you from point a to point b and an ec2 um instance is a virtual machine so they're completely unrelated technology i will refer you back and i'm happy to talk about vpns versus direct connection yesterday we spent about three hours on that and that video is completely free we'd love you to see it um but uh that's what we're referring to now i covered the dedicated hosts for for the dedicated instance um we covered uh okay okay so cancer the second half of your question is related to um so abby you're gonna still have to define your version of network of nfa tell me what you need because we can't answer your question until you tell us exactly what your abbreviation is there's at least 150 nfes out there so just like you wouldn't write a doc a book without defining your term and you wouldn't present to an engineer to the ceo without defining a term i'm training you to be an architect and since i'm training you to be an architect um i'm going to ask you to tell me what you mean by it because there's lots of nfes so thank you so much ahmed um um you create private vlans for your internal uh and they okay so not in the cloud in the cloud surveying you basically have no control over the network whatsoever other than a little bit of their virtualized network see aws does all network virtualization and you're not going to have the control you don't have access to the switches now here's where you would use private vlans in a high security environment and i designed this constantly let's say you have your data center and you've got three subnets of people that are going to be accessing the clouds we would typically put them in so let's say you've got your data center and you've got servers yes users and three vlans i would typically put the users accessing the cloud in a private vlan and then i would also use something called 802.1x authentication and what i would do is i would make sure when the users plug in they're authenticated by the system using 802.1x because if you can get onto the network you can cause damage so because you can get on to the network because you can get onto the network with damage so let me start me explain this so typically you limit the subnets that can access the cloud and then you limit who can even get on the subnets with something called 802.1x authentication and say it'll recognize mike gibbs mac address mike gibbs mac address so because it's going to recognize mike gibbs mac address it knows to allow me on now the reason i would use private vlans in the data center is as follows if my pc were to get worm infected i don't want effect it to in fact to infect somebody else's system so by using private vlans on our data center we can limit the damage of who can damage each other now that way when we connect to the cloud that way when they connect to the cloud then uh we can make sure that if one worm gets infected we don't create a ddos attack on ourselves by having a hundred users in a private vlan coming onto the cloud but sir ken you have no control over anything in the cloud you're not going to be able to look at the network you're not going to be able to optimize the networking all network functions are virtualized in the cloud at least what you see but you're not going to have pretty any kind of pretty visualizations of things like this it's just it's not like a data center there's no net cool there's nothing good for you to do these kind of things it's not part of it thank you for defining the term now i knew what you meant visualization versus versus virtualization versus one of the other 148 terms that use that term so please remember that when you communicate with executives as part of your job as a cloud architect in the future or anytime you do any writing long before you use an abbreviation you have to network up see we've even had people on the call actually misinterpret that because it's so commonly used that's why it's essential always to define it long before you do it that's why as architects we make zero assumptions when someone gives you an abbreviation if it's not clear we ask we must ask if it's clear because otherwise we're going to make a design based on bad decisions we must get that information we make no assumptions that's why your soft skills and communication skills are so essential that's why aws is on my call that's why they're going to be on there on friday that's why they asked me and they want so many of my students why because my students know how to speak they know how to write that and ask the right questions it's more important than your technical competency it's that critical to know how to speak and communicate that's why people have a hard time getting a cloud architect job look cloud architect jobs are very easy to get i get a job offer every day of the week and it's because i can communicate well more than my technical skills it matters more so you have no control over sdn solutions in the cloud um ahmed we're thrilled to have you here it's very i i i don't know where you're at with your name it could be in the middle of the night for you so i'm just grateful that you participated and in any case thank you software defined networking is uh something very different abby i think you can assume that aws is using uh an igp of either ospf or or isis intermediate systems to intermediate systems or versus open shortest path first then you can assume that they're running some rsvp signaling then you can assume they have a tremendous number of mpls based tunnels you can assume they're running ibgp ebgp and multi-protocol bgp and a whole lot more software defined not working not so much okay so we covered this we're going to start talking about databases today because we've got about 40 minutes left and then tomorrow after we do that we are going to have a much bigger much deeper conversation on some more databases my team has informed me that i have not posted the link to our to our cloud architect career development program in a long period of time so i was asked and informed that i should paste it so i'm going to do [Music] so okay getting back to that now that i've made the rest of my team happy i'm doing the things that we're supposed to do let's talk about databases so first and foremost the way i told you that you always define a term before we get something and that way we don't make mistakes we're going to talk about what is a database so a database is basically a software package that you use that enables you to store a tremendously large amount of information it's going to enable you to sort it calculate it report it and share information databases and say this again are considered to be one of the most critical business applications at this point and why we use databases and the types of databases we choose are critically important for a cloud architect you must know why the databases are being used how they're used and their use cases so so so important so we're going to talk about that from an architectural perspective so when you're dealing with aws we're going to talk realistically speaking about three kinds of databases and then we're talking about why you're going to use them so of the three databases that you have access to you've got relational databases no sql databases and data warehousing databases let's talk about relational databases first we're going to get into depth or at least some moderate depth because i'm not a database architect i'm a cloud architect and a network architect but we're going to talk about databases first and foremost let's talk about when you would use them we're going to begin with the relational database which is the most common format database in their use case and then we're going to go back and we're going to start talking more relational databases are used for the following purpose to help an organization determine the relationships between variables that's it relational databases are used to help an organization develop to find the relationships between variables what do i mean by that what does price do to sales that is what relational databases sold if i run this cell did it improve sales if we down this ad did it improve ads is the customer's phone number or geographic things related to how much they purchase relational databases show the relationship of variables how one thing affects another why do organizations care it helps them make better business decisions that's why organizations use relational databases the next type of database that organizations use is something called a nosql database nosql means not only structured query language meaning that it has more functionality and capabilities than an sql or structured query language database what's so special about nosql they scale they scale and they scale so organizations use no sql databases when they have a lot of customization that needs to occur or they need scalability beyond anything you can believe and we're going to talk about this soon but what do i mean by that let's say you've got 50 million internet of things devices all weather sensors all over the world reporting weather information every 10 seconds not going to be capable of doing that in a relational database even though it would be cool you're going to have to use something called the nosql database now we're going to talk about the last kind of database which is used for data warehousing and what is the data warehousing database and aws is going to be redshift but there's lots of versions for this it's a place where you store incredible amounts of information why would an organization want to store incredible amounts of information quite simply they want to take this information that they've collected over a large period of time they want to mine the information and they want to make better business decisions based upon the information think data science that's what data science is it's not machine learning that's a tool data science is the ability to take large amounts of information be able to look at the information and analyze the information to make better business decisions that's why organizations use data warehousing databases so now that we talked about that let's talk a little bit more about the database relational databases are the most common form of databases and they provide storage and access to data that's related to each other in the structure of these things which we call the schema think of a spreadsheet effectively your data is stored with rows and columns and each row is going to have a unique id and each column is going to have a unique value and realistically speaking that's the way a relational database stores in a structure think spreadsheet columns and rows now when you're dealing with aws there's a lot of relational databases that you can choose from relational databases are best when your data is structured relational databases apply to what the acronym the acid model which means atomic consistent isolated and durable well mike just said a whole lot of stuff in a short period of time what did he mean by that it means when you do a transaction it's all or nothing it either gets written or it doesn't consistent means second i write to the database everybody gets to say the same things meaning it's instantly consistent we'll talk about eventually consistent later isolated meaning one transaction will not affect another transaction so it's isolated the next thing in the list is something called durable meaning the data and the database won't be lost so relational databases follow the acid model atomic meaning all or nothing consistent meaning as immediately after you write something everybody else can read it isolated and that one transaction doesn't affect each other and the data in the dashboard will not be lost so going back why are we using these relational databases but you see what we're trying to hear we're trying to show the relationships between variables customer orders names email addresses price we need to know this why we can make better business decisions this way that's why relational databases are so cool now let's talk about the relational database options that you have on the cloud you've got basically amazon aurora which i'll talk about mariadb microsoft sql server mysql oracle db and postgresql so if you're looking for a fully managed service you can use amazon aurora it's a nice fully managed relational database service it's got relatively good performance and scalability and what you're trying to do here with aurora is you're looking to get some of the commercial functionality that you would get saying in an oracle database but for a database where you're not paying for licensing so it combines the benefit of like your commercial databases like an oracle with the the low cost of something like mysql or mariadb or something like that so typically people use aurora because it's you know it's basically an enterprise relational database software as a service kind of thing it scales well it's easy it's easy to use they manage everything this is one of your options the next option you would have would be to use good old-fashioned mysql this is the most common relational database in the world it was it's been out there for a long period of time it's free and open source and this is part of that lamp stack that everybody needs to know which is why we have our students all build it the linux apache mysql php this database is used everywhere for everything it's free it's got some scalability challenges but it's an exceptionally good relational database it's free and it's used everywhere the next type of relational database that tends to scale well and people like is this postgres sql and it's another open source relational database and it's got a really good feature set so think of it as like an advanced mysql still all relational databases we're talking about right now so they're all pretty similar then there's the very common mariadb this is another open source relational database in fact it was created by the same people i created mysql so think of it as like mysql v2 um lots of additional features and functionalities and scalability over mysql made by the same people that made us the sql mysql database in the first place so love this database played with it done it lots of containers use it in lots of environments terrific database and it's free now when you're getting into paid options you're realistically looking at the microsoft options the oracle options most people that i've worked with use oracle i haven't worked with many organizations that have a lot of windows workloads other than microsoft exchange and uh what do you call it active directory and those sort of things they typically use unix linux servers to run everything in microsoft servers to do those two things calendaring now those kind of things microsoft does a beautiful job there are people that do use microsoft sql and it is a very elegant database to use and it is very simple to administer compared to other databases which is what people like and there's an amazon relational database service that supports lots of versions of it supports microsoft sql server 2008 2012 2014 and if you've got users that are working on my sql databases and they've got large numbers of windows workloads which some customers do you're really dealing in a great environment of lots of people doing a lot of things microsoft has different clustering and failover options that most databases completely different so if you're using that you know make sure you understand how those things work and there are four versions of the micro microsoft sql databases you can use not just the version no not just the year version but the express the standard the web and the enterprise so you know if you're part of one of those organizations or you're designing some something for an organization that uses microsoft that's that's your options my cloud lots of options to use now when it comes to enterprise databases you know the king and queen of this for the most part are probably oracle they're the most one of the most popular paid relational databases in the world these databases have an extraordinarily good feature set and functionality is look at it this way when a company does nothing more than make databases for a period of 20 years it's going to be pretty darn good so their database is great now they're doing cloud and other things like the rest of us are but oracle databases extensive features and functionality used by pretty much all your big global enterprises that aren't afraid to pay for things great product the aws relational database has multiple versions of oracle it's the standard the enterprise and i'm sorry the standard one the standard and enterprise and you can use any of these things and each one's going to have different performance flexibility and scalability needs now there's really speaking there's two versions of licensing supported by aws you can use the license included which basically means you buy one of the things from amazon you're buying their systems from amazon and while you're buying your systems from amazon you get this you uh includes their license you don't have to pay for the license now option two is you bring your own license and you put it on your own host and it gives you a lot of flexibility you can choose the standard the enterprise the enterprise one and a price two and it's your license you're in charge of managing so you've got a lot more flexibility you can just pop it on an ec2 instance now we've introduced you to the relational databases on the system i'm going to talk about nosql databases for a few minutes and then we'll talk about data warehousing options and then we're going to break for some questions or maybe even break for the day so let's talk we'll break for some questions and then for the day so let's talk a little bit about nosql databases what do we mean by nosql i told you this means not only structured query language these nosql databases are really great because they allow for a lot of flexibility in the schema remember i told you relational databases use columns and rows just like an excel spreadsheet a no sql database gives you a lot of schema flexibility and therefore can work with structured and non-structured data since the structure is flexible you've got a lot of customization here's the thing with structural databases with nosql databases is they are really big and they offer really really great performance um because they are they enable you to be much more flexible so that's one of those important things to remember is the way these databases actually work highly scalable flexible schema work absolutely terrifically so customization but also more scalable the way these things work is you partition the database and it's almost as if you created a new database so when you're dealing with nosql databases you've got a lot of options every cloud provider has their own choice aws has dynamodb google has their cloud big big table what everybody uses internal organizations is apache cassandra use whichever you want and you can use any of these on the cloud now on the cloud i'm going to say this i typically i'm not the biggest fan of serverless architectures why am i not the biggest fan of serverless architectures generally speaking when you use a serverless kind of environment here's what happens you have to spend an incredible amount of money refactoring your applications to bring them to the cloud and then if you bring them on the cloud and the cloud doesn't work it's too expensive running the problem with the cloud provider you got to spend millions and millions and millions of dollars to come off the call not with this dynamodb is a cloud managed no sql database it's completely managed by aws it's highly available and it's going to be in multiple availability zones by default and it's serverless now typically speaking i don't like serverless but i like dynamodb why do i like dynamodb it's compatible with apache cassandra so it's easy to get your data in and out of should it not work for you in the cloud it scales it grows on demand and because of this you have an environment where you're not constantly swapping out servers and you're not constantly dealing with all these things it makes it simple it makes it elegant and it can scale love this this dynamodb is an exceptionally good database could you use apache cassandra on an ec2 instance sure i've done it with lots of clients could you just set up a dynamodb situation sure done that with lots of clients elegant serverless solution and if you want to come on the cloud and off the cloud in a short period of time it is a very good model so i highly recommend it and i think it's terrific so realistically speaking what's cool about dynamodb is it's got low millisecond latency and uh that could still be a lot um but it's relatively low but the good news which makes this stuff even cooler is that you can actually add the dynamodb accelerator and that can really knock down your latency and can make it so much so much lower latency so i think that's pretty terrific anything that you store on dynamodb is going to be encrypted by default and can be backed up with little or no effect on database so this is really great and you can actually set it up for cross region replication so love dynamodb lots of scalability lots of functionality lots of everything you need so the way these things work is you've got these primary indices and your primary indices are basically your primary database now inside of your primary indices in your primary database what you've got going on is you create what's called the secondary index when you partition the database and you create a secondary index you can increase lots of performance so the secondary indexes can have the same partition key as the base table um but anyway you've got you when you create a secondary index you can make a local or global one now the local setting condera index is going to have the same partition key as the base table a global global index or a secondary index can literally go across all database partitions the only thing is you can't actually make a key value that exceeds 10 gigabytes but you know that's not really a big problem to increase so to increase the scalability that if it does not follow the assam allah false base model which means your data is eventually consistent this means if i write data to the database and someone goes to read it then that's how you access the information so data is written to the database and then pulled for the database eventually consistent so understand that if i write to the database it's not going to be immediately available super important to know it may take a second or two or five seconds to view available when do you use dynamodb actually before we talk about when you should use it let's talk about one more thing dynamodb can scale as needed but there's some real limitations with that in that you can set up auto scaling of your systems and when you take you set up your auto scaling of your systems they go up and they go down that's what they're designed to do they go up and they go down not so with dynamodb dynamodb you need to know your capacity ahead of time you need to make some smart intelligent decisions dynamodb is for example if you enable autoscaling it'll scale up but it'll never scale back down so let's say you you have a retail store that's using a nosql database and then it's on the highest sales day of the year and they allow auto scaling they're then going to be paying for the highest price for the rest of the year because auto scaling won't scale down so with dynamodb you need to provision your read and write capacity before you use it to make sure you have capacity and if you have capacity this is wonderful this is the way it's supposed to be so scale up but it won't scale down so pre-provision it now the photo scaling scales it up you can manually take it down but you need to understand that autoscaling is not working so when you're dealing with dynamodb if you're going to use that pricing schedule you make sure you provision your read and write capacity ahead of time if you do not you're going to pay too much or you won't have the capacity that you need because it scales up but it won't scale back down so when are you going to use this data warehouse dynamic db again when near unlimited scalability is required when low latency is required when you store lots of things from internet of things devices game state for example you saving gamestay player leaderboards like for example for netflix to kind of store you know where you're at in the video would be a great use case for dynamodb financial institutions that are doing thousands and thousands and thousands of transactions all over the place shopping carts e-commerce you know these kind of things big big big data applications big database applications that's where you're going to use this kind of stuff so let's talk a little bit about data warehousing databases data warehousing databases are where you put an incredible amount of stuff so if you've got a if you want to you store data from a wide variety of tools you stick it in a data warehouse with aws it's going to be ratchet but we're just going to talk about these things and by using these things you put lots of data in your thing then you can analyze your data with business analytics tool um business analytics tools can help you with these things so they're designed to handle large large large amounts of historical business data typically it's going to include the co the following components your data warehouse it's going to be a database to store your tools a tool to visualize your data and a tool to prep and load the data like an etl tool so big data big storage environments lots of work to get it in lots of work to process it so let's talk about the data warehousing databases you've got your buckets your s3 buckets which go to elastic map reduce typically speaking and then gets put into redshift for example and then you would then visualize it with quicksite so this is typically what you do you basically have your systems you you run what's called an etl or extraction translating translation loading until you stick it into your data warehouse and then you run a tool like tabula or quicksite or one of these kind of things to go look at your data or power bi or something like that so let's talk so you can see what it actually looks like in this picture you're getting your data from s3 you're copying it with an etl tool you're migrating it to redshift and you're going to visualize with a visualization tool traditional data warehouse that you see in all kinds of environments this is the way your data lakes work this is the way the environment works this is the way things go data warehousing so when you're with aws their version is the red shift and that's really advantage where it matters data warehouse solution what's cool about this is you put all your things in there and then you can take your data and you get some actionable insights why are you doing these actionable insights you're doing it for the following reasons you want to make better business decisions so you're going to look at your data and so redshift is what you're going to do and the way redshift works is you're going to build these clusters of computing nodes and you're going to have one node that's going to be the later node and then you're going to have all these followers which are going to be called compute nodes they all support the later node does that sound like anything familiar when we talk about databases and we talk about the master database and the read replicas kind of the same kind of concept but for a different kind of database you're always directing the main database as the leader node so that's kind of it's uh analogous but not exactly the same but that's realistically the way these things kind of work so when you're dealing with aws and data warehousing you've got a couple options you can buy dense compute node and then storage nodes and they're different dense compute nodes have very high speed ssd rate arrays and dense storage is for if you've got a lot of stuff to start and that's going to use magnetic right array so it's going to be a higher latency but much lower cost the last concept that we want to talk about today when it pertains to this is the concept of a data lake what is a data lake a data lake is a repository that enables you to store structured uncensored data literally everything you would possibly want at the same place in any skill the data lake is where you're going to place your content so think s3 think relational databases take no sql databases think your data warehouse all together sharing information between them so you can make better business decisions the whole point of all this is to create a data lake you don't even have to structure your data until you're ready to use it but it's a place to cut your data for big data applications to use for the future so let's go and talk about the benefits of it you have all your data it's adaptable and changeable it can hold the data in live format until it's needed and data links can be queried so you can search for your information what does it actually look like i'm going to show you real quick you've got a data source a data source and a data source and you pop it into a data lake it gets transformed in some way shape or form through an etl tool with dynamodb then to redshift into rdis by combining your data from multiple sources with databases that all have different functionalities you can make really really really wonderful use of your time so i find this to be a fantastic way to look at things with a data lake future of data big data what are your future cloud computing careers the way i look at it cloud in architect for the infrastructure and the knowledge of putting it together cloud security because security is a gaping hole in the cloud i'm mostly due to not people that are configuring things and they don't understand what they're configuring but it's still a problem and data science and being able to use this and data likes are part of this data science where you're going to get buying your data from your data lake so think of it this way now the last thing i really want to cover for today is when you're using databases except for serverless like dynamodb and amazon aurora um what you're going to be dealing with is you're going to have to put them somewhere and where are you going to put them again your databases are going to be on the ec2 instance so where are you going to store information on ac tools in an ebs file so what kind of ebs files do you have you've got the provision iops which is the best access you're going to get you're going to be a general purpose ssd which we talked about in depth and your magnetic storage which is your standard storage so these are the concepts we talked today i'm going to open it up for about 15 minutes of questions because we've completed the content that i think we should cover for today i see some questions that are actually here and i'm going to answer these questions i'm going to ask all of you sister perez thank you so much for your commentary i really do love reaching out to the community so thankfully happy to be there next question i think i saw maybe another question in here please ask any questions you may have known somebody asked me to explain stateful again and i am very happy stateful beings understanding or watching our knowledge of the connection so let's take it out of tech if i told my wife that i was going to be outside i was going to leave my house and i was going to come back in one minute don't lock the door if she tracked me out the house and then she had a camera seeing me come back in the door so she didn't lock it and i came back in that would be stateful how i because she watched me walk to my car and she'd watch me walk back to my house and that would be allowed that's exactly what firewalls actually do exactly exactly exactly what firewalls do you send your data into the firewall through the firewall it says mike's data went through the firewall mike's data is allowed back so realistically speaking it's no different than if you left your house and then your house remembered that you walked out the door and only allowed you back in where when a stranger would come into your house whose house would say no way you're not allowed in i don't know you you're not on the permitted list and you didn't just leave the house therefore you're not allowed back in i hope i explained the term stateful for you in a way that you can understand sir kind you have no experience with database types that's fine um and services when you buy db services from aws um how they how do they manage or change the storage area where your data is survived they don't um you're basically getting an evs vitamin you're storing your data on your own ebs volume in the database think of it this way sharkcan if you were to just do it on your own where you were to set up an instance and and uh so you're going to set up your instance and you were going to set up your instance and it was going to that was the word and you were going to pop it on the system and then you would store it in an ebs line that's really all these things are except for serverless and where it is on serverless aws stores it on their own rate arrays and gives it to you no different than any other ec2 instance so don't worry about it sir can it's just basically the way you'd set up any compute instance on the cloud astute thank you so much i'm really appreciative of this andrew yes we are absolutely going to talk about database optimizations read replicas caching queuing high availability we're going to do that tomorrow we're going to spend a lot of time on that we will spend a tremendous amount of time talking about high performance high availability and all these wonderful things that's just something that's uh super important to us and we will cover that tomorrow so we definitely definitely definitely will definitely do that and we'll cover that tomorrow super excited to talk about it the average uh human brain can only remember four to six things at a time so i try not to go beyond that but we will definitely definitely have it tomorrow what other questions do you have any other questions anyone else have any questions got upwards of 10 minutes left scheduled for the day so if there's any questions i'm happy to answer them again um when would you use serverless or rdbs quite frankly i don't use serverless or db for anything um i do use serverless dynamodb but i prefer to use something where i can move my data in and out fairly quickly people use serverless and people like serverless and the reasons people like serverless is uh is because it scales automatically and they don't have to think about it people therefore would use aurora a lot what i like about aurora is it gives you a lot of functionality that you might get an oracle type database not all of it but a lot of it you don't have to pay for the license i also like people like that serverless if you're one of those people that wants a serverless architecture aurora is a great place if you're not one of those people that likes a serverless architecture then aurora is not the best place sir ken my training covers everything that you would ever need to get hired as a cloud architect and then 20 or 30 more um we focus on what's necessary to get hired we remove all the fluff all the nonsense and everything else because otherwise it would take us about four to ten years in our 16 week program we have to work really hard my team has asked me to basically uh to ask you where you all found out about us so i place the form if you can we want to make sure we know how to find you so we can we can produce content and get it shared our charitable mission of free content and free training like this we do is very important to us i grew up with an extraordinarily limited means i worked three jobs just to get through college and i worked 80 hours a week or more when i went to school and i know what it was like when i had to buy my first mcse boot camp which was five thousand dollars back in 1999 and i remember my 10 day 11 000 ccnp boot camp which and how much that negatively impacted us so we try and do what i call a freemium model and that we try and provide an incredible amount of free training to the community that's basic training and then what we give our students is something that we would normally charge uh ten to twenty thousand dollars where we just do it for basic for very little the reason we do it is i know what it's like to grow up with nothing and i know what a good knowledge can do and we're really happy to do that so i'm going to drop the link to my program one more time if anybody is interested in discussing the program with me i'm going to leave access to my numbers and i will totally do that um for the q for this here's a coupon code for 20 off if any of you are interested and if any of you guys want to text me or call me and ask questions about our training here's my phone number i want to make sure that all you guys get hired and get your goals as fast as possible you found me on youtube thank you so much super brigadier a lot of people do but we're also on linkedin we've been exploring reddit we're exploring twitter and we want to make sure we know where i have my team focus their time because we want to help as many people as maximum yes they do manage dbs could be better in controlling costs but they might not be so it's better to truly truly truly know um bellwinder thank you for being here um chris thank you for making sure that uh you're you're you've helped us with these things also make sure you've subscribed and hit the bell and if you've liked our video and you don't mind giving a like it really helped us as a business spread our message of free training to the world and anybody that you want to share this content with we'd be more than grateful please share please help please forward please repost anything that you can do to share free content and free training to the world really helps us malek uh cassie good day you have a few picking up uh questions picking up for securities especially which approach should you take uh i would go to the ec council for training so uh when it comes to training i will probably do a free aws advanced security i will probably also do a complete total free aws advanced networking course on youtube live for people when it comes to getting your first cloud security architect job here's what you need to know you need to get certified solution architect professional and you need to know security two best certifications for security and the three are the ceh the cissp and the offensive security professional they're the only security certifications we ever recommend we do teach aws security courses the reason we do it is you know how to do these things on the cloud because the cloud has a lot of weaknesses and there's a tremendous number of work around so these advanced security things are good if you're a security architect like me and they teach you how to work around the limitations of the cloud but definitely if you're trying to get your first tech job or your first cloud job get the ceh or the cissp there's a lot of challenges on how to do that we've got a process to get our students through that but they're the ones that matter stay away from the security plus it's not going to help you stay away from the aws advanced security as a primary as a backup to the ceh or cisp malek that's great don't do it as a primary one it's just not going to help you you want to get your guys hired and paid as much as possible and that's why we're trying to give you this kind of training any more questions for me anywhere thank you talia that's my whole goal my whole goal is to take all the mysticism out of cloud computing all the complexity out of cloud computing and make it to something that every one of you can do i'm alec i will do some more network training and security training i'm going to do them via live streams um if the results are good actually if you guys want some security training or some network training please comment in the comment section before the call ends please comment security training or network training or comment security training and then comment network training and if you and if you're having a good time please comment cloud architect and i want to know that all of you guys are having a wonderful experience i have some members from my team that are going to be looking through the comments and when they look through the comments and the questions they're going to basically tell me what kind of videos i need to serve you better so please comment so we know so my data my data science team can actually mine this data and help us produce better content for you we've got two data scientists on our team and we're really thrilled to have them it's okay and you're welcome really we love doing this okay if i don't see any more questions uh feijoa you're welcome i'm gonna close this and if you guys want me to do any specific training please put it in the comments section below i'll keep the window open for the next two minutes thank you sister event we can we'll if we get enough requests looks like we've got enough requests for network training but we're always happy to security training i also would say if any of you are from a developing nation where places where it's hard for people to afford this kind of training please share this kind of these videos with them i want to make sure we help as many people as possible so taiya after my course if you're referring to my chronicle career development program we give people the certification training throughout the program to be certified it becomes very easy to actually get certified when you actually understand the underlying technology it's very challenging when you don't know what it is but like you know if i just had to learn s3 and and and policies and things like that it would be monstrous for me but it's very easy for me because i built this stuff and i know the data thing center things so tachy we make sure we get everybody trained through the process of going through our program and we'll make sure you do great jobs a great job thank you sweetheart so happy we're here thank you super brigadier thank you uh bellwonder we'll get you some secure we'll probably we'll do some scenery thank you so much alonzo um really appreciate everything uh you've said alonzo along the way and everything you do drop a link to my program one more time keep all the rest of my team happy and coupon code and if any of you guys want to call me and have any questions about training i've like a crazy person posted my phone number on a public youtube video that should tell you exactly how much we care about this community thank you very much thank you thank you i've got your information my team is looking at everyone's comments so they know how to for us to best produce content for you take care everyone i will see you all tomorrow

Channel: Go Cloud Architects

Views: 5,104

Keywords: aws sap, AWS certified solutions architect professional, aws certified solutions architect professional exam preparation, aws certification free, sap-c01, free aws certification training online, aws cloud computing full course, cloud computing complete course, aws solution architect certification, free aws course, cloud architect career skills, aws training free, aws professional solution architect, aws solutions architect professional exam, aws solutions architect professional

Id: Sy5WAvhMegk

Length: 180min 0sec (10800 seconds)

Published: Thu Jun 17 2021