AWS Certified Solutions Architect Associate 2021 (Full Free AWS course!)

hello and welcome to the go cloud architects aws certified solution architect associate course hi i'm michael gibbs and i've been working in technology for 25 years and during those 25 years i've had the opportunity to work for cisco riverstone networks which is now nokia comcast and mci worldcom which is verizon and i've been considered to be a subject matter expert in global network design cloud computing and i.t security and i've been working with others in a teaching coaching manner for over 20 years while working in technology i am the founder of go cloud architects and we created this course with one goal in mind we want to provide the best quality online education available anywhere in the world and we want to give our students real life skills we want to make a big impact in your lives and we want to give our students every advantage possible so they'll be successful in their world we have an approach we're going to have no fluff whatsoever we're going to take all the fluff and remove it because we want you to know exactly what you need for the exams but also real life we're going to make everything simple our motto is we make cloud computing simple and for us we want to know that you have a deep understanding and if you have to have a deep understanding we've got to make it to something that's completely understandable so we do everything we can to take the most complicated technologies and simplify them and lastly we're going to start at the beginning we have no knowledge of your cloud computing background so we want to give you all the foundational information that you need to make certain that you'll be effective in your current role so let's begin so to begin what is cloud computing and how did we get here so let's begin with the history of the enterprise network and data center when i began working in it organizations were really just beginning to figure out that technology could be a competitive advantage and at that point what would happen is there'd be somebody's computer which would actually function as a server for for an office and then there'd be a couple of connected computers and organizations said wow this enhanced productivity so what ultimately happened is they began building large networks and large data centers and what the data centers in those days looked like is they could be 5 10 15 000 square feet filled from floor to ceiling with servers routers switches and an enormous amount of other technologies and these data centers drew an incredible amount of power required an incredible amount of cooling and they required a management staff like you couldn't imagine to try and maintain all these devices and it was still financially successful for organizations they invested more and more in the data center and they also said wow we're getting so much benefit that they started building out networks and all of a sudden an organization that had 100 locations now connected all hundred locations to this data center through their network and that really worked very well but it had with it the complexity of managing an incredible amount of technology and also having the people that could manage that technology along with dealing some space power and cooling problems that were always challenging and then computers became much more powerful with moore's law we saw computing power increase exponentially and as computers became much more powerful all of a sudden it became feasible to have less computers so that's where virtualization came in and the first you know early stages of virtualization or kind of blade servers and what a blade server was you have the server chassis and you'd have these cards that would be placed into the server chassis and each card had their own cpu and memory and they were effectively multiple physical computers inside of a computer chassis and they worked great they decreased the footprint of the data center and to a large degree the power consumption of the data center but then processing power got even more powerful so then we started with the development of a hypervisor and what a hypervisor does is enables a single physical server to be partitioned into multiple logical servers and now that's realistically what we're doing we're taking servers we're chopping them up into logical servers through a hypervisor of some kind and creating virtual machines and that really was the key driver to cloud computing with the ability to virtualize your network and then actually connect to the network which is the next component of where we got to cloud computing which is high-speed networking when i began working with networking a 56k leased line could be thousands of dollars per month and the performance was very slow so what organizations did is they had their data center they would build these big offices right near the data center they would connect them to their land their land and through the land they would get the best performance they possibly could and then they would have remote offices connecting through the lan and that worked well for a while but it's extremely costly to maintain some of these large networks some of these networks would have thousands of remote locations and in the process these organizations effectively became their own service providers they effectively had their own cloud their own data center their own high-speed network backbone and that was the way organizations worked and that was the best we could do at the time but all of a sudden networking speeds enhanced exponentially but the price of networking went down so now that we have really high speed local area networks and we can do wide area networks over ethernet all of a sudden it became possible to have a gigabit connection from your data center to the cloud or a 10 gigabit connection or multiple multiple 10 gigabit connections and now in today's world an organization's office can have such high speed connections to the cloud that they wouldn't even know that their applications are housed in the cloud so by housing applications in the cloud an organization doesn't need to maintain the enormous staff that they would need to maintain if they're maintaining their data center they have much simpler power space and cooling requirements and we'll talk about in a minute but they have a whole lot less technology to build so this is some of the drivers that actually brought cloud computing into feasibility in reality today now we're going to talk about a few more drivers so we've talked about the key technology drivers for cloud adoption but now let's talk about the business drivers so before we discuss the business drivers i want to define two terms as it pertains to cost there is capital cost otherwise known as capex capital expenditures and there's operational costs otherwise known as opex and they're very different and with cloud computing there's a big shift from capital expenditures to operational expenditures so we're going to talk about what they are but one of the key business drivers to moving to a cloud environment is to reduce capital costs so for organizations that have their own data centers they have to buy physical servers they need routers they need switches they need firewalls load balancers racks power distribution units generators and cooling and that's just some of the capital expenses there's others and they still have some operational expenses for example if you have a data center that's got 3 000 servers in it you need a pretty large it staff to manage it and you've got some pretty substantial electric bills some related to both powering the servers but also cooling the data centers because these servers don't like it hot and the performance will degrade and the reliability will degrade dramatically under heat but an organization still has the wan connections they have their internet connections and all these things get more expensive now when you shift to the cloud environment what you actually get is very reduced capital cost for the most part you don't have anything to buy you're just purchasing or i should say leasing the technology on an as needed basis from the cloud provider so in this case amazon will be charging monthly or daily fees for every service used so with a shift from the cloud comes in a shift from the capital expenses that an organization would face to more operational expenses now let's talk about the key business driver to cloud adoption and that's agility and in a traditional data center environment if you need a new server you purchase a server if you have a new application you might need to order 10 servers now you might order the servers they may take you know a month to build by the time they're shipped to you when you get them you have to put an operating system on them you need to put all the dependencies for the application and all your applications and this can take a large period of time now with the cloud all you really need to do is you need to log into your console provisional server and that's it so the agility gain business wise from being able to deploy a new application in minutes versus in days or weeks is enormously beneficial to an organization so that speed of deployment is fantastic and whether that's related to a server a database a network performance optimization that's just a competitive advantage that cannot be matched in any environment other than a cloud computing environment and now we have auto scaling so without auto scaling let's say an organization has a web and application server they have to provision that server for the peak demand it might have one percent of the time now in a cloud computing environment you can purchase the use of a server based on your average computing environment and the server itself can auto scale by adding additional servers as needed so you can purchase what you need and then still scale almost unlimited limits so not only does that increase agility can also substantially decrease decrease costs with that comes serverless environments where basically you can just deploy an application by uploading your code and have unlimited scalability and that's really something you can't do without a cloud environment so the ability to buy unused capacity at a discount if you've got some commuting batch jobs and have a network that's pre-built that you can deploy pretty much anywhere anytime at a reduced cost over building it yourself are really the key business drivers to cloud computing now let's talk about how the aws cloud is organized so the aws cloud is broken into regions and availability zones and a region is a large geographic area inside of each region there are multiple availability zones now many students get this confused so imagine a region is a part of a continent an availability zone is just a data center that's inside of that region so when you realize an availability zone is just a data center then it makes complete sense that you've got this region which is a large geographic area and inside of that area there's multiple data centers and each data center is called an availability zone so since we're talking about cloud computing an organization is effectively going to create their own virtual private cloud so what is a virtual private cloud also known as a vpc and a virtual private cloud is a private network that an organization purchases from the cloud provider now the aws network is a shared network meaning there's multiple customers that are sitting on that network but they're logically separated and since they're logically separated one organization cannot talk to another organization unless they go to several steps to do so which is something called bpc peering which we'll talk about more later but the point being is aws can have thousands upon thousands of organizations and none of them can see each other because they're private unless you do some work to enable them to talk to each other and because each organization's network is private they can use their own private ip address space they don't have to worry about overlapping ips with other organizations they can be confident their security so other organizations can't see what they have and it's really just using the aws network so aws has their own network it's got their own high-speed backbone they have their own storage apparatus and they have their own when and land so by purchasing or at least leasing an organization's virtual private cloud you're just purchasing access to the aws network so let's talk about the two types of cloud architectures you're going to see as a cloud architect the first type of architecture is a hybrid cloud and a hybrid cloud is just simply an organization has a data center but they also connect to a cloud computing environment meaning they're using both and why might this be let's say an organization has invested an incredible amount of financial resources into building their own data center they would like to leverage the investment they've made in this technology while still gaining some of the agility benefits that can be gleaned from the cloud so in many cases there's a migration to the cloud and people have a hybrid cloud for quite a while there are also ultra high performance applications and there are not many but there are some applications that literally benefit from being one nanosecond quicker than than other applications and just that latency requirement might make it better to have certain applications housed in an organization's data center and then house the rest in the cloud this is not your typical customer but there are applications that are that latency sensitive especially for competitive advantages now when it comes to disaster recovery let's say you have your environment and you back up your entire environment to the cloud that creates a perfect low-cost disaster recovery environment and it's a perfect use case for the cloud and then some organizations just want to place certain things on their own network and use the cloud for the rest and that's typically when you'll see a hybrid cloud environment now the next type of cloud architecture is what's called a pure cloud and if you don't have a data center for example an organization has no data center they have no technology but they're deploying new applications new services a pure cloud computing environment is fantastic and it's ideal because when you start an organization you don't completely understand what your technology requirements are you think you do but what happens is you may think you're going to get a certain number of hits on a web server and all of a sudden you receive a thousand times that and you just never really know so the auto scaling capabilities of a cloud architecture are enormously beneficial to a startup now i mentioned auto scaling a few times already and the cloud computing environment promotes near unlimited scalability so you can know that you can deploy your application with the least amount of resources you think are necessary but it'll still grow to meet your needs and that ability is something that can't be offered in any other environment so that scalability of the auto scaling that can be done with the cloud is incredibly beneficial for businesses now let's talk about the speed of deployment if an organization has to purchase servers have the servers shipped to them and then they have to install their operating systems find a place put for them put them in a rack connect them to power then install their applications dependencies in the application this can take a long period of time but with the cloud you can simply just go to the website and literally click a few clicks and all of a sudden you have a server of course you can do it via an api and you can do it via the cli which we'll talk about later but the point is you can pretty much instantly deploy things as needed and if you're on the cloud and you have multiple partner organizations and they're also on the cloud you could just simply hear them and enable them to talk to each other without building other network connections which could take a lot more time effort and money and generally speaking a pure cloud computing environment can be cheaper than a traditional environment or a hybrid environment but that's going to be dependent upon the organization and the application generally speaking cloud computing environments are cheaper than traditional data center environments but i've covered this before you need to understand that with traditional data and center environments there's a lot of equipment to purchase which is a capital expense when it comes to a cloud computing environment you could be paying by the minute the second the day what have you and you have much higher ongoing operational cost it's a shift from capital expenses effectively to operational expenses so let's talk about connecting to the cloud if an organization is going to place their data in a cloud computing environment which is effectively outsourcing their data center then they need to be sure they have reliable and high performance connectivity between the organization and the cloud computing environment so the first way we're going to talk about this is with a direct connection and a direct connection is going to be your highest performance option and it's going to be best for most customers and what a direct connection is it's realistically speaking it's a wire between the cloud provider and your organization so there's direct connectivity in between it and because it's effectively a wire which will be used for ethernet realistically speaking you know exactly the performance you're going to get if you purchase one gigabit per second you're going to get one gigabit per second because it's effectively a wire this is different than a vpn and we'll talk about why vpn performance is typically lower but for right now understand this is guaranteed performance now it's not just guaranteed performance in terms of bandwidth it's going to be guaranteed and consistent performance with regards to latency you'll have an sla that says latency will be less than a certain amount and the latency will be consistent this is very important to have consistent latency for voice or video applications because these applications are very sensitive to something called jitter for which that is realistically speaking just variations in latency if you always have 5 milliseconds of latency that's typically not a problem but if you have 5 milliseconds 50 milliseconds 10 milliseconds and then five it's very hard for real-time applications such as voice or video to perform reliably and properly so the direct connection gives you your best performance and is the best option for high availability high reliability applications now the next method to connect is actually with a vpn connection and what a vpn connection is is an organization organization connects to the internet and aws connects to the internet and they have an encrypted tunnel in between the organization and the cloud provider now this is typically cheaper and the reasons is typically cheaper is you're not buying a private connection between the data center and the cloud all you're buying is a connection to the internet now because the internet is public in order to maintain privacy you have to encrypt that data so the data will be encrypted via something called ipsec and that'll encrypt the data and it'll do a few things it'll make sure that the data has maintained its integrity that it hasn't been changed along the way it'll enable the organization to verify the sender and the receiver so you know it's the organization connecting to amazon and it'll also facilitate something called non-repudiation because of the way the messages are sequenced if the organization connects to aws the message can't be changed and the organization can't say they did not make a trade they did not purchase a product you can guarantee because it's encrypted data and the data is measured then you know that the person on the other end is who they say they are you know that no one else can read it along the way and if you're using it for signatures and the such you can guarantee that the person can say it but you also know that it hasn't been changed imagine trying to send a one million dollar purchase order and have it come out as a billion dollars because someone changed it along the way so that's really why you're using encryption now because you're using a vpn encrypting the data you can use your private ip addresses that are going to be tunneled inside the vpn connection so that enables you to have a private network across a public inter across the public internet now here's the thing with vpn connections they're inexpensive but there's really no bandwidth guarantees and here's the reason if you purchase a gigabit connection to the internet you know that you have a gigabit connection to the internet but the internet is comprised of multiple internet service providers and multiple other organizations called autonomous systems and just because you can guaranteed performance to the internet doesn't mean the internet which is owned and managed by many many many many organizations along the way they don't have to prioritize your traffic so you can get an sla of certain gate certain guarantees to the internet but whatever once it's on the internet whatever happens there's no guarantees it's best definite delivery so with a vpn environment you're going to have variable latency and this can work for some applications but not others so you realistically speaking need to know what are your latency and performance requirements if you have tight demands on latency and on strict performance requirements a direct connection is what you're going to need to use if you have much looser requirements with regards to availability performance latency a vpn vpn connection would work well so the organization will have to determine whether it makes sense to purchase the right connection or whether they can get away with the vpn now let's talk about high availability there's a saying that's called one is none and two is one realistically speaking when it comes to availability you need at least two of everything i mean literally everything if you're using a router you want two routers you want two power sources you want two connections over two different providers and the list could go on so realistically speaking most organizations that require high availability will have a direct connection backed up by a vpn connection so that's your typical high availability architecture when it comes to connecting to the cloud now that you know how to connect to the cloud let's talk about accessing accessing the cloud and managing the cloud so the first thing you're going to need to do if you sign up for a cloud provider is you're going to have to manage your virtual private cloud so with aws realistically speaking you have three options the easiest option for most cases is to use the management console and that's quite simple you're going to log in to the aws website and inside there there'll be a console inside the console though you'll be able to select what you choose to do and there'll be help along the way so this is the easiest option for most new people especially because of the abundance of help that's available for you now the next option is going to be to do it via the command line interface so in this environment what will happen is the organization will set up a secure cell or ssh session between a terminal at their local facility and between the ssh i'm sorry between the aws cloud and you'll be able via the command line to actually be able to monitor as well as configure and make changes to to your systems and that also works very good highly highly efficient very quick but you absolutely have to know the cli and you have to know everything you want to do before you do it because there's not going to be any help it's not going to be a easy to click website and the other option is to do it through the software development kit so it's quite possible and many large organizations can figure things through the api or programming interface through the software development kit and that is your perfect option but you realistically need speaking need to really know what you're doing and this is for larger scale deployments now that's typically the way most servers are managed but if you have a windows server on the aws cloud and many organizations have windows servers whether it be for active directory or some other windows based application then you're going to have to use a remote desktop protocol and that's going to be something where basically you'll be able to see the remote desktop windows typically doesn't do well with it with ssh configuration like a linux or unix system so when it comes to dealing with windows servers you're going to be using remote desktop protocol also known as rdp let's talk about storage on the aws cloud so before we get into the types of storage let's first define storage and storage is the environment where you're going to keep an organization's data so we like to define storage in terms of volatile storage and non-volatile storage volatile storage is something that goes away with a reboot so for example the storage that comes with an instance instant storage is volatile meaning as soon as you reboot or terminate the instance all the data that's there is gone so now you know that instant storage is volatile don't store anything critical on instant storage and then we have non-volatile storage and non-volatile storage stays there after the system is rebooted so for anything that matters you're going to put your data on non-volatile storage and storage is an absolutely critical component to your vpc and choosing the right storage options based on whether you need host storage network file storage or high performance storage can be the matter between an application being successful and an application being a failure so we'll cover those types of storage in depth but let's first talk about the types of storage so the first storage we're going to talk about is block storage block storage is a highly efficient high performance form of storage and with block storage you take the data the data is chopped up into little blocks and each block has an identifier and the data is reconstituted based upon the identifier when it's needed now this scales well and it's extremely high performance the reason it's so high performance is blocked can be placed anywhere on the storage array that's most efficient so by being able to place the blocks anywhere that's most efficient and then being able to reconstitute them on demand by the block identifiers you're getting really good high performance storage and we'll talk much more about block storage when we talk about ebs or elastic block storage now we'll talk about object storage and object storage is a bit different than block storage uh and in blocks in block storage data is broken down into blocks but in object storage data is broken down into objects and each object will have its own unique identifier and each object will have metadata and the fact that each object has metadata gives you some really interesting things that you can do with object storage and we'll talk about that much more when we talk about amazon s3 and then there is file storage now your file storage can be attached to the host or it can be on a network and typically we think of file storage as traditional storage it's the hard drive that you have in your computer system and these are typical volumes that can be formatted based on what a host would use for example a windows host would use ntfs volumes those are the three primary types of storage used on the aws network so let's begin by discussing amazon simple storage otherwise known as s3 now s3 has a tremendous number of use cases on the aws cloud and is typically used for backup and archival of an organization's data static website hosting distribution of content media or software disaster recovery planning and with the integration of things like redshift big data analytics and data lakes so because an organization's lifeblood is its data anytime you have a lot of data that you're going to store you're going to have to make sure your data is secure so if you're going to secure your data in an s3 realistically speaking you have two options with regards to data security you can create a bucket policy which is by far the preferred method and this is very granular and it's based upon iam and you can determine who can access what and what they're allowed to do with the data in a very granular manner now the alternative is more of an access list but this is really not so granular i mean it gives you the ability to do read write or full control like typical unix type permissions but you've got much more security options with a bucket policy now s3 is used for a wide variety of use cases as i mentioned so they also have multiple storage classes and each storage class has different use cases so we'll begin with amazon s3 standard so the typical s3 that you think about is s3 standard it's very high availability it's very high durability and it's very high performance and i'm going to repeat that again it is extremely durable data meaning it's 99.99 all the way to nine places after the 99 so which is considered to be eleven nine's durability so that means that the data is going to be there and not be destroyed even if it's temporarily unavailable and the data is 99.99 or four nines available meaning when you actually want to use the data chances are pretty strong that it's going to be there for you but it's durable enough that even if it's not available at that one time it's extremely likely that it'll be available for you later and you're going to use s3 for data standard for data that you're going to use on a typical environment so this is for frequently accessed data it's also the highest price of the s3 but if it's data that you're going to use on a consistent basis this is by far your best option now there's something called s3 infrequent access and what s3 in frequent access is it's s3 but they give you a discount but you actually have to pay when you retrieve your data but it's still incredibly high availability it's still incredibly durable and you're going to use this when you have data that is very important your organization but you don't need to access it very frequently because you're going to pay every time you access it so if it's frequently accessed data it'll cost too much to use as data storage but if it's infrequently accessed data this is going to be a great option for you now for people that need data but they don't need it to be as completely available and they're still looking to save costs they can do an amazon s3 in frequent access but it's only in one zone so it's going to be reduced availability storage at a lower price and this is good for data that you may not need critical instant access to that you don't frequently access because it's going to give you some better pricing for the reduced availability now there's a new feature and function called s3 intelligent tiering and what that is is amazon will actually monitor how you use your data on s3 and it will move your data to the appropriate storage class meaning if it's something you don't use almost ever it'll move it to uh s3 and frequent access if something you never use it could move it to other forms for you as well but we'll talk about those other forms right now so the last two forms of s3 is amazon s3 glacier and glacier is really for deep archival purposes traditionally speaking you put your data on glacier you just don't access it because you pay for a retrieval fee when you need it and when you need your data it's going to be you know multiple hours three to five hours on average now you can pay an expedited retrieval fee to get it sooner but you don't put anything on amazon glacier that you're typically going to need now you can do a vault lock option with glacier so let's say you're a health care institution or a financial institution and you have to store records for a certain number of years with the glacier vault lock option it'll put it in an immutable vault which means the data can't be modified so you pop it in it can't be changed until you take it out so that's really good for regulatory environments so some things to know about glaciers it's just very low cost but you have to pay for data retrieval so you have to understand all the types of data that you're using when you're using s3 to pick the right option because when it comes to cloud computing you can easily have very high ongoing operational expenses if you make the wrong choices now let's talk about life cycle management in order to save costs and make the best use of their of their of their data you can do what's called life cycle management with s3 and let's talk about this typical use case it's very frequent that organizations have some fresh data that everybody needs to use for a period of time and then they have some data that some people need to use for a period of time and then they have some data that they need to maintain for archival purposes or future machine learning purposes so they want to save the data but they're almost never going to use it so what you can actually set up is you can set up lifecycle management for example let's say you need to use your data every day for 30 years i'm sorry 30 days you could pop it on standard s3 now after 30 days you might only need that data twice a month so you can then migrate that data automatically with lifecycle management to s3 and frequent access and let's say after a period of say 30 days or 90 days you no longer need that data but you want to maintain it you can have that data move to glacier in which case you'll pay the least amount of money to store the data but you'll still have access to it so let's discuss protecting your data so there's a few options to help protect your data and one of them is s3 versioning and what s3 versioning is when you take a file and you store it into your s3 bucket it creates the file with a version number now if an organization updates or modifies that file s3 creates another version number and then if you modify the file again s3 will create another version number so every time a file is modified s3 will store another version of that file and that way if one of the files were to be lost at least you have all the other versions now an excellent way to protect very sensitive important data for which the business is relying on is with s3 multi-factor authentication delete and here what goes on is if you're trying to delete a file it will use a multi-factor authentication delete to to make sure that you're the right person and you truly want to delete that file so here's how it's going to work a user tries to delete something in an s3 bucket they'll receive a challenge which will ask them for a one-time password if you will it could be an authentication device it could be a text message but the point is with doing it this way at least you receive a challenge so they know that you are actually you trying to delete it then now you'll use your authentication response which is the one-time password that you'll use you'll you'll send that challenge and if your challenge is accepted the file will be deleted so this is really a two-step process it's first making sure that you want to delete the file in case it's accidental but it's also making sure you are you instead of you being a hacker now when you organize data in s3 it's a little bit different than traditional file storage so s3 is object storage and not file storage and s3 is organized a whole lot like a database instead of file storage and some of those s3 database type features become very useful when it comes to some big data analytics and some other things which we'll talk about later in the course so s3 is completely flat storage anything that's in your bucket is flat storage so what happens is when you store something new into an s3 bucket there's a pointer in the database like structure that shows how to get to your data that's stored in this flat storage and because of that you have the opportunity to do incredible things in the big data environment and other environments and run sql queries on your data but that's again we'll cover that more at a later portion of this course but understand that s3 is more like a database than traditional file storage but if you want your s3 to feel like traditional files or a traditional fire server or file storage you can use some tricks to make it happen for example you can use a forward slash or a backslash to uh as a delimiter to make it feel like unix or windows based storage for the user that's trying to receive their data on s3 so anytime you have data and the data is valuable to your business or you might have some business secrets in that data you need to protect that data one of the best ways to protect your data is through encryption and you can definitely encrypt your data on s3 so you have to protect sensitive information because if you don't it'll at some point be compromised and if trade secrets are lost an entire business can be lost so data protection is absolutely essential so encryption is a means to make the data you have unusable to anyone unless they have the decryption key and brief brief overview of encryption what encryption typically does is it takes a specialized key and it takes that key and it runs it through a mathematical encryption algorithm and then it decrypts your data and the data will be like i said unusable and then you need that key to actually decrypt the data on the back end and that's how ipsec works that's how so vp that's all vpns work but also that's how many things uh in blockchain work but that's also how you want to protect your data and we cover uh encryption algorithms in much deeper depth in networking and security courses so for now understand that encryption makes your data unusable to anybody without the decryption key so if you want to encrypt your data on s3 there's really two ways you can do it you can encrypt it on the client side before you send it to s3 or you can encrypt it on the server side so let's talk about the first type of encryption keys and the way you can use them with aws the first one is called the sse-kms system and realistically speaking this is where the customer manages their own encryption keys but they're using the aws key management system to do it and it the aws key management system is a full system to manage keys rotate keys and really take a lot of the overhead away of managing encryption keys that occurs when you're using encryption in a business environment so with this environment the user manages the master key and the key management system will basically make a data key and it'll manage it and it will provide an audit trail of how when and who accessed the data so this is a really great way to do things if you want to manage the encryption key now there's also another version called sse-s3 and that's where aws manages all the keys and this is a complete key management solution so what happens is the key management system makes the keys it automatically rotates the keys and every object is actually encrypted by a unique encryption key so this is a great automated way for key management but some businesses for security reasons or regulatory reasons may need to manage the keys themselves and for those reasons some people might want to do you know customer managed keys and let's talk about the last option and that's going to be sse-c which is customer provided keys and in this case the user will have complete autonomy over their security keys an encryption key so they'll have complete autonomy over all keys but then again they also have to manage the keys and all the overhead associated with key management and encryption of your data let's discuss tuning s3 for optimal performance we're going to be discussing pre-signed urls multi-part uploads and cross-region replication but let's begin with pre-signed urls so you have to understand that everything that you store on s3 is private and if you want to share data you're going to need to provide access to the data to anyone that you want to show the data with and the simplest and easiest way to do this is to just generate a pre-signed url so what happens is from your account you'll create a url and it's going to use your own encryption key and your and it's going to create that url and then you'll share that url with the person you desire to receive the data and they'll have access to that in a secure manner and a temporary manner so you can temporarily provide very secure access and it's the simplest means to do this now when you're using a pre-sung url the urls will expire and the expiration length is determined based upon how you sign the url so if you sign the url with an im instance profile it's going to expire in up to six hours now if you signed it with an aws security token device the expiration time you can make up to 36 hours now if it's already an im user then the date is good for up to seven days or you can configure it to be the url up for seven days and if it's a temporary token it expires as soon as the token expires now let's talk about multi-part uploads so s3 is some pretty interesting storage it enables you to store files as big as five gigabytes but a five gigabyte file transmitted over the internet there's just so many things that can go wrong you can reach a congestion point on the internet data can be lost and the whole file can be lost so it is recommended at any time you you send you to send a file that's greater than 100 megabytes that you actually use a multi-part upload and what happens with the multi-part upload as the file is broken down into many little files and little files are sent and once all the little files are received it gets the data gets reconstituted into one but if any of the many files get lost along the way they're just re-transmitted instead of re-transmitting the entire file so that's how this is so much more efficient and it's recommended anytime you have a file larger than 100 megabytes now let's talk about cross-region replication as we previously discussed aws has multiple regions you can have an s3 bucket say for example in the east coast region of the u.s and want to ship it to an asian region and you might do this for two reasons one is you may have a main business in both the us and asia another reason you might want to do this is you've got a tremendous number of customers in asia that are using an s3 based website so by bringing the data to the website that's hosted off s3 locally to asia it'll be much faster for your asian customers than if they had to reach back to the us for the data the third reason you may want to do this is just disaster recovery purposes have your data literally based backed up into automatically by the way into another region geographically that's far away what a great way to protect your data so just understand cross region replication basically takes everything in your bucket and will replicate it to another region so that's cross region replication and great way to really protect your data and make sure you have a perfect backup at all times the next type of storage we're going to discuss is instant storage so instant storage is just storage that's attached to an ec2 instance now you have to remember that instant storage is volatile meaning that as soon as you reboot or terminate an instance all the data that's on that instant storage is lost so as a boot volume you could possibly use instant storage having said that you can't store any data there at least if you want to be able to access that data later now let's talk about elastic black storage elastic block storage is block storage on the aws cloud and elastic block storage which you'll hear called ebs is a very high performance storage and it acts like a virtual hard drive and you typically would use this in ec2 instance when you need high performance storage or storage that does not go away upon instance termination or you would also use it with your database because it can be some very high performance storage and certain databases need access to very low latency high performance storage and this is also for mission critical use it's very high availability it's nine it's five nines available so you're going to have access to your ebs storage when you need it it's optimal for high throughput and high transaction workloads and you're going to have multiple options performance options with ebs volumes which we'll talk about in a minute now ebs volumes are associated with a single availability zone and you really need to remember that they're associated with a single availability zone because that'll come up in examinations then i'll also come up in actual practice and evs volumes are actually automatically backed up to another availability zone so great way to make sure that the volume itself is completely backed up but to another location should anything happen to the location with your data stored and ebs volumes are actually backed up via a snapshot so basically what happens is an image is made of the data in the form of a snapshot and it's typically stored on s3 as i mentioned there were multiple options for the amount of performance that you can get from an ebs volume and the way that occurs is you have to choose the correct ebs volume type for your application and there's volume types based on different levels of performance requirements specifically latency and throughput so latency is realistically speaking how fast the disk can be accessed in terms of the number of times in and out per second whereas throughput is the amount of data that can actually be moved measured in terms of megabit per second and these are very different and you need to make sure that you have the right volume type to support an application's requirements so let's talk about the first and overall best performing volume type available and that's ebs provisioned iops and here's what you need to understand with this option you can actually procure ahead of time the number of input and output operations per second that you can you'll receive and by doing that you can make sure you have guaranteed input and output operation per second which means lower latency and if you've got an application sensitive to latency this is the volume type you're going to be using because it's giving you guaranteed performance in terms of latency it's also the highest throughput of the options so if you've got something that needs high throughput and low latency this is your perfect volume type and again this is going to be used often in databases because databases are very latency sensitive but this could be used for any kind of application that could be latency sensitive like like gaming for example now let's talk about the next volume type and that's going to be ebs general purpose ssd and this is general purpose ssd storage it's got a really good blend of price for performance and you might actually use this as a boot volume instead of using instant storage because ebs volumes are non-volatile meaning they don't go away with instance termination they stay there so this could be a great way to actually build a boot drive that's very high performance for one of your computing instances and this type of volume is exceptional for transactional workloads because it's moderately low latency and it has still relatively good throughput and another place where we use this type of volume is let's say you have a production database and production web application running on provision iops and this is your lab environment where you're going to stress test it but not to the ability to at least the levels that occur in the production environment you can use the ebs general purpose ssd is a way to save costs in your test environment as opposed to using the provision iops that you're using in production now the next type of item we'll talk about is magnetic storage and this is ebs throughput optimized hdd and this is relatively low cost magnetic storage but it's got some relatively good throughput so if you've got an application that needs to actually send and receive a lot of data but it's not sensitive this is an excellent option and it's fantastic for log storage and it's great when you have a lot of data that's going to be sequential read sequential writes and it's got good throughput but not for latency sensitive applications and the last biome type we'll talk about is ebs cold hdd and this is the lowest cost option you could possibly buy in terms of storage of ebs volumes but you're going to use this for workloads that are not accessed frequently so low performance access for for data that you want to store and maintain storage because you don't want to go away with instant termination but it's not for data you're going to access frequently the next type of storage that we'll discuss on the aws platform is the elastic file system also known as efs and the elastic file system is network storage so where an ebs volume looks like a virtual hard drive to a system this is network file storage in fact it's extremely similar in structure to the nfs or network file system that was originally developed by sun micro substances which is now oracle and this is uh and nfs is running on millions upon millions upon millions unix and linux systems throughout the world and it's also important to know that ebs volumes are highly scalable in fact they're so scalable that they actually grow as needed so you don't ever have to worry about running out of network storage on an efs volume they're very high throughput they have very high iops so they're low in latency so these are great performing network storage and it's also something called posix compatible and what you really need to understand about that is it means that it will work with legacy systems now the next type of storage we'll mention is amazon fsx for windows and what amazon fsx is for windows is its fully managed high availability windows storage and because it's windows storage it uses the smb or server message block protocol and this is uh supports you know microsoft features and functions such as active directory so for organizations that have highly windows dependent workloads that need windows dependent type storage amazon fsx for windows is your option to use so let's discuss scalable means to send an organization's data to s3 the first option is with a storage gateway and a storage gateway is an appliance that you place in your data center that directly connects to amazon s3 services this appliance is actually a virtual machine but what happens is the organization puts this virtual machine in their data center and the virtual machine to local servers feels just like a traditional file server so the organizations have their servers they connect to this file server but really it's a storage gateway and then it copies all the data from the organization's data center to amazon s3 so for an organization that wants to use this as a backup disaster recovery purpose it will work perfectly but also it would work let's say an organization has a lot of data they're planning on migrating to the cloud it's going to take a lot of time to copy it over the network connection they put this storage gateway in and data is asynchronously replicated all the way to s3 now the other place where you would use a storage gateway is for a hybrid cloud where an organization has a data center and they also have a cloud and this way they can keep their data synchronized so the storage gateway is a really great way to get your data to aws in a scalable manner now i describe the file gateway where it's copying files and sending it to the aws data center now there's also an option to do this biom gateway in a cached mode and what ultimately happens is you would then place the virtual machine in the data center and let's say an organization's data is stored on s3 what will happen is as users request information from the storage gateway it'll be pulled from s3 to the storage gateway and then distributed to the user but if it's frequently accessed content for example and say another user requests the same data from s3 that the previous user would have that data will already be sitting on the storage gateway in terms of being cached so it will operate like a web cache so the point is is new requests obviously we'll have to go to s3 and then be brought to the storage gateway but for data that's frequently requested it can be stored locally on the storage gateway in the organization's data center without having to reverse the land to actually get to the aws cloud now the storage gateway is a great way to migrate data if you have sufficient network bandwidth and you have sufficient time now there are times where an organization has to move or migrate a tremendous amount of data in a short period of time to the cloud and amazon has a solution for it and it's called a snowball and here's what the snowball is amazon has these highly ruggedized computers with a massive amount of storage and what happens when you have a tight time frame to migrate your data or you have a long time frame but you don't have anywhere close to the network bandwidth that you need to transfer all your data what you do is you request one of these snowballs so amazon will ship you one of these snowballs with their massive storage capacity and you'll place the snowball on your network using a 10 gigabit ethernet port and you'll copy all of your data from your systems to the snowball or multiple snowballs if you've got a lot of data and as soon as your data is copied to the snowball you'll contact aws and then the package or the snowball will get shipped back to aws and when the snowball gets back into aws's possession they will copy it onto s3 and now you have your data on your vpc on s3 now aws offers another option for organizations that have tight time frames and moderate amounts of data but not to the level where they would need something called the snowball and that's called the aws import expert service and realistic what happens you you take an external hard drive you copy your data to it and this could be multiple external hard drives and then you ship the data to aws and then the employees at aws take the data from your hard drive and they copy it onto s3 for you so in this module we've talked about the types of storage block storage object storage file storage we've talked about s3 we've talked about ebs we've talked about efs we've talked about fsx for windows we talked about storage gateways the aws snowball and the aws import export service what we're going to do is we're going to create a budget and the reason we're going to create a budget is we want to train as efficiently as possible in every way from a time perspective but also a cost perspective and i don't like to receive surprise bills at the end of the month and i assume no one else does so by setting a cost budget we'll be in a position to know that what we're going to spend and be able to plan for what we're going to spend and make sure we don't overspend so as soon as you log in we're going to go to the management console and inside of the management console you'll see a section called aws cost management inside of there there's going to be an area called budgets and that's where we're going to click so when we get to this area we have the opportunity to create or manage a budget refine an existing budget or add a notification to a budget that already exists assuming we haven't set it up yet but in this case we're going to create a budget and we're going to create a cost budget because we want to be able to monitor our costs and get alerts once we receive a certain threshold so now we're just going to set our budget and we're going to call this the go cloud architects budget and in your particular case it'll be something different i'm going to have this every month and i'm going to set a fixed amount i'm going to set mine for 50 you realistically speaking don't need to go there but i'm going to go there because uh that's typically uh what i would expect to spend given my situation because i'm constantly doing labs from here we're just going to configure the thresholds and we're going to set up the threshold to know when we reach a percentage of the cost in my case i like to uh get a little more notification so i can see exactly what's going on so when i reach 50 i want to be notified and i'm going to just set up a very simple email now i could set this up with sns and there's other opportunities to do things with chatbot alerts but i'm just going to do a simple email and that's the email address for my company so i'll have that as the location where it's going to set things up and here you can see quite simply i've got a budget i've set it for fifty dollars a month as soon as i reach fifty percent which is twenty five dollars i'm gonna get an email and that's it now you've got a budget it's completely set up and you'll be alerted so pick something that you can afford and now it's time to start beginning our lab so welcome to the program welcome to the labs and we hope you have a great time and learn a lot to be doing is we're going to be creating an s3 bucket just a basic s3 bucket so you know how to do it in the future we'll do more things with s3 like create some lifecycle policies and enable versioning and a few other things but for right now we're going to work through a very simple creation of an s3 bucket so log in and after you log in go to the management console and under storage you'll have your various storage options and we'll talk about much more about them later but you've got your s3 or your simple storage you've got your elastic file system you've got your file system for windows you have your s3 glacier which is long-term storage for things you don't need all the time at least access to we have the storage gateway and we'll talk about them much more but that's a way to connect your data center to your aws cloud and share information and then we'll talk about backup options but for right now it's just s3 so we're going to go to the management console we're going to click on s3 and very simply it's going to give us a tool in terms of how to do this now we're going to go and create a bucket and you have to name the bucket and since we're go cloud architects we're going to use gca for our abbreviations and we're just going to call it bucket this is a very simple exercise to just learn and you can at this point pick the region that you choose to be in i'm going to stay in the us resto ohio area and if you wanted to enable bucket versioning which we'll talk about a lot more you can add tags to help know what's going on with your storage and your types of things that are there and you can enable or disable encryption here but for right now we're just going to simply create this bucket so we're going to click create bucket now you can see we have a bucket that's it so let's do something with this let's place a file in our bucket so we're going to click the bucket and over here it's going to give you a great menu option now we're going to click upload and from here we're going to just add some files or folders i'm going to do it for my computer and i'm going to click add files and i have a cover of an ebook that we give away for free this ebook so i'm going to upload that cover to my s3 bucket and then i'm going to click this and then i'm going to click upload and there you have it we've created an s3 bucket with some things inside of it specifically my book and you can click on the bucket and you can see what's inside there and of course you can look at bucket properties which will tell you what's going on about the bucket permissions we can which we'll talk about later and then you can see other types of associated things but there you go now you have a simple means to create a simple s3 bucket going to be discussing is something called s3 versioning s3 is an ob form of object storage and by nature with object storage anytime a file which is broken down into objects is modified object storage typically creates a new object which can get very expensive in terms of filling up space having said that it can protect you against accidental deletion or file corruption so let's say we have a file and the file is going to be accessed and modified on a daily basis s3 will actually store every version that existed so for example if a new version gets corrupted you can just revert to the previous versions now we'll also talk about throughout this course other ways such as cross region replication to make sure your data is in two places in case your data is actually corrupted but in this particular case versioning is a wonderful way to make sure you have a backup of the previous copy of your data so very simple to set this up we're going to go to the management console once again we can go to storage s3 here or because we were just there we can click on s3 here and from here we're going to access our bucket and if we look at our bucket it's going to give us some information about our bucket and that's all we need to do so we're in this position and let's look at the properties of our bucket and we can see what's going on here but we haven't set up versioning it says versioning is a means to keep it up so let's edit our versioning configuration and let's enable versioning and then we'll click save changes now that's it every time we modify this file it will create a new version of the file and if we modify this file 10 times a day it'll create 10 new versions and if any one of those versions like get corrupted we convert we can revert back to a previous version so that's what you need to know about versioning it's very simple to set up and it is a fantastic way to protect you against accidental or malicious data laws we're going to be doing is we're going to be setting up something called s3 life cycle management and i'll explain what that is there are times where an organization has new data and they access that data really frequently for a period of say 30 days and then that data that's accessed very frequently becomes accessed infrequently but it's still occasionally accessed and by taking your data out of your traditional s3 bucket and sending it to s3 in frequent access you will actually be able to save quite a bit in terms of cost assuming your data is accessed infrequently like we talked about in the lecture discussions if every time you request information from s3 and frequent access there's a charge for it so if it's almost never accessed it's a great way to save some money by placing your data into s3 and frequent access and you'll have immediate access to the data you just have to pay a retrieval fee and then you might find that you don't need access to that data after a period of 30 days 45 days almost never again but you still might need to keep that data for example you could be in the healthcare of the banking industry or another regulated industry that mandates you keep data for seven years you need to hold on to it but you want to do it in the cheapest way possible so you could use glacier you might also be running some machine learning applications and want to save the data that you've gleaned in your s3 bucket for future training data for machine learning so these might be some of the reasons you'd like to keep historical data or you might use it for other business intelligence applications where you can look at trending data and things that you've done in the past so that's why we would use lifecycle management to make sure that we save our data in the most cost effective means possible so we're going to go to the management console and what you can see is we can easily click on s3 here or we could look at reasonably visited services and we just created an s3 bucket so i'm going to just click here make life easy so here you can see the bucket that we created in the previous lab called the gca bucket and in the bucket you can see the whole one file our book cover template that we uploaded so let's talk about some lifecycle management and how we're going to set that up we're going to click on the management tab and under the management tab is going to be quite simple we're just going to click create a lifecycle rule and we can call this rule anything we want so let's take it let's call it s3 data management and we're going to create some kind of a prefix to limit in this case we're just going to do something very simple and something very simple we're just going to call it s3 bucket access now when you do this you need to come up with a tag and when you're creating tags it enables you to identify things that are going on or you can limit the scope for example you could create a tag and tag things as secret and then use that to make sure that certain things aren't taken outside of a region for example you might be working in a government environment and the government may mandate that things are not copied outside of the u.s for example if you're in the u.s environment and you can create a tag and then you can create rules later based upon those tags to protect your data but in this case we're going to come up with a very simple tag and we're going to call it s3 data but we're actually going to put that in the value and we're going to list it over here as a name so like a database kind of environment it goes with key values so there we go and then i'm just going to create a rule and we're going to transition things so now what we're going to do is let's say i have my data on my standard s3 bucket like i do and after 30 days i want to take that bucket because it's not used that often i don't want to send it to s3 and frequent access because it's cheaper so what i'm going to do is i'm going to i'm going to pick my storage class which is going to be s3ia and we're going to put 30 days and it's also pretty common for people to keep their things in s3 for a period of time and then send them the glacier and the reason they're going to send them the glacier is they want to create an environment where it's cheap to store things for a long period of time that they're not going to be accessing frequently now bear in mind when things are on glacier it's going to take a period of time to retrieve them hours now you can pay an expedited retrieval fee to get them sooner but it's going to take hours to get your data we're going to acknowledge that this life cycle rule will increase the cost especially if we're transitioning a lot of small objects and i'm going to create the rule and after we create the rule it'll check and see if we've done everything properly and if we've done everything properly the rule would be created if not it will throw an error and as you can see the life cycle management rules have been completed we have uh data that's going to go to from our current place where it's where it's there to 30 days later to s3ia to uh 60 days out 60 days it's going to go to glacier that's it that's how you set up lifecycle management on s3 buckets video is about s3 cross region replication and there's lots of reasons you might want to actually replicate your data across regions region one could simply be if you have your data in one region and that region were to go out and be compromised you'll still have your data in another region but most frequently the reasons people are setting up s3 cross region replication is actually to make sure that their data is in a bucket in another region and they're going to do this because in a lot of cases people host static websites through s3 and when your website is hosted on s3 that you're going to pay for data across transfers across regions so let's say for example you have a website in the east coast of the us and you've got a lot of customers on the east coast and you also have a lot of customers in asia if your asian customers are accessing your data from an s3 bucket that's sitting inside of the us there'll be latency that's associated with those long cross region transfers and some of this will get better with cloudfront which we'll talk about later but there'll be a latency issue plus there's also going to be a cross region data charge every time that information is accessed so by getting your data closer to the customers it may be more cost effective if you don't have to pay the cross region charges and that's going to be based on the amount of data that's being transferred but it will also give your information in another location for extra redundancy and it can improve a performance because latency will be less as you don't have to travel so far so here's how we're going to set up this s3 cross region replication the first thing we're going to do is we need to have a second bucket to transfer our data to so we're going to do so so we're going to create a new bucket and we're going to call it gca bucket 2 because i have a gca bucket 1. now i'm going to place this in a different region for our purposes in the lab environment we're going to create u.s region west we'll pick something and say california and we can copy our configuration files from a previous bucket or we could set up something different but these are going to be replicated buckets it would be best to have the same configuration on both sides so let's choose the bucket and as you can see it's going to take policies bucket versioning was disabled before but now it's enabled so we're going to quite simply create our bucket and after the bucket comes up you can see we have the primary bucket that we previously created and we have the new bucket so we're going to replicate from this bucket to this bucket so the first thing we're going to do is we're going to click on the current bucket we have and we're going to go to the management tab of this bucket and we're going to create a replication rule to replicate our data from the first bucket to the second bucket so let's do that we'll create a replication rule and like anything else you've got to create a rule name and these names are things that you logically identify so we're just going to call this s3 bucket and i'm bu ck bucket replication and uh i'm not the best speller in the world but we're also doing this for speed and expedience because we want to create the best environment for everyone and we can see that we've this is going to be enabled because we obviously want it to work and we're going to create the source bucket and then we're going to create the debt we're going to send it choose the bucket which is going to be the place that we want to send our data to which is going to be gca bucket two now when we do this we typically have to choose an iam role let's see what we have here i created a role specifically designed this and we're going to just choose that rule and if we wanted to replicate encrypted objects we could do that with the key management system we're just trying to make this as simple as possible we're going to just save this configuration and one of the great things when you're using the console is it will tell you if you errored and occasionally throughout these videos we might intentionally forget something just so you can see how great it is the the guidance that you provided because it can really make life a lot simpler and as you can see our replication configuration settings are up and they're up and running so as we have new data it will be replicated to the new bucket so there you go s3 bucket replication has been set up and now you know how to do it for the future something you definitely want to know especially when you're dealing with s3 hosted websites or when you're looking for additional redundancy in your environment we're going to create an elastic block storage volume and we're going to encrypt it and before we do that i want to do a quick refresher over storage because storage is an essential part of the cloud computing environment so as a reminder we have a few options for storage we have s3 and s3 is object storage and it's a great place to store data or share data between others we have elastic file system volumes which are network volumes it's very similar to the network file system that's typically used on linux and unix systems we have fsx which is the file system that's been designed for windows these are windows servers and they're designed for windows based workloads we have glacier which is really an environment to archive your data and when you're using glacier again remember the data is not immediately available it's going to be there in a period of time it's going to be several hours and if you need your data prior to that you're going to have to pay an expedited retrieval fee we have the option to use storage gateways and what storage gateways do is they sit in your local data center and they enable you to retrieve files from s3 and make it feel like it's standard traditional file storage they also enable you to mount your local servers to the storage gateway appliance sitting in your data center and then it will then take your information and send your information over to s3 and that's a great way to replicate your data to two different environments or to migrate from a traditional data center to the cloud so in this case we're dealing with elastic block storage and elastic block storage is not listed under storage but it's actually listed under the compute services more specifically it's going to be under ec2 so under ec2 what we're going to do we're going to look for where it says elastic block store and we're going to look for the volume area so under volume we have the standard volume that comes with my instance and we're going to create a new volume and i'm going to just choose the standard general purpose ssd we could probably use a cheaper option but we're only going to keep this abs volume up for a little while we're going to pick the size which is 100 gigabytes and then we're going to place the availability zone we could bring it up from a snapshot but in this case we're just going to launch a new volume we're going to encrypt the volume because whenever you're dealing with security you should deal with encryption we're going to click default and we're going to create this volume now realistically speaking we now have a volume and it's now been created and you can see the volume volume's been created and that's really all we have to do now at this point to use it you'll just choose which devices you choose to mount the ebs volume there you go that's how you create an ebs volume and that's how you encrypt it so now let's discuss computing on the aws platform now amazon has what they call a computing instance and they call their computing instances elastic compute cloud or ec2 and what ec2 instances really are is for the most part they're virtual machines and they're sized in the same way you would size any other virtual machine and you're going to size your virtual machines based on cpu cores memory requirements storage requirements and that's in terms of both capacity and performance and then network performance how fast the network access do you need now amazon has a whole lot of different types of virtual machines that are specific for spartan instances i would suggest that you just look on the amazon page because these are updated frequently but you know we have a list and a table for you for you to understand them but they change frequently so at least when it comes to specific computing on the aws platform whether it's a g type instance or an r type instance there's a table in a chart which we'll include and we recommend that you look at that and know them in case you see any questions but in real life you're going to check these on as needed basis so let's talk about giving an operating system to your ec2 instance and ec2 instances support linux and windows and as of a few days ago they actually support mac os which is ron on a mac mini and you have the option to use a pre-built virtual machine or create your own now pre-built virtual machines are available in the form of something called an amazon machine image and this amazon machine image is essentially a fully built machine and an operating system and all you have to do is boot it now the ami has to have a few things it's going to need a storage biome and you're going to specify that i've mentioned before under storage you can have what's called instant storage which is enough to boot the system and run the system but that instant storage completely goes away as soon as the instance is terminated or rebooted or you can use ebs or elastic block storage and with ebs you can choose the performance requirements but it will also not go away upon instance rebooter termination remember if you use instant storage and you upgrade the operating system and cause a reboot the entire drive will be wiped out so don't store anything that you need on instant storage store it on an ebs volume or stored on a network type volume such as an efs share and we'll talk about that more later so while we're working with amis you have to know where to find them because if you're going to build your instance based on an ami where you're going to get it from and you really have four options so amazon publishes amis for pretty much any use case and you can get yours directly from amazon if you want a centos version for example a windows version for example an ubuntu version or an amazon branded linux version it's available for you now you could also buy a pre-built ami from a marketplace and the aws marketplace sells pre-built amis for specific needs and these are typically run by a third-party vendor that's got a specific application and they build this virtual machine to be completely optimized straight out of the box for your application and that's a good option now another way you can get your ami is you can basically take an existing virtual machine that you have running on the aws cloud and you can snapshot that virtual machine create an image of it and from the and from there you basically what you have is you've got an ami which then can be booted or relaunched anywhere in the world on the amazon cloud and lastly let's say you have a traditional data center you've got servers you can basically convert that physical machine to a virtual machine image and then place that on the aws cloud and then boot it up with an ec2 instance so let's talk about the components of an ami so the ami is going to have an operating system it's going to have launch permissions and it's going to have a block device storage mapping so when you set up your ami you're going to tell it boot from instant storage boot from an ebs volume reboot from instant storage but mount these three ebs volumes and that way when it comes up the system knows where the data should be done and what's really interesting and great about an ami is basically you can have a system running on a region in the u.s and then you decide that you want for disaster recovery purposes to basically take a copy of that and copy it to another region in europe or asia for example you can place an identical copy of all your systems in another region and as needed you could just literally boot them right up and if you boot them right up they're going to be completely running and functional so it's a great way to do disaster recovery in a very inexpensive sort of manner so now that you know about ec2 instances and amis let's talk about automating the first boot of an ami so let's say you're taking one of the amazon linuxes or any of your favorite flavors of linux as an as an ami for your first boot when you take that virtual machine there are certain things that you want to do with it in almost every case you want to update that and patch the operating system and whether it's like a pseudo app-get update on an ubuntu type system or whether it's a yum type update on a centos type system no matter what you want to update it to make sure that your system before you apply any of your applications or dependencies is updated to be the most stable and secure version you may also have some applications you want to install maybe you want to install an apache web server for example you may also have some applications that need to be installed and you can do all of these things with a startup script which is called a bootstrap script in the aws cloud and realistically speaking it's a very simple shell script that you would just put as a bootstrap strip for a linux type instance and it's a powershell script that you would use for a windows instance but the key is you're basically taking the operating system and you're going to patch it automatically by your bootscrap bootstrap script as soon as it boots up so now let's talk about purchasing instances on the aws cloud and realistically speaking there's three primary versions and that's on demand instances reserved instances and then spot instances but there's there's two forms of reserved instances and we'll talk about them so let's first begin with an on-demand instance and an on-demand instance is just what it sounds like it's an instance that you create when needed and this is really what makes cloud computing so great is you can have an instance you know it can go up as needed and then it can auto scale meaning you can there's two ways you can scale your instances you can scale them up and you can scale them out scaling up is basically more processor power and more memory in your system scaling out is basically using more systems so one of the best things you can do in a cloud computing environment is you can have these on-demand instances and if the cpu reaches a certain threshold or the memory reaches a certain threshold or disk usage reaches a certain threshold you can have the system open up a new instance and by that you can scale out because overall over time scaling out will give meaning adding more instances will give you much more capacity than scaling up there is a point where no matter what in a single machine you just can't throw enough memory processing power and disk speed but if you are scaling out you definitely can and on demand instances facilitate auto scaling which facilitates scanning out scaling out now i mentioned all the great things with on demand instances you don't need to know exactly how much data you're going to use your compute capacity or how long you're going to need it but with that comes more expensive pricing and you're paying by the second but this is obviously your most expensive of the options because it gives you the most flexibility now we're going to get into reserved instances and what reserved instances really are is you know exactly how much compute power you're going to need you know your memory power that you're going to need you can actually purchase ahead of time a virtual machine or an ec2 instance and you can get a contract for up one year or three years and by having a contract what actually happens is you get a substantially reduced price and the longer the contract the bigger the discount and this is great when you know you have an application that's going to run for three years or one year now what if you have an application that is doing a batch job that you have known computing power requirements that you're going to need every friday through sunday for 24 hours a day while that period is running and it's a critical batch job and you need to know that no matter what under any circumstances the batch job gets completed well you can use what's called the scheduled reserved instance and that's really just a reserved instance where you schedule it for a short period of time or a medium period of time like a few days and by doing that you still get a discount over on-demand pricing but it's guaranteed you own the bandwidth and you're getting a discount now the other main option is something called the spot instance and what a spot instance is is you're purchasing unused capacity from the amazon network and you bid like an auction on this and if your bid is above the spot price just like an auction your bid is filled so let's say you bid on an on-demand instance at a certain price if the price actually rises above your bid price your spot instance will be shut down now there's a few ways you can tweak that but just remember that spot instances shut down if the bid price if the current market price goes above your current bid price so spot instances are a phenomenal way to purchase unused computing power to run any type of a job that actually needs to be performed where you need a lot of computing power but this job has to be able to tolerate being shut down or being stopped and then being rerun later so you don't ever want to use a spot instance for any mission critical applications now there are certain times where you might actually need something that's a little more dedicated to you than the traditional virtual machine and this is say let's say you have an application that's literally tied to physical hardware and it needs or you need access to say i know exactly what the cpu is i know of the hardware i need to know exactly what utilization is of something else then you can purchase a dedicated host which is really just a dedicated server and on this dedicated server you can do realistically speaking anything you want on it because it's your server it'd be no different than any of your servers in your environment so you can install whatever you want in terms of applications operating systems and it's your server now now that you know the instance purchasing options aws has something called tenancy options and what the tenancy options are is really where your machines are actually housed so for the majority of users they're going to have something called shared tenancy and basically with a shared tenancy environment your your virtual machine is on a server but other customers virtual machines are also on that save server and that's fine because they're logically separated they're completely secure but just understand that your systems and somebody else's systems are on the same server and that's fine for normal situations but there might be times where you want to know that you have access to the entire server and for that you can buy a dedicated instance and with a dedicated instance basically all of your virtual machines are going to reside on the same server or the same set of servers if you were to purchase multiple dedicated instances and that means they're all the machines are yours and that's great if you need a little more control and i mentioned previously a dedicated host and a dedicated host is basically just a plain bare metal server it's a server that you're purchasing access to and you can do absolutely anything you want with it and you have full visibility and you have full access to the hardware just like you would any server sitting in your own data center so if you're going to place computing assets on the cloud or even your data center you're going to have to make sure those servers or systems are secure so the way you secure ec2 access is with the security group i'm going to emphasize that again ec2 access is secured via a security group and later on in this course we'll talk about something called the network acl and a network acl keeps unwanted traffic out of the subnet but a security group keeps unwanted traffic out of an ec2 instance or in many cases an aws service but we're talking about ec2 instances right now now the default policy of a security group is deny all traffic so basically if you don't allow any permits all traffic will be denied so whatever your application is if you don't configure the security group to allow what you desire it won't work so remember that if all of a sudden you spin up a new instance and it's not working in its intended way did you put a security group on it and is the security group correct is it allowing the traffic you want and nothing else now when you write a security group policy you're going to write it like you would any type firewall policy or access list type policy and basically what's going to occur is you're going to specify a source and destination subnet you're going to specify a protocol like tcp or udp and you're going to specify a port number say port 80 for example for www or http type traffic now that you understand about security groups now let's talk about providing an iep address to an instance and if an instance is going to communicate on a network it's going to need to have an ip address now something that not everybody understands is every time you have an interface each interface is going to have to be on a different subnet so if you have three interfaces you're going to need three subnets so that's something that's pretty critical we want to make sure that you truly truthfully truly and deeply understand that that all interfaces have to be on a different subnet you can use a private or a public address now private addresses are used for anything that you don't want to be on the public internet that needs higher security public internet public addresses are going to be used that anything is going to be placed on the public internet and if you're going to play something on public internet it's going to need a whole lot more security but for right now understand private addresses are used inside of your vpc public addresses are used outside of your vpc now amazon automatically gives you a fully qualified domain name that matches your instance so for example instead of actually knowing the ip address of the instance they will give you something more clickable that you can just that'll be much easier to reach based on name now we'll talk much more about dns later in this course but understand if you want to go to www.amazon.com you could find out their ip address and put their ip address in your browser or you could just type www.amazon.com and what will happen is you make that request that request will go to a dns server and the dns server will basically say i have the address for www.amazon.com and i'm going to give you that address and therefore you can connect to it now amazon will also give you an ipv6 address but you can disable that but every instance is also going to get its own unique ipv6 address now after you know about securing it and also addressing it how are you going to manage your ec2 instances so you can manage your instances in three realistic ways you can manage it directly from the ec2 console nice browser-based interface very easy to use if you're using a linux-based system you can ssh or you should secure shell to ssh into the machine and manage it the way you would any other virtual machine with a windows system you can use what's called remote desktop protocol and manage it the way you would any other windows machine because you'll have full access to the screen and you can click as necessary and you can also use for certain things like a lot of them but not all of them you can use the software development kit and work through it and manage it through an api interface so let's talk about what we've discussed in this section we've discussed ec2 we've discussed amis we've discussed purchasing options for your specific type of virtual machine meaning do you want like a gpu-paced virtual machine or a memory optimized virtual machine we talked about the tenancy options of what you get like a shared server for example versus a dedicated um host and we also talked about securing your ec2 we talked about addressing your ec2 and then how to manage and access your systems so what we're going to be doing is we're going to be configuring an ec2 compute instance as we taught in the lectures the ec2 is the primary computing platform on the aws cloud so the first thing we're going to do is set up an instance and then we're going to show you how to log into the instance via secure shell or ssh so the first thing we're going to do is we're going to log in and after we log in we'll go to the management console where i'm already out and then under all services under compute we're going to click on ec2 and when we get to ec2 we'll have the ability to launch an instance and we're going to do that right now so the first thing we're going to do is we're going to click launch instance and here we're going to pick an ami and this is going to be a pre-made virtual machine that you're going to just lost from the amazon machine image quite simple we're going to select one we're going to select amazon linux because it's both free and it's a pretty solid form of linux we're going to select that from here we're going to choose our instance type now aws has a variety of instance type based upon your requirements of bandwidth performance and the such memory cpu requirements but in our case this is a lab and we recommend for your own lab you use something that's free tier eligible so we're going to click t2 micro free tier eligible instance now in this case we could just launch it from here or we can go to next configure instance details and while we're not going to really change anything we're going to work with the standard defaults we just want to show you what they are so in this particular case you can pick the number of instances that you want to launch and you could if you wanted to purchase it as a spot instance you'll have that ability we're going to choose the subnet that you want to be on which is part of the network and we can talk about capacity reservation if we need it would we like to start two or three or four instances and here under shutdown behavior we have the opportunity to stop or terminate if we terminate an instance anything that's on instant storage will be completely lost so that may be appropriate under certain circumstances but it may not be and that's why we talk about using ebs volumes as opposed to instant storage for anything that you need to persist beyond the reboot and if you're looking to prevent that kind of termination and instance lost you can click protect against accidental termination when it comes to monitoring we have cloud watch monitoring which begins by default but we can also enable detailed monitoring and we'll talk about that much later during the course of the program and then the tenancy this gives you the option to run it as a shared instance or in different environments and we'll talk much more about that later so in this particular case we're just going to go here and look at some of the advanced details you can place some metadata which is really data about data and under here if you desired you can play some startup scripts or other configuration options but we're not going to really talk about that for right now and then we're going to just review and launch the instance and what you see is it's going to tell us that the security group that the configured one is open to the world and what they mean by that is by default it enables ssh from anyone to access these instances now in reality you only want to allow ssh from the subnets that require access to these instances but in our case it's a simple first lab we're going to just do this with the default security group and then we're going to click launch and from here what's going to happen in order to be able to access these things via a secure shell you're going to need a key pair now in our case you can choose an existing one create a new one or proceed without a key pair you want to use a key pair because it's going to be much more secure i'm going to choose an existing key pair and we're going to launch the instances of course you have to acknowledge this first so our instance is launching it's going to take some time for the instance to boot and here you can see that the instance is pending you can also see that i've had another instance that i've already terminated but in this case our instance is pending meaning it's coming up it's booting so we'll give that a few minutes and we'll keep checking these boot faster than you can see there you go it's already up and running so we're going to click on the instance id and under the instance id we can see the instances private i p address we can also see its public ip addresses what we're going to use to access the instance and you can see that it has a publicly available dns name so we can address this by the dns name we can connect to this name without actually having to connect to the ip address and this is going to be much easier to remember the dns name than the actual ip address now you can see more information about our instance here you can see the details of the instance you can see the security group that's actually running you can get your information regarding the network on this place where it says storage you can actually see what we're using in terms of storage and what the storage device is actually listed as under monitoring you can see what's going on with your instance you can look at the cpu you can look for system failures you can look where network is going and disk reads in and out that's your typical basic cloud watch options there are other options for that you can do for an additional price but these come with it and then if you wanted to tag your instance or add information about your instance you could do that here so imagine you have 10 000 servers if you don't have some kind of information about your instances it's going to be hard for you to remember them but in our particular case we're going to just work with the standard so from here what you would do is you've downloaded those those key pairs now you're going to have to download those key pairs and you're going to have to place them in a directory on your computer and you're going to want to ssh from the directory where you store your key power if you're using a linux system a unix system or a mac now if you're using windows you're going to have to use an ssh client like putty which will be much easier to set up there'll be a place to just import your key power but we're going to do this on a mac because i use a mac and most tech people and many cases are going to be coming from a linux machine or a mac or a unix machine for this purpose so we're going to show you this way so i had already downloaded my key pair and i'll bring my key pair over here for a second and what we'll do is i know where my key pair is located but if you don't know where you're at first go to your pwd will take you to your present working directory and in this case i'm under user slash my gibbs now if i look at my files you can see that i created a folder called aws where i stored my key pairs keep your files in a folder that's going to be very secure so in order to ssh into the system i'm going to go to the directory where my key pair is now if we look very carefully you'll be able to see there my key pairs there but we have to make sure we have permissions we want to use the least number of permissions necessary to get our job done without enabling too much permission so what we're going to do is we're going to do a change mod which enables us to change the permissions and in this case we're going to use 400 or 400 and then we're going to need we're going to actually enter the name press enter and now we have the permissions so now we want to connect to the instance all we have to do is ssh into the instance so we're going to see the address obviously once you do this a lot of times you're going to know what to do but in the case where you don't aws actually works with you with your key power and the dns and they tell you exactly what to type so take this copy this control c command c whatever you're using if you're on a mac versus a windows instance and then come back here and then we're going to have to go back to our terminal window we're in my terminal window brought it back we're going to take the ssh information directly from aws and we're going to click enter now it's going to ask if i want a fingerprint anytime you ssh basically there's a there's like an rsa type shot 256 fingerprint that actually comes through and you're going to have to say yes if you want to be able to connect to this because this is how the authentication and the encryption between your device and the instance is going to occur click yes and as you can see i am now in my ec2 instance and if i were to list ls to see what's in the folders if i want to look at my ip address i can do that now something that's very interesting if you look at the ip address in this video notice that it's a private i p address 172.31.40.43 that is a private ip address now how am i connecting to a private i p address through the internet and the answer is i am not i am connecting to the public i p address that was automatically assigned this ec2 computing instance does not have any knowledge of that public ip address so what's going on is amazon is nodding this private address into a public address and that's what i'm using to connect and over here we could do anything we needed like if we wanted to do a yum update which would update the operating system we could do that and in many cases if you don't have permissions it's going to force you to do a pseudo or super user and as you can see i've been given the ability to do this my permissions are good and all i have to type is y and all of a sudden my ec2 instance is updating itself completely so there you have it that's all that's necessary to set up an ec2 instance and actually secure shell and remote access it we're going to be setting up an ec2 instance and we're going to be setting up a bootstrap script and in this particular bootscrap strip we're going to set it up that it's going to actually upgrade the operating system to the newest packages install the apache web server and we're going to give it a mini mini mini web server um setup so let's go to our ec2 instance and let's launch an instance now we've been using for the last few amazon linux which is a form of linux i really like it's based on the red hat type distributions and it uses the typical yum update functionalities that you would get basically on something like a red header or a centos but in this case let's use ubuntu server just showing you that we can use multiple server types again we're going to configure this for the free tier let's configure the instance details now under the advanced what we're going to actually do is we're going to show we're going to set up a mini shell script and we're going to make sure that we're we're using the appropriate shell so we're going to do pound bang slash bin slash bash so it's a bash script and then when you're working with ubuntu or any of the debians it's going to be an app-get for an update or an upgrade so apt-get update now obviously this would be great to template this you can do the same thing with all of your servers but in this case we're building a lab and we're just building it on the fly especially since i'm recording on a window and it's very hard for me to go to multiple windows without taking the screen off of you what you could see so now let's get let's install the apache key.get install we want to make sure we say yes apac hp 2. and then what we want to do here is we're going to create our web page we're going to do echo very basic website and we're going to say welcome to go cloud architects and we're going to take that we just echoed and we're going to send it to a main web page file called index.html which will be the root of our website and then we're going to copy index dot html to our directory where we actually store the website which is going to be slash var slash www slash html so let's make sure we didn't make any errors i always like to double check app dash get update get install apache 2 echo the web server to form our web page and then we're going to copy the index to the appropriate location and that's it let's just review and launch and we're using the default security group and it's warning us that you typically would not want to use this in reality you'd want to use something that's a little more sophisticated we're going to choose an existing pair there you have it we've set up an instance now the instance is pending so let's wait and see the instance come up we'll give it a minute because not only we ask the instance to come up but we've also asked it to do some upgrades along the way so it looks like our instance is up and running and that's really all that's necessary to set this up so in this section we're going to be discussing databases databases have become an absolute essential tool in today's business environment and they're a mission critical application for which businesses depend many of their applications but also their decision making so databases are really critical so let's talk about what is a database and a database is really a place where you're going to store large amounts of information so you'll store large amounts of information in a database but a database is different than standard storage where you could also store large amounts of information a database facilitates the sharing the sorting the calculating and the reporting of information gleaned from your data so because of the ability to draw such good conclusions and see the relationships between different objects at least within a relational databases businesses have become absolutely dependent upon databases so aws has three primary forms of databases and they all have specific applications aws supports a relational database no sql databases and a data warehousing database and we'll talk about data warehousing and data lakes to some degree later in this course but let's first look at relational databases now relational databases are by far the most common forms of databases that are actually used by businesses currently and what's so interesting about relational databases is they provide a relationship between variables so you can see what happens if one variable is changed how it will relate to something else which gives you great business information and the way they're stored they're stored in a manner that's just like a spreadsheet with regards to rows and column and each row is going to have a unique id which is called the key and each column is going to have something called the value and that's where in relational databases we get this concept of a key value pair now when we're dealing with relational databases they may operate slightly differently than nosql databases at least with regards to the consistency of data so we like to say that relational databases all follow the acid model and the acid model means they're atomic meaning it either works or it doesn't work the reads and writes are consistent and what i mean by that is if i write information to a database every other user that's accessing that database will get the newest information the second or even the millisecond that i write it to that database it's not going to take time to become consistent it's immediately consistent it's isolated meaning one component of the database will not affect another component of the database and the data is durable meaning once it's there it's not going anywhere now aws supports a variety of relational databases they support amazon aurora which is the amazon branded relational database very common database used by many aws customers they support mariadb they support microsoft sql server they support mysql they support oracle's database and they support the postgres database and we'll talk a little bit more about each of these databases and their support on the aws platform so now let's explore the different types of relational databases available on the aws cloud and we're going to begin with amazon aurora which is going to be an exceptional option for most people the reason we like it is amazon aurora takes the feature set and functionality and performance of big commercial databases and they give it to you effectively for free so it's kind of a cross between an open source database and a big commercial database so the performance is great in fact it's up to five times faster than mysql and it's up to three times faster than postgres according to aws documentation and we've seen very good performance in this and this is typically used in enterprise applications or software as a service applications very high performance and great pricing structure now the next uh relational database that's available in the aws platform is mysql database mysql is an open source database and it's been around for a very long time it's an incredibly popular database and it's used by a wide variety of web applications but there's some scalability features or challenges with mysql and that's why some of these other databases were formed and postgres is another type of relational database and it's another open source database like mysql but it has a very sophisticated feature set and functionality compared to mysql and the next version that you can use is actually going to be called mariadb and mariadb is another open source relational database and that's actually created by the people that actually made mysql but it's a newer version designed to address many of the scalability and the problematic shortcomings of mysql but this has a very good feature set and it's really designed for enterprise applications and it's open source so it's free so let's move on to some commercially available databases on the aws platform we'll begin with microsoft sql databases and microsoft sql database is a pay database and is typically used for microsoft specific workloads and it's got a lot of tools like the server management studio which enable organizations to use it quite easy compared to other forms of databases and aws supports the main licensing types and it supports most of the versions we'll show you what the versions are in text but understand that the microsoft database is a paid database and it's available for you if you have microsoft specific workloads placed on the aws cloud now the next thing we'll talk about is oracle database and the oracle database is the most common paid relational database in the world and it's got an incredible feature set and an incredible functionality and there's realistically speaking two ways that you can use oracle databases on the cloud you can use the aws type licenses which is the the standard version the standard one or the enterprise version and these are available what happens is you aws has these licensed and you're actually borrowing their license when you use them and that's completely appropriate completely legal and that's what most organizations do having said that if you need more functionality you can actually do this bring your own license version with you of your oracle database and with this model where you bring your own license as opposed to the license included with the previous method that i described you have access to every one of the oracle database options to your disposal so the next type of database we're going to be discussing are something called no sql databases and nosql means not only sql and no sql databases provide enormous flexibility when compared to sql databases because they allow for much more flexibility in their schema structure which is one it's going to make no sql databases scale much better with much more capabilities than an sql database so there's going to be applications where you're going to use an sql type database or a relational database and there's going to be instances where you're going to use a nosql database what's great about nosql databases they work with both structured and non-structured data and that flexibility really helps a lot one thing to know as i had mentioned previously that sql and relational databases followed an acid model but here we're going to follow a base model and we're not talking about ph what we're talking about is the model is they're basically available soft state and eventually consistent and what all of that really means is that if i were to make a write to the database instantly at the time i make a write to the database someone reading from that database may not get the newest information it may take a few seconds for everything to become consistent and with that comes a lot more flexibility and scalability but with that also comes the ability to get stale information for a very short period of time now no sql databases still typically work with a keypal uver environment and you can see we've drawn that out for you so you understand at least what a key value pair type environment is but the things to remember are they're very flexible and because they are not bound to the same exact schema that you'd find in an sql database or a relational database they can scale much more now aws has their own branded form of a nosql database called dynamodb so let's dive into that so now let's discuss dynamodb which is the aws version of a nosql database now dynamodb is a fully managed service by aws and it's very high ability and by default it's in multiple availability zones and one of the great things about dynamodb is it's completely serverless and since it's serverless you don't have any compute instances to manage no operating systems to patch and a lot of the things that go into managing a compute instance are completely done by aws but furthermore it will scale to almost an unlimited matter because it's serverless so normally with the server you run out of cores you add some additional instances or you can use a bigger server but in this case it's almost unlimited scalability and dynamodb is also very latency because it stores all its information on very high performance ssd drives now one other thing that's pretty incredible about dynamodb is the latency and it's based because the type of drives but also the way the database is architected and it's typically low millisecond latency but you can actually set it up by using dynamodb accelerator to be sub millisecond latency and all data is encrypted by default you don't have to worry about your data being protected and it can be backed up with little to no effect on database performance and some things to remember about dynamodb is it's typically designed to work with a key value pair environment now you can have a primary index or a secondary index and that's one of the reasons you can make this scale so well when you purchase dynamodb you actually provision your capacity ahead of time just like a provisioned iops volume you're going to provision your dynamodb ahead of time so realistically speaking you need to know what your requirements are prior to using dynamodb now you could set up dynamodb with auto scaling and that's a totally good option and it will scale up based upon needs but it doesn't scale back down so if it scales up temporarily you're going to have that bigger instance and be charged for that bigger instance all the time so you really want to know what you're doing so you can manually provision this ahead of time so you have the read and write capacity and the performance capacity you need without relying on auto scaling now the price of this is going to be based on throughput so the way they actually work dynamodb it's priced on throughput so that typically occurs with how much data is moving in and out of the database i just want you to understand that so where would you use db you're going to use this in a wide variety of use cases you're going to use this when you need for the most part unlimited scalability and unlimited performance you're going to use this if you have extremely low latency requirements and if you've got a tremendous number of devices think internet of things think having millions of sensors constantly reporting every few seconds and sending it to a database perfect use case for this so this is ideal for storing say game state it's it's really ideal for any time you have a large number of transactions and you need to move it quickly lots of e-commerce applications lots of financial applications where you're just storing a tremendous amount of data now i mentioned previously that no sql databases and dynamodb is a nosql database are eventually consistent and when i talked about the acid version versus the base version i said that they're eventually consistent meaning if you make a right the read may not be immediately available off the right now that is the default configuration you can change that if you need to be instantly consistent or immediately consistent but with that comes a performance and a scalability cost but it's definitely an option should you need that in your business so let's talk about data warehousing databases and this is a big data application and the data warehouse is really a way to in a very structured manner and the searchable matter store very large amounts of data and it's going to be a combination of using a database it's going to use a company it's going to use s3 it's going to use some tools to try and take some of the data from s3 and put it in the database and then the reason you're doing all this is to gain actionable insights from your data because if you've got all this data the data can help you make inferences on for business decisions but if you don't analyze the data or use the data then you won't get there and amazon has a tool called quicksite that helps you visualize the data that you've gleaned from redshift as well as your your entire data warehouse so what is redshift and redshift is the amazon data warehouse solution and it's a fast powerful fully managed service and it's based on the postgres database which is good because it enables you to do sql queries on your data so it can use any platform they can do sql queries can actually work with amazon redshift and the way it's built it's built around a architecture of computing clusters and the clusters are all of computing nodes so what's going to happen is you're going to have this big cluster and inside of there they're going to be these little mini computing nodes and the primary code node is called the leader node and all the other nodes are compute nodes which are going to support the leader node when you run a query it's going to be directed to the leader whereas the other devices are doing much more processing so when you're dealing with this data warehouse you're going to achieve scalability or increase performance by scaling out by adding additional nodes and realistically speaking the amazon redshift has two types of nodes that are available and you've got the option for dense compute or dense storage now dense compute stores their information on high speed ssd arrays where dense storage means you've got a lot of stuff to store and you're looking for a more cost effective way to store it you're going to store that on magnetic disk arrays so the next big data application that we're going to be talking about is something called the data lake and a data lake is really a repository that allows you to store a tremendous amount of structured and unstructured data in the same place at virtually any skill and because you can put so much data that doesn't have to be structured it enables you to have access to a tremendous amount of data to make better decisions so the reason you're doing this is you're storing your data and you're going to store it in a lot of different forms i mean you're going to have some data that's going to be stored in data storage say for example s3 you'll probably be integrated with both redshift for big data like your data warehouse you probably have a dynamodb you might also have what do you call it a relational database and you're going to have tools that are going to extract and transform this data but the point being is data lakes hold a tremendous amount of data and they keep them there until you need them so you'll be able to do queries on them to search for relevant information when you need them or you can save this data and use it for future applications such as machine learning but the point being is it's a great place to store structured and unstructured data lots of data and keep it available for a future date when you're going to need it so in this section we're going to be discussing how to optimize databases and improve their performance and one of the things we need to start with is storage so amazon databases are stored on ebs volumes now these are basically the same ebs volumes that we talked about when we talked about storage so let's talk about the three ebs volume types that you would use on your databases provision iops and you're going to use this this is the fastest and best performing option on the aws cloud and you're going to use this when you need low latency high performance storage so for any type of database that you need to really scale and you need low latency it's going to be on a provisioned iops volume now your next option is going to be a general purpose ssd and this is a medium latency medium throughput environment and it could be used for databases in a lab environment or it could be used on databases that are not critically latency sensitive and the last option you have is going to be for magnetic storage which is great for storing large amounts of data it's low cost but it's much higher latency so do not use this for latency sensitive applications so i mentioned databases are an absolutely mission-critical application in modern business and anytime you have data and then organization is dependent upon that data you need to maintain that data under any circumstances so we need to back up the data that's on all of our databases now the good news is aws does this automatically for your databases and when it's backed up the database backups which occur automatically it backs up the entire instance it doesn't just back up your data it backs up the operating system it backs up the dependencies it backs up your database it backs up your configuration and all of your data so effectively what's happening is you're going to have a snapshot that's automatically created which is effectively the same as a machine image to to relaunch the instance from a snapshot so you're getting everything now when these automated backups occur you have to understand that it's actually taking resources away from the database so it's possible during the short period of time that a database is being backed up performance is either degraded or even the database is somewhat unavailable for a very short period of time now i mentioned that databases are backed up in the form of a snapshot and you can also perform a manual snapshot so you can take the automated ones or you could snapshot them manually and because it's a snapshot of the entire ebs volume as i mentioned before it's a complete copy of the server and when you do a database snapshot on a manual one they're going to be maintained until you delete them as opposed to an automated database backup which is going to be there from 1 to 35 days based upon configuration a manual backup will be there until you delete them so it's going to look very simply you're going to see the database it's basically going to make a an image of the database and it's going to store it now if you want to restore the database again it's pretty easy you're going to take your database snapshot file you're going to open it up and a new instance is going to be formed now because the new instance is going to be formed it changes a few things all of your information is identical configurations are identical and everything will be the same about the server but it's going to have a new ip address because it's a new instance and with that ip address is going to come a new dns name as well so if you have anything that's connecting to it based upon its dns name or ip address you may need to change that because even though the server is configured identically it's going to have a new ip address and a new dns name because a new instance has been formed so i mentioned databases are a place for organizations to store an enormous amount of data so if an organization is going to store a lot of their important data they need to make sure that that data is secure and one of the ways you optimize your database is with database encryption now relatively speaking there's two ways to do this we're going to start with the primary way which is complete database encryption and that's the way that's going to occur is you're going to create something called encryption at rest which means when the data is stored it's going to be encrypted and the way you'll do this is you'll actually encrypt the ebs volume for which the entire database is stored and with that everything on the on the volume will be completely encrypted you enable that by enabling the key management service and the key management service will automatically take care of the encryption keys and encrypt the item now the other type of encryption that's supported on the aws cloud is something that's called transparent data encryption and transparent data encryption is widely used in oracle databases and microsoft sql server databases and what transparent data encryption really is is the data is encrypted and decrypted on demand so when you write information to the database and it's stored it's encrypted when you pull information from the database it's going to be decrypted so that's a way to basically do encryption on the fly and that's called transparent data encryption so as we're discussing databases we have to discuss scalability since the database is such a mission critical application it's going to grow and at some point no matter what you do it's going to struggle to keep up with demand but there's a lot of tricks that we can do to increase the performance and the scalability to make sure the database can meet the business and customer requirements so there's two means of scaling things out or up one is scaling them up and this is basically putting the database on the largest system you can with the most processing power the most memory and the fastest drives that will work to a point but no matter what you do at some point you're going to exceed the capabilities of the most powerful servers in the world so then what you're going to have to do is you're going to have to scale out and basically scaling out is basically just adding additional compute instances and we're going to talk about how to scale out different types of databases so we'll start with the nosql databases like dynamodb or cassandra now the good news is scaling these things out is very simple all you have to do is partition the database when you partition the database it chops it into multiple more manageable pieces called partitions or shards and the application will have the intelligence to route the correct information to the appropriate shard and by doing this it's going to scale well so that's very easy for the nosql databases like dynamodb now relational databases scaling them out there's a there's a lot of things you can do and a lot of performance optimizations but we're going to start with scaling out and then we'll talk about queuing and caching and things like that so to scale out a relational database you're going to create something called a read replica and a read replica is effectively with the exception of maria db an identical copy of a database instance but there's a difference between the master database and the read replicas the master databases do read write activity and the read replicas do read activity and they're meant to be a read only copy of the database and it's going to get synchronized and effectively near real time to the master database and what this does is by having multiple read replicas you can point your queries to the read replicas and keep and therefore you're not affecting the cpu of the master database so you're going to scale out your your databases with up to say four read replicas and as you scale them out you can increase database performance so when would you use a read replica you're going to use it when there's a lot of read activity if all your database activity is right and you add read replicas it's not going to make much of a difference because it's read activity but what will happen is query traffic is going to slow things down at some point if you've got a lot of read and write activity and by taking all the reads off of the maester database you can substantially reduce the overhead on the master database so anytime you need extra capacity extra performance and you're using a relational database the first thing you're going to do is you're going to add a read replica now i want to touch on one topic before we get off of read replicas rate replicas are not for disaster recovery read replicas are strictly performance so if you see an exam question that's related to read replicas and disaster recovery this is not a solution this is a performance option when it comes to disaster recovery and it comes to high availability we'll actually talk about multi-az and we'll cover that in much more depth when the time comes the next way we can increase the performance of our databases is through database caching so let's describe caching in the following manner i previously mentioned that you can reduce read activity from the master database by adding a read replica but at some point if you direct enough traffic to the read replicas they're still going to be too busy so what can you do to offload traffic to the read replicas imagine a situation where you already had the information for a lot of the content and didn't need to go to the read replicas so then your master database with your four read replicas could be the equivalent of say 10 12 30 based on how frequently you access new data and that's where database caching comes into play database caching is basically a server that sits between you and the database you've got this caching server and here's what happens i'm a user i run an sql query on my database or any type of a query on my database i first go to the cache the cache does not have the results so then it goes to the server the server then responds to me through the cache the cache stores the data for a period of time and then the data is sent back to me now let's say my associate at work 10 minutes later needs the same data he does the same query that i do but in this case he goes to the cash first and the cash says wow i have this information and sends it right back to the person so not only did the person get it faster because it was cached but it didn't have to go to the database server and now let's say the next person requests the same information again they're going to go they're going to go they're going to reach out to the server but they're going to make a stop at the cache the cache is going to say i have this information and send it so that's how caching works caching takes frequently accessed information stores it on the cache so it doesn't need to go to the read replicas or whatever type of database device you're using whether you're using an sql or no sql database by doing this it can promote substantial reduction of read activity so you're going to use caching to you to reduce read activity so i'm going to say that again because you may see that you're going to use caching to decrease read activity and there is an aws caching service and that's called for for databases and that's called elasticash they're going to be two versions of elastic cache you're going to have elasticast for reddish and elastic d now elastic memcache d is a fully functional but very simple cache and it's designed for simplicity now elasticash for redis is a very high performance very features set of caching capabilities if you need that so again caching is going to work by offloading read activity to your database servers and one last thing that we should talk about while we're discussing caches is cache timeout so i previously gave you an example where i pulled information from the database that got stored on the cache and then two of my co-workers ran a similar query and on the way to the database they got their information from the cache and it got brought back and it's perfect but let's say three hours later someone actually updates that record on the database now the cache still has the old information and what's going to happen anytime you use a caching service of any kind in any environment is you're going to have to have something called the cash timeout and what that actually is is the cash will age out or nullify the information stored in the cache after a period of time and you're doing this because you don't want old or stale data any time you use caching you run the risk of having some stale data and that may actually be okay an organization will say okay my data can be stale for six hours or three hours or ten minutes or whatever it needs to be based on the effectiveness of the information and how that information could either help or hurt the organization but they're gonna have a cash timeout now the cash time out or how fast the cache actually ages out and gets rid of the data or purges the data is going to be configurable so we've discussed using database caching to reduce read activity but now let's talk about reducing write activity and we're going to be using database queuing to reduce the effect of write activity on a database so what is queuing healing is simply a means to schedule the data that's delivered and without a queueing service what's going to happen is the data is going to come in and be written as fast as possible directly to the database but by placing a queue we can place a man in the middle to effectively help out so what's going to happen is we're going to take our data we're going to send it to the queue and then the queue is going to find when it's appropriate to deliver a message to the database and by doing that we can make sure that messages don't get lost we can increase the availability of the application because for example what if the application were to go the database were to go down but the queue is still up the messages can still be sent to the queue and when the database comes back up the message could be drained from the queue and placed in the database and killing is used with a tremendous number of applications killing effectively decouples the components of your application and you may see that as a test question somewhere along the line when you take you put in a queue to decouple the applications components so in this case it's decoupling the traffic destined for the database to the actual database now amazon has their own killing system so you don't have to build your own and it's called sqs which we'll cover in more depth in the section where we talk about amazon specific features but we're discussing databases so we need to address it right now and when it comes to queuing you have two options now you have the standard queue and the standard cube basically takes the messages in and takes the messages out and delivers them as fast as possible now smaller messages might get delivered before bigger messages it's very possible with an amazon sqs standard queue that your messages become delivered out of order and if your application can tolerate that that's terrific because it's highly scalable now if you need your messages to be delivered in a specific order you're going to have to go from the standard queue to something called the fifo queue or first in and first out queue and when you do when you set this up you can guarantee that the first message that comes in is the first message goes out the second message comes in is the second message that goes out and this is great but realize if the first message is is still on the way out you may have 30 other messages before behind them that could be delivered quicker with a standard queue than with a fifo queue so you have to determine the right cue to use based upon your application's requirements so when should you use sqs you're going to use it to reduce write contention on the database you're going to use it to keep messages from being lost and you want when you want extra assurance that no no no message is going to be lost and you want to increase the availability of the apple and the performance of the application so sqs is an ideal means to increase database performance by decreasing the effectiveness of the write activity even though the number of writes that would be going to the database will be the same but they'll be spread out over time so they're going to smooth disk performance they're going to it's going to smooth cpu performance and it's going to increase scalability as we've discussed databases are mission critical business applications so if they're so important we need to make sure that they're available and when we talk about high availability what we're talking about is making sure that the application is available when you need it so how do we design an aws database for high availability and simply the answer is we're going to put that database in multiple availability zones so as a quick reminder aws divides their network into both regions which are large geographic areas and availability zones which are basically a data center inside of one of those regions so by placing your servers or your database servers or your database in multiple availability zones even if one availability zone were to lose power and network connectivity or and your server were to crash all at the same time no problem because the other availability zone will take over so some things to remember when you're using multiple availability zones it does not increase performance at all in fact what's going on is your master database and your primary availability zone is going to do everything and it's going to synchronously copy everything that's in the the master database to all the other availability zones so it's not a means to improve performance but it is a means to prove improve availability so i i typically like to saying one is none and two is one you always have to have at least two things of anything that matters and that's when we're referring to high availability so let's talk a little bit about more about if you're using this high availability design when will it fail over and it's going to fail under over the following circumstances if your primary database fails well then the backups in the multi-az environment will take over but if uh an availability zone were to lose connectivity at least the one where the master's house it's the backups one of the backups will also take over so if you change anything on the database instance and make it temporarily unavailable the backups will actually take over if you patch a system and do a reboot especially if the reboot's implemented with failover it will fail over to another availability zone so there you know how we create high availability database designs in aws we basically put them in multiple availability zones so you have a copy running in multiple data centers in this lab what we're going to be doing is configuring a simple database and we're going to start out from the management console and we're going to go to the database section now you've got multiple options here you've got your relational databases which are rds you have dynamodb you have elastic cache which is a caching service and then you have neptune and then you have a few other sets of documentation and things like that and ways that you can optimize your database so let's go create a relational database so we clicked on rds and from here we're going to click create a database you could pick it in one of two places now there's really a couple ways you can do this you can pick a standard create and specify all the options which is totally appropriate or you can use easy create now depending upon your database knowledge and depending upon what's necessary for your world and your environment it may be best to use easy create especially because it's going to be configured with best practices so unless you're an expert on databases this is our recommendation now when you're dealing with some dealing with some complicated web applications you may have to use a standard crate so and then specify every option so you have to choose what's best for you so we're going to pick a mysql database just because it's the most common one that's out there in terms of scalability there's many better options but we're going to do something it's a lab environment we're going to pick a free tier so then we're going to go to easy create and from here it's going to ask us you know is this something that's going to be in a production environment and you can see what's in from the information that you're getting with regards to ec2 instances and the cost and then they'll give you something for the dev test environment but for what we're doing right now we're just going to be using the free tier now it's going to give you the database i identifiers you'll be able to know what it is you can logically tell this what you want it to be it's going to give you a master username and then you're going to have to come up with a password so typically speaking the auto generation of a password is the best way to go because it's going to give you something that's quite good and we're going to create our database and you can see the provisioning is occurring as we're speaking so let's see what's going on let's see what it's telling us so you can see we've got the database identifier the way we set it up we see see that you have the i am role which is based on the instance it's going to be mysql we can see the size of what we're doing and which vpc is located and no we have not set this up as a multi multi-uh az database so let's look a little more into it as we're exploring we can see that it's still coming up and that's why there's really nothing to monitor yet at this point because it's still coming up but it will be in a moment or so but you can see you'll have some logging functionality that you could set up with cloudwatch you can see that it'll give you very good information under the configuration section of what you've actually set up and under maintenance and backups you can set up ebs snapshots and you can take snapshots well of your database with a db snapshot which is very similar to an ebs snapshot because it's going to be on an ebs file and then tags would be things that you specify so that's pretty much it that's all that's really necessary to set up a database you can see it's creating right now it'll take a few minutes and then the database will be up in this lab what we're going to do is we're going to be creating a read replica so recall from the lectures that a database only has a limited capacity and that includes both read and write because they both affect the cpu and they affect the the the hard drive speed and access because you can only have so many input output operations per second no matter what you do you only have so much memory and processing power so any way you can offload the primary database you can make the database scale and you make relational databases scale by reducing read activity from the primary database with the creation of read replicas so in this lab we're going to create a read replica so in the previous lab we previously created a mysql database and now we're going to take that database and we're going to create a read replica so from the management console we're going to click on databases and we're going to click rds or relational database services and what will ultimately happen is we're going to have a dashboard and we can see what databases we actually have so let's click on our databases and you can see this is the database we created before the database one so from here you can see that the status is available you can see what availability zone it's in the subnets it's associated with and uh how to reach it with both the endpoint as well as the port numbers so now what we're going to do to create a replica is actually quite simple we're going to go to the actions and click create read replica now the read replica has been the process of being provisioned and it's going to be the database source because this is going to be the primary database and this gives us an option for a database identifier and then we'll have the region we're going to pick the instant size which is going to be the same as our last one and then the storage type we're just going to use general purpose ssd and we're going to enable automatic storage auto scaling what happens there is if your database is getting full it will just increase the drive space you never have to worry about things and availability and durability you know do we create a standby instance in a multi-ac sort of environment and for high availability you're definitely going to do so this is just a lab environment so we're going to keep the cost as low as possible and we're going to move forward with everything else in the current conditions that they are click create read replica and it needs a it needs a name and that's totally fine and what's great about the console is it reminds you of things that you might not play so we're going to call this for read replica now obviously you're going to pick a name that's going to be descriptive and do something for you but in this case i'm looking for something that's quite simple because i want to know what they are because it's a lab environment and i want to remember how to delete them when we're done so we click read replica now you can see what's going on you can see that under our databases we have the primary database which is what we set up before which is being modified right now to let it know that there's going to be a read replica and you can see what's going on with the cpu it's not quite busy and you can see that our read replica is now being created and realistically speaking that's about all it's going to take to set up both an rds database and and set up a read replica we're going to do is we're going to be taking the read replica that we created in a previous lab and we're going to promote that to the master database now there will be times where you're going to want to do this for example let's say your master database were to fail you could take a pr read replica and promote it to become the master so we're going to start by going to the management console and then we're going to go to the database section and click rds and as soon as this comes up i'm going to show you the databases that we previously created remember in the first lab where we created a mysql database which is the primary and then we created a read replica so let's take the read replica and promote it to be the master database so we're going to click the read replica and then we're going to go to actions and we're going to take promote yes we definitely want automatic backups because we want to make sure that things are good we're going to keep the backup retention period for one day for right now since we're doing automatic backups and we're going to select no preference for the backup window now ideally the backup window is going to occur at a time when your organization is not very busy because during the backup period the system may have severely degraded performance or even be unavailable to take new reads or writes for a very short period of time so ideally you can do the backup window at a time where it's not going to be used heavily by the organization so we're going to click continue they're going to ask you are you sure and you're being promoted so we're going to watch and see if the replica has become promoted okay now you can see what's going on we have to give it some period of time you can see that the databases are changing you can see that the status is being modified so in a few moments what you'll have is a promoted read replica to be the primary master database now that's really all that's necessary to promote a read replica to a primary database we're going to be doing is setting up a dynamodb table if you recall from the lectures dynamodb is an extraordinarily scalable serverless database and it's a nosql database and nosql databases store data in tables and the tables have more flexibility than you would with a relational database so we're going to create our dynamodb setup and a table so we're going to start with the management console and under the management console if we go to databases we can choose dynamodb so we're going to create our dynamodb table and we have to come up with a table name so in this case we're going to pick something simple we're going to call it product catalog and you can see that you have to be careful your spell check might actually fix things for you and in the process of your spell check fixing things for you you can find yourself in a position where um it won't accept things and we're going to create a primary key and we can set this to realistically speaking anything we need it to be so in this case we're going to set it that we're going to set the primary key to to an id and then for that we're going to set the number now we're going to click create and as you can see our dynamodb table is being set up and create it that's actually if you look it's pretty fast you can see that the system is up and it's running we'll be able to look and see what's in this so you can so we don't have anything stored yet you can see what's going on with regards to read write capacity in the system well you'll get your alarms or alerts based on what's going on with regards to the capacity you have to provision your read and write capacity now it could be done on demand and if your read write capacity is done on demand it will grow as we discussed in the lectures but it will not shrink on its own so realistically speaking you want to going to want to provision your read and write capacity units to be as much as you need ahead of time and then you'll be able to look at your indexes and see what's going on or the types of tables that are actually there you'll be able to see backups that are there and really that's all that's necessary for you to create a dynamodb table in this section we're going to be describing the aws virtual private cloud so to begin what is a virtual private cloud a virtual private cloud is a logically separated private network on the aws shared network so what that really means is when you're using a cloud computing environment you're using their network but you don't want amazon for example to be able to see your data so you need something private on the amazon network but not only do not want amazon to see your data you don't want other customers that are sitting on the aws network to also be able to see your data so the vpc is a virtual private cloud and it's effectively your own private network on a shared network but they're logically separated they're logically isolated so you don't have any security concerns on the vpc it is effectively your space through the way it's logically separated now because it's logically separated you can conceivably using be using overlapping ip address space with other clients in the virtual private cloud and in many cases you're going to be because most organizations are going to be using the address space from the rfc 1918 which describes private i p address spaces and internally most organizations are going to be using similar ip address space now externally if you have to connect to the internet you're going to need a globally routable unique ip address or a public address but inside of an organization you're going to use a private ip address and with the amazon vpc addressing is a key component of your vpc now let's talk a little bit about you know what needs an ip address every actual network interface on an ec2 instance is going to need not only an ip address but each interface is going to have to be on a separate subnet so we'll talk a lot more about ip addressing and subnetting in inside of this section but at least for right now it needs to be understood that the vpc is a private network and because it's a private network it's logically separated from everyone else every system on your network will need an ip address and every ip address on your network is going to need to be unique and it works like this let's say for example you live in a home and you want your mail to be delivered the mail gets delivered to you based upon your actual address which is going to include a number a street name possibly an apartment number a state a city and a postal code and through all of that the post office knows how to deliver your mail to your house versus your neighbor's house or at least hopefully now the same thing happens in the ip addressing world the network will deliver the messages to the appropriate destination and this occurs by the ip address therefore every device inside of your network needs to have a unique ip address and in the event an organization were to purchase another organization using the same addresses they'll either need to re-address every one of them or they're going to have to translate addresses through something called nat and we'll talk about nat or at least other applications of net when we're talking about connecting to the internet so when we're discussing ip addressing i'd like to cover the two types of ip addresses we have ipv4 and ipv6 ipv4 are the original ip addresses that were created and for the most part they're the primary addresses we're using on our computing systems today now the original ip address was a 32-bit address and unfortunately we just don't have enough addresses so when people planned this and they came up with the ipv4 addresses the internet was very small it was used by some academic institutions and a few military organizations and that was it and no one expected that today everybody would have multiple computers per home and every one of them needs an ip address so two things occurred the concept from the internet engineering task force came up with private ip address space and that's basically using the 10.0.0.08 they're giving you the 172.16.0.0.16 to the 17231 0.0 16. and you've got the 192.168.0.0.16 and these are private ip addresses and organizations across the world use these private ip addresses on their internal network now with ipv6 it's 128 bit address and because of that the chances of running out addresses are not that small or are not that great we have more than enough addresses for mo for people that right now believe we should be able to address every computer in the world both karen and future we typically see ipv6 addresses in use is with mobile phones every mobile phone needs an ip address and they typically have an ipv6 address so aws loves to use the term citer or classless interdomain routing and where this really comes from is a long time ago we had something called the class a a class b and a class c address and of course there was a class d which was used for multicast and class e which are experimental which aren't being used and the point of these classes is they either gave you a slash eight slash 16 or a slash 24 and that refers to the subnet mask basically the larger the mask meaning as it comes to the the lower the number so a slash eight will have a lot more hosts that are available than a slash 16 and the slash 24 will have less than that and in our professional version of this course we do a much deeper dive on networking but i right now just understand that a network can be subnetted and we'll talk about why networks are subnetted we subnet a network or we take an ip address block and we break it down to smaller blocks for several reasons we have to be useful and careful with the addresses we have so a wan link for example if you had a point-to-point connection only needs two ip addresses so we wouldn't use a class c or a slash 24 and actually waste of the available 253 addresses that are usable for that that subnet we wouldn't want to waste them but waste that whole subnet if we only needed to so we're going to use subnetting to break a large network into many smaller subnets and we're going to do that for ip address space utilization now we need to be so careful with ip addresses because if you have a router that's got 100 interfaces each interface is going to have to be on a separate subnet if you've got a server with six interfaces in it each interface is gonna have to be in a different subnet so if all you have is the slash 24 you could and you deploy that on one subnet you won't have anything for the rest of the interfaces on that device so you're going to have to be very careful with your address space and you're going to do that by subnetting and i've given you an example on subnetting where we took one subnet and we broke it down into multiple other smaller subnets and therefore we can be good stewardes of ip address space usage now one might ask why can't we just use a giant subnet and stick every host in the available subnet and we can't do that because what happens is once we get a certain number of hosts environment the hosts tend to send broadcasts to identify each other on the network and as they send broadcasts it'll fill up the network and fill up the cpus of devices so practically speaking we're never going to use greater than a class c to place our our systems on the network and the class 3 would be a 192 168.1.0 24 where the addresses that are usable or are from the dot one to the dot 254 which like i said gives you 253 usable addresses the 255 or the highest number is used for directed broadcast and the lowest number is used by the router for a subnet no aws does something that's a little different than the rest of the entire networking world and it's pretty essential that you understand this aws reserves the first four ip addresses of a subnet and the last ip address which is the broadcast so effectively out of any subnet you're losing five ip addresses so i had previously mentioned that with a wan typically you would only use two so a wan link typically has a slash 30. but that can't work with an aws network because you have to lose you lose five addresses automatically so if you only have a total of four of which two are usable and you're going to throw away five you you don't that can't work so with aws the smallest subnet is a slash 28 again the smallest subnet on aws is a slash 28 and the way a slash 28 would typically work there'd be 16 total addresses of which the 0 which is the network and the broadcast which is is is the the all one broadcast at the end would be used which would give you a total of 16 minus two which would give you 14. now i remember aws reserves the first five as well as the broadcast so with aws a slash 28 subnet only gives you 11 usable hosts so you may need larger than a slash 28 even if you have a few devices especially if you have autoscaling because if auto scaling scales out the infrastructure and brings up more compute instances they're going to use ip addresses and the same thing can happen with your load balancers and other devices so make sure you have an idea prior to determining a subnet size how many devices may actually need to be on that network and how many of them are going to scale up and make sure you have enough addresses or use a larger subnet meaning a smaller subnet mask to make sure you have more available hosts now if subnetting breaks down a network into smaller networks super netting actually takes smaller networks and brings them up to a bigger network now for network utilization you'd realistically never do this because your smaller subnets have constrained broadcasts but in reality with routing you may want to send a summary address it might not make sense to have an ip address for every single subnet when you're trying to create reachability you might just need to create an address that says hey everybody for this classless interdomain routing range send it to uh this next hop and that's typically what's used so when you can think of super netting you can think of route summarization the reason i'm making a big point of route summarization when we're talking about the aws network as opposed to bigger more more broad scale say cisco or juniper networks and a carrier is when you connect to aws and you use bgp like with a direct connection you are limited to 100 routes and 100 routes is absolutely nothing in the grand scheme of things of a network so if you only have access to 100 routes you need to minimize the space in your routing table and you're going to minimize the space in your routing table through route summarization as i mentioned in the professional cost course on the networking course we'll have a much deeper dive in networking but in the associate course at least understand that the smallest subnet you can use as a slash 28 the first four and the last ip address of a subnet are actually reserved and understand that auto scaling use addresses uses addresses so make sure you have enough available ip addresses if your infrastructure needs to scale up so now you know what a virtual private cloud is let's talk about the key components of the virtual private cloud the key components include routing internet gateways egress only internet gateways not instances and not gateways elastic ip addresses vpc endpoints vpc peering network access list and security groups so let's begin with routing and routing tables so all vpcs effectively have a virtual router and it's this virtual router that means or retains a routing table or a map of the entire network and the routing table builds a map of the the network and that's how the router determines how to forward traffic and routers typically forward traffic based on the following they take the most specific address in the routing table and when they do a matching algorithm so if you have two addresses for example let's say you have a 192 168.0.0.16 with a next top one and you have a 192.168.0.0.24 with next top two the router is going to take next top two because it's always going to use the one that's most specific now the routers can also are also going to have routes be via things that are directly connected meaning they're right attached or they're going to have things that are dynamically learned via say a routing protocol like bgp or something where a static route meaning you manually put a route to go to the internet here's your next hop to the default gateway and they're basically the two or you could be running bgp which is a routing protocol and bgp is the routing protocol that's typically used when you connect to aws so we'll talk a little bit about bgp a deep discussion of bgp is way outside of this course uh for people that really want a big understanding of bgp we recommend the book internet routing argus textures by bassam halabi it's a cisco press book we also recommend reading the relevant rfcs that's a great way to learn them as well but for here which it's a very basic implementation we're talking about and the first and foremost thing you need to know about bgp is it's a tcp based protocol using port 179. so if you're not a routing and switching guy like me but you do know that bgp is on your network you need to be certain that if you're going to allow bgp traffic into your network you can't have an access list or firewall type function blocking tcp port 179 or bgp will just not work so the reason bgp is typically chosen as a routing protocol when you're using an application like this is it's extremely tunable you have the ability to manually configure which direction you want to you want your traffic to go and there's a lot of knobs that you have in bgp to be able to determine you know the preferred way to exit the network the preferred way to enter a network it's very easy to actually summarize ip addresses for example take a bunch of addresses or a bunch of routes and then summarize it into a single route and it's that tune ability and the ability to send specific routes in one direction versus summary routes in another direction to allow you to perform deep traffic engineering again that's beyond the scope of this class but understand that bgp is the routing protocol you're going to use when you connect to aws bg this routing protocol is a very light implementation it only accepts 100 100 routes for example the internet routing table at this time has well over 700 000 routes so to give you an understanding bgp is quite scalable but for here we're using it for very specific purposes and one of the good things are the btp implementation with aws supports some very simple things like the no export community and it exports it works with other communities and many other tuning features as well but with the no export community is a very well-known attribute and basically what happens is you send your routes to somebody else using the no export community and then that organization doesn't send your routes to anybody else and that type of filtering is very helpful because if you send your all of your your routing information to your upstream neighbor and then they pass to someone else all of a sudden your vpc could be an internet service provider for half of the internet if you've made some mistakes in your routing so that well-known no export community that's supported by bgp is terrific but understand that bgp is a highly tunable highly scalable tcp based protocol and if you understand that you'll have the basic knowledge of what you need for bgp for the solution architect exam so now we'll begin to talk a lot more about nat or network address translation and network address translation takes a private address and turns it into a public address and most organizations have their systems internally with private addresses and their firewall will actually translate their private addresses to public addresses at least in a traditional data center type environment now in this case we have multiple nat environments we're going to begin with the nat instance and then that instance is more of a legacy product with the aws cloud not the preferred method of doing things today and much more complicated than the preferred method but we just wanted to cover it just in case you hear about it so a nod instance is effectively a custom instance or a virtual machine that actually runs on your network and it translates your private addresses into your public addresses and in order for the nat instance to work it's going to have to be connected to an internet gateway and it's also going to have to have a route to the internet gateway and you're also going to have to have a public ip address on that instance so there's a little bit of complexity in getting this set up now this is for egress only connections within that instance so basically that means that your internal devices will be able to reach out to the internet and the return traffic will be able to come in but if you under had a web server for example the traffic would be blocked before it comes into your subnet so in that instance is for egress only internet traffic meaning your devices request information on the internet and the data can come back but external devices will not be able to connect to inside of your subnet within that instance now let's talk about the new modern way to do egress only not in the aws environment and that's with a net gateway and a nat gateway is a very simple fully managed nat service and it's going to be a redundant nod instance inside of an availability zone and it's going to be enable you to connect your internal devices to the internet and allow the return traffic back so for example if you have devices in a secure environment and you want them to be able to go to the internet and update their patches for example and you need them to download new software you do this within that gateway they'll be able to reach out to the internet and their traffic will be able to come back but they will not be in an environment where they can be attacked from the internet so this is good security and it provides good connectivity now if you need a high availability architecture every availability zone is going to need its own nat gateway so two things to remember for both the exams but also real life is for high availability with not gateways each availability zone is going to need its own nut gateway and within that gateway you can reach out to the internet but the inter and your return traffic is allowed back for say patching of an operating system now if you're going to connect to the internet somewhere on the line something is going to need a public ip now you could have a web server that needs to be accessed from the internet and that's going to have a public ip of course you could also place the web the ip address on a load balancer or something and have your web server in a private subnet but how you organize that routing is up to you as the architect but realize that anytime you want something accessible from the internet it's going to need a public ip address and amazon does this with something called elastic ip addresses which you may also see referred to as an eip and what an elastic ip address is it's a public ip address that amazon owns and they maintain a pool of these addresses and when you need a public address or say your web server or your load balancer you borrow one from the aws pool and you have access to that as long as you would need it and when you're done with it you've returned it and aws puts it back in the pool and they give it to another customer so elastic ip addresses are public addresses that are available for you to borrow from the aws platform for as long as you need them and then they can return to aws when you no longer need them now we're going to discuss vpc endpoints and vpc endpoints are the means that you're going to connect to other vpcs or other aws services over the aws network and let me give you two ways that you could possibly do this now if you weren't using a vpc gain you're going to use a vpc but if you weren't using a vpc what you'd have to do let's say you wanted to connect to s3 is you'd have to go from your vpc you'd have to go out to the internet you'd go across the internet and you'd come back from the internet into the aws cloud and then you'd go to s3 now that's problematic for a lot of reasons first you're going on the internet which is not secure unless you use some form of encryption you're using up your internet bandwidth which is typically limited so you definitely don't want to use that and when it hits the internet you don't know what's going to happen the reason i say it doesn't know what's going to happen is the internet is a collection of internet service providers connected together and while you can purchase guaranteed connectivity speeds to the internet your internet service provider once it leaves your internet service provider anything can happen on all the different internet service providers for which your data travels on the way to the end so performance is definitely not going to be over the internet what it could be if you had say for example a private line and in this case what we're going to be using we're going to use the aws network effectively as if you had a private connection with a vpc endpoint so by using the vpc endpoint you're going to your data is going to traverse the amazon network as opposed to the public internet amazon can guarantee the performance of their network they can guarantee the latency they can make certain their network is secure and it's going to cost a whole s to ride the amazon network because it's already theirs as opposed to riding the amazon network to the internet and then back into the internet back across the aws network so you want to use a vpc endpoint for security and performance purposes now realistically speaking there's two kinds of endpoints and there's an interface endpoint and there's a gateway endpoint and an interface endpoint pulls a private ip address space from your vpc's address poll it basically uses the private link service and it connects your vpc to either another vpc through vpc peering or to an aws service now the alternative form is something called the gateway endpoint and the gateway endpoint is a high security high availability high performance gateway service endpoint service and what it does is it realistically places a route in your in your vpc routing table to the aws service and you're typically going to use this when you want to connect to something like s3 or dynamodb but the point being vpc endpoints enable your data to traverse the aws network in a private and secure manner whether you're connecting to an aws service or you're going to be connecting to another aws client who has their own vpc through vpc peering so what is vpc pairing vpc pairing is a method to connect avpc to another vpc on the aws network and vpc peering provides secure and high performance transport across the aws network one thing to remember is that vpc peering communicates over private ip address space and if your traffic is in a region it's going to be encrypted but there's a special component of epc appearing in that it's not transitive and because it's not transitive it means if you have two vpcs they can send their information to each other but if the second vpc were to connect to a third vpc vpca that would say go through vpc-b would not be able to connect to vpcc because it's non-transitive so for those of us that are familiar with bgp from the networking world think of the way ibtp works where you have to fully match your ibgp peers well guess what when it comes to wanting transitive connections in a vpc environment you're going to have to do the same thing these are very similar in function so let's talk about the two types of epc pairing architectures if you don't need transit and you want a highly scalable environment with a little less redundancy you're typically going to use a hub and spoke environment and in a hub-and-spoke environment what you have is you have your center and that's going to be your main vpc and that's going to connect ancillary vpcs across the network through vpc pairing it's going to work great and you'll have good connectivity now if you need transitive properties and let's say you have 10 vpcs and you want every one of them to talk to each other well then you're going to have to use what's called a fully meshed environment and in a fully meshed environment every every vpc will ultimately peer with every other vpc now that doesn't sound like that it's that big of a deal and it doesn't if you have two or three and if you have 10 20 30 50 100 connections that you need to peer with this becomes problematic very quickly so a fully measured environment adds up the number of connections super qui fast so in networking we use the term n times n minus 1 divided by 2. so for example let's say you have 10 devices that need to appear you would have 10 times 10 minus 1 which is 9 so 90 divided by two which is 45 so with 10 connection with 10 vpcs you're going to need 45 total connections so you can see how that'll add up very quickly now bring that to 20 30 50 100 and you're going to need a whole lot of epc connections so the work around to that is let's say you want to use a hub and spoke environment which scales well and at the same time you want to allow vpcs to talk to each other aws is a function called cloudhub and cloudhub enables you to break those rules of fully meshing your your vpc pairs and you'll put them in a hub and spoke environment and then cloudhub which uses bgp will propagate the right routing information to all the vpcs so that all vpcs have network layer reachability information or knowledge of each other's network and therefore full network reachability is achieved let's discuss network acls and network acls are used to keep unwanted traffic out of a subnet so i know i mentioned that security goops keep unwanted traffic out of an instance network acls keep unwanted traffic out of a subnet so network acls allow or disallow traffic based upon the configured policy and with a network acl the default policy is to deny all traffic so if you want something to get through in network acl make sure you have some permits so when you're writing a network acl you're going to specify the source and destination address the protocol and the tcp or udp port number and by protocol i mean tcp udp icmp that type of a protocol but it's based on one of the ip protocols now a network acl is stateless meaning it doesn't watch what actually occurs so it's not like a firewall where if you send traffic out the firewall notices that you send traffic out and monitors the connection so it allows you to return traffic but doesn't allow external traffic in network acls are stateless that means you're going to have to create your rules both inbound and outbound for them to work now anytime you're dealing with access control lists as a rule and it's no different with amazon they're processed in order and they're processed in order of the rule set so the order is going to matter and it's going to matter critically because if you deny something and then desire to permit it later the initial deny will will make it that it can't be permitted later so you have to be very specific and very careful how you write them and write them in the correct order so we're going to show you how not to do this let's assume that you have an inbound rule we'll call it 100 that says deny all traffic and then after that allow tcp port 80 which is web traffic to come in with any from any source and now you've got an outbound rule that says deny everything and but allow the tcp port 80 return traffic what do you think is through nothing and the reason nothing gets through is it's all blocked by the first and i long before your permit so if we're going to write this differently let's say we had a rule 110 that says allow tcp port 80 traffic from any source and allow the return of port 80 traffic to any destination now it's going to work and all other traffic will be blocked but the order matters so let's say for example we find we have a subnet call it the 192 168 1.0 24 subnet and we've got a lot of system use from that we would like to permit that but let's say there's also one host on that subnet that's used for security services that's mapping and doing a whole lot of things that we don't want to be accessing our mission critical application no problem we write the first rule that says block that one host address and we can do it with a slash 32 so that specific host address and then the next rule let's permit that subnet but that's the way you would permit an environment like that because what if we said deny the subnet and then permit the host because the host is part of that subnet it will be blocked so network acl matters and it matters a lot and in this diagram we can show you that when it comes to security you're going to use a multi-layered approach it's like dressing for cold weather layer up well with security you're going to use a network acl to keep traffic out of your subnet and then you're going to use a security group on your instance and you're going to also probably use some firewalling at the perimeter of your network too but the point is you're going to access it through multiple errors and network acls keep traffic out of your subnet now a little more on security groups and i keep covering the topic of network acl versus security groups and the reason is when we interview students we find they often don't understand and especially new incoming students by the time they take our program they understand it very well but these topics come up a lot of exams and part of the security architecture you're probably going to use both but you need to understand how and where they work so a security group is basically a host-based firewall in that it's stateful and it monitors the state of the connections but it's applied to an ec2 instance or some aws services and in reality you're going to use both you're going to keep unwanted traffic out of the subnet with your network acl and you're going to keep unwanted traffic out of your instance with your security group now security groups really all you need to do is provide allow rules because the default policy is to deny all traffic now the good thing with security groups is they're stateful so you're only going to have to allow inbound traffic and they can they automatically know to allow the return traffic and another good thing about security groups is they actually process all rules before they make a decision to send the traffic or deny the traffic so the order of the rules in a security group is substantially less important than something you would find in a network acl still we recommend writing all rules with all types of firewalls or securities appliances with good logic but with a security group it's not as critical because the the rules are processed completely not just discard or throw away in like a network acl would be if you've got a deny rule and then permit statements everything could be denied before you even permit it so with a network acl you're going to be very careful and very specific with the security group you're still going to be careful you're still going to use good logic but at least you've got some cushion as it pertains to the order of the rule and you don't have to apply them outbound because they're stateful in this section we're going to be discussing optimizing performance and networking and the three types of networking performance optimizations we're going to be discussing are going to be placement groups route 53 and elastic load balancers let's begin with placement groups so placement groups are really where you're placing your computing instances and where you place your computing instances can have a big impact on performance or availability or both based on how you do it so let's begin with a clustered placement group and this is going to offer the highest performance option and realistically speaking a clustered placement group is a very low latency group and in this situation what you're going to do is you're going to place your servers very close to each other and not only close to each other you may have a lot of your virtual machines sitting on the same physical server and they may be in the same physical rack so when it comes to latency it's going to be the lowest because if it's going across the same physical server the networking is going to be super high performance and low latency and if it's in the same rack it's going to be on the same switch in most cases so this is going to give you the absolute best in terms of performance at least with regards to latency having said that if you've got all your servers sitting in the same rack or all your servers sitting in the same server if you have a power failure if you have a physical server crash your entire operation is down so this is a great option when you have an application that is extremely demanding of latency meaning it has to be very low latency but if you're going to do this perhaps you want to set up another clustered placement group in another availability zone now let's talk about the next option which is a partition placement group and in this case instances are going to be placed across racks and by doing this you're going to spread yourself across multiple servers multiple switch ports multiple power distribution units which is typically where the servers are getting their power so you're going to provide some level of additional redundancy now this is still going to be relatively high performance because your servers and or computing instances whichever you choose to call them are still fairly close to each other so by doing something like this you've got much better availability than if it's all in the same rack for example or the same physical server but it's still going to be very high performance now your next option which is going to be useful for most people and that's called a spread placement group and what happens is you're going to spread your computing in your computing instances across multiple servers multiple racks and you can even do across multiple availability zones and by doing this you create you create an environment where there's no single points of failure so this is going to give you much better availability but as you spread your computing instances over larger distances for example across availability zones which is effectively across data centers you're going to have higher latencies so choosing the correct clustered placement group partition placement group or spread craze placement group can really help you optimize performance but you have to understand the trade-off the closer you get your things together the lower the availability and the farther they are apart the higher the availability but the lower the performance will be now we're going to be discussing the domain name system or dns every device on the internet or network is going to have an ip address and you connect to these devices by their ip addresses but here's the problem how do you remember the ip address for every website and every computer system in the world in practicality you can so the domain name system is effectively a database which maps an ip address to a name and by doing that it becomes much more simple because we can remember amazon.com but we can't remember its ip address unless you're one of those lucky geniuses with a photographic memory so here's what happens you're on your computer system and you're on your computer system and let's say you want to go to www.amazon.com you enter that in your browser and of course your browser doesn't have an ipad so your computer then says how do i get to www.amazon.com and it then sends a message to the dns server and says what's its ip address and the dns server responds to the computer and it gives an ip address and then the computer connects to amazon.com via its ip address so that's what the dns system does it maps a name to an ip address and you can check this on your computer if you're on a windows system or a linux system or a mac from a command prompt type nslookup www.amazon.com and you'll first see the response which is your dns servers your name servers which is who you ask the question to and then you'll get the response of its ip address and that's how you know how you can map a name to an ip address now we're going to be discussing aws route 53 and route 53 is the amazon implementation of the dns protocol so let's dive into amazon route 53 so as i mentioned route 53 is the aws platform for dns and the aws platform is a highly available high performance highly scalable low latency dns service and the reason it works so well is it uses what's called anycast services and what anycast services are is you use the same ip address for your dns servers and that gets a little tricky so let me just explain it in simple terms your computer has a dns server attached to it and that's how your computer need knows how to find how to find the the name that maps to an ip address or the ip address that maps to a name but what if your dns server were to go down or your primary backup your computer wouldn't know how to reach anything but with anycast what happens is the dns servers all publish to the same ip address and you then when you connect to the ip address connect to the one that's closest to your location and hence that gives you the low the lowest latency but for example if your location were unavailable the next location will still be available published to the same ip address so your computer which is going to be programmed to rese to attach to the anycast address even if your local anycast address isn't available a remote one will be so this gives you built-in high availability but it also supports latency now route 53 comes from port 53 which is the tcp and udp port that's actually used for dnf services so if you're curious where that came from most likely route 53 comes from port 53 which is the dns port now something else to know about the route 53 implementation is it supports health checks and we'll talk a little bit more about why health checks matter specifically when you're using dns for routing whether it be performance or failover or other things but we'll talk about that you should know that the aws route 53 implementation supports most of the dns types and records that you would ever need in an environment but there are really about four records that we'd like you to know and we'd like you to know these four records because they often come up on exams but more importantly in your career these four dns record types are going to be pretty important so let's just talk about them the first record type we'd like you to know is something called an a record and a dns a record quite simply just maps a name to an ip address that's it fundamental records used everywhere maps and name to an ip address now the next record we'd like you to know is something called the cname record and a scene name record is really just like a dns redirect or an alias so let's say you have a website and it's existed for a while and your branding department says i'd like a new name but you don't want for ex and you want to use the new name but you don't want to lose all the customers that are coming to your current website you would start your new website with your new name and then you would create a dnsc name record that would point from your old domain name to your new website and that way you'll keep your current customers that you've had from the old name as well as your new and future customers that come from the new branding that you're trying to do so a cname record just maps one domain to another the next record that we really want you to understand is something called the name server record and that really just tells you who your name servers are but it's essential to know who your name servers are and who are your authoritative name servers meaning the name servers that you truly believe the last record we're going to discuss is something called an mx record and an mx record is a male record and without setting up an mx record your organization won't be able to receive email so these are the four key records that we'd like you to know as we begin our discussion into dns routing policies i want to begin this discussion with talking about health checks so the aws route 53 implementation supports health checks and really what a health check is is the dns server will actually reach out to the server and it'll say are you there and if it's not there it can be used to make decisions maybe route to an alternate location because if you map a name to an ip address but the ip address isn't there you'll have an outage so there are routing policies that enable you to fail over or fail or use the one that's closest to you or yes that's the one that's the lowest latency but one of the things that has to be enabling factor for these things to work is you need to be able to do a health check and health checks will also be used when we talk about load balancers but a health check is really just a way to say server are you there so aws route 53 has several route policy options so let's talk about what they are the first is simple routing and simple routing is very simple it maps a dns name to an ip address and that's it the next type of routing policy you can set up with aws route 53 is something called failover routing which basically says go to your primary location and if that's not there go to a backup location and this is uh terrific it's a great way to i fell over to organization it's not that complicated to do now we can start getting into geolocation and geoproximity routing which we're going to do next and the next thing is something called geolocation routing and what's going to happen is when you go to request something your source ip address is going to be looked at and because your source ip is going to be looked at the dns server is realistically speaking going to know your location so what it's going to do is it's going to send you to the server that's in the closest region and that's going to work really well for global organizations they don't want their people in asia coming to new york for example and having the latency of that long-haul network they might still want that to work for example if the asian network is down so using geo-location routing really gives you a great opportunity to find the server with the lowest latency based on region and from there make sure that you also have some failover capabilities now another version of location-based rounding is something called geoproximity routing and while geolocation routing basically works on a region geoproximity routing actually works on like an availability zone so the way this works is you have your dns server set up and it will find inside of your region the closest availability zone so theoretically you should have the lowest latency now you could also set this up by latency and instead of geography which usually equals latency latency but you could do it based on purely latency so there's an inversion to do route 53 called latency-based routing and in this case the dns system looks for the server that has the lowest latency and then it sends you there so this gives you extremely good performance the last two that we're going to talk about is something called multi-value answer and really this is a record that says route to anywhere any available server that you've got use one of them so it's not very specific it's not very performance oriented but it does allow for load sharing and the last one we're going to talk about is something called weighted routing and what weight rated routing really is is you can set it up to say send 75 percent of your traffic to one web server and 25 to another web server so there are your dns routing policy options on the aws platform so let's discuss load balancers load balancers are a fantastic way to increase the performance and the availability of your systems and what load balancers do is they enable you to share load across multiple systems so we previously talked about scaling up versus scaling out and as a refresher scaling up is moving to a bigger server but when you have applications that are really going to grow there's going to be a point where the largest and most powerful servers in the world still won't be able to meet the demands of the application plus if you put it all on one server and the server were to fail your systems would be down so you're going to want to place it on multiple servers so as you would scale out for example by adding multiple compute instances you have to find a way to load share across these compute instances otherwise you'll have one up and the others will be backup and low balancers are really the way that you're going to do that you can do some of this with dns but you're going to do a lot more of this on the load balancer side so load balancers enable utilization of multiple servers to share load at the same time so there's realistically speaking two kinds of load balancers that are available and there's network load balancers which are operate at the network level specifically layer 4 at the t of the osi model which is tcp or udp now the next type of load balancers which have a lot more intelligence in them are actually application load balancers and they work at layer 7 of the osi model or the application layer so they're typically looking at http versus https aws has three types of load balancers available they have application load balancers they have network load balancers and they have classic load balancers which are available in both network as well as application forms now we'll talk about the types of load balancers offered specifically by the aws implementation the aws implementation of a load balancer is called an elastic load balancer now these load balancers are very high performance and they do a great job of distributing traffic across your servers or compute instances there are some things to remember about these one of the things that makes these so great is their auto scaling so if your load balancer is running at full capacity it'll spin up another load balancer so you're always going to have the ability to have extra performance with these load balancers but you do need to remember if the load balancer spins up a new instance and it's another load balancer it's going to need another ip address so plan your subnets carefully because if you have auto scaling of your of your compute systems and your load balancers and many other applications on your network if you don't have a big enough subnet size and auto scaling occurs you could literally run out of addresses and then your systems just won't scale so try and remember that now these load balancers support health checks and that's key for performance as well as availability you don't want the load balancer to send traffic to a server that's down so the load balancers will use health checks and that's how they can increase availability by taking unhealthy servers out of the rotation now these elastic load balancers can also terminate ssl sessions so they can really take a lot of load off of the web servers by terminating ssl sessions so let's talk about the three implementations specifically we're first going to discuss the elastic load balancer which at the network level so this forwarding is realistically based on the destination port and it's going to look at tcp and udp but these network load balancers are really fast i mean they can handle millions of requests per second so if you're looking for the fastest option on the aws cloud it's going to be a network load balancer good thing to remember as these load balancers are stateful meaning they look at the state of the connection and they remember it the entire time so they use something called sticky sessions so they're gonna they're gonna know where the user is and which server their traffic's gonna be and it's gonna stay there for the entire session and we want to do this to make sure the performance is good but we also want to make sure we don't have to go through authentication processes and such so sticky sessions are going to make sure that you stay on the same server during the entire time of your session now let's talk about application load balancers and application load balancers work at layer 7 of the osi model so they typically look at http https but they can actually look at the path provided in the url and they can even look at the method is it a is it a get or a push so these things have a lot of intelligence in them and they're going to work at the application level now they're also stateful and because they're also stateful they're going to make sure that things are on the same service the entire time meaning that if i connect to server b i'm always going to be on server b now application load balancers are really a great way to deal with micro service based applications so that you know keep that in mind if you're asked a question on micro service based applications in a load balancer you're going to want to use an elastic load balancer so let's talk about the aws legacy load balancers they're still available but it's not recommended to use them now you should use an elastic load balancer at least with the aws network but they're still available so when at least cover them the classic load balancers are available in both network as well as application forms for the most part they work very similar to the elastic load balancers and that they're stateful and they can also terminate ssl connections but these are older load balancers and we really recommend that you use the elastic load balancer either the network option if you need extreme performance the application load balancer if you're looking for more intelligence in the load balancer network what we're going to be doing is creating a vpc now we've been working in the default vpc but in this case we're going to create our own vpc and we're going to use that so we can create the networking the way we desire it in future labs so in order to do this we're going to go to the management console and under networking and content delivery we're going to click vpc now the good news is they make it very easy with the launch vpc wizard so what we're going to be doing is we're going to launch our vpc and the first thing we're going to do is we're going to create um the type of vpc and then we're going to add our networking so typically speaking your addresses are going to be one that are specified in that rfc 1918 address space and with that what's going to go along with that is the 10.0.0.8 but we're going to be using the slash 16. we're going to have the 170 to 16 all the way to 172.31 16. so the 172.6 all the way to the 172.31 16 and then we're going to have the 192 168 0.0 16. and these are going to be the private address spaces that we could be using but in this particular case we're going to be using just an ipv4 cider block we're not going to be using ipv6 we're going to keep this as simple as possible and there you go so we're going to create the vpc notice it's automatically creating us a public one which is in the private address space yet what's going to happen is going to be knotted out the public addresses for when we need to use public so there you go quite simply that's how our vpc was created and you'll be able to see the vpc in the vpc dashboard here you can see the default vpc here you can see is the new vpc we just created and under the vpc id you can see certain things you can see its state you can see how the dhcp is being set up you can see the route table and any network acls that are there doing is creating a subnet now that's something that can be created by this cli but it could be created so easily via the management console we're going to use the management console and the first thing we're going to do is we're going to go to the networking and content delivery section and we're going to click vpc and from here you can see the vpcs that actually exist so we're going to go to these vpcs and we're going to look at them now this vpc is the default vpc that's created when you launch your your your organization and you set it up on the cloud and it's going to be something that's going to have typically loose security policies it's designed to be easy to use and set up right from the go typically speaking i like to create my own vpcs so we've already created this vpc so now let's go into this vpc and while we're into this vpc we can see some information about the vpc but in this particular case now that we're in the vpc you can see what we can do we can edit our classless interdomain routing we can edit dhcp option sets host names that we'd use under dns tags that could help us do things or we could of course delete the vpc but in this particular case we're trying to create a subnet so what we're going to do is we're going to click subnets and we're going to make sure we do the subnet inside of this vpc but we're going to do this by creating subnet now if you notice default plus the one we just created and here what we're going to do is we're going to specify a name for the subnet and what we're going to do this is we're going to call this private subnet one and we can pick which availability zone we want to use for the subnet or click no preference in this case we're going to let it pick its default one because we're being very simple and the cider block that we're going to use we're going to use a 10 dot 0 dot 2 dot 0 we're going to go slash 24 because that'll give us enough ip addresses that we don't have anything to worry about we've got the name we've got the private subnet we can add a tag and we can just create the subnet now you now let's look back at our bpc and look at the subnets that are actually there here's the new vpc under vpc let's look at the subnets that exist and there you can see we have two of them we have the 10 slash 0.0.0 24 which is the public subnet that was created and we created a private subnet with a 10.0 there you have it that's how you create a subnet inside of your vpc going to be doing is we're going to be creating an elastic network interface and we're going to be attaching it to an ec2 instance now when you boot up an ec2 instance for the first time it already comes with a network interface but there might be times where you want to dual home your your system for example you might want it to be on a public subnet or a private subnet or there might be times where you're going to place it on two separate private subnets and have different security groups for different access for different people so these are the kinds of things that you could do by creating multiple interfaces so to create the interface there's a few things we're going to have to do we're going to go to ec2 and under ec2 what we're going to do is we're going to create a network interface and you can see that under this section network interface so from here we're going to create a new network interface and we're going to give it a name and i'm into very simple name so we're going to give it something like gca go cloud architects lab int in reality you're going to use something much more descriptive something that's going to help you but in this case today what we're going to be doing is we're going to pick something simple now under subnet you're going to have to create a subnet where you want this to be now recently i just created a new subnet so we're going to choose one of the new subnets that i just created and it's going to be a subnet in us east 2a so i just created a subnet i called it new subnet so i'm going to use that subnet now we're going to have to imagine a touch in the security group and i'm going to use the default one the default one is a bit open because it allows ssh or port 22 in from anyone in the world where in reality or we're going to limit it to just the subnets that need access to ssh access but for right now we're just going to use the default subnet because we're only trying to work on one thing at a time because we want to create a good learning environment so we're going to choose the default security group and then we're going to create this network interface and now we've created a network interface but the network interface is still sitting there sort of floating around in the either we have to attach this network interface to an ec2 instance so it could be used so we're going to do just that we're going to look at our two interfaces and what you can see is this on the bottom is the default interface that came with the instance and you can see the public address you can see its private address and you can see the owner now we're going to go to the new interface that i just created under actions we're going to click attach so now when we choose attach we're going to have to choose the instance that we want to attach it to in this particular case we're going to choose it to the only inner instance that exists click attach now what you should see is you should see these network interfaces and they should both say in use and that's telling you very good they're in use you can also see they're on separate subnets and there you have it so let's go to our ec2 instance let's see what's going on with the instance and on this instance you'll be able to see lots of things under networking for example you'll be able to see the instance id you'll be able to see the two interfaces and the respective subnets that's it fairly simple that's how you create an elastic network interface and you attach it to a running instance is we're going to modify the default security group and typically speaking with a security group when you're building it you want to lock it down to just the subnets that need access to certain things so in other words if you have a direct connection between your data center and your aws cloud and the only people that need to access this are basically say working in a noc where they're doing network operations and part of the network operations actually includes cloud managing the cloud environment and let's say they're in a 10.1.1.0 24 subnet in the security group you would only allow their subnet in because we're not coming from a direct connection or a vpn connection and we're going to be doing things from a remote access environment and because this is a lab we're basically going to set up our security group to allow several protocols and when it comes to security groups we only have to pro without permit statements because everything else is going to be denied and the security groups actually look at the state of the connection so what ultimately happens is they're they're stateful like a stateful firewall so they're going to watch the traffic that comes in and they automatically know to allow the return traffic back out and if they were stateless like a network acl you'd have to specify rules in and allow traffic out but they are stateful like a stateful firewall and therefore we only have to allow in so what we're going to do is we're going to go to our ec2 environment and we're going to look at the instances that we have running we have one instance that's actually running so we're going to click on this instance and under security what we're going to do is we're going to modify the security group and you can see with the current security group that we have we're allowing port 22 and only port 22 and that's ssh but let's say we want a little more so let's uh let's allow in http traffic along with https traffic and you can even give yourself a description to make it a little easier for you so in this case we're going to add a rule and we're going to allow the protocols that we want to come through and in this particular case we want to allow http and https traffic so the first thing we're going to do is we're going to pick out http which is going to be port 80 and we're going to allow it from everywhere and we can say allow wd allow web traffic now we're going to add an additional rule to this and we want to allow https so we're going to look for https which is going to be port 443 again we're going to allow this from everywhere because http and https unless it's an internal web server like for a corporate internet we're going to want these available to the rest of the world because they're going to be a public web server we're going to save the role and that's it as you can see in this inbound security group what we're actually allowing is web traffic via http or also referred to as www we have https and we have ssh so that's what's going to be allowed into this instance that's it that's how you set up a security group on an instance do is we're going to create a network acl remember that security groups are applied to instances but network acls are applied to subnets so because they are part of the networking function from the management console in order to create a network acl we're going to go to network and content delivery and we're going to click vpc so after we click vpc it's a fairly simple process we're going to go to network acls on the left side of the page and here we're going to quite simply create a network aco and we're going to name it so when you do this you're going to want a naming convention that's going to be very descriptive so you can remember which network acls are applied to which subnets but in our case this is a lab environment and because it's a lab environment and you're going to need to take these things down so you're not charged pick something that's going to be very easy for you so for me i'm going to call it new net acl just because i want to be able to remember it so i can delete it and i'm going to create this network aco but of course you have to select the vpc and we're going to we're going to from the drop down select the vpc now from here you can see we've got a new network acl and it's not associated with any subnets yet but let's make our policies before we associate it with a subnet because if we associate it with a subnet before making the policies and we have systems on that subnet we're going to break them because the first rule of the default new acls when you create them is to deny everything which means if you've got a web server that's up and running and serving traffic and you place this on during the time you're editing the rules it will be completely unavailable to serve traffic so let's create our rules right now now when we're creating these rules they're gonna they have to be applied in a sequence because they're gonna be read in sequence and we're gonna leave a space of about 10 in between everyone at least because there's going to be new rules that you're going to need to create a common example is let's say you've got one bad host coming into your web servers you can create a deny rule for that one host by placing it in front of some in somewhere where it needs to be earlier in the sequence because we might want to deny one host but still allow an entire subnet so you would do the deny before the permit for the subnet because if you permit the whole subnet the traffic is going to be through so let's say you had a rule 100 for example that said allow traffic from the 192 168 1.0 subnet in all traffic and then you have one host that's infected with malware you might want to create a simple deny statement for some one single host on that subnet so that's why we want spaces because you're going to find you're going to have to create new rules and if you don't have spaces you're going to have a problem so we're going to pretend that this subnet has web servers and absolutely nothing else and because they're web servers there's only two kinds of traffic we need to allow in http and https so good security practices involve only allowing necessary traffic in and nothing else so we're going to create a rule 100 and we're going to say http is allowed source any allow and then we're going to create a new rule we're going to call it rule110110 and we're going to allow https traffic in from any source and then we're going to save the changes to this and that's the inbound rules that we need for the subnet that only has web servers and c http is in allowed in https is allowed in and everything else is going to be denied so now let's make our outbound rules and because network acls are not stateful we have to create the same outbound rules to allow the return traffic out so in this case we're going to create a new rule we're going to call it rule 100 we're going to allow http return traffic back out and then we're going to add a new rule we're going to call it 1 1 0 and in this particular case we're going to allow the return of the https traffic and we're going to save the changes now we've got a network aco that for the inbound rules allows http and https only also be denied and for the outbound rules we have the same thing http and https return traffic is allowed and everything else is denied now we have to associate the network acl with the subnet so we're going to click edit network acl subnets and i had previously created a subnet called new subnet so we're going to associate that network acl and that's it we're going to save changes that's how you create a network acl and that's how you apply it to the subnet i've created internet gateway and we're going to place a route to the internet gateway called the default route now what a default route is is if there's no more specific route in the routing table send your traffic this way so organizations typically have a route for everything they need to have in the router and then they typically have a default route which sends traffic to the internet the exception would be if the organization's connecting to multiple internet service providers for which they'd be running bgp but very often even on their network while those two while the routers connecting to the internet service providers will have full set of internet routes they'll typically propagate a default route through their network and that basically means if you don't know how to go somewhere follow the default route and the default route goes to the internet because on the internet that's where things will be outside of your organization so we're going to do this by going to the vpc dashboard where the reason we're going to go to the vpdc dashboard to create our internet gateways is that's where these kind of func network functions occur so we're going to click vpc under networking and content delivery now when we get to this area we're going to create an internet gateway so you'll be able to see the internet gateways on the left side of the menu and what's going to happen is we're going to create this and it's going to ask us to create some steps along the way and we're going to do just that and in this particular case we're going to create an internet gateway now we can name it anything we want we're going to call it gca go cloud architects then we're going to pick lab because it's a lab and then we're going to call it internet and then gty and this is going to make it easier easy for us to identify what it is now of course you could add a tag in which case we have name and we've got the value of what we're calling it we're looking for something very simple now we're going to create the internet gateway now the internet gateway actually exists which is terrific now we're going to attach it to a vpc but you'll see what happens when we try to attach it it's going to tell us that we don't have any vpcs to attach it to so we're going to create a new vpc and we're going to call this gca new vpc and typically speaking when i deal with networks i typically delete the default vpc and create my own with my own subnetting and my own security standards so in this case we're just creating a vpc and we're going to give it a cider block and we really want to use a cider block that's not going to overlap with what we currently have so let's try the 192.168.0.0 16. we're going to click create vpc there you go we've created a new vpc inside of the rfc 1918 private address space of 192.168.0.0.16. now that we've created this we can make sure that it's attached to the vpc we just created this now let's attach it to a vpc and under our vpc here's the new vpc that we created there you go we're going to attach an internet gateway now we're going to have to add a route for unknown traffic to go to the internet and what we're going to do here is we we're going to just go to the routing table of our new vpc so here's our new vpc we can see about it we're going to look at our route tables for and for here we're going to have to pick out our new vpc now it's a little bit tricky where they actually place information in terms you're going to actually have to find out what that vpc is called so i wanted to show you what it's going to look like now let's go back to your internet gateways because the internet gateway will show us the one that we just created and it's going to give us some good information our vpc id and it's going to end in d delta 1 f foxtrot and that's how i try and remember them there's too many letters i try to remember them by the end so d1f we get it we know what it's going to look like so let's go back to our route tables we're going to find the one that says d1f which as you can see is over here we're going to go to the routes and we're going to edit our routing table you can see we have a route to the locally connected subnet inside of our vpc but we're going to add a router and we're going to add a 0.0. it automatically populates for the default route because it's a very common route and then what we're going to do is we're going to choose our internet gateway i'm going to pick our internet gateway i'm going to save the route now that's really all you need to do we've created a vpc we've created an internet gateway and we've created a default route so instances inside of that vpc that are on those subnets they're going to look inside of the routing table which and the routing table that of course we'd have to associate the routing table with those subnets but they're going to be able to reach out to the internet because they have a default route that's it that's how you create an internet gateway and that's how you create a default route to the internet gateway we're going to be setting up route 53 which is the amazon dns or domain name system at least the amazon implementation and we're going to set up something called the hosted zone and we're going to set up something called an a record and really we're just going to create a simple record the a record that's going to map an ip address to a domain name very simple the simplest form of dns so under networking and content delivery we're going to click route 53. now in an optimal world we're going to register a domain and we're going to import that into the domain management system having said that this is a lab environment we want to make it as inexpensive as possible for the people that are here so i'm going to go to dns management we're going to click create a hosted zone now while we're here we're going to need a domain name now if you have a legitimate domain name that you own feel free to use it if you register one feel free to use that if you're just making up one to do the lab that's completely fine so we're going to call this gca dnslab.com and you can put a description if you desire and then we're going to just create the hosted zone and as we do this this is great we've created the lab and the hosted zone and you can see we have a name server record and these are already there but we want to create an a record and by creating the a record we're just going to click create record we're going to click simple routing remember you've got multiple options weighted geo-location latency failover and multi-value answer and we covered them deeply in the lectures but we're going to use simple routing here we're going to create an a record and we're going to create this just simple as we'll even use the term blog because it's fine it's the one that was suggested and the value here's where we're going to put the ip address of our system so i'm going to take an ip address that i created 3.12.241.1 and i'm going to create the record that's pretty much it we've created a record that maps this domain name to this ip address and you'll be able to access the website assuming this is a web server that you created from a browser a very simple dns record now if you were going to set up multiple sets of dna multiple sets of web servers behind dns you might want to use a different type of routing whether it be a latency routing a fail over routing and in order to make that work you're going to have something have to set something up called the health check so the first thing we're going to do is we're just going to create a simple health check and we're going to do that because we've shown you how to create hosted zones you can see where you can easily change the routing policy but knowing how to do a health check is absolutely critical so let's create a health check so the first thing we're going to do is going to go to networking and content delivery we're going to click route 53 and here what we're going to do is we're going to click health checks and from here we're going to do something very simple we're just going to click configure a health check and what we need to do here is we need to come up with a name of the health check so we're going to call this the gca lab h e c k h-e-a-l-t-c-h-e-c-k the name i made up and we're going to monitor you know the endpoint or any or whatever we want things that we're looking to monitor and we're going to have to place the ip address and in this case we're going to use our web server which is 3.137.194.184 and realistically speaking we're going to click next we can set up an alarm for example if you want to be notified if something doesn't go well or you don't have to we're not going to do this because this is just a test and then we're going to click create a health check and right now the status is unknown and what's going to happen is going to start sending messages and check the status and the status will come up as complete in a little bit so that's really all you need to do to set up a dns health check you're basically sending a message it's going to check a page on your server and if that page is no longer there it's going to mark the instance of healthy and it'll remove it from your dns rotations load balancer and if you recall from the lectures the reason we're setting up load balancers is twofold if we want to scale up meaning increase the size of the server we can just get a bigger server but no matter what we do there's never going to be a server that's going to be big enough for our needs when we're dealing with global enterprise environments so we have the opportunity to scale out which means add more servers but if you're going to use a dozen servers or several servers you're going to have to find a way to load share across those servers and you're going to load share across those servers with a load balancer now we're going to create a load balancer now with amazon they're considered to be elastic load balancers but they're listed under ec2 because the load balancers are actually running on ec2 instances so let's click ec2 and you'll notice in this area you have the opportunity to set up a load balancer so we're going to set up load balancers and we're going to create one now when you're creating a load balancer you've got a couple of options you've got the classic load balancers and whenever possible their legacy it's recommended you use one of the newer load balancing options so you can use an application load balancer and this is really great if you're looking for intelligence of the application level routing between microservices and the such if you need extreme speed you're going to use a network load balancer a network load balancer can be much faster and maintain extremely low latencies so know which one you want to use if you're dealing with microservices or containers as a rule the application load balancer is a great choice but if you need pure performance you're going to go with a network load balancer i come from the network world so i'm going to just pick a network load balancer because i've been working with them forever and realistically speaking what you're doing is you're going to say that it's internet facing or internal because you could use one for your internal purposes or your external purposes maybe you need to load balance across 10 servers for your internal users and you would be an internal one or it can be an internet facing one for your web servers so we've got our listeners they're going to listen and make sure things are go they're going to listen for connection requests and when we're dealing with http and https they can terminate these sessions on the load balancer which is going to offload a lot of the work from the actual web servers themselves and we're going to just select all of our availability zones because we want to be able to use the load balancer wherever we place our things and now we're going to configure our security settings but before we do that we actually have to name it so like anything else that needs a name now generally speaking come up with a naming convention that's going to make it very easy for you to identify in the lab environment pick something that's going to be easy to identify so you can find it and easily delete it so you won't be billed so let's just call this gca mic elbow now generally speaking heavy heavy security you should set up you should set up your firewalls your intrusion detection prevention system your network acls as well as your security groups and then even on your systems you have a host based firewall but in this particular case we're just going to set up our routing and what we're going to do is we're going to come up with target group and we're going to come up with we're going to set it up the way that we need to do this so we're going to look at the target group we're going to come up with the name of the target group and it's going to call gca mike elbtgt i want it to be something simple and we can use the standard traffic port what it's going to do is it's going to be healthy and it's going to periodically check it every few seconds and if it misses a few health checks it's going to time out we want it to time out faster so we're going to pick an interval of 10 seconds now what we want to do is we want to share the two instances that we have running and basically what's going to happen is the load balancer is going to share amongst all the instances we've set up so i had previously configured two identical instances next we're going to review our configuration and you can see that we've set a load balancer we know it is it's an interfacing one which subnets we're allowing it to be used on and then you can see the targets or the instances now there you go we've successfully created an elastic load balancer which is being provisioned and it's going to load balance traffic between our ec2 instances so this section is on security of the vpc on the aws cloud so we're going to cover a lot of topics in this section and the first topic we're going to cover is who is responsible for security then we'll talk about the principle of least privilege a little bit about industry compliance identity and access management also referred to as iam we'll discuss multi-account strategies network acos security groups and waff and then from there we'll touch upon intrusion detection and prevention distributed denial of service attacks or ddos attacks and then we'll talk briefly about the service catalog and the systems manager parameter store so now let's talk about security and who was responsible and aws uses something called a shared security model we'll talk a lot more about that but in the secured security model the customer is responsible for the security of their vpc and we'll talk a little bit more about that whereas aws is responsible for the security of the cloud so let's talk at least a little bit about the customer responsibility and that the things that you're going to have to do as the customers you're going to have to manage your access lists your security groups your acls and all of the things that you would naturally do to maintain the security of your servers your systems or appliances inside of your data center aws will handle things like making sure the ec2 instances are strong so for example they'll make sure they're up to the latest hypervisor with when it comes to security to make sure the servers are there they'll make sure they have physical security of the cloud they'll make sure that the cloud for example the cpu vulnerabilities that existed with amd and intel's cpus they'll make sure they're patched in firmware so what they're going to do is make sure the infrastructure is secure and what you're going to do is maintain the security of your vpc now on this graphic you can see a little bit more about the the aws shared security model and but really it all boils down to is you're responsible for the security of the vpc and they're responsible for the security and the integrity of the cloud to begin let's cover the most important topic of security and that's simply the principle of least privilege and the principle of least privilege is also referred to in the military as need to know and what that really means is people are given access to the information that's necessary for their job or their mission and absolutely nothing else well when it comes to designing your systems only give whatever is necessary to your people and the reason you're going to do that is you want to make sure they can't accidentally delete system files they can't accidentally create systems that are going to cost a fortune or they you also don't want them accessing critical trade secret information that they could take with them to a competitor so the principle of lease privilege is one of the most critical components of security and you definitely want to use that with your aws vpc only provide access that's necessary for people to do their jobs and if for some reason an employee is terminated instantly remove them as one of your qualified users but also if people's jobs change for example let's say they move from an engineering role to a management role they may not need engineering access to symptoms so you're going to remove privileges that are unnecessary for your users and you're going to make sure your users have the appropriate privileges for them to do their job do their job well and absolutely nothing else now we'll talk a little bit about industry compliance and what i mean by industry and compliance is several industries are highly regulated and they're regulated with regards to security things like encryption and data protection they're regulated with regards to how long you need to maintain your data and aws supports many of the industry compliance standards that are actually out there and a full list of all the standards is far beyond the scope of this course but know that aws supports critical industry compliance requirements we're going to cover a few because you may see them in test questions and they're extremely common one of the standards that are supported is something called pci dss and that's really an industry standard for payment cards like a credit card there's several iso standards related to cloud computing and other matters and aws is definitely compliant with those fedramp is the standard for which aws is compliant and you need to be complying with fedramp to deal with the u.s federal government and aws also meets hipaa requirements and hipaa is basically a u.s privacy law related to health care and how we manage as healthcare providers organization i spent a little bit of my youth practicing internal medicine as a nurse practitioner so i'm very familiar with the hipaa laws but understand that aws supports many of the industry compliance things in this section we're going to be discussing identity and access management and identity access management is a critical component to information security and it's often misunderstood so we're going to describe it the way most of the industry calls it which is actually aaa which is authentication authorization and accounting now this is an older term but it's very clear and once you understand you'll never forget it so we're going to use this means to describe i am so authentication basically identifies the user and when you go to access a system on your on your computer and it asks for a username and a password part of that is because they want to authenticate you and if you have the right username and password there's a good chance that you are who you say you are now the next component is something called authorization authorization basically says are you allowed to access resource if yes grant access if no block you and then accounting is really finding out what you've done once you're actually on the network what systems have you accessed have you gone to this hard drive if you go into this server that's really the accounting components and accounting is a really key way to figure out what people have done and and how to fix things and process uh improvement as well as good security now aws actually takes an iam and they break it down into two things called users and roles now an iam user is a person an iam role is a system accessing another system so if i were to log on to a system i would be an iam user if my computer or my server needed to access dynamodb it would need an iam role so dynamodb would know that it's an authorized system to access that information so let's talk about a few more key components of iam aws actually uses iam to control users groups and access control policies and the way you're going to use this is the easiest and most scalable way because of course you could create an im user for everybody and then assign permissions manually but imagine if you have a hundred thousand employees or more and you literally want to try and make an i am user for every single person and give them all individual levels of access that would be very challenging so what most people do is they're going to create a group and inside that group they're going to provide permissions to the group and then they're going to add users to the group so for example let's say you have a people that are database administrators you would create an iam group called database administrators and you would give those database administrators access to whatever information they needed to do whatever they needed on those databases so that's really how the concept of iam groups work and for the most part that's the scalable way that you could do it you could just create an im user and assign permissions but in most cases especially in an enterprise environment you're going to create groups and then you're going to add your users to the groups and this is something you would do whether you're working with you know microsoft and active directory environment or for the most part any other type of technology systems so aws as part of their im has two concepts they come up with the concept of principles and they break this down into several so there's a root user and the root user you can think of them as the king the root user or queen whichever you like but the root user has access to everything on the system they can do anything they need including delete the entire vpc so you really don't want to use that root user account you want to use that account and then you want to create other accounts so the root user account is effectively the same thing as if you were to take a linux system you wouldn't log into your linux system as the root or the super user and do things you would do that to do administrative work and then you would reduce your privileges as a user to use the system most of the time same thing's really going on here then of the principles you have im users and they are the people that are actually using the systems and then you have i am roles which are for specifically for the systems accessing other systems now we're going to be discussing roles and security tokens and roles specifically iam roles and security tokens play a big part with regards to security on the aws cloud and there's realistically speaking three kinds of roles that we'll discuss we'll discuss ec2 roles we'll discuss cross-account roles and then we're going to discuss identity frinerations and to some degree single sign-on but that's much more of a topic for the professional exam so iam roles specifically for to create a dramatic enhancement and security over what life would be like without it so let's say if you didn't have an im role and you had an nec2 instance that wanted to access dynamodb the ec2 instance would have to maintain a username and password to then access the dynamodb instance or in this case the dynamodb service which would work but what happens if your ec2 instance is hacked you're going to now gain access to the passwords to dynododb as well as any other passwords that are stored on that system so that's really not a great way to do things from security so the way the im role service works with aws is you you assign an im role to an ec2 instance the ec2 instance attempts to access dynamodb and through the api it's given a token that expires but it then uses that token to then access the dynamodb or what other service whatever service you you've set up on the aws cloud and therefore you're not storing passwords and you're using temporary tokens and temporary tokens are fantastic because once they expire they can't be used again so there's no password stored on the system and you're using temporary passwords so even if the server were to be compromised and someone got access to the token it could only be used for a period of time now a cross account role is really used with partner organizations so for example let's say you're an organization and you've developed a partnership with another organization that's in a separate vpc you can create a cross-account role that enables a user in your organization to access symptoms in another organization so that's really what a cross-account role is it enables one vpc to authenticate and and authorize an account for users and another vpc so the way a cross-account role works is a role is created for the external user the external user will then connect to the aws token service and receive the temporary token and then the external user provides that temporary token to access the other vpc assuming they're authorized to get access to what they need to so now you understand the role of roles and security tokens cross-account roles and we'll talk much more about identity federations so let's discuss identity federations managing large environments with hundreds of thousands or millions of users that all need iam can become quite cumbersome and one of the best ways to do this on a scalable matter is to use something called an identity federation an identity federation enables you to federate or connect to an external identity provider and then use their identity management services on your aws vpc so i'll give you two kinds of examples one is you could use something called your windows authentication system which typically works with active directory so you have a user that logs into their windows system they're authenticated by the active directory controllers inside of an organization and then that identity is then passed through so they can access resources on the aws cloud and by doing that they won't have to authenticate to the aws cloud after they've already authenticated to your windows systems another way to make this scale is you can federate to an external identity provider like facebook like amazon like twitter like linkedin even apple so by doing this you can basically take users from another organization authenticate against them and allow them access to your system so this can be something that scales really well now understand when you make connections to an external identity provider or active directory you're typically using something called security assertion markup language or saml 2.0 and that's an industry standard language to connect to other identity providers and that's the way it would work with the aws cloud is the way it would also work with google cloud so one of the benefits of identity federations is it facilitates single sign-on so a single sign-on all that really happens is you sign on once to your to a system somewhere like your active directory controllers and all of a sudden now you don't need to sign on to receive access to aws resources inside your vpc so let's talk about something very briefly it's more of a topic for the professional course but i'd like you to understand it anyway aws has something called cognito and cognito is a service that allows mobile devices to authenticate to the aws cloud and realistically speaking the way this type of authentication works whether it be anything in the identity federations or cognito basically you authenticate to some external identity provider and you're then given a temporary token and that temporary token is used by aws and anytime you can use tokens or one-time passwords you greatly enhance security because you're not dealing with passwords that if they're compromised they can use be used again to attack an organization it's one time so people or a short period of time so if a hacker for example were to obtain access to the token they can't use it for very long a little more about authentication and then we'll move on to the authorization components of iam so let's talk a little bit about the aws director service and quite simply this is just aws will host windows based active directory controllers so these controllers can be used for im for an organization or their vpc or both they're hosted in the cloud and that's it they're high availability active directory real windows servers and they host it in the cloud and these things are used great for windows type workloads now when you're going to authenticate to the system you realistically speaking have three ways you can do it you can access the system with a username or password systems can be accessed with an access key which contains a 20 character key id and a 40 character secret or systems can be accessed via an access key plus a temporary session token and they're really your three authentication options on the aws cloud as it pertains to im now let's talk about authorization and what is authorization we covered a little bit before but authorization is what are you allowed to do on the systems and authentication occurs by using the specific privileges that are associated with im users groups or roles and you do this by you write and i by writing an iam policy now i am policies are actually written in json format and this could be done via manually writing it in json format using the policy generator but there's several ways that you can create iam policies which will ultimately be in json and we'll talk about that more later understand the default policy is to deny access you're going to have to authorize users or groups or roles to be able to access anything so let's discuss authorization policies and really first what we're talking about is the effect which is basically to allow or deny and that's it and then we talk about a service and that's really about what service is being accessed and then we'll talk about resource and that's really the resource that's being accessed and you're going to be very specific and you're going to use the arn name or the amazon resource locator and when we put in a sample i am policy you'll see what we mean by this and then we're going to have an action and that's that action is really referring to things of like whether you're allowed to read or write for example and then we'll have an optional condition which is going to enable you to get very very granular with your iam policies so really what you're looking for is who or what is going to be accessing the resources and then you're going to sign them specific permissions so they can allow they can access what they need to but nothing else and you're going to specify any resource by the amazon resource locator like we talked about now when you create these there's two ways to create your policies and you can create your amazon you can use basically an amazon pre-managed policy which is pre-made for you this amazon managed policy or you can create a customer managed policy and we'll describe how you can do both so aws managed policies are standard policies and they're optimized for common use cases and they can be based upon job role for example i told you previously that the root user can do absolutely anything and a power user didn't have access to iam well there's another user class which i didn't want to talk about until we got to this section called an admin user and an admin user has access to absolutely everything including iam like the root but it's not a root account and then there's a power user account which can do everything other than iam of course there's other pre-managed policies but those two policies we'd like you to remember the admin policy can do anything and the power user can do everything but i am so how would you create a managed policy well this is quite simple you're going to sign in to the iem console and that's going to be https slash console.aws.amazon.com forward slash iam and inside of here you're going to choose your policies and you're going to see a lot of policies and these policies are going to give you tremendous options for flexibility of standard use cases so now let's discuss customer managed policies in more depth and these are policies that are created and managed by the customer for their account now these policies are not going to be visible outside the customer's organization and they're made for a customer's specific needs and you can attach these policies to entities inside of your account so let's talk about making them so if you're going to create the customer manage policy there's realistically speaking about three ways that you can do it the first way is to copy and modify an aws managed policy and this is a great way to do it take an aws standard policy modify it for your specific needs and you're much less likely to make errors by doing it this way than if you tried to do it from scratch and it's going to be much simpler unless you're an expert in json programming the next way you can do this is to use the policy generator and we'll show you what that policy generator looks like but realistically speaking it's a web page that you go to and on this web page it asks you a series of questions you input your answers and then it helps generate a policy and of course you could create a policy from scratch and if you do that that's fine but understand you really need to know what you're doing and you need to be pretty good at json programming and if you make an error here it could obviously affect things so if you're great at this great but otherwise we strongly suggest you know take an aws managed policy and then tune it for yourself or use the policy generator now here as you can see we have a sample policy and we have the effect which is to allow and then if you'll notice we have the resource and we're very specific with the actual amazon resource locator and of course we have the condition topic there as well but typical sample policy that you can look at so now that you've built your policy you're going to have to apply your policy and you're going to apply the policy to a user or a group or use like a managed policy that would attach to everyone so a user policy is for a user specifically user a managed policy is something that's managed by aws and can be applied to users or groups and then of course there's a group policy so here's really what a group policy is as you can see in this diagram we have several groups and then we put users in each group and then inside of that the users inside the group actually have access to specific resources that we've defined by their user specific policy that's of course one of the managed policies that we made specifically for our groups of users so the last iam topic we want to talk about is something called multi-factor authentication and realistically speaking you're familiar with multi-factor authentication as you've been using it in your daily lives whether you've called it multi-factor authentication or not so there's a concept in security of something you have and something you know and it provides much deeper security than just something you know like a password so let's talk about a debit card or an atm card it's something you have so let's say you want to go to a cash machine and you put your atm or debit card whatever you call it inside of the machine and then it asks you for a pin code the card is something you have that's one factor of authentication the pin code is something you know that's a second form of authentication so we're going to take that old concept and we're going to bring it into security for computing systems we're going to set up an token generation authentication application and what that's going to do is it's going to get create a one-time password and this password is going to change every several seconds because it changes every several seconds even if a hacker got the one-time password they could only use it for one authentication session and they'd have to get it within several seconds before that one time password expires so how's how it works you log in with your username and password to aws aws will then send you a challenge asking for the one-time password which is going to come from your authenticator app which could be on your phone or your hardware authenticated device you then provide the one-time password and if the one-time password matches what it's supposed to match you're now allowed into the system and therefore you will have access to resources so multi-factor authentication is a great way to enhance security by using two forms of authentication at the same time so let's briefly discuss multi-account strategies we cover this in much more depth than our certified solution architect professional course but we'd like you to still have some understanding of it and realistically speaking when you design an architecture anytime you can decouple or anytime you can make things more modular the more reliability scalability and security you have so all a multi-account strategy is is it breaking down one giant account into multiple smaller accounts and you're going to do this to be to do something called minimizing the blast radius so what would occur let's say you have four small accounts inside of your one big account if you set up your multi-account strategy correctly and you limit what traffic can go between the components of your organization then a problem that occurs in one of your part of your organization will be less likely to affect other parts of an organization so that's what we mean by limiting the blast radius and if you hear the term blast radius on an exam you know what we mean by that so it's great to chop your organization to multiple small parts but the problem becomes is doesn't aws give you a discount for volume purchasing and the answer is they do so what a multi-account strategy is enables an organization to create a big unit and they can isolate the big organization from the little mini organizational units and by doing that they can create a policy that would allow filtering of traffic in between organizational units all inside of the same master account so you're getting the volume pricing that you would get as if you had one giant unit but you're able to chop or modularize the components of your organization into smaller organizational units and isolate between them and that's really what you need to know about multi-account strategies so you always want to restrict access to your network to only allow traffic into your network that's necessary for things to work and we talked about using network access list and we'll talk about it more throughout the course to keep unwanted traffic outside of your subnet we covered that in the in the vp and the vpc section and we talked about keeping security groups on your services but what firewalls do is firewalls create a strong perimeter around the organization and firewalls block all unwanted traffic and some firewalls the way they're set up can actually be fairly intelligent and they can actually learn and look at patterns of traffic and when they see patterns of traffic they can implement new access lists or new policies to basically deny something that it views is dangerous so using firewalls to protect entry to an organization's network is a great way to to filter traffic and you're going to use firewalls you're going to use network acls and you're going to use security groups so when it comes to security you want to layer it if you were dressing up for cold weather you'd wear multiple layers and security layer after layer after layer we briefly discussed firewalls and firewalls as we described are devices that keep unwanted traffic out of your network and firewalls are stateful so basically they allow traffic to leave the network and they allow return traffic to come back automatically but you create a policy and it could be based upon an access list for example or a rule set of rule groups and the firewall will keep traffic out based upon these rule groups now aws has their own implementation of a firewall called web or web application firewall which we briefly described and waff basically looks at all the requests and based upon the rule set it'll either allow them or deny them and waff can control a cloud front distribution an api gateway or a load balancer specifically an application load balancer and it's constantly looking at the content and because it's constantly looking at the content it can stop very common malicious attacks like sql injection attacks and other things that occur but the key thing is to note is that it's very granular you can write a policy and these things are managing traffic so here's how that works you're going to enable waff on your application or device and then you're going to create a policy that filters all unwanted traffic and as traffic is coming in wife is going to or or out and return traffic coming in waff is going to really look at the traffic and is going to say does it comply with policies is there something wrong and if the traffic complies and it looks like good traffic it's going to allow it to come back so really what waff is is just another layer of protection that you can provide some firewall protection prior network acos and then prior to your security groups to enhance your security posture and if you're going to use a web application firewall you still are going to need some kind of ddos protection so the next thing we're going to talk about is ddos protection on the aws cloud let's discuss prevention of ddos attacks distributed denial of service attacks but i like to describe what they are so a distributed denial of service attack is when a hacker uses multiple systems on the internet to then attack your system and the way this really works is let's look at a web server for example and let's say when you do an http request you're going to open the session with something called the tcp syn this happens in every normal request now a web server can handle a certain number of web sessions at one time or a certain number of sims that would then be acknowledged by normal traffic and under normal circumstances the web server's not going to be overloaded assuming the web server has been sized to handle all the requests of your organization but what if a hacker attacked a thousand servers on the internet and those thousand servers constantly try to start new sessions with your web server by sending send messages send messages send messages and all these servers are doing it all at the same time now that can easily easily overwhelm your web server and take your web server down and that's called a distributed denial of service attack so how do we prevent distributed decile denial of service attacks well for one thing multiple layers so we block unwanted traffic we block it by the perimeter with a firewall we block it at the subnet with a network acl we block it at this at the server or service level with the security group we shut down unnecessary services because they could be used against us and we use ddos protection and in this case aws has a form of detox protection called aws shield now aws shield will automatically block the majority of ddos attacks that are out there in fact amazon says that the basic version or shield standard will actually block 96 of common ddos attacks so let's talk briefly about the two types of aws shield so they're shield standard and shield standard is free for anybody that's using aws waff and it protects against the majority of ddos attacks and it's based upon a policy and it gives you some pretty good basic ddos protection now there's another version called aws shield advanced and this is additional protection but you have to pay for it but with that additional protection you can apply it to elastic load balancers ec2 instances cloud front route 53 so you can apply it to many more things and this is going to look at layer 3 layer 4 and layer 7. so layer 3 is the network layer where the ip addresses would be layer 4 is the transport layer which is where you're going to have your tcp or udp and it's going to also look at the application layer so this is much more substantial protection and because you can apply it to more things and you have better policy access you're going to have much better ddos protection now one of the things is that that shield advance comes with which is fantastic is it has a 24x7 number that you can call for ddos response it may not be a number it may be a website but whatever the point is you've got access to specific experts that can actually help you that you can reach out to when you need it but in order to be able to use that you're going to have to have some kind of enterprise support or business level support with aws but the point being is aws shield is another means that you can layer on to your to your security architecture because with security it's layer upon layer upon layer upon layer in order to improve the security which will ultimately improve the availability of your organization systems the last two things i want to discuss or at least touch on briefly are the service catalog and the systems manager parameter store now we cover these in substantially more depth in the professional version of this program but it and it's really mostly tested in the professional exam having said that we occasionally see questions pop up on this we want to cover it just briefly so we'll begin with the service catalog and the service catalog is a great way to enhance your security so one of the best ways to control security is to control what you place on your network only put things that are necessary for your business on your network and make sure they are hardened patched locked down and you want to make sure that what you put in your network is secure so the service catalog enables you to do that enables you to create a list of approved services ami servers software's databases applications and servers and you allow your systems administrators to choose only from the service catalog and that way you can control what's placed on your network last thing we'll touch on is the systems manager parameter store and when you run an organization you're going to have a tremendous number of passwords you're going to have license keys you're going to have critical information that you don't want in the wrong hands so the systems manager parameter store is a secure encrypted environment for you to secure passwords and other licenses and other relevant important things in this lab what we're going to do is we're going to create a new iam user and we're going to assign them administrator positions and we're going to show you how to do it so the first thing we do from the aws management console is we're going to go to the iem section so now that we're in the iam section we want to create a user so from here what we're going to do is we're going to choose users and then we're going to add a user now in this particular situation you're going to pick a very strong username and a very strong password because whoever has access to this can specifically administrator access can do anything on the account they could do things in your own name they could be highly expensive and set up anything or they could delete instances that you have so you really want to lock down your users so in our particular case we're going to create a simple username because we want to be able to remove them as soon as we're done this lab so we're going to pick the username we're going to type cindy which happens to be my cat's name and then we're going to call it the cat pick something that makes sense and here you have the opportunity to choose whether they can use the cli the api the software development kit or whether they can access the management console so because this is going to be an administrator password we're going to come up with an administrator we're going to come up with something solid now we're going to take the auto generated password but you could create your own custom password make sure you choose a secure password so the next thing we're going to do is go to permissions and under permissions we're going to add the user to a group so let's see what kind of a group we actually have so we have a power user group access so let's create a group and we're going to call these group names admins and we're going to make sure that we select administrator access from the pre-managed policy you could create your own policy but these are pre-made policies and they typically work great for most situations so now we have an admin group with administrator access and we have the power user so now i'm going to add the user the one that we just created to the group click provide users to group we're going to click next we're going to come up with a name and we're going to call it admin group so now we have a username that has programmatic access and console access we've auto-generated their password and we haven't set their permissions boundary yet so we're going to create the user and now that we've created the user you definitely need to make sure you write down your secret access key as well as your password so i'm going to do that real quick in a normal environment this would be completely secured but we're going to need this information to log into things so the first thing that i'm going to do is i'm going to go to an application where i store things and i'm going to store the username and the password or at least the secret access key and the password and obviously you don't ever want to do this sharing a screen but we're doing this in the lab context and i have to show everyone how to do these things and then we're going to close this now we're going to take our iam user and let's verify the group that they're in they're in power user access but i want them to be in something more so let's put them in the admin group that we created which basically has full access so now we can see that our user is a member of the admin group and in this particular case since admin can do everything and power users can do everything except for iam but admin can i'm just going to remove that user from that group and there you have it now we have our user and our user is actually part of the administrator group and that's how you create a user and that's how you place them part of a group what we're going to be doing is we're going to be creating a user access key and we're going to disable the current access key and create a new user access key so what we're going to do is we're going to sign into the management console and then we're going to go to the iam section so from the im section we are going to choose users and we can pick any user but we just created cindy the cat so let's go to cindy the cat as a user and let's just uh because we created this before let's remember that it's a it's it's following they can change their password they're a member of the administrator group so they have access to everything we created a basic tag just to know what it is we can look at their security credentials and from here what we can actually do is we can create new credentials so we currently have an access key which i'm going to make inactive now we're going to create a new access key which is really not that complicated we're going to create access key and that's pretty much it we've created a new access key you're going to want to download this file because that file contains all the information you need you can see your access key as well as your secret want to close that now we have created a new access key for your iam user so in this section we're going to be covering aws applications and services for which there are many we'll cover sqs sns swf elastic mapreduce kinesis ecs eks elastic beanstalk cloudwatch cloudtrail aws config cloudfront lambda step functions cloud formation and certificate matter manager and we're going to begin this with aws sqs and high availability high scalability applications do best when you make them modular and if you segment your applications just like we talked about segmenting our organizations into manageable pieces if we segment and decouple application architectures we can dramatically improve scalability by removing system bottlenecks so aws has a solution for that and it's called the simple queuing service or sqs and it can help decouple your application architectures so what is sqs sqs is a message queuing service that provides temporary storage of messages and it's going to enhance application availability by basically giving messages a place to be stored so for example you could have a web server that's going to an application server that's going to a database and the database may be constrained so we may want to put sqs between the application servers in the database for example and that way messages from the application servers can be stored on the sqs service prior to being delivered to the database should their database be busy so it's really a transient place to store your data and the data can be stored for quite a long time in fact the default retention period is four days but you can configure this all the way to 14 days so if you've got some busy systems the sqs system can really help sqs enables you to right size your applications because without sqs you would have to scale larger because you're going to have to make sure that you can handle all the messages as they would occur but with sqs you could right-size your applications so you could use a smaller computing instance because the messages might be chaotic for a period of time and then slow all day so if you're using the queuing service you could take the messages stick them in a queue and when the database or whatever the other server is not busy you could drain the queue so sqs really facilitates a lot more scalability sqs can also facilitate auto scaling for example you could have something monitor the depth of messages in the sqsq and if it reaches a certain length then auto scale the environment so sqs is a great way to actually add some modularity and scalability into your application architectures so it's a queuing service so there's realistically speaking two ways that you can queue your message actually three but primarily two and we'll talk about all three the first and primary method is the standard queue and that basically says messages come in and messages come out now this is super fast but there's no guarantee of the order for example the first message might be a large one the second three might be small and the second and the the next three messages might be delivered before the first one and in most cases that's fine but if it's not fine for your applications if your applications need to have messages delivered in the order they were received then you're going to need to do something called the fifo queue which is first and in first out and this is a high throughput environment but it's still slower than a standard queue but it does guarantee messages are processed in the order they're received now the last type of a cue you can set up is something called a dead letter queue what a dead letter queue does is if a message never reaches its ultimate destination it can be stored in the dead letter or queue so it's actually not lost but the primary cues that you're going to be using are going to be your standard cues which is basically messages come in and messages come out it's super fast but there's no guarantee of the order or you're going to use the fifo option which is first in and first out so now that you know about sqs at least from an architectural perspective let's talk about how it works so a message is going to be sent from the computing platform to the queue and as soon as the message is in the queue it can be scheduled for delivery now if the ultimate destination where you the messages will be delivered is busy the messages can stay in the queue until they time out now i had mentioned the default time is four days but it can be configured for up to 14 days so you can store your messages for a relatively long time now when the message is deliverable meaning the recipient is ready for it the message is pulled from the queue and then after the message is pulled from the computer queue it's deleted from the queue and that way you can pretty much guarantee that you have a place from the messages to be stored until they're delivered to their ultimate destination now we're going to discuss the amazon simple notification service or sns an sns is a messaging service that's used to deliver messages between systems and it's also a great way to decouple messages between microservice based applications and you're going to use this to send an sms an email or push messages to mobile devices so the way this works is there's publishers and subscribers publishers communicate by sending messages to a topic and the subscriber subscribes to the topic and then receives the messages kind of like a mailing list you enter your information to the mailing list and every time they send you receive it so we'll talk a little bit about the sns platform functionality it's a high availability platform that by default is actually run across multiple availability zones and because it's in multiple availability zones it's going to be secure and reliable in case the data center were to go out and sns is often used to fan out messages so if you've got a large number of subscriber systems or computer endpoints so sns will enable you to create like a filter policy so that only people that need or systems that need to receive a notification will and sns will encrypt your messages immediately to protect from unauthorized access so where are you going to use sns you're going to use it for application and systems alerts so an sns can send you a notification when a predefined event occurs so your cpu utilization hits 80 percent it'll notify systems admins it'll send email and test text messages so these messages can be sent to you for example if a company's ceo is going to be on tv it could even send a message to everybody in the organization watch this and the concept of push messages so sns can send a push message or a notification directly to moto mobile applications and you've seen this with applications you've got an application all of a sudden it sends you an alert and what happens is the alert is a push message that basically tells you what you need to access so sns is the amazon simple notification service that uses a messaging service now we're going to discuss sws and sws is a workflow management solution and it really enables the coordination of different tasks across distributed applications so for example if you've got a multi-step workflow swf is the solution to help you do that because what it does is it coordinates the execution of tasks across an application so by using swf it kind of mitigates the need to code and coordinate these tasks on your own because it's a pre-built workflow management solution so all that's really necessary is to tell swf the steps and it'll manage the workflow so let's take a sample workflow it's pretty common that organizations have video and they do some transcription and other processing to the video prior to actually utilizing it so let's say we were to make a simple swf workflow for a common video solution let's say i want to upload a video but i want the video to be processed so let's say i update raw video and i want that to be converted into a format that's efficient for web distribution so i can have i can first upload the video i can create a second step to have the video processed and converted into like an mp4 file with an h.264 or five codec to make it a much smaller file and then after the video is formatted i might want it transcribed but i might want those transcriptions to be placed as burned-in captions to the video and then after that after all the work's been done to this video i want it to be stored somewhere and then i want to actually be notified hey by the way mr customer video is ready for download and this is the kind of application that you can do with swf and therefore we tend to love it now a little bit about elastic mapreduce it's much more of a topic in the professional example we just want to touch it just so you know what it is and it's an application for processing large amounts of data so the elastic mapreduce or emr is just an application for processing tremendous amounts of data and it's really a managed cluster and it's really for big data frameworks and it's been built on open source tools such as apache spark apache hive apache a-space apache fink and a few others and it offers some fairly high performance compared to some other solutions and it's a great way to actually deal with large amounts of data and you don't have to code it because it's basically pre-made there's not the expenses that are required with heavy management or heavy coding now we're going to discuss amazon kinesis and kinesis is a big topic and it's heavily covered in aws exams so we're going to spend some time here and kinesis is an aws service for collecting processing and streaming real-time data and kinesis can collect this data from a lot of sources such as video audio application logs website click streams internet of things and many other devices and unlike a traditional environment where you're going to take all this data you're going to store it and at some point analyze it with kinesis you've got some type of real-time analysis that you can perform to make better business decisions in real time so there's four versions of kinesis and one is video streams another is data streams then we have data fire hose and we're going to have data analytics and we're going to start with video streams video streams is basically just a service to collect and aggregate data from multiple places and it can be then ingested sent into storage and then indexed so really kinesis video streams is about capturing video now kinesis data streams is pretty important and it's a highly scalable platform to collect real-time data and this can capture gigabytes a second of data and it can capture it from hundreds and thousands of sources so it's a great way to collect data so think of a financial transaction application for a trading organization how they have millions and millions of trades being placed at any one time this is a great way to capture this type of data so what happens is the streaming data is captured by kinesis and then it's sent for processing and if the data is processed it can quickly be input into business intelligence tools and but you can also in some way shape or form use these kinesis streams to do some more analysis and we'll talk about that later but understand for right now we're talking about collecting data from a lot of sources and then being able to ingest it and process it in real time so some use cases would be you've got a large event that you're trying to collect data more real-time analytics capturing gaming data from multiple game players mobile data coming from all different sources but it's really about capturing streaming live data now we mentioned kinesis data streams is for collecting streaming data now kinesis data fire hose is really taking streaming data and loading it somewhere so kinesis data fire hose is a managed service to load streaming data into like a data store a data lake or an analytic service often people will use kinesis to capture the streaming data and put it into s3 and often they'll put it into redshift as well so they've got that concept of big data applications they can use it for and it's a fully managed service and it scales to match the needs of your data through auto scaling and monitoring now throughput is actually based on something called the shard and a shard is actually one megabit per second and when you use this service you pay for it based upon the number of shards that you need now prior to the obviously prior to use the organization is going to try and guess the number of shards they need because if you're going to provision your capacity and you're going to pay for your capacity you would like to know ahead of time what you're going to use now generally speaking you can create up to 10 different charts and you can go beyond that but you're going to have to call aws first their service and they're going to have to authorize you because you only get 10 standard shards which is still 10 megabit a second of streaming data which is really quite a lot and you can set up a policy to auto scale up the throughput in terms of the number of shards and you set kinesis data fire hose up through the console it's essentially just setting up the sources and destinations of the data now we'll discuss kinesis data analytics and kinesis data analytics is a managed service to transform and analyze real-time streaming data so kinesis data analytics uses the built-in apache finc application in order to process the data so kinesis streams is auto scaling and will definitely meet the needs of an organization if it scales up and data on kinesis streams can be queried with standard sql queries so this is really a way to take your real-time data and analyze it in real time so people often use this they'll basically take their streams they'll do a quick analysis and then they'll pump it into business intelligence tools to make long-term better business decisions now we're going to be discussing the amazon elastic container service and i'd like to begin this discussion with what is a container so when container is a modern version of a virtual machine and it's very lightweight because what happens with a virtual machine you need to each virtual machine needs an entire operating system then it's going to need the application dependencies and then it's going to need the application and a container is a much lighter version basically what it is each container is going to have like a little mini version of the operating system and that's really going to be system libraries and dependencies and such and then you put the application but there's still logically separated from the other applications so now you can have one system and the system can help multiple containers and each container is almost as if it's like a virtual machine for just the application so they're higher performance they're much lower system resources because look at it from an architectural perspective with the virtual machine you have your bare metal hardware then you have your hypervisor and after that you're going to have a virtual machine which is basically going to have an operating system the libraries and dependencies you need and then your application whereas the container is going to be a server the operating system and a little container which is just those things that are necessary to make the application work understand when you're dealing with containers if you're going to use a windows system windows containers can do windows applications and if you're going to use a linux system the containers can do windows up linux applications now so let's talk a little bit more about the container service so it is a very high availability serv service to manage containers wherever they would be on your vpc and we'll talk about the locations but it's a four nines available container service and it's deployed in a vpc which gives you the ability to use any knuckles and and security groups you've got some good security you can use with it now it's a management service for your containers and you can host your containers in one of two locations you can host it on an ec2 instance and the containers themselves will be managed by the container service or you can host it on something called aws fargate and fargate is a serverless platform and one of the things that's really great about using a service like fargate is if you put your container on a fargate location and you need more power it'll just scale up in an unlimited manner because it's serverless so whatever resources it needs because it's fargate it's going to be much easier to manage because you don't have to manage the server then you don't have to manage patching the operating system then you have to manage passing all the applications all you're going to have to do is worry about your application you may have to patch your own application but you're not going to be dealing with the operating system and you're not going to be dealing with anything else it's going to all be managed and all you need to do is just update your container so it's going to be much simpler to manage and that scalability component is great so because you can run your containers either on an ec2 instance or the serverless environment you know when would you choose one each other generally speaking most people are probably going to do better with far bait fargate because of its scalability and its simplicity having said that if you need complete total autonomy over the systems that house your containers choose ec2 because it's your own systems and therefore you can manage it but there are the two options to use the elastic container service since we're discussing containers we'll also discuss the elastic kubernetes service eks and realistically speaking eks is a fully managed kubernetes container service and kubernetes is just an open source container management platform and for the most part you could almost call it the industry standard because most of the containers are actually using this service it's very popular but realistically speaking it's just an open source version of it as opposed to ec2 and like i'm sorry ecs and like ecs the eks service can manage containers on ec2 instances or fargate whichever you choose let's briefly discuss elastic beanstalk an elastic beanstalk is a service to provision and deploy scalable web applications and services and it's quite easy to use realistically speaking all you do is you upload your code and then the elastic bean stock will automatically deploy your infrastructure whether it be ec2 containers load balances and the infrastructure that's deployed is actually auto scaling by default and it's also load balance by default so it pretty much sets you up for a great infrastructure by just putting your code in and it supports you know multiple programming languages it works with uh go java dot net node.js php python and ruby so it's pretty flexible with regards to the code that you're actually uploading and the good part of it is that it actually provisions but it also manages the environment and it's watching to see what's going on with the health of your applications and you can get your logs with this because it integrates the cloud watch logs for performance monitoring let's discuss cloudwatch cloudwatch is a monitoring service to monitor what's going on in your vpc and it can monitor applications that you deploy as well as other aws resources and what happens is it will provide metrics and these metrics can not only help you monitor performance but it will also help you troubleshoot issues when they occur and cloudwatch has two kinds of metrics built-in metrics and custom metrics now built-in metrics are the default metrics give you things like cpu utilization disk read and write and network utilization now you can set up a custom cloud watch metric if you need to know something more for example you might want to monitor memory utilization or api performance and when you set up a cloud watch custom metric and can be very useful to understanding what's going on in your systems and making changes to make them run even better cloudwatch has a has an event type of a notification as well that can be set up with thresholds and when something happens past a threshold you could get a notification for example or you could set off a lambda function to change something so cloudwatch events really gives you a really good idea of what's going on but at the same time it helps you mitigate things whether it's a lambda function in sns notification triggering auto scaling so cloudwatch is really an integral part of your service when you're using the aws cloud now there's two versions of cloudwatch there's basic monitoring which gives you data automatically every five minutes and it's included at no charge now if you need more monitoring than that there's an option for detailed monitoring and this provides data every minute but you pay additionally for this and you also have to enable detail monitoring on the ec2 instance level now we're going to discuss cloudtrail which is an auditing service again i'm going to mention again cloudtrail is an auditing service so what cloudtrail does is it provides an audit log of the things that actually occurred and it will tell you what changes were made by what user account or what role or what service so it gives you good information what goes on and you know cloudtrail is enabled when the account is created created but to start using cloudtrail what you're going to do is you're going to create a trail on on the console the cli or through the api now you're going to get an event history that lets you see events that have occurred over the last 90 days and you can configure cloudtrail to store logs in s3 should you need to keep it longer than that so when you make a cloud trail you basically have two trails so you have a cloud trail that's local to one region and you can make one that applies to all region so when you need a simple trial with less features and functions realistically speaking you're going to use the local cloud trail local but if you need much more sophisticated options you're going to use a cloud trail that applies to all regions now we're going to be discussing aws config an aws config is a service that enables assessment auditing and evaluation of configurations you have in aws so it really gives you and i a good idea to see what's actually going on and i say this because you may think you know about everything that occurs inside of a network but every once in a while things pop up configurations pop up devices pop up that you just don't know about it until you look so this will help you see a change inside of your vpc because what can happen is an s s alert can be sent to all the systems admins as soon as something occurs so this is going to provide constant monitoring of your infrastructure and what's going to happen is as a new device comes online it's going to say does this meet the organizational standards does this meet best practices and if the answer is no it can give you the opportunity to do something about it so it's really a great way to find out what's going on and then determine what you actually need to do about it so think of it this way it begins in the following structure a configuration change is made aws config notes it and says okay i see it and then it records it and then it's going to check what's there and see does it conform to organizational policies and if it does that's fantastic and if it does not it's going to notify a systems administrator so this is a great way to know what's going on this next section is on cloudfront and cloudfront is the amazon branded content delivery network and what content delivery networks do is they bring content closer to the users and if the content is closer to the users and it rides over a higher speed network which they typically do on content delivery networks the user experience has improved so what happens is cloudfront is in a it has a network of caching servers that are also at edger locations throughout the entire world and when a request is made for a web page the requests location is determined and they get sent to the closest cloudfront server and these servers are spread throughout the world additionally the cloudfront servers actually cache the content so here's what typically happens a user makes a web request which gets sent to cloudfront if it's a new web request and no one in that region had ever requested the information before what will happen is the web request will go to the web server now the web server will then send that data back to the cloud front location and the cloud front location will deliver it back to the user now the next user that's in that area that's going to use the same cloudfront location will have that frequently accessed information from me available to them so their web request will go from them to cloudfront and if it's the same thing i request it'll get sent right back to them so that can make performance much faster because you're bringing it closer and because you don't have the latency of the distance of the of the packets that need to travel across the network but caching web servers can make them much more scalable and the reason they can make them more scalable is a lot of the content that people request from web servers is typically the same content and if it's cached the content will be coming from the cache servers as opposed to the organization's web server so that will enable many less web requests to go to the web server which will be very helpful in terms of server performance now cloudfront integrates with a lot of aws services specifically s3 ec2 elastic load balancers in route 53 and cloudfront is very frequently used as a front for a static website that's stored on s3 but you can still use cloudfront in front of an ec2 based website provided an elastic load balancer as part of the architecture so let's just summarize cloudfront can really help with performance because it helps with caching there's 217 points of presence at the time we made this course spread throughout the world so this can really get content much closer to the user it improves routing efficiency because instead of data being sent over the internet it can be sent over the aws high speed backbone which is typically faster cloudfront can also substantially help protect an organization from a ddos attack especially when web application firewall and aws shield are enabled but the reason cloudfront can help so much with the ddos attack is first it'll distribute content throughout the 217 points of presence throughout the world at least at the time of this filming and cloudfront is only going to forward legitimate traffic to the actual web server so typically you would see in a ddos attack illegitimate traffic that's sent to the web server and it's flooded but now you're dividing it among 217 points of presence and only the legitimate request will be sent to the server so this can really enhance the availability of a web application because it can prevent against ddos but also improve the performance through caching cloudfront can also provide some encryption in transit for data it can work with and enforce the ssl tls based protocols and it integrates nicely with the certificate manager for certificates now you can tune cloudfront for example because it is a caching system if you change your data quite frequently you might want to cache the timeout sooner so you don't have stale data and you can set the minimum the maximum the default ttl or time to live which is the expiration time of data in the cache and you can clear the cache if necessary from the cli for example if if you have stale data in the caching you need to clear it can definitely be cleared through the cli and we have a command that you can see that'll be we'll post in the video that will show you how to do that now we'll discuss amazon lambda and lambda is a way that you automate simple functions across an organization's infrastructure and lambda is a serverless computing service and it works by simply you write your code you update your code for the lambda function and that's it it just runs and lambda enables you to write your code in c sharp go java node.js in python so most of your scripting type languages one thing to know is that lambda is stateless so you set up a lambda function once the lambda function is finished it's done if you want something else to occur you're going to have to set up a second lambda function and lambda can be used when you want to automate things processing data across multiple systems remediation of security events patching of operating systems so it's a good way to do some automation and let's walk you through an example of how this can work let's say you upload a new file into an s3 bucket and in this case it's going to be a video file that needs processing we can have a set up a lambda function so that when s3 detects the new video file send it somewhere else or do something with house with it or activate another symptom or another system so that's typically the way you would use a lambda function now there's a way you can bring lambda functions closer to the user and that's called lambda edge what lambda edge does it works with the cloudfront content delivery network and this enables small lambda functions to be run very close to the customer so we've already discussed lambda so now we're going to discuss step functions and what step functions are is there really no means to just sequence multiple lambda functions together and like lambda it's completely serverless but it's really about just scheduling multiple lambda functions so here's how it's going to work you're going to design the steps of the application they'll then create your individual lambda functions and you'll configure the workflow with step functions and you'll connect the workflow components to the individual tasks and then just execute the step function so basically what'll happen is step one will occur then a lambda function will occur when lambda function one is completed step two will invoke another lambda function and then when that function is completed step three will invoke another lambda function and so on and so forth until all the necessary steps of the workflow have been completed now we'll briefly discuss cloud formation and cloud formation is really a means to template known good configurations so you can repeat using them and let's say for example you have an application and it's a common application organization that requires the configuration of several servers specific patches specific dependencies in order to work and if you miss any of those steps or you do any of those steps incorrectly the performance might not be the same as you need it to be or the security might not be good well with cloud formation basically you're going to create a template and and and the administrators can then invoke that cloud formation template and they're going to get the same results every time so it's always good to template your your configurations whether they be in a format that's even a text file but at least a means to get to the same place and cloud formation is a great way to do it so you're going to write a cloud formation template in either json or yaml file and you store the content on s3 and basically what happens when you want to actually build something you're going to pull that template from s3 and cloud formation is then going to provision your services for you exactly the way you want it to be based upon the template the aws certificate manager is a service to essentially manage your ssl or tls certificates in the cloud and it's going to add protection to a website by using a certificate to establish safe and secure connections and really this is just a service to simplify provisioning management and deployment of your certificates whether they be public or private so the certificate manager provides free and private certificates to aws services like a load balancer or an api gateway they're realistically speaking two options when deploying your certificates there's the standard aws certificate manager and this is for customers who want encryption and security using tls there's also a version called private certificate authority or private ca and this is really for making private certificates to really secure an organization's internal communications but these are private certificates not for a public website and these private certificates are available at an additional cost we'll be doing is we're going to be setting up cloud watch now we're going to specifically set up a dashboard remember from the lectures and this is very important going into the exam that you remember cloudwatch is used for logging and cloudtrail is used for auditing so in this case we're going to be setting up some widgets so we can look at certain things related to the logging functionality so we're going to go from the management console to cloudwatch under management and governance we're going to click this section under cloudwatch and then we're going to create ourselves our own custom dashboard so we're going to go to the dashboard section and we're going to create one and we're going to call it whatever we want to but i'm going to call it gca dashboard and from here we're going to figure out what we want to monitor so we're going to monitor some metrics and to be fair we're not going to see many metrics on my cloudwatch logs because i keep a lab environment i spin up instances as i need them i shut them down when they're not needed at least for our lab environment and because of that we're not going to have a lot of data but we're going to set it up anyway so you know how to set up a widget and add it to the dashboard so here you have the option to pick which metrics you're interested in you could look at logs or metrics so we're going to pick metrics and when we get here we can find the kind of metrics that we're interested in for example we can go to ec2 and we can look at per instance metrics and we can pick out what we're really interested in looking at so we could look at say cpu utilization because we care about that we can look at disk input and output and see what's going on with our disks we can see uh what's going on with our network we can pretty much pick any of the things that we need and we can look for some serious problems such as system failures and we can create our widget and that's pretty much all there is we've created a dashboard and now you can look and see and when we have more information coming in you'll be able to see exactly what's going on with disk reads disk writes every five minutes that's pretty much all you have to do to set up dashboard under cloudwatch and set up a widget to give yourself easy access to the information you want to monitor setting up cloudtrial as you recall from the lecture cloudwatch is used for logging and cloudtrail is used for auditing so we're going to begin by setting up our cloudtrail by going to the management console in the management console we're going to go to the section where it says management and governance and we're going to click cloudtrail so when we get to this page we're going to create a trail and we're going to click create a trail now we can come up with a name i had cut and pasted in the management events right before this to make sure we had a name that would be simple to use and we're going to create this trail so what you can see here now is we have a trail called management events and we're going to be collecting the information and so the trail has been created now let's see where the trail is being stored because typically we want to store something in a bucket of some kind now in this particular case we could have created our bucket but we can use the bucket that was explicitly made for us so we'll click on the budget good and there's no objects in the cloud trail because we haven't done anything yet that's really it that's about how you set up a cloud trail and you'll go back to your cloud trail to be able to see what's actually going on with regards to who did what on your systems we had set up some cloudtrail and i had mentioned that some of the logs were not available at the time because we hadn't done anything yet but now there should be so in this lab what we're going to be doing is we're going to be taking cloudtrail and actually integrating it into cloudwatch logs so the first thing we're going to do is we're going to go check on our trail and we're going to do that by going to the management and governance area of the management console and clicking on cloud trial from here what you'll be able to see is in the previous lab we had we had set this up but now there's things you can see i've deleted some things i've placed some bucket policies you can see everything that's going on but let's go back to our cloudtrail dashboard we can look at what's going on here and let's actually go to our specific trail go back to the dashboard we'll click on the name of the trail management events so here what you can see is generally what's going on it tells us where our cloud travel location is you can tell you when the last log file is disa was delivered i chose not to use encryption because i wanted to make this as simple as possible but now we're going to set up cloudwatch logs so we're going to click enabled and now we're in a position to generate cloudwatch logs now we're going to create either a new log group or an existing log group come up with some names letters numbers something that's quite easy and we're going to have to create an iam role remember iam users are for users to access a system but iam roles are for systems to access the system so what we're going to do is we're going to create a name of a role and you can pick something that is going to make sense to you i am going to pick cloud trail underscore gca for code cloud architects underscore mic i just want something that's going to be easy for me to remember now you can look at more under the policy document and that will show you what's actually going on from the iam roles that it's allowing access in to cloud watch from cloudtrail we're going to save the changes and you know with some of these these actions it takes a few seconds or so but at this point as soon as this finished we should have completely integrated our cloud trail into cloudwatch logs so we'll give it a few more seconds and now we can verify things so we're going to go back to the management console and under the management console now let's go to cloudwatch now while we're in cloudwatch we're going to be able to see certain things that are going on we're going to have some dashboards we're going to have log groups and you can see this is the group we just created and in here we'll be able to look at the log stream and you can see what the recent events were so that's it that's exactly how you would take a cloud trail and integrate it into cloudwatch logs we're going to be setting up a cloudfront distribution if you recall cloudfront is the aws content delivery network and what the content delivery network does is it brings content much closer to your customers and it brings the content closer to your customers through the aws network background backbone which in many cases is going to be faster than the internet because aws can create some service level guarantees on their network but it's impossible on the internet to create any kind of service level guarantees i mean you can guarantee your connection to the internet but once your data gets on the internet it's going to be routed through so many organizations and internet service providers it's impossible to actually guarantee any level of performance so it'll bring your content closer to the customers but not only would bring your content closer to your customers it's going to cache your data so it's going to store frequently accessed data closer to the customer which is going to mean less content needs to actually traverse the network background backbone which is going to reduce your data transfer costs and at the same time it's going to improve the performance for your customers so you can set up cloudfront to work in front of an elastic load balancer with ec2 web servers or you can use it word in front of an s3 enabled bucket and so let's say we created a website which we already have on s3 and we've showed you how to do that in a previous lab so now what we're going to do is we're going to create a cloudfront distribution to our s3 bucket now we're going to click cloudfront and we're going to create a distribution and this is going to be pretty easy so for the origin domain and we're going to set this up straight out of the s3 buckets if you had set up a fairly complicated s3 route 53 routing policy you could with custom domains what we're going to do just because it's allowed the basic infrastructure to set up a cloud front distribution for your s3 bucket so for the origin domain name you're going to pick your bucket and it's going to give you the origin id it's going to give you connection attempts it's going to tell you the cash behavior do you want it to cash what do you want it to cache and under price class you can have different you can choose different options whether you want it to cash in all edge locations or just certain environments you can set up aws waff acls directly on the cloud front distribution which is fantastic because you'll be able to block bad content at the location of cloudfront prior to look you're even getting on your network so your web servers will never have to see or in this case your s3 bucket and your wan will never have to see the bad traffic that would be associated with this and if you would set it up with it with a load balancer and an ec2 enabled website those servers would never actually see the bad traffic so it's going to be blocked directly with cloudfront which is going to substantially enhance your distributed denial of service protection as well as improve your performance and then we're going to just create our distribution and that's pretty much it in order to set this up we've already set up a cloud front distribution and it's going to point to the s3 bucket and you can see that it's up and it's running and you can see the origins of where they're there now realistically speaking that's all we're going to do in this lab now if you were to set this up for a production web environment you're going to go to route 53 and you're going to create a record that's actually going to point to the cloudfront distribution instead of where your website would actually be which is the s3 bucket or your ec2 instances behind an elastic load bar so that's it that's how you're going to set up cloudfront to work with an s3 enabled website this next section is about cost management when an organization chooses to move their data center to a cloud computing environment it can have a profound effect on the cost of running an organization's technology now in most situations the total cost of ownership will be less in a cloud computing environment than a traditional data center but not always but one thing is for sure when you migrate to the cloud the cost structure changes and it changes dramatically so cloud migrations will shift expenses from predominantly capital expenses also known as capex to operational expenses called opex and in a traditional data center you have incredibly high capital costs and the reason is you have to buy the servers the routers the switches the firewalls load balancers racks power whether that be pdus ups generators transformers and then there's data center cooling and then of course there's the property or the real estate associated with it so this is heavy duty capital expenses and there's a reasonable amount of operational expenses with the data center and that's realistically speaking the large staff the electric bills from both powering the servers and cooling them and the connectivity of the lan but most of the costs are actually for an organizational equipment and its capital costs and the remaining costs are for operating the equipment now with the cloud there's going to be a big shift to operational costs and we're going to talk about that in a moment so while with a data center the costs are very high because there's a lot to purchase and a cloud computing environment there's really nothing to purchase or almost nothing but the organization pays for the cloud computing environment every time it's used so this means that ongoing expenses in a cloud computing environment are going to be very expensive now in most cases the operational expenses of a cloud computing environment are still cheaper than it would be for the organization to buy the equipment run their operating systems meaning power cooling all the other things that are associated with it and paying for the the networking and the wan connections but not always but understand with the cloud environment it's going to be this operational expense because you could be paying for instances by the second for example and every little bit adds up so what we're going to talk about now is how we actually manage cost to the best of our abilities in a cloud computing environment to make sure the move to the cloud is financially viable for organizations as they start using cloud computing keeping costs low or at least as low as feasible is absolutely critical for a successful cloud computing deployment and the first step is to provision only the resources that you need so don't provision things that you don't need and i also want you to monitor your system see what's going on with the cpu see what's going on with disk performance look at your systems because you might have systems that are bigger than they need to be or you have systems that are smaller than they need to be so pick the optimal systems you're going to do this by monitoring your systems so now that you know what your needs are step two is going to be to properly size your resources size your equipment based upon average usage and not peak usage in a non-cloud computing environment you have to plan for the statistical outliers what happens under peak conditions but in a cloud computing environment you don't need to do that you can plan on whatever your normal usages are and you can use auto scaling to scale up as needed along with this anytime you can decouple your systems and your architecture with things like sqs you'll be in a much better position to use smaller systems that'll even scale better so the next step is to purchase the right platform and what i mean by that there's going to be times where on demand is best times where spot instances are best and times where reserved instances are best but the better you know your systems and the better you're able to plan and the better you monitor them you'll be able to make better decisions but on demand are going to be the most expensive but they're perfect if you don't know the application's requirements spot instances are really going to be the cheapest but they can be preempted or terminated should someone outbid you so this may be a great way to get a lot of work done if your application can tolerate a batch job being shut off and turn back on at a later time for example and if you know you have an application that's going to be up and what its utilization is going to be and it's going to be there for a long time and you can purchase a reserved instance you'll be able to get a discount and if you can purchase a reserved instance for three years instead of one year that's fantastic as well so your cost optimization is going to be achieved by a mix of on-demand reserved instances and spot instances step four whenever possible leverage managed services are serverless because they're going to be a lot easier to maintain and for most people they're going to be a lot cheaper and step five is really about managing data transfer cost so if you have an s3 website that's hosted in multiple locations you're going to pay for the the inner region transfer of your s3 data but it might be cheaper to use s3 cross region replication so you can have something say in the u.s and something and say asia that might be cheaper use cloudfront so you can serve content locally because that will decrease the data transfer charges because and think about the type of whether you want a direct connection or a vpn typically speaking a vpn is cheaper but data in and out of a vpc over a vpn can get expensive if there's a lot of it and in some cases it may actually be cheaper to use a direct connection so this is really about planning and understanding your applications understanding your use and knowing how to best plan for the future the service that can help an organization keep to their budgetary requirements is called aws budgets and aws budgets basically helps an organization make sure they don't overspend and it works very simply you create a budget in the aws management console or the billing console and in that budget you can set some custom alerts and if the organization gets close to exceeding their threshold their budget threshold an alert is sent so this can tell you that maybe you need to look into a different instance type perhaps maybe you no longer need on-demand maybe you need to move up to a reserved instance but you can see what's going on with your data and it can help you plan now the next service that can help an organization keep their costs low is something called the trusted advisor and the trusted advisor is a tool that helps organization optimize their spending and what happens is the trusted advisor actually will scan the organization's infrastructure and it'll compare it to best practices and it's going to provide recommendations and in many cases these recommendations can not only help an organization have a better more available more secure network but they can also help an organization lower costs now there's two versions of trusted advisor there's a basic and developer support plan which has some limited advisor benefits and organizations that have the business support plans or aws enterprise plans get a lot more notifications and advice but realistically speaking these are two applications that can help an organization reduce their costs of the aws cloud let's discuss building high availability systems and availability refers to the service being available for use when you need it so a high availability system means that it's highly likely that the systems are going to be available for you when you need them now designing for availability specifically high availability can become relatively complicated and costly based upon the availability requirements most organizations will consider high availability to be 99.99 or greater which is referred to as four nines now some organizations are going to need much more availability than that for example a service provider a bank or a health organization for which if their network or their systems went down then someone could die if they're dependent upon technology so you're going to design your systems based upon that now a four nines available network meaning 99.99 of the time will have about 52 minutes of downtime more specifically 52.6 minutes now organizations that need five nines or better will have meaning 99.999 percent of the time will be limited to basically 5.25 minutes of downtime per year now building high availability networks like this takes a lot of work it takes work with change management it means no single points of failure it means security and we're going to talk a lot about how to build these systems the first tenet of building a high availability system is you can't have any single points of failure and that means redundant power redundant cooling redundant network connections redundant routers redundant switches low balancers firewalls dns systems storage even your applications have to be redundant so this is what we refer to a as a high availability system but when i said no single points of failure that may mean that you need multiple availability zones and that'll get you to four nines availability but if you need truly high availability meaning 99.999 or greater you may have to use multiple availability zones on multiple regions as well so you can always build a high availability system but the architecture can get very expensive so you need to know what your availability requirements are and how to get there now it's going to be much easier to build a high availability system on the cloud computing environment than it would be in a traditional data center and here's the reason aws maintains redundant power they've got redundant cooling in their data center they have redundant connections to both the internet and across their backbone they've got redundant routers and switches so it's going to be much easier to do this with aws plus they also have a highly sealed scaff that can maintain the underlying infrastructure and they also have a change management process to make sure that changes are made in a time that will have less effects on others so let's talk about some best practices when designing your high availability systems multi-az so since every availability zone is effectively a data center you're going to want to use multi a a z any time you want to talk about high availability specifically 99.99 of the time or greater now if you need greater than that greater than the four nines availability you need to get into that five nines range which is about five minutes of downtime each year like we described you're going to have to use multi-region and multi-availability zone so anything that needs high availability you have to just think redundant so if you've got ec2 compute instances they're going to be in multiple azs if you've got databases multi-azs load balancers multi-azs route 53 same thing and if you need to get to that five nines or greater you're going to even have to do some of these things in multiple regions so we're talking about multiple network connections and this means if you're connecting to the aws cloud their network is redundant but if you're going to connect to them you're going to need at least a primary connection and a backup connection so for most organizations that's going to look like a direct connection and a vpn backup but some organizations have critical bandwidth requirements and they may have multiple direct connections to meet their bandwidth requirements but guess what when you have multiple direct connections if one of them goes you still have multiple backups so the point being is you're going to have redundant connectivity whether it be a single direct connection and a vpn backup multiple direct connections in a vpn backup you're always going to have to have some form of backup now if you had a vpn as your primary thing you can conceivably have two internet service providers and two vpns set up but it goes back to that adage that one is none and two is one so you never want to have a single point of failure and anything that matters you need to have redundancy so one is never enough now let's discuss some other critical elements that go into building a high availability system and one is security if your systems are compromised your systems will not be available so let's review a few concepts we want you to use the principle of least privilege meaning grant only the minimum access necessary for people to do their jobs on any of your systems disable unnecessary services unnecessary services can be used against you to help a hacker gain access to your system we talk about using aws organizations to limit the blast radio should something go wrong and you want to keep unwanted traffic out with firewalls with network acls and security groups use amazon waff use as a firewall use aws shield for ddos protection use ids ips for intrusion detection and prevention make sure that access that you of your router that connects to the aws data center is secured in a locked environment so no one can gain access to that because if they can gain access to your network they can hack you if and when passwords are required make sure you're using strong passwords and template known good configurations with cloudformation templates so you can build things that you know work now an organization's data in many cases this is lifeblood so backup backup backup and consistence backups will protect against lost data but create images of your production servers and that way that if your data is lost and your servers are lost you can you have access to your data and your servers you can relaunch them in another region shouldn't he be another availability zone or whatever you need to but you can do it quickly and backup should be stored in at least one alternative and secure location it's no good to secure backups in a building and if the building catches fire you lose your backups another way you can add security is by using auto scaling and this is about performance because look at it this way if you have a distributed denial of service attack and they're trying to take your system offline and the cpu is 80 but it can scale up to 10 more instances it's going to be really hard to take all those systems down where it will be much easier to take one down and anytime you can decouple your application architectures you're going to have better performance and it's also going to protect you caching can do wonderful things to improve performance it can do this because it's going to offload your systems so not only will less requests go to the systems which will make the cpu lower but caching can also help prevent adverse security incidents like a ddos attack dns this gives you great options to optimize system performance but it also can use things like health checks to make sure that it can improve the availability of your system by only allowing servers and services that are functioning to be available for use and removes everything else from the rotation and the same goes with load balancers if you're going to have 10 web servers you're going to need something to balance that load and the load balancer is the ideal way to do it now part of a high availability system is monitoring your systems so you want to monitor them for system health but you also want to make sure they're able to meet up with demands so use logging functions from cloudwatch and use auditing functions from cloudtrail and really look for system alerts and you're going to really want to monitor for security breaches because if hackers gained access to your network and you haven't found it they can do damage so you it's really about monitoring monitor for usage monitor for performance know what's on your systems monitor them and then optimize them as necessary another critical component to high availability is something called change management and what this really means is before you make any changes across an organization's systems make sure all stakeholders are notified and that they agree for example you don't want to reboot a system that's in the middle of a batch job and you also don't want to be doing a reboot of a system or a patching of a system at a time where someone else is doing something else and if you both reboot at the same time and you cause a problem you don't know which is which so you have to be very careful and diligent about what's being changed when it's changed and how it's changed and change management is an absolute requirement for high availability systems but it's not just making sure everybody's there it's finding a time so is it three o'clock in the morning when the systems are most idle when are the systems not being used and that's the time to make changes on the system assuming it's okay with all other key stakeholders in organization now there we've covered the key components of building a high availability system on the aws cloud let's discuss passing the exam so you've completed this class we've given you a tremendous amount of content one of the things to know about aws exams is they can ask a variety of questions on pretty much any topic on the curriculum and these questions can be a bit challenging to understand in fact we find the hardest part of aws exams to actually read the questions they can just be so wordy and when you read the answers there may not be a clear answer that's the best one especially if you've been around in tech for a long period of time because they may give you four options and of the four options three may be acceptable no two may not be the amazon way but they're still acceptable and one may be the amazon way so don't overthink these questions select the best answer so our recommendations are to really go through this at least once and go back to any sections that you find challenging now we also want you to read the aws white papers that are there and the reason we recommend reading the aws white papers is not only going to give you some additional knowledge but it's in aws's words and we find that aws often pulls test questions directly out of white papers so now that you understand all of the tech that's involved reading the white papers are not only going to be easy to read but we think you can get some additional questions answered on the exam as well as deepen your knowledge now we love and hate practice does we find that people that use practice tests as a mean to study don't understand the content much at all and i can tell you in years of interviewing technology professionals i can tell you someone who passed with a practicing test because they have lots of certifications when i ask them what they know and i ask them any questions on it and say describe this to me they can whereas people that read people that take video courses like this will truly understand the technology now i told you that we hate practice tests as a means to learn but we love practice tests as a mean to prep for the exam so our recommendation is prior to the exam find a practice test it doesn't matter which one it is just something with a lot of questions and really what you're doing is we want to take the knowledge that you have from finishing this course and get used to reading questions the way aws would write them because they're going to be complicated they're going to be tricky they're going to be hard to read and at least if you get used to the way they would ask a question by a practice exam it's a great way to ensure your success on the exam we wouldn't want you to study this way but since you've already completed this program and we know you have good knowledge this is just a way to make sure you can do better so we recommend taking a practice exam and when you can consistently score 95 or better go take the exam these exams are expensive and we want to make sure you're going to be successful now one of the hardest parts of these exams is really reading these questions they can be written in a very tricky manner so the day before the exam get a good night's sleep eat well and avoid any kind of alcohol or anything that could affect with your thinking and on the day of the test arrive early and if it's online still arrive early i've taken exams and i've had to wait 30 minutes due to some technology challenges and at least if you're there early when you deal with some computing challenges whether they be at a facility or your own house and you will run into them at least you have time to do it you don't want to worry about losing time in your exam and remember have a valid photo id with you for the exam you'd hate to study you'd hate to prep you'd hate to do everything and then show up and not have a valid photo id and not be allowed to take the test thank you so much for taking this course we love our students we'd love you to reach out to us and tell us about your success and we look forward to seeing you in the next class wonderful luck in your career and we hope to see you soon in another program

Channel: Go Cloud Architects

Views: 83,632

Keywords: AWS certified solutions architect associate 2021, AWS solution architect certification, AWS solution architect interview questions, AWS full course tutorial, free AWS certification training, AWS certification course online, AWS certification course free, free AWS course, cloud architect career, AWS career tips, cloud architect training, cloud computing architect, cloud architect, Go Cloud Architects, SAA-C02, AWS cloud computing full course, cloud computing complete course

Id: keoNi7MmAUY

Length: 344min 1sec (20641 seconds)

Published: Mon Apr 12 2021