Google Cloud Associate Cloud Engineer Course - Pass the Exam!

hey this is anthony tavelos your cloud instructor at exam pro bringing you a complete study course for the google cloud associate cloud engineer made available to you here on free code camp and so this course is designed to help you pass and achieve google issued certification the way we're going to do that is to go through lots of lecture content follow alongs and using my cheat sheets on the day of the exam so you pass and you can take that certification and put it on your resume or linkedin so you can get that cloud job or promotion that you've been looking for and so a bit about me is that i have 18 years industry experience seven of it specializing in cloud and four years of that as a cloud trainer i previously been a cloud and devops engineer and i've also published multiple cloud courses and i'm a huge fan of the cartoon looney tunes as well as a coffee connoisseur and so i wanted to take a moment to thank viewers like you because you make these free courses possible and so if you're looking for more ways of supporting more free courses just like this one the best way is to buy the extra study material at co example.com in particular for this certification you can find it at gcp hyphen ace there you can get study notes flash cards quizlets downloadable lectures which are the slides to all the lecture videos downloadable cheat sheets which by the way are free if you just go sign up practice exams and you can also ask questions and get learning support and if you want to keep up to date with new courses i'm working on the best way is to follow me on twitter at antony's cloud and i'd love to hear from you if you passed your exam and also i'd love to hear on what you'd like to see next [Music] welcome back in this lesson i wanted to quickly go over how to access the course resources now the resources in this course are designed to accompany the lessons and help you understand not just the theory but to help with the demo lessons that really drive home the component of hands-on learning these will include study notes lesson files scripts as well as resources that are used in the demo lessons these files can be found in a github repository that i will be including below that are always kept up-to-date and it is through these files that you will be able to follow along and complete the demos on your own to really cement the knowledge learned it's a fairly simple process but varies through the different operating systems i'll be going through this demo to show you how to obtain access through the three major operating systems being windows mac os and ubuntu linux so i'm first going to begin with windows and the first step would be to open up the web browser and browse to this url which i will include in the notes below and this is the course github repository which will house all the course files that i have mentioned before keeping the course up to date will mean that files may need to be changed and so as i update them they will always be reflected and uploaded here in the repo so getting back to it there are two ways to access this repository so the easiest way to obtain a copy of these files will be to click on the clone or download button and click on download zip once the file has been downloaded you can then open it up by clicking on it here and here are the files here in downloads and this will give you a snapshot of all the files and folders as you see them from this repository now although this may seem like the simple way to go this is not the recommended method to download as if any files have changed you will not be up to date with the latest files and will only be current from the date at which you've downloaded them now the way that is recommended is using a source control system called git and so the easiest way to install it would be to go to this url https colon forward slash forward slash git dash scm.com and this will bring you to the git website where you can download the necessary software for windows or any other supported operating system and so i'm going to download it here and this should download the latest version of git for windows and it took a few seconds there but it is done and no need to worry about whether or not you've got the proper version usually when you click that download button it will download the latest version for your operating system so i'm going to go over here and open this up you'll get a prompt where you would just say yes and we're going to go ahead and accept all the defaults here this is where it's going to install it let's hit next these are all the components that they're going to be installed let's click on next and again we're going to go through everything with all the defaults and once we've reached installing all the defaults it's gonna take a couple minutes to install and again it took a minute or so we're going to just click on next and it's going to ask if you want to view the release notes and we don't really need those so we can click on ok and simply close that and we're just going to go over and see if git is installed we're going to run the command prompt and i'm going to just zoom in here so we can see a little better and there we go and we are just going to type in git and as you can see it's been installed and so now that we've installed git we want to be able to pull down all the folders and the files within them from the repository to our local system and so i'm just going to clear the screen here and we're going to do a cd to make sure that i'm in my home directory and then we're going to make a directory called repos and in order to do that we're going to do mkdir space repos and then we're going to move into that directory so cd space repos and so again here we want to clone those files that are in the repository to our local system so in order to do that we're going to use the command git clone so get space clone and then we're going to need our location of the git repository so let's go back to the browser and we're going to go over here to clone or download and here you will see clone with https so make sure that this says https and you can simply click on this button which will copy this to the clipboard and then we'll move back to our command prompt and paste that in and once that's pasted just hit enter and it will clone your repository into the repos directory and so just to verify that we've cloned all the necessary files we're going to cd into the master directory that we had just cloned and we're going to do a dir and there you have it all of the files are cloned exactly as it is here in the repository now just as a note in order to keep these files up to date we need to run a different command which would be a git pull and this can be run at any time in order to pull down any files or folders that have been updated since you did the first pull which in this case would be cloning of the repository again this will provide you with the latest and most up-to-date files at any given moment in time and in this case since nothing has changed i have been prompted with a message stating that i'm up to date if nothing is changed you will always be prompted with this message if there was it will pull your changes down to your synced local copy and the process for windows is completed and is similar in mac os and i'll move over to my mac os virtual machine and log in and once you've logged in just going to go over here to the terminal and i'm just going to cd to make sure i'm in my home directory then i'm going to do exactly what we did in windows so i'm going to run the command mk dir space repos and create the repos directory and i'm going to move in to the repos directory and then i'm going to run git now for those of you who do not have get installed you will be prompted with this message to install it and you can go ahead and just install you'll be prompted with this license agreement you can just hit agree and depending on your internet connection this will take a few minutes to download and install so as this is going to take a few minutes i'm going to pause the video here and come back when it's finished installing okay and the software was successfully installed so just to do a double check i'm going to run git and as you can see it's been installed so now that we have git installed we want to clone all the directories and the files from the github repository to our local repos folder so i'm going to open up my browser and i'm going to paste my github repository url right here and you'll see the clone button over here so we're going to click on this button and here we can download zip but like i said we're not going to be doing that we're going to go over here and copy this url for the github repository again make sure it says https and we're going to copy this to our clipboard and we're going to go back to our terminal and we are going to run the command git space clone and we're going to paste in our url and as you can see here i've cloned the repository and all the files and folders within it and so as is my best practice i always like to verify that the files have been properly cloned and so i'm going to run the command ls just to make sure and go into the master directory and do a double check and as you can see the clone was successful as all the files and folders are here and again to download any updates to any files or directories we can simply run the command git space poll and because we've already cloned it it's already up to date and so the process is going to be extremely similar on linux so i'm going to simply move over to my linux machine and log in i'm going to open up a terminal and i'm going to make my terminal a little bit bigger for better viewing and so like the other operating systems i want to clone all the files and directories from the github repository to my machine and so i'm going to cd here to make sure i'm in my home directory and like we did before we want to create a directory called repos so i'm going to run the command mkdir space repos and we're going to create the repos directory we're now going to move into the repos directory and here we're going to run the git command and because git is not installed on my machine i've been prompted with the command in order to install it so i'm going to run that now so the command is sudo space apt space install space get and i'm going to enter in my password and install it and just to verify i'm going to run the command git and i can see here it's been installed so now i'm going to go over here to my browser and i'm going to paste in the url to my repository and over here we'll have the same clone button and when i click on it i can get the url for the github repository in order to clone it again make sure before you clone that this says https if it doesn't say https you'll have the option of clicking on a button that will allow you to do so once it says https then you can simply copy this url to your clipboard by clicking on the button and then move over back to the terminal and we are going to clone this repository by typing in the get space clone command along with the url of the repository and when we hit enter it'll clone it right down to our directory so i'm just going to move into the master directory just to verify that the files are there and again they're all here so again if you're looking to update your repository with any new updated changes you can simply run the get space pull command to update those files and so that's the linux setup so you have a local copy of the lesson files now there's just one more thing that i highly recommend you do and to demonstrate it i'm going to move back over to my windows virtual machine now i'm going to open up the web browser again open up a new tab and i'm going to browse to this url https colon forward slash forward slash code.visualstudio.com [Music] and i'll make sure that the url is in the text below there is a version of this code editor available for windows mac os and linux you can simply click on this drop down and you'll find the link to download it for your operating system but in most cases it should automatically show the correct version so just go ahead and click on download and it should start downloading automatically and you should be able to run it right away now the reason behind me asking you to install this utility is for editing code of different sorts whether you're adjusting yaml or python documents for deployment manager or even managing scripts a code editor will give you the ease of use when it comes to managing editing and even syntactical highlighting of code as shown here below it will highlight the code to make it easier to understand now if you have your own editor that you would prefer to use go ahead and use that but for those that don't my recommendation will be to use visual studio code so to install visual studio code we're just going to accept this license agreement and then we're going to click on next and we're just going to follow all the defaults to install it it's going to take a minute or two and for those running windows you want to make sure that this box is checked off so that you can launch it right away let's hit finish another recommendation would be to go over here to the task bar so you can pin it in place so that it's easier to find and so now you have access to all the resources that's needed for this course but with that that's everything that i wanted to cover for this lesson so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this lesson i wanted to discuss the various certifications available for google cloud as this number keeps on growing and i am looking to keep this lesson as up to date as possible so with that being said let's dive in now google cloud has released a slew of certifications in many different areas of expertise as well as different experience levels now there are two levels of difficulty when it comes to the google cloud certifications starting off with the associate level we see that there is only the one certification which is the cloud engineer the associate level certification is focused on the fundamental skills of deploying monitoring and maintaining projects on google cloud this is a great starting point for those completely new to cloud and google recommends the associate cloud engineer as the starting point to undergoing your certification journey this was google cloud's very first certification and to me was the entry point of wanting to learn more as an engineer in cloud in my personal opinion no matter your role this certification will cover the general knowledge that is needed to know about starting on google cloud and the services within it which is why i labeled it here as the foundational level course i also consider this the stepping stone into any other professional level certifications which also happens to be a recommended path by google with a great course and some dedication i truly believe that anyone with even a basic skill level in it should be able to achieve this associate level certification now it is recommended from google themselves that prior to taking this exam that you should have over six months experience building on google cloud for those of you with more of an advanced background in google cloud or even other public clouds this certification should be an easy pass as it covers the basics that you should be familiar with adding a google twist to it at the time of this lesson this exam is two hours long and the cost is 125 us dollars the exam is a total of 50 questions which consists of both multiple choice and multiple answer questions each of the questions contain three to four line questions with single line answers that by the time you finish this course you should have the confidence to identify the incorrect answers and be able to select the right answers without a hitch moving into the professional level certifications there are seven certifications that cover a variety of areas of specialty depending on your role you might want to take one or maybe several of these certifications to help you gain more knowledge in google cloud or if you love educating yourself and you're really loving your journey in gcp you will probably want to consider pursuing them all in my personal opinion the best entry point into the professional level would be the cloud architect it is a natural step up from the associate cloud engineer and it builds on top of what is learned through that certification with a more detailed and more thorough understanding of cloud architecture that is needed for any other certification there is some overlap from the cloud engineer which is why in my opinion doing this certification right after makes sense it also brings with it the ability to design develop and manage secure scalable and highly available dynamic solutions it is a much harder exam and goes into great depth on services available the professional cloud architect is a great primer for any other professional level certification and can be really helpful to solidify the learning that is needed in any other technical role i find it the most common path that many take who look to learn google cloud which is why i personally recommend it to them and at the time of this lesson it also holds the highest return on investment due to the highest average wage over any other current cloud certification in the market google recommends over three years of industry experience including one year on google cloud before attempting these exams with regards to the exams in the professional tier they are much harder than the associate level and at the time of this course is two hours long and the cost is 200 us dollars these exams are a total of 50 questions which consists of both multiple choice and multiple answer questions it's the same amount of questions with the same amount of time but it does feel much harder each of the questions contain four to five line questions with one to three line answers it's definitely not a walk in the park and will take some good concentration and detailed knowledge on google cloud to solidify a pass after completing the cloud architect certification depending on your role my suggestion would be to pursue the areas that interest you the most to make your journey more enjoyable for me at the time i took the security engineer out as i am a big fan of security and i knew that i would really enjoy the learning and make it more fun for me this is also a great certification for those who are looking to excel their cloud security knowledge on top of any other security certifications such as the security plus or cissp now others may be huge fans of networking or hold other networking certifications such as the ccna and so obtaining the network engineer certification might be more up your alley and give you a better understanding in cloud networking now if you're in the data space you might want to move into the data engineer exam as well as taking on the machine learning engineer exam to really get some deeper knowledge in the areas of big data machine learning and artificial intelligence on google cloud now i know that there are many that love devops me being one of them and really want to dig deeper and understand sre and so they end up tackling the cloud developer and cloud devops engineer certifications so the bottom line is whatever brings you joy in the area of your choosing start with that and move on to do the rest all the professional certifications are valuable but do remember that they are hard and need preparation for study last but not least is the collaboration engineer certification and this certification focuses on google's core cloud-based collaboration tools that are available in g suite or what is now known as google workspaces such as gmail drive hangouts docs and sheets now the professional level collaboration engineers certification dives into more advanced areas of g suite such as mail routing identity management and automation of it all using tools scripting and apis this certification is great for those looking to build their skill set as an administrator of these tools but gives very little knowledge of google cloud itself so before i move on there is one more certification that i wanted to cover that doesn't fall under the associate or professional certification levels and this is the google cloud certified fellow program now this is by far one of the hardest certifications to obtain as there are very few certified fellows at the time of recording this lesson it is even harder than the professional level certifications and this is due to the sheer level of competency with hybrid multi-cloud architectures using google cloud anthos google's recommended experience is over 10 years with a year of designing enterprise solutions with anthos then a four-step process begins first step is to receive a certified fellow invitation from google and once you've received that invitation then you need to submit an application with some work samples that you've done showing google your competency in hybrid multi-cloud once that is done the third step is a series of technical hands-on labs that must be completed and is a qualifying assessment that must be passed in order to continue and after all that the last step is a panel interview done with google experts in order to assess your competency of designing hybrid and multi-cloud solutions with anthos so as you can see here this is a very difficult and highly involved certification process to achieve the title of certified fellow this is definitely not for the faint of heart but can distinguish yourself as a technical leader in anthos and a hybrid multi-cloud expert in your industry now i get asked many times whether or not certifications hold any value are they easy to get are they worth more than the paperwork that they're printed on and does it show that people really know how to use google cloud and my answer is always yes as the certifications hold benefits beyond just the certification itself and here's why targeting yourself for a certification gives you a milestone for learning something new with this new milestone it allows you to put together a study plan in order to achieve the necessary knowledge needed to not only pass the exam but the skills needed to progress in your everyday technical role this new knowledge helps keep your skills up to date therefore making you current instead of becoming a relic now having these up-to-date skills will also help advance your career throughout my career in cloud i have always managed to get my foot in the door with various interviews due to my certifications it gave me the opportunity to shine in front of the interviewer while being able to confidently display my skills in cloud it also allowed me to land the jobs that i sought after as well as carve out the career path that i truly wanted on top of landing the jobs that i wanted i was able to achieve a higher salary due to the certifications i had i have doubled and tripled my salary since i first started in cloud all due to my certifications and i've known others that have obtained up to five times their salary because of their certifications now this was not just from achieving the certification to put on my resume and up on social media but from the knowledge gained through the process and of course i personally feel that having your skills constantly up to date advancing your career and getting the salary that you want keeps you motivated to not only get more certifications but continue the learning process i am and always have been a huge proponent of lifelong learning and as i always say when you continue learning you continue to grow so in short google cloud certifications are a great way to grow and so that about covers everything that i wanted to discuss in this lesson so you can now mark this lesson as complete and i'll see you in the next one [Music] welcome back and in this lesson i'm going to be talking about the fictitious organization called bow tie inc that i will be using throughout the course now while going through the architectures and demos in this course together i wanted to tie them to a real world situation so that the theory and practical examples are easy to understand tying it to a scenario is an easy way to do this as well it makes things a lot more fun so the scenario again that i will be using is based on bow tie ink so before we get started with the course i'd like to quickly run through the scenario and don't worry it's going to be very high level and i will keep it brief so bow tie ink is a bow tie manufacturing company that designs and manufactures bow ties within their own factories they also hold a few retail locations where they sell their bow ties as well as wholesale to other thai and men's fashion boutiques and department stores across the globe being in the fashion business they mainly deal with commerce security and big data sets bow tie inc is a global company and they are headquartered in montreal canada they employ about 300 people globally with a hundred of them being in sales alone to support both the brick and mortar stores and wholesale branches there are many different departments to the company that make it work such as in-store staff i.t marketing for both in-store and online sales manufacturing finance and more the types of employees that work in bow tie inc vary greatly due to the various departments and consists of many people such as sales for both in-store and wholesale managers that run the stores and sewers that work in the manufacturing plant and many more that work in these various departments the business has both offices and brick and mortar stores in montreal london and los angeles now due to the thrifty mindset of management concentrating all their efforts on commerce and almost none in technical infrastructure has caused years of technical debt and is now a complete disaster within the brick and mortar location there contains two racks with a few servers and some networking equipment the global inventory of bow ties are updated upon sales in both stores and wholesale as well as new stock that has been manufactured from the factory there are point-of-sale systems in each store or office location these systems are all connected to each other over a vpn connection in order to keep updates of the inventory fresh all office and store infrastructure are connected to each other and the montreal headquarters and the point of sale systems and kiosk systems are backed up to tape in the montreal headquarters as well and like i said before management is extremely thrifty but they have finally come to the realization that they need to start spending money on the technical infrastructure in order to scale so diving into a quick overview of exactly what the architecture looks like the head office is located in montreal canada it has its main database for the crm and point-of-sale systems as well as holding the responsibility of housing the equipment for the tape backups the tapes are then taken off site within montreal by a third-party company for storage the company has two major offices one in london covering the eu and the other in the west coast us in los angeles these major offices are also retail locations that consume i.t services from the headquarters in montreal again being in the fashion business bowtie inc employs a large amount of sales people and the managers that support them these employees operate the point-of-sale systems so we're constantly looking to have the website sales and the inventory updated at all times each salesperson has access to email and files for updated forecasts on various new bowtie designs most sales people communicate over a voice over ip phone and chat programs through their mobile phone the managers also manually look at inventory on what's been sold versus what's in stock to predict the sales for stores in upcoming weeks this will give manufacturing a head start to making more bow ties for future sales now whatever implementations that we discuss throughout this course we'll need to support the day-to-day operations of the sales people and the managers and because of the different time zones in play the back-end infrastructure needs to be available 24 hours a day seven days a week any downtime will impact updated inventory for both online sales as well as store sales at any given time now let's talk about the current problems that the business is facing most locations hold on premise hardware that is out of date and also out of warranty the business looked at extending this warranty but became very costly as well management is on the fence about whether to buy new on-premise hardware or just move to the cloud they were told that google cloud is the way to go when it comes to the retail space and so are open to suggestions yet still very weary now when it comes to performance there seems to be a major lag from the vpn connecting from store to store as well as the head office that's responsible for proper inventory thus slowing down the point of sale systems and to top it all off backups taking an exorbitant amount of time is consuming a lot of bandwidth with the current vpn connection now bowtie inc has always struggled with the lack of highly available systems and scalability due to cost of new hardware this is causing extreme stress for online e-commerce whenever a new marketing campaign is launched as the systems are unable to keep up with the demand looking at the forecast for the next two quarters the business is looking to open up more stores in the eu as well as in the us and with the current database in place providing very inefficient high availability or scalability there is a major threat of the main database going down now when it comes to assessing the backups the tape backups have become very slow especially backing up from london and the off-site storage costs continuously go up every year the backups are consuming a lot of bandwidth and are starting to become the major pain point for connection issues between locations on top of all these issues the small it staff that is employed have outdated i.t skills and so there is a lot of manual intervention that needs to be done to top it all off all the running around that is necessary to keep the outdated infrastructure alive management is also now pushing to open new stores to supply bow ties globally given the ever-growing demand as well as being able to supply the demand of bow ties online through their e-commerce store now these are some realistic yet common scenarios that come up in reality for a lot of businesses that are not using cloud computing and throughout the course we will dive into how google cloud can help ease the pain of these current ongoing issues now at a high level with what the business wants to achieve and what the favorable results are they are all interrelated issues so bowtie inc requires a reliable and stable connection between all the locations of the stores and offices so that sales inventory and point-of-sale systems are quick and up-to-date at all times this will also allow all staff in these locations to work a lot more efficiently with a stable and reliable connection in place backups should be able to run smoothly and also eliminate the cost of off-site backup not to mention the manpower and infrastructure involved to get the job done while scaling up offices and stores due to increase in demand the business should be able to deploy stores in new regions using pay as you go billing while also meeting the requirements and regulations when it comes to gpdr and pci this would also give the business flexibility of having a disaster recovery strategy in place in case there was a failure of the main database in montreal now as mentioned before the business is extremely thrifty especially when it comes to spend on it infrastructure and so the goal is to have the costs as low as possible yet having the flexibility of scaling up when needed especially when new marketing campaigns are launched during high demand sales periods this would also give bowtie inc the flexibility of analyzing sales ahead of time using real-time analytics and catering to exactly what the customer is demanding thus making inventory a lot more accurate and reducing costs in manufacturing items that end up going on sale and costing the company money in the end finally when it comes to people supporting infrastructure automation is key removing manual steps and a lot of the processes can reduce the amount of manpower needed to keep the infrastructure alive and especially will reduce downtime when disaster arises putting automation in place will also reduce the amount of tedious tasks that all departments have on their plate so that they can focus on more important business needs now that's the scenario at a high level i wanted to really emphasize that this is a typical type of scenario that you will face as a cloud engineer and a cloud architect the key to this scenario is the fact that there are areas that are lacking in detail and areas that are fully comprehensible and this will trigger knowing when and where to ask relevant questions especially in your day-to-day role as an engineer it will allow you to fill the gaps so that you're able to figure out what services you will need and what type of architecture to use this is also extremely helpful when it comes to the exam as in the exam you will be faced with questions that pertain to real life scenarios that will test you in a similar manner knowing what services and architecture to use based on the information given will always give you the keys to the door with the right answer and lastly when it comes to the demos this scenario used throughout the course will help put things in perspective as we will come to resolve a lot of these common issues real world scenarios can give you a better perspective on learning as it is tied to something that makes it easy to comprehend and again bow tie inc is the scenario that i will be using throughout the course to help you grasp these concepts so that's all i have to cover this scenario so you can now mark this lesson as complete and let's move on to the next one [Music] hey this is anthony cevallos and what i wanted to show you here is where you can access the practice exam on the exam pro platform so once you've signed up for your account you can head on over to the course and you can scroll down to the bottom of the curriculum list and you will see the practice exams here at the bottom now just as a quick note you should generally not attempt the practice exam unless you have completed all the lecture content including the follow alongs as once you start to see those questions you will get an urge to start remembering these questions and so i always recommend to use the practice exam as a serious attempt and not just a way to get to the final exam at a faster pace taking your time with the course will allow you to really prevail through these practice exams and allow you for a way better pass rate on the final exam looking here we can see two practice exams with 50 questions each and so i wanted to take a moment here and dive into the practice exam and show you what some of these questions will look like and so clicking into one of these exams we can get right into it and so as you can see i've already started on practice exam one and so i'm going to click into that right now and as you can see the exam is always timed and in this case will be 120 minutes for this specific exam there are 50 questions for this practice exam and you will see the breakdown in the very beginning of the types of questions you will be asked now for the google cloud exams at the associate level they are usually structured in a common format they generally start with one or two lines of sentences which will typically represent a scenario followed by the question itself this question tends to be brief and to the point immediately following that you will be presented with a number of answers usually four or five in nature and can sometimes be very very technical as they are designed for engineers like asking about which gcloud commands to use to execute in a given scenario as well as theoretical questions that can deal with let's say best practices or questions about the specific services themselves now these answers will come in two different styles either multi-choice or multi-select the multi-choice is usually about identifying the correct answer from a group of incorrect or less correct answers whereas the multi-select will be about choosing multiple correct solutions to identify the answer as well for this associate exam the overall structure is pretty simple in nature and typically will be either right or wrong now sometimes these questions can get tricky where there are multiple possible answers and you will have to select the most suitable ones now although most of these types of questions usually show up in the professional exam they can sometimes peek their heads into the associate and so a great tactic that i always like to use is to immediately identify what matters in the question itself and then to start ruling out any of the answers that are wrong and this will allow you to answer the question a lot more quickly and efficiently as it will bring the more correct answer to the surface as well as making the answer a lot more obvious and making the entire question less complex so for instance with this question here you are immediately asked about google's recommended practices when it comes to using cloud storage as backup for disaster recovery and this would be for a specific storage type and so quickly looking at the answers you can see that standard storage and near line storage will not be part of the answer and so that will leave cold line storage or archive storage as the two possible choices for the answer of this question and so these are the typical techniques that i always like to use for these exams and so provided that you've gone through all the course content you will be able to answer these technical questions with ease and following the techniques i've just given and applying them to each question can really help you in not only this practice exam but for the final exam landing you a passing grade getting you certified [Music] welcome back and in this section i wanted to really hone in on the basics of cloud computing the characteristics that make it what it is the different types of computing and how they differ from each other as well as the types of service models now in this lesson i wanted to dive into the definition of cloud computing and the essential characteristics that define it now for some advanced folk watching this this may be a review and for others this may fulfill a better understanding on what is cloud now cloud is a term that is thrown around a lot these days yet holds a different definition or understanding to each and every individual you could probably ask 10 people on their definition of cloud and chances are everyone would have their own take on it many see cloud as this abstract thing in the sky where files and emails are stored but it's so much more than that now the true definition of it can be put in very simple terms and can be applied to any public cloud being google cloud aws and azure moving on to the definition cloud computing is the delivery of a shared pool of on-demand computing services over the public internet that can be rapidly provisioned and released with minimal management effort or service provider interaction these computing services consist of things like servers storage networking and databases they can be quickly provisioned and accessed from your local computer over an internet connection now coupled with this definition are five essential characteristics that define the cloud model that i would like to go over with you and i believe that it would hold massive benefits to understanding when speaking to cloud this information can be found in the white paper published by the national institute of standards and technology i will include a link to this publication in the lesson notes for your review now these essential characteristics are as follows the first one is on-demand self-service and this can be defined as being able to provision resources automatically without requiring human interaction on the provider's end so in the end you will never need to call up or interact with the service provider in order to get resources provisioned for you as well you have the flexibility of being able to provision and de-provision these resources whenever you need them and at any given time of the day the second characteristic is broad network access now this simply means that cloud computing resources are available over the network and can be accessed by many different customer platforms such as mobile phones tablets or computers in other words cloud services are available over a network moving into the third is resource pooling so the provider's computing resources are pooled together to support a multi-tenant model that allows multiple customers to share the same applications or the same physical infrastructure while retaining privacy and security over their information this includes things like processing power memory storage and networking it's similar to people living in an apartment building sharing the same building infrastructure like power and water yet they still have their own apartments and privacy within that infrastructure this also creates a sense of location independence in that the customer generally has no control or knowledge over the exact location of the provided resources but they may be able to specify location at a higher level of abstraction so in the end the customer does not really have the option of choosing exactly which server server rack or data center for that matter of where the provided resources are coming from they will only be able to have the option to choose things like regions or sections within that region the fourth essential characteristic is rapid elasticity this to me is the key factor of what makes cloud computing so great and so agile capabilities can be elastically provisioned and released in some cases automatically to scale rapidly outwards and inwards in response with demand to the consumer the capabilities available for provisioning often appear to be unlimited and can be provisioned in any quantity at any time and touching on the fifth and last characteristic cloud systems automatically control and optimize resource usage by leveraging a metering capability resource usage can be monitored controlled and reported providing transparency for both the provider and consumer of the service now what this means is that cloud computing resource usage is metered and you can pay accordingly for what you've used resource utilization can be optimized by leveraging pay-per-use capabilities and this means that cloud resource usage whether they are instances that are running cloud storage or bandwidth it all gets monitored measured and reported by the cloud service provider the cost model is based on pay for what you use and so the payment is based on the actual consumption by the customer so knowing these key characteristics of cloud computing along with their benefits i personally find can really give you a leg up on the exam as well as speaking to others in your day-to-day role as more and more companies start moving to cloud i hope this lesson has explained to you on what is cloud computing and the benefits it provides so that's all i have for this lesson so you can now mark this lesson as complete and let's move on to the next one welcome back in this lesson i wanted to go over the four common cloud deployment models and distinguish the differences between public cloud multi-cloud private cloud and hybrid cloud deployment models this is a common subject that comes up a fair amount in the exam as well as a common theme in any organization moving to cloud knowing the distinctions between them can be critical to the types of architecture and services that you would use for the specific scenario you are given as well as being able to speak to the different types of deployment models as an engineer in the field getting back to the deployment models let's start with the public cloud model which we touched on a bit in our last lesson now the public cloud is defined as computing services offered by third-party providers over the public internet making them available to anyone who wants to use or purchase them so this means that google cloud will fall under this category as a public cloud there are also other vendors that fall under this category such as aws and azure so again public cloud is a cloud that is offered over the public internet now public clouds can also be connected and used together within a single environment for various use cases this cloud deployment model is called multi-cloud now a multi-cloud implementation can be extremely effective if architected in the right way one implementation that is an effective use of multi-cloud is when it is used for disaster recovery this is where your architecture would be replicated across the different public clouds in case one were to go down another could pick up the slack what drives many cases of a multi-cloud deployment is to prevent vendor lock-in where you are locked into a particular cloud provider's infrastructure and unable to move due to the vendor-specific feature set the main downfall to this type of architecture is that the infrastructure of the public cloud that you're using cannot be fully utilized as each cloud vendor has their own proprietary resources that will only work in their specific infrastructure in other words in order to replicate the environment it needs to be the same within each cloud this removes each cloud's unique features which is what makes them so special and the resources so compelling so sometimes finding the right strategy can be tricky depending on the scenario now the next deployment model i wanted to touch on is private cloud private cloud refers to your architecture that exists on premise and restricted to the business itself with no public access yet it still carries the same five characteristics that we discussed with regards to what defines cloud each of the major cloud providers shown here all have their own flavor of private cloud that can be implemented on site google cloud has anthos aws has aws outposts and azures is azure stack they show the same characteristic and leverage similar technologies that can be found in the vendor's public cloud yet can be installed on your own on-premise infrastructure please be aware any organizations may have a vmware implementation which holds cloud-like features yet this is not considered a private cloud true private cloud will always meet the characteristics that make up cloud now it is possible to use private cloud with public cloud and this implementation is called hybrid cloud so hybrid cloud is when you are using public cloud in conjunction with private cloud as a single system a common architecture used is due to compliance where one cloud could help organizations achieve specific governance risk management and compliance regulations while the other cloud could take over the rest now i'd really like to make an important distinction here if your on-premise infrastructure is connected to public cloud this is not considered hybrid cloud this is what's known as hybrid environment or a hybrid network as the on-premises infrastructure holds no private cloud characteristics true hybrid cloud allows you to use the exact same interface and tooling as what's available in the public cloud so being aware of this can avoid a lot of confusion down the road so to sum up everything that we discussed when it comes to public cloud this is when one cloud provided by one vendor that is available over the public internet multi-cloud is two or more public clouds that are connected together to be used as a single system a private cloud is considered an on-premises cloud that follows the five characteristics of cloud and is restricted to the one organization with no accessibility to the public and finally hybrid cloud is private cloud connected to a public cloud and being used as a single environment again as a note on-premises architecture connected to public cloud is considered a hybrid environment and not hybrid cloud the distinction between the two are very different and should be observed carefully as gotchas may come up in both the exam and in your role as an engineer so these are all the different cloud deployment models which will help you distinguish on what type of architecture you will be using in any scenario that you are given and so this is all i wanted to cover when it comes to cloud deployment models so you can now mark this lesson as complete and let's move on to the next one welcome back so to finish up the nist definition of cloud computing i wanted to touch on cloud service models which is commonly referred to as zas now this model is usually called zas or xaas standing for anything as a service it includes all the services in a cloud that customers can consume and x can be changed to associate with the specific service so in order to describe the cloud service models i needed to touch on some concepts that you may or may not be familiar with this will make understanding the service models a little bit easier as i go through the course and describe the services available and how they relate to the model this lesson will make so much sense by the end it'll make the services in cloud easier to both describe and define now when it comes to deploying an application they are deployed in an infrastructure stack like the one you see here now a stack is a collection of needed infrastructure that the application needs to run on it is layered and each layer builds on top of the one previous to it to create what it is that you see here now as you can see at the top this is a traditional on-premises infrastructure stack that was typically used pre-cloud now in this traditional model all the components are managed by the customer the purchasing of the data center and all the network and storage involved the physical servers the virtualization the licensing for the operating systems the staff that's needed to put it all together including racking stacking cabling physical security was also something that needed to be taken into consideration in other words for the organization to put this together by themselves they were looking at huge costs now the advantages to this is that it allowed for major flexibility as the organization is able to tune this any way they want to satisfy the application compliance standards basically anything that they wanted now when talking about the cloud service model concepts parts are always managed by you and parts are managed by the vendor now another concept i wanted to touch on is that unit of consumption is how the vendor prices what they are serving to their customer now just before cloud became big in the market there was a model where the data center was hosted for you so a vendor would come along and they would take care of everything with regards to the data center the racks the power to the racks the air conditioning the networking cables out of the building and even the physical security and so the unit of consumption here was the rack space within the data center so the vendor would charge you for the rack space and in turn they would take care of all the necessities within the data center now this is less flexible than the traditional on-premises model but the data center is abstracted for you so throughout this lesson i wanted to introduce a concept that might make things easier to grasp which is the pizza as a service so now the traditional on-premises model is where you would buy everything and make the pizza at home now as we go on in the lesson less flexibility will be available because more layers will be abstracted so the next service model that i wanted to introduce is infrastructure as a service or i as for short this is where all the layers from the data center up to virtualization is taken care of by the vendor this is the most basic model which is essentially your virtual machines in a cloud data center you set up configure and manage instances that run in the data center infrastructure and you put whatever you want on them on google cloud google compute engine would satisfy this model and so the unit of consumption here would be the operating system as you would manage all the operating system updates and everything that you decide to put on that instance but as you can see here you are still responsible for the container the run time the data and the application layers now bringing up the pizza as a service model is would be you picking up the pizza and you cooking it at home moving on to platform as a service or paz for short this is a model that is geared more towards developers and with pass the cloud provider provides a computing platform typically including the operating system the programming language execution environment the database and the web server now typically with pass you never have to worry about the operating system updates or managing the runtime and middleware and so the unit of consumption here would be the runtime now the runtime layer would be the layer you would consume as you would be running your code in the supplied runtime environment that the cloud vendor provides for you the provider manages the hardware and software infrastructure and you just use the service this is usually the layer on top of is and so all the layers between the data center and runtime is taken care of by the vendor a great example of this for google cloud is google app engine which we will be diving into a little bit later getting back to the pizza as a service model pass would fall under the pizza being delivered right to your door now with the past model explained i want to move into the last model which is sas which stands for software as a service now with sas all the layers are taken care of by the vendor so users are provided access to application software and cloud providers manage the infrastructure and platforms that run the applications g suite and microsoft's office 365 are great examples of this model now sas doesn't offer much flexibility but the trade-off is that the vendor actually takes care of all these layers so again the unit of consumption here is the application itself and of course getting to the pizza as a service model sas is pretty much dining in the restaurant enjoying your pizza now to summarize when you have a data center on site you manage everything when it's infrastructure as a service part of that stack is abstracted by the cloud vendor with platform as a service you're responsible for the application and data everything else is abstracted by the vendor with software as a service again using the pizza as a service analogy on premise you buy everything and you make the pizza at home infrastructure as a service you pick up the pizza and you cook it at home when it comes to platform as a service the pizza is delivered and of course software as a service is dining in the restaurant now there will be some other service models coming up in this course such as function as a service and containers as a service and don't worry i'll be getting into those later but i just wanted to give you a heads up so now for some of you this may have been a lot of information to take in but trust me knowing these models will give you a better understanding of the services provided in google cloud as well as any other cloud vendor so that's all i wanted to cover in this lesson so you can now mark this lesson as complete and let's move on to the next one welcome back in this lesson i wanted to discuss google cloud global infrastructure how data centers are connected how traffic flows when a request is done along with the overall structure of how google cloud geographic locations are divided for better availability durability and latency now google holds a highly provisioned low latency network where your traffic stays on google's private backbone for most of its journey ensuring high performance and a user experience that is always above the norm google cloud has been designed to serve users all around the world by designing their infrastructure with redundant cloud regions connected with high bandwidth fiber cables as well as subsea cables connecting different continents currently google has invested in 13 subsea cables connecting these continents at points of presence as you see here in this diagram hundreds of thousands of miles of fiber cables have also been laid to connect points of presence for direct connectivity privacy and reduced latency just to give you an idea of what a subsea cable run might look like i have included a diagram of how dedicated google is to their customers as there is so much that goes into running these cables that connect continents as you can see here this is the north virginia region being connected to the belgium region from the u.s over to europe a cable is run from the north virginia data center as well as having a point of presence in place going through a landing station before going deep into the sea on the other side the landing station on the french west coast picks up the other side of the cable and brings it over to the data center in the belgium region and this is a typical subsea cable run for google so continents are connected for maximum global connectivity now at the time of recording this video google cloud footprint spans 24 regions 73 zones and over 144 points of presence across more than 200 countries and territories worldwide and as you can see here the white dots on the map are regions that are currently being built to expand their network for wider connectivity now to show you how a request is routed through google's network i thought i would demonstrate this by using tony bowtie now tony makes a request to his database in google cloud and google responds to tony's request from a pop or edge network location that will provide the lowest latency this point of presence is where isps can connect to google's network google's edge network receives tony's request and passes it to the nearest google data center over its private fiber network the data center generates a response that's optimized to provide the best experience for tony at that given moment in time the app or browser that tony is using retrieves the requested content with a response back from various google locations including the google data centers edge pops and edge nodes whichever is providing the lowest latency this data path happens in a matter of seconds and due to google's global infrastructure it travels securely and with the least amount of latency possible no matter the geographic location that the request is coming from now i wanted to take a moment to break down how the geographic areas are broken out and organized in google cloud we start off with the geographic location such as the united states of america and it's broken down into multi-region into regions and finally zones and so to start off with i wanted to talk about zones now a zone is a deployment area for google cloud resources within a region a zone is the smallest entity in google's global network you can think of it as a single failure domain within a region now as a best practice resources should always be deployed in zones that are closest to your users for optimal latency now next up we have a region and regions are independent geographic areas that are subdivided into zones so you can think of a region as a collection of zones and having a region with multiple zones is designed for fault tolerance and high availability the intercommunication between zones within a region is under five milliseconds so rest assured that your data is always traveling at optimal speeds now moving on into a multi-region now multi-regions are large geographic areas that contain two or more regions and this allows google services to maximize redundancy and distribution within and across regions and this is for google redundancy or high availability having your data spread across multiple regions always reassures that your data is constantly available and so that covers all the concepts that i wanted to go over when it comes to geography and regions within google cloud note that the geography and regions concepts are fundamental not only for the exam but for your day-to-day role in google cloud so just as a recap a zone is a deployment area for google cloud resources within a region a zone is the smallest entity of google's global infrastructure now a region is an independent geographic area that are subdivided into zones and finally when it comes to multi-region multi-regions are large geographic areas that contains two or more regions again these are all fundamental concepts that you should know for the exam and for your day-to-day role in google cloud and so that's all i had for this lesson so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back this lesson is going to be an overview of all the compute service options that are available in google cloud how they differ from each other and where they fall under the cloud service model again this lesson is just an overview of the compute options as we will be diving deeper into each compute option later on in this course so google cloud gives you so many options when it comes to compute services ones that offer complete control and flexibility others that offer flexible container technology managed application platform and serverless environments and so when we take all of these compute options and we look at it from a service model perspective you can see that there's so much flexibility starting here on the left with infrastructure as a service giving you the most optimal flexibility moving all the way over to the right where we have function as a service offering less flexibility but the upside being less that you have to manage and we'll be going through these compute options starting on the left here with infrastructure as a service we have compute engine now compute engine is google's staple infrastructure the service product that offers virtual machines or vms called instances these instances can be deployed in any region or zone that you choose you also have the option of deciding what operating system you want on it as well as the software so you have the option of installing different types of flavors of linux or windows and the software to go with it google also gives you the options of creating these instances using public or private images so if you or your company have a private image that you'd like to use you can use this to create your instances google also gives you the option to use public images to create instances and are available when you launch compute engine as well there are also pre-configured images and software packages available in the google cloud marketplace and we will be diving a little bit deeper into the google cloud marketplace in another lesson just know that there are slew of images out there that's available to create instances giving you the ease to deploy now when it comes to compute engine and you're managing multiple instances these are done using instance groups and when you're looking at adding or removing capacity for those compute engine instances automatically you would use auto scaling in conjunction with those instance groups compute engine also gives you the option of attaching and detaching disks as you need them as well google cloud storage can be used in conjunction with compute engine as another storage option and when connecting directly to compute engine google gives you the option of using ssh to securely connect to it so moving on to the next compute service option we have google kubernetes engine also known as gke now gke is google's flagship container orchestration system for automating deploying scaling and managing containers gke is also built on the same open source kubernetes project that was introduced by google to the public back in 2014 now before google made kubernetes a managed service there was many that decided to build kubernetes on premise in their data centers and because it is built on the same platform gke offers the flexibility of integrating with these on-premise kubernetes deployments now under the hood gke uses compute engine instances as nodes in a cluster and as a quick note a cluster is a group of nodes or compute engine instances and again we'll be going over all this in much greater detail in a different lesson so if you haven't already figured it out google kubernetes engine is considered container as a service now the next compute service option that i wanted to go over that falls under platform as a service is app engine now app engine is a fully managed serverless platform for developing and hosting web applications at scale now with app engine google handles most of the management of the resources for you for example if your application requires more computing resources because traffic to your website increases google automatically scales the system to provide these resources if the system software needs a security update as well that's handled for you too and so all you need to really take care of is your application and you can build your application in your favorite language go java.net and many others and you can use both pre-configured runtimes or use custom runtimes to allow you to write the code in any language app engine also allows you to connect with google cloud storage products and databases seamlessly app engine also offers the flexibility of connecting with third-party databases as well as other cloud providers and third-party vendors app engine also integrates with a well-known security product in google cloud called web security scanner as to identify security vulnerabilities and so that covers app engine in a nutshell moving on to the next compute service option we have cloud functions and cloud functions fall under function as a service this is a serverless execution environment for building and connecting cloud services with cloud functions you write simple single purpose functions that are attached to events that are produced from your infrastructure and services in google cloud your function is triggered when an event being watched is fired your code then executes in a fully managed environment there is no need to provision any infrastructure or worry about managing any servers and cloud functions can be written using javascript python 3 go or java runtimes so you can take your function and run it in any of these standard environments which makes it extremely portable now cloud functions are a good choice for use cases that include the following data processing or etl operations such as video transcoding and iot streaming data web hooks that respond to http triggers lightweight apis that compose loosely coupled logic into applications as well as mobile back-end functions again cloud functions are considered function as a service and so that covers cloud functions now moving to the far right of the screen on the other side of the arrow we have our last compute service option which is cloud run now cloud run is a fully managed compute platform for deploying and scaling containerized applications quickly and securely cloudrun was built on an open standard called k native and this enabled the portability of any applications that were built on it cloudrun also abstracts away all the infrastructure management by automatically scaling up and down almost instantaneously depending on the traffic now cloud run was google's response to abstracting all the infrastructure that was designed to run containers and so this is known as serverless for containers cloudrun has massive flexibility as you can write it in any language any library using any binary this compute service is considered a function as a service now at the time of recording this video i have not heard of cloud cloudrun being in the exam but since it is a compute service option i felt the need for cloudrun to have an honorable mention and so these are all the compute service options that are available on google cloud and we will be diving deeper into each one of these later on in this course again this is just an overview of all the compute service options that are available on the google cloud platform and so that's all i wanted to cover for this lesson so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back now in the last lesson i covered all the different options for compute services in this lesson we're going to cover the options that are available that couple well with these compute services by diving deeper into the different storage types and the different databases available on google cloud again this is strictly an overview as i will be diving deeper into these services later on in the course now when it comes to storage options there are three services that are readily available to you in google cloud each of them have their own specific use case that i will be diving into in just a second the first one i wanted to go over is cloud storage now with cloud storage this is google's consistent scalable large capacity and highly durable object storage so when i refer to object storage this is not the type of storage that you would attach to your instance and store your operating system on i'm talking about managing data as objects such as documents or pictures and shouldn't be confused with block storage which manages data at a more granular level such as an operating system not to worry if you fully don't grasp the concept of object storage i will be going into further detail with that later on in the cloud storage lesson cloud storage has 11 9's durability and what i mean by durability is basically loss of files so just to give you a better picture on cloud storage durability if you store 1 million files statistically google would lose one file every 659 000 years and you are about over 400 times more likely to get hit by a meteor than to actually lose a file so as you can see cloud storage is a very good place to be storing your files another great feature on cloud storage is the unlimited storage that it has with no minimum object size so feel free to continuously put files in cloud storage now when it comes to use cases cloud storage is fantastic for content delivery data lakes and backups and to make cloud storage even more flexible it is available in different storage classes and availability which i will be going over in just a second now when it comes to these different storage classes there are four different classes that you can choose from the first one is the standard storage class and this storage class offers the maximum availability with your data with absolutely no limitations this is great for storage that you access all the time the next storage class is near line and this is low-cost archival storage so this storage class is cheaper than standard and is designed for storage that only needs to be accessed less than once a month and if you're looking for an even more cost effective solution cloud storage has cold line storage class which is an even lower cost archival storage solution this storage class is designed for storage that only needs to be accessed less than once every quarter and just when you thought that the prices couldn't get lower than cold line cloud storage has offered another storage class called archive and this is the lowest cost archival storage which offers storage at a fraction of a penny per gigabyte but is designed for archival or backup use that is accessed less than once a year now when it comes to cloud storage availability there are three options that are available there is region dual region and multi-region region is designed to store your data in one single region dual region is exactly how it sounds which is a pair of regions now in multiregion cloud storage stores your data over a large geographic area consisting of many different regions across that same selected geographic area and so that about covers cloud storage as a storage option the next storage option that i wanted to talk about is file store now file store is a fully managed nfs file server from google cloud that is nfs version 3 compliant you can store data from running applications from multiple vm instances and kubernetes clusters accessing the data at the same time file store is a great option for when you're thinking about accessing data from let's say an instance group and you need multiple instances to access the same data and moving on to the last storage option we have persistent disks now with persistent disks this is durable block storage for instances now as i explained before block storage is different than object storage if you remember previously i explained that object storage is designed to store objects such as data or photos or videos whereas block storage is raw storage capacity that is used in drives that are connected to an operating system in this case persistent disks are doing just that persistent disks come in two options the first one is the standard option which gives you regular standard storage at a reasonable price and the other option is solid state or ssd which gives you lower latency higher iops and is just all around faster than your standard persistent disk both of these options are available in zonal and regional options depending on what you need for your specific workload so now that i've covered all three storage options i wanted to touch into the database options that are available on google cloud these database options come in both the sql and nosql flavors depending on your use case now getting into the options themselves i wanted to start off going into a little bit of detail with the sql relational options so the first option is cloud sql and cloud sql is a fully managed database service that is offered in postgres mysql and sql server flavors cloud sql also has the option of being highly available across zones now moving into cloud spanner this is a scalable relational database service that's highly available not only across zones but across regions and if need be available globally cloud spanner is designed to support transactions strong consistency and synchronous replication moving into the nosql options there are four available services that google cloud offers moving into the first one is bigtable and bigtable is a fully managed scalable nosql database that has high throughput and low latency bigtable also comes with the flexibility of doing cluster resizing without any downtime the next nosql option available is datastore and this is google cloud's fast fully managed serverless nosql document database datastore is designed for mobile web and internet of things applications datastore has the capabilities of doing multi-region replication as well as acid transactions for those of you who don't know i will be covering acid transactions in a later lesson next up for nosql options is firestore and this is a nosql real-time database and is optimized for offline use if you're looking to store data in a database in real time firestore is your option and like bigtable you can resize the cluster in firestore without any downtime and the last nosql option is memorystore and this is google cloud's highly available in memory service for redis and memcached this is a fully managed service and so google cloud takes care of everything for you now i know this has been a short lesson on storage and database options but a necessary overview nonetheless of what's to come and so that's about all i wanted to cover in this lesson so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back now while there are some services in gcp that take care of networking for you there are still others like compute engine that give you a bit more flexibility in the type of networking you'd like to establish this lesson will go over these networking services at a high level and provide you with strictly an overview to give you an idea on what's available for any particular type of scenario when it comes to connecting and scaling your network traffic i will be going into further details on these networking services in later lessons now i wanted to start off with some core networking features for your resources and how to govern specific traffic traveling to and from your network this is where networks firewalls and routes come into play so first i wanted to start off with virtual private cloud also known as vpc now vpc manages networking functionality for your google cloud resources this is a virtualized network within google cloud so you can picture it as your virtualized data center vpc is a core networking service and is also a global resource that spans throughout all the different regions available in google cloud each vpc contains a default network as well additional networks can be created in your project but networks cannot be shared between projects and i'll be going into further depth on vpc in a later lesson so now that we've covered vpc i wanted to get into firewall rules and routes now firewall rules segment your networks with a global distributive firewall to restrict access to resources so this governs traffic coming into instances on a network each default network has a default set of firewall rules that have already been established but don't fret you can create your own rules and set them accordingly depending on your workload now when it comes to routes this specifies how traffic should be routed within your vpc to get a little bit more granular routes specify how packets leaving an instance should be directed so it's a basic way of defining which way your traffic is going to travel moving on to the next concept i wanted to cover a little bit about low balancing and how it distributes workloads across multiple instances now we have two different types of load balancing and both these types of load balancing can be broken down to even a more granular level now when it comes to http or https low balancing this is the type of load balancing that covers worldwide auto scaling and load balancing over multiple regions or even a single region on a single global ip https load balancing distributes traffic across various regions and make sure that the traffic is routed to the closest region or in case there's failures amongst instances or in instances being bombarded with traffic http and https load balancing can route the traffic to a healthy instance in the next closest region another great feature of this load balancing is that it can distribute traffic based on content type now when it comes to network load balancing this is a regional load balancer and supports any and all ports it distributes traffic among server instances in the same region based on incoming ip protocol data such as address port and protocol now when it comes to networking dns plays a big part and because dns plays a big part in networking google has made this service 100 available on top of giving any dns queries the absolute lowest latency with google cloud dns you can publish and maintain dns records by using the same infrastructure that google uses and you can work with your managed zones and dns records such as mx records tax records cname records and a records and you can do this all through the cli the api or the sdk now some of the advanced connectivity options that are available in google cloud are cloudvpn and direct interconnect now cloudvpn connects your existing network whether it be on-premise or in another location to your vbc network through an ipsec connection the traffic is encrypted and travels between the two networks over the public internet now when it comes to direct interconnect this connectivity option allows you to connect your existing network to your vpc network using a highly available low latency connection this connectivity option does not traverse the public internet and merely connects to google's backbone and this is what gives it the highly available low latency connection a couple of other advanced connectivity options is direct and carrier peering these connections allow your traffic to flow through google's edge network locations and pairing can be done directly or it can be done through a third-party carrier and so although this is a very short lesson i will be going into greater depth on all these concepts in later lessons in the course so that's all i had to cover for this lesson so you can now mark this lesson as complete and let's move on to the next one welcome back in this lesson we're going to learn about how resources and entities are organized within google cloud and how permissions are inherited through this approach knowing this structure is a fundamental concept that you should know while working in gcp at any capacity so before defining what the resource hierarchy is i'd like to take a little bit of time to define what is a resource now in the context of google cloud a resource can refer to the service level resources that are used to process your workloads such as compute instance vms cloud storage buckets and even cloud sql databases as well as the account level resources that sit above the services such as the organization itself the folders and the projects of course which we will be getting into a little bit deeper in just a minute the resource hierarchy is google's way to configure and grant access to the various cloud resources for your company within google cloud both at the service level and at the account level the resource hierarchy in google cloud can truly define the granular permissions needed for when you need to configure permissions to everyone in the organization that actually makes sense so now that we covered what is a resource i wanted to start digging into the resource hierarchy and the structure itself now google cloud resources are organized hierarchically using a parent-child relationship this hierarchy is designed to map an organization's operational structure to google cloud and to manage access control and permissions for groups of related resources so overall resource hierarchy will give organizations better management of permissions and access control the accessibility of these resources or policies are controlled by identity and access management also known as iam a big component of gcp which we will be digging into a little bit later on in this course and so when an iam policy is set on a parent the child will inherit this policy respectively access control policies and configuration settings on a parent resource are always inherited by the child also please note that each child object can only have exactly one parent and that these policies are again controlled by iam so now to understand a little bit more about how the gcp resource hierarchy works i wanted to dig into the layers that support this hierarchy so this is a diagram of exactly what the resource hierarchy looks like in all of its awesomeness including the billing account along with the payments profile but we're not going to get into that right now i'll actually be covering that in a later lesson so more on that later so building the structure from the top down we start off with the domain or cloud level and as you can see here the domain of bowtieinc.co is at the top this is the primary identity of your organization at the domain level this is where you manage your users in your organizations so users policies and these are linked to g suite or cloud identity accounts now underneath the domain level we have the organization level and this is integrated very closely with the domain so with the organization level this represents an organization and is the root node of the gcp resource hierarchy it is associated with exactly one domain here we have the domain set as bowtie inc all entities or resources belong to and are grouped under the organization all controlled policies applied to the organization are inherited by all other entities and resources underneath it so any folders projects or resources will get those policies that are applied from the organization layer now i know that we haven't dug into roles as of yet but the one thing that i did want to point out is that when an organization is created an organization admin role is created and this is to allow full access to edit any or all resources now moving on to the folders layer this is an additional grouping mechanism and isolation boundary between each project in essence it's a grouping of other folders projects and resources so if you have different departments and teams within a company this is a great way to organize it now a couple of caveats when it comes to folders the first one is you must have an organization node and the second one is while a folder can contain multiple folders or resources a folder or resource can have exactly one parent now moving into the projects layer this is a core organizational component of google cloud as projects are required to use service level resources these projects are the base level organizing entity in gcp and parent all service level resources just as a note any given resource can only exist in one project and not multiple projects at the same time and moving on to the last layer we have the resources layer and this is any service level resource created in google cloud everything from compute engine instances to cloud storage buckets to cloud sql databases apis users all these service level resources that we create in google cloud fall under this layer now giving the hierarchy a little bit more context i want to touch on labels for just a second labels help categorize resources by using a key value pair and you can attach them to any resource and so what labels help you do is to break down and organize costs when it comes to billing now to give you some more structure with regards to the hierarchy under the domain level everything underneath this is considered a resource and to break it down even further everything you see from the organization layer to the projects layer is considered an account level resource everything in the resource layer is considered a service level resource and so this is how the google cloud resource hierarchy is split up and organized and so before i finish off this lesson i wanted to give you a quick run-through on how policies can be applied at a hierarchical level so i thought i'd bring in tony bowtie for a quick demo so just to give you an example tony bowtie is part of department b and tony's manager lark decides to set a policy on department b's folder and this policy grants project owner role to tony at bowtieinc.co so tony will have the project owner role for project x and for project y at the same time lark assigns laura at bowtieinc.co cloud storage admin role on project x and thus she will only be able to manage cloud storage buckets in that project this hierarchy and permission inheritance comes up quite a bit not only in the exam but is something that should be carefully examined when applying permissions anywhere within the hierarchy in your day-to-day role as an engineer applying permissions or policies to resources with existing policies may not end up getting you the desired results you're looking for and may have a chance to be overlooked now i hope these diagrams have given you some good contacts with regards to resource hierarchy its structure and the permissions applied down the chain now that's all i have for this lesson on resource hierarchy so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in this lesson i will be covering a few different topics that i will touch on when creating a new google cloud account i will be covering going over the free tier and the always free options the differences between them and a demo showing how you can create your own free tier account as well i'll also be going into what you will need in order to fulfill this demo so for the remainder of this course all the demos will run under the free tier now when i built this course i built it with budget in mind and having viewed on ways where i can keep the price to a minimum while still keeping the demos extremely useful and so the free tier falls within all these guidelines and will help you learn without the high ticket price and so getting into a quick overview of the differences between the free tier and the always free option i have broken them down here with their most significant differences in the free tier google cloud offers you a 12 month free trial with a 300 u.s credit this type of account ends when the credit is used or after the 12 months whichever happens first and so for those of you who are looking at taking advantage of this on a business level unfortunately the free tier only applies to a personal account and cannot be attached to a business account now moving over to the always free option the always free option isn't a special program but it's a regular part of your google cloud account it provides you limited access to many of the google cloud resources free of charge and once these limits have been hit then you are charged at the regular per second billing rate and i will show you a little bit later how to monitor these credits so that you don't go over using this in conjunction with the free tier account is not possible you have to have an upgraded billing account which can also include a business account now there are a bunch more stipulations in this program and i will include a link to both of them in the lesson text below for later viewing at your convenience now lastly before we get into the demo i wanted to go through a quick run-through of exactly what's needed to open up your free tier account so we're going to start off with a fresh new gmail address so that it doesn't conflict with any current gmail address that you may have you're gonna need a credit card for verification and this is for google to make sure that you're an actual human being and not a robot and you won't be charged unless you go above the 300 credit limit as well i highly recommend going into a private browsing session so whether you're using chrome you would use an incognito session if you're using firefox you would use private browsing and in microsoft edge you would be using the in private mode and so in order to start with this free trial you can head on over to the url listed here and i'll also include this in the lesson text so head on over to this url and i'll see you there in just a second okay so here we are at the free trial url i'm here in google chrome in an incognito session and so we're not going to sign up we're going to go over here to create account you can just click on create account for myself because as i mentioned earlier you're not able to create a free trial account with your business so i'm going to click on for myself and it's going to bring you to this page where it says create your google account and you're going to go to create a new gmail address instead and now you're going to fill in all the necessary information that's needed in order to open up this new gmail account once you're finished typing your password you can hit next and now i got prompted for six digit verification code that i have to plug in but in order to do that google needs my telephone number so i'm gonna type that in now and just to let you know this verification is done to let google know that you're not a bot and you're a real human and google just sent me a verification code and this is a one-time verification code that i'm going to plug in and i'm going to hit verify and you can plug in the necessary information here for recovery email address your birthday and gender and this is so that google can authenticate you in case you accidentally misplace your password and then just hit next and here google gives you a little bit more information on what your number can be used for and so i'm going to go ahead and skip it and of course we're going to read through the terms of service and the privacy policy click on agree and as you can see we're almost there it shows here that we're signing up for the free trial i'm in canada so depending on your country this may change of course i read the terms of service and i'm going to agree to it and i don't really want any updates so you can probably skip that and just hit continue and so this is all the necessary information that needs to be filled out for billing and so here under account type be sure to click on individual as opposed to business and again fill in all the necessary information with regards to your address and your credit card details and once you fill that in you can click on start my free trial and once you've entered in all that information you should be brought to this page with a prompt asking you exactly what you need with regards to google cloud and you can just hit skip here and i'm going to zoom in here just see a little better and so here you're left with a checklist where you can go through all the different resources and it even gives you a checklist to go through but other than that we're in and so just to verify that we're signed up for a free tier account i'm going to go over to billing and i'm going to see here that i have my free trial credit and it says 411 dollars and due to the fact that my currency is in canadian dollars it's been converted from us dollars and so we'll be going through billing in a later lesson but right now we are actually logged in and so that's all i wanted to cover for this lesson on how to sign up for your free trial account so you can now mark this lesson as complete and you can join me in the next one where we will secure the account using a method called two-step verification [Music] welcome back so in the last lesson we went ahead and created a brand new gcp account in this lesson we'll be discussing how to secure that gcp account by following some best practices whenever any account is created in google cloud and this can be applied with regards to personal accounts as well as the super admin account as it's always good to keep safety as a priority this lesson may be a refresher for those who are a bit more advanced as for everyone else these steps could help you from an attack on your account i'd first like to run you through a scenario of the outcome on both secure and non-secure accounts as well as the different options that reside in google cloud when it comes to locking down your account i'll then run through a hands-on demo in the console to show you how you can apply it yourself so in this specific scenario a username and password is used to secure the account here lark a trouble causing manager looks over the shoulder of tony bowtie while he plugs in his username and password so that he can later access his account to wreak havoc on tony's reputation as tony leaves for coffee lark decides to log in and send a company-wide email from tony's account to change an already made decision about next season's store opening in rome italy that would not look good for tony it was that easy for lark to steal tony's password and in a real life scenario it would be that easy for someone to steal your password now when someone steals your password they could do even more devious things than what lark did not just sending out harmful emails they could lock you out of your account or even delete emails or documents this is where two-step verification comes in this can help keep bad people out even if they have your password two-step verification is an extra layer of security most people only have one layer to protect their account which is their password with two-step verification if a bad person hacks through your password they'll still need your phone or security key to get into your account so how two-step verification works is that sign-in will require something you know and something that you have the first one is to protect your account with something you know which will be your password and the second is something that you have which is your phone or security key so whenever you sign into google you'll enter your password as usual then a code will be sent to your phone via text voice call or google's mobile app or if you have a security key you can insert it into your computer's usb port codes can be sent in a text message or through a voice call depending on the setting you choose you can set up google authenticator or another app that creates a one-time verification code which is great for when you're offline you would then enter the verification code on the sign in screen to help verify that it is you another way for verification is using google prompts and this can help protect against sim swap or other phone number based hacks google prompts are push notifications you'll receive on android phones that are signed into your google account or iphones with the gmail app or google app that's signed into your google account now you can actually skip a second step on trusted devices if you don't want to provide a second verification step each time you sign in on your computer or your phone you can check the box next to don't ask again on this computer and this is a great added feature if you are the only user on this device this feature is not recommended if this device is being used by multiple users security keys are another way to help protect your google account from phishing attacks when a hacker tries to trick you into giving them your password or other personal information now a physical security key is a small device that you can buy to help prove it's you signing in when google needs to make sure that it's you you can simply connect your key to your computer and verify that it's you and when you have no other way to verify your account you have the option of using backup codes and these are one-time use codes that you can print or download and these are multiple sets of eight-digit codes that you can keep in a safe place in case you have no other options for verification i personally have found use in using these backup codes as i have used them in past when my phone died so ever since lark's last email tony not only changed his password but added a two-step verification to his account so that only he would have access and would never have to worry again about others looking over his shoulder to gain access to his account as tony leaves for coffee lark tries to log in again but is unsuccessful due to the two-step verification in place tony has clearly outsmarted the bad man in this scenario and lark will have to look for another way to foil tony's plan to bring greatness to bow ties across the globe and this is a sure difference between having a secure account and a not so secure account and so now that i've gone through the theory of the two-step verification process i'm going to dive into the console and implement it with the hands-on demo just be aware that you can also do this through the gmail console but we're going to go ahead and do it through the google cloud console using the url you see here so whenever you're ready feel free to join me in the console and so here we are back in the console and over here on the top right hand corner you will find a user icon and you can simply click on it and click over to your google account now i'm just going to zoom in for better viewing and so in order to enable two-step verification we're gonna go over here to the menu on the left and click on security and under signing into google you will find two-step verification currently it's off as well as using my phone to sign in is off so i'm going to click on this bar here for two-step verification and i definitely want to add an extra layer of security and i definitely want to keep the bad guys out so i'm going to go ahead and click on the get started button it'll ask me for my password and because i've entered my phone number when i first signed up for the account it actually shows up here this is i antony which is my iphone and so now i can get a two-step verification here on my iphone and again this is going to be a google prompt as it shows here but if i wanted to change it to something else i can simply click on show more options and here we have a security key as well as text message or voice call i highly recommend the google prompt as it's super easy to use with absolutely no fuss and so as i always like to verify what i've done i'm going to click on this try it now button and so because i wanted to show you exactly what a live google prompt looks like i'm going to bring up my phone here on the screen so that you can take a look and it actually sent me a google prompt to my phone and i'm just going to go ahead and open up my gmail app so i can verify that it is indeed me that wants to log in which i will accept and so once i've accepted the google prompt another window will pop up asking me about a backup option and so i'll simply need my phone number and i can either get a text message or a phone call and again you have other options as well so you can use the one-time backup codes which we discussed earlier and you can print or download them but i usually like to use a text message and so i'm going to use that i'm going to send it to my phone and so just to verify it i'm gonna now plug in the one-time code that was sent to me and then just hit next so the second step is the google prompt it's my default and my backup options if i can't get google prompt is a voice or text message and again this is for my account antony gcloud ace at gmail.com sending it to my i antony device so turn on two-step verification absolutely and so there you have it there is two-step verification enabled and if i wanted to change the available steps i can do so here i can also edit it i can edit my phone number and i can also set up any backup codes in case i need it in my personal opinion two-step verification is a must-have on any account best practice is to always do it for your super admin account which would be my gmail account that i am currently signed up with but i find is a necessity for any other users and always make it a policy for people to add two-step verification to their accounts i highly recommend that you make it your best practice to do this in your role as an engineer in any environment at any organization again two-step verification will allow to keep you safe your users safe and your environment safe from any malicious activities that could happen at any time and that's all i have for this lesson on two-step verification and securing your account so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back now there are many different ways in which you can interact with google cloud services and resources this lesson is an overview of the gcp console and how you can interact with it using the graphical user interface and so for this hands-on demo i will be diving into how to navigate through the gcp console and point out some functions and features that you may find helpful so with that being said let's dive in and so here we are back in the console up here you can see the free trial status and then i still have 410 credit again this is canadian dollars so i guess consider me lucky so i'm going to go ahead over here and dismiss this don't activate it because otherwise this will kill your free trial status and you don't want to do that so i'm just going to hit dismiss so over here on the main page you have a bunch of cards here that will give you the status of your environment as well as the status of what's happening within google cloud with these cards you can customize them by hitting this button over here customize and you can turn them on or off and you can go ahead and move these around if you'd like and i'm going to put this up here as well i'm going to turn on my billing so i can keep track of exactly what my spend is i don't really need my get starting card so i'm going to turn that off as well as the documentation i'm going to turn that off as well and the apis is always nice to have as well up here on the project info this reflects the current project which is my first project and the project name here is the same the project id is showing and the project number and i'm going to dive deeper into that in another lesson also note that your cards will reflect exactly what it is that you're interacting with and so the more resources that you dive into the cards will end up showing up here and you can add them and turn them off at will so i'm going to go up here and click on done because i'm satisfied with the way that things look here on my home page and over here to your left i wanted to focus on all the services that are available in their own specific topics so for instance all of compute you will find app engine compute engine kubernetes and so on so note that anything compute related you'll find them all grouped together also another great feature is that you can pin exactly what it is that you use often so if i am a big user of app engine i can pin this and it will move its way up to the top this way it saves me the time from having to go and look for it every time i need it and if i'm using it constantly it's great to have a shortcut to unpin it i simply go back to the pin and click on it again as well if i'd like to move the menu out of the way to get more screen real estate i can simply click on this hamburger button here and make it disappear and to bring it back i can just click on that again and i'll bring it back again now i know that there's a lot of resources here to go through so if you're looking for something specific you can always go up to the search bar right here and simply type it in so if i'm looking for let's say cloud sql i can simply type in sql and i can find it right here i can find the api and if anything associated with the word sql if i'm looking for cloud sql specifically i can simply type in cloud sql and here it is another thing to note is that if you want to go back to your homepage you can simply go up to the left hand corner here and click on the google cloud platform logo and it'll bring you right back and right here under the google cloud platform logo you'll see another set of tabs we have dashboard we also have activity and this will show all the latest activity that's been done and because this is a brand new account i don't have much here now because this is my first time in activity this is going to take some time to index and in the meantime i wanted to show you filters if this were a long list to go through where activity has been happening for months i can filter through these activities either by user or by categories or by resource type as well as the date i can also combine these to search for something really granular and beside the activity tab we have recommendations which is based on the recommender service and this service provides recommendations and insights for using resources on google cloud these recommendations and insights are on a per product or per service basis and they are based on machine learning and current resource usage a great example of a recommendation is vm instance right sizing so if the recommender service detects that a vm instance is underutilized it will recommend changing the machine size so that i can save some money and because this is a fresh new account and i haven't used any resources this is why there is no recommendations for me so going back to the home page i want to touch on this projects menu for a second and as you can see here i can select a project now if i had many different projects i can simply search from each different one and so to cover the last part of the console i wanted to touch on this menu on the top right hand corner here so clicking on this present icon will reveal my free trial status which i dismissed earlier next to the present we have a cloud shell icon and this is where you can activate and bring up the cloud shell which i will be diving into deeper in a later lesson and right next to it is the help button in case you need a shortcut to any documentations or tutorials as well some keyboard shortcuts may help you be a little bit more efficient and you can always click on this and it'll show you exactly what you need to know and so i'm going to close this and to move over to the next part in the menu this is the notifications so any activities that happen you will be notified here and you can simply click on the bell and it'll show you a bunch of different notifications for either resources that are created or any other activities that may have happened now moving on over three buttons over here is the settings and utilities button and over here you will find the preferences and under communication you will find product notifications and updates and offers and you can turn them off or on depending on whether or not you want to receive these notifications as well you have your language and region and you can personalize the cloud console as to whether or not you want to allow google to track your activity and this is great for when you want recommendations so i'm going to keep that checked off getting back to some other options you will find a link to downloads as well as cloud partners and the terms of service privacy and project settings and so to cover the last topic i wanted to touch on is the actual google account button and here you can add other user accounts for when you log into the console with a different user as well as go straight to your google account and of course if you're using a computer that's used by multiple users you can sign out here as well and so that's just a quick run-through of the console and so feel free to poke around and get familiar with exactly what's available in the console so that it's a lot easier for you to use and allow you to become more efficient and so that's all i have for this lesson so you can now mark this lesson as complete and let's move on to the next one welcome back in this lesson i'm going to be going through a breakdown of cloud billing and an overview of the various resources that's involved with billing billing is important to know and i'll be diving into the concepts around billing and billing interaction over the next few lessons as well i'll be getting into another demo going through the details on how to create edit and delete a cloud billing account now earlier on in the course i went over the resource hierarchy and how google cloud resources are broken down starting from the domain level down to their resource level this lesson will focus strictly on the billing account and payments profile and the breakdown are concepts that are comprised within them so getting right into it let's start with the cloud billing account a cloud billing account is a cloud level resource managed in the cloud console this defines who pays for a given set of google cloud resources billing tracks all of the costs incurred by your google cloud usage as well it is connected to a google payments profile which includes a payment method defining on how you pay for your charges a cloud billing account can be linked to one or more projects and not to any one project specifically cloud billing also has billing specific roles and permissions to control accessing and modifying billing related functions that are established by identity and access management cloud billing is offered in two different account types there is the self-service or online account or you can also choose from the invoiced or offline payments when it comes to the self-service option the payment method is usually a credit or debit card and costs are charged automatically to the specific payment method connected to the cloud billing account and when you need access to your invoices you can simply go to the cloud console and view them online now when it comes to the invoice account first you must be eligible for invoice billing once you are made eligible the payment method used can be check or wire transfer your invoices are sent by mail or electronically as well they're also available in the cloud console as well as the payment receipts now another cool feature of billing account is sub-accounts and these are intended for resellers so if you are a reseller you can use subaccounts to represent your customers and make it easy for chargebacks cloud billing subaccounts allow you to group charges from projects together on a separate section of your invoice and is linked back to the master cloud billing account on which your charges appear sub-accounts are designed to allow for customer separation and management so when it comes to ownership of a cloud billing account it is limited to a single organization it is possible though for a cloud billing account to pay for projects that belong to an organization that is different than the organization that owns the cloud billing account now one thing to note is that if you have a project that is not linked to a billing account you will have limited use of products and services available for your project that is projects that are not linked to a billing account cannot use google cloud services that aren't free and so now that we've gone through an overview of the billing account let's take a quick step into the payments profile now the payments profile is a google level resource managed at payments.google.com the payments profile processes payments for all google services and not just for google cloud it connects to all of your google services such as google ads as well as google cloud it stores information like your name address and who is responsible for the profile it stores your various payment methods like credit cards debit cards and bank accounts the payments profile functions as a single pane of glass where you can view invoices payment history and so on it also controls who can view and receive invoices for your various cloud billing accounts and products now one thing to note about payments profile is that there are two different types of payment profiles the first one is individual and that's when you're using your account for your own personal payments if you register your payments profile as an individual then only you can manage the profile you won't be able to add or remove users or change permissions on the profile now if you choose a business profile type you're paying on behalf of a business or organization a business profile gives you the flexibility to add other users to the google payments profile you manage so that more than one person can access or manage a payments profile all users added to a business profile can then see the payment information on that profile another thing to note is that once the profile type has been selected it cannot be changed afterwards and so now that we've quickly gone through an overview of all the concepts when it comes to billing i am now going to run through a short demo where i will create a new billing account edit that billing account and show you how to close a billing account so whenever you're ready join me in the console and so here i am back in the console and so the first thing i want to do is i want to make sure that i have the proper permissions in order to create and edit a new billing account so what i'm going to do is go over here to the hamburger menu up here in the top left hand corner and click on it and go over to i am an admin and over to iam now don't worry i'm not going to get really deep into this i will be going over this in a later section where i'll go through iam and roles but i wanted to give you a sense of exactly what you need with regards to permissions so now that i'm here i'm going to be looking for a role that has to do with billing so i'm simply going to go over here on the left hand menu and click on roles and you'll have a slew of roles coming up and what you can do is filter through them just by simply typing in billing into the filter table here at the top and as you can see here there is billing account administrator billing account creator and so on and so forth and just to give you a quick overview on these roles and so for the billing account administrator this is a role that lets you manage billing accounts but not create them so if you need to set budget alerts or manage payment methods you can use this role the billing account creator allows you to create new self-serve online billing accounts the billing account user allows you to link projects to billing accounts the billing account viewer allows you to view billing account cost information and transactions and lastly the project billing manager allows you to link or unlink the project to and from a billing account so as you can see these roles allow you to get pretty granular when it comes to billing so i'm going to go back over to the left hand menu over on iam and click on there and i want to be able to check my specific role and what permissions that i have or i will need in order to create a new billing account and so if i click on this pencil it'll show me exactly what my role is and what it does and as it says here i have full access to all resources which means that i am pretty much good to go so i'm going to cancel out here and i'm going to exit i am an admin so i'm going to click on the navigation menu and go over to billing and so this billing account is tied to the current project and because it's the only billing account it's the one that shows up and so what i want to do is i want to find out a little bit more information with regards to this billing account so i'm going to move down the menu and click on account management here i can see the billing account which is my billing account i can rename it if i'd like and i can also see the projects that are linked to this billing account so now that we've viewed all the information with regards to the my billing account i'm going to simply click on this menu over here and click on the arrow and go to manage billing accounts and here it will bring me to all my billing accounts and because i only have one is shown here my billing account but if i had more than one they would show up here and so now in order for me to create this new billing account i'm going to simply click on create account and i will be prompted with a name a country and a currency for my new billing account and i'm actually going to rename this billing account and i'm going to rename it to gcloud ace dash billing i'm going to leave my country as canada and my currency in canadian dollars and i'm going to simply hit continue and it's giving me the choice in my payments profile and because i want to use the same payments profile i'm just going to simply leave everything as is but for demonstration purposes over here you can click on the payments profile and the little arrow right beside the current profile will give me the option to create a new payments profile and we're going to leave that as is under customer info i have the option of changing my address and i can click on this pencil icon and change it as well i can go to payment methods and click on the current payment method with that little arrow and add a new credit or debit card and as i said before we're going to keep things the way they are and just hit submit and enable billing now as you can see here i got a prompt saying that a confirmation email will be sent within 48 hours now usually when you're setting up a brand new billing profile with an already created payments profile you'll definitely get a confirmation email in less than 48 hours now in order for me to finish up this demo i'm gonna wait until the new billing account shows up and continue with the demo from then and so here i am back in the billing console and it only took about 20 minutes and the gcloud ace billing account has shown up and so with part of this demo what i wanted to show is how you can take a project and attach it to a different billing account and so currently my only project is attached to the my billing account so now if i wanted to change my first project to my gcloud ace dash billing account i can simply go over here to actions click on the hamburger menu and go to change billing here i'll be prompted to choose a billing account and i can choose g cloud a stash billing and then click on set account and there it is my first project is now linked to g cloud a stash billing so if i go back over to my billing accounts you can see here that my billing account currently has zero projects and g cloud a stash billing has one project now just as a quick note and i really want to emphasize this is that if you're changing a billing account for a project and you are a regular user you will need the role of the billing account administrator as well as the project owner role so these two together will allow a regular user to change a billing account for a project and so now what i want to do is i want to take the gcloud a stash billing and i want to close that account but before i do that i need to unlink this project and bring it back to another billing account which in this case would be my billing account so i'm going to go back up here to the menu click on my projects and we're going to do the exact same thing that we did before under actions i'm going to click on the hamburger menu and change billing i'm going to get the prompt again and under billing account i'm going to choose my billing account and then click on set account so as you can see the project has been moved to a different billing account i'm going to go back to my billing accounts and as you can see here the project is back to my billing account and so now that the project is unlinked from the gcloud a stash billing account i can now go ahead and close out that account now in order to do that i'm going to click on gcloud a stash billing i'm going to go down here on the hand menu all the way to the bottom to account management click on there and at the top here you will see close billing account i'm going to simply click on that and i'll get a prompt that i've spent zero dollars and is linked to zero projects now if i did have a project that was linked to this billing account i would have to unlink the project before i was able to close this billing account so as a failsafe i'm being asked to type close in order to close this billing account so i'm going to go ahead and do that now and click on close billing account just as a note google gives me the option to reopen this billing account in case i did this by mistake and i really needed it i can reopen this billing account so now moving back over to billing you'll see here that i'm left with my single billing account called my billing account with the one project that's linked to it and so that covers my demo on creating editing and closing a new billing account as well as linking and unlinking a project to and from a different billing account so i hope you found this useful and you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in this lesson i'm going to be going over controlling costs in google cloud along with budget alerts i will be touching on all the available discounts the number of ways to control costs and go over budget alerts to get a more granular and programmatic approach so starting off i wanted to touch on committed use discounts now committed use discounts provide discounted prices in exchange for your commitment to use a minimum level of resources for a specified term the discounts are flexible cover a wide range of resources and are ideal for workloads with predictable resource needs when you purchase google cloud committed use discounts you commit to a consistent amount of usage for a one or three year period there are two commitment types available and as you can see here they are spend based and resource based commitment types and unlike most other providers the commitment fee is billed monthly so going over the specific commitment types i wanted to start off with spend based commitment now for spend based commitment you commit to a consistent amount of usage measured in dollars per hour of equivalent on-demand spend for a one or three year term in exchange you receive a discounted rate on the applicable usage your commitment covers so you can purchase committed use discounts from any cloud billing account and the discount applies to any eligible usage in projects paid for by that cloud billing account any overage is charged at the on-demand rate spend based commitments can give you a 25 discount off on-demand pricing for a one-year commitment and up to a 52 discount off of on-demand pricing for a three-year commitment now spend-based commitments are restricted to specific resources which is cloud sql database instances and google cloud vmware engine and this commitment applies to the cpu and memory usage for these available resources now the other committed use discount is the resource-based commitment so this discount is for a commitment to spend a minimum amount for compute engine resources in a particular region resource-based commitments are ideal for predictable workloads when it comes to your vms when you purchase a committed use contract you purchase compute resources such as vcpus memory gpus and local ssds and you purchase these at a discounted price in return for committing to paying for those resources for one or three years the discount is up to 57 percent for most resources like machine types or gpus the discount is up to 70 percent for memory optimized machine types and you can purchase a committed use contract for a single project or purchase multiple contracts which you can share across many project by enabling shared discounts and sharing your committed use discounts across all your projects reduces the overhead of managing discounts on a per project basis and maximizes your savings by pooling all of your discounts across your project's resource usage if you have multiple projects that share the same cloud billing account you can enable committed use discount sharing so all of your projects within that cloud billing account share all of your committed use discount contracts and so your sustained use discounts are also pooled at the same time so touching on sustained use discounts these are automatic discounts for running specific compute engine resources a significant portion of the billing month sustained use discounts apply to the general purpose compute and memory optimize machine types as well as sole tenant nodes and gpus again sustained use discounts are applied automatically to usage within a project separately for each region so there's no action required on your part to enable these discounts so for example when you're running one of these resources for more than let's say 25 percent of the month compute engine automatically gives you a discount for every incremental minute that you use for that instance now sustained use discounts automatically apply to vms created by both google kubernetes engine and compute engine but unfortunately do not apply to vms created using the app engine flexible environment as well as data flow and e2 machine types now to take advantage of the full discount you would create your vm instances on the first day of the month as discounts reset at the beginning of each month and so the following table shows the discount you get at each usage level of a vm instance these discounts apply for all machine types but don't apply to preemptable instances and so sustained use discounts can save you up to a maximum of a 30 percent discount so another great way to calculate savings in google cloud is by using the gcp pricing calculator this is a quick way to get an estimate of what your usage will cost on google cloud so the gcp pricing calculator can help you identify the pricing for the resources that you plan to use in your future architecture so that you are able to calculate how much your architecture will cost you this calculator holds the pricing for almost all resources encapsulated within gcp and so you can get a pretty good idea of what your architecture will cost you without having to find out the hard way this calculator can be found at the url shown here and i will include this in the lesson text below now moving right along to cloud billing budgets so budgets enable you to track your actual spend against your plan spend after you've set a budget amount you set budget alert threshold rules that are used to trigger email notifications and budget alert emails help you stay informed about how your spend is tracking against your budget this example here is a diagram of a budget alert notification and is the default functionality for any budget alert notifications now to get a little bit more granular you can define the scope of the budget so for example you can scope the budget to apply to the spend of an entire cloud billing account or get more granular to one or more projects and even down to a specific product you can set the budget amount to a total that you specify or base the budget amount on the previous month's spend when costs exceed a percentage of your budget based on the rules that you set by default alert emails are sent to billing account administrators and billing account users on the target cloud billing account and again this is the default behavior of a budget email notification now as said before the default behavior of a budget is to send alert emails to billing account administrators and billing account users on the target cloud billing account when the budget alert threshold rules trigger an email notification now these email recipients can be customized by using cloud monitoring to specify other people in your organization to receive these budget alert emails a great example of this would be a project manager or a director knowing how much spend has been used up in your budget and the last concept i wanted to touch on when it comes to cloud billing budgets is that you can also use pub sub for programmatic notifications to automate your cost control response based on the budget notification you can also use pub sub in conjunction with billing budgets to automate cost management tasks and this will provide a real-time status of the cloud billing budget and allow you to do things like send notifications to slack or disable billing to stop usage as well as selectively control usage when budget has been met and so these are all the concepts that i wanted to cover when it came to cloud billing budgets now i know this lesson may have been a bit dry and not the most exciting service to dive into but it is very important to know both for the exam and for your role as an engineer when it comes to cutting costs in environments where your business owners deem necessary and so that's all i had for this lesson so you can now mark this lesson as complete and please join me in the next one where i dive into the console and do some hands-on demos when it comes to committed use discounts budget alerts and editing budget alerts as well as adding a little bit of automation into the budgeting alerts [Music] welcome back in the last lesson i went over a few ways to do cost management and the behaviors of budget alerts in this lesson i will be doing a demo to show you committed use discounts and reservations along with how to create budget alerts and as well how to edit them so with that being said let's dive in so now i'm going to start off with committed use discounts in order to get there i'm going to find it in compute engine so i'm going to simply go up here on the top left hand corner back to the navigation menu i'm going to go down to compute engine and i'm going to go over here to committed use discounts and as we discussed earlier these commitments for compute engine are resource based and as you can see here we have hardware commitments and reservations now reservations i will get into just a little bit later but with regards to hardware commitments we're going to get into that right now and as expected i have no current commitments so i'm going to go up to purchase commitment and so i need to start off with finding a name for this commitment and so i'm going to name this commitment demo dash commitment it's going to ask me for a region i'm going to keep it in us central one with the commitment type here is where i can select the type of machine that i'm looking for so i can go into general purpose and 1 and 2 and 2d e2 as well as memory optimize and compute optimized and so i'm going to keep it at general purpose and one again the duration one or three years and we get down to cores i can have as many vcpus as i'd like so if i needed 10 i can do that and i'll get a pop-up here on the right showing me the estimated monthly total as well as an hourly rate for this specific vm with 10 cores i can also select the duration for three years and as expected i'll get a higher savings because i'm giving a bigger commitment so bring it back down to one year and let's put the memory up to 64 gigabytes here i can add gpus and i have quite a few to choose from as well as local ssds and here with the local ssds i can choose as many disks as i'd like as long as it's within my quota and each disk size is going to be 375 gigabytes so if you're looking into committed use discounts and using local ssds please keep that in mind again the reservation can be added here and i'll be getting into that in just a second and now i don't want to actually purchase it but i did want to show you exactly what a committed use discount would look like and how you would apply it again here on the right hand side it shows me the details of the estimated monthly total and the hourly rate so i'm going to go over here and hit cancel and if i were to have applied it the commitment would show up here in this table and give me all the specified configurations of that instance right here now touching on reservations reservations is when you reserve the vm instances you need so when the reservation has been placed the reservation ensures that those resources are always available for you as some of you might know when you go to spin up a new compute engine vm especially when it comes to auto scaling instance groups the instances can sometimes be delayed or unavailable now the thing with reservations is that a vm instance can only use a reservation if its properties exactly match the properties of the reservation which is why it's such a great pairing with committed use discounts so if you're looking to make a resource-based commitment and you always want your instance available you can simply create a reservation attach it to the commitment and you will never have to worry about having the resources to satisfy your workload as they will always be there so again going into create reservation it'll show me here the name the description i can choose to use the reservation automatically or select a specific reservation the region and zone number of instances and here i can specify the machine type or specify an instance template and again this is another use case where if you need compute engine instances spun up due to auto scaling this is where reservations would apply so getting back to machine type i can choose from vcpus as well as the memory i can customize it i can add as many local ssds as my quotas will allow me and i can select my interface type and i'm going to cancel out of here now when it comes to committed use discounts and reservations as it pertains to the exam i have not seen it but since this is an option to save money i wanted to make sure that i included it in this lesson as this could be a great option for use in your environment so now that we covered resource-based committed use discounts i wanted to move into spend based commitments and so where you would find that would be over in billing so again i'm going to go up to the navigation menu in the top left hand corner and go into billing now you'd think that you would find it here under commitments but only when you have purchased a commitment will it actually show up here but as you can see here it's prompting us to go to the billing overview page so going back to the overview page you'll find it down here on the right and so i can now purchase a commitment and as we discussed before a spend based commitment can be used for either cloud sql or for vmware engine i select my billing account the commitment name the period either one year or three years and it also shows me the discount which could help sway my decision as well as the region as well as the hourly on-demand commitment now you're probably wondering what this is and as explained here this commitment is based on the on-demand price and once this is all filled out the commitment summary will be populated and after you agree to all the terms and services you can simply hit purchase but i'm going to cancel out of here and so that is an overview for the spend based commitment and again these committed use discounts i have not seen on the exam but i do think that it's good to know for your day-to-day environment if you're looking to save money and really break down costs so now that i've covered committed use discounts and reservations i wanted to move over to budgets and budget alerts and because i'm already on the billing page all i need to do is go over here to the left hand menu and click on budgets and alerts now setting up a budget for yourself for this course would be a great idea especially for those who are cost conscious on how much you're spending with regards to your cloud usage and so we're to go ahead and create a new budget right now so let's go up here to the top to create budget and i'm going to be brought to a new window where i can put in the name of the budget and i'm going to call this ace dash budget and because i want to monitor all projects and all products i'm going to leave this as is but if you did have multiple projects you could get a little bit more granular and the same thing with products so i'm going to go ahead and leave it as is and just click on next now under budget type i can select from either a specified amount or the last month's spend and so for this demo i'm going to keep it at specified amount and because i want to be really conscious about how much i spend in this course i'm going to put in 10 for my target amount i'm going to include the credits and cost and then i'm going to click on next now these threshold rules are where billing administrators will be emailed when a certain percent of the budget is hit so if my spend happens to hit five dollars because i am a billing administrator i will be sent an email telling me that my spend has hit five dollars i also have the option of changing these percentages so if i decided to change it to forty percent now my amount goes to four dollars and this is done automatically so no need to do any calculations but i'm going to keep this here at 50 percent and vice versa if i wanted to change the amount the percentage of budget will actually change now with the trigger i actually have the option of selecting forecasted or actual and so i'm going to keep it on actual and if i want i can add more threshold rules now i'm going to leave everything as is and just click on finish and now as you can see here i have a budget name of ace budget now because the budget name doesn't have to be globally unique in your environment you can name your budget exactly the same and again it'll give me all the specific configurations that i filled out shows me how much credits i've used and that's it and that's how you would create a budget alert now if i needed to edit it i can always go back to ace budget and here i can edit it but i'm not going to touch it and i'm just going to hit cancel and so the last thing i wanted to show you before we end this lesson is how to create another budget but being able to send out the trigger alert emails to different users and so in order to do that i'm going to go back up here to create budget i'm going to name this to ace dash budget dash users i'm going to leave the rest as is i'm going to click on next again i'm going to leave the budget type the way it is the target amount i'm going to put ten dollars leave the include credits and cost and just click on next and so here i'm going to leave the threshold rules the way they are and right here under manage notifications i'm going to click off link monitoring email notification channels to this budget now because the email notification channel needs cloud monitoring in order to work i am prompted here to select a workspace which is needed by cloud monitoring so because i have none i'm going to go ahead and create one and so clicking on managing monitoring workspaces will bring you to the documentation but in order for me to get a workspace created i need to go to cloud monitoring now workspace is the top level container that is used to organize and control access to your monitoring notification channels in order for your notification channels to work they must belong to a monitoring workspace so you need to create at least one workspace before adding monitoring notification channels and don't worry we'll be getting into greater depth with regards to monitoring in a later section in this course so i'm going to go ahead and cancel this and i'm going to go up to the navigation menu click on there and scroll down to monitoring and then overview and this may take a minute to start up as the apis are being enabled and the default workspace for cloud monitoring is being built okay and now that the monitoring api has been enabled we are now in monitoring the workspace that was created is my first project so now that we have our monitoring workspace created i need to add the emails to the users that i want the alerts to be sent out to and added to the notification channel so in order to do that i'm going to go over here to alerting and up here at the top i'm going to click on edit notification channels and here as you can see are many notification channels that you can enable by simply clicking on add new over here on the right so now what i'm looking for is under email i'm going to click on add new now here i can add the new email address and so for me i'm going to add antony at antonyt.com and you can add whatever email address you'd like and under display name i'm going to add billing admin notification and just click on save and as you can see my email has been added to the notification channel and so this is all i needed to do in order to move on to the next step and so now that i've covered creating my monitoring workspace as well as adding another email to my email notification channels i can now go back to billing and finish off my budget alert let's go over here to budgets and alerts create budget and we're gonna go through the same steps call this billing alert users leave everything else as is and click on next i'm just going to change the target amount to 10 click on next i'm going to leave everything here as is and i'm going to go back to click on link monitoring email notification channels to this budget now if you notice when i click on select workspace my first project shows up and here it will ask me for my notification channels and because i've already set it up i can simply click on it and you'll see the billing admin notification channel and so if i didn't have this set up i can always go to manage notification channels and it'll bring me back to the screen which you saw earlier and so now that that's set up i can simply click on finish and so now that i have a regular budget alert i also have another budget alert that can go to a different email so if you have a project manager or a director that you want to send budget alerts to this is how you would do it and so that about covers this demo on committed use discounts reservations budgets and budget alerts and so that's all i wanted to cover for this lesson so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in this short lesson i will be covering the exporting of your billing data so that you're able to analyze that data and understand your spend at a more granular level i will also be going through a short demo where i will show you how to enable the export billing feature and bring it into bigquery to be analyzed now cloud billing export to bigquery enables you to export granular google cloud billing data such as usage cost details and pricing data automatically to a bigquery data set that you specify then you can access your cloud billing data from bigquery for detailed analysis or use a tool like data studio to visualize your data just a quick note here that billing export is not retroactive and this should be taken into consideration when planning for analysis on this data and so there are two types of cloud billing data that you can export there's the daily cost detail data and the pricing data and these can be selected right within the console depending on your use case and so now that we've gone through exactly what billing export is i wanted to get into a demo and show you how to export your cloud billing data to bigquery and go through all the necessary steps to get it enabled so when you're ready join me in the console and so here we are back in the console and so in order to enable billing export i'm going to be going to the billing page so i'm going to move up to the top left hand corner to the navigation menu and click on billing here in the left hand menu you'll see billing export and you can just click on there and so for those just coming to billing export for the first time there's a quick summary of exactly what the bigquery export is used for and as we discussed earlier there is an option for the daily cost detail and for pricing and i'm going to use the daily cost detail in this demo and export that data to bigquery so the first step i'm going to do is to click on edit settings and it's going to bring me to a new page where it will ask me for my project and this is where my billing data is going to be stored but as you can see here i'm getting a prompt that says you need to create a bigquery data set first now the bigquery data set that is asking for is where the billing data is going to be stored so in order to move forward with my billing export i need to go to bigquery and set up a data set so i'm going to simply click on this button here that says go to bigquery and it's going to bring me to the bigquery page where i'll be prompted with a big welcome note you can just click on done and over here in the right hand side where it says create data set i'm just going to click on there and i'm going to create my new data set and so for my data set id i'm going to call this billing export and just as a note with the data set id you can't use any characters like hyphens commas or periods and therefore i capitalize the b and the e now with the data location the default location is the us multi region but i can simply click on the drop down and have an option to store my data in a different location but i'm going to keep it at default i have the option of expiring this table in either a certain amount of days or to never expire as well when it comes to encryption i'm going to leave it as google manage key as opposed to a customer manage key and i'll get into encryption and key management a little later on in this course i'm going to go ahead and move right down to the bottom and click on create data set and now my data set has been created i can now see it over here on the left hand side menu where subtle poet 28400 is the id for my project if i simply click on the arrow beside it it'll show my billing export data set because there's nothing in it nothing is showing and so now that the data set is set up i can now go back to the billing export page and finish setting up my billing export so with that being said i'm going to go back up to the navigation menu head over to billing and go to billing export under daily cost detail i'm going to click on edit settings and because i have a data set already set up and since it's the only one it has been propagated in my billing export data set field if i had more data sets then i would be able to select them here as well so i'm going to leave the data set at billing export and simply click on save and so now that billing export has been enabled i'll be able to check on my billing as it is updated each day as it says here and to go right to the data set i can simply click on this hot link and it'll bring me right to bigquery and so there is one last step that still needs to be done to enable the billing export to work and that is to enable the bigquery data transfer service api so in order to do that we need to go back to the navigation menu go into apis and services into the dashboard and now i'm going to do a search for the bigquery data transfer service and i'm going to simply go up here to the top search bar and simply type in bigquery and here it is bigquery data transfer api i'm going to simply click on that and hit enable and this might take a minute and you may be asked to create credentials over here on the top right and you can simply ignore that as they are not currently needed and so now that the bigquery data transfer service api has been enabled i'm now able to go over to bigquery and take a look at my billing export data without any issues now it's going to take time to propagate but by the time i come here tomorrow the data will be fully propagated and i'll be able to query the data as i see fit and so although this is a short demo this is necessary to know for the exam as well being an engineer and looking to query your billing data you will now have the knowledge in order to take the steps necessary that will allow you to do so and so that's all i have for this lesson and demo on export billing data so you can now mark this lesson as complete and let's move on to the next one welcome back in this hands-on demo i'm going to go over apis in google cloud now the google cloud platform is pretty much run on apis whether it's in the console or the sdk under the hood it's hitting the apis now some of you may be wondering what is an api well this is an acronym standing for application programming interface and it's a standard used amongst the programming community in this specific context it is the programming interface for google cloud services and as i said before both the cloud sdk and the console are using apis under the hood and it provides similar functionality now when using the apis directly it allows you to enable automation in your workflow by using the software libraries that you use for your favorite programming language now as seen in previous lessons to use a cloud api you must enable it first so if i went to compute engine or when i was enabling monitoring i had to enable the api so no matter the service you're requesting here in google cloud and some of them may be even linked together it always has to be enabled in order to use it now getting a little bit more granular when using an api you need to have a project so when you enable the api you enable it for your project using the permissions on the project and permissions on the api to enable it now since this is a demo i want to go over to the navigation menu and go straight into apis and services and so here is the dashboard of the apis and services you can see the traffic here the errors and the latency with regards to these apis as well up here it has a time frame for the median latency that you can select for a more granular search now when it comes to what is enabled already you can see list here of the apis that are enabled and since we haven't done much there's only a few apis that are enabled now this hands-on demo is not meant to go into depth with apis but is merely an overview so that you understand what the apis are used for in context with google cloud if you'd like to go more in depth with regards to apis and possibly get certified in it the apogee certification with its corresponding lessons would be a great way to get a little bit more understanding but for this demo we're going to stick with this overview and so in order to search for more apis that need to be enabled or if you're looking for something specific you can come up here to enable apis and services or you can do a quick search on the search bar at the top of the page but just as a quick glance i'm going to go into enable apis and services and so you will be brought to a new page where you will see the api library on the left you will see a menu where the apis are categorized and all the apis that are available when it comes to google cloud and other google services so as you saw before when i needed to enable the api for bigquery i would simply type in bigquery and i can go to the api and since the api is enabled there's nothing for me to do but if i needed to enable it i could do that right there and just as a quick note when going to a service that's available in the console the api automatically gets enabled when you go and use it for the first time and so again this is just a quick overview of apis and the api library with regards to google cloud a short yet important demo to understand the under workings of the cloud sdk and the console so just remember that when using any service in google cloud again you must enable the api in order to start using it and so that about wraps up this demo for cloud apis so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in this demo i'll be creating and setting up a new gmail user as an admin user for use moving ahead in this course as well as following google's best practices we need a user that has lesser privileges than the user account that we set up previously and i'll be going through a full demo to show you how to configure it now in a google cloud setup that uses a g suite or cloud identity account a super administrator account is created to administer the domain this super admin account has irrevocable administrative permissions that should not be used for day-to-day administration this means that no permissions can be taken away from this account and has the power to grant organization admin role or any other role for that matter and recover accounts at the domain level which makes this account extremely powerful now since i do not have a domain setup or using a g suite or cloud identity account i don't need to worry about a super admin account in this specific environment as gmail accounts are standalone accounts that are meant to be personal and hold no organization and usually start at the project level and so to explain it in a bit more detail i have a diagram here showing the two different accounts i will be using and the structure behind it now as we discussed before billing accounts have the option of paying for projects in a different organization so when creating new projects using the two different gmail accounts they were created without any organization and so each account is standalone and can create their own projects now what makes them different is that the antony gcloud ace account owns the billing account and is set as a billing account administrator and the tony bowtie ace account is a billing account user that is able to link projects to that billing account but does not hold full access to billing so in the spirit of sticking to the principle of lease privilege i will be using the tony bowtie ace account that i had created earlier with lesser privileges on billing it will still give me all the permissions i need to create edit and delete resources without all the powerful permissions needed for billing i will be assigning this new gmail user the billing account user role and it will allow you to achieve everything you need to build for the remainder of the course so just as a review i will be using a new google account that i have created or if you'd like you can use a pre-existing google account and as always i recommend enabling two-step verification on your account as this user will hold some powerful permissions to access a ton of different resources in google cloud so now that we've gone over the details of the what and why for setting up this second account let's head into the demo and get things started so whenever you're ready join me over in the console and so here i am back in the console and so before switching over to my new user i need to assign the specific roles that i will need for that user which is the billing account user role so to assign this role to my new user i need to head over to billing so i'm going to go back up here to the left-hand corner and click on the navigation menu and go to billing again in the left-hand menu i'm going to move down to account management and click on there and over here under my billing account you will see that i have permissions assigned to one member of the billing account administrator and as expected i am seeing anthony g cloud ace gmail.com and so i want to add another member to my billing account so i'm going to simply click on add members and here i will enter in my new second user which is tony bowtie ace gmail.com and under select a role i'm going to move down to billing and over to billing account user and as you can see here this role billing account user will allow permissions to associate projects with billing accounts which is exactly what i want to do and so i'm going to simply click on that and simply click on save and so now that i've assigned my second user the proper permissions that i needed i am now going to log out and log in as my new user by simply going up to the right hand corner in the icon clicking on the icon and going to add account by adding the account i'll be able to switch back and forth between the different users and i would only recommend this if you are the sole user of your computer if you are on a computer that has multiple users simply sign out and sign back in again with your different user and here i'm asked for the email which would be tony bowtie ace gmail.com i'm gonna plug in my password and it's going to ask me for my two-step verification i'm going to click on yes and i should be in and because it's my first time logging into google cloud with this user i get a prompt asking me to agree to the terms of service i'm going to agree to them and simply click on agree and continue and so now i'm going to move back up to overview and as you can see here i don't have the permissions to view costs for this billing account and so all the permissions assigned for the billing account administrator which is antony g cloud ace is not applied to tony bowtie ace and therefore things like budgets and alerts even billing exports i do not have access to so moving forward in the course if you need to access anything in billing that you currently don't have access to like budgets and alerts you can simply switch over to your other account and take care of any necessary changes but what i do have access to is if i go up here to my billing account click on the drop down menu and click on manage billing accounts but as you can see here i do have access to view all the billing accounts along with the projects that are linked to them now because these gmail accounts are standalone accounts this project here that is owned by antony gcloud ace i do not have access to in order to access the project i would have to have permissions assigned to me directly in order for me to actually view the project or possibly creating any resources within that project now if i go back to my home page i can see here that i have no projects available and therefore no resources within my environment and so to kick it off i'm going to create a new project and so under project name i am going to call this project tony and you can name your project whatever you'd like under location i don't have any organization and so therefore i'm just going to click on create and this may take a minute to create and here we are with my first project named project tony as well as my notification came up saying that my project has been created and so now that this project has been created it should be linked to my billing account so in order to verify this i'm going to go over into billing and under the drop down i'm going to click on manage billing accounts and as you can see here the number of projects has gone from one to two and if i click on the menu up here under my projects you can see that project tony is a project that is linked to my billing account i also have the permissions to either disable billing or change billing for this specific project yet in order to change billing i will have to have another billing account but there are no other billing accounts available and so moving forward i will only have this one billing account and so any projects i decide to create will be linked to this billing account and so this is a great example of trimming down the permissions needed for different users and even though this is not a domain owned account but a personal account it's always recommended to practice the principle of lease privilege whenever you come across assigning permissions to any user now as i said before any billing related tasks that you decide to do moving forward you can simply switch over to your other user and do the necessary changes and so that's all i have for this lesson so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in this short lesson i'm going to be covering an overview of the cloud sdk and the command line interface as it is an essential component of interacting with google cloud for the exam you will need to get familiar with the command line and the commands needed in order to create modify and delete resources this is also an extremely valuable tool for your tool belt in the world of being a cloud engineer as i have found that is a very common and easy way to implement small operations within google cloud as well as automating the complex ones so what exactly is the cloud sdk well the cloud sdk is a set of command line tools that allows you to manage resources through the terminal in google cloud and includes commands such as gcloud gsutil bq and cubectl using these commands allow you to manage resources such as compute engine cloud storage bigquery kubernetes and so many other resources these tools can be run interactively or through automated scripts giving you the power and flexibility that you need to get the job done the cloud sdk is so powerful that you can do everything that the console can do yet has more options than the console you can use it for infrastructure as code autocompletion helps you finish all of your command line statements and for those of you who run windows the cloud sdk has got you covered with availability for powershell now in order to access google cloud platform you will usually have to authorize google cloud sdk tools so to grant authorization to cloud sdk tools you can either use a user account or a service account now a user account is a google account that allows end users to authenticate directly to your application for most common use cases on a single machine using a user account is best practice now going the route of a service account this is a google account that is associated with your gcp project and not a specific user a service account can be used by providing a service account key to your application and is recommended to script cloud sdk tools for use on multiple machines now having installed the cloud sdk it comes with some built-in commands that allow you to configure different options using gcloud init this initializes and authorizes access and performs other common cloud sdk setup steps using some optional commands gcloud auth login authorizes your access for gcloud with google user credentials and sets the current account as active gcloud config is another optional configuration that allows you to configure accounts and projects as well gcloud components allow you to install update and delete optional components of the sdk that give you more flexibility with different resources now after having installed the cloud sdk almost all gcloud commands will follow a specific format shown here is an example of this format and is broken down through component entity operation positional arguments and flags and i'll be going through some specific examples in the demonstration a little bit later on and so that's all i wanted to cover in this overview of the cloud sdk and the cli so you can now mark this lesson as complete and you can join me in the next one where i go ahead and demonstrate installing the cloud sdk [Music] back in this demonstration i will show you how to download install and configure the cloud sdk and i will be using the quick start guide that lies in the cloud sdk documentation which holds all the steps for installing the cloud sdk on different operating systems and i will make sure to include it in the lesson text below this demo will show you how to install the cloud sdk on each of the most common operating systems windows mac os and ubuntu linux all you need to do is follow the process on each of the pages and you should be well on your way so with that being said let's get this demo started and bring the cloud sdk to life by getting it all installed and configured for your specific operating system so as i explained before i'm gonna go ahead and install the cloud sdk on each of the three different operating systems windows mac os and ubuntu linux and i will be installing it with the help of the quick start guide that you see here and as i said before i'll be including this link in the lesson text and so to kick off this demo i wanted to start by installing the cloud sdk on windows so i'm going to move over to my windows virtual machine and i'm going to open up a browser and i'm going to paste in the link for the quick start guide and you can click on either link for the quick start for windows and each quick start page will give me the instructions of exactly what i need to do for each operating system so now it says that we need to have a project created which i did in the last lesson which is project tony so next i'm going to download the cloud sdk installer so i'm going to click on there and i'll see a prompt in the bottom left hand corner that the installer has been downloaded i'm going to click on it to open the file and i'm going to be prompted to go through this wizard and so i'm just going to click on next i'm going to agree to the terms of the agreement it's going to be for just me anthony and my destination folder i'll keep it as is and here's all the components that it's going to install i'm going to keep the beta commands unchecked as i don't really need them and if i need them later then i can install that component for those who are more experienced or even a bit curious you could click on the beta commands and take it for a test drive but i'm going to keep it off and i'm going to click install and depending on the power of your machine it should take anywhere from two to five minutes to install and the google cloud sdk has been installed and so i'm just going to click on next and as shown here in the documentation you want to make sure that you have all your options checked off is to create a start menu shortcut a desktop shortcut you want to start the google cloud sdk shell and lastly you want to run gcloud init in order to initialize and configure the cloud sdk now i'm going to click on finish to exit the setup and i'm going to get a command shell that pops up and i'm just going to zoom in for better viewing and so it says here my current configuration has been set to default so when it comes to configuration this is all about selecting the active account and so my current active account is going to be set as the default account it also needed to do a diagnostic check just to make sure that it can connect to the internet so that it's able to verify the account and so now the prompt is saying you must log in to continue would you like to log in yes you can just click on y and then enter and it's going to prompt me with a new browser window where i need to log in using my current account so that i can authorize the cloud sdk so i'm going to log in with my tony bowtie ace account click on next type in my password again it's going to ask me for my two-step verification and i'm going to get a prompt saying that the google sdk wants to access my google account i'm going to click on allow and success you are now authenticated with the google cloud sdk and if i go back to my terminal i am prompted to enter some values so that i can properly configure the google cloud sdk so i'm going to pick a cloud project to use and i'm going to use project tony that i created earlier so i'm going to enter 1 and hit enter and again whatever project that you've created use that one for your default configuration and it states here that my current project has been set to project tony and again this configuration is called default so if i have a second configuration that i wanted to use i can call it a different configuration but other than that my google cloud sdk is configured and ready to use so just to make sure that it's working i'm going to run a couple commands i'm going to run the gcloud help command and as you can see it's given me a list of a bunch of different commands that i can run and to exit you can just hit ctrl c i'm going to run gcloud config list and this will give me my properties in my active configuration so my account is tony bowtie ace gmail.com i've disabled usage reporting and my project is project tony and my active configuration is set as default now don't worry i'm going to be covering all these commands in the next lesson and i'm going to be going into detail on how you can configure and add other users within your cloud sdk configuration so as we go deeper into the course i'm going to be using a lot more command line just so you can get familiar with the syntax and become a bit more comfortable with it so now that i've installed the cloud sdk on windows the process will be a little bit different when it comes to installation on the other operating systems but will be very similar when it comes to the configuration so now let's head over to mac os and install the cloud sdk there and so here we are in mac os and so the first thing i want to do is i want to open up a web browser and i want to go to the cloud sdk quick start page so i'm just going to paste in the url here and we're looking for the quick start for mac os and so you can either click on the menu from the left hand side or the menu here on the main page and so like i said before this installation is going to be a little bit different than what it was in windows and so there's a few steps here to follow and so the first step asks us if we have a project already created which we've already done and is project tony and so the next step tells us that the cloud sdk requires python and so we want to check our system to see if we have a supported version so in order to check our version we're going to use this command here python minus v and i'm going to copy that to my clipboard and then open up a terminal and i'm going to zoom in for better viewing and so i'm going to paste the command in here and simply click on enter and as you can see here i'm running python 2.7 but the starred note here says that the cloud sdk will soon move to python 3 and so in order to avoid having to upgrade later you'd want to check your version for python 3 and so you can use a similar command by typing in python 3 space minus capital v and as you can see i'm running version 3.7.3 and so moving back to the guide i can see here that it is a supportive version if you do not have a supportive version i will include a link on how to upgrade your version in the lesson text below and so now that i've finished off this step let's move on to the next one where i can download the archive file for the google cloud sdk again most machines will run the 64-bit package so if you do have the latest operating system for mac os you should be good to go so i'm going to click on this package and it'll start downloading for me and once it's finished you can click on downloads and click on the file itself and it should extract itself in the same folder with all the files and folders within it and so just as another quick note google prefers that you keep the google cloud sdk in your home directory and so following the guide i'm going to do exactly that and so the easiest way to move the folder into your home directory is to simply drag and drop it into the home folder on the left hand menu it should be marked with a little house icon and nested under favorites i can now move into my home folder and confirm that it is indeed in here and so now moving to the last step which shows as optional the guide asks us to install a script to add cloud sdk tools to our path now i highly recommend that you install this script so that you can add the tools for command completion and i will get into command completion a little bit later on in the next couple of lessons and so here is the command that i need to run so i'm going to copy that to my clipboard again and i'm going to move back over to my terminal i'm going to clear my screen and so to make sure i'm in my home directory where the cloud sdk folder is i'm going to simply type ls and so for those who don't know ls is a linux command that will list all your files and folders in your current path and as you can see here the google cloud sdk is in my path and therefore i can run that script so i'm going to paste it in here and i'm going to hit enter and so a prompt comes up asking me whether or not i want to disable usage reporting and because i want to help improve the google cloud sdk i'm going to type in y for yes and hit enter and so as i was explaining before the cloud sdk tools will be installed in my path and so this is the step that takes care of it and so i'm going to type y and enter for yes to continue and usually the path that comes up is the right one unless you've changed it otherwise so i'm going to leave this blank and just hit enter and that's it i've installed the tools so now in order for me to run gcloud init i have to start a new shell as it says here for the changes to take effect so i'm going to go up here to the top left hand menu click on terminal and quit terminal and so now i can restart the terminal again i'm going to zoom in for better viewing and now i'm able to run gcloud init in order to initialize the installation again the prompt to do the diagnostic tests and i can see i have no network issues but it shows me that i have to login to continue i would like to log in so i'm going to type y for yes and hit enter and so a new browser has popped open prompting me to enter my email and password and so i'm going to do that now i'm going to authorize my account with two-step verification i'm not going to save this password and yes i want to allow the google cloud sdk to access my google account so i'm going to click on allow and it shows that i've been authenticated so now i'm going to move back to my terminal and so just as a note before we move forward in case you don't get a browser pop-up for you to log into your google account you can simply highlight this url copy it into your browser and it should prompt you just the same so moving right ahead it shows that i'm logged in as tonybowtieace gmail.com which is exactly what i wanted and it's asking me to pick a cloud project to use now i want to use project tony so i'm going to type in 1 and enter and that's it the cloud sdk has been configured and just to double check i'm going to run the gcloud config list command to show me my configuration and as you can see here my account is tonybowties gmail.com my disable usage reporting is equal to false and my project is project tony and again my active configuration is set as default and so that about covers the cloud sdk install for mac os and so finally i'm going to move over to ubuntu linux and configure the cloud sdk there and so here we are in ubuntu and like i did in the other operating systems i'm going to open up the browser and i'm going to paste in the url for the quick start guide and so we want to click on the quick start for debian and ubuntu and so again you have your choice from either clicking on the link on the left hand menu or the one here in the main menu and so following the guide it is telling us that when it comes to an ubuntu release it is recommended that the sdk should be installed on an ubuntu release that has not reached end of life the guide also asks to create a project if we don't have one already which we have already done and so now we can continue on with the steps and so since we are not installing it inside a docker image we're gonna go ahead and use the commands right here now you can copy all the commands at once by copying this to the clipboard but my recommendation is to install each one one by one so i'm going to copy this and i'm going to open up my terminal i'm going to zoom in for better viewing and i'm going to paste that command in and click on enter it's going to prompt me for my password and it didn't come up with any errors so that means it was successfully executed and so i'm going to move on to the next command i'm going to copy this go back over to my terminal and paste it in now for those of you who do not have curl installed you will be prompted to install it and given the command to run it so i'm going to copy and paste this command and click on enter i'm going to type in y for yes to continue and it's going to install it after a couple of minutes okay now that curl has been installed i'm able to run that command again i'm going to clear the screen first and that executed with no errors as well and so now moving on to the last command this command will download and install the google cloud sdk i am prompted to install some packages and so i'm going to type y for yes to continue so now it's going to download and install the necessary packages needed for the google cloud sdk and depending on the speed of your internet and the speed of your machine this could take anywhere from two to five minutes okay and the google cloud sdk has been installed and so now that the cloud sdk has been installed we can now initialize the configuration so i'm going to type in gcloud init again the prompt with the network diagnostics i'm going to type y for yes to log in and i'm going to get the prompt for my email and password i'm going to take care of my two-step verification and i'm going to allow the google cloud sdk to access my google account and success i am now authenticated and moving back to the terminal just to verify it and again i'm going to pick project tony as the cloud project to use and the cloud sdk has been configured as always i'm going to do a double check by running a gcloud config list and as expected the same details has come up and so this is a quick run through on all three operating systems windows mac os and ubuntu linux on how to install the google cloud sdk and this will help you get started with becoming more familiar and more comfortable using the command line interface and so that about wraps up for this lesson so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in the last demo we went through a complete install of the cloud sdk and configured our admin account to be used within it in this demonstration i will be walking through how to manage the cloud sdk and this will involve how to utilize it and how to customize it to your environment as well as configuring our other user account so that we are able to apply switching configurations from one user to another and so i will be going through initializing and authorization configurations and properties installing and removing components as well as a full run through of the gcloud interactive shell so let's kick off this demo by diving into a pre-configured terminal with the sdk installed and configured with my second user tony bowtie ace gmail.com and so here i am in the mac os terminal and just be aware that it doesn't matter which operating system you're running as long as the sdk is installed and you have your user configured and so as you saw in the last lesson after you install the cloud sdk the next step is typically to initialize the cloud sdk by running the gcloud init command and this is to perform the initial setup tasks as well as authorizing the cloud sdk to use your user account credentials so that it can access google cloud and so in short it sets up a cloud sdk configuration and sets a base set of properties and this usually covers the active account the current project and if the api is enabled the default google compute engine region and zone now as a note if you're in a remote terminal session with no access to a browser you can still run the gcloud init command but adding a flag of dash dash console dash only and this will prevent the command from launching a browser-based authorization like you saw when setting up your last user so now even though i have a user already set up i can still run gcloud init and it will give me a couple different options to choose from so i can re-initialize this configuration with some new settings or i can create a new configuration now for this demo since we already have two users and to demonstrate how to switch between different users i want to create a new configuration with my very first user so i'm going to type in 2 and hit enter and it's going to ask me for a configuration name now it asks me for a configuration name because when setting up your first configuration it's set as default and because i know that this user account has full access to billing as well as administration privileges i'm going to call this configuration master and i'm going to hit enter it did the necessary network checks and now it's asking me for which account i want to use this configuration for now if tony bowtie ace had access to two different google cloud accounts i would be able to add a different configuration here and so because i'm going to log in with a new account i'm going to put in two and hit enter and so again it brought me to my browser window and i'm going to log in using another account and so here you can type in the first account that you created and for me it was antony gcloud ace gmail.com i hit next and i'm going to enter my password it's going to ask me for my two-step verification and i don't want to save this password and i'm going to allow the google cloud sdk to access my google account and i am now authenticated so moving back to the console you can see here that i am currently logged in and it's asking me to pick a cloud project to use now since i only have one project in that google cloud account which is subtle poet i'm going to choose one and since i have the compute engine api enabled i am now able to configure a default compute region and zone and so i'm going to hit y for yes to configure it and as you can see there are 74 different options to choose from and if you scroll up a little bit you should be able to find the zone that you're looking for and so for this course we are going to be using us central one dash a and so this is number eight so i'm going to scroll back down and type in eight and so now my master configuration has been configured with my antony g cloud ace account using us central 1a as the compute engine zone now touching back on authorization if i didn't want to set up a whole configuration i can simply type in gcloud auth login and this will allow me to authorize just the user account only so gcloud init would authorize access and perform the cloud sdk setup steps and gcloud auth login will authorize the access only now as i mentioned in a previous lesson you can use a service account for authorization to the cloud sdk tools and this would be great for a compute instance or an application but would need a service account key file in order to authorize it and so moving back to our user accounts when running the cloud sdk you can only have one active account at any given time and so to check my active account i can type in the command gcloud auth list and this will give me a list of all the accounts that have been authorized and so whenever you run a gcloud init it will use that account as the active account and as you can see here the antony gcloud ace gmail.com has a star beside it and this is marked as the active account and so in essence the account with the star beside it is the active account and so i'm looking to change my active account back to tony bowtie ace and in order for me to do that the command is conveniently shown here and so i'm going to go ahead and run that and the account would be the user shown above and so when i do a gcloud auth list i can see that my active account is now back to tony bowtie bowtieace gmail.com now if you wanted to switch the account on a per command basis you can always do that using the flag dash dash account after the command and put in the user account that you want to use and so let's say i wanted to revoke credentials from an account that i don't need anymore i can simply use the command gcloud auth revoke followed by the username and it will revoke the credentials for that account and so doing this would remove your credentials and any access tokens for any specific account that you choose that's currently on your computer and so if we're looking for that specific account we can always use the gcloud info command and it will give us the path for the user config directory and it is this directory that holds your encrypted credentials and access tokens alongside with your active configurations and any other configurations as well now as you can see here running the gcloud info command will also give you some other information everything from the account the project the current properties and where the logs can be found so now moving on to configurations a configuration is a named set of gcloud cli properties and it works kind of like a profile and so earlier on i demonstrated how to set up another configuration through gcloud init so now if i run a gcloud config list command it would give me all the information of the active configuration so as you can see here my user has changed but my configuration has stayed the same now as seen previously in a different lesson tony bow tie ace does not have access to the project subtle poet this project belongs to antony g cloud ace and the configuration was set for that account now if tony bowtie ace did have access to the subtle poet project then i could use this configuration but it doesn't and so i want to switch back to my other configuration and how i would do this is type in the command gcloud config configurations activate and the configuration that i set up for tony bowtie ace is the default configuration and so now that it has been activated i can now run a gcloud config list and as you can see here the configuration is back to default setup during the initialization process for tony bowtie ace now if i wanted to create multiple configurations for the same user account i can simply type in the command gcloud config configurations create but if i wanted to just view the configuration properties i can always type in the command gcloud config configurations describe and as you can see after the describe i needed the configuration name to complete the command and so i'm going to do that now and i've been given all the properties for this configuration now another thing that i wanted to share when it comes to properties is that you can change the project or the compute region and zone by simply typing in the command gcloud config set now if i wanted to change the project i can simply type in project and the project name if it was for the compute instance i can simply type in compute forward slash zone for the specific zone and just as a note only the properties that are not in the core property section are the ones that can be set as well when you are setting the properties this only applies to the active configuration if you want to change the configuration of one that is not active then you'd have to switch to it and run the gcloud config set command and so moving on i wanted to touch on components which are the installable parts of the sdk and when you install the sdk the components gcloud bq gsutil and the core libraries are installed by default now you probably saw a list of components when you ran the gcloud init command and so to see all the components again you can simply type in the gcloud components list command and if you scroll up you're able to see all the components that are available that you can install at your convenience and so if i wanted to install the cubectl component i can type in the command gcloud components install cubectl and a prompt will come up asking me if i want to continue with this i want to say yes and now it will go through the process of installing these components and so just to verify if i run the command gcloud components list you can see here that i have the cube ctl component installed now if i wanted to remove that component i can simply type in gcloud components remove and then the component that i want to remove which is cubectl i'm going to be prompted if i want to do this i'm going to say yes and it's going to go through the stages of removing this component and it's been successfully uninstalled and so if you're working with a resource that you need a component for you can simply install or uninstall it using the gcloud components command and so one last thing about components before we move on is that you can update your components to make sure you have the latest version and so in order to update all of your installed components you would simply run the command gcloud components update and so before i go ahead and finish off this demonstration i wanted to touch on the gcloud interactive shell the gcloud interactive shell provides a richer shell experience simplifying commands and documentation discovery with as you type autocompletion and help text snippets below it produces suggestions and autocompletion for gcloud bq gsutil and cubectl command line tools as well as any command that has a man page sub commands and flags can be completed along with online help as you type the command and because this is part of the beta component i need to install it and so i'm going to run the command gcloud components install beta and i want to hit yes to continue and this will go ahead and kick off the installation of the gcloud beta commands and so now that it's installed i'm going to simply clear the screen and so now in order to run the gcloud interactive shell i need to run the command gcloud beta interactive and so now for every command that i type i will get auto suggestions that will help me with my commands and so to see it in all of its glory i'm going to start typing and as you can see it's giving me the option between g cloud or gsutil and i can use the arrow to choose either one and below it it'll also show me the different flags that i can use for these specific commands and how to structure them and so for now i'm going to run gsutil version minus l and as you can see here it's giving me all the information about this command and what it can do and so i'm going to hit enter and as you can see my gsutil version is 4.52 and along with the version number i'm also given all the specific information with regards to this gsutil version and this can be used with absolutely any command used on the google cloud platform and so i'm going to go ahead and do that again but running a different command so i'm just going to first clear the screen and i'm going to type gcloud compute instances and as you can see the snippet on the bottom of the screen is showing me not only the command and how it's structured but also the url for the documentation so continuing on gcloud compute instances i'm going to do a list and i'm going to filter it by using the flag dash dash filter and i'm going to filter the us east one a zone and i'm going to hit enter and as expected there are no instances in us east 1a and as you've just experienced this is a great tool and i highly recommend that you use it whenever you can now i know this is a lot to take in and a lot of these commands will not show up on the exam but again getting comfortable with the command line and the sdk will help you on your path to becoming a cloud engineer as well it will help you get really comfortable with the command line and before you know it you'll be running commands in the command line and prefer it over using the console and so that's all i have for this demo on managing the cloud sdk so you can now mark this lesson as complete and let's move on to the next one welcome back in this demonstration i'm going to be talking about the always available browser-based shell called cloud shell cloud shell is a virtual machine that is loaded with development tools and offers a persistent five gigabyte home directory that runs on google cloud cloud shell is what provides you command line access to your google cloud resources within the console cloud shell also comes with a built-in code editor that i will be diving into and allows you to browse file directories as well as view and edit files while still accessing the cloud shell the code editor is available by default with every cloud shell instance and is based on the open source editor thea now cloud shell is available from anywhere in the console by merely clicking on the icon showed here in the picture and is positioned in the top right hand corner of the console in the blue toolbar so let's get started with the cloud shell by getting our hands dirty and jumping right into it and so here we are back in the console and i am logged in as tony bowtie ace gmail.com and as you can see up here in the right hand corner as mentioned earlier you will find the cloud shell logo and so to open it up you simply click on it and it'll activate the cloud shell here at the bottom and because it's my first time using cloud shell i'll get this prompt quickly explaining an overview of what cloud shell is and i'm going to simply hit continue and i'm going to make the terminal a little bit bigger by dragging this line up to the middle of the screen and so when you start cloud shell it provisions an e2 small google compute engine instance running a debian-based linux operating system now this is an ephemeral pre-configured vm and the environment you work with is a docker container running on that vm cloud shell instances are provisioned on a per user per session basis the instance persists while your cloud shell session is active and after an hour of inactivity your session terminates and the vm is discarded you can also customize your environment automatically on boot time and it will allow you to have your preferred tools when cloud shell boots up so when your cloud shell instance is provision it's provisioned with 5 gigabytes of free persistent disk storage and it's mounted at your home directory on the virtual machine instance and you can check your disk storage by simply typing in the command df minus h and here where it shows dev disk by id google home part one it shows here the size as 4.8 gigabytes and this would be the persistent disk storage that's mounted on your home directory now if you've noticed it shows here that i'm logged in as tony bowtie ace at cloud shell and that my project id is set at project tony so the great thing about cloud shell is that you're automatically authenticated as the google account you're logged in with so here you can see i'm logged in as tony bowtie ace and so picture it like running gcloud auth login and specifying your google account but without having to actually do it now when the cloud shell is started the active project in the console is propagated to your gcloud configuration inside cloud shell so as you can see here my project is set at project tony now if i wanted to change it to a different project i could simply use the command stated up here gcloud config set project along with the project id and this will change me to a different project now behind the scenes cloud shell is globally distributed across multiple regions so when you first connect to cloud shell you'll be automatically assigned to the closest available region and thus avoiding any unnecessary latency you do not have the option to choose your own region and so cloud shell does that for you by optimizing it to migrate to a closer region whenever it can so if you're ever curious where your cloud shell session is currently active you can simply type in this command curl metadata slash compute metadata slash version one slash instance slash zone and this will give me the zone where my instance is located and as shown here it is in us east 1b now as you've probably been seeing every time i highlight something that there is a picture of scissors coming up the cloud shell has some automated and available tools that are built in and so one of those available tools is that whenever i highlight something it will automatically copy it to the clipboard for me cloud shell also has a bunch of very powerful pre-installed tools that come with it such as the cloud sdk bash vim helm git docker and more as well cloud shell has support for a lot of major different programming languages like java go python node.js ruby and net core for those who run windows now if you're looking for an available tool that is not pre-installed you can actually customize your environment when your instance boots up and automatically run a script that will install the tool of your choice and the script runs as root and you can install any package that you please and so in order for this environment customization to work there needs to be a file labeled as dot customize underscore environment now if we do an ls here you can see that all we have is the readme dash cloud shell text file if we do ls space minus al to show all the hidden files as well you can see that the dot customize underscore environment file does not exist and this is because we need to create it ourselves and so for this example i want terraform installed as an available tool when my instance boots up and so i have to create this file so i'm going to do so by using the touch command and then the name of the file dot customize underscore environment hit enter and if i clear the screen and do another ls space minus al i can see that my dot customize underscore environment file has been created and so now i'm going to need the script to install terraform which means i would have to edit it and so another great feature of cloud shell is that it comes with a code editor and i can do it one of two ways i can either come up here and click on the open editor button which will open up a new tab or i can simply use the edit command with the file name and i'm going to do just that so edit dot customize underscore environment and i'm just going to hit enter and as you can see i got a prompt saying that it's unable to load the code editor and this is because when using code editor you need cookies enabled on your browser and because i am using a private browser session cookies are disabled and because my cloud shell environment persists i'm going to open up a regular browser window and i'm going to continue where i left off and so here i am back with a new browser window again logged in as tony bowtie ace and so just to show you the persistence that happens in cloud shell i'm going to run the command ls space minus al and as you can see here the customize environment is still here and so again i wanted to install terraform as an extra tool to have in my environment and so i'm going to open up the editor by typing in edit dot customize underscore environment and i'm going to hit enter and here is the editor that popped up as you can see here it's built with eclipse thea and this is an open source code editor that you can download from eclipse and this is what the editor is built on now this menu here on the left i can make it a little bit bigger and because the only viewable file on my persistent disk is the readme cloud shell dot text file i'm not able to see my dot customize underscore environment so in order to open it and edit it i'm going to go to the menu at the top of the editor and click on file open and here i'll be able to select the file that i need so i'm going to select customize environment and click on open and so i'm going to paste in my script to install terraform and i'm just going to paste in my script from my clipboard and i'll be including the script in the github repo for those of you who use terraform and i'm going to move over to the menu on the left click on file and then hit save and so now in order for me to allow this to work the customize environment needs to be loaded into my cloud shell so i'm going to have to restart it and so in order to accomplish this i'm going to move over to the menu on the right i'm going to click on the icon with the three dots and click on restart and you'll be presented with a prompt it's saying that it will immediately terminate my session and then a new vm will be provisioned for me and you'll also be presented with an optional response from google telling them why you're restarting the vm and this is merely for statistical purposes so i'm going to click on restart and i'm going to wait till a new cloud shell is provisioned and my new cloud shell is provisioned and up and running and so i want to double check to see if terraform has been installed so i'm going to go over here to the open terminal button on the right hand side toolbar and i'm going to move back to my terminal and i'm going to simply run the command terraform dash dash version and so it looks like terraform has been installed and as you can see i'm running version.12 but it says my terraform version is out of date and that the latest version is dot 13. and so because i really want to be up to date with terraform i want to be able to go into my customize environment file and edit my version of terraform so that when my cloud shell is initiated terraform.13 can be installed and so i'm going to simply type in the command edit dot customize underscore environment and i'm back to my editor and i'm going to change the terraform version from dot 12 to dot 13 and then go over here to the left-hand menu click on file and then save and now i'm going to restart my machine again and come back when it's fully provisioned and i'm back again my machine has been provisioned and i'm going to go back to my terminal by clicking on the open terminal button and so i'm going to type in the command terraform dash dash version and as you can see i'm at version dot 13 and i'm going to run a simple terraform command to see if it's working and as you can see i am successful in running terraform on cloud shell now customizing the environment is not on the exam but it is such an amazing feature that i wanted to highlight it for you with a real world example like terraform in case you're away from your computer and you're logged into a browser and you need some special tools to use in cloud shell this is the best way to do it now as i mentioned before the cloud sdk is pre-installed on this and so everything that i've showed you in the last lesson with regards to cloud sdk can be done in the cloud shell as well so if i run the command gcloud beta interactive i'd be able to bring up the interactive cloud shell and i'll be able to run the same commands so now if i go ahead and run the command gcloud components list i'll be able to see all the components installed and as you can see with the cloud shell there are more components installed than what's installed on the default installation of the sdk i can also run the gcloud config list command to see all the properties in my active configuration and so this goes to show you that the sdk installation that's on cloud shell is just as capable as the one that you've installed on your computer the only difference here is that the sdk along with all the other tools that come installed in cloud shell is updated every week and so you can always depend that they're up to date and so moving on to a few more features of cloud shell i wanted to point out the obvious ones up here in the cloud shell toolbar right beside the open terminal i can open brand new tabs opening up different projects or even the same project but just a different terminal and moving over to the right hand menu of cloud shell this keyboard icon can send key combinations that you would normally not have access to moving on to the gear icon with this you're able to change your preferences and looking at the first item on the list when it comes to color themes you can go from a dark theme to a light theme or if you prefer a different color in my case i prefer the dark theme as well you have the options of changing your text size we can go to largest but i think we'll just keep things back down to medium and as well we have the different fonts the copy settings from which i showed you earlier as well as keyboard preferences you also have the option of showing your scroll bar now moving on to this icon right beside the gear is the web preview button and so the web preview button is designed so that you can run any web application that listens to http requests on the cloud shell and be able to view it in a new web browser tab when running these web applications web preview also supports applications run in app engine now mind you these ports are only available to the secure cloud shell proxy service which restricts access over https to your user account only and so to demonstrate this feature i am going to run a simple http server running a hello world page so first i'm going to clear my screen and then i'm going to exit the interactive shell and again i'm going to paste in for my clipboard a simple script that will run my simple http server and as you can see it's running on port 8080 and now i'm able to click on the web preview button and i'm able to preview it on port 8080 and a new web browser tab will open up and here i'll see my hello world page now this is just a simple example and so i'm sure that many of you can find great use for this and so i'm going to stop this http server now by hitting ctrl c and just as a quick note web preview can also run on a different port anywhere from port 2000 all the way up to 65 000. now moving on to the rest of the features hitting on the more button here with the three dots starting from the top we covered restart earlier when we had to restart our cloud shell you're able to both upload and download a file within cloud shell when the demands are needed as well if i have a misconfigured configuration i can boot into safe mode and fix the issue instead of having to start from scratch again moving on to boost cloud shell also known as boost mode is a feature that increases your cloud shell vm from the default e2 small to an e2 medium so in essence a memory bump from 2 gigabytes to 4 gigabytes and once it's activated all your sessions will be boosted for the next 24 hours and just as a quick note enabling boost mode restarts your cloud shell and immediately terminates your session but don't worry the data in your home directory will persist but any of the processes that you are running will be lost now when it comes to usage quota cloud shell has a 50 hour weekly usage limit so if you reach your usage limit you'll need to wait until your quota is reset before you can use cloud shell again so it's always good to keep your eyes on this in case you're a heavy user of cloud shell and moving back to the menu again you have your usage statistics which collects statistics on commands that come pre-installed in the vm and you can turn them on or off and as well help for cloud shell is available here as well if you wanted to give feedback to the google cloud team with regards to cloud shell this is the place to do it and so one last thing about cloud shell before we end this demo is that if you do not access cloud shell for 120 days your home disk will be deleted now don't worry you'll receive an email notification before its deletion and if you just log in and start up a session you'll prevent it being removed now moving ahead in this course i will be using cloud shell quite a bit and so feel free to use either cloud shell or the cloud sdk installed on your computer or feel free to follow along with me in the cloud shell within your google cloud environment and so if you are following along please make sure that you keep an eye on your quota and so i hope this demonstration has given you some really good insight as to what you can do with cloud shell and its limitations and so that's pretty much all i wanted to cover in this demonstration of cloud shell so you can now mark this as complete and let's move on to the next one [Music] welcome back in this lesson and demonstration i am going to go over limits and quotas and how they affect your cloud usage within google cloud i'm going to quickly go over some theory followed by a demonstration on where to find the quotas and how to edit them accordingly so google cloud enforces quotas on resource usage for project owners setting a hard limit on how much of a particular google cloud resource your project can use and so there are two types of resource usage that google limits with quota the first one is rate quota such as api requests per day this quota resets after a specified time such as a minute or a day the second one is allocation quota an example is the number of virtual machines or load balancers used by your project and this quota does not reset over time but must be explicitly released when you no longer want to use the resource for example by deleting a gke cluster now quotas are enforced for a variety of reasons for example they protect other google cloud users by preventing unforeseen usage spikes quotas also help with resource management so you can set your own limits on service usage within your quota while developing and testing your applications each quota limit is expressed in terms of a particular countable resource from requests per day to an api to the number of load balancers used by your application not all projects have the same quotas for the same services and so using this free trial account you may have very limited quota compared to a higher quota on a regular account as well with your use of google cloud over time your quotas may increase accordingly and so you can also request more quota if you need it and set up monitoring and alerts and cloud monitoring to warn you about unusual quota usage behavior or when you're actually running out of quota now in addition to viewing basic quota information in the console google cloud lets you monitor quota usage limits and errors in greater depth using the cloud monitoring api and ui along with quota metrics appearing in the metrics explorer you can then use these metrics to create custom dashboards and alerts letting you monitor quota usage over time and receive alerts when for example you're near a quota limit only your services that support quota metrics are displayed and so popular supported services include compute engine data flow cloud spanner cloud monitoring and cloud logging common services that are not supported include app engine cloud storage and cloud sql now as a note be aware that quota limits are updated once a day and hence new limits may take up to 24 hours to be reflected in the google cloud console if your project exceeds a particular quota while using a service the platform will return an error in general google cloud will return an http 429 error code if you're using http or rest to access the service or resource exhausted if you're using grpc if you're using cloud monitoring you can use it to identify the quota associated with the error and then create custom alerts upon getting a quota error and we will be going into greater depth with regards to monitoring later on in the course now there are two ways to view your current quota limits in the google cloud console the first is using the quotas page which gives you a list of all of your project's quota usage and limits the second is using the api dashboard which gives you the quota information for a particular api including resource usage over time quota limits are also accessible programmatically through the service usage api and so let's head into the console where i will provide a demonstration on where to look for quotas and how to increase them when you need to and so here we are back in the console and so as i explained before there are two main ways to view your current quota limits in the console and so the first one is using the quotas page and so in order to get to the quotas page i need to go to iam so i'm going to do that now by going up to the navigation menu in the top left hand corner i'm going to go to i am and admin and over to quotas and so here i am shown all the quotas of the current apis that i have enabled as you can see here it shows me the service the limit name the quota status and the details in this panel here on the right hand side shows me a little bit more information with regards to the service and the quota itself and so let's say i wanted to increase my quota on the compute engine api within networks so i'm going to select this service and over here on the right hand panel i'm going to tick the box that says global and i'm going to go back over here to the top left and click on the edit quotas button and a panel will pop up and i am prompted to enter a new quota limit along with a description explaining to google why i need this quota limit increase and so once i've completed my request i can click on done and then submit request and like i said before once the request has been submitted it will go to somebody at google to evaluate the requests for approval and don't worry these quota limit increases are usually approved within two business days and can often times be sooner than that also a great way to enter multiple quota changes is to click on the selected apis let's do bigquery api and cloud data store api and so i've clicked off three and now i can go back up to the top and click on the edit quotas button and as you can see in the panel i have all three apis that i want to increase my quotas on so i can enter all my new limit requests for each api and then i can submit it as a bulk request with all my new quota limit changes and so doing it this way would increase the efficiency instead of increasing the quotas for each service one by one and because i'm not going to submit any quota changes i'm going to close this panel and so again using the quotas page will give you a list of all your project quota usage and its limits and allow you to request changes accordingly and so now moving on to the second way which you can view your current quota limits i'm going to go to the api dashboard which will give me a more granular view including the resource usage over time so to get there i'm going to go back up to the left hand side to the navigation menu i'm going to go to apis and services and click on dashboard and here i will see all the names of the apis and i'm going to click on compute engine api for this demonstration and over here on the left hand menu you will see quotas and in here as i said before you can get some really granular data with regards to queries read requests list requests and a whole bunch of other requests i'm going to drill down into queries here and i can see my queries per day per 100 seconds per user and per 100 seconds and i can see here that my queries per 100 seconds is at a limit of 2 000 so if i wanted to increase that limit i can simply click on the pencil icon and a panel on the right hand side will prompt me to enter a new quota limit but i currently see that my quota limit is at its maximum and that i need to apply for a higher quota so when i click on the link it will bring me back to my iam page where my services are filtered and i can easily find the service that i was looking at to raise my quota limit and i can increase the quota by checking off this box and clicking on the edit quotas button at the top of the page and so as you can see the quotas page as well as the api dashboard work in tandem so that you can get all the information you need with regards to quotas and limits and to edit them accordingly and so i hope this gave you a good idea and some great insight on how you can view and edit your quotas and quota limits according to the resources you use and so that about wraps up this brief yet important demo on limits and quotas so you can now mark this as complete and let's move on to the next section [Music] welcome back and in this section we're going to be going through in my opinion one of the most important services in google cloud identity and access management also known as iam for short and i'll be diving into identities roles and the architecture of policies that will give you a very good understanding of how permissions are granted and how policies are inherited so before i jump into i am i wanted to touch on the principle of least privilege just for a second now the principle of least privilege states that a user program or process should have access to the bare minimum privileges necessary or the exact resources it needs in order to perform its function so for example if lisa is performing a create function to a cloud storage bucket lisa should be restricted to create permissions only on exactly one cloud storage bucket she doesn't need read edit or even delete permissions on a cloud storage bucket to perform her job and so this is a great illustration of how this principle works and this is something that happens in not only google cloud but in every cloud environment as well as any on-premises environment so note that the principle of least privilege is something that i have previously and will continue to be talking about a lot in this course and this is a key term that comes up quite a bit in any major exam and is a rule that most apply in their working environment to avoid any unnecessary granted permissions a well-known and unsaid rule when it comes to security hence me wanting to touch on this for a brief moment so now with that out of the way i'd like to move on to identity and access management or i am for short so what is it really well with iam you manage access control by defining who the identity has what access which is the role for which resource and this also includes organizations folders and projects in iam permission to access a resource isn't granted directly to the end user instead permissions are grouped into roles and roles are then granted to authenticated members an iam policy defines and enforces what roles are granted to which members and this policy is attached to a resource so when an authenticated member attempts to access a resource iam checks the resources policy to determine whether the action is permitted and so with that being said i want to dive into the policy architecture breaking it down by means of components in this policy architecture will give you a better understanding of how policies are put together so now what is a policy a policy is a collection of bindings audit configuration and metadata now the binding specifies how access should be granted on resources and it binds one or more members with a single role and any contact specific conditions that change how and when the role is granted now the metadata includes additional information about the policy such as an etag and version to facilitate policy management and finally the audit config field specifies the configuration data of how access attempts should be audited and so now i wanted to take a moment to dive deeper into each component starting with member now when it comes to members this is an identity that can access a resource so the identity of a member is an email address associated with a user service account or google group or even a domain name associated with a g suite or cloud identity domains now when it comes to a google account this represents any person who interacts with google cloud any email address that is associated with a google account can be an identity including gmail.com or other domains now a service account is an account that belongs to your application instead of an individual end user so when you run your code that is hosted on gcp this is the identity you would specify to run your code a google group is a named collection of google accounts and can also include service accounts now the advantages of using google groups is that you can grant and change permissions for the collection of accounts all at once instead of changing access one by one google groups can help you manage users at scale and each member of a google group inherits the iam roles granted to that group the inheritance means that you can use a group's membership to manage users roles instead of granting iam roles to individual users moving on to g suite domains this represents your organization's internet domain name such as antonyt.com and when you add a user to your g suite domain a new google account is created for the user inside this virtual group such as antony antonyt.com a g suite domain in actuality represents a virtual group of all of the google accounts that have been created like google groups g suite domains cannot be used to establish identity but they simply enable permission management now a cloud identity domain is like a g suite domain but the difference is that domain users don't have access to g suite applications and features so a couple more members that i wanted to address is the all authenticated users and the all users members the all authenticated users is a special identifier that represents anyone who is authenticated with a google account or a service account users who are not authenticated such as anonymous visitors are not included and finally the all users member is a special identifier that represents anyone and everyone so any user who is on the internet including authenticated and unauthenticated users and this covers the slew of the different types of members now touching on the next component of policies is roles now diving into roles this is a named collection of permissions that grant access to perform actions on google cloud resources so at the heart of it permissions are what determines what operations are allowed on a resource they usually but not always correspond one-to-one with rest methods that is each google cloud service has an associated permission for each rest method that it has so to call a method the caller needs that permission now these permissions are not granted to the users directly but grouped together within the role you would then grant roles which contain one or more permissions you can also create a custom role by combining one or more of the available iam permissions and again permissions allow users to perform specific actions on google cloud resources so you will typically see a permission such as the one you see here compute.instances.list and within google cloud iam permissions are represented in this form service.resource.verb so just as a recap on roles this is a collection of permissions and you cannot grant a permission directly to the user but you grant a role to a user and all the permissions that the role contains so an example is shown here where the compute instances permissions are grouped together in a role now you can grant permissions by granting roles to a user a group or a service account so moving up into a more broader level there are three types of roles in iam there are the primitive roles the predefined roles and the custom roles with the primitive roles these are roles that existed prior to the introduction of iam and they consist of three specific roles owner editor and viewer and these roles are concentric which means that the owner role includes the permissions in the editor role and the editor role includes the permissions in the viewer role and you can apply primitive roles at the project or service resource levels by using the console the api and the gcloud tool just as a note you cannot grant the owner role to a member for a project using the iam api or the gcloud command line tool you can only add owners to a project using the cloud console as well google recommends avoiding these roles if possible due to the nature of how much access the permissions are given in these specific roles google recommends that you use pre-defined roles over primitive roles and so moving into predefined roles these are roles that give granular and finer-grained access control than the primitive roles to specific google cloud resources and prevent any unwanted access to other resources predefined roles are created and maintained by google their permissions are automatically updated as necessary when new features or services are added to google cloud now when it comes to custom roles these are user defined and allow you to bundle one or more supported permissions to meet your specific needs unlike predefined roles custom roles are not maintained by google so when new permissions features or services are added to google cloud your custom roles will not be updated automatically when you create a custom role you must choose an organization or project to create it in you can then grant the custom role on the organization or project as well as any resources within that organization or project and just as a note you cannot create custom roles at the folder level if you need to use a custom role within a folder define the custom role on the parent of that folder as well the custom roles user interface is only available to users who have permissions to create or manage custom roles by default only project owners can create new roles now there is one limitation that i wanted to point out and that is that some predefined roles contain permissions that are not permitted in custom roles so i highly recommend that you check whether you can use a specific permission when making a custom role custom roles also have a really cool feature that includes a launch stage which is stored in the stage property for the role the stage is informational and helps you keep track of how close each role is to being generally available and these launch stages are available in the stages shown here alpha which is in testing beta which is tested and awaiting approval and of course ga which is generally available and i'll be getting hands-on later with these roles in an upcoming demonstration so now moving on to the next component is conditions and so a condition is a logic expression and is used to define and enforce conditional attribute-based access control for google cloud resources conditions allow you to choose granting resource access to identities also known as members only if configured conditions are met for example this could be done to configure temporary access for users that are contractors and have been given specific access for a certain amount of time a condition could be put in place to remove the access they needed once the contract has ended conditions are specified in the role bindings of a resources im policy so when a condition exists the access request is only granted if the condition expression is true so now moving on to metadata this component carries both e tags and version so first touching on e when multiple systems try to write to the same im policy at the same time there is a risk that those systems might overwrite each other's changes and the risk exists because updating an im policy involves multiple operations so in order to help prevent this issue iam supports concurrency control through the use of an etag field in the policy the value of this field changes each time a policy is updated now when it comes to a version this is a version number that is added to determine features such as a condition and for future releases of new features it is also used to avoid breaking your existing integrations on new feature releases that rely on consistency in the policy structure also when new policy schema versions are introduced and lastly we have the auditconfig component and this is used in order to configure audit logging for the policy it determines which permission types are logged and what identities if any are exempted from logging and so to sum it up this is a policy in all its entirety each component as you can see plays a different part and i will be going through policies and how they are assembled in statements in a later lesson and so there is one more thing that i wanted to touch on before ending this lesson and that is the policy inheritance when it comes to resource hierarchy and so as explained in an earlier lesson you can set an im policy at any level in the resource hierarchy the organization level the folder level the project level or the resource level and resources inherit the policies of all their parent resources the effective policy for a resource is the union of the policy set on that resource and the policies inherited from higher up in the hierarchy and so again i wanted to reiterate that this policy inheritance is transitive in other words resources inherit policies from the project which inherit policies from folders which inherit policies from the organization therefore the organization level policies also apply at the resource level and so just a quick example if i apply a policy on project x on any resources within that project the effective policy is going to be a union of these policies as the resources will inherit the policy that is granted to project x so i hope this gave you a better understanding of how policies are granted as well as the course structure and so that's all i have for this lesson so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this lesson i wanted to build on the last lesson where we went through iam and policy architecture and dive deeper into policies and conditions when it comes to putting them together in policy statements as cloud engineers you should be able to read and decipher policy statements and understand how they're put together by using all the components that we discussed earlier so just as a refresher i wanted to go over the policy architecture again now as i discussed previously a policy is a collection of statements that define who has what type of access it is attached to a resource and is used to enforce access control whenever that resource is accessed now the binding within that policy binds one or more members with a single role and any context specific conditions so in other words the member roles and conditions are bound together using a binding combined with the metadata and audit config we have a policy so now taking all of this and putting it together in a policy statement shown here you can see the bindings which have the role the members and conditions the first member being tony beauties gmail.com holding the role of storage admin and the second member as larkfetterlogin at gmail.com holding the role of storage object viewer now because lark only needs to view the files for this project in cloud storage till the new year a condition has been applied that does not grant access for lark to view these files after january the 1st an e tag has been put in and the version is numbered 3 due to the condition which i will get into a little bit later this policy statement has been structured in json format and is a common format used in policy statements moving on we have the exact same policy statement but has been formatted in yaml as you can see the members roles and conditions in the bindings are exactly the same as well as the etag and version but due to the formatting it is much more condensed so as you can see policy statements can be written in both json or yaml depending on your preference my personal preference is to write my policy statements in yaml due to the shorter and cleaner format so i will be moving ahead in this course with more statements written in yaml when you are looking to query your projects for its granted policies an easy way to do this would be to query it from the command line as shown here here i've taken a screenshot from tony bowtie ace in the cloud shell and have used the command gcloud projects get dash iam policy with the project id and this brought up all the members and roles within the bindings as well as the etag and version for the policy that has been attached to this project and as you can see here i have no conditions in place for any of my bindings and so again using the command gcloud projects get dash iam dash policy along with the project id will bring up any policies that are attached to this resource and the resource being the project id if the resource were to be the folder id then you could use the command gcloud resource dash manager folders get dash iam-policy with the folder id and for organizations the command would be gcloud organizations get dash iam-policy along with the organization id now because we don't have any folders or organizations in our environment typing these commands in wouldn't bring up anything and just as a note using these commands in the cloud shell or in the sdk will bring up the policy statement formatted in yaml so now i wanted to just take a second to dive into policy versions now as i haven't covered versions in detail i wanted to quickly go over it and the reasons for each numbered version now version one of the i am syntax schema for policies supports binding one role to one or more members it does not support conditional role bindings and so usually with version 1 you will not see any conditions version 2 is used for google's internal use and so querying policies usually you will not see a version 2. and finally with version 3 this introduces the condition field in the role binding which constrains the role binding via contact space and attributes based rules so just as a note if your request does not specify a policy version iam will assume that you want a version 1 policy and again if the policy does not contain any conditions then iam always returns a version one policy regardless of the version number in the request so moving on to some policy limitations each resource can only have one policy and this includes organizations folders and projects another limitation is that each iam policy can contain up to 1500 members and up to 250 of these members can be google groups now when making policy changes it will take up to seven minutes to fully propagate across the google cloud platform this does not happen instantaneously as iam is global as well there is a limit of 100 conditional role bindings per policy now getting a little bit deeper into conditions these are attributes that are either based on resource or based on details about the request and this could vary from time stamp to originating or destination ip address now as you probably heard me use the term earlier conditional role bindings are another name for a policy that holds a condition within the binding conditional role bindings can be added to new or existing iam policies to further control access to google cloud resources so when it comes to resource attributes this would enable you to create conditions that evaluate the resource in the access request including the resource type the resource name and the google cloud service being used request attributes allow you to manage access based on days or hours of the week a conditional role binding can be used to grant time bounded access to a resource ensuring that a user can no longer access that resource after the specified expiry date and time and this sets temporary access to google cloud resources using conditional role bindings in iam policies by using the date time attributes shown here you can enforce time-based controls when accessing a given resource now showing another example of a time-based condition it is possible to get even more granular and scope the geographic region along with the day and time for access in this policy lark only has access during business hours to view any objects within cloud storage lark can only access these objects from monday to friday nine to five this policy can also be used as a great example for contractors coming into your business yet only needing access during business hours now an example of a resource-based condition shown here a group member has a condition tied to it where dev only access has been implemented any developers that are part of this group will only have access to vm resources within project cat bowties and tied to any resources that's name starts with the word development now some limitations when it comes to conditions is that conditions are limited to specific services primitive roles are unsupported and members cannot be of the all users or all authenticated users members conditions also hold a limit of 100 conditional role bindings per policy as well as 20 role bindings for the same role and same member and so for the last part of the policy statements i wanted to touch on audit config logs and this specifies the audit configuration for a service the configuration determines which permission types are logged and what identities if any are exempted from logging and when specifying audit configs they must have one or more audit log configs now as shown here this policy enables data read data write and admin read logging on all services while exempting tony bowtie ace gmail.com from admin read logging on cloud storage and so that's pretty much all i wanted to cover in this lesson on policies policy statements and conditions and so i highly recommend as you come across more policy statements take the time to read through it and get to know exactly what the statement is referring to and what type of permissions that are given and this will help you not only in the exam but will also help you in reading and writing policy statements in future and so that's all i have for this lesson so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this demonstration i'm going to do a hands-on tour working with iam here in the google cloud console we're going to go through the available services in the iam console as well as touching on the command line in the cloud shell to show how policies can be both added and edited we're also going to be bringing in another new user to really bring this demo to life and to show you how to edit existing policies so with that being said let's dive in so if i go over here to my user icon in the top right hand corner i can see that i am logged in as tony bowtie ace gmail.com and as you can see at the top i'm here in project tony so now to get to iam i'm going to go over to the navigation menu and i'm going to go to i am in admin and over to iam now moving over here to the menu on the left i wanted to go through the different options that we have in iam so under iam itself this is where you would add or edit permissions with regards to members and roles for the policy added to your given project which in my case is project tony and i'll be coming back in just a bit to go greater in depth with regards to adding and editing the policy permissions moving on to identity and organization now although we haven't touched on cloud identity yet i will be covering this in high level detail in a different lesson but for now know that cloud identity is google cloud's identity as a service solution and it allows you to create and manage users and groups within google cloud now if i was signed into cloud identity i would have a whole bunch of options here but since this is a personal account i cannot create or manage any users as well i do not have a domain tied to any cloud identity account as well as any g suite account so just know that if you had cloud identity or g suite set up you would have a bunch of different options to choose from in order to help you manage your users and groups and here under organization policies i'm able to manage organization policies but since i am not an organization policy administrator and i don't have an organization there's not much that i can do here just know that when you have an organization set up you are able to come here in order to manage and edit your organization policies now moving under quotas we went over this in a little bit of detail in a previous lesson and again this is to edit any quotas for any of your services in case you need a limit increase moving on to service accounts i will be covering this topic in great depth in a later lesson and we'll be going through a hands-on demonstration as well now i know i haven't touched much on labels as of yet but know that labels are a key value pair that helps you organize and then filter your resources based on their labels these same labels are also forwarded to your billing system so you can then break down your billing charges by label and you can also use labels based on teams cost centers components and even environments so for example if i wanted to label my virtual machines by environment i can simply use environment as the key and as the value i can use anything from development to qa to testing to production and i could simply add this label and add all the different environments and later i'd be able to query based on these specific labels now a good rule of thumb is to label all of your resources so that this way you're able to find them a lot easier and you're able to query them a lot easier so moving forward with any of your resources that you are creating be sure to add some labels to give you maximum flexibility so i'm going to discard these changes and we're going to move on to settings and we touched on settings in an earlier lesson with regards to projects and so here i could change the project name it'll give me the project id the project number and i'm able to migrate or shut down the project now when it comes to access transparency this provides you with logs that capture the actions that google personnel take when they're accessing your content for troubleshooting so they're like cloud audit logs but for google support now in order to enable access transparency for your google cloud organization your google cloud account must have a premium support plan or a minimum level of a 400 a month support plan and because i don't have this i wouldn't be able to enable access transparency now although access transparency is not on the exam this is a great feature to know about in case you are working in any bigger environments that have these support plans and compliance is of the utmost importance now moving into privacy and security this is where google supplies all of their clients of google cloud the compliance that they need in order to meet regulations across the world and across various industries such as health care and education and because google has a broad base in europe google provides capabilities and contractual commitments created to meet data protection recommendations which is why you can see here eu model contract clauses and eu representative contacts as well under transparency and control i'm able to disable the usage data that google collects in order to provide better data insights and recommendations and this is done at the project level and as well i have the option of going over to my billing account and i could select a different billing account that's linked to some other projects that you can get recommendations on and so continuing forward identity aware proxy is something that i will be covering in a later lesson and so i won't be getting into any detail about that right now and so what i really wanted to dig into is roles now this may look familiar as i touched on this very briefly in a previous lesson and here's where i can create roles i can create some custom roles from different selections and here i have access to all the permissions and if i wanted to i can filter down from the different types the names the permissions even the status so let's say i was looking for a specific permission and i'm looking all the permissions for projects this could help me find exactly what it is that i'm looking for and these filters allow me to get really granular so i can find the exact permission and so you can get really granular with regards to your permissions and create roles that are custom to your environment now moving on to audit logs here i can enable the auto logs without having to use a specific policy by simply clicking on default autoconfig and here i can turn on and off all the selected logging as well as add any exempted users now i don't recommend that you turn these on as audit logging can create an extremely large amount of data and can quickly blow through all of your 300 credit so i'm going to keep that off move back to the main screen of the audit logs and as well here i'm able to get really granular about what i want to log now quickly touching on audit logs in the command line i wanted to quickly open up cloud shell and show you an example of how i can edit the policy in order to enable audit logging just going to make this a little bit bigger and i'm going to paste in my command gcloud projects get dash iam dash policy with the project id which is project tony 286016 and i'm gonna just hit enter and as you can see here this is my current policy and as well as expected audit logs are not enabled due to the fact that the audit config field is not present so in order for me to enable the audit config logs i'm going to have to edit the policy and so the easiest way for me to do that is for me to run the same command and output it to a file where i can edit it and i'm going to call this new dash policy dot yaml and so now that my policy has been outputted to this file i'm going to now go into the editor and as you can see my new policy.yaml is right here and so for me to enable the autoconfig logs i'm going to simply append it to the file and then i'm going to go over here to the top menu and click on file and save and so now for me to apply this new policy i'm going to go back over to the terminal and now i'm going to paste in the command gcloud projects set dash iam-policy with the project id and the file name new dash policy dot yaml and i'm just going to hit enter and as you can see the audit log configs have been enabled for all services and because this may take some time to reflect in the console it will not show up right away but either way audit logs usually take up a lot of data and i don't want to blow through my 300 credit and so i'm going to disable them now the easiest way for me to do this is to output this policy to another file edit it and set it again and so i'm going to go ahead and do that i'm going to first clear the screen and then i'm going to paste in my command while outputting it to a new file called updated dash policy dot yaml and i'm gonna hit enter and now i'm gonna go into the editor so i can edit the file now the one thing i wanted to point out is that i could have overwritten the file new dash policy but if you look here in the updated policy the e-tag is different than the e-tag in the old policy and so this allowed me to highlight e-tags when it comes to editing and creating new policies and so when editing policies make sure that the etag is correct otherwise you will receive an error and not be able to set the new policy so going back to the updated policy file i'm going to take out the audit log configs and i'm going to leave the auto configs field there and i'm going to go to the menu click on file and then save now i'm going to go back to the terminal and i'm going to paste in the new command and this will update my policy and as you can see the audit config logs have been disabled and the policy has been updated now this is the same process that you can use when you want to update any parts of the policy when it comes to your members or roles and even adding any conditions so now moving on to the last item on the menu is groups and as you can see here because i do not have an organization i'm not able to view any groups and so if i did have an organization i could manage my groups right here in this page now moving back over to iam i wanted to dig into policies in a little bit of further detail now what we see here are the permissions and roles that have been granted to selected members in this specific project which is project tony now remember an im policy is a total collection of members that have roles granted to them in what's known as a binding and then the binding is applied to that layer and all other layers underneath it and since i'm at the project layer this policy is inherited by all the resources underneath it and so just to verify through the command line i'm going to open up cloud shell and i'm going to paste in the command gcloud projects get dash iam-policy with my project id and i'm going to hit enter and as you can see here the policy is a reflection of exactly what you see here in the console so as you can see here here's the service agent which you will find here and the other two service accounts which you will find above as well as tony bowtie ace gmail.com and all the other roles that accompany those members so as i mentioned earlier i've gone ahead and created a new user and so for those who are following along you can go ahead and feel free to create a new gmail user now going ahead with this demonstration the user i created is named laura delightful now tony needed an extra hand and decided to bring her onto the team from another department now unfortunately in order for laura to help tony on the project she needs access to this project and as you can see she doesn't have any access and so we're going to go ahead and change that and give her access to this project so i'm going to go back over to my open tab for tony bowtie ace and we're gonna go ahead and give laura permissions and so i'm gonna go ahead and click on this add button at the top of the page and the prompt will ask me to add a new member so i'm gonna add laura in here now and here she is and i'm going to select the role as project viewer i'm not going to add any conditions and i'm simply going to click on save and the policy has been updated and as you can see here laura has been granted the role of project viewer so i'm going to move over to the other open tab where laura's console is open and i'm going to simply do a refresh and now laura has access to view all the resources within project tony now laura is able to view everything in the project but laura isn't actually able to do anything and so in order for laura to get things done a big part of her job is going to be creating files with new ideas for the fall winter line of bow ties in 2021 and so because laura holds the project viewer role she is able to see everything in cloud storage but she is unable to create buckets to upload edit or delete any files or even folders and as you can see here there is a folder marked bowtie inc fallwinter 2021 ideas but laura cannot create any new buckets because she doesn't have the required permissions as well drilling down into this bucket laura is unable to create any folders as explained earlier and the same stands for uploading any files and so i'm going to cancel out of this and so in order to give laura the proper permissions for her to do her job we're going to give laura the storage admin role and so moving back over to the open console for tony bowtie i'm going to give laura access by using the command line so i'm going to go up to the top right and open up cloud shell and so the command i need to run to give laura the role of storage admin would be the following gcloud projects add dash iam dash policy dash binding with the project id dash dash member user followed by colon and then the user name which is laura delightful gmail.com dash dash role and the role which is storage admin and i'm going to go ahead and hit enter and as you can see it has been executed successfully so if i do a refresh of the web page here i'm going to be able to see the changes reflected in the console and after a refresh you can see here storage admin has been added to the role for laura delightful gmail.com and so if i go over to the open tab where laura has her console open i can simply do a refresh and if i go back to the home page for cloud storage you can see here that laura now has the permissions to create a bucket laura also now has permissions to create new folders create edit and delete new files on top of being able to create new storage buckets and so that about wraps up this demonstration on getting hands-on with iam in both the console and the command line and i also hope that this demo has given you a bit more confidence on working in the shell running the commands needed in order to create new bindings along with editing existing policies and this will get you comfortable for when you need to assign roles to new and existing users that are added to your gcp environment and so you can now mark this lesson as complete and let's move on to the next one welcome back in this lesson i'm going to take a deep dive into service accounts now service accounts play a powerful part in google cloud and can allow a different approach for application interaction with the resources in google cloud now service accounts being both an identity and a resource can cause some confusion for some and so i really wanted to spend some time breaking it down for better understanding and so i'm first going to start off by explaining what exactly is a service account and so a service account is a special kind of account that is used by an application or a virtual machine instance and not a person an application uses the service account to authenticate between the application and gcp services so that the users aren't directly involved in short it is a special type of google account intended to represent a non-human user that needs to authenticate and be authorized to access data in google apis this way the service account is the identity of the service and the service accounts permissions control which resources the service can access and as a note a service account is identified by its email address which is unique to the account now the different service account types come in three different flavors user managed default and google managed service accounts when it comes to the user managed service accounts these are service accounts that you create you're responsible for managing and securing these accounts and by default you can create up to 100 user managed service accounts in a project or you can also request a quota increase in case you need more now when you create a user managed service account in your project it is you that chooses a name for the service account this name appears in the email address that identifies the service account which uses the following format seen here the service account name at the project id dot iam.gserviceaccount.com now moving on to the default service accounts when you use some google cloud services they create user managed service accounts that enable the service to deploy jobs that access other google cloud resources these accounts are known as default service accounts so when it comes to production workloads google strongly recommends that you create your own user managed service accounts and grant the appropriate roles to each service account when a default service account is created it is automatically granted the editor role on your project now following the principle of lease privilege google strongly recommends that you disable the automatic role grant by adding a constraint to your organization policy or by revoking the editor role manually the default service account will be assigned an email address following the format you see here project id at appspot.gserviceaccount.com for any service accounts created by app engine and project number dash compute at developer.gserviceaccount.com for compute engine and so lastly when it comes to google managed service accounts these are created and managed by google and they are used by google services the display name of most google managed service accounts ends with a gserviceaccount.com address now some of these service accounts are visible but others are hidden so for example google api service agent is a service account named with an email address that uses the following format project number at cloudservices.gerisa and this runs internal google processes on your behalf and this is just one example of the many google managed services that run in your environment and just as a warning it is not recommended to change or revoke the roles that are granted to the google api service agent or to any other google managed service accounts for that matter if you change or revoke these roles some google cloud services will no longer work now when it comes to authentication for service accounts they authenticate using service account keys so each service account is associated with two sets of public and private rsa key pairs that are used to authenticate to google they are the google manage keys and the user manage keys with the google manage keys google stores both the public and private portion of the key rotates them regularly and the private key is always held in escrow and is never directly accessible iam provides apis to use these keys to sign on behalf of the service account now when using user managed key pairs this implies that you own both the public and private portions of a key pair you can create one or more user managed key pairs also known as external keys that can be used from outside of google cloud google only stores the public portion of a user managed key so you are responsible for the security of the private key as well as the key rotation private keys cannot be retrieved by google so if you're using a user manage key please be aware that if you lose your key your service account will effectively stop working google recommends storing these keys in cloud kms for better security and better management user managed keys are extremely powerful credentials and they can represent a security risk if they are not managed correctly and as you can see here a user managed key has many different areas that need to be addressed when it comes to key management now when it comes to service account permissions in addition to being an identity a service account is a resource which has im policies attached to it and these policies determine who can use the service account so for instance lark can have the editor role on a service account and laura can have a viewer role on a service account so this is just like granting roles for any other google cloud resource just as a note the default compute engine and app engine service accounts are granted editor roles on the project when they are created so that the code executing in your app or vm instance has the necessary permissions now you can grant the service account user role at both the project level for all service accounts in the project or at the service account level now granting the service account user role to a user for a project gives the user access to all service accounts in the project including service accounts that may be created in the future granting the service account user role to a user for a specific service account gives a user access to only that service account so please be aware when granting the service account user role to any member now users who are granted the service account user role on a service account can use it to indirectly access all the resources to which the service account has access when this happens the user impersonates the service account to perform any tasks using its granted roles and permissions and is known as service account impersonation now when it comes to service account permissions there is also another method use called access scopes service account scopes are the legacy method of specifying permissions for your instance and they are used in substitution of iam roles these are used specifically for default or automatically created service accounts based on enabled apis now before the existence of iam roles access scopes were the only way for granting permissions to service accounts and although they are not the primary way of granting permissions now you must still set service account scopes when configuring an instance to run as a service account however when you are using a custom service account you will not be using scopes rather you will be using iam roles so when you are using a default service account for your compute instance it will default to using scopes instead of iam roles and so i wanted to quickly touch on how service accounts are used now one way of using a service account is to attach this service account to a resource so if you want to start a long-running job that authenticates as a service account you need to attach a service account to the resource that will run the job and this will bind the service account to the resource now the other way of using a service account is directly impersonating a service account which i had explained a little bit earlier so once granted they require permissions a user or a service can directly impersonate the identity of a service account in a few common scenarios you can impersonate the service account without requiring the use of a downloaded external service account key as well a user may get artifacts signed by the google managed private key of the service account without ever actually retrieving a credential for the service account and this is an advanced use case and is only supported for programmatic access now although i'm going to be covering best practices at the end of this section i wanted to go over some best practices for service accounts specifically so you should always look at auditing the service accounts and their keys using either the service account dot keys dot list method or the logs viewer page in the console now if your service accounts don't need external keys you should definitely delete them you should always grant the service account only the minimum set of permissions required to achieve the goal service accounts should also be created for each specific service with only the permissions required for that service and finally when it comes to implementing key rotation you should take advantage of the iam service account api to get the job done and so that's all i have for this lesson on service accounts so you can now mark this lesson as complete and please join me in the next one where we go hands-on in the console [Music] welcome back so in this demonstration i'm going to take a hands-on tour diving through various aspects of working with both default and custom-made service accounts we're going to start off fresh observing a new service account being automatically created along with viewing scopes observing how to edit them and creating custom service accounts that get a little bit more granular with the permissions assigned so with that being said let's dive in so as you can see here from the top right hand corner that i am logged in under tony bowtie ace gmail.com and looking over here from the top drop down menu you can see that i am in the project of cat bow ties fall 2021 and this is a brand new project that i had created specifically for this demo and so i currently have no resources created along with no apis enabled so now i want to navigate over to iam so i'm going to go up to the left hand corner to the navigation menu and i'm going to go to i am an admin and over to iam and as expected i have no members here other than myself tony bowtie ace gmail.com with no other members and if i go over here to the left hand menu under service accounts you can see that i have no service accounts created so now in order to demonstrate a default service account i'm going to go over to the navigation menu and go into compute engine and as you can see the compute engine api is starting up and so this may take a couple minutes to get ready okay and the compute engine api has been enabled so now if i go back over to iam to take a look at my service accounts as expected i have my compute engine default service account now again i did not create this manually this service account was automatically created when i had enabled the compute engine api along with the api's service agent and the compute engine service agent and the same would happen to other various apis that are enabled as well and so now that i have my default service account i want to go back over to compute engine and i'm going to go ahead and create a vm instance so i'm going to just click on create i'm going to keep everything as the default except i'm going to change the machine type from an e2 medium to an e2 micro and so now i'm going to scroll down to where it says identity and api access now here under service account you can see that the compute engine default service account has been highlighted and this is because i don't have any other service accounts that i am able to select from now when a default service account is the only service account you have access to access scopes are the only permissions that will be available for you to select from now remember access scopes are the legacy method of specifying permissions in google cloud now under access scopes i can select from the allow default access allow full access to all cloud apis and set access for each api and so i want to click on set access for each api for just a second and so as you can see here i have access to set permissions for each api the difference being is that i only have access to primitive roles and so now that i'm looking to grant access to my service account i'm going to grant access to cloud storage on a read-only capacity and so now that i have granted permissions for my service account i'm going to now create my instance by simply clicking on the create button and so now that my instance is created i want to head over to cloud storage to see exactly what my service account will have access to so i'm going to go over to my navigation menu and scroll down and click on storage and as you can see here i have created a bucket in advance called bow tie ink fall winter 2012 designs and this is due to bow tie ink bringing back some old designs from 2012 and making them relevant for today and within that bucket there are a few files of different design ideas that were best sellers back in 2012 that tony bowtie wanted to re-release for the fall winter 2012 collection and so with the new granted access to my default service account i should have access to view these files so in order to test this i'm going to go back over to the navigation menu and go back to compute engine and i'm going to ssh into my instance and so now that i've sshed into my virtual machine i wanted to first check to see who is it that's running the commands is it my user account or is it my service account and so i'll be able to do this very easily by checking the configuration and i can do this by running the command gcloud config list and as you can see my current configuration is showing that my service account is the member that is being used to run this command in the project of cat bow ties fall 2021 now if i wanted to run any commands using my tony bowtie ace gmail.com user account i can simply run the command gcloud auth login and it will bring me through the login process that we've seen earlier on in the course for my tony bowtie ace gmail.com account but now since i'm running all my commands using my service account from this compute engine instance i'm using the permissions granted to that service account that we saw earlier and so since i set the storage scope for the service account to read only we should be able to see the cloud storage bucket and all the files within it by simply running the gsutil command so to list the contents of the bucket i'm going to type in the command gsutil ls for list and the name of the bucket and the syntax for that would be gs colon forward slash forward slash followed by the name of the bucket which would be bowtie inc fw2012 designs and as you can see we're able to view all the files that are in the bucket and so it is working as expected and so now because i've only granted viewing permissions for this service account i cannot create any files due to the lack of permissions so for instance if i was to create a file using the command touch file one i have now created that file here on the instance so now i want to copy this file to my bucket and so i'm going to run the gsutil command cp for copy file 1 which is the name of my file and gs colon forward slash forward slash along with the name of the bucket which is bow tie inc fw 2012 designs and as expected i am getting an access denied exception with a prompt telling me that i have insufficient permissions and so now that i've shown you how to create a default service account and give it permissions using access scopes let's now create a custom service account and assign it proper permissions to not only read files from cloud storage but be able to write files to cloud storage as well so i'm going to now close down this tab and i'm going to go back over to the navigation menu and go back to iam where we can go in and create our new service account under service accounts and so as you can see here this is the default service account and since we want to create a custom one i'm going to go ahead and go up to the top here and click on the button that says create service account and so now i'm prompted to enter some information with regards to details of this service account including the service account name the account id along with a description and so i'm going to call this service account sa hyphen bowtie hyphen demo and as you can see it automatically propagated the service account id and i'm going to give this service account a description storage read write access and i'm going to click on the button create and so now i've been prompted to grant permissions to the service account and i can do that by simply clicking on the drop down and selecting a roll but i'm looking to get a little bit more granular and so i'm going to simply type in storage and as you can see i'm coming up with some more granular roles as opposed to the primitive roles that i only had access to prior to the search so i'm going to click on storage object viewer for read access to cloud storage i'm not going to add any conditions and i'm going to add another role and this time i'm going to add storage object creator and so those are all the permissions i need for read write access to cloud storage and so now i can simply click on continue and so now i'm being prompted to add another user to act as a service account and this is what we discussed in the last lesson about service accounts being both a member and a resource now notice that i have an option for both the service account users role and the service account admins role now as discussed earlier the service account and men's role has the ability to grant other users the role of service account user and so because we don't want to do that i'm going to leave both of these fields blank and simply click on done now i know in the last lesson i talked about creating custom keys for authentication in case you're hosting your code on premise or on another cloud and so if i wanted to do that i can simply go to the actions menu and click on create key and it'll give me the option on creating a private key either using json or p12 format and because i'm not creating any keys i'm going to simply click on cancel and so in order for me to apply this service account to our vm instance i'm going to now go back over to the navigation menu and go back into compute engine and so now in order for me to change this service account that's currently assigned to this instance i'm going to go ahead and check off this instance and click on stop now please note that in order to change service accounts on any instance you must stop it first before you can edit the service account and so now that the instance has stopped i'm going to drill down into this instance one and i'm going to click on edit now i'm going to scroll down to the bottom and at the bottom you will find the service account field and clicking on the drop down i'll find my custom service account as a bow tie demo so i want to select this and simply click on save and so now that i've selected my new service account to be used in this vm instance i can now start up the instance again to test out the permissions that were granted and so just as a quick note here i wanted to bring your attention to the external ip whenever stopping and starting an instance with an ephemeral ip in other words it is not assigned a static ip your vm instance will receive a new ip address and i'll be getting into this in a lot deeper detail in the compute engine section of the course and so now i'm going to ssh into this instance now i'm going to run the same gsutil command that i did previously to list all the files in the bucket so i'm going to run the command gsutil ls for list and gs colon forward slash forward slash bow tie inc fw 2012 designs and as you can see i'm able to read all the files in the bucket now the difference in the permissions granted for the service account is that i'm able to write files to cloud storage and so in order to test that i'm going to use the touch command again and i'm going to name the file file2 and so now i'm going to copy this file to the cloud storage bucket by using the command gsutil cp file2 and the bucket name gs colon forward slash forward slash bow tie inc fw 2012 designs and as expected the file copied over successfully as we do have permissions to write to cloud storage and so before i end this demonstration i wanted to quickly go over exactly how to create service accounts using the command line and so i'm going to close down this tab and i'm going to head up to the top right hand corner and activate my cloud shell i'm going to make this window a little bit bigger and so now in order to view the service accounts i currently have i'm going to run the command gcloud iam service dash accounts list and so as expected the compute engine default service account along with the custom service account that i created earlier called sa bowtie demo is now displaying and in order to just verify that i'm going to go over to iam under service accounts and as you can see it is reflecting exactly the same in the console so now in order for me to create a new service account using the command line i'm going to run the command gcloud iam service accounts create and the name of the service account which i'm going to call sa-tony bowtie along with the display name as essay tony bowtie as well and i'm going to hit enter and my service account has been created so now if i run the command gcloud i am service accounts list i should see my new service account and as well if i did a refresh here on the console i can see that it is reflecting the same so now that we've created our new service account we need to assign some permissions to it in order for us to be able to use it and so if i go over here to iam in the console i can see here that my service account has not been assigned any permissions and so in order to do that i am going to simply run the command gcloud projects add dash iam-policy-binding so we're adding a policy binding and then the name of the project catbow ties fall 2021 we need to add the member which is the new service account email address along with the role of storage object viewer i'm going to hit enter and as you can see my member sa tony bowtie has been assigned the storage object viewer role and so if i wanted to grant some other roles to the service account i can do that as well and so if i did a refresh here i can see that the console reflects exactly the same and so in order for me to use this account in my instance i'm going to first have to stop my instance attach my service account and then start up my instance again so i'm going to go over to my cloud shell i'm just going to clear the screen and i'm going to paste in the command gcloud compute instances stop the name of the instance along with the zone and now that the instance has stopped i can now add my surface account to the instance and so i'm going to use the command gcloud compute instances set service account instance 1 along with the zone and the service account email address i'm going to go ahead and hit enter and it has now been successfully added and so now that that's done i can now start up the instance by using the command gcloud compute instances start along with the instance name and the zone and so now if i go over to my navigation menu and go over to compute engine and drill down on the instance if i scroll down to the bottom i'll be able to see that my new service account has been added and so this is a great demonstration for when you want to add different service accounts for your different applications on different instances or even on different resources and so that's pretty much all i wanted to cover in this demonstration so you can now mark this lesson as complete and let's move on to the next one welcome back in this lesson i'm going to dive into cloud identity google's identity as a service offering for google cloud that maximizes end user efficiency protect company data and so much more now cloud identity as i said before is an identity as a service solution that centrally manages users and groups this would be the sole system for authentication and that provides a single sign-on experience for all employees of an organization to be used for all your internal and external applications cloud identity also gives you more control over the accounts that are used in your organization for example if developers in your organization use personal accounts such as gmail accounts those accounts are outside of your control so when you adopt cloud identity you can manage access and compliance across all the users in your domain now when you adopt cloud identity you create a cloud identity account for each of your users and groups you can then use iam to manage access to google cloud resources for each cloud identity account and you can also configure cloud identity to federate identities between google and other identity providers such as active directory and azure active directory and i'll be getting more into that a little bit later so now when it comes to cloud identity it gives you so much more than just user and group management it provides a slew of features such as device management security single sign-on reporting and directory management and i will be diving deeper into each one of these features of cloud identity now starting with device management this lets people in any organization access their work accounts from mobile devices while keeping the organization's data more secure in today's world employees want to access business applications from wherever they are whether at home at work or even traveling and many even want to use their own devices which is also known as bring your own device or byod for short using mobile device management there are several ways that you can provide the business applications employees need on their personal devices while implementing policies that keep the corporate data safe you can create a white list of approved applications where users can access corporate data securely through those applications you can enforce work profiles on android devices and requiring managed applications on ios devices policies can also be pushed out on these devices to protect corporate data and identities as well as keeping inventory of devices with corporate data present then when these devices are either no longer being used for corporate use or stolen the device can then be wiped of all its corporate data device management also gives organizations the power to enforce passcodes as well as auditing now moving into the security component of cloud identity this is where two-step verification steps in now as explained earlier two-step verification or to sv is a security feature that requires users to verify their identity through something they know such as a password plus something they have such as a physical key or access code and this can be anything from security keys to google prompt the authenticator app and backup codes so cloud identity helps by applying security best practices along with being able to deploy two-step verification for the whole company along with enforcement controls and can also manage passwords to make sure they are meeting the enforced password requirements automatically so single sign-on is where users can access many applications without having to enter their username and password for each application single sign-on also known as sso can provide a single point of authentication through an identity provider also known as idp for short you can set up sso using google as an identity provider to access a slew of third-party applications as well as any on-premise or custom in-house applications you can also access a centralized dashboard for conveniently accessing your applications so now when lisa logs in with her employee credentials she will then have access to many cloud applications that bowtie inc it department has approved through a catalog of sso applications and this will increase both security and productivity for lisa and bowtie inc as lisa won't have to enter a separate username and password for separate applications now getting into reporting this covers audit logs for logins groups devices and even tokens you're even able to export these logs to bigquery for analysis and then you can create reports from these logs that cover security applications and activity now moving on to the last component of cloud identity is directory management and this provides profile information for users in your organization email and group addresses and shared external contacts in the directory using google cloud directory sync or gcds you can synchronize the data in your google account with your microsoft active directory or ldap server gcds doesn't migrate any content such as your email your calendar events or your files to your google account gcds is used to synchronize all your users groups and shared contacts to match the information in your ldap server which could be your active directory server or your azure active directory domain now getting deeper into google cloud directory sync i'd like to touch on active directory for just a minute now active directory is a very common directory service developed by microsoft and is a cornerstone in most big corporate on-premises environments it authenticates and authorizes all users and computers in a windows domain type network signing and enforcing security policies for all computers and installing or updating software as necessary now as you can see here in the diagram the active directory forest contains the active directory domain a bowtieinc.co and the active directory federation services of bowtieinc.co where the active directory forest is the hierarchical structure for active directory the active directory domain is responsible for storing information about members of the domain including devices and users and it verifies their credentials and defines their access rights active directory federation services or adfs is a single sign-on service where federation is the means of linking a person's electronic identity and attributes stored across multiple distinct identity management systems so you can think of it as a subset of sso as it relates only to authentication technologies used for federated identity include some common terms that you may hear me or others in the industry use from time to time such as saml which stands for security assertion markup language oauth open id and even security tokens such as simple web tokens json web tokens and saml assertions and so when you have identities already in your on-premises environment that live in active directory you need a way to tie these identities to the cloud and so here's where you would use google cloud directory sync to automatically provision users and groups from active directory to cloud identity or g suite google cloud directory sync is a free google provided tool that implements the synchronization process and can be run on google cloud or in your on-premises environment synchronization is one way so that active directory remains the source of truth cloud identity or g suite uses active directory federation services or adfs for single sign-on any existing corporate applications and other sas services can continue to use your adfs as an identity provider now i know this may be a review for some who are advanced in this topic but for those who aren't this is a very important topic to know as google cloud directory sync is a big part of cloud identity and is a common way that is used in many corporate environments to sync active directory or any other ldap server to google cloud especially when you want to keep your active directory as the single source of truth and so that's pretty much all i wanted to cover when it comes to cloud identity and google cloud directory sync so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back now i wanted to close out this section by briefly going over the best practices to follow when working with identity and access management so the phrase that was discussed in the beginning of this lesson that will continuously come up in the exam is the principle of least privilege and again this is where you would apply only the minimal access level required for what is needed to be done and this can be done using predefined roles which is a more granular level role than using primitive roles which are very wide scoped roles that are applied to the whole project roles should also be granted at the smallest scope necessary so for instance when assigning somebody the permissions needed for managing pre-existing compute instances assigning a compute instance admin role might be sufficient for what they need to do as opposed to assigning them the compute instance role that has full control of all compute engine instance resources now when it comes to child resources they cannot restrict access granted on its parent so always remember to check the policy granted on every resource and make sure you understand the hierarchical inheritance you also want to make sure that you restrict access to members abilities to create and manage service accounts as users who are granted the service account actor role for a service account can access all the resources for which the service account has access and granting someone with the owner role should be used with caution as they will have access to modify almost all resources project-wide including iam policies and billing granting an editor role might be more sufficient for the needs of most when using primitive roles now when dealing with resource hierarchy to make it easy on how to structure your environment you should look at mirroring your google cloud resource hierarchy structure to your organizational structure in other words the google cloud resource hierarchy should reflect how your company is organized you should also use projects to group resources that share the same trust boundary as well as setting policies at the organization level and at the project level rather than at the resource level now going back to what we discussed earlier about the principle of least privilege you should use this guideline to grant iam roles that is only give the least amount of access necessary to your resources and when granting roles across multiple projects it is recommended to grant them at the folder level instead of at the project level now diving back into service accounts a separate trust boundary should always be applied for any given application in other words create a new service account when multiple components are involved in your application you also want to make sure that you don't delete any service accounts that are in use by running instances as your application is likely to fail so you will want to schedule this during plan down time to avoid any outages now earlier on in this section we discussed service account keys and how they interact with google cloud and that is the main authentication mechanism used for keys so you want to make sure that any user managed keys are rotated periodically to avoid being compromised you can rotate a key by creating a new key switching applications to use the new key and then deleting the old key but be sure to create the new key first before deleting the old one as this will result in parts or even your entire application failing and also when working with service account keys it's always good practice to name your service keys and this will reflect your use for those keys and permissions for those keys so you know what they are used for when you're looking at them now when you are giving access to service accounts you want to make sure that only those who truly need access are the ones that have it others in your environment should be restricted to avoid any misuse now when it comes to keeping your service account keys safe i can't stress this enough you never want to check in these keys source code or leave them in your downloads directory as this is a prime way of not only getting your keys compromised but compromising your entire environment to be accessed publicly now we touched a bit on auditing but we haven't really gone into it in detail and we'll be going into it later on in the course but touching on best practices you want to be sure to check your cloud audit logs regularly and audit all i am policy changes whenever you edit any iam policies a log is generated that records that change and so you always want to periodically check these logs to make sure that there are no changes that are out of your security scope you also want to check to see who has editing permissions on these iam policies and make sure that those who hold them have the rights to do so point being is that you want to restrict who has the ability to edit policies and once these audit logs have been generated you want to export them to cloud storage so that you're able to store them for long term retention as these logs are typically held for weeks and not years getting back to service account keys service account key access should be periodically audited for viewing of any misuse or unauthorized access and lastly audit logs should also be restricted to only those who need access and others should have no permissions to view them and this can be done by adding a role to be able to view these logs now when touching on policy management you want to grant access to all projects in your organization by using an organization level policy you also want to grant roles to a google group instead of individual users as it is easier to add or remove members from a google group instead of updating an im policy and finally when you need to grant multiple roles to a task you should create a google group as it is a lot easier to grant the roles to that group and then add the users to that group as opposed to adding roles to each individual user and so that's all i wanted to cover on this short yet very important lesson on best practices when it comes to iam now i know this is not the most exciting topic but will become extremely necessary when you are dealing with managing users groups and policies in environments that require you to use iam securely and so please keep this in mind whenever you are working in any environment as it will help you grant the proper permissions when it comes to these different topics so now i highly recommend that you take a break grab a tea or coffee before moving on into the next section and so for now you can mark this lesson as complete and whenever you're ready please join me in the next section [Music] welcome back now i wanted to make this as easy as possible for those students who do not have a background in networking or any networking knowledge in general which is why i wanted to add this quick networking refresher to kick off the networking section of this course so with that being said let's dive in so before the internet computers were standalone and didn't have the capabilities to send emails transfer files or share any information fast forward some time people started to connect their computers together to share and be able to do the things that modern networks can do today part of being in this network is being able to identify each computer to know where to send and receive files this problem was solved by using an address to identify each computer on the network like humans use a street address to identify where they live so that mail and packages can be delivered to them an ip address is used to identify a computer or device on any network so communication between machines was done by the use of an ip address a numerical label assigned to each device connected to a computer network that uses the internet protocol for communication also known as ip for short so for this system to work a communication system was put in place that defined how the network would function this system was put together as a consistent model of protocol layers defining interoperability between network devices and software in layers to standardize how different protocols would communicate in this stack this stack is referred to as the open systems interconnection model or you may hear many refer to it as the seven layer osi model now this is not a deep dive networking course but i did feel the need to cover that which is necessary for the understanding of the elements taught in this course for those wanting to learn more about the osi model and the layers within it please check out the links that i have included in the lesson text below so for this lesson and the next i will be covering the specific layers with its protocols that are highlighted here and will help you understand the networking concepts in this course with a bit better clarity so i'll be covering a layer 3 being the network layer layer 4 being the transport layer and layer 7 being the application layer so first up i will be covering layer 3 which is the networking layer along with the internet protocol now there are two versions of the internet protocol and are managed globally by the regional internet registries also known as the rir the first one which is ipv4 is the original version of the internet protocol that first came on the scene in 1981 the second version is ipv6 which is a newer version designed in 2017 to deal with the problem of ipv4 address exhaustion meaning that the amount of usable ips were slowly being used up and i will be covering both versions of the internet protocol in a little bit of depth so let's first dive into ipv version 4. so ipv4 can be read in a human readable notation represented in dotted decimal notation consisting of four numbers each ranging from 0 to 255 separated by dots each part between the dots represents a group of 8 bits also known as an octet a valid range for an ip address starts from 0.0.0.0 and ends in 255.255.255.255. and this would give you a total number of over 4.2 billion ip addresses now this range was viewed as extremely large back then until the number of ip addresses available were quickly dwindling due to the many ipconnected devices that we have today and this is when a new addressing architecture was introduced called classful addressing where the address was split into smaller ranges and this was originally assigned to you when you needed an ip address by one of the registries noted before so for any given ip address they're typically made of two separate components the first part of the address is used to identify the network that the address is a part of the part that comes afterwards is used to specify a specific host within that network now the first part was assigned to you and your business by the registries and the second part was for you to do it as you'd like and so these ip addresses were assigned from the smaller ranges explained earlier called classes the first range of classes is class a and it started at 0.0.0.0 and ended at 127.255 and this would give a total number of over 2.1 billion addresses with 128 different networks class a ip addresses can support over 16 million hosts per network and those who were assigned addresses in this class had a fixed value of the first octet the second third and fourth octet was free for the business to assign as they choose class a ip addresses were to be used by huge networks like those deployed by internet service providers and so when ips started to dwindle many companies return these class a network blocks back to the registries to assist with extending addressing capacity and so the next range is class b and this is half the size of the class a network the class b network range started at one at 128.0.0.0 and end it at 191.255.255.255 and carries a total number of over 1 billion ip addresses with over 16 000 networks the fixed value in this class is of the first and second octet the third and fourth octet can be done with as you like ip addresses in this class were to be used for medium and large size networks in enterprises and organizations the next range is class c and this is half the size of the class b network the class c network range starts at 192 and ends at 223.255.255.255 and carries a total of over half a billion addresses with over two million networks and can support up to 256 hosts the fixed value of this class is the first second and third octet and the fourth can be done with as you like ip addresses in this class were the most common class and were to be used in small business and home networks now there's a couple more classes that were not commonly used called class d and class e and this is beyond the scope of this course so we won't be discussing this and so this was the way that was used to assign public ip addresses to devices on the internet and allowed communication between devices now the problem with classful addressing was that with businesses that needed larger address blocks than a class c network provided they received a class b block which in most cases was much larger than required and the same thing happened with requiring more ips than class b and getting a class a network block this problem introduced a lot of wasted ips as there was no real middle ground and so this was a way to address any publicly routable ips now there were certain ranges that were allocated for private use and were designed to be used in private networks whether on-premises or in cloud and again they are not designed for public use and also didn't have the need to communicate over the public internet and so these private ip address spaces were standardized using the rfc standard 1918 and again these ip addresses are designed for private use and can be used anywhere you like as long as they are still kept private chances are a network that you've come across whether it be a cloud provider your home network or public wi-fi will use one of these classes to define their network and these are split into three ranges first one being single class a with 10.0.0 ending in 10.255.255.255. the class b range ranging from 172.16.0.0 to 172.31 dot and lastly class c which was ranging from 192.168.0.0 to 192.168.255.255. now for those networks that use these private ips over the public internet the process they would use is a process called network address translation or nat for short and i will be covering this in a different lesson later on in the section this method of classful addressing has been replaced with something a bit more efficient where network blocks can be defined more granularly and was done due to the internet running out of ipv4 addresses as we needed to allocate these ips more efficiently now this method is called classless inter domain routing or cider for short now with cider based networks you aren't limited to only these three classes of networks class a b and c have been removed for something more efficient which will allow you to create networks in any one of those ranges cider ranges are represented by its starting ip address called a network address followed by what is called a prefix which is a slash and then a number this slash number represents the size of the network the bigger the number the smaller the network and the smaller the number the bigger the network given the example here 192.168.0.0 is the network address and the prefix is a slash 16. now at this high level it is not necessary to understand the math behind this but i will include a link in the lesson text for those of you who are interested in learning more about it all you need to keep in mind is as i said before the bigger the prefix number the smaller the network and the smaller the prefix number the bigger the network so just as an example the size of this slash 16 network is represented here by this circle its ip range is 192.168.0.0 ending in 192.168.255.255. and once you understand the math you will be able to tell that a slash 16 range means that the network is the fixed value in the first and second octet the hosts on the network or the range are the values of anything in the third or fourth octets so this network in total will provide us with 65 536 ip addresses now let's say you decided to create a large network such as this and you wanted to allocate part of it to another part of your business you can simply do so by splitting it in two and be left with two slash 17 networks so instead of one slash 16 network you will now have 2 17 networks and each network will be assigned 32 768 ip addresses so just to break it down the previous network which was 192.16 forward slash 16 with the first two octets being the network which is 192.168 it leaves the third and fourth octet to distribute as you like and these third and fourth octets are what you're having to create these two networks so looking at the blue half the address range will start at 0.0 and will end at 127.255. the green half will start halfway through the slash 16 network which will be 128.0 and end at 255.255. so now what if i was looking to break this network down even further and break it into four networks well using cider ranges this makes things fairly easy as i can have it again and as shown here i would split the two slash 17 networks to create four slash 18 networks so if i took the blue half circle and split it into two and then splitting the green half circle into this would leave me with four slash 18 networks as seen here the blue quarter would start from 192.168.0.0 ending with the last two octets of 63.255 and the red quarter which starts from where the blue left off starting at the last two octets of 64.0 and ending in 127.255. the green quarter again starting off with the previously defined 128.0 network which is where the red quarter left off and ending with the last two octets being 191.255 and lastly the yellow quarter starting off from where the green quarter left off at 192.0 with the last two octets ending with 255.255 and so this would leave us with four smaller slash 18 networks broken down from the previous two 17 networks with each of these networks consisting of 16 384 ip addresses and we can continue this process continuously having networks and breaking them down into smaller networks this process of dividing each network into two smaller networks is known as subnetting and each time you subnet a network and create two smaller networks the number in the prefix will increase and so i know this is already a lot to take in so this would be a perfect time for you to grab a coffee or a tea and i will be ending part one here and part two will be continuing immediately after part one so you can now mark this lesson as complete and i'll see you in the next one for part two [Music] welcome back and in this lesson i'm going to be covering the second part of the networking refresher now part two of this lesson is starting immediately from the end of part one so with that being said let's dive in now i know this network refresher has been filled with a ton of numbers with an underlying current of math but i wanted you to focus on the why so that things will make sense later i wanted to introduce the hard stuff first so that over the length of this course you will be able to digest this information and understand where this fits into when discussing the different network parts of google cloud this will also help you immensely in the real world as well as the exam when configuring networks and knowing how to do the job of an engineer so getting right into it i wanted to just do a quick review on classless inter-domain routing or cider so as discussed in the first refresher an ipv4 address is referenced in dotted decimal notation alongside the slash 16 is the prefix and defines how large the network is and so before i move on i wanted to give you some references that i found helpful in order to determine the size of a network and so here i have referenced three of the most common prefixes that i continuously run into that i think would be an extremely helpful reference for you so if you look at the first i p address 192.168.0.0 with slash 8 as the prefix slash 8 would fall under a class a network 192 being the first octet as well as being the network part of the address would be fixed and so the host part of it would be anything after that so the address could be 192 dot anything and this cider range would give you over 16 million ip addresses the second most common network that i see is a slash 16 network and this would make this ip fall under a class b network making the first two octets fixed and being the network part meaning that anything after 192.168 would be the host part meaning that the address could be 192.168.anything and this would give you 65536 ip addresses and so for the third ip address which is probably the most common one that i see is a slash 24 network which falls under a class c network meaning that the first three octets are fixed and the fourth octet could be anything from zero to two five five and this would give you 256 ip addresses and another common one which is the smallest that you will see is a slash 32 prefix and this is one that i use constantly for white listing my ip address and because a slash 32 is one ip address this is a good one to know when you are configuring vpn for yourself or you're whitelisting your ip address from home or work and for the last reference as well as being the biggest network is the ip address of 0.0.0.1 forward slash 0 which covers all ip addresses and you will see this commonly used for the internet gateway in any cloud environment and so these are some common prefixes that come up very frequently and so i hope this reference will help you now moving back to the osi model i've covered ipv4 in the network layer and so now it's time to discuss ipv6 now as i noted earlier ipv4 notation is called dotted decimal and each number between the dots is an octet with a value of 0 to 255. now underneath it all each octet is made up of an 8-bit value and having four numbers in an ip address that would make it a 32-bit value ipv6 is a much longer value and is represented in hexadecimal and each grouping is two octets which is 16 bits and is often referred to as a hextet now as these addresses are very long as you can see you're able to abbreviate them by removing redundant zeros so this example shown here is the same address as the one above it so if there is a sequence of zeros you can simply replace them with one zero so in this address each grouping of four zeros can be represented by one zero and if you have multiple groups of zeros in one address you can remove them all and replace them with double colons so each of these ipv6 addresses that you see here are exactly the same now each ipv6 address is 128 bits long and is represented in a similar way to ipv4 starting with the network address and ending with the prefix each hextet is 16 bits and the prefix number is the number of bits that represent the network with this example slash 64 refers to the network address underlined in green which is 2001 colon de3 each hextet is 16 bits and the prefix is 64. so that's four groups of 16 and so this is how we know which part is the network part of the address and which is the host part of the address again notice the double colon here and as i explained previously any unneeded zeros can be replaced by a double colon and so this address would represent a slew of zeros and so adding in all the zeros the ipv6 starting network address would look like this now because the network address starts at 2001 colon de3 with another two hextets of zeros as the network address that was determined by the slash 64 prefix which is four hextets it means a network finishes at that network address followed by all fs and so that's the process of how we can determine the start and end of every ipv6 network now as i've shown you before with all ipv4 addresses they are represented with a 0.0.0.0.0 and because ipv6 addresses are represented by the same network address and prefix we can represent ipv6 addresses as double colon slash zero and you will see this frequently when using ipv6 and so i know this is really complicated but i just wanted to give you the exposure of ipv6 i don't expect you to understand this right away in the end it should become a lot clearer as we go through the course and i promise you it will become a lot easier i had a hard time myself trying to understand this network concept but after a few days i was able to digest it and as i went back and did some practice it started to make a lot more sense to me and so i know as we move along with the course that it will start making sense to you as well so now that we've discussed layer 3 in the osi model i wanted to get into layer 4 which is the transport layer with ip packets discussing tcp and udp and so in its simplest form a packet is the basic unit of information in network transmission so most networks use tcpip as the network protocol or set of rules for communication between devices and the rules of tcpip require information to be split into packets that contain a segment of data to be transferred along with the protocol and its port number the originating address and the address of where the data is to be sent now udp is another protocol that is sent with ip and is used in specific applications but mostly in this course i will be referring to tcpip and so as you can see in this diagram of the ip packet this is a basic datagram of what a packet would look like again with this source and destination ip address the protocol port number and the data itself now this is mainly just to give you a high level understanding of tcpip and udpip and is not a deep dive into networking now moving on to layer 7 of the osi model this layer is used by networked applications or applications that use the internet and so there are many protocols that fall under this layer now these applications do not reside in this layer but use the protocols in this layer to function so the application layer provides services for networked applications with the help of protocols to perform user activities and you will see many of these protocols being addressed as we go through this course through resources in google cloud like http or https for load balancing dns that uses udp on port 53 and ssh on port 22 for logging into hosts and so these are just a few of the many scenarios where layer 7 and the protocols that reside in that layer come up in this course and we will be diving into many more in the lessons to come and so that about wraps up this networking refresher lesson and don't worry like i said before i'm not expecting you to pick things up in this first go things will start to make more sense as we go through the course and we start putting these networking concepts into practice also feel free to go back and review the last couple of lessons again if things didn't make sense to you the first time or if you come across some networking challenges in future lessons and so that's everything i wanted to cover so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in this lesson we will be discussing the core networking service of gcp virtual private cloud or vpc for short it is the service that allows you to create networks inside google cloud with both private and public connectivity options both for in-cloud deployments and on-premise hybrid cloud deployments this is a service that you must know well as there are many questions that come up on the exam with regards to vpcs so with that being said let's dive in now vpcs are what manages the networking functionality for your google cloud resources this is a software defined network and is not confined to the physical limitations of networking in a data center this has been abstracted for you vpc networks including their associated routes and firewall rules are global resources they are not associated with any particular region or zone they are global resources and span all available regions across the globe as explained earlier vpcs are also encapsulated within projects projects are the logical container where your vpcs live now these vpcs do not have ip ranges but are simply a construct of all of the individual ip addresses and services within that network the ip addresses and ranges are defined within the subnetworks that i will be diving into a bit later as well traffic to and from instances can be controlled with network firewall rules rules are implemented on the vms themselves so traffic can be controlled and logged as it leaves or arrives at a vm now resources within a vpc network can communicate with one another by using internal or private ipv4 addresses and these are subject to applicable network firewall rules these resources must be in the same vpc for communication otherwise they must traverse the public internet with an assigned public ip or use a vpc peering connection or establish a vpn connection another important thing to note is that vpc networks only support ipv4 unicast traffic they do not support ipv6 traffic within the network vms in the vpc network can only send to ipv4 destinations and only receive traffic from ipv4 sources however it is possible to create an ipv6 address for a global load balancer now unless you choose to disable it each new project starts with a default network in a vpc the default network is an auto mode vpc network with predefined subnets a subnet is allocated for each region with non-overlapping cider blocks also each default network has a default firewall rule these rules are configured to allow ingress traffic for icmp rdp and ssh traffic from anywhere as well as ingress traffic from within the default network for all protocols and ports and so there are two different types of vpc networks auto mode or custom mode an auto mode network also has one subnet per region the default network is actually an auto mode network as explained earlier now these automatically created subnets use a set of predefined ip ranges with a slash 20 cider block that can be expanded to a slash 16 cider block all of these subnets fit within the default 10.128.0.0 ford slash 9 cider block and as new gcp regions become available new subnets in those regions are automatically added to auto mode networks using an ip range on that block now a custom owned network does not automatically create subnets this type of network provides you with complete control over its subnets and ip ranges as well as another note an auto mode network can be converted to a custom mode network to gain more control but please be aware this conversion is one way meaning that custom networks cannot be changed to auto mode networks so when deciding on the different types of networks you want to use make sure that you review all of your considerations now custom mode vpc networks are more flexible and better suited to production and google recommends that you use custom mode vpc networks in production so here is an example of a project that contains three networks all of these networks span multiple regions across the globe as you can see here on the right hand side and each network contains separate vms and so this diagram is to demonstrate that vms that are in the same network or vpc can communicate privately even when placed in separate regions because vms in network a are in the same network they can communicate over internal ip addresses even though they're in different regions essentially your vms can communicate even if they exist in different locations across the globe as long as they are within the same network the vms in network b and network c are not in the same network therefore by default these vms must communicate over external ips even though they're in the same region as no internal ip communication is allowed between networks unless you set up vpc network peering or use a vpn connection now i wanted to bring back the focus to the default vpc for just a minute unless you create an organizational policy that prohibits it new projects will always start with a default network that has one subnet in each region and again this is an auto mode vpc network in this particular example i am showing a default vpc with seven of its default regions displayed along with their ip ranges and again i want to stress that vpc networks along with their associated routes and firewall rules are global resources they are not associated with any particular region or zone so the subnets within them are regional and so when an auto mode vpc network is created one subnet from each region is automatically created within it these automatically created subnets use a set of predefined ip ranges that fit within the cider block that you see here of 10.128.0.049 and as new google cloud regions become available new subnets in those regions are automatically added to auto mode vpc networks by using an ip range from that block in addition to the automatically created subnets you can add more subnets manually to auto mode vpc networks in regions that you choose by using ip ranges outside of 10.128.0.049 now if you're using a default vbc or have already created an auto mode vpc you can switch the vpc network from auto mode to custom mode and this is a one-way conversion only as custom mode vpc networks cannot be changed to auto mode vpc networks now bringing this theory into practice with regards to the default vpc i wanted to take the time to do a short demo so whenever you're ready join me in the console and so here we are back in the console and if i go here in the top right hand corner i am logged in as tony bowties at gmail.com and in the top drop down project menu i'm logged in under project tony and because this demo is geared around the default vpc i want to navigate to vpc networks so i'm going to go over here to the top left hand corner to the navigation menu and i'm going to click on it and scroll down to vpc network under networking and so as you can see here in the left hand menu there are a bunch of different options that i can choose from but i won't be touching on any of these topics as i have other lessons that will deep dive into those topics so in this demo i'd like to strictly touch on the default vpc and as you can see in project tony it has created a default vpc for me with a one subnet in every region having its own ip address range and so just as a reminder whenever you create a new project a default vpc will be automatically created for you and when these subnets were created each of them have a route out to the public internet and so the internet gateway is listed here its corresponding firewall rules along with global dynamic routing and flow logs are turned off and again i will be getting deeper into routing and flow logs in later lessons in the section now earlier i had pointed out that an auto mode vpc can be converted to a custom vpc and it's as simple as clicking this button but we don't want to do that just yet and what i'd like to do is drill down into the default vbc and show you all the different options as you can see here the dns api has not been enabled and so for most of you a good idea would be to enable it and so i'm going to go ahead and do that now as well you can see here that i can make adjustments to each of the different subnets or i can change the configuration of the vpc itself so if i click on this edit button here at the top i'm able to change the subnet creation mode along with the dynamic routing mode which i will get into in a later lesson and the same thing with the dns server policy and so to make this demo a little bit more exciting i want to show you the process on how to expand a subnet so i'm going to go into us central one i'm going to drill down here and here's all the configuration settings for the default subnet in the us central one region and so for me to edit this subnet i can simply click on the edit button up here at the top and so right below the ip address range i am prompted with a note saying that the ip ranges must be unique and non-overlapping as we stated before and this is a very important point to know when you're architecting any vpcs or its corresponding sub networks and so i'm going to go ahead and change the subnet from a cider range of 20 and i'm going to change it to 16. i'm not going to add any secondary ip ranges i'm going to leave private google access off and so i'm going to leave everything else as is and simply click on save and so once this has completed i'll be able to see that my subnet range will go from a slash 20 to a slash 16. and so here you can see the ip address range has now changed to a slash 16. if i go back to the main page of the vpc network i can see that the ip address range is different from all the other ones now you're probably asking why can't i just change the ip address range on all the subnets at once and so even though i'd love to do that unfortunately google does not give you the option each subnet must be configured one by one to change the ipa address range now i wanted to quickly jump into the default firewall rules and as discussed earlier the rules for incoming ssh rdp and icmp have been pre-populated along with a default rule that allows incoming connections for all protocols and ports among instances within the same network so when it comes to routes with regards to the vpc network the only one i really wanted to touch on is the default route to the internet and so without this route any of the subnets in this vpc wouldn't have access to route traffic to the internet and so when the default vpc is created the default internet gateway is also created and so now going back to the main page of the vpc network i wanted to go through the process of making the ip address range bigger but doing it through the command line and so i'm going to go up to the right hand corner and open up cloud shell i'm going to make this a little bit bigger and so for this demo i'm going to increase the address range for the subnet in us west one from a slash 20 to a slash 16 and so i'm going to paste in the command which is gcloud compute networks subnets expand ip dash range and then the name of the network which is default as well as the region and i'm going to do uswest1 along with the prefix length which is going to be 16. so i'm going to hit enter i've been prompted to make sure that this is what i want to do and so yes i do want to continue so i'm going to type in y for yes and hit enter and so within a few seconds i should get some confirmation and as expected my subnet has been updated and so because i like to verify everything i'm going to now clear the screen and i'm going to paste in the command gcloud compute networks subnets describe and then the subnet name which is default along with the region which would be uswest1 i'm going to click on enter and as you can see here the ipsider range is consistent with what we have changed and if i do a quick refresh on the browser i'll be able to see that the console has reflected the same thing and as expected the ip address range here for us west one in the console reflects that which we see here in cloud shell and so now to end this demo i wanted to quickly show you how i can delete the default vpc and recreate it so all i need to do is to drill into the settings and then click on delete vpc network right here at the top i'm going to get a prompt to ask me if i'm sure and i'm going to simply click on delete now just as a note if you have any resources that are in any vpc networks you will not be able to delete the vpc you would have to delete the resources first and then delete the vpc afterwards okay and it has been successfully deleted and as you can see there are no local vpc networks in this current project and so i want to go ahead and recreate the default vpc so i'm going to simply click on create vpc network and so here i'm prompted to enter in a bunch of information for creating this new vpc network and so keeping with the spirit of default vpcs i'm going to name this vpc default i'm going to put default in the description and under subnet creation mode i'm going to click on automatic and as you can see a prompt came up telling me these ip address ranges will be assigned to each region in your vpc network and i'm able to review the ip address ranges for each region and as stated before the ip address ranges for each region will always be the same every time i create this default vpc or create a vpc in the automatic subnet creation mode now as a note here under firewall rules if i don't select these firewall rules none will actually be created so if you're creating a new default vpc be sure to check these off and so i'm going to leave everything else as is and i'm going to simply go to the bottom and click on the create button and within about a minute i should have the new default vpc created okay and we are back in business the default vpc has been recreated with all of these subnets in its corresponding regions all the ip address ranges the firewall rules everything that we saw earlier in the default vpc and so that's pretty much all i wanted to cover in this demo on the default vpc network along with the lesson on vpcs so you can now mark this lesson as complete and let's move on to the next one welcome back and in this lesson i'm going to be discussing vpc network subnets now the terms subnet and sub network are synonymous and are used interchangeably in google cloud as you'll hear me using either one in this lesson yet i am referring to the same thing now when you create a resource in google cloud you choose a network and a subnet and so because a subnet is needed before creating resources some good knowledge behind it is necessary for both building and google cloud as well as in the exam so in this lesson i'll be covering subnets at a deeper level with all of its features and functionality so with that being said let's dive in now each vpc network consists of one or more useful ip range partitions called subnets also known in google cloud as sub networks each subnet is associated with the region and vpc networks do not have any ip address ranges associated with them ip ranges are defined for the subnets a network must have at least one subnet before you can use it and as mentioned earlier when you create a project it will create a default vpc network with subnets in each region automatically auto mode will run under this same functionality now custom vpc networks on the other hand start with no subnets giving you full control over subnet creation and you can create more than one subnet per region you cannot change the name or region of a subnet after you've created it you would have to delete the subnet and replace it as long as no resources are using it primary and secondary ranges for subnets cannot overlap with any allocated range any primary or secondary range of another subnet in the same network or any ip ranges of subnets in peered networks in other words they must be a unique valid cider block now when it comes to ip addresses of a subnet google cloud vpc has an amazing feature that lets you increase the ip space of any subnets without any workload shutdown or downtime as demonstrated earlier in the previous lesson and this gives you the flexibility and growth options to meet your needs but unfortunately there are some caveats the new subnet must not overlap with other subnets in the same vpc network in any region also the new subnets must stay inside the rfc 1918 address space the new network range must be larger than the original which means the prefix length must be smaller in number and once a subnet has been expanded you cannot undo an expansion now auto mode network starts with a slash 20 range that can be expanded to a 16 ip range but not larger you can also convert the auto mode network to a custom mode network to increase the ip range even further and again this is a one-way conversion custom mode vpc networks cannot be changed to auto mode vpc networks now in any network that is created in google cloud there will always be some ip addresses that you will not be able to use and these are reserved for google and so every subnet has four reserved ip addresses in its primary ip range and just as a note there are no reserved ip addresses in the secondary ip ranges and these reserved ips can be looked at as the first two and the last two ip addresses in the cider range now the first address in the primary ip range for the subnet is reserved for the network the second address in the primary ip range for the subnet is reserved for the default gateway and allows you access to the internet the second to last address in the primary ip range for the subnet is reserved for google cloud for potential future use and the last address and the ip range for the subnet is for broadcast and so that about covers this short yet important lesson on vpc network subnets these features and functionalities of subnets that have been presented to you will help you make better design decisions that will give you a bit more knowledge and flexibility when it comes to assigning ipspace within your vpc networks and so that's all i have to cover for this lesson so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this lesson i'm going to be going through routing and private google access now although routing doesn't really show up in the exam i wanted to give you an inside look on how traffic is routed so when you're building in google cloud you'll know exactly what you will need to do if you need to edit these routes in any way or if you need to build new ones to satisfy your particular need now private google access does pop its head in the exam but only at a high level but i wanted to get just a bit deeper with the service and get into the data flow of when the service is enabled so with that being said let's dive in now google cloud routes define the paths that network traffic takes from a vm instance to other destinations these destinations can be inside your google cloud vpc network for example in another vm or outside it in a vpc network a route consists of a single destination and a single next hop when an instance in a vpc network sends a packet google cloud delivers the packet to the route's next hop if the packet's destination address is within the route's destination range and so all these routes are stored in the routing table for the vpc now for those of you who are not familiar with a routing table in computer networking a routing table is a data table stored in a router or a network host that lists the routes to particular network destinations and so in this case the vpc is responsible for storing the routing table as well each vm instance has a controller that is kept informed of all applicable routes from the network's routing table each packet leaving a vm is delivered to the appropriate next hop of an applicable route based on a routing order now i wanted to take a couple minutes to go through the different routing types that are available on google cloud now in google cloud there are two types of routing there is the system generated which offers the default and subnet route and then there are the custom routes which support static routes and dynamic routes and so i first wanted to cover system generated routes in a little bit of depth and so every new network whether it be an automatic vpc or a custom vpc has two types of system generated routes a default route which you can remove or replace and one subnet route for each of its subnets now when you create a vpc network google cloud creates a system generated default route and this route serves two purposes it defines the path out of the vpc network including the path to the internet in addition to having this route instances must meet additional requirements if they need internet access the default route also provides a standard path for private google access and if you want to completely isolate your network from the internet or if you need to replace the default route with the custom route you can delete the default route now if you remove the default route and do not replace it packets destined to ip ranges that are not covered by other routes are dropped lastly the system generated default route has a priority of 1000 because its destination is the broadest possible which covers all ip addresses in the 0.0.0.0.0 range google cloud only uses it if a route with a more specific destination does not apply to a packet and i'll be getting into priorities in just a little bit and so now that we've covered the default route i wanted to get into the subnet route now subnet routes are system generated routes that define paths to each subnet in the vpc network each subnet has at least one subnet route whose destination matches the primary ip range of the subnet if the subnet has secondary ip ranges google cloud creates a subnet route with a corresponding destination for each secondary range no other route can have a destination that matches or is more specific than the destination of a subnet route you can create a custom route that has a broader destination range that contains the subnet route's destination range now when a subnet is created a corresponding subnet route for the subnet's primary and secondary ip range is also created auto mode vpc networks create a subnet route for the primary ip ranges of each of their automatically created subnets you can delete these subnets but only if you convert the auto mode vpc network to custom mode and you cannot delete a subnet route unless you modify or delete the subnet so when you delete a subnet all subnet routes for both primary and secondary ranges are deleted automatically you cannot delete the subnet route for the subnet's primary range in any other way and just as a note when networks are connected by using vpc network peering which i will get into a little bit later some subnet routes from one network are imported into the other network and vice versa and cannot be removed unless you break the peering relationship and so when you break the peering relationship all imported subnet routes from the other network are automatically removed so now that we've covered the system generated routes i wanted to get into custom routes now custom routes are either static routes that you can create manually or dynamic routes maintained automatically by one or more of your cloud routers and these are created on top of the already created system generated routes destinations for custom routes cannot match or be specific than any subnet route in the network now static routes can use any of the static route next hops and these can be created manually if you use the google cloud console to create a cloud vpn tunnel that uses policy-based routing or one that is a route based vpn static routes for the remote traffic selectors are created for you and so just to give you a little bit more clarity and a little bit of context i've included a screenshot here for all the different routes that are available for the next hop we have the default internet gateway to define a path to external ip addresses specify an instance and this is where traffic is directed to the primary internal ip address of the vm's network interface in the vpc network where you define the route specify ip address is where you provide an internal ip address assigned to a google cloud vm as a next hop for cloud vpn tunnels that use policy based routing and route-based vpns you can direct traffic to the vpn tunnel by creating routes whose next hops refer to the tunnel by its name and region and just as a note google cloud ignores routes whose next hops are cloud vpn tunnels that are down and lastly for internal tcp and udp low balancing you can use a load balancer's ip address as a next hop that distributes traffic among healthy back-end instances custom static routes that use this next hop cannot be scoped to specific instances by network tags and so when creating static routes you will always be asked for different parameters that are needed in order to create this route and so here i've taken a screenshot from the console to give you a bit more context with regards to the information that's needed so first up is the name and description so these fields identify the route a name is required but a description is optional and every route in your project must have a unique name next up is the network and each route must be associated with exactly one vpc network in this case it happens to be the default network but if you have other networks available you're able to click on the drop down arrow and choose a different network the destination range is a single ipv4 cider block that contains the ip addresses of systems that receive incoming packets and the ip range must be entered as a valid ipv4 cider block as shown in the example below the field now if multiple routes have identical destinations priority is used to determine which route should be used so a lower number would indicate a higher priority for example a route with a priority value of 100 has a higher priority than one with a priority value of 200 so the highest route priority means the smallest possible non-negative number as well another great example is if you look back on your default routes all your subnet routes are of a priority of zero and the default internet gateway is of a priority of 1000 and therefore the subnet routes will take priority over the default internet gateway and this is due to the smaller number so remember a good rule of thumb is that the lower the number the higher the priority the higher the number the lower the priority now to get a little bit more granular you can specify a list of network tags so that the route only applies to instances that have at least one of the listed tags and if you don't specify any tags then google cloud applies the route to all instances in the network and finally next hop which was shown previously this is dedicated to static routes that have next hops that point to the options shown earlier so now that i've covered static routes in a bit of detail i want to get into dynamic routes now dynamic routes are managed by one or more cloud routers and this allows you to dynamically exchange routes between a vpc network and an on-premises network with dynamic routes their destinations always represent ip ranges outside of your vpc network and their next hops are always bgp peer addresses a cloud router can manage dynamic routes for cloud vpn tunnels that use dynamic routing as well as cloud interconnect and don't worry i'll be getting into cloud routers in a bit of detail in a later lesson now i wanted to take a minute to go through routing order and the routing order deals with priorities that i touched on a little bit earlier now subnet routes are always considered first because google cloud requires that subnet routes have the most specific destinations matching the ip address ranges of their respective subnets if no applicable destination is found google cloud drops the packet and replies with a network unreachable error system generated routes apply to all instances in the vpc network the scope of instances to which subnet routes apply cannot be altered although you can replace the default route and so just as a note custom static routes apply to all instances or specific instances so if the route doesn't have a network tag the route applies to all instances in the network now vpc networks have special routes that are used for certain services and these are referred to as special return paths in google cloud these routes are defined outside of your vpc network in google's production network they don't appear in your vpc network's routing table you cannot remove them or override them or if you delete or replace a default route in your vpc network although you can control traffic to and from these services by using firewall rules and the services that are covered are load balancers internet aware proxy or iap as well as cloud dns and so before i end this lesson i wanted to touch on private google access now vm instances that only have internal ip addresses can use private google access and this allows them to reach the external ip addresses of google's apis and services the source ip address of the packet can be the primary internal ip address of the network interface or an address in an alias ip range that is assigned to the interface if you disable private google access the vm instances can no longer reach google apis and services and will only be able to send traffic within the vpc network private google access has no effect on instances that have external ip addresses and can still access the internet they don't need any special configuration to send requests to the external ip addresses of google apis and services you enable private google access on a subnet by subnet basis and it's a setting for subnets in a vpc network and i will be showing you this in an upcoming demo where we'll be building our own custom vpc network now even though the next hop for the required routes is called the default internet gateway and the ip addresses for google apis and services are external requests to google apis and services from vms that only hold internal ip addresses in subnet 1 where private google access is enabled are not sent through the public internet those requests stay within google's network as well vms that only have internal ip addresses do not meet the internet access requirements for access to other external ip addresses beyond those for google apis and services now touching on this diagram here firewall rules in the vpc network have been configured to allow internet access vm1 can access google apis and services including cloud storage because its network interface is located in subnet 1 which has private google access enabled and because this instance only has an internal ip address private google access applies to this instance now with vm2 it can also access google apis and services including cloud storage because it has an external ip address private google access has no effect on this instance as it has an external ip address and private google access has not been enabled on that subnet and because both of these instances are in the same network they are still able to communicate with each other over an internal subnet route and so this is just one way where private google access can be applied there are some other options for private access as well you can use private google access to connect to google apis and services from your on-premises network through a cloud vpn tunnel or cloud interconnect without having any external ip addresses you also have the option of using private google access through a vpc network peering connection which is known as private services access and finally the last option available for private google access is connecting directly from serverless google services through an internal vpc connection now i know this has been a lot of theory to take in but i promise it'll become a lot easier and concepts will become less complicated when we start putting this into practice coming up soon in the demo of building our own custom vpc and so that's pretty much all i wanted to cover when it comes to routing and private google access so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this lesson i'm going to be discussing ip addressing now in the network refresher lesson i went into a bit of depth on how i p addresses are broken down and used for communication in computer networks in this lesson i'll be getting into the available types of ip addressing in google cloud and how they are used in each different scenario please note for the exam a high level overview will be needed to know when it comes to ip addressing but the details behind it will give you a better understanding on when to use each type of ip address so with that being said let's dive in now ip addressing in google cloud holds quite a few categories and really start by determining whether you are planning for communication internally within your vpc or for external use to communicate with the outside world through the internet once you determine the type of communication that you're looking to apply between resources some more decisions need to be made with regards to the other options and i will be going through these options in just a sec now in order to make these options a little bit more digestible i wanted to start off with the options available for internal ip addresses now internal ip addresses are not publicly advertised they are used only within a network now every vpc network or on-premises network has at least one internal ip address range resources with internal ip addresses communicate with other resources as if they're all on the same private network now every vm instance can have one primary internal ip address that is unique to the vpc network and you can assign a specific internal ip address when you create a vm instance or you can reserve a static internal ip address for your project and assign that address to your resources if you don't specify an address one will be automatically assigned to the vm in either case the address must belong to the ip range of the subnet and so if your network is an auto mode vpc network the address comes from the region subnet if your network is a custom mode vpc network you must specify which subnet the ip address comes from now all subnets have a primary sider range which is the range of internal ip addresses that define the subnet each vm instance gets its primary internal ip address from this range you can also allocate alias ip ranges from that primary range or you can add a secondary range to the subnet and allocate alias ip ranges from the secondary range use of alias ip ranges does not require secondary subnet ranges these secondary subnet ranges merely provide an organizational tool now when using ip aliasing you can configure multiple internal ip addresses representing containers or applications hosted in a vm without having to define a separate network interface and you can assign vm alias ip ranges from either the subnet's primary or secondary ranges when alias ip ranges are configured google cloud automatically installs vpc network routes for primary and alias ip ranges for the subnet of your primary network interface your container orchestrator or gke does not need to specify vpc network connectivity for these routes and this simplifies routing traffic and managing your containers now when choosing either an auto mode vpc or a custom vpc you will have the option to choose either an ephemeral ip or a static ip now an ephemeral ip address is an ip address that doesn't persist beyond the life of the resource for example when you create an instance or forwarding rule without specifying an ip address google cloud will automatically assign the resource an ephemeral ip address and this ephemeral ip address is released when you delete the resource when the ip address is released it is free to eventually be assigned to another resource so is never a great option if you depend on this ip to remain the same this ephemeral ip address can be automatically assigned and will be assigned from the selected region subnet as well if you have ephemeral ip addresses that are currently in use you can promote these addresses to static internal ip addresses so that they remain with your project until you actively remove them and just as a note before you reserve an existing ip address you will need the value of the ip address that you want to promote now reserving a static ip address assigns the address to your project until you explicitly release it this is useful if you are dependent on a specific ip address for a specific service and need to prevent another resource from being able to use the same address static addresses are also useful if you need to move an ip address from one google cloud resource to another and you also have the same options when creating an internal load balancer as you do with vm instances and so now that we've covered all the options for internal ip addresses i would like to move on to cover all the available options for external ip addresses now you can assign an external ip address to an instance or a forwarding rule if you need to communicate with the internet with resources in another network or need to communicate with a public google cloud service sources from outside a google cloud vpc network can address a specific resource by the external ip address as long as firewall rules enable the connection and only resources with an external ip address can send and receive traffic directly to and from outside the network and like internal ip addresses external ip addresses have the option of choosing from an ephemeral or static ip address now an ephemeral external ip address is an ip address that doesn't persist beyond the life of the resource and so follows the same rules as ephemeral internal ip addresses so when you create an instance or forwarding rule without specifying an ip address the resource is automatically assigned an ephemeral external ip address and this is something that you will see quite often ephemeral external ip addresses are released from a resource if you delete the resource for vm instances the ephemeral external ip address is also released if you stop the instance so after you restart the instance it is assigned a new ephemeral external ip address and if you have an existing vm that doesn't have an external ip address you can assign one to it forwarding rules always have an ip address whether external or internal so you don't need to assign an ip address to a forwarding rule after it is created and if your instance has an ephemeral external ip address and you want to permanently assign the ip to your project like ephemeral internal ip addresses you have the option to promote the ip address from ephemeral to static and in this case promoting an ephemeral external ip address to a static external ip address now when assigning a static ip address these are assigned to a project long term until they are explicitly released from that assignment and remain attached to a resource until they are explicitly detached for vm instances static external ip addresses remain attached to stopped instances until they are removed and this is useful if you are dependent on a specific ip address for a specific service like a web server or a global load balancer that needs access to the internet static external ip addresses can be either a regional or global resource in a regional static ip address allows resources of that region or resources of zones within that region to use the ip address and just as a note you can use your own publicly routable ip address prefixes as google cloud external ip addresses and advertise them on the internet the only caveat is that you must own and bring at the minimum a 24 cider block and so now that we've discussed internal and external ip addressing options i wanted to move into internal ip address reservations now static internal ips provide the ability to reserve internal ip addresses from the ip range configured in the subnet then assign those reserved internal addresses to resources as needed reserving an internal ip address takes that address out of the dynamic allocation pool and prevents it from being used for automatic allocations with the ability to reserve static internal ip addresses you can always use the same ip address for the same resource even if you have to delete and recreate the resource so when it comes to internal ip address reservation you can either reserve a static internal ip address before creating the associated resource or you can create the resource with an ephemeral internal ip address and then promote that ephemeral ip address to a static internal ip address and so just to give you a bit more context i have a diagram here to run you through it so in the first example you would create a subnet from your vpc network you would then reserve an internal ip address from that subnet's primary ip range and in this diagram is marked as 10.12.4.3 and will be held as reserved for later use with a resource and then when you decide to create a vm instance or an internal load balancer you can use the reserved ip address that was created in the previous step that i p address then becomes marked as reserved and in use now touching on the second example you would first create a subnet from your vpc network you would then create a vm instance or an internal load balancer with either an automatically allocated ephemeral ip address or a specific ip address that you've chosen from within that specific subnet and so once the ephemeral ip address is in use you can then promote the ephemeral ip address to a static internal ip address and would then become reserved and in use now when it comes to the external ip address reservation you are able to obtain a static external ip address by using one of the following two options you can either reserve a new static external ip address and then assign the address to a new vm instance or you can promote an existing ephemeral external ip address to become a static external ip address now in the case of external ip addresses you can reserve two different types a regional ip address which can be used by vm instances with one or more network interfaces or by network load balancers these ip addresses can be created either in the console or through the command line with the limitation that you will only be allowed to create ipv4 ip addresses the other type is a global ip address which can be used for global load balancers and can be created either in the console or through the command line as shown here the limitation here is that you must choose the premium network service tier in order to create a global ip address and after reserving the address you can finally assign it to an instance during instance creation or to an existing instance and so as you can see there is a lot to take in when it comes to understanding ip addressing and i hope this lesson has given you some better insight as to which type of ips should be used in a specific scenario now don't worry the options may seem overwhelming but once you start working with ip addresses more often the options will become so much clearer on what to use and when and as i said in the beginning only high level concepts are needed to know for the exam but knowing the options will allow you to make better decisions in your daily role as a cloud engineer and so that's pretty much all i wanted to cover when it comes to ip addressing in google cloud and so now that we've covered the theory behind ip addressing in google cloud i wanted to bring this into the console for a demo where we will get hands-on with creating both internal and external static ip addresses so as i explained before there was a lot to take in with this lesson so now would be a perfect opportunity to get up and have a stretch grab yourself a tea or a coffee and whenever you're ready join me back in the console so you can now mark this lesson as complete and i'll see you in the next [Music] welcome back in this demonstration i'm going to be going over how to create and apply both internal and external static ip addresses i'm going to show how to create them in both the console and the command line as well as how to promote ip addresses from ephemeral ips to static ips and once we're done creating all the ip addresses i'm going to show you the steps on how to delete them now there's a lot to get done here so let's dive in now for this demonstration i'm going to be using a project that has the default vpc created and so in my case i will be using project bowtieinc dev and so before you start make sure that your default vpc is created in the project that you had selected so in order to do that i'm going to head over to the navigation menu i'm going to scroll down to vpc network and we're going to see here that the default vpc has been created and so i can go ahead and start the demonstration and so the first thing i wanted to demonstrate is how to create a static internal ip address and so in order for me to demonstrate this i'm going to be using a vm instance and so i'm going to head over to the navigation menu again and i'm going to scroll down to compute engine and so here i'm going to create my new instance by simply clicking on create instance and so under name i'm going to keep it as instance 1. under region you want to select us east one and i'm going to keep the zone as the default selected under machine type i'm going to select the drop down and select e2 micro and i'm going to leave everything else as the default i'm going to scroll down here to management security disks networking and soul tenancy and i'm going to select the networking tab from there and so under here i'm going to select under network interfaces the default network interface and here is where i can create my static internal ip and so clicking on the drop down under primary internal ip you will see ephemeral automatic ephemeral custom and reserve static internal ip address and so you're going to select reserve static internal ip address and you'll get a pop-up prompting you with some fields to fill out to reserve a static internal ip address and so under name i'm going to call this static dash internal and for the purposes of this demo i'm going to leave the subnet and the static ip address as the currently selected if i wanted to select a specific ip address i can click on this drop down and select let me choose and this will give me the option to enter in a custom ip address with the subnet range that is selected for this specific sub network and so because i'm not going to do that i'm going to select assign automatically i'm going to leave the purpose as non-shared and i'm going to simply click on reserve and this is going to reserve this specific ip address and now as you can see here i have the primary internal ip marked as static internal and so this is going to be my first static internal ip address and so once you've done these steps you can simply click on done and you can head on down to the bottom and simply click on create to create the instance and when the instance finishes creating you will see the internal static ip address and as you can see here your static internal ip address has been assigned to the default network interface on instance 1. and so in order for me to view this static internal ip address in the console i can view this in vpc networks and drill down into the specific vpc and find it under static internal ip addresses but i wanted to show you how to view it by querying it through the command line and so in order to do this i'm going to simply go up to the menu bar on the right hand side and open up cloud shell and once cloud shell has come up you're going to simply paste in the command gcloud compute addresses list and this will give me a list of the internal ip addresses that are available and so now i'm going to be prompted to authorize this api call using my credentials and i definitely do so i'm going to click on authorize and as expected the static internal ip address that we created earlier has shown up it's marked as internal in the region of us east one in the default subnet and the status is in use and so as we discussed in the last lesson static ip addresses persist even after the resource has been deleted and so to demonstrate this i'm going to now delete the instance i'm going to simply check off the instance and go up to the top and click on delete you're going to be prompted to make sure if you want to delete this yes i do so i'm going to click on delete and so now that the instance has been deleted i'm going to query the ip addresses again by using the same command gcloud compute addresses list i'm going to hit enter and as you can see here the ip address static dash internal still persists but the status is now marked as reserved and so if i wanted to use this ip address for another instance i can do so by simply clicking on create instance up here at the top menu and then i can select static dash internal as my ip address so i'm going to quickly close down cloud shell and i'm going to leave the name as instance one the region can select us east one and we're going to keep the zone as the default selected under machine type you're going to select the e2 micro machine type going to scroll down to management security disks networking into soul tenancy and i'm going to select the networking tab from under here and under network interfaces i'm going to select the default network interface and under primary internal ip if i click on the drop down i have the option of selecting the static dash internal static ip address and so i wanted to move on to demonstrate how to promote an internal ephemeral ip address to an internal static ip address and so in order to do this i'm going to select on ephemeral automatic and i'm going to scroll down and click on done and i'm going to go ahead and create the instance and once the instance is ready i'll be able to go in and edit the network interface and so the instance is up and ready and so i'm going to drill down into the instance and i'm going to go up to the top and click on edit i'm going to scroll down to network interfaces and i'm going to edit the default network interface so i'm going to scroll down a little bit more and here under internal iptype i'm going to click on the drop down and i'm going to select static and so here you are taking the current ip address which is 10.142.0.4 and promoting it to a static internal ip address and so you're going to be prompted with a pop-up confirming the reservation for that static internal ip address and so notice that i don't have any other options and so all i'm going to do is type in a name and i'm going to call this promoted static and i'm going to click on reserve and this will promote the internal ip address from an ephemeral ip address to a static ip address and so now i'm just going to click on done and i'm going to scroll down and click on save and so now because i want to verify the ip address i'm going to go ahead and open up the cloud shell again and i'm going to use the same command that i used earlier which is gcloud compute addresses list and i'm going to hit enter as expected the promoted static ip address is showing as an internal ip address in the region of us east 1 in the default subnet and its status is in use and so just as a recap we've created a static internal ip address for the first instance and for the second instance we promoted an ephemeral internal ip address into a static internal ip address and we were able to verify this through cloud shell using the gcloud compute addresses list command and so this is the end of part one of this demo it was getting a bit long so i decided to break it up and this would be a great opportunity for you to get up and have a stretch get yourself a coffee or tea and whenever you're ready join me in part two where we will be starting immediately from the end of part one so you can now mark this as complete and i'll see you in the next one [Music] welcome back this is part two of the creating internal and external ip addresses demo and we will be starting immediately from the end of part one so with that being said let's dive in and so now that we've gone through how to both create static ip addresses and promote ephemeral ip addresses to static ip addresses for internal ips i want to go ahead and go through the same with external ips and so i'm going to first start off by deleting this instance i'm going to go ahead and click on delete and so instead of doing it through the compute engine interface i want to go into the external ip address interface which can be found in the vpc network menu so i'm going to go ahead up to the left hand corner click on the navigation menu and i'm going to scroll down to vpc network and from the menu here on the left hand side you can simply click on external ip addresses and here you will see the console where you can create a static external ip address and so to start the process you can simply click on reserve static address and so here you'll be prompted with a bunch of fields to fill out to create this new external static ip address and so for the name of this static ip address you can simply call this external dash static i'm going to use the same in the description now here under network service tier i can choose from either the premium or the standard and as you can see i'm currently using the premium network service tier and if i hover over the question mark over here it tells me a little bit more about this network service tier and as you can see the premium tier allows me higher performance as well as lower latency routing but this premium routing comes at a cost whereas the standard network service tier offers a lower performance compared to the premium network service tier and is a little bit more cost effective but still delivering performance that's comparable with other cloud providers and so i'm just going to leave it as the default selected and as we discussed in the previous lesson ipv6 external static ip addresses can only be used for global load balancers and so since we're only using it for an instance an ipv4 address will suffice and so just as a note for network service tier if i click on standard ipv6 is grayed out as well as the global selection and this is because in order to use global load balancing you need to be using the premium network service tier so whenever you're creating a global load balancer please keep this in mind as your cost may increase so i'm going to switch this back to premium and so under type i'm going to keep it as regional and under region i'm going to select the same region that my instance is going to be in which is us east 1 and because i haven't created the instance yet there is nothing to attach it to and so i'm going to click on the drop down and click on none and so just as another note i wanted to quickly highlight this caution point that the static ip addresses not attached to an instance or low balancer are still billed at an hourly rate so if you're not using any static ip addresses please remember to delete them otherwise you will be charged and so everything looks good here to create my external static ip address so i'm going to simply click on reserve and this will create my external static ip address and put the status of it as reserved so as you can see here the external static ip address has been created and you will find all of your external static ip addresses that you create in future right here in this menu and you will still be able to query all these external ip addresses from the command line and so now in order to assign this ip address to a network interface i'm going to go back over to the navigation menu and scroll down to compute engine and create a new instance so you can go ahead and click on create instance i'm going to go ahead and keep the name of this instance as instance one and in the region i'm going to select us east one i'm going to keep the zone as the selected default and under machine type i'm going to select the e2 micro machine type i'm going to scroll down to management security disks networking and soul tenancy and i'm going to select the networking tab and here under network interfaces i'm going to select the default network interface i'm going to scroll down a little bit here and under external ip ephemeral has been selected but if i click on the drop down i will have the option to select the ip that we had just created which is the external dash static ip and so i'm going to select that i'm going to click on done and you can go down and click on create and so now when the instance is created i will see the external ip address of external static as the assigned external ip and as expected here it is and because i always like to verify my work i'm going to go ahead and open up the cloud shell and verify it through the command line and so now i'm going to query all my available static ip addresses using the command gcloud compute addresses list i'm going to hit enter and as you can see here the external static ip address of 34.75.76 in the us east one region is now in use and this is because it is assigned to the network interface on instance one and so before we go ahead and complete this demo there's one more step that i wanted to go through and this is to promote an ephemeral external ip address to a static external ip address and so i'm going to go up here to the top menu and create a new instance i'm going to leave the name here as instance two under the region i'm going to select us east one i'm going to keep the zone as the selected default under machine type i'm going to select the e2 micro machine type i'm going to leave everything else as the default and i'm going to scroll down to management security disks networking and soul tenancy and select the networking tab and i'm going to verify that i'm going to be using an ephemeral external ip upon the creation of this instance if i scroll down here a little bit i can see that an external ephemeral ip address will be used upon creation and this will be the ip address that i will be promoting to a static ip through the command line so i'm going to go ahead and scroll down click on done and then i'm going to scroll down and click on create and once this instance is created then i can go ahead and promote the ephemeral external ip address okay and the instance has been created along with its external ephemeral ip address and so now i can go ahead and promote this ephemeral ip address so in order for me to do this i'm going to move back to my cloud shell and i'm going to quickly clear my screen and i'm going to use the command gcloud compute addresses create and then the name that we want to use for this static external ip address so i'm going to call this promoted external i'm going to use the flag dash dash addresses and so here i will need the external ip address that i am promoting which is going to be 104.196.219.42 and so i'm going to copy this to my clipboard and i'm going to paste it here in the command line and now i'm going to add the region flag along with the region of us east one and i'm going to go ahead and hit enter and success my ephemeral external ip address has been promoted to a static external ip address and of course to verify it i'm going to simply type in the gcloud compute addresses list command i'm going to hit enter and as expected here it is the promoted external ip of 104.196.219.42 marked as external in the u.s east one region and the status is marked as in use and so i wanted to take a moment to congratulate you on making it through this demonstration of creating internal and external ip addresses as well as promoting them so just as a recap you've created a static internal ip address in conjunction with creating a new instance and assigning it to that instance you then created another instance and used an ephemeral ip and then promoted it to a static internal ip address you then created an external static ip address using the console and assigned it to a brand new instance you then created another instance using an external ephemeral ip address and promoted it to a static external ip address and you did this all using both the console and the command line so i wanted to congratulate you on a great job now before we end this demonstration i wanted to go through the steps of cleaning up any leftover resources so the first thing you want to do is delete these instances so you can select them all and go up to the top and click on delete it's going to ask you if you want to delete the two instances yes we do click on delete and this will delete your instances and free up the external ip addresses so that you're able to delete them and so now that the instances have been deleted i'm going to go over to the vpc network menu and i'm going to head on over to the external ip address console and here i'm able to delete the external ip addresses and so i'm going to select all of them and i'm going to go up to the top menu and click on release static address and you should get a prompt asking you if you want to delete both these addresses the answer is yes click on delete and within a few seconds these external ip addresses should be deleted and so now all that's left to delete are the two static internal ip addresses and as i said before because there is no console to be able to view any of these static internal ip addresses i have to do it through the command line so i'm going to go back to my cloud shell i'm going to clear the screen and i'm going to list the ip addresses currently in my network and so here they are promoted static and static internal and so the command to delete any static ip addresses is as follows gcloud compute addresses delete the name of the ip address that i want to delete which is promoted static and then i will need the region flag and it'll be the region of us east one and i'm going to go ahead and hit enter it's going to prompt me if i want to continue with this and i'm going to type y for yes hit enter and success it has been deleted and so just a double check i'm going to do a quick verification and yes it has been deleted and so all that's left to delete is the static internal ip address and so i'm going to paste in the command gcloud compute addresses delete the name of the ip address that i want to delete which is static dash internal along with the region flag of us east one i'm going to go ahead and hit enter y for yes to continue and success and one last verification to make sure that it's all cleared up and as you can see i have no more static i p addresses and so this concludes this demonstration on creating assigning and deleting both static internal and static external ip addresses and so again i wanted to congratulate you on a great job and so that's pretty much all i wanted to cover in this demo on creating internal and external static ip addresses so you can now mark this as complete and i'll see you in the next one [Music] welcome back in this lesson i will be diving into some network security by introducing vpc firewall rules a service used to filter incoming and outgoing network traffic based on a set of user-defined rules a concept that you should be fairly familiar with for the exam and comes up extremely often when working as an engineer in google cloud it is definitely an essential security layer that prevents unwanted access to your cloud infrastructure now vpc firewall rules apply to a given project and network and if you'd like you can also apply firewall rules across an organization but i will be sticking to strictly vpc firewall rules in this lesson now vpc firewall rules let you allow or deny connections to or from your vm instances based on a configuration that you specify and these rules apply to either incoming connections or outgoing connections but never both at the same time enabled vpc firewall rules are always enforced regardless of their configuration and operating system even if they have not started up now every vpc network functions as a distributed firewall when firewall rules are defined at the network level connections are allowed or denied on a per instance basis so you can think of the vpc firewall rules as existing not only between your instances and other networks but also between individual instances within the same network now when you create a vpc firewall rule you specify a vpc network and a set of components that define what the rule does the components enable you to target certain types of traffic based on the traffic's protocol ports sources and destinations when you create or modify a firewall rule you can specify the instances to which it is intended to apply by using the target component of the rule now in addition to firewall rules that you create google cloud has other rules that can affect incoming or outgoing connections so for instance google cloud doesn't allow certain ip protocols such as egress traffic on tcp port 25 within a vpc network and protocols other than tcp udp icmp and gre to external ip addresses of google cloud resources are blocked google cloud always allows communication between a vm instance and its corresponding metadata server at 169.254 and this server is essential to the operation of the instance so the instance can access it regardless of any firewall rules that you configure the metadata server provides some basic services to the instance like dhcp dns resolution instance metadata and network time protocol or ntp now just as a note every network has two implied firewall rules that permit outgoing connections and block incoming connections firewall rules that you create can override these implied rules now the first implied rule is the allow egress rule and this is an egress rule whose action is allow and the destination is all ips and the priority is the lowest possible and lets any instance send traffic to any destination except for traffic blocked by google cloud the second implied firewall rule is the deny ingress rule and this is an ingress rule whose action is deny and the source is all ips and the priority is the lowest possible and protects all instances by blocking incoming connections to them now i know we touched on this earlier on in a previous lesson but i felt the need to bring it up as these are pre-populated rules and the rules that i'm referring to are with regards to the default vpc network and as explained earlier these rules can be deleted or modified as necessary the rules as you can see here in the table allow ingress connections from any source to any instance on the network when it comes to icmp rdp on port 3389 for windows remote desktop protocol and for ssh on port 22. and as well the last rule allows ingress connections for all protocols and ports among instances in the network and it permits incoming connections to vm instances from others in the same network and all of these have a rule priority of six five five four which is the second to lowest priority so breaking down firewall rules there are a few characteristics that google put in place that help define these rules and the characteristics are as follows each firewall rule applies to incoming or outgoing connections and not both firewall rules only support ipv4 connections so when specifying a source for an ingress rule or a destination for an egress rule by address you can only use an ipv4 address or ipv4 block insider notation as well each firewall rules action is either allow or deny you cannot have both at the same time and the rule applies to connections as long as it is enforced so for example you can disable a rule for troubleshooting purposes and then enable it back again now when you create a firewall rule you must select a vpc network while the rule is enforced at the instance level its configuration is associated with a vpc network this means you cannot share firewall rules among vpc networks including networks connected by vpc network peering or by using cloud vpn tunnels another major thing to note about firewall rules is that they are stateful and so that means when a connection is allowed through the firewall in either direction return traffic matching this connection is also allowed you cannot configure a firewall rule to deny associated response traffic return traffic must match the five tuple of the accepted request traffic but with the source and destination addresses and ports reversed so just as a note for those who may be wondering what a five tuple is i was referring to the set of five different values that comprise a tcpip connection and this would be source ip destination ip source port destination port and protocol google cloud associates incoming packets with corresponding outbound packets by using a connection tracking table google cloud implements connection tracking regardless of whether the protocol supports connections if a connection is allowed between a source and a target or between a target and a destination all response traffic is allowed as long as the firewalls connections tracking state is active and as well as a note a firewall rules tracking state is considered active if at least one packet is sent every 10 minutes now along with the multiple characteristics that make up a firewall rule there are also firewall rule components that go along with it here i have a screenshot from the console with the configuration components of a firewall rule and i wanted to take a moment to highlight these components for better clarity so now the first component is the network and this is the vpc network that you want the firewall rule to apply to the next one is priority which we discussed earlier and this is the numerical priority which determines whether the rule is applied as only the highest priority rule whose other components match traffic is applied and remember the lower the number the higher the priority the higher the number the lower the priority now the next component is the direction of traffic and these are the ingress rules that apply to incoming connections from specified sources to google cloud targets and this is where ingress rules apply to incoming connections from specified sources to google cloud targets and egress rules apply to connections going to specify destinations from targets and the next one up is action on match and this component either allows or denies which determines whether the rule permits or blocks the connection now a target is what defines which instances to which the rule applies and you can specify a target by using one of the following three options the first option are all instances in the network and this is the firewall rule that does exactly what it says it applies to all the instances in the network the second option is instances by target tags and this is where the firewall rule applies only to instances with a matching network tag and so i know i haven't explained it earlier but a network tag is simply a character string added to a tags field in a resource so let's say i had a bunch of instances that were considered development i can simply throw a network tag on them using a network tag of dev and apply the necessary firewall rule for all the development servers holding the network tag dev and so the third option is instances by target service accounts this is where the firewall rule applies only to instances that use a specific service account and so the next component is the source filter and this is a source for ingress rules or a destination for egress rules the source parameter is only applicable to ingress rules and it must be one of the following three selections source ip ranges and this is where you specify ranges of ip addresses as sources for packets either inside or outside of google cloud the second one is source tags and this is where the source instances are identified by a matching network tag and source service accounts where source instances are identified by the service accounts they use you can also use service accounts to create firewall rules that are a bit more granular and so one of the last components of the firewall rule is the protocols and ports you can specify a protocol or a combination of protocols and their ports if you omit both protocols and ports the firewall rule is applicable for all traffic on any protocol and any port and so when it comes to enforcement status of the firewall rule there is a drop down right underneath all the components where you can enable or disable the enforcement and as i said before this is a great way to enable or disable a firewall rule without having to delete it and is great for troubleshooting or to grant temporary access to any instances and unless you specify otherwise all firewall rules are enabled when they are created but you can also choose to create a rule in a disabled state and so this covers the vpc firewall rules in all its entirety and i will be showing you how to implement vpc firewall rules along with building a custom vpc custom routes and even private google access all together in a demo following this lesson to give you some hands-on skills of putting it all into practice and so that's pretty much all i wanted to cover when it comes to vpc firewall rules so you can now mark this lesson as complete and let's move on to the next one where we dive in and build our custom vpc so now is a perfect time to grab a coffee or tea and whenever you're ready join me in the console welcome back in this demonstration i want to take all the concepts that we've learned so far in this networking section and put it all into practice this diagram shown here is the architecture of exactly what we will be building in this demo we're going to start by creating a custom vpc and then we're going to create two subnets one public and one private in two separate regions we're then going to create a cloud storage bucket with some objects in it and then we will create some instances to demonstrate access to cloud storage as well as communication between instances and finally we're going to create some firewall rules for routing traffic to all the right places we're also going to implement private google access and demonstrate accessibility to the files in cloud storage from the private instance without an external ip so this may be a little bit out of your comfort zone for some but don't worry i'll be with you every step of the way and other than creating the instances all the steps here have been covered in previous lessons now there's a lot to get done here so whenever you're ready join me in the console and so here we are back in the console and as you can see up here in the right hand corner i am logged in as tony bowtie ace gmail.com and currently i am logged in under project tony and so in order to start off on a clean slate i'm going to create a new project and so i'm going to simply click on the project menu drop-down and click on new project i'm going to call this project bowtie inc and i don't have any organizations so i'm going to simply click on create and as well for those of you doing this lesson i would also recommend for you to create a brand new project so that you can start off anew again i'm going to go over to the project drop down and i'm going to select bow tie ink as the project and now that i have a fresh new project i can now create my vpc network so i'm going to go over to the left hand corner to the navigation menu and i'm going to scroll down to vpc network and so because vpc networks are tied in with the compute engine api we need to enable it before we can create any vpc networks so you can go ahead and enable this api so once this api has finished and is enabled we'll be able to create our vpc network ok and the api has been enabled and as you can see the default vpc network has been created with a subnet in every region along with its corresponding ip address ranges and so for this demo we're going to create a brand new vpc network along with some custom subnets and so in order to do that i'm going to go up here to the top and i'm going to click on create vpc network and so here i'm prompted with some fields to fill out so under name i'm going to think of a creative name that i can call my vpc network so i'm going to simply call it custom under description i'm going to call this custom vpc network and i'm going to move down here to subnets and because i'm creating custom subnets i'm going to keep it under custom under subnet creation mode and so i'm going to need a public subnet and a private subnet and you'll be able to get the values from the text file in the github repository within the sub networks folder under networking services and so i'm going to create my public subnet first and i'm going to simply call the public subnet public for region i'm going to use us east one and the ip address range will be 10.0.0.0 forward slash 24 and i'm going to leave private google access off and i'm going to simply click on done and now i can create the private subnet so underneath the public subnet you'll see add subnet you can simply click on that and the name of the new subnet will be as you guessed it private under region i'm going to use us east 4 and for the ip address range be sure to use 10.0.5.0.24 and we're going to leave private google access off for now and we'll be turning that on a little bit later in the demo and so you can now click on done and before we click on create we want to enable the dns api and clicking on enable will bring you to the dns api home page and you can click on enable to enable the api okay so now that we have our network configured along with our public and private subnets as well as dns being enabled we can now simply click on create but before i do that i wanted to give you some insight with regards to the command line so as i've shared before everything that can be done in the console can be done through the command line and so if ever you wanted to do that or you wanted to get to know the command line a little bit better after filling out all the fields with regards to creating resources in the console you will be given the option of a command line link that you can simply click on and here you will be given all the commands to create all the same resources with all the same preferences through the command line and i will be providing these commands in the lesson text so that you can familiarize yourself with the commands to use in order to build any networks using the command line but this is a great reference for you to use at any time and so i'm going to click on close and now i'm going to click on create and within a minute or two the custom vpc network will be created and ready to use okay and the custom vpc network has been created along with its public and private subnet and so just to get a little bit more insight with this custom vpc network i'm going to drill down into it and as you can see here the subnets are respectively labeled private and public along with its region ip address range the gateway and private google access the routes as you can see here are the system generated routes that i had discussed in an earlier lesson it has both the subnet routes to its respective ip range along with the default route with a path to the internet as well as a path for private google access now we don't have any firewall rules here yet but we'll be adding those in just a few minutes and so now that you've created the vpc network with its respective subnets we're going to head on over to cloud storage and create a bucket along with uploading the necessary files so i'm going to go again over to the navigation menu and i'm going to scroll down to storage and so as expected there are no buckets present here in cloud storage and so we're just going to go ahead and create our first bucket by going up here to the top menu and clicking on create bucket and so here i've been prompted to name my bucket and for those of you who are here for the first time when it comes to naming a storage bucket the name needs to be globally unique and this means that the name has to be unique across all of the google cloud platform now don't worry i'm going to get into further detail with this in the cloud storage lesson with all of these specific details when it comes to names storage classes and permissions and so in the meantime you can come up with a name for your bucket something that resonates with you and so for me i'm going to name my bucket bowtie inc dash file dash access and so now i'm going to simply click continue and so just as a note for those who are unable to continue through it is because the name for your bucket is not globally unique so do try to find one that is now when it comes to location type i'm just going to click on region and you can keep the default location as used one and i'm going to leave all the other options as default and i'm going to go down to the bottom and click create and so for those of you who have created your bucket you can now upload the files and those files can be found in the github repository in the cloud storage bucket folder under networking services and so now i'm going to click on upload files and under the networking services section under cloud storage bucket you will find these three jpeg files and you can simply select them and click on open and so they are now uploaded into the bucket and so now i'm ready to move on to the next step so you should now have created the vpc network with a private and public subnet along with creating your own bucket in cloud storage and have uploaded the three jpeg files so now that this is done we can now create the instances that will have access to these files and so again i will go over to the navigation menu in the top left hand corner and scroll down to compute engine and here i will click on create and so again i will be prompted with some fields to fill out and so for this instance i'm going to first create the public instance again i'm going to get really creative and call this public dash instance under labels i'm going to add a label under key i'm going to type environment and under value i'm going to type in public i'm going to go down to the bottom and click on save and under region i'm going to select us east1 and you can leave the zone as us east 1b moving down under machine type i'm going to select the e2 micro as the machine type just because i'm being cost conscious and i want to keep the cost down and so i'm going to scroll down to identity and api access and under service account you should have the compute engine default service account already pre-selected now under access scopes i want to be able to have the proper permissions to be able to read and write to cloud storage along with read and write access to compute engine and so you can click on set access for each api and you can scroll down to compute engine click on the drop down menu and select read write and this will give the public instance the specific access that it needs to ssh into the private instance and so now i'm going to set the access for cloud storage so i'm going to scroll down to storage i'm going to click on the drop down menu and select read write and this will give the instance read write access to cloud storage scrolling down a little bit further i'm going to go to management security disks networking and sold tenancy and i'm going to click on that scroll up here just a little bit and you can click on the networking tab which will prompt you for a bunch of options that you can configure for the networking of the instance so under network tags i want to type in public and you can click enter you can then scroll down to where it says network interfaces and click on the current interface which is the default and here it'll open up all your options and so under network you want to click on the drop down and set it from default to custom the public subnet will automatically be propagated so you can leave it as is and you also want to make sure that your primary internal ip as well as your external ip are set to ephemeral and you can leave all the other options as default and simply click on done and again before clicking on create you can click on the command line link and it will show you all the commands needed in order to create this instance through the command line so i'm going to go ahead and close this and so i'm going to leave all the other options as default and i'm going to click on create and so now that my public instance is being created i'm going to go ahead and create my private instance using the same steps that i did for the last instance so i'm going to go ahead and click on create instance here at the top and so the first thing i'm going to be prompted for is the name of the instance and so i'm going to call this instance private dash instance and here i'm going to add a label the key being environment and the value being private i'm going to go down here to the bottom and click on save and under region i'm going to select us east 4 and you can keep the zone as the default selected under machine type we're going to select the e2 micro and again scrolling down to the identity and api access under the access scopes for the default service account i'm going to click on the set access for each api and i'm going to scroll down to storage i'm going to click on the drop down menu and i'm going to select access for read write and for the last step i'm going to go into the networking tab under management security disks networking and soul tenancy and under network tags i'm going to give this instance a network tag of private and under network interfaces we want to edit this and change it from default over to the custom network and as expected it selected the private subnet by default and because this is going to be a private instance we are not going to give this an external ip so i'm going to click on the drop down and select none and with all the other options set as default i'm going to simply click on create and this will create my private instance along with having my public instance so just as a recap we've created a new custom vpc network along with a private and public subnet we've created a storage bucket and added some files in it to be accessed and we've created a private and public instance and assigning the service account on the public instance read write access to both compute engine and cloud storage along with a public ip address and assigning the service account on the private instance read write access only for cloud storage and no public ip and so this is the end of part one of this demo and this would be a great opportunity for you to get up and have a stretch get yourself a coffee or tea and whenever you're ready you can join me in part two where we will be starting immediately from the end of part one so you can go ahead and complete this video and i will see you in part two [Music] welcome back this is part two of the custom vpc demo and we will be starting exactly where we left off from part one so with that being said let's dive in and so now the last thing that needs to be done is to simply create some firewall rules and so with these firewall rules this will give me ssh access into the public instance as well as allowing private communication from the public instance to the private instance as well as giving ssh access from the public instance to the private instance and this will allow us to access the files in the bucket from the private instance and so in order to create these firewall rules i need to go back to my vpc network so i'm going to go up to the left hand corner again to the navigation menu and scroll down to vpc network over here on the left hand menu you'll see firewall i'm going to click on that and here you will see all the default firewall rules for the default network so for us to create some new ones for the custom vpc i'm going to go up here to the top and click on create firewall and so the first rule i want to create is for my public instance and i want to give it public access as well as ssh access and so i'm going to name this accordingly as public dash access i'm going to give this the same description always a good idea to turn on logs but for this demonstration i'm going to keep them off under network i'm going to select the custom network i'm going to keep the priority at 1000 the direction of traffic will be ingress and the action on match will be allow and so here is where the target tags come into play when it comes to giving access to the network so targets we're going to keep it as specified target tags and under target tags you can simply type in public under source filter you can keep it under ip ranges and the source ip range will be 0.0.0.0 forward slash 0. and we're not going to add a second source filter here so moving down to protocols and ports under tcp i'm going to click that off and add in port 22. and because i want to be able to ping the instance i'm going to have to add another protocol which is icmp and again as explained earlier the disable rule link will bring up the enforcement and as you can see it is enabled but if you wanted to create any firewall rules in future and have them disabled you can do that right here but we're gonna keep this enabled and we're gonna simply click on create and this will create the public firewall rule for our public instance in our custom vpc network and so we're going to now go ahead and create the private firewall rule and so i'm going to name this private dash access respectively i'm going to put the description as the same under network i'm going to select our custom network keep the priority at 1000 direction of traffic should be at ingress and the action on match should be allow for target tags you can type in private and then hit enter and because i want to be able to reach the private instance from the public instance the source ip range will be 10.0.0.1 forward slash 24. we're not going to add a second source filter and under protocols and ports we're going to simply add tcp port 22 and again i want to add icmp so that i'm able to ping the instance and i'm going to click on create and so we now have our two firewall rules private access and public access and if i go over to the custom vpc network and i drill into it i'll be able to see these selective firewall rules under the respective firewall rules tab and so now that we've created our vpc network along with the public and private subnet we've created the cloud storage bucket with the files that we need to access the instances that will access those files along with the firewall rules that will allow the proper communication we can now go ahead to test everything that we built and make sure that everything is working as expected so let's kick things off by first logging into the public instance so you can head on over to the navigation menu and scroll down to compute engine and you can ssh into the public instance by clicking on ssh under connect and this should open up a new tab or a new window logging you in with your currently authenticated credentials okay and we are logged into our instance and i'm going to zoom in for better viewing and so just to make sure that everything is working as expected we know that our firewall rule is correct because we are able to ssh into the instance and now i want to see if i have access to my files in the bucket and so in order to do that i'm going to run the gsutil command ls for list and then gs colon forward slash forward slash along with my bucket name which is bow tie inc hyphen file iphone access and i'm going to hit enter and as you can see i have access to all the files in the bucket and the last thing i wanted to check is if i can ping the private instance so i'm going to first clear my screen and i'm going to head on over back to the console i'm going to copy the ip address of the private instance to my clipboard and then i'm going to head back on over to my terminal and i'm going to type in ping i'm going to paste the ip address and success i am able to successfully ping the private instance from the public instance using the icmp protocol and you can hit control c to stop the ping so now that i know that my public instance has the proper permissions to reach cloud storage as well as being able to ping my private instance i want to be able to check if i can ssh into the private instance from my public instance and so i'm going to first clear my screen and next i'm going to paste in this command in order for me to ssh into the private instance g cloud compute ssh dash dash project and my project name which is bow tie inc dash dash zone and the zone that my instance is in which is us east 4c along with the name of the instance which is private dash instance and along with the flag dash dash internal dash ip stating that i am using the internal ip in order to ssh into the instance and i'm going to hit enter and so now i've been prompted for a passphrase in order to secure my rsa key pair as one is being generated to log into the private instance now it's always good practice when it comes to security to secure your key pair with a passphrase but for this demo i'm just going to leave it blank and so i'm just going to hit enter i'm going to hit enter again now i don't want to get too deep into it but i did want to give you some context on what's happening here so when you log into an instance on google cloud with os login disabled google manages the authorized keys file for new user accounts based on ssh keys in metadata and so the keys that are being generated that are being used for the first time are currently being stored within the instance metadata so now that i'm logged into my private instance i'm going to quickly clear my screen and just as a note you'll be able to know whether or not you're logged into your private instance by looking here at your prompt and so now i want to make sure that i can ping my public instance so i'm going to quickly type the ping command i'm going to head on over to the console i'm going to grab the ip address of the public instance i'm going to go back to my terminal and paste it in and as expected i'm able to ping my public instance from my private instance i'm just going to go ahead and hit control c to stop and i'm going to clear the screen so now we'd like to verify whether or not we have access to the files in the cloud storage bucket that we created earlier and so now i'm going to use the same command that i used in the public instance to list all the files in the cloud storage bucket so i'm going to use the gsutil command ls for list along with gs colon forward slash forward slash and the bucket name which is bow tie ink hyphen file if an access and i'm going to hit enter and as you can see here i'm not getting a response and the command is hanging and this is due to the fact that external access is needed in order to reach cloud storage and this instance only has an internal or private ip so accessing the files in the cloud storage bucket is not possible now in order to access cloud storage and the set of external ip addresses used by google apis and services we can do this by enabling private google access on the subnet used by the vms network interface and so we're going to go ahead and do that right now so i'm going to hit control c to stop and i'm going to go back into the console i'm going to go to the navigation menu and i'm going to scroll down to vpc network and then i'm going to drill down into the private subnet and i'm going to edit it under private google access i'm going to turn it on and i'm going to go down to the bottom and click on save and by giving this subnet private google access i will allow the private instance and any instances with private ip addresses to access any public apis such as cloud storage so now when i go back to my instance i'm going to clear the screen here and i'm going to run the gsutil command again and success we are now able to access cloud storage due to enabling private google access on the respective private subnet so i first wanted to congratulate you on making it to the end of this demo and hope that this demo has been extremely useful as this is a real life scenario that can come up and so just as a recap you've created a custom network with two custom subnets you've created a cloud storage bucket and uploaded some files to it you've created a public instance and a private instance and then created some firewall rules to route the traffic you then tested it all by using the command line for communication you also enable private google access for the instance with only the internal ip to access google's public apis so that it can access cloud storage and so again fantastic job on your part as this was a pretty complex demo and you can expect things like what you've experienced in this demo to pop up in your role of being a cloud engineer at any time so before you go be sure to delete all the resources you've created and again congrats on the great job so you can now mark this as complete and i'll see you in the next one welcome back in this lesson i will be going over vpc network peering and how you can privately communicate across vpcs in the same or different organization vpc network peering and vpc peering are used interchangeably in this lesson as they are used to communicate the same thing now for instances in one vpc to communicate with an instance in another vpc they would route traffic via the public internet however to communicate privately between two vpcs google cloud offers a service called vpc peering and i will be going through the theory and concepts of vpc peering throughout this lesson so with that being said let's dive in now vpc peering enables you to peer vpc networks so that workloads in different vpc networks can communicate in a private space that follows the rfc 1918 standard thus allowing private connectivity across two vpc networks traffic stays within google's network and never traverses the public internet vpc peering gives you the flexibility of peering networks that are of the same or different projects along with being able to peer with other networks in different organizations vpc peering also gives you several advantages over using external ip addresses or vpns to connect the first one is reducing network latency as all peering traffic stays within google's high-speed network vpc peering also offers greater network security as you don't need to have services exposed to the public internet and deal with greater risks of having your traffic getting compromised or if you're trying to achieve compliance standards for your organization vpc peering will allow you to achieve the standards that you need and finally vpc network peering reduces network costs as you save on egress costs for traffic leaving gcp so in a regular network google charges you for traffic communicating using public ips even if the traffic is within the same zone now you can bypass this and save money by using internal ips to communicate and keeping the traffic within the gcp network now there are certain properties or characteristics that peered vpcs follow and i wanted to point these out for better understanding first off peer vpc networks remain administratively separate so what exactly does this mean well it means that routes firewalls vpns and other traffic management tools are administered and applied separately in each of the vpc networks so this applies to each vpc independently which also means that each side of a peering association is set up independently as well so when you connect one vpc to the other you have to go into each vpc that you are connecting to both initiate and establish the connection peering becomes active only when the configuration from both sides match this also means that each vpc can delete the peering association at any given time now during vpc peering the vpc peers always exchange all subnet routes you also have the option of exchanging custom routes subnet and static routes are global and dynamic routes can be regional or global a given vpc network can peer with multiple vpc networks but there is a limit that you can reach in which you would have to reach out to google and ask the limit to be increased now when peering with vpc networks there are certain restrictions in place that you should be aware of first off a subnet cider range in one peered vpc network cannot overlap with a static route in another peered network this rule covers both subnet routes and static routes so when a vpc subnet is created or a subnet ip range is expanded google cloud performs a check to make sure that the new subnet range does not overlap with ip ranges of subnets in the same vpc network or in directly peered vpc networks if it does the creation or expansion will fail google cloud also ensures that no overlapping subnet ip ranges are allowed across vpc networks that have appeared network in common and again if it does the creation or expansion will fail now speaking of routing when you create a new subnet in appeared vpc network vpc network peering doesn't provide granular route controls to filter out which subnet cider ranges are reachable across pure networks these are handled by firewall rules so to allow ingress traffic from vm instances in a peer network you must create ingress allow firewall rules by default ingress traffic to vms is blocked by the implied deny ingress rule another key point to note is that transitive peering is not supported and only directly peered networks can communicate so they have to be peered directly in this diagram network a is peered with network b and network b is peered with network c and so if one instance is trying to communicate from network a to network c this cannot be done unless network a is directly peered with network c an extremely important point to note for vpc peering another thing to note is that you cannot use a tag or service account from one peered network in the other peered network they must each have their own as again they are each independently operated as stated earlier and so the last thing that i wanted to cover is that internal dns is not accessible for compute engine in peered networks as they must use an ip to communicate and so that about covers this short yet important lesson on the theory and concepts of vpc peering and so now that we've covered all the theory i'm going to be taking these concepts into a demo where we will be pairing two networks together and verifying the communication between them and so you can now mark this lesson as complete and whenever you're ready join me in the console welcome back in this hands-on demonstration we're going to go through the steps to create a peering connection from two vpcs in two separate projects as shown here in the diagram and then to verify that the connection works we're going to create two instances one in each network and ping one instance from the other instance this demo is very similar to the custom vpc demo that you had done earlier but we are adding in another layer of complexity by adding in vpc network peering and so there's quite a bit to do here so let's go ahead and just dive in okay so here we are back in the console as you can see up in the top right hand corner i am logged in as tony bowties gmail.com and for this specific demo i will be using two projects both project tony and project bowtie inc and if you currently do not have two projects you can go ahead and create yourself a new project or the two projects if you have none and so i'm going to continue here with project tony and the first thing i want to do is create the two networks in the two separate projects so i'm going to go up to the navigation menu in the top left hand corner and i'm going to scroll down to vpc network here i'm going to create my first vpc network and i'm going to name this bowtie ink dash a i'm going to give it the same description and then under subnets i'm going to leave the subnet creation mode under custom under the subnet name you can call this subnet dash a i'm going to use the us east one region and for the ip address range i'm going to use 10.0 that's 0.0 forward slash 20. and i'm going to leave all the other options as default and i'm going to go down to the bottom and click on create now as this network is being created i'm going to go over to the project bowtie inc and i'm going to create the vpc network there so under name i'm going to call this bowtie inc b and under description i'm going to use the same under subnets i'm going to keep subnet creation mode as custom and under new subnet i'm going to call this subnet subnet b the region will be used 4 and the ip address range will be 10.4.0.0 forward slash 20. you can leave all the other options as default and scroll down to the bottom and click on create as this network is being created i'm going to go back to project tony and i'm going to create the firewall rule for bow tie ink dash a in this firewall rule as explained in the last lesson we'll allow communication from one instance to the other and so i'm going to click on create firewall and under name i'm going to call this project tony dash a under description i'm going to use the same under the network i'm going to choose the source network which will be bowtie inc dash a priority i'm going to keep at 1000 direction of traffic should be ingress and action on match should be allow under targets i'm going to select all instances in the network and under source filter i'm going to keep ip ranges selected and the source ip range specifically for this demo is going to be 0.0.0.0 forward slash 0. and again this is specifically used for this demo and should never be used in a production-like environment in production you should only use the source ip ranges that you are communicating with and under protocols and ports because i need to log into the instance to be able to ping the other instance i'm going to have to open up tcp on port 22. under other protocols you can add icmp and this will allow the ping command to be used i'm going to leave all the other options as default and i'm going to click on create and now that this firewall rule has been created i need to go back over to project bowtie inc and create the firewall rule there as well i'm going to call this firewall rule bowtie inc dash b i'm going to give it the same description under network i'm going to select bow tie ink dash b i'm going to keep the priority as 1000 and the direction of traffic should be ingress as well the action on match should be allow scrolling down under targets i'm going to select all instances in the network and again under source filter i'm going to keep ip ranges selected and under source ip ranges i'm going to enter in 0.0.0.0 forward slash 0. and under protocols and ports i'm going to select tcp with port 22 as well under other protocols i'm going to type in icmp i'm going to leave everything else as default and i'm going to click on create now once you've created both networks and have created both firewall rules you can now start creating the instances so because i'm already in project bowtie inc i'm going to go to the left-hand navigation menu and i'm going to scroll down to compute engine and create my instance so i'm just going to click on create and to keep with the naming convention i'm going to call this instance instance b i'm not going to add any labels for now under region i'm going to choose us east 4 and you can leave the zone as the default selection and under machine type i'm going to select e2 micro and i'm going to scroll down to the bottom and i'm going to click on management security disks networking and sold tenancy so that i'm able to go into the networking tab to change the network on the default network interface so i'm going to click on the default network interface and under network i'm going to select bowtie inc b and the subnet has already been selected for me and then i'm going to scroll down click on done and i'm going to leave all the other options as default and click on create and so as this is creating i'm going to go over to project tony and i'm going to create my instance there and i'm going to name this instance instance a under region i am going to select us east1 you can leave the zone as the default selected under machine type i'm going to select e2 micro and scrolling down here to the bottom i'm going to go into the networking tab under management security disks networking and soul and here i'm going to edit the network interface and change it from the default network to bow tie ink dash a and as you can see the subnet has been automatically selected for me so now i can just simply click on done i'm going to leave all the other options as default and i'm going to click on create so just as a recap we've created two separate networks in two separate projects along with its corresponding subnets and the firewall rules along with creating an instance in each network and so now that we have both environments set up it's now time to create the vbc peering connection and so because i'm in project tony i'm going to start off with this project and i'm going to go up to the navigation menu and scroll down to vpc network and under vpc network on the left hand menu you're going to click on vpc network peering and through the interface shown here we'll be able to create our vpc network peering so now you're going to click on create connection and i'm prompted with some information that i will need and because we are connecting to another vpc in another project you're going to need the project id as well as the name of the vpc network you want to peer with and just as explained in the earlier lesson the subnet ip ranges in both networks cannot overlap so please make sure that if you are using ip ranges outside of the ones that are given for this demonstration the ip ranges that you are using do not overlap so once you have that information you can then click continue and so here you will be prompted with some fields to fill out with the information that you were asked to collect in the previous screen and so since we have that information already we can go ahead and start filling in the fields so i'm going to call this peering connection peering a b and under vpc network i'm going to select bow tie ink dash a under peered vpc network we're going to select the other project which should be bowtie inc and the vpc network name will be bow tie inc dash b and i'm going to leave all the other options as default and so under vpc network name you will see exchange custom routes and here i can select to import and export custom routes that i have previously created so any special routes that i have created before the actual peering connection i can bring them over to the other network to satisfy my requirements and so i'm not going to do that right now i'm going to close this up and i'm going to simply click on create and so this is finished creating and is marked as inactive and this is because the corresponding peering connection in project bowtie has yet to be configured the status will change to a green check mark in both networks and marked as active once they are connected if this status remains as inactive then you should recheck your configuration and edit it accordingly so now i'm going to head on over to project bowtie inc and i'm going to create the corresponding peering connection i'm going to click on create connection once you have your project id and the vpc network you can click on continue and for the name of this peering connection i'm going to call this peering dash ba respectively under vpc network i'm going to select bowtie inc b and under peered vpc network i'm going to select in another project here you want to type in your project id for me i'm going to paste in my project tony project id and under vpc network name i'm going to type in bowtie inc a and i'm going to leave all the other options as default and i'm going to click on create and so now that we've established connections on each of the peering connections in each vpc if the information that we've entered is correct then we should receive a green check mark stating that the peering connection is connected and success here we have status as active and if i head on over to project tony i should have the same green check mark under status for the peering connection and as expected the status has a green check mark and is marked as active so now in order to do the pairing connectivity test i'm going to need to grab the internal ip of the instance in the other network that resides in project bowtie and so because it doesn't matter which instance i log into as both of them have ssh and ping access i'm going to simply go over to the navigation menu i'm going to head on over to compute engine and i'm going to record the internal ip of instance a and now i'm going to head over to project bowtie and log into instance b and ping instance a and so in order to ssh into this instance i'm going to click on the ssh button under connect and it should open a new browser tab for me logging me into the instance okay i'm logged in here and i'm going to zoom in for better viewing and so now i'm going to run a ping command against instance a using the internal ip that i had copied earlier and i'm going to hit enter and as you can see ping is working and so now we can confirm that the vpc peering connection is established and the two instances in the different vpc networks are communicating over their private ips and you can go ahead and hit control c to stop the ping and so just as a recap you've created two separate vpc networks with their own separate subnets in two separate projects you've created the necessary firewall rules in each of these networks along with creating instances in each of those networks you then established a vpc peering connection establishing the configuration in each vpc you then did a connectivity test by logging into one of the instances and pinging the other instance and so i hope this helps cement the theory of vpc peering that you learned in the previous lesson and has given you some context when it comes to configuring each end of the peering connection so i wanted to take a moment to congratulate you on completing this demo and so all that's left now is to clean up all the resources that we created throughout this demo and you can start by selecting the instances and deleting them in each network as well as the firewall rules and the networks themselves i'm going to go over to project tony and i'm going to do the same thing there and so you can do exactly what you did with the last instance here you can select it click on delete and delete the instance and so next we're going to delete the peering connection so we're going to go up to the navigation menu we're going to scroll down to vpc network and on the left hand menu we're going to scroll down to vpc network peering and so we're going to select appearing connection we're going to go to the top and click on delete and then delete the peering connection and so now we're going to delete the firewall rule so we're going to go up to firewall we're going to select the firewall rule at the top we're going to click delete and then delete the firewall rule and last but not least we want to delete the vpc network that we created so we're going to go up to vpc networks we're going to drill down into the custom vpc up at the top we're going to click on delete vpc network and then we're going to click on delete and so now that we've deleted all the resources in project tony we're going to go back over to our second project project bowtie and do the same thing and so we're first going to start off with the vpc peering connection so we're going to go over to vpc network peering we're going to select the appearing connection we're gonna click on delete at the top and delete the peering connection next we're gonna go into firewall we're gonna select the firewall rule go up to the top and click on delete and then delete the firewall rule and finally we're gonna go over to vpc networks we're going to drill down into the custom network we're going to click on delete vpc network at the top and delete the vpc network and so now that you've successfully deleted all your resources you can now mark this lesson as complete and i'll see you in the next one and congrats again on the great job of completing this demo [Music] welcome back and in this lesson i'm going to be discussing the concepts and terminology of shared vpcs i'm also going to go into some detailed use cases and how shared vpcs would be used in different scenarios so with that being said let's dive in now when a vpc is created it is usually tied to a specific project now what happens when you want to share resources across different projects but still have separate billing and access within the projects themselves this is where shared vpcs come into play shared vpcs allow an organization to connect resources from multiple projects to a common vpc network so that way they can communicate with each other securely and efficiently using internal ips from that network when you use shared vpcs you designate a project as a host project and attach one or more other service projects to it the vpc networks in the host project are considered the shared vpc networks so just as a reminder a project that participates in a shared vpc is either a host project or a service project a host project can contain one or more shared vpc networks a service project is any project that has been attached to a host project by a shared vpc admin this attachment allows it to participate in the shared vpc and just as a note a project cannot be both a host and a service project simultaneously it has to be one or the other and you can create and use multiple host projects however each service project can only be attached to a single host project it is also a common practice to have multiple service projects administered by different departments or teams in the organization and so just for clarity for those who are wondering a project that does not participate in a shared vpc is called a standalone project and this is to emphasize that it is neither a host project or a service project now when it comes to administering these shared vpcs we should be adhering to the principle of least privilege and only assigning the necessary access needed to specific users so here i broken down the roles that are needed to enable and administer the shared vpcs a shared vpc admin has the permissions to enable host projects attach service projects to host projects and delegate access to some or all of the subnets in shared vpc networks to service project admins when it comes to a service project admin this is a shared vpc admin for a given host project and is typically its project owner as well although when defining each service project admin a shared vpc admin can grant permission to use the whole host project or just some subnets and so when it comes to service project admins there are two separate levels of permissions that can be applied the first is project level permissions and this is a service project admin that can be defined to have permission to use all subnets in the host project when it comes to subnet level permissions a service project admin can be granted a more restrictive set of permissions to use only some subnets now i wanted to move into some use cases which will give you a bit more context on how shared vpcs are used in specific environments illustrated here is a simple shared vpc scenario here a host project has been created and attached to service projects to it the service project admin in service project a can be configured to access all or some of the subnets in the shared vpc network service project admin with at least subnet level permissions to the 10.0.2.0 24 subnet has created vm1 in a zone located in the us west one region this instance receives its internal ip address 10.0.2.15 from the 10.0.2.0 24 cider block now service project admins in service project b can be configured to access all or some of the subnets in the shared vpc network a service project admin with at least subnet level permissions to the 10.10.4.0 forward slash 24 subnet has created vm2 in a zone located in the us central 1 region this instance receives its internal ip address 10.10.4.1 from the 10.10.4.0 forward slash 24 cider block and of course the standalone project does not participate in the shared vpc at all as it is neither a host nor a service project and the last thing to note instances in service projects attached to a host project using the same shared vpc network can communicate with one another using either ephemeral or reserve static internal ip addresses and i will be covering both ephemeral and static ip addresses in a later section under compute engine external ip addresses defined in the host project are only usable by resources in that project they are not available for use in service projects moving on to the next use case is a multiple hosts project for this use case an organization is using two separate host projects development and production and each host project has two service projects attached to them both host projects have one shared vpc network with subnets configured to use the same cider ranges both the testing and production networks have been purposely configured in the same way so this way when you work with resources tied to a subnet range it will automatically translate over from one environment to the other moving on to the next use case is the hybrid environment now in this use case the organization has a single host project with a single shared vpc network the shared vpc network is connected via cloud vpn to an on-premises network some services and applications are hosted in gcp while others are kept on premises and this way separate teams can manage each of their own service projects and each project has no permissions to the other service projects as well each service project can also be billed separately subnet level or project level permissions have been granted to the necessary service project admins so they can create instances that use the shared vpc network and again instances in these service projects can be configured to communicate with internal services such as database or directory servers located on premises and finally the last use case is a two-tier web service here an organization has a web service that is separated into two tiers and different teams manage each tier the tier one service project represents the externally facing component behind an http or https load balancer the tier 2 service project represents an internal service upon which tier 1 relies on and it is balanced using an internal tcp or udp load balancer the shared vpc allows mapping of each tier of the web service to different projects so that they can be managed by different teams while sharing a common vpc network to host resources that are needed for both tiers now we cover quite a bit in this lesson when it comes to all the concepts of shared vpcs we covered both host and service projects and the roles that they play and their limitations we also went over the different roles that are needed to administrate these shared vpcs and we went over different use cases on how to use shared vpcs for different scenarios and so that about covers everything i wanted to discuss in this lesson so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this lesson i'm going to be discussing vpc flow logs flow logs is an essential tool for monitoring and analyzing traffic coming in and going out of vpcs from vm instances flow logs are essential to know for the exam as you should know the capabilities and use cases and so with that being said let's dive in so vpc flow logs records a sample of network flows sent from and received by vm instances including instances used as google kubernetes engine nodes these logs can be used for network monitoring forensics real-time security analysis and expense optimization when you enable vpc flow logs you enable for all vms in a subnet so basically you would be enabling vpc flow logs on a subnet by subnet basis flow logs are aggregated by connection from compute engine vms and exported in real time these logs can be exported to cloud logging previously known as stackdriver for 30 days if logs need to be stored for longer than 30 days they can be exported to a cloud storage bucket for longer term storage and then read and queried by cloud logging google cloud samples packets that leave and enter a vm to generate flow logs now not every packet is captured into its own log record about one out of every 10 packets is captured but this sampling rate might be lower depending on the vm's load and just as a note you cannot adjust this rate this rate is locked by google cloud and cannot be changed in any way and because vpc flow logs do not capture every packet it compensates for missed packets by interpolating from the captured packets now there are many different use cases for vpc flow logs and i wanted to take a quick minute to go over them the first one i wanted to mention is network monitoring vpc flow logs provide you with real-time visibility into network throughput and performance so you can monitor the vpc network perform network diagnostics understand traffic changes and help forecast capacity for capacity planning you can also analyze network usage with vpc flow logs and you can analyze the network flows for traffic between regions and zones traffic to specific countries on the internet and based on the analysis you can optimize your network traffic expenses now a great use case for vpc flow logs is network forensics so for example if an incident occurs you can examine which ips talked with whom and when and you can also look at any compromised ips by analyzing all the incoming and outgoing network flows and lastly vpc flow logs can be used for real-time security analysis you can leverage the real-time streaming apis using pub sub and integrate them with a sim or security information in event management system like splunk rapid7 or logarithm and this is a very common way to add an extra layer of security to your currently existing environment as well as a great way to meet any compliance standards that are needed for your organization now vpc flow logs are recorded in a specific format log records contain base fields which are the core fields of every log record and meta data fields that add additional information metadata fields may be omitted to save storage costs but base fields are always included and cannot be omitted some log fields are in a multi-field format with more than one piece of data in a given field for example the connection field that you see from the base is of the ip details format which contains the source and destination ip address and port plus the protocol in a single field flows that have an endpoint in a gke cluster can be annotated with gke annotations which can include details of the cluster pod and service of the endpoint gke annotations are only available with a custom configuration of metadata fields now when you enable vpc flow logs you can set a filter based on both base and metadata fields that only preserves logs that match the filter all other logs are discarded before being written to logging which saves you money and reduces the time needed to find the information you're looking for shown here is a sample from the console in both the classic logs viewer as well as the logs viewer in preview and so in the classic logs viewer you can simply select the sub network from the first pull down menu and from the second pull down menu you can select the compute.googleapis.com forward slash vpc underscore flows and this will give you the information that you need to pull up all your vpc flow logs in the logs viewer preview it is done in a similar way but the query is shown here in the query builder and can be adjusted accordingly pulling up any vpc flow logs must be done within the console when viewing them in google cloud and so the last thing i wanted to show you before ending this lesson is a sample of the log itself the log shown here is a sample of what a vpc flow log looks like and as you can see here beside each field you will see a small arrow clicking on these arrows will expand the field and reveal many of the subfields that you saw on the last slide and will give you the necessary information you need to analyze your vpc flow logs in this example of the connection field it shows the five tuple that describes this connection which you can clearly see up here at the top and if i were to go further down and expand more of these fields i would find more information that could help me better analyze more logging info for my given problem that i am trying to solve now i didn't want to go too deep into logging as i will be diving into a complete section on its own in a later section of the course but i did want you to get a feel for what type of data vpc flow logs can give you and how it can help you in your specific use case as well as on the exam and so that's pretty much all i wanted to cover with regards to vpc flow logs so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in this lesson i'm going to cover a high-level overview of a basic foundational service that supports the backbone of the internet as we know it today this foundation is called dns or the domain name system dns is used widely in google cloud from mostly an infrastructure perspective and is used in pretty much any other cloud environment or computer network on the planet now there is quite a bit to cover in this lesson with regards to dns so with that being said let's dive in now dns or domain name system is a global decentralized distributed database that lets you store ip addresses and other data and look them up by name this system uses human readable names like google.com and translates it into a language that computers understand which are numeric ip addresses for example humans access information online through a domain name like google.com computers use ip addresses to access information online like 172.217. now whether you type google.com or the ip address into a web browser both will connect to google.com dns translates the domain name to an ip address so that the web browser knows where to connect to and we know what to enter into the web browser through dns you can connect a domain name to web hosting mail and other services now getting a bit deeper into it as ip addresses are at the core of communicating between devices on the internet they are hard to memorize and can change often even for the same service to get around these problems we gave names to ip addresses for example when it comes to our computer communicating with www.google.com it will use the dns system to do this now in the dns database contains the information needed to convert the www.google.com domain name to the ip address and this piece of information is stored in a logical container called a zone the way that the zone is stored is through what's commonly known as a zone file now within this zone file is a dns record which links the name www and the ip address that your laptop needs to communicate with the specific website and this zone file is hosted by what's known as a name server or ns server for short and i will be going into further detail on this in just a minute so in short if you can query the zone for the record www.google.com then your computer can communicate with the web server and dns is what makes it all happen now i wanted to go into a bit of history of how dns came about so in early computer networks a simple text file called a host file was created that mapped hostnames to ip addresses and this enabled people to refer to other computers by the name and their computers translated that name to an ip address when it needed to communicate with it the problem is as network sizes increased the host file approach became impractical due to the fact that it needed to be stored on each computer as each computer would have to resolve the same host names as well updates were difficult to manage as all of the computers would need to be given an updated file all in all this system was not scalable now to overcome these and other limitations the dns system was developed and the dns system essentially provided for a way to organize the names using a domain name structure it also provided a dynamic system for protocols services and methods for storing updating and retrieving ip addresses for host computers now that i've covered what dns is and why we use it i wanted to dive into the structure of the dns system now the structure all begins with a dot the root if you will and this can be found after every domain name that you type into your browser you will almost never see it and this is because your browser will automatically put it in without your knowing you can try it with any domain in any browser and you will almost always come up with the same result this dot is put in for you and will provide the route for you and this is where we start to break down the dns system now the domain name space consists of a hierarchical data structure like the one you have on your computer each node has a label and zero or more resource records which hold information associated with the domain name the domain name itself consists of the label concatenated with the name of its parent node on the right separated by a dot so when it comes to dns the domain name is always assembled from right to left this hierarchy or tree is subdivided into zones beginning at the root zone a dns zone may consist of only one domain or may consist of many domains and sub domains depending on the administrative choices of the zone manager now getting right into it the root server is the first step in translating human readable hostnames into ip addresses the root domain is comprised of 13 dns systems dispersed around the world known collectively as the dns root servers they are indicated by the letters a through m operated by 12 organizations such as verisign cogent and nasa while there are 13 ip addresses that represent these systems there are actually more than 13 servers some of the ip addresses are actually a cluster of dns servers and so each of these dns servers also consists of the root zone file which contains the address of the authoritative name server for each top level domain and because this is such a big undertaking to keep updated iana or the internet assigned numbers authority was appointed as the authority that manages and administrates this file and i will include a link in the lesson text for those of you who are looking to dive deeper into the contents of this root zone file as well as getting to know a little bit more about the iana organization now while the dns root servers establish the hierarchy most of the name resolution process is delegated to other dns servers so just below the dns route in the hierarchy are the top level domain servers also known as tld for short the top level domain takes the tld provided in the user's query for example www.google and provides details for the dot-com tld name server the companies that administer these domains are named registries and they operate the authoritative name servers for these top level domains for example verisign is the registry for the dot com top level domain over a hundred million domains have been registered in the dot com top level domain and these top level dns servers handle top level domains such as com dot org dot net and dot io and this can also be referred to as the gtld which is the general top level domains and the cctld which is the country code top level domain like dot ca for canada dot uk for the united kingdom and dot it for italy the top level dns servers delegate to thousands of second level dns servers now second level domain names are sold to companies and other organizations and over 900 accredited registrars register and manage the second level domains in the dot com domain for end users the second level of this structure is comprised of millions of domain names second level dns servers can further delegate the zone but most commonly store the individual host records for a domain name this is the server at the bottom of the dns lookup chain where you would typically find resource records and it is these resource records that maps services and host names to ip addresses and will respond with the queried resource record ultimately allowing the web browser making the request to reach the ip address needed to access a website or other web resources now there is one more concept that i wanted to cover before we move on and this is the sub domain now some of you have noticed and wondered where does the sub domain come into play with regards to the dns structure well this is a resource record that falls under the second level domain and in dns hierarchy a sub domain is a domain that is a part of another main domain but i wanted to put it in here just to give you an understanding of where subdomains would fall so now that we understand how dns is structured i wanted to go through the breakdown of the data flow of dns to give you some better contacts now there are eight steps in a dns lookup first we start off with the dns client which is shown here as tony bowtie's laptop and this is a client device which could also be a phone or a tablet and is configured with software to send name resolution queries to a dns server so when a client needs to resolve a remote host name into its ip address in most cases it sends a request to the dns recursive resolver which returns the ip address of the remote host to the client a recursive resolver is a dns server that is configured to query other dns servers until it finds the answer to the question it will either return the answer or an error message to the client if it cannot answer the query and the query will eventually be passed off to the dns client the recursive resolver in essence acts as the middle man between a client and a dns name server which is usually the internet service provider a service carrier or a corporate network now to make sure that a resolver is able to properly run dns a root hints file is supplied with almost every operating system and this file holds the ip addresses for the root name servers this also includes the dns resolver but in case it is unable to answer the query the client will be able to still make the query to the dns name servers now after receiving a dns query from a client this recursive resolver will either respond with cache data or send a request to a root name server and in this case the resolver queries a dns root name server the root server then responds to the resolver with the address of a top level domain or tld dns server such as com or dot net which stores the information for its domains now when searching for google.com the request is pointed towards the dot-com tld so naturally the resolver then makes a request to the com tld then the tld name server then responds with the ip address of the domain's name server google.com and lastly the resolver then sends a query to the domain's name server the ip address for google.com is then returned to the resolver from the name server this ip address is cache for a period of time determined by the google.com name server and this process is so that a future request for this hostname could be resolved from its cache rather than performing the entire process from beginning to end and so for those of you who are unaware cache is a component that stores data so that future requests for that data can be served faster the purpose of this caching is to temporarily store data in a location that results in improvements in performance and reliability for data requests dns caching involves storing the data closer to the requesting client so that the dns query can be resolved earlier and additional queries further down the dns lookup chain can be avoided and thus improving load times dns data can be cached in a variety of locations down the chain each of which will store dns records for a set amount of time determined by a time to live also known as ttl for short and this value is the time to live for that domain record a high ttl for a domain record means that local dns resolvers will cache responses for longer and give quicker responses however making changes to dns records can take longer due to the need to wait for all cash records to expire alternatively domain records with low ttls can change much more quickly but dns resolvers will need to refresh their records more often and so in this final step the dns resolver then responds to the web browser with the ip address of the domain requested initially and once these eight steps of the dns lookup have returned the ip address for www.google.com the browser is able to make the request for the webpage and so the browser will reach out to the ip address of the server and request the web page which will be loaded up in the browser now i know this probably has been a review for those who are a bit more advanced when it comes to understanding dns but for others who are fairly new to the underpinnings of dns i hope this has given you a basic understanding of what it is why we use it and how it works moving forward in the course i will be discussing dns with regards to different services and the needed resource records within zones that are used by these given services and so that's pretty much all i wanted to cover when it comes to the fundamentals of dns so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in this lesson i'm going to be diving into dns record types now dns resource records are the basic information elements of the domain name system they are entries in the dns database which provide information about hosts these records are physically stored in the zone files on the dns server this lesson will go through some of the most commonly used dns records that we will be coming across throughout this course so with that being said let's dive in now the first record that i wanted to touch on are the name server records also known as ns records for short this record identifies which dns server contains the current records for a domain these servers are usually found at a registrar internet service provider or hosting company ns records are created to identify the name server used for each domain name within a given zone in this example we have the dot co zone that will have multiple name server records for bowtieinc.co now these name server records are how the dot co delegation happens for bowtieinc.co and they point at servers that host the inc.co zone that is managed by bowtie inc and the flow shown here of the query starts from the root zone going to the dot co zone where the record lies for the name servers for bowtieinc.com and flows down to the bowtieinc.cozone that contain all the necessary records for bowtieinc.co the next record that i wanted to touch on are the a and aaa records and this is short for address records for ipv4 and ipv6 ip addresses respectively and this record points a domain name to an ip address for example when you type wwe in a web browser the dns system will translate that domain name to the ip address of 52.54.92.195 using the a record information stored in the bowtieinc.co dns zone file the a record links a website's domain name to an ipv4 address that points to the server where the website's files live now when it comes to an aaa record this links a website's domain to an ipv6 address that points to the same server where the website's files live a records are the simplest type of dns records and one of the primary records used in dns servers you can do a lot with a records including using multiple a records for the same domain in order to provide redundancy the same can be said for aaa records additionally multiple domains could point to the same address in which case each would have its own a or aaa record pointing to that same ip address moving on to cname records a c name record short for canonical name record is a type of resource record that maps one domain name to another this can be really convenient when running multiple services like an ftp server and an e-commerce server each running on different ports from a single ip address you can for example point ftp ftp.bowtieinc.co and shop.bowtieinc.co to the dns entry for bowtieinc.co which in turn has an a record which points to the ip address so if the ip address ever changes you only have to change the record in one place in the dns a record for bow tie inc dot co and just as a note cname records must always point to another domain name and never directly to an ip address next up are txt records a text record or txt for short is a type of resource record that provides text information to sources outside your domain that can be used for a number of arbitrary purposes the records value can be either human or machine readable text in many cases text records are used to verify domain ownership or even to provide human readable information about a server a network or a data center it is also often used in a more structured fashion to record small amounts of machine readable data into the dns system a domain may have multiple tax records associated with it provided the dns server implementation supports this each record can in turn have one or more character strings in this example google wants to verify the bowtieinc.co domain so that g suite can be set up and needs verification through the domain to google through creating a text record and adding it to the zone google will then supply a text verification record to add to the domain host's dns records and start to scan for the text record to verify the domain the supplied text record is then added by the domain administrator and behind the scenes google is doing a verification check at timed intervals when google finally sees the record exists the domain ownership is confirmed and g suite can be enabled for the domain and this is a typical example of how tax records are used now moving on to mx records a dns mx record also known as the mail exchange record is the resource record that directs email to a mail server the mx record indicates how email messages should be routed and to which server mail should go to like cname records an mx record must always point to another domain now mx records consist of two parts the priority and the domain name the priority are the numbers before the domains for these mx records and indicate the preference of the order in which the mail server should be used the lower the preference number the higher the priority so in this example laura is emailing tony bowtie at tony at bowtieinc.co the mx records are part of this process as dns needs to know where to send the mail to and we'll look at the domain attached to the email address which is bowtieinc.co so the dns client will run a regular dns query by first going to the root then to the cotld and finally to bowtieinc.co it will then receive the mx record which in this example is two of them the first one being mail representing mail.bowtieinc.co and then the second one is a different mail server outside the current domain and in this case is a google mail server of aspmx.l.google.com and this is a fully qualified domain name as the dot on the right of this record suggests so here the server will always try mail.bowtieinc.co first because 5 is lower than 10. and this will give mail.bowtieinc.co the higher priority in the result of a message send failure the server will default to aspmx.l.google.com if both values are the same then it would be low balanced across both servers whichever is used the server gets the result of the query back and it uses this to connect to the mail server for bowtieinc.co via the smtp protocol and it uses this protocol to deliver all email and this is how mx records are used for email the next record i wanted to cover are the pointer records also known as ptr records for short and this provides the domain name associated with an ip address so a dns pointer record is exactly the opposite of the a record which provides the ip address associated with the domain name dns pointer records are used in reverse dns lookups as we discussed earlier when a user attempts to reach a domain name in their browser a dns lookup occurs matching the domain name to the ip address a reverse dns lookup is the opposite of this process and it is a query that starts with the ip address and looks up the domain name while dnsa records are stored under the given domain name dns pointer records are stored under the ip address reverse and ending in dot i n a d d r dot arpa so in this example the pointer record for the iap address 52.54.90 would be stored under 195.92.54.52 dot in addr dot arpa ipv6 addresses are constructed differently from ipv4 addresses and ipv6 pointer records exist in a different namespace within.arpa ipv6 pointer records are stored under the ipv6 address reversed and converted into 4-bit sections as opposed to 8-bit sections as in ipv4 and as well the domain.ip6.arpa is added at the end pointer records are used most commonly in reverse dns lookups for anti-spam troubleshooting email delivery issues and logging and so the last record that i wanted to cover are the soa records also known as the start of authority records and this resource record is created for you when you create your managed zone and specifies the authoritative information including global parameters about a dns zone the soa record stores important information about a domain or zone such as the email address of the administrator when the domain was last updated and how long the server should wait between refreshes every dns zone registered must have an soa record as per the rfc 1035 and there is exactly one soa record per zone the soa record contains the core information about your zone so it is not possible for your zone to work without that information and i will include a link in the lesson text for those who are interested in diving deeper and understanding all the information that is covered under these soa records a properly optimized and updated soa record can reduce bandwidth between name servers increase the speed of website access and ensure the site is alive even when the primary dns server is down and so that about covers everything that i wanted to discuss when it comes to resource records within dns so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in this lesson i'm going to be covering network address translation also known as nat for short this is a common process used in home business and any cloud networks that you will encounter knowing and understanding that will help you achieve why you would use it and what makes it such a necessary process now there's quite a bit to cover here so with that being said let's dive in now at a high level nat is a way to map multiple local private ip addresses to a public ip address before transferring the information this is done by altering the network address data in the ip header of the data packet while traveling through a network towards the destination as packets pass through a nat device either the source or destination ip address is changed then packets returning in the other direction are translated back to the original addresses and this is a process that is typically used in most home routers that are provided by your internet service provider now originally nat was designed to deal with the scarcity of free ipv4 addresses increasing the number of computers that can operate off a single publicly routable ip address and so because devices in the private ip space such as 192.168.0.0 cannot traverse the public internet that is needed for those devices to communicate with the public internet now ipv6 was designed to overcome the ipv4 shortage and has tons of available addresses and therefore there is no real need for nat when it comes to ipv6 now nat has an additional benefit of adding a layer of security and privacy by hiding the ip address of your devices from the outside world and only allowing packets to be sent and received from the originating private device and so this is a high level of what nat is now there are multiple types of not that i will be covering which at a high level do the same thing which is translate private i p addresses to public ip addresses yet different types of nat handles the process differently so first we have static nat which maps a single private ip address to a public ip address so a one-to-one mapping that gives the device with the private ip address access to the public internet in both directions this is commonly used where one specific device with a private address needs access to the public internet the next type of nat is dynamic nan and this is similar to static nat but doesn't hold the same static allocation a private ip address space is mapped to a pool of public ip addresses and are allocated randomly as needed when the ip address is no longer needed the ip address is returned back to the pool ready to be used by another device this method is commonly used where multiple internal hosts with private i p addresses are sharing an equal or fewer amount of public i p addresses and is designed to be an efficient use of public ips and finally there is port address translation or pat where multiple private ip addresses are translated using a single public ip address and a specific port and this is probably what your home router is using and will cover all the devices you use in your home network this method uses ports to help distinguish individual devices and is also the method that is used for cloudnat in google cloud which i will be covering in a later lesson and so i wanted to get into a bit more detail on how these methods work starting with static not now to set the stage for static not i'm going to start off with a private network here on the left and the public ip space here on the right and the router or not device in the middle in this example there is a server on the left that needs access to external services and for this example the external service we are using is the bowtress service an image sharing site for all sorts of awesome bow ties so the server on the left is private with a private ip address of 192.168.0.5 and this means it has an address in the ip version 4 private address space meaning that it cannot route packets over the public internet because it only has a private ip the beautress service on the other hand has a public ip address which is 54.5.4.9 so the issue we run into is that the private address can't be routed over the public internet because it's private and the public address of the beau trust service can't directly communicate with any private address because public and private addresses can communicate over the public internet what we need is to translate the private address that the server on the left has to a public ip that can communicate with the service on the right and vice versa now then that device will map the private ip to public ip using and maintaining a nat table and in this case of static nat the nat device will have a one-to-one mapping of the private ip address to a public ip address and can be allocated to the device specified which in this case is the server marked as 192.168.0.15 and so in order for the server on the left to communicate with the beautress service the server will generate a packet as normal with the source ip of the packet being the server's private ip address and the destination ip of the packet being the ip of the bowtrust service now the router in the middle is the default gateway for any destination so any ip packets which are destined for anything but the local network are sent to the router so as you can see here with the entry in the table it will contain the private i p address of 192.168.0.15 and mapped to the public address which in this case is 73.6.2.33 and these are statically mapped to one another and so as the packet passes through the nat device the source address of the packet is translated from the private address to the mapped public address and this results in a new packet so this new packet still has beautrest as the destination but now it has a valid public ip address as the source and so this is the translation that happens through nat now this process works in a similar way in the other direction so when the beautress service receives the packet it sees the source as this public ip so when it responds with data its packet has its ip address as the source and the previous server's public ip address as the destination so it sends this packet back to this public ip so when the packet arrives at the nat device the table is checked it recognizes then that the ip is for the server and so this time for incoming traffic the destination ip address is updated to the corresponding private ip address and then the packet is forwarded through to the private server and this is how static nat works the source i p address is translated from the mapped private ip to public ip and for incoming traffic the destination i p address is translated from the allocated public ip to the corresponding private ip all without having to configure a public ip on any private device as they always hold their private ip addresses now i wanted to supply an analogy for nat and so a very common analogy that is used is that of a phone service so in this example laura is the new manager of bow tie inc new location in montreal and has put in a new public phone number of 514-555-8437 although as you can see here laura also has a private extension of one three three seven now if george called laura at that public phone number he would reach laura without ever knowing her private extension so the private extension acts as that private ip address and the public phone number would act as the public ip address and this would be the telephone analogy for static nat and so this is the end of part one of this lesson it was getting a bit long so i decided to break it up this would be a great opportunity for you to get up and have a stretch get yourself a coffee or a tea and whenever you're ready you can join me in part two where we will be starting immediately from the end of part one so you can go ahead and complete this video and i will see you in part two [Music] welcome back this is part two of the network address translation lesson and we will be starting exactly where we left off from part 1. so with that being said let's dive in now moving on to dynamic nat this method is similar to static nat except that devices are not allocated a permanent public ip a public ip address is allocated from a pool of ip addresses as they are needed and the mapping of public to private is allocation base in this example there are two devices on the left and according to the nat table there are two public ip addresses available for use 73.6.2.33 and 73.6.2.34 so when the laptop on the left is looking to access the beautress service it will generate a packet where the source ip is the private address of 192.168.0.13 and the destination ip is 54.5.4.9 so it sends this packet and again the router in the middle is the default gateway for anything that isn't local as the packet passes through the router or the nat device it checks if the private ip has a current allocation of public addressing from the pool and if it doesn't and one is available it allocates one dynamically and in this case 73.6.2.34 is allocated so the packet's source i p address is translated to this address and the packets are sent to the beautress service and so this process is the same as static not thus far but because dynamic nat allocates these ip addresses dynamically multiple private devices can share a single public ip as long as the devices are not using the same public ip at the same time and so once the device is finished communication the ip is returned back to the pool and is ready for use by another device now just as a note if there's no public ip addresses available the router rejects any new connections until you clear the nat mappings but if you have as many public ip addresses as hosts in your network you won't encounter this problem and so in this case since the lower server is looking to access the fashion tube service there is an available public ip address in the pool of 73.6.2.33 thus giving it access to the public internet and access to fashion tube so in summary the nat device maps a private ip with the public ip in a nat table and public ips are allocated randomly and dynamically from a pool now this type of knot is used where multiple internal hosts with private ip addresses are sharing an equal or fewer amount of public ip addresses when all of those private devices at some time will need public access now an example of dynamic nat using the telephone analogy would be if laura and two other bow tie inc employees lisa and jane had private phone numbers and this would represent your private ips in this example bowtie inc has three public phone numbers now when any employee makes an outbound call they are routed to whichever public line is open at the time so the caller id on the receiver's end would show any one of the three public phone numbers depending on which one was given to the caller and this would represent the public ips in the public ip pool now the last type of nat which i wanted to talk about is the one which you're probably most familiar with and this is port address translation which is also known as not overload and this is the type of not you likely use on your home network port address translation is what allows a large number of private devices to share one public ip address giving it a many to one mapping architecture now in this example we'll be using three private devices on the left all wanting to access fashiontube on the right a popular video sharing website of the latest men's fashions shared by millions across the globe and this site has a public ip of 62.88.44.88 and accessed using tcp port 443 now the way that port address translation or pat works is to use both the ip addresses and ports to allow for multiple devices to share the same public ip every tcp connection in addition to a source and destination ip address has a source and destination port the source port is randomly assigned by the client so as long as the source port is always unique then many private clients can use the same public ip address and all this information is recorded in the nat table on the nat device in this example let's assume that the public ip address of this nat device is 73.6.2.33 so when the laptop in the top left generates a packet and the packet is going to fashion tube its destination ip address is 62.80 and its destination port is 443. now the source ip of this packet is the laptop's private ip address of 192.168.6 and the source port is 35535 which is a randomly assigned ephemeral port so the packet is routed through the nat device and in transit the nat device records the source ip and the original source private port and it allocates a new public ip address and a new public source port which in this case is 8844 it records this information inside the not table as shown here and it adjusts the pocket so that its source ip address is the public ip address that the nat device is using and the source port is this newly allocated source port and this newly adjusted packet is forwarded on to fashiontube now the process is very similar with the return traffic where the packet will verify the recorded ips and ports in the nat table before forwarding the packet back to the originating source now if the middle laptop with the ip of 192.168.0.14 did the same thing then the same process would be followed all of this information would be recorded in the nat table a new public source port would be allocated and would translate the packet adjusting the packet's source ip address and source port as well the same process would happen for the laptop on the bottom generating a packet with the source and destination ip with the addition of the source and destination ports and when routed through the nat device goes through its translation recording the information in the nat table and reaching its destination again return traffic will be verified by the recorded ips and ports in the nat table before forwarding the packet back to its originating source and so just as a summary when it comes to port address translation the nat device records the source ip and source port in a nat table the source ip is then replaced with a public ip and public source port and are allocated from a pool that allows overloading and this is a many-to-one architecture and so for the telephone analogy for pat let's use a phone operator example so in this instance george is trying to call laura now george only knows lark laura's executive admin and only has lark's phone number george does not have laura's private line lark's public phone number is the equivalent to having a public ip address george calls lark who then connects george to laura the caveat here is that lark never gives out laura's phone number in fact laura doesn't have a public phone number and can only be called by lark and here's where nat can add an extra layer of security by only allowing needed ports to be accessed without allowing anyone to connect to any port now i hope this has helped you understand the process of network address translation how the translation happens and the process of using a nat table to achieve packet translation along with its destination this is so common in most environments that you will encounter and it's very important to fully understand the different types of not and how it can be used in these types of environments and so that's pretty much all i wanted to cover on this lesson of network address translation so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back so now that we've covered the fundamentals of dns along with the different record types i wanted to focus in on google cloud's dns service called cloud dns now cloud dns is a fully managed service that manages dns servers for your specific zones and since cloud dns shows up on the exam only on a high level i will be giving an overview of what this service can do so with that being said let's dive in now cloud dns acts as an authoritative dns server for public zones that are visible to the internet or for private zones that are visible only within your network and is commonly referred to as google's dns as a service cloud dns has servers that span the globe making it a globally resilient service now while it is a global service there is no way to select specific regions to deploy your zones and dns server policies you simply add your zones records and policies and it is distributed amongst google's dns servers across the globe cloud dns is also one of the few google cloud services that offers 100 availability along with low latency access by leveraging google's massive global network backbone now in order to use cloud dns with a specific publicly available domain a domain name must be purchased through a domain name registrar and you can register a domain name through google domains or another domain registrar of your choice cloud dns does not provide this service and just as a note that to create private zones the purchasing of a domain name is not necessary now as stated earlier cloud dns offers the flexibility of hosting both public zones and privately managed dns zones now public zones are zones that are visible to the public internet and so when cloud dns is managing your public domain it has public authoritative name servers that respond to public zone dns queries for your specific domain now when it comes to private zones these enable you to manage custom domain names for your google cloud resources without exposing any dns data to the public internet a private zone can only be queried by resources in the same project where it is defined and as we discussed earlier a zone is a container of dns records that are queried by dns so from a private zone perspective these can only be queried by one or more vpc networks that you authorize to do so and just as a note the vpc networks that you authorize must be located in the same project as the private zone to query records hosted in manage private zones in other projects the use of dns peering is needed now i don't want to get too deep into dns peering but just know that vpc network peering is not required for the cloud dns peering zone to operate peering zones do not depend on vpc network peering now each managed zone that you create is associated with a google cloud project and once this zone is created it is hosted by google's managed name servers now these zones are always hosted on google's manage name servers within google cloud so you would create records and record sets and these servers would then become allocated to that specific zone hosting your records and record sets and just as a quick reminder a record set is the collection of dns records in a zone that have the same name and are of the same type most records contain a single record but it's not uncommon to see record sets a great example of this are a records or ns records which we discussed earlier and these records can usually be found in pairs and so now to give you a practical example of cloud dns i wanted to bring the theory into practice through a short demo where i'll be creating a managed private zone so whenever you're ready join me in the console and so here we are back in the console and i'm logged in as tonybowties gmail.com and i'm currently in project bowtie inc so now to get to cloud dns i'm going to go over to the navigation menu i'm going to scroll down to network services and go over to cloud dns and because i currently don't have any zones i'm prompted with only one option which is to create a zone and so i'm going to go ahead and create a zone and so here i've been prompted with a bunch of different options in order to create my dns zone and so the first option that i have is zone type and because i'm creating a private zone i'm going to simply click on private and i need to provide a zone name which i'm going to call tony bowtie next i'm going to have to provide a dns name which i will call tony bowtie dot private and under the description i'm just going to type in private zone for tony bowtie and so the next field i've been given is the options field where it is currently marked as default private and so if i go over here to the right hand side and open up the drop down menu i'm given the options to forward queries to another server dns peering manage reverse lookup zones and use a service directory namespace and so depending on your type of scenario one of these five options in most cases will suffice so i'm going to keep it under default private and under networks it says your private zone will be visible to the selected networks and so i'm going to click on the drop down and i'm giving only the option of the default network because it's the only network that i have and so i'm going to select it and i'm going to click on the white space and if i feel so inclined i can simply click on the shortcut for the command line and here i'm given this specific commands if i was to use the command line in order to create this dns zone so i'm going to click on close here and i'm going to click on create and as you can see here my zone has been created along with a couple of dns records the first one being my name server records as well as my start of authority records and so as a note to know for the exam when creating a zone these two records will always be created both the soa record and the ns record and moving on to some other options here i can add another record set if i choose to again the dns name the record type which i have a whole slew of record types to choose from it's ttl and the ip address but i'm not going to add any records so i'm just going to cancel and by clicking in use by i can view which vpc network is using this zone and as expected the default network shows up and i also have the choice of adding another network but since i don't have any other networks i can't add anything so i'm going to simply cancel i also have the option of removing any networks so if i click on this i can remove the network or i can also remove the network by clicking on the hamburger menu and so as you can see i have a slew of options to choose from when creating zones and record sets and so that about covers everything that i wanted to show you here in cloud dns but before i go i'm going to go ahead and clean up and i'm just going to click on the garbage can here on the right hand side of the zone and i'm going to be prompted if i want to delete the zone yes i do so i'm going to click on delete and so that pretty much covers everything that i wanted to show you with regards to cloud dns so you can now mark this lesson as complete and let's move on to the next one welcome back now before we step into the compute engine section of the course i wanted to cover a basic foundation of what makes these vms possible and this is where a basic understanding of virtualization comes into play now this is merely an introductory lesson to virtualization and i won't be getting too deep into the underpinnings it serves as just a basic foundation as to how compute engine gets its features under the hood and how they are possible through the use of virtualization for more in-depth understanding on virtualization i will be including some links in the lesson text for those who are looking to learn more but for now this will provide just enough theory to help you understand how compute engine works so with that being said let's dive in so what exactly is virtualization well virtualization is the process of running multiple operating systems on a server simultaneously now before virtualization became popular a standard model was used where an operating system would be installed on a server so the server would consist of typical hardware like cpu memory network cards and other devices such as video cards usb devices and storage and then the operating system would run on top of the hardware now there is a middle layer of the operating system a supervisor if you will that is responsible for interacting with underlying hardware and this is known as the kernel the kernel manages the distribution of the hardware resources of the computer efficiently and fairly among all the various processes running on the computer now the kernel operates under what is called kernel mode or privilege mode as it runs privileged instructions that interacts with the hardware directly now the operating system allows other software to run on top of it like an application but cannot interact directly with the hardware it must interact with the operating system in user mode or non-privileged mode so when lark decides to do something on an application that needs to use the system hardware that application needs to go through the operating system it needs to make what's known as a system call and this is the model of running one operating system on a single server now when passed servers would traditionally run one application on one server with one operating system in the old system the number of servers would continue to mount since every new application required its own server and its own operating system as a result expensive hardware resources were purchased but not used and each server would use approximately under 20 of its resources on average server resources were then known as underutilized now there came a time when multiple operating systems were installed on one computer isolated from each other with each operating system running their own applications this was a perfect model to consolidate hardware and keep utilization high but there is a major issue that arose each cpu at this given moment in time could only have one thing running as privileged so having multiple operating systems running on their own in an unmodified state and expecting to be running on their own in a privileged state running privileged instructions was causing instability in systems causing not just application crashes but system crashes now a hypervisor is what solved this problem it is a small software layer that enables multiple operating systems to run alongside each other sharing the same physical computing resources these operating systems come as virtual machines or vms and these are files that mimic an entire computing hardware environment in software the hypervisor also known as a virtual machine monitor or vmm manages these vms as they run alongside each other it separates virtual machines from each other logically assigning each its own slice of the underlying computing cpu memory and other devices like graphics network and storage this prevents the vms from interfering with each other so if for example one operating system suffers a crash or a security compromise the others will survive and continue running now the hypervisor was never as efficient as how you see it here it went through some major iterations that gave its structure as we know it today initially virtualization had to be done in software or what we now refer to as the host machine and the operating system with its applications put in logical containers known as virtual machines or guests the operating system would be installed on the host which included additional capabilities called a hypervisor and allowed it to make the necessary privileged calls to the hardware having full access to the host the hypervisor exposed the interface of the hardware device that is available on the host and allowed it to be mapped to the virtual machine and emulated the behavior of this device and this allowed the virtual machine using the operating system drivers that were designed to interact with the emulated device without installing any special drivers or tools as well as keeping the operating system unmodified the problem here is that it was all emulated and so every time the virtual machines made calls back to the host each instruction needed to be translated by the hypervisor using what's called a binary translation now without this translation the emulation wouldn't work and would cause system crashes bringing down all virtual machines in the process now the problem with this process is that it made the system painfully slow and it was this performance penalty that caused this process to not be so widely adopted but then another type of virtualization came on the scene called para virtualization now in this model a modified guest operating system is able to speak directly to the hypervisor and this involves having the operating system kernel to be modified and recompiled before installation onto the virtual machine this would allow the operating system to talk directly with the hypervisor without any performance hits as there is no translation going on like an emulation para virtualization replaces instructions that cannot be virtualized with hyper calls that communicate directly with the hypervisor so a hypercall is based on the same concept as a system call privileged instructions that accept instead of calling the kernel directly it calls the hypervisor and due to the modification in this guest operating system performance is enhanced as the modified guest operating system communicates directly with the hypervisor and emulation overhead is removed the guest operating system becomes almost virtualization aware yet there is still a process whereby software was used to speak to the hardware the virtual machines could still not access the hardware directly although things changed in the world of virtualization when the physical hardware on the host became virtualization aware and this is where hardware assisted virtualization came into play now hardware assisted virtualization is an approach that enables efficient full virtualization using help from hardware capabilities on the host cpu using this model the operating system has direct access to resources without any hypervisor emulation or operating system modification the hardware itself becomes virtualization aware the cpu contains specific instructions and capabilities so that the hypervisor can directly control and configure this support it also provides improved performance because the privileged instructions from the virtual machines are now trapped and emulated in the hardware directly this means that the operating system kernels no longer need to be modified and recompiled like in para virtualization and can run as is at the same time the hypervisor also does not need to be involved in the extremely slow process of binary translation now there is one more iteration that i wanted to discuss when it comes to virtualization and that is kernel level virtualization now instead of using a hypervisor kernel level virtualization runs a separate version of the linux kernel and sees the associated virtual machine as a user space process on the physical host this makes it easy to run multiple virtual machines on a single host a device driver is used for communication between the main linux kernel and the virtual machine every vm is implemented as a regular linux process scheduled by the standard linux scheduler with dedicated virtual hardware like a network card graphics adapter cpu memory and disk hardware support by the cpu is required for virtualization a slightly modified emulation process is used as the display and execution containers for the virtual machines in many ways kernel level virtualization is a specialized form of server virtualization and this is the type of virtualization platform that is used in all of google cloud now with this type of virtualization because of the kernel acting as the hypervisor it enables a specific feature called nested virtualization now with nested virtualization it is made possible to install a hypervisor on top of the already running virtual machine and so this is what google cloud has done now you're probably wondering after going through all the complexities involved with previous virtualization models what makes this scenario worthwhile well using nested virtualization it makes it easier for users to move their on-premises virtualized workloads to the cloud without having to import and convert vm images so in essence it eases the use when migrating to cloud a great use case for many but wouldn't be possible on google cloud without the benefit of running kernel level virtualization now this is an advanced concept that does not show up on the exam but i wanted you to understand virtualization at a high level so that you can understand nested virtualization within google cloud as it is a part of the feature set of compute engine and so that's pretty much all i wanted to cover when it comes to virtualization so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back now earlier on in the course i discussed compute engine at a high level to understand what it is and what it does the goal for this section is to dive deeper into compute engine as it comes up heavily on the exam and so i want to make sure i expose all the nuances as well it is the go-to service offering from google cloud when looking to solve any general computing needs with this lesson specifically i will be going into what makes up an instance and the different options that are available when creating the instance so with that being said let's dive in now compute engine lets you create and run virtual machines known as instances and host them on google's infrastructure compute engine is google's infrastructure as a service virtual machine offering so it being an is service google takes care of the virtualization platform the physical servers the network and storage along with managing the data center and these instances are available in different sizes depending on how much cpu and memory you might need as well compute engine offers different family types for the type of workload you need it for each instance is charged by the second after the first minute as this is a consumption based model and as well these instances are launched in a vpc network in a specific zone and these instances will actually sit on hosts in these zones and you will be given the option of using a multi-tenant host where the server that is hosting your machine is shared with others but please note that each instance is completely isolated from the other so no one can see each other's instances now you're also given the option of running your instance on a sole tenant node whereby your instance is on its own dedicated hosts that is reserved just for you and you alone you don't share it with anyone else and this is strictly for you only now although this option may sound really great it does come at a steep cost so only if your use case requires you to use a sole tenant node for security or compliance purposes i recommend that you stick with a multi-tenant host when launching your instances and this is usually the most common selection for most now compute engine instances can be configured in many different ways and allow you the flexibility to fulfill the requests for your specific scenario and as you can see here there are four different base options when it comes to configuration of the instance that you are preparing to launch and so i wanted to take time to go through them in just a bit of detail for context starting first with the machine type which covers vcpu and memory now there are many different predefined machine types that i will be covering in great depth in a different lesson but for now just know that they are available in different families depending on your needs and can be chosen from the general compute optimize and memory optimize machine types they are available in intel or amd flavors and if the pre-defined options doesn't fit your need you have the option of creating a custom machine that will suit your specific workload now when creating a vm instance on compute engine each virtual cpu or vcpu is implemented as a single hardware hyper thread on one of the available cpu processors that live on the host now when choosing the amount of vcpus on an instance you must take into consideration the desired network throughput as the amount of vcpus will determine this throughput as the bandwidth is determined per vm instance not per network interface or per ip address and so the network throughput is determined by calculating 2 gigabits per second for every vcpu on your instance so if you're looking for greater network throughput then you may want to select an instance with more vcpus and so once you've determined a machine type for your compute engine instance you will need to provide it an image with an operating system to boot up with now when creating your vm instances you must use an operating system image to create boot disks for your instances now compute engine offers many pre-configured public images that have compatible linux or windows operating systems and these operating system images can be used to create and start instances compute engine uses your selected image to create a persistent boot disk for each instance by default the boot disk for your instance is the same size as the image that you selected and you can use most public images at no additional cost but please be aware that there are some premium images that do add additional cost to your instances now moving on to custom images this is a boot disk image that you own and control access to a private image if you will custom images are available only to your cloud project unless you specifically decide to share them with another project or another organization you can create a custom image from boot disks or other images then use the custom image to create an instance custom images that you import to compute engine add no cost to your instances but do incur an image storage charge while you keep your custom image in your project now the third option that you have is by using a marketplace image now google cloud marketplace lets you quickly deploy functional software packages that run on google cloud you can start up a software package without having to manually configure the software the vm instances the storage or even the network settings this is a all-in-one instance template that includes the operating system and the software pre-configured and you can deploy a software package whenever you like and is by far the easiest way to launch a software package and i will be giving you a run through on these marketplace images in a later demo now once you've decided on your machine type as well as the type of image that you wanted to use moving into the type of storage that you want would be your next step now when configuring a new instance you will need to create a new boot disk for it and this is where performance versus cost comes into play as you have the option to pay less and have a slower disk speed or lower iops or you can choose to have fast disk speed with higher iops but pay a higher cost and so the slowest and most inexpensive of these options is the standard persistent disk which are backed by standard hard disk drives the balance persistent disks are backed by solid state drives and are faster and can provide higher iops than the standard option and lastly ssd is the fastest option which also brings with it the highest iops available for persistent disks now outside of these three options for persistent disks you also have the option of choosing a local ssd and these are solid state drives that are physically attached to the server that hosts your vm instances and this is why they have the highest throughput and lowest latency than any of the available persistent disks just as a note the data that you store on a local ssd persists only until the instance is stopped or deleted which is why local ssds are suited only for temporary storage such as caches or swap disk and so lastly moving into networking each network interface of a compute engine instance is associated with a subnet of a unique vpc network as you've seen in the last section you can do this with an auto a default or a custom network each network is available in many different regions and zones within that region we've also experienced routing traffic for our instance both in and out of the vpc network by use of firewall rules targeting ip ranges specific network tags or by instances within the network now load balancers are responsible for helping distribute user traffic across multiple instances either within the network or externally using a regional or global load balancer and i will be getting into low balancing in another section of the course but i wanted to stress that load balancers are part of instance networking that help route and manage traffic coming in and going out of the network and so this is a high level overview of the different configuration types that go into putting together an instance and i will be diving deeper into each in this section as well i will be putting a hands-on approach to this by creating an instance in the next lesson and focusing on the different available features that you can use for your specific use case and so this is all i wanted to cover for this lesson so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back now i know in previous demonstrations we've built quite a few compute engine instances and have configured them accordingly in this demonstration we're going to go through a build of another instance but i wanted to dig deeper into the specific configurations that are available for compute engine so with that being said let's dive in and so i am now logged in under tony bowties gmail.com as well i am logged in under the bowtie inc project so in order to kick off this demo i'm going to head on over to the compute engine console so i'm going to go over to the navigation menu and i'm going to scroll down to compute engine and so here i'm prompted to either create or import a vm instance as well as taking the quick start and so i'm not going to import or take the quick start so i'm going to simply click on create and so i want to take a moment here to focus on the left hand menu where there are a bunch of different options to create any given instance so the first and default option allows me to create the instance from scratch choosing the new vm instance from template option allows me to create a new instance from an instance template and because i don't have any instance templates i am prompted here with the option to create one and so for those of you who are unfamiliar with instance templates templates are used in managed instance groups and define instance properties for when instances are launched within that managed instance group but don't worry i will be covering instance groups and instant templates in a later lesson the next option that's available is new vm instance from machine image and an image is a clone or a copy of an instance and again i will be covering this in a separate lesson and going through all the details of machine images but if i did have any machine images i would be able to create my instance from here but since i do not i am prompted with the option to create a new machine image now the last option that i wanted to show you is the marketplace and so the marketplace has existing machine images that are all pre-configured with its proper operating system as well as the software to accompany it so for instance if i'm looking to create a vm with a wordpress installation on it i can simply go up to the top to the search bar type in wordpress and i will be presented with many different options and i'm just going to choose the one here at the top and i am presented with 49 results of virtual machines with different types of wordpress installations on them and these are all different instances that have been configured specifically for wordpress by different companies like lightspeed analog innovation and cognosis inc and so for this demonstration i'm going to choose wordpress on centos 7 and here i'm giving an overview about the software itself i'm also given information about the company that configured this as well at the top i'm given a monthly estimated cost for this specific instance and if i scroll down the page i can get a little bit more information with regards to this image and as shown here on the right i can see my pricing the usage fee will cost me 109 a month along with the vm instance type that the software is configured for the amount of disk space and the sustained use discount i've also been given some links here for tutorials and documentation and i've also been given instructions for maintenance and support i've been given both an email and a link to live support and of course at the bottom we have the terms of service and this is a typical software package amongst many others that's available in the google cloud marketplace now i can go ahead and launch this if i choose but i'm going to choose not to launch this and i'm going to back out and so just to give you some context with regards to enterprise software software packages like f5 and jenkins are also available in the google cloud marketplace and again when i click on the first option it'll give me a bunch of available options on jenkins and its availability from different companies on different platforms now just as a note to update your existing deployment of a software package you have to redeploy the software package from marketplace in order to update it but other than that caveat the easiest way to deploy a software package is definitely through the marketplace and so now that we've gone through all the different options on how to create an instance i'm gonna go back and select new vm instance so i can create a new vm from scratch and so i am prompted here at the top with a note telling me that there was a draft that was saved from when i started to create in my new instance but i navigated away from it and i have the option to restore the configuration i was working on and so just know that when you are in the midst of creating an instance google cloud will automatically save a draft of your build so that you are able to continue working on it later now i don't really need this draft but i will just hit restore and for the name i'm going to keep it as instance 1 and for the sake of this demo i'm going to add a label the key is going to be environment and the value will be testing i'm going to go down to the bottom click save now when it comes to the geographic location of the instance using regions i can simply click on the drop down and i will have access to deploy this instance in any currently available region as regions are added they will be added here as well and so i'm going to keep it as us east one and under zone i have the availability of putting it in any zone within that region and so i'm going to keep it as us east 1b and just as another note once you've deployed the instance in a specific region you will not be able to move that instance to a different region you will have to recreate it using a snapshot in another region and i will be going over this in a later lesson now scrolling down to machine configuration there are three different types of families that you can choose from when it comes to machine types the general purpose the compute optimized and the memory optimized the general purpose machine family has a great available selection of different series types that you can choose from and is usually the go to machine family if you're unsure about which machine type to select so for this demo i'm going to keep my selection for series type as e2 and under machine type i'm given a very large selection of different sizes when it comes to vcpu and memory and so i can select from a shared core a standard type a high memory type or a high cpu type and i will be going over this in greater detail in another lesson on machine types now in case the predefined machine types do not fit my needs or the scope for the amount of vcpus and memory that i need fall in between those predefined machine types i can simply select the custom option and this will bring up a set of sliders where i am able to select both the amount of vcpus and amount of memory that i need for the instance that i am creating now as i change the course slider to either more vcpus or less my core to memory ratio for this series will stay the same and therefore my memory will be adjusted automatically i also have the option to change the memory as i see fit to either add more memory or to remove it and so this is great for when you're in between sizes and you're looking for something specific that fits your workload and so i'm going to change back the machine type to an e2 micro and as you can see in the top right i will find a monthly estimate of how much the instance will cost me and i can click on this drop down and it will give me a breakdown of the cost for vcpu in memory the cost for my disks as well as my sustained use discount and if i had any other resources that i was consuming like a static ip or an extra attached disk those costs would show up here as well and so if i went to a compute optimized you can see how the price has changed but i'm given the breakdown so that i know exactly what i'm paying for so i'm going to switch it back to general purpose and i wanted to point out here the cpu platform and gpu as you can add gpus to your specific machine configuration and so just as another note gpus can only be added to an n1 machine type as any other type will show the gpu selection as grayed out and so here i can add the gpu type as well as adding the number of gpus that i need but for the sake of this demonstration i'm not going to add any gpus and i'm going to select the e2 series and change it back to e2 micro scrolling down a little bit here when it comes to cpu platform depending on the machine type you can choose between intel or amd if you are looking for a specific cpu but just know that your configuration is permanent now moving down a little bit more you will see here display device now display device is a feature on compute engine that allows you to add a virtual display to a vm for system management tools remote desktop software and any application that requires you to connect to a display device on a remote server this is an especially great feature to have for when your server is stuck at boot patching or hardware failure and you can't log in and the drivers are already included for both windows and linux vms this feature works with the default vga driver right out of the box and so i'm going to keep this checked off as i don't need it and i'm going to move down to confidential vm service now confidential computing is a security feature to encrypt sensitive code and data that's in memory so even when it's being processed it is still encrypted and is a great use case when you're dealing with very sensitive information that requires strict requirements now compute engine also gives you the option of deploying containers on it and this is a great way to test your containers instead of deploying a whole kubernetes cluster and may even suffice for specific use cases but just note that you can only deploy one container per vm instance and so now that we've covered most of the general configuration options for compute engine i wanted to take a minute to dive into the options that are available for boot disk so i'm going to go ahead and click on change and here i have the option of choosing from a bunch of different public images with different operating systems that i can use for my boot disk so if i wanted to load up ubuntu i can simply select ubuntu and i can choose from each different version that's available as well i'm shown here the boot disk type which is currently selected as the standard persistent disk but i also have the option of selecting either a balanced persistent disk or ssd persistent disk and i'm going to keep it as standard persistent disk and if i wanted to i can increase the boot disk size so if i wanted 100 gigs i can simply add it and if i select it and i go back up to the top right hand corner i can see that my price for the instance has changed now i'm not charged for the operating system due to it being an open source image but i am charged more for the standard persistent disk because i'm no longer using 10 gigs but i'm using 100 gigabytes now let's say i wanted to go back and i wanted to change this image to a windows image i'm going to go down here to windows server and i want to select windows server 2016 i'm going to load up the data center version and i'm going to keep the standard persistent disk along with 100 gigabytes i'm going to select it if i scroll back up i can see that i'm charged a licensing fee for windows server and these images with these licensing fees are known as premium images so please make sure that you are aware of these licensing fees when launching your instances and because i want to save on money just for now i'm going to scroll back down to my boot disk and change it back to ubuntu and i'm going to change the size back down to 10 gigabytes as well before you move on i wanted to touch on custom images and so if i did have any custom images i could see them here and i would be able to create instances from my custom images using this method i also have the option of creating an instance from a snapshot and because i don't have any nothing shows up and lastly i have the option of using existing disks so let's say for instance i had a vm instance and i had deleted it but i decided to keep the attached boot disk it would show up as unattached and i am able to attach that to a new instance and so now that i've shown you all the available options when it comes to boot disk i'm going to go ahead and select the ubuntu operating system and move on to the next option here we have identity and api access which we've gone through in great depth in a previous demo as well i'm given an option to create a firewall rule automatically for http and https traffic and as for networking as we covered it in great depth in the last section i will skip that part of the configuration and simply launch it in the default vpc and so just as a quick note i wanted to remind you that down at the bottom of the page you can find the command line shortcut and when you click on it it will give you the gcloud command to run that you can use in order to create your instance and so i want to deploy this as is so i'm going to click here on close and i'm going to click on create and so i'm just going to give it a minute now so the instance can be created and it took a few seconds but the instance is created and this is regarded as the inventory page to view your instance inventory and to look up any correlating information on any of your instances and so this probably looks familiar to you from the previous instances that you've launched so here we have the name of the instance the zone the internal ip along with the external ip and a selection to connect to the instance as well i'm also given the option to connect to this instance in different ways you also have the option of adding more column information to your inventory dashboard with regards to your instance and you can do this by simply clicking on the columns button right here above the list of instances and you can select from creation time machine type preserve state and even the network and this may bring you more insight on the information available for that instance or even grouping of instances with common configurations this will also help you identify your instances visually in the console and so i'm just going to put the columns back to exactly what it was and so now i want to take a moment to dive right into the instance and have a look at the instance details so as you remember we selected the machine type of e2 micro which has two vcpus and one gigabyte of memory here we have the instance id as well scrolling down we have the cpu platform we have the display device that i was mentioning earlier along with the zone the labels the creation time as well as the network interface and scrolling down i can see here the boot disk with the ubuntu image as well as the name of the boot disk so there are quite a few configurations here and if i click on edit i can edit some of these configurations on the fly and with some configurations i need to stop the instance before editing them and there are some configurations like the network interface where i would have to delete the instance in order to recreate it so for instance if i wanted to change the machine type i need to stop the instance in order to change it and the same thing goes for my display device as well the network interface in order for me to change it from its current network or subnetwork i'm going to have to stop the instance in order to change it as well and so i hope this general walkthrough of configuring an instance has given you a sense of what can be configured on launch and allowed you to gain some insight on editing features of an instance after launch a lot of what you've seen here in this demo will come up in the exam and so i would recommend that before going into the exam to spend some time launching instances knowing exactly how they will behave and what can be edited after creation that can be done on the fly edits that need the instance to be shut down and edits that need the instance to be recreated and so that's pretty much all i wanted to cover when it comes to creating an instance so you can now mark this as complete and let's move on to the next one welcome back now in this lesson i'm going to be discussing compute engine machine types now a machine type is a set of virtualized hardware resources that's available to a vm instance including the system memory size virtual cpu count and persistent disks in compute engine machine types are grouped and curated by families for different workloads you must always choose a machine type when you create an instance and you can select from a number of pre-defined machine types in each machine type family if the pre-defined machine types don't meet your needs then you can create your own custom machine types in this lesson i will be going through all the different machine types their families and their use cases so with that being said let's dive in now each machine type family displayed here includes different machine types each family is curated for specific workload types the following primary machine types are offered on compute engine which is general purpose compute optimized and memory optimized and so i wanted to go through each one of these families in a little bit of detail now before diving right into it defining what type of machine type you are running can be overwhelming for some but can be broken down to be understood a bit better they are broken down into three parts and separated by hyphens the first part in this example shown here is the series so for this example the series is e2 and the number after the letter is the generation type in this case it would be the second generation now the series come in many different varieties and each are designed for specific workloads now moving on to the middle part of the machine type this is the actual type and types as well can come in a slew of different flavors and is usually coupled with a specific series so in this example the type here is standard and so moving on to the third part of the machine type this is the amount of vcp use in the machine type and so with vcpus they can be offered anywhere from one vcpu up to 416 vcpus and so for the example shown here this machine type has 32 vcpus and so there is one more aspect of a machine type which is the gpus but please note that gpus are only available for the n1 series and so combining the series the type and the vcpu you will get your machine type and so now that we've broken down the machine types in order to properly define them i wanted to get into the predefined machine type families specifically starting off with the general purpose predefined machine type and all the general purpose machine types are available in the standard type the high memory type and the high cpu type so the standard type is the balance of cpu and memory and this is the most common general purpose machine type general purpose also comes in high memory and this is a high memory to cpu ratio so very high memory a lower cpu and lastly we have the high cpu machine type and this is a high cpu to memory ratio so this would be the opposite of the high memory so very high cpu to lower memory so now digging into the general purpose machine family i wanted to start off with the e2 series and this is designed for day-to-day computing at a low cost so if you're looking to do things like web serving application serving back office applications small to medium databases microservices virtual desktops or even development environments the e2 series would serve the purpose perfectly now the e2 machine types are cost optimized machine types that offer sizing between 2 to 32 vcpus and half a gigabyte to 128 gigabytes of memory so small to medium workloads that don't require as many vcpus and applications that don't require local ssds or gpus are an ideal fit for e2 machines e2 machine types do not offer sustained use discounts however they do provide consistently low on-demand and committed use pricing in other words they offer the lowest on-demand pricing across the general purpose machine types as well the e2 series machines are available in both pre-defined and custom machine types moving on i wanted to touch on all the machine types available in the n-series and these are a balanced machine type with price and performance across a wide range of vm flavors and these machines are designed for web servers application servers back office applications medium to large databases as well as caching and media streaming and they are offered in the standard high memory and high cpu types now the n1 machine types are compute engines first generation general purpose machine types now this machine type offers up to 96 vcpus and 624 gigabytes of memory and again as i mentioned earlier this is the only machine type that offers both gpu support and tpu support the n1 type is available as both pre-defined machine types and custom machine types and the n1 series offers a larger sustained use discount than n2 machine types speaking of which the n2 machine types are the second generation general purpose machine types and these offer flexible sizing between two 280 vcpus and half a gigabyte of memory to 640 gigabytes of memory and these machine types also offer an overall performance improvement over the n1 machine types workloads that can take advantage of the higher clock frequency of the cpu are a good choice for n2 machine types and these workloads can get higher per thread performance while benefiting from all the flexibility that a general purpose machine type offers and two machine types also offer the extended memory feature and this helps control per cpu software licensing costs now getting into the last n series machine type the n2d machine type is the largest general purpose machine type with up to 224 vcpus and 896 gigabytes of memory this machine type is available in predefined and custom machine types and this machine type as well has the extended memory feature which i discussed earlier that helps you avoid per cpu software licensing the n2d machine type supports the committed use and sustain use discounts now moving on from the general purpose machine type family i wanted to move into the compute optimize machine family now this series offers ultra high performance for compute intensive workloads such as high performance computing electronic design automation gaming and single threaded applications so anything that is designed for compute intensive workloads this will definitely be your best choice now compute engine optimized machine types are ideal for as i said earlier compute intensive workloads and these machine types offer the highest performance per core on compute engine compute optimized types are only available as predefined machine types and so they are not available for any custom machine types the c2 machine types offer a maximum of 60 vcpus and a maximum of 240 gigabytes of memory now although the c2 machine type works great for compute intensive workloads it does come with some caveats and so you cannot use regional persistent disks with compute optimized machine types and i will be getting into the details of persistent disks in a later lesson and they are only available in select zones and regions on select cpu platforms and so now moving into the last family is the memory optimize machine family and this is for ultra high memory workloads this family is designed for large in memory databases like sap hana as well as in memory analytics now the m series comes in two separate generations m1 and m2 the m1 offering a maximum of 160 vcpus and a maximum memory of 3844 gigabytes whereas the m2 offering again a maximum of 160 vcpus but offering a whopping 11 776 gigabytes of maximum memory and as i said before these machine types they're ideal for tasks that require intensive use of memory so they are suited for in-memory databases and in memory analytics data warehousing workloads genomics analysis and sql analysis services memory optimized machine types are only available as predefined machine types and the caveats here is that you cannot use regional persistent disks with memory optimized machine types as well they're only available in specific zones now i wanted to take a moment to go back to the general purpose machine type so that i can dig into the shared cord machine type and this is spread amongst the e2 and n1 series and these shared core machine types are used for burstable workloads are very cost effective as well they're great for non-resource intensive applications shared core machine types use context switching to share a physical core between vcpus for the purpose of multitasking different shared core machine types sustain different amounts of time on a physical core which allows google cloud to cut the price in general share core instances can be more cost effective for running small non-resource intensive applications than standard high memory or high cpu machine types now when it comes to cpu bursting these shared core machine types offer bursting capabilities that allow instances to use additional physical cpu for short periods of time bursting happens automatically when your instance requires more physical cpu than originally allocated during these spikes your instance will take advantage of available physical cpu in bursts and the e2 shared core machine type is offered in micro small and medium while the n1 series is offered in the f1 micro and the g1 small and both of these series have a maximum of two vcpus with a maximum of four gigabytes of memory now i wanted to take a moment to touch on custom machine types and these are available for any general purpose machine and so this is customer defined cpu and memory designed for custom workloads now if none of the general purpose predefined machine types cater to your needs you can create a custom machine type with a specific number of vcpus and amount of memory that you need for your instance these machine types are ideal for workloads that are not a good fit for the pre-defined machine types that are available they're also great for when you need more memory or more cpu but the predefined machine types don't quite fit exactly what you need for your workload just as a note it costs slightly more to use a custom machine type than a pre-defined machine type and there are limitations in the amount of memory and vcpu you can select and as i stated earlier when creating a custom machine type you can choose from the e2 n2 and 2d and n1 machine types and so the last part i wanted to touch on are the gpus that are available and these are designed for the graphic intensive workloads and again are only available for the n1 machine type and gpus come in five different flavors from nvidia showing here as the tesla k80 the tesla p4 the tesla t4 the tesla v100 and the tesla p100 and so these are all the families and machine types that are available for you in google cloud and will allow you to be a little bit more flexible with the type of workload that you need them for and so for the exam you won't have to memorize each machine type but you will need to know an overview of what each machine type does now i know there's been a lot of theory presented here in this lesson but i hope this is giving you a better understanding of all the available pre-defined machine types in google cloud and so that's pretty much all i wanted to cover in this lesson on compute engine machine types so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in this lesson i'm going to be reviewing managing your instances now how you manage your instances is a big topic in the exam as well it's very useful to know for your work as a cloud engineer in the environments you are responsible for knowing both the features that are available as well as the best practices will allow you to make better decisions with regards to your instances and allow you to keep your environment healthy this lesson will dive into the many features that are available in order to better manage your instances using the specific features within google cloud so with that being said let's dive in now i wanted to start off this lesson discussing the life cycle of an instance within google cloud every instance has a predefined life cycle from its starting provisioning state to its deletion an instance can transition through many instant states as part of its life cycle when you first create an instance compute engine provisions resources to start your instance next the instance moves into staging where it prepares the first boot and then it finally boots up and is considered running during its lifetime a running instance can be repeatedly stopped and restarted or suspended and resumed so now i wanted to take a few minutes to go through the instance life cycle in a bit of detail starting with the provisioning state now this is where resources are being allocated for the instance the instance is not yet running and the instance is being allocated its requested amount of cpu and memory along with its root disk any additional disks that are attached to it and as well some additional feature sets that are assigned to this instance and when it comes to the cost while in the provisioning state there are no costs that are being incurred moving right along to the staging state after finishing the provisioning state the life cycle continues with the staging state and this is where resources have been acquired and the instance is being prepared for first boot both internal and external ips are allocated and can be either static or ephemeral in the system image that was originally chosen for this instance is used to boot up the instance and this can be either a public image or a custom image costs in the state are still not incurred as the instance is still in the pre-boot state now once the instance has left staging it will move on to the running state and this is where the instance is booting up or running and should allow you to log into the instance either using ssh or rdp within a short waiting period due to any startup scripts or any boot maintenance tasks for the operating system now during the running state you can reset your instance and this is where you would wipe the memory contents of the vm instance and reset the virtual machine to its initial state resetting an instance causes an immediate hard reset of the vm and therefore the vm does not do a graceful shutdown for the guest operating system however the vm retains all persistent disk data and none of the instance properties change the instance remains in running state through the reset now as well in the running state a repair can happen due to the instance encountering an internal error or the underlying machine is unavailable due to maintenance during this time the instance is unusable and if the repair is successful the instance returns back to the running state paying attention to costs this state is where the instance starts to occur them and is related to the resources assigned to the instance like the cpu and memory any static ips and any disks that are attached to the instance and i will be going into a bit of detail in just a bit with regards to this state and finally we end the life cycle with the stopping suspended and terminated states now when you are suspending an instance it is like closing the lid of your laptop suspending the instance will preserve the guest operating system memory and application state of the instance otherwise it'll be discarded and from this state you can choose either to resume or to delete it when it comes to stopping either a user has made a request to stop the instance or there was a failure and this is a temporary status and the instance will move to terminated touching on costs for just a second when suspending or stopping an instance you pay for resources that are still attached to the vm instance such as static ips and persistent disk data you do not pay the cost of a running vm instance ephemeral external ip addresses are released from the instance and will be assigned a new one when the instance is started now when it comes to stopping suspending or resetting an instance you can stop or suspend an instance if you no longer need it but want to keep the instance around for future use compute engine waits for the guest to finish shutting down and then transitions the instance to the terminated state so touching on the terminated state this is where a user either shuts down the instance or the instance encounters a failure you can choose to restart the instance or delete it as well as holding some reset options within the availability policy in this state you still pay for static ips and disks but like the suspending or stopping state you do not pay for the cpu and memory resources allocated to the instance and so this covers a high level overview of the instance lifecycle in google cloud and all of the states that make up this lifecycle now to get into some detail with regards to some feature sets for compute engine i wanted to revisit the states where those features apply now when creating your instance you have the option of using shielded vms for added security and when using them the instance would instantiate them as the instance boots and enters into the running state so what exactly is a shielded vm well shielded vms offer verifiable integrity of your compute engine vm instances so you can be sure that your instances haven't been compromised by boot or kernel level malware or rootkits and this is achieved through a four-step process which is covered by secure boot virtual trusted platform module also known as vtpm measure boot which is running on vtpm and integrity monitoring so i wanted to dig into this for just a sec to give you a bit more context now the boot process for shielded vms start with secure boot and this helps ensure that the system only runs authentic software by verifying the digital signature for all boot components and stopping the boot process if signature verification fails so shielded vm instances run firmware that's signed and verified using google's certificate authority and on each and every boot any boot component that isn't properly signed or isn't signed at all is not allowed to run and so the first time you boot a vm instance measure boot creates the integrity policy baseline from the first set of these measurements and then securely stores this data each time the vm instance boots after that these measurements are taken again and stored in secure memory until the next reboot having these two sets of measurements enables integrity monitoring which is the next step and allows it to determine if there have been changes to a vm instance's boot sequence and this policy is loaded onto a virtualized trusted platform module again known as the vtpm for short which is a specialized computer chip that you can use to protect objects like keys and certificates that you use to authenticate access to your system with shielded vms vtpm enables measured boot by performing the measurements needed to create a known good boot baseline and this is called the integrity policy baseline the integrity policy baseline is used for comparison with measurements from subsequent vm boots to determine if anything has changed integrity monitoring relies on the measurements created by measured boot for both the integrity policy baseline and the most recent boot sequence integrity monitoring compares the most recent boot measurements to the integrity policy baseline and returns a pair of pass or failed results depending on whether they match or not one for the early boot sequence and one for the late boot sequence and so in summary this is how shielded vms help prevent data exfiltration so touching now on the running state when you start a vm instance using google provided public images a guest environment is automatically installed on the vm instance a guest environment is a set of scripts daemons and binaries that read the content of the metadata server to make a virtual machine run properly on compute engine a metadata server is a communication channel for transferring information from a client to the guest operating system vm instances created using google provided public images include a guest environment that is installed by default creating vm instances using a custom image will require you to manually install the guest environment this guest environment is available for both linux and windows systems and each supported operating system that is available on compute engine requires specific guest environment packages either google or the owner of the operating system builds these packages now when it comes to the linux guest environment it is either built by google or the owner of the operating system and there are some key components that are applicable to all builds which can be found in the link that i have included in the lesson text the base components of a linux guest environment is a python package that contains scripts daemons and packages for the supported linux distributions when it comes to windows a similar approach applies where a package is available with main scripts and binaries as a part of this guest environment now touching back on the metadata server compute engine provides a method for storing and retrieving metadata in the form of the metadata server this service provides a central point to set metadata in the form of key value pairs which is then provided to virtual machines at runtime and you can query this metadata server programmatically from within the instance and from the compute engine api this is great for use with startup and shutdown scripts or gaining more insight with your instance metadata can be assigned to projects as well as instances and project metadata propagates to all instances within the project while instance metadata only impacts that instance and you can access the metadata using the following url with the curl command you see here on the screen so if you're looking for the metadata for a project you would use the first url that ends in project and for any instance metadata you can use the second url that ends in instance now please note that when you make a request to get information from the metadata server your request and the subsequent metadata response never leaves the physical host running the virtual machine instance now once the instance has booted and has gone through the startup scripts you will then have the ability to login to your instance using ssh or rdp now there are some different methods that you can use to connect and access both your linux instances and your windows instances that i will be going over now when it comes to linux instances we've already gone through accessing these types of instances in previous lessons and demos but just as a refresher you would typically connect to your vm instance via ssh access on port 22. please note that you will require a firewall rule as we have done in previous demos to allow this access and you can connect to your linux instances through the google cloud console or the cloud shell using the cloud sdk now i know that the use of ssh keys are the defacto when it comes to logging into linux instances now in most scenarios on google cloud google recommends using os login over using ssh keys the os login feature lets you use compute engine iam roles to manage ssh access to linux instances and then if you'd like you can add an extra layer of security by setting up os login with two-step verification and manage access at the organization level by setting up organizational policies os login simplifies ssh access management by linking your linux user account to your google identity administrators can easily manage access to instances at either an instance or project level by setting iam permissions now if you're running your own directory service for managing access or are unable to set up os login you can manually manage ssh keys and local user accounts in metadata by manually creating ssh keys and editing the public ssh key metadata now when it comes to windows instances you would typically connect to your vm instance via rdp access on port 3389 and please note that you will also require a firewall rule as shown here to allow this access you can connect to your windows instances through the rdp protocol or through a powershell terminal now when logging into windows this requires setting a windows password and can be done either through the console or the gcloud command line tool and then after setting your password you can then log in from the recommended rdp chrome extension or using a third-party rdp client and i will provide a link to this rdp chrome extension in the lesson text now once the instance has booted up and your instance is ready to be logged into you always have the option of modifying your instance and you can do it manually by either modifying it on the fly or you can take the necessary steps to edit your instance like i showed you in a previous lesson by stopping it editing it and then restarting it although when it comes to google having to do maintenance on a vm or you merely want to move your instance to a different zone in the same region this has all become possible without shutting down your instance using a feature called live migration now when it comes to live migration compute engine migrates your running instances to another host in the same zone instead of requiring your vms to be rebooted this allows google to perform maintenance reliably without interrupting any of your vms when a vm is scheduled to be live migrated google provides a notification to the guest that a migration is coming soon live migration keeps your instances running during compute engine hosts that are in need of regular infrastructure maintenance and upgrades replacement of failed hardware and system configuration changes when google migrates a running vm instance from one host to another it moves the complete instance state from the source to the destination in a way that is transparent to the guest os and anyone communicating with it google also gives you the option of doing live migration manually from one zone to another within the same region either using the console or running the command line you see here gcloud compute instances move the name of the vm with the zone flag and the zone that it's currently in and then the destination zone flag with the zone that you wanted to go to and just as a note with some caveats instances with gpus attached cannot be live migrated and you can't configure a preemptable instance to live migrate and so instance lifecycle is full of different options and understanding them can help better coordinate moving editing and repairing vm instances no matter where they may lie in this life cycle now i hope this lesson has given you the necessary theory that will help better use the discuss feature sets and giving you some ideas on how to better manage your instances now there is a lot more to know than what i've shown you here to manage your instances but topics shown here are what shows up in the exam as well are some really great starting points to begin managing your instances and so that's pretty much all i wanted to cover when it comes to managing instances so you can now mark this lesson as complete and join me in the next one where i will cement the theory in this lesson with the hands-on demo [Music] welcome back in this demonstration i'm going to be cementing some of the theory that we learned in the last lesson with regards to the different login methods for windows and linux instances how to implement these methods are extremely useful to know both for the exam and for managing multiple instances in different environments now there's a lot to cover here so with that being said let's dive in so as you can see i am logged in here under tony bowtie ace gmail.com as well i am in the project of bowtie inc and so the first thing that i want to do is create both a linux instance and a windows instance and this is to demonstrate the different options you have for logging into an instance and so in order for me to do that i need to head on over to compute engine so i'm going to go over to the navigation menu and i'm going to scroll down to compute engine and so just as a note before creating your instances please make sure that you have a default vpc created before going ahead and creating these instances if you've forgotten how to create a default vpc please go back to the networking services section and watch the vpc lesson for a refresher and so i'm going to go ahead and create my first instance and i'm going to start with the windows instance so i'm going to simply click on create and so for the name of this instance you can simply call this windows dash instance and i'm not going to add any labels and for the region you should select us east1 and you can keep the zone as the default for us east 1b and scrolling down to the machine configuration for the machine type i'm going to keep it as is as it is a windows instance and i'm going to need a little bit more power scrolling down to boot disk we need to change this from debian over to windows so i'm going to simply click on the change button and under operating system i'm going to click on the drop down and select windows server for the version i'm going to select the latest version of windows server which is the windows server 2019 data center and you can keep the boot disk type and the size as its default and simply head on down and click on select and we're going to leave everything else as the default and simply click on create and success our windows instance has been created and so the first thing that you want to do is you want to set a windows password for this instance and so i'm going to head on over to the rdp button and i'm going to click on the drop-down and here i'm going to select set windows password and here i'm going to get a pop-up to set a new windows password the username has been propagated for me as tony bowties i'm going to leave it as is and i'm going to click on set and i'm going to be prompted with a new windows password that has been set for me so i'm going to copy this and i'm going to paste it into my notepad so be sure to record it somewhere either write it down or copy and paste it into a text editor of your choice i'm going to click on close and so now for me to log into this i need to make sure of a couple things the first thing is i need to make sure that i have a firewall rule open for port 3389 the second is i need to make sure that i have an rdp client and so in order to satisfy my first constraint i'm going to head on over to the navigation menu and go down to vpc network here i'm going to select firewall and as expected the rdp firewall rule has been already created due to the fact that upon creation of the default vpc network this default firewall rule is always created and so now that i've gotten that out of the way i'm going to head back on over to compute engine and what i'm going to do is i'm going to record the external ip so that i'll be able to log into it now i'm going to be logging into this instance from both a windows client and a mac client so starting with windows i'm going to head on over to my windows virtual machine and because i know windows has a default rdp client already built in i'm going to simply bring it up by hitting the windows key and typing remote desktop connection i'm going to click on that i'm going to paste in the public ip for the instance that i just recorded and i'm going to click on connect you should get a pop-up asking for your credentials i'm going to type in my username as tony bowtie ace as well i'm going to paste in the password and i'm going to click on ok i'm prompted to accept the security certificate and i'm going to select yes and success i'm now connected to my windows server instance and it's going to run all its necessary startup scripts you may get a couple of prompts that come up asking you if you want to connect to your network absolutely i'm going to close down server manager just for now and another thing that i wanted to note is that when you create a windows instance there will automatically be provisioned a google cloud shell with the sdk pre-installed and so you'll be able to run all your regular commands right from this shell without having to install it and this is due to the guest environment that was automatically installed on the vm instance upon creation and this is a perfect example of some of the scripts that are installed with the guest environment i'm going to go ahead and close out of this and i'm going to go ahead and close out of my instance hit ok and so being here in windows i wanted to show you an alternate way of logging into your instance through powershell so for those of you who are quite versed in windows and use powershell in your day-to-day there is an easy way to log into your instance using powershell now in order for me to do that i need to open another firewall rule covering tcp port 5986 so i'm going to head on over back to the google cloud console i'm going to head over to the navigation menu and i'm going to scroll down to vpc network i'm going to go into firewall and i'm going to create a new firewall rule and under name i'm going to name this as allow powershell i'm going to use the same for the description i'm going to scroll down to targets and i'm going to select all instances in the network and under source ip ranges for this demonstration i'm going to use 0.0.0.0 forward slash 0. and again this should not be used in a production environment but is used merely for this demo i'm going to leave everything else as is and i'm going to go down to protocols and ports i'm going to click on tcp and i'm going to type in 5986 for the port and i'm going to click on create i'm going to give it a second just to create and it took a couple seconds but our firewall rule is now created and so now i'm gonna head over to my windows vm and i'm gonna open up a powershell command prompt and hit the windows key and type in powershell and so in order for me to not get constantly asked about my username and password i'm going to use a variable that will keep my password for me and so every time i connect to my windows instance i won't need to type it in all the time and so the command for that is dollar sign credentials equals get dash credential i'm going to hit enter and i'm going to get a prompt to type in my username and password so i'm going to simply type that in now along with my password and hit ok and if you don't get a prompt with any errors then chances are that you've been successful at entering your credentials and so now in order to connect to the instance you're going to need the public ip address again so i'm going to head on over back to the console i'm going to head on over to the navigation menu and back to compute engine here i'm going to record the external ip and i'm going to head on over back to my windows virtual machine and so you're going to enter this command which i will include in the lesson text and you'll also be able to find it in the github repository beside computer name you're going to put in your public ip address of your windows instance and make sure at the end you have your credentials variable i'm going to simply click enter and success i'm now connected to my windows instance in google cloud so as you can see here on the left is the public ip of my windows instance and so these are the various ways that you can connect to your windows instance from a windows machine and so now for me to connect to my windows instance on a mac i'm going to head on over there now and like i said before i need to satisfy the constraint of having an rdp client unfortunately mac does not come with an rdp client and so the recommended tool to use is the chrome extension but i personally like microsoft's rdp for mac application and so i'm going to go ahead and do a walkthrough of the installation so i'm going to start off by opening up safari and i'm going to paste in this url which i will include in the lesson text and microsoft has made available a microsoft remote desktop app available in the app store i'm going to go ahead and view it in the app store and i'm going to simply click on get and then install and once you've entered your credentials and you've downloaded and installed it you can simply click on open i'm going to click on not now and continue and i'm going to close all these other windows for better viewing i'm going to click on add pc i'm going to paste in the public ip address of my windows instance and under user account i'm going to add my user account type in my username paste in my password you can add a friendly name here i'm going to type in windows dash gc for google cloud and i'm going to click on add and then once you've pasted in all the credentials and your information you can then click on add and i should be able to connect to my windows instance by double clicking on this window it's asking me for my certificates i'm going to hit continue and success i'm connected to my windows instance and so this is how you would connect to a windows instance from a windows machine as well as from a mac as well there are a couple of other options that i wanted to show you over here on the drop down beside rdp i can download an rdp file which will contain the public ip address of the windows instance along with your username if i need to reset my password i can view the gcloud command to do it or i can set a new windows password if i forgotten my old one and so that's everything i had to show you with regards to connecting to a windows instance and so since this demo was getting kind of long i decided to split it up into two parts and so this is the end of part one of this demo and this would be a great opportunity to get up and have a stretch grab yourself a tea or a coffee and whenever you're ready you can join me in part two where we will be starting immediately from the end of part 1 so you can complete this video and i'll see you in part 2. [Music] welcome back this is part 2 of the connecting to your instances demo and we will be starting exactly where we left off in part one so with that being said let's dive in and so now that we've created our windows instance and went through all the methods of how to connect to it let's go ahead and create a linux instance i'm going to go up to the top menu here and click on create instance and i'm going to name this instance linux instance i'm not going to give it any labels under region i'm going to select the us east one region and the zone i'm going to leave it as its set default as us east 1b the machine configuration i'm going to leave it as is under boot disk i'm going to leave this as is with the debian distribution and i'm going to go ahead and click on create okay and our linux instance has been created and in order for me to connect to it i am going to ssh into it but first i need to satisfy the constraint of having a firewall rule with tcp port 22 open so i'm going to head on over to the navigation menu and i'm going to scroll down to vpc network i'm going to head on over to firewall and as expected the allow ssh firewall rule has been created alongside the default vpc network and so since i've satisfied that constraint i can head back on over to compute engine and so here i have a few different options that i can select from for logging into my linux instance i can open in a browser window if i decided i wanted to put it on a custom port i can use this option here if i provided a private ssh key to connect to this linux instance i can use this option here i have the option of viewing the gcloud command in order to connect to it and i've been presented with a pop-up with the command to use within the gcloud command line in order to connect to my instance i can run it now in cloud shell but i'm going to simply close it and so whether you are on a mac a windows machine or a linux machine you can simply click on ssh and it will open a new browser window connecting you to your instance now when you connect to your linux instance for the first time compute engine generates an ssh key pair for you this key pair by default is added to your project or instance metadata and this will give you the freedom of not having to worry about managing keys now if your account is configured to use os login compute engine stores the generated key pair with your user account now when connecting to your linux instance in most scenarios google recommends using os login this feature lets you use iam roles to manage ssh access to linux instances and this relieves the complexity of having to manage multiple key pairs and is the recommended way to manage many users across multiple instances or projects and so i'm going to go ahead now and show you how to configure os login for your linux instance and the way to do this will be very similar on all platforms so i'm going to go ahead and go back to my mac vm and i'm going to open up my terminal make this bigger for better viewing and i'm going to start by running the gcloud init command in order to make sure i'm using the right user and for the sake of this demonstration i'm going to re-initialize this configuration so i'm going to click on one hit enter number two for tony bowtie ace and i'm going to use project bow tie ink so 1 and i'm not going to configure a default compute region in zone and so if i run the gcloud config list command i can see that the account that i'm using is tony bowties gmail.com in project bowtie inc and so because os login requires a key pair i'm going to have to generate that myself so i'm going to go ahead and clear the screen and i'm going to use the command ssh keygen and this is the command to create a public and private key pair i'm going to use the default path to save my key and i'm going to enter a passphrase i'm going to enter it again and i recommend that you write down your passphrase so that you don't forget it as when you lose it you will be unable to use your key pair and so if i change directory to dot ssh and do an ls for list i can see that i now have my public and private key pair the private key lying in id underscore rsa and the public key lying in id underscore rsa.pub and so another constraint that i have is i need to enable os login for my linux instance so i'm going to go ahead and go back to the console and i'm going to go ahead and go into my linux instance i'm going to click on edit and if you scroll down you will come to some fields marked as custom metadata and under key you will type in enable dash os login and under value you will type in all caps true now i wanted to take a moment here to discuss this feature here under ssh keys for block project wide ssh keys now project wide public ssh keys are meant to give users access to all of the linux instances in a project that allow project project-wide public ssh keys so if an instance blocks project-wide public ssh keys as you see here a user can't use their project-wide public ssh key to connect to the instance unless the same public ssh key is also added to the instance metadata this allows only users whose public ssh key is stored in instance level metadata to access the instance and so this is an important feature to note for the exam and so we're going to leave this feature checked off for now and then you can go to the bottom and click on save now if i wanted to enable os login for all instances in my project i can simply go over to the menu on the left and click on metadata and add the metadata here with the same values so under key i type in enable dash os login and under value i type in in all caps true but i don't want to enable it for all my instances only for that one specific instance so with regards to project-wide public keys these keys can be managed through metadata and should only be used as a last resort if you cannot use the other tools such as ssh from the console or os login these are where the keys are stored and so you can always find them here when looking for them here as you can see there are a couple of keys for tony bowtie ace that i have used for previous instances and so i'm going to go back to metadata just to make sure that my key value pair for os login has not been saved and it is not and i'm going to head back on over to my instances and so now that my constraint has been fulfilled where i've enabled the os login feature by adding the unnecessary metadata i'm going to head on over back to my mac vm i'm going to go ahead and clear the screen so now i'm going to go ahead and log into my instance using os login by using the command gcloud compute os dash login ssh dash keys add and then the flag key dash file and then the path for my public key which is dot ssh forward slash id underscore rsa.pub i'm gonna hit enter and so my key has been successfully stored with my user account i'm gonna go ahead and make this a little bigger for better viewing and so in order to log into my instance i'm going to need my username which is right up here under username i'm going to copy that and i'm just going to clear my screen for a second here for better viewing and so in order for me to ssh into my instance i'm going to type in the command ssh minus i i'm going to have to provide my private key which is in dot ssh forward slash id underscore rsa and then my username that i had recorded earlier at and then i'm going to need my public ip address of my linux instance so i'm going to head back over to the console for just a sec i'm going to copy the ip address head back over to my mac vm paste it in and hit enter it's asking if i want to continue yes i do enter the passphrase for my key and success i am connected and so there is one caveat that i wanted to show you with regards to permissions for os login so i'm going to head back over to the console and i'm going to go up to the navigation menu and head over to i am an admin now as you can see here tony bowties gmail.com has the role of owner and therefore i don't need any granular specific permissions i have the access to do absolutely anything now in case i was a different user and i didn't hold the role of owner i would be looking for specific permissions that would be under compute os login and this would give me permissions as a standard user now if i wanted super user access or root access i would need to be given the compute os admin login role and as you can see it would allow me administrator user privileges so when using os login and the member is not an owner one of these two roles are needed so i'm going to exit out of here i'm going to hit cancel and so that about covers everything that i wanted to show you with regards to all the different methods that you can use for connecting to vm instances for both windows and linux instances now i know this may have been a refresher for some but for others knowing all the different methods of connecting to instances can come in very useful especially when coordinating many instances in bigger environments i want to congratulate you on making it to the end of this demo and gaining a bit more knowledge on this crucial part of managing your instances so before you go be sure to delete any resources that you've created and again congrats on the great job so you can now mark this as complete and i'll see you in the next one welcome back in this demonstration i'll be discussing metadata and how it can pertain to a project as well as an instance as well i'm going to touch on startup and shutdown scripts and it's real world use cases in the last lesson we touched the tip of the iceberg when it came to metadata and wanted to go a bit deeper on this topic as i personally feel that it holds so much value and give you some ideas on how you can use it i'm also going to combine the metadata using variables in a startup script and i'm going to bring to life something that's dynamic in nature so with that being said let's dive in so i am currently logged in as tony at bowtie ace gmail.com under the project of bow tie inc and so in order to get right into the metadata i'm going to head on over to my navigation menu and go straight to compute engine and over here on the left hand menu you will see metadata and you can drill down into there now as i explained in a previous lesson metadata can be assigned to both projects and instances while instance metadata only impacts a specific instance so here i can add and store metadata which will be used on a project-wide basis as well as mentioned earlier metadata is stored in key value pairs and can be added at any time now this is a way to add custom metadata but there is a default set of metadata entries that every instance has access to and again this applies for both project and instance metadata so here i have the option of setting my custom metadata for the entire project and so i'm going to dive into where to store custom metadata on an instance and so in order for me to show you this i'm going to first head over to vm instances and create my instance and so just as a note before creating your instance make sure that you have the default vpc created and so because i like to double check things i'm going to head over to the navigation menu i'm going to scroll down to vpc network and as expected i have the default vpc already created and so this means i can go ahead and create my instance so i'm going to head back on over to compute engine and i'm going to create my instance and i'm going to name this instance bowtie dash web server i'm not going to add any labels and under the region i'm going to select us east one and you can keep the zone as the default as us east 1b under machine type i want to keep things cost effective so i'm going to select the e2 micro i'm going to scroll down and under identity and api access i want to set access for each api and scroll down to compute engine i want to select it and i want to select on read write and i'm going to leave the rest as is and scrolling down to the bottom i want to click on management security disks networking and sold tenancy and under here you will find the option to add any custom metadata and you can provide it right here under metadata as a key value pair but we're not going to add any metadata right now so i'm just going to scroll down to the bottom i'm going to leave everything else as is and simply click on create and it should take a few moments for my instance to be created okay and now that my instance is up i want to go ahead and start querying the metadata now just as a note metadata must be queried from the instance itself and can't be done from another instance or even from the cloud sdk on your computer so i'm going to go ahead and log into the instance using ssh okay and now that i'm logged into my instance i want to start querying the metadata now normally you would use tools like wget or curl to make these queries in this demo i will use curl and for those who don't know curl is a command line tool to transfer data to or from a server using supported protocols like http ftp scp and many more this tool is fantastic for automation since it's designed to work without any user interaction and so i'm going to paste in the url that i am going to use to query the instance metadata and this is the default url that you would use to query any metadata on any instance getting a little deeper into it a trailing slash shown here shows that the instance value is actually a directory and will have other values that append to this url whether they are other directories or just endpoint values now when you query for metadata you must provide the following header in all of your requests metadata dash flavor colon google and should be put in quotations if you don't provide this header the metadata server will deny your request so i'm going to go ahead and hit enter and as you can see i've been brought up a lot of different values that i can choose from in order to retrieve different types of metadata and as stated before anything with a trailing slash is actually a directory and will have other values underneath it so if i wanted to query the network interfaces and because it's a directory i need to make sure that i add the trailing slash at the end and as you can see here i have the network interface of 0 and i'm going to go ahead and query that and here i will have access to all the information about the network interface on this instance so i'm going to go ahead and query the network on this interface and as expected the default network is displayed i'm going to quickly go ahead and clear my screen and i'm going to go ahead and query some more metadata this time i'm going to do the name of the server and as expected bowtie dash web server showed up and because it's an endpoint i don't need the trailing slash at the end i'm going to go ahead and do one more this time i'm going to choose machine type and again as expected the e2 micro machine type is displayed and so just as a note for those who haven't noticed any time that you query metadata it will show up to the left of your command prompt now what i've shown you here is what you can do with instance metadata and so how about if you wanted to query any project metadata well instead of instance at the end you would use project with the trailing slash i'm going to simply click on enter and as you can see here project doesn't give me a whole lot of options but it does give me some important values like project id so i'm going to simply query that right now and as expected bowtie inc is displayed and so this is a great example of how to query any default metadata for instances and for projects now you're probably wondering how do i query my custom metadata well once custom metadata has been set you can then query it from the attributes directory in the attributes directory can be found in both the instance and project metadata so i'm going to go ahead and show you that now but first i wanted to add some custom metadata and this can be set in either the console the gcloud command line tool or using the api and so i'm going to run the command here gcloud compute instances add dash metadata the name of your instance and when you're adding custom metadata you would add the flag dash dash metadata with the key value pair which in this example is environment equals dev and then i'm also going to add the zone of the instance which is us east 1a and i'm going to hit enter and because i had a typo there i'm going to go ahead and try that again using us east 1b i'm going to hit on enter and success and so to verify that this command has worked i'm going to go ahead and query the instance and i'm going to go under attributes i'm going to hit on enter and as you can see here the environment endpoint has been populated so i'm going to query that and as expected dev is displaying as the environment value now if i wanted to double check that in the console i can go over to the console i can drill down into bowtie web server and if i scroll down to the bottom under custom metadata you can see the key value pair here has m as the key and dev being the value and so these are the many different ways that you can query metadata for any instances or projects now i wanted to take a quick moment to switch gears and talk about startup and shutdown scripts now compute engine lets you create and run your own startup and shutdown scripts on your vm instance and this allows you to perform automation that can perform actions when starting up such as installing software performing updates or any other tasks that are defined in the script and when shutting down you can allow instances time to clean up on perform tasks such as exporting logs to cloud storage or bigquery or syncing with other systems and so i wanted to go ahead and show you how this would work while combining metadata into the script so i'm going to go ahead and drill down into bow tie web server i'm going to click on edit and i'm going to scroll down here to custom metadata i'm going to click on add item and under key i'm going to type in startup dash script and under value i'm going to paste in my script i'm going to just enlarge this here for a second and i will be providing the script in the github repository now just to break it down this is a bash script i'm pulling in a variable called name which will query the instance name as well i have a variable called zone which will query the instance zone i'm going to be installing an apache web server and it's going to display on a web browser both the server name and the zone that it's in and so in order for me to see this web page i also need to open up some firewall rules and so an easy way to do this would be to scroll up to firewalls and simply click on allow http and allow https traffic this will tag the instance with some network tags as http server and https server and create two separate firewall rules that will allow traffic for port 80 and port 443 so i'm going to leave everything else as is i'm going to scroll down to the bottom and click on save okay and it took a few seconds there but it did finish saving i'm going to go ahead and go up to the top and click on reset and this will perform a hard reset on the instance and will allow the startup script to take effect so i'm going to click on reset it's going to ask me if i really want to do this and for the purposes of this demonstration i'm going to click on reset please note you should never do this in production as it doesn't do a clean shutdown on the operating system but as this is an instance with nothing on it i'm going to simply click on reset now i'm going to head on back to the main console for my vm instances and i'm going to record my external ip i'm going to open up a new browser i'm going to zoom in for better viewing and i'm going to paste in my ip address and hit enter and as you can see here i've used my startup script to display not only this web page but i was able to bring in metadata that i pulled using variables and was able to display it here in the browser and so before i end this demonstration i wanted to show you another way of using a startup script but being able to pull it in from cloud storage so i'm going to go back to the navigation menu and i'm going to scroll down to storage here i will create a new bucket and for now find a globally unique name to name your bucket and i'm going to call my bucket bowtie web server site and i'm going to leave the rest as its default and i'm going to simply click on create and if you have a globally unique name for your bucket you will be prompted with this page without any errors and i'm going to go ahead and upload the script and you can find this script in the github repository so i'm going to go into my repo and i'm going to look for bow tie start up final sh i'm going to open it and now that i have the script uploaded i'm going to drill into this file so i can get some more information that i need for the instance and what i need from here is to copy the uri so i'm going to copy this to my clipboard and i'm going to head back on over to compute engine i'm going to drill down into my instance i'm going to click on edit at the top and i'm going to scroll down to where it says custom metadata and here i'm going to remove the startup script metadata and i'm going to add a new item and i'm going to be adding startup dash script dash url and in the value i'm going to paste in the uri that i had just copied over and this way on startup my instance will use this startup script that's in cloud storage so i'm going to scroll down to the bottom click on save and now i'm going to click on reset i'm going to reset here i'm going to go back to the main page for my vm instances and i can see that my external ip hasn't changed so i'm going to go back to my open web browser and i'm going to click on refresh and success and as you can see here i've taken a whole bunch of different variables including the machine name the environment variable the zone as well as the project and i've displayed it here in a simple website and although you may not find this website specifically useful in your production environment this is just an idea to get creative using default and custom metadata along with a startup script i've seen in some environments where people have multiple web servers and create a web page to display all the specific web servers in their different environments along with their ips their data and their configurations and so just as a recap we've gone through the default and custom metadata and how to query it in an instance we also went through startup scripts and how to apply them both locally and using cloud storage and so i hope you have enjoyed having fun with metadata and using them in startup scripts such as this one i also hope you find some fascinating use cases in your current environments and so before you go just a quick reminder to delete any resources that you've created to not incur any added costs and so that's pretty much all i wanted to cover with this demonstration so you can now mark this as complete and let's move on to the next one [Music] welcome back and in this lesson i'm going to be discussing compute engine billing now when it comes to pricing with regards to compute engine i've only gone over the fact that instances are charged by the second after the first minute but i never got into the depths of billing and the various ways to save money when using compute engine in this lesson i will be unveiling how both costs and discounts are broken down in google cloud as it refers to the resource based billing model and the various savings that can be had when using compute engine so with that being said let's dive in now each vcpu and each gigabyte of memory on compute engine is built separately rather than as part of a single machine type you are still creating instances using pre-defined machine types but your bill shows them as individual cpus and memory used per hour and this is what google refers to as resource-based billing which i will get into in just a bit the billing model applies to all vcpus gpus and memory resources and are charged a minimum of one minute for example if you run your virtual machine for 30 seconds you will be billed for one minute of usage after one minute instances are charged in one second increments instance up time is another determining factor for cost and is measured as the number of seconds between when you start an instance and when you stop an instance in other words when your instance is in the terminated state if an instance is idle but still has a state of running it will be charged for instance uptime but again you will not be charged if your instance is in a terminated state now getting into reservations these are designed to reserve the vm instances you need so after you create a reservation the reservation ensures that those resources are always available for you to use during the creation process you can choose how a reservation is to be used for example you can choose for a reservation to be automatically applied to any new or existing instances that match the reservation's properties which is the default behavior or you can specify that reservation to be consumed by a specific instance in all cases a vm instance can only use a reservation if its properties exactly match the properties of the reservation after you create a reservation you begin paying for the reserved resources immediately and they remain available for your project to use indefinitely until the reservation is deleted reservations are great to ensure that your project has resources for future increases in demand including planned or unplanned spikes backup and disaster recovery or for a buffer when you're planning growth when you no longer need a reservation you can simply delete the reservation to stop incurring charges each reservation like normal vms are charged based on existing on-demand rates which include sustained use discounts and are eligible for committed use discounts which i will be getting into in just a bit now purchasing reservations do come with some caveats reservations apply only to compute engine data proc and google kubernetes engine as well reservations don't apply to shared core machine types preemptable vms sole tenant nodes cloud sql and data flow now as i explained before each vcpu and each gigabyte of memory on compute engine is built separately rather than as a part of a single machine type and is billed as individual cpus and memory used per hour resource-based pricing allows compute engine to apply sustained use discounts to all of your pre-defined machine type usage in a region collectively rather than to individual machine types and this way vcpu and memory usage for each machine type can receive any one of the following discounts sustained use discounts committed use discounts and preemptable vms and i'd like to take a moment to dive into a bit of detail on each of these discount types starting with sustained use discounts now sustained use discounts are automatic discounts for running specific compute engine resources a significant portion of the billing month for example when you run one of these resources for more than 25 percent of a month compute engine automatically gives you a discount for every incremental minute that you use for that instance now the following tables show the discounts applied for the specific resources described here now for the table on the left for general purpose n2 and n2d predefined and custom machine types and for compute optimized machine types you can receive a discount of up to 20 percent the table on the right shows that for general purpose n1 predefined and custom machine types as well as sole tenant nodes and gpus you can get a discount of up to 30 percent sustained use discounts are applied automatically to usage within a project separately for each region so there is no action required on your part to enable these discounts now some notes that i wanted to cover here is that sustained use discounts automatically apply to vms created by both google kubernetes engine and compute engine as well they do not apply to vms created using the app engine flexible environment as well as data flow and the e-2 machine types sustained use discounts are applied on incremental use after you reach certain usage thresholds this means that you pay only for the number of minutes that you use an instance and compute engine automatically gives you the best price google truly believes that there's no reason to run an instance for longer than you need it now sustained use discounts are applied on incremental use after you reach certain usage thresholds this means that you pay only for the number of minutes that you use an instance and compute engine automatically gives you the best price now consider a scenario where you have two instances or sole tenant nodes in the same region that have different machine types and run at different times of the month compute engine breaks down the number of vcpus and amount of memory used across all instances that use predefined machine types and combines the resources to qualify for the largest sustained usage discounts possible now in this example assume you run the following two instances in the us east one region during a month for the first half you run an n1 standard four instance with four vcpus and 15 gigabytes of memory for the second half of the month you run a larger and one standard 16 instance with 16 vcpus and 60 gigabytes of memory in this scenario compute engine reorganizes these machine types into individual vcpu and memory resources and combines their usage to create the following resources for vcpus so because four vcpus were being used for the whole month the discount here would be thirty percent the additional twelve vcpus were added on week two in the month and so for those 12 vcpus they would receive a 10 discount and this is how discounts are applied when it comes to sustained use discounts now moving on to the next discount type is committed use discounts so compute engine lets you purchase committed use contracts in return for deeply discounted prices for vm usage so when you purchase a committed use contract you purchase compute resource which is comprised of vcpus memory gpus and local ssds and you purchase these resources at a discounted price in return for committing to paying for those resources for one year or three years committed use discounts are ideal for workloads with predictable resource needs so if you know exactly what you're going to use committed use discounts would be a great option for this and the discount is up to 57 for most resources like machine types or gpus when it comes to memory optimized machine types the discount is up to 70 percent now when you purchase a committed use contract you can purchase it for a single project and applies to a single project by default or you can purchase multiple contracts which you can share across many projects by enabling shared discounts once purchased your billed monthly for the resources you purchased for the duration of the term you selected whether you use the services or not if you have multiple projects that share the same cloud billing account you can enable committed use discount sharing so that all of your projects within that cloud billing account share all of your committed use discount contracts your sustained use discounts are also pooled at the same time now some caveats when it comes to committed use discounts shared core machines are excluded on this as well you can purchase commitments only on a per region basis if a reservation is attached to a committed use discount the reservation can't be deleted for the duration of the commitment so please be aware now to purchase a commitment for gpus or local ssds you must purchase a general purpose and one commitment and lastly after you create a commitment you cannot cancel it you must pay the agreed upon monthly amount for the duration of the commitment now committed use discount recommendations give you opportunities to optimize your compute costs by analyzing your vm spending trends with and without a committed use discount contract by comparing these numbers you can see how much you can save each month with a committed use contract and this can be found under the recommendations tab on the home page in the console and so i wanted to move on to the last discount type which are preemptable vms now preemptable vms are up to eighty percent cheaper than regular instances pricing is fixed and you never have to worry about variable pricing these prices can be found on the link to instance pricing that i have included in the lesson text a preemptable vm is an instance that you can create and run at a much lower price than normal instances however compute engine might stop or preempt these instances if it requires access to those resources for other tasks as preemptable instances our access compute engine capacity so their availability varies with usage now generally compute engine avoids preempting instances but compute engine does not use an instant cpu usage or other behavior to determine whether or not to preempt it now a crucial characteristic to know about preemptable vms is that compute engine always stops them after they run for 24 hours and this is something to be aware of for the exam preemptable instances are finite compute engine resources so they might not always be available and if you happen to accidentally spin up a preemptable vm and you want to shut it down there is no charge if it's running for less than 10 minutes now another thing to note is that preemptable instances can't live migrate to a regular vm instance or be set to automatically restart when there is a maintenance event due to the limitations preemptable instances are not covered by any service level agreement and when it comes to the google cloud free tier credits for compute engine this does not apply to preemptable instances so you're probably asking when is a great time to use preemptable vms well if your apps are fault tolerant and can withstand possible instance preemptions then preemptable instances can reduce your compute engine costs significantly for example batch processing jobs can run on preemptable instances if some of those instances stop during processing the job slows down but does not completely stop preemptable instances create your batch processing tasks without placing any additional workload on your existing instances and without requiring for you to pay full price for additional normal instances and since containers are naturally stateless and fault tolerant this makes containers an amazing fit for preemptable vms so running preemptable vms for google kubernetes engine is another fantastic use case now it's really critical that you have an understanding for each different discount type and when is a good time to use each as you may be presented different cost-effective solutions in the exam and understanding these discount types will prepare you to answer them understanding the theory behind this resource-based pricing model all the available discount types along with the types of workloads that are good for each will guarantee that you will become familiar with what types of questions are being asked in the exam and will also make you a better cloud engineer as you will be able to spot where you can save money and be able to make the appropriate changes and so that's pretty much all i wanted to cover when it comes to compute engine billing and its discount types so you can now mark this lesson as complete and let's move on to the next one welcome back in this lesson i'm going to be covering the fundamentals as it pertains to storage these concepts are needed to know in order to fully understand the different google cloud storage options that i will be diving into later as well the exam expects that you know the different types of storage that's available for all the various services and so before i get into the different types of storage i wanted to cover the underlying theory behind it so with that being said let's dive in so i wanted to start off by going through the three types of storage and how data is presented to a user or to the server there is block storage file storage and object storage these types of storage tie into the available services that are available in google cloud and they offer different options for different types of workloads and i will be going over each of these in a bit of depth and so the first one i wanted to touch on is block storage now block storage is sometimes referred to as block level storage and is a technology that is used to store data files on storage systems or cloud-based storage environments block storage is the fastest available storage type and it is also efficient and reliable with block storage files are split into evenly sized blocks of data each with its own unique identifier it is presented to the operating system as structureless raw data in the form of a logical volume or a hard drive and the operating system structures it with a file system like ext3 or ext4 on linux and ntfs for windows it would then mount this volume or drive as the root volume in linux or a c or d drive in windows block storage is usually delivered on physical media in the case of google cloud it is delivered as either spinning hard drives or solid state drives so in google cloud you're presented with block storage that consists of either persistent disks or local ssd which can both be mountable and bootable block storage volumes can then be used as your boot volumes for compute instances in google cloud installed with your operating system of choice and structured so that your operating system database or application will then be able to consume it now moving on to the second type of storage is file storage now file storage is also referred to as file level or file based storage and is normally storage that is presented to users and applications as a traditional network file system in other words the user or application receives data through directory trees folders and files file storage also allows you to do the same this functions similarly to a local hard drive however a structure has already been applied and cannot be adjusted after the fact this type of structure only has the capabilities of being mountable but not bootable you cannot install an operating system on file storage as i said before the structure has already been put in place for you and is ready for you or your application to consume due to this structure the service that is serving the file system has some underlying software that can handle access rights file sharing file locking and other controls related to file storage in google cloud this service that serves this type of storage is known as cloud file store and is usually presented over the network to users in your vpc network using the nfs protocol or in this case nfs version 3. but i'll be diving into that a little bit later and the last storage type that i wanted to cover is object storage now object storage also referred to as object-based storage is a general term that refers to the way in which we organize and work with units of storage called objects and this is a storage type that is a flat collection of unstructured data and this type of storage holds no structure like the other two types of storage and is made up of three characteristics the first one is the data itself and this could be anything from movies songs and even photos of men in fancy bow ties the data could also be binary data as well the second characteristic is the metadata and this is usually related to any contextual information about what the data is or anything that is relevant to the data and the third characteristic is a globally unique identifier and this way it's possible to find the data without having to know the physical location of the data and this is what allows object storage to be infinitely scalable as it doesn't matter where the object is stored this type of storage can be found in google cloud and is known as cloud storage cloud storage is flat storage with a logical container called a bucket that you put objects into now although this type of storage is not bootable using an open source tool called fuse this storage type can be mounted in google cloud and i will be covering that a little bit later in the cloud storage lesson but in most cases object store is designed as the type of storage that is not bootable or mountable and because of the characteristics of this storage it allows object storage again to be infinitely scalable and so these are the three main types of storage that you will need to know and understand as each has its use cases so if you're looking for high performance storage you will always look to block storage to satisfy your needs if you're looking to share files across multiple systems or have multiple applications that need access to the same files and directories then file storage might be your best bet if you're looking to store terabytes of pictures for a web application and you don't want to worry about scaling object storage will allow you to read and write an infinite amount of pictures that will meet your requirements so now that we've covered these storage types let's take a few moments to discuss storage performance terms now when discussing storage performance there are some key terms to understand that when used together define the performance of your storage first there is io which stands for input output and is a single read write request and can be measured in block size and this block size can vary anywhere from one kilobyte to four megabytes and beyond depending on your workload now q depth when it comes to storage is the number of pending input output requests waiting to be performed on a disk io requests become queued when reads or writes are requested faster than they can be processed by the disk when io requests are queued the total amount of time it takes to read or write data to disk becomes significantly higher this is where performance degradation can occur and queue depth must be adjusted accordingly now the next term is a common touch point when it comes to discussing storage performance on gcp and on the exam which is iops and this is a metric that stands for input output operations per second this value indicates how many different input or output operations a device or group of devices can perform in one second more value in the iops signifies the capability of executing more operations per second and again this is a common touch point that i will be diving into a little bit later now next up is throughput and this is the speed at which the data is transferred in a second and is most commonly measured in megabytes per second this is going to be another common topic that comes up frequently when discussing storage on gcp as well latency is the measurement of delay between the time data is requested when the data starts being returned and is measured in milliseconds so the time each io request will take to complete results in being your average latency and the last two terms i wanted to bring up is sequential and random access sequential would be a large single file like a video and random access would be loading an application or an operating system so lots of little files that are all over the place it's obvious that accessing data randomly is much slower and less efficient than accessing it sequentially and this can also affect performance now why i bring up all these terms is not about calculating the average throughput but to give you a holistic view on storage performance as all these characteristics play a part in defining the performance of your storage there is not one specific characteristic that is responsible for disk performance but all have a role in achieving the highest performance possible for your selected storage now i know this is a lot of theory to take in but this will all start to make more sense when we dive into other parts of the course where we will discuss disk performance with all these characteristics as it relates to compute engine and other services that use storage it is crucial to know the storage types as well as the performance characteristics as it will bring clarity to questions in the exam and also give you a better sense on how to increase your storage performance in your work environment and so that's pretty much all i wanted to cover when it comes to storage types and storage performance as it pertains to storage as a whole so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this lesson i'm going to be covering persistent disks and local ssds i'm going to be getting into the detail with the most commonly used storage types for instances which are both persistent disks and local ssds this lesson will sift through all the different types of persistent disks and local ssds along with the performance of each knowing what type of disk to use for your instance and how to increase disk performance shows up on the exam and so i want to make sure to cover it in detail and leave no stone unturned so with that being said let's dive in now persistent disks and local ssds are the two available types of block storage devices available in google cloud and the determining factor of what you will use for your particular scenario will depend on your use case and the specific characteristics that you require from each storage medium now by default each compute engine instance has a single boot persistent disk that contains the operating system when you require additional storage space you can add one or more additional persistent disks or local ssds to your instance and i will be going through these storage options along with their characteristics now as you can see here persistent disks and local ssds come in a slew of different types as well with persistent disks they are available in both zonal and regional options so starting off with persistent disks you have three different types you can choose from as well you have the flexibility of choosing from two different geographic options when it comes to the redundancy of your persistent disks and i will be covering the zonal and regional options in detail in just a bit now persistent disks are durable network storage devices that your instances can access like physical disks in a computer so these are not physically attached disks but network disks that are connected over google's internal network persistent disks are independent of your instance and can persist after your instance has been terminated and this can be done by turning on this flag upon creation you can even detach your disk and move it to other instances when you need to scaling persistent disks can be done automatically and on the fly by using the disk resize feature and this gives you the flexibility to resize your current persistent disks with no downtime and even add additional disks to your instance for additional performance and storage persistent disks are also encrypted by default and google also gives you the option of using your own custom keys each persistent disk can be up to 64 terabytes in size and most instances can have up to 128 persistent disks and up to 257 terabytes of total persistent disk space attached and just as a note share core machine types are limited to 16 persistent disks and 3 terabytes of total persistent disk space and so now that i've gone through the details of persistent disks i wanted to dive into the two geographic options that's available for persistent disks first starting with zonal now zonal persistent disks are disks that are available in one zone in one region these disks are the most commonly used persistent disks for general day-to-day usage and used for those whose workloads are not sensitive to specific zone outages they are redundant within the zone you've created them in but cannot survive an outage of that zone and may be subjected to data loss if that specific zone is affected and this is where snapshots should be a part of your high availability strategy when using zonal persistent disks snapshots are incremental and can be taken even if you snapshot disks that are attached to running instances and i'll be going into detail about snapshots in a later lesson zonal persistent disks can also be used with any machine type including pre-defined shared core and custom machine types now when it comes to regional persistent disks they have storage qualities that are similar to zonal persistent disks however regional persistent disks provide durable storage and replication of data between two zones in the same region if you are designing systems that require high availability on compute engine you should use regional persistent disks combined with snapshots for durability regional persistent disks are also designed to work with regional managed instance groups in the unlikely event of a zonal outage you can usually fail over your workload running on regional persistent disks to another zone by simply using the force attached flag regional persistent disks are slower than zonal persistent disks and should be taken into consideration when write performance is less critical than data redundancy across multiple zones now noting a couple of caveats here when it comes to disk limits regional persistent disks are similar to zonal persistent disks however regional standard persistent disks have a 200 gigabyte size minimum and may be a major factor when it comes to cost so please be aware as well you can't use regional persistent disks with memory optimized machine types or compute optimized machine types now these two geographic options are available for all three persistent disk types whose characteristics i will dive into now starting off with the standard persistent disk type also known in google cloud as pd standard now these persistent disks are backed by standard hard disk drives and these are your standard spinning hard disk drives and allows google cloud to give a cost effective solution for your specific needs standard persistent disks are great for large data processing workloads that primarily use sequential ios now as explained earlier sequential access would be accessing larger files and would require less work by the hard drive thus decreasing latency as there are physical moving parts in this hard drive this would allow the disc to do the least amount of work as possible and therefore making it the most efficient as possible and therefore sequential ios are best suited for this type of persistent disk and again this is the lowest price persistent disks out of all the persistent disk types now stepping into the performance of standard persistent disks for just a second please remember that iops and throughput performance depends on disk size instance vcpu count and i o block size among other factors and so this table here along with the subsequent tables you will see later are average speeds that google has deemed optimum for these specific disk types they cover the maximum sustained iops as well as the maximum sustained throughput along with the granular breakdown of each here you can see the differences between both the zonal and regional standard pd and as you can see here in the table the zonal standard pd and the regional standard pd are pretty much the same when it comes to most of these metrics but when you look closely at the read iops per instance this is where they differ where the zonal standard pd has a higher read iops per instance than the regional standard pd and this is because the regional standard pd is accessing two different disks in two separate zones and so the latency will be higher the same thing goes for right throughput per instance and so this would be a decision between high availability versus speed moving on to the next type of persistent disk is the balanced persistent disk in google cloud known as pd balance this disk type is the alternative to the ssd persistent disks that balance both performance and cost as this disk type has the same maximum iops as the ssd persistent disk type but holds a lower iops per gigabyte and so this disk is designed for general purpose use the price for this disk also falls in between the standard and the ssd persistent disks so this is basically your middle of the road disk when you're trying to decide between price and speed moving straight into performance i put the standard pd metric here so that you can see a side-by-side comparison between the balance pd and the standard pd and as you can see here when it comes to the metrics under the maximum sustained iops the balance pd is significantly higher than the standard pd in both the zonal and regional options as well looking at the maximum sustained throughput the read write throughput per gigabyte is a little over two times faster and the right throughput per instance is three times faster so quite a bit of jump from the standard pd to the balance pd and moving on to the last persistent disk type is the ssd persistent disk type also known in google cloud as a pd ssd and these are the fastest persistent disks that are available and are great for enterprise applications and high performance databases that demand lower latency and more iops so this would be great for transactional databases or applications that require demanding and near real-time performance the pd ssds have a single digit millisecond latency and because of this comes at a higher cost and therefore is the highest price persistent disk moving on to the performance of this persistent disk this disk type is five times faster when it comes to read iops per gigabyte than the balance pd as well as five times faster for the right iops per gigabyte and so the table here on the left shows the performance for the pd ssd and the table on the right shows the performance of both the standard pd and the balance pd and so here you can see the difference moving from the standard pd over to the ssd pd the read write throughput per instance stays the same from the standard pd all the way up to the ssd pd but where the ssd outperforms all the other ones is through the read write throughput per gigabyte it's one and a half times faster than the balance pd and four times faster than the standard pd and again you will also notice a drop in performance from the zonal option to the regional option and so this is the end of part one of this lesson as it started to get a little bit long and so whenever you're ready you can join me in part two where i will be starting immediately from the end of part one so you can complete this video and i will see you in the next [Music] welcome back this is part two of the persistent disks and local ssds lesson and we will be starting exactly where we left off in part one so with that being said let's dive in and so now that i've covered all the persistent disk types i wanted to move into discussing the characteristics of the local ssd local ssds are physically attached to the server that hosts your vm instance local ssds have higher throughput and lower latency than any of the available persistent disk options and again this is because it's physically attached and the data doesn't have to travel over the network now the crucial thing to know about local ssds is that the data you store on a local ssd persists only until the instance is stopped or deleted once the instance is stopped or deleted your data will be gone and there is no chance of getting it back now each local ssd is 375 gigabytes in size but you can attach a maximum of 24 local ssd partitions for a total of 9 terabytes per instance local ssds are designed to offer very high iops and very low latency and this is great for when you need a fast scratch disk or a cache and you don't want to use instance memory local ssds are also available in two flavors scuzzy and mvme now for those of you who are unaware scuzzy is an older protocol and made specifically for hard drives it also holds the limitation of having one queue for commands nvme on the other hand also known as non-volatile memory express is a newer protocol and is designed for the specific use of flash memory and designed to have up to 64 000 qs as well each of those queues in turn can have up to 64 000 commands running at the same time and thus making nvme infinitely faster now although nvme comes with these incredible speeds it does come at a cost and so when it comes to the caveats of local ssd although compute engine automatically encrypts your data when it's written to local ssd storage space you can't use customer supplied encryption keys with local ssds as well local ssds are only available for the n1 n2 and compute optimized machine types now moving on to the performance of local ssds throughput is the same between scuzzy and nvme but the read write iops per instance is where nvme comes out on top and as you can see here the read iops per instance is a whopping two million four hundred thousand read iops per instance as well the right iops per instance is 1.2 million over the 800 000 for local ssd now before i end this lesson i wanted to cover a few points on performance scaling as it pertains to block storage on compute engine now persistent disk performance scales with the size of the disk and with the number of vcpus on your vm instance persistent disk performance scales linearly until it reaches either the limits of the volume or the limits of each compute engine instance whichever is lower now this may seem odd that the performance of your disk scales with cpu count but you have to remember persistent disks aren't physically attached to your vm they are independently located as such i o on a pd is a network operation and thus it takes cpu to do i o which means that smaller instances run out of cpu to perform disk io at higher rates so in order for you to get better performance you can increase the iops for your disk by resizing them to their maximum capacity but once that size has been reached you will have to increase the number of cpus on your instance in order to increase your disk performance a recommendation by google is that you have one available vcpu for every 2000 to iops of expected traffic so to sum it up performance scales until it reaches either the limits of the disk or the limits of the vm instance to which the disk is attached the vm instance limits are determined by the machine type and the number of vcpus of the instance now if you want to get more granular with regards to disk performance i've included a few links in the lesson text that will give you some more insight but for most general purposes and for the exam remember that persistent disk performance is based on the total persistent disk capacity attached to an instance and the number of vcpus that the instance has and so that's pretty much all i wanted to cover when it comes to persistent disks and local ssds so you can now mark this lesson as complete and let's move on to the next one welcome back in this demo i'm going to be covering how to manage and interact with your disks on compute engine this demo is designed to give you both experience and understanding on working with persistent disks and how you would interact with them we're going to start the demo off by creating an instance we're then going to create a separate persistent disk and attach it to the instance we're going to then interact with the disk and then resize the disk while afterwards we will delete it and we're going to do this all by both using the console and the command line so with that being said let's dive in so here i am in the console i'm logged in as tony bowties gmail.com and i am in project bowtie inc and so the first thing we need to do to kick off this demo is to create an instance that we can attach our disk to but first i always like to make sure that i have a vpc to deploy my instance into with its corresponding default firewall rules so i'm going to head on over to the navigation menu and i'm going to go down to vpc network and as expected my default vpc has been created and just to make sure that i have all my necessary firewall rules i'm going to drill down into the vpc and head on over to firewall rules i'm going to click on firewall rules and the necessary firewall rule that i need for ssh is created and so i can go ahead and create my instance so i'm going to go back up to the navigation menu and i'm going to go over to compute engine so i'm going to go ahead and click on create and i'm going to name this instance bowtie dash instance and for the sake of this demo i'll add in a label here the key is going to be environment and the value will be testing i'm going to go down to the bottom click on save with regards to the region i'm going to select us east 1 and i'm going to keep the zone as the default for us east 1b and under machine type to keep things cost effective i'm going to use an e2 micro shared core machine and i'm going to scroll down to service account and under service account you want to select the set access for each api you want to scroll down to compute engine and here you want to select read write and this will give us the necessary permissions in order to interact with our disk that we will be creating later so i'm going to scroll down to the bottom here and i'm going to leave everything else set at its default and just before creating the instance please do remember you can always click on the command line link where you can get the gcloud command to create this instance through the command line i'm going to close this up and i'm going to simply click on create i'm just going to wait a few seconds here for my instance to come up okay and my instance is up and so now what we want to do is we want to create our new disk so i'm going to go over here to the left hand menu and i'm going to click on disks and as you can see here the disk for the instance that i had just created has 10 gigabytes in us east 1b and we want to leave that alone and we want to create our new disk so i'm going to go up to the top here and simply click on create disk and so for the name of the disk i'm going to call this disk new pd for persistent disk and i'm going to give it the same description i'm going to keep the type as standard persistent disk and for the region i want to select us east one i'm going to keep the zone as its default in us east 1b and as the disk is in us east 1b i'll be able to attach it to my instance and so just as a note here there is a selection where you can replicate this disk within the region if i click that off i've now changed this from a zonal persistent disk to a regional persistent disk and over here in zones it'll give me the option to select any two zones that i prefer and so if you're looking at creating some regional persistent disks these are the steps you would need to take in order to get it done in the console now in order to save on costs i'm going to keep this as a zonal persistent disk so i'm going to click on cancel i'm going to uncheck the option and make sure your region is still set at us east 1 and your zone is selected as us east 1b we're going to leave the snapshot schedule alone and i'll be diving into snapshot schedules in a later lesson i'm going to scroll down here to source type i'm going to keep it as blank disk and the size here is set at 500 gigabytes and we want to set it to 100 gigabytes but before we do that i wanted to bring your attention to the estimated performance here you can see the sustain random iops limits as well as the throughput limit and so depending on the size of the disk that you want to add these limits will change accordingly so if i change this to 100 my sustained random iops limit on read went from 375 iops to 75 iops and so this is a great demonstration that the larger your disc the better your performance and so this is a great way to figure out on what your performance will be before you create your disk and i've also been prompted with a note here saying that because my disk is under 200 gigabytes that i will have reduced performance and so for this demo that's okay i'm going to keep my encryption as the google manage key and under labels i will add environment as the key and value is testing and so now that i've entered all my options i'm going to simply click on create and i'm going to give it a few seconds and my new disk should be created okay and my new disk has been created and you can easily create this disk through the command line and i will be supplying that in the lesson text i merely want to go through the console setup so that you are aware of all the different options and so now that i've created my disk and i've created my instance i want to now log into my instance and attach this new disk so i'm going to go back to vm instances and here i want to ssh into the bowtie instance and i'm going to give it a few seconds here to connect and i'm going to zoom in for better viewing i'm going to clear my screen and so the first thing i want to do is i want to list all my block devices that are available to me on this instance and the linux command for that is ls blk and as you can see my boot disk has been mounted and is available to me and so now i want to attach the new disk that we just created and just as a note i could as easily have done this in the console but i wanted to give you an idea of what it would look like doing it from the command line and so i'm going to paste in the command to attach the disk which is gcloud compute instances attach dash disk the name of the instance which is bow tie dash instance along with the flag dash dash disk the disk name which is new pd and the zone of the disk using the zone flag with us east 1b so i'm going to go ahead and hit enter and no errors came up so i'm assuming that this had worked and so just to double check i'm gonna run the lsblk command again and success as you can see here my block device sdb has been attached to my instance and is available to me with the size of 100 gigabytes and so now i want to look at the state that this roblox device is in and so the command for that will be sudo file dash s followed by the path of the block device which is forward slash dev forward slash sdb i'm going to hit on enter and as you can see it is showing data which means that it is just a raw data device and so in order for me to interact with it i need to format the drive with a file system that the operating system will be able to interact with and so the command to format the drive would be sudo mkfs which is make file system i'm going to use ext4 as the file system minus capital f along with the path of the new disk so i'm going to hit on enter and no errors so i'm assuming that it was successful so just to verify i'm going to run the sudo file minus s command and as you can see here because the disk now has a file system i've been given the information with regards to this disk whereas before it was simply raw data and so now that we've created our disk and we've formatted our disk to a file system that the operating system is able to read we need to now mount the disk and so in order to do that we need to create a mount point so i'm going to first clear the screen and i'm going to run the command sudo mkdir and the new mount point i'm going to call it slash new pd i'm going to hit enter and now i'm going to mount the disk and the command for that is sudo mount the path for the block device which is forward slash dev forward slash sdb and then the mount point which is forward slash new pd i'm going to hit enter no errors so i'm assuming that it had worked but just to verify i'm going to run the command lsblk and success as you can see sdb has now been mounted as new pd and so now i can interact with this disk so the first thing i want to do is i want to change directories to this mount point i'm in now new pd i'm going to do an ls and so just as a note for those of you who are wondering the lost and found directory is found on each linux file system and this is designed to place orphaned or corrupted files or any corrupted bits of data from the file system to be placed here and so it's not something that you would interact with but always a good to know so i'm going to now create a file in new pd so i'm going to run the command sudo nano file a bow ties dot text so file a bow ties is the file that i'm going to create nano is my text editor and so i'm going to hit on enter and so in this file i'm going to type in bow ties are so classy because after all they are i'm going to hit ctrl o to save i'm going to hit enter to verify it and ctrl x to exit so if i do another ls i can see the file of bow ties has been created also by running the command df minus k i'll be able to see the file system here as well and so this is the end of part one of this demo it was getting a bit long so i decided to break it up this would be a great opportunity for you to get up have a stretch get yourself a coffee or tea and whenever you're ready you can join me in the next one where part two will be starting immediately from the end of part one [Music] welcome back this is part two of this demo and we're gonna continue immediately from the end of part one so with that being said let's dive in and so what i want to do now is i want to reboot the instance in order to demonstrate the mounting of this device and i'm going to do that by using the command sudo reboot it's going to disconnect me i'm going to click on close and i'm going to wait about a minute for it to reboot okay and it's been about a minute so i'm going to now ssh into my instance okay and here i am back again logged into my instance i'm going to quickly clear the screen and i'm going to run the lsblk command now what i wanted to demonstrate here is that although i mounted the new device it did not stay mounted through the reboot and this is because there is a configuration file in linux that points to which partitions get mounted automatically upon startup that i need to edit in order to make sure that this device is mounted every time the instance reboots and so in order to do that i need to edit a file called fstab and i'm going to have to add the unique identifier for this partition also known as the device sdb and this will mount the partition automatically every time there happens to be a reboot so in order to do that i'm going to run the command sudo blk id and the path of the block device forward slash dev forward slash sdb i'm going to hit on enter and here is the identifier also known as the uuid that i need to append to the fstab file so i'm going to copy the uuid and i'm going to use the command sudo nano etc fs tab and i'm going to hit on enter and here you will find the uuid for your other partitions and so you're going to be appending a line here right at the end so i'm going to move my cursor down here i'm going to type in uuid equals and then the uuid that i had copied earlier the amount point which is going to be forward slash new pd the type of file system which is ext4 along with defaults comma no fail i'm going to hit control o to save hit enter to verify and control x to exit and so now i'm going to mount this device by running the command sudo mount dash a and hit enter and this command will mount all the partitions that are available in the fstab file and so when i run a lsblk i can see here that my block device sdb is now mounted on forward slash new pd now i know this may be a refresher for some but this is a perfect demonstration of the tasks that need to be done when creating and attaching a new disk to an instance and is a common task for many working on linux instances and working in cloud this can definitely be scripted but i wanted to show you the steps that need to be taken in order to get a new disk in a usable state okay so great we have created a new disk we had attached the disk created a file system and had mounted the disk along with editing the configuration file to make sure that the device mounts whenever the instance starts up so now that we've done all that i wanted to demonstrate resizing this disk from 100 gigabytes to 150 gigabytes and so just to show you where it is in the console i'm going to quickly go back to my console tab and so here i'm going to go to the left hand menu i'm going to click on disks i'm going to drill down into new pd and at the top i'm going to click on edit and so here i'm able to adjust the disk space size and simply click on save not much that i really need to do here but i did want to show you how to do this in the command line so i'm going to go back to the tab of my instance and i'm going to quickly clear the screen and i'm going to paste in the command gcloud compute disks resize the name of the disk which is new pd and the new size in gigabytes using the dash dash size flag 150 which is the new size of the disc along with the dash dash zone flag of us east 1b i'm going to hit enter it's going to ask me if i want to do this as this is not reversible and please remember when you resize a disk you can only make it bigger and never smaller so i'm going to hit y to continue and it took a few seconds there but it was successful so if i run a df minus k you can see here that i only have 100 gigabytes available to me and this is because i have to extend the file system on the disk so i've made the disk larger but i haven't allocated those raw blocks to the file system so in order for the file system to see those unallocated blocks that's available to it i need to run another command so i'm going to quickly clear my screen again and i'm going to run the command sudo resize to fs along with the block device i'm going to hit enter and as you can see it was successful showing the old blocks as 13 and the new blocks as 19. so if i run a df minus k i can now see my 150 gigabytes that's available to me and so just to demonstrate after resizing the disk along with mounting and then remounting the disk that the file that i've created still exists i'm going to run an ls minus al but first i will need to change directories into new pd clear my screen and run an ls and phyla bow ties is still there and so this is a great example demonstrating how the data on persistent disks persist through the lifetime of a disk even when mounting unmounting rebooting and resizing and so as you can see we've done a lot of work here and so just as a recap where we've created a new disk we attached this disk to an instance we formatted the disk into an ext4 file system we've mounted this disk we've written a file to it added its unique identifier to the configuration file so that it mounts on startup and then we've resized the disk along with extending the file system on the disk and so this is the end of the demo and i wanted to congratulate you on making it to the end and i hope this demo has been extremely useful and again fantastic job on your part now before you go i wanted to quickly walk through the steps of deleting all the resources you've created and so the first thing that i want to do is delete the disk that was created for this demo and so before i can delete the disk i'm going to first detach the disk from the instance and the easiest way to do that is through the command line so i'm going to quickly clear my screen and so i'm going to show you how to detach the disk from the instance and so i'm going to paste in this command gcloud compute instances detach disk the instance name which is bow tie dash instance along with the disc with the flag dash dash disc the name of the disc which is new pd along with the zone i'm going to hit enter and it's been successfully detached and so now that it's detached i can actually delete the disk and so i'm going to head on over back to the console and i'm going to go ahead and delete the new pd disk i'm going to click on delete i'm going to get a prompt asking me if i'm sure yes i am if i go back to the main menu for my disks and this should just take a moment and once it's deleted you will no longer see it here and i'm going to go back over to vm instances and i'm going to delete this as well and so there's no need to delete your default vpc unless you'd like to recreate it again but don't worry for those who decide to keep it you will not be charged for your vpc as we will be using it in the next demo and so that's pretty much all i wanted to cover when it comes to managing disks with compute engine so you can now mark this as complete and let's move on to the next one [Music] welcome back in this lesson i'll be discussing persistent disk snapshots now snapshots are a great way to backup data from any running or stopped instances from unexpected data loss snapshots are also a great strategy for use in a backup plan for any and all instances no matter where they are located and so as cloud engineers and architects this is a great tool for achieving the greatest uptime for your instances so diving right into it snapshots as i mentioned before are a great way for both backing up and restoring the data of your persistent disks you can create snapshots from disks even while they are attached to running instances snapshots are global resources so any snapshot is accessible by any resource within the same project you can also share snapshots across projects as well snapshots also support both zonal and regional persistent disks snapshots are incremental and automatically compressed so you can create regular snapshots on a persistent disk faster and at a much lower cost than if you regularly created a full image of a disk now when you create a snapshot you have the option of choosing a storage location snapshots are stored in cloud storage and can be stored in either a multi-regional location or a regional cloud storage bucket a multi-regional storage location provides higher availability but will drive up costs please be aware that the location of a snapshot affects its availability and can incur networking costs when creating the snapshot or restoring it to a new disk if you do not specify storage location for a snapshot google cloud uses the default location which stores your snapshot in a cloud storage multi-regional location closest to the region of your source disk if you store your snapshot in the same region as your source disk there is no network charge when you access that snapshot from the same region if you access the snapshot from a different region you will incur a network cost compute engine stores multiple copies of each snapshot across multiple locations as well you cannot change the storage location of an existing snapshot once a snapshot has been taken it can be used to create a new disk in any region and zone regardless of the storage location of the snapshot now as i explained earlier snapshots are incremental and i wanted to take a moment to dive into that for just a minute so when creating snapshots the first successful snapshot of a persistent disk is a full snapshot that contains all the data on the persistent disk the second snapshot only contains any new data or modify data since the first snapshot data that hasn't changed since snapshot 1 isn't included instead snapshot 2 contains references to snapshot 1 for any unchanged data as shown here snapshot 3 contains any new or changed data since snapshot 2 but won't contain any unchanged data from snapshot 1 or 2. instead snapshot 3 contains references to blocks in snapshot 1 and snapshot 2 for any unchanged data this repeats for all subsequent snapshots of the persistent disk snapshots are always created based on the last successful snapshot taken and so now you're probably wondering what happens when you decide to delete a snapshot are they dependent on each other well when you delete a snapshot compute engine immediately marks the snapshot as deleted in the system if the snapshot has no dependent snapshots it is deleted outright however if the snapshot does have dependent snapshots then there are some steps that happen behind the scenes so shown here in this diagram snapshot 2 is deleted the next snapshot from the full snapshot no longer references the snapshot for deletion in this example snapshot 1 then becomes the reference for snapshot 3 and any data that is required for restoring other snapshots is moved into the next snapshot increasing its size shown here blocks that were unique to snapshot 2 are moved to snapshot 3 and the size of snapshot 3 increases any data that is not required for restoring other snapshots is deleted so in this case blocks that are already in snapshot 3 are deleted from snapshot 2 and the size of all snapshots are lower now because subsequent snapshots might require information stored in a previous snapshot please be aware that deleting a snapshot does not necessarily delete all the data on the snapshot if you're looking to make sure that your data has indeed been deleted from your snapshots you should delete all snapshots if your disk has a snapshot schedule you must detach the snapshot schedule from the disk before you can delete the schedule removing the snapshot schedule from the disk prevents further snapshot activity from occurring now touching on the topic of scheduled snapshots by far the best way to backup your data on compute engine is to use scheduled snapshots this way you will never have to worry about manually creating snapshots or even worry about using other tools to kick off those snapshots you can simply use this built-in tool by google which is why snapshot schedules are considered best practice to backup any compute engine persistent disks now in order to create any snapshot schedules you must create your snapshot schedule in the same region where your persistent disk resides now there are two ways to create a snapshot schedule the first one is to create a snapshot schedule and then attach it to an existing persistent disk the other way is to create a new persistent disk with a snapshot schedule you also have the option of setting up a snapshot retention policy that defines how long you want to keep your snapshots some options when creating snapshot schedules are both retention policies and source disk deletion rules now if you choose to set up a snapshot retention policy you must do it as part of your snapshot schedule when you create a snapshot schedule is when you can also set a source disk deletion rule the source disk deletion rule controls what happens to your snapshots if the source disk is deleted now a few caveats here on the scheduled snapshots is that a persistent disk can only have one snapshot schedule attached to it at a time also you cannot delete a snapshot schedule if it is attached to a disk you must detach the schedule from all disks then delete the schedule as well after you create a snapshot schedule you cannot edit it to update a snapshot schedule you must delete it and create a new one now before i end this lesson i wanted to touch on managing snapshots for just a minute so when managing snapshots there's a few things to remember in order to use snapshots to manage your data efficiently you can snapshot your disks at most once every 10 minutes you are unable to snapshot your disks at intervals less than 10 minutes so please keep that in mind when creating your schedules also you should create snapshots on a regular schedule to minimize data loss if there was an unexpected failure if you have existing snapshots of a persistent disk the system automatically uses them as a baseline for any subsequent snapshots that you create from that same disk so in order to improve performance you can eliminate excessive snapshots by creating an image and reusing it using this method would not only be ideal for storage and management of snapshots but also help to reduce costs and if you schedule regular snapshots for your persistent disks you can reduce the time that it takes to complete each snapshot by creating them during off-peak hours when possible and lastly for those of you who use windows for most situations you can use the volume shadow copy service to take snapshots of persistent disks that are attached to windows instances you can create vss snapshots without having to stop the instance or detach the persistent disk and so that's pretty much all i wanted to cover when it comes to the theory of persistent disk snapshots their schedules and how to manage them in the next lesson i'll be doing a hands-on demo demonstrating snapshots and putting this theory into practice and get a feel for how snapshots work and how they can be applied to persistent disks so you can now mark this lesson as complete and whenever you're ready join me in the console [Music] welcome back in this demonstration we're going to dive into snapshots and snapshot schedules this demo will give you the hands-on knowledge you need to create and delete snapshots along with how to manage snapshot schedules we're going to start the demo off by creating an instance we're going to interact with it and then take a snapshot of the disk we're going to then create another instance from the snapshot and then create some snapshot schedules for both of these instances by using both the console and the command line so there's a lot to do here so with that being said let's dive in and so i'm currently logged in as tony bowties gmail.com as well i'm in project bowtie inc so the first thing that we need to do to kick off this demo is to create an instance but first as always i like to make sure that i have a vpc to deploy my instance into with its corresponding default firewall rules and so i'm going to head on over to the navigation menu and scroll down to vpc network and because i didn't delete my default vpc from the last demo i still have it here i'm just going to drill down and make sure that i have my firewall rules i'm gonna go over to firewall rules and as expected the ssh firewall rule that i need has already been created and so now that i have everything in order i'm gonna go back over to the navigation menu and head on over to compute engine to create my instance now i figure for this demo i'd switch it up a little bit and create the instance by the command line so i'm going to head on over to cloud shell i'm going to open that up and it took a minute to provision and so what i'm going to do now is i'm going to open it up in a new tab i'm going to zoom in for better viewing and i'm going to paste in my command to create my instance and this gcloud command to create these instances will be available in the github repository and you will find all the instructions and the commands under managing snapshots in compute engine so i'm going to hit enter and you may get a prompt to authorize this api call and i'm going to click on authorize and success our instance has been created and is up and running and so now what i want to do is ssh into the instance and so i'm just going to run the command from here which is gcloud compute ssh dash dash zone the zone that i'm in which is used 1b and the instance which is bowtie dash instance i'm going to hit enter it's going to prompt me if i want to continue i'm going to say yes and i'm going to enter my passphrase and enter it again it's going to update my metadata and it's going to ask me again for my passphrase and i'm in so i'm going to just quickly clear my screen and so the first thing i want to do is i want to verify the name of my instance so i'm going to type in the command hostname and as expected bowtie dash instance shows up and so now i want to create a text file and so i'm going to run the command sudo nano file a text i'm going to hit enter and it's going to open up my nano text editor and you can enter a message of any kind that you'd like for me i'm going to enter more bow tie needed because you can never get enough bow ties i'm going to hit ctrl o to save press enter to verify the file name to write and then ctrl x to exit i'm going to run the command ls space minus al to list my files so i can verify that my file has been created and as you can see here file a bowties.txt has been created and so now that i've created my instance and i've written a file to disk i'm going to now head on over to the console and take a snapshot of this disk and because my session was transferred to another tab i can now close the terminal and you want to head over to the left-hand menu and go to disks and so now i want to show you two ways on how you can create this snapshot the first one is going to disks and choosing the disk that you want for me it's bowtie instance and under actions i'm going to click on the hamburger menu and here i can create snapshot and this will bring me straight to my snapshot menu but for this demo i'm going to go over to the left hand menu and i'm going to click on snapshots and here i'm going to click on create snapshot and so for the name of the snapshot i'm going to type in bowtie snapshot and i'm going to use the same for the description moving down on the source disk the only one that i can select is bow tie instance and that's the one that i want anyways so i'm going to click on that the location in order to cut down on costs we don't need multi-regional we're going to just select regional and if you select on the location i'm able to select any other locations like tokyo and i can create my snapshot in tokyo but i want to keep my snapshot in the same region so i'm going to go back and select us east one where it is based on the source disk location and i'm going to add a label here with the key environment and the value of testing i'm going to leave my encryption type as google managed and i'm going to simply click on create and this will create a snapshot of the boot disk on bow tie instance and that took about a minute there and so just as a note if you have any bigger discs they will take a little bit longer to snapshot okay and now that i've created my snapshot i'm going to go back up to vm instances and i'm going to create a new instance from that snapshot and so i'm going to name this instance bowtie dash instance dash 2 and i'm going to give this a label i'm going to add a label here the key of environment and the value of testing and hit save the region is going to be used 1 and you can leave the zone as its default as us east 1b and under machine type you can select the e2 micro and you want to go down to boot disk and select the change button and here i'm going to select snapshots instead of using a public image so i'm going to click on snapshots and if i select the snapshot drop down menu i will see here my bowtie snapshot so i'm going to select this i'm going to leave the rest as default and i'm going to go down to select and i'm going to leave everything else as its default and i'm going to click on create i'm going to just give it a minute here so bowtie instance 2 can be created okay and it took a minute there so now i'm going to ssh into this instance and i'm going to zoom in for better viewing and even though i know the instance is named bowtie.instance2 i'm still going to run the hostname command and as expected the same name pops up but what i was really curious about is if i run the command ls space dash al i can see here my file of file of bowties.text and if i cat the file [Music] i'll be able to see the text that i inputted into that file and so although it was only one file and a text file at that i was able to verify that my snapshot had worked as there will be times where your snapshot can get corrupted and so doing some various spot checks on your snapshots is some good common practice and so now i want to create a snapshot schedule for both of these instances and so i'm going to go back to the console and on the left hand menu i'm going to head down to snapshots and if i go over to snapshot schedules you can see that i have no snapshot schedules so let's go ahead and create a new one by clicking on create snapshot schedule and so as mentioned in the last lesson we need to create this schedule first before we can attach it to a disk and so i'm going to name this snapshot schedule as bow tie dash disk schedule i'm going to use the same for the description the region i'm going to select it as us east one and i'm going to keep the snapshot location as regional under us east one you scroll down here and under schedule options you can leave the schedule frequency as daily and just as a note for start time this time is measured in utc so please remember this when you're creating your schedule in your specific time zone and so i'm going to put the start time as o 600 and this will be 1 am eastern standard time as backups are always best done when there is the least amount of activity and i'm going to keep the auto delete snapshots after 14 days i'm going to keep the deletion rule as keep snapshots as well i can enable the volume shadow copy service for windows but since we're running linux i don't need to enable this and since we labeled everything else i might as well give this a label i'm going to use the key as environment and the value of testing and once you've filled everything out then you can simply click on create and it took a minute there but the schedule was created and so now that i have my snapshot schedule i need to attach it to a disk so i'm going to head on over to the left hand menu and click on disks and here i'm going to drill down into bow tie instance i'm going to go up to the top and click on edit and under snapshot schedule i'm going to click on the drop down and here i will find bow tie disk schedule i'm going to select that i'm going to click on save and so now that i have my snapshot schedule attached to my disk for the bowtie instance instance i now want to create a snapshot schedule for my other instance and so instead of using the console i'm going to go ahead and do it through the command line so i'm going to go up to the top to my open shell and i'm going to quickly clear the screen and so in order to create my schedule i'm going to run this command gcloud compute resource policies create snapshot schedule the name of the snapshot schedule which is bow tie disk schedule 2 the region the maximum retention days the retention policy and the schedule followed by the storage location and like i said before these commands you will find in the github repository so i'm going to go ahead and hit enter and so i wanted to leave this error in here to show you that i needed the proper permissions in order to create this snapshot schedule a great reminder to always check if you have the right role for the task at hand and so i have two options i can either change users from my service account user to tony bowtie or i can simply head on over to my instance and edit the service account permissions and so the easiest way to do it would be to just switch users and so i'm going to go ahead and do that so i'm going to go ahead and run the command gcloud auth login and remember that this is something that you don't have to do i merely wanted to show you that you require the proper permissions on creation of specific resources okay and i quickly went through the authentication process i'm gonna just clear my screen and i'm going to go ahead and run the command again and as expected the snapshot schedule was created with no errors and so now that my schedule has been created i can now attach it to the disk so i'm going to run the command gcloud compute disks add resource policies the instance name which is bowtie instance 2 and the resource policy which is the snapshot schedule named as bowtie disk schedule 2 in the zone of us east 1b i'm going to hit enter and success and so just to verify that the snapshot schedule has been attached to my disk i'm going to go back to the console i'm going to head back on over to the main page of disks i'm going to drill down into bow tie instance 2 and here it is the snapshot schedule has been attached and so i want to congratulate you on making it to the end of this demo and i hope this demo has been useful as snapshots in the role of an engineer is a common task that can save you from any data loss once set into place and so just as a recap you've created an instance you created a file on that instance and then you've created a snapshot of the disk of that instance and used it to create another instance you then verified the snapshot and then created a snapshot schedule for both boot disks of the instances using the console and the command line well done on another great job now before you go i wanted to take a moment to clean up any resources we've used so we don't accumulate any costs and so the first thing we want to do is we want to detach the snapshot schedules from the disks and so since we're in bow tie instance 2 i'm going to go ahead and click on edit under snapshot schedule i'm going to select the no schedule hit save and i'm going to do the same thing with my other disk now i'm going to head back on over to snapshots i'm going to delete this snapshot and i'm going to head back on over to snapshot schedules i'm going to select all the snapshot schedules and i'm going to click on delete and now that everything's cleaned up with regards to snapshots and snapshot schedules i can now go over to vm instances and delete the instances i'm going to select them all and simply click on delete and so that's pretty much all i wanted to cover in this demo when it comes to snapshots and snapshot schedules so you can now mark this as complete and let's move on to the next one welcome back in this lesson we're going to switch gears and take an automated approach to deployment by diving into google's tool for infrastructure as code called deployment manager now deployment manager allows you to deploy update and tear down resources from within google cloud using yaml jinja and python code templates it allows you to automate the deployment of all the resources that are available in google cloud and deploy it in a fast easy and repeatable way for consistency and efficiency in this lesson we're going to explore the architecture of deployment manager and dive into all the different components that gives it its flexibility and the features that make this tool an easy solution for deploying complex environments so with that being said let's dive in now breaking down the components that i mentioned earlier i wanted to start off with the first component being the configuration now a configuration defines the structure of your deployment as you must specify a configuration to create a deployment a configuration describes all the resources you want for a single deployment and is written in yaml syntax that lists each of the resources you want to create and its respective resource properties a configuration must contain a resources section followed by the list of resources to create and so each resource must contain these three components the name the type and properties without these three components a deployment will not instantiate and so i wanted to take a moment to go over these three components in a bit of depth so the first component of the configuration is the name and the name is a user defined string to identify this resource and can be anything you choose from names like instance one my-vm bowtie dash instance and you can even go as far to use larks dash instance dash don't dash touch and the syntax can be found here and must not contain any spaces or invalid characters next component in a configuration is type and there are a couple of different types that you can choose from a type can represent a single api source known as a base type or a set of resources known as a composite type and either one of these can be used to create part of your deployment the type of the resource being deployed here in this diagram is shown as a base type of compute.v1.instance and there are many other api resources that can be used such as compute.v1.disk app engine dot v1 as well as bigquery.v2 and the syntax is shown here as api dot version dot resource now a composite type contains one or more templates that are pre-configured to work together these templates expand to a set of base types when deployed in a deployment composite types are essentially hosted templates that you can add to deployment manager the syntax is shown here as gcp dash types forward slash provider colon resource and to give you an example of what a composite type looks like here is shown the creation of a reserved ip address using the compute engine v1 api and you could also use composite types with other apis in the same way such as gcp dash types forward slash app engine dash v1 colon apps or bigquery v2 colon data sets and for the last component in a configuration is properties and this is the parameters for the resource type this includes all the parameters you see here in this example including the zone machine type the type of disk along with its parameters pretty much everything that gives detail on the resource type now just as a note they must match the properties for this type so what do i mean by this so let's say you entered a zone but that particular zone doesn't exist or that compute engine machine type doesn't exist in that zone you will end up getting an error as deployment manager will not be able to parse this configuration and thus failing deployment so make sure when you add your properties that they match those of the resource now a configuration can contain templates which are essentially parts of the configuration file that have been abstracted into individual building blocks a template is a separate file that is imported and used as a type in a configuration and you can use as many templates as you want in a configuration and allow you to separate your configuration out into different pieces that you can use and reuse across different deployments templates can be as generalized or specific as you need and they also allow you to take advantage of features like template properties environment variables and modules to create dynamic configuration as shown here templates can be written in a couple of different ways they can be written in either ginger 2.1 or python 3. the example shown on the left has been written in ginger and is very similar to the yaml syntax so if you're familiar with yaml this might be better for you the example on the right has been written in python and is pretty amazing as you can take advantage of programmatically generating parts of your templates if you are familiar with python this might be a better format for you now one of the advantages of using templates is the ability to create and define custom template properties template properties are arbitrary variables that you define in template files any configuration file or template file that uses the template in question can provide a value for the template property without changing the template directly this lets you abstract the property so that you can change the property's value for each unique configuration without updating the underlying template and just as a note deployment manager creates predefined environment variables that you can use in your deployment in this example the project variable will use the project id for this specific project and so combining all these components together will give you a deployment and so a deployment is a collection of resources that are deployed and managed together using a configuration you can then deploy update or delete this deployment by merely changing some code or at the click of a button now when you deploy you provide a valid configuration in the request to create the deployment a deployment can contain a number of resources across a number of google cloud services when you create a deployment deployment manager creates all of the described resources to deploy a configuration it must be done through the command line and cannot be done through the console you can simply use the syntax shown here and a deployment will be instantiated from the configuration file that you have entered where bow tie deploy is the name of the deployment and the file after the dash dash config is your configuration file google cloud also offers pre-defined templates that you can use to deploy from the gcp marketplace and can be found right in the console of deployment manager this way all the configuration and template creation is handled for you and you just deploy the solution through the console now after you've created a deployment you can update it whenever you need to you can update a deployment by adding or removing resources from a deployment or updating the properties of existing resources in a deployment a single update can contain any combination of these changes so you can make changes to the properties of existing resources and add new resources in the same request you update your deployment by first making changes to your configuration file or you can create a configuration file with the changes you want you will then have the option to pick the policies to use for your updates or you can use the default policies and finally you then make the update request to deployment manager and so once you've launched your deployment each deployment has a corresponding manifest as the example shown here a manifest is a read-only property that describes all the resources in your deployment and is automatically created with each new deployment manifests cannot be modified after they have been created as well it's not the same as a configuration file but is created based on the configuration file and so when you delete a deployment all resources that are part of the deployment are also deleted if you want to delete specific resources from your deployment and keep the rest delete those resources from your configuration file and update the deployment instead and so as you can see here deployment manager gives you a slew of different options to deploy update or delete resources simultaneously in google cloud now like most services in gcp there are always some best practices to follow note that there are many more best practices to add to this and can be found in the documentation which i will be providing the link to in the lesson text but i did want to point out some important ones to remember so the first one i wanted to bring up is to break your configurations up into logical units so for example you should create separate configurations for networking services security services and compute services so this way each team will be able to easily take care of their own domain without having to sift through a massive template containing the code to the entire environment another best practice to follow is to use references and references should be used for values that are not defined until a resource is created such as resources self-link ip address or system generated id without references deployment manager creates all resources in parallel so there's no guarantee that dependent resources are created in the correct order using references would enforce the order in which resources are created the next one is to preview your deployments using the preview flag so you should always preview your deployments to assess how making an update will affect your deployment deployment manager does not actually deploy resources when you preview a configuration but runs a mock deployment of those resources instead this gives you the opportunity to see the changes to your deployment before committing to it you also want to consider automating the creation of projects as well as automating the creation of resources contained within the projects and this enables you to adopt an infrastructure as code approach for project provisioning this will allow you to provide a series of predefined project environments that can be quickly and easily provisioned it will also allow you to use version control to manage your base project configuration and it will also allow you to deploy reproducible and consistent project configurations and lastly using a version control system as part of the development process for your deployments is a great best practice to follow as it allows you to fall back to a previous known good configuration it provides an audit trail for changes as well it uses the configuration as part of a continuous deployment system now as you've seen here in this lesson deployment manager can be a powerful tool in your tool belt when it comes to implementing infrastructure as code and it has endless possibilities that you can explore on your own it can also provide a massive push towards devops practices and head down the path of continuous automation through continuous integration continuous delivery and continuous deployment and so that's pretty much all i wanted to cover when it comes to deployment manager and so whenever you're ready join me in the next one where we will go hands-on in a demonstration to deploy a configuration in deployment manager so you can now mark this lesson as complete and whenever you're ready join me in the console [Music] welcome back in this demonstration we're gonna go hands-on with deployment manager and deploy a small web server we're gonna first use the google cloud editor to copy in our code and we're gonna then do a dry run and then finally deploy our code we're gonna then do a walkthrough of deployment manager in the console and go through the manifest as well as some of the other features we're then going to verify all the deployed resources and we get to do an easy cleanup in the end by hitting the delete button and taking care of removing any resources that were created so there's quite a bit to go through here and so with that being said let's dive in and so as you can see here i am logged in as tonybowties gmail.com in the project of bowtie inc now since we're going to be doing most of our work in code the first thing that we want to do is go to the google cloud editor so i'm going to go up here to the top and open up cloud shell and i'm going to then click on the button open editor i'm going to make this full screen for better viewing and so in order to get the terminal in the same viewing pane as the editor i'm going to simply go up to the top menu and click on terminal and select new terminal now for better viewing and this is totally optional for you i'm going to change the color theme into a dark mode and so i'm going to go up to the menu click on file go down to settings and go over to color theme and i'm going to select dark visual studio and for those of you who are working in visual studio code this may look very familiar to you and i'm also going to increase the font size by again going back up to file over to settings and then over to open preferences here under workspace and then scroll down to terminal and if you scroll down to integrated font size i'm going to adjust the font size to 20 for better viewing and my cloud shell font size is a little bit easier to see and so once you've done that you can then close the preferences tab and we're now ready to create files in our editor okay so next up i want to create a folder for all my files to live in so i'm going to go up to the menu here i'm going to select on file and select new folder and i'm going to rename this folder as templates and hit ok and so now that we have the folder that all of our files are going to live in the next step is to open up the github repository in your text editor and have your files ready to copy over and so just as a note for those who are fluent in how to use git you can use this new feature in the cloud shell editor to clone the course repo without having to recreate the files so i'm going to go over my text editor and make sure that you've recently done a git pull we're going to open up the files under compute engine deployment manager and you'll see templates with a set of three files and i've already conveniently opened them up i'm going to go up to bow tie deploy.yaml and this is going to be the configuration file that i'm going to be copying over and once i finish copying all these files over i'll be going through this in a little bit of detail just so you can understand the format of this configuration and so i'm going to select all of this i'm going to copy this head back on over to the editor and here i'm going to select file new file so i'm going to rename this as bow tie dash deploy dot yaml hit okay and i'm going to paste in my code and so this configuration file is showing that i'm going to be importing two templates by the name of bowtie.webserver.jinja as well as bowtie.network.jinja so i'm going to have a template for my web server and a template for the network and under resources as you can see this code here will create my bow tie dash web server the type is going to be the template the properties will have the zone the machine type as well as a reference for the network as well underneath the bowtie web server is the bowtie network and again this is pulling from type bowtie.network.jinja so this is a another template file and under the properties we have the region of us east one and so we're going to copy over these two templates bowtie web server and bowtie network as we need both of these templates in order to complete this deployment and so i'm going to go ahead and do that now head back on over to my code editor i'm going to go to bowtie web server i'm going to copy everything here back to my editor and i'm going to create the new file called bowtie web server it's going to be dot jinja hit enter i'm going to paste the code in and just to do a quick run through of the template the instance name is going to be bow tie dash website the type is compute.v1.instance and as you can see here we are using a bunch of different properties here under zone we have property zone which is going to reference back to the yaml template here under zone you will see us east 1b and so this way if i have to create another web server i can enter whatever zone i like here in the configuration file and leave the bow tie dash web server template just the way it is under machine type i have variables set for both the zone and machine type under disks i'm going to have the device name as an environment variable and it's going to be a persistent disk and the source image is going to be debian9 i also put in some metadata here that will bring up the web server and lastly i have a network tag of http server as well as the configuration for the network interface the network referring to bowtie dash network and a sub network called public which i will be showing to you in just a moment and as well the access configs of the type one to one nat and this will give the instance a public ip address and so now that we've gone through that template we need to create one last template which is the bowtie dash network so i'm going to head back on over to my code editor and open up bowtie network select the code copy it back over to cloud editor and i'm going to create a new file call this bowtie network dot jinja hit enter paste in my code and to quickly walk you through this we're going to be creating a new custom network called bow tie dash network the type is going to be compute.v1.network as the vpc uses the compute engine api it's going to be a custom network so the value of the auto create sub networks is going to be false the name is going to be public here we have the custom ipcider range and you can also use this as a variable but for this demo i decided to just leave it under network i have a reference to the bowtie network the value for private google access is false and the region variable is fulfilled through the configuration file moving right along i have two firewall rules here one for ssh access and the other for web server access one opening up port 22 to the world as well as port 80. as well the web server access firewall rule has a target tag of http server referencing back to the network tag of the bowtie web server instance okay and so now we've finished creating the configuration file along with the templates so i'm going to head back on up to the menu click on file and select save all and since we've finished creating all of our files the next thing to do is to execute a mock deploy using the bowtie deploy configuration but first i know that we haven't used deployment manager before and so i need to go in and turn on the api and so i'm just going to go up here to the top to the search bar and i'm going to type in deployment and you should see deployment manager as the first result and bring this down a little bit and as expected the deployment manager api has not been enabled yet so i'm going to click on enable and after a few moments we should be good to go okay and as you can see here deployment manager is pretty empty as most of it is done through the command line but if you're looking to deploy a marketplace solution you can do that right here at the top and this will bring you right to the marketplace and will allow you to deploy from a large selection of pre-configured templates but i don't want to do that and so i'm just going to bring this up a little bit and i'm going to head on over to the terminal i'm going to run an ls i'm going to run the command ls and you should be able to see the templates folder i'm going to change my directory into the templates folder do another ls and here are all my files and so before we do a mock deploy of this configuration we want to make sure that we're deploying to the correct project i can see here that i am currently in bow tie inc but if you are ever unsure about the project that you're in you can always run the gcloud config list command in order to confirm so i'm going to quickly clear my screen and i'm going to run the command gcloud config list it's going to prompt me to authorize this api call and i'm going to authorize and as expected my project is set to deploy in project bowtie inc and so now that i've verified it i'm going to quickly clear my screen again and so i'm going to paste in my command gcloud deployment dash manager deployments create bowtie deploy which is the name of the deployment along with the configuration file flag dash dash config and then the name of the configuration file which is bowtie deploy.yaml and the preview flag as we're only doing a mock deploy and so if there are any errors i'll be able to see this before i actually deploy all the resources so i'm going to go ahead and hit enter and in just a minute we'll find out exactly what happens and as you can see here the mock deployment was a success and there are no errors and if i do a quick refresh up here in the console i'll be able to see my deployment which i can drill down into and here i will see my manifest file with my manifest name and i can view the config as well as my templates that it imported the layout as well as the expanded config so if i click on view of the config it'll show me here in the right hand panel exactly what this deployment has used for the config and i can do the same thing with my template files so i'm going to open up my network template and i can quickly go through that if i'd like as well i also have the option to download it and if i really want to get granular i can go over here to the left hand pane i can select on vm instance and it'll show me all the resource properties everything from the disks to the machine type to the metadata the network interfaces the zone that it's in and the network tag same thing if i go over here to the network and again because this is a custom network the value for the autocreate subnetworks is false i can check on the public sub network as well as the firewall rules and so because this is a preview it has not actually deployed anything now taking a look at compute engine instances in a new tab you can see here that i have no instances deployed and so the same goes for any of the other resources and so what we want to do now is we want to deploy this deployment and we can do that one of two ways we can simply click on the button here that says deploy or we can run the command in the command line and so i'm looking to show you how to do it in the command line so i'm going to move down to the command line i'm going to quickly clear my screen i'm going to paste in the code which is gcloud deployment dash manager deployments update bowtie deploy now you're probably wondering why update and this is because the configuration has been deployed even though it's a preview deployment manager still sees it as a deployment and has created what google cloud calls a shell and so by using update you can fully deploy the configuration using your last preview to perform that update and this will deploy your resources exactly how you see it in the manifest and so anytime i make an adjustment to either the configuration or the templates i can simply run the update command instead of doing the whole deployment again so i want to get this deployed now and so i'm going to hit enter and i'll be back in a minute once it's deployed all the resources and success my deployment is successful and as you can see here there are no errors and all the resources are in a completed state so i'm going to select my bow tie website in my manifest and i'll have access to the resource with a link up here at the top that will bring me to the instance as well i can ssh into the instance and i have all the same options that i have in the compute engine console and so in order to verify that all my resources have been deployed i'm going to go back over to the tab that i already have open and as you can see my instance has been deployed and i want to check to see if my network has been deployed so i'm going to go up to the navigation menu and i'm going to head on down to vpc network and as you can see here bowtie network has been deployed with its two corresponding firewall rules i'm going to drill down into bowtie network and check out the firewall rules and as you can see here ssh access and web server access have been created with its corresponding protocols and ports and so now that i know that all my resources have been deployed i want to head back on over to compute engine to see if my instance has been configured properly so i'm going to click on ssh to see if i can ssh into the instance and success with ssh so i know that this is working properly and so i'm going to close this tab down and i also want to see whether or not my web server has been configured properly with the metadata that i provided it and so i can directly open up the webpage by simply clicking on this link and success my you look dapper today why thank you tony bowtie and so as you can see the web server has been configured properly using the metadata that i provided so i wanted to congratulate you on making it to the end of this demo and hope it has been extremely useful and gave you an understanding of how infrastructure is code is used in google cloud using their native tools i hope this also triggered some possible use cases for you that will allow you to automate more resources and configurations in your environment and allow you to start innovating on fantastic new ways for cicd for those of you who are familiar with infrastructure as code this may have been a refresher but will give you some insight for questions on the exam that cover deployment manager and just as a quick note for those of you who are looking to learn more about infrastructure as code i have put a few links in the lesson text going into depth on deployment manager and another tool that google recommends called terraform and so now before you go we want to clean up all the resources that we've deployed to reduce any incurred costs and because deployment manager makes it easy we can do it in one simple step so i'm going to head back on over to my open tab where i have my console open to deployment manager and i'm going to head on over to the delete button and simply click on delete now deployment manager gives me the option of deleting all the resources it created or simply deleting the manifest but keeping the resources untouched and so you want to select delete bowtie deploy with all of its resources and simply click on delete all and this will initiate the teardown of all the resources that have been deployed from the bowtie deploy configuration and this will take a few minutes to tear down but if you ever have a larger configuration to deploy just as a note it may take a little bit longer to both deploy and to tear down and so just as a recap you've created a configuration file and two templates in the cloud shell editor you then deployed your configuration using deployment manager through the command line in cloud shell you then verified each individual resource that was deployed and verified the configuration of each resource congratulations again on a job well done and so that's pretty much all i wanted to cover in this demo when it comes to deploying resources using deployment manager so you can now mark this as complete and let's move on to the next one [Music] welcome back and in this lesson we're going to learn about google cloud load balancing and how it's used to distribute traffic within the google cloud platform google cloud load balancing is essential when using it with instance groups kubernetes clusters and is pretty much the defacto when it comes to balancing traffic coming in as well as within your gcp environment knowing the differences between the types of load balancers and which one to use for specific scenarios is crucial for the exam as you will be tested on it and so there's a lot to cover here so with that being said let's dive in now i wanted to start off with some basics with regards to what is low balancing and so when it comes to the low balancer itself a low balancer distributes user traffic across multiple instances of your application so by spreading the load you reduce the risk of your applications experiencing performance issues a load balancer is a single point of entry with either one or multiple back ends and within gcp these back ends could consist of either instance groups or negs and i'll be getting into any g's in just a little bit low balancers on gcp are fully distributed and software defined so there is no actual hardware load balancer involved in low balancing on gcp it is completely software defined and so there's no need to worry about any hardware any pre-warming time as this is all done through software now depending on which low balancer you choose google cloud gives you the option of having either a global load balancer or a regional load balancer the load balancers are meant to serve content as close as possible to the users so that they don't experience increased latency and gives the users a better experience as well as reducing latency on your applications when dealing with low balancers in between services google cloud also offers auto scaling with health checks in their load balancers to make sure that your traffic is always routed to healthy instances and by using auto scaling able to scale up the amount of instances you need in order to handle the load automatically now as there are many different low balancers to choose from it helps to know what specific aspects you're looking for and how you want your traffic distributed and so google has broken them down for us into these three categories the first category is global versus regional global load balancing is great for when your back ends are distributed across multiple regions and your users need access to the same applications and content using a single anycast ip address as well when you're looking for ipv6 termination global load balancing will take care of that now when it comes to regional load balancing this is if you're looking at serving your back ends in a single region and handling only ipv4 traffic now once you've determined whether or not you need global versus regional low balancing the second category to dive into is external versus internal external load balancers are designed to distribute traffic coming into your network from the internet and internal load balancers are designed to distribute traffic within your network and finally the last category that will help you decide on what type of load balancer you need is the traffic type and shown here are all the traffic types that cover http https tcp and udp and so now that we've covered the different types of load balancing that's available on google cloud i wanted to dive into some more depth on the low balancers themselves here you can see that there are five load balancers available and i will be going through each one of these in detail now before diving into the low balancers themselves i wanted to introduce you to a concept using gcp for all load balancers called back end services how a low balancer knows exactly what to do is defined by a backend service and this is how cloud load balancing knows how to distribute the traffic the backend service configuration contains a set of values such as the protocol used to connect to back ends various distribution in session settings health checks and timeouts these settings provide fine grain control over how your load balancer behaves an external http or https load balancer must have at least one backend service and can have multiple backend services the back ends of a backend service can be either instance groups or network endpoint groups also known as negs but not a combination of both and so just as a note you'll hear me refer to negs over the course of this lesson and so a network endpoint group also known as neg is a configuration object that specifies a group of back-end endpoints or services and a common use case for this configuration is deploying services into containers now moving on to the values themselves i wanted to first start with health checks and google cloud uses the overall health state of each back end to determine its eligibility for receiving new requests or connections back ends that respond successfully for the configured number of times are considered healthy back-ends that fail to respond successfully for a separate number of times are considered unhealthy and when a back-end is considered unhealthy traffic will not be routed to it next up is session affinity and session affinity sends all requests from the same client to the same back end if the back end is healthy and it has capacity service timeout is the next value and this is the amount of time that the load balancer waits for a backend to return a full response to a request next up is traffic distribution and this comprises of three different values the first one is a balancing mode and this defines how the load balancer measures back-end readiness for the new requests or connections the second one is target capacity and this defines a target maximum number of connections a target maximum rate or target maximum cpu utilization and the third value for traffic distribution is capacity scalar and this adjusts overall available capacity without modifying the target capacity and the last value for back-end services are back-ends and a back-end is a group of endpoints that receive traffic from a google cloud load balancer and there are several types of back-ends but the one that we are concentrating on for this section and for the exam is the instance group now backend services are not critical to know for the exam but i wanted to introduce you to this concept to add a bit more context for when you are creating low balancers in any environment and will help you understand other concepts in this lesson and so this is the end of part one of this lesson it was getting a bit long so i decided to break it up this would be a great opportunity for you to get up and have a stretch get yourself a coffee or tea and whenever you're ready join me in part two where we will be starting immediately from the end of part one so you can now complete this video and i will see you in part two this is part two of the cloud load balancers lesson and we'll be starting exactly where we left off in part one so with that being said let's dive in now before jumping right into the first load balancer that i wanted to introduce which is http and https low balancer there's a couple of different concepts that i wanted to introduce and these are the methods of how an http and https load balancer distributes traffic using forwarding rules and these are cross region low balancing and content based load balancing now touching on cross region load balancing when you configure an external http or https load balancer in premium tier it uses a global external ip address and can intelligently route requests from users to the closest backend instance group or neg based on proximity for example if you set up instance groups in north america and europe and attach them to a low balancers back-end service user requests around the world are automatically sent to the vms closest to the users assuming that the vms pass health checks and have enough capacity if the closest vms are all unhealthy or if the closest instance group is at capacity and another instance group is not at capacity the load balancer automatically sends requests to the next closest region that has available capacity and so here in this diagram a user in switzerland hits the low balancer by going to bowtieinc.co and because there are vms that are able to serve that traffic in europe west 6 traffic is routed to that region and so now getting into content based load balancing http and https low balancing supports content based load balancing using url maps to select a backend service based on the requested host name request path or both for example you can use a set of instance groups or negs to handle your video content and another set to handle static as well as another set to handle any images you can also use http or https low balancing with cloud storage buckets and then after you have your load balancer set up you can add cloud storage buckets to it now moving right along when it comes to http and https load balancer this is a global proxy based layer 7 low balancer which is at the application layer and so just as a note here with all the other low balancers that are available in gcp the http and https low balancer is the only layer 7 load balancer all the other low balancers in gcp are layer 4 and will work at the network layer and so this low balancer enables you to serve your applications worldwide behind a single external unicast ip address external http and https load balancing distributes http and https traffic to back ends hosted on compute engine and gke external http and https load balancing is implemented on google front ends or gfes as shown here in the diagram gfes are distributed globally and operate together using google's global network and control plane in the premium tier gfes offer cross-regional low balancing directing traffic to the closest healthy backend that has capacity and terminating http and https traffic as close as possible to your users with the standard tier the load balancing is handled regionally and this load balancer is available to be used both externally and internally that makes this load balancer global external and internal this load balancer also gives support for https and ssl which covers tls for encryption in transit as well this load balancer accepts all traffic whether it is ipv4 or ipv6 traffic and just know that ipv6 traffic will terminate at the low balancer and then it will forward traffic as ipv4 so it doesn't really matter which type of traffic you're sending the load balancer will still send the traffic to the back end using ipv4 this traffic is distributed by location or by content as shown in the previous diagram forwarding rules are in place to distribute defined targets to each target pool for the instance groups again defined targets could be content based and therefore as shown in the previous diagram video content could go to one target whereas static content could go to another target url maps direct your requests based on rules so you can create a bunch of rules depending on what type of traffic you want to direct and put them in maps for requests ssl certificates are needed for https and these can be either google managed or self-managed and so just as a quick note here the ports used for http are on 80 and 8080 as well on https the port that is used is port 443 now moving into the next low balancer is ssl proxy an ssl proxy low balancing is a reverse proxy load balancer that distributes ssl traffic coming from the internet to your vm instances when using ssl proxy load balancing for your ssl traffic user ssl connections are terminated at the low balancing layer and then proxied to the closest available backend instances by either using ssl or tcp with the premium tier ssl proxy low balancing can be configured as a global load balancing service with the standard tier the ssl proxy load balancer handles low balancing regionally this load balancer also distributes traffic by location only ssl proxy low balancing lets you use a single ip address for all users worldwide and is a layer 4 load balancer which works on the network layer this load balancer shows support for tcp with ssl offload and this is something specific to remember for the exam this is not like the http or https load balancer where we can use specific rules or specific configurations in order to direct traffic ssl proxy low balancer supports both ipv4 and ipv6 but again it does terminate at the load balancer and forwards the traffic to the back end as ipv4 traffic and forwarding rules are in place to distribute each defined target to its proper target pool and encryption is supported by configuring back-end services to accept all the traffic over ssl now just as a note it can also be used for other protocols that use ssl such as web sockets and imap over ssl and carry a number of open ports to support them moving on to the next load balancer is tcp proxy now the tcp proxy load balancer is a reverse proxy load balancer that distributes tcp traffic coming from the internet to your vm instances when using tcp proxy load balancing traffic coming over a tcp connection is terminated at the load balancing layer and then forwarded to the closest available backend using tcp or ssl so this is where the low balancer will determine which instances are at capacity and send them to those instances that are not like ssl proxy load balancing tcp proxy load balancing lets you use a single ip address for all users worldwide the tcp proxy load balancer automatically routes traffic to the back ends that are closest to the user this is a layer 4 load balancer and again can serve traffic both globally and externally tcp proxy distributes traffic by location only and is intended for specifically non-http traffic although you can decide if you want to use ssl between the proxy and your back end and you can do this by selecting a certificate on the back end again this type of load balancer supports ipv4 and ipv6 traffic and ipv6 traffic will terminate at the low balancer and forwards that traffic to the back end as ipv4 traffic now tcp proxy low balancing is intended for tcp traffic and supports many well-known ports such as port 25 for simple mail transfer protocol or smtp next up we have the network load balancer now the tcp udp network load balancer is a regional pass-through load balancer a network load balancer distributes tcp or udp traffic among instances in the same region network load balancers are not proxies and therefore responses from the back end vms go directly to the clients not back through the load balancer the term known for this is direct server return as shown here in the diagram this is a layer 4 regional load balancer and an external load balancer as well that can serve to regional locations it supports either tcp or udp but not both although it can low balance udp tcp and ssl traffic on the ports that are not supported by the tcp proxy and ssl proxy ssl traffic can still be decrypted by your back end instead of the load balancer itself traffic is also distributed by incoming protocol data this being protocols scheme and scope there is no tls offloading or proxying and forwarding rules are in place to distribute and define targets to their target pools and this is for tcp and udp only now with other protocols they use target instances as opposed to instance groups lastly a network load balancer can also only support self-managed ssl certificates as opposed to the google managed certificates as well and so the last low balancer to introduce is the internal load balancer now an internal tcp or udp load balancer is a layer 4 regional load balancer that enables you to distribute traffic behind an internal load balancing ip address that is accessible only to your internal vm instances internal tcp and udp load balancing distributes traffic among vm instances in the same region this load balancer supports tcp or udp traffic but not both and as i said before this type of load balancer is used to balance traffic within gcp across instances this low balancer cannot be used for balancing internet traffic as it is internal only traffic is automatically sent to the back end as it does not terminate client connections and for forwarding rules this load balancer follows specific specifications where you need to specify at least one and up to five ports by number as well you must specify all to forward traffic to all ports now again like the network load balancer you can use either tcp or udp and so that's pretty much all i had to cover with this lesson on low balancing please remember that for the exam you will need to know the differences between them all in my experience there are a few questions that come up on the exam where you will need to know what low balancer to use and so a good idea might be to dive into the console and have a look at the options as well as going back through this lesson as a refresher to understand each use case this is also a crucial component in any environment that is used especially when serving applications to the internet for any three-tier web application or kubernetes cluster and so that pretty much sums up this lesson on low balancing so you can now mark this lesson as complete and let's move on to the next one welcome back in this lesson i will be going into depth on instance groups along with instance templates instance groups are a great way to set up a group of identical servers used in conjunction with instance groups instance templates handles the instance properties to deploy the instance groups into your environment this lesson will dive into the details of the features use cases and how instance groups and instance templates work together to create a highly scalable and performing environment now there's a lot to cover here so with that being said let's dive in now an instance group is a collection of vm instances that you can manage as a single entity compute engine offers two kinds of vm instance groups managed and unmanaged manage instance groups or migs let you operate applications on multiple identical vms you can make your workload scalable and highly available by taking advantage of automated mig services like auto scaling auto healing regional and zonal deployments and automatic updating and i'll be getting into these services in just a sec now when it comes to unmanaged instance groups they also let you low balance across a fleet of vms but this is something that you need to manage and i'll be going deeper into unmanaged instance groups a bit later right now i wanted to take some time to go through the features and use cases of migs in a bit more detail for some more context starting off with its use cases now migs are great for stateless serving workloads such as website front ends web servers and website applications as the application does not preserve its state and saves no data to persistent storage all user and session data stays with the client and makes scaling up and down quick and easy migs are also great for stateless batch workloads and these are high performance or high throughput compute workloads such as image processing from a queue and lastly you can build highly available stateful workloads using stateful managed instance groups or stateful migs stateful workloads include applications with stateful data or configuration such as databases legacy monolith type applications and long running batch computations with checkpointing you can improve uptime and resiliency of these types of applications with auto healing controlled updates and multi-zone deployments while preserving each instance's unique state including instance names persistent disks and metadata now that i've covered the type of workloads that are used with migs i wanted to dive into the features starting with auto healing now when it comes to auto healing managed instance groups maintain high availability of your applications by proactively keeping your instances in a running state a mig automatically recreates an instance that is not running and managed instance groups also take care of application-based auto healing and this improves application availability by relying on a health check that detects things like freezing crashing or overloading if a health check determines that an application has failed on a vm the mig auto healer automatically recreates that vm instance the health check used to monitor the migs are similar to the health checks used for low balancing with a few little differences low balancing health checks help direct traffic away from unresponsive instances and towards healthy ones these health checks cannot recreate instances whereas mig health checks proactively signal to delete and recreate instances that become unhealthy moving on to managed instance groups regional or multi-zone feature now you have the option of creating regional migs or zonal migs regional migs provide higher availability compared to zonal migs because the instances in a regional mig are spread across multiple zones in a single region google recommends regional migs over zonal migs as you can manage twice as many migs as zonal migs so you can manage 2 000 migs instead of 1000 you can also spread your application load across multiple zones instead of a single zone or managing multiple zonal migs across different zones and this protects against zonal failures and unforeseen scenarios where an entire group of instances in a single zone malfunctions in the case of a zonal failure or if a group of instances in a zone stops responding a regional mig continues supporting your instances by continuing to serve traffic to the instances in the remaining zones now cloud low balancing can use instance groups to serve traffic so you can add instance groups to a target pool or to a back end an instance group is a type of back end and the instances in the instance group respond to traffic from the load balancer the back end service in turn knows which instances it can use and how much traffic they can handle and how much traffic they are currently handling in addition the back-end service monitors health checking and does not send new connections to unhealthy instances now when your applications require additional compute resources migs support auto scaling that dynamically add or remove instances from the mig in response to an increase or decrease in load you can turn on auto scaling and configure an auto scaling policy to specify how you want the group to scale not only will auto scaling scale up to meet the load demands but will also shrink and remove instances as the load decreases to reduce your costs auto scaling policies include scaling based on cpu utilization load balancing capacity and cloud monitoring metrics and so when it comes to auto updating you can easily and safely deploy new versions of software to instances in a mig the rollout of an update happens automatically based on your specifications you can also control the speed and scope of the deployments in order to minimize disruptions to your application you can optionally perform rolling updates as well as partial rollouts for canary testing and for those who don't know rolling updates allow updates to take place with zero downtime by incrementally updating instances with new ones as well canary testing is a way to reduce risk and validate new software by releasing software to a small percentage of users with canary testing you can deliver to certain groups of users at a time and this is also referred to as stage rollouts and this is a best practice in devops and software development now there are a few more things that i wanted to point out that relate to migs you can reduce the cost of your workload by using preemptable vm instances in your instance group and when they are deleted auto healing will bring the instances back when preemptable capacity becomes available again you can also deploy containers to instances in managed instance groups when you specify a container image in an instance template and is used to create a mig each vm is created with the container optimized os that includes docker and your container starts automatically on each vm in the group and finally when creating migs you must define the vpc network that it will reside in although when you don't define the network google cloud will attempt to use the default network now moving on into unmanaged instance groups for just a minute unmanaged instance groups can contain heterogeneous instances and these are instances that are of mixed sizes of cpu ram as well as instance types and you can add and remove these instances from the group whenever you choose there's a major downside to this though unmanaged instance groups do not offer auto scaling auto healing rolling update support multi-zone support or the use of instance templates and are not a good fit for deploying highly available and scalable workloads you should only use unmanaged instance groups if you need to apply load balancing to groups of these mixed types of instances or if you need to manage the instances yourself so unmanaged instance groups are designed for very special use cases where you will need to mix instance types in almost all cases you will be using managed instance groups as they were intended to capture the benefits of all the features they have to offer now in order to launch an instance group into any environment you will need another resource to do this and this is where instance templates come into play an instance template is a resource that you can use to create vm instances and managed instance groups instance templates define the machine type boot disk image or container image as well as labels and other instance properties you can then use an instance template to create a mig or vm instance instance templates are an easy way to save a vm instances configuration so you can use it later to recreate vms or groups of vms an instance template is a global resource that is not bound to a zone or region although you can restrict a template to a zone by calling out specific zonal resources now there is something to note for when you are ever using migs if you want to create a group of identical instances you must use an instance template to create a mig and is something you should always keep in the front of mind when using migs these two resources both instance templates and managed instance groups go hand in hand now some other things to note is that instance templates are designed to create instances with identical configurations so you cannot update an existing instance template or change an instance template after you create it if you need to make changes to the configuration create a new instance template you can create a template based on an existing instance template or based on an existing instance to use an existing vm to make a template you can save the configuration using the gcloud command gcloud instance dash templates create or to use the console you can simply go to the instance templates page click on the template that you want to update and click on create similar the last thing that i wanted to point out is that you can use custom or public images in your instance templates and so that's pretty much all i had to cover when it comes to instance groups and instance templates managed instance groups are great for when you're looking at high availability as a priority and letting migs do all the work of keeping your environment up and running and so you can now mark this lesson as complete and whenever you're ready join me in the next one where we go hands-on with instance groups instance templates and load balancers in a demo welcome back in this demo we're going to put everything that we've learned together in a hands-on demo called managing bow ties we're going to create an instance template and next we're going to use it to create an instance group we're then going to create a low balancer with a new back end and create some health checks along the way we're then going to verify that all instances are working by browsing to the load balancer ip and verifying the website application we're then going to stress test one of the instances to simulate a scale out using auto scaling and then we're going to simulate scaling the instance group back in now there's quite a bit to do here so with that being said let's dive in so here i am logged in as tony bowties at gmail.com under project bowtie inc and so the first thing that you want to do is you want to make sure that you have a default vpc network already created and so just to double check i'm going to go over to the navigation menu i'm going to scroll down to vpc network and yes i do have a default vpc network so i'm going to go ahead and start creating my resources and so now what i want to do is i want to create my instance template and so in order to do that i'm going to go back up to the navigation menu i'm going to go down to compute engine and go up to instance templates as you can see i currently have no instance templates and yours should look the same and so you can go ahead and click on create instance template and so just as a note there are no monthly costs associated with instance templates but this estimate here on the right is to show you the cost of each instance you will be creating with this template okay so getting right into it i'm going to name this instance template bowtie template and since we're spinning up a lot of vms you want to be conscious on costs and so under series you're going to click on the drop down and you're going to select n1 and under machine type you're going to select f1 micro and this is the smallest instance type as well as the cheapest within google cloud you can go ahead and scroll down right to the bottom here under firewall you want to check off allow http traffic next you want to select management security disks networking and sold tenancy you scroll down a little bit and under startup script you're going to paste in the script that's available in the repo and you will find a link to this script and the repo in the lesson text and so you can leave all the other options as its default and simply click on create it's going to take a couple minutes here okay and the instance template is ready and so the next step that you want to do is create an instance group and as i said in a previous lesson in order to create an instance group you need an instance template hence why we made the instance template first okay and our instance template has been created and so now that you've created your instance template you can head on over to instance groups here in the left hand menu and as expected there are no instance groups and so you can go ahead and click on the big blue button and create an instance group you're going to make sure that new managed instance group stateless is selected and here you have the option of choosing a stateful instance group as well as an unmanaged instance group and so we're going to keep things stateless and so for the name of the instance group you can simply call this bowtie group i'm going to use the same name in the description and under location you want to check off multiple zones in under region you want to select us east one and if you click on configure zones you can see here that you can select all the different zones that's available in that region that you choose to have your instances in and so i'm going to keep it under all three zones i'm going to scroll down here a little bit and under instance template you should see bow tie template you can select that you can scroll down a little bit more and here under minimum number of instances you want to set the minimum number of instances to 3 and under maximum number of instances you want to set that to 6 and so this is going to be double the amount of the minimum number of instances so when you're scaled out you should have a maximum of 6 instances and when you're scaled in or you have very low traffic you should only have three instances so you can scroll down some more and under auto healing you want to select the health check and you're going to go ahead and create a new health check under name you can call this healthy bow ties i'm going to use the same for the description and i'm going to leave the rest as its default and go down and click on save and continue i'm going to scroll down some more and i'm going to leave the rest as is and simply click on create and it's going to take a couple minutes here and so i'm going to pause the video and i'll be back in a flash okay and my instance group has been created and so to get a better look at it i'm going to click on bow tie group and i can see here that three instances have been created if i go up to vm instances you can see here that i have three instances but under instance groups because i have health check enabled it shows that my instances are unhealthy and this is because i still need to create a firewall rule that will allow google's health check probes to reach my vm instances and so you're going to go ahead and create that firewall rule so you can bring the health check status up to healthy so i'm going to go over to the navigation menu and scroll down to vpc network and go over to firewall here under firewall as expected you have the default firewall rules from the default created vpc network and so i'm going to go up to create firewall and you can name this firewall rule allow health check i'm going to use the same for the description i'm going to scroll down here a little bit and under targets i'm going to select all instances in the network source filter i'm going to leave as i p ranges and so here under source i p ranges i want to enter in the ip addresses for the google cloud health check probes and you can find these in the documentation and i will also be supplying them in the instructions and there are two sets of ip addresses that need to be entered and just as a note you don't need to know this for the exam but it's always a good to know if you're ever adding health checks to any of your instances i'm going to scroll down a little bit to protocols and ports and under tcp i'm going to check it off and put in port 80. that's pretty much all you have to do here so whenever you entered all that information in you can simply click on create and so now i have a firewall rule that will allow health checks to be done and so it may take a minute or two but if i head back on over to my compute engine instances and go over to my instance groups i'll be able to see that all my instances are now healthy and so whenever you're creating instance groups and you're applying health checks this firewall rule is necessary so please be aware okay so now that we've created our instance templates we've created our instance groups and we created a firewall rule in order to satisfy health checks we can now move on to the next step which is creating the load balancer so i'm going to go back up to the navigation menu and i'm going to scroll down to network services and over to load balancing and as expected there are no load balancers created and so whenever you're ready you can click on the big blue button and create a new low balancer here you have the option of creating an http or https load balancer along with a tcp load balancer or a udp load balancer and because we're serving external traffic on port 80 we're going to use the http load balancer so you can click on start configuration and i'm being prompted to decide between internet facing or internal only and you're going to be accepting traffic from the internet to your load bouncer so make sure that from internet to my vms is checked off and simply click continue and so next you will be prompted with a page with a bunch of configurations that you can enter and so we'll get to that in just a second but first we need to name our load balancer and so i'm going to call this bowtie dash lb for low balancer and so next step for your load balancer is you need to configure a back end so you can click on back end configuration and here you have the option of selecting from back-end services or back-end buckets so you're going to go ahead and click on back-end services and create a back-end service and here you will be prompted with a bunch of fields to fill out in order to create your back-end service and you can go ahead and name the backend service as bowtie backend service back-end type is going to be instance group and you can leave the protocol named port and timeout as is as we're going to be using http under instance group in new back-end if you select the drop-down you should see your available bow tie group instance group select that scroll down a little bit and under port numbers you can enter in port 80 and you can leave all the other options as default and simply click on done and so if you're ever interested you can always add a cache using cloud cdn now i know we haven't gone through cloud cdn in this course but just know that this is google's content delivery network and it uses google's global edge network to serve content closer to users and this accelerates your websites and your applications and delivers a better user experience for your user okay and moving on here under health check if i click on the drop down you should see healthy bow ties you can select that for your health check and so just as a note here under advanced configurations you can set your session affinity your connection draining timeout as well as request and response headers and so we don't need any of that for this demo and so i'm going to go ahead and collapse this and once you've finished filling in all the fields you can simply click on create okay and so you should now have your back end configuration and your host and path rules configured and so the only thing that's left to configure is the front end so you can go up and click on front-end configuration and you can name your front-end bowtie front-end service gonna keep the protocols http and here is where you would select the network service tier choosing either premium or standard and if you remember in the load balancing lesson in order to use this as a global load balancer i need to use a premium tier okay and we're going to keep this as ipv4 with an ephemeral ip address on port 80 so once you've finished configuring the front end you can simply click on done and you can go and click on review and finalize and this will give you a summary on your configuration and so i'm happy with the way everything's configured and if you are as well you can simply click on create and this may take a minute or two but it will create your low balancer along with your back end and your front end so again i'm going to pause the video here for just a minute and i'll be back before you can say cat in the hat okay and my load balancer has been created and to get a little bit more details i'm going to drill down into it and i can see here the details of my load balancer along with my monitoring and any caching but i don't have any caching enabled and therefore nothing is showing so going back to the details i can see here that i have a new ip address for my load balancer and i'll be getting into that in just a minute i'm going to go back here and i'm going to check out my back ends click on bow tie back end service and here i can see the requests per second as well as my configuration and if you do see this caution symbol here showing that some of your instances are unhealthy it's only because the low balancer needs time to do a full health check on all the instances in the instance group and so this will take some time okay and so i'm going to go back over and check out my front end and there's nothing to drill down into with the front end service but it does show me my scope the address the protocol network tier and the low balancer itself so this is the end of part one of this demo it was getting a bit long so i decided to break it up this would be a great opportunity for you to get up have a stretch get yourself a coffee or tea and whenever you're ready part two will be starting immediately from the end of part one so you can now mark this as complete and i'll see you in part two this is part two of the managing bow ties demo and we will be starting exactly where we left off in part one so with that being said let's dive in and so before you move forward you want to make sure that all your instances are considered healthy by your load balancer and as i can see here all my instances in my instance group are considered healthy by the load balancer and so just to verify this i'm going to go ahead and copy the i p address and you can open up a new tab in your browser and simply paste it in and success as you can see here managing the production of many bow ties can be automated but managing the wearer of them definitely cannot another fine message from the people at bow tie inc now although this is a simple web page i used a couple variables just to show you the low balancing that happens in the background and traffic will be load balanced in between all of the instances in the instance group so if you click on refresh then you should see the machine name and the data center change so every time i click refresh the traffic will be routed to a different instance in a different zone and so a simple simulation on how traffic is low balance between the different instances in their different zones okay so now that we've verified the website application i'm going to close down this tab and so now that we've created our instance template we've created our instance group and we've created our low balancer with the back end and front end service and it looks like everything seems to be working together nicely we're going to go ahead and simulate a scale out using auto scaling and so in order to simulate this we're going to do a stress test on one of the instances so i'm going to head back on over to the navigation menu scroll down to compute engine and here you can ssh into any one of these instances and run the stress test from there so i'm going to pick here the one at the top and so whenever you're logged in you can simply paste in the command that i've included in the instructions that will run the stress test and so this is a stress test application called stress that was included in the startup script and this again will put stress on the server itself and trigger a scale out to handle the load and it'll do this for 30 seconds so you can go ahead and hit enter and head back over to the console and in about a minute or two you should see some new instances that will be created by your instance group in order to handle the load okay and after about a couple minutes it's showing here that instances are being created and it will be scaling out to the maximum amount of instances that i've set it to which is six i'm going to drill down into this and yes a scale out is happening and some new instances are being created to handle the load so i'm going to give it just a minute here okay and as you can see here all the instances have been created they've been added to the instance group and all of them are marked as healthy and so just to verify that all the instances are working i'm going to go ahead and open up a new tab i'm going to plug in the ip address on my load balancer and i'm going to simply cycle through all these instances to make sure that all them are working and it looks like i have no issues and so now that you've simulated a scale out i wanted to go ahead and run a scale in and so i'm first going to close up these tabs now with regards to scaling there is a 10 minute stabilization period that cannot be adjusted for scaling and this is a built-in feature into google cloud now because i respect your time as a student i'm going to show you a work around to trigger a scale in sooner strictly for this demo and i also wanted to caution that this should never be done in a production or production-like environment you should always wait for the scaling to happen on its own and never force it this method is being used strictly for learning purposes to save you some time and so i'm going to go ahead to the top menu and click on rolling restart and replace and this will bring up a new page where you will have the option to either restart or replace any instances in your instance group and so for your purposes under operation make sure that you have restart checked off and this will restart all of your instances and only bring up the ones that are needed so i'm going to go ahead and click on restart i'm going to go back to my instance group console and i'm just going to give this a few minutes to cook and i'll be right back in a flash okay so it looks like the instance group has scaled in and we are now down left to three instances the minimum that we configured for our instance group and so that pretty much covers the managing bow ties demo so i wanted to congratulate you on making it through this demo and i hope that this has been extremely useful in excelling your knowledge on managing instance templates managed instance groups and creating load balancers with back-end and front-end services now this was a jam-packed demo and there was a lot to pack in with everything you've learned from the last few lessons and so just as a recap you created an instance template with your startup script you then created a new instance group with a health check to go with it configuring auto scaling for a minimum of three instances you then created a firewall rule so that the health check probes were able to connect to the application and you then created a load balancer with its back end and front-end service and verified that the website application was indeed up and running you then ran a stress test to allow a simulation of a scale out of your instance group and then simulated a scale in of your instance group great job and so now that we've completed this demo you want to make sure that you're not accumulating any unnecessary costs and so i'm going to go ahead and walk you through the breakdown of deleting all these resources so first you're going to go ahead and delete the load balancer go back up to the navigation menu and scroll down to network services and go over to load balancing so i'm going to go ahead and check off bow tie lb and simply go up to the top and click on delete it's going to ask me if i'm sure i want to do this i'm also going to select bow tie back end service and i can delete my load balancer and my back end service all at once i'm going to go ahead and delete load balancer and the selected resources and this should clear up within a few seconds okay and our load balancer has been deleted i'm going to just go up here to the back end make sure everything's good yeah we're all clean same thing with front end and so now you can move on to instance groups so i'm going to head back up to the navigation menu go down a compute engine and go up to instance groups and here you can just simply check off bow tie group and simply click on delete you're going to be prompted with a notification to make sure you want to delete bow tie group yes i want to delete and again this should take about a minute okay it actually took a couple minutes but my instance group has been deleted and so now i'm going to go over to instance templates and i'm going to delete my template and check off bow tie template and simply click delete you're going to get a prompt to make sure you want to delete your instance template yes you want to delete and success you've now deleted all your resources although there is one more resource that you will not be billed for but since we're cleaning everything up we might as well clean that up as well and this is the firewall rule that we created and go over to the navigation menu and scroll down to vpc network i'm going to go to firewall here on the left hand menu and here i'm going to check off the allow health check firewall rule and simply click on delete i'm going to get a prompt to make sure that i want to delete it yes you want to delete i'm going to quickly hit refresh and yes we've deleted it and so this concludes the end of this demo so you can now mark this as complete and i'll see you in the next one welcome back in this next section we will be focusing on google cloud's premier container orchestration service called kubernetes but before we can dive right into kubernetes and the benefits that it gives to containers you'll need an understanding as to what containers are and what value containers provide in this lesson i will be covering the difference between virtual machines and containers what containers are how they work and the value proposition they bring so with that being said let's dive in now for those of you who didn't know container technology gets its name from the shipping industry products get placed into standardized shipping containers which are designed to fit into the ship that accommodates the container's standard size instead of having various sizes of packaging now by standardizing this process and keeping the items together the container can be moved as a unit and it costs less to do it this way as well the standardization allows for consistency when packing and moving the containers placing them on ships and docks as well as storage no matter where the container is it always stays the same size and the contents stay isolated from all the other containers that they are stacked with and so now before we get into the details of containers i wanted to cover how we got here and why so a great way to discuss containers is through their comparison to virtual machines now as we discussed in a previous lesson when it comes to vms the systems are virtualized through a hypervisor that sits on top of the underlying host infrastructure the underlying hardware is virtualized so that multiple operating system instances can run on the hardware each vm runs its own operating system and has access to virtualized resources representing the underlying hardware due to this process vms come with the cost of large overhead in cpu memory and disk as well can be very large due to the fact that each vm needs its own individual operating system there also lacks standardization between each vm making them unique due to the os configuration the software installed and the software libraries thus not making it very portable to be able to run in any environment now when dealing with containers things are run very differently the underlying host infrastructure is still there but instead of just using a hypervisor and abstracting the underlying hardware containerization takes it one step further and abstracts the operating system thus leaving the application with all of its dependencies in a neatly packaged standardized container this is done by installing the operating system on top of the host infrastructure and then a separate layer on top of the host operating system called the container engine now instead of having their own operating system the containers share the operating system kernel with other containers while operating independently running just the application code and the dependencies needed to run that application this allows each container to consume very little memory or disk making containers very lightweight efficient and portable containerized applications can start in seconds and many more instances of the application can fit onto the machine compared to a vm environment this container can now be brought over to other environments running docker and able to run without having the worries of running into issues of compatibility now although there are a few different container engines out there the one that has received the most popularity is docker and this is the engine that we will be referring to for the remainder of this course now a docker image is a collection or stack of layers that are created from sequential instructions on a docker file so each line in the dockerfile is run line by line and a unique read-only layer is written to the image what makes docker images unique is that each time you add another instruction in the docker file a new layer is created now going through a practical example here shown on the right is a docker file and we will be able to map each line of code to a layer shown on the docker image on the left the line marked from shows the base image that the image will be using the example shown here shows that the ubuntu image version 12.04 will be used next the run instruction is used which will perform a general update install apache 2 and output a message to be displayed that is written to the index.html file next up is the working directories and these are the environment variables set by using an env instruction and this will help run the apache runtime next layer is the expose instruction and this is used to expose the container's port on 8080 and lastly the command layer is an instruction that is executing the apache web server from its executable path and so this is a great example of how a docker file is broken down from each line to create the layers of this image and so just as a note here each docker image starts with a base image as well each line in a docker file creates a new layer that is added to the image and finally all the layers in a docker image are read only and cannot be changed unless the docker file is adjusted to reflect that change so now how do we get from a docker image to a container well a running docker container is actually an instantiation of an image so containers using the same image are identical to each other in terms of their application code and runtime dependencies so i could use the same image for multiple copies of the same container that have different tasks what makes each individual container different is that running containers include a writable layer on top of the read-only content runtime changes including any rights and updates to data and files are saved in this read write layer so in this example when using the command docker run fashionista a docker container will be instantiated from the docker image and a read write layer is always added on top of the read-only layers when a container is created writing any necessary files that's needed for the application and so just as a note here docker containers are always created from docker images and containers can use the same image yet will always have a different read write layer no matter the amount of containers running on a given host so now when your containers have been created you need a place to store them and so this is where a container registry comes into play now a container registry is a single place for you to store and manage docker images now when you create your docker file and then build your image you want to store that image in a central image repository whether it be a private one or a public one a popular public container registry is docker hub and this is a common registry where many open source images can be found including those used for the base layer images like the ubuntu example that i showed you earlier and so once you have your containers in a container registry you need to be able to run these containers so in order to run these containers you need docker hosts and these can consist of any machine running the docker engine and this could be your laptop server or you can run them in provided hosted cloud environments now this may have been a refresher for some but for those of you who are new to containers i hope this has given you a lot more clarity on what containers are what they do and the value that they bring to any environment and so that's pretty much all i wanted to cover on this short lesson of an introduction to containers so you can now mark this lesson as complete and let's move on to the next one welcome back so now that you've gotten familiar with what containers are and how they work i wanted to dive into google cloud's platform as a service offering for containers called google kubernetes engine also known as short as gke now although the exam goes into a more operational perspective with regards to gke knowing the foundation of kubernetes and the different topics of kubernetes is a must in order to understand the abstractions that take place with gke from regular kubernetes in this lesson i will be getting into key topics with regards to kubernetes and we'll be touching on the architecture components and how they all work together to achieve the desired state for your containerized workloads now there's a lot to get into so with that being said let's dive in now before i can get into gke i need to set the stage on explaining what kubernetes is put simply kubernetes is an orchestration platform for containers which was invented by google and eventually open source it is now maintained by the cncf short for the cloud native computing foundation and has achieved incredible widespread adoption kubernetes provides a platform to automate schedule and run containers on clusters of physical or virtual machines thus eliminating many of the manual processes involved in deploying and scaling containerized applications kubernetes manages the containers that run the applications and ensure that there is no downtime in a way that you the user can define for example if you define that when a container goes down and another container needs to start kubernetes would take care of that for you automatically and seamlessly kubernetes provides you with the framework to run distributed systems resiliently it takes care of scaling and failover for your application provides deployment patterns and allows you to manage your applications with tons of flexibility reliability and power it works with a range of container tools including docker now although this adoption was widespread it did come with its various challenges this included scaling at cd load balancing availability auto scaling networking rollback on faulty deployments and so much more so now google cloud has since developed a managed offering for kubernetes providing a managed environment for deploying managing and scaling your containerized applications using google infrastructure the gke environment consists of compute engine instances grouped together to form a cluster and it provides all the same benefits as on-premises kubernetes yet has abstracted the complexity of having to worry about the hardware and to top it off it has the benefits of advanced cluster management features that google cloud provides with things like cloud load balancing and being able to spread traffic amongst clusters and nodes node pools to designate subnets of nodes within a cluster for additional flexibility automatic scaling of your cluster's node instance count and automatic upgrades for your clusters node software it also allows you to maintain node health and availability with node auto repair and takes care of logging and monitoring with google cloud's operation suite for visibility into your cluster so as you can see here gke holds a lot of benefits when it comes to running kubernetes in google cloud so i wanted to take a moment now to dive into the cluster architecture and help familiarize you with all the components involved in a cluster so a cluster is the foundation of google kubernetes engine and kubernetes as a whole the kubernetes objects that represent your containerized applications all run on top of the cluster in gke a cluster consists of at least one control plane and multiple worker machines called nodes the control plane and node machines run the kubernetes cluster the control plane is responsible to coordinate the entire cluster and this can include scheduling workloads like containerized applications and managing the workload's life cycle scaling and upgrades the control plane also manages network and storage resources for those workloads and most importantly it manages the state of the cluster and make sure it is at the desired state now the nodes are the worker machines that run your containerized applications and other workloads the nodes are compute engine vm instances that gke creates on your behalf when you create a cluster each node is managed from the control plane which receives updates on each node's self-reported status a node also runs the services necessary to support the docker containers that make up your cluster's workloads these include the docker runtime and the kubernetes node agent known as the cubelet which communicates with the control plane and is responsible for starting and running docker containers scheduled on that node now diving deeper into the architecture there are components within the control plane and nodes that you should familiarize yourself with as these components are what ties the cluster together and helps manage the orchestration as well as the state now the control plane is the unified endpoint for your cluster the control plane's components make global decisions about the cluster for example scheduling as well as detecting and responding to cluster events all interactions with the cluster are done via kubernetes api calls and the control plane runs the kubernetes api server process to handle those requests you can make kubernetes api calls directly via http or grpc or can also be done indirectly by running commands from the kubernetes command line client called cubectl and of course you can interact with the ui in the cloud console the api server process is the hub for all communications for the cluster moving on to the next component is cube scheduler the cube scheduler is a component that discovers and assigns newly created pods to a node for them to run on so any new pods that are created will automatically be assigned to its appropriate node by the cube scheduler taking into consideration any constraints that are in place next up is the cube controller manager and this is the component that runs controller processes and is responsible for things like noticing and responding when nodes go down maintaining the correct number of pods populating the services and pods as well as creating default accounts and api access tokens for new namespaces it is these controllers that will basically look to make changes to the cluster when the current state does not meet the desired state now when it comes to the cloud controller manager this is what embeds cloud-specific control logic the cloud controller manager lets you link your cluster into any cloud providers api and separates out the components that interact with that cloud platform from components that just interact with your cluster the cloud controller manager only runs controllers that are specific to your cloud provider in this case google cloud and lastly we have fcd and this component is responsible to store the state of the cluster at cd is a consistent and highly available key value store that only interacts with the api server it saves all the configuration data along with what nodes are part of the cluster and what pods they are running so now the control plane needs a way to interact with the nodes of the cluster thus the nodes having components themselves for this communication to occur this component is called a cubelet and this is an agent that runs on each node in the cluster that communicates with the control plane it is responsible for starting and running docker containers scheduled on that node it takes a set of pod specs that are provided to it and ensures that the containers described in those pod specs are running and healthy and i will be diving into pod specs in a later lesson next up is cube proxy and this is the component that maintains network connectivity to the pods in a cluster and lastly the container runtime is the software that is responsible for running containers kubernetes supports container runtimes like docker and container d and so these are the main components in a cluster covering the control plane and nodes with regards to communication within the cluster now before i end this lesson there is one more topic i wanted to touch on with regards to the architecture of a gke cluster and that is the abstraction that happens and what exactly does gke manage with regards to kubernetes well gke manages all the control plane components the endpoint exposes the kubernetes api server that cubectl uses to communicate with your cluster control plane the endpoint exposes the kubernetes api server that cubectl uses to communicate with your cluster control plane the endpoint ip is displayed in cloud console and this ip will allow you to interact with the cluster when you run the command gcloud container clusters get dash credentials you see that the command gets the cluster endpoint as part of updating cubeconfig an ip address for the cluster is then exposed to interact with and is responsible for provisioning and managing all the infrastructure that is needed for the control plane gke also automates the kubernetes nodes by launching them as compute engine vms under the hood but still allows the user to change the machine type and access upgrade options by default google kubernetes engine clusters and node pools are upgraded automatically by google but you can also control when auto upgrades can and cannot occur by configuring maintenance windows and exclusions and just as a note a clusters control plane and nodes do not necessarily run the same version at all times and i will be digging more into that in a later lesson and so i know this is a lot of theory to take in but is as i said before a necessity to understanding kubernetes and gke and as we go further along into kubernetes and get into demos i promise that this will start to make a lot more sense and you will start becoming more comfortable with gke and the underlying components of kubernetes knowing kubernetes is a must when working in any cloud environment as it is a popular and growing technology that is not slowing down so knowing gke will put you in a really good position for your career as an engineer in google cloud as well will give you a leg up on diving into other cloud vendors implementation of kubernetes and so that's pretty much all i wanted to cover when it comes to google kubernetes engine and kubernetes so you can now mark this lesson as complete and let's move on to the next one welcome back in this lesson i will be covering cluster and node management in gke as it refers to choosing different cluster types for your workloads cluster versions node pools as well as upgrades and the many different options to choose from it is good to familiarize yourself with these options as they may be the deciding factor of having to keep your workloads highly available and your tolerance to risk within your environment so with that being said let's dive in now in the last lesson we touched on nodes and how they are the workers for the kubernetes cluster so now that you are familiar with nodes i wanted to touch on a concept that builds on it called node pools now a node pool is a group of nodes within a cluster that all have the same configuration and using node config specification to achieve this a node pool can also contain one or multiple nodes when you first create a cluster the number and type of nodes that you specify becomes the default node pool as shown here in the diagram then you can add additional custom node pools of different sizes and types to your cluster all nodes in any given node pool are identical to one another now custom node pools are really useful when you need to schedule pods that require more resources than others such as more memory more disk space or even different machine types you can create upgrade and delete node pools individually without affecting the whole cluster and just as a note you cannot configure a single node in any node pool any configuration changes affect all nodes in the node pool and by default all new node pools run the latest stable version of kubernetes existing node pools can be manually upgraded or automatically upgraded you can also run multiple kubernetes node versions on each node pool in your cluster update each node pool independently and target different node pools for specific deployments in that node now with gke you can create a cluster tailored to your availability requirements and your budget the types of available clusters include zonal both single zone or multi-zonal and regional zonal clusters have a single control plane in a single zone depending on what kind of availability you want you can distribute your nodes for your zonal cluster in a single zone or in multiple zones now when you decide to deploy a single zone cluster it again has a single control plane running in one zone this control plane manages workloads on nodes running in the same zone a multi-zonal cluster on the other hand has a single replica of the control plane running in a single zone and has nodes running in multiple zones during an upgrade of the cluster or an outage of the zone where the control plane runs workloads still run however the cluster its nodes and its workloads cannot be configured until the control plane is available multi-zonal clusters are designed to balance availability and cost for consistent workloads and just as a note the same number of nodes will be deployed to each selected zone and may cost you more than budgeted so please be aware and of course when you're looking to achieve high availability for your cluster regional clusters are always the way to go a regional cluster has multiple replicas of the control plane running in multiple zones within a given region nodes also run in each zone where a replica of the control plane runs because a regional cluster replicates the control plane and nodes it consumes more compute engine resources than a similar single zone or multi-zonal cluster the same number of nodes will be deployed to each selected zone and the default when selecting regional clusters is three zones now if you're dealing with more sensitive workloads that require more strict guidelines private clusters give you the ability to isolate nodes from having inbound and outbound connectivity to the public internet this isolation is achieved as the nodes have internal ip addresses only if you want to provide outbound internet access for certain private nodes you can use cloudnat or manage your own nat gateway by default private google access is enabled in private clusters and their workloads with limited outbound access to google cloud apis and services over google's private network in private clusters the control plane's vpc network is connected to your clusters vpc network with vpc network peering your vpc network contains the cluster nodes and a separate google cloud vpc network contains your cluster's control plane the control plane's vpc network is located in a project controlled by google traffic between nodes and the control plane is routed entirely using internal ip addresses the control plane for a private cluster has a private endpoint in addition to a public endpoint the control plane for a non-private cluster only has a public endpoint the private endpoint is an internal ip address in the control plane's vpc network the public endpoint is the external ip address of the control plane and you can control access to this endpoint using authorized networks or you can disable access to the public endpoint as shown here in the diagram you can disable the public endpoint and connect to your network using an internal ip address using cloud interconnect or cloud vpn and you always have the option of enabling or disabling this public endpoint now when you create a cluster you can choose the cluster specific kubernetes version or you can mix the versions for flexibility on features either way it is always recommended that you enable auto upgrade for the cluster and its nodes now when you have auto upgrade enabled you are given the choice to choose from what are called release channels when you enroll a new cluster in a release channel google automatically manages the version and upgrade cadence for the cluster and its node pools all channels offer supported releases of gke and are considered in general availability you can choose from three different release channels for automatic management of your cluster's version and upgrade cadence as shown here the available release channels are rapid regular and stable release channels the rapid release channel gets the latest kubernetes release as early as possible and be able to use new gka features the moment that they go into general availability with the regular release channel you have access to gke and kubernetes features reasonably soon after they are released but on a version that has been qualified two to three months after releasing in the rapid release channel and finally we have the stable release channel where stability is prioritized over new functionality changes and new versions in this channel are rolled out last after being validated two to three months in the regular release channel and so if you're looking for more direct management of your cluster's version choose a static version when you enroll a cluster in a release channel that cluster is upgraded automatically when a new version is available in that channel now if you do not use a release channel or choose a cluster version the current default version is use the default version is selected based on usage and real world performance and is changed regularly while the default version is the most mature one other versions being made available are generally available versions that pass internal testing and qualification changes to the default version are announced in a release note now if you know that you need to use a specific supported version of kubernetes for a given workload you can specify it when creating the cluster if you do not need to control the specific patch version you use consider enrolling your cluster in a release channel instead of managing its version directly now when it comes to upgrading the cluster please be aware that control plane and nodes do not always run the same version at all times as well a control plane is always upgraded before its nodes when it comes to zonal clusters you cannot launch or edit workloads during that upgrade and with regional clusters each control plane is upgraded one by one as well with control planes auto upgrade is enabled by default and this is google cloud's best practice now again if you choose you can do a manual upgrade but you cannot upgrade the control plane more than one minor version at a time so please be aware as well with any cluster upgrades maintenance windows and exclusions are available and so this way you can choose the best times for your upgrades and so like cluster upgrades by default a clusters nodes have auto upgrade enabled and it is recommended that you do not disable it again this is best practice by google cloud and again like the cluster upgrades a manual upgrade is available and maintenance windows and exclusions are available for all of these upgrades now when a no pool is upgraded gke upgrades one node at a time while a node is being upgraded gke stops scheduling new pods onto it and attempts to schedule its running pods onto other nodes the node is then recreated at the new version but using the same name as before this is similar to other events that recreate the node such as enabling or disabling a feature on the node pool and the upgrade is only complete when all nodes have been recreated and the cluster is in the desired state when a newly upgraded node registers with the control plane gke marks the node as schedulable upgrading a no pool may disrupt workloads running in that pool and so in order to avoid this you can create a new node pool with the desired version and migrate the workload then after migration you can delete the old node pool now surge upgrades let you control the number of nodes gke can upgrade at a time and control how disruptive upgrades are to your workloads you can change how many nodes gke attempts to upgrade at once by changing the surge upgrade parameters on a no pool surge upgrades reduce disruption to your workloads during cluster maintenance and also allow you to control the number of nodes upgraded in parallel surge upgrades also work with the cluster auto scaler to prevent changes to nodes that are being upgraded now surge upgrade behavior is determined by two settings max surge upgrade and max unavailable upgrade now with max surge upgrade this is the number of additional nodes that can be added to the no pool during an upgrade increasing max surge upgrade raises the number of nodes that can be upgraded simultaneously and when it comes to the max unavailable upgrade this is the number of nodes that can be simultaneously unavailable during an upgrade increasing max unavailable upgrade raises the number of nodes that can be upgraded in parallel so with max surge upgrade the higher the number the more parallel upgrades which will end up costing you more money with max unavailable upgrade the higher the number the more disruptive it is and so the more risk you are taking and so during upgrades gke brings down at most the sum of the max surge upgrade added with the max unavailable upgrade so as you can see here there are a slew of options when it comes to deciding on the type of cluster you want as well as the type of upgrades that are available along with when you want them to occur and so your deciding factor in the end will be the workload that you are running and your risk tolerance and this will play a big factor in keeping up time for your cluster as well as saving money in any type of environment and so that's pretty much all i wanted to cover when it comes to gke cluster and node management so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this lesson i will be diving into some more theory within kubernetes and gke this time touching on objects and how objects are managed pods are only one type of object but there are many other parts that are involved in the management of these objects and this is what this lesson is set out to teach you now there's quite a bit to cover here so with that being said let's dive in now kubernetes objects are persistent entities in kubernetes kubernetes uses these entities to represent the state of your cluster for example it can describe things like what containerized applications are running and on which nodes and what resources are available to those applications a kubernetes object is a record of intent once you create the object kubernetes will constantly work to ensure that object exists by creating an object you're effectively telling kubernetes what you want your cluster's workload to look like and this is your cluster's desired state and you've heard me speak about this many times before and this is what i was referring to now almost every kubernetes object includes two nested object fields that govern the object's configuration the object spec and the object's status for objects that have a spec you have to set this when you create the object providing a description of the characteristics you want the resource to have its desired state the status describes the current state of the object supplied and updated by kubernetes and its components the kubernetes control plane continually and actively manages every object's actual state to match the desired state you supplied now each object in your cluster has a name that is unique for that type of resource every kubernetes object also has a uid that is unique across your whole cluster only one object of a given kind can have a given name at a time however if you delete the object you can make a new object with that same name every object created over the whole lifetime of a kubernetes cluster has a distinct uid these distinct uids are also known as uuids which we discussed earlier on in the course now when creating updating or deleting objects in kubernetes this is done through the use of a manifest file where you would specify the desired state of an object that kubernetes will maintain when you apply the manifest each configuration file can contain multiple manifests and is common practice to do so when possible a manifest file is defined in the form of a yaml file or a json file and it is recommended to use yaml now in each yaml file for the kubernetes object that you want to create there are some required values that need to be set the first one is the api version and this defines which version of the kubernetes api you're using to create this object the kind described in this example as a pod is the kind of object you want to create next up is the metadata and this is the data that helps uniquely identify the object including a string name a uid and an optional namespace and the last required value is the spec and this is what state you desire for the object and the spec in this example is a container by the name of bow tie dash web server and is to be built with the latest nginx web server image as well as having port 80 open on the container now when it comes to objects pods are the smallest most basic deployable objects in kubernetes a pod represents a single instance of a running process in your cluster pods contain one or more containers such as docker containers and when a pod runs multiple containers the containers are managed as a single entity and share the pods resources which also includes shared networking and shared storage for their containers generally one pod is meant to run a single instance of an application on your cluster which is self-contained and isolated now although a pod is meant to run a single instance of your application on your cluster it is not recommended to create individual pods directly instead you generally create a set of identical pods called replicas to run your application a set of replicated pods are created and managed by a controller such as a deployment controllers manage the life cycle of their pods as well as performing horizontal scaling changing the number of pods is necessary now although you might occasionally interact with pods directly to debug troubleshoot or inspect them it's recommended that you use a controller to manage your pods and so once your pods are created they are then run on nodes in your cluster which we discussed earlier the pod will then remain on its node until its process is complete the pot is deleted the pod is evicted from the node due to lack of resources or the node fails if a node fails pods on the node are automatically scheduled for deletion now a single gke cluster should be able to satisfy the needs of multiple users or groups of users and kubernetes namespaces help different projects teams or customers to share a kubernetes cluster you can think of a namespace as a virtual cluster inside of your kubernetes cluster and you can have multiple namespaces logically isolated from each other they can help you and your teams with organization and security now you can name your namespaces whatever you'd like but kubernetes starts with four initial namespaces the first one is the default namespace and this is for objects with no other namespace so when creating new objects without a namespace your object will automatically be assigned to this namespace cube dash system is the next one and these are for objects created by kubernetes cube-public is created automatically and is readable by all users but is mostly reserved for cluster usage in case that some resources should be visible and readable publicly throughout the whole cluster and finally cube node lease is the namespace for the lease objects associated with each node which improves the performance of the node heartbeats as the cluster scales and so like most resources in google cloud labels are key value pairs that help you organize your resources in this case kubernetes objects labels can be attached to objects at creation time and can be added or modified at any time each object can have a set of key value labels defined and each key must be unique for a given object and labels can be found under metadata in your manifest file and so the one thing to remember about pods is that they are ephemeral they are not designed to run forever and when a pod is terminated it cannot be brought back in general pods do not disappear until they are deleted by a user or by a controller pods do not heal or repair themselves for example if a pod is scheduled on a node which later fails the pod is deleted as well if a pod is evicted from a node for any reason the pod does not replace itself and so here is a diagram of a pod life cycle that shows the different phases of its running time to give you some better clarity of its ephemeral nature when first creating the pod the pod will start impending and this is the pod's initial phase and is waiting for one or more of the containers to be set up and made ready to run this includes the time a pod spends waiting to be scheduled as well as the time spent downloading container images over the network once the pod has completed the pending phase it is moved on to be scheduled and once it is scheduled it will move into the running phase and this is the phase where the pod has been bound to a node and all of the containers have been created the running phase has at least one container in the pod running or is in the process of starting or restarting and once the workload is complete the pod will move into the succeeded phase and this is where all the containers in the pod have terminated in success and will not be restarted now if all the containers in the pod have not terminated successfully the pod will move into a failed phase and this is where all the containers in the pod have terminated and at least one container has terminated in failure now there's one more phase in the pod life cycle that i wanted to bring up which is the unknown phase and this is the state of the pod that could not be obtained this phase typically occurs due to an error in communicating with the node where the pod should be running so now when you're creating pods using a deployment is a common way to do this a deployment runs multiple replicas of your application and automatically replaces any instances that fail or become unresponsive deployments help ensure that one or more instances of your application are available to serve user requests deployments use a pod template which contains a specification for its pods the pod specification determines how each pod should look like for instance what applications should run inside its containers which volumes the pods should mount its labels and more and so when a deployments pod template is changed new pods are automatically created one at a time now i wanted to quickly bring up replica sets for just a moment you'll hear about replica sets and i wanted to make sure that i covered it replica sets ensures that a specified number of pod replicas are running at any given time however a deployment is a higher level concept that manages replica sets and provides updates to pods along with other features and so using deployments is recommended over using replica sets unless your workload requires it and i will be including a link to replica sets in the lesson text so speaking of workloads in kubernetes workloads are objects that set deployment rules four pods based on these rules kubernetes performs the deployment and updates the workload with the current state of the application workloads let you define the rules for application scheduling scaling and upgrading now deployments which we just discussed is a type of workload and as we've seen a deployment runs multiple replicas of your application and automatically replaces any instances that fail or become unresponsive deployments are best used for stateless applications another type of workload is stateful sets and in contrast to deployments these are great for when your application needs to maintain its identity and store data so basically any application that requires some sort of persistent storage daemon sets is another common workload that ensures every node in the cluster runs a copy of that pod and this is for use cases where you're collecting logs or monitoring node performance now jobs is a workload that launches one or more pods and ensures that a specified number of them successfully terminate jobs are best used to run a finite task to completion as opposed to managing an ongoing desired application state and cron jobs are similar to jobs however cron jobs runs to completion on a cron-based schedule and so the last workload that i wanted to cover are config maps and these store general configuration information and so after you upload a config map any workload can reference it as either an environment variable or a volume mount and so just as a note config maps are not meant to store sensitive data if you're planning to do this please use secrets now i know this lesson has been extremely heavy in theory but these are fundamental concepts to know when dealing with kubernetes and gke as well as the objects that it supports so i recommend that if you need to go back and review this lesson if things aren't making sense so that you can better understand it as these concepts all tie in together and will come up in the exam and so that's pretty much all i wanted to cover in this lesson on pods and object management within gke so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this lesson i'm going to be diving into kubernetes services now services are a major networking component when it comes to working in kubernetes and can play a major factor when it comes to deciding on how you want to route your traffic within your kubernetes cluster as well in my experience services show up on the exam and so an understanding of how they work and the different types to use are essential to understanding the big picture of kubernetes this lesson will cover an overview on what services are what they do and the different types that are available along with their use cases now there's a lot to cover here so with that being said let's dive in now as i had discussed earlier kubernetes pods are ephemeral pods are created and destroyed to match the state of your cluster so these resources are never permanent a perfect example of this is by using a deployment object so you can create and destroy pods dynamically now when it comes to networking in kubernetes each pod gets its own ip address however in a deployment a pod that is running once destroyed will be recreated with a new ip address and there is no real way to keep track of these i p addresses for communication as they change very frequently and this is where services come into play now a service is an abstraction in the sense that it is not a process that listens on some network interface a service can be defined as a logical set of pods an abstraction on top of the pod which provides a single persistent ip address and dns name by which pods can be accessed it allows for routing external traffic into your kubernetes cluster and used inside your cluster for more intelligent routing with services it is also very easy to manage load balancing configuration for traffic between replicas it helps pods scale quickly and easily as the service will automatically handle the recreation of pods and their new ip addresses the main goal of services in kubernetes is to provide persistent access to its pods without the necessity to look for a pod's ip each time when the pod is recreated and again services also allow for external access from users to the applications inside the cluster without having to know the ip address of the individual pod in order to reach that application now in order for a service to route traffic to the correct pod in the cluster there are some fields in the manifest file that will help determine the end points on where traffic should be routed shown here on the right is the deployment manifest for reference and on the left is the services manifest now as you can see here in the service manifest on the left the kind is clearly defined as service under metadata is the name of the service and this will be the dns name of the service when it is created so when it comes to the spec there is a field here called a selector and this is what defines what pods should be included in the service and it is the labels under the selector that define which pods and labels are what we discussed in the last lesson as arbitrary key value pairs so any pod with these matching labels is what will be added to the service as shown here in the deployment file this workload will be a part of the service and its labels match that of the selector in the services file for type this is the type of service that you will want to use in this example type cluster ip is used but depending on the use case you have a few different ones to choose from now at the bottom here is a list of port configurations protocol being the network protocol to use with the port port being the port that incoming traffic goes to and finally the target port which is the port on the pod that traffic should be sent to and this will make more sense as we go through the upcoming diagrams so touching on selectors and labels for a moment kubernetes has a very unique way of routing traffic and when it comes to services it's not any different services select pods based on their labels now when a selector request is made to the service it selects all pods in the cluster matching the key value pair under the selector it chooses one of the pods if there are more than one with the same key value pair and forwards the network request to it and so here in this example you can see that the selector specified for the service has a key value pair of app inventory you can see the pod on node 1 on the left holds the label of app inventory as well which matches the key value pair of the selector and so traffic will get routed to that pod because of it if you look at the label for the pod in node 2 on the right the label does not match that of the selector and so it will not route traffic to that pod and so to sum it up the label on the pod matching the selector in the service determines where the network request will get routed to and so now i will be going through the many different service types that are available for routing network traffic within gke starting with cluster ip now a cluster ip service is the default kubernetes service it gives you a service inside your cluster that other apps inside your cluster can access the service is not exposed outside the cluster but can be addressed from within the cluster when you create a service of type cluster ip kubernetes creates a stable ip address that is accessible from nodes in the cluster clients in the cluster call the service by using the cluster ip address and the port value specified in the port field of the service manifest the request is forwarded to one of the member pods on the port specified in the target port field and just as a note this ip address is stable for the lifetime of the service so for this example a client calls the service at 10.176 on tcp port 80. the request is forwarded to one of the member pods on tcp port 80. note that the member pod must have a container that is listening on tcp port 80. if there is no container listening on port 80 clients will see a message like fail to connect or this site can't be reached think of the case when you have a dns record that you don't want to change and you want the name to resolve to the same ip address or you merely want a static ip address for your workload this would be a great use case for the use of the cluster ip service now although the service is not accessible by network requests outside of the cluster if you need to connect to the service you can still connect to it with the cloud sdk or cloud shell by using the exposed ip address of the cluster and so i wanted to take a moment to show you what a cluster ip manifest actually looks like and i will be going through the manifest for each service type for you to familiarize yourself with we first have the name of the service which is cluster ip dash service we then have the label used for the selector which is the key value pair of app inventory and then we have the service type which is cluster ip and we have the port number exposed internally in the cluster which is port 80 along with the target port that containers are listening on which again is port 80. and so the next service type we have is node port so when you create a service of type node port you specify a node port value the node port is a static port and is chosen from a pre-configured range between 30 000 and 32 760 you can specify your own value within this range but please note that any value outside of this range will not be accepted by kubernetes as well if you do not choose a value a random value within the range specified will be assigned once this port range has been assigned to the service then the service is accessible by using the ip address of any node along with the no port value the service is then exposed on a port on every node in the cluster the service can then be accessed externally at the node ip along with the node port when using node port services you must make sure that the selected port is not already open on your nodes and so just as a note the no port type is an extension of the cluster i p type so a service of type node port naturally has a cluster i p address and so this method isn't very secure as it opens up each node to external entry as well this method relies on knowing the ip addresses of the nodes which could change at any time and so going through the manifest of type node port service we start off with the name of the service which is node port dash service the label used for the selector which uses the key value pair of app inventory the type which is node port and notice the case sensitivity here which you will find in most service types along with the port number exposed internally in the cluster which is port 80 and again the port that the containers are listening on which is the target port which is port 80 as well and lastly and most importantly we have the no port value which is marked as you saw in the diagram earlier as port 32002 the next service type we have up is low balancer and this service is exposed as a load balancer in the cluster low balancer services will create an internal kubernetes service that is connected to a cloud provider's load balancer and in this case google cloud this will create a static publicly addressable ip address and a dns name that can be used to access your cluster from an external source the low balancer type is an extension of the no port type so a service of type load balancer naturally has a cluster ip address if you want to directly expose a service this is the default method all traffic on the port you specify will be forwarded to the service there is no filtering or routing and it means you can send many different types of traffic to it like http https tcp or udp and more the downside here is that for each service you expose with a low balancer you pay for that load balancer and so you can really rack up your bill if you're using multiple load balancers and shown here is the manifest for type load balancer it shows the name of the service load balancer dash service the label which is used for the selector which is the key value pair of app inventory the service type which is low balancer again notice the case sensitivity along with the port and the target port which are both port 80. and so this is the end of part one of this lesson it was getting a bit long so i decided to break it up this would be a great opportunity for you to get up and have a stretch get yourself a coffee or tea and whenever you're ready part two will be starting immediately from the end of part one so go ahead and mark this as complete and i'll see you in the next one [Music] welcome back this is part two of the kubernetes services lesson and we're going to continue immediately from the end of part one so whenever you're ready let's dive in and so the next service type we have is multiport services now for some services there is the need to expose more than one port kubernetes lets you configure multiple port definitions on a service object so when using multiple ports for a service you must give all your ports names and if you have multiple service ports these names must be unique in this example if a client calls the service at 10.176.1 on tcp port 80 the request is forwarded to a member pod on tcp port 80 on either node 1 or node 2. but if a client calls the service at 10.176.133.7 on tcp port 9752 the request is forwarded to the pod on tcp port 9752 that resides on node 1. each member pod must have a container listening on tcp port 80 and a container listening on tcp port 9752 this could be a single container with two threads or two containers running in the same pod and of course as shown here is a manifest showing the multi-port services the name of the service the label used for the selector as well as the service type the port node exposed internally for each separate workload as well as the port that containers are listening on for each workload as well and as you saw before nginx was using target port 80 where appy was using port 9752 moving on to another service type is external name now a service of type external name provides an internal alias for an external dns name internal clients make requests using the internal dns name and the requests are redirected to the external name when you create a service kubernetes creates a dns name that internal clients can use to call the service in this example the internal dns name is bowtie.sql when an internal client makes a request to the internal dns name of bowtie.sql the request gets redirected to bowtie.sql2 dot bow tie inc dot private the external name service type is a bit different than other service types as it's not associated with a set of pods or an ip address it is a mapping from an internal dns name to an external dns name this service does a simple cname redirection and is a great use case for any external service that resides outside of your cluster and again here is a view of a manifest for type external name here showing the internal dns name along with the external dns name redirect and moving on to the last service type we have the headless service type now sometimes you don't need or want low balancing and a single service ip in this case you can create headless services by specifying none as the service type in the manifest file this option also allows you to choose other service discovery mechanisms without being tied to kubernetes implementation applications can still use a self-registration pattern with this service and so a great use case for this is when you don't need any low balancing or routing you only need the service to patch the request to the back end pod no ips needed headless service is typically used with stateful sets where the name of the pods are fixed this is useful in situations like when you're setting up a mysql cluster where you need to know the name of the master and so here is a manifest for the headless service again the service type is marked as none and so to sum it up kubernetes services provides the interfaces through which pods can communicate with each other they also act as the main gateway for your application services use selectors to identify which pods they should control they expose an ip address and a port that is not necessarily the same port at which the pod is listening and services can expose more than one port and can also route traffic to other services external ip addresses or dns names services make it really easy to create network services in kubernetes each service can be backed with as many pods as needed without having to make your code aware of how each service is backed also please note that there are many other features and use cases within the services that have been mentioned that i've not brought up i will also include some links in the lesson text for those who are interested in diving deeper into services this lesson was to merely summarize the different service types and knowing these service types will put you in a great position on the exam for any questions that cover services within gke now i know this has been another lesson that's been extremely heavy in theory and has been a tremendous amount to take in but not to worry next up is a demo that will put all this theory into practice and we'll be going ahead and building a cluster along with touching on much of the components discussed within the past few lessons and so that's pretty much all i wanted to cover when it comes to kubernetes service types so you can now mark this lesson as complete and whenever you're ready join me in the console [Music] welcome back in this lesson i'll be going over ingress for gke an object within gke that defines rules for routing traffic to specific services ingress is a well-known topic that comes up in the exam as well as being a common resource that is used in many gke clusters that you will see in most environments something that you will get very familiar with while diving deeper into more complex environments so whenever you're ready let's dive in now in gke an ingress object defines rules for routing http and https traffic to applications running in a cluster an ingress object is associated with one or more service objects each of which is associated with a set of pods when you create an ingress object the gke ingress controller creates a google cloud http or https load balancer and configures it according to the information in the ingress and its associated services gke ingress is a built-in and managed ingress controller this controller implements ingress resources as google cloud load balancers for http and https workloads in gke also the load balancer is given a stable ip address that you can associate with a domain name each external http and https load balancer or internal http or https load balancer uses a single url map which references one or more back-end services one back-end service corresponds to each service referenced by the ingress in this example assume that you have associated the load balancers ip address with the domain name bowtieinc.co when a client sends a request to bowtieinc.co the request is routed to a kubernetes service named products on port 80. and when a client sends a request to bowtieinc.co forward slash discontinued the request is routed to a kubernetes service named discontinued on port 21337 ingress is probably the most powerful way to expose your services but can also be very complex as there are also many types of ingress controllers to choose from along with plugins for ingress controllers ingress is the most useful and cost effective if you want to expose multiple services under the same ip address as you only pay for one load balancer if you are using the native gcp integration and comes with a slew of features and so shown here is the ingress manifest which is a bit different from the other manifest that you've seen as it holds rules for different paths explain in the previous diagram in the manifest shown here one path directs all traffic to the product's service name while the other path redirects traffic from discontinued to the back end service name of discontinued and note that each of these service names have their own independent manifest as it is needed to create the service and are referenced within the ingress manifest so the more rules you have for different paths or ports the more services you will need now i wanted to touch on network endpoint groups or any g's for short for just a second now this is a configuration object that specifies a group of back-end endpoints or services negs are useful for container native load balancing where each container can be represented as an endpoint to the load balancer the negs are used to track pod endpoints dynamically so the google low balancer can route traffic to its appropriate back ends so traffic is low balanced from the load balancer directly to the pod ip as opposed to traversing the vm ip and coupe proxy networking in these conditions services will be annotated automatically indicating that a neg should be created to mirror the pod ips within the service the neg is what allows compute engine load balancers to communicate directly with pods the diagram shown here is the ingress to compute engine resource mappings of the manifest that you saw earlier where the gke ingress controller deploys and manages compute engine low balancer resources based on the ingressed resources that are deployed in the cluster now touching on health checks for just a minute if there are no specified health check parameters for a corresponding service using a back-end custom resource definition a set of default and inferred parameters are used health check parameters for a back-end service should be explicitly defined by creating a back-end config custom resource definition for the service and this should be done if you're using anthos a backend config custom resource definition should also be used if you have more than one container in the serving pods as well if you need control over the port that's used for the low balancers health checks now you can specify the backend services health check parameters using the health check parameter of a back-end config custom resource definition referenced by the corresponding service this gives you more flexibility and control over health checks for a google cloud external http or https load balancer or internal http or https load balancer created by an ingress and lastly i wanted to touch on ssl certificates and there are three ways to provide ssl certificates to an http or https load balancer the first way is google managed certificates and these are provisioned deployed renewed and managed for your domains and just as a note managed certificates do not support wildcard domains the second way to provide ssl certificates is through self-managed certificates that are shared with google cloud you can provision your own ssl certificate and create a certificate resource in your google cloud project you can then list the certificate resource in an annotation on an ingress to create an http or https load balancer that uses the certificate and the last way to provide ssl certificates is through self-managed certificates as secret resources so you can provision your own ssl certificate and create a secret to hold it you can then refer to the secret as an ingress specification to create an http or https load balancer that uses this certificate and just as a note you can specify multiple certificates in an ingress manifest the load balancer chooses a certificate if the common name in the certificate matches the host name used in the request and so that pretty much covers all the main topics in this short lesson on ingress for gke so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in this lesson i'll be going over gke storage options now kubernetes currently offers a slew of different storage options and is only enhanced by the added features available in google cloud for gke we'll also be getting into the different abstractions that kubernetes offers to manage storage and how they can be used for different types of workloads now there's quite a bit to go over here so with that being said let's dive in now as i stated before there are several storage options for applications running on gke the choices vary in terms of flexibility and ease of use google cloud offers several storage options that can be used for your specific workload kubernetes also provides storage abstractions which i will be getting into in just a bit the easiest storage options are google cloud's managed storage products if you need to connect a database to your cluster you can consider using cloud sql datastore or cloud spanner and when it comes to object storage cloud storage would be an excellent option to fill the gap file store is a great option for when your application requires managed network attached storage and if your application requires block storage the best option is to use persistent disks and can be provisioned manually or provisioned dynamically through kubernetes now i wanted to first start off with kubernetes storage abstractions but in order to understand kubernetes storage abstractions i wanted to take a moment to explain how storage is mounted in the concept of docker now docker has a concept of volumes though it is somewhat looser and less managed than kubernetes a docker volume is a directory on disk or in another container docker provides volume drivers but the functionality is somewhat limited a docker container has a writable layer and this is where the data is stored by default making the data ephemeral and so data is not persisted when the container is removed so storing data inside a container is not always recommended now there are three ways to mount data inside a docker container the first way is a docker volume and sits inside the docker area within the host's file system and can be shared amongst other containers this volume is a docker object and is decoupled from the container they can be attached and shared across multiple containers as well bind mounting is the second way to mount data and is coming directly from the host's file system bind mounts are great for local application development yet cannot be shared across containers and the last way to mount data is by using temp-fs and is stored in the host's memory this way is great for ephemeral data and increases performance as it no longer lies in the container's writable layer now with kubernetes storage abstractions file system and block based storage are provided to your pods but are different than docker in nature volumes are the basic storage unit in kubernetes that decouples the storage from the container and tie it to the pod and not the container like in docker a regular volume simply called volume is basically a directory that the containers in a pod have access to the particular volume type used is what will determine its purpose some volume types are backed by ephemeral storage like empty dir config map and secrets and these volumes do not persist after the pod ceases to exist volumes are useful for caching temporary information sharing files between containers or to load data into a pod other volume types are backed by durable storage and persist beyond the lifetime of a pod like persistent volumes and persistent volume claims a persistent volume is a cluster resource that pods can use for durable storage a persistent volume claim can be used to dynamically provision a persistent volume backed by persistent disks persistent volume claims can also be used to provision other types of backing storage like nfs and i will be getting more into persistent volumes and persistent volume claims in just a bit now as you saw in docker on disk files in a container are the simplest place for an application to write data but files are lost when the container crashes or stops for any other reason as well as being unaccessible to other containers running in the same pod in kubernetes the volume source declared in the pod specification determines how the directory is created the storage medium used and the directory's initial contents a pod specifies what volumes it contains and the path where containers mount the volume ephemeral volume types live the same amount of time as the pods they are connected to these volumes are created when the pod is created and persist through container restarts only when the pod terminates or is deleted are the volumes terminated as well other volume types are interfaces to durable storage that exist independently of a pod like ephemeral volumes data in a volume backed by durable storage is preserved when the pod is removed the volume is merely unmounted and the data can be handed off to another pod now volumes differ in their storage implementation and their initial contents you can choose the volume source that best fits your use case and i will be going over some common volume sources that are used and you will see in many gke implementations the first volume that i want to bring up is empty dir now an empty dir volume provides an empty directory that containers in the pod can read and write from when the pod is removed from a node for any reason the data in the empty dir is deleted forever an empty dir volume is stored on whatever medium is backing the node which might be a disk ssd or network storage empty der volumes are useful for scratch space and sharing data between multiple containers in a pod the next type of volume that i wanted to go over is config map and config map is a resource that provides a way to inject configuration data into pods the data stored in a config map object can be referenced in a volume of type config map and then consumed through files running in a pod the next volume type is secret and a secret volume is used to make sensitive data such as passwords oauth tokens and ssh keys available to applications the data stored in a secret object can be referenced in a volume of type secret and then consumed through files running in a pod next volume type is downward api and this volume makes downward api data available to applications so this data includes information about the pod and container in which an application is running in an example of this would be to expose information about the pods namespace and ip address to applications and the last volume type that i wanted to touch on is persistent volume claim now a persistent volume claim volume can be used to provision durable storage so that they can be used by applications a pod uses a persistent volume claim to mount a volume that is backed by this durable storage and so now that i've covered volumes i wanted to go into a bit of detail about persistent volumes persistent volume resources are used to manage durable storage in a cluster in gke a persistent volume is typically backed by a persistent disk or file store can be used as an nfs solution unlike volumes the persistent volume life cycle is managed by kubernetes and can be dynamically provisioned without the need to manually create and delete the backing storage persistent volume resources are cluster resources that exist independently of pods and continue to persist as the cluster changes and as pods are deleted and recreated moving on to persistent volume claims this is a request for and claim to a persistent volume resource persistent volume claim objects request a specific size access mode and storage class for the persistent volume if an existing persistent volume can satisfy the request or can be provisioned the persistent volume claim is bound to that persistent volume and just as a note pods use claims as volumes the cluster inspects the claim to find the bound volume and mounts that volume for the pod now i wanted to take a moment to go over storage classes and how they apply to the overall storage in gke now these volume implementations such as gce persistent disk are configured through storage class resources gke creates a default storage class for you which uses the standard persistent disk type of ext4 as shown here the default storage class is used when a persistent volume claim doesn't specify a storage class name and can also be replaced with one of your choosing you can even create your own storage class resources to describe different classes of storage and is helpful when using windows node pools now as i stated before persistent volume claims can automatically provision persistent disks for you when you create this persistent volume claim object kubernetes dynamically creates a corresponding persistent volume object due to the gke default storage class this persistent volume is backed by a new empty compute engine persistent disk you use this disk in a pod by using the claim as a volume when you delete a claim the corresponding persistent volume object and the provision compute engine persistent disk are also deleted now to prevent deletion you can set the reclaim policy of the persistent disk resource or its storage class resource to retain now deployments as shown here in this diagram are designed for stateless applications so all replicas of a deployment share the same persistent volume claim which is why stateful sets are the recommended method of deploying stateful applications that require a unique volume per replica by using stateful sets with persistent volume claim templates you can have applications that can scale up automatically with unique persistent volume claims associated to each replica pod now lastly i wanted to touch on some topics that will determine the storage access that is available for any gke cluster in your environment now i first wanted to start off with access modes and there are three supported modes for your persistent disks that allow read write access and are listed here read write once is where the volume can be mounted as read write by a single node read only many is where the volume can be mounted as a read only by many nodes and lastly read write many is where the volume can be mounted as read write by many nodes and just as a note read write once is the most common use case for persistent disks and works as the default access mode for most applications next i wanted to touch on the type of persistent disks that are available and the benefits and caveats of access for each now going through the persistent disks lesson of this course you probably know by now about the available persistent disks when it comes to zonal versus regional availability and so this may be a refresher for some now going into regional persistent disks these are multi-zonal resources that replicate data between two zones in the same region and can be used similarly to zonal persistent disks in the event of a zonal outage kubernetes can fail over workloads using the volume to the other zone regional persistent disks are great for highly available solutions for stateful workloads on gke now zonal persistent disks are zonal resources and so unless a zone is specified gke assigns the disk to a single zone and chooses the zone at random once a persistent disk is provisioned any pods referencing the disk are scheduled to the same zone as the disk and just as a note using anti-affinity on zones allows stateful set pods to be spread across zones along with the corresponding disks and the last point that i wanted to cover when it comes to persistent volume access is the speed of access now as stated in an earlier lesson the size of persistent disks determine the iops and throughput of the disk gke typically uses persistent disks as boot disks and to back kubernetes persistent volumes so whenever possible use larger and fewer disks to achieve higher iops and throughput and so that pretty much covers everything that i wanted to go over in this lesson on gke storage options so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in these next few demos i'm going to be doing a complete walkthrough and putting all the theory we learned into practice through building and interacting with gke clusters and you'll be building and deploying your own containerized application on this cluster called box of bowties so in this demo we're going to be setting up our own gke cluster in the console along with going through all the options that are available when deploying it we're also going to use the command line to configure the cubectl command line tool so that we can interact with the cluster so with that being said let's dive in and so here in the console i am logged in as tonybowties gmail.com under the project of bow tie inc and so before launching the cluster i need to make sure that my default vpc has been created so i'm going to go over to the navigation menu and i'm going to scroll down to vpc network and as expected the default network is here so i can go ahead and create my cluster and so in order to get to my kubernetes engine console i'm going to go up to the navigation menu and i'm going to scroll down under compute and you will find here kubernetes engine and you'll see a few different options to choose from and over here on the left hand menu i will be going through these options in the upcoming demos but for now i want to concentrate on creating our cluster now gk makes things pretty easy as i have the option to create a cluster to deploy a container or even taking the quick start and so we're going to go ahead and click on create our cluster and so here we are prompted with our cluster basics now if i really wanted to i can simply fill out all the fields that you see here and click on create and it will use all the defaults to build my cluster but we're going to customize it a little bit so we're going to go ahead and go through all these options so first under name we're going to name this cluster bowtie dash cluster and so under location type we want to keep things as zonal and if i check off the specify default node locations i'll be able to make this a multi-zonal cluster as i have the option of selecting from multiple zones where i can situate my nodes and so i can select off a bunch of different zones if i choose but we want to keep it as a single zonal cluster and so i'm going to check these all off and under zone i'm going to click on the drop down menu and i'm going to select us east 1b and just as a note for each zone that you select this is where the control plane will live so if i was to create a multi-zonal cluster as you can see the master zone is the zone where the control plane will be created and is selected as us east 1b as that is the zone that i had selected and so if i change this to let's say us east 1d you can see that the control plane will change with it so i'm going to change it back to us east 1b and you also have the option of creating a regional cluster and the location selection will change from zone to region and here you will have to specify at least one zone to select but please also remember that the same number of nodes will be deployed to each selected zone so if i have three nodes in this cluster and i decide to select three zones then i will have nine nodes in this cluster and so doing something like this could get quite pricey when you're looking to be cost conscious okay so moving on i'm going to uncheck specify default node locations i'm going to change the location type back to zonal and make sure that my zone is at us east 1b moving down to the master version this is where we would select either a static version or opt-in to a release channel for the version of kubernetes that you want for your cluster and so with the static version i can choose from a bunch of different versions here all the way back from 1.14.10 all the way to the latest version and so with the release channel i have the release channel selection here and i can choose from the rapid channel the regular channel or the stable channel and so i'm going to keep things as the default with the regular channel as well i'm going to keep the default version as the version of my choice now i could go ahead and simply click on create here but as this demo is a walkthrough i'm going to go ahead and go through all the available options so i'm going to start by going over to the left hand menu and clicking on default pool under no pools now here i have one node pool already with three nodes and this is the default node pool that comes with any cluster but if i was doing something specific i could add another node pool and configure it from here but because i don't have a need for two node pools i'm gonna go ahead and remove nodepool1 so i'm going to go up here to remove nodepool and as you can see gke makes it really easy for me to add or remove node pools so i'm going to go back to the default pool and i'm going to keep the name as is i'm gonna keep my number of nodes as three and if i wanted to change the number of nodes i can simply select this i can choose six or however many nodes you need for your workload and so because we're not deploying a large workload i'm gonna keep this number at 3 and moving right along we do want to check off enable auto scaling and so this way we don't have to worry about scaling up or scaling down and here i'm going to put the minimum number of nodes as one and i'm going to keep my maximum number of nodes at 3. and so here i'm given the option to select the zone location for my nodes but again for each zone that i select it will run the same amount of nodes so basically i have another option in order to choose from having a zonal or multi-zonal cluster and because we're creating our cluster in a single zone i'm going to uncheck this and under automation as you can see enable auto upgrade and enable auto repair are both checked off and this is due to the fact that the auto upgrade feature is always enabled for the release channel that i selected but as i pointed out in a previous lesson that this is google's best practice to have auto upgrade and auto repair enabled and so moving down to the bottom are some fields to change the surge upgrade behavior and so just as a refresher surge upgrades allow you to control the number of nodes gke can upgrade at a time and control how disruptive those upgrades are to your workloads so max surge being the number of additional nodes that can be added to the node pool during an upgrade and max unavailable being the number of nodes that can be simultaneously unavailable during that upgrade and because we're not worried about disruptions we'll just leave it set as the default and so moving on we're going to move back over to the left hand menu and under no pools we're going to click on nodes and here is where i can choose the type of instance that i want to be using for my nodes and so i'm going to keep the image type as container optimize os and this is the default image type but i also have the option of choosing from others like ubuntu or windows and so i'm going to keep it as the default and under machine configuration i'm going to keep it under general purpose with series e2 but i do want to change the machine type to e2 micro just to be cost conscious and under boot disk size i want to keep it as 10 gigabytes as we don't really need 100 gigabytes for what we're doing here and you also have the option of choosing from a different boot disk type you can change it from standard persistent disk to ssd but i'm going to keep things as standard as well i also have the option here to use customer manage keys for encryption on my boot disk as well as selecting from preemptable nodes for some cost savings and so i'm going to now move down to networking and here if i wanted to get really granular i can add a maximum pods per node as well as some network tags but our demo doesn't require this so i'm going to leave it as is and i'm going to go back over to the left hand menu and click on security and under node security you have the option of changing your service account along with the access scopes and so for this demo we can keep things as the default service account and the access scopes can be left as is i'm going to go back over to the left hand menu and click on metadata and here i can add kubernetes labels as well as the instance metadata and so i know i didn't get into node taints but just to fill you in on no taints when you submit a workload to run in a cluster the scheduler determines where to place the pods associated with the workload and so the scheduler will place a pod on any node that satisfies the resource requirements for that workload so no taints will give you some more control over which workloads can run on a particular pool of nodes and so they let you mark a node so that the scheduler avoids or prevents using it for certain pods so for instance if you had a node pool that is dedicated to gpus you'd want to keep that node pool specifically for the workload that requires it and although it is in beta this is a great feature to have and so that pretty much covers no pools as we see it here and so this is the end of part one of this demo it was getting a bit long so i decided to break it up this would be a great opportunity for you to get up and have a stretch get yourself a coffee or a tea and whenever you're ready part two will be starting immediately from the end of part one so you can now mark this as complete and i'll see you in the next one [Music] this is part two of creating a gke cluster part 2 will be starting immediately from the end of part 1. so with that being said let's dive in and so i'm going to go back over to the left hand menu and under cluster i'm going to click on automation and here i have the option of enabling a maintenance window for aligning times when auto upgrades are allowed i have the option of adding the window here and i can do it at specified times during the week or i can create a custom maintenance window and so we don't need a maintenance window right now so i'm going to uncheck this and as well you have the option of doing maintenance exclusions for when you don't want maintenance to occur ngk gives you the option of doing multiple maintenance exclusions for whenever you need them and because we don't need any maintenance exclusions i'm going to delete these and here you have the option to enable vertical pod auto scaling and this is where gke will automatically schedule pods onto other nodes that satisfy the resources required for that workload as well here i can enable my node auto provisioning and enabling this option allows gke to automatically manage a set of node pools that can be created and deleted as needed and i have a bunch of fields that i can choose from the resource type the minimum and maximum for cpu and memory the service account as well as adding even more resources like gpus but our workload doesn't require anything this fancy so i'm going to delete this and i'm going to uncheck enable auto provisioning and lastly we have the auto scaling profile and i have the option from choosing the balance profile which is the default as well as the optimize utilization which is still in beta and so i'm going to keep things as the default and i'm going to move back on over to the left hand menu over to networking and so here i can get really granular with my cluster when it comes to networking i have the option of choosing from a public or a private cluster as well i can choose from a different network and since we only have the default that's what shows up but if you had different networks here you can choose from them as well as the subnets i can also choose from other networking options like pod address range maximum pods per node and there's a bunch of other options which i won't get into any detail with but i encourage you if you're very curious to go through the docs and to check out these different options now the one thing that i wanted to note here is the enable http low balancing and this is a add-on that is required in order to use google cloud load balancer and so as we discussed previously in the services lesson when you enable service type load balancer a load balancer will be created for you by the cloud provider and so google requires you to check this off so that a controller can be installed in the cluster upon creation and will allow a load balancer to be created when the service is created and so i'm going to leave this checked as we will be deploying a load balancer a little bit later and so moving back over to the left hand menu i'm going to now click on security and there are many options here to choose from that will allow you to really lock down your cluster and again this would all depend on your specific type of workload now i'm not going to go through all these options here but i did want to highlight it for those who are looking to be more security focused with your cluster and so moving down the list in the menu i'm going to click on metadata and so here i can enter a description for my cluster as well as adding labels and so the last option on the cluster menu is features and here i have the option of running cloud run for anthos which will allow you to deploy serverless workloads to anthos clusters and runs on top of gke and here you can enable monitoring for gke and have it be natively monitored by google cloud monitoring and if i was running a third-party product to monitor my cluster i can simply uncheck this and use my third-party monitoring and there's a whole bunch of other features that i won't dive into right now but if you're curious you can always hover over the question mark and get some more information about what it does and so now i've pretty much covered all the configuration that's needed for this cluster and so now i'm going to finally head down to the bottom and click on create and so it may take a few minutes to create this cluster so i'm going to go ahead and pause this video here and i'll be back faster than you can say cat in the hat okay and the cluster has been created as you can see it's in the location of us east 1b with three nodes six vcpus and three gigabytes of memory and i can drill down and see exactly the details of the cluster as well if i wanted to edit any of these options i can simply go up to the top click on edit and make the necessary changes and so now you're probably wondering what will i need to do in order to create this cluster through the command line well it's a bit simpler than what you think and i'm going to show you right now i'm going to simply go over to the right hand menu and activate cloud shell and bring this up for better viewing and i'm going to paste in my command gcloud container clusters create bow tie dash cluster with the flag num nodes and the number of nodes that i choose which is three and so like i said before if i wanted to simply create a simple cluster i can do so like this but if i wanted to create the cluster exactly how i built my last cluster then i can use this command which has all the necessary flags that i need to make it customize to my liking a not so very exciting demonstration but at the same time shows you how easy yet powerful gke really is and so i'm not going to launch this cluster as i already have one and so now i wanted to show you how to interact with your new gke cluster so i'm going to simply clear my screen and so now in order for me to interact with my cluster i'm going to be using the cube ctl command line tool and this is the tool that is used to interact with any kubernetes cluster no matter the platform now i could use the gcloud container commands but they won't allow me to get very granular as the cubectl tool and so a caveat of creating your cluster through the console is that you need to run a command in order to retrieve the cluster's credentials and configure the cubectl command line tool and i'm going to go ahead and paste that in now and the command is gcloud container clusters get dash credentials and the name of my cluster which is bow tie dash cluster along with the zone flag dash dash zone followed by the zone itself which is us east 1b i'm going to go ahead and hit enter and as you can see cubectl has now been configured and so now i'm able to interact with my cluster so just to verify i'm going to run the command cubectl getpods and naturally as no workloads are currently deployed in the cluster there are no pods so i'm going to run the command cube ctl get nodes and as you can see the cubectl command line tool is configured correctly and so now this cluster is ready to have workloads deployed to it and is also configured with the cubectl command line tool so that you're able to manage the cluster and troubleshoot if necessary now i know that there has been a ton of features that i covered but i wanted to give you the full walkthrough so that you are able to tie in some of the theory from the last few lessons and get a feel for the gke cluster as we will be getting more involved with it over the next couple of demos and so that's pretty much all i wanted to cover when it comes to creating and setting up a gke cluster so you can now mark this as complete and whenever you're ready join me in the console in the next one where you will be building your box a bow ties container to deploy to your new cluster but if you are not planning to go straight into the next demo i do recommend that you delete your cluster to avoid any unnecessary costs and recreate it when you are ready to go into the next demo [Music] welcome back now in the last lesson you built a custom gke cluster and configured the cube ctl command line tool to interact with the cluster in this lesson you're going to be building a docker image for a box of bow ties using cloud build which will then be pushed over to google cloud container registry so that you can deploy it to your current gke cluster and so as you can see there's a lot to do here so with that being said let's dive in so now the first thing that you want to do is to clone your repo within cloud shell so you can run the necessary commands to build your image so i'm going to go up here to the top right and i'm going to open up cloud shell i'm going to make sure that i'm in my home directory so i'm going to run the command cd space tilde hit enter and i'm in my home directory if i run the command ls i can see that i only have cloud shell.txt and so now i'm going to clone my github repository and i'll have a link in the instructions in the github repo as well as having it in the lesson text below and so the command would be git clone along with the https address of the github repo and i'm going to hit enter and it's finished cloning my repo i'm going to quickly clear my screen and i'm going to run the command ls and i can see my repo here and now i'm going to drill down into the directory by running cd google cloud associate cloud engineer if i run an ls i can see all my clone files and folders and so now the files that we need are going to be found in the box of bowties folder under kubernetes engine and containers so i'm going to change directories to that location and run ls and under box of bow ties is a folder called container which will have all the necessary files that you need in order to build your image we have the jpeg for box of bow ties we have the docker file and we have our index.html and so these are the three files that we need in order to build the image and so as i said before we are going to be using a tool called cloud build which we have not discussed yet cloudbuild is a serverless ci cd platform that allows me to package source code into containers and you can get really fancy with cloud build but we're not going to be setting up any ci cd pipelines we're merely using cloud build to build our image and to push it out to container registry as well container registry is google cloud's private docker repository where you can manage your docker images and integrates with cloud build gke app engine cloud functions and other repos like github or bitbucket and it allows for an amazing build experience with absolutely no heavy lifting and because you're able to build images without having to leave google cloud i figured that this would be a great time to highlight these services so getting back to it we've cloned the repo and so we have our files here in cloud shell and so what you want to do now is you want to make sure the cloud build api has been enabled as this is a service that we haven't used before now we can go through the console and enable the api there but i'm going to run it here from cloud shell and i'm going to paste in the command gcloud services enable cloudbuild.googleapis.com i'm going to hit enter and you should get a prompt asking you to authorize the api call you definitely want to authorize should take a few seconds all right and the api has been enabled for cloud build so now i'm going to quickly clear my screen and so because i want to show you exactly what cloud build is doing i want to head on over there through the console and so i'm going to go over to the navigation menu and i'm going to scroll down to tools until you come to cloud build and as expected there is nothing here in the build history as well not a lot here to interact with and so now you're going to run the command that builds the image and so you're going to paste that command into the cloud shell which is gcloud builds submit dash dash tag gcr.io which is the google cloud container registry our variable for our google cloud project along with the image name of box bow ties version 1.0.0 and please don't forget the trailing dot at the end i'm going to go ahead and hit enter cloud build will now compress the files and move them to a cloud storage bucket and then cloud build takes those files from the bucket and uses the docker file to execute the docker build process and so i'm going to pause the video here till the build completes and i'll be back in a flash okay and the image is complete and is now showing up in the build history in the cloud build dashboard and so if i want to drill down into the actual build right beside the green check mark you will see the hot link so you can just simply click on that and here you will see a build summary with the build log the execution details along with the build artifacts and as well the compressed files are stored in cloud storage and it has a hot link right here if i wanted to download the build log i can do so here and i conveniently have a hot link to the image of box of bow ties and this will bring me to my container registry so you can go ahead and click on the link it should open up another tab and bring you right to the page of the image that covers a lot of its details now the great thing i love about container registry is again it's so tightly coupled with a lot of the other resources within google cloud that i am able to simply deploy right from here and i can deploy to cloud run to gke as well as compute engine now i could simply deploy this image right from here but i wanted to do it from gke so i'm going to go back over to gke in the other tab i'm going to go to the navigation menu go down to kubernetes engine and i'm going to go up to the top menu and click on deploy it's going to ask for the image you want to deploy and you want to click on select to select a new container image and you should have a menu pop up from the right hand side of your screen and under container registry you should see box of bow ties you can expand the node here and simply click on the image and then hit select and so now the container image has been populated into my image path and you want to scroll down and if i wanted to i could add another container and even add some environment variables and so we're not looking to do that right now so you can simply click on continue and you're going to be prompted with some fields to fill out for your configuration on your deployment and so the application name is going to be called box of bow ties i'm going to keep it in the default namespace as well i'm going to keep the key value pair as app box of bow ties for my labels and because this configuration will create a deployment file for me you can always have a look at the manifest by clicking on the view yaml button before it's deployed and this is always good practice before you deploy any workload so as you can see here at the top i have the kind as deployment the name as well as the namespace my labels replicas of three as well as my selector and my spec down here at the bottom as well this manifest also holds another kind of horizontal pod auto scaler and is coupled with the deployment in this manifest due to the reference of the deployment itself and so it's always common practice to try and group the manifest together whenever you can and so this is a really cool feature to take advantage of on gke so i'm going to close this now and i'm actually going to close cloud shell as i don't need it right now as well you can see here that it's going to deploy to my kubernetes cluster of bow tie cluster in us east 1b and if i wanted to i can deploy it to a new cluster and if i had any other clusters in my environment they would show up here and i'd be able to select from them as well but bow tie cluster is the only one that i have and so now that you've completed your configuration for your deployment you can simply click on deploy this is just going to take a couple minutes so i'm just going to pause the video here and i'll be back as soon as the deployment is done okay the workload has been deployed and i got some default messages that popped up i can set an automated pipeline for this workload but we're not going to do that for this demo but feel free to try it on your own later if you'd like and we will want to expose our service as we want to see if it's up and running and we're going to take care of that in just a bit and so if i scroll through some of the details here i can see that i have some metrics here for cpu memory and disk the cluster namespace labels and all the pods that it's running on basically a live visual representation of my deployment if i scroll back up to the top i can dive into some details events and even my manifest i can also copy my manifest and download it if i'd like so as you can see a lot of different options and so now i want to verify my deployment and so i'm going to use the cube ctl command line tool to run some commands to verify the information so i'm going to open back up my cloud shell and make this a little bit bigger for better viewing and i'm going to run the command cubectl get all and as you can see here i have a list of all the pods that are running the name of the service the deployment the replica set everything about my cluster and my deployment and you should be seeing the same when running this command and so next you want to pull up the details on your deployments in the cluster and so the command for that is cube ctl get deployments and it came out kind of crammed at the bottom so i'm going to simply clear my screen and run that command again and as you can see the box of bowties deployment is displayed how many replicas that are available how many of those replicas achieve their desired state and along with how long the application has been running and so now i want to dive into my pods and in order to do that i'm going to run the command cube ctl get pods and here i can see all my pods now if i wanted to look at a list of events for a specific pod the command for that would be cubectl describe pod and then the name of one of the pods so i'm going to pick this first one copy that i'm going to paste it and i'm going to hit enter and here i can see all the events that have occurred for this pod as well i also have access to some other information with regards to volumes conditions and even the container and image ids and this is a great command to use for when you're troubleshooting your pods and you're trying to get to the bottom of a problem and so now the final step that you want to do is you want to be able to expose your application so you can check to see if it's running properly and so we're going to go ahead and do that through the console so i'm going to close down cloud shell and i'm going to go to overview and scroll down to the bottom click on the button that says expose and if i wanted to i can do it from up here in the top right hand corner where it says expose deployment so i'm going to click on expose and this probably looks very familiar to you as this is a graphical representation of the services manifest and so the port mapping here will cover the ports configuration of the services manifest starting here with port target port as well as protocol for target port i'm going to open up port 80. here under service type you have the option of selecting cluster ip node port or load balancer and the service type you want to use is going to be low balancer and we can keep the service name as box of bowties service and again you can view the manifest file for this service and you can copy or download it if you need to but we don't need this right now so i'm going to close it in a pretty simple process so all i need to do is click on expose and within a minute or two you should have your service up and running with your shiny new low balancer okay and the service has been created and as you can see we're under the services and ingress from the left hand menu and if i go back to the main page of services in ingress you can see that box a bow tie service is the only one that's here i also have the option of creating a service type ingress but we don't want to do that right now so i'm going to go back to services and here you will see your endpoint and this is the hot link that should bring you to your application so you can click on it now you'll get a redirect notice as it is only http and not https so it's safe to click on it so i'm going to click on it now and success and here is your box of bow ties what were you expecting and so i wanted to congratulate you on deploying your first application box of bow ties on your gke cluster and so just as a recap you've cloned your repo into your cloud shell environment you then built a container image using cloud build and pushed the image to container registry you then created a deployment using this image and verified the deployment using the cube ctl command line tool you then launched a service of type low balancer to expose your application and verified that your application was working so fantastic job on your part and that's pretty much all i wanted to cover in this part of the demo so you can now mark this as complete and whenever you're ready join me in the console for the next part of the demo where you will manage your workload on the gke cluster so please be aware of the charges incurred on your currently deployed cluster if you plan to do the next demo at a later date again you can mark this as complete and i'll see you in the next welcome back in the last couple of demo lessons you built a custom gke cluster and deployed the box of bowties application in this lesson you will be interacting with this workload on gke by scaling the application editing your application and rebuilding your docker image so you can do a rolling update to the current workload in your cluster now there's a lot to do here so with that being said let's dive in so continuing where we left off you currently have your box of bow ties workload deployed on your gke cluster and so the first thing you want to do is scale your deployment and you are looking to scale down your cluster to one pod and then back up again to three and this is just to simulate scaling your workload so whether it be ten pods or one the action is still the same so now we can easily do it through the console by drilling down into the box of bowties workload going up to the top menu and clicking on actions and clicking on scale and here i can indicate how many replicas i'd like and scale it accordingly and so i wanted to do this using the command line so i'm going to cancel out of here and then i'm going to open up cloud shell instead okay and now that you have cloud shell open up you want to run the command cube ctl get pods to show the currently running available pods for the box of bowties workload and you may get a pop-up asking you to authorize the api call using your credentials and you definitely want to authorize and here you will get a list of all the pods that are running your box of bow ties workload and so now since you want to scale your replicas down to one you can run this command cube ctl scale deployment and your workload which is box of bowties dash dash replicas is equal to one you can hit enter and it is now scaled and in order to verify that i'm going to run cube ctl get pods and notice that there is only one pod running with my box of bow ties workload and in order for me to scale my deployment back up to three replicas i can simply run the same command but change the replicas from 1 to 3. hit enter it's been scaled i'm going to run cube ctl get pods and notice that i am now back up to 3 replicas and so as you can see increasing or decreasing the number of replicas in order to scale your application is pretty simple to do okay so now that you've learned how to scale your application you're gonna learn how to perform a rolling update but in order to do that you need to make changes to your application and so what you're going to do is edit your application then rebuild your docker image and apply a rolling update and in order to do that we can stay here in cloud shell as you're going to edit the file in cloud shell editor i'm going to first clear my screen i'm going to change directory into my home directory and now you want to change directories to your container folder where the files are that i need to edit i'm going to run ls and here's the files that i need and so what you're going to do now is edit the index.html file and the easiest way to do that is to simply type in edit index.html and hit enter and this will open up your editor so you can edit your index.html file and if you remember when we launched our application it looked exactly like this and so instead of what were you expecting we're going to actually change that text to something a little different and so i'm going to go back to the editor in my other tab and where it says what were you expecting i'm going to actually change this to well i could always use something to eat then i'm going to go back up to the menu click on file and click on save and so now in order for me to deploy this i need to rebuild my container and so i'm going to go back to my terminal i'm going to clear the screen and i'm going to run the same command that i did the last time which is gcloud build submit dash dash tag gcr dot io with the variable for your google cloud project followed by the image box of bowties colon 1.0.1 and so this will be a different version of the image also don't forget that trailing dot at the end and you can hit enter and again this is the process where cloud build compresses the files moves them to a cloud storage bucket and then takes the files from the bucket and uses the docker file to execute the docker build process and this will take a couple minutes so i'm going to pause the video here and i'll be back before you can say cat in the hat okay and my new image has been created and so i want to head over to cloud build just to make sure that there are no errors so i'm going to close down cloud shell because i don't need it right now i'm going to head back up to the navigation menu and scroll down to cloud build and under build history you should see your second build and if you drill down into it you will see that the build was successful and heading over to build artifacts you should now see your new image as version 1.0.1 and so now i'm going to head over to the registry and verify the image there and it seems like everything looks okay so now i'm gonna head back on over to my gke cluster i'm gonna go to the navigation menu down to kubernetes engine and here i'm gonna click on workloads i'm gonna select box of bowties and up at the top menu you can click on actions and select a rolling update and here you are prompted with a pop-up where you can enter in your minimum seconds ready your maximum search percentage as well as your maximum unavailable percentage and so here under container images i am prompted to enter in the sha-256 hash of this docker image now a docker image's id is a digest which contains a sha-256 hash of the image's configuration and if i go back over to the open tab for container registry you can see here the digest details to give you a little bit more context along with the sha 256 hash for the image that i need to deploy and so you can copy this digest by simply clicking on the copy button and then you can head back on over to the gke console head over to the container images highlight the hash and paste in the new hash and so when you copy it in make sure it's still in the same format of gcr dot io forward slash your project name forward slash box of bow ties the at symbol followed by the hash and so once you've done that you can click on the update button and this will schedule an update for your application and as you can see here at the top it says that pods are pending as well if i go down to active revisions you can see here that there is a summary and the status that pods are pending and so just as a note rolling updates allow the deployments update to take place with zero downtime by incrementally updating pods instances with new ones so the pods will be scheduled on nodes with available resources and if the nodes do not have enough resources the pods will stay in a pending state but i don't think we're going to have any problems with these nodes as this application is very light in resources and if i open up cloud shell and run a cube ctl get pods command you will see that new pods have started and you can tell this by the age of the pod as well if you ran the command keep ctl describe pod along with the pod name you could also see the event logs when the pod was created and if i close cloud shell i can see up here at the top of my deployment details it shows that my replicas have one updated four ready three available and one unavailable and if i click on refresh i can see now that my replicas are all updated and available and so now in order to check your new update you can simply go down to exposing services and click on the endpoints link you'll get that redirect notice you can simply click on the link and because the old site may be cached in your browser you may have to refresh your web page and success and you have now completed a rolling update in gke so i wanted to congratulate you on making it to the end of this multi-part demo and hope that it's been extremely useful in excelling your knowledge in gke and so just as a recap you scaled your application to accommodate both less and more replicas you edited your application in the cloud shell editor and rebuilt your container image using cloud build you then applied the new digest to your rolling update and applied that rolling update to your deployment while verifying it all in the end fantastic job on your part as this was a pretty complex and long multi-part demo and you can expect things like what you've experienced in this demo to pop up in your role of being a cloud engineer when dealing with gke and so that's pretty much all i wanted to cover with this multi-part demo working with gke so before you go i wanted to take a few moments to delete all the resources you've created one by one so i'm going to go up to the top i'm going to close all my tabs i'm going to head on over to clusters and so i don't want to delete my cluster just yet but the first thing that i want to do is delete my container images so i'm going to head up to the top and open up cloud shell and i'm going to use the command gcloud container images delete gcr dot io forward slash your google cloud project variable forward slash along with your first image of box of bow ties colon 1.0.0 hit enter it's going to prompt you if you want to continue you want to hit y for yes and it has now deleted the image as well you want to delete your latest image which is 1.0.1 so i'm going to change the zero to one hit enter it's going to ask if you want to continue yes and so the container images have now been deleted and so now along with the images you want to delete the artifacts as well and those are stored in cloud storage so i'm going to close down cloud shell i'm going to head on up to the navigation menu and i'm going to head down to storage and you want to select your bucket that has your project name underscore cloud build select the source folder and click on delete and you're going to get a prompt asking you to delete the selected folder but in order to do this you need to type in the name of the folder so i'm going to type it in now you can click on confirm and so now the folder has been deleted along with the artifacts and so now that we've taken care of the images along with the artifacts we need to clean up our gke cluster so i'm going to head back on up to the navigation menu and i'm going to head on over to kubernetes engine and the first thing that i want to delete is the low balancer so i'm going to head on up to services and ingress and you can select box of bow tie service and go up to the top and click on delete you're going to get a confirmation and you want to click on delete and it's going to take a couple minutes you do quick refresh and the service has finally been deleted i now want to delete my workload so i'm going to go over to the left hand menu click on workloads select the workload box of bowties and go up to the top and click on delete and you want to delete all resources including the horizontal pod auto scaler so you can simply click on delete and it may take a few minutes to delete gonna go up to the top and hit refresh and my workload has been deleted and so now all that's left to delete is the gke cluster itself so i'm going to go back to clusters so you're going to select the cluster and go up to the top and click on delete and you're going to get a prompt asking you if you want to delete these storage pods and these are default storage pods that are installed with the cluster as well you can delete the cluster while the workload is still in play but i have this habit of being thorough so i wanted to delete the workload before deleting the cluster and so you want to go ahead and click on delete and so that's pretty much all i have for this demo and this section on google kubernetes engine and again congrats on the great job you can now mark this as complete and i'll see you in the next one [Music] welcome back and in this lesson i will be covering the features of cloud vpn an essential service for any engineer to know about when looking to connect another network to google cloud whether it be your on-premises network another cloud provider or even when connecting to vpcs this service is a must know for any engineer and for the exam so with that being said let's dive in now cloudvpn securely connects your peer network to your vpc network through an ipsec vpn connection when i talk about a peer network this is referring to an on-premises vpn device or vpn service a vpn gateway hosted by another cloud provider such as aws or azure or another google cloud vpn gateway and so this is an ipsec or encrypted tunnel from your peer network to your vpc network that traverses the public internet and so for those who don't know ipsec being short for internet security protocol and this is a set of protocols using algorithms allowing the transport of secure data over an ip network ipsec operates at the network layer so layer 3 of the osi model which allows it to be independent of any applications although it does come with some additional overhead so please be aware and so when creating your cloud vpn traffic traveling between the two networks is encrypted by one vpn gateway and then decrypted by the other vpn gateway now moving on to some details about cloud vpn this is a regional service and so please take that into consideration when connecting your on-premises location to google cloud for the least amount of latency it also means that if that region were to go down you would lose your connection until the region is back up and running now cloud vpn is also a site-to-site vpn only and therefore it does not support site-to-client so this means that if you have a laptop or a computer at home you cannot use this option with a vpn client to connect to google cloud cloudvpn can also be used in conjunction with private google access for your on-premises hosts so if you're using private google access within gcp you can simply connect to your data center with vpn and have access as if you were already in gcp so if you're looking to extend private google access to your on-premises data center cloud vpn would be the perfect choice and so when it comes to speeds each cloud vpn tunnel can support up to three gigabits per second total for ingress and egress as well routing options that are available are both static and dynamic but are only available as dynamic for aha vpn and lastly cloudvpn supports ik version 1 and ike version 2 using shared secret and for those of you who are unaware ike stands for internet key exchange and this helps establish a secure authenticated communication channel by using a key exchange algorithm to generate a shared secret key to encrypt communications so know that when you choose cloudvpn that your connection is both private and secure so now there are two types of vpn options that are available in google cloud one being the classic vpn and the other being h a vpn and i'm going to take a moment to go through the differences now with classic vpn this provides a service level agreement of 99.9 percent also known as an sla of three nines while h a vpn provides a four nines sla when configured with two interfaces and two external ips now when it comes to routing classic vpn supports both static and dynamic routing whereas havpn supports dynamic routing only and this must be done through bgp using cloud router classic vpn gateways have a single interface and a single external ip address and support tunnels using static routing as well as dynamic routing and the static routing can be either route based or policy based whereas with havpn it can be configured for two interfaces and two external ips for true ha capabilities and as mentioned earlier when it comes to routing for havpn dynamic routing is the only available option now the one thing about classic vpn is that google cloud is deprecating certain functionality on october 31st of 2021 and is recommending all their customers to move to h a vpn and so know that this has not been reflected in the exam and not sure if and when it will be but know that when you are creating a cloud vpn connection in your current environment h a vpn is the recommended option and so now i wanted to dive into some architecture of how cloud vpn is set up for these two options starting with classic vpn now as i said before classic vpn is a cloud vpn solution that lets you connect your peer network to your vpc network through an ipsec vpn connection in a single region now unlike h a vpn classic vpn offers no redundancy out of the box you would have to create another vpn connection and if the connection were to go down you would have to manually switch over the connection from one to the other now as you can see here when you create a vpn gateway google cloud automatically chooses only one external ip address for its interface and the diagram shown here shows that of a classic vpn network connected from the bowtie dash network vpc in bowtie project to an on-premises network configured using a static route to connect now moving on to h-a-v-p-n again this is a highly available cloud vpn solution that lets you connect your peer network to your vpc network using an ipsec vpn connection in a single region exactly like classic vpn where havpn differs is that it provides four nines sla and as you can see here it supports double the connections so when you create an h a vpn gateway google cloud automatically chooses two external ip addresses one for each of its fixed number of two interfaces each ip address is automatically chosen from a unique address pool to support high availability each of these ha vpn gateway interfaces supports multiple tunnels which allows you to create multiple h a vpn gateways and you can configure an h a vpn gateway with only one active interface and one public ip address however this configuration does not provide a four nines sla now for h a vpn gateway you configure an external peer vpn gateway resource that represents your physical peer gateway in google cloud you can also create this resource as a standalone resource and use it later in this diagram the two interfaces of an h a vpn gateway in the bowtie network vpc living in bowtie project are connected to two peer vpn gateways in an on-premises network and this connection is using dynamic routing with bgp connecting to a cloud router in google cloud now when it comes to the times when using cloudvpn makes sense one of the first things you should think about is whether or not you need public internet access so when you're sharing files or your company needs a specific sas product that's only available on the internet vpn would be your only option as well when you're looking to use interconnect and your peering location is not available so you're not able to connect your data center to the colocation facility of your choice vpn would be the only other option that you have as well if budget constraints come into play when deciding on connecting to your peer network vpn would always be the way to go as cloud interconnect is going to be the more expensive option and lastly if you don't need a high speed network and low latency is not really a concern for you and you only have regular outgoing traffic coming from google cloud then vpn would suffice for your everyday needs and so the options shown here are also the deciding factors to look for when it comes to questions in the exam that refer to cloudvpn or connecting networks and so that's pretty much all i have for this short lesson on cloudvpn so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this lesson i'm going to go over another connection type that allows for on-premises connectivity to your google cloud vpcs which is cloud interconnect other than vpn this is the other connection type that allows connectivity from your on-premises environment to your google cloud vpc cloud interconnect is the most common connection for most larger organizations and are for those that demand fast low latency connections this lesson will cover the features of cloud interconnect and the different types that are available so with that being said let's dive in so getting right into it cloud interconnect is a low latency highly available connection between your on-premises data center and google cloud vpc networks also cloud interconnect connections provide internal ip address connection which means internal ip addresses are directly accessible from both networks and so on premises hosts can use internal ip addresses and take advantage of private google access rather than external ip addresses to reach google apis and services traffic between your on-premises network and your vpc network doesn't traverse the public internet traffic traverses a dedicated connection or through a service provider with a dedicated connection your vpc network's internal ip addresses are directly accessible from your on-premises network now unlike vpn this connection is not encrypted if you need to encrypt your traffic at the ip layer you can create one or more self-managed vpn gateways in your vpc network and assign a private ip address to each gateway now although this may be a very fast connection it also comes with a very high price tag now unlike vpn this connection type is not encrypted if you need to encrypt your traffic at the ip layer you can create one or more self-managed vpn gateways in your vpc network and assign a private ip address to each gateway now although this may be a very fast connection it also comes with a very high price tag and is the highest price connection type cloud interconnect offers two options for extending your on-premises network dedicated interconnect which provides a direct physical connection between your on-premises network and google's network as well as partner interconnect which provides connectivity between your on-premises and vpc networks through a supported service provider and so i wanted to take a moment to highlight the different options for cloud interconnect starting with dedicated interconnect now dedicated interconnect provides a direct physical connection between your on-premises network and google's network dedicated interconnect enables you to transfer large amounts of data between your network and google cloud which can be more cost effective than purchasing additional bandwidth over the public internet for dedicated interconnect you provision a dedicated interconnect connection between the google network and your own router in a common location the following example shown here shows a single dedicated interconnect connection between a vpc network and an on-premises network for this basic setup a dedicated interconnect connection is provisioned between the google network and the on-premises router in a common co-location facility when you create a vlan attachment you associate it with a cloud router this cloud router creates a bgp session for the vlan attachment and its corresponding on-premises peer router these routes are added as custom dynamic routes in your vpc network and so for dedicated interconnect connection capacity is delivered over one or more 10 gigabits per second or 100 gigabits per second ethernet connections with the follow-on maximum capacity supported per interconnect connection so with your 10 gigabit per second connections you can get up to eight connections totaling a speed of 80 gigabits per second with the 100 gigabit per second connection you can connect two of them together to have a total speed of 200 gigabits per second and so for dedicated interconnect your network must physically meet google's network in a supported co-location facility also known as an interconnect connection location this facility is where a vendor the co-location facility provider provisions a circuit between your network and a google edge point of presence also known as a pop the setup shown here is suitable for non-critical applications that can tolerate some downtime but for sensitive production applications at least two interconnect connections in two different edge availability domains are recommended now partner interconnect provides connectivity between your on-premises network and your vpc network through a supported service provider so this is not a direct connection from your on-premises network to google as the service provider provides a conduit between your on-premises network and google's pop now a partner interconnect connection is useful if a dedicated interconnect co-location facility is physically out of reach or your workloads don't warrant an entire 10 gigabit per second connection for partner interconnect 50 megabits per second to 50 gigabits per second vlan attachments are available with the maximum supported attachment size of 50 gigabits per second now service providers have existing physical connections to google's network that they make available for their customer to use so in this example shown here you would provision a partner interconnect connection with a service provider and connecting your on-premises network to that service provider after connectivity is established with the service provider a partner interconnect connection is requested from the service provider and the service provider configures your vln attachment for use once your connection is provisioned you can start passing traffic between your networks by using the service providers network now there are many more detailed steps involved to get a connection established along with traffic flowing but i just wanted to give you a high level summary of how a connection would be established with a service provider now as well to build a highly available topology you can use multiple service providers as well you must build redundant connections for each service provider in each metropolitan and so now there's a couple more connection types that run through service providers that are not on the exam but i wanted you to be aware of them if ever the situation arises in your role as a cloud engineer so the first one is direct peering and direct peering enables you to establish a direct peering connection between your business network and google's edge network and exchange high throughput cloud traffic this capability is available at any of more than 100 locations in 33 countries around the world when established direct peering provides a direct path from your on-premises network to google services including google cloud products that can be exposed through one or more public ip addresses traffic from google's network to your on-premises network also takes that direct path including traffic from vpc networks in your projects now you can also save money and receive direct egress pricing for your projects after they have established direct peering with google direct peering exists outside of google cloud unless you need to access google workspace applications the recommended methods of access to google cloud are dedicated interconnect or partner interconnect establishing a direct peering connection with google is free and there are no costs per port and no per hour charges you just have to meet google's technical peering requirements and can then be considered for the direct peering service and moving on to the last connection type is cdn interconnect now i know we haven't gotten into cdns in the course as the exam does not require you to know it but cdn standing for content delivery network is what caches content at the network edge to deliver files faster to those requesting it one of the main ways to improve website performance now moving on to cdn interconnect this connection type enables select third-party cdn providers like akamai and cloudflare along with others to establish and optimize your cdn population costs by using direct peering links with google's edge network and enables you to direct your traffic from your vpc networks to the provider's network and so your egress traffic from google cloud through one of these links benefits from the direct connectivity to the cdn provider and is billed automatically with reduced pricing typical use cases for cdn interconnect is if you're populating your cdn with large data files from google cloud or you have frequent content updates stored in different cdn locations and so getting into the use cases of when to use cloud interconnect a big purpose for it would be to prevent traffic from traversing the public internet it is a dedicated physical connection right to google's data centers so when you need an extension of your vpc network to your on-premises network interconnect is definitely the way to go now in speed and low latencies of extreme importance interconnect is always the best option and will support up to 200 gigabits per second as well when you have heavy outgoing traffic or egress traffic leaving google cloud cloud interconnect fits the bill perfectly and lastly when it comes to private google access this travels over the backbone of google's network and so when you are connected with interconnect this is an extension of that backbone and therefore your on-premises hosts will be able to take advantage of private google access and so i hope this has given you some clarity on the differences between the different connection types and how to extend your google cloud network to a peer or on-premises network so that's pretty much all i had to cover when it comes to cloud interconnect so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back in this lesson i'm going to be covering an overview of app engine now this is not a deep dive lesson for app engine as there is so much to cover with this service but i will be listing a lot of the features of app engine to give you a good feel for what it can do and what you will need to know for the exam so with that being said let's dive in now app engine is a fully managed serverless platform for developing and hosting web applications at scale this is google's platform as a service offering that was designed for developers so that they can develop their application and let app engine do all the heavy lifting by taking care of provisioning the servers and scaling the instances needed based on demand app engine gives you the flexibility of launching your code as is or you can launch it as a container and uses runtime environments of a variety of different programming languages like python java node.js go ruby php or net applications deployed on app engine that experience regular traffic fluctuations or newly deployed applications where you're simply unsure about the load are auto scaled accordingly and automatically your apps scale up to the number of instances that are running to provide consistent performance or scale down to minimize idle instances and reduces costs app engine also has the capabilities of being able to deal with rapid scaling for sudden extreme spikes of traffic having multiple versions of your application within each service allows you to quickly switch between different versions of that application for rollbacks testing or other temporary events you can route traffic to one or more specific versions of your application by migrating or splitting traffic and you can use traffic splitting to specify a percentage distribution of traffic across two or more of the versions within a service and allows you to do a b testing or blue green deployment between your versions when rolling out new features app engine supports connecting to back-end storage services such as cloud firestore cloud sql and cloud storage along with connecting to on-premises databases and even external databases that are hosted on other public clouds app engine is available in two separate flavors standard and flexible environments and each environment offers their own set of features that i will get into in just a sec now as i mentioned before app engine is available in standard and flexible environments and depending on your application needs either one will support what you need for your workload or you could even use both simultaneously the features shown here will give you a feel for both types of environments and i'm going to be doing a quick run through summarizing the features of each starting with the standard environment now with the standard environment applications run in a secure sandboxed environment allowing app engine standard to distribute requests across multiple servers and scaling servers to meet traffic demands your application runs with its own secure reliable environment that is independent of the hardware operating system or physical location of the server the source code is written in specific versions of the supported programming languages and with app engine standard it is intended to run for free or at a very low cost where you pay only for what you need and when you need it with app engine standard your application can scale to zero instances when there is no traffic app engine standard is designed for sudden and extreme spikes of traffic which require immediate scaling and pricing for standard app engine is based on instance hours and so when it comes to features for app engine flexible the application instances run within docker containers that includes a custom runtime or source code written in other programming languages these docker containers are then run on compute engine vms app engine flexible will run any source code that is written in a version of any of the supported programming languages for app engine flexible and unlike the standard environment unfortunately there is no free quota for app engine flexible as well app engine flexible is designed for consistent traffic or for applications that experience regular traffic fluctuations and pricing is based on the vm resources and not on instance hours like app engine standard and so where app engine flexible really shines over app engine standard are how the vms are managed so instances are health checked healed as necessary and co-located with other services within the project the vm's operating system is updated and applied automatically as well vms are restarted on a weekly basis to make sure any necessary operating system and security updates are applied ssh along with root access are available to the vm instances running your containers now deploying applications to app engine is as simple as using the gcloud app deploy command this command automatically builds a container image from your configuration file by using the cloud build service and then deploys that image to app engine now an app engine application is made up of a single application resource that consists of one or more services each service can be configured to use different runtimes and to operate with different performance settings services and app engine are used to factor your large applications into logical components that can securely share app engine features and communicate with one another these app engine services become loosely coupled behaving like microservices now within each service you deploy versions of that service and each version then runs within one or more instances depending on how much traffic you configured it to handle having multiple versions of your application within each service allows you to quickly switch between different versions of that application for rollbacks testing or other temporary events you can route traffic to one or more specific versions of your application by migrating traffic to one specific version or splitting your traffic between two separate versions and so the versions within your services run on one or more instances by default app engine scales your application to match the load your applications will scale up the number of instances that are running to provide consistent performance or scale down to minimize idle instances and reduce costs now when it comes to managing instances app engine can automatically create and shut down instances as traffic fluctuates or you can specify a number of instances to run regardless of the amount of traffic you can also configure how and when new instances are created by specifying a scaling type for your application and how you do this is you specify the scaling type in your application's app.yaml file now there are three different types of scaling choices to choose from and the first one being automatic scaling and this scaling type creates instances based on request rate response latencies and other application metrics you can specify thresholds for each of these metrics as well as a minimum number instances to keep running at all times if you use automatic scaling each instance in your application has its own queue for incoming requests before the queues become long enough to have a visible effect on your app's latency app engine automatically creates one or more new instances to handle the load the second type is basic scaling and this creates instances when your application receives requests each instance is shut down when the application becomes idle basic scaling is fantastic for intermittent workloads or if you're looking to drive your application by user activity app engine will try to keep your costs low even though it might result in higher latency as the volume of incoming requests increase and so the last scaling type is manual scaling and this is where you specify the number of instances that continuously run regardless of the load so these are instances that are constantly running and this allows complex startup tasks on the instances to have already been completed when receiving requests and applications that rely on the state of the memory over time so this is ideal for instances whose configuration scripts require some time to fully run their course so now that i've gone over managing the instances i wanted to take a few moments to go over how app engine manages traffic starting with traffic migration now traffic migration switches the request routing between the versions within a service of your application moving traffic from one or more versions to a single new version so when deploying a new version with the same name of an existing version it causes an immediate traffic migration all instances of the old version are immediately shut down in app engine standard you can choose to route requests to the target version either immediately or gradually you can also choose to enable warm-up requests if you want the traffic gradually migrated to a version gradual traffic migration is not supported in app engine flexible and traffic is migrated immediately now one thing to note is that when you immediately migrate traffic to a new version without any running instances then your application will have a spike in latency for loading requests while instances are being created and so another way to manage traffic on app engine is through traffic splitting now you can use traffic splitting to specify a percentage distribution of traffic across two or more of the versions within a service so in this example if i'm deploying a new version of my service i can decide on how i want to distribute traffic to each version of my application and so i decide that i want to keep my current version in play but roll out the new version of my application to 10 of my users leaving the old version was still 90 of the traffic going to that version and so splitting traffic allows you to conduct a b testing between your versions and provides control over the pace when rolling out features and just as a note when you've specified two or more versions for splitting you must choose whether to split traffic by either by either ip address http cookie or do it randomly now again this has not been a deep dive lesson on app engine but i hope this has given you an overview of the features that are available as the exam touches on these features i also wanted to give you some familiarity with the service itself as coming up next i will be going into a demo where we will be launching an application using app engine and trying on some of these features for yourself and so that's pretty much all i wanted to cover when it comes to app engine so you can now mark this lesson as complete and whenever you're ready join me in the console where you will deploy an application on app engine and try out some of these features for yourself [Music] welcome back and in this demo you're going to build another application to deploy on app engine called serverless bowties this demo will run you through the ins and outs of deploying a website application on app engine along with managing it while experiencing no downtime so there's quite a bit of work to do here so with that being said let's dive in and so here in my console i am logged in as tonybowtieace gmail.com under project bowtie inc and so the first thing i want to do here is i want to head on over to app engine so in order to do that i'm going to go to the top left-hand navigation menu and i'm going to go down to app engine and because i haven't created any applications i'm going to be brought to this splash page now in order to deploy this application we're not going to be doing it through the console but we will be doing it through the command line and so to get started with that i'm going to go up to the top and open up cloud shell i'm going to make this bigger for better viewing and so in order for me to get the code to launch this application i'm going to be cloning my github repository into cloud shell and so for those of you who haven't deleted your repository from the last demo you can go ahead and skip the cloning step for those of you who need to clone your repository you will find a link to the instructions in the lesson text and there you'll be able to retrieve the command which will be git clone along with the address of the repo i'm going to hit enter and because i've already cloned this repo i'm receiving this error i'm going to do an ls and as you can see here the google cloud associate cloud engineer repo has already been cloned so i'm going to cd into that directory and in order to get the code i'm going to simply run the command git pull to get the latest and i'm going to simply clear my screen and so now that i've retrieved all the code that i need in order to deploy it i need to go to that directory and that directory is going to be 11 serverless services forward slash 0 1 serverless bowties and hit enter you're going to run ls and here you will find two versions of the website application site v1 and site v2 along with the instructions if you want to follow straight from here and so i want to go ahead and deploy my first website application so i'm going to cd into site v1 ls and here you will see the app.yaml which is the configuration file that you will need in order to run the application on app engine and so before i go ahead and deploy this i wanted to take a moment to show you the application configuration so i'm going to go ahead and open it up in cloud shell editor so i'm going to type in edit app.yaml enter and as you can see here my runtime is python 3.7 and as you can see i have a default expiration of two seconds along with an expiration underneath each handler and this is due to the caching issue that happens with app engine and so in order to simulate traffic splitting between the two website applications in order to make things easy i needed to expire the cash and this is an easy way to do it now there may be applications out there that do need that caching and so the expiration may be a lot higher but for the purposes of this demo two seconds expiration should suffice as well explain the two handlers here the first one showing the files that will be uploaded to the cloud storage bucket as well as the second stating what static files will be presented and so i'm going to go ahead back over to my terminal and i'm going to go ahead and clear my screen and i'm going to go ahead and run the command gcloud app deploy with the flag dash dash version and this is going to be version one so i'm going to go ahead and hit enter and you may get a pop-up asking you to authorize this api call using your credentials and you want to click on authorize and you're going to be prompted to enter in a region that you want to deploy your website application to we want to keep this in us east one so i'm going to type in 15 hit enter and you're going to be prompted to verify your configuration for your application before it's deployed you're also going to be prompted if you want to continue definitely yes so i'm going to hit y enter and so now as you've seen the files have been uploaded to cloud storage and app engine is going to take a few minutes to create the service along with the version so i'm going to let it do the needful and i'll be back before you know it okay and my application has been deployed now although you don't see it here in the console it has been deployed all i need to do is refresh my screen but i wanted to just point out a couple things that is shown here in the terminal the first one being the default service now the first time you deploy a version of your application it will always deploy to the default service initially and only then will you be able to deploy another named service to app engine now here where it says setting traffic split for service this is referring to the configuration for traffic splitting being applied in the background which i will be getting into a little bit later and lastly the url shown for the deployed service will always start with the name of your project followed by.ue.r.appspot.com which is why in production google recommends to run app engine in a completely separate project before this demo running it in the same project that we've been using will suffice okay so let's go ahead and take a look at the application so i'm going to go back up to the top here to the navigation menu and i'm gonna go down to app engine and go over to services and so here you will see the default service with version one and if i go over to versions i will see here my version the status the traffic allocation along with any instances that it needs the run time the specific environment and i'll have some diagnostic tools here that i could use and so because this is a static website application we won't be using any instances and so this will always show a zero so now i want to head back on over to services and i'm going to launch my application by simply clicking on this hot link and success serverless bow ties for all and so it looks like my application has been successfully deployed so i'm going to close down this tab now there's a couple of things that i wanted to run through here on the left hand menu just for your information so here i can click on instances and if i was running any instances i am able to see a summary of those instances and i can click on the drop down here and choose a different metric and find out any information that i need as well i can click on this drop down and select a version if i had multiple versions which i do not clicking on task queues here is where i can manage my task queues but this is a legacy service that will soon be deprecated clicking on cron jobs here i can schedule any tasks that i need to run at a specific time on a recurring basis i can edit or add any firewall rules if i need to and as you can see the default firewall rule is open to the world now you probably noticed memcache as being one of the options here in the menu but this is a legacy service that will soon be deprecated memcache is a distributed in-memory data store that is bundled into the python to runtime acting as a cache for specific tasks and google recommends moving to memory store for redis if you're planning on applying caching for your app engine application and so i'm not sure how much longer this will be here and lastly under settings here is where you can change your settings for your application i can add any custom domains any ssl certificates as well as setting up email for any applications that want to send email out to your users okay and now that we've done that walkthrough i want to go ahead and deploy my second version of the application and so i'm going to go ahead back down to cloud shell i'm going to quickly clear my screen and i want to move into the site v2 directory so i'm going to hit cd dot dot which will bring you back one directory you do an ls and i'm going to change directories into site v2 and do an ls just to verify and yes you will see serverless bow ties too i'm going to quickly clear my screen and i'm going to run the same command as before which is gcloud app deploy with the version flag dash dash version and instead of one i'm going to launch version 2. so i'm going to hit enter i'm going to be prompted if i want to continue yes i do and as you can see the files have been uploaded to cloud storage for version 2 of the website application and app engine is going to take a few minutes to create the service along with the version so i'm going to let it cook here for a couple minutes and i'll be back before you can say cat in the hat okay so version 2 has been deployed and so if i go up here to the console and i click on refresh you should see version 2 of your service and as you can see 100 of the traffic has been allocated to version 2 automatically and this is the default behavior for whenever you launch a new version of your service the only way to avoid this is to deploy your new version with the no promote flag and so if i go back to services here on the left and i click on the default service you should see success for version two and so i know that my website application for version 2 has been deployed successfully so i'm going to close down this tab again and i'm going to go back to versions and so what i want to do now is i want to simulate an a b test or blue green deployment by migrating my traffic back to the old version in this case being version one so in production let's say that you would release a new version and the version doesn't go according to plan you can always go back to the previous version and app engine allows you to do that very easily and so i'm going to click on version 1 and i'm going to go up to the top menu and click on migrate traffic you'll be prompted if you want to migrate traffic yes i do so i'm going to click on migrate and it should take a minute here and traffic should migrate over to version one and success traffic has been migrated and so we want to verify that this has happened i'm gonna go back to services i'm gonna click on the default service and yes the traffic has been allocated to version one okay so i'm going to shut down this tab i'm going to go back to versions and so now what i want to do is i want to simulate splitting the traffic between the two versions and so in order for you to do this you can go up to the top menu click on split traffic and you'll be prompted with a new menu here and here i can choose from different versions and because i only have two versions i'm going to add version 2 and in order to allocate the traffic between the two i can either use this slider and as you can see the allocation percentage will change or i can simply just type it in and so i'm going to leave this at 50 percent so fifty percent of version one fifty percent of version two i'm going to split traffic randomly i'm gonna move this down just a little bit and so that's exactly how you wanna allocate your traffic and so once you've completed that you can simply click on save it's going to take a moment to update the settings and it's been successful so if i head back on over to the previous page you can see here that traffic has been allocated to both versions and so now in order to verify this what you're going to do is go over to services and click on the default hot link and you'll see version one but if i continuously refresh my screen i can see that here i have version two so because it's random i have a 50 chance of getting version 1 and a 50 chance of getting version 2. and so this is a simulation of splitting traffic to different versions and usually with a b testing only a small percentage of the traffic is routed to the new version until verification can be made that the new version deployed has indeed been successful and this can be done by receiving feedback from the users and so now i wanted to take a quick moment to congratulate you on making it through this demo and hope that it has been extremely useful in excelling your knowledge in deploying and managing applications on app engine so just as a recap you've cloned the repo to cloud shell you then deployed version one of your application into app engine you verified its launch and then you deployed version two of the application and verified its launch as well you then migrated traffic from version two over to version one and then you went ahead and split traffic between both versions and allotted 50 of the traffic allocation to each version and so now before you go i want to make sure that we clean up any resources that we've deployed so that we don't incur any unnecessary costs and so the way to do this is very simple so first step you want to go over to the left hand menu and click on settings and simply click on disable application you're going to be prompted to type in the app's id for me it's bowtie inc so i'm going to type that in and i'm going to click on disable now unfortunately with app engine you can't actually delete the application it can only be disabled and so now here i'm going to hit the hot link to go over to the cloud storage bucket and as you can see here i have no files but i'm going to move back to my buckets and i'm going to move into the staging bucket which is appended with your project id.appspot.com and as you can see here there's a whole bunch of different files as well if i drill down into the directory marked as ae for app engine i can see here that i have some more directories along with the manifest and so now if you want to keep your application in order to run it later you don't need to delete this bucket but because i don't need it i'm going to go ahead and delete the bucket hit delete paste in my bucket name hit delete as well under us.artifacts you will find a directory called containers and as explained in the last lesson code build builds a container for your application before deploying it to app engine so i'm going to drill down into images so here's all the container digests and i don't need any of these so i'm gonna go ahead and delete this bucket as well and so this is the last step in order to delete all the directories and files that we use to deploy our application in an app engine okay and so i'm gonna head back on over to app engine and so now that cleanup has been taken care of that's pretty much all i wanted to cover in this demo for deploying and managing applications on app engine so you can now mark this as complete and i'll see you in the next one and again congrats on a job well done [Music] welcome back in this lesson i will be diving into another serverless product from google cloud by the name of cloud functions an extremely useful and advanced service that can be used with almost every service on the platform now there's quite a bit to cover here so with that being said let's dive in now cloud functions as i said before are a serverless execution environment and what i mean by this is like app engine there is no need to provision any servers or updating vms as the infrastructure is all handled by google but unlike app engine you will never see the servers so the provisioning of resources happens when the code is executed now cloud functions are a function as a service offering and this is where you upload code that is purposefully written in a supported programming language and when your code is triggered it is executed in a fully managed environment and your billed for when that code is executed cloud functions run in a runtime environment and support many different runtimes like python java node.js go and net core cloud functions are event driven so when something happens in your environment you can choose whether or not you'd like to respond to this event if you do then your code can be executed in response to the event these triggers can be one of a few different types such as http pub sub cloud storage and now firestore and firebase which are in beta and have yet to be seen in the exam cloud functions are priced according to how long your function runs and how many resources you provision for your function if your function makes an outbound network request there are also additional data transfer fees cloud functions also include a perpetual free tier which allows you 2 million invocations or executions of your function now cloud functions themselves are very simple but have a few steps to execute before actually running so i wanted to give you a walkthrough on exactly how cloud functions work now after selecting the name and region you want your function to live in you would then select the trigger you wish to use and you can choose from the many i listed earlier being http cloud storage pub sub cloud firestore and firebase a trigger is a declaration that you are interested in a certain event or set of events binding a function to a trigger allows you to capture and act on these events authentication configuration is the next step and can be selected with public access or configured through iam now there are some optional settings that can be configured where you would provide the amount of memory the function will need to run networking preferences and even selection for a service account now once all the settings have been solidified your written code can then be put into the function now the functions code supports a variety of languages as stated before like python java node.js or go now when writing your code there are two distinct types of cloud functions that you could use http functions and background functions with http functions you invoke them from standard http requests these http requests wait for the response and support handling of common http request methods like get put post delete and options when you use cloud functions a tls certificate is automatically provisioned for you so all http functions can be invoked via a secure connection now when it comes to background functions these are used to handle events from your gcp infrastructure such as messages on a pub sub topic or changes in a cloud storage bucket now once you have put all this together you are ready to deploy your code now there are two things that will happen when deploying your code the first one is the binding of your trigger to your function once you bind a trigger you cannot bind another one to the same function only one trigger can be bound to a function at a time now the second thing that will happen when you deploy your function's source code to cloud functions is that source code is stored in a cloud storage bucket as a zip file cloud build then automatically builds your code into a container image that pushes that image to container registry cloud functions accesses this image when it needs to run the container to execute your function the process of building the image is entirely automatic and requires no manual intervention and so at this point of the process the building of your function is now complete now that the function has been created we now wait for an event to happen and events are things that happen within your cloud environment that you might want to take action on these might be changes to data in cloud sql files added to cloud storage or a new vm being created currently cloud functions supports events from the same services used for triggers that i have just mentioned including other google services like bigquery cloud sql and cloud spanner now when an event triggers the execution of your cloud function data associated with the event is passed via the functions parameters the type of event determines the parameters that are passed to your function cloud functions handles incoming requests by assigning them to instances of your function now depending on the volume of requests as well as the number of existing function instances cloud functions may assign a request to an existing instance or create a new one so the cloud function will grab the image from cloud registry and hand off the image along with the event data to the instance for processing now each instance of a function handles only one concurrent request at a time this means that while your code is processing one request there is no possibility of a second request being routed to the same instance thus the original request can use the full amount of resources that you requested and this is the memory that you assign to your cloud function when deploying it now to allow google to automatically manage and scale the functions they must be stateless functions are not meant to be persistent nor is the data that is passed on to the function and so once the function has run and all data has been processed by the server it is then passed on to either a vpc or to the internet now by default functions have public internet access unless configured otherwise functions can also be private and used within your vpc but must be configured before deployment now there are so many use cases for cloud functions and there are many that have already been created by google for you to try out and can be located in the documentation that i've supplied in the lesson text below now the exam doesn't go into too much depth on cloud functions but i did want to give you some exposure to this fantastic serverless product from google as it is so commonly used in many production environments in a simple and easy way to take in data process it and return a result from any event you are given and i have no doubt that once you get the hang of deploying them that you will be a huge fan of them as well and so that's pretty much all i had to cover when it comes to cloud functions so you can now mark this lesson as complete and whenever you're ready join me in the next one where we go hands-on in the console creating and deploying your very first function welcome back and in this demo we will be diving into creating and deploying our very first cloud function we're going to take a tour of all the options in the console but we're going to do most of the work in cloud shell to get a good feel for doing it in the command line so with that being said let's dive in and so i'm logged in here as tony bowties gmail.com and i'm in the project of bowtie inc and so the first thing i want to do is head on over to cloud functions in the console so i'm going to go up to the top left to the navigation menu and i'm going to scroll down to cloud functions and as you can see here cloud functions is getting ready and this is because we've never used it before and the api is being enabled okay and the api has been enabled and we can go ahead and start creating our function so you can go ahead and click create function and you will be prompted with some fields to fill out for the configuration of your cloud function and so under basics for function name i'm going to name this hello underscore world for region i'm going to select us east one and under trigger for trigger type we're gonna keep this as http although if i click on the drop down menu you can see that i will have options for cloud pub sub cloud storage and the ones that i mentioned before that are in beta so we're going to keep things as http and here under url is the url for the actual cloud function under authentication i have the option of choosing require authentication or allow unauthenticated invocations and as you can see this is clearly marked saying that check this if you are creating a public api or website which we are and so this is the authentication method that you want to select and so now that we have all the fields filled out for the basic configuration i'm going to go ahead and click on save and just to give you a quick run through of what else is available i'm going to click on the drop down here and this will give me access to variables networking and advanced settings the first field here memory allocated i can actually add more memory depending what i am doing with my cloud function but i'm going to keep it as the default if you have a cloud function that runs a little bit longer and you need more time to run the cloud function you can add additional time for the timeout and as well i have the option of choosing a different service account for this cloud function and so moving on under environment variables you will see the options to add build environment variables along with runtime environment variables and the last option being connections here you can change the different networking settings for ingress and egress traffic under ingress settings i can allow all traffic which is the default i can allow internal traffic only as well i can allow internal traffic and traffic from cloud low balancing now as well when it comes to the egress settings as i said before by default your cloud function is able to send requests to the internet but not to resources in your vpc network and so this is where you would create a vpc connector to send requests from your cloud function to resources in your vpc so if i click on create a connector it'll open up a new tab and bring me to vpc network to add serverless vpc access and so i don't want to do that right now so i'm going to close down this tab and i'm going to go ahead and leave everything else as is and click on next and so now that the configuration is done i can dive right into the code and so google cloud gives you a inline editor right here along with the different runtime environments so if i click on the drop down menu you can see i have the options of net core go java node.js and python 3.7 and 3.8 and so for this demo i'm going to keep it as node.js 10. the entry point will be hello world and i'm going to keep the code exactly as is and this is a default cloud function that is packaged with any runtime whenever you create a function from the console and so if i had any different code i can change it here but i'm not going to do that i'm going to leave everything else as is and click on deploy and it'll take a couple minutes here to create my cloud function and so i'm going to pause the video here for just a quick sec and i'll be back in a flash okay and my cloud function has been deployed and i got a green check mark which means that i'm all good and so i want to dive right into it for just a second so i can get some more details here i have the metrics for my cloud function the invocations per second execution time memory utilization and active instances i have my versions up here at the top but since i only have one version only one version shows up if i click on details it'll show me the general information along with the networking settings the source will show me the code for this cloud function as well as the variables the trigger permissions logs and testing and here i can write in some code and test the function and so in order for me to invoke this function i can simply go to trigger and it'll show me the url but a quick way to do this through the command line is to simply open up cloud shell and make this a little bigger for better viewing and i'm going to paste in the command gcloud functions describe along with the function name which is hello underscore world along with the region flag dash dash region with the region that my cloud function has been deployed in which is us east one and i'm going to hit enter it's going to ask me to authorize my api call yes i want to authorize it and this command should output some information on your screen and so what we're looking for here is the http trigger which you will find here under https trigger and it is the same as what you see here in the console and so just know if you want to grab the http url trigger you can also do it from the command line and so i'm going to now trigger it by going to this url and you should see in the top left hand side of your screen hello world not as exciting as spinning bow ties but this example gives you an idea of what an http function can do and so i'm going to close down this tab and so now what i want to do is i want to deploy another function but i want to do it now through the command line and so i'm going to now quickly clear my screen and so since i've already uploaded the code to the repo i'm going to simply clone that repo and run it from here so i'm going to simply do a cd tilde to make sure i'm in my home directory for those of you who haven't deleted the directory you can simply cd into it so i'm going to run cd google cloud associate cloud engineer hit enter and i'm going to run a get pull command and it pull down all the files that i needed i'm going to quickly clear my screen and so i'm going to change directories into the directory that has my code and so you're going to find it under 11 serverless services under zero to you called hit enter and again i will have a link in the lesson text for the full instructions on this demo and it will list the directory where you can find this code okay so moving forward i'm going to run ls and you should see three files here main.py requirements.txt and the text file with the instructions and so now that i have everything in place in order to deploy my code i'm going to paste in the command to actually deploy my function which is gcloud functions deploy the name of the function which is you underscore called the flag for the runtime dash dash runtime and the runtime is going to be python 3.8 the flag for the trigger which is going to be http and because i'm a nice guy and i want everyone to have access to this i'm going to tag it with the flag dash dash allow unauthenticated so i'm going to hit enter okay and this function should take a couple minutes to deploy so i'm going to sit here and let it cook and i'll be back before you can say cat in the hat okay and our function has been deployed i'm going to do a quick refresh here in the console and it deployed successfully as you can see the green check mark is here okay and so now that it's been deployed we want to trigger our function and so because i just deployed this function the url trigger is conveniently located here in my screen so you can go ahead and click on it and hello lover of bow ties you called now although this may be similar to the hello world demo but i did add a small feature that might spice things up and so if you go up to the url and you type in question mark name equals and your name and since my name is anthony i'm going to type in anthony hit enter and hello anthony you called and so this is a perfect example of the many different ways you can use functions and although i've only highlighted some very simple demonstrations there are many different ways that you can use functions such as running pipelines running batch jobs and even event driven security now although the exam doesn't go into too much depth on cloud functions it's always good to know its use cases and where its strengths lie for when you do decide to use it in your role as a cloud engineer now before you go be sure to delete all the resources you've created by deleting the functions and the storage buckets that house the code for the cloud functions and i will walk you through the steps right now okay so first i'm going to close down this tab and next you're going to select all the functions and you're going to simply click on delete you're going to get a prompt to delete the functions you're going to click on delete and it's going to take a minute or two and the functions are deleted i'm going to close down my cloud shell and i'm going to head over to cloud storage and as you can see here both these buckets that start with gcf standing for google cloud functions can be safely deleted as inside them are the files that were used for the cloud function so i'm going to go back out i'm going to select both of these and i'm going to click on delete you get a prompt to delete two buckets you can simply type in delete and click on delete and the buckets have now been deleted and you've pretty much finished your cleanup and so just as a recap you created a default cloud function that was available from the console and then verified it by triggering the http url you then deployed another function from the command line by pulling the code from the repo and using it for deployment and then you verified that function by triggering it using the http url as well and then you modify the url for a different output great job on another successful demo so you can now mark this as complete and let's move on to the next one [Music] welcome back in this lesson we're going to dive into cloud storage the go to storage service from google cloud if you're an engineer working in google cloud you've probably used this many times as a storage solution and if you haven't this is definitely a service that you will need to know for both the exam and your day-to-day role as a cloud engineer now there's quite a bit to cover here so with that being said let's dive in now cloud storage is a consistent scalable large capacity highly durable object storage and this is unlimited storage for objects with no minimum object size but please remember that this is object storage and is not designed to store an operating system on but to store whole objects like pictures or videos cloud storage has worldwide accessibility and worldwide storage locations so anywhere that there is a region or zone cloud storage is available from there and can be accessed at any time through an internet connection cloud storage is great for storing data from data analytics jobs text files with code pictures of the latest fashion from paris and videos of your favorite house dj at the shelter cloud storage excels for content delivery big data sets and backups and are all stored as objects in buckets and this is the heart of cloud storage that i will be diving into so starting with buckets these are the basic containers or construct that holds your data everything that you store in cloud storage must be contained in a bucket you can use buckets to organize your data and control access to your data but unlike directories and folders you cannot nest buckets and i'll get into that in just a minute now when you create a bucket you must specify a globally unique name as every bucket resides in a single cloud storage namespace as well as a name you must specify a geographic location where the bucket and its contents are stored and you have three available geography choices to choose from from region dual region and multi-region and so just as a note choosing dual region and multi-region is considered geo-redundant for dual region geo-redundancy is achieved using a specific pair of regions for multi-region geo-redundancy is achieved using a continent that contains two or more geographic places basically the more regions your data is available in the greater your availability for that data after you've chosen a geographic location a default storage class must be chosen and this applies to objects added to the bucket that don't have a storage class explicitly specified and i'll be diving into storage classes in just a bit and so after you create a bucket you can still change its default storage class to any class supported in the buckets location with some stipulations you can only change the bucket name and location by deleting and recreating the bucket as well once dual region is selected it cannot be changed to multi-region and when selecting multi-region you will not be able to change the bucket to be dual region and lastly you will need to choose what level of access you want others to have on your bucket whether you want to apply permissions using uniform or fine grained access uniform bucket level access allows you to use iam alone to manage permissions iam applies permissions to all the objects contained inside the bucket or groups of objects with common name prefixes the find green option enables you to use iam and access control lists or acls together to manage permissions acls are a legacy access control system for cloud storage designed for interoperability with amazon s3 for those of you who use aws you can specify access and apply permissions at both the bucket level and per individual object and i will also be diving more into depth with access control in just a bit and just as a note labels are an optional item for bucket creation like every other resource creation process in gcp now that we've covered buckets i wanted to cover what is stored in those buckets which is objects and objects are the individual pieces of data or data chunks that you store in a cloud storage bucket and there is no limit on the number of objects that you can create in a bucket so you can think of objects kind of like files objects have two components object data and object metadata object data is typically a file that you want to store in cloud storage and in this case it is the picture of the plaid bow tie and object metadata is a collection of name value pairs that describe the various properties of that object an object's name is treated as a piece of object metadata in cloud storage and must be unique within the bucket cloud storage uses a flat namespace to store objects which means that cloud storage isn't a file system hierarchy but sees all objects in a given bucket as independent with no relationship towards each other for convenience tools such as the console and gsutil work with objects that use the slash character as if they were stored in a virtual hierarchy for example you can name one object slash bow ties slash spring 2021 slash plaid bowtie.jpg when using the cloud console you can then navigate to these objects as if they were in a hierarchical directory structure under the folders bow ties and spring 2021 now i mentioned before that the part of the bucket creation is the selection of a storage class the storage class you set for an object affects the object's availability and pricing model so when you create a bucket you can specify a default storage class for the bucket when you add objects to the bucket they inherit this storage class unless explicitly set otherwise now i wanted to touch on these four storage classes now to give you a better understanding of the differences between them the first one is standard storage and is considered best for hot data or frequently accessed data and is best for short-term use as it does not have any specified storage duration and this is excellent for use in analytical workloads and transcoding and the price for this storage class comes in at two cents per gigabyte per month next up is near line storage and this is considered hot data as well and is a low-cost storage class for storing in frequently accessed data nearline storage has a slightly lower availability a 30-day minimum storage duration and comes with the cost for data access nearline storage is ideal if you're looking to continuously add files but only plan to access them once a month and is perfect for data backup and data archiving the price for this storage class comes in at a penny per gigabyte per month now cold line storage is considered cold data as it enters into more of the longer term storage classes and is a very low cost storage class for storing and frequently accessed data it comes with slightly lower availability than nearline storage a 90-day minimum storage duration and comes with the cost for data access that is higher than the retrieval cost for nearline storage coldline storage is ideal for data you plan to read or modify at most once a quarter and is perfect for data backup and data archiving the price for this storage class comes in at less than half of a penny per gigabyte per month and finally archive storage is the lowest cost highly durable storage service for data archiving online backup and disaster recovery and even coming in at a lowest cost the data access is still available within milliseconds archive storage comes in at a higher cost for data retrieval as well as a day minimum storage duration and is the best choice for data that you plan to access less than once a year archive storage also comes with the highest price for data retrieval and it is ideal for archive data storage that's used for regulatory purposes or disaster recovery data in the event that there is an oopsies in your environment the price of the storage class comes in at a ridiculously low price per gigabyte per month at a fraction of a penny per gigabyte per month now when it comes to choosing your geographic location this will determine the availability of your data here as you can see the highest availability is the standard multi-region whereas archive has the lowest availability when stored in a regional setting now when it comes to the durability of your data meaning the measurement of how healthy and resilient your data is from data loss or data corruption google cloud boasts 11 9's durability annually on all data stored in any storage class on cloud storage so know that your data is stored safely and will be there holding the same integrity from the day you stored it now when it comes to granting permissions to your cloud storage buckets and the objects within them there are four different options to choose from the first is iam permissions and these are the standard permissions that control all your other resources in google cloud and follow the same top-down hierarchy that we discussed earlier the next available option are access control list or acls and these define who has access to your buckets and objects as well as what type of access they have and these can work in tandem with im permissions moving on to sign urls these are time limited reader write access urls that can be created by you to give access to the object in question for the duration that you specify and lastly is sign policy documents and these are documents to specify what can be uploaded to a bucket and i will be going into each one of these in a bit of detail now cloud storage offers two systems for granting users permission to access your buckets and objects iam and access control lists these systems act in parallel in order for a user to access a cloud storage resource only one of the systems needs to grant the user permission im is always the recommended method when it comes to giving access to buckets and the objects within those buckets granting roles at the bucket level does not affect any existing roles that you granted at the project level and vice versa giving you two levels of granularity to customize your permissions so for instance you can give a user permission to read objects in any bucket but permissions to create objects only in one specific bucket the roles that are available through iam are the primitive standard storage roles or the legacy roles which are equivalent to acls now acls are there if you need to customize access and really get granular with individual objects within a bucket and are used to define who has access to your buckets and objects as well as what level of access they have each acl consists of one or more entries and gives a specific user or group the ability to perform specific actions each entry consists of two pieces of information a permission which defines what actions can be performed and a scope which defines who can perform the specified actions now acls should be used with caution as iam roles and acls overlap cloud storage will grant a broader permission so if you allow specific users access to an object in a bucket and then an acl is applied to that object to make it public then it will be publicly accessible so please be aware now a signed url is a url that provides limited permission and time to make a request sign urls contain authentication information allowing users without credentials to perform specific actions on a resource when you generate a signed url you specify a user or service account which must have sufficient permission to make the request that the sign url will make after you generate a signed url anyone who possesses it can use the sign url to perform specified actions such as reading an object within a specified period of time now if you want to provide public access to a user who doesn't have an account you can provide a signed url to that user which gives the user read write or delete access to that resource for a limited time you specify an expiration date when you create the sign url so anyone who knows the url can access the resource until the expiration time for the url is reached or the key used to sign the url is rotated and the command to create the sign url is shown here and as you can see has been assigned for a limited time of 10 minutes so as you've seen when it comes to cloud storage there are so many configuration options to choose from and lots of different ways to store and give access and this makes this resource from google cloud such a flexible option and full of great potential for many different types of workloads this is also a service that comes up a lot in the exam as one of the many different storage options to choose from and so knowing the features storage classes pricing and access options will definitely give you a leg up when you are presented with questions regarding storage and so that's pretty much all i wanted to cover when it comes to this overview on cloud storage so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this lesson i will be covering object versioning and life cycle management a feature within cloud storage that is used to manage and sort through older files that need to be deleted along with files that are not in high need of regular access knowing the capabilities of these two features can really help organize accumulated objects in storage buckets and cut down on costs so without further ado let's dive in now to understand a bit more about objects i wanted to dive into immutability and versioning now objects are immutable which means that an uploaded object cannot change throughout its storage lifetime an object's storage lifetime is the time between a successful object creation or upload and successful object deletion this means that you cannot edit objects in place instead objects are always replaced with a new version so after the upload of the new object completes the new version of the object is served to readers this replacement marks the end of one object's life cycle and the beginning of a new one now to support the retrieval of objects that are deleted or replaced cloud storage offers the object versioning feature object versioning retains a non-current object version when the live object version gets replaced or deleted enabling object versioning increases storage costs which can be partially mitigated by configuring object lifecycle management to delete older object versions but more on that in just a bit cloud storage uses two properties that together identify the version of an object the generation which identifies the version of the object's data and the meta generation which identifies the version of the object's metadata these properties are always present with every version of the object even if object versioning is not enabled these properties can be used to enforce ordering of updates so in order to enable object versioning you would do that by enabling it on a bucket once enabled older versions remain in your bucket when a replacement or deletion occurs so by default when you replace an object cloud storage deletes the old version and adds a new version these older versions retain the name of the object but are uniquely identified by their generation number when object versioning has created an older version of an object you can use the generation number to refer to the older version this allows you to restore a replaced object in your bucket or permanently delete older object versions that you no longer need and so touching back on cost for just a minute these versions can really add up and start costing you some serious money if you have thousands of files with hundreds of versions and this is where life cycle management comes into play now cloud storage offers the object lifecycle management feature in order to support some common use cases like setting a time to live or ttl for objects retaining non-current versions of objects or downgrading storage classes of objects to help manage costs now in order to apply this feature to your objects you would assign a lifecycle management configuration to a bucket the configuration contains a set of rules which apply to current and feature objects in the bucket when an object meets the criteria of one of the rules cloud storage automatically performs the specified action on the object and so some example use cases are shown here so if you're looking to downgrade the storage class of objects older than 365 days to cold line storage for compliance purposes along with saving money life cycle management is perfect for this another use case is when you want to delete objects created before january 1st of 2020 and this is another great use case to save money as well with keeping only the three most recent versions of each object in a bucket with versioning enabled to keep from version objects building up object lifecycle management has so many other use cases across a myriad of industries and when used correctly is a great way to achieve object management along with saving money now i wanted to take a moment to dive into the lifecycle management configuration each lifecycle management configuration contains a set of components these are a set of rules conditions and the action when the conditions are met rules are any set of conditions for any action conditions is something an object must meet before the action defined in the rule occurs on the object and there are various conditions to choose from that allows you to get pretty granular and finally the action which is where you would have the option to delete or set storage class now when you delete current versions this will move the current version into a non-current state and when you delete a non-current version you will permanently delete the version and cannot get it back and so when you set the storage class it will transition the object to a different storage class so when defining a rule you can specify any set of conditions for any action if you specify multiple conditions in a rule an object has to match all of the conditions for the action to be taken so if you have three conditions and one of those conditions have not been met then the action will not take place if you specify multiple rules that contain the same action the action is taken when an object matches the conditions in any of these rules now if multiple rules have their conditions satisfied simultaneously for a single object cloud storage will either perform the delete action as it takes precedence over the set storage class action or the set storage class action that switches the object to the storage class with the lowest at rest storage pricing takes precedence so for example if you have one rule that deletes an object and another rule that changes the object storage class but both rules use the exact same condition the delete action always occurs when the condition is met or if you have one rule that changes the object storage class to near line storage and another rule that changes the object storage class to cold line storage but both rules use the exact same condition the object storage class always changes to cold line storage when the condition is met and so some considerations that i wanted to point out when it comes to cloud storage is that when it comes to object life cycle management changes are in accordance to object creation date as well once an object is deleted it cannot be undeleted so please be careful when permanently deleting a version as well life cycle rules can take up to 24 hours to take effect so be aware when setting them and always be sure to test these life cycle rules in development first before rolling them out into production and so that's pretty much all i had to cover when it comes to versioning and object life cycle management and so you can now mark this lesson as complete and whenever you're ready join me in the console where we go hands-on with versioning object life cycle management and cloud storage as a whole [Music] welcome back in this demo we're going to cement the knowledge that we learned from the past couple lessons on cloud storage and really dive into the nitty gritty when it comes to the features and configuration you're first going to create a cloud storage bucket and upload some files to it and then interact with the bucket and the files using the console as well you're going to get your hands dirty using the gsutil command line tool and this is the tool for managing cloud storage from the command line now there's quite a bit of work to do here so with that being said let's dive in and so i am logged in here as tony bowties at gmail.com along with being in project bowtie inc and so the first thing i want to do is i want to create a cloud storage bucket so in order for me to do that i'm going to head over to the navigation menu and i'm going to scroll down to storage and here i already have a couple of buckets that i created from earlier lessons and you may have a couple buckets as well but you're going to go ahead and create a new bucket by going up to the top here and click on create bucket now i know that we've gone through this before in previous lessons but this time i wanted to go through all the configuration options that are available and so the first thing that you're prompted to do here is to name your bucket as explained in an earlier lesson it needs to be a globally unique name and so you can pick any name you choose and so for me i'm going to call this bucket bowtie inc dash 2021 i'm going to hit continue and if it wasn't a globally unique name it would error out and you would have to enter in a new name but since this bucket name is globally unique i'm able to move forward for location type you can select from region dual region and multi region with multi region under location you can select from either the americas europe or asia pacific and under dual region you have the options of again choosing from america's europe and asia pacific and you will be given the regions for each and so for this demo we're going to go ahead and choose region and we're going to keep the location as u.s east one and once you've selected that you can go ahead and hit continue and you're going to be prompted to choose a default storage class and here you have the option of selecting from the four storage classes that we discussed in an earlier lesson and so for this demo you can keep it as standard and simply click on continue and so here you're prompted to choose access control and because we're going to be diving into acls you can keep this as the default fine grain access control you can go ahead and click continue and under encryption you can keep it as the default google manage key but know that you always have the option of choosing a customer manage key and once you've uploaded your customer manage key you can select it from here and because i have no customer managed keys no other keys show up so i'm going to click on google manage keys and here under retention policy i know i haven't touched into that but just to give you some context when placing a retention policy on a bucket it ensures that all current and future objects in the bucket can't be deleted or replaced until they reach the age that you define in the retention policy so if you try to delete or replace objects where the age is less than the retention period it will obviously fail and this is great for compliance purposes in areas where logs need to be audited by regulators every year or where government required retention periods apply as well with the retention policy you have the option of locking that retention policy and when you lock a retention policy on a bucket you prevent the policy from ever being removed or the retention period from ever being reduced and this feature is irreversible so please be aware if you're ever experimenting with lock retention policies so if i set a retention policy here i can retain objects for a certain amount of seconds days months and years and for this demo we're not going to set any retention policies so i'm going to check that off and i'm going to go ahead and add a label with the key being environment and the value being test and just as a note before you go ahead and click on create over on the right hand side you will see a monthly cost estimate and you will be given an estimate with storage and retrieval as well as how much it costs for operations your sla and your estimated monthly cost and so before creating any buckets you can always do a price check to see how much it'll cost for storage size retrieval to get a good idea of how much it'll cost you monthly okay so once you're all done here you can simply click on create and it'll go ahead and create your bucket and so now that your bucket is created we want to add some files and so we first want to go into copying files from an instance to your cloud storage bucket and so in order to do that we need to create an instance and so we're gonna go back over to the navigation menu we're gonna scroll down to compute engine and we're gonna create our instance and for those who do not have your default vpc set up please be sure to create one before going ahead and creating your instance i'm going to go ahead and click on create i'm going to name this instance bowtie instance going to give it a label of environment test click on save the region is going to be east one and you can keep the default zone as us east 1b the machine type we're going to change it to e2micro and you're going to scroll down to access scopes and here your instance is going to need access to your cloud storage bucket and so it's going to need cloud storage access so you're going to click on set access for each api scroll down to storage and for this demo we'll select full gonna leave everything else as the default and simply click on create and so we'll give it a couple minutes here for instance to create okay and my instance has been created and so now i want to create some files and copy them over to cloud storage so i'm going to first navigate over to cloud storage and into my bucket and this way you can see the files that you upload and so next you're going to open up cloud shell and make this a little bigger for better viewing and so now you're going to ssh into your instance by using the command gcloud compute ssh along with your instance name the zone flag dash dash zone with the zone of us east 1b i'm going to go ahead and hit enter and you may be prompted with a message asking to authorize this api call and you want to hit authorize and you're going to be prompted to enter a passphrase for your key pair enter it in again and one more time and success we're logged into the instance i'm going to quickly clear my screen and so i know i could have sshed into the instance from the compute engine console but i wanted to display both the console and the shell on the same screen to make viewing a bit easier as i add and remove files to and from the bucket okay and so now that you're logged in you want to create your first file that you can copy over to your bucket so you can enter in the command sudo nano file a bow ties dot text hit enter and this will allow you to open up the nano editor to edit the file of bowties.txt and here you can enter in any message that you'd like for me i'm going to enter in learning to tie a bow tie takes time okay and i'm going to hit ctrl o to save hit enter to verify the file name to right and ctrl x to exit and so now i want to copy this file up to my bucket and so here is where i'm going to use the gsutil command so i'm going to type in gsutil cp for copy the name of the file which is file of bowties text along with gs colon forward slash forward slash and the name of your bucket which in my case is bow tie ink dash 2021 and this should copy my file file a bowties.txt up to my bucket of bow tie inc 2021 i'm gonna hit enter okay and it's finished copying over and if i go up here to the top right and click on refresh i can see that my file successfully uploaded and this is a great and easy method to upload any files that you may have to cloud storage okay and so now that you've copied files from your instance to your bucket you're going to now copy some files from the repo to be uploaded to cloud storage for our next step so you're gonna go ahead and exit out of the instance by just simply typing in exit i'm gonna quickly clear the screen and so here i need to clone my repo if you already have clone the repo then you can skip this step i'm going to cd tilde to make sure i'm in my home directory i'm going to do an ls and so i can see here that i've already cloned my repo so i'm going to cd into that directory and i'm going to run the command git pull to get the latest files fantastic i'm going to now clear my screen and i'm going to cd back to my home directory and so now i want to copy up the files that i want to work with to my cloud storage bucket and they are two jpegs by the name of pink elephant-bowtie as well as plaid bowtie and these files can be found in the repo marked 12 storage services under zero one cloud storage management and i will be providing this in the lesson text as well as can be found in the instructions and so i'm going to simply cd into that directory by typing in cd google cloud associate cloud engineer 12 storage services and 0 1 cloud storage management i'm going to list all the files in the directory and as you can see here pink elephant dash bow tie and plaid bow tie are both here and so i'm going to quickly clear my screen and so now for me to copy these files i'm going to use the command gsutil cp for copy star.jpg which is all the jpegs that are available along with gs colon forward slash forward slash and the bucket name which is bow tie inc dash 2021 i'm going to hit enter and it says that it's successfully copied the files i'm going to simply go up to the top right hand corner and do another refresh and success the files have been successfully uploaded another perfect example of copying files from another source to your bucket using the gsutil command line tool and so this is the end of part one of this demo it was getting a bit long so i decided to break it up and this would be a great opportunity for you to get up and have a stretch get yourself a coffee or tea and whenever you're ready part two will be starting immediately from the end of part one so you can complete this video and i will see you in part two [Music] this is part two of the managing cloud storage access demo and we'll be starting exactly where we left off in part 1. so with that being said let's dive in and so now that we've uploaded all these files we next want to make this bucket publicly available now please know that leaving a bucket public is not common practice and should only be used on the rare occasion that you are hosting a static website from your bucket and should always be kept private whenever possible especially in a production environment so please note that this is only for the purposes of this demo and so i'm going to quickly show this to you in the console so i'm going to shut down the cloud shell for just a minute and i'm going to go to the top menu and click on permissions and under permissions i'm going to click on add here you can add new members and because you want to make it publicly available you want to use the all users member so you type in all and you should get a pop-up bringing up all users and all authenticated users you want to click on all users and the role that you want to select for this demo is going to be storage object viewer so i'm going to type in storage object viewer and here it should pop up and select that and then you can click on save you're going to be prompted to make sure that this is what you want to do that you want to make this bucket public and so yes we do so you can simply click on allow public access and you will get a banner up here at the top saying that this bucket is public to internet and is a great fail safe to have in case you were to ever mistakenly make your bucket public and if i head back over to objects you can see that public access is available to all the files in the bucket and so just to verify this i'm going to copy the public url for pink elephant dash bowtie i'm going to open up a new tab paste in the url hit enter and as you can see i have public access to this picture and close this tab and so now that we've done our demo to make the bucket publicly accessible we should go ahead and remove public access so in order to remove public permissions i can simply go up to permissions and simply click on remove public permissions i'm going to get a prompt to make sure this is exactly what i want to do and yes it is so you can click on remove public permissions a very simple and elegant solution in order to remove public access from your bucket and if you go back to objects you'll see that all the public access has been removed from all the files and so now that you've experienced how to add public access to a bucket i wanted to get a little bit more granular and so we're going to go ahead and apply acl permissions for one specific object and because i like pink elephants let's go ahead and select pink elephant dash bow tie and so here i can go up to the top menu and click on edit permissions and i'll be prompted with a new window for permissions that are currently available for this object you can click on add entry click on the drop down and select public from the drop-down and it will automatically auto populate the name which is all users and the access which will be reader i'm going to go ahead and click on save and a public url will be generated and so just to verify this i'm going to click on the public url and success i now have public access to this picture yet once again i'm going to close down this tab and so now that you've configured this object for public access i want to show you how to remove public access using the command line this time so you're going to go up to the top right hand corner and open up cloud shell i'm going to quickly clear my screen and i'm going to paste in the command here which is gsutil acl ch for change minus d which is delete the name of the user which is all users and if this was a regular user you could enter in their email address along with gs colon forward slash forward slash the bucket name which in my case is bow tie ink dash 2021 and the name of the file which is pink elephant bow tie dot jpeg i'm going to hit enter and it says that it's been successfully updated and so if i go back up here to the console and i back out and go back into the file i can see here that the public url has been removed okay and now there's one last step that we need to do before ending this demo and this is to create a signed url for the file so in order to create a signed url we first need to create a private key and so we're gonna do this using a service account and so i'm gonna head on over to iam so i'm going to go up to the navigation menu i'm going to go to i am an admin and here with the menu on the left i'm going to click on service accounts here up at the top menu you're going to click on create service account and under service account name you can enter in any name but for me i'm going to enter in signed url i'm going to leave everything else as is i'm going to simply click on create i'm going to close down cloud shell because i don't really need it right now just select a role and i'm going to give it the role of storage object viewer i'm going to click on continue and i'm going to leave the rest blank and simply click on done and you should see a service account with the name of signed url and so in order to create a key i'm going to simply go over to actions and i'm going to click on the three dots and i'm going to select create key from the drop down menu and here i'm going to be prompted with what type of key that i want to create and you want to make sure that json is selected and simply click on create and this is where your key will be automatically downloaded to your downloads folder i'm going to click on close and so once you have your key downloaded you're able to start the process of generating a signed url and so i'm going to go ahead and use cloud shell in order to generate this signed url so i'm going to go ahead back up to the top and open up cloud shell again and then you can open up the cloud shell editor going to go up to the top menu in editor and click on file and you're going to select upload files and here's where you upload your key from your downloads folder and i can see my key has been uploaded right here and you can rename your key file to something a little bit more human readable so i'm going to right click i'm going to click on rename and you can rename this file as privatekey.json hit ok and so once you have your key uploaded and renamed you can now go back into the terminal to generate a signed url i'm going to quickly clear the screen i'm going to make sure that the private key is in my path by typing in ls and as you can see here privatekey.json is indeed in my path and so before i generate this key i'm going to head back on over to cloud storage i'm going to drill down into my bucket and as you can see here pink elephant dash bow tie does not have a public url and so when the sign url is generated you will get a public url that will not be shown here in the console and will be private to only the user that generated it and the users that the url has been distributed to okay and once you have everything in place you can then go ahead and paste in the command gsutil sign url minus d the allotted time which is 10 minutes the private key which is private key dot json along with gs colon forward slash forward slash your bucket name which in my case is bow tie ink dash 2021 along with the file name of pinkelephant-bowtie.jpg i'm going to hit enter and so i purposely left this error here so you can see that when you generate a signed url you need pi open ssl in order to generate it and so the caveat here is that because python 2 is being deprecated the command pip install pi openssl will not work pi open ssl needs to be installed with python3 and so to install it you're going to run the command pip3 install pi open ssl and hit enter and so once it's finished installing you can now generate your signed url i'm going to quickly clear my screen paste in the command again hit enter and success you've now generated a sign url for the object pink elephant bowtie.jpg and because this is a signed url you will see under public url there is no url there available even though it is publicly accessible and so just to verify this i'm going to highlight the link here i'm going to copy it i'm going to open up a new tab i'm going to paste in this url hit enter and success this sign url is working and anyone who has access to it has viewing permissions of the file for 10 minutes and so again this is a great method for giving someone access to an object who doesn't have an account and will give them a limited time to view or edit this object and so i wanted to congratulate you on making it through this demo and hope that it has been extremely useful in excelling your knowledge on managing buckets files and access to the buckets and files in cloud storage and so just as a recap you created a cloud storage bucket you then created an instance and copied a file from that instance to the bucket you then clone your repo to cloud shell and copy two jpeg files to your cloud storage bucket you then assigned and then removed public access to your bucket and then applied an acl to a file in the bucket making it public as well as removing public access right after you then created a service account private key and generated a signed url to an object in that bucket congratulations again on a job well done and so that's pretty much all i wanted to cover in this demo on managing cloud storage access so you can now mark this as complete and let's move on to the next one [Music] welcome back in this demo we're going to be getting into the weeds with object versioning and life cycle management using both the console and the command line we're going to go through how versioning works and what happens when objects get promoted along with creation configuration and editing these life cycle policies and so with that being said let's dive in so we're going to be starting off from where we left off in the last demo with all the resources intact that we created before and we're going to go ahead and dive right into versioning and so the first thing that you want to do is turn on versioning for your current bucket so in my case for bow tie ink dash 2021 and we're going to do this through the command line so i'm going to first go up to the top right hand corner and open up cloud shell and so you first want to see if versioning is turned on for your bucket and you can do this by using the command gsutil versioning get along with gs colon forward slash forward slash with your bucket name and hit enter and you may be prompted with a message asking you to authorize this api call you definitely want to authorize and as expected versioning is not turned on on this bucket hence the return of suspended and so in order to turn versioning on we're going to use a similar command gsutil versioning and instead of get we're going to use set on gs colon forward slash forward slash and the bucket name and hit enter and versioning has been enabled and so if i run the command gsutil version in get again i'll get a response of enabled okay great now that we have versioning enabled we can go ahead with the next step which is to delete one of the files in the bucket and so you can go ahead and select plaid bowtie.jpg and simply click on delete you can confirm the deletion and the file has been deleted now technically the file has not been deleted it is merely been converted to a non-current version and so in order to check the current and non-current versions i'm going to use the command gsutil ls minus a along with the bucket name of g s colon forward slash forward slash bow tie inc dash 2021 i'm gonna hit enter and as you can see here plaid bow tie still shows up the ls minus a command is a linux command to show all files including the hidden files and so what's different about these files is right after the dot text or dot jpg you will see a hashtag number and this is the generation number and this determines the version of each object and so what i want to do now is bring back the non-current version and make it current so i'm going to promote the non-current version of plaid bowtie.jpg to the current version and so in order to do this i'm going to run the command gsutil and v for move along with the bucket of gs colon forward slash forward slash bowtie inc hyphen 2021 and the name of the file of plaid bow tie dot jpeg along with the generation number and i'm going to copy it from the currently listed i'm going to paste it in and so now we need to put in the target which is going to be the same without the generation number and paste that in then hit enter okay operation completed and so if i go up to the top right hand corner and click on refresh i can see that now there is a current version for plaid bow tie now just know that using the move command actually deletes the non-current version and gives the new current version a new generation number and so in order to verify this i'm going to quickly clear my screen and i'm going to run the command gsutil ls minus a along with the bucket name a bow tie inc dash 2021 and the generation number here is different than that of the last now if i use the cp or copy command it would leave the non-current version and create a new version on top of that leaving two objects with two different generation numbers okay so with that step being done you now want to log into your linux instance and we're going to be doing some versioning for file of bowties.text so i'm going to go ahead and clear my screen again and i'm going to run the command gcloud compute ssh bowtie instance which is the name of my instance along with the zone flag dash dash zone of the zone us east 1b i'm going to hit enter and you should be prompted for the passphrase of your key and i'm in and so here you want to edit file a bowties.txt to a different version so you can go ahead and run the command sudo nano file a bow ties dot text and hit enter and you should have learning to tie a bow tie takes time and what you want to do is append version 2 right at the end ctrl o to save enter to verify the file name to right and control x to exit and so now we want to copy file a bow ties dot text to your current bucket mine being bow tie ink dash 2021 so i'm going to go ahead and run the command gsutil cp the name of the file which is file of bowties dot text and the target which is going to be bowtie inc 2021 and hit enter and it's copied the file to the bucket and so if i hit refresh in the console you can see that there is only one version of file of bowties.text and so to check on all the versions that i have i'm going to go back to my cloud shell i'm going to quickly clear my screen and i'm going to run the command gsutil ls minus a along with the target bucket hit enter and as you can see here there are now two versions of file of bowties.text and if i quickly open this up i'm gonna click on the url you can see here that this is version two and so this should be the latest generation of file of bowties.txt that you edited over in your instance i'm going to close this tab now and so what i want to do now is i want to promote the non-current version to be the current version in essence making version 2 the non-current version and so i'm going to run the command gsutil cp and i'm going to take the older generation number and i'm going to copy it and paste it here and the target is going to be the same without the generation number and paste it and hit enter okay and the file has been copied over so i'm going to do a quick refresh in the console i'm going to drill down into file a bowties.txt and when i click on the url link it should come up as version 1. and so this is a way to promote non-current versions to current versions using the gsutil copy command or the gsutil move command i'm going to close on this tab now i'm going to quickly clear my screen and if i run the command gsutil ls minus a again you can see that i have even more files and so these files and versions of files will eventually accumulate and continuously take up space along with costing you money and so in order to mitigate this a good idea would be to put life cycle policies into place and so you're gonna go ahead now and add a life cycle policy to the bucket and this will help manage the ever-growing accumulation of files as more files are being added to the bucket and more versions are being produced something that is very common that is seen in many different environments and so we're going to go ahead and get this done in the console so i'm going to close down cloud shell and i'm going to go back to the main page of the bucket and under the menu you can click on lifecycle and here you'll be able to add the lifecycle rules and so here you're going to click on add a rule and the first thing that you're prompted to do is to select an action and so the first rule you're going to apply is to delete non-current objects after seven days so you're gonna click on delete object you're gonna be prompted with a warning gonna hit continue and you'll be prompted to select object conditions and as discussed in an earlier lesson there are many conditions to choose from and multiple conditions can be selected so here you're going to select days since becoming non-current and in the empty field you're going to type in 7. you can click on continue and before you click on create i wanted just to note that any life cycle rule can take up to 24 hours to take effect so i'm going to click on create and here you can see the rule has been applied to delete objects after seven days when object becomes non-current and so now that we added a delete rule we're going to go ahead and add another rule to move current files that are not being used to a storage class that can save the company money and so let's go ahead and create another lifecycle rule but this time to use this set storage class action and so the files that accumulate that have been there for over 90 days you want to set the storage class the cold line so this way it'll save you some money and so you're going to click on add a rule you're going to select set storage class to cold line and as a note here it says archive objects will not be changed to cold line so you can move forward with the storage class but you can't move backwards in other words i can't move from cold line to near line or archive the cold line i can only move from near line to cold line or cold line to archive so i'm going to go ahead and click continue for the object conditions you want to select age and in the field you want to enter 90 days and here you want to hit continue and finally click on create and so in order to actually see these rules take effect like i said before it'll take up to 24 hours and so before we end this demo i wanted to show you another way to edit a life cycle policy by editing the json file itself so you can head on up to the top right and open up cloud shell i'm going to bring this down a little bit and you're going to run the command gsutil lifecycle get along with the bucket name and output it to a file called lifecycle.json and hit enter and no errors so that's a good sign next i'm going to run the command ls and as you can see here the lifecycle.json file has been written and so i'd like to edit this file where it changes the set to cold line rule from 90 days to 120 days as tony bowtie's manager thinks that they should keep the files a little bit longer before sending it to coldline and so in order to edit this file you're going to run the command sudo nano along with the name of the file of lifecycle.js you hit enter and it's going to be a long string but if you use your arrow keys and move down and then back you'll see the set to cold line rule with the age of 90 days so i'm going to move over here and i'm going to edit this to 120 and i'm going to hit ctrl o to save enter to verify file name to write and ctrl x to exit and just know that you can also edit this file in cloud shell editor and so in order for me to put this lifecycle policy in place i need to set this as the new lifecycle policy and so in order for me to do that i'm going to run the command gsutil lifecycle set along with the name of the json file which is lifecycle.json along with the bucket name and hit enter and it looks like it said it and i'm going to do quick refresh in the console just to verify and success the rule has been changed from 90 days to 120 days congratulations on completing this demo now a lot of what you've experienced here is more of what you will see in the architect exam as the cloud engineer exam focuses on more of the high level theory of these cloud storage features but i wanted to show you some real life scenarios and how to apply the theory that was shown in previous lessons into practice and so just as a recap you set versioning on the current bucket that you are working in and you deleted a file and made it non-current you then brought it back to be current again you then edited a file on your instance and copied it over to replace the current version of that file in your bucket you then promoted the non-current version as the new one and moved into lifecycle rules where you created two separate rules you created a rule to delete files along with the rule to set storage class after a certain age of the file and the last step you took was to copy the lifecycle policy to your cloud shell and edited that policy and set it to a newer edited version and so that pretty much covers this demo on object versioning and lifecycle management congratulations again on a job well done and so before you go make sure you delete all the resources you've created for the past couple of demos as you want to make sure that you're not accumulating any unnecessary costs and so i'm going to do a quick run through on deleting these resources and so i'm going to quickly close down cloud shell and i'm going to head on over to the navigation menu go to compute engine i'm going to delete my instance and i'm going to head back on over to cloud storage and delete the bucket there i'm going to confirm the deletion i'm going to click on delete and so that covers the deletion of all the resources so you can now mark this as complete and i'll see you in the next one welcome back and in this lesson i'm going to be covering cloud sql one of google cloud's many database offerings that offers reliable secure and scalable sql databases without having to worry about the complexity to set it all up now there's quite a bit to cover here so with that being said let's dive in now cloud sql is a fully managed cloud native relational database service that offers mysql postgres and sql server engines with built-in support for replication cloud sql is a database as a service offering from google where google takes care of all the underlying infrastructure for the database along with the operating system and the database software now because there are a few different types of database offerings from google cloud sql was designed for low latency transactional and relational database workloads it's also available in three different flavors of databases mysql postgres and the newest edition is sql server and all of them support standard apis for connectivity cloud sql offers replication using different types of read replicas which i will get into a little bit later and offers capabilities for high availability for continuous access to your data cloud sql also offers backups in two different flavors and allows you to restore your database from these backups with the same amount of ease now along with your backups comes point in time recovery for when you want to restore a database from a specific point in time cloud sql storage relies on connected persistent disks in the same zone that are available in regular hard disk drives or ssds that currently give you up to 30 terabytes of storage capacity and because the same technologies lie in the background for persistent disks automatic storage increase is available to resize your disks for more storage cloud sql also offers encryption at rest and in transit for securing data entering and leaving your instance and when it comes to costs you are billed for cpu memory and storage of the instance along with egress traffic as well please be aware that there is a licensing cost when it comes to windows instances now cloud sql instances are not available in the same instance types as compute engine and are only available in the shared core standard and high memory cpu types and when you see them they will be clearly marked with a db on the beginning of the cpu type you cannot customize these instances like you can with compute engine and so memory will be pre-defined when choosing the instance type now storage types for cloud sql are only available in hard disk drives and ssds you are able to size them according to your needs and as stated earlier can be sized up to 30 terabytes in size and when entering the danger zone of having a full disk you do have the option of enabling automatic storage increase so you never have to worry about filling up your disk before that 30 terabyte limit now when it comes to connecting to your cloud sql instance you can configure it with a public or private ip but know that after configuring the instance with a private ip it cannot be changed although connecting with the private ip is preferred when connecting from a client on a resource with access to a vpc as well it is always best practice to use private i p addresses for any database in your environment whenever you can now moving on to authentication options the recommended method to connecting to your cloud sql instance is using cloud sql proxy the cloud sql proxy allows you to authorize and secure your connections using iam permissions unless using the cloud sql proxy connections to an instance's public ip address are only allowed if the connection comes from an authorized network authorized networks are ip addresses or ranges that the user has specified as having permission to connect once you are authorized you can connect to your instance through external clients or applications and even other google cloud services like compute engine gke app engine cloud functions and cloud run now i wanted to focus a moment here on the recommended method for connecting to your instance which is cloud sql proxy now as mentioned before the cloud sql proxy allows you to authorize and secure your connections using iam permissions the proxy validates connections using credentials for a user or service account and wrapping the connection in an ssl tls layer that is authorized for a cloud sql instance using the cloud sql proxy is the recommended method for authenticating connections to a cloud sql instance as it is the most secure the client proxy is an open source library distributed as an executable binary and is available for linux macos and windows the client proxy acts as an intermediary server that listens for incoming connections wraps them in ssl or tls and then passes them to a cloud sql instance the cloud sql proxy handles authentication with cloud sql providing secure access to cloud sql instances without the need to manage allowed ip addresses or configure ssl connections as well this is also the best solution for applications that hold ephemeral eyepiece and while the proxy can listen on any port it only creates outgoing connections to your cloud sql instance on port 3307 now when it comes to database replication it's more than just copying your data from one database to another the primary reason for using replication is to scale the use of data in a database without degrading performance other reasons include migrating data between regions and platforms and from an on-premises database to cloud sql you could also promote a replica if the original instance becomes corrupted and i'll be getting into promoting replicas a little bit later now when it comes to a cloud sql instance the instance that is replicated is called a primary instance and the copies are called read replicas the primary instance and read replicas all reside in cloud sql read replicas are read-only and you cannot write to them the read replica processes queries read requests and analytics traffics thus reducing the load on the primary instance read replicas can have more cpus in memory than the primary instance but they cannot have any less and you can have up to 10 read replicas per primary instance and you can connect to a replica directly using its connection name and ip address cloud sql supports the following types of replicas read replicas cross region read replicas external read replicas and cloud sql replicas when replicating from an external server now when it comes to read replicas you would use it to offload work from a cloud sql instance the read replica is an exact copy of the primary instance and data and other changes on the primary instance are updated in almost real time on the read replica a read replica is created in a different region from the primary instance and you can create a cross region read replica the same way as you would create an in-region replica this improves read performance by making replicas available closer to your application's region it also provides additional disaster recovery capability to guard you against a regional failure it also lets you migrate data from one region to another with minimum downtime and lastly when it comes to external read replicas these are external mysql instances that replicate from a cloud sql primary instance for example a mysql instance running on compute engine is considered an external instance and so just as a quick note here before you can create a read replica of a primary cloud sql instance the instance must meet the following requirements automated backups must be enabled binary logging must be enabled which requires point-in-time recovery to be enabled and at least one backup must have been created after binary logging was enabled and so when you have read replicas in your environment it gives you the flexibility of promoting those replicas if needed now promoting replicas is a feature that can be used for when your primary database becomes corrupted or unreachable now you can promote an in-region read replica or cross-region re-replica depending on where you have your read replicas hosted so when you promote a read replica the instance stops replication and converts the instance to a standalone cloud sql primary instance with read and write capabilities please note that this cannot be undone and also note that when your new primary instance has started your other read replicas are not transferred over from the old primary instance you will need to reconnect your other read replicas to your new primary instance and as you can see here promoting a replica is done manually and intentionally whereas high availability has a standby instance that automatically becomes the primary in case of a failure horizontal outage now when it comes to promoting cross-region replicas there are two common scenarios for promotion regional migration which performs a planned migration of a database to a different region and disaster recovery and this is where you would fail over a database to another region in the event that the primary instances region becomes unavailable both use cases involve setting up cross-region replication and then promoting the replica the main difference between them is whether the promotion of the replica is planned or unplanned now if you're promoting your replicas for a regional migration you can use a cross region replica to migrate your database to another region with minimal downtime and this is so you can create a replica in another region wait until the replication catches up promote it and then direct your applications to the newly promoted instance the steps involved in promotion are the same as for promoting an in-region replica and so when you're promoting replicas for disaster recovery cross-region replicas can be used as part of this disaster recovery procedure you can promote a cross-region replica to fail over to another region should the primary instances region become unavailable for an extended period of time so in this example the entire u.s east 1 region has gone down yet the reed replica in the europe region is still up and running and although there may be a little bit more latency for your customers in north america i'm able to promote this read replica connect it to the needed resources and get back to business now moving along to high availability cloud sql offers aha capabilities out of the box the aha configuration sometimes called a cluster provides data redundancy so a cloud sql instance configured for ha is also called a regional instance and is located in a primary and secondary zone within the configured region within a regional instance the configuration is made up of a primary instance and a standby instance and through synchronous replication to each zone's persistent disk all rights made to the primary instance are also made to the standby instance each second the primary instance writes to a system database as a heartbeat signal if multiple heartbeats aren't detected failover is initiated and so if an ha-configured instance becomes unresponsive cloud sql automatically switches to serving data from the standby instance and this is called a failover in this example the primary instance or zone fails and failover is initiated so if the primary instance is unresponsive for approximately 60 seconds or the zone containing the primary instance experiences an outage failover will initiate the standby instance immediately starts serving data upon reconnection through a shared static ip address with the primary instance and the standby instance now serves data from the secondary zone and now when the primary instance is available again a fail back will happen and this is when traffic will be redirected back to the primary instance and the standby instance will go back into standby mode as well the regional persistent disk will pick up replication to the persistent disk in that same zone and with regards to billing an ha configured instance is charged at double the price of a standalone instance and this includes cpu ram and storage also note that the standby instance cannot be used for read queries and this is where it differs from read replicas as well a very important note here is that automatic backups and point in time recovery must be enabled for high availability and so the last topic that i wanted to touch on is backups and backups help you restore lost data to your cloud sql instance you can also restore an instance that is having problems from a backup you enable backups for any instance that contains necessary data backups protect your data from loss or damage enabling automated backups along with binary logging is also required for some operations such as clone and replica creation by default cloud sql stores backup data in two regions for redundancy one region can be the same region that the instance is in and the other is a different region if there are two regions in a continent the backup data remains on the same continent cloud sql also lets you select a custom location for your backup data and this is great if you need to comply with data residency regulations for your business now cloud sql performs two types of backups on-demand backups and automated backups now with on-demand backups you can create a backup at any time and this is useful for when you're making risky changes that may go sideways you can always create on-demand backups for any instance whether the instance has automatic backups enabled or not and these backups persist until you delete them or until their instance is deleted now when it comes to automated backups these use a four hour backup window these backups start during the backup window and just as a note when possible you should schedule your backups when your instance has the least activity automated backups occur every day when your instance is running at any time in the 36 hour window and by default up to seven most recent backups are retained you can also configure how many automated backups to retain from 1 to 365. now i've touched on this topic many times in this lesson and i wanted to highlight it for just a second and this is point-in-time recovery so point-in-time recovery helps you recover an instance to a specific point in time for example if an error causes a loss of data you can recover a database to its state before the error happened a point in time recovery always creates a new instance and you cannot perform a point in time recovery to an existing instance and point in time recovery is enabled by default when you create a new cloud sql instance and so when it comes to billing by default cloud sql retains seven days of automated backups plus all on-demand backups for an instance and so i know there is a lot to retain in this lesson on cloud sql but be sure that these concepts and knowing the difference between them as well as when to use each feature will be a sure help in the exam along with giving you the knowledge you need to use cloud sql in your role as a cloud engineer and so that's pretty much all i had to cover when it comes to cloud sql so you can now mark this lesson as complete and let's move on to the next one welcome back and in this lesson i wanted to touch on google cloud's global relational database called cloud spanner now cloud spanner is the same in some ways as cloud sql when it comes to asset transactions sql querying and strong consistency but differs in the way that data is handled under the hood than cloud sql and so knowing this database only at a high level is needed for the exam but i'll be going into a bit more detail just to give you a better understanding on how it works so with that being said let's dive in now cloud spanner is a fully managed relational database service that is both strongly consistent and horizontally scalable cloud spanner is another database as a service offering from google and so it strips away all the headaches of setting up and maintaining the infrastructure and software needed to run your database in the cloud now being strongly consistent in this context is when data will get passed on to all the replicas as soon as a write request comes to one of the replicas of the database cloud spanner uses truetime a highly available distributed atomic clock system that is provided to applications on all google servers it applies a time stamp to every transaction on commit and so transactions in other regions are always executed sequentially cloud spanner can distribute and manage data at a global scale and support globally consistent reads along with strongly consistent distributed transactions now being fully managed cloud spanner handles any replicas that are needed for availability of your data and optimizes performance by automatically sharding the data based on request load and size of the data part of why cloud spanner's high availability is due to its automatic synchronous data replication between all replicas in independent zones cloud spanner scales horizontally automatically within regions but it can also scale across regions for workloads that have higher availability requirements making data available faster to users at a global scale along with node redundancy quietly added for every node deployed in the instance and when you quickly add up all these features of cloud spanner it's no wonder that it's available to achieve five nines availability on a multi-regional instance and four nines availability on a regional instance cloud spanner is highly secure and offers data layer encryption audit logging and iam integration cloud spanner was designed to fit the needs of specific industries such as financial services ad tech retail and global supply chain along with gaming and pricing for cloud spanner comes in at 90 cents per node per hour with the cost of storage coming in at 30 cents per gigabyte per month definitely not cheap but the features are plentiful now this isn't in the exam but i did want to take a moment to dive into the architecture for a bit more context as to why this database is of a different breed than the typical sql database now to use cloud spanner you must first create a cloud spanner instance this instance is an allocation of resources that is used by cloud spanner databases created in that instance instance creation includes two important choices the instance configuration and the node count and these choices determine the location and the amount of the instances cpu and memory along with its storage resources your configuration choice is permanent for an instance and only the node count can be changed later if needed an instance configuration defines the geographic placement and replication of the database in that instance either regional or multi-region and please note that when you choose a multi-zone configuration it allows you to replicate the databases data not just in multiple zones but in multiple zones across multiple regions and when it comes to the node count this determines the number of nodes to allocate to that instance these nodes allocate the amount of cpu memory and storage needed for your instance to either increase throughput or storage capacity there is no instance types to choose from like cloud sql and so when you need more power you simply add another node now for any regional configuration cloud spanner maintains exactly three read write replicas each within a different zone in that region each read write replica contains a full copy of your operational database that is able to serve rewrite and read only requests cloud spanner uses replicas in different zones so that if a single zone failure occurs your database remains available in a multi-region instance configuration the instance is allotted a combination of four read write and read only replicas and just as a note a three node configuration minimum is what is recommended for production by google and as cloud spanner gets populated with data sharding happens which is also known as a split and cloud spanner creates replicas of each database split to improve performance and availability all of the data in a split is physically stored together in a replica and cloud spanner serves each replica out of an independent failure zone and within each replica set one replica is elected to act as the leader leader replicas are responsible for handling rights while any read write or read only replica can serve a read request without communicating with the leader and so this is the inner workings of cloud spanner at a high level and not meant to confuse you but to give you a better context of how cloud spanner although it is a relational sql database is so different than its cloud sql cousin now before ending this lesson i wanted to touch on node performance for a quick moment and so each cloud spanner node can provide up to 10 000 queries per second or qps of reads or 2000 qps of writes each node provides up to two terabytes of storage and so if you need to scale up the serving and storage resources in your instance you add more nodes to that instance and remember as noted earlier that adding a node does not increase the number of replicas but rather increases the resources each replica has in the instance adding nodes gives each replica more cpu and ram which increases the replicas throughput and so if you're looking to scale up automatically you can scale the numbers of nodes in your instance based on the cloud monitoring metrics on cpu or storage utilization in conjunction with using cloud functions to trigger and so when you are deciding on a relational database that provides global distribution and horizontally scalable that handles transactional workloads in google cloud cloud spanner will always be the obvious choice over cloud sql and so that's pretty much all i have to cover when it comes to this overview on cloud spanner so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this lesson we will be going over the available nosql databases available in google cloud this lesson is meant to be another overview just to familiarize you with the nosql database options as they show up in the exam this lesson is not meant to go in depth on databases but an overview and will give you a good understanding on what features are available for each and their use cases so with that being said let's dive in now there are four managed nosql databases available in google cloud and i will be briefly going over them and i'll be starting this off by discussing bigtable now cloud bigtable is a fully managed wide column nosql database designed for terabyte and petabyte scale workloads that offers low latency and high throughput bigtable is built for real-time application serving workloads as well as large-scale analytical workloads cloud bigtable is a regional service and if using replication a copy is stored in a different zone or region for durability cloud bigtable is designed for storing very large amounts of single keyed data while still being able to provide very low latency and because throughput scales linearly you can increase the queries per second by adding more bigtable nodes when you need them bigtable throughput can be dynamically adjusted by adding or removing cluster nodes without restarting meaning you can increase the size of a bigtable cluster for just a few hours to handle a large load and then reduce the cluster size again and do it all without any downtime bigtable is an ideal source for map reduce operations and integrates easily with all the existing big data tools such as hadoop dataproc and dataflow along with apache hbase and when it comes to price bigtable is definitely no joke pricing for bigtable starts at 65 cents per hour per node or over 450 dollars a month for a one node configuration with no data now you can use bigtable to store and query all of the following types of data such as cpu and memory usage over time for multiple servers marketing data such as purchase histories and customer preferences financial data such as transaction histories stock prices and currency exchange rates iot data or internet of things such as usage reports from energy meters and home appliances and lastly graph data such as information about how users are connected to one another cloud bigtable excels as a storage engine as it can batch mapreduce operations stream processing or analytics as well as being used for storage for machine learning applications now moving on to the next nosql database is cloud datastore and cloud datastore is a highly scalable nosql document database built for automatic scaling high performance and ease of application development datastore is redundant within your location to minimize impact from points of failures and therefore can offer high availability of reads and rights cloud datastore can execute atomic transactions where a set of operations either all succeed or none occur cloud datastore uses a distributed architecture to automatically manage scaling so you never have to worry about scaling manually as well what's very unique about cloud datastore is that it has a sql-like query language that's available called gql also known as gql gql maps roughly to sql however a sql role column lookup is limited to a single value whereas in gql a property can be a multiple value property this consistency model allows an application to handle large amounts of data and users while still being able to deliver a great user experience data is automatically encrypted before it is written to disk and automatically decrypted when read by an authorized user now this does not reflect in the exam as of yet and i will be updating this lesson if and when it happens but firestore is the newest version of datastore and introduces several improvements over datastore existing datastore users can access these improvements by creating a new firestore database instance in datastore mode and in the near future all existing datastore databases will be automatically upgraded to firestore in datastore mode now moving right along cloud datastore holds a really cool feature for developers that's called datastore emulator and this provides local emulation of the production datastore environment so that you can use to develop and test your application locally this is a component of the google cloud sdks gcloud tool and can be installed by using the gcloud components install command that we discussed earlier on in the course and so moving on to use cases for datastore it is ideal for applications that rely on highly available structured data at scale you can use datastore for things like product catalogs that provide real-time inventory and product details for a retailer user profiles that deliver a customized experience based on the user's past activities and preferences as well as transactions based on asset properties for example transferring funds from one bank account to another next up we have firestore for firebase and so this is a flexible scalable nosql cloud database to store and sync data for client and server side development and is available for native c plus unity node.js java go and python sdks in addition to rest and rpc apis pretty much covering the gamut of most major programming languages now with cloud firestore you store data in documents that contain fields mapping to values these documents are stored in collections which are containers for your documents that you can use to organize your data and build queries documents support many different data types as well you can also create sub collections within documents and build hierarchical data structures cloud firestore is serverless with absolutely no servers to manage update or maintain and with automatic multi-region replication and strong consistency google is able to hold a five nines availability guarantee and so when it comes to querying in cloud firestore it is expressive efficient and flexible you can create shallow queries to retrieve data at the document level without needing to retrieve the entire collection or any nested subcollections cloud firestore uses data synchronization to update data in real time for any connected device as well it also caches data that your application is actively using so that the application can write read listen to and query data even if the device is offline when the device comes back online cloud firestore synchronizes any local changes back to cloud firestore you can also secure your data in cloud firestore with firebase authentication and cloud firestore security rules for android ios and javascript or you can use iam for server side languages and when it comes to costs firestore falls into the always available free tier where you can use one database holding five gigabytes or if you need more you can move into their paid option now firebase also has another database sharing similar features like having no servers to deploy and maintain real-time updates along with the free tier in this database is called real time database and is used for more basic querying simple data structure and keeping things to one database it's something i like to call firestore lite real time database does not show up in the exam but i wanted to bring it to light as it is part of the firebase family just know that you can use both databases within the same firebase application or project as both can store the same types of data client libraries work in a similar manner and both hold real-time updates now although firebase is a development platform and not a database service i wanted to give it a quick mention for those of you who are unfamiliar with the tie-in to firestore with firebase firebase is a mobile application development platform that provides tools and cloud services to help enable developers to develop applications faster and more easily and since it ties in nicely with firestore it becomes the perfect platform for mobile application development okay so moving on to our last nosql database is memorystore and memorystore is a fully managed service from google cloud for either redis or memcached in memory datastore to build application caches and this is a common service used in many production environments specifically when the need for caching arises memory store automates the administration tasks for redis and memcached like enabling high availability failover patching and monitoring so you don't have to and when it comes to memory store for redis instances in the standard tier these are replicated across zones monitored for health and have fast automatic failover standard tier instances also provide an sla of three nines availability memory store for redis also provides the ability to scale instant sizes seamlessly so that you can start small and increase the size of the instance as needed memory store is protected from the internet using vpc networks and private ip and also comes with iam integration systems are monitored around the clock ensuring that your data is protected at all times and know that the versions are always kept up to date with the latest critical patches ensuring your instances are secure now when it comes to use cases of course the first thing you will see is caching and this is the main reason to use memory store as it provides low latency access and high throughput for heavily accessed data compared to accessing the data from a disk common examples of caching is session management frequently accessed queries scripts or pages so when using memory store for leaderboards and gaming this is a common use case in the gaming industry as well as using it for player profiles memory store is also a perfect solution for stream processing combined with data flow memory store for redis provides a scalable fast in memory store for storing intermediate data that thousands of clients can access with very low latency and so when it comes to nosql databases these are all the available options on google cloud and as i said before it will only show up on the exam at merely a high level and so knowing what each of these databases are used for will be a huge benefit along with being an entry to diving deeper into possibly using these services within your day-to-day job as a cloud engineer and so that's pretty much all i wanted to cover when it comes to nosql databases available in google cloud so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this lesson we'll be going over the big data ecosystem in an overview just to familiarize you with the services that are available in google cloud and are the services that will show up in the exam this lesson is not meant to go in depth but is an overview and will give you a good understanding on what these services can do and how they all work together to make sense of big data as a whole so getting right into it i wanted to first ask the question what is big data i mean many people talk about it but what is it really well big data refers to massive amounts of data that would typically be too expensive to store manage and analyze using traditional database systems either relational or monolithic as the amount of data that we have been seeing over the past few years has started to increase these systems have become very inefficient because of their lack of flexibility for storing unstructured data such as images text or video as well as accommodating high velocity or real-time data or scaling to support very large petabyte scale data volumes for this reason the past few years has seen the mainstream adoption of new approaches to managing and processing big data including apache hadoop and nosql database systems however those options often prove to be complex to deploy manage and use in an on-premises situation now the ability to consistently get business value from data fast and efficiently is now becoming the de facto of successful organizations across every industry the more data a company has access to the more business insights and business value they're able to achieve like gain useful insights increase revenue get or retain customers and even improve operations and because machine learning models get more efficient as they are trained with more data machine learning and big data are highly complementary all in all big data brings some really great value to the table that is impossible for any organization to turn down and so now that we've gone through that overview of what big data is i wanted to dive into some shorter overviews of the services available for the big data ecosystem on google cloud and so the first service that i'd like to start with is bigquery now bigquery is a fully managed serverless data warehouse that enables scalable analysis over petabytes of data this service supports querying using sql and holds built-in machine learning capabilities you start by ingesting data into bigquery and then you are able to take advantage of all the power it provides so big data would ingest that data by doing a batch upload or by streaming it in real time and you can use any of the currently available google cloud services to load data into bigquery you can take a manual batch ingestion approach or stream using pub sub etl data and with bigquery data transfer service you can automatically transfer data from external google data sources and partner sas applications to bigquery on a scheduled and fully managed basis and the best part is batch and export is free bigquery's high-speed streaming api provides an incredible foundation for real-time analytics making business data immediately available for analysis and you can also leverage pub sub and data flow to stream data into bigquery bigquery transparently and automatically provides highly durable replicated storage in multiple locations for high availability as well as being able to achieve easy resource bigquery keeps a seven day history of changes in case something were to go wrong bigquery supports standard sql querying which reduces the need for code rewrites you can simply use it as you would for querying any other sql compliant database and with dataproc and dataflow bigquery provides integration with the apache big data ecosystem allowing existing hadoop spark and beam workloads to read or write data directly from bigquery using the storage api bigquery also makes it very easy to access this data by using the cloud console using the bq command line tool or making calls to the bigquery rest api using a variety of client libraries such as java.net or python there are also a variety of third-party tools that you can use to interact with bigquery when visualizing the data or loading the data bigquery provides strong security and governance controls with fine-grained controls through integration with identity and access management bigquery gives you the option of geographic data control without the headaches of setting up and managing clusters and other computing resources in different zones and regions bigquery also provides fine grain identity and access management and rest assured that your data is always encrypted at rest and in transit now the way that bigquery calculates billing charges is by queries and by storage storing data in bigquery is comparable in price with storing data in cloud storage which makes it an easy decision for storing data in bigquery there is no upper limit to the amount of data that can be stored in bigquery so if tables are not edited for 90 days the price of storage for that table drops by 50 percent query costs are also available as on-demand and flat rate pricing and when it comes to on-demand pricing you are only charged for bytes read not bytes returned in the end bigquery scales seamlessly to store and analyze petabytes to exabytes of data with ease now there are so many more features to list but if you are interested feel free to dive into the other features with the supplied link in the lesson text now moving on to the next service is pub sub and pub sub is a fully managed real-time messaging service that allows you to send and receive messages between independent applications it acts as messaging oriented middleware or event ingestion and delivery for streaming analytics pipelines and so a publisher application creates and send messages to a topic subscriber applications create a subscription to a topic and receives messages from it and so i wanted to take a moment to show you exactly how it works so first the publisher creates messages and sends them to the messaging service on a specified topic a topic is a named entity that represents a feed of messages a publisher application creates a topic in the pub sub service and sends messages to that topic a message contains a payload and optional attributes that describe the content the service as a whole ensures that published messages are retained on behalf of subscriptions and so a published message is retained for a subscription in a message queue shown here as message storage until it is acknowledged by any subscriber consuming messages from that subscription pub sub then forwards messages from a topic to all of its subscriptions individually a subscriber then receives messages either by pub sub pushing them to the subscriber's chosen endpoint or by the subscriber pulling them from the service the subscriber then sends an acknowledgement to the pub sub service for each received message the service then removes acknowledged messages from the subscriptions message queue and some of the use cases for pub sub is balancing large task queues distributing event notifications and real-time data streaming from various sources and so the next service that i wanted to get into is composer now composer is a managed workflow orchestration service that is built on apache airflow this is a workflow automation tool for developers that's based on the open source apache airflow project similar to an on-premises deployment cloud composer deploys multiple components to run airflow in the cloud airflow is a platform created by the community to programmatically author schedule and monitor workflows the airflow scheduler as you see here executes the tasks on an array of workers while following the specified dependencies and storing the data in a database and having a ui component for easy management now breaking down these workflows for just a sec in data analytics a workflow represents a series of tasks for ingesting transforming analyzing or utilizing data in airflow workflows are created using dags which are a collection of tasks that you want to schedule and run and organizes these tasks to ensure that each task is executed at the right time in the right order or with the right issue handling now in order to run the specialized workflows provision environments are needed and so composer deploys these self-contained environments on google kubernetes engine that work with other google cloud services using connectors built into airflow the beauty of composer is that you can create one or more of these environments in a single google cloud project using any supported region without having to do all the heavy lifting of creating a full-blown apache airflow environment now when it comes to data flow dataflow is a serverless fully managed processing service for executing apache beam pipelines for batch and real-time data streaming the apache beam sdk is an open source programming model that enables you to develop both batch and streaming pipelines using one of the apache beam sdks you build a program that defines the pipeline then one of apache beam's supported distributed processing back-ends such as data flow executes that pipeline the data flow service then takes care of all the low-level details like coordinating individual workers sharding data sets auto scaling and exactly once processing now in its simplest form google cloud data flow reads the data from a source transforms it and then writes the data back to a sink now getting a bit more granular with how this pipeline works data flow reads the data presented from a data source once the data has been read it is put together into a collection of data sets called a p collection and this allows the data to be read distributed and processed across multiple machines now at each step in which the data is transformed a new p collection is created and once the final collection has been created it is written to async and this is the full pipeline of how data goes from source to sync this pipeline within data flow is called a job and finally here is a high-level overview of what a data flow job would look like when you involve other services within google cloud and put together in an end-to-end solution from retrieving the data to visualizing it and finally when it comes to pricing data flow jobs are billed in per second increments so you're only charged for when you are processing your data now moving on to data proc this is a fast and easy way to run spark hadoop hive or pig on google cloud in an on-premises environment it takes 5 to 30 minutes to create spark and hadoop clusters data proc clusters take 90 seconds or less on average to be built in google cloud dataproc has built-in integration with other google cloud platform services and use spark and hadoop clusters without any admin assistance so when you're done with the cluster you can simply turn it off so you don't spend money on an idle cluster as well there's no need to worry about data loss because data proc is integrated with cloud storage bigquery and cloud bigtable the great thing about dataproc is you don't need to learn new tools or apis to use it spark hadoop pig and hive are all supported and frequently updated and when it comes to pricing you are billed at one cent per vcpu in your cluster per hour on top of the other resources you use you also have the flexibility of using preemptable instances for even lower compute cost now although cloud data proc and cloud data flow can both be used to implement etl data warehousing solutions they each have their strengths and weaknesses and so i wanted to take a quick moment to point them out now with dataproc you can easily spin up clusters through the console the sdk or the api and turn it off when you don't need it with dataflow it is serverless and fully managed so there are never any servers to worry about and when it comes to having any dependencies to tools in the hadoop or spark ecosystem data proc would be the way to go but if you're looking to make your jobs more portable across different execution engines apache beam allows you to do this and is only available on data flow moving on to the next service is cloud data lab now cloud data lab is an interactive developer tool created to explore analyze transform and visualize data and build machine learning models from your data data lab uses open sourced jupyter notebooks a well-known format used in the world of data science it runs on compute engine and connects to multiple cloud services easily so you can focus on your data science tasks it also integrates with all of the google services that help you simplify data processing like bigquery and cloud storage cloud data lab is packaged as a container and run in a vm instance cloud data lab uses notebooks instead of text files containing code notebooks bring together code documentation written as markdown and the results of code execution whether it's text image or html or javascript like a code editor or ide notebooks help you write code and they allow you to execute code in an interactive and iterative manner rendering the results alongside the code cloud data lab notebooks can be stored in google cloud source repository this git repository is cloned onto persistent disk when attached to the vm now when it comes to prepping your data before consumption whether it be data cleansing cleaning prepping or alteration this is where data prep hits it out of the park dataprep is a serverless intelligent data service for visually exploring cleaning and preparing structured and unstructured data for analysis reporting and machine learning it automatically detects schemas data types possible joins and anomalies such as missing values outliers and duplicates so you don't have to the architecture that i'm about to show you is how data prep shines the raw data that's available from various different sources is ingested into cloud data prep to clean and prepare the data data prep then sends the data off to cloud data flow to refine that data and then sent off to cloud storage or bigquery for storage before being analyzed by one of the many available bi tools now these big data services are used by many data analysts in the field and it's great to know what services that can be used to help process the data needed for their specific job as well for the exam you only need to know these services at a high level and not to know them in depth but if you seem interested in diving into any of these services to know more about them i highly encourage you to dive in after the course and really take a look at them and that's pretty much all i have to cover in this lesson on the services that are available for the big data ecosystem in google cloud so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back this lesson is going to be based on the foundation of machine learning i'm going to go over what machine learning is what it can do for us the machine learning ecosystem on google cloud and hopefully answer any questions along the way this lesson will be a high level overview of the services available on google cloud yet these services that are available are a need to know as they come up in the exam and hopefully will give you some really cool ideas on the possibilities of building something truly fantastic on google cloud so what is machine learning well machine learning is functionality that helps enable software to perform tasks without any explicit programming or rules traditionally considered a subcategory of artificial intelligence machine learning involves statistical techniques such as deep learning also known as neural networks that are inspired by theories about how the human brain processes information it is trained to recognize patterns in collected data using algorithmic models and this collected data includes video images speech or text and because machine learning is very expensive to run on-premises is an efficient place for machine learning due to the use of massive computation at scale and as explained before machine learning is always better with big data so now i wanted to touch on what can machine learning do for us well it can categorize images such as photos faces or satellite imagery it can look for keywords in text documents or emails it can flag potentially fraudulent transactions when it comes to credit cards or debit cards it can enable software to respond accurately to voice commands it can also translate languages in text or audio and these are just some of the common functions that machine learning can do for us so getting into google's machine learning platform itself machine learning has been a cornerstone of google's internal systems for years primarily because their need to automate data-driven systems on a massive scale and doing this has provided unique insight into the right techniques infrastructure and frameworks that help their customers get optimal value out of machine learning the originally developed open source framework for use inside of google called tensorflow is now the standard in the data science community in addition to heavily contributing to the academic and open source communities google's machine learning researchers helped bring that functionality into google products such as g suite search and photos in addition to google's internal operations when it comes to data center automation now here is an overview of all the machine learning services that we will be covering and that you will need to know only at a high level for the exam and we'll start off with the site api services starting with the vision api the vision api offers powerful pre-trained machine learning models that allow you to assign labels to images and quickly classify them into millions of pre-defined categories vision api can read printed and handwritten text it can detect objects and faces and build metadata into an image catalog of your choice now when it comes to video intelligence it has pre-trained machine learning models that automatically recognizes more than 20 000 objects places and actions in stored and streaming video you can gain insights from video in near real time using the video intelligence streaming video apis and trigger events based on objects detected you can easily search a video catalog the same way you search text documents and extract metadata that can be used to index organize and search video content now moving on to the language apis we start off with the natural language api and this uses machine learning to reveal the structure and meaning of text you can extract information about people places and events and better understand social media sentiment and customer conversations natural language enables you to analyze text and also integrate it with your document storage on cloud storage now with the translation api it enables you to dynamically translate between languages using google's pre-trained or custom machine learning models translation api instantly translates text into more than 100 languages for your website and apps with optional customization features following another grouping of machine learning is the conversation apis first up we have dialog flow dialog flow is a natural language understanding platform that makes it easy to design and integrate a conversational user interface into your application or device it could be a mobile app a web application a bot or an interactive voice response system using dialogflow you can provide new and engaging ways for users to interact with your product dialogflow can analyze multiple types of input from your customers including text or audio inputs like from a phone or voice recording and it can also respond to your customers in a couple of ways either through text or with synthetic speech now with the speech-to-text api this api accurately converts speech into text it can transcribe content with accurate captions and deliver better user experience in products through voice commands going the other way from text to speech this api enables developers to synthesize natural sounding speech with over a hundred different voices available in multiple languages and variants text to speech allows you to create lifelike interactions with their users across many applications and devices and to finish off our machine learning segment i wanted to touch on auto ml automl is a suite of machine learning products that enables developers with very limited machine learning expertise to train high quality models specific to their business needs in other words using automl allows making deep learning easier to use and relies on google's state-of-the-art transfer learning and neural architecture search technology so you can now generate high quality training data and be able to deploy new models based on your data in minutes automl is available for vision video intelligence translation natural language tables inference and recommendation apis now i know this has been a lot to cover for this machine learning lesson and the ecosystem around it but is a necessity for the exam and will also help you build really cool products when it comes to your role as an engineer again all the services that i have discussed in this lesson should be known at a high level only although my recommendation would be to dive deeper into these services by checking out the links in the lesson text below and having some fun with these products getting to know these services will really help up your game when it comes to getting to know these services a little bit more in depth and will really help you gain more momentum when it comes to building any applications or applying them to any currently running applications i personally found it extremely valuable and really cemented my knowledge when it came to machine learning i also had a ton of fun doing it and so that's all i have for this lesson on machine learning so you can now mark this lesson as complete and let's move on to the next one [Music] welcome back and in this lesson we'll be diving into a suite of tools used on the google cloud platform that allow you to operate monitor and troubleshoot your environment known as operation suite and previously known as stackdriver this lesson will be mostly conceptual and gear more towards what the suite of tools do as it plays a big part not only in the exam but for the needs of gaining insight from all the resources that exist in your environment now there are a few tools to cover here so with that being said let's dive in now the operation suite is a suite of tools for logging monitoring and application diagnostics operation suite ingests this data and generates insights using dashboards charts and alerts this suite of tools are available for both gcp and aws you can connect to aws using an aws role and a gcp service account you can also monitor vms with specific agents that again both run on gcp for compute engine and aws ec2 operation suite also allows the added functionality of monitoring any applications that's running on those vms operation suite is also available for any on-premises infrastructure or hybrid cloud environments operation suite has a native integration within gcp out of the box so there's no real configurations that you need to do and integrates with almost all the resources on google cloud such as the previously mentioned compute engine gke app engine and bigquery and you can find and fix issues faster due to the many different tools an operation suite can reduce downtime with real-time alerting you can also find support from a growing partner ecosystem of technology integration tools to expand your operations security and compliance capabilities now the operation suite comprises of six available products that covers the gamut of all the available tools you will need that allows you to monitor troubleshoot and improve application performance on your google cloud environment and i will be going over these products in a bit of detail starting with monitoring now cloud monitoring collects measurements or metrics to help you understand how your applications and system services are performing giving you the information about the source of the measurements time stamped values and information of those values that can be broken down through time series data cloud monitoring can then take the data provided and use pre-defined dashboards that require no setup or configuration effort cloud monitoring also gives you the flexibility to create custom dashboards that display the content you select you can use the widgets available or you can install a dashboard configuration that is stored in github now in order for you to start using cloud monitoring you need to configure a workspace now workspaces organize monitoring information in cloud monitoring this is a single pane of glass where you can view everything that you're monitoring in your environment it is also best practice to use a multi-project workspace so you can monitor multiple projects from a single pane of glass now as i mentioned earlier cloud monitoring has an agent and this gathers system and application metrics from your vm and sends them to cloud monitoring you can monitor your vms without the agent but you will only get specific metrics such as cpu disk traffic network traffic and uptime using the agent is optional but is recommended by google and with the agent it allows you to monitor many third-party applications and just as a note cloud logging has an agent as well and works well together with cloud monitoring to create visualize and alert on metrics based on log data but more on that a little bit later cloud monitoring is also available for gke and this will allow you to monitor your clusters as it manages the monitoring and logging together and this will monitor clusters infrastructure its workloads and services as well as your nodes pods and containers so when it comes to alerting this is defined by policies and conditions so an a learning policy defines the conditions under which a service is considered unhealthy when these conditions are met the policy is triggered and it opens a new incident and sends off a notification a policy belongs to an individual workspace and each workspace can contain up to 500 policies now conditions determine when an alerting policy is triggered so all conditions watch for three separate things the first one is a metric the second one is a behavior in some way and the third one is for a period of time describing a condition includes a metric to be measured and a test for determining when the metric reaches a state that you want to know about so when an alert is triggered you could be notified using notification channels such as email sms as well as third party tools such as pagerduty and slack now moving on to cloud logging cloud logging is a central repository for log data from multiple sources and as described earlier logging can come not just from google but with aws as well as on-premises environments cloud logging handles real-time log management and analysis and has tight integration with cloud monitoring it collects platform system and application logs and you also have the option of exporting logs to other sources such as long-term storage like cloud storage or for analysis like bigquery you can also export to third-party tools as well now diving into the concepts of cloud logging these are associated primarily with gcp projects so logs viewer only shows logs from one specific project now when it comes to log entries log entry records a status or an event a project receives log entries when services being used produce log entries and to get down to the basics logs are a named collection of log entries within a google cloud resource and just as a note each log entry includes the name of its log logs only exist if they have log entries and the retention period is the length of time for which your logs are kept so digging into the types of logs that cloud logging handles there are three different types of logs there are audit logs transparency logs and agent logs now with audit logs these are logs that define who did what where and when they also show admin activity and data access as well as system events continuing on to access transparency logs these are logs for actions taken by google so when google staff is accessing your data due to a support ticket the actions that are taken by the google staff are logged within cloud logging now when it comes to agent logs these are the logs that come from agents that are installed on vms the logging agent sends system and third-party logs on the vm instance to cloud logging moving on to error reporting this looks at real-time error monitoring and alerting it counts analyzes and aggregates the errors that happen in your gcp environment and then alerts you when a new application error occurs details of the error can be sent through the api and notifications are still in beta error reporting is integrated into cloud functions and google app engine standard which is enabled automatically error reporting is in beta for compute engine kubernetes engine and app engine flexible as well as aws ec2 air reporting can be installed in a variety of languages such as go java.net node.js python php and ruby now moving into debugger this tool debugs a running application without slowing it down it captures and inspects the call stack and local variables in your application this tool debugs a running application without slowing it down it captures and inspects the call stack and local variables in your application this is also known as taking a snapshot once the snapshot has been taken a log point can be injected to allow you to start debugging debugger can be used with or without access to your application source code and if your repo is not local it can be hooked into a remote git repo such as github git lab or bitbucket debugger is integrated with google app engine automatically and can be installed on google compute engine gke and google app engine debugger is integrated with google app engine automatically and can be installed on gke debugger is integrated with google app engine automatically and can be installed on google compute engine google kubernetes engine google app engine and cloud run and just as a note installation on these products is all dependent on the library and again debugger can be installed like trace on non-gcp environments and is available to be installed using a variety of different languages next up is trace and trace helps you understand how long it takes your application to handle incoming requests from users and applications trace collects latency data from app engine https load balancers and applications using the trace api this is also integrated with google app engine standard and is applied automatically so you would use trace for something like a website that is taking forever to load to troubleshoot that specific issue trace can be installed on google compute engine google kubernetes engine and google app engine as well it can also be installed on non-gcp environments and it can be installed using a variety of different languages as shown here and coming up on the last tool of the bunch is profiler now profiler gathers cpu usage and memory allocation information from your applications continuously and this helps you discover patterns of resource consumption to help you better troubleshoot profiler is low profile and therefore won't take up a lot of memory or cpu on your system as well in order to use profiler an agent needs to be installed profiler can be installed on compute engine kubernetes engine and app engine as well and of course it can be installed on non-gcp environments and profiler can be installed using the following languages just go java node.js and python and so just as a note for the exam only a high level overview of these tools are needed and so this concludes this lesson on a high level overview of operation suite so you can now mark this lesson as complete and let's move on to the next one

Channel: freeCodeCamp.org

Views: 969,125

Keywords:

Id: jpno8FSqpc8

Length: 1202min 40sec (72160 seconds)

Published: Mon May 02 2022