Mastering AWS DevOps: Migrating SQL Data to RDS with Flyway | Step-by-Step Guide

foreign the first thing I want to say is for those of you that have your ticket in review I have provided my update so feel free to look at it and once you are familiar with since we are doing the reference architecture you can either take a mental note or use the sample reference architecture I've provided to update your reference architecture the main point of why we are creating the reference architecture is so that you get familiar with how to create it and also understand when you are creating one where to put the service and what the service are used for it ties into the knowledge of what we are building because when you are deploying these services in AWS having a reference architecture also helps you understand it better so when I was learning that was what I did for every project I did I always have a reference architecture so I would advise you that for every project you do always have a reference architecture you create for that project by understanding how to create a reference architecture and also understanding where in your reference architecture does each service belongs that helps you in the long run and it helps you understand everything you are doing this is how you can develop the talking points of things you can talk about during your interview my first interview that I ever did I used to print my reference architecture and and when they asked me tell me about your day-to-day I used to just describe whatever I did on that reference architecture but now I can talk about it without looking at it so that's the reason why we are doing the reference architecture before we start the next project so for example if I go into this ticket I have added comment of what the reference architecture should look like for the assignment that are assigned for creating the reference architecture so this is what the reference architecture should look like one of the main things I say is when you guys are creating the reference architecture take your time to create it these things are just drag and drop so the more time you spend on it the better you'll get at it so another thing to keep in mind is you don't just want to create a reference architecture just to create one you want to create one that is very clean and easy to read and you also want to make sure that you are putting the services in the right space where they belong so for example if you are creating an RDS instance you are not going to put it somewhere out here make sure that all the resources you are creating are in the right space for everyone that has their ticket in review I have provided the same comment so feel free to review it and move your ticket to done so that is the update for the reference architecture just as a reminder the program is self-paced meaning that you are going according to your own pace in your account you are going to see a list of all the projects you are supposed to complete what we are doing now is start going through the projects in your account once you complete the first project let me know I will assign the appropriate assignment for that project to you and once I assign it to you you can come to the jira board create your ticket here so you can track your progress with that work once that is done you'll move on to the next project once you complete the next project I'm going to assign it the tags to you then you come to the jira board you create your ticket and that's how we are doing it I'm going to create a Google document that has everybody's name all the projects so where you can mark the projects you have completed and you can see the ones you have left to do also the group thing is something you guys suggested and you guys have been working on some of the groups meet regularly some don't so I will leave that up to you guys but what I would do is I'll create a Google document with everyone's name on it with all the projects so if you need help with any assignment you can look on the document and see who has complete what project you can feel free to reach out to them if you have any issues while you are working on the assignment or you can reach out to me and if you guys still wants to keep the group that is up to you guys you can do that as well because as new people are joining people are living it is going to be difficult for me to be adding people to the group removing people from the group so I'll just create a space where you'll see all the groups and feel free to join any group you want also feel free to reach out to anyone you've seen on the Google document that has complete the assignments that you are working on so I think that is what I want to try for that and I'll create that document sometime today or tomorrow and I'll share it with everyone that is it for the jira board the next thing I want to bring to you guys attention is in our slack Channel if you look in our devops Channel you are going to see a thing that looks like a file up here let me close this just to show you so when you come to the devops channel you'll see this thing that looks like a file here click it here you will see important links for different things we've been doing for example our Zoom link for our meeting you'll see what we discussed last week like the process to deploy any application in the cloud here this is the access to the jira board and so on so if there's any information you are looking for I would be including those information in this space so that way you guys can access it instead of scrolling up here to look for everything so there's still more links I'm going to add here and if there's any link you want that you are looking for that I have an idea let me know I will make sure I add all the important links here so that is it for this I and the next thing I want to discuss is the next assignment for those that have completed their reference architecture this is the next application you will be working on so this is the next application you'll be working on as you can see I've already worked on it I've deployed it you see this is my domain name it is encrypted and everything is working properly so we are going to assume we are working for a company that owns a grocery store and we are deploying their website for them so this is what their website looks like as a cloud engineer it will be your job to deploy this application on AWS using the reference architecture we created this is what the application will look like when you fully deploy it all right one thing I want to mention is also the details for the application so I'm going to all right this is the detail for the application for the assignment you guys will be completing right so here you have the reference architecture the assets that I'm providing to you guys is the application code so if you click on this link it will take you to the slack Channel where I have the application code you can and download it from there for the application you will also be migrating the data for the application into your RDS database here's the link to download the SQL script that you will migrate into your ideas the requirements this goes with what we discussed last week about to deploy an application in the cloud these are the simple steps you can follow and for this requirement I have uploaded that video to YouTube I would also include the link so you have it as reference so you just understand what's going on so as we discussed last week the framework for the application is a lamp stack the hardware for this next assignment you will be completing is going to be easy to instance once you finish this one the next one is going to be Docker but the first one we'll work on is the ec2 instance the hardware is going to be easy to instance and I want you to use the Amazon Linux 2023 Ami then software installation and server requirements once you launch your ec2 instance these are the so softwares you must install on your ec2 instance first you must install Apache PHP version 8 PHP extensions and these are all the extensions your server must have you must install MySQL version 8 you must enable mode rewrite on your server and you must also increase the PHP memory to these numbers and enable mode rewrite so here you will enable the PHP core module enabled then here you will set the settings for memory limit and Max execution time for your PHP to these numbers and you will enable mode rewrite on your ec2 instance so these are your server requirements once you launch your server you want to make sure you have every one of these things on your server and remember we said how to find it is you have to launch a server probably in the public subnets do a research for example on how to install Apache on an Amazon Linux 2020 3 Ami once you find the command tested on the ec2 instance you launch in the public subnet if that command Works add it to your note then move on to the next one how do I install PHP do a research on it whatever command you find test it on an Institute instance you launched in the public subnet if it works write it down and that is how you keep going down the next one is how do I install this PHP extensions do a research find the command test it if it works add it to your notes that is how you will come up with all the commands to do all these things the only command I am open to providing to you is this command so if you've tried and you can find the command to do this let me know I'll give you that command but everything else you should be able to find the commands to do it by you doing the research then once you've installed all these things the next step is the installation of the application and the configuration so here this is where we store our application code in S3 we are using S3 to store application code then we will be migrating our SQL script into our RDS database then here you guys know we'll download the application code from S3 into our HTML directory then we set some permissions then here I'm pretty much telling you what you have to do on your server for your application to be up and running after you've copied your code into the HTML directory for example here I'm telling you you have to set the permission on the HTML directory and the storage directory to 777 do a research on how to do that and also please keep in mind that everything I'm giving you here it is the example for this assignment is in your account in the lab stack project so if you watch that project you've completed and you say you are ready to take on the assignment there are pretty much 95 of those the things in those projects are applicable to your assignment so you should be able to pick things out of that project to answer all most of these questions then here you need to open your dot EMV file and update these credentials so that your database can connect to your application and here we are going to modify our appservice provider.php file and I'm pretty much telling you where to get the code here and where to add the code once you open that file on your server so these are the things you'll be doing to complete this application and to fully deploy it on AWS so the last thing is the acceptance criteria so once you are done I want to see the URL of the site fully encrypted just like I showed you guys here paste the URL in the slack Channel and on your jira ticket if I get to it on on Time fine I will let you know but if I don't get to it on time you can delete your resources and but make sure you provide a screenshot of your application deployed with your site URL showing that is showing that it is fully encrypted another then I'm asking for from the acceptance criteria is the screenshot which I mentioned here and also a screenshot of your auto scaling group the final server that will be deployed should be in an auto scaling group so your auto scaling group should be deploying your servers again the example of this is in the lab stack project if you need example of it make sure you complete that project to understand what you are doing here and as you guys see when we did the first assignment HTML now we are doing this application there is no difference in what we are doing the only thing we have had here is the RDS database everything else is the same so you should be able to take the knowledge from completing the lampstat project and also from completing the previous assignment you did into this project it might take you some time to come up with your list of commands you have to do some testing to make sure you have the commands and that your command works when I was deploying this application for the first time it took me some time to come up with all the comments because I had to test it sometimes it worked sometimes it didn't work I just kept testing different results I was getting to find a list of the commands that work so that you will be doing I don't want you to deploy it the first time and when your first command don't work and you give up no you have to sit there and do a research and keep trying until you figure it out but if there's something absolutely that is hindering you from being able to complete it let me know I'll be able to point you in the right direction but the project I meant to mimic what your day-to-day will be like as a cloud engineer meaning that when something is assigned to you you are 100 responsible for completing it and your co-worker will be there for you to rely on but it is your responsibility to complete that project so that is why the assignment I've made them that way and I've also made them very similar to the projects you are completing right now in your account um does anyone have any questions on the assignment before I move on to the demo yes hello this site is similar to one of the projects we have in our account is it a dynamic um requirement or yes for this one yes okay so this was okay yeah so every assignment I've been giving you guys is an assignment that is relating to a project in your account so once you finish the dynamic project this would be the assignment for that project a lot of the things we did in that project 90 to 90 percent is transferable to this project the only thing we are doing different is we are using an Amazon Linux 2023 Ami and the commands you have to run on that Amazon Linux 2023 for example the repository that the Amazon Linux 2023 downloads packages from is different from the Amazon Linux too so that is why some of the command may be slightly different but the process we used in that project is the same Ubuntu no it's not Ubuntu I think that uh Amazon Linux 2023 is you are still using yum is not Ubuntu I think it's based on like Fedora if I'm not if I'm correct Linux 2 it should be sentos yeah the commands when I was testing it out and I was doing the research to find the command they we are still using yum here but the repositories where the package you may want to download is being downloaded from on Amazon Linux 2023 may be different so which is why I said test your command you cannot for some of the commands we use in the project in your account on the Linux 2 it may work or it may not work on the 2023 you would have to test it out to make sure that the command works and this app my question is on the chart okay if we consider that this project is a real world situation and the interviewer asks us questions like um how many uh what was the composition of your team to deploy this application and the delay that you people to implement it what can I say because among other people we would not be the same the the only the only person to deploy it and each person will have a role how can we situate ourselves among other people in this project okay yeah so if you get asked questions like this if you get asked a question about how what is your team composition for an assignment like this attacks like this there are two ways you can answer it if they are at work you will not be the only one doing all these things which is why we are using the jira board because you guys will be using an agile methodology where for every Sprint something in the application will be assigned to you maybe this print we want to complete the building of the VPC they may assign it to you that your tax for this print is to build the VPC they may assign other things as part of the application tags to other engineers and once the Sprint is over in the next print whatever assignment is left to complete the project they will keep assigning it until everything is completed so that is one of the ways you can answer that to deploy an application like this this is how it is at work when you work on a team an application like this would not just be assigned to just you even though just you can complete it but it will be undoed in Sprints which is similar to the jira board we are creating just imagine that in this print I work you are creating the VPC so on your jira board your ticket for create the VPC will be there and you will give your updates once you completed the Sprint ends in the next print it will move on to the next tax but since you are working on this project if you want to present this project as something you've completed it is always good when they ask me questions like that I will always tell them that yes yes I can deploy this application everything on this application end to end like create the VPC all the resources in it and deploy the application end to end so if you are being asked a question during your interview lean more towards letting them understand that you can deploy this application end to end by yourself let's say at this referring to the same question let's say we have a team composed of project manager solution architect web developer data base administrator devops engineer and maybe a security security specialist or quality insurance and then they ask you as a developer as a devops engineer what was your specific text in your tax it will be everything on this reference architecture to deploy so a security engineer all those other people that you've mentioned they would not be responsible for deploying the application your job as a devops engineer is to deploy everything you are seeing on this reference architecture but that would be your job so anything that has to do with creating resources in AWS whether using terraform or you will not use the Management console because nobody at work used to Management console deploying the application on AWS using terraform and making sure the application is available end to end that is your job as a cloud engineer the security Engineers might be there to discuss with you some of the security things you need to implement in the VPC in the AWS account to make sure that the application is compliant for example they might tell you for the S3 bucket you've created here you should encrypt it and these are the encryption mechanism you should use they may have other requirements for you so they are just there to make sure from a security standpoint the security engineer is there to make sure that the application is secure your project manager is there to manage the project but in terms of deploying the application on AWS and the services in it that is your job as a cloud engineer so basically everything you are seeing on this reference architecture is your job yeah sorry as is I think I um reviewed the comments you made on my architecture and I corrected it yeah yeah that's fine yeah so so once you review it comment it still continue to practice your reference architecture moving forward with other projects you work on for this I don't want you guys to keep going back and forth with or review it and send it to me you've done it this is the first one you've done it looks good for some of you guys you might still need practice with getting comfortable with drawing your lines or just your spacing continue to work on that and the more you work on it the better you get at it so for those that have commented on this you can move your ticket to complete and start the next assignment but I would say the assignment you must complete the previous assignment before you start the next one none of the assignments are meant to be skipped even if you feel you know the previous one I still want you to complete it so don't start the dynamic application but you still have the ticket for the other assignment on the jira board so all your ticket for the previous assignment must be fully completed before you start the next one and that is not pertaining to anyone that's just a general information for everyone all right yes subscribe yeah considering this reference architectural we can consider that here we as a devops engineer we are acting as uh on the on the development part if we have to talk about the operations part what can we say talk about the operations part in terms of how the application works yeah by building it you will understand what the services are used for how the services are connected to each other which is why we are the reference architectures assignment to do and that is what you talk about you might not be able to talk about the code for the application itself because you didn't you didn't design that so the code for this application I'm not going to be able to describe what the code here is doing so for the code for the application I won't be able to describe it you go to inspect yeah I think so the code for this application I won't be able to just describe what the code is doing but as a cloud engineer I can explain what this application looks like from end to end in terms of the services we are using and how those services are being used so that is what I will focus on as my talking point all right so does anyone else have any questions okay yeah yeah basically we can't um since these parts without doing the position right and um you have to like let you know if we [Music] actually I can't really hear you your voice is very low sometimes when you are speaking to your computer Mike is probably not picking it up very well but your voice is very low yeah so can you hear me now yes I can yeah so some of some of the projects I completed right because at the national levels it did them but I have not put them on so why have to like um meet again then let you know limit actually before I move on that's a big way for me to confirm that you've completed the project because I I don't want to make it a standard of someone saying they completed it I believe you completed it but I don't want to make that a standard so if there's any project you are working on any tax you are working on it is preferably to create your ticket on the material board so we can track it there so if I choose it put that right now so I have to like put them through everyone yes so if you start any of the assignments that has been assigned you have to create a ticket on the jira board so we can track it there that's the only way I can confirm that you complete the assignment before I move to any tax at all yes you don't start this one if you haven't completed the other one because I'll be honest with you guys if you skip the other projects and you ask me questions on this and the question I feel is something you should know I'm not going to respond so the point of the other projects is to build your knowledge up if I show you the reference architecture for that previous project you will see that it is the same thing as this there is nothing has changed we are just adding the RDS to it here so by completing that project you would have picked up some knowledge there that and I'm telling you guys this is 100 How I Learned AWS and how I got my breakthrough it's like I just kept learning little concept here and then I'll go work on the next thing I'll add a little bit stuff to it those little knowledge is what turns into a big knowledge so that is why I when it comes to the assignment I am holding you guys accountable for completing it so for example if you are working on this Dynamic application and you are asking me how to create your night Gateway I'm not going to respond because you should have picked up that Knowledge from the last project with it in terms of creating this reference architecture for this project you shouldn't be asked send me any question about a VPC because that we should have addressed in the first project if you feel you don't understand how to create your VPC then I'll say revisit the first project do the first project very well and try to understand everything in it because you only have to understand it once you guys know that majority of this project it seems like it's repetitive because things you've done before you're doing it again so I just want you guys to take your time I'm not saying I'm not going to like 100 not respond to you but I also want to hold you guys accountable for not just completing the project but picking up the knowledge you are supposed to pick up by completing that project I don't just want us to follow along and complete the project and at the end you still feel like you don't know anything so this is where you would have to struggle do all the research you can do do what you can do to make sure you understand but I am certainly here to assist you but I would be minimizing the type of assist I gave you like I said for some of these commands I am willing to provide this command to you if you can't find it and I'm also willing to provide the command to do this for you if you can't find it but if you do research the answers are online for all this I had to do a research too so everything you guys are doing now I did it this week and it actually took me a couple of days to get all my commands straight I launched several ICT instance testing my commands to make sure it works so I am also going through the same thing you guys are going through and that is exactly what your day-to-day would be like as a cloud engineer so yes make sure you complete the previous assignment before you move on to the next one okay do you guys have any other questions okay the one thing I want to present though is in this application I said that you will be migrating your data for your RDS instance into the RDS database the SQL script is provided here for every application you guys will deploy at work there is always data migration because think about it when your developer writes the code for your application they write the code there is also code for the database which is the SQL script the SQL script is there for the application right but when you guys launch a brand new RDS instance it is a brand new RDS instance AWS does not know what the SQL script for your application looks like so you would have to migrate the SQL script for your application into the RDS database this is for any new application you guys are deploying in AWS you would 95 of the time would have to migrate the data for the application so which is why I've also included that in the project for you to be able to learn how to migrate any data into an RDS database and I'll be updating the course as we proceed because with AWS releasing the ec2 instance connect endpoint which allows you to connect to any resource in your private subnet it just even makes the data migration very very easy and I want to show you guys how you can migrate your data into your RDS database using the ec2 instance connect endpoint for those of you that have completed the project you guys noticed that we were using my SQL workbench and this one just even makes this step a little bit simpler where we don't have to do the MySQL workbench and all that stuff so I want to present that to you guys as something to keep in mind when you get to this step where it says migrate your data into the RDS database you can go back to reference this demo and use it to migrate your data into your RDS database and even for the terraform project when we start the terraform project you guys will create all your resources and at the end you step to migrate your data you can use the same process to migrate your data okay so to migrate your SQL script into the RDS instance in the private subnet using the ec2 instance connect endpoint first you have to create your VPC so here we have a VPC with public and private subnet another thing we have done is we have create the net gateways in the VPC because the resources in the private subnet we want them to have access to the internet and the way the resources in the private subnets are able to have access to the internet is by using the night Gateway so as a prerequisite before you start make sure you have a VPC with public and private subnets you've set up the routing tables for your VPC and you also have not gateways in your VPC so how we are going to migrate the data into our RDS instance is we will launch an ec2 instance in our private subnet this is just an easy to instance once we launch the ec2 instance in the private subnet we will use the ec2 instance connect endpoint to SSH into that ec2 instance and once we have SSH into that easy to instance we we can now run the migration from this ec2 instance to the RDS this is the solution I came up with there might be a better one but for now this is the solution I came up with once you've set everything up it is very easy to migrate your data into the RDS database another thing we're also using here is we have to make sure we set our security group correctly between the ec2 instance connect endpoint the ec2 instance and the RDS we would also upload our SQL script into the S3 bucket so that our ic2 instance can download it and I'll show you that when we do the demo okay so to start the first thing we need to do is create an S3 bucket and upload our SQL script into it when you are working on this project as part of your reference architecture you already have an S3 bucket in your AWS account but this is a separate S3 bucket just to migrate your data into the RDS database so I'm going to come to the console and I'm going to type S3 here then I'm going to click create bucket I gave the book edit name once I give the book edit name I'm going to scroll down and create the bucket once you create the bucket go into the bucket and we are going to upload the script we would migrate into our database the script we are going to migrate into our database is just a SQL script it is this file that I'm going to provide to you and we will use FlyAway to migrate the data into the RDS database so the script we want fly away to migrate for us is this SQL script that our developer created with our application this is the script that I said the developer creates with the application that you must migrate into your RDS database once you've created the S3 bucket this is the script I'm going to select the script and drag and drop it here I've dropped the script in the S3 bucket I just created you can see it here I'm going to scroll down and click upload that's the first thing we are going to do create an S3 bucket and upload our script into it the next thing we have to do is create an IAM role to allow the ec2 instance to download the script from the S3 bucket so here we've created the S3 bucket we upload our script into it when we launch this ec2 instance this easy to instance us to be able to download the script from the S3 bucket so that is why we are creating an IEM row that will allow this to instance to download the script from the S3 pocket here you guys already have that role in your account from the previous project you did but I'm just showing you this end to end again showing you how to create the IEM rule to create an IAM rule we are going to type IEM in the search box select IAM under services we are going to click rows then click create row so on that trusted entity type it is going to be AWS service and under use case we are going to select ec2 then scroll down and click next then on this page we are going to select the policy we want to attach to the ec2 instance we want the ec2 instance to be able to download files from our S3 pocket so here we can type S3 and press enter and under your search results we can give it the Amazon S3 full access I'm going to select it by checking that box then scroll down and click next on this page we are going to give the IEM row a name I'll call it data migrate rule once you give the row a name this is the permission we added to the row scroll down and click create row and there you go it has created the IEM row to see the IEM rule I just created if I type the name call the data my grade row it is here that's the next thing we need to do then the next thing we are going to do is create the security groups remember what I said in order for you to be able to SSH into the ec2 instance you will launch in your private subnet and in order for this ec2 instance to have access to the RDS instance you have to control that traffic with Security Group which on the security group for each resource you have to allow the necessary reports that will allow you to connect to it so that is why we are creating our security group and these are the security groups we will create we will create a security group for the endpoint we will create a security group for the ec2 instance let me show you on the reference architecture we will create a security group for the endpoint the ec2 instance and the RDS so in our Management console I'm going to type VPC in the search and select VPC under services on the VPC dashboard here first I'm going to filter by my VPC make sure the VPC you are creating the security group in is the VPC you have created that has public and private subnets once you've select your VPC I'm going to scroll down here under security I'll select security groups and I'm going to click create Security Group I'll give the First Security Group a name and this is the security group we will add to the ec2 instance connect endpoint for this Security Group I'm going to call the security group eiceg then I'm going to copy the name and add it as the description on that VPC click this x to remove it and select your VPC the VPC that I created that has public and private subnet is this Dev VPC I'll selected then scroll down on that inbound rule for the ec2 instance connect endpoint we don't need to add any inbound rule according to the AWS documentation and we also saw this when we did the demo on the ec2 instance connect endpoint for the easy to instance connect endpoint what we need to add is an outbound rule since we will use this to instance connect endpoint to SSH into our ec2 instance in the private subnet then here we are going to allow SSH on the outbound on Port 22. so for the outbound select this drop down and select SSH ssh is on Port 22 and for the destination we are going to remove this 0.0 this means anyway we are going to remove this and we are going to limit the destination to our VPC cider block that is because the ec2 instance connect endpoint I should have the endpoint here I'll update the reference architecture but the ec2 instance connect endpoint is going to be in our VPC and when we use it it is going to be initiating the traffic from within our VPC that is why we have just the outbound and we want to limit the destination to our VPC cider so I'm going to open a new console here and copy my VPC slider this is the VPC I'm using the cider block is 10.0.0.016 I'm going to select it copy it and go back here and for the destination I'm going to paste it there this is exactly what you guys would do when you get to that step of migrating your data into the RDS database this is one option you can use if you like my SQL workbench you can use my SQL workbench too but just understand that with that you have to use SSH keys and all that stuff but in this project we are not using SSH Keys we'll be using the easy to instance connect endpoint to connect to our ec2 instances and resources in the private subnet so that is why using my SQL workbench would not it would work but you would have to go the route to set it up but the way we've set up this project we've set it up not to use my SQL workbench or SSH keys so for the security group for the endpoint this is all we need to add SSH Port 22 and the destination is going to be the cider block of our VPC scroll down and click create Security Group once you create that Security Group I'm going to select Security Group here again then I'll click create Security Group again the next Security Group I'm going to create will be the security group we will add to the easy training status and for this Security Group I'll call it server SG then I'm going to copy the name paste it here then for the VPC I'm going to remove that select the right VPC under inbound rule I'm going to click addro and for this security group we are going to open port 22. so here select this drop down and select SSH and for the source the source is going to be the ec2 instance connect endpoint because remember the ec2 instance connect endpoint is what we will use to SSH into our ec2 instance so we need to open the security group on this easy to instance to allow traffic from the ec2 instance connect endpoint on Port 22. that is what we are doing here so we've add SSH import 22 for this Source we will select in here and scroll down to Security Group section and look for the security group we created which we call ec2 instance which we called a iceg this is it I'm going to select it and scroll down here and click create Security Group we've created the second Security Group remember on that Security Group this is the security group we will add to the web server we open SSH Port 22 and we are allowing traffic from the endpoint Security Group then the last security group we are going to create is the security group we will add to the RDS database here I'm going to scroll up and select security groups then click create Security Group I'm going to give it a name then select it copied I'm going to paste it here make sure you put it in the right VPC then on that inbound rule we are going to add group and we are going to add rule on MySQL and Aurora so look for that which is here that is always on 3306 and the source according to our reference architecture this RDS instance will be allowing traffic on 3306 from the ec2 instance so here we have to select this drop down scroll down to the security groups section and find the server SG Security Group we created that is how you allow traffic from that easy to instance then scroll down and click create Security Group those are the security groups we need to create the next thing we are going to do is create our RDS instance so this RDS instance see it and type RDS and select RDS on the RDS dashboard click create database then here we are creating a mySQL database engine selected scroll down we are going to use the MySQL community and under the version the version for the application we need to use is version 8. so here when you select MySQL you can select whatever latest version of version H you see here the latest one I'm seeing is 8.0.33 Select that one then scroll down on that template we can use free tier and scroll down remember that when you are using free tier you won't get the option for creating a multi-az DB cluster and all that stuff we don't need it for this we just need a single database then here this is the database instance identifier so for the database instance identifier I'll call it DB and just call it update under master username I'll give the database a name then under password I'm going to create a password then I'll confirm the password then scroll down under the database instance class we can check this button to use the previous generation so that way you get the DBT to micro that is fine then storage we will leave all this option as default and connectivity we would leave all this option as default then on that VPC we'll select this drop down and select the VPC we want to create it in then subnet group it is going to create a subnet group for us that's fine then here Public Access will leave all this other option as default on that security group we are going to choose an existing Security Group remove this default one and select database SGI created then click out availability Zone we will just leave that as default and scroll down here keep scrolling down on the additional configuration here select this drop down and give your database a name once you've given your database a name scroll down and that's all we need to do and I'm just going to click create database I think I might have okay okay I guess I can have that so I'll just call it up to be like that all right so I am creating the database which is this one and when you create your database if you want to see your information it is here your username and password this is what I created I'll just copy this so I don't forget while you are creating the RDS database the next thing we are going to create is the easy to instance connect endpoint we'll create it in our VPC on my reference architecture I should have the easy to instance connect endpoint here you can create it anywhere in your private subnet it doesn't matter so we can create it here we can create it here we can create it here but I'll create it in any of these subnets in my VPC then then the last thing we'll do is launch the ec2 instance and we should be able to migrate our data I'm going to go back to the Management console and I'll type VPC again select VPC under services and on the left side under the VPC dashboard scroll down and select endpoint here then we are going to click create endpoint we'll give the endpoint in name under service category it is going to be an easy to instance connect endpoint select that then on that VPC we will select this drop down and select our VPC scroll down for the security group we are going to select the security group we created for this endpoint which is this one so make sure you check the box for the security group you created for the endpoint scroll down on that subnet we are going to select this drop down and this is how you pick which Subnet in your VPC you want to create it you want to create the endpoint whether it is any of these subnets but make sure it is your private subnet I'm going to select this drop down and I'll select my private app subnet easy one once you select your subnet click create endpoint that is all you need to do to create the end point the last thing we have to do is create this easy to instance that we will use to migrate the data into the RDS instance so yeah I'm going to go in the Management console and type ec2 on that services and select easy to then I'm going to click launch instance and launch instance again I'm going to give the instance a name I'll call it data migrate instance once you give the ec2 instance the name scroll down under Ami we are going to use Amazon Linux and to migrate our data make sure you use the Amazon Linux 2023 Ami then here under instance type it is going to be T2 micro since we are using the endpoint to connect to our ec2 instance we don't need a keypad so here I'm going to click proceed without a keeper then scroll down here under network settings I'm going to click edit select my VPC then select the subnet I want to launch the ec2 instances it doesn't matter which Subnet in your private subnet you launch the ec2 instance so I'm going to pick the private app subnet easy one then on that firewall Security Group we will choose an existing Security Group scroll down and look for the security group you created for the easy to instance I called it server SG I'm going to select it then scroll down here under Advanced details we are going to select this drop down and remember we need to add the IAM rule that will allow the ec2 instance to download the script from the S3 bucket so on that instance profile here select this drop down and you can see the DB migrate rule I created I'm going to select it and that is all I need to do here and I'm going to click launch instance I've launched the ec2 instance I'll click here to go into the ec2 instance that is all we need to do in terms of setup so I'm going to wait for the ec2 instance to run and also I will verify that my RDS instance is running yeah so this is the RDS instance I created it is available and also I would also verify that the endpoint is available so the endpoint is still pending I'll wait for the endpoint to be created once the status of the endpoint is available then we can use it to SSH into our ec2 instance does anyone have any questions while we are waiting for the endpoint to be available the question is what script is it are we going to write to migrate this to migrate the resources into the the yeah that would be the next thing that'll be the next thing we'll work on that's a good question that's the next thing we work on but while I guess I don't have to wait for this we can just start writing the script so again that's why I'm saying like when you guys are working as a cloud engineer and you are coming up with these things you'll notice everything is piece by piece even for me I just don't write the script just like that so when you guys see me give you like a full command trust me I didn't write that full command just by sitting there and typing away I had to like do it one by one and by getting each command that is what turns into a full script so we said that a good tool you can use to migrate your data into the RDS database is Flyway it is it's you that I've used that work to migrate data for many applications so yeah this is how it works we are going to type Flyway download when I type fly we download you can say download Fly Away here I'll select it and that will bring me to the documentation to download Fly Away in their previous documentation this is the command they gave you so this W get command which you just use it to download the file from the internet so that's the command I will provide to you guys to use my ec2 instance should be up and running now and let me check my end point okay my endpoint is available so what this is what I would do I'm going to open my terminal then I will SSH into my vc2 instance this is the CLI command to use the ec2 instance connect endpoint to SSH into your ec2 instance so it is AWS easy to instance connect SSH then you specify the instance ID so here I'm going to remove this and copy the ID of my new ez2 instance once I add that I'm going to press enter I'll give you some time to load the first time it loads it will actually this question type yes and press enter so I have SSH into that easy to instance so once you SSH into your ec2 instance the next thing we are going to do is download Flyway on the ec2 instance like I said when I reference their documentation this is the command you can use to download flywave version 9.20.0 so I'll give you guys this command too so I'll copy this command and it's always good to update your ec2 instance the first time you log in like as you can see now there is nothing to update everything is complete once you update your easy to instance then the next thing we need to do is download flyweight so that is that First Command uh thank you adewali I think adewali found the documentation thank you so this is the link to the documentation you guys can copy it and I also copied and paste it here you can see here to download Flyway on a Linux computer this is the command you have to run and now they are on version 9.20.1 but when I just did it a couple of days ago they were on nine points 0.2 so I'll stick with the one that I already tried that works so I'm downloading the 9.20.0 so you guys can see this command that I'm about to run I got it straight from their documentation so I paste our Command in there see it is downloading Flyway it has successfully downloaded once you download it now if you type LS you see a flyweight folder in there if you read their documentation this is the standard that fly away as specified to migrate your data it is always download the zip file unzip it but that command I just run did everything for me it downloaded the azip file unzip it that's why you are saying the unzip version here and anytime you download flyweight this is what you will see you will see Flyway iPhone the version of Flyway you downloaded it is always the same the next thing you need to do is go into the directory of the Flyway that you see remember I downloaded it I type LS now I need to go into this directory so I'm going to type CD Flyway then press enter now I am in that directory if I type LS again you can see all the Assets in this directory for the Assets in this directory you don't need to do anything here the only thing you need to do is in this SQL directory here this is where you will add the SQL script you want to migrate so remember that SQL script I uploaded into my S3 bucket Flyway wants you to add that SQL script in this SQL directory so when we run our Flyway migrate command Flyway will go into this directory find the script we want to migrate and migrate it for us so what I'm doing is when you go into this directory so for example if I'm sitting into the SQL directory we'll just see the SQL and the ls fly we always have some text file in there just telling you what to do put your SQL migration here right so this is what I just mentioned that you need to put your script in this directory so here I'm going to CD back to my previous directory so I am in my Flyway 9 Point twenty point zero right in order to run the Flyway command you must be in the Flyway directory so in this directory what I'm doing is I'm just going to delete their own SQL directory here because when you download it there's always something that comes in it so I'm just going to delete that directory and create my own so here I'm going to type RM minus RF and SQL so I'm saying I want to delete this directory you press enter now if I type LS that's SQL directory is gone then I'm going to recreate that directory by typing MK dir this is the command to make a directory which is equivalent to making a folder on your ec2 instance so I'm going to type mkdir SQL and press enter and now if I type LS you can see I've created that directory again the only difference is now if I go into that directory CD SQL I don't have anything in there that is the reason why I deleted the the one fly away as just to remove whatever they have in there here so I'm going to CD back into my Flyway directory so I know I'm doing a lot of back and forth here just to show you guys what's going on but as you are doing all these things you can type history here one should type history you can see here we downloaded Flyway then the two important thing I did is ICD into the Flyway directory so here what we can even do is copy this and in your command here paste it here I downloaded Flyway it created this directory I set it into it Ilse I don't need to I don't need that I set it there just to show you guys so these ones I don't need the main one I need is I removed the SQL directory I need that and I created another one so this is how I come up with my script by testing it one by one so so far this is my script what I've done so now that I've created my directory the next thing I need to do is copy this SQL file from my S3 bucket into the SQL directory that Flyway wants me to put my script in the command to copy a file from S3 is AWS S3 copy and if you Google it or use charge GPT you'll find this command it is AWS S3 copy then let's go to our S3 console yeah I'm going to type S3 and select S3 on that Services then I'm going to select the pocket I think I call it data migrate AOS data micro pocket right so when you come in the S3 bucket that your script is in select the script and it is the URI you see yeah so click this box to copy it and that is how we get this first part after you type S3 CP the URI of the file you want to copy and here is the destination where you want to copy it to and how you get this destination is on your ec2 instance remember let me clear my screen so you can see remember I am in the Flyway directory in the Flyway directory if you type LS you will see all the files in it and folder we said in this directory we need to put our script in the SQL directory so here if I see the into the SQL press enter now I am in the SQL directory once you are in the SQL directory if you want to get the full parts to the SQL directory type PWD that will give you the full part this is the easiest way I get it so I'll select it copy it and that is how I come up with this one so we are running an S3 CP command it would copy the file file from the S3 bucket and it would put it in this directory on the ec2 instance so I'm going to copy this and it doesn't matter what directory you run it from that command board remember when we run our fly we migrate we need to be in the flyweight directory not the SQL directory so here I'm just going to CD back into my flywheel directory if you type CD dot dot it will bring you back to the previous directory you're in now you can see I'm in my flywheat directory so I'll run that command from there then once you paste your command press enter now it has download the SQL script from my S3 bucket into this directory because that is what we specified in our Command so again if I LS in my flyweight directory this is my SQL directory if I go into it by typing CD SQL and LS you can see my script is in there once you've copied your script into the SQL directory remember so far all we did is if we review our Command we download Flyway then we CD into the folder the flyweight folder it is up to you however you want to copy your script into the SQL for me I remove whatever SQL directory they had and I created a new one then I copied my script into that directory so far that is what we've done then the next thing we'll have to do is run our flyweek migrate command and this is the Flyway migrate command I'll provide this command to you in this command the only thing we have to change here is here I'm going to add something here so you can see here we have to add the end point of our RDS instance that we want to connect to here we have to add the name of the database we want to connect to here we have to add the username of our RDS database here we have to add the password and this other two will leave them as the default so this location is where your script is located and our script is located in the SQL directory so for this I'm going to copy the endpoint of my RDS instance I'll go to the Management console I already have the RDS database open here I'll go to the RDS database this is my RDS database once you select it again once you come to the RDS dashboard select database instance select your database and here you see the endpoint selected and copy it and for this first part I'm going to replace here up to the.com paste it there then the next thing I need to enter is the name of my database so I'm going to go back to the RDS console to get the name of your database please note that it is not the instance identifier name it is the name of your database which you will find under configuration and here under DB name this is the name of the database we are going to copy it once we copy it we'll replace this and this Flyway migrate command came straight from the flyweight documentation I will include all the links for you guys so you can see where this command came from so we have we have update our RDS endpoint our database name then we need to add our database username remember I called my username this and we also need to update our password and here this is also what I called my password and and we leave this order to as default our SQL script is in the SQL directory and the command we want to run is migrate so this is how you update this Flyway command once you update it I'm going to copy it and before you run it here right now I am in this SQL directory I need to City back into the flywheel directory you must be in the Flyway iPhone whatever version you downloaded to run the fly with my grid command not in the SQL directory so please remember that once you are in the Flyway directory paste your command and run it press enter you can see it is now migrating the data and there you go it has successfully migrated the data into the database appdb and now it is on version one so when you get to this Dynamic assignment and the part where you have to migrate your data into the RDS database this is the process you use do you guys have any question on this do you guys like this process or do you prefer the MySQL workbench method we've been using before as I said I do have a question real quick yeah um when you uh downloaded and install fly away and when you install fly away and um you deleted SQL why did you delete it I deleted it because I don't want you guys to run into an issue when you are running your script let me go back here I think I cleared it yeah I remember you deleted it and then reinstalled oh yeah there's no way for us to say I can't I can't go back but I the reason why I deleted is because when you download FlyAway and you unzip it in their own directory that comes in there they have like a text file in it okay so why don't you why don't you just uh simply delete the text file rather you can you can do that too it is whatever method you you want to use you you can delete the text file yes you can run the command to delete the text file but what if they change the name of that text file in the future and you need to run this script in your CI CD maybe we can have a hide file some high five and then it will cause problem it is good to delete and then come back and create it again exactly because the reason why I deleted it is because here I'm just only showing you how to run the commands one by one to show you how I come up with the script right so that is why I was able to go into that directory and I saw the name that they are calling the sample script they put in there so let's say this command you've included it descript you've included it in your CI CD pipeline to run it for you you are not going to be there when um the script run what if Flyway in the future you guys saw last week I was just there there are documentation page was somewhere this week I went back is in another place what if fly with this SQL directory will always be there but let's say now in this SQL directory the file you said I should delete is called my SQL file right but what if in the future fly we now change it to my SQL data file then your pipeline is going to fill Because by the time you run that command it is not going to find it so without me not having to worry about whatever name they have called whatever file they've put in that directory I just delete the entire directory and recreate my home gotcha yeah I think this FlyAway evaluation method is a lot easier than the SQL workbench yeah yeah I know it seems like a lot because we had to do this setup but you guys remember the setup is just the first time the easy to instance connect endpoint once you guys create your VPC with your ec2 instance connect endpoint that's when you clean up your environment you don't have to delete those I don't think those those don't cost you money my assistant's connect endpoint is running in my VPC and I'll check my bill at the end of the month but so far I don't think they are charging me for leaving it running so you guys understand that the work is only on the setup once you've set up your security groups you're easy to instance connect endpoint and the necessary things you need to set up all you have to do is just if I wanted to do another one I only did this to show you guys how to get the commands but let's say I am migrating my data once I launch my ec2 instance I'll just add my script remember I will put beam bash up here once we put bin bash up here I'll just copy this entire script into my ec2 user data so when this ec2 instance that I'm using to migrate my data is coming up it will just do everything for me so by time this easy to instance come up my data is already migrated into the RDS I don't have to ssh in here I'm only just doing this to show you guys how we come up with the command so the process is very simple and straightforward all right that is the main thing I wanted to show you guys you are going to need it when you start working on this assignment I would add this assignment to the slack channel right after um our meeting and that is pretty much it for what I have today do you guys have any other questions before we go okay well thank you guys continue your work and if you have any questions please reach out to us I have a question yeah sure please let's see if you know this task now is with the click the what you call it the clicking clicking so if let's say someone is done with it if you want to do it with the terraform you know that yeah um how let's say on the side not as assignment since uh how um can we do that the person do it on his own yeah you can yes you can we are still going to get to terraform so again yeah yeah there's still so much we'll go we have to work on right in order for you to work in terraform you must understand what you are doing in the console right so that's why we are here now but certainly everything we are doing here can be done with terraform if you actually when I deploy this the application you guys are going to be deploying when I did this I did it with terraform I did it manually just to know that it works so if I'm asking you guys to do it I know it works because I've deployed it by clicking but I've also deployed it using terraform 100 so I don't want to start you guys on terraform without you doing it manually first and understanding every process of how to deploy this application because once you understand that it will make you deploying it entire form much easier but suddenly once we do this on the ec2 we will do it on Docker then the next one is terraform so we do have a lot of work to do so we're just scratching this office thank you sir uh I have a question sir yes sir yeah to deploy so application in a company how long it can take to deliver it at the regime for the whole team to work on it depends on the deadline I would say we are moving much at a faster Pace but at the company the deadline for the application might be two months it might be three months it might be a month but also at the company there's always there's also always moving pieces because you would have to reach out to other people for certain things that you need for example you would have to wait for the developers to provide the updated version of the code that is functioning properly for you to test you would have to first test everything in a Dev environment to make sure everything is working before you create it in prod so there's so much moving pieces going on but I'll say for an application it depends on any company and it also depends on how complex the application is but from what I've seen I can see I can say anywhere from two months to four months it just depends so we are moving much at the faster pace and you guys please note that your company's print is always in two weeks I mean my company is always in two weeks you know at every company is different so whatever they've assigned to you in your Sprint you have two weeks to complete it and they will just keep tackling it like that until the entire application is completed and they've also Factor those times two weeks Sprint into the deadline for the application when they're supposed to deliver the application so they're not going to start an application today and tell you by the end of the week it should be done it usually takes longer time than that and it's always a continuous process because there's always things they are updating in it so it's always an ongoing process but from what I've seen is like I said two months to four months it just depends please I would like you your help just after this course okay for a situation okay okay all right um we're already over time so thank you everyone can I hang on for five minutes as well yeah sure thank you Aziz thank you all right uh thank you guys have a have a good weekend no problem bye thank you yeah I'm still here but I'm just waiting for everyone to end if anyone has any questions you guys can do wait after oh okay

Channel: AOS Note

Views: 1,249

Keywords: aosnote, alx, Mastering AWS DevOps: Migrating SQL Data to RDS with Flyway | Step-by-Step Guide, aws, amazon web services, aws cloud, aws re:invent, aws tutorial, cloud computing, events, webinars, amazon cloud, aws summit, aws re:inforce, awsome day online, aws demo, aws webinar, cloud, amazon aws, database migrations, flyway, java, how to, tutorial, sql, database, flyway database migration, flyway tutorial, flywaydb, springboot, database migration, amazon, rds sql server, amazon rds

Id: fIbio8VhLj8

Length: 70min 46sec (4246 seconds)

Published: Fri Jul 21 2023