What is DevOps? REALLY understand it | DevOps vs SRE

if you know my videos you know that i cover various devops tools and concepts on this channel but to address the main question what exactly is devops and that's what we're going to focus on in this video first we'll see why is devops even needed in the application release process and what are the challenges in this process that devops solves of course we will talk about what devops concept actually is we will also see the devops as a separate role and how it evolved as well as what are the tasks and responsibilities of a devops engineer and finally we will briefly talk about sre and how sre fits in the whole devops process well devops is a relatively new concept which has been gaining a lot of popularity and taking over the traditional way of software development devops term itself is so broad and includes so many things that it became difficult to exactly define it and clearly set the boundaries of devops compared to other it fields so it's encompassing a lot of things the simplest definition is that devops is an intersection of development and operations but where do boundaries of devops start and end which part of development is not devops or what part of operations is not devops and why was there even a need for something between development and operations development and operations are two main components in the whole application release process so let's look in detail at this release process starting from the very beginning whenever we're developing an application we always have the same process of delivering that application to the end users so this is the main goal no matter if you use waterfall or agile or whatever approach at its core you create an application and you want to deliver it to your end users so that they can use it so let's say you have a great idea about a cool application you define its functionality or in other words what features it will have you coded you tested and now that you have a tested application you want to actually deploy it on a public server and let users access it for that you build and package your application in some kind of executable form so that it can run you configure the public server with all the needed stuff like installing any tools the application needs and deploy your application there you configure firewall rules to allow access to the application on the server and you have launched users can start using it so that's the simplified basis of any application release but that's not the end of the journey while in use you of course have to check in on your application is everything running fine are users experiencing any issues maybe there are bugs in the application that you didn't catch when testing also can application handle high user loads etc so after launching it you have to actually make sure that your application is accessible and usable by end users and if there are any issues for users of course you should fix them now that was the initial launch of your application but the application development is not done yet if you see users like your application you would want to make it even cooler add new features maybe optimize the performance by getting better servers or making your application faster and so on so you still have a lot of things to do and every time you improve your application either the code itself or the server configuration you want to make this improvement accessible to the end users immediately so after the initial launch you do multiple updates to your application and to keep track of these updates you version those changes there are many ways to version changes to the application one common way of versioning is with three numbers one for major changes like you replace the framework you use for coding another one for minor changes like you edit one small feature and one for quick small changes or maybe small bug fixes and you do that over and over again you have an idea of improvement you implement it in code you test it build and package it you deploy it and once released you observe it in the production to see whether there are any new improvement possibilities or any issues that need to be fixed right away so this gives you a process of continuous delivery of changes an endless cycle of improvements to your application and devops is about making this process of continuous delivery fast and with minimal errors and bugs so with devops improvements get created and delivered to users fast but also those improvements are of high quality and well tested and that is a big challenge quickly delivering high quality code now let's see what are exactly the challenges that teams may face during this process and which devops tries to solve during this whole release process we have roadblocks and frictions that slow down the process make it too much effort and allow errors to slip through all the way to production now what are the frictions and roadblocks in the release process first and the most important challenge is miscommunication and lack of collaboration between developers and operations so releasing application has two main parts you code the application you deploy and run the application developers are responsible for coding operations are responsible for running the application and between these two there might be a gap of i wrote an application but i can't run it or i'm running the application but i don't know how it works so developers would code without considering where or how the code will be deployed while operations would try to deploy without really understanding what and why they are deploying or how the application even works and this would result in miscommunications between these two developers finish coding but the deployment guide for the operations team is not good enough or well documented enough so operations team struggles deploying it so release takes longer or developers finish coding but the feature cannot be deployed because it has a lot of issues so the operations throws it back with improvement suggestions this kind of miscommunication could cause stretching the release periods for days and weeks and in complex badly maintained projects maybe even month so between the developer is done with the feature and operation starts deploying it there is no clearly defined automated process of handover it's based on a complex bureaucratic process of what checklists need to be completed and what needs to be documented and who needs to manually approve what for the release and so on so no streamlines or automated processes here apart from miscommunications between development and operations in a traditional setup where one team is only responsible for development and other team only for operations these two have seemingly different incentives that make it hard for them to work together developers want to push out new features fast that's their incentive while operations want to make sure those changes don't break anything because operations are incentivized to maintain stability in production their main focus is to make sure the application is available doesn't crash doesn't show 500 errors to the users and so on this means that operations need to resist the speed of release and check all the aspects of a new release to make sure it's 100 safe which again slows down the process especially considering that operations don't really understand the code or the application so it's even more effort for them to evaluate this new release so for example let's say developers developed a new feature which was released but this feature consumes so much resources in the production environment that the servers got overloaded and the application crashed now operations team needs to fix that so because it's the operations who needs to put out the fires when something like this goes wrong developers may not be as careful as operations about the changes they release and again focus on releasing new features as fast as possible without really thinking so much about stability so even though the main common goal of everyone in a company should be to deliver high quality applications to the end users fast in practice the more immediate goals are for each role to do their job and developers job is to quickly create new features and push them out operations job is to maintain the system stability and resist new changes being pushed out and this gives us a conflict of interest so this kind of setup naturally makes it difficult for those two to collaborate another showstopper when releasing a feature is security just like operations team carefully evaluates and changes to make sure they won't affect system stability security team will evaluate any changes to make sure they don't affect systems security and in a traditional setup this is the same manual bureaucratic process as operations which takes days or weeks and slows down the release process and as i mentioned devops is about removing any roadblocks that slow down the process so it includes this one as well however even though this is part of the devops solution there was a separate term created for that called devsecops in order to highlight and just remind the teams of the importance of security because it somehow got left out i actually have a separate dedicated video on devsecops which you can also check out if you're interested now adding to the list of show stoppers is application testing in many projects there are separate teams or roles for testers who test the application changes on different levels like testing just the feature testing the whole application testing on multiple environments etc and often these tests are done manually when teams cannot totally rely only on their automated tests and only after manual testing is done can the change be released and even though this may not be done by development or operations role but rather a separate tester role this is an important part of the release process and may also slow it down considerably as i mentioned many of the tasks during the release process like testing security checks deployment etc used to be done manually for example operations would do most of the operations tasks manually either by directly executing commands on the servers to install tools configure stuff do patches or have scripts or small programs they execute but in both cases this is manual work so application release tasks are not automated so we're manually deploying the application manually preparing the deployment environment creating infrastructure configuring servers etc manually configuring jenkins jobs for example manually configuring access to the servers to jenkins build to all the tools etc and this manual work is slow and more error prone because of human error plus with manual work you have a disadvantage that knowledge sharing is very difficult because people who do the tasks would have to document it and others would have to read it it's also very intransparent because it's hard to trace who executed what when and finally when infrastructure configuration and so on was done manually if something happens to the infrastructure it may be really hard to recover and replicate the exact state fast you would have to remember exactly what was done on the servers in which order to get to that previous infrastructure state so you see the main characteristic of all these issues is that they all slow down the release cycle and create roadblocks on the way and you also see in case of security and testing that devops may even go over only development or only operations responsibilities and tasks and that's why to understand devops instead of focusing on the name and what it means we're focusing on what it tries to achieve devops tries to remove all these roadblocks and things that slow down the release process whatever that may be and instead of manual inefficient processes helps create fully automated streamlined processes for release cycles and this can be done step by step removing one roadblock at a time until you have a fully optimized and automated device process that makes your application release super easy there are many companies who have optimized the process to the level that they can release multiple times a day of course not every project needs multiple releases a day but having this kind of streamlined release process is obviously beneficial for everyone so how does devops help achieve this and solve all these challenges well by the official definition and this was the original idea of devops devops defines a combination of cultural philosophies practices and tools for doing that so devops is not just one set of tools or one specific concept it's a combination of anything that creates the process of releasing the software fast and with high quality and the main part of the concept was that developers and operations people should work together more often talk to each other more often and collaborate better to achieve that but actually this definition is too broad and too high level and makes it hard to imagine how it works in practice so it's just not specific enough so naturally different companies implemented devops in different ways so the actual implementation of devops looked pretty different from company to company but since companies started adopting it gradually it got a more concrete form with some of the common patterns across many companies and one of these patterns was that devops evolved into an actual role called a devops engineer where either developers are doing devops as a job next to development or operations are doing it or someone is doing devops exclusively as their only job and a set of technologies that were used to implement the devops principles became devops technologies which now devops engineers would need to learn and i understand that many people are resisting the idea of devops engineer and the creators of the devops concept didn't see it used this way but the reality is often different from the theory we see that concept was adjusted and bent to meet the needs of the end goal and devops engineer role is what came out of it and that devops role is responsible for creating a streamlined release process without any roadblocks slowing down the release and that's why in the center of devops is the well-known continuous integration continuous delivery process so let's see exactly what makes up a fully streamlined cicd pipeline and generally what are the tools and concepts you need to learn as a devops engineer what tasks and responsibilities it has as well as where is the line and boundaries of devops from development and from operations it all starts with the application developers team will program an application with any technology stack different programming languages build tools etc and they will of course have a code repository to work on the code in a team one of the most popular ones today is git now you as a devops engineer will not be programming the application but you need to understand the concepts of how developers work which git workflow they are using also how the application is configured to talk to other services or databases as well as concepts of automated testing and so on now that application needs to be deployed on a server so that eventually users can access it right that's why we're developing it so we need some kind of an infrastructure on-premise servers or cloud servers and these servers need to be created and configured to run our application again you as a devops engineer may be responsible for preparing the infrastructure to run the application and since most of the servers where applications are running are linux servers you need knowledge of linux and you need to be comfortable using command line interface because you will be doing most of the stuff on the server using command line interface so knowing basic linux commands installing different tools and software on servers understanding linux file system basics of how to administer a server how to ssh into the server and so on you also need to know basics of networking and security for example to configure firewalls to secure the application but also open some ports to make application accessible from outside as well as understand how ip addresses ports and dns works however to draw a line here between it operations and devops you don't have to have advanced super operating system or networking and security skills and be able to administer the servers from start to finish there are own professions like network and system administrators security engineers and so on that really specialize in one of these areas so your job is to understand the concepts and know all this to the extent that you're able to prepare the server to run your application but not to completely take over managing the servers and whole infrastructure nowadays as containers have become the new standard you will probably be running your application as containers on a server this means you need to generally understand concepts of virtualization and containers and also be able to manage containerized applications on a server one of the most popular container technologies today is docker so you definitely need to learn it great so now we have developers who are creating new features and bug fixes on one side and we have infrastructure or servers which are managed and configured to run this application the question now is how to get these features and bug fixes from development team to the servers to make it available to the end users so how do we release the new application versions basically and that's where the main tasks and responsibilities of devops comes in with devops the question is not just how we do this in any possible way but how we do this continuously and in an efficient fast and automated way so first of all when the feature or bug fix is done we need to run the tests and package the application as an artifact like jar file or zip etc so that we can deploy it that's where build tools and package manager tools come in some of the examples are maven and gradle for java applications for example npm for javascript applications and so on so you need to understand how this process of packaging testing applications work as i mentioned containers are being adopted by more and more companies as a new standard so you will probably be building docker images from your application as a next step this image must be saved somewhere right in an image repository so docker artifact repository on nexus or docker hub etc will be used here so you need to understand how to create and manage artifact repositories as well and of course you don't want to do any of this manually instead you want one pipeline that does all of these in sequential steps so you need build automation and one of the most popular build automation tools is jenkins of course you need to connect this pipeline with git repository to get the code so this is part of continuous integration process where code changes from the code repository get continuously tested and you want to deploy that new feature or bug fix to the server after it's tested built and packaged which is part of continuous deployment process where code changes get deployed continuously on a deployment server and there could be some additional steps in this pipeline like sending notification to team about the pipeline state or handling failed deployment etc but this flow represents the core of the cicd pipeline and the cicd pipeline happens to be at the heart of the devops tasks and responsibilities so as a devops engineer you should be able to configure the complete ci cd pipeline for your application and that pipeline should be continuous that's why the unofficial logo of devops is an infinite cycle because the application improvement is infinite new features and bug fixes get added all the time that need to be deployed now let's go back to the infrastructure where our application is running nowadays many companies are using virtual infrastructure on the cloud instead of creating and managing their own physical infrastructure these are infrastructure as a service platforms like aws google cloud azure linux etc one obvious reason for that is to save costs of setting up your own infrastructure but these platforms also manage a lot of stuff for you making it much easier to manage your infrastructure there so for example using a ui you can create your network configure firewalls route tables and all parts of your infrastructure through services and features that these platforms provide however many of these features and services are platform specific so you need to learn them to manage infrastructure there so if your applications will run on aws you need to learn the aws and its services now aws is pretty complex but again you don't have to learn all the services that it offers you just need to know those concepts and services that you need to deploy and run your specific application on the aws infrastructure now our application will run as a container right because we're building docker images and containers need to be managed for smaller applications docker compose is enough to manage them but if you have a lot more containers like in case of big micro services you need a more powerful container orchestration tool to do the job most popular of which is kubernetes so you need to understand how kubernetes works and be able to administer and manage the cluster as well as deploy applications in it now kubernetes is a powerful but also a very complex tool so it's usually a lot of effort to set up and manage multiple kubernetes clusters for different teams in a company so before moving on i want to give a shout out to our sponsor loft which is a platform that helps you build self-service kubernetes clusters easily platform teams can deploy loft connect clusters and then let engineers create isolated development in ci city environments on demand whenever they need them so it puts developers in charge and gives them direct self-service access to kubernetes one of the great features and benefits of loft is that that loft can save you more than 70 percent of the cloud costs by automatically putting virtual clusters to sleep when nobody is using them and automatically waking them up again once engineers interact with it if you want to learn more about how loft works i actually did a separate video on it for the devops tool of the month series now love has a lot of other great use cases for working with kubernetes so if you want to try it out yourself for my followers loft actually provides six months free for their paid subscription for the first 500 people so check out my special link and use my promo code for that now back to our devops roadmap now when you have all these maybe thousands of containers running in kubernetes on hundreds of servers how do you track performance of your individual applications or whether everything runs successfully whether your infrastructure has any problems and what's more important how do you know in real time if your users are experiencing any problems one of your responsibilities as a devops engineer may be to set up monitoring for your running application the underlying kubernetes cluster and the servers on which the cluster is running so you need to know a monitoring tool like prometheus or nagios etc now let's say this is our production environment well in your project you will of course need development and testing or staging environments as well to properly test your application before deploying it to the production so you need that same deployment environment multiple times creating and maintaining that infrastructure for one environment already takes a lot of time and is very error prone so we don't want to do it manually three times as i said before we want to automate as much as possible so how do we automate this process creating the infrastructure as well as configuring it to run your application and then deploying your application on that configured infrastructure can be done using a combination of two types of infrastructure as code tools infrastructure provisioning tool like terraform for example and configuration management tool like insibol or chef etc so you as a devops engineer should know one of these tools to make your own work more efficient as well as make your environments more transparent so you know exactly in which state it is and easy to replicate and easy to recover in addition since you are closely working with developers and system administrators to also automate some of the tasks for them you would most probably need to write scripts maybe small applications to automate tasks like doing backups system monitoring tasks cron jobs network management and so on in order to be able to do that you need to know a scripting language this could be an operating system specific scripting language like bash or powershell or what's even more demanded a more powerful and flexible language like python ruby or golink which are also operating system independent again here you just need to learn one of these languages and python without a doubt is the most popular and demanded one in today's devops space easy to learn easy to read and very flexible python has libraries for most of the databases operating system tasks as well as for different cloud platforms now with this automation tools and languages you write all of these automation logic as code like creating managing configuring infrastructure that's why the name infrastructure is code now how do you manage your code just like the application code you manage this also using version control like git so as a devops engineer you also need to learn git so at this point you may be thinking how many of these tools do i need to learn do i need to learn multiple tools in each category also which ones should i learn because there are so many of them well you should learn one tool in each category one that's the most popular and most widely used because once you understand the concepts well building on that knowledge and using an alternative tool will be much easier if for example you need to use another tool in your company or project now you may be thinking these are a lot of things to learn and it may be hard to know where to start or what to learn first or what resources to use and so on well there are many resources out there to learn individual devops technologies but ideally you want to follow a well-structured step-by-step roadmap and more importantly learn how to use these technologies together in combination because that's what devops engineers do they use and integrate multiple technologies together to create devops processes and of course you want to learn all these with actual real-life project examples to know how it will look like in a real job and very few courses and learning resources offer this and that's exactly why we created a complete devops bootcamp with a clear structure and lots of hands-on projects so if you're thinking about becoming a devops engineer or slowly transitioning into devops you should definitely check out our devops bootcamp in the video description to get a full picture of devops i want to mention one more concept which is sre or site reliability engineering and how it fits into devops in this video we learned that there are two definitions of devops the original definition which is more high level and more broad and doesn't specify how exactly devops should be implemented and a more practical one which evolved over time with its own devops engineer role which is what you learned in our devops bootcamp so when we compare devops with sre it's important to know which definition of devops we're using for this comparison so with the first broader definition of devops devops is a more high-level concept that defines what needs to be done to achieve the automated streamlined release process while sre is more specific about how to exactly implement this process and how to implement devops principles so many people would say that sre is a specific implementation of the devops concepts but as we saw devops itself also became more practical with its own role and specific technologies and ways to implement it so what's the comparison here well in many companies devops implementation this practical devops implementation became more focused and concentrated on the speed of delivery for the application changes and of course even though it's part of the devil's principles to not only release fast but release quality code many devops teams in practice again seemed to optimize more for the speed than reliability so as a great complementary part of devops sre emerged with the same principles and goals in mind which is release quality code fast but as the name suggests more focused on reliability and keeping systems stable while allowing for fast changes so sre is its own role with its own set of tools for making systems reliable so these two were kind of parallel developments and are now often seen as two sides of the same coin and it's not uncommon for teams to have both devops engineer and sre helping implement the devops principles so this was just a short look at sre to understand it in comparison to devops but since i have received many questions about what sre is i will release its own follow-up video on sre in the next weeks to explain in more detail like how sre works in practice what are the tasks and responsibilities of a site reliability engineer and so on so be sure to subscribe to my channel and activate the notification bell to be notified when i release the video and for devops i hope i could clarify all your questions about it if not leave a comment with a question and i will try to answer them with that thank you for watching and see you in the next video

Channel: TechWorld with Nana

Views: 369,567

Keywords: devops, what is devops, devops vs sre, sre vs devops, sre vs devops what's the differen, devops engineer, devops roadmap, techworld with nana, what devops engineer do, devops explained, what devops means, what devops, devops concepts, devops tutorial, devops tools, devops for beginners, devops explained for dummies, why devops, why devops is needed, why devops is important, how devops works, devops meaning, devops in practice, what devops is, devops tutorial for beginners

Id: 0yWAtQ6wYNM

Length: 35min 32sec (2132 seconds)

Published: Tue Jan 18 2022