What does larger scale software development look like?

okay so this video is going to be a little bit different um I just want to give more insight into like what enterprise software development kind of looks like and I'm going to walk you through how like a system and a team of like 10 developers all collaborate together basically code integrate build test and deploy their applications and like the environments that what they might look like okay so again the scope of what I'm going to be saying is like a team of 10 developers because that's kind of what I have most experience with but again there's there are teams and projects that have like a lot more developers than that and there's a lot of projects that have one or two developers so really I want to kind of give you insight into like what I've typically seen in the industry so the first thing I want to do is we have developers okay so I'm just going to make a bunch of different developers all right so we got like nine developers over there that's probably good enough and then also every team depending if you're using agile you might have like a scrum master so I'll say like a scrum Master here and then you might also have like a project manager um and then you know some teams will have a designer or one or two ux designers our team actually has a ux UI I think we have like two members okay so this is like your core team and then typically there's a client and this could be one or could be multiple people but your job is to basically build software for this client right the client has requests they have things that they need built and those clients typically what I've seen is that they the client has to basically be the buffer between your users and typically what you're doing is you are trying to satisfy a customer or a client sometimes you might have multiple of these people sometimes you have one person who's like kind of leading the project and you're trying to basically daily have stand-ups and kind of demo your work to that client and get feedback from the client at the same time the client is the one with the vision of what they want built so they're going to kind of give you some requirements and tell you hey I need you know stuff to do this and that now a lot of things that I've read about is that like your Dev team here should be kind of like directly integrating and talking to users but most of the time there's like this middle man the client who's basically doing that talking for you right from what I've seen okay every team is different okay so like the users will basically use the system the client is going to give you you know stories and tasks that you have to work on and whatnot and then your team is supposed to take down the needs of the clients and needs of the users find the best solution and Implement that now there's a little bit more Nuance of this like technically the UI ux people would be like you know doing user research talking to users you know Direct talking directly to the clients the devs would be demoing to users and clients the project manager is basically working directly with the client in the devs and the ux like make sure everything is going smoothly that we're hitting deliverables on time that the roadmap is good and then the scrum Master if you're doing agile is just trying to make sure that all the meetings you do are trying to follow like the you know the scrum guides so that's kind of like the overall like what I've seen feel free to leave a comment if you've seen some other but that's generally what happens right whether you have one or nine devs like you're going to be talking to someone who's basically dictating what you got to build whether that's your boss um whether that's your client if you're working as like a consultant all right so ultimately the goal is these clients and these users are going to ask you to do something so like if I were to go ahead and right they're gonna they're gonna write down some things that they want and you got to figure out how to basically ship stuff to them so that your users can actually interact with this stuff again like I said this is going to be all over the place so don't judge me too hard about my lack of planning so your goal as a developer is ship features so your users will use the feature that you're shipping so your clients and your users basically kind of dictate what you're building your Dev team and your uiu extras and everything to get together and try to ship these features to production your users will use them and you will continue this feedback loop over and over again until you get a well-polished system and you kind of keep on going back and forth of like making new features improving old features fixing bugs and stuff like that okay so that's like the the whole project management life cycle kind of I'm glossing over the terms now the next step is how do we take this requirement here and how do we convert it into a Deployable package because there is a lot that goes on under the hood with how this works so typically I'm gonna move down here you have some type of Version Control right so I'm gonna go ahead and just put like git and the developers are going to get a story or a task or whatever you want to call it what we kind of tried to do on our team is we have the devs basically break off into smaller groups we do a lot of Ensemble programming or pair programming or mob programming whatever you want to call it where we'll grab like two or three developers and we'll try to work on a story and as we're working on the story what we do is we will check out a feature Branch okay so if you're familiar with Git this is a Version Control software more specifically we use GitHub but using git you can basically have your master branch or I guess we call it main these days the main branch is basically what should have your production code right but what we do is we'll check off a feature from Main onto a we'll call a feature Branch I guess feature branch and this group of developers will basically continuously every day they will add features they will push that feature Branch they will try to test out that code and then once they feel like that code is good they will create something called a pull request okay let's go ahead and do a pull request to try to get that feature merged into main so at this point you have three devs who have basically worked every day together we're in a zoom we're pairing together we're sharing screens we're writing the code trying to implement test cases make sure everything works and then we make a pull request and at this point other members of the team will go and try to review that pull request you want to try to do as much knowledge sharing as possible so the more people you have kind of understanding what's changing the system and how the code is changing the better it is right because you know people leave companies people come and go so the more knowledge that you share amongst all your developers the better your team health is going to be but anyway we have uh constraints where like we try to get like one or two devs to review this pull request and once it's good we merge it in the main and that kicks off a chain reaction in the CI CD pipeline we use GitHub actions that are work but people use Circle CI people might use Jenkins but basically this kicks off a trigger I can go like this and I'll say like let's just say get up actions where we basically build test and then we deploy the code okay and that is basically the the whole like deliverable right this thing once it's built and everything's tested and deployed okay that's where we get this little shippable feature that makes sense so we work on the work we merge in the main that kicks off a thing to basically build everything tested deploy it and that gets into the hands of users where they can play around with it this is if you're doing like trunk based development off of Main but often with Enterprise software you can't just start merging features into main unless you have a very very very robust testing Suite often most teams I've seen you want to have QA Engineers kind of verify stuff you want to have the product manager be able to click through a test environment or a QA environment to make sure everything's good and the way that you can achieve that is by using different branches so in our case we have various branches we have like a test Branch we'll have like a Dev Branch now instead of just checking off a main every time you can check off of a Deb Branch okay and this Deb Branch basically you're going to work off of the Dead Branch you'll do some features and then you are going to do a pull request to your Dev Branch when everything is good like that okay and the dev branch is mainly for developers right it's a it's a environment and a branch that basically has your latest development code and you want developers to basically integrate their features together as much as possible okay every day you guys should be pushing different changes over the story that you're working into some type of centralized location like integrate as often as possible is the idea okay so you might say okay well you have this dead Branch what is the what is the point of this okay again you have this entire workflow with GitHub actions or whatever CI CD pipeline where you have a build deploy and test step when code is merged into debt and what this does is this is actually going to deploy out an entire environment so in our case we use AWS for everything but inside of AWS when this thing is deploying it's going to be running a bunch of different scripts we use terraform as our infrastructure as code tool but there's like AWS cdk there's like the serverless framework you can just do a bunch of awsc like commands if you want um but in our project everything is automated so when it gets this deploy step it basically runs one command and that goes into Amazon and it starts spinning up tons of different Services right we get like serverless lambdas like we'll get Lambda spun up we get open searches open search clusters spun up um Dynamo a lot of friends for the CDN I probably could have used real icons for this instead of just boxes but hopefully you get the idea okay so again like the purpose of this is that we have a Dev environment and now all these developers can basically try to play around with they can click through this is a live deployed environment with fake data this doesn't have like real production data but it's everything is fully deployed and integrated together so that you're confident that when developers are working on stuff together that like nothing is breaking okay because when you start getting to like Cloud deployments everything might work locally in your machine but the moment you start deploying stuff if you have like one configuration that's off it just starts breaking okay so we do this daily you integrate and push stuff to Dev daily you're deploying and updating your infrastructure on a Dev environment so I'll say AWS Dev and once everything is good at some point you decide I'm going to cut a release right so you're basically going to take your branch and you're going to make a pull request merge your changes to a higher tier environment now in our case you'd merge it to like test or QA or staging a lot of people say like staging um it doesn't really matter like this convention is a little bit different for every team but basically once you have like a a nice set of features that you want to get in front of users you cut a release you make a pull request just like we did before people might review it and then you get that merge into a higher environment now what happens when you merge all these recent Dev changes into another environment well it kicks off yet again another GitHub action pipeline which is going to spin up an entire environment I'll put staging it's probably too small to read but now we've got a staging environment so we have basically the exact same infrastructure completely replicated in a higher tier environment and the main difference between this environment and the other environment is that staging you may have pre-production data pre-prod data right so this might have um more realistic data sets okay so if Deb was just like you know just enough to test to make sure stuff is working you can click through this has a bunch more data so you're verifying against like a larger data set but basically when you deploy these two environments you can actually ask your clients or your users or you have like a set of test users to go and just poke around in this environment play with the feature get some feedback the best way is just pull them into a zoom and like show them the feature you're working on and get feedback then let them control your screen and click around with the feature but depending on the way your team works you might have access to users that easily so the next best step is you give them access to like a deployed environment where they can go and play around with the test user now some other things you can do on your staging environment is that you might have a bunch of like load tests okay you might want to run a bunch of load tests on this real production like data make sure that hey when everything is deployed and we get a lot of traffic does this stuff actually hold up okay so you might do some load tests you might do some like smoke tests I'll say load tests and sometimes this will happen inside of the deployment pipeline so like you'll do a build test deploy and then you'll do smoke tests and load test all in an automated fashion right and at that point you look at staging you know your product owner your clients or your users will come in they'll start playing around features and if everything is good you'll do this process again to basically get those features merged to like a main branch you guys super confused yet which again when that pull request has finally merged in that's going to kick off a deploy and basically you're going to have another identical environment that has all the same resources that you had in staging and Dev and that's going to get updated and deployed and your users are going to actually use that deployed environment over here okay so this is one of the main benefits of like basically using something like terraform or infrastructure as code because all of these resources like you could have hundreds of these resources you could have like sqs cues SNS topics um SCS configurations you never want your developers actually going into your AWS dashboard and like clicking things by hand because you can see how quickly that if you were to go and manually update something here you're going to forget to deploy it to prod right so you want to use infrastructure as code tools so that when you do a deployment it basically just takes a snapshot of the current configuration of your environments and just deploys it out to whatever environment you're trying to go to and another beauty of infrastructure as code is that you can spin up as many environments as you want right so we have actually a bunch of different experimental environments where I work where developers can go and you know push their changes to some experimental Branch just to verify that when stuff is integrated together everything works fine so that's kind of the flow that I've seen during my career we've done this a lot it works pretty well but this again this is like a smaller scale project right this is like 10 Developers now what happens is when you get into a larger team and a larger project let's say you have 50 developers this whole process is literally duplicated for different teams so you might be doing this like every team might have their own set of clients their own users their own project that they're working on and the only difference is and they might all have their own unique way of basically getting their code deployed out some teams might be more efficient than others and when you're at an Enterprise setting you know there's higher up managers that are kind of like overseeing all these different teams and trying to like coordinate and get all this stuff working together because these prod environments here I know it's super small to see but these things are all integrated together in different ways right you might have microservices you might have different environments needing to access data and from different databases and stuff like that so you have these three different prod environments in this example and you have managers on top trying to figure out okay how is this team going to be able to deploy a change and hopefully allow this team to be successful and stuff like that and it just keeps scaling up and up the bigger the project incorporation that you're at the more these things just keep breaking out into you know more teams and more stuff it gets crazy right it gets crazy so depending on when your project you might just be on like a one-man team and you just push the main every single every single day and then that kicks off of her cell deployment and then your users see that when you get to an Enterprise setting it it doesn't work as beautifully as that it's not as simple as that let me undo most of that stuff um I'm gonna go back to like the basic view here where we have one one team and again there's like nuances to all this like this whole this whole cutter release from Dev the staging to main it has issues as well because you can have unfinished work getting all the way up to production and you can apply things like feature Flags um in wrap code to make sure that code never runs in production until a certain lag is turned on right you might have like a database over here uh you might have like a feature flag database over here where you have like um you know your users or your clients can go and update different booleans and when they update a certain feature flag it turns a feature on for all the users that's one scenario you can do to basically allow you to kind of do this workflow a lot of teams just work directly off of Main and if your cicd pipeline and you have tons of tests you have a lot of confidence that like when your your builds pass you know that everything will be good so there's two other things I kind of want to want to hit on right there's how do you handle bugs and actually zooming in a little bit this is actually more complicated because if you were to look at the infrastructure of your deployments of your prod environment often at Enterprise you have requirements saying that you need to have like five nines uptime right so you can only be down for like an hour out of the whole year you can only be down for like 30 minutes out of the whole year some projects have higher requirements and depending on the requirements you may have to set up basically a multi-region deployment where you have one entire infrastructure set up in east coast of AWS so like we'll say this is East and then you might have in another infrastructure set up in the west coast so that if for whatever reason the East Coast goes down which it does once in a while sometimes all your users are basically redirected to the West Coast that's called like a active passive I believe um setup but there's also active active where you have east coast users always hitting the East Coast and West Coast users always hitting the West Coast so they have the lowest latency and you'll have like your data replicated between your databases in between these things and if you have like more different Services you know there's like replication going on between those just so that if whatever reason if something were to go down there is ways for your system to be resilient and not completely fall over for different users right so for example if let's say open search inside this box were to go down right all the users on the East Coast your whole system will basically start failing right so you have to have ways internally to basically say okay well the East elastic search is down let's just route to the West Coast and grab that and of course there's always like rules with like data and like where can data live are you allowed to pass data around a lot of country laws like prevent you from having user data sent over to various regions so if you're working on an Enterprise setting and you have a bunch of user data if you were to try to replicate your data to like Europe or vice versa if you were to try to replicate from Europe to us East one you could get in a lot of trouble A lot of times we hear about like Edge Computing and multi-region and stuff like that it's not as simple as just flipping a switch because there are laws that you have to be in place to make sure that user data is not replicated so I just want to touch on that because even inside these little boxes it gets way more complicated with how everything is set up and interacting with each other on AWS with multi-region okay and then the second thing I want to talk about I don't know if I have more stuff I want to talk about the second thing is how do you handle bugs okay when a user is kind of working on a feature let's say you just deployed a feature and you start getting a ton of Errors okay how do you even know that there's errors well the first line of fence is the user will contact you and say hey you know I'm getting I'm getting tons of errors and you'll get a bunch of like incident reports from your client or from your users saying hey there's like an issue with this new feature we deployed your first line of defense if you're using feature Flags is you just go into this database and you turn the feature off just turn it off so that no one is using that feature second line of fancy you could do a rollback deployment so all the changes you just made to this environment find a way to roll it back and sometimes it's usually just a matter of like rolling back an individual change here like maybe you just change some API code so all you need to do is roll back your Lambda from one version to the previous version but as far as like the the code for fixing the bug sometimes you can't roll back sometimes you can't just turn the feature flag off you have to do something called like a hotfix right so what we do or what I've seen done in the industry is instead of doing this whole flow where you work on dev staging in Main you could potentially just check off a branch directly off of Main okay so like you'll have like a set of developers basically check off a branch here they'll do their Quick Fix and then they will try to get that redeployed out to main as fast as possible depending on how risk adverse your team is maybe you'll do it the staging maybe you'll do it to some pre-prodden environment first that has to you know get checked off by some higher ups to say okay we're good to go and then you do that whole process again but now that you've put code inside of these branches and it doesn't exist in this Branch what you have to do now is you have to back merge your changes from Main all the way down to your Dev environment or back merch from staging to your Dev environment so that this is up to date with the latest changes that was just hot fixed on Main okay and again this is it just gets complicated right Enterprise development gets complicated which is why like a lot of stuff I do on my channel I don't want to get into this stuff I just want to be able to push the main have it build have it deploy and my users can use it right then and there and I would say like smaller teams and startups probably don't do all that complexity they probably just you know work directly off of Main and then they get that auto built through I don't know or sell or something like that that gets deployed out if you're using next and that's also one of the beauties of using something like nexjs and versel is that if you can avoid deferring all this complexity do so and then only start adding in complexity as you find it needed unless you have like explicit requirements that you have to do all this stuff and it actually improves your team and your process and your quality um then I would say avoid it like if you ever watch Theo he talks about basically instead of doing all the work I just described just get good at fixing bugs right when you deploy a new feature if a user says that something's broken get good at deploying a fix within 30 minutes and which basically allows you to avoid all that complexity it's not necessarily unnecessary complexity it's complexity for a particular reason if you're working on a high stakes system you can't risk shipping a bug if you ship a bug and you break I don't know some major piece of functionality for millions of users that's no good right that's when you start having all these other like workflows come in because you're trying to avoid breaking production as much as possible you start adding tons of unit tests tons of integration tests tons of end-to-end tests you start doing these multi-tiered environments because you want to make sure that when you go live with this new change it doesn't just start blowing up for everybody because that could be very very bad for your users maybe you're working on actually like Health Care Systems or something and if you have three days of downtime because you broke people's insurance or you you know you broke some type of software that's used for monitoring People's Health that could have huge impacts on people's lives right so again just keep in mind that whatever your project you're working on if you're working on a project that does something basic like this you're either not that high risk of a project or you guys just don't care and you guys can fix bugs really fast when they come anyway I've been talking for like 30 something minutes so I'm going to wrap this up I hope you guys enjoyed this random video sporadic video but I think it was important to share because most people that you watch on YouTube and read on Twitter I feel like they're all just startup companies or Indie hackers and you don't really get good insight as to like how the large-scale Enterprise systems actually work under the hood cool have a good day happy coding

Channel: Web Dev Cody

Views: 1,341,135

Keywords: web development, programming, coding, code, learn to code, tutorial, software engineering

Id: Dl-BdxNRUqs

Length: 24min 14sec (1454 seconds)

Published: Mon Jul 10 2023