Software Engineering is Overwhelming

what's going on guys my name is hussein and software engineering is an overwhelming field especially for newcomers to the engineering field and experienced and seasoned engineer too we get we get confused all the time because there are so many uh stuff to choose from and why that's because software engineering is a very creative field it's a very creative playground a lot of people you can build a lot of tools very quickly uh to solve your particular problem and that results in in thousands and thousands and thousands of software and thousands of solutions and thousands of architectures right that's it's unlike physical medium like i don't know art of painting there's there's a style right and and poetry there's certain style you can do but if you see this you know this is poetry and like hardware design even that is kind of limited because you're limited with with the rules of physics right you just you have this and you try to make things a little bit better right however software you can you can build anything with software right it's it's it's relatively simpler to build and as a result you have different architecture and you get a lot of choices as a result so in this video what i want to do is i want to talk through talk through three points of why software engineering is an overwhelming field and what can someone do to kind of ease up on that and the third three sections are the learning of new technologies and the second one is bug fixing some people spend days if not weeks fixing a single bug right and how to help relieve that a little bit because that could be a frustrating experience definitely for for newcomers and experienced ones but but once you get out of it you you feel really good feeling if you understand what what you were you fixed and the final thing is just design tasks and designing a system and what to do for these kind of things and how to basically design a system so it fits your use case how about we jump into it so first thing guys learning software engineering topics and concepts i said this many many times i can't say it enough if you want to learn a new technology first of all the first question you have to ask is what does this technology give you and why does it exist you have to ask these two questions and the why question is more than the what and the reason is because once you ask the why you start having certain clues and cues to why this technology exists to to reveal the problem the original problem and once you look at the problem you can start deriving different sets of first principles that you can lay on top of i'll give you an example so if you for example start learning a technology let's say kubernetes right you want to learn kubernetes and you don't know anything else you just first thing you want to learn is kubernetes you will be extremely lost why because you'll you'll stop memorizing things because what the heck is a pod why do i have a pot you don't know you just know that it's a thing and you have to memorize it and you this is how it used to work and this is how all those work and that's that that's what you're going to do as a software engineer you don't care what it does that's that's that's a false mindset to have in my opinion so you will get easily lost in these semantics but if you understood that well why does this technology exist kubernetes and can i choose something else to fix my problem right because why would you learn kubernetes to begin with there's always a reason that sometimes there is no reason it's just your your manager told you to do that and you have to learn them but that doesn't stop you from breaking down this thing into its basic atomic principles just like math do you know guys if imagine memorizing equations in math like how many possible permutation of equations are you going to memorize that's why it's not that's why a lot of people do bad in math because they don't understand what when to use when right once you you go back to the original first principle methods or or or components i guess and operations you can start proving anything that comes on top of that if you follow the same principle with with software engineering you will easily understand every single technology and why it does exist kubernetes exists because we spin up we we found a way to spin it my application in a container which is a jailed version of of an entire vm so it's a single process or set of processes that are completely isolated that's what we want that's why we wanted the application but now it became easier and easier to spin up multiple applications so it became really hard to hoard those cats right so we needed some tool to manage those containers that's why it existed but if you think about it if you don't have so many containers to begin with then why do you need this you don't right if you have a few containers yeah i can manage them myself yeah you know you come to questions like high availability what if i spin up my container on different machines you can write quick tools to do that for you kubernetes was written for for for for people who have massive and if you're going with kubernetes i don't have anything against that that's that's just a solution but just understand why you're going there just don't implement something blindly snowflake is another thing right just snowflake what is that when you hear something it just sounds nice and sexy and spicy but once you dive into it it's like okay this is nothing but a database column based storage and the the the software vendor don't make this easier for you they try to convolute this thing over cloud-based database scalable horizontally all this it's all it's a little marketing thing because they don't want to say oh we're just a column-based database that happens to be running on the cloud so once you understand the first principles he says oh that makes sense that makes up absolute sense oh you chose this side because of this yeah you have other features like security but that's that's that's that's the another thing that i i have some beef with the dock of every other software out there because if you go to the documentation every one of the software tries as much as possible to to beef up the and sound their software is the best right so you're going to start throwing some buzz confusing words just to make yourself like that's the famous thing if you can confuse them then they will take you seriously right that's that's that's what happened in the software engineering industry so we're trying to confuse everybody by by throwing words that's that's that's basically what we're doing right but once you just look through the and understand the first principles you are deadly nobody can't stop you because it's like okay this is just this basics and if someone tells you oh it's not that easy then you thought okay explain it to me what is it then if they start throwing another buzzword then they're full of simple so don't don't take things uh very seriously right try to relax when it comes to learning don't learn so many things because i can understand how frustrating that can be because yeah it could be very very frustrating to learn so many things serverless is another buzzword right i'm not minimizing the effect and and the the challenges that serverless architecture solves but just understand what does that mean it's just a marketing term right because nobody actually explains how serverless work under the hood there's a lot of work there the the beautiful design of serverless is underrated the buzz of the word is overrated hope that makes sense right the the art of adding a proxy that measures and have knowledge of whenever you call this function or or endpoint or not and how can i start a particular instance or container or a vm doesn't matter right so that i can execute your function as fast as possible that's the trick and it doesn't matter you i'm going to scale it for you if you if you're calling it three times a day i'm gonna have one instance but if the moment i start increasing the instance so so the smart logic is written in the reverse proxy but nobody talk about that right good because it's not fun to talk about this stuff but yeah there's a lot of smart logic built in the reverse proxy that accepts your request to begin with that's the clue here and then the and how if you think about it how can you build this so that it's it's low latency because that knowledge that queries because the serverless have to query and has its own database to store this knowledge to store this state that hey uh joe blow calls this function seven times a day and then now he's calling it more she is calling it even more that's another thing that's another art right so learn about these basic functionalities once you break everything into these fundamentals these fundamentals are not much they're very few things that we know very few things fundamentals once you can the trick is to break down these big components into these basic fundamentals so it becomes instantly less overwhelming once you look at the biggest compound right it's like chemicals like it's chemistry like like if i look at the iphone yeah it's a very complex device but if you break it you can essentially break it into the table of elements it is a bunch of table of elements at the end of the day right i'm oversimplified obviously but you get my idea and always ask why always always ask why am i learning this why does this technology exist why react exist why angular exists why socket io exists why websockets exist website because i consider websockets almost like a fundamental to me because it's a standard right i follow rfcs and standards that's what i like to follow standards these tools in beltrav can be broken into these standards and fundamentals and once you realize that it clicks immediately once you realize that everything can be broken into these small fundamentals and first principles you're deadly you're unstoppable because that said nothing will be scary and overwhelming anymore you can look at anything and you can challenge people who try to you because there's okay what does that mean if they if they tell you oh or just use i don't know graffana for example okay what's that it's it's a fancy war right it's getting thrown a lot it's nothing but dashboard right dashboards like okay so i'm thinking never used it by the way but i i think it's it's a web interface so that means it's so there's a web server spinning up hosting that application and that web server is communicated with the backend right so pure client server architecture simple simple simple right so so once you understand the first principle and why and why something exists you start asking more intriguing questions and you will be surprised how you can cut through anything that doesn't make any sense quickly you'll understand everything right that's why i always understand the fundamentals learning and take it easy these technologies are not going anywhere you're gonna learn everything you want just pick one thing spend a month or two or four months learning that thing and that thing only and then that learning experience is going to drive you down to the first principle fundamentals and you're going to start asking a question oh i don't understand for example i don't understand tls over tcp so i need to understand that so this is a fundamentals you have to understand it to me you have to understand tls you have to understand tcp doesn't mean you have to understand how the headers of tcp look like to me i don't care as a software engineer and network our security engineer might care and should care i don't i know how it works i know the latency i know how much i'm opening and closing tcp connection and how many trips a tls 1.3 does and how many what is zero round trip time all that stuff i need to know it in case my breaking up experience of of this big technology into this first principle become necessary and that's it so always ask why and take it easy pick one thing learn it take all the time you want get off twitter because that just complete complicated things because companies are trying to throw in more and more and more and more buzzword every single day every single day you find new buzzwords every single day and that just confuses us even more i run into this problem all the time sometimes like okay i need to take a chill all right a lot of people invent new word for no particular reasons so we try to limit this and er if you can't if it's a first principle thing that you that's never been invented before which is i really doubt it right then then learn it right example like cloudfood color built a layer 4 kernel level load balancer i never heard about this to me this is a new ground breaking thing i never heard of a layer 4 kernel level kernel level load balancer the old the layer 4 load balancers or proxies that i know about are all living in the application user space they reinvented this thing so that's a first principle to me that's a new thing that i'm going to add to my list of things that i need to know right but they change things this is this is innovative to me building on top of a different thing is obviously innovative but it's not as innovative as reinventing a new thing that didn't exist before poof circuit io is just a library on top of websocket yeah it does a great things it does retries once we understand why does it exist using web sockets raw is hard because you have to understand uh when to upgrade the web sockets whether the server supports it whether you can retry whether if the connection closed then you should re-establish the connection so circuit i is almost like statelessly built on top of warp sockets which is stateful on top of http which is stateless on top of tcp which is stateful so cycle turtle on turtles so once you understand these things you understand why it exists so only use something that you and look through it look through these learning experiences and and and cut through the things that don't make sense to you always be curious and take it easy take it easy these technologies are always there cockroachdb is going to be there tomorrow snowflake is going to be there tomorrow sella db cassandra all of this stuff it's going to be there take it easy guys take it if you're 20 years old and your software engineer who you have all the time in the world my friend yeah all the time the award just take it easy if you're 30 if you're 40 you still have all the time in the world just learn the fundamentals take it easy take a break from now for every now then right just don't don't overwhelm yourself so the second point i'm gonna talk is that and this is now you're an engineer now you are about to fix a bug that is in your system so now you i don't know you encounter the bug and here's the thing about bug fixing bug fixing guys is can be a little bit overwhelming and can be really frustrating and especially if you do not understand how technology works and it can get really really frustrating right sometimes it's all it's it's out of your hands like the compiler does really weird things in debug mode versus release unless you're a compiler expert or your compiler engineer you wouldn't get that and i wouldn't expect anybody to be everybody could be adept in compile engineering right so so that would be that would be like high level really well priced skilled high commodity engineers right there who understand how compilers work and if they want a bug oh oh i i moved to debug that means i'm single threaded i mean i was like i don't know i have my own my own buffer and then it's safer there versus release whereas the memories are located a little bit differently god knows i don't know so that's that could be really challenging so when you try to fix a bug try to first of all understand how how things work i mean if you are living at the application space most of the time these are trivial bugs most of the time the nastiest bug are with with regard to caching so if you're building a stateless application tough luck man tough luck stateless application are the hardest to debug because if you're storing a state and you and you claim that your application is stateless then debugging this thing is extremely difficult because how do you reproduce the behavior that could cause the bug because a request could go to a given container or even a vm or given physical machine and then another request could go to an and sits a state but it goes to another request and sits a different state and that causes the bug right when that state doesn't exist or even worse you're you're going to to a container that is fresh or you're going to have vm machine that is fresh and that works right but the moment you sit something you see the state and you get another request and goes to another machine that works because it's a fresh machine but then this next time the low balance of the round robin drops you into the same machine that has the state then it fails so that's a stateful behavior in a stateless application that is of obviously you added for for for performance reasons you added this cash but could real to these disastrous unpredictable behaviors that's the worst thing of the box fixing so my opinion when you when you see something like that try to mimic this by attaching the debugger into multiple into multiple processes that's not easy by any means but try that but in general most of the bugs could be fixed by if you if you spend most of the time trying to fix one bug like you spend three hours four hours trying to fix one bug and you're you're you're always circling the same thing i suggest i suggest you really consider taking a break and move to a completely different thing and then come back because you will look at the problem differently and you try to understand it from a bird eye view and that might help obviously i love to know what do you guys think junior senior engineers what do you guys do when you face a bug and that takes a long time one of the most interesting bugs that you actually faced so yeah bug fixing is definitely not fun sometimes but if you fix the bug and you absolutely know how you fixed it man that brings you up like with you come out with scratches and and battle scars that makes you as a developer and makes you as an engineer those are the best right if you spend like three days straight trying to find a bug oh my god and i i remember spending some time like that most of the time i spent like three four days to fix a bug most of them are stupid caching problems most of them are cache invalidation most of them right some of them has to do with with the way databases defer right it's like a oracle versus sql server and we're getting the behavior in oracle but not on sql server and slow performance on this and once you understand my oh my god it makes perfect sense and that's when that clicks you just my friend just learned a new experience that you're adding to your catalog as a as as a battle scored engineer the third and the final point is design activities and and this is my favorite because i do i do most of the stuff i i don't know if i mentioned that i don't code in my daily job as i don't write i don't contribute software code just to the software than in my company i test i design mostly i'm a principal engineer so i own the product and i communicate with with the stakeholders and decide what what goes into the product right so so i i design i have acceptance right here i have i i do the full designs right so when i do the full designs the first design mostly is the best design however i found out that if i sleep on it and come the next day i i have different perspective and i appreciate that initial design more sometimes it it it works sometimes i i just decide to say you know what the initial design that comes out to my mind is the is the one we're going to use however don't just do the design yourself right try to come up with something and present it to your team you have no idea how i appreciate working with the team especially when it comes to the design because every engineer gives you their own unique perspective and think and boy that is beautiful engineers will challenge you it's like okay why did you come up with this and you have to come up with an answer oh i i did this because of this i did this because of this and when you say that the x the question of why is very interesting comes back to why if you ask why and then you give an answer and then follow it up with another y you get another answer you will three y's will get you into a corner that will tell you whether you pick this because of an emotion or because of an actual valid design choice sometimes it goes back to you oh why why why why why and this is i don't know because it's a it's a best practice you start saying these kind of things oh because of practice also because it's the design pattern where's that thing can you see it it's a design pattern that means it's right right it's a design pattern must be right don't fall into that trap this is great those guys are smart as f but it does not mean they they say it themselves doesn't mean every design pattern you have to fit everything into design pattern no or architectural pattern no if you can't answer why using that design oh it's a design pattern because it's proven yeah but why would what would what value will it deliver that's the question you have to answer if you just memorize these puppies and then you're blindly using them you didn't learn anything because you're memorizing back to our mathematical concept right if you memorize mathematical equation if you memorized how look natural logarithm looks like then then you you kind of just take it for granted but if you know why those people came up with this design pattern when you built your application then you can answer this question more confidently instead of saying oh it's the design pattern you can say hey oh it's a strategy design pattern or it's observer whatever the observer design pattern and i picked it because of this it allows minimum the minimum coupling it almost removes the coupling that's why i did this okay and as a result it becomes the application becomes available so people will buy this and i have smart people in my team so they call out the bs really quick so so definitely guys think about this and take your time yeah you don't have to have an answer for everything but definitely i recommend when you design application uh try to do it like before before bed just think about it and then sleep and then your brain will start working out the problem and i i do this all the time in my sleep these design problems comes in it's like oh i should have used this instead of this it's countless time where i wake up in the morning i say oh that makes perfect sense or i remember that oh my one of my teammates will say oh use this instead of this try that what will happen sometimes i the more i listen to it it's okay mine makes sense but i have a bad feeling about this and then when you have that big feeling just sleep on it sometimes it will go away sometimes you will find what that bad feeling is and you go to it so guys to summarize this long video software engineering field is very complex and very overwhelming but it doesn't have to be if you know the basic basics of the fundamentals break down anything in life really can be breaking down into this fundamentals right once you understand the basics first principles of everything and that you take for granted obviously and then you start you start boiling down everything into these first problems and you will basically prove everything and and if you can't bring the component or the tool or the framework down to the first principle you have a misunderstanding gap that you need to fill just try to fill it try to identify it that's tough and difficult i i totally agree with that try to identify it and then try to fill that gap usually it's just a principle if you feel there is a gap sometimes it's a gap in you knowledge sometimes it's a flaw in the tool itself that nobody actually claimed and you actually called it out just called it up because hey what what does that mean why do we have this what why well it doesn't make any sense why do we have this right it's just okay oh we built this because of this because of this because once you start asking question you get to the bottom now oh that problem i don't care about that problem i never had that problem to begin with you guys had it and and you built all of this stuff because of this screen savers nobody uses screen saver anymore but why was it built it was fun to look at do you think no it was built because people with spreadsheets would leave their screens and go back to go to lunch at 12 or 11. they come back and they move their mouse and the screen is frozen why because because of the emitting of the electrons to the crt screen would just burn the screen as it is and it will corrupt it so that they invented screen savers to actually save the screen from burning and the idea is just keep something moving keep suddenly moving so so you don't just emit the same electron all this all the time on the same light i don't know how how this thing works right that's the idea so if people started using screen saver all the time without knowing why it exists i think people still do but this is not a problem anymore we fixed our monitors so they they you can leave the monitor for days running it's just the only limitation is the power and we don't have that problem right if you're plugged in for your laptop yeah we turn off the screen just to save in screen time and then power the better that's what we do so once you understand everything just it makes perfect sense so try to do that and still it's going to be it's going to continue to be overwhelming so take it easy and just try to enjoy try to find the field that you enjoy in software engineering and go deep into that whatever that is i'm gonna reference that back back-end engineering free playlist actually okay because i i made it this back-end engineering video that i discussed where i discussed the different technologies in the back end again these are first principle fundamentals right and then you can pick any one of them and then dive deep and then you can derive more principles off of those but once you understand those use this oh i i love database engineering i'm going to dive deep into that one oh i love uh communication protocols i'm going to learn all about web sockets i'm going to learn all about tcp i'm going to learn all about udp i'm going to learn all about grpc and i'm going to be the best at when to pick this versus this we need people that tells us this no one exists today that tells us to use this or this everyone just makes up stuff right but once we understand why does grpc exist why does warp socket exist why does army exist ermes is a very grpc like protocol that spotify used they built it customly and they they dumped it eventually in favor of grpc just because grp is more standardized now and popular but they built it for a reason and now they they moved for a reason that's you need to understand these these small things and obviously guys i'm going to leave the question back to you what do you guys think of being overwhelmed engineer what do you do to get less overwhelmed how do you tunnel through these difficult times when it comes to like to difficult task or difficult learning curves i'm gonna i wanna learn uh i wanna know more about that in the comment section below thank you so much for watching or listening if you're listening on this in the podcast i'm gonna see you in the next one you guys stay awesome goodbye

Channel: Hussein Nasser

Views: 15,879

Keywords: hussein nasser, backend engineering, software engineering challenges, software engineering interview, junior engineering, software engineering design interview, backend engineer job, jobs in software

Id: MbDjrztWtX4

Length: 34min 17sec (2057 seconds)

Published: Sun Oct 18 2020