How Important are Algorithms and Data Structures in Backend Engineering?

what is going on guys my name is asain and this video I want to talk about our algorithms and data structures relevant for back in engineering the short answer is absolutely yes but it also depends on what kind of back in engineering career are you taking and what what tools are or what infrastructure is already building so if you're interested to know stay tuned if you're new here welcome my name is Hussein and this channel we discuss all sorts of back in engineering so if you want to become a better software engineer consider subscribing I had that bill I liked and so you get notified every time I upload one of those beautiful videos that's it let's just jump into this video so some of you guys asked me how important are those complex algorithms Dyk Astra's and and those mergesort and quicksort and do i have to learn all of that stuff to become a back-end engineer and then the answer is really really it really depends right so let's take an example and before we do that I want to talk about the different types of back in engineering I say career you can take and you can either be an integrator so you'll integrate existing solution in the backend I'm gonna take an example or you are building core infrastructure back-end such as you are building a database or you are building a web server or a proxy and depends on where are you in the stack those algorithms can come in handy okay so let's think about the first one if you are an integrator you are back in engineer but you are doing an integration kind of a work most of your work is integration an example is you're building a crud application for say you're building an online cinema system or a URL shortener that's still a system but most of this work is if you really pay attention to it it's integration work so what you're doing is if you're building an online system you're not gonna invent your own database are you you're not gonna invent your own web server you are going to spin up our observer could be caddy could be nginx could be a niche a proxy wallet a proxies not a web server but sure right so you're gonna spin up these pieces and you're gonna choose a database could be MongoDB if you're gonna no one going no sequel router could be post class we got a relation or route and then you can start building your application logic and you're integrating an existing core back-end infrastructure that you just spun up you might throw in a write as cash or memcache D right so you're in this case you're an integrator and for that particular case it's kind of a straightforward job not easy by I'm not I'm not minimizing the work they're right but you're you're accepting API requests and you're making some TCP connection on the back end and you're managing your resources you're making sequel statements and getting back the results and processing this result and maybe restoring it back to the database and yeah you're maintaining locking you're maintaining in the multi multi concurrency and you're doing this work right but most of this work if you think about it is just your integration right you're integrating different pieces right and those most of those jobs and a 90% of 80% of the work out there is this is integrating different things and you're building an API or building an application and this doesn't really involve writing algorithms I have noticed personally so you you're not gonna write another example if you're gonna build a social network right you're gonna bail the social network you're gonna start asking this question this could be any an integration kind of a back-end job or it could be a core back-end job so what could this could fall in both category to be and it really depends on your requirement let's take an example if you're gonna build a social network you might say well social networking is a very complex problem right because you have friends and and those friends have other friends and and those friends can have other friends and immediately what pops up a graph right and computer science people love graph and there's a whole genre of just graph theory and then really complex algorithm that comes with them so you can decide to pick up a back-end technology a core back in technology that does that job for you I graph databases right I graph database that does these stuff and really does it really well and you can just write your logic on top of them again I'm not saying your logic is gonna be gonna be a little bit less but you still have to write your application logic on top of this existing database so that's one solution but you might find with time that well that particular graph solution has some limitation and a lot of people are running into this right now you will start seeing this trend people are building their own databases and if you are falling in this category you better know these algorithms right and let's say that graph database that you picked has a huge limitation that real requirement just doesn't really it doesn't cut it right because it will die because of some limitation because your requirement have let's say you have a kind of a depth-first kind of use case that this graph database just dies and there is no graph database that solves this problem and guess what you can work around this problem in your application logic and you still use that database but if you're serious about it you're gonna spin up and build your own graph database in this case write a database is a huge - you're building a core component now as a backing engineer you're not just an integrator you're building a core infrastructure piece and to do that that piece is extremely complex you need to understand the computer science behind it you need to understand different data structure you probably gonna need to throw in bloom filters you need to understand how diecast your algorithm works right - to find the shortest path and there are many you have to read up and here's the thing guys don't freak out because nobody out there memorizes any of that stuff anyway right that's why I get mad on this interview questions like okay that's all this sort this array using the best sorting looks like what I don't memorize or less stuff nobody does even if you run into this if go back to our example if you run into this graph building this graph database you think that your managers gonna stand there and says okay I'll go ahead build it build this thing now right now because you have one hour to build it know this is an extremely complex problem you gotta sit down you get ad read a reset you spend weeks reading and pick up the right algorithm compare I'll go to them together this is not an easy task yes so you probably gonna fall into that category one one way or another so don't freak out and yeah I've gotta come to that point when - no learn these algorithms and all that stuff so yeah guys if you're building a core technology a core back in technology algorithms you will need to learn them and I'm not saying have them handy and memorized all know just have keep in mind that you're gonna learn that if you're building an integration system like an API or a crud I rarely run into a problem where you get a need these complex algorithms to be honest my there might be some exceptions obviously guys right but most of the time you're stitching things together so that's the first thing we you know there are two categories there is integrators and there is core integrators most of the time don't need those algorithms or data structures however those the core technology used them heavily use those data structures and use the algorithms very very heavily again I'm not saying that's not a vice-versa you can use those algorithms in an integration back-end job that being said guys I want you to also be an open mind and keep an open mind and be pragmatic with these algorithms and and data structures a better algorithm or switching algorithm is not always the solutions and and here is a perfect example assume that you have certain sorting algorithm let's say quicksort right and quicksort is sorting around a million rows right and those million rows are we're sorting it and we're picking up the first hundred rows and returning them to user it's it's a request that pulls up it's a paging let's say it's does paging it makes a request to get the first page hundred results all right a hundred rows and that turns around and queries the entire table and then sorts that in the logic and uses quicksort to sort those and then return the first hundred rows right and those obviously is getting slow because one hundred mainly 1 million resorting 1 million is obviously slow so you can argue says ok merge sort is going to increase the performance or three percent because it's faster for this particular case but you can and here's and here's the thing guys if you tweak one small thing you can fix this problem right there why are we even querying million records and sorting them in the application that's just wrong right switching the algorithm is not gonna do you any good it's not gonna do it's gonna give you some better performance but you have to be pragmatic and think about why am I doing this the right way of doing this is like hey keep those million rows in the taya database and let the database does its job query give me order by whatever role you want whatever field you want and give me the top hundred results the database of the database will do that optimally correctly using an index and gives you only what you need and you as a thin layer you just ship it to the user so that's the correct fix right the fix in this case is not to argue which algorithm is the best quick sort or merge sort or bubble sort probably bad but no sometimes algorithms are not always the solution guys right so you're gonna think about that real hard before just changing the algorithm or just implementing another algorithm because it's better or or it gives better performance right this is this can give you into this situation sometimes you have to be program pragmatic and think about what is the application doing and does it make sense and finally guys algorithms and data structure and learning those is a great idea regardless of what you're doing you don't have to memorize them you don't have to do anything crazy read about them if you know this is knowledge knowledge is key knowledge is king anything you can do to get knowledge by all mean obtain it right if you know how diet kestrel work pull it up and just read through how it works and see it will not harm you it will not slow you down just read about because one day in the future says look oh I ran into this problem I'm building this core back in technology and you will say oh wait a minute I read one three years ago about this algorithm let me search back obviously you're gonna index back and search it and read it more right but just the idea of reading I know pertaining knowledge is always always always always a good you never hurt you keep reading keep learning guys stay awesome I'm gonna see you on the next one goodbye

Channel: Hussein Nasser

Views: 42,182

Keywords: backend engineering, backend algorithms, algorithms and datastructures

Id: pG_twkFN6iE

Length: 13min 30sec (810 seconds)

Published: Sat May 09 2020