Why NOT to become a Data Engineer

hi guys for those of you who don't know me i'm carolina and i work with data on my channel i often talk about data engineering i made a video about you know what data engineers do i made a video about how to become a data engineer and i even created a data engineering course using spotify data but in today's video i'm going to play devil's advocate but before i do it let's just get some facts straight data engineering is booming it has officially overtaken data science as the sexiest job of the 21st century according to the 2021 data science interview report the number of data engineering interviews grew by 40 over the last year and that can be compared to the growth of 10 for data science jobs so you can clearly see that the hiring activity for data engineers is much higher that of course means that the salaries for data engineers are growing as well and according to glassdoor they currently range between 110 000 to 155 000 per year and look most importantly there is a massive shortage of data engineers there is only 2.53 on average applicants per place if you think about it it's almost like you know if you learn the required skills you are going to get a job because there is such a demand and such a shortage and to put that into perspective the competition for data science jobs is almost twice as high so are there any reasons why you would not want to become a data engineer personally i think there are at least three good reasons why despite all those you know salaries and growth and demand and everything why you might still want to choose not to become a data engineer but obviously these are my personal thoughts but still i think it's worth discussing them just in case it helps someone to you know get on the right career path so let's dive into it reason number one for me personally this is probably the biggest reason i can think of as a data engineer you're not at the heart of the business in the majority of cases let me explain you probably can't imagine a hospital without doctors or a court without lawyers or school without teachers you also probably can't imagine a tech giant like google without software engineers or you can't imagine an investment bank without bankers as a data engineer you are often behind the curtains you are there to kind of support other business functions you are the infrastructure you build the infrastructure and you do it to support other functions in the business you are not the revenue generating function of the business of course in some cases this might not be true for example if you work for a company that specializes in providing data engineering solutions to other businesses then surely you are the key employee you are generating the revenue you are you know at the very heart of that kind of business but those cases are quite rare and in majority of companies majority of industries your role is to be the infrastructure and to support other business functions but you might be asking you know why is it so important to be at the heart of the business i think it all comes down to your personality and what you expect to get out of a job and what you personally value but i would say that there are you know two dimensions to this question first of all again depending on your personality you might feel more valued being in a business generating function you might feel like you are the key employee and your self-esteem might be you know a little higher because you are you know the uh the lobster that the big lobster which is you know a nice thing a nice feeling especially if you are after you know fame and glory and you just like being at the front you like speaking with stakeholders you like being in the center of whatever is happening so yeah the first reason is basically your self-esteem and how you see yourself the second reason is a bit more pragmatic um often if you are in a business generating function in a revenue generating function um your salaries might be better or your bonuses might be better because you are you know directly contributing to the growth of the organization but i've just said that data engineers earn a lot and yes they really do they earn a lot of money also in data engineering the work-life balance can be amazing so if you are someone who loves technology and who just wants to kind of get the job done then data engineering is amazing it is growing the salaries are great the work-life balance is great it's a perfect job but if you are someone who doesn't like technology as much or someone who doesn't enjoy being in the shadows or someone who is in it just for the money i suppose then you know there are other ways in which you can make money and i would think twice reason number two data engineering has very little to do with machine learning i know the data job titles can be quite misleading what's machine learning for one employer won't be machine learning for another employer what data science means for an employer x won't be the same as it is for an employer why etc but look what data engineers do and what data scientists do are actually very different things no matter what definition you use no matter what employers you ask these are very different functions the only common thing they have is data in the title so if you're someone that somehow got misled that data engineering is about training neural networks or that it is about drawing insights from data then i just want to clarify that data engineering is not about that data engineering is about data infrastructure and infrastructure is the key word here and to make it very clear i'm not saying that this is a reason why you should not choose data engineering as a career no what i'm saying is that if you enjoy working with data infrastructure then sure that's great i know a lot of people who do and they're very satisfied and happy what i'm saying is that if you thought that machine learning is involved then it isn't so if you are choosing data engineering because you thought that it involves you know data science then in majority of cases it doesn't some people want to go into data engineering because then they want to move to data science for example it's not a path that is unheard of there are accounts of many people who did exactly that so it is possible and some people kind of prefer to go this way for whatever reason but one thing i would just say is that it can be quite difficult later on to move from a data engineering function to let's say data science function and the reason why it is difficult is because the world the world will want you to stay a data engineer and the reason why the world will want you to stay a data engineer is because those skills are in demand so it will be very difficult the world will try to bribe you to stay to kind of keep improving your existing skills because you will already have some experience in data engineering so it will be way easier to find another similar job you will already have some contact in data engineering it will be difficult to move to a different function and the prime reason for that is because it is so hard to find a skilled data engineer the demand for data engineering skills is huge the shortage is huge so you will be bribed to stay and i can back it up with a very recent example i know a person who is hiring both data scientists and data engineers for his team and he told me that he's already filled the data science positions because it's not hard to find a skilled data scientist in the market currently but the data engineering positions are still open because it is so hard to find a skilled professional for these positions so look obviously there are pros and cons to this situation right like on one hand that's amazing because if you learn data engineering you will be like uh there's a saying like a hot bun that everyone just wants to grab you and eat you i don't know if you say that in english anyway anyway if you know that you're interested in data science as opposed to data engineering then i would advise you to go for data science straight away and finally reason number three data standardization and data virtualization efforts that are currently happening are going to reduce the demand for data engineers in the future there are many companies such as databricks azure data factory dinodo that are aimed or whose aim is to unify the data sources and simplify the etl process and just make everything nicely unified which is great obviously that's you know technological advancement that's something we we cherish and we we want but that also means that the demand for typical data engineering tasks or position skills such as writing etl processes from scratch such as unifying data sources cleaning it all up that might not be as hot in the future as it is at the moment but to be honest i wouldn't really treat this as a huge negative because there are two things that you should keep in mind one someone will still need to oversee adoption of those platforms oversee those platforms oversee the data on these platforms so it's not like data engineers will extinct and the second reason which i think is probably even more important one is that you cannot really protect yourself from jobs becoming obsolete in the future all right you know this is the reality we live in everything is moving pretty fast new jobs come in all jobs become obsolete everything is just going faster and we have to adapt we have to take the skills that we earned in our previous positions and apply them to new jobs new opportunities it's nothing to be afraid of and i think it would be foolish to expect that you can have one job for the rest of your life and just learn one skill set and never have to learn again that's not going to happen you are this is just not how the world works at the moment so these are the three points that i wanted to make and to be honest i hope i only encouraged you to become a data engineer because honestly what are the reasons whether when you sum it up when you think about it what are the reasons why someone might consider you know to avoid this career path well like i said if you don't like technology if you don't see yourself working in the data infrastructure if you thought that data engineering involves data science then yes perhaps stay away from this career choice but if none of these apply to you then data engineering is amazing and it is growing it is booming you will have amazing job security amazing salary amazing job opportunities so yeah you know the world is your oyster data engineers are ruling the world at the moment i hope this was useful and i'll see you next thursday bye

Channel: Karolina Sowinska

Views: 58,898

Keywords: why not to become a data engineer, become a data engineer, data engineering cons, data engineering career choice, is data engineering a good career choice, data engineer career, should you become a data engineer, should i become a data engineer, why you should not become a data engineer, why you should become a data engineer, why become a data engineer, data engineer salary, data engineer demand, data engineering salary, data engineering vs data science

Id: UjYc8uH6lHw

Length: 13min 44sec (824 seconds)

Published: Thu Mar 25 2021