The Harsh Reality of Being a Data Engineer

thank you HubSpot for sponsoring this video data engineering is one of the hottest careers out there right now it gets a lot of attention from content creators and everyone who wants to get in tech there is no denying that the number of jobs that have opened for data engineering has increased over the years and there is a rising demand for data engineering so in this video I wanted to talk about the harsh reality of being a data engineer so you can make a well-rounded decision if you are considering a data engineering career I have been in the industry for over over 10 years now and have had experience across different data science job family I actually started as a data engineer before I transitioned out of the data engineering job family to data science you can watch one of my videos where I talk about in detail why I left there are four points I will be covering in this video so keep watching reason number one is difficult entry point even though the demand for data engineering has increased there are no degree programs that actually teach you to become a data engineer so there is a clear gap on how you become a data engineer engineer especially when the schools haven't caught up and they haven't been offering data engineering this makes it very difficult for somebody to transition into a data engineering role or start as a data engineer after graduating from school and often times you will see that data Engineers who are in the industry they have been working in the industry in a different role and then they decide to transition to data engineering so they do have some technical background whether that is software engineering or some other technical work where they make that transition from software engineering to data engineering Ro or from another role to data engineering role and that makes it very difficult for somebody who is a junior or who is an entry level to start get into Data engineering because there is no defined program that help you learn data engineering and get a job as a data engineer and this point is specifically important for people who are students and people who are just starting out how to get into dat engineering is not as clear as some of the other roles such as data scientist software engineering data analyst but there are tons of resources on YouTube and other platforms that help you learn about data engineering but that means you have to do your own work to understand what topics you need to learn and where you need to learn it so you can Define your path to learn data engineering this means you have to be self-driven motivated to be a data engineer because you'll have to do a lot of work to figure out what curriculum you need to learn in order to become a data engineer especially if you're just starting out if you're already working in the industry even in that case you'll have to figure out a path where either another team is willing to take you in as a data engineer and teach you or some other way where you can like do a side project in data engineering and pivot your path from whatever role you are currently in to a data engineer and this is actually a harsh reality because it's not as clear-cut as some of the other roles in Tech the second reason and this is going to sound really bitter the work that a data engineer does can be easily done by a software engineer yes I'm going to say this again the work that a data engineer does can be easily done by a software engineer often times companies hire data Engineers so they can spare up time of software Engineers so they can work on other important work and data Engineers can focus on building data pipelines and other data engineering related work now this begs the question that in the future in the world of AI where things will get automated software Engineers will likely have less work to do what are the jaob families that they will be able to absorb in quickly data engineering definitely stands out and this is something you should keep in mind as you consider a role in data engineering on the topic of data engineering and coding I wanted to share this intro to python ebook which is the beginner guide for learning python especially if you are planning to use python in a data role this ebook is created by HubSpot who is also sponsoring this portion of the video the ebook covers libraries such as numpy Panda matte plot lib which are essential libraries when it comes to data analysis and any data related job it walks you through basic ideas gives you coding snippet so you can plug and play it's available to download for free I'm linking it in the description below feel free to check it out now let's talk about the the third harsh reality of being a data engineer building on my previous point on data engineering versus software engineer another topic that we need to discuss is the salary even though data engineering are meant to be very technical and are supposed to do the same level of work often that a software engineer does they do not get paid the same this is an average salary for a software engineer and then this is average salary for a data engineer in the US you can clearly see that despite the work despite the technical depth and the knowledge that is required the salaries are not unpar are and you get paid less than a software engineer so this is something that you should definitely keep in mind if money is important to you which it should be you should definitely keep in mind that data Engineers do not get paid the same as a software engineer the fourth harsh reality of being a data engineer is the lack of career growth now you must be like what are you saying if data engineering is similar to software engineer why is there not a career growth why is there a lack of career growth because in software engineer you have software engineer 1 software engineer 2 software engineer 3 and then you get to principal and then you become SDM which is software developer manager in the data engineering space often times you will find that a lot of data Engineers Peak at senior data engineer however despite the negatives data engineering is a great career to be especially if this is something that you enjoy doing especially in the current environment when there are limited job opportunities out there and data engineering is one of the job families that has a lot of open roles it's definitely a job family that everyone should be considering if they want to work in a data and Tech space what are your thoughts on this topic anything mention resonated with you let me know in comments and I'll see you in the next video bye

Channel: Sundas Khalid

Views: 27,497

Keywords: data science, data scientist, self-taugh data scientist, big tech, Machine Learning, Python, data science projects, data science tutorials, data science jobs, data science day in the life, AI, big data, data analyst, data analytics, business analyst, sundas khalid, people of color in tech, data science at google, data science 2024, data analyst 2024, data engineer roadmap 2024, learn data engineering, data engineer vs data scientist, how to become a data engineer

Id: Gniq-YEJvfc

Length: 5min 34sec (334 seconds)

Published: Thu Nov 30 2023