Why I Left Data Engineering for Data Science

hi friends welcome back to another youtube  video in today's video I'm going to   share my journey and my reasoning for leaving data  engineering as a career for some of you who are   familiar with my career journey you already know  that i worked as a daily engineer for about two   years before i decided to become a self-taught  data scientist before we jump into the video as   always make sure to give this video a thumbs up if  you like what you hear and subscribe to my content   subscribe to my content subscribe to my channel  if you want to hear more about data science tech   lifestyle career journey and everything else in  between and i'm also starting a new series on my   channel where i'm going to invite other industry  professionals who are in the data science domain   and in tech domain i am super excited to share  those content i'm super excited to talk to all   those people and share more insights with all of  you i think you'll love that so with that let's   not waste any more time let's get started  okay so let's step back back in terms of my   journey as a data engineer a brief history i  started my first job as a data engineer if i   don't include my internship experience and my  pro bono experience my first full-time paid job   was working as a data engineer and i did that  for about two years i was working at one of the   fan companies what are the manga companies  oh my god i've been meaning to say that   okay the new name is mama i don't  know like what what is happening   yeah for those of you don't know facebook  changed their name to meta so now the fang   whole fang thing is all messed up there is a  twitter meme going on some people are calling it   manga and the original guy who this who basically  founded the term fang has called it mama which is   i don't even know if that's how you  pronounce it i need to hear somebody   else say it but apparently that's the new  name anyways i got a little bit sidetracked   so i worked as a date engineer for two years  at one of the mama companies that's so so weird   mama companies okay sure i guess  i'll get the hang of it eventually   um and um i did that for two years okay  it was it was great learning experience   there's a lot of work that goes behind the scenes  of like how the data pipelines are built how you   actually get to see those like pretty dashboards  and stuff like there's a lot that happens in the   back end and i was basically part of that back  end um of like building those pipelines making   sure the data is flowing in and we have all the  correct set of data that we need to understand   where whoever is like doing the analysis whether  those are like financial analysts data scientists   product managers whoever is accessing the  data so i did that for about two years   and then after two years i decided to make the  transition out of the data engineering domain   and um became a data scientist so there were three  reasons why i left data engineering domain and i'm   going to talk about them one by one number one  for me was obviously the type of work i was like   i supported one of the mama companies i just can't  get it one of the mama companies who has like   presents globally and i basically supported all  their data pipeline basically how customers are   coming to the site and how they're engaging with  the site um and where they are coming from where   how they're entering the site what do they do  after they enter the site so a lot of like traffic   data um i was ingesting and building pipelines  around that whole whole basically domain after   doing that for two years i realized like a lot  of the work was somewhat repetitive for example i   would build one pipeline it would go to production  people will start using it then i get assigned on   another project and then i'm doing pretty much the  same thing i guess like i was i was like in the   entry level role so that's why maybe it did not i  did not get to do like a lot of like thinking work   but um most of it was like very repetitive work i  would say on top of it all the data pipelines that   i created supported the team's weekly business  review which goes out on monday 12 pm that means   whoever is going to prepare the data monday one  morning has to has the data by sunday night so   i for that two years i actually was working on  sundays trying to make sure that the the pipelines   are running and everything else is is is in in  the database that it needs to be and if it's not   there then i'm like opening ticket talking to the  data warehouse teams and trying to figure out like   what how can we get that data in time by monday  morning or what is the reasoning that the data is   delayed because if data is missing i need to give  reasoning that was my reason i got actually kind   of bored i felt pretty comfortable in the space  i got kind of bored so i was like i need to try   something different and that was the time when  i actually did a project um which was more like   data science focused project and that's where  i got the flavor of like what the interesting   word could look like which means like i don't do  the same thing over and over again so that like   brings me to my second reason visibility so one  of the things that i struggled with like working   in the data engineering space was it was it's  a back end role like you work in the back end   you prepare data and then you're putting in a  database and somebody else is either accessing   it directly from the database and presenting it or  if you are working on a data poll you are actually   pulling the data and then you're giving it to  somebody else like a product manager or somebody   else so for me i realized like i want to be the  front-facing person because i was realizing that   i was doing a lot of work like i was pulling a lot  of data i was like doing a lot of like work in the   back end but i wasn't getting the visibility in  the front end by the time somebody takes that   data or writes a paper on it and presents it  to leadership my name was lost in that process   so i wanted to be in the front i didn't want to  be in the back so that was like my other reasoning   because i wanted that visibility i wanted to be  part of those decisions that were being taken   with the data so that was my second reasoning  of the lack of visibility which led me to   um make that decision the third and the last  one was it's it's a bit of open-ended and   some not all of you are gonna agree with me  on that that data engineering overlaps with   software engineering so let me say that again the  the data engineering work overlaps with data with   software engineering in some companies they don't  even have data engineers they only have software   engineers and software engineers are doing the  data engineering work that data engineering   the role itself exists in very big companies  but it does not exist in all companies   even if you just take example of like the big tech  companies and you just search role by each company   data engineer role by company you won't find it  in every company which means you either have to do   that maybe they have some other role that is that  is considered a data engineer but they're calling   it something else so you have to like do your  own kind of research and figure out what that is   or you're stuck where you are pretty much  the create opportunities i did not find that   in data engineering like there is like entry  level data engineering one data engineering   two senior data engineer or principal and  then there were not like many principal data   engineers like actually i don't even know if  there were there were any data principal data   engineer in that company where i was so i didn't  see a lot of people in the leadership position   who were data engineer previously and now are  in the leadership position and for me like i   want to be in the leadership position i  uh so for me like that was kind of like   also like a reminder to self like what is my what  is my trajectory in this role if i were to like   invest 10 more years in it all those things  combined basically led me to transition   out of the data engineering role the money was  good i'm not going to complain the money was good   pretty much same pretty much same as i made as  a data scientist in that company but the type of   work was obviously is very different between the  data engineering and the data scientists work i've   been doing data science work for about like seven  eight years now and i love it like i'm not bored   i know this is like the right field for me  my work is super interesting even though like   probably from my last videos you saw you saw like  how frustrated i was with like the job search   specifically yeah there are frustrations but i  love this space i don't think i'm going anywhere   i love working on data science projects i love  working on experiments i love working on documents   i love writing my analysis i love storytelling  and i love being the person who is in the front   presenting those insights and being part of the  whole decision i think that just that's just like   more of my work style and what i enjoy most this  video is not supposed to bash a specific career   choice i think data engineering is great for  somebody who enjoys that type of work for me   personally it did not make sense because i had  different career aspirations and i wanted to do   different type of work that's all I wanted to say  and if you have data engineering experience let me   know in comments how has your experience been what  are your thoughts on the data engineering field   in general like i would love to know your  thoughts but that thank you so much for   watching i hope you're having a beautiful  day and i will see you in the next video bye

Channel: Sundas Khalid

Views: 66,514

Keywords: data science, data scientist, self-taugh data scientist, tech jobs, amazon, women in tech, big tech, google, Machine Learning, Python, data science projects, data science tutorials, data science jobs, data science day in the life, AI, vlog, data science 2021, data science 2022, big data, data engineer, data engineering

Id: wN_eu04kXhk

Length: 9min 21sec (561 seconds)

Published: Sun Oct 31 2021