How I use ChatGPT as a Data Analyst

nerds I've been using chat gbt to analyze data for my job and it's been saved me a lot of time let me show you what I mean so here we have a spreadsheet with some data we want to analyze first I asked it to connect to the data by providing it with a link to the spreadsheet it gives me the first few reads of the data so I know it's connected so next I ask it what's the status at even about and it looks like it's a bunch of job postings specifically for the role of data analysts they even provides some basic stats about the columns in the spreadsheet let's do some quick Eda to see what are the 10 most common jobs and it provides me this bar graph with data analysis the most common job title followed by senior data analysts let's wrap this analysis up and check out what the median salary is for these 10 most common jobs with this calculation let's actually time [Music] and in just over a minute we're able to generate both these visualizations you can see that data scientists have some of the highest while data analysts have some of the lowest hmm I don't like this so I did this entire analysis before filming this without chat gbt and it took me almost an hour and here in less than a few minutes we've generated all of these different visualizations and insights without even coding one bit now that we've shown chat gpd's capabilities let's move on to an even bigger problem that we're going to solve all right so I stayed up a little bit too late last night is this how's my audio my audio good and I put together this PowerPoint of how we're going to solve this problem and there's a lot of hype around AI lately and whether it's going to actually automate and take away jobs and is it going to take over the world probably but more importantly we want to focus on how is AI impacting data jobs and when I look at two main aspects first from the people's aspect from the persons that has the job how satisfied are they with AI impacting their jobs the next one is businesses is there going to be an actual impact such as Financial or are there going to be actually looking for those with skills of this Ai and how we're going to accomplish this all well here's the game plan I have a two-step approach the first is using chat gbt for data research we're going to be using different plugins and internet searching methods in order to find research papers data whatever it may be in order to find insights into those two main areas that we're trying to look at from there we're going to move into Data analysis and we're going to once again use chat gbt to actually analyze this data that we found in order to find insights and maybe prove one way or another whether how bad AI is impacting data science jobs but before we just dive into this analysis we need to understand the capabilities of this tool chat gbt so um BRB [Applause] so what the heck is Chad gbt well I asked it and come to find out it's an artificial intelligence that's running on one of the models from open AI specifically this one right here I'm using model gbt4 which is their newest and greatest one and in a recent survey from stack Overflow over 50 000 respondents reported that they prefer Chachi BT over any other AI chatbot at the moment I've also tried all these other options and done a color comparison video and I've found that chat gbt is the best at performing data analysis at the moment but there is a catch with chat gbt and that comes to pricing so you can use the free version of chat CPT and go in and prompt it and get different insights from it but you're really going to be limited in what you get now if you go with the pro version not only do you have access to an improved model of gpt4 but also you have access to plugins and plugins are really the money plug-ins allow these chat Bots to connect to different applications so recently I've been using with Wolfram in order to connect and do analysis on the back end and then feed it back into Chachi BT so I can get my numbers crunched all right enough of this green screen break let's get back to the video all right I made some progress on using chat GPT to go and search the internet to find things around AI impacting data science jobs and first up is one of the limitations that I found with chat gbt and that's that the model itself was only trained on data upped up to September of 2021. this has been really big problem because you can't actually ask the model directly about things that have happened right now because it won't know that's where plugins come in the first main one that I've been using to search the internet is their Bing plug-in that allows it to go in and search and find different articles research papers and Publications and I can find out what's most impacting now chat gbt isn't really that great I also trusted out Bing and also the Google competitor Bard and they're not really search engines right so I still rely on Google itself to go in and search but I was able to find some recent articles on this specifically this one from McKenzie talking about the economic impact of generative AI now this bad boy is almost a hundred pages long and if you think I'm going to go through and read it you're probably wrong so what's good about chat gbt is they have plugins in order to read these type of PDFs I just give it the link or the PDF itself and it will go in and actually read it summarize it and provide me with insights now generative AI has to deal with AIS that produce new content so basically what jot GPT is anyway it went through and found this pretty interesting stat on generative Ai and it showed that it had really high potential for automating data related tasks specifically around processing and collecting data I mean this is all sort of meta right I'm using chat gbt to analyze this PDF and provide Data Insights to me on Data Insights of generative AI so pretty crazy okay another key source of information was this website right here from stack Overflow where they actually surveyed a bunch of developers around 90 000 on what their sentiment was towards AI one of the key findings from this was that 77 of those surveyed had a favorable opinion towards using AI in their workflow that's four out of five that's pretty high considering how much negativity you wear towards AI recently now the results of this report are pretty damn long and so once again I can use chat gbt to go in and analyze it and actually asked it to summarize it better around data science professionals and how they felt AI was impacting their job and chat CPT was able to point out a pretty interesting Insight compared to developers data analysts and scientists and Engineers had some of the most positive sentiment towards this technology now the document itself has a lot of numbers around how these different job subtitles feel around AI so I asked it to analyze these numbers provide a comparison but arranged some issues where I started hallucinating and making up a lot of the values that it saw in this document right here so I was really confused with this especially since I had provided it with the source document I don't know why I was fabricating these numbers I couldn't figure that out but anyway nonetheless I was able to find the actual table that was referring to and with another plug-in I was actually able to go in pull these numbers out and then visualize it in order to see how it compared to other developers all right we're back to the green screen so we discovered that data nerds have some of the most favorable opinions about using AI Tools in their job and next slide we also this also satisfies what we needed here in understanding what people's thoughts were towards this technology and overall what we're seeing four out of five prefer I mean I'd say that that's a pretty good score and frankly I think this is good but what about this side for the businesses what are their thoughts about it I mean they're the ones that are actually going to be driving this type of change within organizations and ensure people are adapting to this new technology the McKinsey study that I referenced at the beginning of a research basically confirmed that these AI tools are going to increase our economic output so it's a win-win situation for companies to actually utilize this for their employees yes I do understand that there's different privacy concerns around what can be put into these type of tools but that's something that we need to start thinking about now so with the research we did with chat gbt we weren't able to find any data that related to how companies are implementing these AI tools but I have an idea so I've been collecting job postings since last year and I have more details about it in this video right here anyway I've collected over a million jobs for data science job postings this should show indirectly how companies feel about implementing these AI tools whether they're even asking new employees to have these skills now this data set includes things like job title salary and even job description so we're going to do is we're going to use chat gbt to perform some NLP and go in and extract out any data related to AI being used in these jobs so let's get into it all right it's been a few days and I've made a lot of progress using chat gbt and specifically using the notable plugin and not only is it providing visualizations through the front end but you can also see the code behind how this all worked and that's pretty cool that you can double check it with this one quick note is that chat GPT will be releasing the code interpreter plugin in the near future and this plugin will be as powerful full or even more powerful than this notable plugin so getting into the analysis all of the data is located in a database that has a secure connection so I asked chatgpt how do I go about giving these credentials so that way it can access this data and at first it tried to give me code to perform this but I re-prompted and told it to use the installed plugin and with that it gave me a link that I could input in the credentials of the database so to make sure that it was connected to the database I asked it how many entries it had in this table which it told me it was about a million so I knew we were connected next I prompted chatgbt to provide me some descriptive statistics about the data set and it not only provided information about what columns were in the data set but it also showed some statistics around the salary information I'm not going to really use that right now but I thought it was interesting that I provided nonetheless next I want to get into analyzing these job postings for keywords around AI so I prompted chat gbt for a list of keywords related to this and although good it's not really what I wanted as these are more theoretical terms and not necessarily tools I've analyzed terms like Ai and artificial intelligence before and I didn't really think the insights from those keywords were really that great instead I want to use tools that relate to AI for this we're going to be using those tools that were included in that stack Overflow survey not only are we going to focus on search tools but also on developer tools so I gave Chachi BT this list of tools and ask it to analyze whether these keywords were included in the job descriptions this is where I ran into a limitation of this notable plugin because this data set is over a million rows it had tried to extract those a million rows of data and import it into a pandas data frame within the notebook I did find an option to upgrade my small ass computer but it looked like it was behind a waitlist so I reached out to notable on Twitter and ask them if I could access these larger computers and they told me to wait so we're only limited around a hundred thousand dollars for this and so we used a random sampling for this analysis and double checking all the results that I got at the end of this it doesn't look like the results really change or fluctuate that much so I think we're fine with using this sampling so the first thing I jumped into is telling chat GPT that each one of these rows was a job posting and I was curious to find out what percentage of these jobs contained one of those tools and less than two percent of jobs is even requesting the skill of AI tools in the job description so we're not really off to a good start I thought next let's actually look into what are the actual tools that are appearing in those job descriptions and I had to reprompt chat gbt several times getting it to sort highest to low displaying the y-axis as a percentage and then formatting it to include one decimal place but eventually we got this visualization which shows that Bing and you.com are some of the highest requested tools I am a little confused by these results as they don't really correlate really well to that stack Overflow survey by but they are less than one percent so maybe this does make sense so next I was curious with this low percentage are they even paying those with these type of skills a higher salary once again I had to re-prompt it to correct column names and also format the y-axis as a currency and we finally get this showing data scientists do you get paid more but data Engineers machine learning engineers and data analysts don't wow and such a low percentage of these jobs including these Search terms I'm not ready to call anything conclusive off of this visualization just think that we maybe should explore this more in the future now I did want to get into one last thing looking at how these different job titles compare to each other for containing these AI tools and looks like machine learning Engineers are leading the way followed by data analysts and data scientists so with these results I think we've shown pretty conclusively that companies aren't caring as much as people are about implementing AI Tools in their jobs and I think this is a pretty big problem but there's such a big disparity between those that want actually Implement AI Tools in their jobs and then the company is not really requesting it in these jobs anyway I digress so we've got value of this video and you like AI smash that like button if you don't like day I like those companies hit that dislike button I'm curious to know anyway I'll see you in the next one

Channel: Luke Barousse

Views: 151,869

Keywords: data viz by luke, business intelligence, data science, bi, computer science, data nerd, data analyst, data scientist, how to, data project, data analytics, portfolio project, sql, excel, python, power bi, tableau, data engineer

Id: WSkcsrFCCXM

Length: 13min 40sec (820 seconds)

Published: Fri Jul 07 2023