Natural Language Processing in Python

okay thank you all for coming today this is the natural language processing in Python by our speaker Alice Chou she's a senior data scientist at Metis and I believe that she has a bit of a tutorial that we can follow along with so please join me in welcoming her thanks everyone it's so great to see a full house here today I'm very excited to speak at the PI Ohio conference one of my colleagues spoke here last year and he loved it and so he recommended that I come as well so today I'll be talking about natural language processing in Python and the first thing I want to mention is this is a two hour long tutorial so you just make you so that everyone signed up for sitting here for the next two hours also I'm going to be walking through a couple of Jupiter notebooks so Jupiter notebooks are this nice way of doing Python code and we use it a lot when we teach our data science boot camps so if you go to github.com slash at - of data there's a readme file up there as well that has some setup instructions so that includes downloading anaconda which is how we recommend our students use Python because it comes along with a lot of data science packages as well so I recommend you download that and then once you download that you can try to see if you can open up some of the notebooks so we won't be touching the notebooks for about 20-30 minutes so over the next 20-30 minutes try to get that set up and I'll check in with you again to see if you have any issues all right also today we're going to be walking through an end-to-end project so I see a lot of tutorials out there that our algorithm specific so they'll be like okay we're going to teach you this algorithm or this algorithm but it's hard to see how it all comes together so my goal for you guys today is to show you how to start with a question and then go through all the steps of cleaning the data and do some exploratory analysis and then applying algorithm so you can see how an end-to-end project works okay so here's a schedule for today so first I'm gonna give you an introduction to NLP and then also data science because I'm a data scientist so I'm going to show you how I approach problem and then we're going to go into the tutorial which is gonna be in the Jupiter notebooks and then I'll end with the conclusion so let's start with natural language processing so all of you signed up for this tutorial do you know what NLP is any here any suggestions I see some nods yeah yeah I love that finding meaningful information from random text so when I think of a natural language processing I think of it in two parts so the first part is now a natural language so what is a natural language well when you think languages you see all these languages up here which one of these is not like the other Python so Python here is not a natural language so Python was written for coding whereas natural languages are languages that have naturally evolved over time that humans actually use to communicate and then processing I think of like a processor on a computer so it's how the computer carries out instructions and so natural language processing is how a computer is able to process these natural languages and then more simply put just like you said earlier I think of natural language processing is just simply how to deal with text data so today we're going to be working with text data and then the other thing I want to point out is that NLP falls under the greater branch of artificial intelligence I want to mention this because AI is such a hot term these days and AI is all about a computer performing tasks that a human can do so that includes things like image processing right humans can see so how do you make a computer see well in this case since we're dealing with language humans can interpret language so how can we make the Machine interpret language okay so now I'm going to go through a couple examples of NLP so let's say that you're the manager for customer service center so you are working for a company and you sell hats and you also sell shirts and then you get a bunch of calls from people that talk about like them loving your hats or them hating your shirt and you want to get an idea of how people are feeling about your hats and shirts overall so let's say you have a thousand calls you're the manager you could go and listen to all those thousand calls but that would take a very long time you might have your customer care reps maybe take notes on who's happy about what but another thing you can do is you can use NLP techniques to actually read that text and figure out is this a positive emotion or is this a negative emotion so by applying NLP you can see that people are generally pretty happy about your hats but they're not really happy about your shirts so this idea is called sentiment analysis and this is one of the things that we'll be going through today okay so another example so say you're working at a legal firm and you're working on a case where someone at a company may have embezzled some money and so you need to figure out who is doing all the bad things at the company and so you go through all the emails at the company and then you realize there's like 10,000 emails that you have to go through so how do you prioritize which emails are likely related to embezzling money versus something random and so what you can do is you can look at all your emails and then you can label them not you manually labeling them but you can have a machine using an Opie technique to label them and say okay these emails up here are about project work I see these are about money this one's about someone's honeymoon it's very personal so this is probably a low priority email that you don't need to read but then maybe the ones about money are the ones that you should make higher priority and read first out of the whole 10,000 email set so this idea here is a very powerful a topi technique called topic modeling and I find this very very interesting and this is one of the ones we'll be talking about today as well and then finally this is a kind of fun example but say that your writer and you work for an inspirational quote company and you have been tasked to write inspirational quotes for 365 days of the year and you've written it for 11 months but you're just completely out of ideas and so you're thinking okay I have all these inspirational quotes I'm a Python developer why don't I just code something up that will write inspirational quotes for me based on my past codes so you can do just that with something called text generation which is the final thing that we'll be talking about today so just to recap the three NLP examples we'll be going through today our sentiment analysis topic modeling and text generation so these are just some of the NLP techniques out there but they're ones that I find very interesting and also a good introduction to data file and so before we go through all these techniques we're actually gonna go through a lot of data cleaning a lot of people think okay these are such cool techniques let's use them right away there's a lot of stuffs we have to do beforehand why should I'll be walking you through so that's natural language processing what is it again yeah out of language out of text data okay so that's a no fee alright so now I'm going to talk about data science and I'm very passionate about data science because I'm a data scientist and I like to give a little intro about this because this is how I see these problems so has anyone here heard of data science okay what is data science exactly I love that getting meaning out of data or using I vote here using data to make decisions so falls under analytics which is generally using data to make decisions or getting meaning out of data but other under analytics there's all other things like business intelligence which is like creating dashboards and then there's data science so when I think data science I always think the data science min diagram so this was created by Drew Conway data scientist about 10 years ago and I always have this at the beginning of my lectures so what he says is that every data scientist should have three types of skills programming skill math skills and communication skills so if you have all three your so let's say you have two of the three let's say you have programming and math skills with no communication skills so then you're more of a machine learning engineer because you can code you can you know all the algorithms but you don't necessarily have to create presentations right on this side if you have math skills and communication skills you're probably more of a researcher so you don't have to deal with a lot of data but you know the algorithms you have the domain expertise and you can do great research so over here this is the danger zone and I want to mention this because I've seen a lot of people fall into this danger zone so these are people who know how to code like everyone in this room and also have communication skills so like know how to communicate ideas and so if you have these two skills without the math skills then you fall into the danger zone because what ends up happening is you'll just pull packages or libraries out there use them and not know how to interpret the results or interpret them incorrectly so because of that today we're going to be going over all three of these types of skills so first we're gonna be talking about a couple different Python libraries so for data analysis my favorite is pandas so I'll be talking out pandas proximately talking a little bit about regular expressions does anyone have regular expressions before so powerful right it's great for text data and then also scikit-learn which is used a lot in data science and it's a great way to has much a lot of machine learning tools available but we're gonna be using it to format some of our data we're gonna be using a couple NLP libraries so the most popular one by far is NLT Kay natural language toolkit and then text blob is built on top of NLT K and then Jensen is specifically used for topic modeling and don't worry about knowing all this right away we're gonna walk through all this today and so that's from the Python side from the math and staff side this is where I really want you guys to try to understand these concepts what's so important in interpreting the results so we're gonna be cleaning the data and we're going to be putting it into a couple of formats so first is just a general corpus format what go over in detail and then also a document term matrix and then we're going to do some exploratory data analysis which I'll be calling EDA from now on and that's gonna be a lot of word counts and then finally we're gonna be going through those three techniques that I mentioned earlier so sentiment analysis topic modeling and text generation and then finally there's this communication piece and usually all the soft skills kind of get lumped into this area but I think of it as two parts so there's a design piece which is all about how you design a project and this is so important whenever I advise my students I find that this is where they struggle the most so just figuring out if you have a question like where to start how do you scope out your project what insights can you draw from that data what visualizations can you make to communicate things more effectively so that's all about design and we're gonna be doing that throughout the tutorial and then finally having some type of domain expertise is really important so I'll be sharing with you the project we're going to be working on throughout today where the question we're gonna be answering and hopefully you'll know a little bit about the subject area that you can help me interpret the insights ok so before we move into the toriel the last thing I want to talk about is the data science workflow so today the science workflow this is the order of steps that we usually take to solve a problem and this is the order that we're going to be following today so the first is to start with the question it's a lot of people think well data science that has data in the title so you have to start with the data but that's not the case you should always start with a question mine because that will influence what data you're actually going to collect the first step is to start with the question then you're going to get the data perform EDA which stands for exploratory data analysis great then you're going to apply some NLP techniques and finally share all those insights so let's walk through a simple example so let's say I had the question if I study more well I get a higher grade what are your initial thoughts Yeah right so that's your guess but the within the data science workflow the next thing you're gonna do is you're gonna get some data so what type of data money you get number of our studied grades perfect yeah let's start there so let's say I go when I talk to a teacher and I get this information so for all these students this is the number of hours they studied this is the grade they got so take a look at this data what do you think of it what's not consistent about it yeah there's there's definitely bad things happening here right we see a two here we see someone might have had a typo here and so at this point I need to clean the data this is what we mean by clean the data it get it all in consistent format so at this point got him have to make a few assumptions so I'm going to assume that that two means that two and that nine I'm eight was actually 98 it might not be always correct but I think it's a pretty good assumption to make at this point okay great so I clean the data so the next thing I want to do is perform EDA so what this means is I want to quickly understand and quickly see if my data makes sense so this is typically done through visualization techniques so knowing that what would you do at this point with this data yeah scatter plot perfect I'd create a scatter plot so just by looking at this scatter plot what what are some things you can learn from this data yeah great we see a positive correlation right you see that the more hours you study the higher grade you're gonna get anything else yeah they did yeah they did pretty decent the other thing that I found here was this person up here that was Charlie he did pretty well without studying that much right so those are my two findings that's what you'll typically see with numerical data first you want to see if there's any correlation and then you want to see if there are any outliers so in that case that's an outlier great so that's my EDA my data makes sense right more I study higher grade I get great so only after you've done your EDA now it's time to apply some techniques so if the most basic data science technique is applying a linear regression so fitting a line to our model and you can see here that was the linear regression I can actually get a specific equation to see what that relationship is and so at the very end now I can share all the insights basically summarize everything I've done so my question was if I study more will I get a higher grade and the answer is yes there's a positive correlation and then specifically with when you're aggression we saw that this is the relationship and finally we know that Charlie was really smart and then he was kind of an outlier so at the end of the day even if a linear regression predicts you're gonna get a T you're probably gonna get slightly less because of that so that right there was our whole data science workflow make sense great okay so at this point for this section I talked about data science being part of analytics and it's all about using data to make decisions talked about the Venn diagram and not fall in the data too as a danger zone because you need to know the math and then finally the workflow all right so let's get into the tutorial which is what you've all come for so was everyone able to open up a Jupiter notebook does anyone have issues that's pretty amazing nice job guys that's amazing okay great so at this point we are going to go through all of these steps so the first thing we're gonna do is I want to note that for the getting clean data step the only difference with NOP analysis is that this step is going to be using text data and this is going to be the format for today so I'm gonna have a presentation at the beginning of every section and then you see here there are five notebooks for you to follow through I follow along with and so I'll be doing a notebook after these middle three steps here okay so the first step is defining a question so this is the question that we're going to be focusing on today does anyone recognize this woman anyway she's a comedian okay so her name is Ali Wong and about two years ago I was not into stand-up comedy at all and I saw her special on Netflix and I really liked it and I was kind of surprised because I I'm not into stand-up comedy and then at the time I was thinking okay yeah do I like her because she's also female and Asian and I was also pregnant at the time I'm thinking maybe but maybe there's something more to it maybe there's something actually in the language that she's using that makes her different from other comedians and so that's the question that we're going to focus on today okay so our goal is to see what makes Ali Wong's comedy routine stand out that sound good okay great all right so now we've got our question great so the next step is we have to get our data so there's two parts to this we have to get our data somehow and then we also have to clean that data so the input into this step is okay how is Ali Wong different other comedians and the output to this step is we want clean organized data that looks something like this because this is what a machine can work with okay so for this first step data gathering my question to you guys is how are you gonna get this data you haven't the github ago but how did I get it how did I get this data transcripts so luckily when I was thinking of doing this project I just googled Ali Wong baby Cobra transcript and then it came up so very very lucky there and then the second question is how much data are you gonna get so I said I wanted to compare Ali Wong with other comedians so how would you decide how many comedians to compare her to so who are the top 10 comedians how do you do to find that so I actually did exactly that I googled who are the top 10 comedians and every list was different right so how are you know yeah that's actually a great idea so you could do that so I tried to make this as much of a data centered approach as possible so the way I did this was okay first this is where I got the transcripts from and then second I went to IMDB because IMDB has this advanced search option and I tried to adjust the filters so that I would get about 10 comedians and it would include Ali Wong and I think those are my two comments I would get about 10 comedians and it would include Ali Wong and so I ended up kind of tweaking the features tweaking the filters to make this make sense so at this point is one of pods because these questions that I've asked you seem really simple they're very hard and this is always where you have to start your analysis is thinking like how do I want to scope my project and this is where your domain expertise matters a lot because I'm gonna get this top list of top ten comedians I also have to make sure that my list makes sense like I don't know too much about stand-up comedy but my husband knows a lot about stand-up comedy and so I asked him and he's like okay this list it makes sense you got you can start here so the way I decide to limit my scope was I looked at comedy specials from the last five years the hats had at least a 7.5 rating out of 10 on IMDB with over 2,000 votes and then I ended up with about 20 comedians and then some of them showed up multiple times so I had to figure out how to deal with that so then I decided I'm just gonna take for every comedian their top rated special and that's a good place to start so these are the comedians that I ended up with this is anyone here into stand-up comedy okay so looking at this set is this a pretty representative set so like the ones I recognize here there's Ali Wong and then I know Dave Chappelle is popular I know Lou CK is famous I know I don't know many of the others but having a few in there makes me feel good about my data set so great I've locked in my scope let's move to data cleaning okay well actually before we move into data cleaning I just wanted to mention a few of the Python packages that I've used here and we're gonna walk through this more in the notebook but throughout my presentation anything that you see in teal that's going to be a Python package so for a web scraping has anyone done any web scraping here before okay so then you'll recognize these so I used requests so request basically allows you to enter in a URL and then you can get all the data from that URL and then beautifulsoup is great too because then it actually looks at that HTML page and then it can you can pick out certain sections from that HTML page so just think of a web page that has transcripts on it but it also has a ton of other stuff with requests and beautifulsoup you can just pull out the transcript text that's the web scraping side and then also I did a lot of pickling so you'll see I've done five I created five different notebooks here what I do is at the end of every notebook I'll pickle some objects what that means is I can save the object for later so think of an object like a list you can pickle a list so that you save it and then in the next notebook you can load it up and weed it and use it again so you'll see a lot of pickling throughout okay so we've gathered our data we have 12 comedians now we have to clean that data so our goal for this step is to get the data in a clean standard format that we can use for further analysis and we're gonna get our data in two types of formats the first is just a general corpus I'll talk about in a bit and the second that involves a little bit more work is a document term matrix so the first is a corpus and a corpus is just a collection of text very simply so the goal for this step is just to get the data in this nice table here and the way we're gonna do this is using pandas so again pandas is the Python library for data analysis and specifically in pandas there's an object called a data frame so a data frame is essentially just a table looks like this so every row of a data frame will have an ID and then every column of the data frame has the same data type so we're gonna be creating a panda's data frame to create this corpus dress so that's the first format very easy so the second thing we're going to create is a document term matrix and this is a little bit more complex so the things we need to do here first we have to clean our text then we have to tokenize it and then finally put it into matrix form so let's walk through this step by step so this is the first line of John mulaney's stand-up routine and if you look at it it looks pretty messy so if you are a computer and you want it to only understand the most important parts of this line right here what would you do to make this data cleaner great idea get rid of punctuation any other ideas what's that mm-hmm yeah yeah yeah these are all great ideas and these are all things that you can do to this text so there are many different ways that you can clean your text but there are a couple of standard ones that we're gonna be going through so the ones that we're gonna be doing today is first removing punctuation also making everything lowercase and a very common thing to do is to remove numbers and any words that have numbers in them all the things you guys said can be done but this is just like a very quick first pass so if you did all of those things then your text would look something like this so at this point we've done a first round of cleaning the text so everything's lowercase and we we have them in a mostly a standard format so the way we're going to do this is using regular expressions in Python so regular expressions are very powerful if you think of like control find on your computer we can search for one word with regular expressions you can search for patterns so you can search for anytime a word starts with a capital letter do something with it that's the power of regular expressions I'll walk you through that later as well and so now that we've clean the data the next step is to tokenize the data and tokenization is very standard term and NLP so to tokenize something is to break it down into smaller parts so you can tokenize things by sentence you can tokenize things by to word combinations they're called by grams but the most common way to tokenize is by word so at this point if we tokenize this sentence up here by word it would look something like this so now every word is its own item okay so at this point we've done some tokenization and now every single item is its own word so at this point we can remove things called stop words so stop words are words in a language that have very little meaning so every language out there has its own set of stop words so you'll see that English has its own set of stop words including things like the or words like that so it's very common at this point you have all these words you can remove these stop words because they're not gonna add much meaning for the machine to process so if we move the stop words these are the words we end up with and at the end of the day you end up with just these words here so we went from that really messy looking data to just six words here and it's much easier for the computer to process so this type of format for the words with an NLP it's called the bag of words format because if you think about it it's saying that this document here is essentially just a bag of words and the order of the words doesn't really matter it's just a group of words thrown into a bag so it seems like an oversimplified way to represent text data but it's actually really powerful just using bag of words you can do a lot of analysis which I'll show you later as well okay and then finally we're gonna put this all into a matrix so this texture was just for John Mulaney skit but what if I want to within one table include data for John Mulaney and Ali Wong and Dave Chappelle well I'd have to put that all into a matrix so the reason we put this into matrix is because we want to store these terms for multiple documents so at the end of the day you get the document term matrix so this was our goal we wanted to create a matrix that contained contains for every row a different comedian or a different transcript or a different document and then every column here is a different term and then all the values inside are the word counts and so if you think about it we started with for every comedian a big transcript of really messy data but using those steps of cleaning tokenizing and putting in a matrix form we've been able to really simplify it and put it into this document term matrix that we can now easily use for analysis it's almost just like having numerical data at this point it's very easy to cross so at the end of the step we created this document term matrix and the way that we do this is within scikit-learn again this is pythons machine learning library there's this function called count vectorizer which helps us create this document term matrix so you can do a little bit of data cleaning ahead of time and then you put it in count vectorizer it creates this matrix for you and then you're even able to remove stock words along the way count vectorizer is a great tool for this so again our goal at the beginning of this was to get our data in a clean store clean standard format for analysis and now you see that we've got it in two standard formats just a general corpus with every comedian in the transcript and also this document term matrix with all the documents the terms all the word counts so to summarize the input into this data this data set down here was how is Ali Wong's comedy different and we first gathered our data we've scoped our project then we put that data into a standard format using data cleaning techniques and our output is a corpus and a document term matrix so let's get into the jupiter notebook okay so for those of you who have a Windows machine what you should do at this point I set it in the steps but you want to launch anaconda navigator so I open anaconda navigator and then launch Jupiter notebook and then within the browser that opens up you want to navigate to that to the first notebook data cleaning and then if you're on a Mac you can just go to a terminal and then type in Jupiter notebook and then you see this window pop up and if you can navigate to the folder you can see all of the duper notebooks here so at this point I just wanted to point out this extension IP ynb that stands for interact I found notebook so it's been rebranded as Jupiter notebook but you'll see whenever I talk about a notebook it's anything that has that extension dot I py + B so we're gonna go to this first one here called data cleaning okay has anyone here used Jupiter notebooks before okay a lot of people great so one of my favorite shortcuts in Jupiter notebook is shift enter and what shift enter does is allows you to run a cell so at the very beginning we have our introduction here our problem statement and I'm doing shift enter the whole time so you're able to actually run the cells so I'm going to walk through the code here of what's actually happening so for this first section my goal is to get the data do you guys remember what the library is were that I used for this section to get the data requests and beautifulsoup yep those are the two most important ones for web scraping so what I'm doing here is I am first importing requests and beautifulsoup and then here I'm creating a function that will pull the transcript data specifically from the scraps from the loft website so if you take a look at that I can go to scraps from the left and the way I scrape data is you can infect an element in this browser here so that is command shift C and then you can't really see it right here but you'll see that behind every webpage there's a bunch of code here and so for me the code that I specifically want is this transcript data so if I highlight over this you can see here that all this transcript data is in this div class called post content so that's how I was able to identify where that transcript data was so if I go back to this function here you can see what I'm doing is first I'm using a request to get all the data from that webpage that URL and then I'm super fiying it next in this step called beautifulsoup basically telling python that this text is actually an HTML document I want to read it as an HTML document and then this part is where I find that specific class called post content which I knew by hovering over that section of the webpage I knew that that section was called post content and then I want to find all the paragraphs in there and pull out the text from those paragraphs and that's it that's my simple web scraper just from this little function here and so at this point I've listed all the 12 URLs that I want to scrape and then I scrape them so this cell here if you uncomment it you could actually scrape it it takes a few minutes so instead of all of you trying to hit this website which is already a little bit slow to begin with I've pickled the files here and then you can just load the pickled files here all right so once you've loaded the pickled files you can see that I've created a dictionary here and in my dictionary every key is a comedian and then every value is the transcript anyone have any questions so far okay great so that was the data gathering step so looked really simple there but that actually took me a whole day to figure out because I was trying to figure out like where on the website all the state content was and how to write it in a short format so that it does take a while just want to put that out there okay so the next step is to clean the data and so what were some of the things you can do to clean data again that are not that I'll hide there yeah remove words with numbers lowercase punctuation right so these are a lot of the common data cleaning stepped up here and then there's some more data cleaning steps for later as well that I'm not going to talk about today but things like limitation and stemming so what that means is you can take words like driving Drive drives and it knows that those are all the same words and I could group them together parts of speech tagging diagrams and so on there's a lot you can do but you want to start simple and so we're gonna start here with just these common data cleaning steps so I always like to keep looking at my data to make sure that it looks right so I'm just gonna again see that I have this dictionary called data I've called data all the keys are the comedian's and then all the values are the transcripts and so the next thing I'm going to do is I'm going to create a function that takes this transcript text and puts it into one large chunk of text so right now if you remember when I scraped the data every the data was split into lots of chunks of paragraphs and so I put that all into a lip so I had a list of lots of text which is kind of tricky to deal with with text data a lot of times you just put it into one giant string it's a lot easier to deal with so in this case I'm using this function to put all that data into just one giant string and that's this part here so now I've created this data set that has a key of a comedian and then the value is instead of a list text is just one giant string of text and at this point you can keep your data in this dictionary format or what I like to do is I like to put it into a data frame so I'm very used to working with pandas data frames so what I'm gonna do here is I'm gonna import pandas and pandas has this nice function that allows you to take a dictionary and make it into a data frame so at this point if I run this you can see that I had that dictionary of of comedians two transcripts and now I've been able to put it all into this nice data frame and this was one of the format's that I wanted which format was this I think I heard a corpus so at this point this is not the document turn matrix just yet this is just the raw corpus right because I have just all of the transcripts so my column is just transcripts so now yeah the document turn matrix this is just a corpus okay so that was the the easy thing to create was just the corpus so the bit harder thing to create is that document term matrix where instead of every column being so in that case my column was just a transcript and I want to create a different column for every single term so the way I'm gonna do that is I need to first clean the data remember we have to use regular expressions to clean that data so again I just like to look at my data to make sure it looks good it does and at this point I'm going to start applying data cleaning techniques to create that nice document term matrix so I'm importing re which stands for regular expression and then string which allows me to get all the types of punctuation out there so I'm going to create a function that allows me to clean my data so you can see at this part I am making all the text lowercase this next regular expression here is a bit trickier so what i'm doing here is if you look at this data up here let's see okay so if you look at this data up here you'll see that there's data in brackets and I wanted to get rid of the data in that brackets in those brackets because those aren't actually part of the routine you can see that they're just sounds and so that's what this regular expression here is doing it's saying anything that's within these square brackets if you have square brackets and there's some character in them then get rid of that whole thing that's what this line is saying what this regular expression is saying is we have string that punctuation which is just literally like a list of punctuation marks and so it's saying if anything is any of those punctuation marks then get rid of it said you want to get rid of this part can you give an example oh yeah yeah I see what you're saying yes so that would be great so I didn't do that but yes that would be great so you'll notice like in this these two sections it says announcer and then it has that other one yes that would be a great second round of cleaning to do okay so going back here though so I'm getting rid of just very easily for a first pass I'm getting rid of everything in square brackets the punctuation then does anyone know what's happening in this one right here the numbers yeah so what this is saying is - D these are all the digits and then W is alphanumeric characters so A to Z 0 9 and then the star is 0 more times so it's saying if you have a number and then there are any letters or numbers surrounding that number then that means that that's a word that contains a number in it and you want to get rid of any words that contain numbers in them so these regular expressions are a bit harder to decipher so I encourage you after this tutorial to go back and look at them to understand them a bit more so right here this is my first round of text cleaning right so it's just those three basic things and then the one extra thing I added there was the square brackets so let's run that and then you can see here that our data looks a little bit cleaner do you notice anything here that hasn't been clean yeah there's some single letters there that are kind of weird the first thing I noticed when I looked at this was some of my Koston disappear and it's because the string that punctuation didn't capture this particular type of quote so then I went back and I like copy pasted these quotes and then I did a second round of cleaning so I added those specific quotes and then I also noticed that these line breaks are in here the backslash ends are in there so then I did that as a second round and actually those square brackets I mentioned I actually didn't add those in to my day to clean this up until like three days after I did this analysis and I realized it actually helped my case but it helped me better more if I added that in there so there are always more ways for you to clean the data but I'd say like two rounds is a good place to start so if you apply that second cleaning step I would say this is looking cleaner there are these like weird EW io things there there's things that the announcers saying but it's pretty good to start so we're gonna move on from here so this is what I mentioned here this could go on forever just stop at some point and move on with your analysis at this point we've organized the data into a corpus and a document term matrix and we're going to pickle them because we're going to use them in later files so first we have this corpus which is just every comedian and the transcript and then you see down here I'm going to add the full names next to it as well because I want to use that for my data visualizations later and then I'm pickling it so you can see Corp this topical I'll use that later and then there's also this document term matrix so this is where we're going to use count vector right here from the scikit-learn and the way they use count vectorizer is you first instantiate a count vectorizer object and what you can put in there is stop words so if I say stop wars equals English it automatically knows that I don't want those stop words as columns which is really really helpful there's a ton more that you can do with count vectorizer you can even say I want to include by grams as well so anything that's one word I want to have that as a column any combination of two words I want that as a column as well so it's super powerful if you another trick that I like in do print notebook is if you do shift tab know here let me run this first if you do shift tab you can see all the documentation behind a function so if I expand this you can see there's a ton of thing you can do in here you can strip the accents it defaults to making everything lowercase I mentioned the Ngram range here they're just a lot you can do in there so after you create that count vectorizer you can fit it on to your data and we're specifically fitting it onto our transcript data and then these two lines are a bit trickier but you convert it into an array and then you label all the columns but just know that if you're you can just copy and paste this this code for a future use the main thing you have to know here is you have to specify the stop words and then you have to put in your clean data text data here and at the end of the day you get this term matrix so remember every document and all the terms so you can see that this is a pretty pretty big matrix it has seven thousand columns so if you include things like diagrams those pairs of words this will really blow up but I found for that a lot of analysis it's really useful to include those that at the second pass because then you'll find really meaningful things in your data so at this point you can pickle it I'm going to pickle this for later use and then I'm also going to pickle two other things I'm pickling you data clean and data clean is this data frame from up here so you can see this is before I put it into count vectorizer form it's just the clean data where I removed all the punctuation so I've pickled this for later use because we're gonna use it for one other thing and then at the very end I also picture pickled the count vectorizer object which we're going to use for a later thing as well so that's it there's some additional exercises you can do on your own after this tutorial mmm-hmm so what's happening at this just sterilizing it and so it's just think of it as you're taking that object and you're just saving it as a file on your computer so that later you can just call that and then you can open it up in the next document does that make sense so like if you see that I've pickled a file here so once I ran that pickle step you can see that it created this pickle file so in the next notebook I'll open a pickle file and then you can see that the object pops up again so pickle file is just the easiest thing you can create like you can also maybe like compress it in some way but pickle files not compress at all so it's just like a very simple way to just dump that object into a file that you can use for later yep it's just it just stays there so okay how's everyone feeling pretty good okay so then let's move on okay so we've cleaned our data so the next step is ETA and for EDA the input into this step is your text in some standard format so again who created our corpus and our document term matrix and then the goal for EDA is to summarize the main characteristics of the data preferably in a visual way so if you remember from our simple example that's when we created the scatterplot and we were like okay there's definitely positive correlation and also we see an outlier and then the output of this section is just to see if our data makes sense or not so you might have heard the term or the phrase garbage in garbage out so it that means if you put in garbage data into your model even if you have the coolest model it's not gonna be your results are not gonna be good so you have to do the step to make sure your data is actually good so if you think about our data set with all our transcripts what are some ways that you think you can explore that data perfect frequency counts like word counts anything else nape is a great idea they seem like the actual proper nouns of these what's that sentiment I would I would I would make I would say that's like a little bit more advanced analysis but yes these are these are great ideas like you just want to get an idea of what what's going on in your data and so I thought of three things that I want to get out of this data so the first thing I'm gonna do is I'm just gonna look for top words that's the most obvious one so for every comedian what are the top words that they use right the second thing I thought of is taking a look at the vocabulary so maybe some comedians have a larger vocabulary than others that might be interesting to look at and the third thing is the amount of profanity so I didn't realize that this would be really a thing until I looked at the top words and there's a lot of swearing it's almost like maybe I should see how much comedians swear so those are just some things that I decide to look at this is really up to you whatever is interesting to you that would help you figure out the data make sense or not would be good to do here so how did I find the top words well the first thing you want to do is figure out okay we have our data in two standard formats right the corpus and the document term matrix which one do you think would be better for figuring out the top words yeah the document term matrix right because you already have everything in this every word as a column so it's very easy to aggregate your data at this point so the whole reason we did all that data cleaning and standardization was so our life would be really easy at this point okay so we got this data the next thing we have to do is aggregate so how do we aggregate this data to find the top words perfect you have to for every comedian select the columns with the largest value basically like sort across the row and then find which one which words have the highest word counts okay so we've got got we've got the data we've aggregated the data and now we have to visualize it in some way so let's say we found the top 30 words for every comedian what's the best way to visualize that word clouds okay so it's funny because where I teach half of us love word clouds and half of us hate word clouds and I love word clouds so I never expect people to say word clouds when I'm teaching five so glad you brought them up because I really like word cuts so it's just it's a great way to visualize text so you can see here this is Allie Wong's word cloud and here's John mulaney's word cloud and so what we'll do in the notebook is I'll show you how to create word clouds for all the comedians yep what's that sure you can do like a bar plot so something like more classic words are specific to text but yeah you can create like a bar plot or so the bar plot is the easiest if you're comparing two words you can do a scatter plot okay so you visualize this but what was the whole point of this why are we visualizing this what's that we want to find the top words do you remember why we want to find the top words yeah we want to know why she's different from everyone else we want to see if these results are actually making sense so when I first did this analysis I didn't do as much data cleaning as I showed you and when I was getting these work these word clouds like they did not make sense at all they just had like very common words in there I hadn't removed stop words and so they didn't look very good so it took a lot of cleaning to get to this point where it made sense so that's where ETA can help so you look at these word clouds and you're like okay do they make sense that they don't make sense and you have to go back and have to do more data cleaning until this part makes sense yeah yeah that could be the case so I'm not sure the exact logic behind the word cloud package but it has to do something with that I'm sure you can tweak the parameters okay so I mentioned the last thing you want to do is you just want to take a look at the visualizations and future data make sense and figure out if you have to clean your data somewhere and then also you can get some initial findings from this so how are the comedians differ from each other well you saw that Ali Wong let's see oops sorry guys you see that she says get a lot and then John Wayne talks about Bill Clinton the line is good so those are some ways you can see that it's different mmm-hmm it is so it is a bigram that I detected okay so at this point these are the EDA steps that we followed so we figured out what format we need the data in so we're finding the top words we need the data in the document turn matrix format then we had to aggregate that data visualized it and then see if it made sense so what we're gonna do in the notebook next is we're gonna do this for three things so top words which is word cut which we create word clouds and then look at the size of the vocabulary for every comedian and then the amount of profanity so the way I'm gonna do this is I'm gonna use word cloud specifically in its the word cloud package and then there's also map pot Lib which is the most popular data visualization package and Python so one thing I want to know is within the readme file I updated the readme file about two days ago and there's an extra step you have to take to download word clouds so if you look at that readme file there's a Conda install step for you to actually download word cloud because it's not included with the base anaconda navigator okay to summarize what we did for this section the input into this section was our standard document formats what we're gonna do here is we're gonna do a ton of EDA to see if our data makes sense and then the output is to find some big trends in the data and see if they make sense so do the notebooks alright I'm gonna go to the second notebook here and what we're gonna do here the first thing is I'm gonna read in the pickle files so someone asked earlier about how this part works so within pandas what's really nice is if you use pickling in pandas like you can pickle a dataframe it's really easy to read it again so if I import pandas you can see within pandas there's a reed pickle function and I can specifically use that to read in a data frame so here I've read in my document turn matrix but one extra thing I've done here is I've transposed it so now it's a term document matrix and the reason that I did that is because now my aggregations will be a little bit easier so I talked about like I wanted to find the top words for every transcript well if I didn't transpose it it it's harder to do things across rows than it is to do things over columns so that's the reason I transpose it here I just wanted to mention that so my data here is now a term document matrix and so let's say I wanted to find the top 30 words for every comedian so what I've done here is for every comedian I've looked in that data frame and I see for a Lee Wong these are her top words and how often they occur and so I'm going to do that for every comedian so these are the top 15 words said by every comedian so if you take a look at this what do you think about these top words yeah people say like a lot and so like is not part of the standard English stock word list but if everyone says like a lot it's not going to be really meaningful in your data right and so what you can do is you can actually add these common words to your stock word list so what that means is in my document turn matrix I'm gonna remove this term and in your case like so what I want to do here is I'm going to for show you so I'm gonna first pull the top 30 words for every comedian and then I'm going to use counter which is super helpful so what's going to do is going to look at my list of the top 30 words for every single comedian and then I'm going to see how many documents contain that word so at this point you can see all 12 comedians have like as one of their top 30 words all 12 comedians have I'm as one of their top 30 words so these words aren't really meaningful so I'm just gonna set a limit I'm gonna say if over half of the comedian's have one of these words as their most common words I'm gonna say it's a stop word I'm going to add it to my stop word list so you'll see here these these words are in a lot of the words from the top comedians so I'm going to here what I'm going to do is add them to my stock word list so I'm gonna walk through you you through a couple steps here so first I'm importing count vectorizer which we used before I'm also importing text so text is what actually contains the stop word list what I'm gonna do here is I'm going to first read in the clean data so if you remember that was my corpus that had all the punctuation stripped and then I'm gonna add new stock words so here what I'm gonna do is there's already Engler stock words out there and I'm gonna Union I'm gonna add to it my new stock word list and so now I'm going to recreate this document term matrix with my new stock word list along with my clean data from my last notebook and then at the end of that I'm going to have a new count vectorizer object that includes my new stop words and a new document turn matrix that excludes those staff words as well so I run all that at this point I have a new document term matrix that doesn't have those stop words in anymore so at this point we can make some word clouds so this part will only run if you've done this additional install so if you're on a Mac you can in your terminal if you're on Windows machine you can do this in your anaconda prompt and you just type in Conda install Conda Forge word cloud and so I've created my word cloud object and then to actually plot it you can see here so these are the word clouds for all the comedians so what can you tell from this there's so much swearing like I was so surprised with the amount of swearing which is why I added the profanity step yeah there's a lot of swearing and what I found from this remember my original question is how is Ali Wong different I noticed that she says okay a lot I see okay lodge talks about her husband and I talked about her as my husband I guess I think this is all funny so if I look at this this makes sense to me and I've come up with a couple of findings here okay so the next thing I'm going to do is look at number of words so my goal here is to see how big of a vocabulary everyone has and so the way I did this is first I went for every comedian I looked at the number of unique words that they used and then the other thing I did I was thinking maybe I can look at the speed of each comedian so I went on IMDB and I looked at how long everyone's routine was so I added that here so this table I created has every single comedian the number of unique words they've used the total words that they use throughout their routine how long that routine was and so I could calculate this words per minute so the two columns that I care about are the unique words that they use the number of unique words that they use and then also the number of words per minute that they say so now that I have this table what's the way I commission yep so I just had the time of the show but you're absolutely right like if I want to be more specific I would need to know the specific time that the community comes on stage I went on IMDB and won and I just like manually like went to each commedia and like looked at their run time yeah I could add it to my scraper too so ok so at this point what what can I do like this is kind of hard to read so yeah that's a great idea I could take the average of the unique words and then see who's above or below that I don't have any other ideas oh do some clustering - we'll talk more about that later so for me I like to visualize in very simple ways so what I've done here is I've just created some bar plots this is pretty simple a great idea would be to add that average piece like a line here or a color here for average but you can see that Ricky Gervais and Bill burr have a big vocabulary and Louie CK and Anthony Chile's Nick don't and here are some things about words per minute so I did this and I found that my analysis wasn't really interesting but it's something to try and not finding anything is something in itself so I just did this analysis who didn't find it very interesting and so then I went to my last bit which was looking at the amount of profanity so when I looked at all the word clouds I found that the S word and the F word were said a lot so I was like okay why don't I just create a scatterplot to see how often those things are saying so that's what I'm going to do in this section I'm just going to create a scatter plot of bad words so this here this is number of f words and the best words and what can you tell from this Dober used a lot of s words and just profanity in general yeah I was really surprised I don't know about I don't know Joe Rogan but he's using to F words per minute it's comedy which is crazy and then on the other and you have Mike Birbiglia who has zero swear words in his whole routine which I thought was pretty interesting so just from eda I haven't done any fancy analysis you can see I've come up with some interesting things here I do think my profanity pot is my most interesting thing from my EDA section so yeah that's a great question so I didn't do any stemming so what I did was I looked at the most common words and then when I looked here I saw that there were variations of the F word and I just put them all in one if you you could do stemming although I don't know what the summers in NL Tek include profanity so might that's probably thought they had to manually okay so the side note I wanted to mention here is the whole goal of EDA was to take an initial look at our data and see if it made sense and my conclusion is it does for a first pass and yes there's always they make things that could be better but it's good to get something done quickly and so my all my students know my data science motto is let go perfectionism because I always have to do that come in they want everything to be absolutely perfect before they move on to the next step but I'm like you got to move quickly so just do something you know there are things wrong in it but it's okay so my data science and my life motto is let go of perfectionism I just wanted to throw that out there okay so we're at about the halfway point and we've been here for a while so I want to get you give you all a break so let's get back together about five minutes at is it 11th we have now completed the EDA step where we've explored and visualized the data and we've gotten a couple interesting findings from it as well so this last part now second to last part is the most interesting part so this is actually looking at all of the NLP techniques so now we're going to be looking at the NLP techniques so our input into this is we have clean data and we verify that the data makes sense and now we're going to do these more advanced analytics techniques including sentiment analysis topic modeling and text generation and our output is going to be additional insights to help us answer our original question so these are the three techniques we're going to be going through starting with sentiment analysis so who here has heard of sentiment analysis okay who here has applied sentiment analysis okay okay that works so the input into sentiment analysis so member we have our corpus and we have our document term matrix for sentiment analysis it's going we want to input in our corpus and the reason for that is because remember when we had our document term matrix it was in a bag of words format so what is a bag of words format again yeah it's a collection of all the words an order doesn't matter but the thing is with sentiment analysis order does matter right so if we have the work that's termed great and we have the term not great those are going to mean very different things so we want to preserve that order so our input here is a corpus and what we're going to use for sentiment analysis here is a library called text blob so earlier I talked about NLT K and NLT K is the library that everyone uses for natural language processing in Python but text blob was built on top of animal TK and it makes it a lot easier to use and it also includes some additional functionality such as sentiment analysis also it like fixes typos so it does all those things but I would say it does it in a very very basic way so keep that in mind so for the output of this for every comedian what we're gonna do is we're gonna look at their entire transcript and then we're going to give that transcript and overall sentiment score and an overall subjectivity score so the sentiment score is how positive or negative they are and then the subjectivity score is how opinionated they are so let's take a look at what this code looks like so this is literally all the code for a sentiment analysis so you would import text blob and then you would say text blob of some text you do dot sentiment and then you get this as an output so polarity is a number between negative 1 and positive 1 and so a polarity of 0.5 would mean I love PI Ohio is generally a positive statement which is right and then subjectivity is how opinionated you are about something so that's a number between 0 & 1 so the more subjective you are the more opinionated you are so love is an opinion so you can see that that's a bit higher on that scale so great let's jump into the notebook but wait don't jump into the notebook so I mentioned this a little bit before but if you jumped into the notebook at that point then you would fall into that danger zone I talked about because you only understand at a very high level what's going on but before you import and use some model i module I highly encourage you to understand what's going on behind the scenes so I'm gonna go into a little bit more information of what's actually happening so the way that text blog sentiment works is there's this great linguist Tom de smet so what he's done is he's gone through all these words in the English language and manually labelled them as positive sentiment negative sentiment and so on and this might seem kind of crazy but there are a lot of linguists out there and this is what they do they've created these amazing incredible databases of the words of the English language so you might have personally called word Annette so word now is created by this group of researchers at Princeton and what they've done is they've said these words are all very similar to each other these are the definitions of these words and so on so they've actually kind of mapped out the English language manually and so there are a lot of sentiment I would say like lexicons out there so people who have gone through all the words researchers linguists who have gone through all the words and labelled them so the one that texts blob uses is a specific one that was labeled by Tom de smet and the reason I know that is because I went to github and like read some of the documentation and it would tell you this so what does this mean so let's take the word great so if you look into that big dictionary from the last slide if you just pull out some key features here you'll see that the word gray actually shows up in that list four times these are the word Annette IDs so again word Annette is this big dictionary that was created by Princeton and a lot of people are referenced in the NLP world and so they'll have different word net IDs and everything in the word net dictionary is labeled with parts of speech so JJ stands for adjectives and then different meanings of that word and then you can see here that this specific linguist has labeled these as a certain polarity and a certain subjectivity and so how do you think text blob aggravates all of that so overall if text blob saw the word great how do you think it would decide what to do you would think it does something fancy like that it just literally takes the average yeah so this was saying like text blobs features it sounds really fancy like fixing typos and sentiment alysus but it's very very basic but it's okay because it's a good first pass so these are all good things to know before you just use the module so let's look at some examples so if I just did text blob of great you would see that that polarity pointing is just the average of that column and the son subjectivity is just the average of that column if I did not great what do you see as a change here polarity what's changed it's negative so if you look at the documentation what's actually happening is whenever it seems not before a word it multiplies the polarity by five so this is a rules-based approach to sentiment analysis if you see very before a word what it does is multiplies both those scores by 1.3 and then it caps it at 1/4 polarity so again rules based approach and then if you do I am great you can see that that has the same score as that first line because I am don't have any sentiment yep I'm not sure but I would I would hope so I'd have to look at the actual code to see what's happening but that makes sense like an explanation mark would and some meaning to it so so this kit kind of gives you an idea of what's happening behind the scenes and it makes this when you do dig deep into what's happening in the modules you don't just take the module to be magic that's great that's great there's the answer to your question thanks for answering okay so at the end of the day what text blob does is it first if it sees a word it will average all the subjectivity and polarity scores and then at the end of the day will look at your entire text and then average together all those scores so not the most okay well before I say that so the output will be every comedians gonna be assigned one polarity and one subjectivity score and it's not the most sophisticated technique like I said but it's a really good starting point because so this is a rules based approach and you can also do knowledge based techniques so this is more if you're into data science and you know about classification techniques that are out there you can use one of those classification techniques as well so what how that would work is the most popular data set to do sentiment analysis on is this movie reviews data set so for every movie review it has a bunch of text and then it's labeled as this is a positive review this is a negative review and then you'll use some more advanced technique like naive Bayes or logistic regression to see what's the combination of words that that gives a movie a positive review and so that's a more advanced technique that if you're interested on and I highly recommend you look into after this yep yeah that's a great question so with naivebayes the major assumption there is that all your features are independent so it's saying like if you're trying to predict how good you will do on a test all the things that could predict how well you're going to do on tests like number of hours studied or how high were you on your last test like that those are not related to each other so typically if you use a standard algorithm that's not naivebayes that independent have to assume that and sorry let me just summarize my thoughts so if you're using something like naive Bayes it assumes that each of those features is independent so that they're not related to each other at all and with other types of analysis you usually can't make that assumption but with text data it is okay to assume that those are independent and so it just ends up working very well with text data any other questions alright ok so now that we've kind of understood how text blob works let's jump into the notebook all right so for this we are going to go into notebook three and for this you're also going to have to do an extra Conda install so again instead of reading in the document term matrix we're going to read in the the actual corpus that has all of the words in order so again it's very easy to do text blob all you have to do is do text blob sentiment and you end up with those scores so again that was very easy to do just encourage you to figure out what's happening on behind the scenes and then if I map this this is what I see so on the x-axis I have positive and negative on the y-axis I see facts and opinions so what can you tell from this graph yeah he's pretty negative yes yes and so if I'm if I want routines I want to see comedians similar to Ali Wong who what might I recommend yeah like whoever's close by right family activity and so on so there you go with sentiment analysis so that was pretty simple and so another thing I did was I wanted to see the sentiment of a routine over time because one of my friends actually did this analysis on Disney movies and saw that like there was like a common pattern throughout all Disney movies gets like starts very sad I'm gonna get sad they're very happy I want to see if there was anything with Tommy routines than that so what I did was I took every data every transcript and then I split it into 10 pieces of text and so I created this new list and it has 12 it has a different element for every one of the 12 comedians and then you can see every comedian their text has been split into 10 different parts and what I want to do is calculate the polarity and the subjectivity for every single piece in there and so if you plot that out for one person you can see this is their sentiment over time and then at the end I created this sentiment all these pots for all the different comedians so anything you can tell from these pots immediately yeah I mean maybe I mean this yeah there's like a peak here what's up yeah that's kind of why I took out at this it's like Ali Wong is pretty consistently positive and so I wanted to see it are there any other comedians that are pretty consistently positive timely again who seek a microp eagle yeah and so with this extra sentiment analysis I was able to find some extra interesting things about my data so here's some additional exercises for you as well alright so that was sentiment analysis so the next thing I want to talk about is topic modeling so this is probably the most complex thing we'll talk about today but I'll try my best to explain the idea so for topic modeling the input into topic modeling is a document term matrix because order doesn't matter here so again what topic modeling what we're trying to do is we have all these documents we want to see what are the topics that are being said in these documents and so we don't want we don't really care about the order of the words here we just want to know what are the words what's the bag of words that's in each topic and then the library we're going to use here is called Jensen so Jensen was built specifically for topic modeling and it's it it actually has different ways of doing topic modeling but the most popular one is called latent DHE allocation which is called help which is most commonly known as lva and we're also gonna use NLT cave for some additional parts of speech tagging which we'll go through in the notebook and then our goal at the end is to find the various topics or themes across the comedy routines so I'm talking about topics and pretty vague way right now so I want to show you an example before I go through that so this technique is called latent dere Schley allocation so latent just means hidden and dere Schley is a type of probability distribution and that's those are the basic ideas that you have to understand this so I'm going to talk about probability distributions coming up and then hidden it means that what's happening is I'm looking at the text in terms of probability distributions I'm gonna find hidden topics from that so let's go through a concrete example because I think this will make it make more sense so let's say I have these five documents here if I applied Lda on this set of documents what would happen is I would get an output that looks something like this so would say that document is a hundred percent topic a this is 100 percent topic B this is split between topic and topic B okay so the way to think about this is earlier I was talking about latent trait latent means hidden so these are topics that were hidden in these documents before that I couldn't find but using Lda I'm able to find those topics that's the latent part dear Schley is all about probability of distributions and the way to think about this is you see that this document here is a mix of topics topic and topic B or Vance you were to think of that is this document contains a probability distribution of topics so when I say probability distribution I just mean it's a mix of these words so again this document is a mix or probability distribution of these topics yep great question that's all go through next okay so at this point what you would do is so I talked about this document ends up being a probability distribution or a mix of these topics in addition every topic ends up being a mix or probability distribution of these words so this is what you would get it as an output to Lda you would see this topic contains this much about bananas and kale this topic contains a lot of words about kittens so what do you think topic a might be about food right and what would topic B be about animals so now you can go back to your documents and say that documents about food that's about animals this one's about food and animals so that's the whole idea of topic modeling I have these documents I don't know what the topics are but then I can figure out what the topics are and then see for every document what's the mix of topics and then for every topic what's the mix of words okay so with Laettner I shall do your slate allocation it's all about these probability distributions or these mixes so again just to reiterate I said this a couple times but every document is a mix of topics and then every topic is a mix of words so you can see here this food topic it has a lot of things about bananas and kalynn frogs and then animals has a lot about kittens and puppies so that's a general idea behind Lda and the way that it actually figures out which what the topics are is it goes through these steps so first is again you want Lda to learn about the topic mix in every document and then the word mix in every topic so within each document what are the topics in there within each topic what are all the words that are in there so the first thing you do is you choose the number of topics that you think are in your corpus so people typically start with two and then you just build on from there and then what it does is it goes through every single word in every document so here's my example up here let's say we had this document it goes through every single word and it randomly assigns it a topic so it might say I is part of topic a like is very topic B and so on it just makes the random distribution of topics okay and then this is the most complex part so this is where it goes through every word and it updates those topic assignments so what's going to do is it's going to look at a word so let's say we're gonna say bananas let's see what it looks at bananas and right now banana is assigned to topic a sir sorry banana assigned a topic be so banana is signed to animals does that sound right no right so you want it to update so this is what how this is how Lda works it looks at every word and the topic Simon so like bananas let's say it's the sign to topic B it's gonna look at that and say should I reassign that to topic a or should it stay topic B and the way it decides whether it should reassign it or not is oh look at that word and it'll see how often does that topic that it's assigned to occur in that document so right now banana is assigned to animals how often does animals actually occur in that document that's the first thing it looks at and then it looks at that word and see how often is that word occur in that whole topic overall so how often does banana occur in the topic about animals and in those cases those probabilities are both pretty low and so it's saying okay well I should reassign that to the other topic then so I'll read ate that topic assignment the topic a which is about fruit or food so that that's like the bulk of what's happening so what's gonna what's gonna happen is it's gonna go through every single word every single document and then try to figure out which topic should be part of it might end up in both great question it would depend on your corpus but it could absolutely be part of those so what happens at this point is it goes through multiple iterations of this so it does the random assignments I'll go through every single word in the whole corpus and that will be iteration one and then all the topics will be slightly better and then we'll go through it again iteration two so you should start you try like probably a couple dozen iterations and then you'll see the topics evolve and eventually the topics will start making sense hopefully if they don't make sense then you're gonna have to some more data cleaning so this all seems a very complex but luckily Jensen does this whole part for you so all you have to do is you have to say how many topics you want how many iterations you want to go through and do the interpretation so our input is our document term matrix our number of topics and our number of iterations Jensen will find those probability distributions that are best and then the output is you'll see for every topic what are the top words in that topic and then you have to figure out do these make sense or not and so this here is probably one of the most famous implementations of topic modeling late into your slate allocation it was created in 2003 paper by David Bly and also Andrew aim if you've heard of him from the Coursera so they all created this so this is one way to do topic modeling which is specific on text data but you can also use general matrix factorization techniques so those of you who know a little bit more about linear algebra so that's basically you take a big matrix and you can decompose it into two or three matrices and that's another way of finding topics because you take let's say like a bunch of columns and when you decompose it you can squeeze that down into smaller columns and and up being your topics so those techniques are called LSI which is essentially a singular value decomposition for text you know what that is and then nmf which is non non-negative matrix factorization those are also included in Jensen so you can use that as well but I think Lda is I like how specific to text so that's the one I like to teach okay so let's run through topic modeling in the jupiter notebook okay so with topic modeling you'll see that this is absolutely iterative process so I have attempt one two and three here because my topics don't make sense that takes a long time to try to make them make sense so what I'm gonna do here is first I'm gonna import the document turn matrix because again order doesn't matter so because we're working with this bag of words model and then all of this here like these three cells here these are specific to using Jennsen so I'm gonna import some modules from Jennsen I'm also going to import so I've comment this out for now you can also import logging so logging will help you debug so when you create a topic model it'll log what's happening at every step and you can actually look at the hyper parameters and tune them some more so that's for you to look at later and then at this point for these two cells what I'm doing so for these couple cells here what I'm doing is preparing for Lda so here the first thing I have to do is for Lda it actually requires a term document matrix so and so the document turn matrix we're gonna be using we've been working with its transposed so i've transposed it to a term document matrix and specifically for Lda you have to convert it into a sparse matrix first and then into a specific jensen corpus so a sparse matrix is basically so here we have this matrix lot of zeros it's not the most efficient way to store that matrix what a sparse matrix will do is saying for this row this column there's a value in there and here's the call about you so it stores it in that format and then that's just the format that Jenson reads it in and then Jenson also requires a dictionary of every single term and their location in that document term matrix and you can get that from the count vectorizer so if you remember our count vectorizer it was a big document term matrix it remember it has those vocabulary items thing in it that has for every single column what's the ID for that column it's just something that LD it requires so at this point this is so we've done all the setup this is actually where we're running l da so L da here you can see it requires four things as input so the first is our corpus which is our in this case it's actually our term document matrix but they call it a corpus ID to word which is our dictionary of for this term where is its location in that term document matrix our number of topics which will start at 2 and then our number of passes which will start at 10 and the reason I'm starting at 10 is 10 is actually not that many passes to make all those updates but just for this tutorial I want to make sure we have some time to run through all this so if you run those three it's gonna take a while because it's going through every single term in your entire corpus and making those updates so at the end of the day this is what's happening so you see these are the two topics that emerged so for this first topic these are all the words there and the second topic you see here anything interesting from this yeah absolutely so you start to see those overlaps and those topics so when I look at this I don't find this super meaningful and so at this point I did a lot of work to try to figure out how to make this better the first thing I always do is I try to increase the number of topics but that didn't help me too much I saw so much I saw it like so much overlap between the topics and I was like I wish I could do better so the second thing I did was I looked at nouns only and so this is a very popular trick that you can use and which 104 nouns it doesn't it doesn't specifically weight nouns yeah it doesn't it doesn't do that automatically because it looks at all the words the same but yeah so it's very yeah yeah yeah which is which is why we filter by them so it's like the it's like the easy clean way to do exactly you're saying like look at the topics by nouns so the way I'm doing that here is from NLT kay I'm gonna import this parts of speech tag what I'm gonna do is for my text I'm only gonna look at nouns and the way I know that nouns is NN is because you pen has created their linguist have created this tags all these tags so you see there's there's a lot of manual work here like when I took my first text analytics class that's when I learned that it's not all magic it's a ton of manual work people have spent a lot of time labeling this stuff so let's do this just for nouns so at this point I'm gonna run through this because they're running a little bit low on time but you'll see here that once I filter out all the nouns or if I filter it so it's just the nouns some of these words are a bit more meaningful and so if I scroll down here you'll see these topics a little bit more about nouns they're getting a little bit more meaningful there yeah I have Bible talk a little bit about Spacey at the end but you can absolutely do you Spacey for that as well I could see that cuz Spacely is like the newer version of adults he can he could do a lot of the same things but better and faster so that's probably if he could remove the apostrophe thank you I guess they've tagged it yeah they've tagged as a noun no I don't I don't really see it clearing up either so I didn't see it really clearing up either so the next thing I did my attempt three was including nouns and adjectives it was like not just about nouns but like how do people feel about those nouns so at this point I've included nouns and ends and JJ's again I know that because it's from the you you penned tag set and then I ran through this a couple times and this was starting to make a little bit more sense to me but this point I'd spent like a day on this I'm like I think this is the best I'm gonna get within a day so at this point I decided okay out of all my topic models I think nouns and adjectives worked pretty well and I settled on four topics because it was as good as I could get for that day and so my final Lda model what I did was I ran my topic model for four topics and I ran it for a lot of iterations and the reason for that is so that you've reached that steady-state over time and it can get more fine-tuned topics so what I like to do is try a lot of things for a few iterations and then once I Zone in on the best one I'll do that one for a lot of iterations do you have another question yeah these are the four topics I and yeah so like the first one was like talking about mom and parents second one was talking about husbands and wives this was talking about guns and then this one is a lot of profanity and so those are the topics that I ended up with definitely could be better but the first path yeah absolutely yeah you can even absolutely like just tag things with topics as well so this you find useful when you have really large documents that you have like no idea what they're about and then this can pull out so so LDN you when you would always do as unsupervised or if you know what the topics are you can't go the supervisor out and say you already know what these topics are and so then you can just see like what words are times tend to be part of that topic not using all da yeah so all the topic mining techniques are unsupervised include a matrix factorization all unsupervised yep oh I think that so within the error message it says you have to download something or import something just add that import to the line above it I think there were two that you had to go through so there's two there's only two so it's you've imported NLT kay but there's a lot more that you have to do and I'll TK has a ton of stuff in it and there's more you have to download yeah you have to do that two times in the cell before and then it should work okay so yeah so so yes Jenson was able to figure out this topic contains all these words and then you have to figure out what those topics are so it's so it's always one thing about as much as a compression as like a mix of words or a probability distribution so it's like figuring out what the most common words aren't in that probability distribution and then it's like doing a better job of curating that list so the food that doesn't come from Jensen you have to label it as food yeah okay so at the end of the day the next the other thing you can do is for now that you've come up with the topics for every comedian you can see what topic they talk about and so it just ended up happening that every comedian was only had one topic they were assigned to but a comedian can actually one document one comedian can actually contain multiple topics but in this case every comedian is assigned one topic and you see that Ali Wong is part of that second group that talked about husbands and wives along with Mike Birbiglia and John Mulaney all right yeah which is pretty amazing with just like a short amount of time with this analysis okay so we have about seven minutes left and I want to I'm not gonna go through this one in detail but I think it's pretty cool so text generation your input into this is a corpus so order matters right because you're trying to figure out what the next word is and there's nothing special and pi thought that we're gonna do here for text generation but the output would be a new comedy routine so this is purely for fun and the way you can do this is using something called Markov chains so Markov chains are basically a way of showing how states change over time so the main assumption of Markov chains is that the next state of a process only depends on its previous state so instead of going through this example I will just go through the text example so with the text example how it works is it takes every word as a state and then it looks at the next word and it says how likely is this going to be the next word based on this word and then for this word how likely is this gonna be the next word based on this word and then you can see it even wrap around like if this is a word there's a 10% chance that this is gonna be a the next word so the idea of Markov chains is the next day is only based on the current state so with in Python you can put this make this a dictionary and you can just say the keys are the current words and then the values are a list of the next words so it's very easy way for text generation it seems like a pretty simple model it is and you end up with some really funny comedy routines at the end so I'll let you discover that on your own in the notebook but if you want to do this in a more complex way then you would have to look at something like deep learning so with Markov chains you're only looking at the previous state but with deep learning you're also looking at not just a prior word but words before that and then with deep learning it also predicts those words and so it includes those predictions as well so specifically for a text generation you would look at long short-term memory okay so we don't have to have to go to that notebook but I encourage you to look at it yourself and then I want to summarize all this by I won't ask you what you learned today sorry we don't have time for a discussion but this is what you did learn today you learned the data science workflow how to complete this end-to-end project all these things here and then my motto and you can think this is what we learned today here in this column and then this is what you can go into next so instead of just looking at a document term matrix look into tf-idf it's the next thing to look at so instead of word counts it weights some of the rare words higher and then these are other data visualization tools we talked about classification techniques like nave Bayes and linear logistic regression more topic modeling techniques and then deep learning I also want to talk about the libraries so we use a little bit of n ltk with the parts of speech tagging but n ltk has a ton of built-in functionality so parts of speech tagging limit ization and so on text blob is built on top of that makes it easy to use Jennsen is for topic modeling and you mentioned Spacey so Spacey is like the next new thing so a lot of my students are using Spacey now but it's a lot faster than these other libraries and I feel like it's it's gonna replace the null TK one day that's just my okay so to summarize this was our question for today what makes her routine stand out and be for all the techniques that we used and so now we can answer that question what makes her a routine stand out though we saw that she talks about her husband and I talked about my husband a lot in my lectures I mentioned him once and then but yeah I have a lot of lessons and then she has the highest s word to s worst ratio which I don't like the F word at all but I don't mind the S word so and then she tends to be pretty positive and what's opinionated like me so who are some other comedians I might like so these people don't say the F word that often Michael beyblade doesn't at all he says no swear words these people have that similar sentiment flow these people talk about similar topics and you can see Mike Birbiglia comes up a lot I just want to mention I finished this analysis on Thursday and then I got this text from my husband as I was looking at the comic I say which one and he says Mike Birbiglia and I was like coding my analysis he is most similar to Ali Wong so my analysis worked yes no but I just that would be so funny Mike Birbiglia just randomly I got that text two days ago amazing right so next steps so try the text generation notebook it's super cool apply this analysis to other comedians and then also try this ir text data yep Oh interesting probably because I don't think there yeah I'm thinking there recommendation engine probably doesn't actually look at transcripts yeah absolutely so I just wanted to finally give a shout-out to my company Metis because they gave me time over the last two weeks to actually create all this we are a data science training company so we do live online trainings where you have a professor and a TA teaching you and then we do in-person boot camps which I'm teaching this fall and that's it thanks so much for coming [Applause]

Channel: PyOhio

Views: 1,034,400

Rating: 4.9311018 out of 5

Keywords: pyohio, pyohio_2018, AliceZhao

Id: xvqsFTUsOmc

Length: 111min 3sec (6663 seconds)

Published: Sun Jul 29 2018