NLP Tutorial 16 - CV and Resume Parsing with Custom NER Training with SpaCy

hello everyone welcome to your new lesson in this lesson I'm gonna really take a very interesting topic it's a resume a are you can say a CV summarization so its resume and a CV Sandra Jason in this we will be taking a PDF resuming our CV and then we will be summarizing that CV to get the name of our son it's a emails and the organization in which that person has worked and it's a skill etc so it is really very helpful for the HR people who receives a lot of the resumes and series and they are not able to go through all the resumes at once so with this summarization or they can go through all the resumes very quickly and they can get the required information and it can be extracted but the one thing all of you should all of you should remember that the collecting the training data is incredibly painful so it's not as simple as it seems okay yeah we are gonna do our today resume a and the CV summarization whatever we do in the data analysis are in deep learning trainings or NLP etc we need to first prepare a data if we are a beginner we get the data directly from the kegels and the other from other websites etc and we start building our model on the already prepared data set so that seems a very easy task but whenever you try to implement something which is not available in the market then you need to first prepare your training data for the machine learning algorithm and this is the where most of the work need to be done collecting and preparing the data is most and most important part of deep learning ok so the data for this resume and the CV summarization is a prepare for around the 200 CV on which we will be training our algorithm yes it's around 200 the series on which we will be training our data set but the 200 a 200 CV and the resumes data is not enough for a large organization you need to get the thousands and thousands of the seaweeds and prepare it manually so that you can train it on the larger data set for a better accuracy so let me show you the data which I have here the currently I have a data in the form of let's say here the train data dot txt if you open this you see here the data so this data I had taken from the internet somewhere avatar not able to find the link from where I had downloaded this data and then on this data written the algorithms in his speci for its Processing's and for the classification so I'll upload this data on the github and he then you can download it so let's say you have this data ok so this data seems the pretty pretty mature bureau but it's not making a sense all right so let me explain you explain it to you in in the parts then definitely it will make some sense so with this whole data you see this is at Apple and in this topple if you notice that there is topple which starts from here okay from this entities all right and it ends here and after after before this you see before this you have this whole later okay so this is the whole later so in this topple this is the first element of this topple and this is the second element of this topple which is in the form of dictionary and then the list of tuples so this is you can say a key of this dictionary into th and then there is list and this list in list is having here these are the component of the list and these are dutiful okay and these tuples are these third index are the second index you can see here the second index says that here the label okay the named entity recognization are a label for this particular data set okay and it is similarly for all other data set so how it is done let's say there is CV it is given in a PDF format a PDF is converted into a text format and then that text is here this is a text format and these are the automatic process a manual process starts from this from this location so these things should be done manually how manually you need to first identify on yourself so that the machine can learn so you need to identify that this task companies work and add so at which company this person has has worked you need to first identify it manually algorithm is not gonna do everything automatically for you so it is identified that okay this person is worked here Oracle okay and if we count the characters from here to here - one two three four kind of that so it will start from 1749 okay the characters or rectal character so it says that here January 2018 - president present senior consultant orator alright so this Oracle word starts from this character position to this character position so it says that this is the companies at which this person is work and there are many mentions of this the Oracle so you need to identify that okay what are the other position of that that what so that you can put that here and then you see here also companies were that there are three mention here then there is a mention of skills you can get those skills from here okay so it has skills and skills starts from this location to this location so the probability starts from this location to this location all right and then similarly companies worked again and if you see here the graduation here so it says that the graduation here then you need to first manually find out where is that getting graduation here in which that person has graduated so it says that this person education is B in computer science auditor Institute of Technology Tamilnadu it graduated in September 2008 to 2012 so you see the graduation year is this one 2012 and this is the so these things you need to do the manually first of all you need to understand this okay and then the finally the college name so the college name here it says that this is your Aditya technology okay and then the location designation except I you need to first manually find out from this text and you need to prepare this kind of double alright if you do not have the data so I'm gonna push this data on github you can get this data and then you can follow along the video and then you can write the code I'm not gonna the push to the code today I haven't found that the people have been iam just making a bundle of the codes and the my videos and they have been selling it at the high price in the market so but that was not the my intention my intention was to distribute all these codes and the videos in the free of cost to the everybody but there are some nasty people who do not like these I mean they take the course and the videos and they bundle it and then it starts they selling it although so so so I had taken some the action so the talk to the few people's the legal people's like the police acceptor to find out those people who are really doing the infringement of the copyrights and and violating it although so let's go ahead and start it so in this what you get here let me show you again this whole data so there you see we have this one let's go ahead and see this PDF if we can see I think this cannot be seen there but anyway let me get it there just a second so the PDF is here let's say ok so what we need to do we need to convert this PDF ok this PDF into the text format and then you need to get these texts and then you need to prepare all those data so let's suppose that we have those data in this the training data dot txt and the same data is stored here it's a pickle format pickle is very useful to preserve its data structure so I also stored it into the pickle format all right now let's go ahead and back to the this code okay so the roughly I will be following this example here speci dot R you just let me open this for you ok so the roughly I'll be following this how do I get training data ok so the training data if you read these it says that you need to prepare your training data manually so you have here the text format let's say in this text format let's say entity is here uber and it's level is organization so it says that this entity starts from the zeroth character position and it ends at the 4th character position and the similarly you see here the level is a person here this Android okay so Android Play expands to the Canada so this is not the person here if you say the entire play okay but if you talk hope out only the androids it these enjoy it's like in the technology the androids like a but in that case you can say this is a person but the actually this is the wrong here if you see here sorry this is the Android P expands to the Canada or you can say now I think now it is a converted to I think Rd Google pay so this would be the organization here and the Android pay expands to the Canada again here the Canada here you see the Canada is here the GP graphical geolocation and that's okay this is the country and the Spotify steps up easy expansions now this is the organisation you see here this starts and ends how it is encoded so this is encoding what is the start character and what is the end characters alright so this is how you do this you can also do this with the rule-based mature to only make something like this but that is not always accurate let's say if you say that let's say if you say that this google is a person sorry this Android is a person then it will even Android operating system you will also take as a person and into RPA will also take as a person enroll base and then you finally prepare your training data do you remember here how we had prepared we had exactly like this okay we had first with the text and then we had the labels okay now there we had our custom label like the companies that work skills graduation year the college names degree etc so all those things so this is how we did and the cord which shall be following there are flea that code is here so they are few changes which I had made in the code to make it easy for you to understand but now you see here this is the training data now this is exactly as we had seen previously now you see here it has the first test data and then finally it has this year into teach from here okay so this is the training data and this training data then we will be feeding here in NLP speci model so we'll first read read they're blanks passing model and then that blanks passing model to that blank space model we will be retraining our algorithm okay all right perfect so let's go ahead and start coding okay so the first of all I am going to load that training lay down so here it says that we have trained underscore data is equal to let's go ahead first run this one so we have imported here speci imported here pickle and the random if you do not have a speci and the pickle you need to install this you can go ahead at my channel KGB talkie and see I have taken many videos on this Pasi and then you can see there how to install in the work with this speci so here we have a trend data is equal to pickle dot load and then finally I'm gonna say that here the open and sorry open train data dot PKL and then finally here we have our V okay so we have here a train data train data then we have here a 0 now you see this is the first training data in this the first training data if you see it has the first text and after that these entity which we had seen in the jumper so these are the things on which we will be training our algorithm okay perfect so let's go ahead and roughly follow this code okay so I'm gonna has to prepare here a function so this is going to be a little a little this is going to take a little time so we have a deaf before deaf we need to first load that model NLP is equal to speci dot blank I'm going to load here Ian model that's the English model and then I'm going to say that here D F train underscore model and there I'm gonna pass their train underscore data so this is the data which I will be passing there and once we have this train data now what I'm going to do I'm going to remove all the pipelines and I'm just going to add their T name the entity pipeline okay since he is this is this this is a blank model so it do not have anything so there I'm gonna add their janilla named entity recognition model there so I'm going to say that here if that's the image okay if any are named entity recognition not sorry if if any are not in NLP dot pipe underscore names if you do not know what are these a pipe etc I had taken NLP tutorials around 1516 tutorials in which the perhaps fourth or the fifth lectures are there in which I have talked about the NLP pipelines in the especi you can go ahead and watch that video so there you will get all the ideas what are these in ER and pipelining etc in the NLP so we have if any are not in NLP dot 5.5 underscore names then any R is equal to NLP dot create create underscore pipe so we have any air pipe lining here and then finally we are gonna add this NLP dot add underscore pipe and then here we have any R and I'm gonna add this NER at the last position of the pipe lining all right perfect so we have this here the pipe lining has been added now I'm gonna add the labels in this NLP pipeline so how we can add that label we have here for a mutation let's say annotation in train data now you might be wondering that what is this so this is nothing I am just skipping that because you see this is the topple and this couple has a two part at the zero index it has a text so the currently we do not need that text and and then it has the labels so the label resides in to the annotation that starts from here okay so this is the annotation so I'm gonna add this indentation there so for underscore annotation in train data for ent II in notations dot get into teach okay dot get into this otherwise you can just write here into T since this is your dictionary you can just type it in reason and then here we have entities okay now it is going to once we get these entities now if we return this list and then we are going here each object okay so one object after the another so we'll go first companies work that and then accept all other things perfect then we are going to say that in ER yeah then er dot ad actually this would be the NER dot add label okay any r dot add label then we have here ENT at the second position at the second position these are the levels so this is these two first our end and they start character position and then finally these are the labels which we are gonna add our custom labels in this alright perfect so once we do this now we are going to prepare our data for the training so I'm just going to copy and paste this code since this is going to take a lot of the time and then I'll explain you eat a line by line let's say I have pasted this code from here okay alright so now you see we have got this in ER in the pipeline and then I'm going to see that if there is other pipelines and you see here this is exactly we are following here let me show you yes you see here other pipelines etc and then we will be doing these things so for the detailed documentation and the understanding of this lesson definitely you need to read this later after watching this video so we have here other pipes there PI for pipe in an LP pipe names if pipe is not in here I mean it is saying that if there is more than one pipe lines after this NER then remove those pipe lining okay so the NLP door disable other pipelines and train only the named entity recognition and here optimizers we are getting near NLP door begin training and then this says that the iteration so we are going to train it for the 10 iteration and random dots a police the training data at every tracin and then here we have a losses and the index and then finally you see here again we are gonna read this train data okay so we are again gonna wrote read this train data and earlier we had not read the text but now we are also going to read this text and uh and from this text it is gonna match this annotation and then finally here text and annotation this is the drop that's mean we are just making it a little harder to remember exactly how it is doing since it will add the regularization in your model and here a city optimizer is being used there and then finally the losses so this is all let's go ahead and run this now we have our train data as you remember we have this train data now we are gonna pass this train data in this model which is a train model okay so let's go ahead and copy this train model and then we are gonna pass their training data so once we do this you will see it starts the execution but at the first step it is saying that it is not defined perhaps here okay all right let's go ahead and run this so it starts the training it might take around 20 to 30 minutes to complete it's a training you see here there is only the 200 resumes the data but it still it takes around the half an hour so if you have a large data set then definitely it is going to take a lot of the time for some of my projects it sometimes took around up the two they are the three days to complete the training so you have to the person and to complete its a training all right now you see the training has been completed it took some time to complete the training save this model so that you can also use this NLP model for for your future use not only not only to these projects and all so it will avoid to retrain your model every time so to to to save this model your NLP model what you need to do you can just right there NLP dot two discs they reach dot two discs and then I'm gonna say that here NLP underscore model so this will write this to the desk now you see here we have here NLP underscore model okay perfect now this model was already retrained so I'm going to just remove this this is that okay this cannot be removed from here but let me delete it from the original position okay now you see model is removed there now let's go ahead and load our this NLP model okay although this model is already loaded into the NLP but I'm gonna also show you that how you can train all how you can load already trained model to already to load this say already trained model let's go ahead and load it into a variable there we have an LP model is equal to speci dot load do you remember we had loaded first the blank water but now I'm going to load here already trained model which is NLP model okay so now this is loaded into the NLP model okay now you see here we already have this NLP model instance here in the NLP but here we are also having it into the NLP underscore model so it will help us to load this model even though if you are you are working at some other time and you do not also need to training every time okay so let's go ahead and see our training data we already have our training data train zero let's say this one and the Train one is something like this one so let's go ahead just take the train zero okay so this was our the training data train zero data earlier it was something but since we did here train random dots up well so this data data has been changed okay it I mean it has been suffered the data is same but the position of each of these data has been changed earlier if you remember at the train 0 something else was coming but now since the sahel has occurred so it has changed and it is giving a another name alright so it is not it is not preferable to test our model on the training data but to just understand how it is working and to get this some idea let us go ahead and pass this data to this NLP model so that we can understand how it is identifying these things so we are going to just pass this text and then our models should should written something the call is named agree etc from this text okay since the we have already trained our model alright so we are going to say here doc doc is equal to we have here NLP underscore model this is the recent model which we loaded here and then I'm gonna pass the train data okay so this is the train data underscore 0 and then we need to down or we need to load only this text value which we can get from the 0th position so this is the 0th index so with this we have got it and now I'm gonna see here say here for in a poor ent so here we have a for ENT in dog dot ents that's the entities name and then I'm gonna print it with some the padding so I'll tell you how it is working so this formatting tells that let's go ahead first print here ent label that's mean the predicted label which it is predicted and then finally the text I mean which label which text is corresponding to which level ok so we run it once we run it now you see here it is saying that ok Samir cuz you would is name location is Arisa it is email designation degree college name and these skills alright so this is how it is working okay so this is we have tested it from this is we have tested it from the already the data which is available on which we had trained it now let's go ahead and and and and do a classification from unseen data so here I have a few few here dog files and the PDF on which I'm gonna test it so let me show you this dog file so currently I am having this dog file Alaska Clark so this file is not included this file is not included into training data so this should work on this file our algorithm should work on this file so the first of all what we need to do we need to convert this PDF into the text format we need to extract the text from this PDF and once we get the text from this PDF then we need to pass it for the for the classification so to converting the text into a PDF we need to install here PI mu PDF do remember you need to install this or some other I had also tried I tried well I tried the PI PDF too but it does not work okay it does not work properly it had extracted PI PDF to had extracted the information from the PDF but not all the information and it was also returning some garbage characters as well and it was skipping the most of the information from the extraction so I do not recommend this PI PDF to and I have found that the pipe you PI mu PDF has worked better than the PI PDF in case of this project but if you have some other project and PI PDF to is working great why Kingdon it's a great all right so you need to install this I have already installed this so I'm gonna just uh now the commented otherwise you need to install this PI mu PDF so having said that let's assume that you have installed this PI mu PDF then what you need to do you need to import this from here and you also need to import here PG and once you get here the feed G then we are gonna give here the file name so file name which we have currently is LS Clark CV dot PDF so this is the file which on which we are going to work this one okay L is Clark CV PDF and then I'm going to say that here dog is equal to Fiji okay dot open and then F name so it will open this file then I'm gonna say that okay get imported text then for page in this doc it will get all the page the text is equal to text plus STR I'm doing a steer because sometimes only integer character can be it can only return some integer characters or non steer characters so that ste are correct non steer characters will not be appended with this this have this string so it might throw an error so here STR paste dot get text okay so we have here this one now once you have this let's go ahead and print this text so once we print this text now you see there are many new lines and if you use here this print you will see now this PDF all this information from this PDF has been extracted and this information is now printed into the text format okay great so I have found that this pi mu PDF fetch world better than the PI PDF alright so this is all here it has extracted but now you see here there is a lot of the special characters and the newline so what we are going to do we are going to remove those new lines so to remove those new lines I'm gonna use here join and then text dot split with the newline so how it is going to work it will split from a new line and the new line will be discarded and and without a new line then this text will be joined with a space thats mean new line will be replaced with a space and then finally I'm gonna print this T X now if you see here that new line has been removed and all other things are exactly the same as it is okay so now I'm gonna copy this one once I copy it and then I'm gonna paste it here and in this as you see we have our text in the form of TX all right so the TX is gonna pass into an LP model and then finally I'm gonna run it once I run it wow this is super this is great now you see you have got your name location designation it says the designation is just three manila tix and also designation is a software engineer you see here there is a little error but all those things are kind of the same you do remember if you are a data scientist and if you have already a data and you are just developing algorithm believe me then you are not true data scientist that the true data scientist job is not only the develop algorithm but also to understand the data and prepare this data okay but sometime you might have already the data but to prepare that the data according to your need and get the information desired information from that data is a real task of data scientist so it says that designation software engineer companies work at the Microsoft you see here there is a little later it says that the degree is Indian Institute of the Technology Mumbai although this is the Institute from where that this person has passed out okay and it skill it seems that it has given a skill quite good and additional information it has also predicted a quite good so with this having said all these things now I am sure it will help you to understand the CV parsing and the resume a parsing in a better way and if you increase your data set and if you include a various kind of CV and resume in your data set then definitely all kind of the series and the resume a you can pass without any problem okay do remember this is small data set is not going to always help you to pass CV and the resume a this is a big project you need a lot of the time you need to give a lot of the time to prepare the data and after that you will be getting a good result but for the vagueness for getting started how you can do the CV parsing and resume a parsing this is definitely going to give you a great head start thank you so much for watching if you are visiting first time this channel and if you have not subscribed my channel please do subscribe like and share this video to show your support thank you so much bye bye and the take care

Channel: KGP Talkie

Views: 32,094

Rating: 4.9129171 out of 5

Keywords: ner, spacy, nlp, nlp with spacy, resume parsing, cv parsing, resume parsing python, cv parsing in python, kgp talkie, nlp tutorial for beginners, nlp tutorial python, nlp tutorial python for beginners, free nlp lesson, free nlp codes, kgp talkie nlp lesson, custom ner training, ner training, ner spacy training, cv parsing in nltk, cv summarization in nltk, resume summarization

Id: HJy11kOlgvk

Length: 37min 2sec (2222 seconds)

Published: Sun Apr 19 2020