So far in our deep learning tutorial series we looked at Artificial Neural Network and Convolutional Neural Network which is mainly used for image processing. In this video we will talk about Recurrent Neural Network which is used mainly for natural language processing tasks so if you think about deep learning overall, CNNs are mainly for images, RNNs are mainly for NLP. There are other use cases as well so we'll understand how Recurrent Neural Network works and we'll look at different applications of RNN in the field of NLP as well as some other domains. We will be looking at some real-life use cases where sequence models are useful. You must have used Google mail-Gmail. Here, when you type in a sentence it will auto complete it. So just see, when I type "not interested at this time" is something it auto completed. So google has this RRN or Recurrent Neural Network embedded into it where, when you type in a sentence "not interested at" it will auto complete with "this time". If you say "we'll let you know if it changes" it will also say "in the future" so this saves you time. It will write the sentence for you. Another use case is translation. You must have used Google Translate where you can translate sentence from one to another language easily. Third use case is Named Entity Recognization where in the X you know you give Neural Network a statement and in the Y Neural Network will tell you the person name the company and time. Rudolph Smith must be a millionaire with Tesla's prices skyrocketing. So these are various use cases where using sequence models or RNN-Recurrent Neural Network helps. The fourth use case is Sentiment Analysis where you have a paragraph and it will tell you the sentiment whether this product review is One star, Two star and so on. Now you would think - Why can't we use a simple Neural Network to solve this problem? See all these problems they are called Sequence Modeling problem because the sequence is important. When it comes to human language sequence is very important. For example when you say, "how are you?" versus "you are how" doesn't make sense, right? So the sequence is important here and you would think - Why don't we use simple neural network for that? Well, let's try it. So for language translation how about we build this kind of neural network we know where input is the English statement and the output could be Hindi statement Once I build this network, what if my sentence size changes? So i might be inputting different sentence size and with a fixed neural network architecture it's not going to work because you have to decide how many neurons are there in the input and output layer. So with language translation, number of neurons becomes a problem. Like what do you decide as a size of neurons? Now one would argue okay I would decide let's say a huge size let's say 100 neurons and remaining if I am saying, did you eat biryani? So it will occupy 4 neuron. Remaining 96 I will just say 0 or you know blank statement. That might work but still it's not ideal. The second issue is too much computation. You all know neural networks work on numbers, they don't work on string. So you have to convert your word into a vector. So one of the ways of converting that into a vector is -l et's say there are 25000 words in your vocabulary and you will do one hot encoding where you know "how" let's say is at 46th position "are" is let's say second position "you" is let's say at 17000th position. So at that position you put 1, remaining position you put 0 and that's called one hot encoding. You have to do similar thing for output as well. But you realize this will increase too much computation. Each of the word when you convert it to a vector you know how many neurons you need in the input layer itself. Its humongous. The third issue is this - Sometimes when you translate language you for let's say two different English statements you might have a Hindi statement. So in this case when I say "On sunday I ate golgappa" let's say I train this network based on this statement and then for 'On Sunday' let's say it will adjust the weights of all these edges which I have highlighted in yellow color. Same statement I can say differently. I can say "I ate golgappa on Sunday". So now on Sunday the meaning of on Sunday is same but here neural network has to learn different set of edges you see all these edges are in yellow color. So the parameters are not shared. We looked at in our Convolutional Neural Network tutorial as well that by using convolution operation we can share the parameters. Here, the use of ANN or Artificial Neural Network doesn't allow you to do that okay. Also the most important part in all this discussion is the sequence. See when you have structured data, for example you're trying to figure out if the transaction is fraud or not and let's say your features are transaction amount, whether the transaction was made out of country or whether the SSN that customer provided is correct or not. Now here if you change the order of this features, let's say 'ssn correct?' I supply you know
my first neuron it's not going to affect anything
you know because the sequence in which you supply the input doesn't matter. Whereas if you have English to Hindi translation and instead of saying "I ate golgappa
on sunday' and if I say
181
00:06:25,360 --> 00:06:31,600
"I ate Sunday on golgappa" the meaning becomes totally different. So now you cannot say that the Hindi translation is 'ravivar ko mene golgappe khaye" because it becomes invalid so sequence is very very important that's why Artificial Neural Network doesn't work in this case. Just to summarize these are the three major problems with using ANN for sequence problems. Let's once again talk about Named Entity Recognition. Let's say 'Dhaval loves baby yoda' I love my baby grogu. I love Mandalorian series and we have got this nice baby grogu at our home which actually talks with us. In this statement Dhaval and baby yoda are person names, okay. So the whole purpose of Named Entity Recognization is to find out the entity you know like 'Dhaval' as an entity is a person 'baby yoda' as an entity as a person so that's the whole goal of NER. Now you can represent this as ones and zero. So if the word is person's name you would mark it as one and if it is not a person's name
you would mark it as zero.
so let's see how RNN works here. RNN is also called Recurrent Neural Network.
so first of all you have to convert Dhaval into some vector.
It doesn't matter how you convert
216
00:07:57,199 --> 00:08:01,680
it you can take a vocabulary and use one hot encoding and there are other ways of vectorizing a word. Then you have a layer of neurons. So these are all individual neurons. Let's say this is one layer. It's a hidden layer you supply that and you get one output okay. So each neuron all you know has a sigma function and activation function. So now while processing the statement 'Dhaval loves baby yoda' now I will process it word by word. So I supply 'Dhaval', get the output and then I go back again. Now I supply 'loves' converted into vector and the previous output which I got which was y Dhaval I now supply that as an input to this layer. So you see the input of the layer is not only the next word but the previous output because the language makes sense. Language needs to carry the context if I have just a word loss and if I don't have Dhaval in front of it it might mean a different thing. So there is a context that you need and this kind of architecture provides your context or a memory. In the third word again you supply 'baby' to the same network right. So our network has only one layer. it has only one layer, so there is input layer output layer and the hidden layer is just one and it has bunch of neurons. In that we are repeatedly processing word one by one okay and you keep on doing this. Now the benefit of this is when i'm processing 'baby' when i get why loves that 'why loves' carries the state, the previous state or previous memory of 'Dhaval loves' you know the whole statement. Now i'm presenting this in a different way make sure these are not four different hidden layers. This is a time travel okay so actual hidden layer is only one. I am just doing a time travel. So first when I supplied word 'Dhaval' i got this output and output was nothing but the activation function which I am denoting with a1 and you need some previous activation a0 as well. Let's say it's a vector of all zeros then you supply second word 'loves' and use the previous output which was yDhaval so yDhaval and a1 they are both same here and then you get another output a2 where that you supply along with the third word 'baby' to the same network. So these four neurons it's the same single layer. I am just showing the status of it at different times okay. So you have to be very clear on this that these are not four different layers. It's just one layer just because I am showing different time steps that's why I'm showing you almost a time travel here and once the network is trained of course it will output like 'Dhaval' is one 'loves' is zero 'baby' is one and so on okay. So you get your NER output individually here. One other way of representing the same network okay because just to avoid confusion and to make presentation little more clear. Many times in literature you will see presentation like this - Where each word which is an input comes from the bottom and there is activation. So again this and these two diagrams are exactly same okay I'm just putting this word at the bottom. Generic Representation of RNN is this. So this is the real representation. You have only one layer and you are you are kind of almost in a loop. You are supplying the output of previous word as an input to the second word. So now let's talk about training. So again the problem we are talking about is NER where these are my training samples okay x and y. 'x' is a statement 'y' is whether a given word is person name or not so we are processing first training sample 'Dhaval loves baby yoda' so this one I will first initialize my neural network weights with some random values, then I supply each word, then I calculate y hat which is predicted y, then I compare with the real y so real y here is 1 0 1 1 so I compare that with here so 1 0 1 1 I compare that with y hat and I find out the loss okay and then I sum the loss. So that will be my total loss. You all know about grade and descent right. So we compute the loss then we back propagate the loss and we adjust the weights. So now I take the second statement 'Iron man punched on hulk's face' he was very angry with hulk. Again i calculate all the losses then I find total loss and then I do grid and decent to reduce the loss. So i keep on doing this for all my training samples. Let's I have 100 training samples. Passing all hundred training samples through this network will be one epoch. We might do let's say 20 epochs and at the end of the 20 epoch my loss might become very minimum. At that point we can say my neural network is trained. Let's take a look at language translation. So in language translation what happens is you supply first word to your network then you get the output then again same network you supply second word and the output from previous step as an input and of course when you supply first where you have to pass in some activation values let's say all a vector of all zeros. Then you supply third word for fourth word and so on and when you're done with all the words that's when the network starts to translate it because you cannot translate one word by one, because after the statement I can push maybe one more word and that will just totally change my translation. That's why for language translation you have to supply all the words and only then the network can translate for you. So the network will translate it like this and the first part is called encoder the second part is called decoder. We will go more in depth into all this but I want to quickly demonstrate how the neural network looks in the case of language translation. Now this layer doesn't have to be just single layer. It can be a deep RNN as well where the actual network might have multiple hidden layers okay. So I hope that clarifies the architecture behind RNN and you understand why you can't use simple neural network here and you have to use specialized neural network called RNN which can memorize for you, which can remember previous state because language is all about sequence. If you change the sequence the meaning changes so if you like this video please give it a thumbs up and we'll be having more Recurrent Neural Network and NLP type tutorials in the future videos Thank you.
