Coding Challenge #105: Polynomial Regression with TensorFlow.js

hello coding challenge the last one part 2 of the last one something maybe I don't know this is actually my second attempt I started this earlier today and I'm now gonna try it again you shouldn't watch this video go do something else but I'm making it because I want to make this video they've got to feel like I did something today so in the previous video I did a demonstration of linear regression with ten approaches and the idea of linear regression is I have some data set right my data set or a bunch I'm making up a data set just by clicking points but you could imagine the x-axis representing something in the y-axis representing something and then I'm trying to fit a line I want my line to fit to that data set and I'm doing that by creating intent to vote yes these variables that represent the end and the B of the formula for a line and then I create an optimizer and I I try to like figure out the loss like well if I had a line there what's all the differences between all the actual data points and where the line is and minimize that all that stuff that I did in the previous video if you watch that but linear regression can only ever fit a line so what if for example my data looks something like this well you can see the best line you can figure out to fit that data is this line which is it really accurate but I could see pretty easily like oh I could create a polynomial function probably a quadratic function that is a curve that looks like this that fits all those points so this brings me to the topic of what everybody wants to do with their lives so what is a polynomial equation well in some sense this is a polynomial equation it's a degree one I could also have a polynomial equation that's like a constant of degree 0 y equals 5 if I had one of degree 2 what I have is y equals a x squared plus B Plus see quadratic cubic would be y equals ax cubed plus BX squared plus C now what are these values what is this what does this mean so the degree has to do with the number of solutions whether they're real solutions or complex solutions and there are little bunch of solutions across that hello is math stuff which is not my forte I'll try to find a good resource for you know the fundamental theory of algebra or whatever you could read about but the point is if I just want to now adjust my code to use a polynomial equation instead of a linear equation alright so that's not going to be too hard there's actually very little I need to change into code so let's go back here and let's go to the code and let's look so here's the thing now I'm not gonna do this but really what I want to I'll make sure that's at the end I have an exercise for you to do if you make it all the way into this video I've got an exercise for you to do which would be totally fun again in the sense in the world we're polynomial regression is fun alright so I had em in B as variables now I need a B and C okay so now I need instead of m and V a B and C a B you know one thing I don't like about this you know I'm gonna fix something first that I never liked about this example which is that my space my two-dimensional space goes between 0 & 1 I think it'd be better just for the world if it went between negative 1 and 1 so let me look at everywhere where I map I mean it's really gonna be the same negative 1 1 negative 1 and 1 probably gonna miss something negative 1 or 1 negative 1 and 1 just think that I should have like a Cartesian plane where zero zeros in the middle as my data space just for what I'm doing here so I think and then this should be this so let's look at this whoops okay I missed something yes no there's the mistake that you all saw negative 1 and 1 there we go ok ok so we're my linear regression is working just to be really sure up into a point here oh no look at this is the quickness down there I have it backwards zero too high ah what a negative one all right we're really gonna get this there we go okay there we go all right now everything's fine I can now go to changing these too sorry for that little digression a b and c a b c okay then what's the other thing that i need to change so clearly the thing that i need to change is my predict function wherever that is there it is right so this is the formula y equals MX plus B expressed with tensorflow j/s now I just need to express this formula also we tend to float is this I feel confident I know how to do cuz I did it earlier today y equals ax carrot or I like to say hat look at x squared plus BX plus C so I'm gonna say constant wise equals so X's squared right x squared multiplied all the stuff in tensorflow J's can be chained the mathematical operation so the X is squared multiplied by a adding the X is multiplied by B that's B X and then finally adding C so this you know takes some getting used to how to like put all this stuff together and I could put it in multiple steps to make it more clear but this is the kind of thing you want to if you want to get into little low-level tension flow digest so if you want to practice so now I can get rid of I'm just gonna get rid of this this is the predict function now I have another really significant issue here so I'm not gonna let me just run the code to make sure there's no syntax errors but it's obviously not going to do anything that makes any sense because it is what it's that this code down here is designed to just draw a lot by the way good see I was doing something earlier in other words I'm picking this point and this point and just drawing a line but in order for me to draw a curve I can't you know I have to sample a lot of points along the x-axis so I need to make a loop and I'm now going between negative 1 and 1 to sample all these points I need to say begin shape vertex to your vertex to your vertex here for protection bobba bobba and shape i'll see that line so instead of having X 1 and X 2 I want an array of X values and I'm going to call that instead of line X curve X and that's going to be an array and I am going to start X at negative 1 go all the way up to 1 say X + equals and let's use some increment like 0.05 and then I'm going to say curve X dot push X so I'm just trying to make let me just show you what I'm trying to do here console.log curve X and there's gonna be all sorts of errors here but let's see you can see here that I'm just trying to make an array that has lots of X values but between negative 1 all the way up to 1 so I can now I need to get the Y values that go with that so I can draw this curve and I have the predict function does that so predict curve X get all those wise then curve Y is then again maybe I shouldn't be using data sync here that's kind of gonna be an animation slow down a problem but hopefully it'll be fine curve Y is now the regular number the floating point numbers not the tensor anymore version of the Y's then I can get rid of that tensor I'm done with it and instead of drawing a line what I'm gonna do is say now ok now I just need to go through and look at all of the X points and say the x value is map curve X index I which goes between negative 1 and 1 to 0 to width and Y map curve y between negative 1 0 to and then I want to say before this begin shape no fill stroke 255 stroke wait for two or whatever I had it and then n shape oh I need to actually set the vertex vertex X comma Y so let's see let's see what happens here look at that so this by the way is the random curve that the coefficients and I should get rid of this console.log so what's interesting about this is that every time i refresh this i'm gonna get a new polynomial a quadratic equation because it's picking random a b and c and you know what i didn't pick a random any negative numbers so this would also probably make more sense for it to write couldn't technically these be anywhere like I should start between like negative one and one does that make more sense right yes that makes more sense so you could see I'm getting all these random curves that's what it's starting with now when I click you can see it's trying to approximate the curve whoa weird huh that looks exactly right but backwards why because I made this mistake again height is flipped negative one to one oh no no not there this is it's hard to keep track of these pixel mappings between height and zero let's run this one more time there we go you can see now it's you can see it's taking a lot let me let me be more methodical about this I'm gonna click like a bunch like this it's actually finding it quite nicely but you can see it's good that it's quite lovely let me do it the other way you know what's bothering me is that you can see how this is not getting all the way to the edge it's because when I created those points I really want to say less than or equal to I want to get the last point in there as well Oh like do this that's a little bad trick there there we go let me let me write draw these now there we go so you can see this is working hooray so now we have polynomial regression now one thing I probably would want to do do is this you know now that I'm getting past just sort of like basic linear regression the optimizers the optimizer that I'm using if we look at the tensorflow code is train dot s GED stochastic gradient descent let's just try for a second let's look at the Tetra flow jsapi and i'm actually already here because i was looking at this earlier you can see there's actually different optimizers that we could try a well-known one atom the a da here being for adaptive i believe like a de grad a dad a de Delta but if I look at this we can see oh this constructs an atom optimizer that uses the atom algorithm I could click on this link here and I could find this whole paper that explains it beyond the scope of what I'm doing right now but just out of curiosity all I would need to do if I owe maybe stochastic gradient descent isn't the best algorithm for fitting this curve let's just change this to Adam and I kind of have a feeling that this learning rate being so high is gonna not work forever let's leave it so warning it might flicker quite a bit let's just see what happens with like a very high learning rate yeah you can see look at it like whoa that is really fun actually I kind of like it with the high learning rate because look how quickly but you can see it's kind of like bouncing around is because it's high learning rate it's gonna overshoot the optimal spot so let's just make that point one and let's also I think it would be fun to change this to Mouse dragged although I'm worried I'm gonna get so many points that it's gonna really slow down yeah Oh while I'm adding the points that's interesting so while I'm adding the points let's see this is a little bit weird but let adding points equals false I'm just gonna have this is like a terrible idea adding points equals true adding points equals false this doesn't make any sense why would that be really slow while I'm adding the points because what I want to say is like if and if not adding I was thinking I could like don't run the training if I'm not adding the points only run the training if I'm not adding the points yeah that didn't really help why what am I missing here all right I'm kind of off in the weeds here a little bit of something that's not irrelevant to this example but I've been noticing that it's like really slow as I'm like drawing and then as soon as I like let go it's perfectly happy to like animate very quickly I'm trying to figure out why this is so one thing I'm gonna try is I'm gonna just change I'm gonna add mousepressed and I'm gonna say let adding whoops I'm gonna create a new variable let adding points I'm gonna just actually call it let dragging equal the false and I'm going to say I'm gonna say my mouse press I'm gonna say dragging dragging equals true and then I'm gonna add mouse released and say dragging equals false and then in draw I'm gonna I'm gonna do this now in draw so this way I'm gonna kind of like really keep everything in draw and I'm gonna say if dragging then add points and if you're adding points maybe don't try minimizing the just wait till you're done adding points to minimize the function let's see how that works yeah there we go so it's not happening in real time in the same way but at least it's letting me add the points so I I'm intrigued why I wonder if there's probably a different way to think about this I really love how you that animation of watching it fit oh it's really just very satisfying so in any case let's try I'm just you know let me just try a smaller learning rate just because I'm curious so you can see with a smaller learning rate it's moving much more slowly so the higher move learning rate it's gonna get there much more quickly abouts good about you could do something that's called annealing I believe of the learning rate meaning start with a high learning rate but lower it over time we'd have to look at the tensorflow das API to see how that's done but here's thing I really want to do this I want to do this with a equation of degree three just because so I'm gonna do that manually so I'm gonna add a D here and then I'm gonna add a D here and this is gonna lead to my exercise for you and then I'm gonna say ax cubed plus BX squared plus CX Plus D so I want oh is there a cube mathematical operation in tensorflow TAS square operations operations there's a square a POW I guess I could do I'm just wondering if there's a TF dot square is there a cube no so I probably want to do power where is that it's funny how that's under it react so I want to do this POW base exponents so I would do X's tau 3 multiplied by a adding X's squared multiplied by B adding C X's multiplied by C and maybe I should start to do something where I put these on different lines so so let's do this X's pal 3 multiply adding X's squared multiplied by B adding X is multiplied by C adding D right did I get this right this is now of degree 3 X is power to the power of 3 multiplied by a plus X is squared multiplied by B plus X's plus CX Plus D ok and then I think actually I'm good because this everything else is the same so let's go check this ooh argument exponent passed Oh must be a tensor but got a number of course so if I'm saying power 3 I need to say TF scalar 3 so everything's got to be everything's got to be a tensor I can't just use numbers like I'm used to so that's to be TF to scalar dot 3 and let me put the learning rate back up higher let me put it back up to like point - wow that is immensely satisfied so now you can see and by the way if I happen to draw a line it should still be really happy to sort of like fit a line there because it could just make those coefficients 0 oh this is great oh hi perfect lad I've made this video for myself at least so here's the thing here's my exercise to you how can you make this so that the degree of the polynomial is like something that can be interactive as well so could I have a drop-down that allows me to try it with a degree 2 degree 3 degree 4 or maybe a slider what other kinds of interactive features could I add to this - and how could I maybe like have the data set come into this in a more interesting way besides just kind of drawing with the mouse so could you make what kinds of things could you could you do with this I think there's some interesting visual possibilities so I hope you enjoyed this video I'm going to publish this code now as a coding challenge called it's really part two of the linear regression now polynomial regression and I'll see you the future videos I'm going to work on that I'm gonna look at that tensorflow digest layers API soon enough ok goodbye I forgot something important that I was told by the chat that I should mention and I'm referring of the dangers of overfitting of polynomial equation and that's what this being represented on this is a Wikipedia page for overfitting which is a term that you'll hear a lot in machine learning meaning what is overfitting so if my data set if those points are my data set a really fancy polynomial equation of some high degree is gonna be able to draw something like really accurately connecting all those but it doesn't actually have any real meaning as applied to the data the data can have a lot of noise in it and a line might actually be an appropriate way to make a prediction even though it doesn't fit it has on like a higher loss than the polynomial so so this idea of generalization you don't want your machine learning model to work so accurately with your known data set that it cannot make good predictions with an unknown data set and this is something that will come up more and more as I do more videos and tutorials with things here's the thing though I kind of like this example that I made for the fact that it does something kind of interesting visually in that it just looked figures out how to make this polynomial function fit this arbitrary set of points so I think we're sort of like some kind of artistic output there might be some value here but yes thinking about actual machine learning and how you want to make predictions is important so that's why I'm here at the with this public service announcement the more you know [Music] you

Channel: The Coding Train

Views: 30,487

Keywords: tensorflow tutorial, tensorflow tutorial for beginners, tensorflow js, tensorflow basics, tensorflow example, tensor, tensorflow, machine learning, machine learning basics, machine learning tensorflow, tensorflow basics tutorial, JavaScript (Programming Language), programming, daniel shiffman, tutorial, coding, the coding train, nature of code, artificial intelligence, gradient descent, stochastic gradient descent, polynomial functions, polynomial regression

Id: tIXDik5SGsI

Length: 21min 38sec (1298 seconds)

Published: Mon Jun 04 2018