Massively Speed-Up Python Code With Numba Compilation

[Music] what is going on guys welcome back in today's video we're going to learn how to speed up your python code with number in just in time compilation so let us get right into it alright so the first thing we want to do is we want to open up a terminal for example cmd on windows and we're going to say pip install number and as i said number is a just in time compiler a high performance just in time compiler which basically means that we compile certain pieces of python code during execution time so we have a certain function this function is a an ordinary python function and the first time we run that function it's going to be compiled to machine code and the second time we call that function it's not going to be interpreted by the python interpreter it's going to be run as machine code which is significantly faster so in order to do that we need to save from from number import it just in time so this basically means just in time like that um and we're going to import random and we're going to import math because we're going to do a very simple function here it's not going to be too fancy it's just going to be some function that has a parameter n and we're going to do some random calculations so this is not anything meaningful here i'm just constructing a function that has some uh computations to make so that it takes some time so that we can then measure speed up so basically i'm just going to say z equals 0 for i in range n we're going to say x equals random dot random y equals random dot random and then z equals dot sqrt x squared plus y squared so nothing too fancy and in the end finally we're going to return z and of course we're not just going to say equals but plus equals so again there is no real logic behind this computation here it's just something that we do of course you can also come up with something that makes sense uh but that is just some computation we can go ahead now uh we should import time i guess we can go ahead now and measure the time it takes so we can say start equals time time and equals time time as well and then we can say uh print or actually we we don't want to print this because printing itself takes quite a lot of time uh we're going to say some function and let me just see what number is meaningful to measure those are three zeros six zeroes okay ten million one two three one two three there you go so we can measure the time it takes to run this and we can see without any speed up we're not doing any compilation yet we're just running the ordinary python function here uh it already takes some time here we'll probably take like four or eight seconds or something like that maybe we should remove one zero oh it takes quite some time oh there you go and of course i'm stupid because i didn't print the result and minus start so we can wait again but you saw that it takes quite some time so if we manage to speed this up significantly uh then number obviously works so this takes like eight seconds or something or six seconds seven nine i don't know uh but it takes some time it doesn't work just like that oh 13 seconds actually okay so we have 13 seconds here we can maybe write that down 13 seconds now all we need to do in order to speed that up is we need to add an annotation a decorator that says just in time now the thing is that this jit tag has the option called no python we can set this to true and we can set this to false now if we don't set anything this basically means that if number is not able to compile this to machine code for some reason we're going to fall back to a mixed mode to uh to the python interpreter and no python means no python no python means uh compile it to machine code and then use it as machine code only the interpreter doesn't have anything to say here so with this function it's going to work fine with numpy it's going to work fine with pandas we're going to encounter some problems so uh we're going to just run this again now with no python equals true and the jit tag and you're going to see a significant speed up um hopefully there you go 1.779 seconds this is way faster um and this is only because we compiled it now i think if we are able to run this a second time it's going to be even faster because you need to think about this the first time we call this function we also compile it so in those 1.7 seconds we already had the compilation time not just the execution time so if i run this again now we should be able to execute this way faster so let's see what happens here um run so the first one should be way there you go 0.29 seconds this is the execution time of the machine code this is not the python interpreter and this is also not the compilation time this is just the raw execution time so this is 13 seconds compared to 0.293 seconds this is extremely fast all right so that is just a simple python function with calculations uh and in the documentation it says number likes loops number likes broadcasting number likes calculations and number also likes numpy so if we're working with numpy this should also uh work without a problem import numpy snp we're going to now have another function uh and this function is going to be actually quite similar we're going to do the same thing but with numpy erase not just with individual digits so first of all let's delete this jit tag here and we're going to have the same function um sum function and z is now going to be np dot zeros of shape n n and then we're going to actually we need to surround those with parentheses here because that's a shape and then we're going to say for i in range n and here we're going to now define or generate random numpy array so we're going to say np random dot rand of shape n n we're going to do the same thing for y and for z we're now going to do np dot sqrt and the rest stays the same so that's actually the same function but we're doing it now with numpy arrays which is a lot of more computation needed here um but that's actually quite simple so we can go ahead and measure the actu execution time but since we're dealing with a race here we cannot just go with 10 million we have to pick a smaller value let's go with 500. and let's remove all that now let's see how long it takes without any optimization without any compilation it should take like a couple of seconds for 500 elements now this is not 500 elements it's 500 times 500 so it's quite a big number here so there you go 8.48 seconds in order to process 500 or n equals 500 uh if we now go ahead and add let me just write that down again we had 8.48 come on 8.48 seconds and if we now go ahead and add a jit tag no python equals true by the way no python equals true is the best practice mode you don't want to use object mode which is the other mode uh so you want to use the no python mode because that's the mode that you want for best performance the only reason to not use it is if there are any problems whenever you can you use no python equals true so now we're going to do this again and we should hopefully see a speed up i don't see a speed up why don't i see a speed up come on takes actually longer okay no it's not longer but it was not really a speed up maybe because of the compilation maybe if you run this a second time it's going to be faster it doesn't really speed up a lot let's see what happens if i do this one more time let me cross check my code here should actually work maybe it works a second time maybe the compilation time is way too too large for this one okay we have at least some speed up here let me just see if i didn't make any mistakes here we have jit no python equals true we have a basic function we say np zeros we say for iron range n we do all these calculations we return z uh so maybe we can see a better speed up if we go to lower or higher numbers i'm not sure let's do it without this tag first okay 0.03 now if i do it with the tag okay actually takes longer because we need to compile uh maybe the opposite is true maybe if we go with i don't know i want to go too high here but maybe if we go to 600 let's go with 600 and we're going to see better results but yeah sometimes it doesn't work as well as it should work sometimes it works better it depends you just need to to evaluate it i think whenever you have a complicated logic where python where the python interpreter can mess up a lot of things uh then usually you don't want to then usually you want to use number and if it's just some short code maybe number is not the best solution here okay so 11.33 is already with the optimization so let's see what it is without but i think we're not going to get as good as of a speed up as we got in the beginning so probably it doesn't work too well on this particular case but the documentation says that number likes numpy so if we're using numpy okay we have at least some speed up here i'm going to terminate this now but last but not least i want to also show you here that it does not work with pandas at least not the way we want it to work so i'm going to delete all that and we're going to do we're going to say import pandas spd by the way for those of you who didn't figure that out already you need to install numpy and you need to install pandas if you don't have it already installed and i'm also going to use the pandas data reader this is not going to be anything uh related to to this particular number compilation number compilation but i'm just going to load some data so you don't need to use the data reader if you don't want to you can also generate data um i'm going to just load some stock data and the function is going to be pandas function and we're going to pass data to it and what we're going to do is we're just going to do some operations we're going to say okay result equals data dot sort values i'm going to say bye and we're going to sort by volume so just some stuff so that it has to do something result equals result dot apply map math.sqrt so we're going to take the square root of all these things then we're going to just say result plus equals 2 which is you know result plus equals 2 what does that even do number doesn't know what that does but it's a valid python code so it's going to be confused it's not going to be able to compile this properly uh then result equals result dot apply map and we're going to do a lambda expression here so this is not going to work at all by the way we're going to see that this is the case in a second um and then we're just going to take the transposed version here and we're going to return result so just a useless function that does something and we're going to do is we're going to say data equals web dot data reader i'm going to load apple stock data from the yahoo finance api and we're going to then just say start equals time time and equals time time and then we're saying pandas function on data and in the end would print end minus start there you go so first of all we're going to do this without uh without any tags and we're going to see how long it takes uh 0.02 seconds so the large or the waiting time that we had was because of the api so you can see this is actually quite quick and do it again they go it doesn't take too long now if we want to optimize that we can go ahead and say jit no python equals true and you're going to see that there are going to be some problems there you go so first of all where was it we had something failed in no python mode yeah first of all it failed in no python mode because it says on the documentation it does not work with pandas usually and because of that when we use pandas we just use jit without any no python because that means that it can fall back to object mode if needed now if we run this here i think we should still get some problems because of the lambda they go global name lam does not define so we can get rid of that line doesn't is not supported by jit and i think now at least it kind of works they go but it takes longer of course uh now if we run this a second time maybe we have a slight speed up but i don't even think so we can see if that is the case but all in all what you need to to know here is that when you're working with pandas with some more complicated mappings and functions or operations like this one jit is not going to work so number is not going to work properly it works good with ordinary mathematical functions it works good with loops and with basic algorithms it works perfectly fine with numpy it does not work with pandas and other more complicated libraries so that's it for today's video hope you enjoyed hope you learned something if so let me know by hitting the like button and leaving a comment in the comment section down below and of course don't forget to subscribe to this channel and hit the notification bell to not miss a single future video for free other than that thank you very much for watching see you next video and bye [Music] you

Channel: NeuralNine

Views: 8,226

Rating: 4.909091 out of 5

Keywords: jit, compile, compiler, compilation, jit compilation, jit compiler, numba, python, speed-up, speed up, faster, fast code, fast python

Id: bZ5G-RZoE6Q

Length: 16min 27sec (987 seconds)

Published: Sun Jun 20 2021