Custom Indicators for Reinforcement Learning Trading Tutorial | $GME Python Trading PT2

what's happening guys my name is nicholas renate and in this video we're going to be building up from part one of my reinforcement learning for trading series and what we're going to be doing is we're going to be adding custom indicators to our gme trading environment so this is going to hopefully allow us to get slightly better performance when it comes to applying reinforcement learning for trading let's take a deeper look as to what we'll be going through so the main goal of this video is to build a trading bot that performs a little bit better than the original trading bot that we built in part one of our series now the way that we're going to tackle this is to start adding some custom signals to our environment so a lot of you in the comments on part one asked if we could add different types of signals like rsi obv and even a simple moving average so that's exactly what we're going to do so in order to do that we're going to be using finter to calculate custom signals so this is a financial technical analysis library in python makes it really easy to calculate these custom indicators we're then going to take those indicators and integrate it into our existing gme trading environment so we'll actually be building technically a new one but we'll have those new signals integrated and then what we'll do is we'll retrain and we'll test that model and see what performance looks like so let's take a look as to how this is all going to fit together so again really similar to what we did in part one of the video we're going to be using open ai gym and the any trading environment to be able to work with our gme trading data the core difference however is that we're going to apply our own custom overlay to this to be able to calculate our custom signals so specifically we're going to be calculating sma rsi and obv so three additional technical indicators which you are asking for in the comments so we're going to be adding these to our environment and then exactly the same training process is going to be followed when it comes to actually trading our bot and then we'll test it out hopefully fingers crossed we get some significantly better performance than we did in part one of the series ready to do it let's get to it alrighty so in order to get started with part two of our reinforcement learning tutorial we're going to be building up from everything that we did in part one so specifically we're going to be leveraging the code that we wrote inside of the reinforcement learning for trading video and in here we did a whole bunch of stuff but specifically we used jim any trading and stable baselines to train a reinforcement learning bot that ideally should have been able to make a little bit of profit off of gme now we also brought in some gme data from marketwatch and a whole bunch of other good stuff so again we're going to be trying to build up on top of this but add some new custom indicators so in order to do that i'm going to jump into the notebook and this is the exact same notebook that you're able to get off github so if you want access to that by all means check the link in the description below so we'll also have all the code for this video available in the description below via github if you want the code from the initial tutorial just go to github.com forward slash knick knock knack forward slash reinforcement dash learning dash 4 dash trading so again if you just go to my github account i've linked it a bunch of times in the description you should be able to get all of this code particularly the code that we had in the previous video including the data as well which is obviously quite important okay but in this case we're going to be building up from that now specifically so there's not too many changes from step zero one we'll also change one thing there though um two the main changes that we have are gonna come around here so i've got a couple of new steps so in terms of what we need to do to add our custom indicators and when i say custom indicators we're going to be adding rsi so if you're a technical analyst or a particularly interested technical trader so you're going to have relative strength index we're also going to have a simple moving average and we're also going to have the obv indicator which is tied to volume so in order to do that we need to install and import some new dependencies we're then going to fix our volume column in our gme data because what i notice is that those are actually strings so we're going to need to fix that up again it's just a single line then we'll easily calculate sma rsi and obv using a python package again i'll show you how to do that and then last but not least we're going to create our custom environment overlay and then pretty much everything from there on out is very much the same so we're just going to plug in our new environment to our stable baselines algorithm and then we're going to test it out so first things first though what i did notice is that inside of the data set that we actually had from marketwatch this is actually in reverse order so we actually want it to go from older state to newest right now it's going from newer state to oldest which might have explained why we had some weird patterns in our previous trading block what we're going to do is we're going to flip this around so we're just going to re-import our dependencies and then we're going to bring in our data set which again is inside of the data folder forward slash gme data.csv so if we actually open that up you can see that we've got our gme data and again this code is available in the github repo so if you want to pick that up you can okay so what we are going to do is we are going to resort our data however so that rather than having it start from newest to oldest we want it to go from oldest to newest so let's go on ahead and fix that up first up okay so that's our data resorted so you can see here that it was starting out at the what is that the 12th of march 2021 and now after we've sorted it we're now starting out at the 16th of march 2020. so we're now in the appropriate uh sorting order in order to sort that what we've done is we've written two lines of code so the first one is df dot sort values so this uses the sort values method on top of the pandas data frame to be able to sort values and then we've passed through our date so this is the column that we want to sort on and we specified ascending equals true so this is going to give us oldest to newest then we've also passed in place equals true and this allows us to apply it to our data frame in place and then we've gone and showed the first five rows of that data set by typing in df.head so all up we're sorting out values in this line and then we're showing the first five rows so that's the core change that we needed to do to our data set and then we can go on ahead and run through the rest so again all this code we wrote in part one of the tutorial so we can test out our environment and you can see now that it looks like it's in a better order because what we're expecting with our gamestop data is that we'd have a big spike towards the end of the data set and that's exactly when the short squeeze started to happen so if we actually take a look and change our frame down so this is changing how much data we've actually got available within our trading environment so if i type in 200 and again we explain all of this in part one of the tutorial so again all we're doing here is by changing frame bound equals five to two hundred we're going to show 200 days on exactly 195 days worth of data because we're windowed so let's take a look at that and you can see in fact that the short squeeze is happening up here so this is when the stock price did start to rise and if we go a little further i think we'll see an even more significant rise if we go to i think we've got 250 days of data so if we do that yeah so again so now you can see that we're definitely going in the right pattern so when we built the original trading bot we were trying to trade or teach our bot to trade on this period here which is obviously going to be really really difficult because we've got these huge swings in this case we've solved that now so we've got the right order so we can actually get on and start building our custom indicators so first things first we're going to need to install and import some new dependencies now there's one key dependency that we need to install and that is finter so finta gives us a whole bunch of technical indicators that we can apply to our pandas data frame just makes things a whole heap easier so in this case we're going to be doing simple moving average rsi is somewhere down here but again you could add in a bunch of these as well if you wanted to you could add in all of them if you really wanted to this gives you a whole heap of capability to add in a bunch of additional indicators so let's go ahead and install finta and then we'll be able to start trialing it up so we'll need to import it then we can try it out alrighty that's finter installed now again i already had it installed so but in order to install it for the first time you just need to type in exclamation mark let's just make sure we zoomed in exclamation mark pip install finta so f-i-n-t-a and that's going to install all of those custom indicators that we can begin to use within our environment now we actually need to import it into our notebook so let's import finta but we're also going to import the stocks env class from jim any trading and we need this because we're going to use it in our custom overlay that we do down here but again more on that later so let's import our dependencies all righty those are our two dependencies or two new dependencies now imported so i've written two lines of code there so the first one is from jim underscore any trading dot envs import stocks env and this is going to give us our custom overlay so we'll be able to customize our environment to be able to add our new indicators using this class then the next import that we've done is from finta import ta and this gives us our technical analysis library or technical indicators technical analysis library so we're going to be able to use ta.a whole bunch of other different technical indicators to actually calculate a new indicator so if we type in ta dot and hit tab you can see that we've got a whole bunch of different indicators there that we could try out on our pandas data frame in this case we're going to delete that cell because we don't need it just there we're going to get onto step 2.1.2 so i added this new numbering notation in case we wanted to do more stuff with this tutorial later on so let's go ahead and fix our volume column now by fix what i mean is that if i we type in df.d types you can see that our volume isn't actually a number right now it's an object so if we type in df.d types it's going to give you the data type of every column within your data frame now this is particular to the market watch data set that i've got but again this might be perfectly fine if you're using a different data set and again remember that you can use a whole bunch of different types of data in this example so if you wanted to do different stocks different securities wanted to work with crypto you could definitely do that here so if we actually take a look at our volume column you can see that we've got these commas in there now you might be you might actually not notice that this is actually a string but if we pass through our column so df and then we pass through our volume index dot unique you can see that these are actually wrapped in strings so as of right now we're not going to be able to calculate our obv technical indicator because it relies on our volume column so what we need to do is convert this to a number specifically a float to be able to use it so let's go ahead and do that and then we'll have a number that we can work with okay before we run that let's take a look at what we're actually doing so first up what we're doing is we're grabbing our volume column so by passing through df and then passing through our volume column this is going to allow us to work with all the values in our volume column and then we're using a combination of the apply function and lambda to loop through each value within that column then this line over here is actually doing two things so first up what we're doing is we're getting our values so say for example we're actually working with the four eight six six six nine six we're then replacing all of the commas in that string with a blank value so effectively we're stripping out all of these commas so they will not be there anymore then we're applying a float type over the top of it so this is going to convert it from a string so str to a float so we'll now actually be able to work with it inside of our data frame and then we're overriding our existing volume column with the values that we've now just gone and pre-processed so all up what we've written is df square brackets volume so this gives us or this overwrites our volume and then we're resetting it using equals then we're getting our volume column again so df square brackets and volume inside dot apply lambda x so that means we're going to be able to leverage x as our iterator or actual value then a float x dot replace and then inside of our replace method we're using quotes comma and then comma and then quotes again so basically we're passing two values here so replace has the comma first and then the blank string next so effectively it's going to replace that so if we take a look at our df volume column now you can see that they are indeed numbers and if we type in df.d types you can see that in fact our volume is now a float so we're all good to go now the next thing that we want to do is actually go on ahead and calculate our simple moving average our rsi and our obb but again you could add in a whole bunch of different technical indicators if you wanted to you could even calculate your own in this case we're going to calculate these new indicators and then plug them in so let's go ahead and do that alrighty so those are our technical indicators now calculated so we've written three lines there so df oh we'll come back to that in a second let's take a look at our results first so you can see that inside of our data frame we now have a three new column so let's take a look at some more rows so we can see it all so we've got our simple moving average column our rsi column and we've also got our obv column as well now in this case you can see some not a number values and this is because we obviously need some lagging data to be able to calculate some of these indicators what we're going to do is we're going to fill these with a zero so let's go on ahead and do that and then we'll explain all of the code that we just wrote so if we go back into here so we're just going to use the fill n a function pass through 0 and then specify in place equals true and then if we take a look at it again we've filled in those missing values so this just shouldn't cause any issues when it comes to actually applying it to our trading function okay so four lines of code that we've written there so the first one is creating a new column called sma so df and then we're passing through a string called sma which is going to create a placeholder for our sma column and then we're using our ta package up here which we brought in from finta to be able to calculate sma so remember you've got a whole bunch of different technical indicators that you can use and really all you need to do to calculate the different ones is just type in t8 dot and then whichever function you actually want to apply to that we've passed through our data frame and the number of periods that we want from our simple moving average so in this case we want a 12 period simple moving average then we've done similar things for rsi and obv so in this case we've created a new column for rsi df and then inside of our square brackets we've created a string called rsi use the ta package again dot rsi and pass through our data frame again we've done the exact same thing for obb in that particular case we're subbing out rsi for obv so this gives us our three new indicators that we're going to apply into our environment and then the last line that we've gone and written is df dot fill in a and then we're filling in all our missing values with a zero and we're passing through in place equals true to apply it to our data frame in place and then this gives us this so we can take a look at our first 15 rows by typing in df.head and then passing through the value 15. so this is going to allow us to take a look at everything that's actually in there and so you can see now that we've got a couple of different columns so open high low close volume sma rsi and obv but again if you wanted to you could plug in a whole bunch of additional columns you could even calculate your own indicators using just standard pandas functionality as well if you wanted to so if you had some really specific technical indicators that you're using your trading or you're using your current algorithmic setup then you can definitely do that as well here okay so that is now step 2.1.3 done so so far what we've done is we've reordered our data set installed and imported our new dependencies fixed our volume column and calculated sma rsi and obv now the next thing that we need to do is actually go ahead and apply these changes to our existing environment so for this we're going to be using this stock cnv that we just brought in so effectively what we're going to be doing is we're going to be grabbing our environment adding a custom overlay to be able to bring in our new signals so let's go ahead and kick this off okay so that's the first part of creating our new environments so what we're doing is we're creating a new function here called add signals and to that we're going to be passing through our existing environment so what's then going to happen is that we're actually going to call this add signals function within our environment so first up what we're going to do is calculate the starting index so for this we're grabbing frame bound 0 which is effectively grabbing let me show you where it is it's effectively grabbing this value here so we're grabbing our first frame value and then we're grabbing our second one in the next line of code so if i come back down here so we're grabbing our start index which is grabbing frame bound zero minus our window size in this case if we had a window size of five and we had our starting point for our environment as five then this particular value is going to be zero which means we're starting at our first row then the next line that we've written is grabbing our ending index so env dot frame bound 1 which is effectively going to be grabbing this value here so 250 if our environment had that particular frame bound so why we're doing those is because we're effectively just trying to grab our starting and ending indexes because when we create our environment we still want to be able to get the same range of data which you can see in this next line here so what we're then doing is we're grabbing our prices so in order to do that we've written prices equals env df dot loc so we're using our location indexer and then we're basically passing through that we want all rows and then we only want the low column then we're converting that to a numpy array and then we're grabbing our start and end ranges so really this is where these two values come in because we only want to get the prices from our start to our end components we don't want to go any further then we're doing a really similar thing but in this case so for our signal features rather than just grabbing our prices we're grabbing our low value our volume sma rsi and obv so once you've actually gone and calculated all these custom indicators you can actually choose what to bring in so if we didn't want to bring in rsi or if we didn't want to bring in obv we didn't need to or we don't need to and then again we're converting it to a numpy array and we're passing through a specific index so we're grabbing our start and our end values and then we're returning it as a result of our function so returning prices and our signal features so if we run that so it doesn't look like we've got any errors there now what we actually need to do is we need to apply this to our environment so let's go ahead and do that we'll take a step back and we'll see how it all fits together foreign okay so that is our custom environment created now what we've gone and done is we've gone and created a new class called my custom environment and then we've passed through the existing stocks environment to that so if we take a look uh up here so effectively got a bit of inheritance happening so we're grabbing our stocks environment and we're applying it to that particular custom environment then what we're doing is we're going and resetting our process data function so this is a native function available within the stocks environment and we're basically saying look rather than use the native one which is just going to give us back our existing signal features which i believe is price and then price difference we want to apply this function that we've just gone and created which is instead of giving us just those two values we're now going to get our prices which is just our low value and all of our custom signal features then what we've gone and done is we've gone and created an in order to do that sorry we're basically going underscore process underscore data equals add signals which effectively means that we're going to be running this function when our new custom environment is created so to summarize all of that so we've typed in class my custom env we've passed through our stocks environment to that colon and then we're resetting our process data function there so underscore process underscore data equals add signals which is this function which we just created here so ideally we should be returning back our new prices and our new signal features values once we go and create this environment then what i've gone and done is actually created a new instance of that particular environment so by typing in e v2 so we're now going to call it emv2 we're creating a new instance of the my custom environment class which we had over here and then we're passing through a number of different keyword arguments so first up is our data frame so this is this data frame that you can see here so in order to do that written df equals df so the first one is what parameter you're setting and then the second one is actually the value i always used to get confused with this when i first started adding coding but this is the keyword argument this is the actual value to set that keyword argument then the next value is window size equals 12. so in this case we're changing how large our window size is so in the previous video we actually had a window size of five here we're just going to change it to 12 but again you could play around with this if you wanted to and then we're setting our training data so in this case we've set frame underscore bound equal to and we've set this inside of a set of parentheses 12 comma 50. so this effectively means that we're going to have our data all the way out to day 50 and we're going to be windowing from day 12. so effectively that equates to what 38 days worth of windowed data so if we actually take a look at our environment now so emv emb2 you can see that looks like we don't have any errors there if we take a look at our prices we've got all of our low prices which should match df.head so we've got our data starting from low which should be 3.9 4.11 3.5 so you can see these values here are matching what we've got here then if we take a look at our signal features again now we've got all of our values so if we try to match these up we've now got 3.9 which is our low value for 866 which our vol is our volumes which you can see there sma 0 rsi is 0 and then obv is 0 in this first array and if we take a look at the next one so 4.11 3.5 or 3.56 to the power of 06 which is this value over here so low volume and you can see they are matching so now it looks like we've gone and created our custom environment we've got our custom environment inside of env2 we can actually go in ahead and train now so we already had our build and train functionality from part one of this series and again we can just reuse this we don't need to go and create anything new here so what we are going to do however is rather than using this standard gym.make functionality or gym.make environment creation we're actually just going to set this out for env2 so this is now going to create a new environment here and plug it in so if we set that and then we go and train on it this is now going to kick off our training and you can see that now it's actually training using our custom indicators and what i actually noticed when i was testing this out is that it does look a lot more stable so this is probably a combination of factors particularly the fact that we've now gone and sorted our data correctly and we now have some additional indicators which should ideally try to train our data so again no changes to this code here you can just kick off the training so we'll let that run and then we'll be right back in a second alrighty so what i did is actually stopped our training so we were actually getting a really really high explained variance value so if you actually take a look we're getting explained variance of 0.999 explained variance of 1 which seems ridiculously high and again these are all pretty high values again it's performing way better than what we had in a series or part 1 of this series so now what we want to do however is actually go on ahead and test this out because right now we just have our training values and again i stopped this at 48 000 time steps but again you can play around with this see what actually works well for your particular set of data so in order to actually go on ahead and run our evaluation script down here we just need to change this line and change it to our custom environment so let's go ahead and scroll on up and what we actually want to do is copy this line here and if we scroll down we're just going to paste that over and then what we can do is we can change our frame bound because obviously we don't want to use the exact same training data that we used for our particular environment we want to go and test this out so if we set this to 80 and then maybe like 120 that should give us day 120 to day 80 and then we're obviously gonna have window data so we can actually go and test this out now so if we run it so it looks like we're getting a total profit of 1.17 so this is giving us 1.7 1.8 percent profit over what is that like 40 days so not too bad and again it looks like it's a lot more stable so if we plot this out so you can see it looks like it's shorting up there which is a little bit strange let's push it out a little bit further and see what that looks like so let's go to day 250 for example and it looks like it's actually shorting appropriately when it's got up to the top but it looks like it's actually shorting quite a fair bit but again our total profit now is significantly higher than what we had in our first training run so remember in part one of this series i think the maximum that we got was about 50 or sorry not even 50 it was 5 in this case we're now up to 0.359 so what is that 35.9 profit now again this is for demonstration so don't take this as golden so do test it out and do do your own research this again just shows what's possible with the capabilities and reinforcement learning but again that really sort of summarizes how to build in your own custom indicators how to fix up the data and reorder it and ideally how to go on ahead and build a more successful trading bot using reinforcement learning so if we summarize what we went and did in this video so in step one of the original tutorial we went and fixed up an error and we went and resorted our data to be in the correct date format and then we went in and we went and did all of this custom stuff so we added our custom indicators we installed finter to give us our different indicators to calculate we reformatted our volume column but again if you're using your own custom data you might not need to do that this was in the particular case of the marketwatch gamestop data that i was using it looked like our volume column was a string we then went and calculated all of our custom indicators so sma rsi and obv and you could add in your own ones if you wanted to we then went and created our new environments we went and trained a new model and again this looked like it was performing a lot better than what we had in our original environment and then we went and tested it out so we just replaced this line we went and tested it out on our new data and that about wraps it up thanks so much for tuning in guys hopefully you found this video useful if you did be sure to give it a thumbs up hit subscribe and tick that bell and let me know what other reinforcement learning trading videos you'd like to see maybe we could integrate this into an actual trading platform and test it out in the real world thanks again for tuning in peace

Channel: Nicholas Renotte

Views: 39,813

Keywords: reinforcement learning, algorithmic trading, reinforcement learning trading bot, reinforcement learning trading python, reinforcement learning trading algorithm, reinforcement learning trading environment, reinforcement learning trading tutorial, deep reinforcement learning trading, deep reinforcement learning for algorithmic trading

Id: q-Uw9gC3D4o

Length: 28min 55sec (1735 seconds)

Published: Thu Mar 18 2021