AI For Trading Forex (What You Can & Can't Do!) - Ernest Chan

what's up welcome back to another trailer interview my goal with these interviews to be able to teach you things about all the aspects of trading so that you are able to learn about all these different things and these possibilities you have as a trader this week i'm sitting down with dr ernest chan to discuss about artificial intelligence and trading how that can be used and also the history of ai how that came to be many many years ago how that evolved today and what you can do and cannot do with it also how to best use it for your trading ai is topic that's really fascinating to me so without further ado let's dive right into the interview welcome everyone to tv i'm sitting down with dr ernest p chan you are an expert into the into actually trading and enterprising models into trading so we'll talk about i feel ai software and things to sort quantitative training so dr john can you tell people who you are and a bit about yourself yes i started my career at ibm tj watson research center in the human language technologies group doing machine learning research that i i had a very uh productive three years there but the interesting thing about that group was that it produced a lot of very high profile quantitative hedge fund managers i mean the group has nothing to do with investment management it has only thing to do is machine learning and natural language research but for some reason the algorithms and the techniques that we deployed uh seems to found a lot of use in managing money so you know there are some high-profile hedge fund managers that grew out of that group and i too after three years of very enjoyable years there i too decided to move into finance not because i didn't like the group but because i prefer to live in manhattan rather than in the suburb of new york city so i joined morgan sandy's data mining and artificial intelligence group that was back in 1997 so in case those who think that machine learning as applied to finance is a recent phenomenon people have been trying to apply ai to finance since uh 20 30 years ago and monsanto was certainly one of those pioneers so after i joined that group a number of my colleagues decided that they do not wish to consult that was an internal consulting group by the way so a number of colleagues decided to manage their own money instead using the algorithm so we started a proprietary trading group at credit suisse in new york at the time and after that i worked for a number of different hedge funds until 2006 where i started to trade for myself and then shortly thereafter i wrote books and started my first hedge fund called exp uh with a partner in chicago it had a few good years but my partner decided that our aum wasn't growing fast enough so he retired from the industry managing his own money instead of others and i started qts capital in 2011 trading a fairly high frequency forex strategy which for about six months it practically has no losing days and then so we leveraged 14 times that was yes i know that was a newbie mistake and we were promptly hit by the first ever black swan event namely the u.s treasury debt downgrade no one in their right mind could have believed that the u.s treasury that would be downgraded but it did happen things can happen so we were hit by 35 drawdown uh in that quarter uh and but you know so we decided to um look for ways to hedge this kind of a tail risk because certainly our high frequency geology was subject to such tail risk and we found one in the following year and that grew to today's tail ripper strategy it is a intraday trend falling strategy that trade only the e-mini futures no overnight holdings and yet it was able to hatch all manners of nasty market regime so we had a very good quarter firstly in the 2015 chinese stock market crash in the fourth quarter we had a very good quarter in the 2018 former garden where some of the leveled for volatility etf went up etn went out of business uh and then of course this year the strategy and also our fund had some very amazing returns as well so the strategy was initially created to hatch our own tail risk and now it grew to become a standalone product which many institutional and individual investors have used to hatch their own long beta portfolio as well and also it is interesting as well that um in the course of improving our on our tailhead strategy you know the tail reaper strategy uh we went back to well i went back to my own roots in machine learning and tried to improve it using the techniques in machine learning and it's very interesting that um well you know for tail hair strategy a lot of people were able to uh you know make money in a crisis it's easy just by put option on the index and that's it uh the question is how do you prevent laws during better times and the strategy was able to limit its loss or in fact make small returns during calm markets because of machine learning essentially we applied a algorithm to make prediction about the probability of each trade in our tailhead strategy and if the probability is too close to 0.5 which means it's essentially a coin flip we would not enter sec into a new position we would not trade it and that has saved us from a lot of grief um during actually car markets you know when the market is very volatile you know you just let it run but when the market has come we let the machine learning this slayer to feed to our trades and that system we find to be universally applicable to any strategy and so we actually started an independent company called predict now.ai to capitalize on that so that we can offer it to other investment managers and there's no conflict of interest because we are not trying to generate trading signals for these other managers the machine learning system will learn from the manager's own trading strategy learn from its own past track record and you know and supplement it with external data to improve on their own strategies so it's a win-win situation we get to you know the managers get to improve the system very easily using a machine learning service it's a no code machine learning service you don't even need to know how to program and we get to commercialize some of the intellectual property that we developed in the fund as well that's very interesting we have a lot to talk about you mentioned the fact that people would start to use ai 20 years ago and even beyond that what did it look like back then compared to today is it similar or is it different what people were trying to do back then that is very good question uh in the old days people were very um naive i think and i certainly was one of those naive people in applying ai to finance we think my goodness the back class is like a sharp of four the machine is so smart much smarter than human and then we will you know run the black box and to trade in day in there and we promptly lost millions using that black box because what we did not uh understood was that there is a huge room for overfitting using ai um you know the machine can does not have any sort of intuition it does not have any context so it learn every little bit of history in the data if you let it run wild unconstrained and those noise are not repeatable so there are all kinds of patterns that they learn were unique to a particular period or particular year or particular market condition that is not generally applicable whereas you know and certainly not repeatable in the future whereas for a human they understand a lot of the contacts for example they know that this year 2020 is a very unusual year you do not try to learn from the pricing pattern you know you're not going to see a 10 daily move you know several times in a month going forward but the machine doesn't know that uh so um the the problem of overfitting or data snooping bias was rampant in those days and we certainly suffer from that and that's why for the next 10 20 15 15 years i have barely touched machine learning i've barely used machine learning in trading because it was so difficult to keep it from overfitting but since then there are advances in both machine learning per se to to overcome overfitting because overfitting is not just a problem for finance it's a problem for any applications of machine learning but there are increasing knowledge a body of knowledge that were developed to overcome that in particular in the new network world we know that neural network was already developed again 30 years ago it was not new and but the problem was mainly again of overfitting you know it's very difficult to make it to learn something uh truly enduring in general and not noise but in in the past decade professor hinton at the university of toronto developed this technical dropout that he applied to neural network and which really reduced significantly this problem of overfitting in neural network and this really launched a renaissance in neural networks and which we now call deep learning i mean you know you can effectively say the deep learning this renaissance of new the application of neural network to many practical commercial problems has started with two changes in the current market one is of course the increased computational power now we can easily utilize a huge network of cloud computing resource to solve this problem whereas before we couldn't we need to buy a supercomputer which few people have but the second major advance is the fact that your dropout techniques and other techniques related that have been able to overcome overfitting in machine learning and that really brings about a huge revolution in the application of ai to many commercial problems whether you are talking about self-driving car or the um the siri or the natural language understanding and so on so forth you know that that is you know these two major advances have really propelled us to a new technological age quite frankly even though machine learning is nothing new at all and the second advance most pertinent to finance is the fact the realization that maybe some of the best way to apply ai to finance is not to directly try to generate trading signal but to actually for example a better way to allocate capital among various components of a portfolio or to generate a probability of profit like what we do to learn from your own human trading strategy rather than directly from the market because the market as everybody know has a very low signal to price ratio if you could find a signal likely many other people can find it too and very soon the signal will be arbitraged away so the signal directly learned from the market is unreliable and have severe alpha decay very short half-life so machine learning is very you have had very tough time directly learning from the market no matter whether you're using deep learning or using reinforcement learning or you you know all kinds of fancy technique have been flown at it and the result is you know at least a publicly published result are unimpressive however uh it would many people including the one of the giants in this field dr lo marcus lopez prado whose book you know advances in financial machine learning has been the best seller he advocated idea in fact that machine learning can be most profitably applied to learn from the trader's own past history instead of directly from the market and we adopted that approach as well and with a lot of success and so that that's i believe is the second event um that that's the second aspect which made machine learning and finance much more powerful today than before and it is the fact that you do not need to use ai to directly learn from the market and of course another factor that would make ai very useful in finance is somewhat indirect and that is to learn from alternative data because traditionally people you know use price and volatility as input very simple factors uh earnings and so forth you know in a simple linear factor model uh to make prediction but it is exceedingly hard for this kind of linear model to take into account um alternative data alternative sources of offer like credit card transactions satellite inventory or uh cell phone uh location data and so forth so forth um those data is very hard to develop a mathematical model for it as opposed to developing a mathematical model for prices you know it you can but it takes tremendous amount of time and the data are so heterogeneous a mathematical model that works for the credit card transaction may not work for freight traffic and it might not work for central imagery so all those alternative data can really only practically be explored using machine learning and because of the proliferation of alternative data you know machine learning has found a very fertile field to work on and to add value to um to investment management as well so that's i think is the um the third link of the tripod which propel machine learning in finance to the current level of enthusiasm one of the questions that i get a lot since i begin to publish things about ai from the guest interview is a lot of people ask do you think ai will replace manual trading in the future do you think at traders menu will have a chance to be able to compete against ai or do you think traders will kind of disappear well you know i think that ties in with the observation i made earlier which is that it is not particularly fruitful to apply ai directly to learn from the market so you you know the the the most fruitful use from from my experience of applying machine learning is to learn from a trader's own um investment strategy so somebody has to create investment strategy first and that would not be ai so i think a human trader is a crucial first step in any investment strategy it's only after the human use their sense of intuition and context and you know the the understanding about what where we're living in and filter out those signals before machine learning can really be applied and also the human will be a keenly aware of the uniqueness of situation and also of the potential for arbitrage activities that cause alpha decay so i i'm not afraid that human portfolio manager will suffer declining value it's actually i think that their value um will be you know the good portfolio managers will still be in extreme demand because you need them for the machine learning system to learn from the machine learning is my learning from them not from the market i'll be curious to hear about the process that you take to be able to create systems from the beginning to system working in the market how does that look like what are the steps people would have to take in their great systems yes so we um start with a conventional quant approach um and we you know we read widely you know there are a lot of academic publications of various arbitrage opportunities in the market so you know practically you can if you have time you can spend all your waking hours reading papers and do nothing else at this point a lot of those favor have been overfitted a lot of those people have failed to take into account transaction costs or illiquidity or difficult you know many many pitfalls but you know i would say 10 of those papers will still remain valid and from those papers and also from our own trading experience we will distill a few trading ideas they are essentially similar to scientific scientific hypothesis you know let's say newton observed an apple falling down and he hypothesized there's a force called gravity that takes the apple below downward trajectory instead of going upward so and you you know based on that idea you test the other consequences and we adopt a similar scientific uh process in development trading strategy we read paper we observe the market and we say oh maybe for example a recent idea was that oh maybe uh most of the return happened overnight most of return in the u.s equity market or other market happen overnight uh is that true of course we will use our own data to backtest that idea you know because it's been talked about on the social media and even on published research and we'll test that idea and we say oh yeah it does work it seems that indeed most of return were overnight but is this a practical offer is this something that actually one can exploit and then you apply transaction cost to that idea so without transaction cost that id may have a sharp ratio of 1.5 wow it's a time to get money into it but after you apply your appropriate transaction course you will discover that the shop might be dropped to points 0.2 most of that return is so tiny even though it's consistent but it's so tiny that if you actually trade on it it will not generate much return after your uh you know account for bit ass course and celebration marketing bank and so forth and um and you abandon it so that's the process but you know eventually you will find some arbitrage opportunity that can withstand tracks and costs and it works in consistently over different regimes and it makes sense to you and then you would paper trade it you know even though it might withstand backtested you know there are always things that escape a back test system alone and will fall apart in paper trading for example we used to trade a uh we used to backtested a japanese small cap strategy that has just amazing consistency practically no losing days in back test and then we tried to paper trade it and even light trade it in small size because it's a little bit hard difficult to paper trade and it doesn't work at all why because whenever we see a good shot that should have made tremendous amount of money the broker won't lend lend it to us so we had hard to borrow issues almost every day all the profitable trades were hard to borrow cannot execute well maybe we didn't have a very good broker probably if we were you know a 1 billion hedge fund and you know sign up with goldman sachs prime broker maybe we have a different um experience but we aren't at 1 billion fund and so we are stuck with some prime broker that would not let us borrow stock and so all this practical concern would destroy a strategy even so the strategy is sung and the the backtest is not biased or you know does not have any pitfalls but the practical aspect of the market have prevented many strategies from earning offer because of the many other issues such as hard to ball stock landing and boring issues and that is in addition to other operational issues such as margin calls so some broker in during the pandemic suddenly find it necessary to raise the margin three times or two times overnight without much of a warning and many technologies that are supposed to be profitable are forced to liquidate because of this kind of haphazard situation so again this is not something that you can backtest or you know even paper trading paper trade you probably have quite amount unlimited amount of margin but live trading is a totally different matter so um after a period of paper trading we at least eliminate potential look-ahead buyers we eliminate any sort of data snooping bias although we couldn't eliminate any unforeseen operational issues we'll go into live trading after a few months of paper trading and then live trading will reveal whether operational problems are exist to prevent the strategy from capturing the offer and if it still works after a few months we'll scale up so and then but that's not the end of process yes we might scale up but there are always periods of drawdown people will always uh we you know think that this is uh purely a flash in the pen phenomenon and the offer already decayed you know when the strategy entered into trolling you never know if this is true alpha decay or is a regime issue uh is it because it's just we are in a rough patch or is it because it truly has gone away for good it's very tough call you never know so um one of the things we always do to strategy is that whenever you know whether it's in the tough patch or not we are constantly improving it uh whether if improvement is in the execution side or it is in the strategy side or in the risk management side you know there are three angles to locate it up execution of course can always be better the strategy itself can always be better adding other inputs or looking for different regime indicators and then thirdly risk management can always be better whether risk management is based on machine learning or based on some fundamental insights they can be applied to any strategy to improve it so no strategy is static for us you know people would say oh you know we want a strategy with a long track record and say sure this strategy has a long track record like tail reapers had the track record since 2012. and um they thought okay that's good we want to invest in something that is a long tracker but beware just because it has long track record doesn't mean the strategy is static because every quarter we add new features to the strategy it is never the same strategy you know it's like you you dip your toe in the river and you know it's not the same river and you know any day you know so so um it's um you know a lot of allocators are under the um mistaken uh notion that uh you know you want to invest in a strategy long track right but long tracker first of all doesn't uh predict that in the future truly that it will continue to stay that way because of alpha decay and secondly you don't want a strategy that decided you want a strategy that evolves being constantly improved and so um and i think we take pride in that process we never stop research on a so supposedly successful uh strategy because it may be successful today may not be successful next month and how do you make sure then that you don't over optimize it the strategy won't try to improve it because i know a few people that are building systems and they add new things but they come to a point where they added too many things to their strategy so how do you do that balance yes that's very good question uh also um so some of the uh improvements the strategy make intuitive sense so for example people would say oh um you know your tail risk strategy doesn't work when the market is calm okay so i want this kind of office to say that well we maybe we should add a volatility filter to it so that it doesn't trade when 40 is low right that sounds sensible and you try it and you can see quickly whether it will work in the past and if it worked in the past and it also make intuitive sense by all means add it and we will see we will track it in the future to see if it makes correct decision this filter right and and if it doesn't we will have less confidence in it but you know if it works then it has both proof in terms of concept proof in terms of back test and proof in and now the sample test good to go we will keep that improvement and that applies to many other improvements does is it sensible does it withstand back tests and does it work out of sample that also applies to machine learning so in machine learning we always add features the machine learning program again is not static practically every month i'll research at the feature and test well at not necessarily add the future in production but at least at the feature in our research program to see if it will improve the prediction and once in a while we find a feature that actually improve in back test and we'll add it to the production system as well and again those features are all sensible and that's beauty of the way that we do machine learning is that it is not a black box if we are not just blindly alerting the machine learning system to make trades because the features are added are carefully curated by human traders based on their years of experience and observation of market we don't just randomly oh you know let's add 100 different technical indicators to see which one hit no we add indicators that we believe are unique uniquely suitable for predicting this market and sometimes they work so if i understand the world that means the ai will give you some ideas of things to modify or improve and you could decide if you want to use them or not yeah there are actually two ways to use ai one is just like you said um it's called a process of future importance ranking and we wrote a paper on it um the paper will be published soon um actually it's in available on pre-pin form it's called um can remember what it is called what features it's like if you google my name earning the chain you'll find that paper online um the idea is that machine learning can offer two things one is you allow you to pick what variables are important to your strategy and that once you take it you don't have to use machine learning anymore you can add that feature directly into your strategy you know let's say machine learning pick volatility to be an important determinant of the profitability and you say well that makes sense you know really you know you shouldn't trade this tailhead strategy when foreign slow well you might just directly add that as a filter you don't care about machine learning anymore anybody know how to add a volatility field to your strategy that's one way and that depends on features importance ranking which is a topic of active study and that's actually what our paper is about the second way is of course automatically include only the top features in in your predictive program and that is the other way that we have applied feature selection to and using machine learning too it is uh you know we don't necessarily have to take those important features and put it in our base strategy but we can simply automatically only include the top strategies as top features in order to build a model do you feel like investors are comfortable with investing in ai or do you feel like they have a resistance to prefer maybe investing in something manual there have been a increasing and tremendous increase in investor interest in ai funds in fact a recent study i believe by jp morgan have noted that funds that utilize machine learning has fastly outperformed those that does not in recent years uh that is a you know objective study you know jpmorgan is you know basically a sales cypher they don't care if you trade by machine learning or not so they make the same amount of um commissions um or other fees so uh and uh you know we we have uh you know seen really much less resistance to uh applying machine learning uh to fund especially the way that we use it because our fundamental idea is very simple it's very intuitive and no one will dispute that but the machine learning is used to enhance the basic investment strategy make it much harder to copy for example you know if anybody can do trend following buy high and sell higher like how tough is that any high school student can trade a portfolio with that kind of technique but are you able to select 170 features that monitor global markets in order to decide how much to allocate to a trade or or whether at all today we should trade at all that's not something that any 17-year-old can do even though she or he might have um some programming experience and because they may not have the necessary market experience because again these 170 features are not random we just we don't just follow in every single tech communicators and hope that it works they are carefully curated by human trafficking so the combination of human intelligence and artificial intelligence is powerful in this area of commerce as in others for example people talk about self-driving car you know actually it is much safer if the car will work with the human driver you know you hear about story about tesla their radar or computer vision couldn't differentiate a truck that has been painted white they think that it is just a sunlight and they just drive straight into it it's a very sad story but we take the same attitude towards applying machine learning to trading we don't take everything we have a human trader behind every machine decision we don't let the machine just take over completely and we go on a beach in the bahamas no we are watching the trade every day maybe at least one if not three of us are watching the trades every day to make sure that the machine doesn't make crazy decision and that's the same principle that i believe should be applied to autonomous vehicles is that you should not just say that the human is useless let's not judge his judgment no the human judgment plus the machine judgment is what makes the driving experience much safer we don't want to get rid of that layer of risk management in my opinion of course many self-driving car producers totally disagree what do you think is going right now because we already have a lot of progress like we mentioned before we went from 20 years ago to now it's different and what do you think is the next step in ai the next step never um you know the progress never stops so the next step is like next hour next day so it's not like there's a certain quantum jump that we're waiting for there is always more alternative data to learn from that requires machine learning there's always better technique to select features so originally there's only one feature selection method now we have at least three probably more that i haven't even heard of techniques of feature selection improves if you keep up with the machine learning literature you know essentially you cannot possibly finish reading everything you even you default your entire entire waking hours to reading machine learning papers or even machine learning applied to finance paper you will never finish them so progress is happening at a rapid clip every hour not every month every hour because as i said the number of paper published is impossible to keep up no one can keep up with them so you can imagine that you know uh the next step is is right now is in the next hour there will be several new papers that are published and one of them might make a major advance in this field that's amazing and someone being as in new to ai where would you recommend them to kind of go to start learning about things oh there are a lot of very good books uh in this in this area as i said you will not be able to finish reading everything so let's start with the most pertinent books i would say that for quantitative training per se you know before you go on to machine learning of course i would recommend my own books the three books that i've written starting with quantitative trading and then algorithmic trading and machine trading and my book also referenced many other highly uh readable books and blogs and articles as well so it's not just my own ideas i i make an effort to make them educational and point people in different direction with in terms of learning resources in the books and then in terms of the financial machine learning i mentioned dr marcus lopez departures book they're highly readable highly acclaimed and then he recently published a follow-up on that book i believe it's called machine learning for asset managers and which go into some of the even newer techniques as i said research happened every hour so just maybe two years after he published his first book already there are many improvements to the techniques he discussed in the first book and then i'm sure that next year there will be another book that pushed the boundary forward another major step so but i would suggest starting with those books and and actually one of the best way to get a list of all the best books in this field is to visit my blog epchen.blogspot.com i have a book list a recommended book list and those are the books that i read and love and find them very useful not just my books obviously but many books in the quantitative finance world that i have i found highly valuable and uh so readers can just go to my blog and check them out perfect i'll make sure to put the link in the channel for the podcast and other videos as well in the description people check it out all those books definitely recommend people to read your book as well like you said quantity of trading so to talk about things like how to evaluate strategies how to back the strategies the right way it's so definitely useful for people that are looking into kind of starting to be marketing data driven and kind of starting that world of of using data to trade so definitely good book on that thank you so what can people find you know connect with you or reach out after this interview it's very easy i my email is on epchen.com so go to my website my name my initials e p and then my last name chen chan dot com you will find my email there and all the resources that i put together uh for traders uh experience or new to to get involved excellent and ernest what are your goals for the future in trading or outside of trading our goal is to develop more strategies um not just tail hedging but um in all directions you know volatility trading has been one of our focus so we have a full city trading strategies uh that doesn't trade very often so we are under taking an active effort in go moving into volatility trading involving either faulty futures or options we find it is a extremely fertile field for algorithmic and machine learning approach in this field and that's in terms of the investment management business that's our our current focus and then of course for the machine learning focus as i said our new company predict now.ai is gaining a lot of traction uh surprisingly fast um i just published a blog post about um maybe a week or so ago and it was already there's a bit of a bus out there uh on using this server this is so easy to use um so that's going to be um also the secondary focus that we have which is to commercialize our ai technology that we use in-house with a lot of success what is your main motivation to keep creating strategies and keep working on ai what motivates you to to go beyond it's a very intellectually exciting field i i mean but you know my education was a as a theoretical physicist so i have not done a single day of research in theoretical physics after i got my phd unfortunately but because at that point i already know that my passion is in machine learning and statistical pattern recognition and has been my passion even while i was a physics graduate student i always try to look for projects in statistical pattern recognition because it's so very close to statistical mechanics a lot of the major advances in neural network were done by physicists dr hinton for example undergraduate degree was in physics not in computer science so um so that has been my passion ever since graduate school machine learning and statistical pattern recognition and what and there's no more challenging problem for statistical pattern recognition than the financial market as everybody has come to recognize it's much harder than jail driving car much harder than playing gold much harder than responding to your siri queries on your iphone it is truly one of the toughest if not the toughest machine learning challenge and i'm very excited to be part of it and even to this day after 20 years of research that's very awesome to hear and it's been a really good discussion really interesting for me to learn these things and how people like it as well so i suggest people reach out to you connect with you on your blog and also read the book we discussed and thank you for your time ernest i appreciate it and we'll hopefully talk soon thank you very much for inviting me it's my pleasure [Music] you

Channel: Etienne Crete - Desire To TRADE

Views: 13,878

Keywords: ai in trading forex, ai in trading, ai in trading course, ai in trading stocks, trading forex, forex, trading, forex for beginners, forex signals, forex strategy, day trading, how to trade forex, algorithm, business, english, finance, money, desire to trade, ernest chan algorithmic trading, ernest chan trading, ernest chan quantitative trading

Id: pLslJHWpIiQ

Length: 39min 53sec (2393 seconds)

Published: Sun Sep 06 2020