Multi-touch Attribution: What am I training for? - Sri Sri Perangur

hello everyone can you hear me okay thank you all for coming here today and so yes my name is Sri Sri just to introduce myself and yes I'm working currently at Spotify I just successfully finished my first week yay and before that I was working with Skyscanner up in Edinburgh in Scotland and and that's whom all this research has been done with and before that I was here again in London and working for a social network Capital Markets defense industry but throughout the experiences data has been core to every job I've done and and who has programming so it was inevitable that I went into data science and so how many of you here know what Skyscanner is awesome this is really cool and I love it every time I meet anyone in the yeah Skyscanner we totally knew that this video what's that and so this guy's kind of put those of you don't know is a travel search site and where you can get great prices for your flights hotels or car hires and and they have over 70 million users every single month and the concept of multi-touch attribution is a really important one for them because they're very much an ecommerce industry but I'll explain why it's important in a second so just to give you an overview first and we'll be going through what multi-touch attribution is so if you don't know don't worry about it and and then some of the key challenges around multi-touch attribution which is to do with the training data and and then we'll talk about the solution that's been designed which is a simulated training data environment and so most of this talk is predominantly data science heavy we won't be like looking into the code and but it's getting you to think about these concepts in a new perspective so let's begin so for those of you who would have seen the hottest hour would know what a great performance Gary Roman made of Winston Churchill well deserving the Sears Oscar for Best Actor his shiny new toy which he seems to be really attached to and now when we think about who in Gary's life gets credit for his Oscar there are a few ways to think about this so you could say right let's give credit to the last person who had something to do with his win and the director or you could say let's give his mum the credit after all if it wasn't for her he wouldn't exist she wouldn't exist and or you could say well the dad certainly had something to do with his existence so you've got to give him some credit as well so if you want to keep the piece you could just say let's just give everyone even credit now but that doesn't necessarily make us any smarter so now we could say okay let's give the last person a little more credit and the first person less at this point we're designing an algorithm but still quite making up the rules here so instead what we really want to aim for is this custom values that for the various players which is hopefully a true representation of their contribution to the scenario and this in effect is the attribution problem and you'll see this in a few different forms and across the e-commerce industry and in other situations like when it comes to Oscars and but this is what the industry is trying to achieve today and this is again a storyline from segment IO who did a great job of it so I highly recommend you read their blog as well and so where do we see the situation really occur in e-commerce so when you think of Netflix a whole load of amazing content if you're anything like me you can't go a day without it so when we think about all the various contents what kind of content is really keeping users coming back is it the Netflix originals is it and the BBC productions like Sherlock or extraordinary Holmes or is it the more cutting-edge stuff like Black Mirror who would love that and so what kind of content is it and or is it just the classic and she was like friends which people don't mind watching for like the hundredth time so where should Netflix really be investing to get more content to keep you coming back so this again we see a situation like this with Amazon again tons of different products they started off with the books and but now they have music and video and the popular shopping but even in shopping you have different kinds of products you have everyday products like groceries or you have small tech and then you have big-ticket items so which one gets you coming back is it the big-ticket item you buy really what's bringing you back or is it the feasibility of getting everyday small products as well and that makes your life easier and again and you see this it's 45 as well whole lot of amazing music but everyone's really different so you see the music surface in few different styles so it could be your mood playlists or it could be discovery playlist which is more tuned to your taste are just the classics maybe you just want to have a platform where you could just go see what's popular in which country classic chops that's it so what kind of music is again making that impact so in all these scenarios we're seeing multiple players and we're trying to figure out who gets how much credit so now we see how this comes across in the marketing space and this is where the attention is of the e-commerce industry significantly so I've tailored this use Kies particularly for Skyscanner but it's generalizable to any e-commerce industry so think about say you you're on a social network and you see an ad for an article that comes through which says the ten places the Brits are bound to be this year and you go okay what is this okay let me go check it out where are they bound to be and you're just kind of exploring what's happening there and you see maybe Budapest there you go Budapest okay I didn't expect them now you're just exploring the idea no real intentions of travel or booking now from there a few days after or a day after you see again Budapest kind of come up in different contexts and you go now it looks like I see Budapest everywhere okay let me go read this other article about to do cool things in Budapest this time it's not Skyscanner article just an article aimed by a blogger now as you're reading the blog to the right you'll see ad which says hey Skyscanner remember us do you want to see where the flights are the prices are to go to Budapest maybe this weekend and you go yeah well let me just explore it I've never really considered the option then a few days go by and you've been talking to your friends about Budapest or your family and and turns out this might actually be fairly interesting so you wanted to share that article you read a few days ago but you don't quite remember that company's name sky something and so you go okay let's just ask Google so ask Google 10 places Brits are bound to be this year and organically you get like a whole lot of results in the top one of Skyscanner and so you go to go take the leg share it across spread the word okay now the pods brewing so you're actually thinking about Budapest and a holiday it's all coming together and it's been a really long winter and you want to get away on a holiday be it to Budapest or a time Sonny in place and so at this point now you and your mates have decided you definitely want to do this and you just go right let's just go to Google ask for cheap flights to be the pass see where it goes so you do that and this time as you get the results at the very top you get some results which they add and one of it is Skyscanner cheap flights to be the pest and you go ah that's what those guys are I almost forgot about them they were pretty decent so let me just go and check it out so you come back one final trip to finalize your prices and this time you've heard of Skyscanner a good few times so you just come there directly and book your flight now this is a multi-touch situation that in the commerce industry so which of these points gets the credit for that booking so the industry norm is to give the last one full credit now in this situation since the last one was you directly coming to Skyscanner when you look at the financial reports that doesn't seem quite right we need no marketing people are just coming directly okay so the rule they have tailored it a bit to say give the last paid touch point the full credit now this teaches us a little bit more about the situation but not enough what about the other three touch points they get no credit whatsoever and the industry is well aware that this is not ideal what we ideally would like to see is something more like this where the credit is distributed amongst all players and this is where we can actually start strategizing and seeing what's happening how much investment you can stop making various touch points however the industry is still stuck at this because we didn't have the computational power or big data to enable us so now that we have those what can we do about it so now we come to designing a solution so the first thing always to do when you come to designing a solution is start with the simplest thing so the simplest thing would be to just give everyone even credit or give them buy time decay or custom rules however we know we can do better so let's take that 70 plus million users data study it understand what's working for the users or not did design a data driven attribution so now when we come to the detailer of an attribution we still have many choices in the industry out there but some which have really stood out our two methods one is Markov chain attribution where you basically study the various life cycles of the user and put design that into a probabilistic graph and now see I remove channel 1 what is the loss or the impact in terms of conversion and this is the Markov chain attribution and then there's a very compelling competitor which is the Shapley value attribution which is a game theory approach where you say let me consider all the different situations the channel has been seen in just seenin we don't care about the sequences and average out the payoff it would receive in all those scenarios so computationally it's seen it would seem like Shapley value would be easier to implement than Markov chain however Shapley value does well up to you like 10 to 15 channels after that the payoffs per channel get really exponentially complicated to compute so Markov chain do initially a bit more cumbersome to implement Steel's really well now we have some strong approaches both with its pros and cons however we don't know still the answer to the key question which is which is the best method which is the best method for the problem of multi-touch as well as for Skyscanner in particular so we have the data driven approaches we have the rule-based approaches everyone deserves a fair chance at winning but we need a way to tell which is the best solution so now we have the methods ready to train where's the training data so let's take a step back and take a look at a typical machine learning and problem say houses in San Francisco houses in New York and we have some attributes there about the houses that don't quite giveaway which city its indirectly and now we need to design a classifier using a training method to predict the label of the city so clearly we have a goal we have a map and we have a route that's going from that from our current position to the goal and hopefully it's the fastest one so thinking about it from this perspective when we take a look at multi-touch attribution would see that we have the multiple different approaches we just discussed and when it comes to where we stand we know that because we have tons of user data now to understand the user flows the marketing channels the sequence of events occurring and so on however when we come to the goal we have no estimates of the channel we don't actually have the actual values so what do we do with them we don't have the actual values because those values are hidden in our real world they are what are known as latent variables now latent variables we can't observe directly but we can observe them through byproducts so when you think about say the value of your university degree or courage or your help this is where and we cannot observe it directly but we can infer it by byproducts so for university degree it could be the kind of job the candidate gets or the kind of place they live in or how much they earn and for courage making a daring leap or getting the courage to speak up or with health and putting your body to the test to see what you're capable of and your blood pressure your fat levels and so on so even mathematically we have such variables and one such is a multi-player payoffs so in this example of the Oscar we don't know that the director should be getting forty five percent credit or the moment should be getting five these are just estimates and similarly for the marketing channels as well and as label there these are hypothetical values and so what we really rely on is the knowledge of our field experts in this case marketing experts in order to understand what would be the payoffs for the various channels and ideally what would happen is that we would be able to keep everything still change one channels perspective and do an a/b test but this if anyone has been working in the marketing scene would know that is really not possible and first of all marketing is happening off your site and on top of that it's a very dynamic environment so business strategy is actively changing and there are so many variables out there that you don't control reason why it becomes extremely challenging to try and put and the real value of these various champ so that's why we really need our field experts been working in this industry for years and understand the value of various channels through their experience so we could say right brilliant we have a gap let's plug in our experts and now we have a closed-form a problem that can be solved however even with our experts presence we wouldn't be able to say how much a method is better or worse than another it wouldn't be possible for them to say ok it is 10% better or 1% better better or 7.5 right so it's far too much owners on the expert excuse me to try and think of all the different variables that could be potentially impacting the situation and then make that decision it's far too much onus on one expert even or even a group of them so this is where we've designed a solution of simulated training data so what we do is create a simulated marketing environment where we take all these expectations from our experts as well as the data to capture and then we start simulating different goals where we can start training so to understand that a bit better when it comes to the expert again key expectations and the main one which is the estimated goals so we can say right marketing channel values let's put that in and there are other assumptions like say brand advertisement and adds value but not immediately but over a long term vendor now that is something we again need to understand assumptions like this from our experts and again we can get the data to support that but these are key factors we need to make sure we incorporate into any paradigm that we're creating and similarly we can take the known data traits we see in our Big Data and add that in so we could say right distribution of the various marketing touch points first initially top-level let's make sure that's alright and and these are the results actually we had where the blue one here is the real data and the orange one is a simulated data I did have to abstract out the access unfortunately though I absolutely hate unlabeled access and and then we have the distribution of the various marketing touch points so now that we got the overall expectation set now let's go in and understand what kind of distributions do we see for these various marketing touch points marketing channels so do we see an alpha distribution or a beta is it normal is it negative binomial and we can do a bit better rather than just see that's the distribution we can actually tune it a bit better to see how exactly that would look like and then we go from a user's perspective and say right now let's make sure that the expectations on what you see for the first touch point of the second are set and and then finally again from the channels perspective let's make sure that the key metrics we know of as of now such as session conversions conversion rates all of those expectations are as required from the various channels perspectives so these are the key perspectives we've used to create the simulated marketing environment and you could do the same thing so now now that we have a simulated environment we know what is our environments where what kind of features we have we know what is the goal we're trying to aim for and we have the various different approaches that we can start working with to aim for the goal and optimize this so we can start comparing the different methods and once we know which is the winning solution start actually tuning it to make it more accurate this again would be exceptionally hard if you didn't have and an accurate way to measure this and rather dependent purely on see an expert so now and when it comes to the results and again a the mock just another point for the simulated marketing environment we've designed this and tailored it particularly for Skyscanner and and even in doing so this approach is quite unconventional so you'll see a lot of researchers working on estimating latent variables and this is not the traditional approach taken however we wanted a pragmatic approach with which we can work and make better but our main benefit what it's doing is taking all the expectations we have in our in our minds and putting it into code somewhere we can and set the expectations evolve it over time make it better and as we get to know our reality more so this is very much like as an MVP that we improved on with a few iterations and as we learn more about our reality we start putting those expectations in so the results for multi-touch attribution again will be focused for Skyscanner so with all the various methods now the simulated approach we can see that the best solution has a ninety percent accuracy in estimating the simulated marketing and channel values and here we see the top results and the best method a actually yes so our current best industry approach is last touch attribution and that's where it is at the Third Point and our baseline is even touch which is again just giving whatever is seen the most highest frequency more credit right and then our best method is has been given a codename Z and because it's proprietary and it can't reveal them and however we see that even the best method is 10% better than the baseline and what's really interesting is also that this is the first time we can see that even touch which is our baseline is two point six nine percent better than last touch where the industry is up so this to me was really interesting and if there's anything I'd like you to take away from this talk it would be this one slide the ability to clearly measure and see where the various methods down has been exceptionally useful and and this has really helped us to realize where the various methods dan what's good for us and how to actually make that more accurate and and also that that's two point six nine percent is your compelling argument that you should be moving more towards data-driven solutions than two or other rule base and and it's again only like two point six nine percent better but that again is fairly compelling and have you been using the wrong rule even though we have the choice have you been using their or wrong rule as well and what's really comforting to know is that the current best data-driven method is much better than the last touch method we're not going for one percent or two percent accuracy it's ten percent so now you guys could ask me what if the channel estimates are wrong simple as that and maybe it's not meant to be X but it's meant to be 100 times that or maybe 1% of that what that so good question and what we could do again is leverage the simulated marketing environment where we could put the best solution along with his competitors under the test so here we have the ability to paint different scenarios now that we put all the different variables we know our key so we design stress testing scenarios on the three key teams one let's just change the probability of the channel ever being exposed to the user to the probability of purchase maybe we've been saying that the purchase order is ABC with a having the highest conversion rates whereas it's supposed to be CBA so we can do that we can paint that scenario added to the stress testing and finally the one where we fluctuate relative touch points so earlier I mentioned how you have advertising where brand advertising has an impractical later date here we could say you know what brand advertising doesn't have impact at a later date it just is what you see on that day and that's there so let's paint that scenario and or that the brand advertising doesn't have a long enough tail and actually we think it's a few days whereas its impact lasts a few weeks or two a year so we paint as different scenarios and then put it to the test and and what we see is as we see here so how to read this job so if you're close to the go most accurate you'd be here to the center right here whereas if you're am overestimating to the goal you'd be to the right and if you're under estimating to the goal it'd be to the left and the scores here are the average per channel and in the twenty different scenarios and I've just shown like the top five methods again we had to censor the results and but the method Z we just saw previously is the one in green so when we see here what method is winning the most and I've just put little stars to make it easy to read in most channels it's the green method that's winning throughout but not in all and where it's no winning it's not winning by a very small margin that's why overall still it's the best solution to adapt to change so this is where and we now have a solution that is 90% accurate and the best at adapting to change and now we come to the power of applying this to production so we've learned from a simulated environment how to how to what method to use to to design a solution and what particular tunings as well that would be most ideal and now we can apply the same tunings with the same method in reality in practice so for the simulation stack and we've taken Skyscanner data to understand the attributes of the various channels and users and so on and designed all those with those expectations purely in Python thanks to the N number of libraries we have and to produce the simulated values and for the production stack and we have again Skyscanner data but all the entire implementation has been done for the bidding solution with the tunings all in apache spark and absolutely love apache spark it's made my life really easy and I could do a whole talk on it and so now we have the final results for multi-touch attribution that the team can start using and implementing it in practice in order to test out life to see what's happening and as we learn more about our reality we start placing those expectations in in the simulated data and thank you in conclusion sorry and so we've been able to make understand the accuracy of different approaches comparison comparison of different approaches tunings of the different solution stress testing the solutions and ease of production and this approach has been really invaluable to us so thank you so much here are all the links of the content and the images and thank you to my Skyscanner team that have made this possible it's been really exciting to work on this if any of you are interested in the slides I have the link right there feel free to and have a look and let me know if you have any questions and see ya thank you thank you so much thank you for a presentation I have a question do you have also like real results because why I'm asking is those 90% of gray or better better results are only because of the simulation and when I'm seeing like a little bit hook in this that's you were computing the same I don't know let's say a conversion rate of each channels but those channels acts differently when you put like more traffic into it like when you are struggling with saturation of the channels so it's like it could be a little bit tricky to deal with it so if my question I do deal with yeah no that's a very fair question and this is why we also use the stress testing scenarios to kind of simulate that but you're right in that you can't really expect the saturation level unless you put it into practice so that is what Skyscanner is actively looking into now but as you can imagine to understand the saturation levels and you need to run the test for a long time in order to get that feedback at least a good few days so or even weeks so that's where we created the stress testing scenario in order to get a quick feedback so we can actually start shooting it a bit better and and then once it's ready we're implementing it now in production but now in we are still in the process of testing it out and validating it which could take a while because it just can't be in one test that you do and it's like all right done it's it's a done deal it's gonna be taking a few good phases of it to test it out and understand how and if it's actually working and in that as you might imagine it's not easy to run a test in the marketing space so again their challenges with that and we have a really interesting approach to deal with it so it's happening however those results I wouldn't be surprised if it didn't come out for like a few months until we are conclusive so that's actively happening so I would say you should definitely watch out these guys kind of engineering space on medium they're very good with blogging things out there and when the results are out they would put it there so it'll be really interesting to see what happens thank you great question thank you do you have more questions thank you I've been actually dealing with these things as well and I was wondering how you have whether you have thought about what to do with interactions I said you you couldn't get so if you think about a journey of a customer to conversion I mean in a deuce in the ideal situation we would have all the interactions with any channel but in reality in particular was a big players like Google and Facebook they don't really want to give the data where did you sort of try to go around that what did you think about it that is a fair question again and and it's the kind of problems again anyone would see if they're trying to understand the value of say TV add value again because again and we don't really have exposure to all the touch points true so this is where we introduce fair amount of randomization however again we work with the experts to understand where those potential gaps are and put those assumptions in and this is where we wouldn't know and everything we need to to get a complete picture and this is why again for Skyscanner there are a lot of users who don't sign on so we again are base cookie cookie based predominantly so understanding that space again and needs elements of randomization learning more what's gonna happen and as we get more expectations and tested out light we put those expectations back into simulated data to make that solution stronger thank you okay we probably have time for a few more questions thank you this is not it's more of a business question other than a data science one how do you manage to communicate these wonderful results to the senior business stakeholders in a convincing way are they already sort of told to listen to your do you have a secret to share with us so very good question and it is a core part of my job to make this understandable so again if it hasn't been comprehensible please do email me and like let me know if you have any tips or feedback and the team at Skyscanner have been very good in their data-driven approach in mentality and they make a conscious effort to bridge the gap not just from the tech side to business but even business try to meet you halfway so stakeholders wise I've had some of the best there and who try and understand the gaps and really bridge their understanding and what's also really good is to understand what's happening in the industry out there because you could see that and even big labels like Google and Facebook are making their solutions and they're moving towards this direction so that kind of is compelling but it's also good to know speaking to the industry out there that everyone is uncertain which is the best solution so they've been very proactive and really take one of the core values to be faithful with so tested learn share those learnings and go forward so that way it's been amazing to even invest in this understand this and all the senior leadership have been behind it we thank you so much for a wonderful talk let's give a big round of applause to three you

Channel: PyData

Views: 12,179

Keywords:

Id: LDU87WlOhWg

Length: 37min 24sec (2244 seconds)

Published: Mon May 28 2018