Andrew Patterson - Outlier detection methods for detecting cheaters in mobile gaming

I was I was wondering how many people would find it interesting but I was told on Friday night that there's probably a spurious correlation between people to enjoy pythons and data science and clearly enjoy mobile games so it's a pretty good pretty good thing there and so I'm Andrew Karsen on a date scientist at natural motion I actually need all the analytics data science for one of our games done at Titans and but this is kind of out these are the kind of games that we've got so don't ins is kind of a mid core game where you kind of battle but battles like a social battling game so you're building a city upgrading your troops upgrading these Titans and battling each other and trying to basically just win the game so this r2 is a more of a racing game so you're doing a similar thing but you're upgrading your cars and then you're racing against your friends and things like that we have the CSR franchise which obviously sort have been built on and then we have some other smaller games that kind of we start out with I would implore you all to download clumsy ninja because fun really fun and so I'm going to talk to you about how we deal with cheaters specifically with Don times but actually this this whole concept has happened across dollar times and CSR - and they are obviously in social games a huge huge issue because it's not just that they are cheating us they're actually cheating the other players that are playing the game first one I just go through quickly the free-to-play model I think everyone understands most mobile games now in this free-to-play model which is great because we can get our games into the hands of as many people as possible who have a good enough to advice to play them but it comes with a problem so does anybody understand what this percentage might be yeah well not necessarily Wales but this is actually just the percentage of people who convert in free-to-play games and so it's I mean amazingly we hope that we have a huge amount of his user base playing but a very small amount of people that actually give us revenue as a business we still you know it's to knee to make money and actually that only comes from about 2% of our players so if we're thinking about detecting cheated in that game it's a real problematic thing because if we're thinking about them as out in the data that we're looking at actually 100% of our revenue is also coming from outliers so it's them it comes to you know it's a problem that we have so I'm gonna quickly talk through just quickly how what you would cheat in a game this is donna titans this is my daughter titans on my mobile i'm crash kid only level 4 started on my start on a new mobile recently and so a lot lot of games of them a lot of games have currencies in the game so obviously that's a really really obvious area that you had hype the game give yourself more currency and to do things buy things in the game things like that potentially gems which is kind of a hard currency that's where you skip waste come from like skipping through the game comes from and basically buying some of the bigger items and this is where kind of our IPS are attached we have vp which is social currency and so what you use that for is oh any of you battle somebody you get more vp you rise at the ranks get rewards and things like that and then there's loads of other things like your actual general player profile which gives you extra wasn't another so i just want to show you this for a second because here this is the kind of social map in our game and we've got carlos 8 9 or 5 here now is he a cheater as a player I don't see him as a cheater he looks very similar to me he's got a few more lands but it looks very similar to me now he could be cheating but Ness this isn't necessarily the problem like actually don't mind if I really care if he is if it has ways of nominally cheating in our game because it's not really affecting me but I'm not sick perceiving him to have a hair kind of to be better than me because of it because of that I'm saying hey how you know how is he upgrading himself so quickly uh-oh and when he's obviously a cheater you know and I'm going to as a player not like the game because of it and so what we're trying to do is find it we don't have to care about all of this cheats in the game we actually just wanna care about the player facing ones the ones that are really going to affect other players and in the game so just gonna quickly run through I'm from an engineering level how people cheat so these games are always online so they're always talking to the game server every transaction you do every kind of buttons like you do with we're talking to the game server so from a client side there's the obviously obvious tax you can actually download hacked builds off the off online this is not one of our games I'm not going to show you show you the apk for our own games this one's for clash Royale please don't also don't download that one either and then so that these already have like it's basically the easiest way they're pre companies so like you'll have loads of gems to play you might even just start at a higher level and you know it's effectively you haven't spent any money and you can just rule again these are actually generally quite heated a to em to stop there's also these assistant applications so you could have a completely fine gaming on your mobile and use an assistant application to just change values in our metadata and this is this is a harder problem because actually people generally do it in smaller smaller increments like just when they need it they'll change the value and and so you may even you know from looking through the data you may look quite normal but actually you're kind of just you're edging you're just edging there with your cheatin there's also like Network edits which is really really tough so your client can be completely fine and the transaction is sent to the server is then just captured and you can then edit that so you don't even need to be online at this point people can actually just capture these them capturing these kind of transactions and then just send them to the server obviously the survey is robust enough to understand most of them but this is an arms race like every time we find every time we find a way to stop something some clever clogs finds a new way to do it so it's a constant arms race and this is why rather than dealing with from an engineering point of view which we do we should also need to double down on kind of catching users within game so this is where it comes from an outlier detection problem so as the transactions come in the easiest thing to do which is so a lot of the ways that we deal with cheaters is just deal it from a transaction level so this transaction has changed our currency by X amount and you know whether that was an outlier or not but as I've said like a lot of the time it's just scope creep it's like creeping creeping creeping cheating so actually what we need to do is aggregate the user level data to kind of understand overtime how big a difference if they made that Network than we were expecting them to do so I'm going to walk through some kind of different ways that we've dealt with you know cheating as an outlier detection problem start from the simplest approach and really just move forward with that so obviously you have got single variate solutions and so if you look here we've kind of got battle win ratios here or play a player progression on your first day and you this is in in terms of the mathematics is something called kurtosis which is really simple thinking it's just really the amount of tail that you have on your distribution you need to assume that high kurtosis means that this variable has a lot of outliers compared to your normally distributed data and so we can kind of use that to understand what variables perceive as outliers versus versus not so you can see here for player progression there's a huge long tail here so it's very easy to expect that you know these are these are huge outliers here and and it's not possible to get to some of these levels on your first day so we know for a straight away that these people are cheating and so it's a really really simple method and in fact actually we don't necessarily need to do some deep level data science here we can just put in theoretical maximums and say you can't you know it's not possible for you to get here whereas actually with the battlin ratios it's a bit harder like you could you could be winning a hell of a lot of battles compared to what you should be doing but but there's more nuances and more issues there so if we look at from a multivariate solution you can obviously see that the battle win ratios may affect to be affected by the troops that you take into the game so obviously that's quite correlated but they're still going to be out lies there so we've created in a single variable create a distance metric from a kind of averaging I mean so how do we do that in the multivariate sense it's not just a kind of Euclidean distance so what we want to kind of want to do is create that ellipsoid around so that we can kind of understand that this is an outlier just as this is an outlier even though for example they are actually very very different in terms of distance from the average point of our data what we use is the maillot notice distance and what this is doing is is really just creating a similar distance metric but taking into account the variance of each of the M variables the easiest way to think about that is if we on the two-dimensional space if we actually just took a simple principle component so in fact actually we've just scaled down our variance not necessarily the fact that they're non correlated that doesn't really matter here but and the fact that we scale down our variance means that it's now just a simple kind of nuclear lien distance around that space so that projection now if you think about the malarious distance it's basically just a Euclidean distance in the kind of projection a scale invariant projection cool so yeah so K back to the problem is are these cheaters I mean we've got people who who are winning a lot of battles but they've also got a high troop score maybe the troop score is the problem that's the thing but you've actually also got outliers here that are just people who are really bad AI they're not very good at the King so this is the thing like whenever we do as we going through this it's like outliers doesn't equal cheaters at all it's actually just so we need to be really really careful on on what we're actually detecting as an outlier and what we're actually pushing through as a potential cheater in the game there's also a notion on this so for it this outlier is kind of around around completely around the distribution so even if we calculated the like the ellipsoid the mean is probably even if we take some of those outliers out that mean is probably going to be in a similar place so we can kind of properly understand that a little not necessarily having to worry about the outliers this is a really really actually just nicked it straight from the scikit-learn website but only because it was just it is the simplest approach simplest kind of graphical representation of the importance of kind of understanding a robust measurement of that kind of ellipsoid around your distributions so if you took actually an ellipsoid and included those outliers it's going to be very very highly skewed so what you need to do is kind of understand what the ellipsoid might look like through kind of samples of your data such that you're trying to take out those outliers and try and get a good graphical representation so the mean of this ellipsoid is not necessarily the mean of your data it's actually it's actually the mean of kind of your non out Lyday to your inliers you can use two approaches for that there's a minimum covariance determine which is actually inside Caitlin there's also a minimum volume ellipsoid which is a slightly more robust mechanism but effectively the minimum covariance term is a kind of direct calculation a minimum volume ellipsoid is is an iterative method where actually what we going to do is just sample smaller sets of our data calculate an ellipsoid that covers a certain amount of of our data points and then just take the smallest one - the one with the least outliers in it that's the kind of a sort of calculation of what what I kind of mean and the very covariance of our data could be so this so just skipping back this works well for normal uni-modal data but as you get to higher dimensions this is not necessarily what you're going to see like even and so you need to kind of abstract this and think about a different way of doing it so rather than a distance measure I'm going to kind of show you an error measure and what we're going to do is use neural nets so I'm guessing you've probably seen picture like this about 10 times this weekend but what's them so what we're going to do is actually just create a really really simple neural net which is going to take our input vector of our data of our users and we're going to learn the kind of structure of this data and output and try and basically replicate that data from a lower dimension similar to what you'd see is maybe an autoencoder the work the reason this is good for outliers is actually as it learns the structure of the data any variable that any any kind of sample that we put into the neuron there that doesn't actually fit that structure is going to it compute it's going to be really really bad at predicting that data so in fact actually what we can do is create an outlier factor or an outlier score similar to the distance metric but actually it's just an error metric you know we've built on your net to understand the structure of our data but the as you put a sample through in it and it's never seen before it's not going to be able to predict it and therefore it is an outlier based on kind of what no data normally looks like so it's reasonable to assume that and the way the way that this works is actually is you want to represent your data in a lower dimension so you don't want to actually 100% replicate your data you just want to kind of estimator it's reasonable to assume that in your nets I mean you can you can optimize in your very very well to the point where even if you take a sample of your data and maybe it has one or two of your outliers in it even if it has a couple of samples it's probably going to try and understand you know if there's a good chance it may understand the structure of that data and so actually you may find might find two outliers in your data not having a huge outlier factor so what we kind of do with this replicate in your net is there's from the paper directly there's a activation function which effectively just creates a step function so rather than this middle hidden layer having a fully continuous output it could take any value we're actually only going to give it a certain set of values so effectively just clustering our data down into these into this hidden layer so for example here we've got three inputs and effectively what we're saying bit of three continuous variables but we're effectively going to try and represent that as just two neurons four steps eight piece of eight bits of data and then I met is going to kind of try and predict out that data again and there's going to be a an outlier factor based on kind of how badly we did so there's a big problem of this in an implementation standpoint this completely makes sense for for out like prediction from a theoretical standpoint but from a so we're kind of actually implementing this is really tough these step functions if you're for example training you honor on kind of gradient descent it's not possible to move through this activation function because the different the M if you differentiate these values they're all just zero on those kind of flat planes so in practice what we actually did is just which is actually just kind of relax these conditions and bring it back to a continuous variable which is you're probably thinking what like you just talked about how important this is and you just thrown it away now so but if it but it's an important thing to just note because we can still use this continuous distribution so you have two likely continuous activation function but given that we know the problems with it it means that we just make sure that when we're training it we kind of have a very very early stopping criteria and in practice in practice like when we actually did this we probably had a very very early stopping criteria because we actually had a lot of false positives in terms of what we thought were outliers but it was M but we found that this was probably the best approach this wasn't definitely the best approach and going back to this versus the distributions is we were able to feed it much more much different data and you know kind of both bimodal data or very very high skewed data and it dealt with it and very very well so that's kind of a whirl whirl whirl wind that goes through em out lie detection and so let's just touch on a few other points of just kind of a whole process so before we even put into the outlay detection system like whatever model we use how do we deal with variable selection when we're talking about outliers it's not it's not a simple cut and dry process you do a classification you're just going to kind of take the mutual information it's like a very very simple approach is just take the mutual information of each variable and you take your top X variables and run that through your model or that topics just being a high parameter to kind of optimize but in out in kind of out lie detection and particularly out lie detection trying to find cheaters we can't necessarily just take variables that have a high a high difference between their inliers and outliers so this is one way you can do it you can actually algorithmically understand this by taking a variable the high kurtosis knowing that the difference between your outliers and your in lies are very very high but you can't necessarily do that in practice because you're outliers don't necessarily the outliers I'm not necessarily definitely the things you want to find they're just a medium to find to find some of them basically so with the variable searching you still have to be very very yeah I still has to be very very intuitive and you still have to craft these variables which obviously you're adding biased your in and are kind of model here but there's not really a foolproof way or if there is I'd love to look to speak to anybody who knows that foolproof way of doing so you still have to kind of really really manually craft these variables to make sure that they are going to give you give you actual cheaters rather than just outliers so once we've got our outliers what do we actually do with them we actually kind of need a ground truth so we work with you know and again this is not there's no easy win here unfortunately we have to go through a sample of our data and just understand what a cheater looks like versus a non cheater we know that these outliers that we found are most probably our cheaters but there are going to be a lot of false positives there there's going to be high spenders and they're just going to be people who are very very good at eye game who have found kind of completely legitimate ways through the problem with that also is there's kind of a needle in the haystack problem if we just sample down our data and we've only got a 1% outlier then there's only gonna be a few people that are outliers so we're only going to know a little bit about them and we then will have a load of other people that we don't see as outliers who are probably not cheaters and we're not going to see anything in between either so if we look at the variables we're not going to see like the tipping point in which people are a cheater or in line cheater so a really really simple way of dealing with this is just resampling the data based on the actual outlier score the distance score or the error score that we found from the neural net and that means we get a really really good representation across the board of outliers in Liars and everything in between and we can really really understand the nuances and the tipping points between who cheater and who's not which is awesome and I mean it's a bit of hard work it took me a few days of just a lot of coffee going through a lot of a lot of variables at this point but we kind of had a labeled data set at that point which we can kind of analyze and model and we have kind of a truth of who they're cheatin who aren't so in terms of yeah just quickly so two examples here of what we did actually after it there's two games these are two and Don Titans and me they actually diverged in what we did it after this so one of the games actually after going through that process we actually only found a finite amount of ways that people were able to cheat the game and so we kind of took a step back um as I well we could obviously just create classify here but given the fact that our ground truth is only eight facts about teachers we can in fact just create very very simple heuristics on those eight facts and and building a model on top of those is is ridiculous because you it's never going to be as good as those eight facts that we found for that ground truth so so we haven't so in fact we just create very very simple single Verity aristokraft that data another game actually had much more nuances and there wasn't much more there's a lot of very very small ways that people were cheating in such a way that we weren't going to be able to just craft a rule for each one of those so in fact we actually just use that labeled data set to build a classifier to classify who is a cheater who wasn't and that's kind of how we output that I'm not going to kind of go into that cuz this was more about kind of the outlet out lie detection method but in practice that's kind of what you end up doing with your with your ground truth of data once you've found those scores course so barely so summary the fact is to find and ban the cheaters there's obviously issues with that our we're never going to be able to just find cheaters because actually all of our really really high value players look exactly the same as them so we were never going to be able to do this in a simple way so there's always going to be a lot of fun crafting involved but we kind of used these out lie detection methods to try and pinpoint actual potential cheaters and potential and high-value players such that we can kind of we can understand them better build models and build infrastructure so we can remove that players and kind of let people enjoy the game that the way they should and not feel like their gain getting the short stick of the game so kind of using into it intuitive variables detecting the outliers kind of either through very very simple mechanisms or actually using kind of neural nets to do so and then and then there's all kind of like writing set resampling that we use to to get a good view of what an outlier looks like versus not and then if everything in between and that everything in between it was the most important bit because we really need to understand the tipping points so and after that we kind of just you know we've now got a ground truth in it's just a it's just an exercise of either building a classifier off it or or creating new heuristics so yeah that's kind of the journey that we took through data fine using data science to kind of take out find cheated in our game has anybody got any questions oh sorry note quickly knew that there's one more and so just quickly you can actually just find all of these stuff really easy so cyclones got loads of packages especially you're looking below minimum covariance determinant and the minimum volume ellipsoid there's a implementation on github by Joanne's one of our old analysts and the replicate in your net doesn't currently have an open source thing but if you're actually thinking about the relaxed version of what we'd built it's actually a very very simple feed-forward neural net so you can kind of build that either through tend to flow on your own or actually there's a very very simple classify that they've built through tensorflow to builder which is quite good and so yeah questions [Applause] let's just say I get your game and I am all right applause let's let your game Judith but I'll only do it when we we still miss a day of Thanksgiving today I think it's on progress as much now which is a look like and I've Eric's play it I was a kid that's gonna make me possible to do that it will but this is that a necessary an issue I think I kind of broached on it earlier on in the in the presentation is actually if if we can't necessarily detect you can i users realize you're a cheater because that's the only thing that we actually care about we don't we don't generally who's someone who's going to cheat in our game is we're not gonna be able to slap them on the hand and say don't cheat again and they won't cheat again and they might then spend in our game you know it's not going to it doesn't do that so we can either try and find absolutely everybody or actually we just want to focus on what is detrimental to our game and the detrimental part is actually just what what you can perceive is cheating and that's the bit that one of the biggest kind of points away from this is actually like the perception of cheating in the game over what's hit over actually what cheaters are I think that was one sorry we do we do so the way that we actually build the kind of end process is there's not it's not completely we don't just completely automate it there's certain levels of it if we builds very very simple heuristics of facts of like this is a cheater we can kind of automate it but there's a lot of levels of kind of well there's a gray area here and it goes to our support team to kind of make a decision if people come back and we are so that's so that kind of rules out necessarily the kind of firm the false positive of banning somebody but if they come back we kind of have to just say you're banned we can't tell them why because they'll just learn how their bands you know they'll learn the constructs that we've put together to stop them and so it's a case of just not really talking to them this is it's kind of just like I'm sorry you were banned because we found that you would you were cheating our game but we can't necessarily tell them why because then they'll just find new ways to do it it's that similar arms race so what we're trying to do is stop the arms race by not telling them exactly not giving them showing them everything so the target variable is actually the data itself so what we're trying to do is M so what this neural net is doing it is not trying to trying to predict whether someone's a cheater on on we're just taking our data so our three days say for example our three data points that we know if a user and we're running it through a model such that we're optimizing this neuron X such that this this neural net can predict our input data really really well so any normal bit like so that's going to learn the structure of the data any normal data there'll be a very low error to it and any outlier data which the neuro net doesn't necessarily know is going to then come out with a very very high error rate and that's that's what we use as a kind of an outlier factor yeah it's kind of set yes so the replicate the replicate in your net is is the theoretical replicate in your net is using the step function inside but actually we've relaxed the assumptions so effectively we're using an autoencoder for a slightly different purpose yeah there are so I mean the first thing to think about is like hey I mean we're talking my suspenders here we've got a piece of data in the in the fact that they've spent and so it's kind of naively assumed that we could probably just use that very very easily but there's a lot of nuances in-between for a fact that there's actually people who decide to spend in our game and then they go on to cheat and so we need to kind of we can find ways in which we can kind of separate them out but actually we kind of have to treat them as similar because they are similar behaving players our moral weight and there is I can't I don't know whether I can give that away so there's certain things yeah sorry but it's a good question there are there are different ways that we can do it and but I can't really go into it so okay and I think right back you just canvas replace a metadata attitude of two ASIS or we'll have you might see formulas each game as rules you might political because all the excited just heading some as a seller of like associations or some yet on the South Texas one both distances if you can start testing is critical results as numerical vary where variables how would you think so that's so the first question about the software engineering problem so yeah you're right I think for a lot of games you can deal with cheated in that manner but the games that we build are quite deep and level in kind of complexity and actually we can most of it is kind of authoritative in terms if we can stop a lot of these things but it's an incredibly deep problem and to the point where you know there are always going to be some ways in which people can find ways around it and also as I said it's an arms race there's people that are legitimately like spending days trying to find out how they can don't get around our system and I guess you groom the step function of the pan and the last yeah because you know differentiation yeah and that space what kind of magical process at some point doing here that is true yes that is an imagine that maybe the piece of rose that is still to the clustering and then also you get confirmations of clusters and Ibiza and yes you're right and there are that is definitely a way we could go around it I don't think that was really going to necessary solver problem here though because in essence if we find clusters in our data then they are a ABA player behavior that is probably quite normal you know not everybody plays the game exactly the same and so actually what we're trying to find is the people that don't fit into clusters about about data that answer your question yeah so I was thinking maybe the supposedly fosters there was not in one of those clusters that might be in red red single in the same thing I guess you do yeah so now you're catching and finally teachers how does that Peyton so it companies probably busy and it's so the biggest thing is over resource costs mainly and so for a while we had a big team having to deal with a lot of kind of you know a huge valley manual process to do this and what we've done is kind of found ways to completely automate that and take that away from them so it's a resource cost and also it's just kind of consumer facing kind of I think it you were saying is I was the bottom line metric that we've improved and the revenues there's probably an implicit increase in revenue given the fact that we if not the people on perceiving cheated in our game anymore and therefore not playing but the one kind of metric that we were trying to do is just kind of the amount of complaints of cheaters that we're having can we reduce that and we reduce that by well fast amounts but more like yeah yes so that's kind of I didn't really go much into it I kind of touched on it at the end is like once we have that ground truth that's kind of what we did but the problem being is that right at the start is we don't have that view and so we need ways in which to be able to to filter through kind of what could be a cheater what couldn't be a cheater so that we can then go and check and analyze that data yeah so this is back to here so what we're doing is using the outlier score to resample the data such that we actually have more yeah yeah yeah we have more cheaters or more outliers in our data that when we go and find the ground truth will therefore have a larger amount of things so it kind of deals with that class imbalance it seems like the issue is really I really use experience fighting approval levels really strong players telling you yeah I used to be able to tell it in standing or cheating and insider is not an implicit thing that it spend us what's really good for you and then that's the game is a whole thing between billing with you there's definitely there's definitely in a slight issue at that because because they look so similar people can perceive cheating when actually there's just people who are very very active and I game and I'm kind of high level so that's really that's kind of a slightly different prom is more like structurally how do we build the game such that everyone can enjoy it there's another point sorry what was the other point to the question before you yeah John's games not ruining user experience and that was it the identification so I mean that's definitely an approach we can do we can have a feature that kind of badges the IPS but as a VIP do I want to know tell people how much I've spent in this game it's not necessarily like that's not necessarily something that you'd probably want to see be be showing as as somebody who has kind of spent some money in our game so it's yes it's a very very tough one and that player perception thing is a really a completely different problem of you know once we believe we've actually solved the problem there's still that level of somebody thinking that there's cheating going on when actually they're just seeing kind of very very active players buying the game colors already so yes so you're saying so why do we why do we not just have labels straight away because we would we would end up having users that we've already found before this method that we know is cheaters the problem is is we only have one label that we don't have the the opposite to it because then we just have a pool of people that we think on cheating but they could potentially be you know it's one I think that it's it's similar to the problem we had with the game in fact that we didn't end up building a classifier because we had ground truth that we could attribute as these are the exact ways that people are cheating and so if we just tried to approximate and build a classifier that approximated those ground truths it would never be as good as the ground truth itself but there is that other and it also we're kind of then also missing the extra level of like the cheating we don't know about and so that's why we kind of went through this method of just fully understanding the outliers and I game to fully understand all of the different practice of cheating yeah that is still money I'm fortunate I mean but it's worthwhile as well because not because we've now taken a statistical view of this now we need a real like game view of why people are doing this and we can and that's not just affected cheating but it's affected kind of how we built our game going forward because we now understand not just how to eat as a cheating but how our kind of high level players are also playing and what the differences are yeah and no and so it's it can be a problem and it's definitely a problem amongst these kind of deeper level games because people people will spend a lot more and definitely our social level it's a huge problem because even if you have a small amount of cheaters they can cause a lot of negative feelings for all of the legitimate players in your game but in terms of so you know it's not necessarily a fact of whether you've got a high problem with teaching although problem any problem with cheating is a huge problem because it's in social games it's going to affect this legitimate place yeah for example that people like spender are also tired or that you want some broken and in the previous episode zero was a typical way to make sure whereas how do you make sure that that's a very good point and so you think how can I just rephrase the question a little bit are you saying how how did we know the replication yarn that was better than the distribution based methods how do you make sure you some of the concerns you have in the bracelet which were that okay so the first question on the kind of replicate in your net is a it kind of had there is no there's no one there's no one way to do this it's not it's not a very simple mechanism of just having a value and making sure it's minimized for the for the right model and what we actually found is we actually used a distribution based method on the first on the first of our games that we did this with um and that was implemented but we learned a lot from it and realized that there was certain areas that we needed to address in in the way that we calculated the outliers so when we went to the you know net we kind of made sure that it was as good as in terms of the things that it was doing well and it was actually taking care of the problems that we've seen in them in the in the first implementation so it was a very very that's quite subjective isn't it it was it was kind of yeah it was it was a relatively subjective approach what we found is is it seemed to differentiate true outliers a lot better but that yeah generally it's quite subjective approach and the second one was based on the distributions so that again this was kind of an implicit assumption but we knew it would hold is there are very very good at Ennead to be for the distributions to work whereas actually on on the net we knew that we wouldn't have to worry as much of those at those points but again it comes back to the same thing we knew having implemented it the problems that we'd found and the reasoning because of some of our some of our data wasn't actually we weren't able to bring it into a kind of Gaussian you know modal space we were then able to check that once we implemented the replicating on that that those problems didn't no longer arrived okay there's no more questions [Applause] [Music]

Channel: PyData

Views: 8,054

Rating: 4.816092 out of 5

Keywords:

Id: Q2HLPCBStLQ

Length: 42min 2sec (2522 seconds)

Published: Mon May 15 2017