Kaggle Grandmaster Panel - H2O AI World London 2018

[Music] that is mine [Applause] what a treat said all Kaggle grandmasters eight of them and I think there's two more in the audience maybe the left already but if you're around somewhere you're welcome to join the stage this is the largest concentration of grandmasters in the world I think so close to those guys let's start with the youngest who can guess his age very good seventeen please introduce yourself yeah so I'm Mikkel I'm the youngest chiral gram mass that's my gimmick 17 started kaggle about three years ago and then since then sort of just been self learning and doing competitions and getting better at it I'm Dara and I've been a cowgirl it's a bit three or four years and I'm an AI engineer in the health care space with optim so doing some interesting work there and also trying to learn as much as I can on coggle hello i'm mateus i'm data scientist here at h2 and working primarily on developing 12 let's say I and my name it kegels feminine and it's just an awesome platform and if you're not competing already just get started hi I'm Brandon customer data scientists at h2o I joined kaggle five years ago but I only knew excel at the time so I spent another year and half or so learning our and so I've been on Keiko for competing actively for about three and a half years and hello I'm Mario's my kagu name is Casanova and yeah I've been working for h2o almost one year and a half now as a competitive data scientist mostly on driverless AI and yeah have my fair share of kagu competition I think I have participated more than 120 yeah that's me hi my name is Mattel our Co I am Cal Grant Master I know I participate on a lot of different competitions I opened my will coggle to myself like seven years ago and I'm a hdall data scientist I'm sure was appreciate known as CPM P&K girl and double Grand Master and I'm better at speaking than competing gram first under on the forum I work for IBM doing machine learning and machine learning is a long term interest because I had a PhD from previous millennium as you can see I'm not the youngest here but machine learning is really my interest for many years thank you hi I'm so late autumn are called as a sake in cattle so I am working as a data scientist at Rodarte I've been cackling for about five years now so I'm a Grand Master in both competitions as well as in Colonel section all right let's get going with questions if you have questions please submit into slide oh but what is the number one advice you would give to a average data scientist that you already know they need to notice like it's it's something they just have to listen to I think what I would say is that there's a big difference between learning something in theory and actually trying to do machine learning in practice because like you've seen the rest of the day it's all about feature engineering it's all about figuring out what's going to be good and sort of just getting your own ideas and implementing them as opposed to actually knowing the equations not that important you know because someone's done the equations for you right it's still programmed for you so it's just about getting my intuition and so my advice is to get stuck in dukale competitions and get that intuition yeah I totally agree with that and one thing I find myself struggling with is when you develop a pipeline or something trying to not get them stuck in that for too long trying to move to something new because I think we're all in for learning well what we can learn at a new methods feature engineering models to try and constantly push yourself to move on to the next thing instead of spending a lot of time just tweaking one pipeline because then yeah generally get more out of it during the competition and also you learn more that you can apply in the future a man so really one upon seeing for myself was after after I finished my studies I started after that was kicking and I was quite confident actually because that oh I have so good educational background now and then I got into my first competition and I got grounded a bit and then all the fun experience kick in so I really wanted to learn what I'm missing and I figured out that I missed a lot so that academia teaches you not not the things of really necessary out there so you have to figure it out on your own I mean that's always AB in the cases University a lot of things you really need you have to teach yourself and carefully the best platform to get started if you want to get better at all the things which part of a data science pipeline I don't know if much to add and heard about all these guys just said but I think I agree with dark about trying different things I feel like sometimes in a competition you just get stuck doing one thing and trying to perfect it and then you realize after the competition that someone tried something else that you we're thinking about trying but you just never had the time to do so I think like trying to taking more of a shotgun approach towards things rather than I guess focusing on one single thing I'm not sure what the question was the single the single best yeah I mean you just okay I see now I then your nonsense make sense I think yeah I mean you need to get your hands dirty don't get demotivated because you know even they work you know science it might be a bit tough sometimes you see all the pile of notations people use and he might feel a bit intimidated to enter into the field but you know you shouldn't I mean obviously takes a certain amount of time but I've been able to do it from an accounting background I know a lot of people who have been able to do this with very little not very strong background so I think it's a matter of you know putting the time you know trying to learn it is a very open community the data science community is one of the best more generous communities out there dominated by open sore walks people will help you don't be afraid to ask and generally that's my message can you hard 30 and don't be intimidated yeah so basically actually no we have a lot of advice advice asses already so if I can actually make like some sort of a summarization you supposed to know your algorithms right but that's something that everybody knows right he also supposed to know the metric you trying to optimize and not the data right and especially keep in mind like a difference between train and test data set on cargo computation because in some computations back to actually can give you a lot of insights I would say the hardest skill to get is to properly evaluate model performance it's very easy to get full download install open source train models get 99% whatever accuracy or any other metric and your model is just learning the training data and not generalizing and entering a kegger competition is a wake up call if you don't do it right you believe you're along the competition and the last day when the private data set is is used you discover that your model was offered and so learn how to evaluate properly your mother and it's hard because you must not overfeed to the training data so practice Kagura is one way to learn that to miss his most important skill so in my case like as people who are mentioning so like being hands-on is one thing which which has helped me a lot like in GRU grooming my career so like say in case of Kegel competitions I try to participate in multiple competitions so I try to get to solve different types of problems so and in in that way like I improved on my data science skills I wonder how many hours do you spend the day it's a point one or is it five what would be your your over over the last year or two your average oh yeah asking average unfortunately probably over an hour a day for me but it really depends on if I'm doing competition seriously and it's coming up to like the last week of the competition then I think we can all say that we spend way too long doing it sort of missing sleep and all that but then again there are times when you're not working on a competition or you're just doing one for fun and it's really you know whenever you feel like it but I think overall if you're trying to be in sort of a prize winner or a top ten it really if most of it's not like we just press a button it works right you really spend a lot of time doing it yeah it's the answer is a bit embarrassing sometimes they're long-standing I'd say when when I find like if I'm in a competition and I start to do well early on I end up spending a lot of time and it may be two or three hours a day maybe a bit more at the weekend and they'll be on top of work so a lot of time but I you know enjoy it and do it mainly out of learning learning new techniques because I find it interesting so sorry to find instead of watching Netflix do coggle yeah it takes 10,000 hours to be a grand master at anything right so they're probably under estimating the hours even it's yeah I cannot tell how much hours to spend an average I don't know but it's it's definitely if I'm really into a competition it's a bit like an addiction so for me it's like have so many ideas in mind I want to try them out and it's really painful to to have to wait because you know I have to implement the stuff and then I have to try them and then I see okay it's not working dammit next next one and this gets longer and longer and so sometimes it's like it's really more the way how can I get find sleep and get out of it because if you're really diving into a competition and you have actually a chance to compete towards top ten or better then it's for me personally it's like I use every few minute I got but for me it could be sometimes ten hours it I get to do it for sometimes I'm doing research and stuff like that so one of the things we're working on at h2o is reading documents or trying to figure out where a document and invoice date is so I was using faster CNN's for that which happened to be what I used for the right now that is the our SNA making bounding boxes around where pneumonia might be so for that competition you know probably eight hours a day just looking through the code trying to make changes trying to figure out how it works and how I can what changes I would need to make to make it work for our process so for something like that might spend yeah eight to ten hours a day but for the home credit competition which my large team of twelve we got second place but for that I didn't do a whole lot I just I wrote some code that it was kind of in a loop and made 20,000 features or something like that would ridiculous and that was pretty much it I didn't do like maybe an hour two hours a week maybe on that one not too much probably more time than most yeah I should not embarrass myself further but I should say in my defense but there is that there is good overlap with my work well the last competition which I was able to make like a end up with six out of two thousand something it was my latest solo gold medal out of course I calculate how many hours I actually spent I spent four hours a day for three months so that's a lot depends if you include dreaming or not at the end of my current competition last night so it was not a nightmare so that's good I would say like use three hours every day job there is some overlap but maybe not as much as fall you Mario Mario's so spending a bit every morning preparing runs watching them during the day and finalizing in the evening I spend around three to five years on an average but my system used to spend around 15 to 20 years so what's your process do you do you just do random attack you try all kinds of stuff what do you first setup a validation scheme or think about the metric maybe you reframe the target column first how do you do it this doesn't depend on the problem of course it always depends right but is there something like a systematic way to make a better process how is it that you made it into this stage let's say I think that in an ideal world like when you start a competition you should first thing setting up validation is really important because you want to be able to work out locally how could your model is so you'd have to keep uploading to Kaggle and also sort of visualizing the data and trying to figure out sort of find interesting bits of information that you can use feature engineering things like that in practice from myself what I tend to do is I tend to download the data stick it into XG boost upload it and then sort of sit happily for a couple hours while I'm at the top of the leaderboard because no one no one else has downloaded the data yet that's that's my approach then I will go and go back once I'm on there now I'll go back and I'll start looking the data more and trying to do it I really should do more of what I preach I think the pipeline's pretty similar at the start to try and get an idea of how well people are doing on kernels and Dan set up with validation set I think that's something that's very important and it's often underlooked having a good validation set run run a simple model yourself locally to make sure you can get close to that whatever the scores on the leaderboard are and then just spend a lot of time trying to understand the data a lot of time I'm working working with that part yeah it's getting already repetitive but first step is to set up a good foundation for cross-validation so I always spend my first couple of submissions to China model and I I know that it's not good and I improve it a bit and I want to see a relationship between my validation score and the leaderboard and if it's not the other than I have to think again about okay what's going on why is validation not working and that's the first step of the whole process so I'm not starting with dependences more modeling before I don't have a solid validation framework established because that's a very matter of everything as well as following if your validation is bad everything what comes after that is not working pretty much the same as fairness I'm not going to bore you with repeating it CV a I think on top of what the guys have said I will add what determines the winner is obviously the time you put in you have more time to search for different things patterns in the data making good partnership that the science is a team sport being able to find people that can complement your skill set have different experience on different techniques or they're able to seize the data from different angles can lead to very diverse solutions and normally better results having access to the right tools and having good hardware support you can run more experiments at larger scale cover more ground some level of automation I think it's quite important you know you you don't have the power to search for everything you need to kick in automated things well you focus on the things where it can really matter where you can extract more information so some and individual understanding of the data try to get a bit more context and get specific ideas of how to solve the specific problems I think and can really help then a little element of like I think is also you know a good boost they said it all already I would just add trying to see if what our people have not worked on the same problem and published papers especially fuse deep learning it's evolving so fast I see in the deep learning competition the one that win are the one using the latest papers alright let's now move on to the slider questions thanks again grandmasters for answering all the questions so far let's pick the first one why aren't there any women cackle grandmasters on the stage are there any in the audience does anybody know anyone we would love to encourage participation I mean it seems like it's male-dominated and that's possibly true this bias in the state anything we can do and I think that it's important that people don't think that you need to do a degree or whatever to get into data science or into Kaggle I think that it's almost a softer science than people think at first because it's really about experience and all that so I think in car urging people maybe here to have a background in machine learning to also you know give it a go see if the interests then thing I think it's to open it up to more people is would be quite good because the best software engineers are women in the early days right the Apollo meeting space space mission was all programmed by women so it's it's definitely the attention to detail is actually better of women I think that's the state of the art of the research so men might be just more bullet but is it called the shotgun approach maybe there's a there's some kind of a brain chemistry that makes it and to be fair there is improvement and there are you know I know female grandmasters I mean I don't I don't see any serious obstacles right now to participate in that in that platform that would be based on gender for example it's just a matter of you know feeling comfortable that you know you you you as long as you don't feel bad if you don't get very good if you don't get it with the first try that's the only thing that can keep you back in my opinion right now for in a platform like goggle and what's the point now that we have driver let's say I live in competing cargo or actually I would rephrase it is there any news of driverless in cattle well we kind they use drivers here to get insight about the data we do in the cattle computations right we're not allowed to use it for final submissions that's for sure because cattle is a commercial software but it doesn't stop us from using it you know to get inside with features actually interact the most but it's still I mean you know I think seeing his presentation actually compare Drive I say as a to autopilot basically right and it is basically not a pilot but you still have to you know you have to take off your plane you know and actually and land it back it's still a result of actually additional things you need to do you still have to think about you know like a carefully designing your experiment your validation schema you still need to you know try things by yourself like pilot actually launch fly even if they do have another pilot available they still actually know how to land it and what what how to operate the airplane he is pretty much the same so Java CI it's a extremely useful sophisticated automated tool but it doesn't have any insights about the data you can actually use it doesn't have any other my knowledge it's just you know it's it's do it perform like a huge relentless search so it's kind of help you to speed up your research but doesn't replace your insights and your looking into the data it's also meant to be production ready which means that it may not be always suitable to win a cargo competition what I mean for example in a cargo contest you're allowed to look at the test data obviously you don't know the answer what you're predicting but you can see the distribution of the test data and you can use that information to enhance on and reach your your training data and this is something that quite often we do I mean you probably don't do it in a production environment but you're allowed to do it in this competitive context because this data is available so you see there is still room for manual intervention and you know but in any case and not only the process I agree that race is the bar but we can if have anybody everybody has access who can just push the boundaries even further with individual effort so in general like when I start cattle competitions what I used to do is like say I take the data set and run XT boots model or like GPM models so previously now what I'm doing is like say I am just running a try velocity and getting a baseline and also getting the important features out of it so now that the baseline is high I need to walk for heart and to beat that so that's how I'm improving myself by using driver safe and another thing maybe I said tools like - or let's say I can actually take the poem part right so actually is that what the data scientist has to do what doesn't want to do all the time like dealing with hardware to prime my model and stuff like I said such things can I mean I'm okay with an automated system makes it for me because I don't want was that from its bobbing I want to I want to get more data inside so actually what's human access that I want to spend time and debt and tools like do Arliss I just help to find more time for the really cool things I think it's a little bit like in chess I mean nowadays the best chess engines know grandmaster and the world can beat them but yes playing chess so we could ask question okay why is still has two people playing chess now that the computers are so much better Sam okay and doesn't use those computers to improve their own games finding more insight so combining the power of artificial intelligence and the human brain and I think that is also interesting a point of view and so on to the whole thing because currently it's still like really separated right so we don't have that deep understanding of human brain yet and artificial intelligence is still kind of shallow as ever change and that is more like I would look at this there was a presentation earlier today we showed that chess enhanced by humans at the robots plus the humans together beat the robots always so fifty eight to zero or something so the the combination of humans with their machines is always better and I think the same is true here right so your your cattle insights will help your performance as a data scientist and it doesn't hurt that the machine is there as well do you use R or Python can you give a very quick short overview of what you what tools you're using or do you scale our what is it you have a super computer at home or just a laptop so I'm using Python Metin II pretty much for everything I think we may have a couple our users here he was still sort of stuck in those dark ages but never mind terms of computing power having access to a lot of it is helpful and the thing you will have access to quite a lot so but there's a lot you can do with just a normal desktop or a laptop it's really just that extra edge which I will admit is there from having a lot more computing power tons of favorite tools I'm still quite a big fan of X G boost they sort of seems to work on everything I know everyone else has switched to like GBM so there's me in the dark ages but you know I still seem to do better with and I prefer it so I'm like XG boost so for myself I'm an R user and I find data tables very good so it's great to see the presentations here today and I find out very good form on gene data picots is coupled with or it does very good visualizations so you can quickly iterate over ideas and it's quite flexible to manage ideas and and see see the results of that they see how it looked like within a model and at the moment but I'm spending a lot of time we care us and tensorflow to try and because I think there's a lot of levels to what you can do with deep learning so spending a lot of time with that and I enjoyed that I was thinking it's sort of a love-hate relationship I like it but it hates me yeah if amyot pison mostly coupled with c++ if it's like time-critical code and basically all the libraries like actually bills like to be a more PI torch care etc so there's a common stake which is not a use by almost everyone I would say that's what I'm using as well yeah so I my or user mostly I do deep learning stuff in Python but in our use a lot of data table and shout out to Matt you know does a lot of work on that I'm at the yes so for deep learning been using PI torch lately I used to use Karis and using pi torch a lot more then yeah obviously actually booster my gbn for other machine learning stuff XC boost light CBM for gradient boosting scikit-learn for random for implementations lip liner lip SVM for linear models and support vector machines and follow the regular ice fleet leader normally in Python implementation tensorflow which cares for deep learning it's kinda interesting nobody mention h2 actually so I'm going to do that h2o h2o I kinda love PI torch but usually i mostly use carrots don't know why it's kind of strange relationship so you guys actually mentioned everything else so is it basically it's a toggle you always have like a pretty more like a lesser more or less classical set of how it goes you always going to use python in the same but i spent time writing efficient Python code it's very easy to write very slow code in Python and if you have a limited time to compete it will be bad so I use so in addition recently I mean I start also started using tribal assay to get that feature interactions which features are useful and so on so that like I can just use them in my models so like that second question does the validation set design match real-world validation sets or more generally what you doing CAG was that useful for real-world production use or are you really just tweaking some kind of a leakage problem and playing Sherlock Holmes in that local CSV file or is it actually useful in for me that you're learning I think sometimes the of course there are leakage problems and that's happened a lot lately actually on cable we're sort of there's accidentally been additional information in the data which lets you do Elm leader board the perhaps you wouldn't have in real life I think for the most part it is sort of applicable to real world and what tends to happen is when when you do a Carol competition the top solutions will be to the monstrosity zuv in Samba 'ls that actually when you rerun them you'll get a different result but really then the organizers can do is then from the top solutions they can take those the insights that are in there and sort of make like a diet model with sort of just the useful stuff that is maybe a little bit worse and but so I think it's still useful for the organizers and in terms of whether the validation set matches I think once again that's up to the sort of the organizers if you want to be able to brick predict the future then you should give us a test set that isn't in the future but yeah I think the most part Cargill is quite useful for companies I mean that's why they keep coming back but it's you know they will have to do some tweaking on those solutions yeah it's it's a good question I think there's a lot of overlap Morden is sort of recognized sometimes I think the it's the thinking of how to represent data for a problem and is is quite important and I think getting getting comfortable with tools is it it makes you better able to think about other things outside of the tool if you're not as worried about am I using the tool correctly or whatever so I'd say from those perspectives like I've seen a lot of use cases professionally that could be drawn on ID is taken from from coggle yeah I would I would also say that's really useful because actually the problems in Kegel actually we'll all data sets so sometimes they are not probably set up but and leakage can make a huge impact on our cause of we as competitors find a way to exploit leakage we do it because it makes us gospel that's how the game works actually but consider the same problem now as on your day day day job and you are not able to identify that leakage and I don't want to know how many models out in production was really bad because trained on leaky data and actually that is a really important skill to to to develop for your own to identify he said model what I'm bullying is it actually useful is it say any leakage I have to think about because most of the times those leakage issues start rising it's a very beginning of the pipeline and the data collection or data storage states and not necessarily only it's a stage of the modeling itself so I think that makes it even more useful because you acquire skills there that you would really need as well in your day to day job if you are data scientist and you have to build a data sense partner I also have very little to add here I mean it was basically pretty much covered but obviously there is a lot more going on from the point you collect the data you set up the problem and then how you take the results and you and you products nice and you utilize that and sometimes it is very difficult and people are critical about goggles having only one data set it's it's just you need to make it so that it is it can be using a competitive context so you have a common measure to test every one and quite often there are pitfalls you know people introduce liquids introduce stuff with they shouldn't be they are something having that experience you definitely learn a lot about how you know what is the best way to represent your data to be able to avoid such problems but but yeah I mean I think generally this is this is a very obviously important matter which needs to be you know very analyzed I think we inserted have you hacked the kegger scoring what people do is not scoring this is documented what people hack are the target of the of the test data this is leakage all this discussion it's amazing how good the kegger community is good at finding leaks finding signal that the organizer thought they had removed it's amazing if there is a leak it would be found and this happens in reality as well so the ability of finding a leak leak is you believe you you have new data you know nothing about the target but in fact you included the target somehow in preparing the data and this happens in reality as well it's a big danger so learning how to detect this is also useful even if it looks like a drawback it's actually very useful good maybe pick another question is due to the nine of you grandmasters everything yourself like Ocean's eleven and wanted to predict the next loitering when we wish we could it's more difficult but I think most of us we are engaged I think in some form of activity to predict something which may yield some return I know some people do Bitcoin predictions over there right Mikkel something like that yeah I also try to optimize my own port I mean we all I mean maybe that's the problem if we work together we can make more yeah to be three oceans I also would like to add to you you know it's not very efficient to predict the next lottery win because the pool is too small right we have to wait for you know to pool to increase basically so on as soon as jackpot actually becomes bigger that's up would be more effective you know spending your time for so next lottery win is useless to predict we also know that half the money is gone already right it's not a very good way to make money the expected return is pretty low is it a science or is it all just random hackery trying to make gold out of rock it's both to be honest right so it does has actually science inside because you still have to create hypotheses and test them you still have to you know be very rational and not emotional about you know well you believes on this data in the prediction of your model you still have to you know basically criticize your approaches over and over again built right validation so so very alt of actually of science is scientific approach to the to the in the data science basically into the competition but it does require some hacker as well like deployment for example with the tons of hacks you know and actually not been proved by by like some feel field without any theoretical proof at all but they work so we just use them I think it's so science by the method you make hypotheses you design experiment to test it and depending on the result your purchase is confirmed on that and the basic I put this is this feature is going to add my model so it's it simple I put it is but you have to test it in a scientific way to make sure that it's true or not so if you are if you come if you've trained in a scientific domain you can do well in data science for instance physicists they usually become great data scientists it's not by chance then of course there's also black magic as well but there is some science have you hacked the car goalscoring algo what was that answer sorry but in any case I mean this is how we made it that high right I mean we we found a way to secretly increase her scores no but but but occasionally people do find faults in the crop either in the scoring system in general in specific competition and generally it is quite an honest community these faults are identified and resolved competition actually what happened was the metric was a you see so what people I mean some one of the other brand master did was to actually predict actual predictions I mean and then did them one - of those prediction so it gave a score of point zero something of the third so finally on the very last day he just reversed this course and that way like he hacked the system to get into the first place so no one else knows that like he is actually improving his score so that's one way like people had I want to point out a question someone asked whether they should fear joining cago and that's something I hear quite a lot but I sort of doesn't make sense to me cuz it's not like you get kicked out the secret Club if you're at the bottom right so when I join Kaggle I sort of barely knew how to write Python I had no idea what I was doing I was going to cackle Colonels taking code that people have posted like chaining the parameters and just sort of clicking run over and over again and so back then I wasn't doing very well were sort of doing that that led me to actually figure out what was going on so it's not a case of learning how to do machine learning okay now I'm ready for Cargill I think that's actually part of the journey and even if you have no idea what you're doing it's still good to have a go in German isn't saying that the path is the target or something so keep doing what you like doing and of course I guess you have to like it right you can't force somebody to do this but if you have a little bit of this addictive personality you will definitely enjoy it and it I think it's good for your career it's good to learn this stuff and even though there are tools out there like driverless you you don't need to stop thinking about the depth of the trees it's better if you understand what that is right but you shouldn't have to do everything by hand all the time and it's there any last quick hacks that you can share like some some kind of a secret trick like I know Demetri has a secret trick that he applies at the end of cargo I'm not sure if he's willing to share it but it is are there any like one short summary tricks that you say just do this or give us a hint of what you did to make your score go up sleep let's try more no no sharing want to keep our positions right [Applause] great now never give up until the last minute you can make progress it's solid how about taking other people's submissions and averaging it and submitting that I mean now everybody is doing that not it no tip beware with whom you team now some rules on kegger and if you break them and you'll get caught you're removed it happened to me look for last minute I was not the guilty one look for last minute kernels that have high scores if you're quick we might just click he didn't get a good score keeper submission till the last minute yeah last minute keep following the the results there's also actually good advice you know do not start a competition earlier because somebody's can find the leakage happens you waste your time base just wait for a couple of weeks you know people all right Colonels people will find the leakage and you will just you know use they found eggs what do you think about the fact that you I can make two submissions it's not quite fair right you should have one shot not two shots they can always have like some kind of a backup and in the real life you might not have two shots that go into production I mean so yeah Cargill has a thing where for the private leader board you choose to any two submissions and in the best school from those two is the one that actually counts and I think that yeah it's not that indicative of production but it's nice because it allows you to try to have two different ideas you know that you don't know you know one might actually be better than the other I think it gives you that peace of mind instead of lets you sleep before the competition ends and a good strategy gets to select which these two submissions might be because you might have thousands of models where you could select this you normally win something that I have done in kind of helps is I normally pick the one which work best for me internally in my internal validation framework and I also select which one they want that work best on the public part of the leaderboard so I select these two that was a tip Thanks [Applause] you can go thanks again for coming [Applause] [Music]

Channel: H2O.ai

Views: 7,708

Rating: 5 out of 5

Keywords:

Id: BNAiHpH_gMM

Length: 45min 38sec (2738 seconds)

Published: Thu Nov 01 2018