DLRLSS 2019 - Career Panel feat. Rich Sutton, Yoshua Bengio & Martha White

yoshua of NGO and rich Sutton who'll be here to answer some of those questions and also probably take some live questions from the audience so please join me with a round of applause let's give a warm welcome to our panel all right okay it's working so I'm actually not on this panel I'm just here to moderate this panel rich in Yosh you're gonna answer all the questions and mostly what I've done here is just taking all the wonderful questions you put off on slack and sort of summarized them and sub-select it a little bit so I have a longer list here we're gonna get through some of them but if there's a burning question you have in the audience we'll also just interrupt this and let you guys ask your questions okay so we're gonna start this off with them just answering the very first question but Oh both of you have had long successful careers what do you think has been the key to that go ahead rich well I just want to say a few things even before what's been successful about it but just that I have worked actually almost equally in in corporate labs and in and in academia I've spent the last sixteen years at U of A University of Alberta but before then I was at 18 tea labs and I was a GTE labs which is another phone company in the States and I think success just comes from you know I've been at this a lawfully long time like more than 40 years and so just determination and and focusing on a problem any of us you know if we focused on one problem and put the time in we could become an expert on it and I think that's that's I guess I was lucky to pick a topic that's that's still top that's still popular after 40 years but even that wasn't really a great stroke of insight you know I thought I would try to understand the mind for 40 years and surely that would still be interesting you know probably in another 140 years I think you're right determination is very central to it and listening to you know your your gut feeling while also trying to see if your ideas are wrong by taking experimental evidence and other people's opinion and consideration is a fine line it's very easy especially in the machine learning community to just follow the trend and it you know keeps moving around and so on so sometimes the majority think found something but you think otherwise well if you don't see any like really strong evidence that your idea is wrong just go for it so that's one important thing another why I was successful I guess I was very lucky to meet the right people people who helped me understand a lot of things and I I made a choice at some point to go to academia rather than industry mostly for personal reasons and it was a very good choice for me I would say I guess there are more questions about this later the things that really helped me a lot being in academia is interacting with students I get a lot of pleasure from interacting with students I feel like it's you know my group is like a family or something and of course there's an amazing leverage to be able to influence all these young minds like just in the first couple of years of their groceries then they do whatever they want and so so yeah this has been really key ingredient in my success nice great answers determination and networks okay the next question that was a general question many of you had was what are important skills for researchers and in a related question what are some common misconceptions or mistakes made by young researchers or earlier researchers or maybe just all researcher important skills for researchers many things it's like I feel like you you you have to have lots of different tats sometimes I feel like I'm an artist sometimes I feel like I'm a teacher something sometimes I feel like a counselor something I'm you know I feel like I'm somebody who has to be really really super well organized which I'm not so so there are many many skills and I'm not sure if there is one dominant skill but one thing that really helped me a lot is being able to step back from [Music] all of the things that are happening all the meetings all the discussions and so on once in a while and and you know look at the big picture and allow yourself to be a little bit on the crazy side with new ideas so so research is exploration this is really the key thing if you go back to that I think you're you're okay very good yeah sure let me just pause you there because while you were thinking of things while you were speaking I was thinking of things and maybe you'll think of some more things while I speak because there are there are as you say there are many things you have to learn how to do research you have to learn how to do experiments you have to learn a little bit of theory perhaps depending upon what you're focusing on you have to learn how to communicate communication so it's so important you know to give a good talk to present your ideas just to get to present your ideas to others and the key to that is being able to imagine how your ideas would look if you didn't have them okay you have to mad you have to it's tough yeah it's just the same in writing and when you write something you have to you have to forget everything you know and imagine you didn't know it and you had so you have to be able to read your writing with fresh eyes this is really important you have to be ambitious it's really important to be ambitious yes Canadians we have to we have to take that as a special goal it might not always come so easily we really have to be ambitious but we don't want to be arrogant if your arrogance you can't think straight okay you've got to be better retain your humility and yet be ambitious at the same time which is a challenging thing okay but it's certainly possible there's no reason you have to be arrogant so maybe there are different meanings of the word ambitious I think ambitious is in looking at the big picture like big goals it could have a big impact but not ambitious in the sense of trying to do better than the others I don't think that really helps very good you want to have a big impact you wanna you wanna you wanna pick a topic so often you know when somebody comes to me with a problem I ask but why like why do you care and like I realize well they don't know and and and if there is no strong motivation like I'm not interested I really need to feel like why am I doing this motivation is so important for a researcher very very important now those are one second what you said about being able to subject your ideas to tests you've got to be really determined but you also have to balance that to test you've got to be well you got to be really really interested it's just my idea have a flaw does it not work because then I really want to know that because then I would come up with some other end in that line of thinking when things don't work as expected it's an opportunity right especially if others expected it not to work good sounds great so maybe just to add summarize where you guys said there one of the misconceptions might be to focus more on what everyone else is doing and how your work compares to that versus focusing on the ideas and looking at an opportunity to have impact that's a big problem in our field now is it there's there's a there's a herd effect yeah have to watch that and I think somehow and I don't fully understand why people in our field today are a lot more competitive than they oughta be because we're all gonna get a good job right so like what's why why be so anxious right we should like step back and look at the hard problems that may take a long time that others haven't looked at and if others look at them great we can collaborate and you know build together how did this happen why did our field become so competitive I don't know I feel it was not true before deep learning I don't know it's bad but you can all all of you can influence it by how you act in what you say and if you review papers to do it in a collegial fashion and trying to see what's the new idea and what's positive about each idea great actually and one more thing to that just to rat sometimes when you're doing new research you might think that person's work is really great and you know the flaws of your own work and your misconception or mistake is to think what I'm working on is not nearly as good as that other person but remember that in your mind you know your work well so you know its flaws you don't know the philosophy I know some people who have exactly the exact opposite attitude great Martha we did get you to person I'm gonna change it up a little bit and ask a different question about whether it's worthwhile to do a PhD if you don't plan to stay in academia you know maybe you're gonna do applied AI work in the real world I think you should only do a PhD if you're gonna enjoy the PhD alright don't do it but it's very very enjoyable that's that's really true but it's also I also usually say that if you want to do and go into research it's better to go to the whole PhD route rather than try to do shortcut it just on average that's usually the best yeah in the long run career-wise more and more having a PhD has a very positive impact even if you continue your career in industry maybe as a follow-up question that maybe if you already started a PhD you know it's a couple years in and you're not sure you're enjoying it that much do you have any advice for those people it's hard so so it's true that when you were doing a PhD often you you're in a period of questioning and certainty and not when knowing if you're in the right place and right you know a job so I guess it goes back to the motivation question so you have to pick problems that you really care about that you you you find interesting that you you'd like to understand and once you latch on these things it's a it's a source of motivation for the rest of your life at least that's what happened to me I just want to say something obvious you know you guys should know you are like the luckiest people in the world because you are interested in in machine learning and it's 2019 you've won the lottery in terms of syncing up with your with the universe you have so many job opportunities you should do something you enjoy you know you pick the best of the good choices you're not in the situation of picking the least bad of bad choices you're in a really good situation okay another question was both of you have started companies or been involved in the creation of startups should students start companies and should they do it while in school well starting companies is an amazing adventure and I would encourage anyone who has you know an appeal I mean is appealed by this to try it out now you how also have to be careful it's not for everyone it's you know you're taking risks potentially getting larger rewards and you can learn a lot maybe not so much on the science side but on a human side and so yeah the answer is yes and there was something about doing it while you're doing your studies yeah yeah oh well I I tend to say wait for you know when you're near the end of your studies because otherwise it's gonna be really painful yeah yeah I think startup life is all encompassing I really like Joshua's whole answer if you're all if you're all inclined I would encourage you but it's not for everybody and it's amazing how possible it is though today to make a company it's much easier now than when we were younger I can tell you all right and if you fail which is very like expected to fail yeah you would have learned a lot and you can you know continue whatever else you would have done I should have mentioned that that I I'm in the corporate world now I'm at deep mind in Edmonton and I'm enjoying that a lot enjoyed helping bring that into being a couple years ago and so right now I'm working you know primarily at deepmind only minority time at the University I'm seeing all kinds of different people different people want to do full-time University some people wanted to do full-time company some people like to split it half and half or different ways there's a lot of possibilities you can pick broadly speaking you can look forward to at some point being able to pick your distribution and and along those lines I want to mention at me line Montreal we have set things up so that would be easy for students to start a company while they're finishing there's their studies so we have space for them we are making it easier for them to take advantage of the common resources and so on even bringing startups from outside academia in the same physical space to encourage this rubbing of shelters that can that can help companies really take advantage of what's going on in research and I think at the level of different countries like Canada but also in many other countries the AI startups are gonna be a very strong component of economic growth and so if you want to help the economy wherever you do it it's actually a very good way to create value to create jobs it's also very demanding like if you're not ready to spend a lot of time if you thought that research was very demanding starting a company is even more demanding so you but but it you know some people really enjoy that right and it's you you have to find your space let me say something else it's little bit obvious I know yasha would agree with me that teaching the next generation is really really important and right now we have the best of both worlds we have all sorts of companies that are interested in doing real things and that will matter in the world and and providing resources and funding for for for research and and yet we have also the opportunities within academia that's really important to keep new people being trained and and as you said earlier we're now at a time when universities and companies are more ready than ever to accommodate people who want to be part time in one place part time in the other place used to be like a fairly rigid choice that you had to make and it's not so anymore good okay I feel like a natural follow question is is there a room for a healthy work-life balance in fast-paced field an AI and are there any personal tips on how to maintain such a balance I don't lead by example absolutely worried about this but I do think it's really important I can feel like I'm not yeah it's not for me to tell people to work less when I'm working so much I'm not a good example either but but we we do always make a big point at deep deep mind too that it's fine to have have a good balance this it's healthy to have a good balance without requiring it I don't know it's fine so so let me mention a personal thing I have two children and when they were born I was about the time when I started as a professor which was fairly demanding and I did spend quite a bit of time with them compared to what I thought you know I could oh my life was really stressed to try to spend all the time I could with them but then in retrospect like five years later ten years later I thought oh I should have spent like twice as much time with him it was a huge mistake right I I hadn't put the values in the right place but of course you realize this and it's too late they're not babies anymore so so maybe as part of that story I feel like I have an okay work-life balance and one of the things is just to accept that you don't have to do everything this you probably have a standard up here and the real standard everyone else has for you as much down here as long as you realize that you can sort of find a place where you're happy with the work you're doing and everyone else is probably very happy with the work you're doing okay another question was are there any important applications or provider I want to go back to the life balance so they know the way that we be talking about it up to now is like it's just individual choices but it's not I think that by reorganizing the rules of the game how institutions like universities or a company work we could make it easier for individuals to find more happiness in their work and a better balance and I think we should take more of that point of view so there will be room because we're gonna fix our structures so this question is going to be easier in the future oh look mike has a comment I'm gonna say nothing what do we need to change to achieve what I suggested so for for staff it's easy because you can tell them oh this is how many hours you're supposed to work the problem it doesn't work with researchers they would work even if you tell them stop working I don't know so so maybe the way that we reward research right now is pushing us into this rat race but how do we fix that I don't know I think in the case of grad students and so on if we could instill more of a sense of security because a lot of reasons why people are working so hard is it is insecurity like am I gonna do well enough but I don't have like hard answers all right I'm good all right are there any important applications or problems getting less attention from the research community but you think should receive more attention why I gave a talk about that Saturday but you know every researcher has their own view about you know what's important and and that's the beauty of research right if we were all thinking it's the same thing well I wouldn't be as efficient as a you know collective enterprise to search we're explorers so if if we all go in the same place it's not good we have to have diversity of points of views and of topics you just look at any researcher you to go to their webpage or their recent papers and you'll see what what they think is interesting and you should like find your own that being said that being said I know that both you and I are very interested in and how learning systems can go beyond like following datasets and learn the way the world works yes it was something they don't have now and is an obvious thing that one should learn understanding how the world works is interesting because it's both something that I want to do for myself and what I think AI should be doing and it's not really a coincidence if you could understand how if our machines could understand how the world works and then they could plan the way alphago plans okay then we would have I think a really cool that's good the main chunk of what it means to be intelligent so I'm excited about that so maybe to turn that is a to the question of what kind of applications are we not focusing enough applications maybe the applications where we could have agents that interact with the world the way that we do maybe if we had those applications we wouldn't even drive the research even faster so I'm thinking about applications from a different angle which is what impact if we're successful say for a particular application will it have on society is it gonna be positive how much is it going to be negative and and that has driven me in recent years to things like medical applications environmental applications education things that as an academic I can work on without having to think about is it going to be profitable but you know we can contribute through our research so you know you have to look at where you are what can I do to make a better world that's the way I think about it that's for applications and the science part is like there's so much we just want to understand better like intelligence for example for example yeah that's that's all very very good I'm you know I often get the question about what's gonna happen what's the next application that's going to be really important and I always have to chuck a little bit because you know my area of expertise is sort of research and knowing what's going to be important in 20 years you know it's not knowing what's gonna be important than the next year and that's a really important skill that other people have I totally agree and we were talking about startups earlier and I think that a lot of the inventions and innovations that are come that are kind of come out of the eye and I'm going to influence our societies are not really gonna come from guys like me or rich but rather from entrepreneurs engineers founders of CEO of startups who really focus on on that particular question like they make it their life to think what is it that's missing in society right now and we have those tools like how could we build a new gadget that's gonna have a big impact okay so that being said there is one thing that I like to say which is that I would like to see applications where the machine actually continues to learn after it's fielded not like speech recognition systems that that are have lots of learning gone into them amazing deep learning deep networks and then when they're fielded they're fixed they don't have we don't have what anima systems basically right now that there are in in products right we have them and in our papers yeah but there would be all kinds of ways to use online adaptation just in the user interface you know use your phone and like a series comes on right when I'm just trying to switch from one application to another and Siri comes on it's because I my finger is a little bit too slow and I'm it doesn't it doesn't learn it doesn't learn that Siri keeps coming up and I tell it to go away there's lots of feedback you could learn from that there's lots of opportunities for learning online adaptive interfaces or interfaces that learn how to work better with you okay next question is what role do air researchers have an ethical uses of AI and also similarly how about AI engineers yeah and so why don't you start I know you have some clear views on this I mean I do too but I need a little time all right all right so I think we have a really important role and I have changed my mind on this when I was younger I thought oh you know I'm doing all this Mathi stuff and it has nothing to do with society and in a way I was right because it didn't feel that there was a connection between what I was doing and what was happening outside of the university context but but it's different now AI is out there and it is already influencing many things in society having complicated feedback loops that we don't understand and it's gonna be even more so in the future and so for this reason even though we do like even like learning theory we have at least a responsibility to ask the question how is what am I gonna what I'm doing potentially going to influence society and is it gonna have a beneficial impact or negative impact or maybe you know the answer is I don't know but at least you ask the question and then so that's that's one part sometimes you do things that are more applied so you we're talking about engineers well indeed as you go from more like thirty cool stuff to more apply stuff that the question becomes even more pressing when you choose even well the projects are gonna be involved in the companies that you're gonna be working for you're you're sending a signal to the world right about what you think is right and what you think is wrong and then you can be especially if you get a bit more senior but nowadays even my PhD students they get interviewed by journalists and even sometimes they you know they get invited in panels with all kinds of people that are outside of our research circles and so when we interact with the rest of society I think again as people that have expertise in our field we have a responsibility to be part of a discussion it's not like we have all the answers obviously we don't but but we we have our share of knowledge that is part of how to fix the world and make it a better place and choose for example the social norms of the future that are going to be compatible with our values as well as the technology that we're putting into the world so so that's a responsibility as well I don't think that AI researchers have a special responsibility to do this but but maybe we all have some I know I not maybe we all have some responsibility for our society and how its ethical norms are generated I'm a real believer in specialization of function so different people can be good at different things we really are interested in six we really care about it and we should want people to do it that our are dedicated to it and there are and there are they exist these people so all that's true and I want to say one last thing was that you know how society should be structured how ethics should work these are really interesting questions so you know we should we don't all have to do it but but there are really interesting questions and if they interest you you might want to you know dedicate a certain part of your career to it actually I think sometimes that the sort of concepts that we're coming up with in machine learning might be helpful when we think of how society could be reorganized in different ways because we're thinking about computation we're thinking about adaptation we're thinking about multiple agents and what complexity and and and in fact a lot of these things really are topics that matter when you think about how societies organized and how different agents and society and track other disciplines of course are looking at is like economics and so on but but we we might bring our own sort of style to these problems very good okay well I'm gonna switch topics a little bit and ask you a question that was highly uploaded on slack which is how do you keep on top of everything that's going on in the field you don't that's thats related to do a work balance yeah don't we feel guilty about that though and i feel like i feel like a bit of an impostor feeling because you know i feel all of these people know about this paper and I didn't read it and but well we do our best and and then you realize sometimes that you know stuff the other guys don't know that you've read these paper or this paper that they didn't read so you know read the things that you're interested in and and take advantage of the social network that you're in so like the people I operate with my students the other professors my collaborators anywhere in the world the discussions that are happening online the all sources information for me to figure out what I should be reading not necessarily what is published at Europe's okay that's that's one criterion but it's much broader than that and in a way there will are too many papers just you know in the best conferences they're just too many papers so you can't really use that yeah thus the best clue I've had about what to read comes from people usually that I interact with you've got to make choices and specialize a little bit and say I'm gonna know all about whatever it is I tried to read papers on te lambda and know what's going on in that theory and and then people you know they might know they can ask me if they're missing something about TD lambda becoming an expert in something is what a PhD is about by the way and maybe as part of that you know when you've decided I'm really interested in this problem then a natural thing to do is go do a deep dive into the literature in that area and that's one way to at least keep on top of all the literature that is out there not just all the literature from this year oh and I have another recommendation when you start your grand studies you kind of have no choice but spend a lot of time reading but then at some point you start focusing on something you do experiments and you try new things and you get into collaborations with other people and and then there's a temptation of just working on these algorithms and these experiments and forget about reading and that's a terrible mistake alright so you know cut some time off in your Raja and your calendar like every weeks spend I don't know at least ten hours or something reading make sure well it's like sacred thing that you you will do and you can't just read the latest stuff you've got to read stuff that's before 2015 what before 2015 yeah you go back 20 40 a hundred years sometimes you should pick a topic don't pick TD lambda pick something like perception and learning from interaction or state construction and then think about how all the different fields have tried to have tried to say something about that topic unfortunately to use other words in the past sad that that's true in that which makes it extra challenging but that it's extra important because there might be some really important stuff that you don't know about it's going under a different name and the other field or in the same field sometimes yeah good if we reach the end of your list one more question it's sort of provocative a little bit ok so do you think AI might face another winter and is machine learning in an investment bubble and do we have any lessons from the previous winter lesson is don't despair I think it's possible that we are in a bubble yeah I guess you never know and maybe were the worst people that ask but so so I really feel that the phase in which we are is very different from the previous acceleration fails phases that we had in the past for a very very simple reason that the the tools we've built they are out there they are being used and they are making hundreds of millions or billions of dollars to some companies and that's that's a huge difference so that's not gonna go away and even better than that that the the derivative like the slope of how many applications are coming up taking advantage of machine learning is just increasing very rapidly so I think we're exactly in the opposite of a bubble we are just at the beginning of the sigmoid I think I agree it's unlikely that we're gonna have swings like have happened in the past with neural networks what what we may have our usual ups and down in like the job market or the investment market I mean these things go for all kinds of reasons political reasons just demand and and you know things that are swinging with time naturally but I think the underlying current is very strong okay that was the main questions that I have listed I have a few backup questions but maybe now's a good time to see if you guys have any questions so does anyone have any questions are we gonna use a mic yeah I think so so your should they raise their hand yes gently right here's a question somebody has mics all right say it really loud and I'll repeat it wow that's a great philosophical question okay the summary for the question is essentially an ethics question if you have a project that you were a problem you could work on that seems a little it could be unethical but could also be you know I'm just gonna say it could be unethical should you join it and try to make it so that it goes down a more ethical route or should you choose not to work on it at all and I'm gonna leave out the money part of that because I think that maybe wasn't relevant maybe the first one gives you lots of money okay so now I don't you cool and make money yeah I don't think we should leave out the money part okay because people think about this so I think you should leave out the money part in the sense that you should make your choices so that you choose to work on something you are comfortable with that you you you feel good about and don't work on something that you feel queasy about you will find a non-jew job don't worry there are lots and lots of jobs in in our field so just go and find something else it's hard to recommend being unethical and making money I'm sorry no but of course other people will do it but that's not a good reason you know there are lots of unethical things happening in the world doesn't mean that you have to do it too thank you so I was wondering do see if there's anything psychology or little science can add to AI and if there's anything only it has already but it seems to me involve most of them I just added a philosophical directions or some inspiration no no it's it's much more in philosophical let me give you an example that people don't know about you know the the piecewise non the Erinyes that we use in deep learning that actually make a huge difference compared to what people use before which were there like sigmoid and hyperbolic tangent we tried doing this in deep Nets because we thought oh this is more biologically plausible let's try and see what happens another one is catastrophic forgetting the best papers I know about that are written are by psychologists and there that's a little strong but psychologists definitely have something just to say about this most one of the most important problems in our field all right let me give you an example very recent one attention mechanisms I mean it's like transform the field of deep learning at least for NLP applications completely and obviously it comes from an inspiration from psychology right so I have to go and do the obvious one which is I started in psychology and it was very clear to me the psychologist knew much more about learning and in any of the AI people and things like temporal difference learning came directly out of psychology and now let's come back and you know they they they look at parts of the brain the the reward systems in the brain are all the theorized in terms of temporal difference learning so that's that's a really clear example of strong positive interactions both ways so so let me add something else which is more philosophical in my way of thinking about interesting problems in machine learning often I apply a filter which is does this make sense from the point of view of humans or animals yeah and that eliminates a lot of algorithms I can tell you yeah very good pleased to hear that Yahshua okay how much of the of the things you've said about their prospects for people working in AI machine learning how much do you think it applies to outside Canada like in Europe or East Asia or Africa or South America like if you if you're going to you know not choose to move to Canada what no way no seriously I think it's happening all across the world of course we are you know ahead of time here but I see a lot of movement in Europe I see even in in developing countries and I think we have a responsibility to help that whether it's in Africa or even in South America in Asia I mean obviously so yeah I mean machine learning and AI are super hard all across the world and it's gonna have a tremendous economic impact we should be thinking about the whole world absolutely if you could pass her hi yeah I had a question so you were talking a lot about how do you pick projects that are worth pursuing but how do you pick projects that you should abandon like how do you decide that okay this is not working at which point do you decide ever that's hard so I noticed one thing with research ideas and projects that you you create a sort of emotional attachment to them and then well initially you're like in love right and then it doesn't work [Laughter] but you want to give it a chance and then the worst thing is when you say ah it doesn't work I'm gonna drop it and then two years later if somebody picks it up and it works really well so in my experience I would generally have multiple problems that I'm interested in and and some of them go for long pauses right and some some of them you have to you you ended up abandoning maybe you abandoned them just because they live on for an eternal pause yeah you can't you can't you can't cry over if the milk is spilled you know it's it's it's a son cost just because you've worked on a whole lot doesn't mean you should work on it some more you gotta always look at the future and you're gonna have many things you want to have a variety of things and then you decide one day oh this idea is ready I can just push it through and and and get the results and make a publication and this is the right time for that idea yeah so so it it has happened to me a lot that I have an idea at some point and then I work a bit on it and then maybe the first experiments don't work and I move on to something else but it stays in the back of my mind and then something else comes up maybe some someone else I meet who's interested in in pursuing this or new results that come in that suddenly could be useful and I go back to it right so you don't have to completely abandon something so this it's a very hard question you're asking I mean there's a this whole spectrum between being too opportunistic and they're not having a long-term vision and and then not listening to the evidence that well your idea is not working as an additional point to that if you're working on a problem that you think is really important and the thing that's not working is one idea to try to solve that problem then you probably haven't really wasted your time you're not really abandoning something you're gonna take some understanding that maybe this wasn't the best algorithm but you presumably learned a lot about your problem and then you actually only care about solving your problems so you're still working on your problem you're just not working on that specific algorithm to solve that problem so maybe the if you're working on problems are important to you that's likely not going to change that quickly oh and another suggestion when you you work on an idea and it doesn't work trying to figure out why and that in itself could be a very very important contribution and so even the negative results right we don't we don't reinforce sufficiently their importance in in our community but but really we should in fact some of my most cited papers have been negative results let me just say one or thing about that I mean maybe it's obvious intelligence is a large data thing and as many different parts we can't work on all the parts at once and that makes it challenging because you you're gonna choose some part to work on and how can you tell if it's really done right if it's failed or succeeded when you don't have all the other parts that it needs to work with and so this is a fundamental problem of our topic that we're working on something that we will only see it fully working when we have all the parts together which means we should respect other people's choices that are different from ours of course yeah so I had a question about more of on the what type of incentives there could be to kind of promote more value in the quality of the research instead of kind of quantity say key performance indicators now our paper base how many papers how many projects how many things and it doesn't necessarily equate quality it's an excellent question yeah so I have a related suggestion our field has focused a lot on conference papers and the advantage of conference papers is that the turnover I mean like the the the time to publication is very quick and we've left journal papers which usually take more time to be reviewed but the thing with conference papers is to come with deadlines and that makes us work like crazy on those deadlines and if there was only one deadline per year it wouldn't be so bad but now we have like six deadlines per year and in my group right I'm sure it's the same in many labs and so we just keep rushing like crazy all the time with little breaks in between if we were to shift our publication process so that the end product would be well if if if we were to say we we do like in other communities you would basically submit to a journal and that could happen at any time during the year or like usual and then if the paper is accepted then it could be selected for presentation at a conference that's like putting things upside down so the the effect would be that we would have better papers at conferences and we would not be always rushing for these deadlines now I don't know if this is gonna happen but in order to make it happen we would need to find a review process for journals that would be as efficient as the one we have for conferences so that you know in a period of three months we could make sure that papers are reviewed this is a really good problem like how can we structure our field so that it it is it moves in a good direction I have no idea how to do it I want to I want to get the microphone to Adam white here because I know he's he's like the world's best reviewer and he probably has some ideas about how to fix the field yeah I think I agree with the a show the moving to the journal thing is exactly what we've been thinking about in our group is switching to journal papers and and thinking how we can rearrange our conferences so that it isn't just this huge crush every sixth every six months of all these papers and it's it's not clear what every two months yeah and and you know we can use good metrics to to evaluate good research right like the ten year research awards those things really matter right the paper that was good in the last ten years that that's what matters yeah long term so that's another problem with conferences is that it puts us in a mode of short-term little projects and we do incremental research and it would never have the time to embark on something that might take two years before we get a result it just doesn't happen in our field anymore like every few weeks I have a student come coming into my office and asking me okay so there are five weeks left to deadline X what do you suggest I do and and I think I want to actually maybe turn it back on people like me and rich who are in these companies now where we have more freedom to to work on these long term projects and to show off what it means to invest over a multi-year project and and how that can really affect the scientific community and the progress of the field so we should be doing a better job setting a good example and not just following the trends also well I've heard that in companies also you get these whatever quarterly kind of reporting which also put people in this mode and then they are part of the community which is driven to publish so the the publication disease is sort of it spreads it's all over the place I have heard one suggestion given that we shouldn't evaluate people we should be asking for the best two papers that the person thinks that they did and it would yeah that should be enough you know either forgetting a faculty position or forgetting tenure or forgetting hire two companies so a number of citations of your two best papers rather than h-index maybe or maybe a justification for letters from someone saying these two papers are great for these reasons our seminal in our field for these reasons I have a question that I'm going to play devil's that ad advocate here not because I disagree but just to spur on the conversation do you think switching to such a method would stifle innovation and maybe slow down the progress and in what why I guess it just simply because people aren't seeing as many papers perhaps maybe getting as much out of them maybe coming up with more do you think we have enough papers well that's very true our field has grown enormously and we have to deal with that somehow or celebrated which as you should suggesting so so I think we could build new tools for organizing our results in something that might be slightly different from the current format of papers for example there is a sort of line image of papers that follow each other often from the same group sometimes they have there are revisions to the same paper and and it's hard to actually put these revisions in in a journal paper or in in a conference paper so so maybe we can like restructure the way that the knowledge is being presented so that it's not broken in so many pieces that are disconnected from each other but I'm not sure exactly how to do that yeah so if you were to recommend one or two or treat like a few activities that wouldn't take like a whole lot of time that would improve the efficiency what you do what would you recommend like the individual researcher what should they do some activities that people may not be doing and that would be respond to email immediately there's a thing in Gmail where you can like delay by whatever time you want really because the problem is like I spend all my time on email instead of doing research I mean some of it is research but like email is like destroying our ability to focus and concentrate on one thing for hours and hours which is what research should be about and number one is is writing I think Mohamed knows I'm a big proponent of every researcher having a notebook they write in ideally every day shoot for about a page a day just put your thoughts are explain to yourself what your thoughts are and put it in two sentences and paragraphs if you do that consistently over a period of time years you will have much more interesting things to say to other researchers how the coherence of thought one other thing I think sometimes people don't put enough emphasis on is collaboration like you can be a lot more effective if you work with groups of people and so if you have the tendency to think I want to work on this thing by myself because it's easier or I want this to be my only paper like I'm the sole author you will be less effective along that same vein what do you think is like a productive way to handle competition between researchers say like sharing ideas and you know how do you you know we did the same group or group or across organizations or I mean maybe even just within the same group I think it can happen so one option would be to remove author names from papers I think there's so much around the ego here but I don't know how to fix that I mean I'm kind of half joking but but really we should we should collaborate because it's fun to do it like enter into a discussion with somebody who has something a different view and and brings you something and really it you can see how it makes the science advance it's such a such an amazingly human and positive thing to do I think what part of the answer is we should make sure that the social norms including the laws make it easy to do that we want to remove barriers for collaboration as much as possible some of that is cultural like in some companies there's a lot of collaboration from with people outside the company and other companies it's not so some of that is sort of legal barrier some of that is cultural but we should try to minimize all of these barriers my experience may be special but it's it's valid experience and my experience has been very much that that you should always that you should not worry about someone stealing your idea if you if they do steal your idea you should celebrate because then they have the burden of carrying it out and talking about it and publicizing it and and and you know it's like you leave doubled your efforts because they're doing that and now you can go do something else it's it's really true and if your idea is really good you'll have to like force them to understand it you know it's like it has never worked out to be a problem for me say the same thing it's always wonderful if someone wants to hear your recent idea because you get a chance to say it out loud in the disadvantage to not saying I don't allowed is you won't catch errors in it and even just saying it out loud is extremely helpful much more likely to help than it is that someone takes your idea I think I'm gonna have to end it because I see the timer is done so let's thank our panelists here and I'm gonna hand it off to Warren thank you perfect all right thanks so much some awesome conversations going on today and I'm excited that we are about to transition into the evening one last word from one of our sponsors tonight never would have been able to happen without the support from Alberta innovates a long time Amie board member and CEO of Alberta innovates please join me with a warm round of applause to welcome to the stage Laura Gilcrease what an impressive panel Ashley what also an impressive audience now I have a speech for you but I don't think you probably won another speech right so if you may hold this number one I want to thank you for being here if you don't already know you are special over 1500 people applied and there's just over 300 here more importantly I want you to know about the province of Alberta and the province of Alberta this is a place for you to come learn actuate what you want to do and grow here and this event tonight is allowing you to interact with more of Industry to find the connections you want you know we are a big oil and gas province but we're also a big agriculture province and we're a big healthcare province and we're just a big-budget province and even though it's not about the money it's about what we do it's people like you that allow the future to be held and to grow here and this is why Alberta innovates has been a funder of Amy and the University of Alberta for 17 years in the art in the artificial intelligence space to date we've already invested 46 million and I think as we go forward there's going to be many many many many more opportunities even last week we just were awarded a hundred and eight million for the smart farm of the future in this province that will need your help in artificial intelligence with that data with the new robotics that the farmer of the future will need and that's just one tiny example you affect every business and every industry that will be in this province if not here today we'll be here over the next five to six years so my my plea to you is did you have fun are you gonna have fun tonight who's gonna find a new opportunity in Alberta or a deal here raise your hand come on ladies 30 percent of the startups come from Alberta in Canada right so I really really want want you to have a good time from my team at Alberta innovates the team we work with Amy the University of Alberta I can't be more ecstatic that you're here and if there's something you think we can do for you let us know but my challenge to you now is twofold first find one thing out tonight that's good for you a new connection a new company to talk about a new product to build a new startup and the second one is you're gonna have to race me to get to the bar Thanks all right with that enjoy the NED everyone there's a bunch of companies and really great people waiting for you just on this side of the curtain food and drink and I hope to see you at the bar as well thank you

Channel: Amii Intelligence

Views: 2,457

Rating: 5 out of 5

Keywords: Machine Intelligence, Reinforcement Learning, Amii, CIFAR, DLRLSS, Deep Learning Reinforcement Learning Summer School, AI, Artificial Intelligence, RL, ML, Machine Learning, Rich Sutton, Richard Sutton, Yoshua Bengio, Martha White

Id: kL5GJag6Ipo

Length: 60min 57sec (3657 seconds)

Published: Tue Oct 15 2019