State of AI Report 2020 (review)

i'm so so excited you know why partially because it's halloween so if we go here let me check out the date 31st of october 2000 oh it's my face in the way there doesn't really matter but anyway it's halloween little piece of advice if you are getting dressed up i'm not really dressed up if you are getting dressed up and going to some sort of party for halloween make sure that everyone else is getting dressed up too because if you go there and you're dressed as like a goblin and everyone else is in plain clothes you're going to stand out now that might be depending on how inclined you are that might be a good thing anyway that's not what this video is about happy halloween by the way we are going to go through i've been waiting to the end of the month to go through this the state of ai report for 2020. how exciting so this actually came out at the start of the month um october 1st 2020 again we're at october 31 2020. now what i figured i'll be fun to do is just go through it for probably an hour or so and then actually let's get a stopwatch out six stopwatch zero there we go let's go through it for the next hour or so um of course if you don't want to listen to me talk over this you can go read it through yourself www.stateofai.com i mean state of dot ai a big shout out to nathan benny bennett bernice for creating this and ian hogarth so without any further ado let's jump into it by the way i'm just offering my commentary on here i'm not gonna claim that i'm some sort of uh expert and know everything the insides and outs of what's in this report i actually don't know what's in this report so yeah take anything i say with a grain of salt and be sure to do your own research because that's what i'm excited to do that's what i use these things for i use them as information and then i go research it myself to figure out more but let's get started thank you thank you nathan beniac ben and ian hogarth just make sure my head is in there where we got here anyway i'll change that as we go through so introduction so we've got some definitions about ai multiple multi-disciplinary field of science and engineering whose goal is to create intelligent machines yes yes yes these are the key dimensions in the report so i did actually go through last year's report and it was phenomenal um so i am really excited to go through this years so it's broken down into research talent industry politics and predictions how many slides do we have here holy crap okay daniel you're probably gonna have to be quick no rush we've got some contributions jack clark if you haven't signed up to his newsletter yet make sure you do jeff ding chip hewitt chip hewn also has an amazing blog check her blog out andre capathi some names that i'm not going to pretend that i can pronounce some definitions i'm not going to read all them out i'll let you read them out algorithm model supervised learning unsupervised learning transfer learning natural language processing an executive summary oh this might be worth going through let me just i'm going to shrink myself down real small that way it's a focus on the content and not just my face all right so what do we have here research a new generation of transfer language models are unlocking nlp news cases use cases so yeah i think that's that's a big thing right transformers are kind of the architecture of choice for nlp at the moment but that being said i did watch a weights and biases podcast with um someone who's working in the nlp field and they found that transformers were were too big to use in practice for their for their application so again it's about tailoring the tools that you have to the problem that you have huge models large companies and massive training costs dominate the hottest area of ai today nlp yep gpt-3 i think that costs like 11 million dollars to train biology is experimenting its ai moment from medical images to genetics protein proteomics chemistry to drug discovery ai is mostly closed source only 15 of papers publish their code which harms accountability and reduce ability in ai 15 we need to step those numbers up team come on talent industry politics let's get into the slides scorecard reviewing the predictions from 2019 new natural language processing companies raise a hundred million dollars in 12 months so what happens with these reports is at the end they make our predictions for the next year i probably might make my own prediction why don't we do that um did they get this correct yes they did so a fair few companies are getting funding there in the nlp space no autonomous vehicle companies drive above 15 million miles waymo 1.5 million cruise nearly a million baidu 108k miles i wonder is is tesla self-driving car company could you include them in there because they've done a fair few i don't know how many privacy preserving ml is adopted by a f-2000 company yup machine learning well that kind of makes sense like it's a privacy i think that's like the trend of of 2020 is privacy i mean that's where apple's differentiating itself from things like facebook google um amazon and whatnot i mean google and facebook has sort of taken a step to go you know what we can make your data private if you want apple's sort of front and foremost everything's private from the get go um but it kind of makes sense just going forward like if you could do ml privately why not just do it um unis build de novo undergrad ai degrees so this is like a if a uni has a university has a dedicated ai specialization which i mean it's kind of the buzzword like you could use ai like i mean at the moment you could just dedicate it towards like a degree that teaches mathematics and and coding and data processing um so they got that right google has a major breakthrough in quantum computing they sort of got that right um and governance of ai becomes a key issue and one major ai company makes substantial governance model change nope business as usual research let's do it ai is less open than you would think only 15 percent of papers publish their code see that's that's pretty astonishing to me oh and this is this is a source from papers with code um great website by the way you need to check that one out so research code paper implementations uh important for accountability reproducibility and driving progress in ai i totally agree so there's been times where i've read a paper and i'm like this is amazing results and then go on you know what i'm gonna go look at their code and the code wasn't available i'm like oh you know what i'll spend a week trying to reproduce these results i actually did this in the past two weeks um and i i thought i did as close as possible to the the paper itself and i was getting like two two percentage points of f1 score uh below what they said they got and i thought i'd got everything correct so i emailed the authors i'm still waiting back on a reply notable organizations that don't publish all of their code open ai and deepmind so this is yeah open ai you kind of got it in your name um open ai but if you're keeping all your code um and research private well then is that really open again i'm not the one making the decisions to to put these things out there but in terms of the space itself um i know a lot of these companies are sort of private entities as well so a lot of their their work in ai is proprietary it's how they kind of make their money so if another company was to leverage that they would become competition but again it seems going forward if people like you or i could start working on the things that large companies are working on right now just in a room like this i mean chances are you just increase the the opportunity for new discoveries papers with code tracks openly published code and benchmarks model performance yeah papers with code's a great website i love that website so it's tracking state of the art facebook's pie torch is fast outpacing google's tensorflow in research papers ooh which tends to be a leading indicator of production use down the line wow that's pretty cool so where's percent of pie torch papers of total tensorflow pie torch papers holy crap 75 percent above 50 so 55 oh there we go 55 of them have switched to pie torch you know what the more and more i hear about pie torch so i'm i'm more versed in tensorflow than i am pytorch i mean i've been a long advocate for like you should you should know a little bit of both but the more i'm seeing this is pie torch is potatoes is coming up you know well well this is clear i mean it's it's more than coming up it's taking over so i think i think i've got some uh some learning to do over the summer but that's pretty cool pie torch is also more popular than tensorflow in paper implementations on github 47 of these implementations are based on pi torch versus 18 for tensorflow wow that's that's pretty full on i mean jax is another framework there's a google framework that is more math friendly and favored for work outside of convolutional met models and transformers mxnet caf2 caf2 is like a i think it's in pure c plus plus maybe only one repo um so yeah that's that's very interesting to see is that pi torch is becoming more more popular well it is more popular than tensorflow but that being said does the tool you're using get the job done you don't necessarily have to use a tool because it's more popular language models welcome to the billion parameter club we're talking billies now baby who's going to be the first language model company or the first company to build a language model with a trillion microsoft um open ai deepmind google who's going to hit the trilli first i want to see this truly so first models large companies and massive training costs dominates the hottest area of nlp today look at that gpt 3 175 billion parameters where's burt is burt on here burt large so bert large i mean that was the state of the art i mean that kind of cascaded the whole transformer revolution right but um 340 million parameters and that came out at the end of 2018 so 2018 to 2020 onwards look at that we have a quite a large step change an order of magnitude or more 10x 100 really nearly two orders of magnitude here bigger models data sets and compute budgets clearly drive performance empirical scaling laws of neural language models show smoother smooth power law relationships which means that as model performance increases the model size and amount of computation has to increase more rapidly so this is this is very interesting research um and it's kind of clear like what you've seen over the past few months in the nlp world is basically if you have the ability to build a bigger model and the ability to to run ridiculous amounts of compute you're going to win is that the is that like the the way to go forward like if we if you think about moore's law if compute's always going to increase some would argue that it's dead i don't know enough um but if you just imagine compute power is continually going to increase um is that how we get to to more and more state-of-the-art nlp models we just as compute power increase the models get better not entirely sure tuning billions of model parameters cost millions of dollars yeah here we go this is so this is where it's like limited to like if you unless you're here we go open ai's 170 billion parameter gpt3 could have cost tens of millions to train tens of millions that's a pretty that's a pretty costly experiment like i mean imagine being the engineer at openai who like hit the hit the enter key to like kick off that experiment it's like all right have we got all the code lined up we're going to train this full big dog 175 billion parameter model and it's going to cost 10 to 20 mil so did you get did you make sure you've you've tuned all your hard parameters did you make sure you've uh you're saving the model results um budget was 10 million dollars so yeah that's like what we said so google's t5 costs well above 1.3 million dollars for a single run i mean yeah this is like these are the kind of experiments that only large scale companies can run um so what i'm excited for like going forward is research like this but making it like super super efficient that's what i'm really excited for like because i mean of course i'm excited to see like large companies throw millions of dollars at training runs but i'd like to be able to see you like someone you know what we got 95 percent of the results of gpt3 on a single gpu is it possible i don't know um to achieve the needed quality improvements in machine translation google's final model trained for the equivalent of 22 tpu v3 core years or five days with 2048 cores non-stop whoa quality gain okay that's what you want so you have to train for a ridiculous amount of time well in terms of uh computation cost to get oh where's the improvement from what's that an 8 2048 divided by 128 16. so 16 times you need a 16 times more compute to get double the double the performance gain to me that's uh that doesn't really add up you know 16 times more compute but only double the performance game we're rapidly approaching outrageous computational economic and environmental costs to gain incrementally smaller improvements in model performance yeah this is what i'm saying right it's like if we're without major new research breakthroughs dropping the image net error rate from 11.5 to 1 would require over 100 billion billion dollars oh my god 100 billion billion where we got today the target the today the error rate is 11.5 percent so right now to train state of the art on imagenet it looks like it costs can you see that ten ten to the power of six that's a million well i can't zoom in here you're just gonna have to check out this line so yeah that's what i'm saying where where although massive massive models uh like exciting to like read about and see or whatever um they're just getting like outlandishly expensive and costly on the environment to train like if they're releasing so much there i mean you could argue that yeah google's data centers are all uh renewable energy i'm not entirely sure if they are but i think they're moving that way but still massive increase in cost for not that great an increase in performance a larger model needs less data than a smaller pier to achieve the same performance yeah this is very interesting research as well so this has implications for problems where training data samples are expensive to generate which likely confers an advantage to large companies entering new domains with supervised learning based models yeah so this graph here what is showing we've got test loss here tokens processed if you have 10 so a billion parameters 10 to the power of 9 your your test loss is lower but if you only have a thousand parameters your test loss is higher so larger models require fewer samples to reach the same performance the optimal model size grows smoothly with a lost target and compute budget so what do we got here compute pf days line color indicates number of parameters this is actually beautiful graphs once you sort of dig in and figure out what they're saying so the larger this is the large model here compute efficient training stops far short of convergence so the larger the model the few fewer samples it requires and as the model size increase the amount of compute increases that makes sense low resource languages with limited training data are a beneficiary of large models so google made use of their large language models to deliver high quality translations for languages with limited amounts of training data for example hansa and uzbek this highlights the benefits of transfer learning yeah so this is a big thing like it's very and this is my own like cognitive bias like in the field of of natural language processing and just just in life in general is you kind of get narrowed down like if i'm fluent in english and i read english and i write english so i kind of if i'm not interacting with with languages like hansa and uzbek i i don't have experience with what it's like to to work in those those sort of if i was working on a natural language processing problem with with a language like that and so i really really do like even though i personally don't work with many foreign languages at all i really do like seeing research where it's including languages with with limited training data or that aren't as popular as english so although this is not my field of of work i love seeing this stuff even as deep learning consumes more data it it continues to get more efficient okay so this is this is good to hear since 2012 the amount of compute needed to train a neural network to the same performance on imagenet classification has been decreasing by a factor of two every 16 months wow okay two distinct areas of compute in training ai systems so the first era was uh prior i think this is about 2000 and what's yeah 2012 this line here and the modern era so i'm not sure what this access is but it just looks like the modern networks are increasing in compute but the training and fish efficiency factor so you got over here i don't know if you can see that but that's efficient net b0 which is um a computer vision model is getting has a i'm not sure how this efficiency factor is measured but it's right up the top there compared to uh an architecture like mobile net v2 which was state of the art like two years ago so this is good to see yet for some cases like dialogue small data efficient models can trump large models yeah so this is what i was saying i um poly ai i'm not too familiar with with their company let's uh let's get a poly ai let's see what they do enterprise ready voice assistance for customer service okay so you could call up oh wow that's cool except human level conversations all right wonderful so they're doing oh wow we're only up to slide 23 and we're like 20 minutes in faster daniel faster okay um so poly ai is crushing it in how good their conversational ai is so intent accuracy oh they're out doing bird and amount of data so on low data they're still getting about 80 accuracy yeah this is the stuff that i i mentioned before i really like to see is that like hey we got sometimes like especially when i was working at my my previous role um if a really efficient model so say it took 100 less times compute but would get 95 or 97 98 of the results of a model that took 100 times more compute we would usually go with the more efficient model even though its results were um weren't as good because on the long run uh even though its results weren't as good it was it ran it was easier to deploy it actually ran in production yada yada so i like seeing these things like getting good good performance with not as much data um so what have they done polyai a london-based conversational ai company open source their con con vrt model a pre-trained contextual re-ranker based on transformers transformers coming at it again their model outperforms google's burp model in conversational applications especially in low data regimes suggesting bert is far from silver bullet for all nlp tasks but although is an amazing model oh there we go model size but 1.3 gigabytes yes that's the exact problem we ran into we tried to use bert it was too big and now convert rt convert convvrt is only 59 megabytes that's what's up like an order of magnitude two nearly two orders of order of magnitude and a half times smaller a new generation of transformer language models are unlocking new nlp use cases i mean gpt3 is like you've obviously seen the um oh here we go code generation gpt3 examples.com here's a sentence describing what google's homepage should look like and here's gpt generating the code for it nearly perfectly okay so describe a layout two light grey buttons that say search google and i'm feeling lucky with padding in between them and it recreated google's website that's wild absolutely wild you know someone was asking like if someone was arguing i heard that gpt3 okay it's it can take like instructions like this but it can't tell you why um it produced like the output that it did and i would argue as well if you think back to humans if you give someone humans like an instruction like walk up this hill and then the person walks up that hill is like why did you walk in the left side of that hill and so you don't really know anyway computer please convert my code into another programming language this is really cool so i don't know c plus i know python but even if i look at that i would have to um i would have to go through it that would take me a significant amount of time to like reverse engineer and figure out you know but apparently an unsupervised machine translation model trained on github projects with a thousand parallel functions can translate 90 of these functions from c plus plus to java and 57 so 90 of c plus plus to java and 57 of python functions into c plus plus and successfully pass unit test what that's wild and see this is what excites me about machine learning right is um is if you can frame your problem really really well like what inputs do you have okay a bunch of code on github and what outputs do you want in other words translate this code into this code language if you can frame that really well and then you know sort of like the the basic premise of of unsupervised learning supervised learning what type of problem you have you can very quickly or not very quickly but you can start to bridge the gap between their inputs and the outputs you desire using machine learning it just requires you to frame your problem very well once you get that i mean you're seeing examples of this all over the place is turning turning code into a translation problem which essentially is is what it is right if we if i was a python expert and a c plus expert and i wanted to convert this this function here sum of k sub array into c plus plus i could go through this and translate it just like i would translate something like english to french computer can you automatically repair my buggy programs too oh here we go we're all done computers are now starting to repair code a sota is set on deep fix which is a program repair benchmark for correcting correct intro programming assignments in c wow yeah so unless you can write bug free code you're going to get overtaken by deep learning algorithms no i don't think so i think we're we're still a little while off of a maybe who knows gpt4 might be out of right fluent python code nlp benchmarks take a beating over a dozen teams outrank human glue baseline that's wild it was only 12 months ago that the glue benchmark was beat by one point now super glue is inside so i believe glue is like a way to evaluate your nlp model glue nlp so glue benchmark here we go glue the general language understanding evaluation glue benchmark is a collection of resources for training evaluating and analyzing natural language understanding systems i'll let you read about that now oh and there's also super glue there we go i'm imagining super glue is just a harder version of glue and 12 months ago one company or one research group broke the human baseline and see that's what happens in a cascade it's usually when someone sort of makes a little breakthrough you're gonna watch a whole bunch of other people do it as well because it's like that kind of uh follow the leader effect you know so that's really cool nlp is just taking over let's step this up let's uh we're only up to slide 28 what's next after super glue let's let's do a few of these a bit faster what's next after superglue more in challenging nlp benchmarks zero in on knowledge yeah so again if we're trying to evaluate these these super crazy big models we need harder harder ways to evaluate them need harder tests for them to if they're just blazing through the old test i mean future models are just well if you're getting all these models that are just crushing them well is the test really helpful transformers ability to generalize is remarkable it can be thought of as a new layer type that is more powerful than convolutions because it can process sets of inputs and fuse information more globally yeah so this is wild gpt2 which was originally a natural language model was used for image completion so it was fed say half an image and then it completed to this and then this that is wild actually i want to show you a paper that i found um what's it called images transformers for computer vision am i going to see it um oh this is one end-to-end object detection with transformers go check that out but i believe i saved it to my pocket there we go bring this over here and images were 16 by 16 words so transformers for image recognition what does it say here when pre-trained on large amounts of data and transferred to multiple recognition benchmarks vision transformer attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train that is a major key right there could transformers be the architecture to solve them all who knows and it's actually it's actually worth diving into because the way i view transformers is just a whole bunch of attention layers stacked on top of each other um very primitive description there but go check it out biology is experimenting experiencing its ai mode moment over 21 000 papers in 2020 alone this is what i'm i'm really excited for the crossover of ai and health um we've got a bunch more papers coming who knows what's going to come into it but that's it's exciting to see the research grow from physical object recognition to cell painting decoding biology through images so now we have a large image database of cells treated with very various chemical agents so instead of getting great results on imagenet you can now run your computer vision models on different images of cells and find some insights there deep learning on cellular micro microscopy accelerates biological discovery with drug screens this is wild use deep learning to just to do to discover new drugs um oh my goodness hard word op ophthalmology advances as a sandbox for deep learning applied to medical image imaging let me uh i don't know what this word is how do we spell that op op fell what does this do what's an ophthalmologist do an ophthalmologist is a specialist doctor who diagnoses and manages eye conditions and disorders of the video visual system i thought that was an optometrist maybe i'm getting that wrong anyway ai based screening mammography reduces false positives and false negatives in two large clinical clinically representative data sets from us and uk yeah i think i saw this this research the google health and deep mind the eye system and his ensemble of three deep learning models operating on individual lesions individual breasts and the full case was trained to produce a cancer risk or between zero and one for the entire mammography case the system outperformed human radiologists and could generalize to u.s data that is really cool when trained on uk data only so this is i mean this kind of goes to show is not only are we all like human is that if if one like country has some sort of breakthrough so on mammography for us data we can bring that to the uk or vice versa this is what happened here the the model could generalize to u.s people when it was only trained on uk data so that's the exciting thing about biology is that although we are all different this this kind of shows that it can a use case developed in one area of the world can be adjusted for another area of the world causal inference taking ml beyond correlation uh most ml applications utilize statistical techniques to explore correlations between variables this requires that experimental conditions remain the same and that trained ml system is applied to the same kind of data as the training data yeah this is a tough one right so it's like if you you train a machine learning algorithm on a certain training set and you deploy that into production and the data that you're you're getting in your your production system is not from the same distribution as your your training data set well then your model is not going to perform as well but again this ignores a major component of how humans learn by reasoning about cause and effect so if we learn something right if you go if you learn how to ride a bike and it's a small bike and then you go to try and ride a bigger bike at the start it might not be that great but then you can you can leverage the skills you learned on the smaller bike to adjust to the bigger bike so if you imagine like an ml system trained on one set of training data and then it finds out that it's uh in its it's in test production on the test set it's not performing as well how can it leverage what it's learned from the training data to improve its performance on the test data so many pioneers in the field including judea pearl pictured and joshua bengio believe that this will be a powerful new way to enable ml systems to generalize better be more robust and contribute to just more to decision making causal reasoning is a vital missing greeting for applying ai to medical diagnosis yeah that's really important right so you want causal reasoning when you're making some sort of medical diagnosis model explainability is an important area of ai safety a new approach aims to incorporate causal structure between input features into model explanations okay um so this is okay so it says shapley values have a floor asymmetric sharply values are proposed to to incorporate this causal information so if we look up um chat values i think chat values i've only had a little bit of experience shap here we go shop is a game theoretic approach to explain the output of any machine learning model it connects optimal credit allocation with local explanations using the classic shapley values from game theory and the related extensions so yeah if you want to explain the outputs of your machine learning model you're probably going to to look into shap and apparently there's a better version now an improved version asymmetric shapely values reinforcement learning helps ensure that molecules you discover in silico can actually be synthesized in the lab this helps chemists avoid dead end dead ends during drug discovery yeah there'll be nothing worse than finding a promising drug and then figuring out you know what we can't actually make that in the lab have you have you have your desired molecule ml will generate a synthesis plan faster than you can wow so this is yeah where well this is i mean this is more so chemistry than biology isn't it again repress repurposing the transformer architecture i need to get i need to get my head fully around the transformer architecture and maybe a future project for me going forward is um using transformers for just just figuring out how to apply them to almost anything graph neural networks i've heard of these i haven't looked into them much but i hear they're making some great uh insights particularly on 3d data so ie non-euclidean space so yeah most deep learning methods focus on learning from 2d input data see i get confused when i read that because i i imagine tensors to be multi-dimensional but you think about it yeah an image is only length by width a sequence is like one dimension hmm i need to look into more into to graph neural networks graph neural networks learn to guide antibiotic drug screening leading to new drugs in vivo oh i always get these mixed up in vivo studies that are in vivo are those which the effects of various biological entities are tested on whole arca in vivo is on actual living organisms enhancing chemical property prediction using graft neural networks well so this is yeah this is definitely something that i need to um so here's what i do when i see something that i don't know i go graph neural networks it's amazing what you can learn if you google it um a general introduction to graph neural network basics here we go i'm going to bookmark this and read that later ai sifts through chemical space using dna encoded small molecule libraries dell uh again graph neural networks trained on dell so um i imagine there's some so this is again this is um this is the the formulating your problem what are your inputs so in the case of graph neural networks i'm assuming the data is is not suitable for traditional deep learning neural networks so they had to adjust uh the bridge between the desired inputs and outputs of an ml system language models show promise in learning to predict protein properties from amino acid sequences alone i remember when i first started learning about language models i thought that well if we imagine dna is just a sequence of um a c's g's and t's um why can't we just train a language model on that and then figure out sequences in there of course com way easier said than done obviously but this is showing some promise well as it says shows promise in learning to predict protein properties from amino acid sequences alone so if you have acgtta ghcta can you predict what kind of protein is going to come from that sequence covert 19 analyzing symptoms from over 4 million contributors detects novel disease symptoms ahead of public health community and could inform diagnosis without test wow so loss of smell is the most predictive symptom of covid19 if you've lost your smell go and get a covert test oh zoe i love that company what is it join zoe zoe.com i actually applied for a job here i got denied so i believe they're working on personalized nutrition yeah here we go learn the groundbreaking science of predicting nutrition that's that's what i'm slowly moving towards you know i'm not in a rush not in rush the trajectory is going that way we come here might have to start my own company um drug discovery goes open source to tackle covert 19 this is a rare example of where ai is used actively used on a clearly defined problem that's part of the covert 19 response i think this is this is yeah one of the the cool things that like open sourcing um ai techniques can be used for so we we at the start of the video we talked about how what are the stopwatch we talked about how um only 15 of research papers codes are available but if we have like a global scale problem like something like covert 19 then in that case it makes sense to just open source everything and get all hands on deck basically you never know some teenager in a in a room who's just an absolute prodigy just uh spending eight hours a day working on some sort of problem might stumble upon some sort of breakthrough of course rare but that's if you look back in history that's that's how a lot of discoveries have been found and if you ever say oh i'm too young that's a that's a lame excuse whatever age you are you can you can get into these things i mean it's it's code and it's math math is a language of nature [Music] missed out on strawberries and cream this year a controllable synthetic video version of wimbledon tennis matches what combining a model of player and tennis ball directories pose estimation and unpaired image to image translation to create a realistic controllable tennis match hold on a realistically controllable tennis match video between any players you wish vid to player i've got to check this one out vid to player stanford here we go what's this going to show me that can't be is that fake what you're kidding me this is generated okay that's wild this is where what do they use for that i'm going to have to read that i'll save that this is what i do i save things and i tag it with ml to pocket great app and we're only 47 through let's do a burner attention turns to computer vision boom transformers for computer vision and object detection there we go detail detection transformer detail uses 2d images features from a cnn flattens them into a sequence and then uses transformers to model pairwise interactions between the features holy crap that's so cool um computer vision predicts where an agent can walk behind what is seen i think yeah computer there's so nlp is having its moment but there's still a lot like computer vision is just getting wild lately the things you can do with one camera i mean look at like the new iphones have like that depth thing and whatever will that be helpful who knows that's again if you can define the inputs and outputs of your system you can work out the bridge between computer vision learns stereo from single images so that's depth from single images um enabling the use of consumer grade 360 cameras and construction using deep learning um learning dynamic behaviors through latent imitation imagination wow so dreamer is an rl agent that solves long horizon tasks so things that are in the in the future um or tries to predict steps that are in the future that's how i think of long horizon from images purely through an imagined world though this reminds me of world models great paper a few years ago uh dreamer predicts both actions and state values by training purely and in an imagine oh my god and imagined latent space i think this is where we're also going to see a big a big step in the coming years is um is yeah late in space so synthetic uh a massive improvement to synthetic data learning learning to drive by predicting and reasoning about the future predicting how a given driving situation will unfold ranging from what the driver will do and the behavior of dynamic agents in the scene can help autonomous agents learn how to drive from videos yeah so this is this is what i think yeah it can be if you imagine self-driving cars to me the the data labeling problem um if you're collecting data for self-driving cars you've got so many different variables you've got lights cars roads lanes etc etc uh a human a human of course when i drive i'm like registering those things but one more one way that i heard that i think comma ai especially george hotz is talking about training it is to um take all the training data and the labels are what a human driver does in a given situation rather than labeling every single car and i think that's a really cool approach to it so i think this is kind of what what this is talking about learning to drive by predicting and reasoning about the future what the driver will do visual question answering about everyday images again this is really cool this is helpful this is from uh laura it's an approach that reads text in an image and jointly reasons about the image and text content to answer a question from a fixed or by selecting one of the ocr strings derived from the image so this is yeah this is really cool it's like can you take a photo of something and not only grab the image details but grab the text details that are in there because that's so much of what we if you imagine a food package what's the image of the food and what is the information on there can you capture that in in one go learning a multi-purpose generative model from a single natural image wow on device computer vision models that won't drain your battery yeah this is what this this is what gets me really excited is efficient debt d7 so efficient debt is a object detection algorithm achieves state of the art on coco object detection tasks with four to nine times fewer model parameters than the best in class and can run two to four times faster on gpus and five to 11 times faster on cpus than other detectors so this is this is the stuff that i like to see um is getting incredible results way more efficient i mean who let's be real who doesn't like to see that um evolving entire algorithms from basic mathematical operations alone with automl zero that was a wild paper so it starts off automl zero from how i understand it was uh let's look this up it starts with basic like mathematical operations like addition plus minus uh automl zero go read this paper or the blog post and then what i usually do is i read the blog post and then i read the paper so the blog post is typically uh worded in in not as a technical way as a paper so that kind of gives me the the ground base of understanding what's going on and then i i read the paper and sometimes a lot of the time i read a paper the first time and it's like well this is this is like a foreign language to me but then i read back through it again and i go okay someone's put a lot of work in here to describe their work um where we go yeah search for rather than just uh searching for um the best neural network architecture automl zero searches for an entire algorithm from scratch crazy but it did take a lo of a large amount of compute power to do kicked off by 2000 and google in 2016 federated learning research is now booming federating learning is so imagine i have my computer and you have your computer and you have an xbox and they're not getting used rather than use a whole bunch of of deep learning uh or gpu chips on a server why don't we harness the power of computers that aren't in use so instead of training on 10 super computers in google server why don't we train on a million android phones that people aren't using while it's on charged overnight so you imagine your your phone could run a very small part of a model while you're not using it and it's plugged into charge so it's taking advantage of compute power that exists but isn't being used which to me makes a lot of sense good to see that's growing open mind the leading open source community for privacy preserving preserving ml yeah if you want privacy preserving ml open mind um founded i believe by andrew trusk who is someone you should definitely follow on twitter um i am trust here we go so he posts he posts some great stuff um where we go here yeah lead open mind senior research signed a deep mind phd student uh teacher so just phenomenal person and but yeah he posts some great tweets about like getting started in ml basically oh there we go federated learning cavemen had super computers in their brain wanting the world's most powerful machine learning algorithm for thousands of years what held them back for so long access to training data beyond their family and village books libraries internet wikipedia got federated learning there we go speaking of federated learning so yeah go give our andrew trust a follow um yeah big shout out to andrew charles thank you for the work that you do far out we need to really bust through here prospective testing begins for privacy preserving ai applied to medical imaging again that just makes sense privacy preserving ai for medical stuff gorgeous processes struck back i don't know enough about gaussian processes um but quantified uncertainty and faster training speed so this is yes this is what we want uncertainty in so if your model makes a prediction how certain of it is of that prediction is it of that prediction so in a system you would you would like to know okay maybe um you so again a problem that i worked on we had a system where our model was 99.8 accurate on a binary classification test so really good but it was kind of like if it gets something wrong well that's not good so we wanted to know rather than we wanted to know something more than just how confident a model was in a prediction we wanted to know how much uncertainty was around that and the way we did that was through monte carlo dropout monte carlo dropout um what is monte carlo dropout etc etc and i believe the reason why we couldn't use gaussian processors is because they weren't fast enough and now we can get quantified uncertainty here we go gp training time is reduced from 15 minutes to 40 seconds beautiful i'm gonna look that up i'm gonna save that again second mind ai gaussian process boom i'll save that for later here we go 2019 prediction outcome google quantum supremacy section two talent the great brain drain ai professors depart u.s universities for technology companies to me this is kind of like okay university is paying you a hundred grand a year if you come work at google we'll give you a million dollars a year and you can still research the stuff that you're passionate about and that's how that's how i see that i mean a lot of people who who wouldn't make that decision almost um tech companies endow ai professorships in return for poaching but is it really enough new professorships may free the ladder for young academic talents to rise meanwhile some companies including facebook champion the dual academic industry affiliation as a solution some academics don't buy it this model assumes people can slice their time and attention like a computer but people can't do this okay so they want them to spend spend some of your time 80 in industry and then 20 of your time at the university no thank you see humans don't operate very well when we half-ass things that's like number one rule in life don't don't half-ass always full ass the loss of ai professors seems to matter departures correlates with reduce graduate entrepreneurship across 69 universities wow this is interesting um so the professor leaves the people don't start businesses i'm not sure how that well it's correlation right so this is what i want to i would like to see is uh of course more businesses can 100 million euros buy you 50 professors for a new ai institute uh the idhoven artificial intelligence systems institute in the netherlands plans to recruit 50 professors so they're dropping 100 ml if you're a ai professor and you want to go work in the netherlands they've got a big cash stack 100 mil donation from silver lake 29 prediction outcome abu dhabi opens the world's first ai university wow the beautiful thing well i to me the internet is the ai university the code university the tech university all the imagine this right imagine you have all the learning resources that you need to to learn a subject oh wait you do you have google like there's no shortage of learning resources there's a shortage of uh uh like the ability to get it dedicate oneself to learn that's that's what the shortage is chinese educated researchers make increasingly significant contributions at new reps yeah this is amazing china are just just pumping out ai papers after leaving university in china 54 of graduates go on to publish in europe's to move to the usa again i can't really comment on this i've never lived in china or the u.s the us is an incredibly strong talent retainer post phd actually speaking of the us i did have the idea when i was studying my ai master's degree online i couldn't think i could find a job in australia and then i decided to think about going to the us and lo and behold before i did that i got a job in australia so i'm not sure where i was going with that oh yeah i was attracted to the us because i think you see all the large tech companies you see all the research a fair fairly large chunk if not the majority comes out of the u.s so it makes sense to to want to go there three of the biggest questions that you can ask yourself in life is what to do who to be with and where to live it's worth spending a fairly long time on those three questions foreign national graduate graduates of usai phd programs are most likely to end up in large companies um that makes sense because i think large companies can just pay you more foreign nationals are two times more likely to join large companies in part due to their h1 be sponsoring power we only have four minutes left the uk and china the biggest beneficiaries of american educated ai phds uk or the uk and china so people from the us go to uk and china the majority of top ai researchers working in the us were not trained in america oh wow so 27 come from china holy that's very interesting given how dependent america's ai industry is on immigrants there's been a strong backlight backlash to trump's proclamation to suspend h-1b visas i don't know enough about that but to me it kind of it makes sense if you're if you're allowing smart people into your country to work on and help businesses make sense again i don't make policy around that american institutions and corporations continue to dominate nurit's 2019 papers google stanford cmu mit and microsoft research own the top five look at that google is just crushing it but then again it's a quantity or quality the same is true at icml which is another massive machine learning conference american organizations cement their leadership position google doing a lot stanford mit berkeley cmu microsoft facebook leading universities continue to expand ai course enrolment stanford now teaches 10 times the students per year during 1999 to 2004 and twice as many as 2012 to 2014. well you know what all of these stanford courses i'm pretty sure all of them are online so if we look at this cs224n boom if you want to do stanford's natural language processing with deep learning you can you can go through that you've got the coursework here you've got the lectures on youtube create your own assignment from it share your work on github share your work on your own website and then start applying to companies using natural language processing demand outstrips supply for ai talent here we go analysis of u.s data shows that three times more job posting uh than job views for ai related roles job postings grew 12 times faster than job viewings uh in the last in the last couple of years from late 2016 to late 2000 2018. i'm losing my ability to say 2000 and something um so yeah this is saying that three times more job posting so postings per million searches per million so a lot more there's a lot of demand out there for ai ai talent while hot the ai talent market is not immune to covert 19 during the pandemic here we go oh this is a cool little um graph so tensorflow and keras has more job postings on linkedin than pie torch but high torch is more common in as we saw at the start of the video pie torch is more common in in research oh this is from francois chalet who is the founder of tf keras so might be a little biased there i mean founder of maker of keras which is now a part of tensorflow which is kind of confusing but i digress industry we're only up to section three we've been going for almost a full hour um the first phase one clinical trial of an ai designed drug begins in japan to treat ocd patients wow ai designed drugs are now coming to to the real world that's awesome emergent evidence is that large pharma is validating ai first therapeutic discovery outputs ai drug discovery 2019 prediction outcome large pharma and startups ally around privacy preserving machine learning for drug discovery yeah well that makes sense right as we've said privacy preserving ml for medical stuff if it can be done it probably should be done deep learning models interpret protein biology to find new therapeutics this is what we were saying before is um a combining ml with carefully designed experiments has enabled lab genius to increase the number of potential drug candidates by up to a hundred thousand fold so that's improving how many people can take a certain type of drug there we go that one hour timer is done you know what i think what i might do is because we have a whole bunch of slides to go i have to go to uh jiu jitsu training right now but i might come back later today and finish this off we might go for another half an hour or so why not this is fun and again if you want to just read this thing yourself look for the top link in the description so i will be back uh in a couple of hours for me but it'll only be a second for you well well well fancy seeing you here training's done massive session let's finish off this where were we we were on slide at number 80. i just have to tick something off give me a second uh today's goal snap next and cash checks yep i'll take that off jiu jitsu training did snap a few necks and here we go let's get uh let's get 30 30 minutes back on the clock boom and we're gonna finish off the rest of these slides using genetic metabolic and metagenomic and meal contest information from 1 100 study petitions to predict individuals metabolic response to food at scale now this is what i'm i'm really interested in because my like formal education is in nutrition so i went to university i studied science and nutrition for well food science and nutrition for a few years um so i have a deep interest in that and i love seeing this so zoe i did apply for a job there once but then i realized you know what i prefer to to work on my own for the time being you know what i might re-see see what they have but uh big shout out to uh to zoe i love the work that you're doing and ml predictions of glucose triglyceride response two hours after meal consumption correlate seventy percent of the time with the actual measured responses zoe's commercial ai test driven kit uh launched in the us in august 2020. so only recently so i've got to go check out their paper papers or recent findings in 2019 the fda acknowledged that the traditional paradigm of medical device regulation was not designed for ai first software which improves over time i totally agree so this is what i'm thinking like just in terms of regulatory systems that have been set up if things are like 20 years old like a lot has changed in the last 20 years uh we didn't have smartphones everyone didn't have a supercomputer in their pocket we didn't have the internet wasn't as prevalent as it was and i'm not saying all regulatory systems are set up like 20 years ago who knows when they were set up maybe longer maybe shorter but we're starting to move into a world where the interconnectedness is is just extreme so old laws may not apply to such a severe amount of of interconnectedness so i think going forward um there has to be some sort of updated system to to regulate good machine learning practices new international guidelines are drafted for clinical trial protocols spirit ai and reports consort ai that involve ai systems in a bid to improve both quality and transparency i'm not sure what that is uh ai based medical imaging studies have a major problem twenty thousand recent studies in the field found that less than one percent of these studies had high quality design and reporting yeah so hmm there's another big thing like you could do a study um and think that you find valuable insights but if it wasn't structured in the correct way like say it wasn't built up with a with a science foundation all of your findings could be completely wrong you know the worst thing than discovering something that doesn't work is finding something that does work but for the wrong thing so that's a big big problem so if you're designing a study um make sure you do your research and make sure the system is set up or the study that you're doing is correct uh so that you don't get to the end and think oh oops uh the floor in my methodology has rendered all of our results useless the first reimbursement approval for deep learning based medical imaging product has been guaranteed by the centers for medicine and medicaid services cms in the u.s wow that's really cool so vs ai was granted a new technology ad on payment of up to one thousand forty dollars per use in patients with suspicious strokes so i believe it might be an app for some sort winning reimbursement from the cms is a critical step towards any new system becoming implemented in clinical medicine because it creates a needed financial insistence to drive use yeah so this is one of the things it's it's one thing to to be able to code uh like an app like you could you could build a prototype of something in a day the hard part will be uh scaling it up of course and then the even harder part will be implementing it into an actual system so getting it into to people's hands if you're building a medical device getting it into people's hands i mean in theory it could work really well but then in practice totally different things so really cool to see viz ai getting something uh working in practice um u.s states continue to legislate autonomous vehicle policies uh this is just making sense as autonomous vehicles start to arise we're going to need to figure out how they interact with the rest of the world even so driverless cars are not so driverless only three of 66 companies with automated vehicle testing permits in california are allowed to test without safety drivers since 2018. hmm so waymo neuro and auto x i haven't even heard of auto x to be honest are there only three companies who can have their cars driving without safety drivers well that's i suppose if you want full autonomous vehicles right i think we're gonna a better thing is before full autonomous vehicles we're going to go uh level three so with like i'm not sure what the exact levels are but i'm not sure we're gonna ever who knows ever is a long time ever move towards full autonomy because there's just so so many different variables i mean highway full autonomy that kind of makes sense but when you're when you're coming into uh like suburbia different scenarios who knows maybe there'll be zones right so like you can only fly your drone in a certain area you can only use your autonomous vehicle in a certain area that that kind of makes sense um anyway self-driving mileage in california remains microscopic compared to human driving so we got here wow but this is it's a slow it's a slow burn right like of course self-driving is going to be smaller than human driving but then you imagine if if autonomy does become a thing the curve of the amount of kilometers driven by or miles driven by uh autonomous vehicles is going to exponentially increase uh compared to human mileage because if you have a lot of autonomous vehicles out on the road you can just scale them up all of a sudden this two thousand two million eight hundred seventy four thousand miles could easily turn into twenty million miles sketchy metrics tracking av price progress is complicated by the industry's focus on miles per disengagement which is hard to benchmark and is not reported across all u.s states that's a good that's a good uh a good statement you know it's like how do you judge um how good a self-driving car is at self-driving so disengagement means uh how many miles did the car drive before it had to go you know what i can't handle this scenario human takeover and baidu looks like it's it's had a mammoth improvement over the last wow over the last year nearly plus nine thousand percent but again use your gut trust your gut if improvements on year on year of plus nine thousand percent if they sound too good to be true self-driving when even a billion dollars isn't enough so amazon acquired zoox which is a self-driving car company um i think amazon want to get into the the self the autonomous fried hailing uh they want a piece of the pie you know if autonomous vehicles are going to be a thing this is basically amazon's model it's like if an industry is making money and we can do it better we're going to come into that so i think they acquired some sort of self-driving car company zeus but that apparently this car company was burning 30 million dollars per month in early 2020 so this is a the challenge right autonomous vehicles yes they have a massive payoff but you also need a ridiculous amount of capital to start off so 30 million dollars per month and 900 full-time employees with no like deliver like they haven't of course they deliver prototypes but they haven't delivered like a working product this is why you got to hand it to people like tesla and comma ai who are uh i'm not sure if tesla's profitable but they're definitely delivering vehicles and they're they're at least have massive amounts of revenue and comma ai deliver shippable intermediates and are also profitable and so they're sort of they're making profit while be building self-driving cars that's the that's if you can ever that's to me that's getting paid to learn a really good scenario in life is getting paid to learn if you can set that up for yourself do it the main self-driving contenders raised almost 7 billion in private rounds since july 2019. holy crap look at all these self-driving i didn't know there was this many self-driving car companies voyage auto x pony dot ai that's a good name waymo 5ai ai motive dd another self-driving group dd so db dd is like uh uber in in china we've got dd in australia too i'm not sure if you have dd where you live but um you can order an uber or a dd they basically offer the same thing but again it's very uh they're very interested in self-driving cars because if they could replace human drivers they don't have to pay the human driver spins off from its parent and raises 500 million dollars dd's self-driving unit raised 500 million from softbank vision fund grew its team from 200 to 400 last year and launched its ride hailing service to consumers in shanghai in late july 2020. wow capital is used vertically integrate and deepen technology modes aeg in-house lidar so a lot of companies self-driving car companies using lidar which is to me it's like it's depth sensing so it's uh rather than just a camera um it's sensing depth from the environment but again to me this is just adding more pieces to the puzzle of course i'm not an expert in self-driving cars but the more uh the way i the way i think about it is the more pieces you add to the puzzle the more the more complex the problem becomes um so if you remember from before there was some computer vision research that was generating depth from from a single image so um i was listening to a conversation from george hotz uh who's from comma ai was saying that if we can't push a single camera to its limits so comma ai use a single camera then adding multiple cameras or adding lidar and all that sort of stuff is not going to to help us so i'm a real big fan of of embracing constraints meanwhile lidar incumbent velodine and challenger luminar both go public on the nasdaq via reverse mergers spac to compete with hardware and adas software so there's some lidar companies going uh going public supervised learning and cost of edge cases new technology approaches are needed yeah i think this is um this is what we've been discussing right improvements in supervised ml systems over time expectation so rather than following this exponential this power curve we're following an s curve here so we've seen a massive amount of ai performance but now we're sort of tapering off so rather than seeing exponential improvements in the quality of ai performance moore's law we're instead seeing exponential increases in the cost to improve ai systems supervised ml seems to follow an s-curve yeah i think this is what we're going to see going forward is more far more unsupervised and semi-supervised methods just just taking over of course we're going to squeeze supervised for as much as it's worth but going forward over the next couple of years that that to me is where where i'm seeing the trends that are heading leading companies crowdsource ideas from open source using data they've generated i like this approach too right this is leveraging the power of the internet a real good business model is to invert the internet it's what amazon have done so what they did is they were creating all these data servers uh for their own store when amazon was just an online bookstore then they realized hold on a lot of other companies have problems creating data servers all over the world so why don't we just create some data servers and then rent them out to other companies and so amazon web services was born so these companies here have open sourced their data sets and now they've made things like kaggle competitions or got people like you and me to see what kind of work they can do on these data sets and then leverage the things that the people around the world figure out for them the next step new models and a shift in focus from perception to motion prediction hmm use of ml and self-driving is still mostly limited perception with large parts of stack stack large parts of the stack hand engineered yeah this is what we were talking about before much of ml's self-driving focus on what's going around the vehicle but think about it when you drive what the self-driving car should do is mostly hand engineered making development difficult and slow yeah this is what i was thinking right if if i'm a self-driving car company and i'm hand engineering features like this is what a car looks like this is what the car should do if you see a car in front of you that's like using a lot of uh software 1.0 all right like so a lot of rule based stuff so of course you're using computer vision and machine learning to to perceive what's around but then you have to do a lot of rule based stuff to go you know what okay yep you can see that red light stop if you see a red light so a bunch of if statements to go along with your machine learning where i think going forward is and makes sense to me is something like the mu zero approach so mu zero um here we go so this is reinforcement learning sort of style now constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence where's mu0 however in real world problems the dynamics governing the environment are often complex and unknown so this is this is like a self-driving environment if we're in if we're driving along the road uh the dynamics governing that environment are often complex and unknown so you never know what's going to happen in a in a road basin of course there are some things that are going to be quite similar depending on where you're driving but what happens if a car comes out of nowhere these are the edge cases going to be really hard for supervised ml to tackle because uh a machine a supervised machine learning algorithm doesn't do too well on things that are outside of the distribution of its training set now if we go here mu0 learns a model that when applied iteratively predicts the quantities most directly relevant to planning the reward the action selection policy and the value function when evaluated on 57 atari games the canonical video game environment for testing ai techniques in which model based planning approaches have historically struggled our new algorithm achieved a new state of the art when evaluated on go chess and shogi without any knowledge of the game rules mu0 matched the superhuman performance of the alpha zero algorithm that was supplied with game rules boom so look at that without any knowledge of the game rules so what if we tilt what if we treated again i'm not an expert not an expert this is just me thinking out loud what if we treated self-driving and this is where things like tesla's uh dojo tesla dojo like a simulated uh way yeah so simulating driving what if we treated driving like a game right and applied an algorithm like mu0 instead of hand labeling all these features of what to do just your your labels could be what the human does in a certain scenario so it comes up to red light stops you don't have to hand hand craft the the feature of a red light like a bounding box around a red light just have the whole scenario as as the game world again my thinking here is just out loud if you want to see more on this sort of thinking i'd go watch all the videos uh of george hot talking about self-driving cars he knows a lot more than i do the new frontier of self-driving development is machine learning for planning ah here we go new algorithms working akin to alphago and trained on large amount of human driving demonstrations are being developed yeah okay that's where i see the future of self-driving cars the consumer first approach to self-driving tesla has hundreds of thousands of autopilot-enabled cars in the wild and consumers help it towards inch towards full self-driving yeah so this is this is tesla's mode they have the whole full stack they have the hardware of the car they have the software and they have the data oh there we go comma dot ai so these are my two favorite companies in the self-driving world i think the other ones like lyft waymo and uber are just like to me they're just burning cash like vc cash it's hard to me it's hard for a problem uh to be like fully involved in a problem when you're just burning someone else's cash but when you're a tesla or comradei and making you have profits off your customers your actual customers who are using your product then you kind of got like skin in the game if the ship goes down you go down but if you're at one of these funded companies if it goes down while it doesn't really matter to you because it was someone else's money uh ai problems like self-driving thrive on compute new providers of specialized ai compute platforms are already onto their generation 2 products well this is exciting all right because nvidia need nvidia need competition because they're they're just they own the basically the ai chip market what amd have great gpus i'm pretty sure they just released some new ones i haven't checked them out yet but um a lot of the tooling isn't there for them so i have a amd gpu in my macbook pro here but i can't use libraries like tensorflow and pi torch without a whole bunch of uh hacking away or a framework like played ml which is which is truth be told i checked out the github it's coming along whereas with a nvidia gpu i just run one one command basically and i can start writing deep learning code straight away so this is where i see if they can get the tooling right not only just the hardware but the software to run the deep learning libraries well then i'm i'm all for that see look at this there we go they're already nvidia watch out for graph core they're coming for you what's that so that's about a 12x reduction in cost for the same amount of compute 16 times faster training time for image classification model efficient net see this is what i'm talking about using using more efficient effective dedicated compute google's new tpu v4 delivers up to a 3.7 training speed up over tpu v3 sure i haven't found like have you ever tried to use a tpu sure you can connect it to colab but if you want to use a tpu on a large scale it takes quite a bit of engineering especially a little bit of fairness the video will not rest either up to 2.5 training speed ups with the new a100 gpu versus v100 wow so four times performance gains on ml perth in 1.5 years so this is where we're like it's inevitable right like to well who knows who who knows what we can predict but compute you can just if you just imagine compute is consistently going to go up and somehow because even if if one like uh section of the computing so like fabrication uh of an ai chip doesn't sort of uh improve maybe the algorithm improves so then it'll look like even though it's running off off hardware the algorithm will be able to go faster the rise of ml ops devops for ml signals an industry shift from technology r d how to build models to operations how to run models yeah so this is what i'm seeing too is um so the model like building stage of machine learning pipelines has as it's getting pretty mature so we know like okay computer vision use convolutional architecture you can use transfer learning for almost any problem you can imagine these days now it's how do we get that model that we're building pretty succinctly into the hands of others um and so yeah this is where a lot of my time is going to be going in the future is is upskilling myself on ml ops so if you'd like to see a video on that let me know as ai adoption grows regulators regulators give developers more time more to think about again making sense any kind of uh new technology should not be slowed down but it also should not be sort of i don't think it's a good thing to have just the wild wild west i've just been letting anything happen without sort of thinking about how this is going to affect people because it can be too easy for nerds like us like engineers just sitting away tapping and writing code and not thinking about what what happens to to an actual person on the other end of that code enterprises report that ai drives revenue and sales and marketing while reducing cost in supply chain management makes sense rpa and computer vision are the most common deployed techniques in the enterprise rpa is robotic process automation i don't have much experience there ai dialogue assistants are live handling calls from uk customers today poly ai we've seen them throughout this before so it's rolled out its voice assistant for hospitality in the uk wow that's pretty cool so i think like things that we see for like just booking in like appointments or or just calling a place and just getting some basic frequently asked questions like that's where that's where we want to just use like a uh a chatbot or an ai voice assistant to just be able to give someone information about a business computer vision unlocks faster accident and disaster recovery intervention oh wow that's really cool tractable ai captures and processes images imagery of the damage to automatically predict its repair costs wow so we had a potential proof of concept on that at one stage can we capture images of it looks like they've done that capturing images of damaged cars uploading it to some system and then using computer vision to estimate how much it'll cost 30 days to one week oh i really like that that's a great use case no code ml automation a universal prediction api for 360 customer data i think 360 might be like a what's it called like a black book for customers i can't remember a database of some sort for you different customers nlp is used to automate quantification of a company's environmental social and governance esg perception using the world's news oh wow so understanding nlp can derive esg perception scores by assessing the relationships and sentiments of products and companies with respect to client specific esg reputation pillars oh wow so just uh just by analysing analyzing the news about different companies you can tell how much pollution they have what's their sustainability wow that's a really cool way to sort of monitor uh if companies are staying true to their word right if if they're reporting something but the the algorithm disagrees it's like why is there such a discrepancy here machine learning protects humans from email spear phishing attacks and this is just spam detection on 2020 you know computer vision detects subtle evidence of tampered identity documents yep that makes sense use computer vision to see if uh if you've got a fake id yo ai is a key to web scale content analysis for money laundering and terrorist financing i think nlp again article collection and classification as well as entity recognition and disambiguation to support downstream risk classification of people and organizations wow uses deep learning techniques to cover up to 85 of the risk data in all key geographies wow i'm not sure what kind of like websites they'd be analyzing but i'm guessing it's yeah it's fraud detection and seeing if like if a certain website is uh is funding a terrorist organization that's a great use case machine translation unlocks financial crime classification group globally machine translation is used to generate multilingual training data for financial crime classification wow so they use so if they don't have any data for a certain type of uh what is it financial crime on a certain language they can use translation to translate i'm guessing english to another language and use that data to train on that's phenomenal but language model goes mainstream upgrading google and microsoft's bing search query understanding okay so this makes sense open source public publication so bert came out at the end of 2018 and now it's used in in large scale production within 12 months unless google already had it in production before they they publicly said that they did i think microsoft are using it as well robotic installations are achieving millions of robotic picks per month so robots going into factories well that makes sense right it's like if you're working in a factory is that the most fulfilling work that you can do get the robots to do the least fulfilling work get the humans to do the creative stuff manufacturing cnc machine programming starts to be automated so this is a another massive thing right is uh i don't think about these things because it's like we like write code and we don't well well you might make things but for me personally i write code on a machine that's already been made for me i i rarely think about how the machine got made and from what i've heard uh we all just think that yeah laptops just come out of a factory but how do you design the machine to build the laptop so the machine that builds the machine i've heard elon say like from from car manufacturing is the machine that builds the machine is way harder to design than the machine itself so the machine that builds the car is harder to design than the car itself open source model and data set sharing is driving nlp's cambrian explosion big shout out to hugging face here hugging faces transformers library in production five mil pip installs two and a half thousand community transformer models trained in over 164 languages how good is that by five 430 contributors so hugging face transformers look them up if you want to use transformers for your nlp task um they could if transformers get repurposed for computer vision hugging faces transformers for nlp could get a it could have a separate arm could be forked now we've got hugging face transformers for nlp hugging face transformers for computer vision hugging face could become the new actual open ai since open ai are basically closed ai now open source conversational ai expands its footprint across industry rasa's libraries and tools have clocked up two million downloads and have open source uh have open source 400 plus contributors so i believe rasa um and we've we've only got one minute left on the timer how many slides are we through 128 oh let's just go to the end you know what we're we're almost there we'll just keep going rasa rasa open source conversational ai this is cool build contextual assistance that really help customers wow okay so you can build your own chat bot compare plans how much is this free and open source wow if you're enterprise you pay big i like that model let's go here well done rasa private 15 mil funding rounds are above 15 mil for ai first companies remain strong despite uh in spite of covert 19. that's good to hear if you're building an ai company covert 19 isn't really impacting how your opportunity to get funding politics now again this is probably my least uh experience of uh expertise in this whole report so yeah i'm probably going to burn through this one fairly fast if you want more in depth of course you can check out the report yourself ethical risk a group of researchers have spent years helping to frame the ethical risks of deploying ml in certain sensitive contexts this year the issues went mainstream oh the perpetual lineup facial recognition is remarkably common around the world 50 of the world currently allows the use of facial recognition holy crap that's more than i thought again well i mean we all the iphone uses facial recognition now right potential risk to wrongful arrest yeah here's where it can go bad when facial recognition recognizes the wrong person two known examples of wrongful arrests may 19 and january 2020 likely just the tip of the iceberg yeah see imagine if you got i mean it's going to happen almost in any system right it's going to be a human even if a human recognizes someone they're going to have some some percentage of error the same goes for a facial recognition system it's not going to be 100 perfect you might be able to reduce the error rate but a perfect system facial recognition facebook central's class action lawsuit for 650 mil my goodness who sued them it went against illinois's biometric privacy law ah i see maximum exposure via the lawsuit was 47 billion in the end the suit is likely to net each affected user 200-400 so this is like kind of facebook's motto is just uh uh just do things and ask for permission later even if they're illegal clearview exposes what is now technically possible with facial recognition a search engine for face for faces oh great so clearview i think are uh a facial recognition system like at a an airport clear view here i think if you go to an airport instead of like scanning in you can like um scan your face and go through quicker or something like that anyway you can check that out but how they got their data set was we'll just scrape the internet of everyone's faces and now we'll just resell that those images back to law enforcement agencies scraped photos from facebook youtube venmo and millions of other websites now i mean hmm do people know that when they're putting their stuff online like i'm making this video now my face is probably in all of these these databases everywhere but the people know that their faces are now in this database and i think many people would know that facial recognition more thoughtful approaches gather steam large technology companies are taking a more careful path yep microsoft deleted 10 million faces amazon announced a one-year pause on letting police use its facial recognition system okay apple is asked by new york's mta to enable face id for passengers while they wear a mask yep a lot of people wearing mastering covered more thoughtful approaches reducing bias in data sets yep totally agree that should definitely happen we should do our best to do that even though the argument is even though data sets are bias that's just because human nature is bias well think about this is is that human nature we can also adapt to scenarios that we have to adapt to so if removing bias is a good thing then should we not do that facial recognition a new legal precedent in the uk emphasizes that facial recognition tools cannot move fast and break things haha if you're doing facial recognition do not use the facebook motto um washington state passes new law with active support from microsoft legal challenge in china oh deep fakes increased awareness of deep fakes causes a rush of activity by china and california california passed law ab-730 aimed at deep fakes which criminalizes distributing audio video that gives a false damaging impression of a politician's words or actions yeah you wanna you want these laws to happen you want these laws for deep fakes just find a prominent politician create a deep fake of them and then just send it to them and make sure they see it because once they see it they'll be like oh crap this stuff is is pretty full on and then lo and behold shock the law of our passing legislation against deep flags probably be sped up algorithm decision making regulatory pressure builds new zealand's prime minister says the first in the world to produce a set of standards for how public agencies should use algorithms to make decisions i like new zealand's prime minister what's her name i think she's doing a great job new zealand prime minister and look i don't agree with all of her decisions of course it's rare for anyone to agree with anyone's jacinta ardern but i think she seems to be the most thoughtful world leader at the moment put it that way in my view and i don't really pay too much attention to politics gpt-3 like gp22 still outputs biased predictions when prompted with topics of religion ah okay again this biases you could argue the researchers could argue gpt3 was trained on the internet so the bias is from the internet well again we can engineer ways to remove this bias from deepmind to u.s army research lab ai research agendas start to overlap holy crap three months after deep blind starcraft 2 breakthrough the us army publishes interesting starcraft results so now deepmind's research in closed mind is uh going into the u.s army that is incredible incredible while it is notable that cutting-edge resource ideas are migrating from academic and corporate research labs to military labs wow u.s army continues to make major investments into implement military ai systems of course this makes sense if we're using ai for a whole bunch of different things especially like computer vision i'm imagining a drone flying over of of where it needs to fly uh rather than have a human in that just use computer vision to understand the landscape startups at intersection of ai and defense raise large financing rounds yes i think i saw this drone before and dural it took uh took a shot at at dji america now builds the best drones because before um what's it called pj lucky is it pj lucky and dural yes so there's a blog post somewhere which talks about this drone which apparently before this drone came out the ghost four dji built the best drones in the world and of course they're china and so china and us as much as i'd love to believe they're friends they still have some sort of uh military sort of angst between each other so if dji building the best drones and drones become a thing in in military combat which i'm sure they will it'd be a good thing to know that the us have great military drones as well is u.s china competition in weakening the missile technology control regime for alphago and alpha star alpha dogfight big dog's gotta eat but this is looks like it's um deep reinforcement learning for fighter pilots holy crap the top ai developed by heron systems beta human pilot five nil what that's insane the winning ai used hyper aggressive tactics of flying very close to its opponent whilst continually firing with lower regard for the survival of its own plane ah the anonymous pilot said the standard things that we do as fighter pilots aren't working hmm well the winning ai it could use hyper aggressive tactics of flying very close to its opponent because it doesn't have consideration for its own like well-being whereas a pilot a fighter pilot uh doesn't want to fly too close to someone else because if they do they'll they'll get dusted and they'll be gone whereas an ai is just like i don't care i'm just going to fly as close to you as i want and just try to destroy you the us secretary of defense targets 2024 for real life ai versus human dog fight holy crap 2024 the ai agent's restoring victory demonstrated the ability of advanced algorithms to outperform humans and virtual dogfights these simulations will culminate in a real world competition involving full-scale tactical aircraft in 2024 we're going to have ai fighter pilots versus human fighter pilots in 2024 that's mental to me many actors attempt to define principles for responsible use of ai totally agree it's very hard problem right two of the leading ai conferences adopt new ethics codes neurops and iclr both proposed new ethical principles and expectations of researchers but no mandatory code and data sharing hmm again i i kind of understand this i would like to see of course code and data sharing but again some of the people and authors and and companies that go there um if they do share their code and data is that affecting the business of the the company they represent so if google and facebook are sponsoring their researches authors are required to provide an explicit disclosure of funding and competing interests so yeah we built this computer vision model that's the best in the world at understanding faces but we're also paid by facebook and so that improves facebook's business google is leading into fairness interpretability privacy and security of ai models people ai guidebook that's a good resource actually i checked that one out so it tells you explainability and trust data collection and valuation user needs and defining success white house extends its ban on chinese companies with ties to surveillance in xinjiang i'm pronouncing that wrong for sure i'm not i'm again i don't know too much about this huawei is increasing dominant player in the smartphones and investing heavily in machine learning technology wow huawei is taking over samsung but i'm pretty sure huawei's banned from um the us now isn't it president of huawei's consumer unit declared no chips and no supply okay foreign companies that use the u.s chip making equipment would be required to obtain a us license before supplying certain chips to huawei okay semiconductors amplify the geopolitical significance of taiwan and particularly tsmc so tsmc make like the majority of the world's like smartphone chips i'm pretty sure yeah the u.s technology industry and tm tsmc are significantly co-dependent with 60 of tm tsmc sales coming from the us tsmc said it would spend 12 billion to create a chipped fab in arizona see to me this makes sense if you're going to be creating something fundamental for your business it'd be good if you could create that close to your business so if something like what happens in the world like what happens in 2020 covert like borders shut down and stuff like that you don't have access to your supply chain it'd be good that if you didn't necessarily need to cross so much borders so if if you're in the us and your apple and you're making everything in china and you want your business to keep running it'd be good if some of what you made in china was also made at home taiwan's tsmc remains dominant in r d expenditure and semi semiconductor manufacturing so it's the only fabricator with a five nanometer nanometer manufacturing process and is now working on three nanometers holy crap chinese government sets up an additional 29 billion state-backed fund to reduce its dependency on american semiconductor technology yeah see this is what i'm seeing a lot of this what's happened in the in the world over the last year is that uh a lot of like companies and business and entities have realized hold on how much do we rely on external parties to to to drive our fundamental business and so if the chinese government is relying too much on america and something goes wrong in their relationship of course i don't want that to happen but they probably want chips to be made in their own country china hires over 100 tsmc engineers in push to close the gap in semiconductor capabilities so yeah this is what i'm i don't know enough but from what i know tsmc is like the one of the best chip manufacturers in the world us senate proposes the chips for america act makes sense half the world's advanced chips are designed in america but only 12 are manufactured there hmm sounds like that could be improved given mounting concerns over chips cross-border mma remains highly politicized the vast majority of acquisitions have been blocked so i believe mma is merge and acquisition so these ones have been blocked all these companies trying to merge and acquire but april 2020 china allows nvidia usa 6.9 billion dollar acquisition of mellanox uh israel okay so nvidia the potential acquire acquisition of arm by nvidia will be a major test of where things stand so didn't it i'm pretty sure they bought them didn't they nvidia buys arm okay there we go is this legit nvidia buys arm from softbank for 40 billion there we go well done nvidia the video said on sunday that it would require the british chip designer arm from softbank okay now again i'm not sure that's what's at new york times hopefully that's trustworthy who knows what you can trust these days ai nationalism governments increasingly plan to scrutinize acquisitions of ai companies the likely state of arm is to nvidia is questioned by many including its founder hermann hosser a leading founder and investor argues it would be bad for the uk if arm is acquired by nvidia well i think they just did according to the news aoi nationalism in the us ai budgets continue to expand federal budget for ai for non-defense ai r d one and a half billion in 2021 a major new bipartisan act is proposed ai nationalism china decentralizing policy experimentation to cities another wave of countries declare national ai strategies yep this is making sense as ai becomes more prevalent countries need to know where they stand on it u.s tax code incentivized replacing humans with robots hmm i don't know what i think on that jobs at risk of automation in the eu in 19 countries yeah see i see these things and a lot of this looks to me just like i'd have to spend a fair bit of time trying to interpret this these graphs and so what i'm thinking here is of course a lot of jobs are at uh at risk of ai automation but again people can adapt right there's there's people have the ability to adapt that's the most important point to remember is that people have the ability to adapt to new scenarios they have to believe it first themselves of course benjio hasabis um and other ai research leaders united europe's 2019 and a call to action for climate change a position paper and workshop explored various high leverage problems where ml methods can be applied tackling climate change with machine learning oh so there's a paper on that um i think these are scenarios where ml is high leverage automatic monitoring with remote sensing uh scientific discovery optimized systems new battery materials carbon capture reducing food waste oh i like that one i wonder how we could do that accelerate physical simulations climate models and energy scheduling the authors note that ml is a part of the solution it is a tool that enables other tools across fields this is why ml interests me so much again what we talked about before the information theory problem set up if you have inputs and you have some sort of desired output ml is a part of your tool set that can be the bridge between those two predictions my favorite eight predictions for the next 12 months the race to build larger language models continues and we see the first 10 trillion parameter model who's going to be who's it going to be microsoft facebook google who do you think hugging face maybe attention based neural net or open ai or deep mind closed ai attention-based neural networks move from nlp to computer vision and achieving state-of-the-art results yes i would agree with that transformer models i i don't i don't doubt that we'll see in the next sort of 12 months transformer models take over state of the art of cnns because that's just where the trends are going i'm pretty sure a major corporate ai lab shuts down as its parent company changes strategy oh interesting in response to u.s department of defense activity and investment in u.s based military ai sign-ups a wave of chinese and european defense-based ai startups so yeah okay other countries say that u.s is investing heaps in the ai military space they're like yeah we need some ai in our military space one of the leading ai first drug discovery startups uh either ipos or is acquired for over one billion so this is going to be massive like big pharma is is huge obviously and so if ai first drug discovery startups are doing good the big pharma companies will be like yep we'll take over deepmind makes a major breakthrough in structural biology and drug discovery beyond alpha fold so alpha fold is currently their model for looking at protein data facebook makes a major breakthrough in augmented and virtual reality with 3d computer vision i think this is where facebook have to head right because to me facebook as a platform is just i don't even use it like it's i like they they facebook could have like groups service and event service and just get rid of the timeline and that's all you have you have groups and you have events and i would pay for that don't let me don't don't make me watch heaps of ads just allow me to create a private group and private events and i can invite people through a link nice and simple but yeah i think that's where they're going they acquired oculus they have a whole bunch of image and whatever data with um uh instagram and facebook and whatnot so that kind of makes sense if they got a lot of smart people in the right places nvidia does not end up completing its acquisition of arm hmm well that's interesting to see i mean this was made on october 1 and that article we saw of nvidia acquiring arm was september 14 or something like that conclusion congratulations on making it to the end of the state of the ai report thanks for reading thank you for watching this video if you followed along the whole way through we're probably at two hours or something now i've talked for a long time um of course yes it is a snapshot of of progress um last year's issue was published on june 26 2019 of course this is uh not capturing everything but it is a good snapshot to sort of just tell how the field is progressing i mean it might seem the things that you do day to day might not be making large exponential gains but year on year that's that's a good time period to evaluate what's happening big shout out to the authors thank you to nathan benaj i'm probably saying that incorrectly and ian hogarth um who are both uh deep into the field of ai and investing and company creating and whatnot so thank you very much we have one more thing i will be back in a second ah you thought we'd talk about the state of ai for 2020 without an without future dan showing up well i'm here i'm always here let's be real i've got some predictions future dance 2020 or 2021 more so ai predictions now the transformer overflows traditional cnns as the best architecture for computer vision now that that that aligned with the state of ai report 2020 pi torch and tensorflow have a baby and call it pi flow or tensor torch must be real it could happen nvidia faces serious competition like actual competition from an ai chip company it could be graphcore graphcore i'm rooting for you and the video i'm reading for you as well uh competition is a good thing semi-supervised and unsupervised methods like simclr2 takeover supervised for state of the art less data sim clr2 is a framework for uh semi-supervised computer vision so check that out a self-driving car company shows state-of-the-art results with mu0 like setup so again rather than all these handcrafted features they use alpha alphago type two billion dollars plus invested into health plus ai startups i think this is where this is the year that that sort of ai and data starts really moving into healthcare and full stack ml so not just building models which is a mature which is to me becoming a mature science becomes the norm as as model architecture's designed so full stack ml getting your stuff out to the world that's where i'm going to be spending a lot of my oh actually i'm futureden so i've already spent a lot of my time there so that's that's what i can tell you for what the future's like where's uh let's go back to actual dan whoa um you didn't see that it is halloween anyway but uh that's it the state of iii report 2020 review i'll probably do another one next year leave a comment below of what your favorite thing was what are your predictions for for upcoming in 2021 what are you most excited about and as always don't forget to check it out big shout out to nathan and ian for creating it and keep learning keep creating i'll see you next time but i know for sure i speak swedish

Channel: Daniel Bourke

Views: 13,849

Rating: 4.9022403 out of 5

Keywords: what's the state of artificial intelligence, latest artificial intelligence research, state of ai report 2020, state of ai

Id: o2fYsrV-YlQ

Length: 112min 6sec (6726 seconds)

Published: Sun Nov 01 2020