Well, good afternoon everyone. Got to love the buzz
in the room here today. Welcome to the MaRS Discovery District, this wonderful complex for this very special
radical AI founders event, co-hosted by the University of Toronto. My name is Meric Gertler and it's my great privilege
to serve as president of the University of Toronto. Before we begin, I want
to acknowledge the land on which the University
of Toronto operates. For thousands of years, it
has been the traditional land of the Huron-Wendat, the Seneca, and the Mississaugas of the Credit. Today, this meeting
place is still the home to many Indigenous people
from across Turtle Island, and we are very grateful
to have the opportunity to work and to gather on this land. Well, I'm truly delighted
to welcome you all to this discussion
between Geoffrey Hinton, university professor Emeritus
at the University of Toronto, known to many as the
Godfather of Deep Learning, and Fei-Fei Li the
inaugural Sequoia Professor in Computer Science at
Stanford University, where she's Co-Director of
Human-centered AI Institute. I want to thank Radical Ventures and the other event partners
for joining with U of T to create this rare and
special opportunity. Thanks in large part to
the groundbreaking work of Professor Hinton and his colleagues, the University of Toronto
has been at the forefront of the academic AI community for decades. Deep learning is one of
the primary breakthroughs propelling the AI boom, and many of its key
developments were pioneered by Professor Hinton and
his students at U of T. This tradition of excellence, this long tradition
continues into the present. Our faculty, students and graduates, together with partners
at the Vector Institute and at universities around the world are advancing machine learning
and driving innovation. Later this fall, our
faculty, staff, students, and partners will begin
moving into phase one of the beautiful new Schwartz
Reisman Innovation campus just across the street. You may have noticed a
rather striking building at the corner with the
official opening plan for early next year. This facility will accelerate
innovation and discovery by creating Canada's largest university based innovation hub, made possible by a
generous and visionary gift from Heather Reisman and Gerry Schwartz. The innovation campus
will be a focal point for AI thought leadership, hosting both the Schwartz
Reisman Institute for Technology and Society, led by Professor Gillian Hadfield and the Vector Institute. It is already clear that
artificial intelligence and machine learning
are driving innovation and value creation across the economy. They're also transforming research in fields like drug discovery,
medical diagnostics, and the search for advanced materials. Of course, at the same time, there are growing concerns over the role that AI will play in
shaping humanity's future. So today's conversation clearly addresses a timely and important topic, and I am so pleased that
you have all joined us on this momentous occasion. So without further ado, let me now introduce today's
moderator, Jordan Jacobs. Jordan is managing partner and co-founder of Radical Ventures, a leading venture capital firm supporting AI-based
ventures here in Toronto and around the world. Earlier he co-founded Layer 6 AI and served as Co-CEO
prior to its acquisition by TD Bank Group, which he
joined as Chief AI Officer. Jordan serves as a Director
of the Canadian Institute for Advanced Research, and he was among the founders
of the Vector Institute, a concept that he dreamed up with Tomi Poutanen, Geoff Hinton, Ed Clark, and a few others. So distinguished guests, please join me in welcoming Jordan Jacobs. (audience applauding) - Come on up. Thanks very much, Meric. I wanted to start by
thanking a number of people who've helped to make this possible today, University of Toronto and Meric, Melanie Woodin, Dean of Arts and Science, and a number of partners that
have brought this to fruition. So this is the first in
our annual four part series of founder AI masterclasses
that we run at Radical. This is the third year we've done it, and today's the first one of this year. We do it in person and online. So we've got thousands of
people watching this online. So if you decide you
need to start coughing, maybe head outside. We do that in partnership
with the Vector Institute and thank them very much for their participation and support with the Alberta Machine
Intelligence Institute in Alberta, and with Stanford AI, thanks to Fei-Fei. So thank you all of you for
being excellent partners. We're hoping that this is gonna be a really interesting discussion. This is the first time
that Geoff and Fei-Fei, who I like to think of as
friends, and I get to talk to, but this is the first time they're doing this publicly together. So it's, I think, gonna be a really interesting conversation. Let me quickly do some deeper explanations of their background. Geoff is often called the Godfather of Artificial Intelligence. He's won the touring award. He is a Professor Emeritus
University of Toronto, co-founder of the Vector Institute, also mentored in a lot of the people who have gone on to be
leaders in AI globally, including at the big companies and many of the top research
labs in the world in academia. So when we say godfather, it really is, there are many kinds of children
and grandchildren of Geoff who are leading the world in AI and that all comes back to Toronto. Fei-Fei is the founding Director
of the Stanford Institute for Human-Centered AI, Professor at Stanford. She's an elected member
of the National Academy of Engineering in the US, the National Academy of Medicine in the American Academy
of Arts and Science. During a sabbatical from
Stanford in 2017/18, she stepped in for a role as
a Vice-President at Google as Chief Scientist of
AI/ML at Google Cloud. There's many, many other things
we could say about Fei-Fei but she also has an
amazing number of students who have gone on to be
leaders in the field globally. And really importantly, and so for those of you
who haven't heard yet, Fei-Fei has a book coming
out in a couple of weeks. It is called, it's coming
out on November 7th, it's called "The Worlds I See, Curiosity, Exploration, and
Discovery at the Dawn of AI." I've read it, it's fantastic. You should all go out and buy it. I'll read you the back
cover slip that Geoff wrote 'cause it's much better than
what I could say about it. So here's Geoff's description. "Fei-Fei Li was the first
computer vision researcher to truly understand the power of big data, and her work opened the
floodgates for deep learning. She delivers an urgent, clear-eyed account of the awesome potential and danger of the AI technology
that she helped to unleash. And her call for action and
collective responsibility is desperately needed at this
pivotal moment in history." So I urge you all to go
and pre-order the book and read it as soon as it comes out. With that, thanks Fei-Fei
and Geoff for joining us. - Thank you, Jordan. (audience applauding) - Okay, so I think it's
not an exaggeration to say that without these two people, the modern age of AI does not exist, certainly not in the way
that it's played out. So let's go back to what I
think is the big bang moment. AlexNet ImageNet, maybe Geoff, do you want to take us
through from your perspective that moment which is 11 years ago now? - Okay, so in 2012, two of my
very smart graduate students won a competition, a public competition, and showed that deep neural networks could do much better than
the existing technology. Now, this wouldn't have been possible without a big data set that
you could train them on. Up to that point, there
hadn't been a big data set of labeled images, and Fei-Fei was responsible
for that data set. And I'd like to start by asking Fei-Fei whether there were any problems in putting together that data set? (audience laughing) - Well, thank you Geoff,
and thank you Jordan, and thank you University Toronto for this, it's really fun to be here. So yes, the data set that
Geoff you're mentioning is called ImageNet. And I began building it 2007 and spent the next three years pretty much with my graduate students building it. And you asked me was there
a problem building it, where do I even begin? (Fei-Fei laughing) Even at the conception of this project I was told that it really was a bad idea. I was a young Assistant Professor. I remember it was my first year actually as a Assistant Professor at Princeton and for example, a very respected mentor of mine in the field, if you know the academic jargon, these are the people who will be writing my tenure evaluations, actually told me really out of their good heart that please don't do
this after I told them what this plan is back in 2007. - So that would've been Jitendra right? (audience laughing) - The advice was that, "You might have trouble
getting tenure if you do this." And then I also tried to
invite other collaborators and nobody in machine learning or AI wanted to even go close to this project, and of course no funding.
- Sorry. (audience laughing) - Just describe ImageNet to us for the people who are not
familiar with what it was. - Yeah, so ImageNet was
conceived around 2006, 2007, and the reason I conceived
ImageNet was actually twofold. One is that, and Geoff, I think we
share similar background, I was trained as a scientist, to me, doing science is
chasing after North Stars. And in the field of AI,
especially visual intelligence, for me, object recognition, the ability for computers to recognize there's
a table in the picture or there's a chair is
called object recognition, has to be a North star
problem in our field. And I feel that we need
to really put a dent in this problem. So I want to define that North Star problem, that was one aspect of ImageNet. Second aspect of ImageNet was recognizing that machine learning was
really going in circles a little bit at that time, that we were making
really intricate models without the kind of data to
drive the machine learning. Of course, in our jargon, it's really the
generalization problem, right? And I recognize that we
really need to hit a reset, and rethink about machine learning from a data driven point of view. So I wanted to go crazy and make a data set that
no one has ever seen in terms of its quantity and
diversity and everything. So ImageNet after three
years was a curated data set of internet images that's totaled 15 million images across 22,000 concepts, object category concepts. And that was the data set - Just for comparison, at the same time in Toronto we were making a data set called CIFAR-10 that had 10 different
classes and 60,000 images, and it was a lot of work, with general generously paid for by CIDAR at five cents an image. - And so you turn the data
set into a competition, just walk us through a little bit of what that meant, and then we'll kind of
fast forward to 2012. - Right. So we made the data set in 2009. We barely made it into a poster
in a academic conference. And no one paid attention. So it was a little desperate at that time. And I believe this is the way to go. And we open sourced it, but even with open source, it wasn't really picking up. So my students and I thought, well let's get a little more
drive up the competition. Let's create a competition to invite the worldwide research community to participate in this
problem of object recognition through ImageNet. So we made a ImageNet competition and the first feedback we got from our friends and
colleagues is, it's too big. And at that time you can not
fit it into a hard drive, let alone memory. So we actually created a smaller data set called the ImageNet challenge data set, which is only 1 million images across 1000 categories
instead of 22,000 category, and that was unleashed in 2010, I think. You guys noticed it in 2011, right? - Yes.
- Yeah. - And so in my lab we already
had deep neural networks working quite well for speech recognition. And then Ilya Sutskever said, "What we've got really ought to be able to win the ImageNet competition." And he tried to convince
me that we should do that. And I said, well, you know,
it's an awful lot of data. And he tried to convince
his friend Alex Krizhevsky, and Alex wasn't really interested. So Ilya actually
pre-processed all the data to put it in just the
form Alex needed it in. - You shrunk the size of the images. - Yes.
- Yeah. - He shrunk the images a bit. - Yeah, I remember. - And got it pre-processed
just right for Alex, and then Alex eventually agreed to do it. Meanwhile, in Yann
LeCun's lab in New York, Yann was desperately
trying to get his students and postdocs to work on this data set. 'Cause he said, "The first person to apply convolutional nets to
this data that's gonna win." And none of his students were interested. They were all busy doing other things. And so Alex and Ilya got on with it, and we discovered by running on the previous
year's competition that we were doing much better
than the other techniques. And so we knew we were gonna
win the 2012 competition. And then there was this political problem, which is we thought if we
showed that neural networks win this competition, the Computer Vision people, Jitendra in particular will say, well that just shows it's
not a very good data set. So we had to get them
to agree ahead of time that if we won the competition, we'd proved that neural networks worked. So actually called up Jitendra and we talked about data
sets we might run on. And my objective was to
get Jitendra to agree that if we could do ImageNet, then neural nets really worked. And after some discussion and him telling me to do other data sets, we eventually agreed, okay,
if we could do ImageNet then we'd have shown neural nets work. Jitendra remembers it
as he suggested ImageNet and he was the one who told us to do it, but it was actually a
bit the other way round. And we did it and it was amazing. We got just over half the error rate of the standard techniques. And the standard techniques
have been tuned for many years by very good researchers. - I remember standard
technique at that time, the previous year is
support vector machine with sparsification.
- Right. - That was, so you guys submitted
your competition results, I think it was late
August or early September. And I remember either
getting a phone call, or getting an email late
one evening from my students who was running this because we hold the test data we were running on the server side. The goal is that we have
to process all the entries so that we select the winners, and then by, I think it was
beginning of October that year that Computer Vision Fields
International Conference, ICCV 2012 was happening
in Florence, Italy. We already booked a workshop, annual workshop at the conference. We will be announcing the winner, it's the third year. So a couple of weeks before
we have to process the teams. Because it was the third year and frankly the previous two years results didn't excite me, and I was a nursing mother at that time. So I decided not to go to the third year, so I didn't book any tickets. I'm just like, too far from me. And then the results came in, that evening, phone call or email, I really don't remember, came in. And I remember saying to
myself, darn it Geoff, now I have to get a ticket to Italy. Because I knew that was a
very significant moment, especially with a
convolutional neural network, which I learned as a graduate student, as a classic algorithm. And of course by that time there was only middle seats economy class flying from San Francisco to Florence with a one stop layover. So it was a grueling
trip to go to Florence- - I'm sorry.
- But I wanted to be there. (audience laughing) Yeah, but you didn't come. - No (audience laughing) Well, it was a grueling trip. - But did you know that
would be a historical moment? - Yes, I did actually. - You did, and you still didn't come. But you sent Alex. - Alex, yes.
- Yeah. - Who ignored all your advice? - Who ignored my email for multiple times, 'cause I was like, Alex, this is so cool, please do this visualization,
this visualization. He ignored me. But Yann LeCun came and it was because, for those of you who have attended these academic conference workshops tend to book these smaller rooms. We booked a very small room, probably just the middle section here. And I remember Yann had to
stand in the back of the room because it was really packed, and Alex eventually showed up 'cause I was really nervous that he wasn't even gonna show up. But as you predicted at that workshop ImageNet was being attacked. At that workshop there were
people vocally attacking, this is a bad dataset. - In the room? - In the room . - During the presentation?
- In the room. - But not Jitendra, 'cause Jitendra has already
agreed that it counted. - Yeah, I don't think
Jitendra was in the room, I don't remember. But I remember it was such
a strange moment for me because as a machine learning researcher, I knew history was in the making, yet ImageNet was being attacked. It was just a very strange, it was exciting moment. And then I had to hop in the middle seat to get back to San Francisco because then the next morning. - So you mentioned a few people that I want to come back to later. So Ilya who's founder and
chief scientist at OpenAI, and Yann LeCun who subsequently went on to be head of AI at Facebook now Meta, and there's a number of other
interesting people in the mix. But before we go forward and kind of see what
that boom moment created, let's just go back for a little bit. Both of you started in this with kind of a very specific goal in mind that is an individual and
I think a iconoclastic, and you had to persevere
through the moments that you just described, but kind of throughout your careers. Can you just go back,
Geoff maybe and start, give us a background to why did you want to get
into AI in the first place? - I did psychology as an undergraduate. I didn't do very well at it. And I decided they were
never going to figure out how the mind worked unless they figured out
how the brain worked. And so I wanted to figure
out how the brain worked and I wanted to have an
actual model that worked. So you can think of
understanding the brain as building a bridge. There's experimental data and things you can learn
from experimental data, and there's things that will
do the computations you want, things that will recognize objects. And they were very different. And I think of it as you
want to build this bridge between the data and the competence, the ability to do the task. And I always saw myself as starting at the end
of things that work, but trying to make them more
and more like the brain, but still work. Other people tried to stay with things justified by empirical data, and try and have theories that might work. But we're trying to build that bridge and not many people were
trying to build a bridge. Terry Sejnowski was
trying to build a bridge from the other end, and so we got along very well. A lot of people doing, trying to do computer vision, just wanted something that worked, they didn't care about the brain. And a lot of people who
care about the brain wanted to understand how
neurons work and so on, but didn't want to think much about the nature of the computations. And I still see it as we
have to build this bridge by getting people who know about the data and people who know about
what works to connect. So my aim was always to make
things that could do vision, but do vision in the
way that people do it. - Okay, so we're gonna come back to that 'cause I want to ask you about the most recent developments and how you think that
they relate to the brain. Fei-Fei, so Geoff just to
kind of put a framework on where you started,
UK to the US to Canada, by mid to late '80, you
come to Canada in '87, along that route, funding
and interest in neural nets, and the way the approach
that you're taking kind of goes like this, but I'd say mostly like this- - It went up and down. - Fei-Fei you started your
life in a very different place. Like can you walk us through a little bit of how you came to AI? - Yeah, so I started my life in China, and when I was 15-year-old, my parents and I came to
Parsippany, New Jersey. So I became a new immigrant and where I started was first English as second language classes, 'cause I didn't speak the language, and just working in laundries, and restaurants and and so on. But I had a passion for physics. I don't know how it got into my head. And I wanted to go to Princeton because all I know was Einstein was there, and I got into Princeton, he wasn't there by the
time I got into Princeton. - You're not that old.
- Yeah. But there was a statue of him. And the one thing I learned in physics, beyond all the math and all that is really the audacity to
ask the craziest questions, like the smallest particles
of the atom world, or the boundary of space time
and beginning of universe. And along the way I discover brain as a third year Roger
Penrose and those books. Yeah, you might have opinions, but at least I've read those books. - It was probably better that you didn't. (audience laughing) - Well it at least got
me interested in brain. And by the time I was graduating I wanted to ask the most
audacious question as a scientist. And to me the absolute most
fascinating audacious question of my generation that was
2000 was intelligence. So I went to Caltech to get a dual, pretty much a dual PhD in neuroscience with Christof Koch, and in AI with Pietro Perona. So I so echo Geoff, what
you said about bridge because that five years allow me to work on computational neuroscience and look at how the mind works, as well as to work on
the computational side, and try to build that computer program that can mimic the human brain. So that's my journey,
it starts from physics. - Okay, so your journeys
intersect at ImageNet 2012. - By the way, I met Geoff when I was a graduate student.
- Right, I remember, I used to go visit Pietro's lab. - Yeah. - In fact he actually offered me a job at Caltech when I was 17. - You would've been my advisor. - No, I would not, not when I was 17. - Oh, okay. - Okay, so we intersected at ImageNet, I mean in the field everyone knows that ImageNet is this big bang moment and subsequent to that first
the big tech companies come in and basically start buying
up your students and you, and to get them into the companies. I think they were the first ones to realize the potential of this. I would like to talk
about that for a moment, but kind of fast forwarding, I think it's only now since ChatGPT that the rest of the world is catching up to the power of AI. Because finally you can play with it. You can experience it, in the boardroom they can talk about it, and then go home, and then the 10-year-old kid has just written a dinosaur essay for fifth grade with ChatGPT. So that kind of transcendent experience of everyone being able to play with it, I think has been a huge shift. But in the period in
between which is 10 years, there is kind of this
explosive growth of AI inside the big tech companies, and everyone else is not really
noticing what's going on. Can can you just talk us
through your own experience? Because you experienced a kind of a ground zero post ImageNet. - It's difficult for us
to get into the frame of everybody else not
realizing what was going on, 'cause we realized what was going on. So a lot of the universities you'd have thought would
be right at the forefront were very slow in picking up on it. So MIT for example, and Berkeley, I remember going even talking in Berkeley in I think 2013 when already AI was being very successful in Computer Vision. And afterwards a graduate
student came up to me and he said, "I've been
here like four years and this is the first talk I've heard about neural networks. They're really interesting." - Well, they should have gone to Stanford. - Probably, probably. But the same with MIT, they were rigidly against
having neural nets. And the ImageNet moment
started to wear them down and now they're big
proponents of neural nets. But it's hard to imagine now, but around 2010 or 2011 there was the Computer Vision people, very good Computer Vision people who were really adamantly
against neural nets. They were so against it that, for example, one of the main journals,
the IEEE PAM recognition- - PAM?
- PAM. Had a policy not to referee papers on neural nets at one point. Just send them back, don't referee them, it's a waste of time,
it shouldn't be in PAM. And Yann LaCun sent a
paper to a conference where he had a neural net that
was better at identifying, at doing segmentation of pedestrians than the state of the art. And it was rejected. And it was one of the
reasons it was rejected was one of the referees said, "This tells us nothing about vision." 'Cause they had this view of
how computer vision works, which is you study the nature
of the problem of vision, you formulate an algorithm
that'll solve it, you figure out how to
implement that algorithm, and then you publish a paper. In fact, it doesn't work to it - I have to defend my
field, not everybody, - Not everybody. - So there are people who are- - But most of them were
adamantly against neural nets. And then something remarkable happened after the ImageNet competition, which is, they all changed within about a year. All the people who have
been the biggest critics of neural nets started doing neural nets, much to our chagrin, and some of them did it better than us. So this (indistinct)
in Oxford, for example, made a better neural net very quickly. But they behaved like
scientists ought to behave, which is that the strong
belief this stuff was rubbish. Because of ImageNet we
could eventually show that it wasn't and then they changed. So that was very comforting. - And just to carry it forward, so what you're trying to show, you're trying to label
using the neural nets, these 15 million images accurately, you've got them all
labeled in the background so you can measure it. The error rate when you did it dropped from 26% the year before, I think to 16% or so. - Yep.
- I think it's 15.3. - Okay. And then it subsequently keeps- - 15.32. (audience laughing) - I knew you'd remember.
- Which randomization? - Geoff doesn't forget. And then in subsequent years people are using more powerful neural nets and it continues to drop to the point where it surpasses- - 2015. So there's a Canadian, very smart Canadian
undergrad who joined my lab, his name is Andrej Karpathy. And he got bored one summer and said, "I want to measure how humans do." So you should go read his blog. So he had all these like human doing image that test parties, he had to bribe them with pizza I think. with my students in the lab. And they got to a accuracy
about 5%, and that- Was it five or 3.5? - Three.
- Three. 3.5 I think. - So humans basically make mistakes about 3% of the time?
- Right, right. And then I think 2016, I
think a resonant passed it. - Yeah.
- Right, it was resonant, is that year's winning algorithm passed the human performance. - And then ultimately you
had to retire the competition because it was so much
better than humans that had- - We had to retire 'cause
we run out funding. - Okay, alright. It's a different reason. - A bad reason.
- Still run outta funding - Instantly that student started life at the University of Toronto.
- Yes. - Where he went to your lab, and then he went to be
head of research at Tesla. - Okay, first of all, he came to Stanford to be a PhD student. And yesterday night we were talking, actually there was a
breakthrough dissertation, in the middle of this. And then he became part of
the founding team of OpenAI. - But then he went to Tesla.
- And then he went to Tesla. - And then he thought better of it. - He's back. But I do want to answer your
question of that 10 years. - Well there's a couple of
developments along the way. - Right.
- Transformers. - Right. - So the transformer paper is
written, the research done, paper written inside Google, another Canadian is a
co-author there, Aidan Gomez, who's now the CEO and
co-founder of Cohere, who I think was a 20-year-old
intern at Google Brain when co-authored the paper. So there's a tradition of Canadians being involved
in these breakthroughs. But Geoff, you were at Google
when the paper was written, was there an awareness inside Google of how important this would be? - I don't think there was,
maybe the authors knew, but it took me several years to realize how important it was. And at Google people didn't realize how important it was until BERT so BERT used transformers, and BERT then became a lot better at a lot of natural language
processing benchmarks for a lot of different tasks. And that's when people realized
transformers were special. - So 2017 the transformer
paper was published. I also joined Google, and I think you and I actually met on my first week.
- Right. - I think most of 2017 and 2018 was neuro-architecture search.
- Right. - I think that was Google's bet. - Yep.
- And there was a lot of GPUs being used. So it was a different bet. - So just to explain that
neural architecture search essentially means this, you get yourself a whole lot of GPUs, and you just try lots of
different architectures to see which works best
and you automate that. It's basically automated evolution for neural net architectures. - It's like hyper parameter to new. - Yeah.
- Yeah. - And it led to some-
- Good way. - Quite big improvements.
- Yeah. - But nothing like transformers. And transformers were a huge improvement for natural language - Neural architecture search
was mostly the ImageNet. - Yeah.
- Yeah. - So I'll tell you our
experience of transformers. So we were doing our
company Layer 6 at the time, I think we saw a pre-read of the paper and we were in the middle of a fundraising and a bunch of acquisition
offers and read the paper. And I mean, not just me, but my partner told me
who had studied with you, and Maksims Volkovs who came
out of the group lab. And we thought this is the
next iteration of neural nets, we should sell the company, start a venture fund and
invest in these companies that are gonna be using transformers. So we figured it would take five years to get adopted beyond Google. And then from that moment forward, it would be 10 years for all the software in the world to get replaced or embedded with this technology. We made that decision
five years and two weeks before ChatGPT came out. So I'm glad to see we
were good at predicting, but I have to give
credit to my co-founders who I thought I understood
what the paper was, but they were able to explain it fully. - I should just correct you, I don't think Tomi ever studied with me. He wanted to come study with me, but a colleague in my department told him if he came to work with me, that would be the end of his career and he should go do something else. - So he took the classes, and this is my partner
who in the late '90s was doing a master's at U of T, and he wanted to go study with
Geoff, studied neural nets. And his girlfriend, now wife's father, who was a engineering
professor, said, "Don't do that, neural nets are a dead end." So instead he took the classes but wrote his thesis in cryptocurrency. (audience laughing) Okay, so- - Are you still gonna
talk about the 10 years? Because I think there's
something important. - Yeah, so go ahead. - So I do think there's
something important the world overlooked this 10 years
between ImageNet, AlexNet and ChatGPT. Most of the world sees
this as a tech 10 years, or we see it as a tech 10 years, in the big tech there's things brewing. I mean it took sequence
to sequence transformer, but things are brewing. But I do think for me personally and for the world, it's
also a transformation between tech to society. I actually think personally,
I grew from a scientist to a humanist in this 10 years. Because having joined
Google for that two years in the middle of the transformer papers, I begin to see the societal
implication of this technology. It was post AlphaGo moment and very quickly we got
to the AlphaFold moment. It was where bias it was creeping out, there was privacy issues. And then we're starting
to see the beginning of disinformation and misinformation. And then we're starting to see the talks of job within a small circle, not within in a big public discourse. It was when I grew personally anxious, I feel, you know 2018- Oh, oh, it was also right
after Cambridge Analytica. So that huge implication of
technology, not AI per se, but it's algorithm driven
technology on election, that's when I had to
make a personal decision of staying at Google or
come back to Stanford. And I knew the only
reason I would come back to Stanford was starting this
human-centered AI institute to really, really understand the human side of this technology. So I think this is a
very important 10 years, even though it's kind of not
in the eyes of the public, but this technology is
starting to really creep into the rest of our lives. And of course 2022, it's all shown under the daylight how profound this is. - There's an interesting footnote to what happened during
that period as well, which is ultimately you and
Ilya and Alex joined Google, but before that there was
a big Canadian company that had the opportunity to
get access to this technology. Do you want us, I've heard this story but I don't think it's
ever been shared publicly. Maybe do you want to share
that story for a second? - Okay, so the technology
that we were using for the ImageNet, we developed it in 2009 for
doing speech recognition, for doing the acoustic modeling,
bit of speech recognition. So you can take the sound wave and you can make a thing
called a spectrogram, which just tells you at each time how much energy that is at each frequency. So you're probably used
to seeing in spectrograms. And what you'd like to do
is look at a spectrogram and make guesses about which part of which phonamium is being expressed by the middle frame of the spectrogram. And two students, George Dahl and another student who I shared with Gerald Penn called Abdo, he had a longer name,
we all called him Abdo, who was a speech expert,
George was a learning expert. Over the summer of 2009, they made a model that was better than what 30 years of speech research had been able to produce, and big, big teams working
on speech research. And the model was slightly better, not as big as the ImageNet
gap, but it was better. And that model was then
ported to IBM and to Microsoft by George went to Microsoft and Abdo went to IBM, and those big speech groups
started using neural nets then. And I had a third student who'd been working on something else, called Navdeep, Navdeep Jaitly. And he wanted to take
this speech technology to a big company, but he wanted to stay in Canada for complicated visa reasons. And so we got in touch
with Blackberry, RIM, and we said we've got this new way of doing speech recognition and it works better than
the existing technology and we'd like a student to
come to you over the summer and show you how to use it, and then you can have the
best speech recognition in your cell phone. And they said after some discussions, a fairly senior gap Blackberry said, "We are not interested." So our attempt to give it
to Canadian industry failed. And so then Navdeep took it to Google, and Google were the first
to get it into a product. So in 2012, around the same time as we won the ImageNet competition, George and Abdo's speech
recognition acoustic model, the acoustic model was in, there was a lot of work
making it a good product and making it have low latency and so on, that came out in the Android. And there was a moment when
the Android suddenly became as good as Siri at speech recognition and that was a neural net. And I think for the people
high up in the big companies, that was another ingredient. They saw it get this
dramatic result for vision, but they also saw that it
was already out in a product for speech recognition was
working very well there too. So I think that combination
of it does speech, it does vision, clearly
it's gonna do everything. - We won't say anymore about Blackberry.
- It was a shame. It was a shame that
Canadian industry didn't- I think we might have
still had Blackberries if that happened. (audience laughing) - Alright, we'll leave that one there. (audience laughing) I thought it was a story,
I've heard this story before, but I thought it was important
for the rest of the world to know some of what went
on behind the scenes, why this technology didn't stay in Canada even though it was offered for free. Okay, so let's advance forward. We now have post transformers, Google is starting to use this and develop it in a
number of different ways. OpenAI, where your former
student Ilya had left Google, been a founder of OpenAI with
Elon Musk and Sam Altman, Greg Brockman and a few others. Ilya is the chief scientist, and Andrej your student as a co-founder. So they are working
together a very small team to basically take turns, well initially the idea
was we're gonna build AGI, artificial general intelligence, ultimately the transformer
paper comes out, they start to adopt at
some point transformers, and they start to make
extraordinary gains internally, they're not really sharing publicly in what they're able to do
in language understanding and a number of other things. They had efforts going on
in robotics that spun out. Pieter Abbeel ended up
spinning out Covariant, a company we subsequently
invested in and other things. But so the language part of it advances, and advances and advances. People outside OpenAI don't really know to the extent what's going on. And then ChatGPT comes out
November 30th last year. So 10 months ago.
- Well, GPT-2 caught the attention of some of us. I think actually, I think
by the time GPT-2 came out, my colleague Percy Liang,
an LP Professor at Stanford, I remember he came to me and say, "Fei-Fei I have a whole
different realization of how important this technology is." So to the credit of Percy, he immediately asked HAI to
set up a center to study this. And I don't know if this
is contentious in Toronto, Stanford is the university
that coined the term foundation models, and some people call it
LLM- large language model. But going beyond language, we
call it a foundation model. We created the center of research for foundation model before,
I think before 3.5 came out. So definitely before ChatGPT.
- Just describe what a foundation model is just for those who are not familiar. - That's actually a great question. Foundation model, some people feel it has to have transformer in it. I don't know if you use-
- No, it just has to be a very big huge amount of data. - Very large, pretrained
with huge amount of data. And I think one of the
most important thing of a foundation model
is the generalizability of multiple tasks. You're not training it for
example, machine translation. So in NLP, machine translation
is a very important task, but the kind of foundation model like GPT is able to do
machine translation, is able to do conversation, summarization, and blah blah blah. So that's a foundation model and we're seeing that
now in multimodality. We're seeing a vision, in
robotics, in video and so on. So we created that. But you're right, the
public sees this in the- - 10 months ago.
- What did you say? - October 30th.
- November, I think. - November. - One other very important
thing about foundation models, which is for a long time
in cognitive science, the general opinion was
that these neural nets, if you give 'em enough training data, they can do complicated things, but they need an awful
lot of training data. They need to see thousands of cats. And people are much more
statistically efficient. That is they can learn to do
these things on much less data. And people don't say that so much anymore because what they were
really doing was comparing what an MIT undergraduate can learn to do on the limited amount of data with what a neural net that
starts with random weights can learn to do on a
limited amount of data. - Yeah, that's an unfair comparison.
- And if you want to make a fair comparison, you take a foundation
model that is a neural net that's been trained on
lots and lots of stuff and then you give it
a completely new task, and you ask how much data does it need to learn this completely new task? And that's called few shot learning 'cause it doesn't take much. And then you discover these things are statistically efficient. That is, they compare
quite favorably with people in how much data they need
to learn to do a new task. So the old kind of innatist idea that we come with lots
of innate knowledge, and that makes us far
superior to these things, you just learn everything from data. People have pretty much
given up on that now because you take a foundation model that had no innate knowledge
but a lot of experience and then you give it a new task, it learns pretty efficiently. It doesn't need huge amounts of data. - You know, my PhD is
in one-shot learning, but it's very interesting, even in Beijing framework
you could pre-train, but it's only in the neuro network kind of pre-training really
can get you this multitask. - Right. - Okay, so this basically
gets productized in ChatGPT, the world experiences it,
which is only 10 months ago, although for some of us it feels like-
- Seems longer. - Much longer.
- It feels like forever. - Because you suddenly you have this, you had this big bang that
happened a long time ago that I think for a long time no one really saw the results of it, suddenly, I mean my comparison would be there's planets that are formed, and stars that are visible, and everyone can experience the results of what
happened 10 years before, and then transformed, etc. So the world suddenly
becomes very excited about what I think feels to a
lot of people like magic. Something that they can
touch and they can experience and gives them back a feedback in whatever way they're asking for it. Whether they're putting in text prompts and asking for an image to be created, or video, or texts, and asking for more texts to come back and answer things that you
would never be able to expect and getting those unexpected answers. So it feels a little bit like magic. My personal view is that, we've always moved the goal line in AI. AI is always the thing
that we couldn't do, it's always the magic. And as soon as we get there then we say that's not AI at all, or there's people around that
say, that's not AI at all. We move the the goal line. In this case what was your
reaction when it came out? I know part of your
reaction is you quit Google and decided to do different things, but when you first saw
it, what did you think? - Well, like Fei-Fei said, GPT-2 made a big impression on us all. And then there was a steady progression, also I'd seen things
within Google before GPT-4 and GPT-3.5 that were
just as good like PaLM So that in itself didn't
make a big effort. It was more PaLM made an
impression on me within Google 'cause PaLM could explain
why a joke was funny, and I'd always just use that as a, we'll know that it really gets it when it can explain why a joke is funny. And PaLM could do that. Not for every joke but for a lot of jokes. - And so-
- Incidentally these things are quite good now at
explaining why jokes are funny but they're terrible at telling jokes, and there's a reason which is they generate text one word at a time. So if you ask them to tell a joke, what they do is they're
trying to tell a joke. So they're gonna try and tell
stuff that sounds like a joke. So they say, a priest and
a badger went into a bar and that sounds a bit like
the beginning of a joke and they keep going telling stuff that sounds like the beginning of a joke. But then they get to the point where they need the punchline. And of course they haven't thought ahead, they haven't thought what's
going to be the punchline. They're just trying to make it sound like they lead into a joke, and then they give you a
pathetically weak punchline, 'cause they have to come
up with some punchline. So although they can explain jokes 'cause they get to see the whole joke before they say anything, they can't tell jokes, but we'll fix that. - Okay, so I was going to ask you if comedian is a job of the future or not. You think soon? - Probably not. - All right.
- So anyway- - So what was your reaction to it? And again, you've seen things behind the scenes along the way. - A couple of reaction. My first reaction is of
all people I thought I knew the power of data, and I was still old by the power of data. That was a technical reaction. I was like, darn it, I should
have made a bigger ImageNet. No, but maybe not, but that was really-
- You still could. - Funding is the problem. Yeah, so that was first. Second, when I saw the
public awakening moment to AI with ChatGBT, not just the GPT-2 technology moment, I generally thought, thank goodness we've
invested in human centered AI for the past four years. Thank goodness we have built a bridge with the policy makers,
with the public sector, with the civil society. We have not done enough, but thank goodness that that
conversation had started. We were participating it, we were leading some part of it. For example, we as a
institute at Stanford, we're leading a critical
national AI research cloud bill that is still going
through Congress right now. - [Geoff] Not right now actually. - Senate, Senate, it's by camera, so at least it's moving the senate because we predicted the societal moment for this tech. We don't know when it would come, but we knew it would come, and it was just a sense
of urgency honestly. I feel that this is the moment we really have to rise to, not only our passion as technologist, but responsibility as humanists. - And so you both, I think the common reaction
of you both has been, we have to think about both
the opportunities of this, but also the negative consequences of it. - So for me, there was
something I realized and didn't realize until very late, and what got me much more
interested in the societal impact was like Fei-Fei said, the power of data. These big chatbots have seen thousands of times more data than any
person could possibly see. And the reason they can do that is 'cause you can make thousands of copies of the same model, and each copy can look at a
different subset of the data, and they can get a gradient from that of how to change their parameters, and they can then share
all those gradients. So every copy can benefit from what all the other
copies extracted from data, and we can't do that. If suppose you had 10,000 people and they went out and they
read 10,000 different books, and after they've each read one book, all of them know what's in all the books. We could get to be very smart that way, and that's what these things are doing and so it makes them far superior to us.
- And there is education. There's some schooling that
we're trying to do that but not in the way.
- Yes. But education's just hopeless, I mean hardly worth paying for. (audience laughing) - Except University of
Toronto and Stanford. (audience laughing) - I've tried to explain to friends that Geoff has a very
sarcastic sense of humor and if you spend enough time
around it, you'll get it. But I'll leave it to you to decide whether that was sarcastic. - So the way we exchange
knowledge, roughly speaking, this is something of a simplification, but I produce a sentence and you figure out what you
have to change in your brain, so you might have said that, that is if you trust me. We can do that with these models too. If you want one neural net architecture to know what another architecture knows, which is a completely
different architecture, you can't just give it the weights. So you get one to mimic
the output of the other, that's called distillation and that's how we learn from each other. But it's very inefficient, it's limited by the
bandwidth of a sentence, which is a few hundred bits. Whereas if you have these models, these digital agents which
have a trillion parameters, each of them looks at
different bits of data and then they share the gradients, they're sharing a trillion numbers. So you are comparing an
ability to share knowledge that's in trillions of numbers with something that's hundreds of bits. They're just much, much
better than us at sharing. - So I guess Geoff that- So I agree with you at
the technology level, but it sounded like for you that's the moment that got
you feeling very negative. - That's the moment I
thought, we are history, yeah. - Yeah, I'm less negative than you. I'll explain later, but I think that's where we-
- Well let one sec actually, let's talk about that. Explain why you are optimistic and let's understand why
you are more pessimistic. - I'm pessimistic 'cause the
pessimists are usually right. (audience laughing) - I thought I was a pessimist too. We have this conversation. So I don't know if I should
be called an optimist. I think I'm- Look when you came to a
country when you're 15 now speaking a single bit of
language and starting from $0, there's something very
pragmatic in my thinking. I think technology, our human relationship with
technology is a lot messier than an academia typically would predict, 'cause we come to academia
in the ivory tower, we want to make a discovery, we want to build a piece of technology, but we tend to be purist. But when the technology
like AI hit the ground and reach the societal level, it is inevitably messily
entangled with what humans do. And this is where maybe
you call it optimism is my sense of humanity. I believe in humanity. I believe in the, not only
the resilience of humanity, but also of collective will, the arc of history is dicey sometimes. But if we do the right thing, we have a chance, we
have a fighting chance of creating a future that's better. So what I really feel is not delusional optimism at this point, is actually a sense of
urgency of responsibility. And one thing Geoff, I think, I really hope you do feel positive is you look at the students
of this generation, in my class I teach a 600 undergrad class every spring on introduction of deep
learning and computer vision. This generation compared to even five years ago is so different. They walk into our class not only wanting to learn
deep learning transformers, gen AI, they want to talk about ethics, they want to talk about policy, they want to understand privacy and bias. And I think that really is where I see that the humanity rising to the occasion. And I think it's fragile. I mean look at what's going on
in the world, in Washington, it's very fragile, but I think if we recognize
this moment, there's hope. - So I see the same thing. - Oh good.
- I don't teach undergraduates anymore, but I see it in sort of
more junior faculty members. - Yeah.
- So at the University of Toronto for example, two of the most brilliant young professors went off to anthropic
to work on alignment. Roger Grosse is coming back again, I hope. And AI, for example, is now
full-time working on alignment. - Yes.
- So there really is a huge shift now, and I think
I'm unlikely to have ideas that will help solve this problem, but I can encourage these
younger people around 40. - Thank you. - To work on these ideas and they really are working on 'em now, they're taking it seriously. - Yeah, as long as we put
the most brilliant minds, like many of you, I'm looking
in the audience and online, onto this problem, this is where my hope comes from. - So Geoff, you left Google in large part to be able to go and
talk about this in freely in the way that you wanted to. And basically -
- Actually, that's not really true, that's the media story and it sounds good. I left Google 'cause I was old and tired and wanted to retire and watch Netflix. (audience laughing) And I happened to have the
opportunity at that time to say some things I'd been
thinking about responsibility, and not have to worry about how Google would respond. So it's more like that.
- If we have time we'll come back to the
Netflix recommendation. - I was going to say. - But in the meantime, but you did go out, and start speaking pretty significantly- - Yes.
- In the media. I think you've both spoken
to probably more politicians in the last eight months
than in your lives before, from presidents and prime ministers, right through congress, parliament, etc. Geoff can you explain
what your concern was, what you were trying to
accomplish in voicing it, and whether you think
that has been effective? - Yeah, so people talk about AI risk, but there's a whole
bunch of different risks. So there's a risk that
it will take jobs away and not create as many jobs. And so we'll have a whole
underclass of unemployed people. And we need to worry hard about that because the increasing productivity AI is going to cause is not going to get shared with the people who lose the jobs. Rich people are going to get richer and poor people are going to get poorer. And even if you have basic income, that's not going to solve the
problem of human dignity of many people want to have a job to feel they're doing something important, including academics. And so that's one problem. Then there's a problem of fake news, which is a quite different problem. Then there's a problem of battle robots, that's a quite different problem again. All the big defense departments want to make battle robots, and nobody's going to stop them and it's going to be horrible. And maybe eventually after we've had some
wars with battle robots, we'll get something like
the Geneva Conventions like we did with chemical weapons. It wasn't until after they were used that people could do something about it. Then there's the existential risk. And the existential risk
is what I'm worried about. And the existential risk is
that humanity gets wiped out because we've developed a
better form of intelligence that decides to take control, and if it gets to be much smarter than us. So there's a lot of hypotheses here. It's a time of huge uncertainty. You shouldn't take anything
I say too seriously. So if we make something
much smarter than us because these digital intelligences
can share much better, so can learn much more, we will inevitably get those smart things to create sub goals. So if you want 'em to do something in order to do that they'll figure out, well, you have to do something else first. Like if you want to go to Europe, you have to get to the airport. That's a sub goal. So they will make sub-goals and there's a very obvious sub-goal, which is if you want to get
anything done, get more power. If you get more control, it's going to be easier to do things. And so anything that has the ability to create sub-goals will create the sub-goal
of getting more control. And if things much more
intelligent than us want to get control, they will, we won't be able to stop them. So we somehow have to figure out how we stop them ever
wanting to get control. And there's some hope. These things didn't evolve, they're not nasty competitive things. They're however we make
them, they're immortal. So with a digital intelligence, you just store the weight somewhere and you can always run it
again on other hardware. So they really, we've actually discovered
the secretive immortality. The only problem is it's not for us, we are mortal. But these other things are immortal. And that might make them much nicer 'cause they're not worried about dying and they don't have to sort of- - Like Greek gods. - Well, they're very like Greek gods, and I have to say something
that Elon Musk told me. This is Elon Musk's belief that, yes, we are the kind of bootloader for digital intelligence. We are this relatively
dumb form of intelligence that is just smart enough
to create computers and AI, and that's going to be a much
smarter form of intelligence. And Elon Musk thinks it'll keep us around 'cause the world will be more interesting with people in it than without. Which seems like a very thin thread to hang your future from. But it relates to what Fei-Fei is said, it's very like the Greek god's model, that the gods have people
around to have fun with. - Okay, can I comment on that? - Yes. (audience laughing) - Nothing you said was controversial. - Yeah, no, not at all. So I want to bucket your four concerns, economy, labor, disinformation
and weaponization, and then the extinction Greek gods- - (indistinct) discrimination and bias. - Okay, so I want to bucket
them in two buckets. The Greek god extinction
is the extinction bucket, everything else I would call catastrophic. - [Geoff] Yeah, merely catastrophic. - Catastrophic danger. And I want to comment on this, I think that one thing I really
feel is my responsibility as someone in the AI
system, the ecosystem, is making sure we are not
talking hyperbolically, especially with public policy makers. The extinction risk is,
Geoff, with all due respect, is a really interesting thought process that academia and think
tanks should be working on. - That's what I thought for many years, I thought it was a long
way off in the future and having philosophers and academics working on it was great. I think it's much more urgent. - It might, but this process
is not just machines alone. Humans are in this messy process. So I think there is a lot of nuance. For example, we talk about nuclear. I know nuclear is much more narrow, but if you think about nuclear, it's not just the theory of
fusion or fission or whatever. It's really obtaining
uraniums or plutonium, the system engineering,
the talents and all that. I'm sure you watch the
movie "Oppenheimer." So here, if we're going towards that way, I think we have a fighting
chances more than fighting because we are human society. We're going to put guardrails,
we're going to work together. I don't want to paint the picture that tomorrow we're going to
have all these robots, especially in a robotic
form, in physical form creating the machine overlords. I really think we need
to be careful in this, but I don't disagree with you that this is something we
need to be thinking about. So this is the extinction bucket. The catastrophic risk bucket, I think it's much more real. I think we need the smartest people and the more the merrier to work on. So just just to comment
on each one of them, weaponization, this is really real. I completely agree with you. We need international partnership, we need potential treaties, we need to understand the parameters. And this is humanity's, as much as I'm optimistic about humanity, I'm also pessimistic about
our self-destruction ability as well as the destroying each other. So we've gotta get people working on this, and our friend Stuart Russell, and many of the even AI
experts are talking about this. And second bucket you talk
about is disinformation. This is again, I mean 2024, everybody's watching the US election and how AI will play out. And, I think we have to get
on the social media issue, we have to get on the
disinformation issue. Technically I'm seeing more work now. Digital authentication technically is actually a very
active area of research. I think we need to invest in this. I know Adobe is, I know
academia is, I think we need to, I hope there's startups
actually in this space looking at digital authentication. But we need also policy. And then jobs. I cannot agree more. I actually, you use
the most important work that I think it's really at
the heart of our AI debate is human dignity. Human dignity is just beyond
how much money you make, how many hours you work. I actually think if we do this right, we're going to move from labor
economy to dignity economy in the sense that humans with the help of machines
and collaboratively will be making money because of passion, and
personalization, and expertise rather than just those jobs that are really grueling and grinding. And this is also why
human, HAI at Stanford has a founding principle
of human augmentation. We see this in healthcare, one of the biggest
earliest day of ChatGPT. I've got a doctor friend
from Stanford Hospital who walked to me and said, "Fei-Fei I want to thank you for ChatGPT." I said, I didn't do anything. But he said that we are using medical summarization tool from GPT because this is a huge
burden on our doctors, it's taking time away from patients. But because of this, I get more time. And this is a perfect example, and we're going to see this more. We might even see this
in the blue collar labor. So we have a chance to make this right. I would add another concern
in the catastrophic concern is actually you talk
about power imbalance. One of the power imbalance
I'm seeing right now, and it's exacerbating that as a huge speed is leaving public sector out. I don't know about Canada, not a single university in the US today can train a ChatGPT in
terms of the compute power. And I think combining
all universities of US, GPT-A100 or H100, probably nobody has it, but A100 cannot train a ChatGPT. But this is where we
still have unique data for curing cancer, for
fighting climate change, for economics and legal studies. We need to invest in public sector. If we don't do it now, we're going to fail an entire generation, and we're going to leave that power imbalance in such a dangerous way. So I do agree with you. I think we've got so
many catastrophic risks and we need to get on this. This is why we need to
work with policy makers and civil society. So I don't know if I'm saying
this in an optimistic tone or in a pessimistic, some were pessimistic to myself now, but I do think there's a
lot of work to do this. - Well, optimistically,
since you've both been very vocal about this over
the last six, eight months, there has been a huge shift
both as Geoff as you said, key researchers going and
focusing on these issues and then public and policy shifting in a way that governments are
actually taking it seriously. So I mean, you are
advising the White House and US government, you've spoken to them as well, and you've sat with the prime minister or multiple prime ministers maybe, and they're listening right, in a way that they
wouldn't have necessarily 10 months ago, 12 months ago. Are you optimistic about the
direction that that is going? - I'm optimistic that
people have understood that there's this whole bunch of problems, both the catastrophic risks
and the existential risk. And I agree with Fei-Fei completely, the catastrophic risks are more urgent. In particular 2024 is very urgent. I am quite optimistic that
people are listening now yes. - Yes, I agree. I think they're listening. But I do want to say, first of all, who are you listening from? Again, I see a asymmetry between public sector and private sector, and even private sector
who are you listening from? It shouldn't just be big
tech and celebrity startups, there is a lot of agriculture sector, education sector. And second is then after all this noise, what is a good policy? We talk about regulation
versus no regulation, and I actually don't
know where Canada sits, there's always America
innovates and Europe regulates. Where's Canada?
- Probably in between. - Okay, good, good for you. So I actually think we need
both incentivization policy, building public sector, unlocking the power of data. We have so much data that
is locked in our government, whether it's forest fire
data, wildlife data, traffic data, the climate data, and that's incentivization. And then there's good regulation, for example, we're very vocal about, you have to be so careful in regulating, where do you regulate
up upstream, downstream? One of the most urgent
regulation point to me is where rubber meets the road, is when technology is now in the form of a product or service. It's going to meet people,
whether it's through medicine, food, financial services, transportation. And then you've got
these current framework, they're not far from perfect, so we need to empower
these existing framework and update them rather than wasting time and possibly making the wrong decision of creating entirely
new regulatory framework when we have the existing ones. - Okay, so we are almost out of time for the discussion part, but we're going to have a
long session of Q and A. Before we started though, I'll ask two last questions. One is, I mean, our
view is this technology is going to impact virtually everything, and some of the positive
impacts are extraordinary. It is going to help cure
diseases like cancer, and diabetes and others. It's going to help mitigate climate change. There's just an enormous number of things, invent new materials. I see over here someone
who's focused on that, that can help in the energy
sector, and aerospace, and pharmaceuticals. And that's a big effort
at University of Toronto. But there's this entire
world of new things that could not be done
before that now can be done. So it's basically advancing science in a way that was part of, either fiction or imagination before it. Are you optimistic about that part of it? - I think we're both very
optimistic about that. I think we both believe it's
going to have a huge impact on almost every field. - So I think for those in this room who are actually studying, it's an incredibly exciting
moment to be coming into it because there's the opportunity to get involved in limiting the negatives, the negative consequences, but also to participate in
creating all those opportunities to solve some of the problems
that they've been with us as long as we've been around as a species. So there's, I think, at
least from our perspective, this really is one of the
most extraordinary moments in human history. I hope that those of you who
are embarking on your career is actually go out and go after
the most ambitious things. You can also work on like
optimizing advertising and other things, or making more Netflix
shows, which is great. But also-
- Geoff would like that. - Yes. So would my mom, who I
think has exhausted Netflix. If there's a Turkish or
Korean show out there, she's seen the very last episode of all. But for those of you who
are embarking the career, my recommendation is, try and think of the
biggest possible challenge and what you could use this
technology to help solve that is incredibly ambitious. And you have both done that, and kind of fought against
barriers all the way along to achieve that. There's a room full of people and a lot of people online, and others who will see this subsequently, I think, who are at the beginning stages of making those decisions. I think, I'm guessing
you would encourage them to do that too, right,
think as big as possible, and go after the biggest,
hardest challenges. - Absolutely. I mean, embrace this, but I also would encourage, this is a new chapter of this technology. Even if you see yourself as a technologist and a scientist, don't forget there's
also a humanist in you because you need both to make this positive change for the world. - Okay, last question and then we'll get into Q
and A from the audience. Are we at a point where
these machines have understanding and intelligence? - Wow, that's a last question. How many hours do we have? - Yes. - Okay, I'll come back to the yes. (audience laughing) - No. (audience laughing) (audience applauding) - Okay, we have questions
from the audience. Let's start on the far side. Do you want to stand up and
you're going to be given a mic. - Hi, thanks, my name's Ellie. This was awesome and thank you so much, Geoff your work really
inspired me as a U of T student to study cognitive science, and it's just amazing to
hear both of you speak. I have a question, you mentioned the
challenges for education, and for enabling universities to empower students to use
this technology and learn. And you also mentioned Fei-Fei
like the opportunity for this to become a dignity economy and empower people to just
focus on personalization, and passion, and their expertise. I'm wondering if either of you have a perspective on the challenge that could emerge with overuse
and over reliance on AI, especially for kids and students as they're on their education career, and they need to be building skills, and using their brain, and exercising the meat
sack in their head. Our our brains don't just continue to work and not accrue cobwebs
if they're not learning. And yeah, I wonder your
thoughts on burnout and over-reliance, and just what happens around de-skilling and the ability to learn to paint when you can use stable diffusion, or learn to write like Shakespeare when you can have ChatGPT do it for you, and then as those systems progress and can accrue greater insights and more complex problem solving, how that impacts our
ability to do the same? - So I have one very
little thought about that, which is when pocket
calculators first came out, people said, "Kids will
forget how to do arithmetic." And that didn't turn out
to be a major problem. I think kids probably did
forget how to do arithmetic, but they got pocket calculators. But it's maybe not a very good analogy because pocket calculators
weren't smarter than them. Kids could forget doing arithmetic and go off and do real math. But with this stuff, I don't know. For myself, I found it's actually made me much more curious about the world. 'Cause I couldn't better go to a library and spend half an hour
finding the relevant book and look something up, and now I can just ask ChatGPT anything and it'll tell me the answer, and I'll believe it, which maybe isn't the right thing to do. But it's actually made me
more curious about the world 'cause I can get the answers more quickly. - [Ellie] But you've had
years to learn what to ask. - Well, you have to-
- Yeah, but normally I ask questions about plumbing and things like that. So, well the (indistinct). - So I'll answer this
with a very quick story. I don't know about you guys, ever since I've become Stanford professor, I'm always so curious, there's a mysterious
office in the university, which is the office of college admission. To me, they're the most mysterious people and I never know where
they are, who they are, where they sit till I got a
phone call earlier this year. And of course they wanted
to talk to me about ChatGPT and college admission. And of course the question is related to, do we allow this in the
application process? And now that there is
ChatGPT how to do admission? So I went home and I was
talking to my 11-year-old, I said, well I got this phone call and there's this college
admission question, what do we do with ChatGPT and students? What if a students wrote
the best application, using ChatGPT and blah blah blah? And then I said, what would you do? I asked my 11-year-old and he said, "Let me think about it." He actually went back and slept on this, or I don't know what happened. And the next day, the next day in the morning,
he said, "I have an answer." I said, what's your answer? He said, "I think Stanford should admit the top 2000 students who knows how to use ChatGPT the most." It was actually, at the beginning I thought
that was such a silly answer. Like, it's actually a
really interesting answer, is kids already are seeing this as a tool and they're seeing their
relationship with this tool as a enabling empowering tool. Clearly my 11-year-old had
no idea how to measure that, what that means and blah blah blah. But I think that's how we
should see it in education. And we should update our education. We cannot shut the tool
outside of our education, like what Geoff said,
we need to embrace it, and educate humans so that they know how to use
the tool to their benefits.Q - I've incidentally I've met
Fei-Fei's 11-year-old son. He might be the president of
Stanford by the time he's 18. (audience laughing) - If Stanford still exist. - Maybe let's go to this side of the room in the far corner. - Yeah. I want to ask about like, so we have really good
foundational models right now, but in many of the applications we need kind of a real time
performance of the model. So like how do you see this area of research going in the future of using the abilities of
this expert foundation models to train fast, smaller models? That's the question.
- Maybe you should answer this question. - I'll leave it to you. - Well you're talking
about the inference, right? We need to start thinking
about the performance, the inference, and also
fit the model on devices depending on which, well, I mean, without getting
into the technical details, all these research as well as, like even outside of
research it's happening, you want to talk about? Okay, you don't want to
talk about, okay, okay. It's happening but I
mean it'll take a while. But yeah- - We talk about things, he invests. - That's true. - I can't talk about it until the company says that
it's okay to talk about it. - Okay, let's go back in the middle. Just right here. - Thank you. Yeah, Hi, my name is Ariel, I'm a third year Engineering Science student, majored in machine
learning at U of T as well, and then that conversation
was pretty great, and then thank you Prof.
Hinton and Prof. Li. I just have a question that maybe a lot of
undergrad or grad students are interested in this where in this room? So just like in your 20s, like what drove you to
be like a researcher, and what drove you into
the area of academia in AI? Because I'm kind of
like confused right now, like should I continue with like industry or a direct entry PhD, or like take a master and
then go back to industry. And I have like one more question that usually what do you look for? Like if I apply for a direct
entry PhD to your lab, is that like GPA, or publication,
or recommendation letters? Could you just like
elaborate a bit more on that? Thank you.
- I think there are about 300 people in the room, and about 6,000 online who want to ask that question to you Fei-Fei. - You want to start? Your 20s. - Oh, I got interested in how the brain works
when I was a teenager, 'cause I had a very smart friend at school who came into school one day
and talked about holograms, and how maybe memories in the
brain were like holograms. And I basically said, what's a hologram? And ever since then I've been interested in how the brain works. So that was just luckily having a very smart friend at school. - I'm going to be very shamelessly, if you read my book that's actually what the
book is about is that- (audience laughing) - It's a very good book. - Yeah, thank you. No, seriously. I actually, I told Jordan and Geoff, there's so many AI books about technology and when I started writing
this book about AI technology, I want to write a journey,
especially to the young people, especially to the young
people of all walks of life. Not just a certain look. And that book talks about
the journey of a young girl, and in different settings realizing or coming to
understand her own dream and realizing her dream. And it's not very different
from what Geoff said, it starts with a passion. It really did start with a passion. A passion against all other voices. The passion might come from a friend, it might come from a movie you see, it might come from a book you read, or it might come from the
best subject in school that you felt most fun, whatever it is. And in the students I hire, I look for that passion. I look for ambition, a healthy ambition of
wanting to make a change, not wanting to get a degree per se. And of course technically speaking, I look for good technical background, not just test scores. But honestly, I would have
never got into my own lab. So the standard today is so high. So by the time you apply for a PhD, or a graduate school program, you probably have some track record, some, it doesn't have to necessarily, of course if it's Geoff's student, I'll take them without
even asking question. But even if you, and I'm saying this not
only to U of T student to every student online, you can have a very different background. You can come from an
underprivileged background. What I look for is not where you are, but the journey you take, that track record shows
the journey you take, shows your passion and conviction. - Having read the book, I will say that it is a
very surprising journey, I think to most people, who will read it. And just a plug, if you're in
Canada, go buy it at Indigo, you can go to indigo.ca and
order to pre-order the book. But I think that people will be surprised, and really enjoy reading and understanding that experience, and you'll get a very
good understanding kind answering that question. - Thank you. - Okay, there's about 50 hands up. All right, let's go over
here right in the corner. - Hey, thank you for the great talk. My name's Chelav, I'm at
Vector Institute. So I think benchmarks are very important. Benchmarks are like questions. ImageNet was basically a question and then people are trying
to answer it with models. And so right now LLMs
are very hard to evaluate and generalist agents that
take actions are even, it's so hard to start thinking
about how to evaluate those. So my question's about, my
question is about questions. It's about these benchmarks. So two things. One, if you sat down
with GPT-5, GPT-6, GPT-7 and you had five minutes to play with it, what questions would you
ask that would tell you this is the next
generation of these models? And the second is more of
a comprehensive benchmark, what is the more comprehensive
not five minutes benchmark that we need in order to evaluate LLMs, or generalist agents? You can choose which one you want to, I guess think about or answer. Thank you. - Thank you for your question. It's a very good question. I will answer a different question that's just vaguely related. So this issue arose with GPT-4. How do you tell whether it's smart, and in particular, I was talking to someone called Hector Levesque who used to be a faculty
member in computer science, and has beliefs that are almost the diametric opposite of mine but is extremely intellectually honest. And so he was kind of
amazed that GPT-4 worked and he wanted to know how
it could possibly work. And so we spent time talking about that. And then I got him to give
me some questions to ask it and he gave me a series
of questions to ask it so we could decide whether it understood. So the question was does
it really understand what it's saying, or is it just using some fancy statistics to predict the next word? One comment about that is, the only way you can predict
the next word really well is to predict what the person is to understand what the person said. So you have to understand
in order to predict, but you can predict quite
well without understanding. So does GPT-4 really understand? So a question Hector came up with was "The rooms in my house are painted white, or yellow, or blue. I want all the rooms to be
white, what should I do?" And I knew it would be able to do that. So I made the question more difficult. So I said the rooms in my
house are painted white, or yellow, or blue, yellow paint fades to white within a year, in two years time I'd like
all the rooms to be white. What should I do? And ChatGPT- Oh, and I said, and why? If you say and why, it'll
give you the explanation. ChatGPT just salted it said, you should paint the blue rooms white. It said you don't need to
worry about the yellow rooms 'cause they'll fade to white. It turns out it's very
sensitive to the wording, if you don't use fade, but you use change, I got a complaint from somebody who said, I tried it and it didn't work. And they use change instead of fade. And the point is, if we understand fade to mean change color and stay changed. But if you say change, it will change color but
it might change back. So it doesn't give the same answer if you change rather than fade. It's very sensitive to the wording. But that convinced me it
really did understand. And there's other things it's done. So there's a nice question that
people came up with recently that many chat bots don't get right and some people don't get right, but GPT-4 gets right, which is, so you see I'm answering the question, does GPT-4 understand, which is does have some relation
to what you asked, right? - [Chelav] Yeah, yeah, precisely. - So the question goes like this, Sally has three brothers, each of her brothers has two sisters, how many sisters does Sally have? And most chatbots get that wrong. - What about humans? - Well, I just gave a
fireside chat in Las Vegas and the interviewer asked
me for an example of things that chatbots got it wrong. So I gave him this example
and he said, "Six," and that was kind of embarrassing. - We won't ask his name, I'm just kidding. - No, so people get it wrong. - Yeah. - But I don't see how
you can get that right without being able to do a
certain amount of reasoning. It's gotta sort of build a model. - Yeah.
- And Andrew Ying has these examples where playing Othello, even if you just give it strings as input, it builds a model of the board internally. So I think they really do understand. - And to take that a step further is that understanding cross
the line into intelligence? - Oh no.
- You said yes. - Yeah, I mean I accept the
Turing test for intelligence. People only started
rejecting the Turing test when we passed it. - So that's the moving goal
line that I was talking about. Okay, do you want to answer- - I want to quickly answer, first of all also applaud you for asking such a good question. I'm going to answer in addition to Geoff's 'cause I think what Geoff
is trying to push is really how do we assess the
fundamental intelligence level of these big models. But there are a couple
of other dimensions. One is, again, Stanford
HAO's center for research of foundation model is creating these evaluation metrics. You are probably reading the papers by Percy Helm and all that. I think also this technology
is getting so deep that some of the benchmark is more messier than what you think
the ImageNet benchmark, for example, in collaboration
with government now, for example, NIST, the US the National
Institute for Standard, what's the T?
- Technology. - And the technology,
testing or something. We need to start benchmarking against societally relevant issues, not just core fundamental capability. One more thing, I want to open
your aperture a little bit, is that beyond the LLMs there are so many technology
towards the future of AI that we actually haven't built
a good benchmarks for yet. I mean, again, my lab is doing some of the robotic learning one, Google just released the paper yesterday on robotic learning. So there is a lot more research
coming up in this space. - [Chelav] Thank you. - Okay, I know we have a
lot of questions online. I'm going to maybe take
another few in the room and then maybe someone from Radical could read out a question
or two from online. Okay in the room, let's go for one that's not too far away from the last one here. Just right here. Okay. (indistinct chattering) Here's the mic coming. - Hello, I'm Vishaam, and I'm a graduate student
at University of Guelph and I'm doing my thesis
in AI and agriculture. So building upon something you mentioned that universities don't
have enough funding to train kind of foundation models. So same question, I want to work in AI and agriculture. I'm passionate about it, but I don't have enough
resources to do that. I might think of a very good architecture, but I can't train it. So maybe I can go to industry then pitch them the idea, then I don't have the
control over the idea. I don't know how they're going to apply it. So do you have some advice on how to handle the situation? - Do a start up.
- If you can get- - Do a startup, that's
what we're here for. Oh, sorry, I'll let you answer. - If you can get your hands on an open source foundation model, you can fine tune one of those models with much less resources than
it took to build the model. So universities can still do
fine tuning of those models. - That's a very pragmatic answer for now. But this is where we
have been really talking to the higher education leaders as well as policy makers
invest in public sector. We've gotta have national research cloud, I don't know if Canada has
national research cloud, but we're pushing the US, we need to bring in the
researchers like you to be able to access the
national research cloud. But you do have an advantage
by not being a company is that you have more opportunity to get your hands on unique data sets, data sets, especially for public good and play up that card. You could work with government agencies, work communities and whatever because public sector still has the trust and take advantage of that. But for now, yes, fine
tune on open source models. - [Vishaam] Thank you. Thank you so much. - Okay, we're going to
take a couple questions. We have thousands of
people watching online watch parties at Stanford and elsewhere, so let's see if we can get a question from some people online. Leah's going to ask this question on behalf of someone online. By the way she's done an
enormous amount of work to make this happen
along with Aaron Brindle, so thank you both. - Thank you Jordan. (audience applauding) All right, thank you. So we do have hundreds
of AI researchers online and they're folks who are
building AI first companies. And so the first most upvoted question was from Ben Saunders or Saunders. He's currently CEO of an AI startup. And his colleague was actually student of Geoffrey Hinton's in 2008. And he has asked about
building responsibly and a lot of these
questions have to do about building responsibly, and they're thinking about what measures can help them as teams be proper stewards for good versus bad, and what it actually
means to be a steward? - Great question. So responsible AI framework, there's a lot of framework, and I think somebody has estimated, a few years ago there
were like 300 framework from state nation, state
all the way to corporate. I think it's really
important for every company to build a responsible framework. There is a lot you can borrow, even Radical is is making one, and create the value
framework that you believe in and recognize that AI product is a system. So from the upstream,
defining problem, dataset, data integrity, how you build models, the deployment, and create a multi-stakeholder ecosystem or multi-stakeholder, whatever team to help you to build this responsible framework and also create partnerships. Partnerships with public
sector like academia, like us, partnership with the civil society that worries about different dimensions from privacy to bias to this. So really try to take, both have a point of view as a company, but also be part of the ecosystem and partners with people
who have this knowledge. So that's my current suggestion. - I'll add to- Do you want to? - No, that was a much better
answer than I could've given. - I'll just add a little bit. I think to Fei-Fei's point, by working with people who
are interested in this, I think there are people
in the investment community who are thinking and leading on this, in our case Radical, we've written into every single term sheet an obligation for the company
to adopt responsible AI. Initially when we did that, some of the lawyers who
read it and were like, what is this? And tried to cross it out,
but we put it back in. But we're also, we've been working on a responsible AI investing framework that we are going to
release pretty broadly. And we've done this in partnership with a number of different
organizations around the world. We've met with 7,000 AI
companies in the last four years and I think we've invested in about 40. So we've seen a lot and tried to build a framework that others can use going forward. And we'll open source it so we can develop it and make it better. But I think there's a lot that
individual companies can do by just reaching out to others who are thinking in a like-minded way. Do you want to ask another question? - Yeah, great, there's so many questions, so we'll only get to a
couple of them unfortunately. But playing off of that, a lot of these questions have to do with the relationship with industry, considering how big of a role industry, and the private sector's now
playing in model development. And some folks are even asking, should researchers and
different engineering roles also be taking management courses today? - Sure. - I have to tell you a story
of when I was at Google, I managed a small group and we got reports every six months from the people who worked for us. And one of the reports I got was, "Geoff is very nice to work for, but he might benefit from
taking a management course, but then he wouldn't be Geoff." (audience laughing) That's how I feel about
management courses. (audience laughing) (audience applauding) - I don't have a better story than that. (audience laughing) - We have about a minute and a half left, so maybe let's do one more
in the room if we can. Let's see. Do you want take? Yeah, no, beside you. Sorry. All right, hopefully ask quickly and then we'll get a quick answer. - Thank you. And it's
a pleasure to be here. Good to see you Fei-Fei. My name's Elizabeth I work at Cohere. So my question is, from a private sector perspective, we work with everybody to
take NLP large language models to the broader society, on the specific public sectors and research institutions, universities who has a lot
of talent, a lot of data. What is the best way to find the mutual kind of beneficial relationship that we can contribute
and they can contribute? Thank you. - Give them some money. (audience laughing) - Thank you. (audience applauding) - Or H100s, we'll take H100. But look, it's very important. I advocate for public sector investment, but I also actually probably more so advocate for partnership. We need government, private sector, and public sector to work together. So the past four years at Stanford HAI, this is one of the main
things we have done is create an industry ecosystem. And there's a lot of
details we can talk offline, but if I'm talking to university leaders or higher education, is that I think we need to embrace that. We need to embrace that responsibly. Some people will have
different ways of calling it, but I think this
ecosystem is so important. Both sides are important. Create that partnership, be the responsible partner for each other. And resource is a big thing. We would appreciate that. - [Elizabeth] Thank you. - Okay, with that we're
exactly out of time. I want to thank you both. I feel very privileged always to be able to call you both friends, and Fei-Fei you're a partner, and Geoff, you're an investor, and have these conversations
privately with you. So it's great to get you both together and let other people hear
what you have to say. So thank you both so much for doing this. Hopefully it was as informative
for you as it was for me. (audience applauding) And we'll turn it over to Melanie Woodin, Dean of Arts and Science at U of T. - Thank you so much Jordan. So Geoff and Fei-Fei and Jordan on behalf of everyone in the
room tonight here in MaRS, and the thousands joining us online, we are deeply grateful for such a profound
conversation this evening. I can say, and I think many of us know that being part of a university community offers a never ending set of opportunities for engaging conversations and lectures. And as Dean of a Faculty
of Arts and Science, I have the pleasure of
attending many of them. But I can say without reservation that tonight's conversation
was truly unparalleled. And of course this conversation
couldn't be more timely. Geoff, when you shared your concerns with the world about the
threats of super intelligence, we all listened and we
all did what we could to try and understand this complex issue. Whether it's reading opinion pieces, watching your video, or reading long for journalism, we really tried to understand
what you were telling us. So to hear directly from
you and from Fei-Fei who spent so many years now leading the way in human-centered AI is really, truly powerful. So with that, thank you both and thank you everyone here
for attending this afternoon, and big thanks to Radical Ventures and the other partners
that make tonight possible. And so with that, the talk has concluded and we invite those of you
that are here with us in person to join us out in the foyer
for some light refreshments. Thanks for joining us. (audience applauding) (audience indistinctly chattering)
