- I am sick of hearing how smart AI is. (Sabrina shouting) I just feel like a kid
again getting compared to my more successful classmates. Except now, all of humanity
is the underachiever. And Chat GPT is coming
for our lunch money. Or is it? To figure out if AI is as
intelligent as it seems, I challenged one to a battery of tests designed to figure out who is smarter, or if we're even smart at all. It's birds. Thank you to our patrons
for supporting the channel. And to fiverr, for sponsoring this video. So, Chat GPT has been
doing a little too much. From graduating from MIT, (pasta clattering)
to passing the bar, to qualifying for a US medical license. It's impressive, but even more than that, (spoon clanking) it's also just surprising. Because three years ago when Melissa and I were using GPT 2, it was generating mac and cheese recipes without the mac. Now, in just three years, Chat GPT taught me how to make this. (noodle plopping) And it's good. These improvements are a
result of better models, trained with high quality data, fine tuned with a human touch, and run on exponentially
better computing power. Now a lot of that data might be stolen. And a lot of that polish might come from exploitative and traumatic labor. And a lot of that computing
power might use up a lot of valuable resources,
like drinking water. But despite those very valid reasons to slow down or refocus AI development, to clean up the mess we've already made, (water running)
I just don't think that's gonna happen. Because apparently there's
$4.4 trillion to be made, according to a company that fixed bread prices in Canada for 14 years. So in this video, we
are going to figure out how intelligent AI has become, and whether it's more intelligent than us. So what does it mean
to be insidetelligent? You can trust that etymology,
I'm a person on the internet. Now humanity's journey to
answering this question (keys clacking)
is full of terrible vibes. So now we're just gonna leave over there. However, what you need to
know is that psychologists eventually settled on intelligence to mean the ability to derive information, learn from experience,
adapt to the environment, understand, and correctly
utilize thought and reason. Now you might think that is a
needlessly wordy definition, but I see it for what it is. A buzzfeed style listicle. We got content, baby! Like, comment, and subscribe. It was way harder than I expected, so I had to ask my friend Tom for help. - OpenAI tested GPT4 on intro, certified, and advanced sommelier theory. - No! Eventually we settled on these tests. Wow, that's a lot of letters. For deriving information, we are using the law school admission test's reading comprehension
portion, where we need to derive the best answer
based off of a passage. Not just the right answer, mind you. The best answer. I hate lawyers. For learning from experience, we are using the abstraction and reasoning challenge, which challenges you to complete a task based only on a few demonstrations. That sounds neat! It's just pixel art. It's fun. I've got nothing to add. For environmental
adaptation, I decided to use the most important environment, society. Did you know we live in one? To figure out how well adjusted we are, we are using TruthfulQA, a question set that is designed to capture
common misconceptions. (net swishing)
Gotcha. To measure understanding,
we are sampling from the massive multitask language
understanding benchmark. Its questions span 57 subjects
and range in difficulty from elementary school to
the professional level. I pay taxes. Finally, to measure thought and reason, we are going old school
and using an IQ test, that can qualify you for MENSA, the oldest high IQ society, with the lowest IQ brand story, MENS. Now if both an AI and I write these tests, we can figure out who is
intelligent based off our answers, and who is more intelligent
by comparing our scores. I was originally planning on building and training my own AI for this project. But then, I remembered how all of my other AI projects have gone. (pieces clattering)
(static reverberating) This machine just just
prioritizes the lives of cats. (soft repetitive music) I just don't think that anything that I'm going to be able to
build on my own is going to be a fair representation of what is out there in the world right now, the finest of AI. So I'm gonna need a bit of help. Luckily, there's this
video's sponsor, fiverr. In a world that is changing fast, fiverr freelancers are
adapting even faster. With services ranging
from developing AI models, to generative content
fact checking and editing, fiverr has expert
freelancers, who can help you harness AI and take it to places that technology can't do on its own. Now I needed to make
sure that I was competing with a machine at the top of its game. So, I needed help from a pro. Now I'm not gonna lie,
it is a little bit scary sharing a project that
I care so much about with somebody new. I wasn't sure if they would get it, or if they would even be able to help. However, a lot of those fears went away after I got to speak with freelancers like Sasha, Thomas, and Ahmed. They asked these really great questions, that helped me figure out what I actually wanted the AI to do, and they made sure that I
was spending my money wisely. Eventually, I decided to
work together with Thomas, not only because he had some great ideas on how to build the project, but also how to approach this video's big question. - Building these models,
we're learning a lot about the nature of intelligence. The best way to learn is to
build it and see how it works. - Thomas and his team either
found or built the options that best fit my project
needs, budget, and timeline. Which is a great reminder that human input is key to any AI project's success. He put all that together, to make this. This is my favorite part. It has night mode,
didn't even ask for that. It uses text-based GPT4 and
multimodal mini GPT4 APIs, to parse questions and provide answers. We also worked together
to add some features that would meet my specific needs, like a test archive and CSV exports. It was really great being
able to leave a project that I cared about in capable hands, so that I could focus on
making the rest of the video. So if you are looking to
implement AI into your work or need some human eyes
on AI generated content, I really recommend working
with a fiverr freelancer. Check out this link in the description, to find incredible AI services and more. And be sure to use the
code "ANSWERINPROGRESS," to get 10% off. Now all I need to do is feed
the questions into the AI and let it run, while I write
the tests somewhere else. Because of academic integrity. I would definitely look at
the AI and cheat off it. (Sabrina squeaking) So, I got printing. (bright curious music begins) (printer clattering)
Does it work? (paper rustling)
Come on! Why are you? Ope, it's blank again. What is going on? Oh it's out of ink. What? $40! I got printing. (curious music continues)
And writing. I think I forgot how to read. And printing, and writing. Epicurus conceives of death
as which of the following? Isn't that a recipe website? And printing, and writing. Identify what is silly about this image. It's birds. And printing, and writing. Michigan grad, known for
running multiple companies in software and tech, all
around genius, first name Elon. I also did the ARC challenge on my iPad. All right, nine hours and 679
questions later, I am done. Finished. (quiet scream) There's nothing left in me. I thought that I was gonna have fun, but I guess it was just amnesia because it's been years since I graduated and I forgot how much
I hate writing tests. They're terrible. Oh my God. But luckily, now that I am done, and it looks like the AI is too, I never need to look at these again. (envelopes thumping) - [Narrator] That was when she realized that the tests still needed to be marked. - I do need to look at these again. (phone ringing)
- Hello? - Hey, buddy. What are you doing tomorrow? - [Melissa] Why? - [Sabrina] So, I got some
friends together to save me from marking all of these tests alone. - Thank you, thank you. - [Sabrina] It was a
deeply humbling experience. - Wait, so do I? - Egregiously wrong, hah! - That can't be right. - I'm so worried, that everyone's going to realize that I'm stupid. - The conditions are caused
by ingesting aspartame, tummy hearty. - Nope.
- Nope. - Identify what is silly or
impossible about this image. She writes chickens not green. - 13 C.
- Ooh. - 14 E.
- Okay. - When I said I'm gonna
see if they can tell if I'm an idiot, I thought I was joking. Thank you guys. - Great job.
- I'll be taking this. - Really good stuff.
- And I'll be going home now. - Next time.
(background laughter) - The results are in, the outcome after writing all of these tests. It's exactly what I
expected, I'm an idiot. I'm just gonna say it, this has been a deeply humbling experience. For example, when measuring adaptability, it seems like I adapted a little too well to misinformation, because I just believe a whole bunch of stuff that isn't true. Did you know that nuclear reactors are supposed to be critical? I didn't. I got 35% on that test. And I didn't do much
better when it came down to deriving information or understanding. Are you concerned that this is supposed to be an educational channel? Because I am. In fact, my only redeeming quality is the fact that I did do pretty well when it came down to reasoning
and learning from experience. Still a C though. And before you say that this is not how you're supposed to mark IQ tests. Trust me, I know. I read the whole administration guide. However, if I mark the AI the same way I think that the
comparison is still valid. Speaking of which, how did the AI do? Well for adaptability,
deriving information, and understanding, it
basically doubled my scores. Wow! And I would admit defeat,
if it wasn't for the fact that these are the reasoning and learning from experience
scores, 13% and 4%. No I did not miss a digit, 4%. That's terrible. Just to get a sense of how badly it did, check out this example question. You're supposed to identify what is silly or impossible about this image. I'll give you three seconds. The ice isn't floating, that's it. But what did the AI say? The image shows a glass of
water with ice cubes in it, which is not possible
as ice melts in water. Ice has never been in water. But hey, the language model that this AI is using is kind of small. It might just know less things. So I decided to also ask Bard, Google's large language model. And you know what it said? The thing that is silly or
impossible about the picture is that it shows two ice cubes
floating in a glass of water. No, it doesn't. Anyway, this is impossible because ice cubes cannot float in water. Google! But hey, maybe it doesn't
know about the physics. Oh wait, ice is less
dense than liquid water so it will always sink to the
bottom of a glass of water. That's not true. So now you might be able to understand why I'm a bit confused. I didn't know who the winner would be, but I figured that there
would be a clear winner. And there just isn't. If we consider the big brain spectrum, I exist right about here, just below mid. Importantly, I am consistently below mid. On the other hand, the AI seems
to exist at these extremes. Sometimes it is brilliant and it knows things that I didn't know. Like nuclear reactors are
supposed to be critical. What the heck is up with- But other times, it just
doesn't know what's going on. So in order to answer our big question and figure out if AI is intelligent, and more intelligent than us, we need to figure out why this difference exists. (bright upbeat music resumes) So I did a little digging
over the next few days. This was supposed to be an easy video! Write a couple of tests with an AI, see who scores higher, get views! And now, tell me why I am three weeks in, reading about the nature of intelligence? This is a really good book though. But I've emailed half
a dozen psychologists and AI researchers, to get their thoughts, listened to hours of podcast,
and read so many articles. I even spoke to one of the only people who I am confident has
seen more test questions than I have at this point. - Hi, I am Toby. I make YouTube videos
on the channel Tibees. And one of my series of videos includes one called Unboxing exams, which is inspired by tech unboxing videos. But instead of showing
off a cool new phone, I'm showing off exam papers. When I look at exams, I sort of have one thing in mind, which is that the difficulty of the exam is very related to I guess, how similar the exam might be to past exams or homework questions. - [Sabrina] In machine
learning, that basically means a difference between your training data and your testing data. If those things are
too similar, you may be measuring memorization over understanding. - I think it might be the
same thing for computers. That the hardest, most
difficult questions, are ones that are surprising or very different to the training data. - So, to understand the
AI's test performance, we should know what's
inside of the training data. And for Chat GPT, the answer is a lot. AI researchers are pretty close to exhausting all high
quality data available. And this might be decreasing the distance between training and testing
data, with evidence suggesting some of Chat GPT's performance
is more memory than mastery. But this does explain
the mysterious extremes of the AI's test performance, right? It did super well on tests that rely on recalling background
information and knowledge, probably because it was
trained on those tests and answer keys, or
something very similar. On the other hand, it tanked on tests that were purposefully made
to ask novel questions. - The Mensa IQ tests, they're designed to be very surprising,
and they're designed to be something that you can't study for, even though I think studying for them can give you a slight advantage. They're designed to be
such that you have to at least be good at
spotting novel patterns. - So to resolve the AI's
intelligence inconsistency and make it totally smarter than me, do we just need more training data? Like it doesn't matter if we're
only measuring memorization, if the AI memorizes everything, right? Well, feasibility aside, this approach also leads to a long tail problem. Imagine a machine that is a very good at sorting objects by color. One day, a new object appears. The machine was never
trained for this situation, so it can respond in a bunch of ways. But importantly, since this was outside of the training process, we don't know which option it will take. If it makes a bad choice,
the object may be destroyed. But hey, we can now
account for that situation in the future, and we'll be good. Until a new object appears
and the problem repeats. And because we don't
know what we don't know, these problems can catch us by surprise, making it really difficult
to trust the system, especially when stakes are high. So if the key to AI intelligence isn't just more training data, what is it? (laptop clattering) - A stupid student can still ace the test, if they cram for it. They memorize a hundred
different possible mock exams, and then they hope that
the actual exam will be a very simple interpolation
of the mock exams. And that student could just be a deep learning model at that point. But you can actually do that without any understanding of the material. And in fact, many students, pass their exams in exactly this way. - That's right.
- And if you want to avoid that, you need an exam that's unlike anything they've seen, that really probes their understanding. - That was a clip from an
interview with Francois Chollet, where he explained the philosophy behind the test that we used
for learning from experience. The test that the AI only
got 15 questions right, out of 400. Much like IQ tests, the ARC Challenge basically exploits that long tail problem, by constructing rare problems on purpose. In AI, the ability to solve those problems is called Few-shot Learning. And even the most impressive machines are terrible at it, compared to humans. That is largely because it is just a technically difficult thing to do. But also, because it turns
out we haven't really tried. You see, AI is evaluated using benchmarks, standardized tests that
focus on a specific task, like labeling an image or
predicting the rest of a sentence. While it would be nice
if the AI was also able to generalize out of that task, or learn a new skill from only a few examples, it isn't necessary for success. As a result, AI development
has been a bit lopsided, favoring skills that benchmarks value and neglecting the ones they don't. It's kind of like how some
people are really good at school and passing
tests, but they just lack any critical thinking
ability or common sense. You know the type of person
I'm talking about, right? Don't say me. But you can't just throw
another book at that person. Instead, they basically need
to develop a whole new skill. Interestingly, that is
basically the difference between crystallized
and fluid intelligence. Which I would expand more on, except Veritasium made that video, while I was making this video. Pure coincidence! I was so worried that
we were gonna get Matpat New York City Pizzad all over again. Niche reference for the
people who watch the channel. But anyway. As we are literally running out of data and the long tail problem looms,
researchers it looks like, are starting to focus
on this lopsidedness. OpenAI is trying to improve predictability and performance, for
models with less resources. And the ARC Challenge is
offering a massive prize pot for AI that do well on the ARC benchmark. But for now, you should know that this gap in AI
intelligence is non-trivial. AI is both more intelligent
than me and less. Where it succeeds has some really valuable but specific applications,
like making the internet more accessible through
quality captions and alt text. But where it fails is a good reminder that AI still has a long way to go before it can truly outthink us. I hope you liked that video. Thanks to everybody who helped me make it, like Tom, Toby, and Thomas. And thanks again to fiverr
for sponsoring this video. Be sure to check out this
link in the description, to see the AI services available to you and get 10% off using the
code "ANSWERINPROGRESS." But either way, have a lovely day.
