So there's a lot of public interest in this
recently and it feels like hype. Is this the same, or is this something where
we can see that this is a real foundation for future application development? We are living in very exciting times with
machine learning. The speed of ML model development will really
actually increase. But you won't get to that end state that we
want in the next coming years unless we actually make these models more
accessible to everybody. Swami Sivasubramanian oversees database, analytics
and machine learning at AWS. For the past 15 years, he has helped lead
the way on AI and ML in the industry. Swami’s teams have a strong track record
of taking new technologies and turning these into viable tools. Today, Generative AI is dominating news feeds
and conversations. Consumers are interacting with it and brands
are trying to understand how to best harness its potential for their customers. So, I sat down with Swami to better understand
the broad landscape of this technology. Swami, we go back a long time. Tell me a bit.
Do you remember your first day at Amazon? I still remember because it's not very common
for PhD students to join Amazon at that time because you were known as retailer or ecommerce. We were building things. And so that's also quite a departure from
a foreign academic. Definitely, for a PhD student to go from
thinking to, actually, how do I build this? So you brought actually DynamoDB to the world
and quite a few other databases since then, but under your purview now is also AI and
machine learning. So tell me a bit about how does your world
of AI look like? After building a bunch of these databases
and analytics services, I got fascinated by AI because that literally
AI and machine learning because that puts data to work. And if you look at machine learning technology
itself broadly, it's not necessarily new. In fact, some of the first papers of deep
learning was written even like 30 years ago. But even in those papers, they explicitly
called out for it to get large scale adoption. It required massive amount of compute and
massive amount of data to actually succeed. And that's what cloud got us to actually unlock
the power of deep learning technologies. So which led me to early on this is like six,
seven years ago to start the machine Learning organization because we wanted to take machine
learning, especially deep learning style technologies, from not just in the hands of scientists to
everyday developers. If you think about the early days of Amazon,
the retailer with similarities and recommendations and things like that, were they the same algorithms
that we're seeing being used today or is that, I mean that's a long time ago, 30 years. Machine learning has really gone through huge
growth in actually the complexity of the algorithms and applicability of the use cases. Early on the algorithms were a lot more simple,
a lot more like linear algorithms based or gradient boosting. If you see last decade, it was all around
like deep learning early part of last decade, which was essentially a step up in the ability
for neural nets to actually understand and learn from the patterns, which is effectively
what all the image based image processing algorithms come from. And then also personalization with different
types of neural nets and so forth. And that's what led to the invention like
Alexa, which has a remarkable accuracy compared to others. So the neural nets and deep learning has really
been a step up. And the next big step up is what is happening
today in machine learning. So a lot of the talk these days is around
generative AI, large language models, foundation models. Tell me a bit why is that different from,
let's say the more task based like vision algorithms and things like that? I mean, if you take a step back and look at what's - How this foundation models -
large language models - is all about These are big models which are trained with hundreds of millions of parameters if not billion A parameter, just to give context, is like an
internal variable where the ML algorithm has learned from its data set. Now, to give a sense, what is this
big thing suddenly that has happened? Few things - One, if you take a look at Transformers, has
been a big change. Transformer is a kind of neural net technology
that is remarkably scalable than the previous versions like RNNS or various others. So what does this mean? Why did this suddenly lead to this transformation? Because it is actually scalable and you can
train them a lot faster now you can throw a lot of hardware and lot of data. Now that means now I can actually crawl the
entire World Wide Web and actually feed it into these kind of algorithms and start actually
building models that can actually understand human knowledge. At a high level, a generative AI text model
is good at using natural language processing to analyze text and predict the next word
that comes in a sequence of words. By paying attention to certain words or phrases
in the input, these models can infer context. And they can use that context to find the
words that have the highest probability of following the words that came before it. Structuring inputs as instructions with relevant
context can prompt a model to generate answers for language understanding, knowledge,
and composition Foundation Models are also capable of what
is called “in-context learning,” which is what happens when you include a handful
of demonstration examples as part of a prompt to improve
the model’s output on the fly. We supply examples to further explain
the instruction And this helps the model adjust the output based
on the pattern and style in the examples. When the models use billions of parameters
and their training corpus is the entire internet, the results can be remarkable. The training is unsupervised and task agnostic. And the mountains of web data used for training
let it respond to natural language instructions for many different tasks. So the task based models that we had before
and that we were already really good at, could you build them based on these foundation
models? You no longer need these task specific models
or do we still need them? The way to think about it is the need for
task based specific models are not going away. But what essentially is how we go about building
them. You still need a model to translate from one
language to another or to generate code and so forth. But how easy now you can build them is essentially
a big change because with foundation models, which are the entire corpus of knowledge of,
let's say huge amount of data, now it is simply a matter of actually building on top of this
with fine tuning, with specific examples. Think about if you're running like a recruiting
firm as an example and you want to ingest all your resumes and store it in a format that
is standard for you to search and index on, instead of building a custom NLP model
to do all that. Now using foundation models and give a few
examples of here is an input resume in this format and here is the output resume. Now you can even fine tune these models
by just giving few specific examples and then you essentially are good to go. So in the past, most of the work went into
probably labeling the data and that was also the hardest part because that drives the accuracy. Exactly. So in this particular case, with these foundation
models, no longer labeling is needed? Essentially, I mean, yes and no. As always with these things, there is a nuance. But majority of what makes these large scale
models remarkable is they actually can be trained on a lot of unlabeled data. You actually go through what I call as a pretraining
phase, which is essentially you collect data sets from, let's say, the World Wide Web,
like common crawl data, or code data and various other data sites, Wikipedia, whatnot. And then you don't even label them, you kind
of feed them as it is. But you have to of course go through Sanitization
step in terms of making sure you cleanse data from PII or actually all other stuff like
negative things or HP and whatnot. But then you actually start training on large
number of hardware clusters because these models to train them can take tens of millions
of dollars to actually go through that training. And then you actually finally you get a notion
of a model and then you go through the next step of what is called inference. When it comes to building these LLMs, the
easy part is the training. The hardest part is the data. Training models with poor data quality will
lead to poor results. You’ll need to filter out bias, hate speech,
and toxicity. You’ll need to make sure that the data is
free of PII or sensitive data. You’ll need to make sure your data is deduplicated,
balanced, and doesn’t lead to oversampling. Because the whole process can be so expensive
and requires access large amounts of compute and storage, many companies feel lost on where
to even start. Let's speak object detection in video that
would be as a smaller model than what we see now with the foundation models. What's the cost of running a model like that? Because now these models with these hundreds
of billions of parameters are probably very large pieces of data. That's a great question because there is so
much talk only happening around training these models, but very little talk on the cost of
running these models to make predictions, which is inference, which is a signal that
very few people are actually deploying it and runtime for actual production. Or once they actually deploy in production,
they will realize oh no, these models are very expensive to run and that is where few
important techniques actually really come into play. So one, once you build these large models
to run them in production, you need to do a few things to make them affordable to run
at cost, run at scale and run actually very in an economical fashion. One is what we call as quantization. The other one is what I call as distillation,
which is that you have these large teacher models and even though they are trained hundreds
of billions of models, they kind of are distilled to a smaller fine grained model and speaking
in a super abstract term, but that is the essence of these models. Of course, there’s a lot that goes into
training the model, but what about inference? It turns out that the sheer size of these
models can make inference expensive to run. To reduce model size, we can do “quantization,”
which is approximating a neural network by using smaller, 8-bit integers instead of 32-
or 16-bit floating point numbers. We can also use “distillation”, which
is effectively a transferring of knowledge from a larger “teacher” model to a smaller
and faster “student” model. These techniques have reduced the model size
significantly for us, while providing similar accuracy and improved latency. So we do have this custom hardware to help
out with this that happens at I mean, normally this is all GPU based, which are expensive
energy hungry beasts. Tell us what we can do with custom silicon
that makes it so much cheaper both in terms of cost as well as in,
let's say, your carbon footprint of the energy. When it comes to custom silicon, as mentioned,
the cost is becoming a big issue in these foundation models because they are very
expensive to train and very expensive also to run at scale. You can actually run like build a playground
and test your chatbot and at low scale and it may not be that big a deal, but once you
start deploying at scale as part of your core business operation,
then these things add up. So since in AWS we did invest in our custom
silicones for training with Trainium and with Inferentia with inference. And all these things are like ways for us
to actually understand the essence of which operators are making are involved in making
these prediction decisions and optimizing them at the core silicon level and software
stack level. I mean, if cost is also a reflection of energy
used because in essence, that's what you're paying for, you can also see that they are,
from a sustainability point of view, much more important than running it on general
purpose GPUs. So there's a lot of public interest in this
recently and it feels like hype. Is this the same or is this something where
we can see that this is a real foundation for future application development? First of all, we are living in very exciting
times with machine learning. I have probably said this now every year.
But this year is even more special because these large language models and foundation
models truly can actually enable so many use cases where people don't have to staff as
separable teams to go build task specific models. The speed of ML model development will really
actually increase. But you won't get to that end state that we
want in the next coming years unless we actually make these models more accessible to everybody. And this is what we did with SageMaker early
on with machine learning and that's what we need to do with Bedrock and all its applications
as well. But we do think while the Hype cycle will
subside like with any technology, but these are going to become a core part of every application
in the coming years. And they will be done in a grounded way, but
in a responsible fashion too, because there is a lot more stuff that people need to think
through in a generative AI context. Because what kind of data did it learn from
to actually what response does it generate? How truthful it is as well? These are stuff we are excited to actually
help our customers. So when you say that this is the most exciting
time in machine learning, what are you going to say next year? Well, Swami, thank you for talking to me. I mean, you educated me quite a bit on what
the current state of the field is. So I'm very grateful for that. My pleasure. Thanks again for having me, sir. I'm excited to see how builders use this technology and continue to push the possibilities forward. I want to say thanks to Swami.
His insights and understanding of the space are a great way to begin this conversation. I'm looking forward to diving even deeper
and exploring the architectures behind some of this. And how large models can be used by engineers
and developers To create meaningful experiences.
