(music) Have you ever wanted to know how large language models
work when you connect them to the data in your organization? At Microsoft, we recently demonstrated Microsoft 365 Copilot, which transforms how we work by leveraging large language models that interact with your
organizational data. Copilot works alongside you. For example, in Word, Copilot can easily write
an entirely new document, like a business proposal using content from your existing files. Or in Outlook, based on
the content you select, Copilot can compose your
email replies for you. In PowerPoint, you can
transform your written content into a visually beautiful presentation with the click of a button. In Teams, Copilot can
generate meeting summaries with discussed follow-up actions. Or while using Business
Chat in Microsoft Teams, it can help you catch up on
something you may have missed, bringing together information
from multiple sources to bring you up to speed. If you're wondering how
large language models know what they know in these scenarios, let me break down the mechanics of what makes this possible, and how the process respects your privacy, and keeps your data safe
with Microsoft 365 Copilot. First, let's look at where
large language models, or LLMs, get their knowledge. LLMs are trained on massive
amounts of public data, including books, articles, and websites to learn language, context and meaning. You can interact with
large language models using natural language with
what's called a prompt. A prompt is typically a
statement or question. When you ask a question in the prompt, the LLM generates a response based on its public data training and understanding of context, which can come in part from
how you phrase your prompt. For example, you might
give it more details to generate a response. As you continue to ask
questions and get responses, the large language model is temporarily getting more context. Your full conversation gets sent with each subsequent prompt, so the LLM can generate relevant responses as you chat with it. It's processing natural language and referring to its knowledge like we would in conversation. A key difference is that it
only remembers the conversation while it's in that conversation. The chat history is wiped clean
with each new conversation. And it won't use the knowledge
from your conversations and interactions to train the model. That said, you can also write your prompt to include additional information, which the large language
model will refer to as it generates its response. This is how you can give the
LLM a little more knowledge it might need to answer your question. I'll show you how this works using Microsoft Bing Chat's
GPT-enabled public service that has no affiliation with
your organization's data. First, I'll ask it a
completely random question that it can't answer, "What
color shirt am I wearing today?" And it responds intelligently. It knows what a shirt
is but it can't see me to answer my question so
it responds accordingly, which is an accurate response. Let me ask the question again, this time including some
additional information in my prompt. I'll describe my outfit. Now you can see it responds using the information I gave it, which is more in line with
what I was looking for. And now that it has the context, I can keep asking it
related questions like, "What color shoes?" Again, that's because the prompt builds with each interaction. And to prove that the large language model doesn't retain the information, I'll start a new chat
session and ask it again, "What color shirt am I wearing today?" And now it again says, "I can't
see you, so I don't know." It knew what shirt I was wearing before only because I temporarily provided that additional limited information. In this new session,
it no longer has access to what I said before, and I
never told it my shirt color, so it doesn't know. So how does this work then in the context of Microsoft 365 Copilot? In my previous example using Bing Chat, I provided the prompt more
information and context to give the LLM what it needed to generate the right response. This is what the Microsoft
365 Copilot system does automatically for you as you interact across different app experiences. To do this, Copilot has
several core components. First off, are the large language models hosted in the Microsoft Cloud
via the Azure OpenAI service. To be clear, Copilot is not calling the public OpenAI service
that powers ChatGPT. Microsoft 365 Copilot uses
its own private instances of the large language models. Next, Microsoft 365 Copilot has a powerful orchestration engine that I'll explain in a moment. Copilot capabilities are surfaced in and work with Microsoft 365 apps. Microsoft Search is used
for information retrieval to feed prompts, like I
did in the example before where information I provided in my prompt was used to help generate an answer. Then the Microsoft Graph, which has long been
foundational to Microsoft 365, includes additional information
about the relationships and activities over your
organization's data. The Copilot system respects
per user access permissions to any content and Graph
information it retrieves. This is important because
Microsoft 365 Copilot will only generate responses
based on information Now let's go back to the
example you saw earlier in Microsoft Teams where a user asked, "Did anything happen
yesterday with Fabrikam?" Copilot didn't just send that
question or prompt directly to the large language model. Instead, Copilot knew that
it needed more knowledge and context, so using clues
from the user's question, like Fabrikam, it inferred
that it needed to search for content sources private
to the organization. The Copilot orchestrator
searched the Microsoft Graph for activities, ensuring it
respected the user's permissions and access to information,
in this case, the user Kat. It found the email thread
from Mona that Kat received, activities in the Project Checklist and March planning presentation, which are files that Kat had access to, as well as the sharing action
where the final contract was sent to Fabrikam for review, again, where Kat would have
been part of the share activity. And Copilot cited each
source of information so Kat could easily validate the response. These are all individual steps that Kat could have done manually, like searching her inbox
for emails from Mona looking at recent project
file activities in SharePoint or reading the sharing notifications sent to Fabrikam for the contract. Copilot removed the tediousness of performing these steps manually and formulated a natural easy-to-follow and concise response in a single step. So that's how Business
Chat with Copilot works. Now, in the examples I showed you earlier, you also saw how Microsoft 365 Copilot can help save you time in the apps you're working
in by generating content. In fact, let's go back to
the Copilot and Word example to explain how that worked. Microsoft 365 Copilot can
help generate a draft proposal by using content you've been
working on, for example, in OneNote or other documents
that you have access to, like Word or PowerPoint files. Here we combine the large
language model's training on how a proposal document is structured and written with Microsoft
365 Copilot orchestration, which scans and takes relevant inputs from additional documents you've selected, adding the information to the prompt. The LLM is then able to generate an entirely new proposal document with the additional
information from those files, providing a first draft that
you can use to save time and quickly get started. And just like with the
Business Chat example, the important thing to remember here is that the enterprise data used to generate informed responses
is only present as part of a prompt to the large language model. These prompts are not retained by the large language models
nor used to train them, and all retrieved information is based on the individual data access and permissions you have
while using Copilot. So hopefully that explains
how Copilot capabilities in Microsoft 365 work. For more information on how Microsoft operates its AI services, check out aka.ms/MicrosoftResponsibleAI. Please keep checking back
to Microsoft Mechanics for the latest in tech updates,
and thanks for watching. (gentle music)
