Bring your data to Copilot for Microsoft 365 with .NET plugins and Azure AI Search

[Music] [Applause] welcome back everyone to on.net I'm your host James Monto Magna and today we're going to be talking about how we can extend co-pilot for M365 with the teams toolkit and.net this is a super awesome awesome piece of technology that I really can't wait to talk about and here from one of my best friends in the world AA how's it going AA hi James so great to be here again onnet channel so I'm so excited to share what's new with co-al for microsof for 65 I'm sure this topic is super new for everyone and looking forward to talk about more yeah and we've talked a lot about the team's toolkit we've had some of your colleagues on so this is sort of some new tech that I'm excited to to show off so I'm gonna let you just go right into it sounds good sounds great awesome let's jump right cool go for it all right so hi everyone H quickly introduce myself my name is chash I'm A Cloud developer Advocate at Microsoft and today we will talk about how we can extend co-pilot for Microsoft 65 using team soit and.net we will bring our Enterprise data in co-pilot for Microsoft 365 and to do that we will actually create some teams apps and I'll show you how we can do it but before we jump in how we can extend coil for Microsoft 365 I just want to introduce what is co pilot for Microsoft 365 it's actually AI power productivity tool that uses llm behind the scenes and integrates with your data in Microsoft 365 using Graff so that means that any Microsoft 365 app you use today Outlook H one drive teams then coiled for Microsoft 365 will will have access to it through Microsoft graph and you will be able to summarize your emails or ask for some files to co-pilot for Microsoft 3 65 so the question comes when what if I want to extend co-pilot for Microsoft 365 experience and what if I want to bring some external data to it so I don't need to go to multiple different llm applications so that's what today's session is about we are going to talk about how we can extend co-pilot experiences and generally we have two main ways of extend and co-pilot the first way is Microsoft graph connectors and the second way is plugin and today specifically we will focus on plugins but I will tell you the differences and when to use which one and the main reason why we are again extending co-pilot is to bring your external data into co-pilot as well so that not only you can search your emails documents but you can also search your external data using co-pilot for Microsoft 365 without going to any other application that's the main idea behind it and which to use when is a critical area because both of the options help you bring your external data into it but plugins if you focus on plugins it is for Real Time data access and uh the data is read and write so we can actually work with the data and we can send some adaptive CS ask ask some information and we can send some data out and data lives anywhere not only on Azure Microsoft cloud but it can be also in external uh storages databases you can bring your existing data wherever it lives through plugins using on copile from Mar 3 65 another way of bringing the external data is graph connectors it is definitely easy to create graph connectors and we ingest data from your external data into Microsoft graph and data lives inside Microsoft 365 this way of uh ingesting data through graph connectors is read only if you were looking for only searching your data and bringing your existing data and maybe looking up for it co-pilot for Microsoft 65 gives you this option as well and um you can use graph connectors too so today as I mentioned earlier we going to talk a bit deep dive about plugins and um the reason is that we want to bring Enterprise data and we want to interact with our data real Time That's the biggest reason uh why we are using plugins and um I will also show you how we can create one so how do we build a plugin I'm sure most of you are aware of teams toolkit and we also had a lot of sessions about team toit for visual studio the way of creating a plug-in is is actually dividing into two areas one is pro code the other one is low code in proc code um we create mic moft 65 app and the app is actually message extension maybe you're already aware of message extension it's definitely not new message extensions were out there for a really long time and we are able to create them using teams toolkit and today we will create one using visual studio with c and also you have low code options you can use per platform and Microsoft co-pilot studio um if you're interested in going to low code um enir as well okay before before we dive a bit deeper with the scenario I just want to show you how you can get started building a plug-in meaning a message extension using visual studio and team soit and.net okay let me quickly switch to my other screen just one second okay there we are in Visual Studio 2022 okay what I will do quickly is I will create a new project and then I'll select teams app and next I can call this anything for example sample plugin I'll select create here as I mentioned earlier we're going to create message extension and the type of message extension is going to be custom search results I will create this and quickly show you how a message extension looks like if you've never seen before don't worry about it um it's pretty straightforward and we will F five and show you how the experience looks like on teams so quickly here under search folder is our main file search app. CS this is the main place where we do this where we create the search experience and this template comes with on teams messaging extension query Asing and here we we use one adaptive card and then we create a text and text being the query user input and we get the value of user input in text and we search using find packages function let's quickly go there and check what this function does this mainly makes an HTTP call to an API we make a search in uh we make a search in Azure search nugat meaning that we actually searching nugat packages by using me this message extension and we return um ID version description project URL and I URL so this is a very straightforward experience we are making one API call and then we are retrieving the um data through this API call and then we are just showing the result with adaptive card let me quickly run this before that I need to select a Dev tunnel I created the one before and then I'll right click to the project select teams toolkit prepare teams app dependencies you should select the e account you want to create your uh app dependencies so that we'll be able to create ad app for our bot and um we will create B framework and everything actually behind the scenes is generated by just this app dependencies you can learn more about this life cycle provisioning and everything behind the scenes H it's actually under infra you can check and dive deeper what we're doing with Team soit behind all this yeah and that's pretty cool too because we did some videos on that so if people do want to Deep dive really into the details there we have a whole bunch of videos and I'll make sure that I link them below as well um if you're interested in those deep details for sure definitely yes um that was so great James thank you okay seems like we're good to go I will hit you can use F5 or Microsoft teams browser and then this will initiate our message extension or any teams app actually on teams um and we will be able to debug and see how the experience looks like uh for today I will only show you how a message extension experience looks like so we will just Deep dive deeper with another scenario but for this I will add my teams app and then I'll start using it right away or I can debug and test if I want to okay so there we are in a chat environment I will click plus and then I will select sample plug-in local I can search any new get package I'll type Azure it's the safest and I'm sure I have bunch of them already okay there we go we have list of the good packages available I can pick one and send it to my colleague Rabia so this is pretty much what plug-in is and as of today if you create a message extension you should be able to use this message extension as a part of co-pilot for Microsoft 365 too so I should be able to test this message extension in co-pilot for Microsoft 365 but I'll show you in couple of scenarios so you'll understand why would you actually build one and uh use it as a plugin okay so I'll switch back to my slides quickly so this scenario I'm sure Mo most of you are already aware it's a pretty popular scenario custom chat GPT with Enterprise data using Azure open Ai and AI search meaning that developers can build their app ux chat app and then use orchestrator and also use Azure open AI to build their own CH GPT experiences and they can also plug in Azure AI search and plug in any data sources through as AI search to make a query to their data sets so what we're offering today is actually a bit simpler than uh this uh overall architecture meaning what if you don't need to create your custom chat GPT instead you use co-pilot for Microsoft 365 as this custom chat GPT H so in this case our app um and also chat experience and orchestrator open AI like any kind of cat GPT experience will be the responsibility of co-pilot and the only thing we are going to build is a message extension and we will plug in Azure search and we will make search to any data source we have it can be files videos images databases meaning that any sort of data living in on Azure today I can bring Azure search into my message extension and I can make a search and um reach out to my data sources right through co-pilot for Microsoft 365 and another um important way I want to mention is that I'm using Azure AI search here and the biggest reason is that I can bring in hybrid search experience in co-pilot for Microsoft 365 and I also quickly want to highlight why hybrid search is pretty important because H the advantage of vector search is actually finding the information through similarity instead of making a direct keyword search and also advantage of keyword search is precision every time I make a keyword search I'm 100% sure that with semantic ranking I'll be able to reach out to that data and as of today copilot for Microsoft 365 has already semantic index behind the scenes hybrid search combination of vector and keyword will bring will bring me the strength of Both Worlds actually it can perform better together with both experiences vector and keyword so that's the reason why I want to enable azri search and also with AI search I should be able to use hybrid search experience in my projects let me give you an example let's say that I have a data source to do all this architecture I'm going to have to create embeddings for my data and then that data with that data I will create index for my data in Vector DB using Azure AI search to create embeddings I'm going to use Azure open AI a model and once my data is ready with all the embeddings then I will create embeddings for my query user input so every time user asks something I will I will actually create embeddings version of that text and then make hybrid search with that embeddings um on Azure search and then I will return response and then um co-pilot will respond to the user with text document image video anything they want so let's think about a specific scenario for example call center scenario let's think that um we have a e-commerce company and uh there's always customers calling uh to check their orders or uh give feedback complaint and so on so forth what we want to do is we maybe record all these call uh Center Communications between agents and client in a a text format in documents so let's say a manager or a customer agent wants to search something from the previous calls they ask like any customer complaints recently and we convert this text into embeddings so it's going to look like an embedding we will send this to Azure search we will make hybrid search behind the scenes and then Azure search will return us some data about this question and I just want to highlight even though customers calling don't use the word as complaints we will be able to find this information and as you see here uh in the text it's it's saying I've been an echo groceries customer for years but I've never had such a problem uh with order before there's no complaints as a word but with uh the power of vector search and semantic search together I should be able to find customer complaints right away in text which is super powerful and that's the scenario we are going to talk about today sort of the overview of what a lot of developers have to do today and how sort of the co-p palet for M365 fits in which is awesome yes thanks J this is definitely amazing so I'm just going to show you how developers can get started with this sample and there are more samples available I'll also share that information okay so this sample is available in a repository copile for microsof 365 and the first one is doour T any developer today can go to this repo and clone this and get started using this plugin right away I already cloned this and I should be able to show you how how hybrid search works behind the scenes Let me jump right into Visual Studio okay here I have two main files let me close this read me and then I'll zoom in I have the search app it's a regular search app and then I have ai search to handle aszure AI search client let's quickly go to search app here instead of making an API call you're actually making a c sending the query to semantic hybrid search function let's go to this function this function initializes asure AI search client and then generate embeddings with our cury and here to generate embeddings we're using open AI client and we also use um SE search options we send the cery uh and our embeddings through search options once we create the embeddings we get response with title content URL and file pad okay so I have eight documents here before we jump into the hybrid search experience I just want to show you how you can test the sample how you can create your uh embeddings from your data these documentations are just um a prototype you can use these documentations or your own documentations um just to test this sample so also show you have you can uh create AI search and create your m settings with this data let's go to Azure I already have Azure open Open studio open I will choose upload files in the add data source here you will need blob storage and then you will also need aure AI search resource you need to put a unique index name here I'll put call record STS I will select add Vector search to this search resource and I'll CH choose uh Ada model and select next here you can drag and drop your files and upload files I already did that step I just uh want to show you that I select hybrid Vector key uh keyword and we will save let me go to a search and show you how it looks like actually once you upload it we chunked all the data and it looks like in an embedding format what we're going to do right now is whenever are users asking a question to to co-pilot we will convert their data to embeddings we will make search and then we will retrieve the related data from Azure AI search and um show it to our customers but before all this and showing this in co-pilot I also want to show you how this existing message extension looks like in a normal message extension way okay so let's go back and I will close this folder right click team soet prepare teams app dependencies and once we done with the app dependencies I will run this app just like another message extension and show you how we actually do hyper search in this message extension okay we are done I will hit oh okay first I'll select tunnels and then I'll hit Microsoft teams and here I should be able to show you that even though the words we use are not available in the documents we should be able to get the related uh documents from the call records I add Echo groceries call center and then I will hit plus Echo groceries C and then I'll type any customer complaints here we have bunch of files I'll select this one as you can see it shows up in an Adaptive card we also share the PDF you can open the PDF and see what is inside this call record let me show you how things work when we debug let's debug this hybrid search so that you will understand the flow a little bit better again I will type the same thing any customer complaints and here let's see that uh semantic hybrid search we create the search index client and after that we generate embeddings here we are using um Ada model from Azure open Ai and once we create our response uh from our text it should look like an embedding as you can see here it's list of embeddings okay and after this we will send search options and return response so in that point I'm just going to click F5 and show you that the results are showing up here okay so the same experience is supposed to work on Co pilot for Microsoft 365 before we jump into uh showcasing this plugin this message extension working on co-pilot for Microsoft 365 I just want to explain you one thing how does co-pilot work with plugins as you can see we have co-pilot ux V actually our chat uh experience is co-pilot here that means that co-pilot whenever user asks something in natural language input co-pilot searches for Relevant tool in this case our relevant tool is Echo grosseries call center and then finds reasoning map users intent to slots and then execute the tool one once we executed the tool we return the response from Echo grosseries call center and then send the response to our user in this case our um echor grossers call center behind the scenes we actually run Azure AI search and then retrieve the relevant data through Azure AI search using hybrid search okay let me show you how this experience works on co-pilot so anyone can um enable their message extension as long as you have the permissions to use co-pilot you can enable your message extensions and you can start testing it out I will enable my message extension and then I can ask are there any complaints about Echo groceries recently this is supposed to return me relevant results and the reason why we are getting three responses every time is because I limited the search options to three only maximum we are going to see three as you can see here we again got three files so here you can see the Adaptive card by hover over to the numbers the first one is again client agent the second one you can see and read through the Adaptive card the third one you should be able to see in an Adaptive card so all of these are also available as a part of the reference you can also get the PDF files in the reference okay let's ask something more specific for example we can ask um something special to the order number such as can you share more information about order number s 6 5432 at Echo groceries obviously this is a fake number then we should be able to see an information about the specific order as well as you can see John Williams called and we have a record uh with his request as well so here um as we're seeing we can see multiple response from uh our sample or we can retrieve actually one uh single response and we can show this in adaptive card experience too so this is only one sample available in our Gallery I also want to show you that we have sample solution gallery and we have a bunch of other plugins and connectors available for you to test out also in net if you go to akms community samples you should be able to find out this sample and also others available so looking forward to hear what you think what is your experience and also don't forget to reach out if you see any challenges or if you have any feedback um definitely here to talk about more awesome that's so cool to kind of see the full endtoend story as well because I think sometimes the you know when I I even I come into building an application I think to myself like how much work do I need to do but it's actually really nice to see that not only if I'm used to building teams apps the amount of code that you actually need to generate these sort of queries um to talk to open Ai and then to integrate into M365 co-pilot are like very minimal and it kind of just works into your flow so really advancing that technology uh is cool to see one question I have for you is let's say now right you've you've taken those documents right you showed how to index them what's the normal process so now at this time if I want to just add more would it just reindex basically everything so it's like okay hey we're every night or every whenever we're uploading those is that the noral flow that they would go through and then they would just know about it I guess um exactly so what I showed is definitely not for the production I drag and drop and use a Azure open a studio but you can actually automize that behind the scenes by just creating an indexing you can use net for that you can just create a job for um recurrence and create new files behind the scenes um we can also suggest that for example you keep all of your data in a blob storage and then you create recurrence on Azure and you create Vector index every time blob storage renews with the new files so I think that would be more suitable for production level um but this is only a sample just to showcase the scenario no that's great yeah it's examp exactly like as a developer I like the the give me the easy button right just like drag and drop some things let me you Dev it and then talking about that Pro scenario is great IA thank you so much for coming on showing this all off of course we'll put links to everything below and we'll also link to those other teams toolkit videos as well AA thank you so much I really appreciate it thank you James thanks for having me maybe I'll see you again another time thanks awesome bye and thanks everyone for tuning in and don't forget if you're over here on the donet YouTube to hit that subscribe button so you stay up toate every single time we put out videos right here on the donet YouTube well that's going to do it for this one so until next time I'm James and this has been on on [Music] [Applause] that

Channel: dotnet

Views: 2,082

Keywords: .NET

Id: DvqnTunnJkQ

Length: 27min 36sec (1656 seconds)

Published: Thu Feb 29 2024