Build AI-First Apps Faster with Semantic Kernel

questions so we're going to start the meeting now officially recording is in progress uh so let everybody know we're going to start with the transcription uh my name is Scott Locke I am the co-founder CEO of infernal red technology who is sponsoring this meeting I'll make sure I get into the transcript so I'm going to say it again if you're looking for a change of career in terms of what the company work for if you want to hire somebody to build pretty amazing software look no further than for Inferno red technology you can find us at infernored.com Inferno red.com uh if you'd like to reach out to us and um get a chance to work with our amazing team which includes Lee Richardson who's sitting here Greg Winograd is sitting in the back um feel free to reach out and we're happy to provide pizza and and hosting for for the Meetup and the account so with that um Santosh why don't you take the opportunity to introduce yourself and then roll right into it thank you and you are muted yes yes it's been a long day um thanks thanks for the intro so uh my name is Santosh I am currently based out of Orlando and I say currently because I may be moving in a few days a little bit north of here not not too far so uh I currently work at Microsoft I work as a consultant so you know I'm one of those people that go from customer to customer uh Microsoft loans me out to customer and I go Implement different Azure Solutions customer sites so that's kind of my day job I am also organizer for the orlando.net user group so you know user groups.net easy groups in particular are very dear to my heart and I'm organizer for the Orlando Court Camp which hap which just took place uh exactly one month from one month back from today so that was a great event um if you ever get the chance you know come to Orlando code Camp next year it's in March uh whether it's nice in Orlando at that time yeah and you know I a little bit about myself I love speaking at user groups uh conferences meetups you know different uh different uh stages and scales I also give talks on a wide variety of topics so I can speak about uh this is a talk on AI but a previously I've spoken about uh deaf topics like you know no SQL databases with Cosmos serverless I mean Azure devops you name it as probably given a talk on that one so um you know we're looking forward to this one this talk in particular um a few weeks back um I was invited to do a uh hackathon with a customer involving semantic kernel so I have prep for the hackathon I never ended up doing it because of some because you know personal reasons caused me to miss the hackathon but then you know I was sitting on a treasure chest of knowledge and uh I figured I would convert that into a talk and when Lee approached me to do a talk on a short turnaround I had I kind of had this one in my back pocket so I've decided to build this into a talk and uh here I am so uh just for fair uh warning this is a fairly new library you will not find much information on this one and that's it's probably a a side effect of it being so new um when Microsoft released open AI they also release semantic kernel and it is uh you know it's Brands packing here you'll see that I'm using a preview version of it in my uh preview version of the nuget package so looking forward to the stock with that uh if you're okay with it I'll go ahead and share my screen [Music] please tell me you can see a screen with a kernel on it okay that that was deliberate you know you see the kernel in the background semantic kind of so come on yes love emojis would be nice for my bad jokes but enough for that so let's get moving so that's a little bit about myself uh best place to contact me is probably LinkedIn these days uh everything else is too um chaotic in terms of social media I will um give some resources at the end of this talk uh I am still building out my GitHub repo for this talk I'm not as well organized as I would normally be because it's uh quick turnaround on this one but I'll I'll probably put this up on my GitHub repo at some point so just a quick uh screen I wanted to start this talk with so this is a common saying that's been going around and you know it's you can kind of see it from different perspectives you know some people will say oh it's political some people will say it's you know this is what it's going to be and you know I kind of on the camp of people who think that yeah you know this is kind of a true statement you know it's there's things that make you more efficient and it behaves of us to adopt those new technologies and use them and you know people are afraid that AI will take over the world and you know Terminator style kill people but that's not what we are worried about I think what we should be worried about and I'll talk a little bit about responsible AI later on and this stock but I think that we should generally be worried about people misusing Technologies and you know if you are a uh if you are a consensus person and you you don't want to see it misused then you might as well be early in the field and start setting Trends so that's how I see this uh today's agenda is pretty straightforward um you know I will I won't even wait to get into code we the first thing we'll do is we'll write a simple chat GPT app using semantic kernel and I will show you how uh in a fast and efficient it is uh then I'll discuss some basic details um you know all of us know a little bit about AI we heard about chat GPT and Dali and a bunch of other uh uh I guess models lately but uh Slash software lately but we uh want to know you know what's Behind These software so I'll I'll discuss that a little bit and by the way when you say when you see a SK that's semantic kernel I got tired of writing out the whole thing um and you know we'll discuss Basics on uh llm AI models semantic kernel I will write a function uh simple uh semantic kernel function and then we'll see how to chain them together which is going to be a really interesting uh I will show you how to build skills so you know like if you have ever coded for voice assistance uh you know they have skills so Alexa and so on you can similarly create skills for semantic kernel and the best part is that recording language is English for these skills uh we I will also give a couple of other uh features of semantic kernel you know how to use some pre-existing skills and how to write quote unquote native functions so you know you write to write in English but I'm also a C sharp coder at heart so I want to write my uh for some of my skills with our functions and C sharp and I'll show you how to do that and whenever I do a talk on here I always send it with some simple responsible AI principles and throughout the stock I will be monitoring the chat um if I don't respond to a question for more than a couple of minutes feel free to stop me and I will be happy to respond to that I like I like to keep it in practice any questions so far I think you're in the room so far all right cool so what we're gonna do is we will be writing a GPT application from a scratch so before I start coding I want to kind of show you what's covering this application so what's powering this application is a instance of azure opening act uh if you can see my screenwriter uh let's go it's currently uh you know this one is not available to everyone but but you still can make use of these same principles and I will show you how to do that you can use run this against open AI um API and just do the same exact same things but you know Azure Azure open Ai and open AI you know are the same thing Azure is you want to use Azure basically for Enterprise reasons so you know uh a few weeks back I read a story where Samsung developers submitted their code to child GPT and it kind of went into public uh if you want to prevent that from happening what this is what you want to use you want to use uh Azure open Ai and then there's a whole bunch of options you get for networking and identity and other items which makes your traffic private and it it doesn't expose it to the public so definitely uh you know if you're doing private if you're not doing a talk like me and if you're doing some kind of company in a private proprietary data obviously is azure open AI Consular other regular open AI thank you and definitely don't ask GPT where about you know don't address code secrets to the chat GPT software that's out in the open here's Azure open AI so when you use a provision Azure open AI instance all you need to know is the couple of things you need to know what models you're using and I will talk about models later but for the purposes of this uh application I'm using a chat GPT model as you can see it says gbd 3.5 turbo and you can create more models so for instance I could use any of these other ones like text Ada text query um DaVinci has code and text and so on so you'll see yeah and again you know I'll talk about all these models later but for the purposes of this application we're using chat GPD because that's what everyone wants to talk about and the other thing you need to know is this part so for the purposes of this uh talk I will be using the endpoint and I'll be using the key so I can use the primary key one or key two and the other item I'll be using is the name of the deployment so these are the three things I'll take away from my Azure open AI instance other than that it's just sitting out there it it's probably incurring some cost because I've been hammering it lately but it's not that expensive I do want to you know stress that if you're using any kind of API like open AI or Azure open air it is going to incur some amount of costs so be you know don't run like million data points against that and B shocked when you get a build so that's all you know Okay cool so so this is what I'm using right so next we'll jump over to a window and is this visible at all so you can can you read this if you're in the room definitely in the back here I can I can try to increase the size now all right it's good all right cool so multiple people including chats as you can read it so I will take that and run with it okay so the first thing I want to create is I want to create a console application I will collect I like writing console applications because I don't like dealing with any I just want to show the input and output and C sharp then not have anything else get in between so console let us so I'll create a dotnet new console uh we will call it uh let's see Capital Area dot net and chat jpt uh where it's extremely Innovative name but bear with me and then for the framework we'll be using.net 7.0 it really doesn't matter I think we can use dotnet core the oh of course all right all right and please excuse my error so I'm just you know trying to do a little bit of live scoring so bear with me okay so that's it so now I have a capital area.net Child GPT console project I'm going to open up and visual studio code there it is so the first thing you need in here is obviously you know I already showed you that you need to have a Azure open AI instance or an open AI instance so the next thing you need is the packages so we will add the semantic kernel package and as I add the semantic kernel package you will see that I'm using a 0.12 preview version so this is a really really early so again you know I want to stress that please do not take this and try to write production uh business production code with this because that could end up in a bad situation then I'll have integrated terminal over here just for just because I like to do that okay so so I have added semantic kernel to this one uh so the next thing I want to do is kind of get rid of the boilerplate code and um I will start with let's see I want to add a few items and these are essential because these are the configurations for me to connect to my chat GPT instance I'll quickly flash this this is the Alias uh the uh deployment ID open AI endpoint and the key and I'm going to close that up that's why I put it in the region region can actually be used for good purposes believe it or not that's good and restore this share Crystal okay okay so we have all the you know Azure uh open AI chat completion service all the configuration values in here so next thing I want to do is I want to start building a kernel so this is the semantic kernel Escape kernel kernel so uh I built this but then you know this is going to be it's going to throw an error because I don't I don't know what this kernel is right so I'm going to add the namespace so it knows what I'm talking about and that fixes the error so next thing I want to do is uh I want to add this kernel configured so I'm adding a configuration and in the configuration I have all these options so I can add Azure chat completion service click and add Azure open AI chat completion service text completion service embedding uh you know generation there's a whole bunch of other items that you can do but you know quick [Music] quick tip like if you want to use the Azure open AI obviously you should use the library that's called Azure if you have open you can also sign up for an open air instance and get a few free calls you could if you have the key from that you should probably use the open AI chat completion service so in this case I'll be using the azure open AI chat completion service oh there it is it already I'm also using copilot so it's actually completing my code right here but you can see that I have the model Alias which is the name of the model the deployment the end point and the key so this model Alias is Just some name I gave it from my local code the deployment ID endpoint and key I got from the Azure portal okay so that's what we have so far it doesn't appear to be breaking I will switch over and just make sure this bills and then keep going so I promise to build a chat GPT like interface uh it's not going to be as fancy as the one you see online but it'll kind of you know give you an idea of what it looks like so let's let's instead of building a broad one let's build a simple chat so in this chat what it's going to do is uh It'll ask me what I'm interested in learning about and then it will try to narrow down my area of focus so that's the first one uh what what are you interested in learning about uh it'll um so you know we'll somehow send that to the chat uh it will come back with a reply so let's do this send the chat and get reply and see what to do with that and then I want the assistant to ask me is there anything you like allow in particular about this topic is is there anything in particular I can love about this topic and again you know I'll send this to chat and get the reply and then I will print the entire chart great so what do I need to for to accomplish this so that's the question so I have semantic kernel I have configured semantic kernel to use my chat GPT instance in Azure open AI and I'm kind of you know asking it a couple of questions but you know this is where the whole concept of prompting or prompt engineering comes in so before I would do that I want to tell charging that I'm interested in a certain area so how do I do this so I wanted to first ask you know what is your profession and what is your expertise so let's do this it's going to ask for my profession and my expertise or I'm going to touch our GPT that this is your professional expertise and then it'll ask me uh I'm going to tell it this is what you are and then it'll ask me oh yes I am but what what are you interested in learning about this and then I will send something and it will reply and then I'll it'll prompt me for a particular topic and it'll reply right okay so I think we kind of had the skeleton but now we need to hook this up with chat GPT so the first thing I need to do is I need to configure my kernel to use to use the Azure Azure chat completion Service uh well it's just use the generic chat completion service I guess technician and this is throwing an error well why is that because something is missing all right we'll go ahead and add that let's see it had an eth at the end of the completion yeah okay there you go thank you that's why I love live coding because ah there you go cool so that takes care of the red squiggly or not yet let's see if services to get service all right this is why we should also be careful of what you use uh with uh copilot so there you go so this returns something what does it return it returns our eye chart completion all right so I'm gonna just call it um see I turn it completion GPT okay that's good yeah so far it's good with that so the next is I want to take these um above input and create a prompt so my prompt is that let's see a prompt us he I'm telling the assistant that you you are a profession right you are a uh proficient okay I guess it should be professional but that's fine and you're an expert and at this particular area of expertise and that's what I will feed in so using this prompt I am going to create a chat so we will call this chair GPT dot create new chat and prompt does this work so create new chat chat history okay that's fine we will see how this works I don't know right so I created a new chat uh and I'm prompting it saying uh you are a you know this is your profession and you're an expert and such maybe I should say this is your profession all right so now it's going to ask me what I'm interested in learning about and when I answer it it's going up what we'll do is we'll add this message but it's not just any message it's a user message does not continuous message okay so let's see if I'm missing an assembly problem where does it look like over here let's just build this does not contain a definition for add user message okay so it's trying to convert it to maybe it's a different type of chat history so let's see each other history but that's not working so insert the relationship start instrument work I can call this a open-air changes ah okay all right so I had to type cast it to opening a chat history but does that mean it will also work for permission let's see no I guess we'll just have to go with opening a check history like I said this library is continuing so still some rough edges there should have probably been Azure chat history maybe feedback for the team okay so cool so we took the profession uh and expertise we created a prompt we instantiated a new chat and then it added a uh that it asked me a question it read my answer to that and it's adding that as a user message now once you send the message the next step is to get the reply so let's look at this so string reply and then the reply will be uh chair GPT dot something right dark generate message async okay [Music] um so College okay so that should do it but it doesn't like that oh that's okay so I guess I need to add in a weight over here [Music] and now it's like that oh that requires this request settings okay provided that new chat request settings okay so that's my reply but even the reply needs to be added and this is added as assistant message so the assistant is the charge equity and the user is here so I you know I answer the question it adds the answer of the question over here as we use a message uh the assistant response and it adds an assistant message okay so that's the first chat so let's go ahead and combine this make sure we have no issues then we'll add a second message just to make it look like a chat and we should be good to go after that so the next question is do you like anything in particular yes so then it wants to read that so it again I'll read that into the console again I need to add that to send that to the chat so that we'll do that over here and once I send it I will get the reply in a similar fashion and we'll add the reply as an assistant message how are we doing any questions so far is it all straightforward simple complicated all right so I guess you know we'll wait and see if this is simple or complicated right so what we want to do is we will now do a console Dot the right line and we will see the chat output right and then what I want to do is for each message and chat messages I want to write that to the output so we'll we'll write this and then see if we can pretty it up further right so I'm gonna go ahead and switch to the logic console and we will see if we can run that so we will click it here I'm going to do a DOT click build yeah it built with some warnings but that's that's okay as far as the warnings don't cost my code to break I can live with that during live coding and we'll run this foreign developer or chat GPT is a software developer expert and Microsoft Technologies all right so now I'm sending this to this is the moment of truth right I'm opening up a chat session and I'm telling it that you you assistant are a software developer with expert in Microsoft Technologies it's asking me what are you interested in learning about I want to say dot match seven right so it's going out and doing the whole big blah blah blah bloop sound for calculations is there anything in particular you'd like to what in dot Net 7 what I want to learn anyone you wanna help me out here I forget what are the new features references records the guy product yeah records so again you know this is all happening in the background it's talking to the open AI instance and uh assuming my code doesn't Bond it's going to work oh wait uh I did get an output but it is really not useful at all why did that happen so let's go back and look at my code [Music] so I'm outputting message but message is really that's not text so instead what I want to do is get some useful uh information from the message oh there you go so let's see if co-pilot actually leads me right this time so it's sync message dot sender message dot text but this is really not it so what can I do message dot okay author role and the sage dark content so let's see I'm going to rerun this all right let's let me go with a different profession um let's see I am a driver let's see I'll just stick to software developer I don't know expertise is I'll change it around that's uh I'll still go with Microsoft Technologies and what are you interested in learning about I'm interested in learning about azure so this is where it actually goes out and starts collecting the data uh anything in particular I will say Azure opening I have no idea this will work because this is so new and I don't know if the data it's getting us old so we will see City oh okay so great uh Microsoft is okay so it's giving me a bunch of texts talking about different Azure Services as a software developer you can leverage this certain uh and then I want to talk about oh Azure open Ai and then it goes out and tells me that it's a partnership between Microsoft and open AI and I should open a office a range of Air Services cognitive I'm not quite sure if this can be classified and I should open AI but okay we're getting a response which is what we wanted to see and we also learned that we should not always trust the response so that's another valuable lesson over here and this this talk about gpt3 um cool so there you go yeah first uh chat GPT application loaded live with semantic kernel and let's look at the number of lines of code that I actually had to write or this so basically I had three usings um I had to get some values and ideally they should not be in code here they should come from you know.net Secrets or something I didn't want to go you know that was not the point of this exercise so I did not do it but maybe I'll update my GitHub reporter to do that the right way um so you instantiated a version of the kernel you configured it with the values for Azure open AI and then he created a prompt you created a chat completion object and it created a chat session so once you created the chat session uh you add you know you expressed your topic of interest and your topic that you really liked and both of these were added as user messages the assistant reply came back as and when the reply came back from the chat GPT object you added it as an assistant message and that was pretty much it and then we collected the messages and we basically displayed them so now in this says 50 lines of code but I want you know if I I bet you if I remove the spaces and all my console statements are probably more like 25 lines of code at the most so you know with 25 lines of code in a semantic kernel basically let you write allowed you to do charge GPT like chatting so that is pretty cool um at least I think so do you need an account to use the API so a couple of things if you're using Azure open Ai and I see this message in the chat from Victor so that's the one I'm responding to if you're using uh Azure open air you need a Azure account this is true and currently Azure open AI um is not available to everyone like I literally had to beg and plead and you know threaten people to get the the access to Azure open here uh you can however get a account with open AI this is these are the people who actually provide chat GPT in in public um when you get a developer account with open AI which is Free by the way uh you can go to the developer portal and configure a key and use it the same way it's literally the same uh things that you you'll need like a model uh like a deployment name for child GPT uh they also have options to create deployment names for Dolly and like the image generation service and other other models but you will need to create a chat GPT model and then it will provide you with a URL and it will provide you with a key so it's exact same uh the only difference is that you would be using a ad open AI chat completion service over here in Smith Azure chat service so that's the only difference does that answer your question okay cool uh the next question can I build this in Linux in a container I don't see why not I mean the dotnet is cross-platform uh you're basically um using.net 7 which is definitely cross-platform in fact uh if I had time today's demo if I was better prepared uh you today's demo would have been in Dev containers if you haven't used Dev containers it's a really cool feature that you can access from GitHub and you can basically code in your browser and you can run all these all this code in your browser so yes you can do this in Linux in a container you can also do it in a Dev container which is which is actually much more we are going and one thing I want to say I did actually just get my account by uh the Microsoft Azure Ai and there's a form you need to fill out I just have a couple Pro tips for anyone who's looking to fill that out one is uh you can't have an email address that's Google or Yahoo or whatever you have to have a company one and it has to match the exact um URL you have to put in a URL for your company and your your company's uh address and phone number I think it might just be an automated form though but you have to make sure those two things match because the first time I filled it out they didn't and it you know came back and two days later and then I had to you know resubmit it all and took a couple more days that the whole process is a pain but you can get through it um and then I had a question about this that I'm just I'm kind of struggling a little bit to understand centouch what's the difference between just using the pure Azure open AI um SDK and using this semantic core like what is what is the semantic Court getting you on top of the the regular SDK uh for this particular instance I think you know it's just a simple hook back and forth I am not I haven't played around with the open AI SDK to know that it offers like uh ability to create a chart object and add user message and so on so you know you get these type of uh simple um abstractions uh again you know that may be part of the open AI SDK I'm not I'm not sure um maybe maybe if you're playing around with it we can learn from each other right I don't know I don't know everything and I have never played around with the open AI SDK itself awesome so that's that's one thing but you will see that you know this is just one part of it right so I this is you know I can actually build a bunch of these type of prompts and other items as skills and I can chain them to each other and other things so this is where you get really start getting into uh you know open I guess um the azure I'm sorry semantic kernels power so with that I'll quickly dive into a semantic kernel itself so let me pick up from where I left off uh suddenly did I answer a question and I'm not quite sure I did but you know it's I'm also learning some of these things and I don't know for sure if it's Superior to open aisdk or not because I really haven't played around with that yeah I think we'll get there all right so I want to quickly uh go through some llm AI Basics so uh you know GPT basically stands for generalized uh pre-trained Transformer but I also called it uh easier way to remember is GPT is good at predicting text so you know that's uh if you you know want to know what it is you know that's you can certainly use that as acronym um what it does is it learns from a lot of Text data that defined on the internet and other sources and then it tries to predict what comes next in a sequence of words or symbols or code for example if you give the uh a GPT model uh the input saying the sky is uh it's going to try to predict what's come next as you'll probably get like blue sky is blue this guy's cloudy or Sky star depending on the context so if you if you say something like it's night the sky is it may say dark uh or you know it's in Seattle and the sky is it may say cloudy so it also uses a little bit of context in it also GPT models are you know based on a special type of neural network called a Transformer uh the Transformer what it does is it basically captures uh the sorry the GPT model captures the meaning and structure of language uh it does this with a Transformer that has two parts the encoder and decoder and we will certainly see how this works but the encoder basically takes an input sequence or input text and converts uh breaks it up and converts it into a vector representation and the decoder takes this Vector representation and generates an output sequence that is kind of related to the Specter sequence uh and these are some of the models that you will see in fact uh when I was going through the Azure open AI portal you got to see some of these models so you know gpt3 is you know it's a large language model it can basically so it's the more recent one gpt4 is out but no one knows what it exactly has so but we do know what gpt3 has so this is kind of an evolution of GPT so it went from Ada to Babbage to curated DaVinci uh it's the way to think about this is uh you know it's a b c d and they kind of went up uh or became more complex and improved as it went on so Da Vinci is the latest uh one for gpd3 and uh you know there's no gpd4 so they moved on since then but uh Ada was uh basically had 350 million parameters uh that were used and 40 GB of data that it trained out um data is really good at let's see classification uh sentiment um I think it was also good at like summarizing things so you know the simple or it could do simple things really well and then they introduced uh damage which uh actually used three billion parameters and we'll talk about parameters a little bit actually have a slide on this one but these three billion parameters a parameter inputs and I was trained on 300 GB of data and it's really good at things like arithmetic uh giving word analogies um also like reasoning things so that's what Babbage was good at and it but it was built on top of ADA I believe uh Curie is uh you know it again you'll see that it uses exponentially more parameters and data and this is where it started to get into the question answer uh translating uh paraphrasing uh text to speech speech to text and so on and finally Da Vinci is you know miles better than any of these and this is where he started to get into creative stuff so the da Vinci is where you could actually you know tell it to write a poem and so on and it would start doing that so uh Da Vinci is usually the one that's used these days especially with you know if you connect with open AI or even Azure open AI you'll basically see Da Vinci being the latest ones uh one thing to keep in mind you know ABCD is the cheapest and D is the most expensive and that's a good reason this is why because the computer computationally DaVinci is far more complicated than uh Ada so problem sweet uh you know I created a prompt you saw what a prompt to us prompt is basically you know English is recording language is how I put it it's the input or queries that a program uh gives to Ai and it uses this to elicit responses um it's basically you know natural language sentences questions code Snippets commands and so on um you can use any combination of text code or depending on the domain and the task um prompts can you'll see that we will be nesting and chaining prompts uh more chaining the nesting with semantic kernel but they can certainly be nested also so think of it like a query like a SQL query can be nested uh within another SQL query so similarly you know prompt is similar to that I kind of think of problems like you know A bash shell command or a Powershell command and so on so you know you can you can certainly you know you can pipe a bachelor comment on the one similarly with prompts you can do some kind of chaining um The Prompt engineering is an interesting term and you know generally people tend to put prompt engineering you know the whole two words in a double quote I put Engineering in double quotes because I feel like prompt engineering is basically it's the art it's the art of crafting effective prompts you know that's what it is and you you try to tweak your problems and kind of you know it's until it it you know becomes this wonderful thing so it's it's basically art um this but you know once you discover something you try to repeat it and I guess that's what the engineering part comes in so that's uh you know and finally the parameters you know we talked about all the parameters and the values and uh you know this is what it comes down to parameter just like you can tweak a Powershell um or bash command uh by parameterizing it you can similarly tweak a prompt with parameters and these are some of the parameters that you can provide so for instance if you look at the top three temperature so these are kind of the ones that control the randomness and creativity of your GPD model and you know the uh other to the frequency uh and the presence penalty uh these are the ones that uh can you can control repetition so the way this works is you know if you have higher values then we have less repetition or it will plagiarize uh less from itself but uh you know the answer may not be as coherent or consistent but if you want to the answer to apply more consistent you have to uh increase your penalty uh which means you know you'll be penalized and that it will start repeating some of the words from the input or the output so that's uh that's that's how you tweak some of your uh prompts oh and you know we will see how some of these are used and you know these are things that again it is very dependent on what you're trying to do it depends on your domain so for instance you know you may want to be you may want to write poems so if you're writing a poem you should probably uh increase the creativity of the model and uh you may want to set it at a high temperature so the temperature values are between 0 to 1.0 so you may want to set it to 0.8.9 or even one depending on how creative you want the approve generator to be but if you're doing more of a fact checking type thing you may want to set your temperature to be close to zero so that makes it more fact based and less creative fact and again keep in mind that the fact is something that the model knows it does not mean it's a real world fact that is something you have to double check yourself question so far if not I'll jump into my next month um so prompt engineering again you know these some of the uh prompt engineering kind of looks like this you have to give specific instructions to the AI uh and then you know it will make sure that uh and the instruction keep in mind that yeah you can also tell the AI that if the user asks you a question and you don't know the answer then say I don't know so that is called giving it an out you you know otherwise it's going to try to you know fix uh definitely you know prompt engineering involve some of these items you know be very specific leave but don't leave as much to interpretation be very descriptive use analogies uh you know you may want to repeat yourself or repeat the instructions sometimes uh also the order in which you provide uh instruction really matters so if you want to say summarize the following then you want to add all your data after that if you want to say summarize the above then you may want to add your data above that um and if you give an example it may use the most recent example if you give 100 examples so so the order matters quite a bit uh you'll see a lot about tokens when you write code our tokens are basically basic when you give the input you cannot basically take the whole text and insert it into the model you have to break it down into some kind of understandable format for the air model and that's what tokens are tokenization is the task of splitting tokens into smaller units um and uh you know the tokenization cost really affects the memory and computational resources so for instance if you look at the tokenizer if you look at the cost for DaVinci versus Ada it has different it has cheaper and DaVinci is more expensive so but you know DaVinci can also handle larger larger token inputs and so on so the you know there's a lot of things that you have to play around with and certainly you know these are some things that you'll have to know when you're doing it uh useful Link at the bottom of the page if you click on this one uh let's see if it opens up in a window thank you so uh this is a useful link from open AI in which you can actually for it allows you to calculate token for gpt3 so if you type in some sentence it'll tell you how many tokens it requires to process this so you know I'm just giving a quote from a movie and it's telling me I require six tokens IE token they say is approximately four characters so 29 is six tokens but uh you know that's that's just a rough measure so you know uh just watch out for um watch out for how you know you think it's you know how many tokens you're using because that is how your computation is calculated in the months uh this is these are the things that you hear and again not to dive into any debt with them but you know when you have tokens tokens are basically you know your input text is broken down and then it's converted into a vector format and stored in Vector databases now what is Vector it's more of a spatial representation so for instance you know if I have uh you know words you know a bunch of words that are related it may store those in the same space in the vector database and uh unrelated words may be stored in a different space so if I say you know uh medicine doctor um X-ray and so on it may store that in a similar uh in a close enough space in the vector database to where if you make queries it may return some some of these together but you know if I put in football I may store it in a completely different part of the database and that's not going to be written if that's not part of your query so basically you know the model captures the semantic meaning of the data such that similar data points have you know similar embeddings and embedding is kind of like a numeric representation and it's stored in a vector database and then the queries are they query against the vector database and they find the most similar or relevant data based on the semantic or contextual meaning again you know you're really not involved with any of these most times but just this is just like a high level understanding now for the topic of today's semantic kernel uh this is basically a lightweight SDK uh and the the thing about semantic kernel is it allows you to easily mix your conventional programming languages so yeah you know most AI stuff previously was written in Python and you know semantic kernel still allows pattern but we can write C sharp code and it pretty much works with you know doing it gets the job done with uh with the semantic kernel so it's cement these are the different parts of the semantic kernels so you know the kernel has something called a planner again will probably not ever use a like refer to a planet directly but it does have it and the planner makes use of the SM and C are basically skills um what's the the C is for connector uh one second out skills so memories so M is for memories and connector so uh the planner basically invokes these three to get the job done so uh it's it's uh it's a highly abstracted um uh Library uh like it abstract survey most of your operations and you simply write the code and get the results so that's why I like it so much because you know I'm also relatively new to Ai and this helps me get you know focus on business problems solving business problems as opposed to worrying about you know what what I'm doing with a memory or something like that um a skill so this is where it starts to become interesting uh with semantic kernel you can create you can write functions a function is basically like the name suggests it's like a small section of code that does something so you can write functions in English and uh like those are llm AI problems they're called semantic functions and you can write functions in Native computer code like C sharp and those are called native functions um so the way it works is a skill is a group is a single function or a group of function it's like a container container for this so memories again you probably never have to deal with memory but again uh it's good to know what these memories are something that provide context uh for your ask so ask is basically the input and you know memories are uh basically the oh the you know when you store your data in in a air in a vector database and access it back that's that's how you get the context so for instance if I'm you know asking chat GPT a question you know you um you know I asked it a question about um let's say a topic like semantic kernel and then I say oh provide me the basics then it knows to give me the basics of semantic kernel and that's that's what that means by memory uh so to me the concept of memory is similar to Ram uh in like in a computer there's a CPU or a ram a ram kind of keeps context of all the data that's being processed by the CPUs so that's that's similar to a memory okay so that's all I have not from I'm not done obviously uh other how am I doing on time really and I have uh 20 minutes 30 minutes or less than that yeah 45 minutes cool I don't know I have enough material for 45 minutes but you know I will certainly you can always generate it yeah so what I will do is we will kind of dive into this and start writing more code I do have pre-cand code if I run out of 10 but I want to show how to build up some of these so I will start with a simple one uh you know where you are using um again you know you you know you're using a complete uh completion service but um uh semantic kernel what it allows you to do is it allows you to do things like you know perform different uh create different functions and then chain them together so I want to write an example of chaining things together so I will get my command prompt and go back and uh I'll create a dotnet console application and you'll see that you know that's what I'll be creating constantly so I'll call this um no SK chicken and and greater than net 7.0 and I should have that in here there you go that gives me access to more of this projects so as usual uh what I'm gonna do is I'll start right writing code for this so I will need to add the package reference so I will add the package reference in the c-sharp project and then I will start doing my usual item where these are some of the boilerplate code that you'll see I write for most of the programs so you know some name spaces instantiation of the kernel configuration and then hooking it up to the Azure open AI instance so this I'll be using for most of my code and so think of semantic kernel as a thin layer that's sitting between your code and um you know the kernel so in the first instance what I'll do is I'll ask it for a movie name oh God understood this and then um I am going to give it up an interesting one that says given this movie name uh generate the most popular code from the movie so and then let's [Music] do this so once I do that I have the what I will need to do is create a function so I'll create a function and I'll call it my code function and this code function it uses the kernel that creates a semantic I don't know symptom lost intelligence semantic function and it will do this it will do my code prompt I don't know what this is I'm gonna delete this now because I'm talking with my um I'm talking with open AI uh Library I also can do something like I can say um now for my open air Library I want to use the tokens I want to set the max tokens to send with each message and let's say we send it at 500 I mean enter uh just for reference you know for the max tokens for gpt3 I think it's 4096 and the max tokens for gpd4 is 819 so 500 is well within the range so I created this created cost function uh so next I'm going to provided some context so I will say of my context and I mean and the context I want to set is for the move that's not the track it's interesting how copilities bravely gives the wrong um okay so I'm setting uh the context as you know this this movie variable that's being specified here this is where I'm hooking it up to the actual value in C sharp so I had this so far and then it tells me but my recollection is I have to actually use the kernel to run this so I call back semantic Colonel I say run async and uh I use insert my contacts and then I call my port function okay but and we will give that being a output and then what I'm going to do is this is probably an awake right here because I used async all right and then I will write this out to the console so I'm writing the console out so what this is doing is it's taking the name of a movie and it's getting the most popular quote from the movie right so this is pretty standard charging beauty stuff that you see in everyday life where you know people asking at what support from the movie so um maybe I can just use the integrated until now so you're switching back and forth yeah there you go we're able to see the integrated terminal or is this text too small to read yep we're good all right cool so in the integrated terminal I'm going to do I'm going to do a build to see if this works all right cool and then we will do a when dot net runs so it should come up and ask me the code for a movie let's see a movie name I'll say dominator because you're talking about Ai and you know it comes back and says so I'll be back as a um the most popular code from the Terminator which kind of checks up uh okay so that's fun you know that's fun yeah sure yeah anyone can do this uh but this is what starts to get interesting so this is what I want to do I want to um generate a review for this movie and I want to use this most popular code from the movie to generate that review so okay so how do I do this so first you know I am going to generate a quote uh I want to create a my create a prompt for the uh review and okay so we'll call this and uh give a review of the movie uh no no so take the squad [Music] take this code and write a review about it so the code does not run the movie because that doesn't make sense so that code is actually I'm going to insert the previous uh and we'll show how to insert this into here but what I'm doing is I'm inserting the quote as the input and then uh write a review there's a small page so so what I'm doing is instead of saying that uh go back to this uh look at the movie and then write a code for this movie I'm saying that write a code uh that you the right way view that uses this popular code from the movie so let's see if this works so I need to set let's see the next one I will need to do is expand this so like how I created the code function and I have to create a review function so let's see wire review prompt okay that checks out I think copilot actually work in this time for me okay and then I don't need to set any further context because I'm not providing any more inputs but what I'm doing in here is I have it's first going to call my code function and then I'm going to add my review function and what this does it performs this thing called chaining so a change code function to review function which means that the output of the code function is the input of the review function so that's a Paul claim let's see if that works so I'm going to do this dotnet [Music] so we'll use the same movie Dominator and see it should be I'll be back now does it use I'll be back in the review okay so there you go so accumulator I'll be back it's a famous quote so I'll use that in a review so okay so maybe maybe that's just for Terminator maybe it doesn't work for other movies right so let's just try a different movie can someone give a movie name or type it in the chat ddlj okay we will certainly do that I don't know how much do you know about ddlj but let's give it a shot no that's not it didn't hallucinate and yes I don't know Interstellar okay I'll take intestinal I'm sorry did someone say something yeah enter I don't know what code it's going to use from Interstellar because it doesn't have this single quote that defines the whole movie you know it's like the only thing that we shut off is that love is the one thing that transcends time and space Okay cool so and we'll also do Fight Club just to be sure kidding I cannot say for work alone oh yeah that's true I forgot to do that hopefully you know comes back with a safe forward quote the first rule of the fight Club is we don't talk about right which is the most popular quote from Frank Thompson there you go so look at that okay so cool so chaining functions this is really cool uh you can also do something like you know take this uh code and generate uh the generate slogans for a software service product or something and set it to creative mode and it would be you know really uh you really do crazy stuff over there so I can you can change multiple you can you know I just showed chaining uh two together you can go as deep as you want I haven't tested the limit so if you do 20 will it work I would think it would but I don't know that for sure okay so that's chaining so let me see what I have next any any questions um sorry go ahead please so you know I was having this conversation with my boss like three weeks ago on how to implement the the AI you know but he was telling me he doesn't want to share the the data that we have because he's actually going you know to somewhere else and then so we we couldn't find an actual uh application for the business you know so we end up more discussing the functionality to improve the coding you know the the code that we have already but we couldn't find like a business application like hey let's give this data so we can get some kind of prediction but he said hey we cannot do that so have you seen like uh real business application for in your I don't know in your last two or three months that you've been you know around the yeah so a couple of things good question so you know people um that's one of the I guess advantages of working for you know the company that's at the Forefront for this because yes we do get those kind of questions and uh I'm gonna kind of put on my consultant hat at this point but uh yeah so one of the top concerns customers have is they don't want to send their proprietary data to a public facing chat GPT because they don't know you know if it'll consume that data and learn from it and then spread it out to others which is what happened with Samsung so if you read up a story and maybe someone can find that story and post it in chat but you know Samsung developers basically try to improve their code by sending it to chat GPD and you know that made that code basically the equivalent of making it public because you know chat GPT now learns from the code and it has their code in there so that's not what you want to have happened which is why you know uh chat GPT is you know obviously uh it is open Ai and open AI partners with the Microsoft so Microsoft itself has a Azure opening instance and that's the one that I was coding against now this one is basically you can also deploy the same model so you know you have a model for chat GPT which is the one we used we also have uh you know there's a Code DaVinci which is the one that you would probably use to improve your code so that that's the model for that one and uh you know you can configure it in Azure instance and once you configure it you know you you can actually set it to be a private within a private Network and you can connect you know if you have a Azure uh v-net you can connect that to this this one and that way all your traffic is within your um you know within your instance um or within your own network or if you even if your data is on-prem you can uh connect it to this Azure network uh and uh that way your data stays within your own network and this as for this instance you know this model will basically consume a data okay to learn from it it will improve over time but it's not going to share it with other models so that's the other part of this that's why you should probably look at a private model deployment as opposed to you know just banging the Open chat GPT instance but hopefully that answers your question so that data stays in Azure Azure it's it stays local to that model it will not be shared with like this it will stay local to this deployment it's not going to be shared outside of this deployment okay okay so good thank you yeah cool good question uh can you point the SK to on-prem program like I do not know the answer to that question I'll have to find out for you Brad sorry cloud I will I will ask the team about that I'll ask the team and hopefully you know they will be able to point me to resources whether it can be shared or not so okay so I have a couple more coding exercises but I want to you know it's a close to 8 15. uh so I want to see how to best use maybe I'll stop at 8 30. does that sound goodly or do you want me to stop on you that's when the meeting should officially end okay got it cool so we'll do a few more I I want to you know what I'll do is I'll just pull in um some of the code that I previously have and I'll just walk through them and then you know we can have discussions and questions so um what I want to do is uh uh before you start I just have a question on the thing that you showed before uh so when you when you say um the the data goes into that particular model um so is there like a way to promote so suppose I have like a test environment staging production is there a way to promote that model along these environments oh but right now it's very limited uh keep in mind that this is a brand new service and preview from I'm sure that they will come with um those type of separations you can certainly currently the Restriction is that you can only do I forget if it's one or three you can only have one or three deployments per region per subscription there's a lot of restrictions like that but yes ultimately the advantage of doing it and Azure is that you can write like arm template or byset code and you can deploy that to different environments um that is not the I I don't think that's the case right now because it's you know again it's it's a previous service and they are still building it out but ultimately that is the goal and they have stated it so so uh probably you know if you check back in three months that that will you can probably do that so my advice is to start you know start writing your uh pocs and your development code against that and then uh probably prep for a launch maybe three six months from now foreign by learning and if I have to create a new model can I inherit from the current model so I don't have the I don't have to like uh you know train the new model can I inherit some of the features of the current model upon the new model or no currently there's not uh currently there's not a way to do that um okay it's whether that's uh uh that's an excellent question on whether that's an upcoming feature but currently I don't know of a way to do that then azure and I don't even know if openai allows for that uh I will have to double check but pretty sure you cannot do it in Azure currently all right cool so we talked about chaining uh functions we don't charge GPT uh and then so the next one I want to do is talk about skills so skills you know like we we created functions so far but you know functions are great but you know you want to templatize these functions and reuse them so that's where skills comes in handy so the way skills works is you have and you'll kind of see this you know you have uh skills directory and under the skills directory can have different skills so in this particular example what I'm doing is someone is visiting a particular place and so I'm creating visiting skills and the visiting skills could be things like you know you learn about flights you learn about the food at the location and you learn about the weather at the location right so I'm creating three different skills so these are how do I create these skills so to create a skill you need uh two uh two files so you know you have to create this type of folder structure uh you'll have a directory that can have multiple skills and under each skills directory you can have multiple functions so these are basically functions so let's take this function so this config.json uh you know these have some of the parameters that we talked about tokens temperature uh the presence penalty frequency penalty and so on so uh you know you you need a config.json file that has some of these parameters and then there you have a prompt file called SK Prime dot text and in this prompt file is where you know you can templateize this so this is a weather skill so I'm saying this weather skill I am visiting City and staying near landmark around timeline what will the weather be like so uh chat GPT is not um it it's going to refuse to give you live weather it's not going to go out and ask but it kind of give a historical pattern saying um you know there's a joke going around that today is April 25th so it's the perfect uh there's something about a perfect date because you don't it's not too hard not to call and anyway it's the name going around today right so it's going to give the weather based on historical patterns um so that's that's a weather skill and similarly I created a food skill saying I'm visiting City staying around landmark and I like a certain type of Cuisine provide the names of three highly ranked restaurants and near where I'm staying so try to be again as specific as possible and the flight skills I'm visiting uh this city from my hometown what flight option do I have so three to three different uh functions these functions I have I should probably call them functions folder so these are flight function food function and leather function and I put them under visiting skills right and how does this work in my program so in my program I do my usual Plumbing you know I do the namespace instantiate the kernel hook it up to Azure open AI um and uh then hooking uh you know using it to configure the uh or hook it up to Azure open Ai and then this is where it starts to get interesting so you know I'm importing semantic skills from the directory so I gave the name of the root directory and the functions that I want to use so I'm going to the visiting skills and then I'm using a bunch of functions from them then I'm setting context for these functions uh the hometown so the city that I'm visiting so let's say that's DC or Hometown is Orlando um The Landmark I'm staying around is the capital building I'm visiting around May 1st and I like Indian Cuisine right so um then you know I'm running the kernel and I am invoking the skill uh the I guess the functions one by one and you'll see that I provided the name of the root directory and the skills directory and then I'm accessing them as I would access the elements like a key value pair or something of that sort so uh you know I'm running I've been working the flight function the food function and weather function so let me switch to this folder and we will go ahead and run it in our integrated kernel itself server.net plan and what I want to show with this exercise was that you can create highly customized prompts um previously we did prompt chaining now we're creating uh sorry we did function chaining now we're creating prompts we can you can you kind of see how these things start to add up and you can build more complicated uh you know uh systems that use outputs of certain problems and you know templatized prompts and so on so this one in this case it went out and it gave me the options of different airlines uh and then it told me that you know here are some restaurants that's uh Indian food and then it says as an AI model I don't have access to real-time weather data but it gives me the historical patterns of the data uh it does it could be rainy so I think it's just predicted rain for you all on May 1st let's see how that works okay so that's uh you know templatizing problems we'll also do a couple more uh lots of dive into a couple more uh you know code examples and then we can take questions or if you have questions so feel free to go out and ask them um okay uh so um so far what we have done is we have used uh Dynamic uh skills right we have really created these skills uh literally during coding time and run it but uh semantic uh semantic semantic card also comes with its own set of core skills and uh that are pre-built so for instance it has a time scale uh it has a text skill that you can use to trim text left and right and other things so there's a bunch of these type of inbuilt built-in skills so one exam one type of skill it has is a conversational summary skill and this is basically it takes a conversation and then it will do things like you know you can call the different skills by using the key value pair type approach so you can call uh you can say that I want to summarize the conversation I want to get a different action items from the conversation and I want to get what the conversation what topics the conversation was about and this is a lengthy conversation that I also generated using chair GPT so you know it did a five minute uh transcript generated a five minute imaginary transcript of a call and what we're going to do is we're going to summarize this conversation and get action items and other other things from it so I will go ahead and run this month and the first one it's going to do is it's going to process that again all of these are hooked up to my open AI so you know it's still using that open air instance or it Banks it um should have switched to when they're on the folder shame on week get biggercourskill so.net Brown okay so it's summarizing my conversation so that's the first one and that's the next thing it'll do is it'll give action items so it provided a list of action items as Json output and thirdly it's going to list all the topics so these are the topics that were discussed so again you know there's another inbuilt or the core skills of semantic kernel uh that you can use uh so the way I look at it is you know if you want to summarize something why not just use a existing one that's well fleshed out has code written for it and so on so maybe you know that might be a good uh use case for this it also has a time scale that you can use to do things like you know or time of the day and so on so again you know that's uh you know rather than writing your own prompt maybe you can use this pre-existing skill to just do that because that may be more efficient because the you know it's well tested for things like using the optimal tokens and so on the prompt engineering part is also you know they got it down for this uh built-in skills so this might be a good use case for that uh the other one is you know text Trends so you know if you have like a text with spaces at the beginning and end you can say trim they can trim and and you can call this text skill to take care of that so that would be a good use case for that instead of trying to create your own prompt so just just some patterns to consider uh lastly I want to talk about something called native functions and but these are important because so far we have removed all of our functions so you know under prompts and everything else in English uh but what this does is it actually allows you to write your own C sharp code so you know this one is pretty simplistic so I call it my Mad Skills and I am um you know writing um I'm sending an input and it's calculating the length of the string so nothing fancy but you'll see that I gave it its own namespace right and then uh in my program I'm actually using that namespics so when I use that namespace and you know again give enter a movie and uh what it's doing is it is getting the uh I'm creating a prompt to get the most popular code from the movie um and then I'm in instantiating uh you know the code prompt as a semantic function but then I'm also importing my custom skill uh by calling it by its name um and a strongly typed object over here I'm importing that as a skill and then I'm running my code function and injecting that into my um injecting that into this mad skill so what it'll do is it'll take my movie give the code and this will calculate the length of the string length of the coat so uh you know and it's do you know you're doing using C sharp to do that so you can also write custom C sharp code for your skills um again and I'll run this but I think I'm kind of at the end of my presentation uh I will certainly take any questions I also have have one slide to show on responsible Ai and then you know we can drop off disk have further questions for the discussions maybe past sweeping statements about how AI is bad and will take over the world or you know anywhere you guys want to take it so you know I put in Scarface and it's getting the most popular quote from Scarface which would be say hello to my little friend and that is 32 characters long so that's uh you know not only did I write a skill in C sharp but I also did function chaining to uh invoke that combine a templated skill with a c I know native skill so uh or a llm AI prompt with the native skills so it this allows you to mix and match so you know again semantic function is kind of like a thin layer is abstraction it takes care of a lot of the plumbing so uh those are some of the reasons I really like uh semantic prompt and I feel like it can benefit C sharp Developers uh and you know I I already see this at work so I expect that many people are going to start seeing um um questions from Management on how to use chat GPT or any of the new AI items in your code and there are you know this is not the only way to do it but it's certainly an interesting uh library that you can leverage to do some functions so again consider the use cases you know use that uh you know just calculate the use cases see if it works for you and if it works for you definitely use it and finally this thing on responsible AI so you know these are kind of based on Microsoft students full of responsible AI so fairness you know treating all individuals without biased discrimination uh reliability safety make sure that you know you operate reliably and maintain user trust privacy is definitely very important inclusiveness so regardless of you know people's uh backgrounds abilities should be able to uh you know to cater to everyone and transparency I think this is very important and that you know if for instance if one of the things that is recommended is if you're using an AI to make any kind of determination you have to be very transparent in letting people know that this is what is happening but also accountability in that you know if AI has decided something there has to be some kind of checks and balances in place to make sure especially if you're making things like little decisions that you know like someone's College admissions or loan approval or something uh it would always be good to have a human double check that to make sure that it's you know AI is not just the AI That's doing the calculation is not just uh being unfair or biased on Sunday so some of the responsible principles so that's all I have for today happy to take questions uh the resources uh by the way uh I will put put this put these resources in chat these are the ones I learned from uh and if you go to these links you have a whole bunch of other resources that it points to I will uh tidy up my code and I will put it on my GitHub repo and I will post the link for my GitHub repo I really don't want to do that right now because uh it's not it's not anywhere close to complete or in good shape but this is my GitHub repo and I will put my code samples and everything else that in a few days that I'll take questions or criticisms anything else how can I improve on this talk happy to discuss anything that was fantastic so thank you very much thank you questions about embarrassing ourselves to the internet

Channel: CapArea User Group & NoVA CodeCamp

Views: 5,125

Keywords:

Id: xOy60FpgI2Y

Length: 106min 14sec (6374 seconds)

Published: Wed Apr 26 2023