How to Chat With Knowledge Graphs (python tutorial)

hello and welcome to this new part in our miniseries on how to use language models together with graph databases like Neo forj in my previous video I covered how we can use language models to extract entities and relationships from unstructured text documents and Json and then runs some python to automatically create a graph database uh from those entities and relationships now this video is the obvious cql the that one where we are now going to build a chat interface to that graph database where we will again use a language model to ask natural language questions uh then convert those into Cipher queries run those queries against the neo4j database and then finally answer that original question by using the results from the database and if you're new here I'm Johannes I'm a freelance data engineer and data architect and on this channel I create content on how to create Advanced language model applications from that data perspective and so if you haven't seen this previous video video that I just mentioned I highly recommend you check it out otherwise if you have seen it already then let's jump into this one I hope you enjoy and find it useful okay so let's get started we are here in our familiar repository from the previous parts here's the notebook with all the code to generate the knowledge graph again if you want to go through the Deep dive on that um do watch the previous part uh now for the UI and the chat interaction we're going to create a new file and we'll be using stream lit uh to build this simple demonstration application with a chat to our knowledge graph so we'll be importing that few things actually and then we'll start by defining some Basics uh for the for the website uh some columns to have a page title maybe some image and we'll just put a neo4j logo up there on top of the page and then uh we'll have the user input elements and then when the user does plus enter a question and then we are going to run some action and now if you aren't familiar with uh stream lits we're just going to run this now which we can simply do by running the command uh stream lits run and then the path to the file and then this way we can just very easily run this kind of a simple web application locally and uh do some testing and play around with it and then see the changes uh in real time and so now this starter application looks like uh this so we're going to start adding functionality on top of this basic frame uh the image is a little small but we'll fix that we'll make it a little more spacious for starters we're going to set the page configuration to use the wide layout option and then again if we refresh this we can immediately see the uh effect like so now for the actual functionality first first of all when we get a new question from the user we're going to add that to a session State object and what this is going to do is just maintain a history of the of the entire chat the conversation throughout our whole session and additionally when building these kinds of stream L applications it's a good practice to initialize these kinds of uh session State variables which just means that at the beginning of the application uh we initialize it as an empty list just to make sure that these variables are always defined when we start to refer to them and then next up we're going to start a timer so we can see how long it takes to handle uh each request and this going to require another input from the time it library and now we can start looking at the actual main function that we're going to use to query the graph and we're going to start defining it here mainly from L chain uh these are going to be really useful as you're going to see in a second because blank chain in fact has this um class for doing exactly what we're about to do which is to translate natural language into Cipher searches and then also translate the results back into natural language so this going to make our life a lot easier nice so first of all we're going to Define our language model wrapper using the Azure chat open AI wrapper from Lang chain and because we are using open as models here running on Azure uh here we are setting our end point and our credentials for uh accessing the API and the open AI API type that's going to be Azure as opposed to just using the open AI API directly and temperature zero to reduce the amount of Randomness in the responses and then also the deployment name which is going to specify exactly what model we are going to use and in this case we're going to use uh GPT 4 originally I tried to use 3.5 like we did uh when creating the knowledge graph but I found that for generating Cipher uh gp4 really performs a lot better 3.5 does make quite a lot of mistakes uh in the syntax uh so this is going to be very very useful uh also going to add some missing Imports uh to get the environment variables which I have stored here in the EnV file uh I go into more detail in the in the previous video on how where do you actually get all the credentials and how do you store them in the EnV file I'm not going to go through it again in this video but if you do want the deeper look at how you can get started using neo4j how you can create the database in in just a few moments and then how you can set up all the credentials both for neo4j as well as for using open AI on Azure I cover all that in my previous video the one I posted before this uh so you can go check that out and this we can actually copy from our notebook that we used previously these are going to be exactly the same we get the endpoints uh username and then the password and now we can start defining the function uh so first of all we're going to define the ne 4J graph object here that we import and it's going to take three arguments it's going to take the URL the username and then the password and this is an easy way for us to interact with NE forj through Lang chain similarly we're going to define the graph Cipher Q&A chain and this is going to take a couple more parameters it's going to take the language model first of all that we want to use for the conversation it's going to take the graph that we just defined here uh setting verbos the true and this is going to print out all the intermediary steps that the that the chain is going to do during the conversation and then we also going to want to uh return those intermediate steps in the final uh result that's going to be useful for us because we want to display those uh in the in the UI so that's things like the generated Cipher the results that we get from the database uh instead of just the final final answer and then finally we're just going to have a prompt for generating the cipher so there's going to be some custom instructions there that are going to be necessary and that's going to be it we're going to Define this prompt in a second but otherwise we're going to get the results uh by running the chain with the user question with the user inputs and then the return like I said it's going to contain everything we need uh to show in the UI so we're just going to return all of that and that's basically the function now where there is a lot of interesting stuff here is in this prompt so we're going to look at that next so we're going to Define that here and I'm just going to paste this here and then explain in more detail so the first line pretty standard you are an expert neo4j Cipher translator converting English to Cipher based on the schema provided and then we're also going to Define some extra instructions here to address some of the mistakes uh and confusion that I've noticed the language model does when generating Cipher so the first one we specify which version of neo4j to use five being the latest so we want to use that because of course over time syntaxes can change as with any language so we want to make sure that we don't mix different versions that aren't compatible and secondly we're going to mention some very specific keywords not to use uh because these don't exist in Cipher but they exist in other database languages uh like SQL and I've noticed that sometimes this causes confusion with the language models so when we specify not to use these words uh then that's going to help I noticed um to reduce this kind of hallucination quite quite reliably and these kinds of instructions can sometimes be a bit of a hit and miss sometimes the language model goes and does the mistake anyway but here I found that this actually very effective for getting rid of this issue next up another failure mode I've noticed is that sometimes the language model refers in the cipher to node types and relationship types that don't actually exist in our database so for example in our graph we have the has client relationship that's defined between the projects and the clients but sometimes the language model would instead reverse the direction and think that there is a pass project relationship from the client to the project now the this direction of the relationship doesn't have any impact on which direction we can search um regardless of how you define the relationship and its direction you can still search either way uh but the name still has to be what we have defined what's in the schema as well as the arrow in the syntax indicating the direction so this needs to be correct and so that's why we have the instruction number three to try and reduce this issue now number four here we Define that we want to do case insensitive and fuzzy search uh for any anything that we are looking for and so what this basically means is instead of looking for exact matches uh we are always wanting to use keywords like contains and this is super important because although the language model here has information of the schema of the graph it doesn't know exactly what each field contains in all the nodes uh so if it's going to try and look for exact matches like client ID equals something it's just going to try and guess exactly what the fields are and this is just going to miss most of the time so we Define that here and for the projects we actually want to use two different fields we want to look for things in the summary field and we want to look for things in the name field and we want to get the results if either of these fields contains the thing we are searching for instead of having an and keyword here so this is all just to make the searches more flexible uh so then we are more likely to get get some results for our search now these two last ones are basically just repeating some of these earlier instructions never use relationships that are not in this given and never use relationships that are not in the given schema that's basically the same as number three in different words and then number six just reiterates how to search for projects now this is admittedly pretty hacky but sometimes when you repeat these kinds of instructions more than once in the prompt then it makes it more likely that the language model is actually going to follow the instructions because like I said it's not quite as clear as just giving instructions and then being able to trust that the language model is always going to follow those instructions so this kind of repetition can also be useful sometimes and then here we have a placeholder for the schema of the graph that we're going to pass to these templates and then finally we're just going to do some few shot prompting which just means that we are giving some example questions and answers to help help Orient Orient the model and then finally another placeholder where we are going to pass the actual user question so that's the prompt and so you might get the impression already from the size of the prompt and all the elaborate Specific Instructions we need to give that generating Cipher is something that these language models even the Top ofth Line GPT 4 I really aren't the best at and they do still make mistakes even with an elaborate prompt like this we can only do so much and so this wouldn't fly in a production setting unfortunately but before you fall into despair there is a solution which is fine tuning and you probably heard of fine tuning at least in some capacity but what fine tuning is basically it's the process of training a base model further essentially making it perform better uh in some specialized task that your fine-tuning data set is focused on and so in this case our special case would be to generate Cipher and the good news is that this has already been done and share shared by Thomas bratanic who's a researcher at NE forj and he's been very active in this space posting a lot of great articles like this one already way before I was talking about this stuff uh this one is back in April last year and so what he's done here is he's fine-tuned an open source language model uh to perform better in generating Cipher statements and in this article which I'm going to share a link to he walks you through the process in in great detail uh but the bottom line for us here is his conclusion in the end which is that he feels confident that you could find to a model to generate Cipher um good enough for a production setting and so this is what we want to know if we are dreaming of making something like this in production in the near future and so I feel very confident that with the combination of finetuning as well as near future advances in the capabilities of the base models themselves in the next year or two years I think this is certainly not going to be an issue but for now because finetuning is a more involved process uh this kind of a prompt engineering is going to allow you to just get started a little faster and simpler experimenting with this kind of stuff uh so let's carry on with our example here so with this template defined we can now Define The Prompt itself which we're going to do with the prompt template class from L chain and this way or this extra step of defining prompts might seem a little confusing but the bottom line is just that lung chain and these uh chain objects defined by L chain expect these kinds of object objects prompt templates to be passed uh here so that's the reason why it's done this way and with that we are ready to send some results uh from this function back to our UI and then we're going to pause those results a little like I said it's going to contain a bunch of stuff that we are interested in because this result here is a dictionary and among other things it's going to contain a key with the intermediate steps and this is further going to contain our Cipher query that we can access like so and then additionally it's going to contain our database results which we can access here and then finally of course it's also going to contain our final answer under this kind of a key and so with the final answer just like with the user messages we're also going to save these all the responses into the session State I we just call it system messages and then we're going to append the answer every time we get a new answer and also for this uh session variable we're going to initialize it as an empty list and now that we are maintaining and building this history of the of the chat the conversation then we can Now display that conversation and so if there are messages in the history we're going to Loop through those messages and we're going to do it in a reverse order that's what these minus ones are for so we start from the newest message uh display that first all the way to the oldest oldest message in the history and this is actually a system message and then between these uh system messages we're going to show the users uh input and that should do it and let's see how that looks like uh in our application now oh we have a little uh import error here uh this one new 4J graph oh yeah this should be a lower case J and then this one as well now let's try that again refresh and there we go and now let's try a simple question U great except both of these are listed as the user sending these messages so we made a mistake somewhere but but we'll fix that oh yeah this should be the key uh so what we're also going to do here is we're going to create three columns in the UI and then this way we can have the chat history in the First Column and then in the second column we're going to have the cipher query that was last generated and then finally in the third column we're going to have the last retrieved database results and then that's going to give us some nice visibility behind the scenes what's actually being done under each query like so beautiful now the issue that some people do find with Lang chain is that although it can help you get started very fast when you want to do some customization to how the components work that can be a little tricky their documentation isn't always up to date and it is a very fast moving Library so it can be hard to understand uh what exactly this component here is doing behind the scenes for example if we wanted here to customize uh how the final answer is formatted uh from the language model uh how would we go about doing that because this prompt here this is only used for generating the actual Cipher but then when we get the database results and we use that to create the final answer uh then this prompt uh isn't isn't used anymore in that step so even if we added some instructions to this prompt let's say give your answer in bullet points or a table or German and then then it wouldn't matter to the final response there are two different steps and they use different prompts uh so if we go to the Lang chain documentation for these component we can actually find out that there is another argument aside from the cipher prompt there's also a Q&A prompt uh that we can pass to this uh class and then there is no explanation exactly what this uh argument does in the class but we can assume that this probably does what we want uh so what we can try and do is we can pass a prompt like this this and then just do something similar as with the cipher prompt create another prompt template with something simple give your answer in bullet points and then we don't we don't Define any input variables let's see how this works going back to our UI and now we run into issues uh we can see that the cipher works fine we get the correct response from the database but for some reason this information doesn't get passed onto the language model and therefore we get this kind of an answer and the documentation doesn't really help us here because like I said there's no information about this Q&A prompt whatsoever and then this is the point where especially a lot of beginner people run into a frustration they don't really have anything to go on and what they will often then do is try and Google find out if somebody else has run into the same issue if somebody else has done the same thing and sometimes you just can't find an actual example of how this is used but what's also useful to understand is that we can actually find all of the components and all of the classes in L Chain by going to the L chain GitHub repository and this might seem obvious to some but again for more beginner people who are just used to importing a library and using some functions and not really having any additional questions beyond that uh it might not be so clear and so I wanted to use this opportunity to talk about this for a little bit and show how you can navigate at these kinds of Library repos and find the components that you want to look into in more detail and so here we have the libraries under the libs uh Direct directory and they also as well as the main L chain Library they have the experimental library for more still in development features um what we are looking for is in the Lang chain main library and where exactly we find the component here we can see that by looking at our import uh in this case this is the component we are interested in so it's l chain. chains so it's the subm module chains in the main L chain module so in the repo if we go to L chain here we see the chain module and then here in the graph Q&A we can find what we're looking for so under the cipher py we can find uh the Q&A chain and then looking for the specifications for the Q&A prompt and we can find that under this class and this part here is going to tell us that the chain is going to use the Q&A prompt that the user passes if they pass it but if they don't then there's a default prompt that's used and so where exactly we find this one then we can see that this is also imported from yet another Lang chain library or sub module uh so we're going to have to do a bit more digging still going under chains and graph Q&A and prompts so it's in this same folder in the prompts and then somewhere here we can find the default Q&A prompt that's being used and here it is uh it's pretty generic for the most part it's a good decent prompts but most of all what it's going to tell us is is going to tell us the expected input variables that we should pass so it's expecting context which is the results we get from the database and then the question from the user so with this information we are now in a better place to then Define our own custom Q&A prompt because we can just copy this into our own code and then we can make any necessary adjustments here we'll keep the expected uh we'll keep the expected input variables that are going to make sure that our database results and the user question will get passed on to also the Q&A or the final anwers step and then we can also still make any additions or changes to this prompt here for example we can add here that The Final Answer should always be in Spanish and now if we try it again then we get the result we want now I hope that little detour to Lang chains inner world was useful but I think in the context of working in this kind of a fast moving and Cutting Edge space uh it's good to have this understanding because you can often be working with these kinds of niche librar where the documentation isn't always perfect and they are changing a lot uh it's good to have this understanding that all the code typically is going to be available in a place like GitHub and you can actually navigate it find all the classes all the functions in full detail see what they do and then you are in a better place to understand them and then make any adjustments as you need for your specific use case Okay so just jumping back to our code for some final touches we are basically finished here but there's a few more things I want to add first of all we are going to add some uh exception handling in case we get some errors with the cipher as sometimes happens uh so we're just going to have that here put this in a tri block write out the exception and then we're going to put this under a spinner element uh which is just you'll see in a second um it's just this visual element in stream L to indicate that something is being done and so we're going to see that visual elements while uh it takes us a while to process all of this call the language model call Ne forj Etc and then finally we're also going to use the timer we set here to show how long each question takes to answer and then finally we're going to take away this uh Spanish line and uh we're just going to ask for the answer to be easily readable and structured and let's see what that's going to give us and then I'm just going to ask one more question and this is the spinner element that I mentioned uh because this default running indicator up here it's very easy to miss and it's nice to give users some indication that something is being done and here we go now we are getting the answer in a slightly more structured way the paragraphs are being separated nicely uh with our additional prompt and that's going to wrap up this tutorial on how to create a chat interface to a graph database

Channel: Johannes Jolkkonen | Funktio AI

Views: 9,363

Keywords:

Id: Kla1c_p5v0w

Length: 24min 26sec (1466 seconds)

Published: Thu Dec 07 2023