Make AutoGen Consistent: CONTROL your LLM agents for ACCURATE Postgres AI Data Analytics

the channel has absolutely exploded after our previous two autogen agentic postgress AI videos while this is great I got to be honest I don't really care about the views I care about building a gentic software and I care about helping you push your engineering into the future of programming this means using the best tool for the job and right now as you know we have a whole new playground of llm driven AI tools to explore experiment and build with we're building real valuable software with autogen AER Advanced prompt engineering all on top of gp4 I say this to set the stage and welcome everyone new to the channel but also to make it absolutely clear this channel is about finding the best tool for the job it's about raw engineering and it's about true product Centric value creation through software this is not a hype train overly edited Channel or report the latest overhype GitHub repost or AI related news drop every single week there are many channels that do that you know where to find them if you want them on this channel I share Tech that I truly believe will help you solve your engineering problems as for all the views and attention and you know uh new subscribers really appreciate you being here but I'll be here every single week with serious skin in the game just like I was when I was getting less than 100 views per video that out of the way huge thanks once again I really appreciate you being here now let's build some agentic software all right so first things first let's recap our current postgress agent so I'm just going to go ahead and run a full example of our application I'm going to kick up our poetry environment and then I'm going to run our program with a prompt give me all Outlook email users so in natural language we're going to query our postgress database you can see here we have the table definitions at the top we have users and jobs and our current flow of llm Agents they're going to read from our postgress database figure out what query needs to be run run it and then as you can see here they're going to report the results I asked for all Outlook emails you can see here we got Joe and we got Jane Smith if we hop over to table plus and we look at all of our Outlook emails you can see we have two Joe and Jane so we correctly got the right results back and how do we accomplish this we utilize a couple of environment variables here we have our database URL and our open AI key we pass in a prompt using AR parse and we have a postgress manager I check out the previous two videos to catch up I'm just going to flash through this we have our GPT config that we pass in the autogen we have a single function map that gives one of our agents the ability to run SQL against our database we then have prompts and we have our flow of AI agents that perform specific tasks to help get the job done and as you can see here at the bottom we set up Auto jens's group chat and group chat manager and then we initiate the chat with all of our agents and this is how we're able to type in natural language and get results out of our postgress database so this is really great but as you may have found working with autogen is not a simple walk in the par especially when you start running your own unique agent combinations each with their own detailed prompts and functions I ran into similar problems and found it quite difficult to consistently get the output I was looking for when I was building out more detailed and more advanced agents with their own functionality in this video I want to share two architectural patterns I've used to largely solve these problems with consistent agent conversations and consistent function calls and I want to show you a glimpse into the future and and see how we can use these two patterns to truly start building out real agent systems that can solve your engineering problems so the first thing we need to do is get control of our agent conversations how and in what order our agents Converse to solve our problem is critically important right now as you can see we're relying on auto Jen's group chat and group chat manager the backbone of this application is of course open ai's GPT 4 model and Microsoft's autogen which is a framework for building multi-agent applications shout out to everyone who's worked on this from the contributors to everyone at Microsoft Penn State and University of Washington so right now we've been relying on auto Jen's group chat and group chat manager to direct the flow of our conversations and while this is a great starting point to really get control of how our agent system runs we need to get closer to the metal if we dig into the group chat manager you'll find this run chat function which essentially controls the flow of the conversation it does a decent job for specific use cases but it's not great for other use cases and I found it not great at consistently calling functions when it needs to so to improve this we're going to code our own group chat like class that we'll call the orchestrator I don't want to burn up your time I'm going to speed through this to show you exactly how this works okay now we have a brand new orchestrator class that can control the flow of our conversations between our agents so we're going to go into more detail here in a second but first let's just rerun our poetry command give me all Outlook email users and let's make sure that our new orchestrator flow works so just like before we have the table definitions we have the admin passing it to the engineer we have the engineer generating the SQL that needs to be run and then we have the SQL analyst reporting the results to the product manager and as you can see here we have our results and we have an approved message and we have our orchestrator reporting success as you can see here we now are assembling a team we have our data engineering agents we pass them to a new orchestrator class and then we take this orchestrator class and we kick off a specific conversation type this is really important we kick off a sequential conversation type and if we dig into this function here we can see that what we're doing here is passing the message from A to B to B to C C to D and D to e between our agents from start to finish add a prompt to our list of messages we can hop up to the definition here in the a nit basically the core of our orchestrator is this we have the name our agents and the messages that are spoken between our agents we then have a bunch of properties we can dive into these one by one quickly we have total agents last message is a dictionary this allows us to know if the last message we're getting is a function call or a function call response we have last message is string and then we have last message is a function call so this is gp4 saying we need to call a specific function and then last message is content is saying we just called a function get the result from the function and then of course we just have latest message it'll give us none or the actual result that just occurred and this is optional string this should actually be a union between dict and string but I'm just going to leave this like this for now so if we jump back to the sequential conversation you can see that we're basically just running this flow right if the last message was just a string just continue the chat conversation right we go from A to B but if the last message was a function call we instead of doing this basic chat where we just pass a message from A to B we run this function call chat which allows a and I'm actually going to call this function chat just to keep it more accurate and contextual agent a to communicate and call its function before passing the message to Agent B and we do this check so that we know that if agent a has a function we expect it to call that function this adds some really key logic where if your agent has a function and you want it to run make sure that it always runs we go up to the second to last agent because because we don't want our product manager at the end of our list to pass its message to someone that doesn't exist right and then we check our complete keyword which is just approved thanks to the completion prompt so if everything looks good product manager will just respond with approved right and that's basically that let's quickly dive into the basic chat I just want to show you exactly what this looks like so that you can replicate it we're taking two agents and a prompt message we're mimicking conversation so agent a is going to send this message to Agent B so it's like agent a is saying something to Agent B we then have Agent B generate a reply this is where the open AI gp4 request actually occurs and then we're just adding this message to our list of messages and the only thing our messages list is doing is just keeping track so that it's easier to debug when you need to go in and see what's going on and what has been said between agents and then the function call is where things get a little more interesting we have a basic chat between agent a and itself to basically continue to trigger the function call that needs to occur and then after that's complete we have agent a pass the content from the function call to Agent B with open AI function calls it first needs to prompt the function call and we can see this in our working example the senior analyst first suggests a function call and then it executes the function call and so in the suggestion we need agent a to send that back to itself so that it can then execute that function call and send the response of the execution to B so that concludes our orchestrator basically what we're doing here is we're getting control over the flow of conversation between our agents and if we want to see exactly what this looks like at a high level the sequential conversation is just this agent a passes a message to b b goes to C C goes to d d goes to e and e goes to F just exactly as you would imagine right after agent e to F finishes the conversation ends so I hope you can see see the importance of needing to control the conversation right if you're not if you're not in control of the conversation that your agents run and you're not directing the flow with an orchestrator you're just up to the mercy of the functionality that you know autogen group chat and group chat manager forces you down right you want to be building your own specific flows to handle that specific flow of logic right so let's keep pushing this forward now that we have a better pattern for the control over the flow of our agent conversation using our new orchestrator class let's give our agents more unique functionality let's say that after we generate this um data orchestrator result right we want to do some reporting and like data visualization on our postgress response right so for example just to keep things simple let's say we want to Output the result to Raw text Json and yaml all together so let's say you know we're generating a bunch of different types of reports maybe our clients want different reports of the data right or maybe we're Genera ing the uh data for our product managers and they want it in a couple different formats for whatever reason with our new conversation flow in our orchestrator we can build a new type of conversation where we essentially just broadcast out what we want our agents to do right so let's take a look at a agent conversation that handles message broadcasting versus a sequential conversation flow right so if we hop over to a broadcast conversation where we have agent a just saying Hey I want you guys to uh report this data in this format we can then have Agent B just be a raw text agent while agent C is a you know Json agent which handles and reports Json data and then agent D can be you know yaml and then you know so on and so forth so with this you can kind of see how having specific conversation types really matters when you're putting together your agents it's not just a free-for-all because passing information along like this through an organization through a system of Agents through a through any system is very different than broadcasting it out right let's see what this really looks like so I'm going to build this new broadcast conversation flow and then we can take a look at how that helps us build more concise agentic systems all right I'm just finished this is where things get really cool let's collapse our code and look at the changes we made here so first things first we added a brand new type of conversation to our orchestrator we now have a broadcast type conversation where agent a just yells out at all the agents hey I need you to do this and then they all do it in their own unique way right if we hop back to this chart this is what this looks like right it's a different conversation flow than our sequential flow where we're passing information along in this conversation flow we're broadcasting a message out to all of our agents as you can imagine we have the broadcast agent just yelling out its message using basic chat and this should actually be a memory chat and the only difference between the basic chat and the memory chat is that memory chat will have Agent B Store its own reply when agents generate a reply they're not actually saving this in their own personal memory then if we pick up a function call agent iterate which is you know our base level agent here Agent B C D E and F it's going to then given that information run its function right and we're going to basically assume that it's going to be a function call and so there's no one else to pass a message to in the function chat so we're just going to send the message to itself which is totally fine it's not going to do anything with it this is our new broadcast conversation type and if we jump up to the top level we have a couple new things here let me go ahead and expand this and you can see previously we had our data orchestration pipeline here data engineer orchestrator which contains all of our agents and then we're running this sequential prompt I'm going to go ahead and just leave this commented out for now because we now have a brand new pipeline here that I want to show you we're doing two things here we're expanding the types of conversations our agents can have and we're going to give our agents a bunch of new specific functionality as you can see in the prompt we have a text file report analyst a Json version and a Amo version and the prompt is simple you exclusively call the function you've been provided to call right we then create three new agents respectively and here's the really cool part you're going to specifically provide the type of function you want that agent to run and it's exclusive to that agent right so this is how we get really cool agents really built out agents running specific functionality solving specific jobs just as I said in the previous video you want to keep to that clean Unix philosophy do one thing and do it well this is a Json file reporter agent it just writes Json that's all it does so here we're going to build out a mock data to report this is just a run from our previous agent flow with some results so that we can pass it and test our new data visualization agent team and you can see here I'm putting together the agents we're just going to go Ahad and use the user proxy as we did before but then as our receiver agents we're going to set up the text Json and yaml Report analyst we then wrap them in the orchestrator class with their team name pass in the agents and then we create the prompt here is the data to report pass in that data from above and then we run our broadcast conversation one last thing to mention I quickly just imported this file class which contains the specific functions to help our agents write so we have the right file we have the right Json file and we have the right EML file and that's what we pass into to our function Maps which have been expanded as you know we need a configuration object to be passed into our agents so that they know what functions they can run as well as a function map so we've broken apart the base version of the configuration to be passed into agent specific configurations so you can see here we have the Run SQL configuration which contains the mapping for the front SQL function we now have a right file config which contains the definition for the right file in our file. py you can see that here and then moving along we have the right Json file and we have the right yaml file and if we go down to our agents of course in their respective llm config you pass in the proper configuration file and their own specific configuration Map There's a much better way to organize this information this code right now I'm just doing it in a single file just to keep it really clean really simple really obvious we just go and collapse all this you can see we have base config which builds out the specific configs and then we have the individual function Maps we then have our prompts our agents and now our new set of agents and prompts the report analysts and then those agents we have the mock data to report data VI agents where we build a list of our agents we set up our orchestrator set up a brand new prompt and now we're going to run the broadcast conversation so this prompt is just going to be skipped over make sure we import our file module here let's fix this lowercase true need to make agent agents so we can get the first broadcast agent here from our list of Agents we need to change the self latest message to prompt when we're passing our message from our broadcast agent to Our receiver agents in the broadcast conversation flow we need to send the prompt to every agent not the latest message let's go ahead and run our data Vis agent team so the output here should be three files we have a text file we have a Json file and we have a yaml file okay so we can see that we had the right file run happen and now now the admin's talking to the Json agent that file right went through now we're talking to the yaml agent and let's see what we got here so okay so it looks like the yaml agent had some error looks like a Json error I'm not going to dig into this exactly I'm sure there's some legitimate reason for it but I just want to prove the point of setting up agents that can broadcast messages out right so here's a user report so we did get that clean right to the user report text file and here's the user report as Json you can see here we have this data V team operating on specific types of data right so this is pretty cool but most importantly it shows how important controlling the agent conversation is to really get control of the flow of conversation between our agents we need to build our own conversation flows out with some type of orchestrator structure and the real time will be spent in understanding how to send prompts and run functions between your agents there's definitely a good argument for doing all this manually and just cutting out autogen to keep it simple but I think autogen gets you a decent way there with the conversation B agent and the base agent class that can help you and you know again the beauty of code is once you kind of have this working um in a clean modular fashion you can solve this problem forever right like these agent pipelines can be reused to solve any configuration of problems between agents right that's going to be the next push and the next area to really focus on after you understand you know basic prompt engineering you want to push your agents to the next level so that they can autonomously solve problems for you the next place to go is going to be to understand this process this idea of orchestrating the conversations that your agents have full complete honesty and transparency it took some time to build out this class the exact right way it needs to be to run the agent flows but again after you have this in place after you have your mode of orchestrating Agents it's there and you've done it and it's reusable uh I hope you can see where this is going right let me just do one last thing in this video I know this is a super long video but hopefully I can edit it down get it as short as possible for you guys so you can get the maximum value and the least amount of time now what we're going to do is put our two agent teams together so we get the result back from our sequential conversation what we're going to end up with is success and and then messages right so I'm going to call this data in messages we want to get the second to last message which has our response from our senior data analyst which is the SQL statement right so data and result and this is going to be -2 and it's going to be in the content part of that dictionary right so this is the SQL response that our data analyst gets from running the Run SQL call and now that we have this which is essentially going to be our data to report we can do something that functional programmers will love we can compose our agent pipelines our agent conversations together we take the result from the data engineering team and instead of using this mock data I'm going to comment this out we're going to use the data Ang result right so with that done we can go ahead I'm going to code collapse get rid of our mock data to report and now we have our data engineering team writing content directly into our data visualization team I'm going to update this give me all unof users right just keep it simple and now we're going to run the entire flow right so we're going to have team one hand up the result to team two so already you can see Team One finished right our data engineering team finished they handed the result to our data visualization team and now they are writing their content respectively to you know whatever output form they're supposed to write to okay and if we look oh look looks like we did get that yo right open up our users report right uh that our team of Agents created for us um you know we got authenticated is no so we'll you know we got both of our um unauthenticated users we can check our database here in table plus we can see we're expecting Bob and Charlie and we have both Bob and we have Charlie we can go to Json now we have a Json format we can work with you know this will be a good good file to just like pull data out from really quickly and then looks like our for whatever reason our agent uh you know pushed through with our yaml response and we have the same thing right so here I'm going to paste in that entire run and let's quickly just look over everything that just happened there right so give me all unof users all right so first things first we instantiated our data engineering team and we're going to run with these iterations right so we can keep track by searching these iterations but first thing first the admin to the engineer passes the table definitions right the engineer then generates the SQL needed and then on the next iteration our agent is now talking to our senior data analyst remember this is sequential chat goes top to bottom and it passes the SQL and it's saying you know execute this command on the next iteration we have the senior analyst reporting to the product manager suggested function to run the senior analyst is going to then run this function first and then it's going to send the result to the product manager the product manager then in the the sequential conversation confirms the result with the approved and then this marks the end of that team right so the results complete and then we pull the result from the senior data analyst and then we kick off our data visualization team right on the first iteration here you can see that the admin is the broadcast agent and the iteration agent is going to be our text report analyst right we have Bob and Charlie both unof then our text report analyst is going to run its function right to a file we get none here because the function you know right file it returns nothing on the second iteration again this is a broadcast conversation so our admin is now broadcasting to the Json report analyst and it does the same thing for Json our Json report analyst writes the Json file and then lastly we have conversation iteration 2 index 2 broadcast agent admin is now uh you know saying to the yaml report analysts do the same thing and then our yaml analysts WR that file so that's the overall flow that is exactly everything that's happened here debugging and logging is going to be really important so I recommend using you know believe B logging and writing out to a file so you can see exactly how your agents and your different teams are working together and this is all through our orchestrators right this is all through our prompt engineering work this is all through our composition work this is all through our conversation work right building blocks engineering is all about building blocks you know this already these are the next level building blocks right when we have systems that can build for us this is a new paradigm shift for the way that we engineer I hope this makes sense I know there's a lot of complexity kind of abstracted away in this orchestrator class it's really important to get your chat down so that you're passing the right information between agents it's really important to set up your own specific conversation flows if you take nothing away from this video I really hope that you take that the order and the conversations that your agents are having really matters and you need to control when they're just chatting and when they're calling functions so our orchestrator is really important I'm going to be reusing this class building upon it setting up different types of conversation flows that can solve different problems in this video we just use two conversation flows these are probably going to be two really common scenarios you're going to see right top to bottom sequential and then broadcast conversations where we just fan out the information from our primary agent to all of our other agents and at the center of the center we have Auto gen which which is helping us out with our agent specific functionality then again at the center of the center of the center we have GPT for llm technology helping us drive arbitrary content generation and you know kind of wrapping that layer we have good prompt engineering right good concise simple prompt engineering we built out two flows of Agents two teams of Agents right we're separating we're we're dividing and we're conquering right so we have the data viz team that takes input directly from the data engineering team or data visualization team which you know the de VI team right now was just writing to files just want to prove a point that you can set up systems of Agents teams of agents to do specific things right again let's remember orchestrating our llms gives us superpowers to recreate any organizational structure right let's open up the read me just like we were talking about in the last video right this is important because it allows us to create a more accurate model of the world and allows us to become orchestrators enabling less engineering and more product level work right that's what the orchestrator is all about we got some agents going guys we're building some agentic software I've been really passionate SL obsessed uh with where this is all going and where this can take us there's a lot of rough edges to this technology it's really non-deterministic by Nature so it's kind of hard to work with but once you get your flows going once you get your nice orchestrator going the results are a lot more consistent a lot more consistent than you would think all great things uh require effort grit focus and patience on this channel we're going to the edge we're pulling the future of engineering into the present so we can build agentic software that works on our behalf faster than we ever could and even better they can work with us while we're working in parallel a lot of talking a lot of content in this one again thanks for all the views all the subs all the likes on the previous couple videos like I said in the beginning I'm just going to keep rolling 10 views 100 views 1,000 10K 100K I don't really care I'm going to keep shipping this type of content because I know that there are a couple key people watching this might be you that are going to take this and push it to the next level and start building out your own agenting software so in the next video I want to get back on the original track we need to expand the number of postgress tables that our agent can consume and as you know there's a context limit to the window that we can pass to our llms using gbd4 whatever likely we'll be using Vector embeddings stick around for that it's going to be a real Hands-On way to use Vector embeddings to get concise results I made it to the end of this video and got value out of this drop a like sub hit the notification Bell and I'll see you in the next one

Channel: IndyDevDan

Views: 29,190

Keywords: microsoft autogen, autogen, ai agents, llm agents, postgres ai, ai data analytics, ai engineering, agentic engineering, agentic software, agent conversation, aider, gpt-4, openai functions, consistent prompts, consistent llm, prompt orchestration, agent orchestrator

Id: 4o8tymMQ5GM

Length: 28min 57sec (1737 seconds)

Published: Mon Oct 16 2023