3. Prompting AI Agents - SCAI017

Hey guys, so today I wanted to talk about prompting AI agents and obviously specifically within N8n and it's really interesting because prompting is kind of a black box and you may have heard some other people say that and it's very different. It's a huge testing part, both testing during development, after development, all that kind of stuff. When you switch out the models sometimes they interpret the system prompts differently so you know there's not a formula but at this point with all the different use cases and the different tools and all this kind of stuff I feel like I've got a pretty solid framework of the way you go about prompting and one thing that's really important to keep in mind, not just prompting, whenever you're building agents this whole philosophy of building things, focus on building the foundational skills of creative problem solving, testing and refining. These are the skills that are going to go farther than focusing on oh like which tool should I be worried about, N8n or make, what if one of them goes away and I learned the wrong one. It's not about that, it's about workflow logic, prompting, problem solving, understanding models, understanding how they work. These skills are the ones that are going to be helpful over the next few years and the ones that are going to separate people when it comes to developing these types of solutions. So what I want to talk about here, the main thing I want to talk about today is proactive versus reactive and this goes to not just prompting but also building things. But yeah, let's break it down. Anyways, let's just get into it. I'm sure I'll ramble a little bit here but either way, so if you're new to agent building and you're new to prompting, what is it? Within your agents you're going to give it a system prompt and the system prompt is basically just you're coding the agent but you're using natural language to code it rather than Python or something. So general rules, you want to keep the prompts simple, very clear, very specific and ultimately you don't want to make it too long and that's for two reasons. One would be just be like the tokens that you're sending over to these APIs and second of all, if you were explaining to an intern how to behave, this is your role, this is what you should do. The longer it is, the more confusing it's going to be and it's just going to be less effective. So keep it, what do they say, kiss? Keep it simple, stupid, right? Anyways, adopt that framework or that methodology. Anyways, so real quick, just laying the foundations here, be specific, clear and actionable, don't leave room for confusion and this is because when you're talking to chatgbt and you say like, hey, read this transcript and then help me analyze it, it'll give you an output and then you can say, oh, let's, can you focus more on this aspect and then it will come back again and you say, oh, can you make that shorter, it'll come back again. We don't have that luxury with agents. We are under the assumption that we're going to put in this prompt, we're going to give it tools and then it's going to be able to run on the back end a thousand times with consistent outputs. That's the goal. So we have to test and refine this prompt and have trust in it. We don't have the ability to go back and forth like a conversation. So obviously a well-structured prompt is going to help them perform their tasks accurately, autonomously and consistently. So that's kind of the goal here. Yeah. Unlike chatgbt, you can't ask follow-up questions. Okay. So this is the biggest thing I learned over the course of the past couple of months, building these AI agents is proactive versus reactive building and prompting. So I don't know, I'm not like a software developer, but most of the time in life, I feel like when you're doing things, you kind of behave proactively or you want to be proactive about things that are coming up and the way that you're approaching different strategies or whatever. But I think when it comes to this kind of stuff, you want to behave the exact opposite. You want to be reactive. And this is because it's going to save you a huge headache. It's going to make things very, very clear as far as what's my current task, what's my next task. When it comes to, a lot of you guys have been asking about how do you find the errors and you debug by, by reactively building things, it's going to cut down on that debugging confusion because you know exactly what caused issues. So proactive means that you're writing a long detailed prompt upfront, you're pasting it in there and you're trying to hit run on your agent. And maybe this is my fault because of that custom GPT I released where you could explain to the GPT, this is what my agent does. This is the tools it has, help me write a prompt. And, you know, it's, it's, it's cool, but ultimately that's not the way you should build is, you know, you don't want to connect all these tools and then throw in that system prompt. You want to take it step-by-step. So some downsides initially of proactive prompting is you don't know what are going to be the common errors. You don't know all the possibilities. And so you want to be reactive because at that point, you can just basically react to whatever just happened. Let's prompt it to not do that. If we're seeing common errors, let's prompt specifically those errors. So I'll dive into exactly what I mean. Sorry, this, this just gets me fired up anyways. So by reactively prompting, you're going to have a step-by-step approach. So when I get in there and let's say, we're going to look at the example of my ultimate assistant, which in this case, right, you've got four agents, you've got a tool. You would think the prompt would be pretty long and robust, but this is all I have in there. And it's super effective so far. Anyways, keep in mind, this isn't like a client build, so it's not as robust, but pretty robust. I start with nothing at all in the prompts. I'll hook up one tool and I'll run it. And a lot of times the agent knows to use that tool before I even prompted at all, which is cool. I'll add another tool. I'll keep running it multiple, multiple times until I get some errors. Once I start getting errors, then I start to add in the prompts to directly counter those errors. So obviously I'm going to start with an overview. So when I built this personal assistant, I gave it an overview and said, you're a personal assistant and you have these tools, right? But that was pretty much it. So this makes debugging more, way easier because you know exactly which new sentence or which tool broke the automation. Because if it's running well, and then you threw in a whole system prompt that a GPT operated or GPT created and you hit run and it was breaking, you would have like, I don't know, maybe 50 sentences to look through and think, okay, which one here did the AI misinterpret and caused it to error? That's bad. So by adding a sentence one by one, let's add one tool. Let's add one action of this is when you use the tool. We'll hit run. If it's working, we know we're good. Then we can add another tool and add another system prompt sentence about that tool. If it errors there, we know that this line killed the automation. Let's take out that line and let's rewrite it. Okay. So that's kind of the methodology to have both in building and prompting. So I'm going to show you guys exactly why I built this prompt. So real quick, let's read it out. Ultimate assistant, right? That's the overview. Your job is to send the user's query to the correct tool. You should never be writing emails or creating event summaries. You just need to call the correct tool. So the ultimate assistant's job is to delegate and take the query from the user and just send it to the right tool. Here are the tools you have, email, calendar, contact creator, and tably. And then we just said, use this tool to blank. And it's very, very clear. Use this tool to take action in email. Use this tool to take action in calendar. That's as simple as we're giving it because there's different steps going on here. The first step is understanding when to call which tool. And then the second step is the data transfer between parent agent and child agent. So that's what we're going to prompt in later. First step is just calling the tools correctly. And so I have rules in here. Some actions require you to look up contact information first. For the following actions, you must get contact information and send that to the agent who needs it. Sending emails, drafting emails, creating calendar events with attendees, because that's something where we need the email address of the person to do that action. So we listed that out there. And you may be thinking this is kind of specific stuff. If you were proactively doing this, you wouldn't know to build in these specific guidelines because you just wouldn't think that those scenarios would happen. So by being reactive, I saw an error, and I directly countered it. So I'll touch on this later, but this input output, this was an exact error. So I was testing and I texted on Telegram, the assistant, and I said, send an email to Nate Herkelman, asking him what time he wants to leave. What happened in this case was it didn't get Nate Herkelman's email. So then I basically just baked in an example. Okay, this is the input you get. What you're going to do is you're going to take action with the contact agent to get Nate Herkelman's email. Then there's another action. Use the email agent to send an email. You will pass the tool of query, like send Nate Herkelman an email to ask what time he wants to leave. Here's the email, email address. And then the output, the email has been sent to Nate Herkelman, anything else I can help you with. So that was an exact example that I had tested and it wasn't working. So I threw in the actions of what to do. And this is only one example. If you're getting multiple errors, throw in three or four examples and do your actual testing data. Put it through, put the actions it should take and put the output you want. And that's how you're going to be able to train this thing on these different scenarios. So touching on what I just showed, right? When I was building the assistant, at first it was incorrectly modifying emails before passing them to the email agent. So the issue was, I would say, write an email to Michael Scott asking him how the project is coming along. It wasn't just sending that query to the email agent. The ultimate assistant agent was trying to make that email. So it was passing over a two field, a subject field and an email field. And this was messing with the way that the email agent was processing the data because the email agent was supposed to just be getting a query. So that's why I added this specific line into the overview, which is your job is to send the user's query to the correct tool. You should never be writing emails or creating event summaries. You just need to call the correct tool. So I threw that in there and it worked. And then I also, like I said, I threw in an actual example where we're showing this as well, because then I said, you will pass the tool, a query, like send Nate Herkelman an email to ask what time he wants to leave rather than sending over a query, like two subject email bot. So I hope that makes sense. So, because I did that reactively, three things. First of all, I knew that that exact sentence was fixing that error and I knew why the error happened. So I added a single rule instead of having to rewrite the whole system prompt or try to figure out where in the prompt I should delete some stuff and put more stuff in. I could just add it because there was nothing much really to work off of yet. And then because of this, the agent learned what not to do and it didn't break anything else. Because sometimes if you have a huge prompt and you're changing things out, you may fix your first issue, but then a new issue is going to pop up. So this is why you want to do it like this. The other example, my AI agent was calling the email tool before calling the contact lookup tool. So all we had to do is we just knew that we had to come in here and give it rules on when to call contact tool, which in this case, what I did was I saw the error and then I reacted by adding this rule, which says some actions require you to look up contact information first. For the following actions, you must get contact information and then send it to the agent who needs it. So I listed these three events that could happen or actions that could happen. And in all of these cases, it needs to hit contact agent first. So sometimes it knows like, okay, so here are my tools. I have email agent and I have contact agent. And I know that I need to get Nate's email and I know that I need to send an email, but it may hit the email agent first. Or if it does it simultaneously, it's going to try to send an email before it gets the contact info. So by specifically saying call contact agent first, boom, that cut out that error. So the moral of the story is rather than guessing what the errors could be upfront and trying to prompt everything at once, be reactive and fix them one at a time. Because the reality is you're never going to build a workflow and not have an error. That's a huge part of the process. It helps you learn how things work. It helps you learn how to debug. And it just helps you understand like what goes on within each node and how data transfers. And you need to really get good at like reading the logs as well. Because I was able to read the logs and see, oh, in this case, it called email agent first rather than contact agent. So being able to see how the agent reasoned. And then it's as simple as let's say you're trying to teach your kid how to ride a bike. Okay. Maybe that's not a great example because proactively like you're going to do things like put a helmet on him, probably give him training wheels and like get ready to hold him. But reactively, you're not going to give him specific advice until you see what he's doing wrong. So if he's leaning to the left, you're going to tell him, okay, center your weight, lean a little more to the right. You're not going to just immediately throw him on a bike and say, lean more right because then he's going to fall to the right. Anyways. So it saves time because proactive approach is going to be, you can get really frustrated when you're changing lines and errors are popping up. You fix some, but a different one pops up. Confusion about which part of the prompt is breaking the agent and then just more time debugging. But reactive lets you find and fix issues as they come up. You can bake in super specific examples and rules that directly counter common errors that might happen in the workflow. And then it helps you add on new tools and increase the system prompt without rewriting everything and wanting to put your fist through the monitor. So key components of a good prompt. We took a look at the one that I have for the email agent, but we've got background and context. I like to chunk up background and context into just one field called overview. And as you can see, I really like to do markdown. And I like to do markdown because it helps me break up sections better. And it also, I think that the agent's able to determine, okay, like here's my tools, here are my rules. You know what I mean? So I like to use markdown. Yeah. So I like to use markdown. The background explains the agent's role and purpose. It gives the agent a sense of who it is and how it should behave. And then the context gives the specific situation the agent's working in and helps the AI understand its rules and boundaries. So that's why in this case, our background and context was your personal assistant. Your job is to send queries to the correct tool. You shouldn't be writing emails or creating event summaries. You just need to call the correct tool. From there, you usually give it a set of instructions or rules, the exact action the AI must take, and it prevents misunderstanding. So in this case, the main rule was just you need to get contact information before you try to do email agent or calendar agent actions. Tools. So this is obviously a really important one, not only telling the agent the tools it has, but when to use them. So this is pretty self-explanatory. Email agent, use it then. Calendar agent, use it then. All this kind of stuff. Yeah. Just outlining the tools and when to use them. You could also kind of do how to use them as well. So in the contact agent, maybe here I could say, and sorry, my keyboard's going to be loud, but call this before email or calendar actions. That's another way you could bake in when to use the tool. Sometimes I've said, once you get the contact data, pass it over to the agent in this format. And I could do that as well in the tools, but just kind of depends on the use case. But then arguably the most important and one of the most reactive bits is the examples. And like I said, the reason that I gave this example was because this was an exact input I had tried and it failed on. So fixing that input, showing the actions, and then it's able to take similar inputs and understand what it needs to do. It's really going to guide the AI, not only in the actions, but also in the output that you're looking for. And so the reason I put output in why I highlighted output is because depending on the use case, sometimes output can be a whole section in itself. A lot like specifically if you're having an email or an agent write emails or create content or anything where the output really matters, having an actual markdown section called output and then listing like, here are the criteria for your outputs that can be helpful as well. And then another kind of bonus, a final notes and reminder section at the bottom, which can be basically just final notes. Usually what I do at the bottom with my final notes are just like saying, here's the current date and time. A lot of times with email things, I'll tell it never to sign off. With placeholders, I'll tell it how to sign off. This can just be final context. Sometimes it's a good place to add little tiny tweaks. If you're seeing little things pop through, just chuck it down at the final notes and reminders. But yeah, I will say I usually just always throw it down there and give it the current date and time because you never know when it might need that. Okay. And now just to wrap up with essential prompting techniques. And this isn't like, okay, I'm going to build an agent and this is going to be role prompting. Like that's the type of prompting I'm doing here is role prompting. Sometimes it comes like within a system prompt, you're going to use different techniques in bits and pieces of different headers. So anyways, role prompting is just assigning a role or persona to the AI, which helps them adopt the right tone and expertise. So if I was doing a customer support agent, I would basically tell it to be friendly. I would tell it, this is your name, this is your job, all this kind of stuff. So in this case, friendly customer support agent handling refund requests, which makes the AI just sort of adopt that tone a little bit better. A few shot is giving the AI examples so it can learn from the patterns. So we talked about that with examples, but maybe it's classifying things. You'd give examples of how to classify. So in a text classifier node, if you guys have seen like the human in the loop type feedback one, or a routing agent where we're using text classifier to discern, determine which type of email is coming in, we would give examples of, here are messages that are, here are wordings that typically align with customer support emails. I mean, give different words or finance emails, or in the case of the human in the loop sales team that I did, what we classified in there was approved. And like, here are some typical approved type messages. So that just helps with the consistency and the accuracy. And then chain of thought, which we kind of did with like the whole rules slash instruction section, which is explaining how to do something step-by-step. So this kind of bakes into examples, kind of bakes into instructions. But when you're seeing errors, especially when there's a multi-step process, is explaining to it, based on this input, you'll call this action to do that, then you'll call this action to do that. So like in my, actually that's a great example that I will, let me pull that in real quick. So bear with me here. But for example, within the ultimate assistant build, we had an email agent and there were specific ones where we had to have it like grab different, what's it called? Grab different tools before calling other ones. So let me just paste in here. This is the email agent system prompts, right? So overview, you're an email management assistant. All emails must be formatted professionally in HTML and signed off as Nate. So that's the overview, that's the background, the context. Now here are the tools you have. So send email, send emails. Create draft if the user asks for a draft. Get emails to retrieve emails when you're requested to do so. Use get labels to retrieve labels. And then we have these three tools that are a little more specific. So you're going to use mark on red to mark an email as on red. You must use get emails first. So you have the message ID of the email to flag. So this is the case where in the tool, we're saying basically you can never call mark on red unless you've called get emails first. Same thing with label email, but in label email, it's a little more complicated because you must use get emails first. And then you also must use get labels. And then once you have label ID and message ID, then you can finally actually label the email with label email tool. And then similarly with the use email reply, if you want to email reply, you first have to get emails so that you have the message ID of the email to reply to. And then as you can see at the bottom, final notes, here is the current date and time. So that was basically 20 minutes of me rambling about proactive versus reactive. I hope this one was helpful. The thing about prompting is the agents are only going to be as good as the prompt. It's always data in, data out. And what happens in between the data in and out is prompting and tools. And that's the black box, but the more you do it, the better you're going to get at it. There's not just one framework because yeah, we have these techniques, we have these core components, and we have the approach of being reactive rather than proactive. But every agent I do, I prompt it a little differently and it's just because I react to how it works and I throw things. I could throw the same sentence in the overview or the same sentence in the rules and it would change the behavior a little bit. So you really just got to get in there and test. I've always said upfront, there's a big part of data ingestion. The actual plugging in tools is very small. And then the final piece is just so, so huge on that iterating and refining of tool add, tool delete, line add, in the system prompt, line delete, just continuing to refine that process. And then once you finally have something that you're working with, you still test the heck out of that thing. So thanks for listening to me ramble, guys. I hope this one was helpful. And yeah, I'm excited to see how you guys can start reactively prompting and see some cool community builds. So thanks guys. Appreciate it.