CrewAI Flows: AI Feedback Loop

Hey guys and welcome to this Flows deep dive where you're going to learn how to add in feedback loops to your flows just like this so that your AI automations will deliver the perfect result every time. And in this tutorial we're going to break up into two major parts. First we're going to cover what is this feedback pattern, how does it work, why is it so important, and when should we use it. And then from there we're going to take everything that we just learned and we're going to go build a flow completely from scratch that's going to generate Twitter posts for us and it's going to use this AI feedback loop to evaluate and improve our tweets until they're absolutely perfect. And don't worry if you're new to Flows because I just released a complete crash course that covers all the fundamentals. I'll have a link to it down in the description below. And also because you guys are awesome I'm going to be giving away all the source code that you're about to see in this video completely for free. Once again just check that link down the description below. But with all that out of the way let's go ahead and dive in to start learning how we can start working on these feedback loops inside of our Flows. Alright guys so welcome to part one where we're going to dive deep into covering what is this Flows feedback loop so we can understand the core principles and cover all the important things I mentioned earlier like why is this important, when to use it, and we'll actually cover a few different examples that I think you guys would find impressive. Then what we're going to do afterwards is quickly cover the actual example that we're going to be building together and then once we have all the core concepts down we're going to hop over and build out everything ourselves completely from scratch. So with all that out of the way let's go ahead and cover what the heck are these AI feedback loops using Flows. Okay so the core principles that we're taking inside of a Flow that we're going to be building here in just a little bit is we're going to be using a crew and that crew's whole job is to go off and take action. Then what we want to do is then have another crew that's just listening. Hey once you have completely finished generating whatever you're doing so this is usually some sort of generation, summarization, any shun is usually what we're doing taking action. What we want to do is just make sure that it did what it was supposed to do. So this is where reviews and evaluations come in and what we're going to do is have a second crew and that's their whole job. We just want to make sure it did exactly what it said it was going to do. Did you hit a certain number count? Did you you know make sure it was a certain length? Whatever criteria you're trying to do and evaluate we're going to go ahead and check that. Then once we're done we're going to a few different states. Three specifically. First if things went well great. What we're going to do is just say yep that was a perfect and whatever you're supposed to do it was perfect. We're going to go ahead and say success and we're going to you know take action. You can save it you can post it to the internet whatever you're trying to do initially. The other option number two is if it failed and this is where we're going to get a little bit intelligent and you're going to see later on but we're going to do a track our maximum attempts taken. So here's what I mean by that. Every time we fail we're going to quickly capture hey it looks like you failed one time. Okay well if you failed let's say less than three times we're just going to continually provide feedback so that the next time you go to do another generation summarization you know any shun we're going to incorporate some additional feedback from our evaluation crew. That way the next time it goes and takes action it will know oh I need to remember that last time I generated something too short or I forgot to talk about this. That way it next time it does another generation summarization it goes cool I now know how to properly do this and once we go ahead and take action again we'll come back and evaluate and review all of the new results and we're going to continue this feedback loop right here and we'll just say for three times we'll attempt to make this action and make sure it's perfect three times and we can bump that number up if we like and once again if it goes through and everything is a success fantastic and in the case where let's say we try three times and for whatever reason we just cannot get the crew to do what we want well then we'll go to a failure state where we'll just log what the result was so in our case it might just be like hey like you weren't properly summarizing it and here's what you were leaving out well we'll just log that failure and what we can do is because we've logged it we can come back later and start making some attempts and updates to our initial crew that was taking action so it's more for us to understand what was going on wrong so we can maybe go update our prompt in the future to better handle this type of feedback so i know that was a lot but i just want to quickly recap on the core loop and then we're going to look at a real world example that we're going to be building together so core loop take action using ai generation summarization then we're going to use a fall web crew and a loop that's going to just be listening for results once it's listening for results it's going to evaluate make sure it looks good if it is success we're great if not we have two things that can happen first we can send back some feedback to the initial crew saying hey dude you didn't do it right and we can try that a few times and if it's still that crew is being too stubborn and it's not working after three attempts five attempts we'll see we'll set some numbers later on we're just going to go ahead and count it as a failure and we're going to log what went wrong with our crew so i hope you guys this is all clicking and i hope you're excited about this feedback loop now what i want to do is just show you exactly what we're going to build here in just a second in our case we are going to build a shakespeare x post generator so what it's going to do is it's going to take in a topic and then it has some specific rules for like hey this is what the ideal tweet should look like this many characters it should follow this style that's what we're holding it to the standard and we're going to say okay cool you just generated it great let me review it let me count the characters let me make sure you followed this rhythm pattern that we're going to talk about in a tweet then per usual we can go ahead and pass some feedback into our previous flow if it failed and then what we can do is if it did work we'll just go ahead and save the post so we can go ahead and you know post it to twitter later on and then if it failed too many times in a row we're just going to go ahead and log the error so hopefully this is making sense and i hope you guys are kind of seeing how this just pattern as a whole is going to make basically allow you to go off and do a bunch of different things so let me just walk you through you know a few of the key elements i want to talk about over here so when and why is this feedback loop so important and really why do we even need this in the first place and it really comes down to like ai is so smart but it just doesn't do what you want it to do always it's very frustrating it feels like it's rebelling against us sometime but these feedback loops help us keep it in check and it helps us allow us to have multiple attempts for multiple shots on goal to make sure that we're hitting the target that's one of my favorite reasons to use these types of flows so and the next thing i just want to point out is when do you want to use this type of flow well it really comes down to when you have hard requirements you want to make sure that whatever you're generating is accurate and you want to avoid inconsistent outputs so in the hard requirements for our twitter example well we need it to be less than 280 characters but we want it to be more than 200 and you can see that like you know these are some hard requirements and we just want to be able to check and then going off to examples where these requirements will become more prevalent well whenever we're writing a proposal for example for a company let's say we're placing in a bid well we want to make sure that our proposals accurately report the name of our company we want to make sure that we have some pricing thresholds that we want to make sure it never goes below you know a fifty thousand dollar offer and it never should see exceed a million dollars so you can kind of see like okay like i can clearly see now that like all right these are some hard requirements and this flow step here will help make my life so much easier because it's going to review it for me and take multiple shots and provide feedback so we have multiple attempts at generating the perfect output outside of that i do just want to mention there's a few other ones email marketing is another one such as if you're sending out a link to your customers or sending out email sometimes you just want to make sure that like hey did we provide valid links you could have you know some sort of web scraper to make sure like yeah that link did exist and it had the proper content so you can just review what you're doing because sometimes you know these ais will hallucinate and just come up with random things and you can you know add in some tools to your evaluation step to actually go explore links and review to make sure links are actually accurate and not just you know made up and then finally this was when i just came across this from our weekly ai developer accelerator calls someone was looking at generating meal plans and what's interesting apparently sometimes ai can poison people because it'll recommend to cook meat at inappropriate temperatures so inside of your review evaluation step you could have your crew basically be a health critic and food critic and make sure that you're producing food that is of proper health standards and cooking standards so you're not accidentally recommending recipes that will go off and get people sick so i'm hoping you can kind of see like man this this pattern here can be applied to you know every niche basically every problem type where you need some sort of checks and balances before spitting out a result so hopefully this is making sense but enough of this you know high level talking let's actually go ahead and hop over to part two where we're going to dive into building this flow completely from scratch to where you can see us setting up the flows adding in the crews and setting up this feedback structure that you see right here on the screen so let's go ahead and head over to part number two all right guys welcome to part two where we're actually going to get our hands dirty and start coding up our flows and just a quick heads up before we just dive into coding out i want to give you a roadmap of where we're going so first things first we're going to follow this readme that i've set up for you guys where we're going to set up your environment once we have your setup your environment we're going to get all the dependencies needed to run your flow and i'm going to show you step by step how to create the flow add the crews and once we have everything ready to go then what we're going to do is first we're going to stub out our flow so you can see how the overall feedback loop looks like then we're going to dive into actually building out each of our crews that are going to go off and generate post for us on x and then do the self-evaluation so that we can get perfect tweets every time so with all that out of the way let's go ahead and dive in to start setting up our project and it is important to mention real fast that you can access everything that you see right here i have a link down the description below where you can go ahead and download the source code but basically i've just created an empty folder with a readme and a requirements text all right so let's go ahead and start following these instructions to go ahead and set up your environment so right now i'm just in a empty terminal and what i'm going to do is go ahead and create a brand new virtual environment from scratch and the reason why we're doing this is because we want to install cray in its own environment so that it's not messing up and interfering other dependencies aren't interfering with crew ai so because i have conda i'm going to follow these instructions right here to create a new flow sorry to create a new conda environment called cray i flows using python version 3.12 but if you prefer to use virtual environment you can follow these instructions instead so let's go ahead and go ahead and create our new virtual environment now i already have this virtual environment so it's going to ask me to delete it but in my case we do want to go ahead and start from scratch and create that new conda environment then it's going to ask us hey do you want to go ahead and install these dependencies of course we do and then finally once it's done installing all those dependencies we want to go ahead and activate our virtual environment let's go ahead and close out of these errors that pop up awesome so yeah we just go ahead and copy and paste that conda activate environment and this is going to go ahead and put us into our environment so you can see right here we are inside of cray i flows now this is where we're going to go ahead and start installing all of our dependencies so we're going to go ahead and hop down here to the requirements step now that we're inside of our environment and we're going to go ahead and install all of our requirements which you can see right here are really just two the first one is cray i and the next one is just going to be python.emb which will allow us to access our environment variables now it is important to mention that cray i we ship code fast so at the time of this recording we're at 076.9 but you know later this week we'll probably be shipping even more features so make sure to go ahead and click that link right here to see what is the latest version of cray i and make sure you update to the latest so let's go ahead and install these requirements right here and this will take a little bit just to go ahead and pull down all the necessary dependencies to go off and run your crew so we're going to go ahead and pause and whenever it's done we're going to hop into actually going ahead and creating our flows all right guys so we just got done setting up our overall conda environment which has all of our dependencies for going off and creating any future crew that we want or flow and now that we have that set up what we can do is dive into actually creating our new flow project so now that we have everything installed we can go ahead and actually start using the crew ai cli so what i mean by that is you can now start running commands such as like cray i create flow and then the name of the flow and whenever we run this what it'll do is it'll go off and create a flow for us that has all the proper crews and the rest of the folder structure set up for us to go ahead and run our project super cool now that we have that out of the way what we can do because you can notice we're starting to get all these like squiggly marks well it's because we don't have for this specific project all the dependencies installed so let me show you how we can go ahead and fix that first let's hop back over to our readme and we're going to scroll down just a little bit and first we're going to cd into this new flow project that we've created and now we can go ahead and install all the dependencies for this specific project so i know it's a lot of dependencies set up but just a quick recap the first conda environment was so that we can go off and create any crew any flow that we want and these dependencies are going to be specifically for this project and you might be asking well what are we actually installing well if you hop back over to our pyproject.toml we're going to be installing these dependencies that you see right here cool so let's go ahead and run this command which will go off and install in our case cray and cray tools this will take just a few seconds then once it's done we can actually move in to the next part which is getting rid of these squiggly marks so if you are on a windows computer or mac don't worry i have you covered what we're going to do is just now activate our environment with the proper dependencies for this project so i'm just going to run this command right here and you'll see now it says oh yeah you are in the self evaluation loop flow environment super cool and one thing that you'll notice is as inside of your main.py file you'll notice like okay well that still didn't fix that issue of these squiggly marks and the reason why is because we need to update our ide to say hey man you need to be using this specific version of python so what we can do is if you're on a mac you can run this command right here which python it'll go ahead and call out where your instance of python is running so you can see it's a long ways but you can see yep it is this virtual environment so i can go ahead and copy this and then i can paste it in right down here so if you go to a python file you'll just click down here and say all right i would like to enter interpreter path i'm gonna go ahead and paste it in i would like to use the one we just found awesome so now we are set up for success and you can see our projects aren't complaining at us anymore and everything is looking great so let's hop back over to our readme and i'm gonna walk you through the final few parts which are going to be going off and creating our crews for the specific flow and just a quick recap of where we're going is first we are going to go ahead and create that shakespeare crew which is going to generate post in the style of shakespeare and then we're going to after that go off and create our evaluation crew that's going to make sure whatever we generate actually works and follows all the requirements that we set out so let's go ahead and knock these out one at a time once again we are going to be using the create icli but this time what we're going to do is use it to add a new crew to our flow so i'm going to go ahead and paste it in here and then i'm going to clear things out so you can see exactly what's happening so what we would expect after running this is we will add a new crew inside of our crews folder so let's go ahead and run this and you can see it'll just pop up here in just a second with all right cool yeah you have now access to the new shakespeare crew this takes just a second to go in and add all the projects templates set up to where everything is looking nice great so you can now see my crews folder i now have two different crews and that's still not everything that we need we also need to add in another crew which is going to be our evaluator crew so let's go ahead and run that too but hopefully you can start to see like oh yeah this is a super nice way to go off and just add as many crews as i want to my flows to go off and handle different parts of my flows all right give yourself a pat on the back we just finally got done with all the setup so now we're going to actually head over to the fun part where we're going to start working on our main.py and start you know stubbing out this template to be repurposed for our feedback loop that we've talked about so far so let's go ahead and start clearing this up and following those initial templates that you saw earlier where we had everything inside of a feedback loop all right so before we go ahead and just start updating this template i just want to show you once again what is our end goal so you can visualize it first and understand what we're trying to build and then we're going to hop back and actually make it happen and build out each one of these different steps in our flow so as you can see first things first we are trying to generate that shakespeare x post then we're going to pass the result over to evaluate so we can see like hey did you make sure we were in the proper character count did you follow this style if not we're going to send feedback to our original post to say hey please try to make it again but listen to these you know additional instructions that you missed then we're just going to continue through that feedback loop a few times and eventually we can end up in two terminal states state one is as a success where we're going to go ahead and save the result option two is if hey you tried let's say three times and it's still not working well we don't want to cause like an infinite loop so we're just going to say max retry exceeded go ahead and log what went wrong and how we can you know maybe go off and update our initial prompt just for feedback for us as the creator of this flow all right so this is what we're trying to build now let's hop over and start updating this main.py to account for this and i think the best place let's just go ahead and pretty much nuke everything in here and what we're going to do next is we're going to go ahead and start creating each one of those key components so i think it might be helpful to put these side by side so you can see how to go from this drawing right here over to an actual flow so what you'll notice is the first thing up top is a orange method which is going to be our start step so inside of flows we to create a starting point you can just use the start decorator just like this and then in our case we're just going to call this our generate shakespeare i'm going to go ahead and copy this shakespeare has such a weird name i need to work on english then what we're going to go ahead and do is close it out just like this and this is where we're going to we'll put it to do for future us but this is where we're going to add our crew add shakespeare crew awesome we're going to put a pass right now because we're just trying to stub things out so we can see how to go from a you know if you're drawing stuff out on a piece of paper for a flow how to come back over here and programmatically create it awesome so the next thing we want to do is go all right i would now like to evaluate this x post so we have to listen for whenever this is done whenever this function is done and once it's completed what we can now do is go all right i would like to evaluate this post so def evaluate x post great and then once again we'll add it to do and this is where we will add in our add evaluation crew awesome put a pass so we stop getting errors now what we can do is start adding in our terminal states because we're going to do some logic in here eventually to go off and actually go ahead and make sure that we are listening to these different crews oh and sorry quick mistake this was supposed to be a router let me explain why real fast so inside of our crews what we can do is say all right i would like to redirect to multiple different paths so path one is going to be our max retry exceeded option two is save result and option three is to go back and add feedback so we're actually going to be routing to one of these three different states that could potentially happen as next steps so that was my bad and what we can do is come back up here and when we're working with our crew ai flows we're just going to import router awesome so now this should go ahead and say cool yeah no more errors and what we can do next now that we've checked off these two we can now start working on max retry exceeded and save result so how do we do this well this is where we are going to use the listener so we're going to go ahead and say i would like to listen to whenever we and this is where we're going to start getting a little bit fancy using routers like i said this is a deep dive but what we're going to do is have a world to where if our router succeeds what we're going to do is trigger the complete path and the complete path will be like yeah this was a success and we're going to go ahead and save the result so to do we'll just say save save the result awesome and then we'll just put a pass and then the other option to wrap things up is going to be whenever we exceed uh the number of attempts so if this takes more than three attempts we're going to say hey we tried and it just is not working for whatever reason so we will inside of our router which you'll see later we're going to go ahead and trigger the max retry exceeded exceeded awesome now this is looking pretty good and what we can go ahead and do oh sorry uh silly me i meant to add a function there then what we can do for our max retry is we can go ahead and log it so we'll say def max retry exceeded and this is an exit now one thing i do just want to point out two things that are important so we'll just exit and what's important is you don't want to accidentally call this function the same as this because what's going to happen is once you're done with this function it would reach listen for that function to get done and you'd end up in a feedback loop so it's you have to be intentional with your flows i actually made that mistake when i was building this and i was like why do i get infinite loops and it's because whenever this function finished it was listening to it and it would re-trigger it so infinite loop so that's why this name is slightly different than what you see right here okay so just a quick recap we now have our code working to where we're listening if things fail we're listening if things complete successfully the only thing left is how the heck do we add back in that feedback loop which is going to say all right our starting function needs to also listen for evaluate crew so what we can do is hop back up initial start function and we can go ahead and listen to another event which will be retry so this retry is going to get triggered here in our router so i might go ahead and just put all the possible options that our router could do so option one if things go successful we will trigger the completed path so completed option two is max retry exceeded that so if you know we try three times and it fails we'll go down that path and option three will just be a regular retry so hopefully this is kind of making sense to where this router right here is actually doing all the heavy lifting for us to where we're first going to evaluate and then if things are a success great save it if we need to add some feedback we'll do that and you'll see how we do it with state in just a minute and we'll go to regenerate the post again and we'll go through this feedback loop right here up to three times five times we'll set that in just a minute and for whatever reason it still cannot generate a valid post we're going to go down to the max retry exceeded step and this is where we say hey we tried too many times we're not going to do an infinite loop i'm stopping and here's why so hopefully you're kind of seeing like this is pretty powerful and that's why this router is so important and you'll see this being built out in just a minute but i just want to show you first how we can go from this graphic to an actual stubbed out flow to where we have all the proper steps set up in here with all the different listeners and routers all right cool now let's just do a little bit more cleanup where we're first going to go ahead and get all of this set up properly so right now you can see everything's kind of set up for the poem flow just because that's initially what the template comes with in our case we are going to be making a x post in the style of shakespeare and the other thing that you're going to notice is we are passing default state into our flow now this is going to be different because we are going to be using instead of poem state we're going to be using our shakespeare x post flow state now i'm going to walk you through why we're adding in each one of these different states in here and just do a quick crash course on what the heck you're actually looking at okay so first things first what is this well inside of flows you can go down two different paths when it comes to state management and they are unstructured and structured we're going to be using structured and all that means is whenever we are kicking off our flow we are saying hey these are all the possible values that you could have inside of your state so whenever you're in this step you could access any of these values same down here when you're evaluating the you could access any of these values and what's nice is it just gives you some guide rails and it allows for auto complete super cool and the other option was unstructured and then unstructured basically you're just given an empty dictionary that you can update any key value inside of your flow and just access this state across all the steps in your flow super straightforward we cover this in much more detail in the initial crash course so once again definitely check that out if any of this is confusing okay so now that you've kind of seen all right we've added in state let's actually dive quickly into what are each of these different values and why do we need them as we go off and start implementing each one of these steps so the first one is the actual post that we're generating so in a little bit once we go off and actually update our crew to generate a post what we're going to do is save the result that is from our generation from our crew that gets generated and save it in our state and this is nice because now when we're going off and evaluating our post we can just do something like just like this we can do self dot state dot and i could go ahead and access our x dot post so you can see it's right here so you can see that's how we're going to be able to grab that post and pass it into our next crew and then what's super cool is what we can also do is actually go ahead and start a valuation step we're going to go ahead and actually use these three values right here so let's zoom out so you can kind of see all this in one so let's actually tie these to our three possible outcomes in a completed state well what's going to is our crew that we're going to build in just a minute is actually going to say hey the post that you just generated it was completely valid and because it was valid i'm going to go ahead and map you down to the completed step and the completed step it's going to go oh okay cool i now know that what you generated was valid i'm going to go ahead and access once again that state right here i'm going to go ahead and save whatever post was saved right here i know i can go ahead and save it to the computer awesome now other option is every time we try to evaluate we're going to first check our retry count so if our retry count goes above three we'll say that's our limit we'll then go oh okay you you definitely are self.state.retrycount it's over our limit so i'm going to kick us down to this path right here so hopefully this is starting to make sense why we're using each one of these variables and then finally for the third potential option which is retry we're going to go okay whenever i evaluated your post that saved a state it didn't pass uh it was too long it was 300 characters and our max was 280 so what i could do is go ahead and say all right we're gonna save whatever feedback was generated down here the fact that it wasn't valid it wasn't valid and our feedback for new rules such as hey it was 280 characters or 300 characters let's go ahead and bump that down to our constraints so we'll just pass it in our feedback and what's so cool is you'll see in just a minute that our crew that we're going to build will actually whenever it's generating a new post it'll actually go ahead and incorporate this feedback so that's like grand scheme high level what we're going to do and enough talking let's go ahead and actually start going and building out each one of these steps from top to bottom so you can see okay yeah that is how we are saving the post this is how we are adding in the feedback so with all that out of the way let's go ahead and dive in to first going ahead and adding in our first crew which is going to be adding in our shakespeare crew which is going to generate that first post and to add in our flow it's super simple all we need to do is up here in our imports we're just going to say from and then we're going to do our crews folder because if you remember that's where every oops sorry that's where everything is saved so over here in our crews folder you can see we have our shakespeare crew our x review crew and everything else so we're going to try and access this crew right here so from crews dot our shakespeare crew and specifically our shakespeare crew file that's right here that's exactly what we're trying to we're going to import our shakespeare crew awesome now that we have access to our shakespeare crew we can go ahead and add it in here so what we're going to do is go ahead and say i would like to use my shakespeare crew then what we can do and we cover this in the crash course but think of this as a class like hey this is like the template of a crew and let's go ahead and create an instance of that crew awesome well now what we can do is we actually want to go ahead and kick off that crew and we're going to be passing in some inputs into this crew so for inputs what we're going to do is go and say all right well i would like to go ahead and kick this crew off to talk about a specific topic so in our case we're just going to go ahead and type in flying cars then what we can do next is start adding in feedback so this is where we're going to start saying like all right out the gate there's not going to be feedback but if something goes wrong in our evaluation step we want to add in our feedback this is exactly how we're going to do it we're going to say all right cool here is the feedback and i'm going to access it from self state feedback which is going to grab this value right here okay so now that you can see how we're going to kick off our crew with valid inputs now let's go ahead and actually hop over into this crew and actually start you know coding up the task and the agents so you can see this you know come to life so let's go ahead and hop over here real fast and start updating these agents and tasks to actually go off and generate an x-post force real fast all right so first things first whenever i'm beginning to work on a crew what i like to do is first go ahead and actually define all the agents and tasks and the way we have this set up inside of cray is you'll notice in your folder structure you now have this config folder this config folder will contain all the definitions for your agents and tasks so you can see right now we have a boilerplate with a researcher and reporting analyst agent and in our task we have a research task and reporting task so we're going to go ahead and actually get these updated to actually use our use case which is you know hey we're trying to generate a post in the style of shakespeare so i'm going to go ahead and just show you what the final version of this agent looks like and let me walk you through it real fast so you understand why i set up this agent to look like it does so i'm going to call this a shakespearean bard who's you know responsible for going off and generating post in the style of shakespeare so we're going to say all right here's your role now what's the goal well i would like you to craft sarcastic playful hot takes in the style of shakespeare and it's important that all your responses fit within 280 characters and it contains no emojis then we're just going to give it a little bit of backstory like hey here's who you are as an agent so keep this in mind as you're performing task all right hopefully this makes sense now what we're going to do is we need to start giving this agent task so we can hop over to our task yaml file and let's go ahead and paste in our task which is going to be all right you now need to go off and write a post on certain topics so what you'll notice is we're going to say all right given a topic and what you'll notice is this topic is wrapped in curly brackets well this is how we allow you to pass in variables into your task so this is basically string interpolation inside of a task so in our example we said the topic was going to be flying cars so whenever you actually run the crew flying cars will appear right here cool hopefully that makes sense and then we're just going to say all right make sure you compose a humorous take in the style of shakespeare and the final short form media post should be over 200 characters and under 280 so over and then under and make sure no emojis happen and then here's where the cool feedback loop starts passing in because now we're going to say all right if there was any feedback please incorporate it so what's going to happen the first time we run this crew this feedback will be empty because there's no feedback it's our first attempt but if something goes wrong in our evaluation step we're going to capture that feedback and then pass it right back in here so this is what's creating the self-improving feedback loop using our agent all right cool hopefully this makes sense and then what's pretty cool is now you can go ahead and actually define your agents so you can see right here we're going to use the shakespearean bard so whenever we hop back over to our crew we need to just go ahead and define this agent so you can see it's right here so shakespearean bard you can see that we're going to go ahead and say hey please use that agent all right cool now that we have that set up let's go ahead and actually start updating our crew to use our agents and tasks that we just defined so first things first what i'm going to do is i'm going to go ahead and clear out everything that you see in here when it comes to agents and tasks because we've created our own agents and tasks so here's what we're going to do first things first we need to go ahead and say where the heck are our config files because we want to start using these config files inside of our agents and tasks so if you look we're in our crew and you can see our agents and tasks are right here in this config folder so we're just going to say all right cool agents config is in the config folder and it's this agents yaml file right here basically the reason we do this is because if you'd like to go off and eventually maybe let's just go ahead and say put your agents all of your config files at like a root level to where you can access them amongst multiple crews you have the ability to go ahead and define where exactly everything's at right here all right now what we can do is go ahead and start working on adding in our two crews so we're going to first work on adding in sorry our agent and task so first things first we're going to work on adding in our agent which is going to be our shakespearean bard and basically what we're doing here is turning our code template which is defined right here like this is our template these are instructions and we're going to go ahead and say all right well i would actually like to turn this into a code agent so exactly what you can see is happening right here we're going to say all right i would like to create a shakespearean bard agent and all it's responsible for doing is returning the agent that's defined in this configuration file right here so you can see all right if i was to hop over to that file and i to look it up you can see all right this is this agent right here and as you build more complex crews you would just add all of that right in here you'd add other agents roles goals everything else cool hopefully this makes sense and we're doing the exact same thing for our right x post so this is the actual task and what you can see is we're just calling the name of the task like the code version of it the exact same name as our task that was defined right here all right cool makes sense and this agent name matches our agent name right here all right things are looking great now what we can go ahead and do is actually start adding in one additional piece of information which is going to be a custom tool for our agent why do we need to do this well the root problem is agents suck specifically llm suck at counting things so if you were to tell an agent to like count the number of characters in a paragraph for whatever reason just normal math and counting doesn't work so what we're going to do is we're going to create a simple code custom tool that will go off and count characters for us so it's super easy to do what we'll do is inside of our agent we'll just come in here and we're going to say all right you now have access to this character count tool and does exactly what you think it would do it will count the number of characters basically inside of a post so we're just giving and supercharging our agent to have access to but now it's up to us to go ahead and create that specific tool so what we're going to do is first inside of our tools folder we're going to come in here and create a new tool so just right click new file and we're going to call this character count tool.py this is a super straightforward tool all it's going to do let me just show you and walk you through exactly what's happening in here line by line so what you're looking at is a way to create structured tools inside of corei so what you can see is there's really two different components going on inside of one of these files first we are defining what is the actual input that you want to be passed into this tool and the reason why this is so important is because as your llms are running they need to know okay well it does sound like i'm generating a post and i can see inside of your instructions that i need to make sure a post is above 200 characters and below 280 characters that's a need okay do i have any available tools that will help me go off and actually count the number of characters and the answer is yes because i have a character count tool and i can see you know what does it do what's the name of it and i can see what is the tool actually doing in this case it's going to count the number of characters in a given string that sounds exactly like what i want to do when i'm making sure my tweets above 200 and below 280 so this is like how the llm determines that like yes i have a tool that can do it and then once it goes okay i want to use that tool the next thing that llm does is it goes all right what information do i need to give you in order for you to go off and run properly and in our case it is we want to say all right you need to pass in one parameter this parameter is going to be a string and it is the description of it is just like hey it's actually the string that you want to count the characters in so the way we actually kind of tie these things together our tool and our tool input is by updating the argument schema for this tool so you can just go ahead and see okay cool i would like the schema for this tool to basically just be this and you'll notice under the hood we're using pedantic to build all of our models in our inputs to make sure that like you know hey this is a text string this could be a integer it could be a float all those different types that come with pedantic we can just set them here all right and i know that might have been a lot don't worry we have a ton of documents on structured tools unstructured tools tool decorators everything that's possible with gray i we have all these tool in our documentation so definitely recommend checking that out if any of this tool part is complex all right the final thing that you have to do when building a tool is you have to say all right what's the actual logic that you want to run whenever this tool gets called so in our case you just need to create and update the underscore run function to actually say like all right this is what the tool needs to do in our case all it's really doing is just calling the length function that comes with python and that's how we're grabbing the character count and then we're just returning it as a string back to our llm so we're going to go ahead and tell it this many characters so hopefully this is starting to make sense of like okay this this does make sense whenever an agent wants to use this tool it's going to pass in a text string that text string we know that our tool we give the ability for our elements to understand that it needs to pass in a text string because we've given it the argument schema right here all right hopefully this is making sense let me know if you have any questions drop a comment down below or hop over to the free school community super happy to help you guys out over there as well all right so now we've created this tool it's now time to actually get our agent to properly import it so if you just go ahead and click command dot on mac you can see all right it's going to go ahead and give us a few different options to import this tool right here so i'm going to go ahead and import it this way you can now see we have properly imported our character count tool let's go ahead and clear up some of these comments so this just looks a little cleaner all right this looks great guys so we now have our agent that's properly going to generate a tweet for us it knows how to count characters and in addition to that it has one task to execute in our case actually going off and writing the post now what's super cool is you'll see down here in the crew section you'll see that we have our agents and tasks but what's so nice is instead of you having to manually come in here and add in agents here one by one basically anything that was wrapped by an agent decorator will automatically get added in to self dot agents so basically that what that's saying similarly for our task anything that's wrapped in a task decorator will automatically get added right here for you guys and what's so cool is if you were just to come down here and add a few more tasks each one of these tasks would be executed from top to bottom so the order you create the task is the order that they'll get executed inside of your crew all right so this crew is officially done and ready to work so what we can do next is we're going to hop over to our next step which is going to be working on our evaluation post right here so this is we're going to work on our second crew which is going to go off and say cool you've made a post let me make sure that it followed all the rules if not i'm going to say it's not valid and here's feedback so let's go ahead and start working on this crew right here so first things first we're going to go ahead and import the crew just like we did the other one so up top we're going to say all right from my crews folder what i would like to do is go ahead and import my x post review crew and i'm going to go ahead and import it and specifically we're going to import the x post review crew cool now this is set up and we're going to do exactly like we did over here so we're going to go ahead and say all right here's my crew i'm going to go and create an instance of this crew and what i can do is go ahead and kick it off now what we're going to do inside of this one is we first want to make sure that everything is getting passed in properly so all we're going to do is go ahead and say crew.kickoff and in this time what we're going to do is update our input to only include the generated post from up here and we're going to clean this up too as well but i just want to show you everything first so we're going to say all right you are going to accept an x post and this x post is going to come from self dot state and then it's going to come from our x post great now you're probably wondering like well how do we pass information from this step over to this step great question so we actually need to go up just a little bit and save the result from this kickoff so we're going to say result is equal to this and now what we need to do is go ahead and save it to our state so first let's go ahead and actually print the result so you can see it in action and because we're just generating text you can go ahead and just get the raw response so that's how you're going to get from a crew the raw response later on in the next step you're going to see how we're going to say all right crew don't just give me back text i want you to give me back an actual object that i can actually pull out like is it valid what is the feedback so we're actually going to get back an object in this follow-up crew so you're going to see later on how we don't just get the raw result we're going to get a basically a pedantic object back so now that we're printing the result what we need to do is actually go ahead and save it to our state for x post so how do we do that well we're going to do self.state and i want to go ahead and store the result just like this so this is going to go ahead and make sure that our x post property will now be accessible down here all right cool hopefully this is making sense so now that we have gone ahead and passed in and saved that value let's go ahead and actually hop over to this crew and start building out just like we did last time we're first going to go ahead and build out the agents and then we're going to start building out the task and then merge everything to get it working inside of this file right here so the first thing we can do let's go ahead and open up our file explorer and you can see we're in the x review post crew file and we just need to hop over to our config folder because this is where we're storing our agents and tasks so let's go ahead and open up both of these and let's go ahead and add some space and then what we're going to do first is we're basically just going to go ahead and update our agents first update our tasks and like i said time together back in our crew so in this case all we're going to do is we just want to have one single agent the purpose of this agent is basically just like the other one but it's going to take more of a review role instead of a generation role so you can see it's gold now it's just like ensure so it's not generating it's ensuring like hey let's go ahead and make sure that it follows these strict guidelines and just to go ahead and recap them 280 characters no emojis and and we're going to go ahead and throw in some additional commentary just so that we can go ahead and kind of force that like failure loop just so hey let's also make sure that when you're generating the post you don't do what llm's doing like oh of course i'd be happy to generate a post for you here it is don't do any of that just give me back the post great now what do we need to do next well now we can go ahead and update our task so this is where we're going to get really in the weeds with requirements so let's go ahead oh sorry about this we're really going to get in the weeds with requirements so what you can see now what we're going to do is just list in bullet points exactly what you need to be checking for so this is a great approach is if you're going off and generating crews every time you run into an error just you know hey it did something wrong what you can do is come back in here and update your reviewer task and say like hey you did this in the past don't do it again so it's kind of like a feedback loop to where it's taking action it's generating results if you're not happy with the results you can just add them here into the review step to make sure that it doesn't happen again all right so let me walk you through exactly what's being done inside of this setup so we're going to just turn everything into hard requirements so you're generating a post it needs to be within these character limits no emojis and don't include additional commentaries and then just to go ahead and like i said force that first feedback loop so you can kind of see it in action we're going to add an additional requirement that we hadn't talked about in the first task which is like all right whenever you're going off and generating a post let's go ahead and say that it must follow the 131 rule to where you know we come out with a bold statement that hooks the reader then we follow up with some supporting information and then a quick sentence to summarize it just to go ahead and generate a hard-hitting tweet so we're going to follow this structure right here and then we're just going to say if there's any other ways you think you can improve the tweet just go ahead do it let me know and update the feedback then this is where things get super important we're going to go ahead and say all right if the post is not valid we didn't meet any of the requirements above if any of them aren't met you need to let me know that it's not valid and go ahead and mark it that way and then what you also need to do is provide actionable changes about what went wrong and what actions need to be taken to fix it now this is super critical because we're basically having two agents work together so we're having crew one do work and then pass results back to the generation and the reason why we're passing in the basically what went wrong and actions is because we want to give context back to the original crew so that it goes oh whenever i'm generating a post i was doing this this is the problem i don't need to do that anymore and here's the actual actions i need to take to make sure in the future i generate valid post so that's why i think it's very important to add what went wrong for context and what actions you need to take to make sure that it actually knows what to do differently when generating the next round of post and then here's what we're going to do we're going to say all right whenever you're generating the final response you only need to give me back two pieces of information the first is basically was it valid and you're going to be back true or false and then additionally if it wasn't valid please provide commentary if what went wrong so that's exactly what it's going to that's the final thing that we want to be returned and now we've given it everything it needs to do we're going to go cool now that we're on the same page what i want you to do next is go ahead and verify this post and once again we're going to do string interpolation because if you remember whenever we were going to go ahead and kick off this crew we're just passing in the x post so this is where we're going to go ahead and access it right here all right so hopefully this is all making sense now what we can do is actually go ahead and hop over to our reviewer crew and then now we're going to start adding in our agents and our task and you're going to see us do a little bit more fanciness where we're going to make sure our task actually returns instead of a string result we're going to have it actually return back a pedantic object that's going to be like contain a valid property contain a feedback property so let's go ahead clean everything out and let's go ahead and hop back into this crew and start getting it working so first things first we need to once again go ahead and be able to point to where our configuration files are so in our case our agent config is over here so you can see config agent that's exactly where it's at same for our task all right awesome makes sense now what we want to do is go ahead and update our agents so let's go ahead and clear all this out because we're going to be putting in our new agents and tasks make some space so first things first we're going to add in our x post reviewer agent and what you'll notice is basically it's the same structure as beforehand you know you're going to pull out your configuration from the x post verifier agent which is right here x post verifier important that the names match up then what we're going to do is we are also going to give this agent the same tool as beforehand and what's super nice is the way we've set up this folder structure for you guys we have a tools folder and you can see that we are able to easily share this tool counter with all of our crews so easily come down here and we're going to go ahead and import that character counter so it's a really nice way to share tools amongst your crews all right now here's where things are going to get interesting we also want to go ahead and add in our task and if you remember our task just review it and tell me is it valid if it is true false great if it's not valid go ahead and give me back feedback now where the heck do we go ahead and define like what is this output value well how do we get back that structured object that's the problem that we're trying to figure out well what you can do inside of a task is we have this property called output pedantic and what it does is it says once you're done with this task i want to make sure the results actually spit out whatever values whatever structure i give you so in our case it's up to us to go off and define that structure so let me go ahead and add it up here for you guys so you can see it in action so what we're doing is we are creating a new pedantic model so this is all coming sorry import it from the wrong place what we're going to do is go ahead and update our imports to include pedantic base model and we're going to update this to come from typings great and here's exactly what's happening we just created a new model that has two properties the first one is valid which will accept a boolean and it's funny because this looks exactly like our task over here so let me just show you side by side why we did what we did so we made a new base model and the first property is valid because it's exactly what we wanted right here and the next one is feedback and just to remember if things go right there's no feedback but if things go wrong and there's something that we need to change we need to include a string that says hey here's what wrong guys and here's what action we need to take next time so that's why we're saying feedback is an optional string so now that we've told our crew exactly what this task should spit back everything's going to work and we'll log it later so you can kind of see everything working but just know whenever this task is done it's going to spit back out an object that follows this format perfectly which is going to be so nice because you'll see in just a second we're going to pull out these you know is valid is is their feedback and we're going to start using this information to actually go off and you know either retry go to success or go to fail so hopefully this is all making sense and we can go ahead and come down here and we are pretty much done because this section right here is exactly like what we had earlier all right so now that we have this review crew set up and working properly let's head back to our main flow file where we were going ahead and connecting everything together and start adding in some additional logic to capture the result from this crew so capture our feedback and is we're going to start working with the retries and we're going to start shooting us over to different states to retry say we're done or to say hey you've exceeded your maximum limit so let's go ahead and start populating this to work properly all right so first things first we need to go ahead and once again we're going to capture the result that comes back from this crew and what we can do is first we're going to extract all the data from it so what we're going to do is go ahead and extract was it valid or not and what we can do is go dot valid great and what we're going to be able to do is grab it directly so from our results we can go ahead and grab the valid property and the reason we can do this is because back in our review crew we made sure that this task spit out a pedantic model that looked just like this so we're able to actually grab this value right here and we can go ahead and also on the line below we can go ahead and grab feedback as well just like this this is looking awesome so feedback great so now that we've extracted all the information that we've defined let's go ahead and just print it out so you guys can see that like oh yeah it did do exactly what we wanted to so valid let's go ahead and look at that feedback let's go ahead and look at that all right so what do we want to do next well we want to go ahead and start pointing to each one of these different paths because if you remember we are in a router that allows us to say all right based on certain conditions go this way or go this way or go that so that's exactly what we're going to start implementing next so what we're going to do first well up to this point what did we do we actually took an attempt at you know reviewing the crew so what we're going to do is first bump up our retry count that makes sense now what we can do is go ahead and start going down different paths so first things first are we valid because if we're valid we're done we can go ahead and not do anything else and this will kick us down to this path right here great well what happens if we weren't valid what we want to do is first go ahead and send us back up to this retry state so we're just going to return retry okay so this looks good we've gone ahead and checked off this path and we've also went down this path now how do we go down the max retry path well super straightforward what we're going to do is if the maximum attempts were exceeded we're just going to go ahead and check that right out of the gate so we're going to say all right where the maximum attempts exceeded in our case we're just going to hard code three then we're going to kick people down this path right here so hopefully this is all making sense and i'm just going to quickly recap it just because like i know routers are kind of confusing the first time you see them so what have we done whenever we evaluate a post we are first going to check to see did we exceed the retry count because we don't want to end up in an infinite loop if our retry count from our state exceeds our you know allowable limit we're going to say hey guys i'm done this isn't working anymore we're going to go ahead and just say we're over we can't get this post to work properly all right that's the thing one if this is our first few times making the crew and we're under our limit great we're going to go ahead and kick it off once we kick it off we're going to save our results we're logging them just so we can see in action we're going to update like hey we took an attempt just to keep track of our retry state and then we can go down two different paths path one is if things worked we'll go down the completed path and if things didn't work we're going to go back to retry and we're going to regenerate using our feedback all right cool so hopefully this is making sense so now we're just going to go ahead and wrap up these two different steps right here of saving the result and max retry exceeded let's go ahead and start working on completed so when we go down the save results step all we want to do is go ahead and it's super simple we're just going to go ahead and post and log that like hey everything worked here's the post that we're going to save and then all we're going to do is actually save the final generated result over to our xpost.txt file so this will actually hop up over in here and you'll be able to review the final generated tweet then what we're going to do next is go down the other option which is like all right if things broke what's going to happen is we really just want to go ahead and log exactly what we were trying to generate so here's the post that was generated and here's our final feedback just so we can see like oh yeah it really isn't working as we expected and here's why so we're going to go ahead and just throw a bunch of print statements in here to look at our post and look at the feedback that was generated now we are almost done guys the only thing left that we have to do is update down here the fact that we are using instead of a poem flow which is comes from the you know the actual uh the template we're going to be doing the shakespeare flow so we're going to go ahead and update all of that things are looking great let's make sure everything is imported property cleaned up now that it is what we can do is actually going ahead and start running our flow so you can see everything in action so what we're going to do let's go ahead and open up our terminal and let's go ahead and try running super fast and real quick before we actually go off and kick off this crew it's important that we update your environment variable file to actually include an open ai key because right now it's just a blank key and what we're going to go ahead and do is paste in a real key that you can see just like this awesome now what we can do now is go off and actually kick off this flow so you can see in action so let me go ahead and make this bigger and then we're going to go ahead and type a command to kick off our flow so you're going to use cray because that's our cli tool and because we're working with flows you're going to say flow and then kick off and this will actually go off and kick off the flow so we're going to go ahead and kick this off and then what you'll see it do is in action it's going to start running the flow and it's going to start generating these posts for us so all right talking about flying cars and you can see in real time that it's trying to generate this so the first time it said not valid then it's going to incorporate that feedback that you can see and it's going to go ahead and try re-verifying it and this time we went ahead and second time it didn't get it either so you can see it's going through the feedback loop over and over and over again until we end up in a valid state like this to where it goes ahead and says all right this is valid and it goes ahead and shows us the final tweet and you can see over here in our x post you can see everything right here so let's go ahead and show what this looks like and open this up a little bit more so you can see this is all in all a pretty much exactly what we wanted to happen so we have a nice looking tweet things are looking great and it goes ahead and pretty much exactly what we asked for it's a Shakespeare tweet it's under the character limit you can see back in our post over here let's go ahead and down here you can see that yep it's only 259 characters things are following the format that we wanted so all around this is exactly what we wanted so hopefully you're starting to see the power of setting up a flow using feedback loops just like this to where we now have a someone who's creating tweets someone who's evaluating it you can see how this time it actually took three attempts at generating a valid post and then because it was a success it went off and saved it but don't worry if it wasn't able to generate a valid post it would have just logged exactly what you kind of saw earlier in the section to where whenever it was generating feedback and it wasn't right it was going to go ahead and say things like let me just show you real fast back up here inside of our reviewer yeah things like this like hey it's not modern basically just lacks a clear structure it's not following this to improve you know so here's the problem and here's the actionable feedback so all around you can see it's doing exactly what we set it up to do so i don't know i'm pumped i think you guys will actually really like this as you go off and build your own flows and you want to make sure that the quality that you get from these flows is perfect and that's a wrap for this video guys i hope you enjoyed this flows deep dive where we start looking into feedback loops with our flows and just a few quick reminders i do have that other flows crash course that covers all the fundamentals and if you're looking for some additional support and to meet fellow like-minded ai developers i have that free school community where we have weekly coaching calls every tuesday and i can't wait to see your awesome face inside of the group but outside that i have a ton of other ai related content and full stack tutorials on this channel so i definitely recommend checking out whatever video pops up after this but have a great day and i cannot wait to see you guys in the next video see ya