How To Connect Local LLMs to CrewAI [Ollama, Llama2, Mistral]

Hey guys, in today's video, I'm going to be teaching you how you can run crew AI completely for free using Ollama so that you don't rack up a huge open AI bill like I just did. So by the end of this video, you're going to learn how to run LLMs locally on your machine like Llama2 and Mistral, and you're going to learn how to connect these LLMs to your crew AI agent so that you can run crew AI completely for free. And don't worry whether you're beginner or advanced, you're going to get a ton of value through this video because I'm going to be walking you guys through everything step by step. And I'm also going to be giving away all the source code for this video completely for free. All you have to do is click the link down description below. And if you want to see more content just like this, it'd mean a lot to me if you guys could also hit that like and subscribe button. But enough of that, let's go ahead and dive into the rest of the video. So here are the three major parts of this video. First, we're going to do a quick recap of the four technologies that you're going to be using in this video just to make sure we're on the same page. Second, I'm going to show you how to set up and run LLAMA2 and Mistral on your machine using OLLAMA. And third, we're going to connect crew AI to OLLAMA so that you guys can see everything in action. So let's go ahead and dive into the first part where we're going to cover the four core technologies that we're going to use in this tutorial. So the first technology that you're going to be using in this tutorial is going to be OLLAMA. And OLLAMA is a tool that makes it super easy to modify, download, and run LLMs locally on your machine. When you visit the models page on this website, you'll actually be able to see that there are a ton of different LLMs that you can run using OLLAMA. And actually in this tutorial, we're going to be downloading, modifying, and running these two different LLMs that you see right here. The second technology that we're going to be using in this tutorial is going to be LLAMA2. Now LLAMA2 is an LLM that was released by Meta. And as you can see, this model was trained on 2 trillion tokens. And what I think is super important for you to know as a developer is that there are three different models and each one of these models comes with different RAM requirements. So in this tutorial, I'm basically going to be using the 7B model, 7 billion model, which requires about 8 gigabytes of RAM. So that's what we're going to be using. So if you have a beefier machine, you can go up to the 70 billion model, which requires up to 64 gigabytes of RAM. But don't worry, OLLAMA will figure out what your system has and use the right model based on your system requirements. And something else that's super important for you to know is that LLAMA2 was trained on a larger dataset compared to Mistral. However, when it comes to actually handling tasks, I've noticed that LLAMA2 doesn't always perform as well as Mistral. So that's just something to keep in the back of your mind when you're trying to pick between which LLM to use, LLAMA2 or Mistral. But we'll talk about more of this later on. And the third technology that we're going to be using in this tutorial is going to be Mistral. Now Mistral is another large language model that we can run locally on our computers. And this thing packs a punch. As you can see, Mistral is a 7.3 billion parameter model, and you can see that it actually outperforms the LLAMA2 13 billion parameter model and the LLAMA1 34 billion parameter model. So as you can see, this thing packs a punch and you're going to be able to see it later off in this tutorial. And the fourth technology that we're going to be using in this tutorial is Crew AI. And simply put, Crew AI is a framework that allows us to create and manage AI agents that work towards solving complicated tasks such as analyzing stocks, planning trips, writing newsletters and much more. So if you're looking to learn specifically more about Crew AI, I actually just released two tutorials that you'll definitely want to check out. The first video is a complete crash course for beginners on Crew AI where you'll cover the basics of working with agents, tasks and tools. And the second video is an updated video on Crew AI where we cover some of the newly released features of Crew AI where you'll learn how to build more reliable and faster crews. So those are the four technologies that we're going to be working with this in this tutorial. So let's go ahead and dive in and to setting up our local LLMs on our machines using LLAMA. So the first thing we need to do so that we can run our large language models locally on our machine is to head over to LLAMA.com so that we can download LLAMA. So what you'll need to do is click that download button. And what it'll do is it'll take about a few seconds to download the package and application because it's about 400 megabytes. But once it's finished downloading, what you'll need to do is just double click on the application and it's going to ask you to go ahead and move LLAMA over to your applications folder. So that's what I'm going to go ahead and do real fast. Now this will take a few seconds because I've already done it in the past, but you know, that's exactly what will happen on your screen. Once it moves over to the application folder, you'll see this little intro pop up saying, hey, welcome to LLAMA. So you're just going to click next and then you're going to be prompted to install the LLAMA to your command line. So you're just going to click install. So in my case, it might ask you to go ahead and put in your password. So do that real fast. And then once it's done, you'll actually see the LLAMA icon in the top of your computer. So that's how you know it's working. And then finally, it's going to say, hey, let's go ahead and run your first model. But we're going to ignore that for now because we're going to open up our terminal. And in my case, I'm going to use warp as my terminal. We're actually going to start actually testing out LLAMA to make sure it works. So if I type in LLAMA right now, I should actually see that it runs. And what I can do is to check to see which models I have. You can actually type in LLAMA list and this will actually list all of your different models and large language models you have installed on your computer. Now yours will be empty at first because you haven't installed anything yet. But what I'm going to do is just go ahead and show you how you'll be able to download your first LLAMA. So what you'll do is you'll type in LLAMA and then what you'll do is type in pull and then you'll just type in LLAMA to. Now how did I know to do this? Well, if you head back over to LLAMA.com and click on the model section and click on the model you want to download, you'll see in the top right hand corner, it'll say, hey, this is basically the command you'll do to run it. But we're just trying to pull it right now. So I'll go ahead and pull LLAMA2 down and this should take, you know, it's a big package, almost four gig, right? So it'll take a minute to actually download. But once it's actually finished downloading, you can actually run it. So we'll just now type in LLAMA run and we'll type in LLAMA2. Fantastic. So this should say, oh, I misspelled it. And this should actually go ahead and actually run the LLAMA locally on your computer. So you can just say, you know, tell me to dad jokes just to make sure that it's running and this will take a few seconds. And yeah, you can see it works just like ChachiBT normally would on your computer. So that's awesome. So now that we've went ahead and installed LLAMA, what we're going to do next is actually go ahead and start configuring our LLAMAs to get them ready to use with Crew AI. So before we can start connecting our large language models over to Crew AI, we need to do a little bit of configurations to our large language models. And don't worry, this is actually super simple to do. What we're going to do is create a thing called a model file. And let me show you what this looks like real fast. It's basically just a set of instructions that say, hey, I want a specific large language model to use a few specific parameters to make sure that everything runs smoothly. Now in our case, this is actually going to be super straightforward because if you head back over here, this guide actually tells us exactly what we need to do. In our case, it's saying, hey, I need you to add the result word as a key stop word inside of our parameters. Now let me actually show you what this looks like in action so that you, you know, this all makes sense. So what I've done so far is I'm actually over here inside of our GitHub code that I actually have for this project. And if you remember from earlier, you can actually download all the source code for this video completely for free. All you need to do is just go click the link down in the description and you can have access to all this code. So in our case, what we're doing is we want to make a LLAMA2 model file. And all we're going to do is say, hey, for my model, I want to grab LLAMA2 and I want to add two different parameters. In our case, I want to set the temperature. That's just, you know, how consistent or creative the results are going to be. And then I want to add a stop keyword so that our agents know to stop producing and executing on a specific task whenever it gets and sees that word, right, that result. And outside of that, I'm just going to say, hey, you're starting with a blank slate for, you know, the initiating and starting of the conversation with the LLM. Pretty much you can ignore that if that didn't make sense. Okay, cool. So now that you know what it looks like to actually customize an LLM with a model file, let me show you what we need to do next. Well, if you head back over to the instructions, what it says to do is just go and basically run two specific scripts. The first one is just, hey, using LLAMA, I need to create a new model using this specific model file. And after that, you can actually go off and run that model file. And what I've done to make your life easier is I went ahead and actually made that script for you. So what we're going to do is we are going to pull down that model, in our case LLAMA2, and then we're going to create our new crew AI LLAMA model using that model file. So I'm going to go ahead and run it and show you what it does. So all we're going to do is just type in, we're just going to run this code. And if this is your first time running it, actually, you might have to do something like this. It's called chmod plus X. And all that does is it actually says like, hey, turn this into an executable. But what we're going to do is just run our setup file. And this is our create LLAMA2. LLAMA2. Fantastic. So that should make it executable. And now we can run it. So I'm going to do setup, create LLAMA2. Now this should go start running. It'll pull down the LLAMA2 model because that's what we said to do. And then from there, what it's going to do is actually go ahead and create our new file. And I was actually in the wrong directory. So that's why I got that error. So I'm actually just going to change over to my setup directory, which you can see right here. So then I'm working from the same file path. So that was the reason I got the error. But if I run it again, what you'll notice is, let's change this. Fantastic. So I run it again. It'll actually now re-pull it down. And now it knows how to point to the right model file. So now if I run o LLAMA list, you'll now see that I have my new crew AI LLAMA2 latest and it was made four seconds ago. And now we're going to go ahead and do the same thing to go off and create our Mistral large language model that's custom for crew AI2. So once again, all we need to do is just type in, you know, chmod to make our script executable because we want this script to be executable. And then let's do that. Chmod plus X. And then we need to tell it which file. In our case, it's the create Mistral file. Fantastic. So now that's executable. And let's make sure our Mistral file looks great. I need to rename that real fast. Mistral. Fantastic. So this is basically the same one. By the way, we just added the word stop to this file. Now I can go ahead and run this one, but instead I'm just going to say create Mistral. Fantastic. So that should go off and create our new large language model, but it's going to make sure it's using the new Mistral large language model and it's going to specify it for our crew. So what we're going to do now, if I run o LLAMA list again, you'll now notice that we also had o LLAMA. You'll now notice that we also have a Mistral large language model that we can start using in our crews so that everything works smoothly. So I know that was a little bit of a lot, but once again, don't worry if you had any issues, actually feel free to just drop a comment down below and I'll do my best to help you guys out as quickly as possible. And feel free once again, to go grab all the source code for this video so that you can actually just run these scripts over here and that way everything will work smoothly. Okay, cool. Well, now that we have our large language model set up and ready for crew AI, let's actually start diving in to connecting these over to our simple example and actually get everything running. All right, guys. So I'm super excited to go ahead and show you our first example where we're going to connect our local language model over to a simple crew AI example. Now what we're going to do is I just want to show you a quick overview of the crew that we're building and then show you a quick output and then I'm going to show you how we got everything running and then we're going to repeat this for the advanced example in just a second. Okay. So what I have done is I have actually cloned down a example that was provided in the crew AI examples list. This is an amazing repository that actually shows you guys a few different ways that you can use crew AI. So what I've done is I've pulled down the markdown validator and what you'll notice is this markdown validator basically allows you to use a crew to go and analyze a markdown file and it will provide recommended changes that you need to make to the document. So it's super straightforward and what you'll notice is that like in the main.py file where we're actually running the crew, there's really only two things that happen. There's one agent and then there's one task and that's pretty much just how the entire thing runs. That one task is the only thing that's executed and it just, you know, it returns the final feedback that you need to make for your markdown file. So what I've done is I've actually went ahead and run this for you guys using the llama two that we just made it the custom one that works with crew AI. So I just ran it and I just want to show you what the final output looks like because I think you guys are going to be super impressed. So this took about a few minutes to run, you know, running some of the local language models actually takes a little bit longer, but Hey, it's free. So that's awesome. But what I did is I said, Hey, I want you to analyze this markdown file that you see right here and provide me feedback. And what you'll notice is it actually went ahead and use tools and everything to find that there was actually five errors in the document and then it tells you feedback on what you need to change in order to make this markdown file valid. So it's like, Hey, add some brackets over here, add some new lines and so forth and so forth. So like I said, I thought this was a pretty awesome example to show you guys that like, Hey, it actually works. Okay, cool. Well, now that we've like seen that these large language models that are running locally work, let me show you what I had to do to get this language model running. So what I'm gonna do is I'm just going to go ahead and clear all this out. So we're starting from a blank slate and I want to go ahead and actually show you guys what I did. So what I did first is I changed my directory over to the simple example folder so that I was set up right here. And just to give you guys a quick overview, we'll cover a lot more of this in that crew AI basics project tutorial that I created for you guys. But just in a nutshell, what I did is I installed all the dependencies that were in my pyproject.toml and this is super easy to do. All you need to do is just type in, you know, poetry install dash dash no root. And when you do that, it'll go ahead and grab all these dependencies for you and it'll make a Python environment that you can use to actually go ahead and use. And that's super simple. You would just type in poetry shell and whenever you do that, you'll basically get the same thing right here where it's like, Oh yep, you were using the markdown validation crew. Cool. So that's what I did. Now let's go ahead and dive into how I actually got our local language model working with this crew. Well, if you actually head over to the document that crew AI provides, you'll notice that it's very easy to set up. What you need to do is actually update your environment variables to point to your local host. That's actually running Olama and Olama actually basically sets up a server that will help facilitate all the communications between your crew and the local language models that you just set up. So it's up to you to update your environment files to point to this local host and then provide your model name. So let's go ahead and look at our project so you can see this working in real life. So what you'll notice is if you head over to the dot ENV file or in your case, they'll probably be a dot ENV and I'll go ahead and create it for you guys. So you'll have it, but you'll have a dot ENV example file and you'll just clone this to, you know, rename it to be a dot ENV. But what you'll do and you'll notice is we copied exactly what they said. We set up our base URL to point to our local host where Olama is running. And then outside of that, we pointed our OpenAA model name to be the newly configured large language model that we just set up. So that's how we were able to get it to run. And it's actually very simple. Whenever you head back over to our main.py file where we set up our agent, you'll notice that we did not set up an LLM where we actually pointed to, you know, hey, this is the large language model I want you to use. By setting up our environment variables, basically any new OpenAI chat large language model that we set up will actually just use this base configuration right here. And that's how you can tell it's working. And another thing that you can do just to like prove to you guys that it's working. If you actually go over to your root directory and go into Olama, what you'll notice inside of here is you actually have some logs. So if I go into our logs folder, let me go into our logs folder. What you'll notice is that we have our server.logs and that's all the logs that are going to come from here. So I'm just going to go ahead and check to see what's inside of here and our server logs. And what you'll notice is, yeah, we have a ton of different chat requests that we're just running. So, you know, this was just literally just a few minutes ago when I was running it before recording this part of the video. So, yeah, you can see there's a ton of stuff and it also just has some other general information about your computer as a whole. But yeah, so that's how you can actually get your crew to work with local language models. The other thing that I want to do is just show you option number two. So if you just go ahead, I'm actually going to just delete .env file just to show you that this guy is working. The other option that you can do if you want to be more explicit is you can actually create a large language model and you can say, hey, I want to create a new chat open AI. And then what you can do is actually provide your model name and your base URL. This is basically the same information that was, you know, defined in our .env file just a second ago. But we can actually just go ahead and hard code in here. Now that we're importing everything, what we can do inside of our agent is actually go ahead and point to it. Fantastic. So now we're actually going to be using this large language model that we just defined. So I'm going to go ahead and start running it again. Let's go ahead and CD back into our right file. I'm actually just going to close out of this and open up a new one just because it's easier. Fantastic. So now that we're back in our file, let's CD into our crew AI. Simple example. And then I'm just going to run Python main and I'm just going to point to our readme file. Great. So that's going to go ahead and kick off. This should work. I actually messed up. I should have done was just you actually still need to say that your open AI key is in a that should work. Fantastic. Now it's going to go off and execute. And I just want to pull up in a parallel task real fast just to show you guys that it's running in real time. So if we go over to Olama, once again, we go over to our logs. What we can do is cat that server file. Oops. Sorry, I'm struggling on the keyboard today. What I can do is just once again, cat that server log file and you'll see that things are happening in real time. So this is 455 local time. And if you look, it is 455. So you can see that's actually running in real time. Cool. So, yeah, it's just running. So those are the two ways that you can get your crews to start connecting to your local language model that's been configured to work with crew AI. So we're going to do next is we're going to hop over to the more advanced example next. And I'm going to provide you guys some more general tips and feedback that I have found when working with these local language models. And we're also going to use Mistral in the next example. So let's go ahead and dive on over. Hey, guys. So welcome to the second example where we're going to connect up our crew AI to our local LLM. In this case, we're going to be using Mistral. So let me give you a quick overview of this crew and what it's trying to do, show you the outputs. And then we'll actually dive into how we got everything set up. OK, so the first thing you'll notice is now we're in the advanced example folder and the synopsis or just quick summary of what we're doing is we're making a financial crew. And the whole point of this financial crew is for us to be able to pass in a stock. And then it's going to go off and research the Internet and find us everything it can about the stock and then return some, you know, hey, this is our opinion on based on what you should do for that specific stock. So I just ran this basically this crew for Tesla. And as you can see, here's the final output that we got from running this crew using Mistral locally. And here's everything that you need to know. So our agents came to an answer where we say, hey, Tesla's got some strong growth. Here's some few things that we just found out about this company as we looked at, you know, their quarterly reports to the SEC. And, you know, here's our summary where we say, like, hey, you know, we're cautiously optimistic about Tesla going forward. So all around, this is awesome work that I was able to do this. This one took a little bit longer because, as you can see, we have a bunch of different agents going off and performing a bunch of different tasks. We're looking on the Internet, so forth and so forth. OK, so once again, I was able to grab this example once again from the crew examples link. I'll have a link to this repo down below in the description so you can browse it for yourself. But let me go ahead and show you actually what I had to do in this code base to go ahead and get it to work with our Mistral local LLM. So this one was once again, pretty straightforward. All I had to do, let's just go and open up a new terminal so you can see what I had to do. So the first thing that I had to do was once again, go ahead and install all the dependencies for this crew. And what you'll notice is this one has a lot more. You know, we're working with a lot of finance data and SEC information, so forth and so forth. So what I had to do next, once I went ahead and, you know, did the normal poetry. First off, I had to obviously see CD into this file. So crew advanced. Now, once I'm in here, I'm going to zoom out just to make the terminal more not so cut off for you guys. But the next thing I did, obviously, same thing as before. You know, I did the poetry install dash dash, no route to go grab everything, no route. And then from there, what I did is I went ahead and actually just did poetry shell to go ahead and activate this crew. OK, cool. So once I had all that running, what I was able to do, I want to show you what I had to actually change in the code base. So the few major things is you'll notice the way that this file is structured. It's very straightforward. We are building a crew and as we're building a crew, we need to import agents and we need to give those agents task and then we need to compile everything together in a crew. OK, well, where are we passing in our LLM? Well, in this case, where we're doing that is if you go over to the agent's file, what you'll notice is for each one of our different agents that we have created, what we're doing is we are passing in which LLM we would like to use. And in this case, what we're doing is just now using our crew, a mistral LLM. And this one, like I said, basically exactly what you saw beforehand. We're passing in the model, passing in the base URL. And in this case, I actually just went ahead and defined API key. Basically, all you have to do to avoid the error you saw on the last screen when we were working with a simple example is you just have to put string in here. It just can't be empty. We're not even going to be using it, but it's just that's how you get around, you know, open AI giving you issues or sorry, the chat open AI giving you issues. So, yeah, this is literally all you have to do to go ahead and start working with your local LLM, but to actually get results that are meaningful. What I have found is that you have to provide a lot more context in your task to make sure that they run. So, for example, when you look at a task. Let me show you an example task. So when you look at like our research task, what you'll notice is we're having to pass in our agent and our company. OK, so that's how we're going to make sure that when we're researching a company, we're going to make sure we research for the right company. Well, in the example that you get from this examples list, you'll notice when you go look at theirs, you'll notice that they only provide the company here. And they were able to get away with that when setting up a crew because they were using open AI, which has specifically chat GPT-4, which seems to have much larger context window, meaning open AI is able to look at and understand a lot more data at one time. I noticed whenever I was running this crew, if I didn't put the company in every single task like I have set up here, the crew would go off and start researching random stuff on the Internet. So the workaround that you need to keep in mind when working with these local LLMs that don't have as large of a context window. You need to be very, very specific in your task on what you want them to do, because, you know, they're not going to remember things that were being worked on a few seconds ago. So that's just a tip that I'd recommend as you guys work on more advanced crews. And the other thing that I want to point out is as you get more involved with working with crews, you'll start working on, you know, newer features that are coming out with crew, such as you'll you know, you'll start to work with making processes that are hierarchical or you'll start working with making sure that your research task and all of these other tasks that you're looking at, you'll make sure that they're going to be asynchronous. So it's important to note that you can't actually use some of these more advanced features when working with your local language models. And the reason why is if you actually go over to what features are natively supported with each of the LLMs, you'll notice trying to go down to open AI for you guys, you'll notice with open AI, everything is supported. And by everything I mean, you're able to do invoke new requests. You're able to do them asynchronously. You can stream back data, so forth and so forth. And you can't really do that using some of the local language models that we're working with, such as, like I said, OLAMA. It really only allows for you to invoke a, you know, the local language model. So I just wanted to provide that for you guys, just as you go off and become experts at working with local language models. I just want you to know that this is a current bottleneck of working with them. But hey, as time goes on, I guarantee you that they're going to improve what OLAMA can do. So eventually, I'm sure these will also become green check marks. But yeah, that is everything for working with your advanced example. And if you have any questions, feel free to drop a comment down below. I'll be sure to get back to you guys as fast as possible so we can get you guys up and running and building your own local language models. But that's a wrap for this video, guys. I hope you all learned a ton about CRU-AI, OLAMA, LLAMA2 and Mistral. And if you did enjoy this video and want to see a bunch more AI related content just like this, you'll definitely want to check out some of my other CRU-AI or full stack tutorials that I have on this channel. So I want to definitely click out one of those videos that pops up on the screen right after that. But enough of that. I hope you guys have a great rest of your day. And I can't wait to see you all in the next video. See ya.