I tested DeepSeek vs. o3-mini for developers: Here’s what I found.

It's been a wild week in AI. DeepSeek is all over the news. People are losing their minds, and now OpenAI is fighting back to prove they're still the best and just released their new O3 mini model. And in this video, we're gonna answer the question that's on every developer's mind, which model should I use with all of these new options popping up left and right? And to answer this question, I've spent the last 24 hours researching and stress testing these models against the most common tasks developers do to figure out which one is the best. And at the end of this video, I'm gonna be sharing my recommendation and actionable insights on how I'm gonna start using these models at my AI software job and on my personal projects so that we can all hopefully level up and become better developers using AI. And to make things even easier for you guys, I've attached timestamps throughout this video so you can hop between the research and all the different tests that I perform on these different models. And because you guys are awesome, I'm also going to be giving away all the source code that you'll see inside of this video completely for free. Just check out the first link in the description below. But enough talking, let's go ahead and actually start testing out and figuring out which model is the best. All right, so before we just hop in and start testing out all these different models, it's important for us to quickly look at the core stats of each of these models, specifically their performance, the price, and their context windows so that we can see how each of the models compares to one another and go ahead and figure out which model is the best, at least on paper. All right, so let's go ahead and look at performance first. So the first thing you'll notice is we're using a score called the SUI Bench Score. Basically, there's this huge GitHub repository that has a bunch of issues in it and these models go off and try and solve those issues and that's how they get a score. So right out the gate, you'll notice all of the Reasoning models have super high scores. O3, O1, and DeepSeek R1 all have like that 49-ish and above score and shockingly, CloudSonic, a chat model, also has a super high score. So, you know, these are the best models. I was also shocked to see the worst model is GPT-40. It has a 33, which is just embarrassing. So if you're using 40 for coding, I would go ahead and cut that out now. All right, so next, the thing I wanna talk about is price. So right out the gate, by far, DeepSeek is the winner. You can see it's been taken over the news with how cheap it was to make and how cheap it was to run. So you can see right now, if you're using DeepSeek R1, it costs $2 almost to do a million tokens worth of output. The next best model right now would be O3 Mini, so the new model from OpenAI. It's about twice the price exactly and it costs about a little over $4 to get a million tokens. The worst performer in this category was Clod 3.5 Sonnet. I was shocked at this because I knew it was a great model, but I did not know how expensive it was. And as you can see, it's $15 for a million tokens of output, which puts it at roughly seven times more expensive than DeepSeek R1. So I'm excited to go ahead and test this out a little bit just to see you guys how well these models stack up, just see if Clod actually justifies that huge price. And then another thing that I do wanna talk about is actually the context windows of these different models. So right now you can see the best models have that 200,000 token max for the context window, and that's roughly a 500-page book, which is huge. So you can see the top tier O1 reasoning models have the 200,000 tokens, and Clod 3.5 also has that 200,000 token max context window. What is important though, besides just the amount of information you're allowed to cram in there, is the max amount of information you're allowed to get out of these models. So right now you can see, once again, the O1 reasoning models, O1 and O3 reasoning models crush it. They can spit out 100,000 tokens, which is just insane amount of information. This is gonna make sure that these agents do super well when it comes to agents, which we'll test in another video. But for right now, that's just amazing. It's able to put out that much information. The loser in this category though, I would definitely say are the DeepSeek and Clod models. The fact that they're only able to spit out 8,000 tokens means that they're pretty limited and you cannot single-shot build a whole project. But if you're just solving specific issues or specific files, 8,000 tokens should be more than enough. But it is just important to know out the gate the limitations of these different models. So right out the gate, it is important to go ahead and say, DeepSeek is, specifically DeepSeek R1, looks like the best model for developers on paper, but we're gonna put that to test here through the rest of the video. And just to go ahead and put some concrete numbers around all of this, I just wanna go ahead and show you guys what actually running these models would look like on a very common scenario. So let's say you're working on a huge file, probably around 1,000 lines of code or more. And let's just go ahead and say it had 10,000 tokens worth of information and it spit out an answer that was 2,000 tokens. Well, for price-wise, it would take DeepSeek R1, a penny, to go ahead and build a solution for that fix. It would cost 03mini two pennies and it would cost Clod roughly around six cents. So this is super important as you're using these models to help you build out solutions, just to have a rough idea of like, oh yeah, a penny or six cents to go off and solve these different common problems that I would have in my code base. And that stacks up over time because I know I'm calling these models like 100 times a day as I'm coding out stuff. But enough comparing the stats, let's go ahead and start testing out these different models and let's see if actually DeepSeek is the best model based on the performance that we've looked at so far. All right guys, so let's go ahead and dive into testing out these models to figure out which one is the best. And as a quick reminder, we're gonna be doing three tests which are focused on the most common activities that developers do. So the first test is about building a new project from scratch. The second test is about building a new feature in an existing application. And the third test is refactoring an existing file and generating tests for that new code. And just a quick overview, every time we dive into one of these tests, I'm gonna show you the prompt that we pass in. We're gonna analyze the LLM and see what it does. And then we're going to review the final output to see was it able to do the task that we actually wanted it to do. All right, so let's go ahead and dive into our first task which is all about building a new project from scratch. All right guys, so now it's time for test number one where we're going to see how these new models do when it comes to building a new project completely from scratch. And here's how this test is gonna work. First, I have one uniform prompt that I'm gonna be passing over to DeepSeq and O3 Mini to see if they're able to actually build the project. But because I wanted to test this out as much as possible, we're also going to be passing the same prompt over to DeepSeq in the web and in cursor. And same for O3 Mini. We're gonna be giving it to the web interface and cursor just to see the performance, see if it changes and actually see what the final outputs are. Then what we're gonna do is go ahead and save the code to our computer and then run it and analyze to see, hey, was it able to actually build a project from scratch? And in our case, our project is going to be, can you please generate a chat GPT style interface that allows me to talk to local models on my computer with OLLAMA. So basically I just wanna be able to chat with like OLLAMA3, DeepSeq R1, DeepSeq V3. Like I wanna be able to chat with these local models on my computer. So that's the prompt we're gonna be giving them. And to get even more specific, like I've said earlier, I have all the code available for this project for you to download completely for free. Right now we're in the first test. So you can see we're in the, I have a huge massive prompt for this because we're all about making sure we get all of our thoughts out of our head into a prompt to make sure these big reasoning models understand what we're trying to do and see if they can go off and tackle it. So let me quickly walk you through the prompt. And then after this, we're gonna hop over and just quickly see how all the models actually perform. So what we're trying to do, please go make a chat GPT style website that works with OLLAMA. I want you to build it using these three technologies. Now, what do we want to happen? Well, I basically want you to have a few core requirements inside the project. I want to be able to select which model to use in order to chat with it. And instead of just like directly reaching out to, you know, the normal port, you know, 11,434, we can't do that. I actually have a nice little proxy website. So we're throwing in a little bit of complexity of like, hey, you need to reach out to this address. Then I also give it some instructions on what the chat interface should look like. You know, you should have user messages, bot messages. You should be able to, you know, go ahead and like see as things are changing. It should say like, oh yeah, the AI is generating a response. From there, you should be able to have like a nice text input. You should be able to click a new chat button. So once we're done with the conversation, we should be able to hop over to another chat. I would like to store and see my chat history. So I can say like I had one conversation and then I hopped to a next. And then finally, I, like I said, just a quick few notes that I just wanted the AI to really pay attention to. We are using a proxy. So you're not directly communicating with Olama on the computer. I have this nice local host application on port 3000. And this is where you can send all of your requests to and it should just work. You're not gonna get any court issue. Yeah, so that's the prompt that gets passed over to cursor and passed over to basically over to, I have a second one that's very, very similar that gets passed over to the web. And the reason that there's a difference is because I am going to actually like really stress test these models. And what I mean by that is I'm actually going to use in cursor. You can actually point to some docs and I'm going to overload it and just put as much doc information as I can in there to figure out if the model can cipher out which information is important for building this when it comes to like, you know, working with pulling out information from an Olama request. So I have to let cursor handle that. And instead of for the web prompts, actually just pulled out the code snippets that were, you know, kind of important because the web browser has a much more limited input window and much more limited output window. So enough talking, let's go ahead and actually see how these models performed locally and on the web and actually see if they're able to generate the results. So we're gonna go one by one. So for our first test, we're trying to see if O3 mini in the web can turn our prompt into a full blown application in a single shot. So let me quickly run you through the exact steps that I did. So I pasted in the prompt that you can see right here and I made sure that I picked to use the O3 mini model. And this is going to, you know, use the reasoning to make sure it's using the best version that can code. And let me quickly walk you through what it was able to do. So as you can see, this is a monster prompt that we passed in. It is important to note that the web specifically on O3 mini does have a much smaller input window. Cause as you saw earlier, it had a 2000, 200,000 input token window. And here we're like limited to like maybe a few thousand characters. So not what I was expecting from the web interface, but hey, it is what it is. But then what I love is you can see the models thinking about, hmm, I need to go off and do each one of these steps. So it's walking through its exact thought process. It's not as detailed as DeepSeek that you'll see in a second, but I do like how it plans and then goes off and actually execute. So this is the whole point of a reasoning model. Now, what I want to show you next is it walks you through a little bit of its code snippet before it dives in and gives you the code. But then what you'll notice is this entire web application, it stores it in one single HTML file. I believe this is due to the fact that the web browser experience does not allow you to use the full context window output, which should have been a hundred thousand tokens. So it had to put everything in as condensed of a format as possible. So now that you kind of see what it can do, let's actually hop over to the code that I have set up and walk you through real fast what it looks like. So as you can see, it's one nice website with styles. It has, you know, pretty much everything it needs. It has a bunch of buttons, event listeners. It looks great. So this is pretty much the standard for going forward, but I just want to show you that, yeah, it did do the thing. It built a website. And if we actually load this website, we end up with this. And so let me walk you through what it did right and what it did wrong. So you can kind of see like, oh, wow, these models are insanely powerful. So right out the gate, what did it do? Well, it allows me to see all of my different models that I have running locally on my computer. So this is awesome. I can go ahead and say, yeah, I would like to use Llama3 or we can use DeepSeek. Now what I can do is go ahead and start chatting with that model. So I can say, hey, which model are you? Then I can go ahead and send it. The message will get popped up right here. But then if you actually dig into the console, what you'll notice is it's kind of hard to see right here, but basically one of the button event listeners, it actually messed up. So in short, it was able to generate a really great website, but not in a single shot, but still super impressive. So let's go ahead and hop over to the cursor version of this website so you can see it in action in our second run through using cursor. So now we're going to look at what we are able to do with O3 mini and cursor to build a chat GPT like style website. So what you can see is once again, I was using basically that same prompt, but what I did do is like I mentioned earlier is I passed in docs. So these docs are actually pointing to the O Llama API. It's a huge page that includes information for using everything inside of, you know, whenever you're running O Llama on your computer. So everything from chatting, getting version numbers and all sorts of other O Llama inquiries that you can make. So basically this is a huge request. Now, what I did after I went ahead and let it run. And once again, O3 mini lightning fast, it just instantly hops in. And one of the big notices what you'll realize is this time it actually generated different files. So we have our HTML file, we have a styles file. After this, we're going to have a JS file. So all in all, it was able to produce a lot more code. And I believe this is because we were able to, you know, actually fully utilize the model and get that 100,000 put token window. And, you know, you can easily change for models. All I did was just click O3 mini and it just went off and worked. So as you can see, it made three different files. I have them over here, the index script and styles. You can play with them yourself, but now let me go ahead and show you what the updated version of this looks like inside the website. So here's the really cool website that we were able to make with cursor and O3 mini. So let me walk you through what it's able to do. So first off, did it nail that we should be able to pick our different models? Check. I can go ahead and say, yep, I'd love to talk with Llama3. Then I can say, hey, which model are you? I can go ahead and send it. And then it's going to go off and say, it's generating an information. And then it's going to come back with an answer. And then I can say, can you please share more info about it? And then it's going to be able to like chat with it, just like I normally would be able to any other model and have that chat should be teach brands. Is it perfect UI? No, it's kind of gross, but it was able to actually do what I wanted it to do. So I now have the ability to have chat histories. It's giving me more responses and this is all running locally on my computer. So we're going to give a huge check and a pass to when it comes using cursor and O3 mini. So now that we got O3 mini out of the way, let's go ahead and see how DeepSeek compared performing the same single shot of generating an entire application from scratch. All right, guys. So now it's time to see how DeepSeek performs when it comes to building an application in a single shot. So like I mentioned, we're running the exact same task. We pass in the same prompt where we say, please build a website. We give a little information on Olama and the API library around it. And then we let DeepSeek go off and perform its magic. Now it is important to mention that like when you are using DeepSeek, it really does have an awesome like thought chain. This is by far my favorite model. I find myself reading through it's reasoning all the time. Cause like, I like the way it's like corrects itself and keeps pointing itself back on track and really thinks through everything it needs to build. And what you'll notice is once again, because we're in the web interface, there is definitely more limitations around input tokens and output tokens. So we ended up generating a single file like we did over using O3 mini in the web. And this one actually is a little bit shorter. Let me show you what I mean. So in this project, DeepSeek R1 web, you can see that this project, they are putting everything in a single file, but you can see it only ended up doing 291 lines of code. Whereas back when it comes to O3 mini, you can see it did almost 500 lines of code. So basically like, you know, three fifths as much information out. And if the 500 lines of code didn't work, we can guess that the 300 lines of code didn't work. So let's go ahead and look at it in action. So this is what DeepSeek was able to make. So, you know, it has the core elements there. It has the new chat functionality. It does have the ability for me to pick new models that I want to work with. So let's, once again, do a little bit too. And I can say, hey, what model are you? And then I can go ahead and press enter, and then it's going to go off and actually trying to respond. And it does. So even though it generated less code, it's actually able to interface and was able to build it in a single shot. But, you know, some of the features really don't work. Like there's not chat history, but the fact that it does in a single shot, allow me to talk to different models is impressive. So I'm going to go ahead and say right now, DeepSeek R1 web is a green, it's a pass. And now what we're going to do is we're going to unleash DeepSeek R1 and actually hop over to Cursor and see what it's able to do. So for our fourth mini test, what we're trying to do is see if DeepSeek R1 inside of Cursor can build the entire application in a single shot. So you can see we're passing in the same prompt. We're also giving it access to a lot of docs and what we're trying to do, you know, and this was obviously set to DeepSeek R1. And then what we did is we kicked it off to the races to let it go off and do its thing. Now, as you can see, just like it did in the website, it does the normal thought process where it's like, hey, I need to tackle this problem, chat interface. I need to build an API. It needs to have this and this. So it thinks through the full problem. It is a lot slower. It's probably about half the speed as O3 mini, but what is unfortunate is it actually just did an awful job. So after it's done thinking, it was like, cool, even though it pointed out everything it needed to do, it's like, cool. And all it did is it made a single file that's a JavaScript file. So we can't even actually run this one and actually go off and like try it out. So it completely just did not do what it was supposed to do. It did not build an application in a single shot. And just an observation that I will note is I have noticed that DeepSeek does actually struggle with huge context windows when there's a task that involves a lot of steps. So I actually tried to run this one a few times to see if it was just a fluke, but it really did just kind of get stuck on making a single file. So even though it was given the same prompt as the web, it made a single file. It just happened to pick, I'm gonna make a JavaScript file instead of a full on website. So again, I'm gonna give this one a fail. All right, now just as a bonus, let's go ahead and actually hop over and I wanna show Claude real fast just so you guys can see exactly how it performs doing the exact same test. All right guys, so now it's time to go ahead and dive into our fifth mini test where we're gonna see if Claude can hang with the big boys and build an entire application in a single shot. Now, we do the exact same tests, give it the instructions, give it the additional documentations and let it rip. And as you can see, Claude starts strong. It's like, we need to build some API. It's understanding the problem, but it's like understanding the problem in bits and pieces. And it's not really connecting the full dots of like it's supposed to be an entire application where things are integrating seamlessly. It's just a bunch of small code snippets of like what should get done. So this is a huge X red flag, Claude completely burned and crashed on this one. So lessons learned, if you're trying to build an entire project from scratch, stay away from Claude, it failed, it bombed. And a few other lessons learned, O3 mini was by far the best when it comes to building an entire project from scratch. So I would definitely recommend doing that. And DeepSeek wasn't far behind, but obviously just the O3 one just absolutely crushed it. It made sure it had the history, it had the chat bar, it allowed selecting the models, everything just worked. So O3 mini inside a cursor was the way to go. Now, enough of building a project from scratch, let's go ahead and hop over to test number two, where we're gonna start building out features in an existing application. And these results actually shocked me after running the first test. So I'm excited for you guys to see how things perform over here in test number two. All right, guys, now we're moving on to test number two, which is building out a new feature inside of an existing code base. This is probably one of the most common things that developers do on a day-to-day basis at their jobs. So this is probably by far the most important test. Now, what I wanna do is just walk you through what we're actually trying to test in this one. So you have like a good understanding of like, oh yeah, this is a very real example. So what we're gonna do is we are once again going to be testing DeepSeek R1 and O3 mini. But what we're gonna do in this test is we are going to be passing in a prompt saying like, hey, I want you to build this new feature. And we are going to be referencing multiple code files so we can understand like what we've done in the past, a few relevant code files, just so it has a good understanding of like where we're going and the current state of the code so it knows what changes to go off and build. Then we're gonna analyze after we run both of these, the code that was generated and actually try and run it to see if it can work. Now I will say right out the gate, this was definitely not a single shot. I could not get either model to instantly go off and build the feature I wanted. It was definitely a more complex feature with the moving parts. So I do wanna at least talk about as we look at the cursor logs and talk about exactly like my experience going and working with these models so you can understand before you go off and start using them yourself, what you can expect. So now let's go ahead and actually dive into the prompt and what we're trying to build before we actually look at what was going on inside a cursor. All right, so first things first, I work at Crew AI, I'm a software developer there. We just recently released a new feature where you can actually chat with your crews and so crews, if you've never heard them before, they're basically just a way to automate tasking using AI agent. Awesome platform, recommend you checking it out. But we recently built a feature called Crew AI chat. And basically what it does is it allows you to chat with your crew. So if your crew is responsible for going off and writing like an AI newsletter or going off and performing some work, you can actually just talk to that crew and say, hey, please go do work. And it goes off and does it. And we have it set up where you can actually chat with it right now through a CLI like you can see right now. But what I actually want to do is for it to go off and actually generate a web UI. It's something I need to work on this week anyway. So I was like, huh, I think this is a good test because I'm going to be doing it myself. Now, what do we actually want to do on our prompt? Well, I basically want to go ahead and say, hey, that UI, basically that chat feature I just built, we're going to turn it into an actual user interface that our users can use. So I went ahead and used a model and to go ahead and generate basically like a requirements doc to say, hey, LLM, go forth and build this for me. So I basically said, hey, it is your job to go off and build a Streamlit application. So an entire new feature inside of a code base, you can run it using this command and here's some of the key requirements you need for the application. Here's what the UI should look like. Here's what the chat functionality should look like. Here's what state management should look like. So all around, it's just a ton of requirements that the LLM needs to take into consideration and then go off and actually build inside of it. And then also, like I said earlier, I am referencing a bunch of relevant code sources so that it knows exactly how things work now and what it needs to do to update that code to get it to work for this new feature. So enough talking, let's go ahead and hop over to O3 Mini inside of Cursor and see how it actually did at building out this new feature. All right, so now it's time to go ahead and test O3 Mini at building out a new feature inside of an existing code base. Now, this feature is pretty complex. So we do have like a pretty big prompt that describes what's going on. And like I mentioned earlier, we are passing in a bunch of relevant files that it needs to analyze so it knows the current state, it knows the future state, and now it's trying to actually build new files and update existing files to get everything to work. So let's go ahead and review the results together and I'll explain my thought process and how I thought the model actually performed. So right out the gate, I was so impressed. It went ahead and generated the new files. So in our case, we'd need an entire new UI that's going to host all of our Streamlit code that we can actually go ahead and start chatting in a nice UI to work with our crew. So it crushed this. I was so excited. It also knew it needed to update our current command line file to actually handle this new flag because this is what should kick off the new Streamlit application. So it honestly crushed it. I thought we were like, man, it did it in a single shot, but this is where things started to go south. Just, you know, in this example, what happened is for the next like maybe four or five cycles of chatting with it, it could not understand for the life of it that whenever we are working with the crew AI repository, we are actually making changes because eventually people are going to create their own crews. So like crew AI is a package you install. And then from there, you go off and create your own crews. It could not understand how crews were actually run by actual users. So over and over and over again, I had to continually explain like, hey, you cannot run this file here. Here's what a project file from a client would look like. Over and over and over again, I kept telling it, nope, you're not doing it right. You're not doing it right. Eventually it figured out, I was like, oh, okay. I now understand what's going on. But then we started to run into additional code issues. Specifically, it struggled with Streamlet. So when it comes to using Streamlet, it just was not managing state. It wasn't properly passing around messages. So if I added a new message to the conversation, it just didn't care. Outside of that, it also didn't properly create the way we should interface with a crew. Specifically, it completely ignored that it needed to actually run the crew. It would just actually, all it did was make like a chat GPT style conversations where all it was was talking to the AI and the AI never ran the crew. Even though it knew it should do it. So all around, I'm gonna go ahead and say like, yes, it did eventually get there. So pass, however, pass with a lot of friction because a lot of this time was like, here's the error I'm getting, please fix it. Here's the error I'm getting, please fix it. And final note, just as you're going off and working on this on your own, it does have a cutoff date of 2023. So it was recommending a lot of features that were experimental, but they're actually not experimental anymore. They're actually real life fixes. So no more experimental. So that's just a kind of a quirk with working with it, but eventually it did work. And I was able to actually type in crew AI chat dash UI. So you can see, I ran it down here, but now what I could do is actually go ahead and start chatting with this UI. So I can say, please write a report on AI and healthcare. We're gonna write this, please write a report on AI and healthcare in 2025. And then what's gonna happen is it's actually gonna go ahead and kick off, it's gonna read the message, it's gonna kick it off and it's gonna clarify, do you want to write this message? And I'm gonna say, yes, please generate the report. Now, what it's gonna do from here is actually go off and do what it's supposed to do, which is go ahead and kick off the entire crew. And you can see in the background, I'll actually like move this over real fast, but you can see the crew is running in the background. So this is where it's actually going off, researching the internet. It's putting together everything into a nice outline. Then it's gonna populate the outline with results. So this usually takes up to like 30 seconds to a minute, depending on how much information it finds on the internet. But just know once it's done, it did do its job and it showed the final report. Yeah, just like this. It shows the final report in a way and I can continue to chat with it. So I'm gonna give it a pass, but a pass with high friction and wasn't the biggest fan. So now let's hop over to DeepSeek and see how it is able to handle and perform at building out new features in an existing code base. All right, so now it's time to go ahead and put DeepSeek to the test and see if it's able to build out complex new features inside of an existing code base. So what do we do? We did the exact same prompt. We gave it the exact same documents. It should go off in reference and we let it run. Now, what I wanna do is go ahead and explain my thoughts as this was happening. So obviously I loved the fact that it was telling me exactly what it was thinking as it was going and planning it out. It was slower, so I was not a huge fan of that. But what I can go ahead and say is like, it generated just like the old one. It generated changes it needed to make to the existing file base. Quick thing I did not like, normally it tells you where you should make this file. Like, hey, this is in your CrayI source CLI folder. You need to update this. It just inserted code. It didn't tell me where this needed to go. So I kind of didn't like that. But next it did go ahead and also realized it needed to make an entire new UI application, which was going to be responsible for hosting and showing the Streamlit application. But it didn't actually generate the whole thing. It just said, I need to do it, but it didn't do it. So it's like it needed to be reminded that it needed to make the UI. It's like, hey, you didn't do it. So I gave it a little bit information like, hey, you need to actually do the thing you said you were gonna do in your thinking step. Also, quick reminder, just like O3mini failed, this is a, Crew is a library that people actually use inside of their own projects and gave it a little bit of context. And the second, I kind of gave DeepSeek a little bit more guidance after its initial thinking, off to the races. This model absolutely crushed it. So it did a lot more thinking and it's like, I know exactly what to do now. It went ahead, it went ahead and made the Streamlit file. It added everything that needed to go in there, absolutely crushed it. And it explained some of the key integrations and changes it made. Then outside of that, it went ahead and improved the UI, basically, sorry, the UI flag. And it wasn't, like, still, once again, it needed reminding like, hey, just one more time, buddy. Like, this is not correct. And it was like, oh, thank you for the reminder. And it fixed it. So right out the gate, it was like, cool, I know exactly what to do now. And then it was ripping after this point. It was a little messy with its code. That's a common theme that you'll see. It generates code that's working after being reminded a little bit. The code it generates is super high quality, but it's sometimes a little careless with what changes it makes. So it's like, cool, I'll clean that up for you. Everything is good to go. And I had one Streamlit error because it just forgot one missing attribute. And that's it, it worked. So it actually probably took maybe eight to 10 iterations of chatting with it to build this entire new feature where working with O3 Mini took over 20. So it needed more reminder. It felt more like a genius short-term memory software developer. Absolutely love working with it, but you are more involved in the process. Whereas I know sometimes if you're being lazy, you just wanna fix this error, fix this error, fix this error. So O3 Mini, if you're trying to be lazy, got there eventually, less brainpower. DeepSeq required more brainpower, but got there way faster. So let me show you what I was able to do. So the UI, beautiful. I'm gonna refresh it. And you can see it actually has some spinning while it's actually doing some of the important things behind the scenes. So it feels more responsive. And then I can do the same thing. So please generate a report on AI in healthcare in 2025. And then the UI for this is absolutely beautiful. I think DeepSeq crushed it way more on the Streamlit application. It didn't run into any of the experimental issues. It just, like I said, boom, crushed it. So all around, my thoughts are DeepSeq is the winner when it comes to building out additional features inside of an existing code base. You, just being the software developer who understands how everything's working, needs to be a lot more involved. But you'll get there faster. And I will personally be using this inside of my code base to go off and make sure that I'm building high quality code and I'm getting there faster so I can just crank out features left and right. So that's it for test number two. Now let's go ahead and hop over to the third task, which is going to be cleaning up existing code base and generating tests. Something that all of us software developers do. And we're gonna see how these two models stack up against each other. All right, now it's time to work on our third test where we're gonna see who performs better, O3 Mini or DeepSeq, when it comes to refactoring existing code and generating tests. So basically what we're doing this time is we're gonna give one prompt and we're gonna pass in a very small amount of code and let the models go off and try to refactor and add tests. So this one is like, as you see, we're like slowly dropping and dropping and dropping in the context window to see which one performs better. So I'm super excited to dive into it. And let me just show you the prompt that we're trying to work on. Once again, all of this code is inside of the code that you can download for free. Link down below. But basically just saying like, hey, you're a professional developer. I want you to refactor this file. And also, can you just please move some code over here? So that's just a part of the refactor that I know needs to be done. So I wanna see if it can like on its own, figure out what changes it needs to make while also taking in my, you know, my desires of taking and moving some code from in this file over to this file to make sure that things are just where they're supposed to be. So let's go ahead and hop in and see how O3 Mini and DeepSeq compare and hold up to the task. All right, so let's quickly look at what O3 Mini was able to do for our prompt. So as you can see, we're referencing two files, nothing crazy. And we're just saying, hey, please go off and refactor them. So right out the gate, it was able to go off and refactor our crew chat file. So it's adding in all the changes. It's making sure we're following best Python practices. And if you actually look at what the change file look like, I'll just go ahead and show you real fast. So you can see when it comes to our crew chat file, we're just adding additional commentary. We're cleaning things up, make sure, you know, basically just making sure that we're following best practices. A lot of these are comment changes, but you can see, you know, we're trying to follow using proper translation. So that was the main thing that I wanted to support in this one. You can also see that we moved over all translations to our internationalization file, where we have all the different languages and all the different, you know, commands that could probably come up. And then finally we said, go off and generate tests. Now, it did not crush test. Specifically, it messed up a, you know, when it comes to a type, it made up its own type that I would probably have to refactor, or I can just say like, hey, here's what our other tests in our test file look like. Please just make a change to follow our current testing best practices. So I think with one additional command, we could get O3 mini where it needs to be. So all around, was I happy? Yes, it quickly cranked out the code on this one and it did exactly what I wanted it to do and makes my life easier because this would have took forever moving everything around. So we're going to give O3 mini a huge pass on refactoring and adding tests. All right, so we're on our final test to review and we're now checking to see if DeepSeq R1 does a good job of refactoring and adding tests. So I went ahead and added in the exact same prompt. As you can see, DeepSeq went off, thought through all the best changes we need to make to make sure we're following Python best practices. And once it was done with that, it went off to the races and started implementing changes. It wanted to really improve error handling and really focus on adding translations. Now, here's one thing that it actually failed. I'll just go ahead and say it. It actually kind of failed pretty hard. It looks like on a refactor, what ended up happening is all this like existing version check functionality just was not properly integrated. So you can see, I'll go ahead and close out of this so you can see what went wrong. So it basically, it ended up cutting a lot of important code that was needed to actually run the file. So let me show you now, if I just go to my run crew, sorry, crew chat, what ended up happening is it ended up dropping some of the like important code that was needed. So like this start loop, it just kind of made that, that's like one of the most important functions and it just completely dropped that. So like, yes, you refactored and like, yes, you did add in a lot of error exception handling, but you only like partially did that. Cause like, as you can see, like, this is just a lot of like, yeah, pass or pass. So it's not really doing the full refactor. It just kind of did a stub out. So that was kind of a disappointment. And like I said, it actually just hardcore broke the code right here by dropping and adding additional stuff that it just was not supposed to. So all around huge, huge, you know, red flag. And that's a note to all of us developers. If you are refactoring with DeepSeek, it struggles with like, you know, I guess understanding complete context. It's more of like a sniper at like solving one specific problem. So without specific instruction, it kind of just went a little haywire. So all around not super impressed with it. So it was like automatically like fail on this one, just cause like, you know, if you were not a hundred percent paying attention, you were just like, oh yeah, it did what I asked it to do. It would just broke your entire application right then and there. I will say though, it did, you know, handle the scope of adding different things over to our, you know, our internalization file internationalization file for different languages. And it did go off and create some tests. So like they did do most of it. It's tests did pass. Like there's no errors over here in the test file, but the fact that it broke the main file it was supposed to work on, you know, that's very alarming and definitely means you need to pay very close attention when using DeepSeek. So now that we've like looked at all the tests, let's go ahead and look at my final recommendations and how I'm going to use everything that we just learned about the quirks of these different models and my own development life cycle, and a few ideas for how you can start using it as well in your own project. So let's go ahead and hop to the quick summary. All right, guys, that's a wrap. You guys are officially experts when it comes to DeepSeek and O3 mini, and I quickly just want to recap some of the core things that we talked about today when it comes to the research. And as you guys got to see these different models performing against the three core tasks that developers do on a day-to-day basis. So in my opinion, the winner overall, like asterisks, not by much is O3 mini. It does beat out DeepSeek R1 barely. And now here's my reasons why. Let me know if you have a different opinion down below, but the main reason is I liked it that it was faster. It gave me consistent, reliable results. I felt like DeepSeek was a little hit or miss on some of the bigger tasks. Like sometimes it would like completely crush it out of the park. And then other times it would just kind of just like not do what I wanted at all. It needed like additional remindering, whereas O3 mini was just more consistent in the long run. And then outside of that, that bigger context window that O3 mini gives us, as you're going off and making bigger file changes, that's happening a lot for a startup or just moving very fast, a lot of changes. So a bigger changes. So more output tokens was a huge reason for me to really like O3 mini in my case. And I do just want to say like DeepSeek is phenomenal. I absolutely love it. And I do want to talk in a second of how I'm actually going to use these different models together in my workflow. And I would definitely recommend you trying out as well. So we'll talk more about that in just a second. But I do want to cover also the biggest losers when it comes to these new model updates. And that is clearly CLAWD 3.5 and 4.0. CLAWD 3.5, it is so expensive. It cannot generate a ton of tokens. So at this point, I do not see a reason why you would keep CLAWD at all anymore going inside of your cursor terminal. If anything, just go O3 and DeepSeek and you are golden. All right. So now how am I actually going to start actually using these models in my workflows going forward and how you can use them too? So my new approach after testing everything and going deep and getting a PhD on these new models is I'm going to have what I'm going to call a like go wide and then deep approach. So what I mean by that is every time I'm going to start working on a new anything inside of cursor, my approach is going to be using O3 mini to go ahead and explain what I'm trying to do, brain dump, have the model ask me questions to make sure we're on the same page. And then from there, I'm going to let O3 mini go as far as it possibly can. And as soon as it's done building out what I'm going to call like a V1 of the code, I'm going to hop over to DeepSeek. DeepSeek is a much better like sniper of a code editor. So like if you're working on an individual, like a few functions, it's a phenomenal, but when it comes to like connecting all the dots and understanding how everything relates, it really seems like O3 mini is phenomenal at that. So O3 mini is going to kind of push the core stubbed out code out. And then once I went to like deep dive into building out individual functionality, I'm going to hop into DeepSeek. So the root answer is I'm going to use both, but that's my approach. But I would love to hear how you guys are going to use these models. If you want to actually just go all DeepSeek, all O3 mini, please let me know. I'm excited to hear what you guys have in the community. And then I'm also curious to hear what projects you're working on with these models. So, you know, let me know if I'm right. Let me know if I'm wrong. I'd love to see if there's any other cool approaches you guys come up with as well. And that's a wrap for this video, guys. I hope you all enjoyed this DeepSeek versus O3 mini comparison. I can't wait to hear what you guys think about it and which model you end up choosing to go off and start using as your primary model. And just a few quick reminders, all the source code in this video, I was giving away completely for free. Just check out that first link down in the description below. Also, I do have a free school community where like-minded AI developers all meet up and help each other out. I also have some free weekly coaching calls in there where I get to hang out with you guys every Tuesday to help you move forward on your AI projects. So definitely check the link down in the description below so that we can all hang out. But besides that, I have a ton of other related AI content here on this channel. Everything from Lang chain, crew AI, just everything that you can think of, I've been working on it. So I definitely recommend checking out whichever video pops up next on your screen. And until the next video, I can't wait to see you. Bye.