New Agentic Coding Model Ornith 9B — Is It Worth Running Locally?

So there's an interesting new open source model called Ornith 1.0 which is meant to be really good at agentic coding. So in this video we'll be testing out the 9 billion parameter dense model on a 16 gig Mac Mini to see just how good it is. So Ornith 1.0 is a brand new family of open source models which is specialized for agentic coding. They were released by a company called Deep Reinforce which is based in San Francisco and Ornith comes out in four different sizes. The 9 billion parameter dense which is what we'll be testing today on a 16 gig Mac Mini. We also have the 31B dense, 35B MOE and 397B MOE. I'm pretty sure these three the 9B, 35B and 397B are open weight which means you can just download them and run them today on your own device and just the 31B is not released maybe not released yet and it might be released later in the future. So there's some pretty interesting claims throughout this release paper. The first being that Ornith 1.0 was built on top of a pre-trained Gemma 4 and Quen 3.5. Gemma 4 in my opinion is one of the best agentic task completion models. So you plug it into your Hermes agent or your OpenClaw and you get amazing results. And Quen 3.5 in general the Quen family of models is one of the best local AI coding models that you can get. So throughout this paper we have three different performance evaluation charts. We got one for the 397B which we're not focusing on for this video. We have the 35B MOE as well. Actually this is really interesting I've downloaded this model already to my computer mainly because they claim that this model can outperform the Quen 3.6 35B but for this video we're focusing on the 9B. Now this performance evaluation chart in general I'm always skeptical of these kinds of evaluations. The best evaluation in my opinion is testing it on your own use case using your own hardware. We have here I just want to look at the terminal bench 2.1 and the SWE bench verified and pro. So a good distinction here is that these benchmarks don't actually test you building something from scratch. They test the possibility to either one work in an existing harness and be able to like execute tool calls do like basically run around in the terminal doing tasks. And the other test is for you get a problem you have a bug in an existing code base which you have never seen before and then it's the possibility to like read the code base and find the bug and implement a fix for the bug. So this is a that's a different category of using the model because on one hand you have can it work in a harness can it find an issue and the other you have can it build something from scratch. And I think a lot of people at home especially myself I want to be able to get a model that can build me something from scratch so we'll be testing that out a little bit later. And then finally we have the claw eval and I think this is the open claw evaluation so the model's ability to work inside an open claw and be able to like run around and do your tasks for you as well. And as you can see for all like for these three types of evaluations the Ornith 9B is the orange one and then I just really want to focus on the 35B and the 31B. So 35B is in the middle 31B is on the edge and you can see it's like the performance is very very similar for these. So yeah I guess it's like pretty interesting in general the 9 billion parameter model is very small it is very small and we're going to be highlighting some of the I guess some of the issues that come with using a small 9 billion parameter model when you want to use it for coding versus using a 35 billion parameter model when you want to use it for coding so I will actually rip out some of these bigger models and just compare them side by side for the output versus the 9B. So just to speak about our device quickly we have an M4 Mac mini with 16 gigs of RAM I'm going to leave four of those gigs available for the operating system so I can continue recording on here browsing and using different applications and we'll be using roughly around 12 for the actual model itself. And then to run our local AI models we'll be using LM Studio this is a free desktop app that you can download onto your computer it lets you download local AI models run those models and then plug those models into different tools. So I've just opened up LM Studio if this is your first time using this app I'm going to link a video somewhere on the screen that you can go ahead and watch that will teach you how to get started but let's go across to our model search and I'm going to type in Ornith and let's see what we come out with so we have the 35B up top but we're interested in the 9B 9 billion parameter model this is by Deep Reinforce AI so the actual company that released the model and over here we have different quantizations we have the which is 5.63 gigs I've already downloaded this to this machine all right so let's load up the model into our memory as you can see I've got two different variants of the Ornith 9B the MLX and the GGUF. MLX is just a little bit smaller and typically you want to use this format when you have your Apple silicon because it's going to be faster especially for pre-fill but for this video let's just stick with GGUF. Now we can see we've got 6.2 gigs consumed for the model weights and just 8,000 tokens we have a maximum of 260,000 tokens for our context length so let's go up on the scale and see what we reach at max okay 14.45 gigs from our maths before we want to stick roughly around 12 gigs and we can see here 186,000 tokens which is exceptional that's absolutely fantastic in a previous video we did the Gemma 412B and I think we could only get like 70,000 tokens before we hit you know our 12 gig limit but if you do want to get a little bit more context you can go down to flash attention you can enable this and you can quantize down your KV cache or cache whatever you have you say it to Q8 actually for both of these and then you can get let's just go to the top and you can increase this to the full context window and just consume 10.29 gigs so yeah it's pretty good for us I'm not going to do this I actually want to leave it as full quantization for my KV cache let's bring it down to yeah 185,000 tokens and let's load it into our memory and now that we've loaded up the model into our memory let's open up our activity monitor and inspect what the damage is so you can see the massive spike here as we load it into our memory and we're currently just over 15 gigs out of 16 gig available now that we have it loaded up let's just test out what kind of tokens per second we can get gonna send in a basic request as we were looking at before we had the GGUF and the MLX variants of this model for a very small conversation or a small message like this the tokens per second output is going to be very similar but as you increase the total like context or you have like if you're sending 4000 tokens versus four tokens that's where the MLX is going to be faster so we can see over here we have roughly 16 tokens per second this is a bit underneath my benchmark yesterday I was testing this out as well and I had roughly around 18 tokens per second and I think that might be partly because I've got some more applications open and we were like being throttled somewhere because we're just using too much of our memory so now that we have the model loaded up in our LM studio I want to test the claim over here that the Ornith 1.0 is specialized for agentic coding in general I don't expect much from a 9 billion parameter model I do expect a lot more from 31b and 35b but for now we'll just give it a relatively basic task so on the screen right now you see a youtube short that I recently posted I gave quen 3.6 35b a very basic prompt and in under four minutes I had a working tower defense game so we'll give a very similar prompt to our 9b model and see what we can get so to set up our coding environment I'll be using vscode as our code editor and then I'll be using pyagent as our harness to plug our model into actually my favorite code editor is claude code but unfortunately when you run some of these small models on smaller hardware just the wait time is really long but on pyagent it's a lot faster and it's it's much more usable so let's just kickstart py let's forward slash and make sure we can choose our model and we're just going to go down to ornith 1.09b let's send in a sample request and let's make sure we get a response from our model in LM studio there we go we're just processing it right now so it is pretty quick like the actual processing of the prompt if we had it in cloud code for that first request will probably take us like three to four minutes so this is definitely much more usable I'm just going to give a very simple prompt basically asking to create a new html file called 9b-demo and to build a replica of tower defense as we saw before that game was was pretty simple or at least it looked pretty simple to build nothing too intense and to deliver that in a single html file and it looks like ornith has already gotten to work so the user wants to build a tower defense game as a single html file we have a bit of a plan over here from my previous testing for the 9b model if i'm ever trying to do like debugging or something there's a lot of thinking tokens applied so this is this is really short but in general there's a lot of thinking tokens and when you read through some of the messages it does sound like the 9b model is pretty smart and over in LM studio we can see we've done a couple of generations and we're currently at 2000 tokens so yeah hopefully we can get a really good result actually while we're here let's also load up one of the bigger ornith models to do a quick little test i'm going to drop down go to my remote models so i've got a couple different devices plugged into my LM studio and by using this thing called LM link i can actually pull the pull my models across any device so i'm going to try and find my ornith 35b 4-bit mlx community and also pipecat are some of my favorite providers for mlx models and this is on my mac studio so it's a bigger device i can go full context window without having to worry about anything i'm just going to copy this for myself for now and just load model and looks like our 9b model is still working away so we're at like 16 tokens per second so i think it's going to take us some time to get that initial file depending on how comprehensive the file of the implementation is going to be it might take us i don't know maybe five minutes to get that initial file back but while we're waiting for this i'm just going to pull up another pi session and just see if we have our next model configured here just going to go forward slash model what models do i have okay 35b so i do have it here as well let's test it out let's send a hi message okay so that was too fast for us to see so it would have come from this mlx let's give it the exact same prompt here we go create me a html file there it goes called ornith mlx demo so we don't get to see the generation speeds here but it's it'll be going very very fast and you can see the 9b model is still going at seven and a half thousand tokens i wonder when it's going to deliver that file actually while we're here let's test out the 35b for speed as well and let's go high there and let's see what kind of okay so yeah 100 tokens per second so this is like five times faster and don't forget we will have the 9b on a small bandwidth device and now we have the 35b on a bigger bandwidth device but we're not dealing with like apples to apples for parameters and it looks like we are also done so let's go to vs code and check this out open the file in any browser let's give ourselves some space over here and we had the mlx demo so we just right click and open in integrated browser all right so now we have the full screen so you can see here we have the archer cannon ice mage and sniper let's start the game we're selecting in an archer here oh it's not shooting anything okay well that's uh that's obviously a problem for us this is not good uh let's just close this so i've just come back and i let the 35 billion parameter model know that the game is cool but when i place the archers we're not actually they don't actually shoot at the enemy so let's see if you can fix that and as you can see there's a lot there's a lot going on here uh which is pretty surprising so i tried this exact same prompt yesterday using the same model on my mac studio and it worked in one shot i was able to get a really cool game um that was uh that was working perfectly fine for me before we keep testing that out i actually want to see this is kind of it's kind of hung here we're at 800 tokens or maybe it's not okay maybe it actually created the first version could not find the edits let's see what kind of output we had here so i think at this stage this is a similar experience to what i've had when i've been using this model for the past couple of days uh let me just verify so the let's open it in a browser and yeah okay it doesn't really doesn't really look good at all and nothing can happen yeah okay so this is this is uh this is as expected to be honest it's as expected we'll see what what happens here but let's go back to our main boss mode and see what we had let's put this back into a browser we'll right click see if this actually gives us a nice result zoom out a couple times start game okay and now we're shooting at these guys yeah really cool so it actually works can i place that i don't have enough cash yeah maybe not we'll just place this here okay so that's a working game it's actually uh it's pretty simple let's see how big this file is okay 600 700 yeah okay 750 lines whereas our demo here was also just as big looks like five six seven hundred eight hundred nine hundred lines so yeah pretty much like for like so what you see here is me struggling to debug the 9 billion parameter model code in general orinth produces good code at 35 billion parameters we have a fantastic game and it's very fun to play but at 9 billion parameters these smaller models not just orinth but like quen and other models they don't have enough precision so what i found across the last couple days is that they struggle with functions they'll create a function but not declare it they'll maybe have an empty function sometimes their tool calls will be broken then py agent cannot understand what's trying to like what are they trying to do but when they do produce some kind of code so let's say this html page i initially tried to debug with the 9b model itself but it just kept going in loops and loops and i think we just made things a lot worse but so then i had to get the 35 billion parameter model and i gave it the code base and i went through like four or five iterations and ultimately came out with this okay so looks like we have uh canvas we have the track here now let me set up some of these little whatever these things are so no enemies are coming in but we're on wave two and wave two out of seven six five okay so the enemies are going down we're probably just getting them off off the view it looks like yeah i guess this game it kind of works okay now we have more enemies coming through but wait why aren't they wait where's a sniper is it getting these guys can this get it okay so we can get it all right that's not that bad i mean we've we've been able to salvage the game from whatever we had before which was not working with the 9 billion parameter model to now being able to work and play it's not as nice as the 35 billion original model but i mean it's something so for me the main takeaway is and by the way this is the 35 billion parameter game the main takeaway is that when you're using a 9 billion parameter model for coding tasks you either have to reduce your expectations or you have to reduce the scope of work because as you saw here oh man i want to lose these lives as you saw here we weren't able to build the app correctly in the first place the 35 billion parameter model has the capacity so the model has the capacity to do good work but at 9 billion parameters the accuracy isn't there you have misspelling you have functions that are not correctly written you have functions that are defined but never used later so all these little little kind of zipping the app up together is not it's not really possible with this 9 billion parameter model there might be some like tricks that you could do oh no there might be some tricks that you could do to try and change how you prompt to change the work that you do with this model but it looks like for building a tower defense game it's not possible right now