Cactus Needle - The 26M Function Calling Model

Okay, so this is a tools JSON definition. You've probably seen these many times. And if I come in here, you can see I can do things like set a timer for 12:30 p.m. today. I can put in a query. I can run it. And sure enough, I get back a function call response. If I ask it for something like take a quick note for me about the new model from cactus. And you can see, sure enough, function call. Again, we've got create note being the function. And then we've got some text and a title in there. What about email Matt about the podcast? And you can see sure enough it's got send email arguments is who podcast and then we've got the body in there. Now none of this is super surprising here. All the Frontier models can do this. A lot of open- source models can do this except this model is only 26 million parameters big. Not 26 billion, not 1 billion, 26 million. In this video, I'm going to talk about Cactus Needle. So, Needle is an opensource 26 million parameter function calling model. They claim that it can run up to sort of 6,000 tokens a second for prefilled 1,200 tokens per second on consumer devices. So, this is still from Gemini 3.1, specifically from 3.1 flashlight. And my guess is that if this had been created by a Chinese model, people would be freaking out about the distilling, but it's created by one of the YC startups called Cactus. So that makes it perfectly fine. Now, this model is available on HuggingFace. You can see that it's up here. They've got a whole bunch of things about it. I'm going to talk about the architecture in a second. And the cool thing is they've got a GitHub repo up as well where not only can you download the model and use it, you can actually fine-tune it even without a GPU. This model is so small that you can just fine-tune it with a CPU. Just quickly, I joke about the whole thing about the company Cactus not getting in trouble for distillation when other people are, but they are actually doing cool things, too. So, Cactus is a startup that's focused on how to make use of models both at the edge with sort of backup and redundancy to the cloud. So they've put out some things already where they can do sort of transcription where a large amount is done on device but then for parts where it gets harder for the model it actually falls back to perhaps a better model on the cloud. So that idea alone is really interesting and clearly they're championing the whole idea of very small models on the edge. In fact their cactus compute engine is pitching kind of like being lama for mobile devices. Now that brings us back to needles. So the argument here is that agentic experiences are really just built on tool calling and in many ways having massive models for that is really kind of overkill. The argument is that tool calling is fundamentally retrieval and assembly. You're basically matching a query to a tool name. You extract some arguments. You basically respond in JSON. This doesn't need hardcore reasoning. And this is where they kind of differ with the previous video that I did around the mini CPM5 which was a 1B model which is still fundamentally a normal large language model. What Cactus have built is what they're calling a simple attention network. The entire model is just attention and gating. No MLPS or dense layers anywhere in the network. And that's definitely something that's quite unusual. So there's no FFN layers in here. There's none of the sort of standard feed forward transformer block in here. Normally what you would have in a transformer layer is something where you've going from a layer of a certain number of neurons, let's say 512, you expand that to four times that, so maybe 20 48, and then you bring it back down and that part of the transformer has actually been shown to be the area that sort of contains or stores the knowledge. And there have been some really nice papers out there around mechanistic interpretability etc. where if you sort of just play with those neurons you can change whole things of where a model suddenly thinks that the capital of a country is a different city. It remembers facts differently etc. So what cactus have done is basically get rid of all of that sort of knowledge store in there. And this really allows them to get a model that's really tiny but still has a lot of the ideas around attention etc from the standard transformer models. Now once they had this architecture done they've then pre-trained that on 200 billion tokens interestingly just using 16 TPUs for 27 hours and then they postrain it on two billion tokens of synthetic function calling and that data set has 15 tool categories in there. When they compare this to models like Function Gemma, which is over 10x bigger, the IBM Granite models, the LFM models, the Quinn 600M models, which are 20 times bigger on singleshot function calling, they're beating those models. Now, I do think the key thing there is singleshot function calling. If you're basically trying to use this for doing parallel function calling or some of the more fancier things that you can do with function calling, it may not work or you may find that it just doesn't work as well. Now, like I said before, they've released a full repo and stuff like that. You can basically use this with Python. In their repo, they've even got the Jax code for actually doing the pre-training in here. And it's a nice small repo if you want to go and sort of see how some Jax code actually works with using Flax, with using Optex, which is the optimizer library for Jax. And I will note for those of you that know Jax, I'm not seeing NNX here anywhere. The other thing they've got in the repo is just a simple diagram of how this actually works. So you've basically got a 12 encoder stack going on here of self attention with group query attention etc. But obviously none of the FFN layers in there. They've also got some interesting things in here why they went for an encoder decoder rather than just a typical GPT architecture of just decoders alone. And they got some nice comments in here about how they did the training with their optimizer and how they did quantize aware training in here to keep it so that the model would actually work when it was quantized and even smaller. And overall, it's a really wellp put together idea that you can actually test out yourself and see what it's going to work for. That this is not trying to replace the larger language models. If your type of function calling needs a lot of reasoning over it and you've got perhaps lots of tools in there, then this is the thing that's not going to work. I doubt this is going to work as is for things like skills where you've got progressive disclosure in there. But when your models are so small like this, you could imagine having multiple models where you use some kind of cascade architecture to decide that okay, this tool call belongs to this set of functions and go to another model for that. Anyway, let's have a play with it. And I put together a couple of demos that we can try some things out as well. And the cool thing is, like I mentioned earlier, you can even fine-tune this on a CPU machine. All right. So, if we come into the demo, you can see over on the left here, we've got a tools JSON file. So, this is basically the function calling that the model can support. You can see we've got one function set timer, another one send email, another one create note, and that's the three that we've got in there. So, if we come in here and try these out, and I ask it, give me a timer for 2 hours from now. And you can see that it's going to basically run. It's not super fast here, although it, you know, it's loading up. It's not running on a GPU, by the way. This is just running on a on a CPU. Set timer arguments for time human hours. Create a note for myself for next Tuesday to to interview Harrison Chase on stage at the Venturebe conference. Okay. So, now we've asked, you know, create a note. We've got the create a note. I'm writing to discuss the interview on stage. Not kind of what I said, but okay. I guess it's, you know, it the title is certainly right, you know, in this case. Send an email to Google to complain about the quot for anti-gravity. Okay, email. It's got the company name. Okay, what if I change it to something like this where now I've actually put the email in there. Can it extract this out? And the answer is yes. It did, you know, a good job of sort of extracting that out. So here we've only got three functions going on. Let's look at if we wanted to do something a bit more. Okay. So what we can do now is we can actually upload a set of JSON for our own set of functions. Okay. So here I've uploaded a set of functions that relates to a smart home. In here you can see we've got some functions around like garage door, alarm system, call emergency. There's a whole bunch of different functions in there. And if we come and try one of these out, turn off the lights in the living room, please. We can see that even just using their model once we give it our functions, it's actually doing pretty good, right? Control light room living room action off. Quickly call the police. Again, it's done pretty good at, you know, being able to call emergency service police. Can you turn the temperature up to 65°, please? Okay, it was able to do set the thermostat. What if we change that to something like Celsius and it still is able to do that, you know, quite easy. You can see here that the descriptions for these functions really help to sort of describe what it can actually do in here. Let's say please make sure all the doors are locked. And again, it's got for for front door, okay, in this case, but you can see that okay, we haven't given it any training data. We're just using the the sort of standard model that they had. Now, this data that I put in there was not from the Cactus team. I literally asked Claude to write me up a JSON file. I gave it their three functions as examples and asked it to basically write these up and give me a set of these for a smartome. So, it's not like their model has seen this exact data before, right? It's probably seen things very similar to it and that's why it's able to do it. So you can see that going through this it's it's able to work out most of these. Let everyone in the house know that it's time for dinner. And you can see going through this we've got yep broadcast message right it's it's time for dinner. Okay so this has done pretty well at new functions that I've given it for this use case. Now this may be something that they've kind of trained on. In fact, I think it it it pretty much is for some this kind of use case makes a lot of sense, right? But imagine you picked something totally different for a toy or for something like that and then you weren't getting great results. You can just come in here and fine-tune on these functions. And so in that case, you paste in your Gemini key. From what I understand, it's going to go and make a data set for you. it will then fine-tune these functions into the model so that suddenly it should be able to do your tasks and stuff, you know, in here and and then it's just going to save it out as a as a pickle file. And like I said, I don't think you even need to have a GPU. When I did some training with it, it seemed to, you know, work pretty well and it seemed to generate pretty good results out there. So, this is definitely an interesting concept of having a really small model that can do function calling. it doesn't have a lot of intelligence. So perhaps if your functions required a lot of reasoning, maybe it's not going to, you know, do super well and stuff like that. But for a lot of simple tasks like we would want to put into hardware for things like on off, for things like change the volume, all those sorts of things, this is going to be fine. And this is only 26 million parameters. This can run on a lot of sort of consumer home hardware. So things like, you know, Samsung fridges, that kind of thing. You can imagine that they're looking at putting these things into their devices to suddenly make their devices just more intelligent. Now, you could also team this up with a bigger model like we looked at last time with the mini CPM 1B model, and then that could actually empower even more intelligence. And you could have it so that like using the bigger model was a tool and so that the model itself would decide when it goes to a bigger model for doing something else, right? So that's something you could think about doing some kind of cascading system in there and training this. You could also have multiple versions of these things where you've got, you know, cascading stuff going through here. Now I would say this model definitely is a step further in the direction of us having lots of specialized little models to do very specific things perhaps built into hardware perhaps just software rather than just having the huge models that can do everything. Your inference costs for this are going to be very close to zero when they're running on your own hardware. And I know that this is something that a lot of organizations are looking at. Actually, I know there are other things like this coming that I'm not allowed to talk about publicly, but there are other teams out there working on developing very small models that can just do this kind of function calling. So, if you want to get started and try it out, I'll put a link to their repo in the description there. It's pretty easy to get the demo up that I showed you. The demo actually may even be on HuggingFace now as well. And then once you've got their repo, it's actually super easy to do the fine-tuning and and that part of it as well. So, if you're looking at doing a project around this kind of thing, where you want to run something like this on a Raspberry Pi, on a phone, something like that, this is definitely a project that's worth checking out. Anyway, let me know in the comments where you could see applications of this. And are you looking for this kind of thing? Right, this is very much a different path than the really big models that can do everything. So, I'd love to hear from people what are the most interesting use cases that you think will be possible with this kind of model going forward. And always if you like the video, please click like and subscribe and I will talk to you in the next video. Bye for now.