Needle 2

Needle 2 is a 45 million parameter language model designed to run on edge hardware like smartwatches, budget phones, IoT devices, VR headsets, and robotics platforms. The entire model fits in a 14 megabyte binary and peaks at 28 megabytes of RAM during inference, which means it can run on hardware that would choke on any cloud connected model requiring gigabytes of memory. It decodes at around 500 tokens per second on a Raspberry Pi 5, hits 400 to 1500 tokens per second on VR devices like Meta Quest 3S and Apple Vision Pro, and manages 300 to 700 tokens per second on budget Android phones that cost under 200 dollars. It's built specifically for tool calling, device control, and structured data extraction rather than general conversation or open ended reasoning tasks.

The core problem it solves is latency and privacy on low power devices. Sending every voice command to a cloud API adds hundreds of milliseconds of delay at minimum, often more depending on network conditions, and requires constant connectivity to function at all. That latency kills the experience on wearables where users expect instant feedback when they tap their watch or speak a command. The connectivity requirement makes offline use impossible, which breaks the device the moment you walk out of WiFi range or enter an area with poor cell signal. Needle processes commands locally on the device itself, eliminating the round trip and working even when there's no internet connection. For a wearable or smart home hub, that's the difference between feeling responsive and feeling sluggish. For a robot navigating a space or a medical device where reliability matters, it's the difference between usable and unusable.

What sets Needle apart from other small language models is its focus on structured outputs rather than freeform text generation. It uses something called Cactus Quants CQ2 bit compression along with byte level grammar constraints to produce schema compliant JSON every single time. If you define a function signature or a data extraction schema, the model will follow it exactly rather than occasionally hallucinating extra fields, omitting required parameters, or returning malformed output. This predictability is essential for device integration where a wrong function call might turn off the lights when you asked for the weather, or trigger an unintended action that can't be undone. The grammar enforcement happens at the byte level during decoding, so it's not a post hoc validation step that can be circumvented by a misbehaving model.

The model uses a Simple Attention Network architecture with a 256 token sliding window, which bounds memory usage regardless of how long the conversation gets. For device control use cases, you don't need the model to remember what you asked twenty minutes ago. You need it to parse the current command correctly and execute it immediately. The sliding window trades off long context memory for predictable RAM usage, which is the right tradeoff for embedded applications where memory is scarce and you can't afford the model to allocate more as conversation history grows. This architectural choice is deliberate and fits the use case.

Needle includes confidence scoring and a hybrid edge cloud architecture for handling edge cases gracefully. When the model is unsure about a command because it's ambiguous, uses vocabulary outside its training, or falls into a gray area, it can route the query to a larger cloud model rather than guessing wrong. This lets manufacturers deploy edge first for the common cases that account for ninety percent of usage while keeping a safety net for the long tail of edge cases the small model can't handle confidently. The confidence scores are output alongside the structured response, so routing logic can be as simple as a threshold check or as nuanced as the application requires.

Target users are hardware manufacturers building consumer devices at mass market price points where adding dedicated AI accelerators isn't economically viable. Smartwatch makers, IoT device companies, wearable startups, and robotics teams building products that can't afford to spend an extra fifty dollars per unit on inference hardware are the obvious fit. Pebble is already shipping Needle in their Index 01 wearable app, so there's real world production validation beyond synthetic benchmarks and controlled demos. The model can also be fine tuned on custom tool vocabularies if a manufacturer needs it to understand their specific device commands, APIs, or domain language that wasn't in the general training set.

Needle 2 is open source under the Apache 2.0 license. The model weights, code, and documentation are available on Hugging Face and GitHub for anyone to download and use. There's no SaaS pricing for the model itself since it runs entirely on device without calling any external service operated by Cactus Compute. The company likely offers enterprise support, custom fine tuning services, and integration consulting for companies that need hands on help getting the model deployed on their specific hardware, but for developers and tinkerers who want to prototype or build on their own, everything needed to download, run, and test the model is completely free.