Getting a small local language model running on a notebook or even smartphone in 2026 is trivial. But what about something even smaller and lower-power. Say, like an ESP32 microcontroller that costs less than $10? It might sound impossible — the device is primarily designed for things like remote sensors, IoT, and other embedded applications, not running generative AI models — yet, that's exactly what a developer who goes by the handle SlvDev has managed to do. In a process detailed on GitHub, and recently showcased on the Better Stack YouTube channel, SlvDev documented how he managed to get a small language model running at nearly 10 tokens a second locally on a microcontroller that costs about the same as a fancy cup of coffee. Tiny stories on a tiny microcontroller Cramming a large language model (LLM) onto something as small as a ESP32 microcontroller isn't a trivial task. There's a reason that these models are trained and run on GPUs. LLMs are memory-hungry beasts that typically require between one and four bytes per parameter just to hold their weights in memory. With just 520 KB of SRAM and 8 MB of pseudo SRAM (PSRAM) on the ESP32-S3, you aren't going to be running a model like DeepSeek V4 Flash . To make it work, the dev had to drop the "large" from the language model and settle for something nearly 10,000 times smaller: TinyStories, a 28.9 million-parameter model originally developed by Microsoft Research. However, even this model is asking a lot of an ESP32-S3 module. At 16-bit precision, the model requires about 60 MB of memory that the ESP32 simply doesn't have. So, the dev employed several techniques, some of which we've previously explored, to shrink the model’s footprint. The first is quantization, a process by which weights are compressed by reducing their precision from something like 16-bits of precision to eight, or even four. This enabled SlvDev to trade a bit of accuracy for a 75 percent reduction in memory required. Instead of about 60 MB of memory to hold the weights, they now require just 14.9 MB. But that still wasn’t enough. Thankfully, in addition to the 8.5 MB of working memory, the ESP32-S3 can also be bought with up to 16MB of flash storage. By borrowing a technique called per-layer-embedding (PLE) from Google's Gemma family of models, the dev was able to offload the majority of the model's weights, about 25 million parameters or about 12 MB worth, to flash with minimal performance degradation. Offloading model weights to NVMe storage is an old trick for getting massive frontier-class models like DeepSeek V3 running on hardware that wouldn't have the necessary memory and GPU capacity to serve it otherwise. The downside of this approach, historically, is that it murders performance. Instead of tokens a second, you're usually looking at seconds, or in some cases minutes, per token. It works but it's not remotely practical. PLE manages quite a bit better because these weights are accessed rather sparingly, which keeps the flash's glacially slow bandwidth relative to DRAM or SRAM from nerfing performance. The result is that rather than trying to cram 14.9 MB into the ESP32's memory, the model now only needs about 2 MB. Specifically the output head, embeddings, and KV cache are kept in the chip's PSRAM, while activations are handled in the chip's 520 KB of SRAM. Using this approach, the dev says they were able to get 9.88 tokens a second out of the microcontroller, which is faster than the average person can read. So, what can I do with it? While you may be able to get a small generative AI model running on a microcontroller like an ESP32, you won't get much from the practice beyond dumb simple pride. Tiny Stories is a great proof-of-concept, but aside from generating short, reasonably coherent stories on demand, it can't do much. You aren't going to build a chatbot with it, generate code, or power an agent. There is another model, called Barista, that can answer questions at roughly twice the performance, but only on topics pertaining to espresso. TinyStories and Barista are, well, just too tiny to do much else. Yet the fact these models run on an ESP32 at all is impressive in itself That said, if you've got a device with just a bit more memory and compute, say a Raspberry Pi or a smartphone, there are far more capable models out there. Google's Gemma 4-E2B-it, launched back in April, employs the same quantization and PLE offload techniques to cram a 5.1 billion-parameter vision language model into just over a gigabyte of memory when using 4-bit weights. Higher degrees of quantization and offloading can get this down to around 500 MB. While its memory footprint is several orders of magnitudes less than what's required to run a frontier model, it is still capable enough to power local chatbots, orchestrate local agents for things like managing a user's calendar, and free you from your reliance on OpenAI or Anthropic so long as you can put up with the occasional hallucination. ®
