I'm making LLMs work on small and old devices

🤏 smolmaxxing Every device around you is more capable than it looks. TV boxes, old phones, single-board computers and routers all ship with CPUs, RAM and flash that sit idle most of the day. smolmaxxing is about unlocking that capability: finding the biggest language model each device can really run, then pushing it further with custom kernels, smarter memory use and weight streaming. Every claim is measured on the hardware itself. Device #1 is a Xiaomi TV Box S (3rd Gen) : Amlogic S7D, 4× Cortex-A55 @ 2.5 GHz, 1.9 GB of RAM (about 1 GB usable), eMMC flash, Android 14, no root. On it, this repo runs models from 260 K to 16 B parameters: Qwen3.5-0.8B: 16.8 tok/s (2.2× the stock 7.5; measured in the evening of 11-05, see §17). Qwen3.5-2B: 7.7 tok/s in real chat. The box's speed drifts by ~10 % over a day, so every step is measured back to back: §16 gave +11 %, §17 another +7 %. An 8B mixture-of-experts model (Moon-Beam-8B-A1B): 7.2 tok/s in real chat, its experts paged from flash on demand and prefetched one layer ahead (§19). A dense 1-bit 8B model (Bonsai-8B): 5.2 tok/s in real chat, up from 1.2: fully resident in a "headless" mode plus a lookup-table kernel (§19). A 16B mixture-of-experts model (Ling-mini-2.0): 5.1 tok/s in real chat, coherent over 600-word answers, the same expert pager applied unchanged (§20). Twice Moon-Beam's size, but not better per byte: bigger only helps on this box when the model is also better. A dense 9B model: 0.27 tok/s , streaming most of its weights from flash. Device #2 is a first-generation iPad Air (2013) : Apple A7, 2 cores, NEON without dotprod, 1 GB of RAM , iOS 12.5.7, jailbroken, CPU only. A 13-year-old tablet runs a 1.7B-parameter model at 24-26 tok/s (18-22 in long chats), a 4B one at about 10 tok/s , an 8B mixture of experts at about 3.5 tok/s (5 with half its experts, at a quality cost), and a 16B one at 2.6-4.5 tok/s , with their experts paged from flash, from a chat page in its own Safari. See Results on device #2 . Every step is written up experiment by experiment in EXPERIMENT.md (E1-E160). Devices Device Status Xiaomi TV Box S (3rd Gen): Amlogic S7D, Cortex-A55, 2 GB ✅ done: 260K → 16B parameters, 8B at 5-7 tok/s, 16B at 5 tok/s iPad Air (2013): Apple A7, 2 cores, 1 GB, iOS 12.5.7, jailbroken ( ipad-air/ ) ✅ running: Bonsai-1.7B (1-bit) at 24-26 tok/s, 18-22 in long chats; Bonsai-4B at about 10 tok/s; Moon-Beam-8B-A1B (MoE) at about 3.5 tok/s; Ling-mini-2.0 (16B MoE) at 2.6-4.5 tok/s; CPU only More TV boxes, phones, single-board computers, routers… 🔜 coming soon The techniques aren't tied to one box. The 1-bit lookup-table kernel needs only base NEON, as the iPad shows; any ARM chip with dotprod gets the other kernels; and any Linux or Android device with more model than RAM can use the streaming engine. Scoreboard Class Best model First run → now (tok/s) Gain Bits/byte What got it there Download 0.8B Qwen3.5-0.8B 7.5 → 16.8 bench ×2.2 ≈1.00 all-Q4_K, padded 65K head, fp16 DeltaNet state HF 2B Qwen3.5-2B 0.1 → 7.7 chat ×77 0.909 all-Q4_K, padded 65K head, fused DeltaNet HF 4B Ternary-Bonsai-4B 2.2 → 4.3 chat ×2.0 ≈0.965 ternary weights in 2-bit fields, merged scales HF 8B dense Bonsai-8B (1-bit) 1.21 → 5.2 chat ×4.3 0.974 1-bit LUT kernel, fully resident in headless mode HF 8B MoE Moon-Beam-8B-A1B 1.87 → 7.2 chat ×3.9 0.996 Q2_K experts paged from flash, cache-aware routing, prefetch HF 9B Qwen3.5-9B 0.16 → 0.27 chat ×1.7 0.885 weight streaming plus its own MTP head rebuild ( q9 ) 16B MoE Ling-mini-2.0 4.1 → 5.1 chat ×1.2 1.054 the Moon-Beam recipe, unchanged HF "First run" is the first time each model ran on the box: 0.8B: the stock Q4_K_M file. 2B: the stock file, streamed from flash because it didn't fit in RAM. Its first build that fit ran at 4.8, so from there the gain is ×1.6. 4B: the stock file, which thrashed. 8B dense: the last 0.1 GB streamed from flash. 8B MoE: the first expert-pager run, without routing bias. 9B: streamed. 16B: started on the Moon-Beam recipe, so its gain is small. On this box, the 2B is the best speed-quality balance. Ling-mini-2.0 shows that bigger isn't automatically better: it is twice Moon-Beam's size, but scores worse per byte and runs slower. What we built, grouped by the bottleneck it attacks (details and numbers in What's in here ): Bottleneck Innovations Biggest wins Compute (in-order A55, 10 items) Q2_K and ternary repacks for vdotq_laneq , 1-bit bit planes, a 1-bit lookup-table kernel, ternary in 2-bit fields, Q8_0 attention with a skip threshold, fused Gated-DeltaNet, DeltaNet state read by id / fp16, the LM head on the fast kernel, guided matmul scheduling Q2_K 7.8 → 13.9 G weights/s; Bonsai-8B 3.77 → 5.1-5.4; 4B 3.2 → 4.3; long answers no longer halve the speed RAM (~1 GB, 5 items) pruned LM head, lossless 1.875-bit ternary, headless mode, a fork-free memory guard, a repack cache on flash −0.2 to −0.5 GB per model; free RAM ~1,000 → ~1,300 MB; Bonsai-8B 1.2 → 3.6 before any kernel work Flash (278 MB/s, 5 items) weight streaming with O_DIRECT, MTP speculation when flash-bound, expert pager, cache-aware routing, pre-gated prefetch 9B +69 %; expert misses 13 % → ~2 %; Moon-Beam 4.4 → 7.2 Output quality (2 items) a long-answer degeneration test, a repeat penalty of 1.15 over 256 tokens degenerate long answers 12/13 → 1/13 Results on device #1 Quality is English perplexity on a fixed 22 KB text, converted to bits per byte so models with different tokenizers compare fairly (lower is better). Speed is generation on the TV, measured either through llama-server with real chat requests ("chat") or with llama-bench ("bench"). Model Size on the box Quality (bits/byte) Generation Prompt Notes stories260K (llama2.c) 1 MB n/a ~650 tok/s n/a the first thing that ran (§1-8) Gemma 3 270M Q4_K_M 253 MB n/a 22 tok/s 73 tok/s Qwen3.5-0.8B Q4_K_M (stock) 497 MB n/a 7.5 bench 28 tok/s Qwen3.5-0.8B, all-Q4_K + 65K head 442 MiB (566 MB RSS) ≈1.00* 16.8 bench 33 tok/s best speed for its quality so far (§15-17) Qwen3.5-2B, all-Q4_K + 65K head 0.99 GB 0.909 7.7 chat 17.8 the default: best overall (§11, §16, §17) Ternary-Bonsai-4B, merged scales + 2-bit fields 1.03 GB ≈0.965 4.3 chat (4.8 bench) 9.6 best 4B: was 3.2 chat / 7.0 prompt on the same build (§18) Moon-Beam-8B-A1B (MoE), Q2_K experts + 60K head 0.48 GB GGUF + 2.6 GB repack cache on flash, ~1.1 GB RSS 0.996 7.2 chat n/a fastest 8B : 128 experts, 6 used; expert pager, cache-aware routing, pre-gated prefetch, repeat penalty against long-answer loops (§19, headless mode) Ling-mini-2.0 (MoE, 16.3 B), Q2_K experts + 60K head 0.47 GB GGUF + 5.3 GB repack cache on flash, ~1.13 GB RSS 1.054 5.1 chat n/a biggest model at a usable speed : 256 experts of 1 MB, 8 used, pool 43 per layer, bias 0.3 (§20) Bonsai-8B, 1-bit, LUT kernel 0.96 GB, all resident 0.991 (0.974 on the head) 5.2 chat (5.1-5.4 bench), 4.0 at ~900 tokens of context 5.8-6.1 best dense 8B: headless mode, Q8_0 attention (§19) Bonsai-8B, 1-bit + compact scales 0.84 GB + 0.12 GB streamed 0.991 1.21 chat (2.2 bench) 4.9 §14, before headless mode Ternary-Bonsai-8B 0.89 GB + 0.80 GB streamed 0.919 0.31 bench 3.45 §14 Qwen3.5-9B UD-IQ2_XXS + MTP 0.63 GB + 1.6 GB streamed 0.885 0.16 chat, 0.27 with MTP 0.8 best quality; flash-bound (§14) * measured on the pruned head's positions (1.3 % of tokens fall outside it), so not exactly comparable; the stock full-vocabulary Q4_0 file scores 1.024. Other small models, stock files (screening, §15 E54): Model Generation (bench) Prompt Bits/byte LFM2.5-350M (QAD Q4_0) 23.7 tok/s 104 tok/s 1.805 LFM2-700M (Q4_0) 11.5 tok/s 52.8 tok/s 1.169 Qwen3-0.6B (Q8_0) 9.35 tok/s 50.6 tok/s 1.059 LFM2.5-1.2B (QAD Q4_0) 8.3 tok/s 31.6 tok/s 1.133 Llama 3.2 1B (Q4_0) 7.0 tok/s 29.5 tok/s