AI Tech Inspire

A 20MB, 44M-parameter CPU LLM that Calculates and Remembers Offline

What if a tiny local model could run at desktop-CPU speeds, do on-the-fly math without calling a tool, and recall facts from disk in microseconds—without re-reading the source text? At AI Tech Inspire, we spotted a compact experiment that pushes exactly on those questions. The result: a 19.8 MB large language model trained from scratch that emphasizes fast CPU inference, deterministic calculation, and persistent memory.

At-a-glance facts

  • Model: SHADOW-50M (44M parameters), trained from scratch on 45B tokens.
  • Footprint and speed: 19.8 MB total; ~1,900 tok/s on laptop CPU; ~41 MB RAM; compiled kernel is 159 KB; runs fully offline.
  • Browser: Same kernel compiled to WebAssembly runs in a tab at ~500 tok/s.
  • Weights: Ternary {-1, 0, +1} throughout.
  • Vocabulary: 73,880 tokens via a fixed 512‑bit fingerprint table (no learned embedding). New word pieces can be appended without retraining; reported Spearman 0.594 vs random -0.057.
  • Computation: Model emits tags like [calc]347*86[eq]; a fixed readout circuit fills the correct digits inline. Circuits for arithmetic, percentages, dates/weekdays, units, counting, sorting, comparisons, and a small program machine.
  • Persistent memory: Stores per-token attention state on disk at 1 bit, 288 B/token. Lookup via a 22 B/token index reportedly finds a record ~1 μs; reinjects attention in ~0.03 ms; text isn’t re-read. Memory-mapped I/O keeps runtime RAM ~28–41 MB even with a large archive.
  • Scale example: At 100M tokens, the archive is ~28.8 GB + 2.2 GB index. An index “reinforcement” trail increased repeated-query top‑1 accuracy from 0.571 to 0.743 without retraining.
  • Benchmarks vs a bf16 51.8M Llama‑style baseline (“Supra‑50M‑Reasoning”): SHADOW underperforms on ARC‑Easy (0.307 vs 0.435), PIQA (0.570 vs 0.600), WikiText‑2 perplexity (186 vs 165), but exhibits strong task behavior in arithmetic, dates, and retrieval examples.
  • Quantization contrast: The 51.8M Llama‑style model reportedly breaks under ternary and degrades at int4, while SHADOW is trained for ternary from scratch.
  • Harnesses: Video-memory (~55–56/60 on queries after watching+storing 298 clip moments), inventory recall (159/160 + correct “no record” negatives), speeding llama.cpp for Qwen3‑32B (19.7 → 28.5 tok/s as a drafting helper), and an MCP memory server for Qwen3‑14B (5/5 fact retrieval with quotes).
  • Limits: Thin general knowledge, weak creative writing, occasional miscopying of 7‑digit operands, rare retrieval cross‑talk in large archives.
  • Community: A contributor proposed a CUDA engine, a quantization tutorial, and a local talking toy demo (mic/speaker, ~$35 board). Not merged yet.
  • Availability: MIT licensed; master weights and fine‑tuning/export kit public. Code: GitHub, Weights: Hugging Face, Browser demo: try it here.

Why this tiny model is interesting for engineers

Most practitioners assume smaller LLMs are for latency or cost, but will always lose on reasoning. This project flips that expectation by separating “language modeling” from “calculation” and “memory,” then implementing the latter two as deterministic systems adjacent to the model. The approach echoes classic systems design: use a statistical core where fuzziness helps, and bolt on exact machines where precision matters.

Key idea: treat math and retrieval as deterministic circuits, not as predictions; treat long‑term memory as serialized attention, not as re‑prompting.

That division is what enables a 19.8 MB footprint to still answer “15% of 240” correctly and pull patient records from disk in microseconds, even while the base model’s general world knowledge stays modest.

Ternary weights and a frozen vocabulary table

Two unusual design choices stand out:

  • Ternary weights: The entire network uses {-1, 0, +1} weights. Many Llama‑style baselines degrade when pushed to ternary post‑training. Here, ternary is a first‑class citizen trained from scratch, which helps keep the binary size tiny and the CPU throughput high.
  • Frozen 512‑bit fingerprints for tokens: Instead of a learned embedding, tokens are mapped via a fixed table. When ~8,600 English word pieces were missing, the developer appended them directly—no fine‑tune needed—and prior outputs reportedly stayed stable. That’s a very different trade‑off than learned embeddings, which can’t just “accept” thousands of new rows without retraining.

For anyone building domain‑specific tooling (medical terms, industrial part numbers, logline tokens), the ability to expand vocabulary without a full training run could be a practical lever.

Computation as a circuit, not a guess

When the model decides something needs math, it writes tags like [calc]347*86[eq]. A fixed circuit at readout injects the correct digits back into the same token stream—no tool call, no API, nothing pasted back. Circuits also handle percentages, dates (“45 days after December 20, 2026” → “February 3, 2027”), weekdays, unit conversions, counting, sorting, and simple programs.

Why it matters: in production, the devops cost of reliable tool calls can be non‑trivial. If your task distribution includes lots of basic arithmetic or calendaring, a circuit may cut latency and remove failure modes. It’s not a universal solution, but it’s an intriguing engineering pattern.

Memory as serialized attention

Instead of re‑feeding long context windows, SHADOW writes a compressed attention state per token to disk at 1 bit, 288 B/token, with a 22 B/token index. A retrieval query (e.g., [need]condition of Patient P-204) uses the index to locate the right record in ~1 μs and reinserts the prior attention in ~0.03 ms. The text isn’t re-read; the process stays memory‑mapped and RAM‑light.

For edge deployments, that’s compelling: the archive scales on disk (e.g., 100M tokens ≈ 31 GB including index) while active memory usage remains modest. A “reinforcement trail” in the index itself bumped repeated‑query top‑1 from 0.571 → 0.743 without retraining—suggesting a path to post‑training memory adaptation via the index alone.

Benchmarks vs behavior

On standard leaderboards, SHADOW trails a bf16, 51.8M Llama‑style baseline: ARC‑Easy (0.307 vs 0.435), PIQA (0.570 vs 0.600), WikiText‑2 perplexity (186 vs 165). Yet in side‑by‑side qualitative tasks—basic arithmetic, date math, and disk‑backed retrieval—SHADOW’s answers were crisp while the baseline sometimes meandered or hallucinated. The takeaway isn’t that small beats big; it’s that task‑specific systems design can overshadow raw benchmark deltas, especially for utility workflows.

For developers, this dichotomy raises good questions: which KPIs matter—ARC scores or exactness + latency on your specific task corpus? And where can circuits or memory externalization remove whole classes of errors?

Harnesses: pairing with larger models

  • Video memory: A larger model (Gemma 3 4B; see Gemma) watched a 10‑minute clip and wrote 298 “moment” descriptions (e.g., “M‑0039: A chubby white rabbit reaches for a purple butterfly.”). SHADOW stored these and later answered structured queries scoring ~55–56/60, all locally in ~42 MB RAM.
  • Inventory recall: On 1,600 records, it answered 159/160 correctly and returned “no record” on 20/20 negatives—useful for checklists, asset logs, or field notes.
  • Draft model for a big LLM: Acting as a drafting helper for Qwen3‑32B (Qwen) in llama.cpp, throughput rose from 19.7 → 28.5 tok/s while the larger model still chose the final tokens.
  • MCP memory server: With Qwen3‑14B, it stored mid‑chat facts and later retrieved 5/5 with the original quotes.

The pattern here is attractive: let a tiny, reliable memory+math layer handle structure and speed, while a larger model handles semantics and open‑ended language.

Where it struggles (and why that’s useful to know)

The developer documents real limitations: thin general knowledge, lackluster creative writing, occasional miscopying of seven‑digit numbers, and rare retrieval cross‑talk when archives get large. For engineers, these are not deal‑breakers—they’re design constraints that inform fit. If your app needs imaginative prose or encyclopedic recall, pair SHADOW with a larger model. If you need deterministic calculations, fast lookups, and small‑RAM operation, the trade‑offs look favorable.

Community energy and availability

The project is MIT licensed with public master weights and a fine‑tune/export kit. A community contributor submitted a CUDA engine, quantization notes, and even a local talking toy prototype using a low‑cost board—a reminder that portability and tiny footprints unlock playful, offline form factors. The maintainers haven’t merged the large CUDA PR at the time of writing.

Links if you want to dive in right now:

What to try and what to question

If you’re a developer or researcher, a few practical experiments could be insightful:

  • Slot SHADOW as a memory+math coprocessor beside a larger LLM for enterprise Q&A, RAG, or summarization—measure latency, token costs, and correctness deltas.
  • Replace parts of a vector DB workflow with the serialized‑attention archive—does it reduce token re‑prompting and improve throughput under I/O constraints?
  • Append domain‑specific tokens to the frozen table (e.g., ICD codes, SKU patterns) and test how quickly the system adapts without fine‑tuning.
  • Stress‑test the date/units circuits with adversarial inputs to map boundary conditions.

More broadly, this work asks a timely question: instead of cramming everything into a monolithic model, what if we blend a small language core with specialized circuits and a durable memory substrate? For some classes of applications—edge devices, privacy‑sensitive tools, rugged offline systems—that architecture may be more practical than pushing context windows and parameter counts ever higher.

As always, measure on your own data. But for anyone chasing low‑RAM, CPU‑first workflows, this is a rare, hands‑on template you can clone, run, and tinker with today.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.