AI Tech Inspire

Tiny Transformer Generates Executable Drawings for an RP2040, Bytecode Not Pixels

What if a model didn’t output pixels at all—but a compact, executable drawing program that a microcontroller could run exactly? At AI Tech Inspire, we spotted a compact research prototype exploring precisely that angle: an ~825k-parameter autoregressive transformer that emits drawing bytecode, which is then executed on a Raspberry Pi Pico (RP2040) via a tiny fixed-point virtual machine.

Quick facts from the project

  • An ~825k-parameter autoregressive transformer generates ~100 bytes of drawing bytecode instead of pixels.
  • The bytecode is transferred to a Raspberry Pi Pico, where a small fixed-point VM executes it and streams geometry back over UART.
  • The model runs on the host; the Pico only stores and executes the generated program. There’s no claim that the transformer itself runs on the microcontroller.
  • Execution validation: 12,670/12,670 generated traces matched a Python reference VM exactly.
  • Interpreter footprint on Pico: 1,862 bytes of flash, 0 bytes static RAM, 492 bytes peak stack, and 7,334 cycles per drawing at 12 MHz (~0.61 ms for measured QuickDraw programs). No floating point unit or tensor runtime is needed on the Pico.
  • Representation experiments compared token, byte, bit, typed-token, and delta-coordinate formats. Outcome: it depends on the corpus. On a synthetic corpus, bit-level was essentially equivalent to bytes at the converged budget; on real QuickDraw sketches it incurred an ~11.6-bit penalty per drawing.
  • Tests examined whether the model can discover repeated structure (e.g., loops) from flat bytecode and whether hierarchical stroke planning helps. The planner did not improve likelihood but significantly improved termination and generated-length behavior.
  • Under teacher forcing, the model showed a strong preference for compatible relational context, yet it still struggled to produce the exact compatible continuation when sampling freely.
  • Current direction: add an explicit source-span / affine-relation / copy-or-emit action while keeping the final output as flat drawing bytecode. The aim is to test whether explicit relations help exact generation on unseen combinations.
  • Repo with demo instructions, figures, captured RP2040 traces, and experiment details: drawing-machine on GitHub. Feedback requested on evaluating novelty vs. memorization, measuring exact program generation beyond teacher-forced likelihood, and experiments to make the microcontroller result more meaningful.

Why generate bytecode instead of pixels?

Vector-style generation can be extraordinarily compact, explainable, and verifiable. Rather than rasterizing a 2D image, the model emits a short sequence of drawing primitives—lines, curves, positions—in bytecode. A microcontroller-side interpreter then renders that program deterministically. This is different from the pixel-first mindset popularized by systems like Stable Diffusion or the token-heavy language modeling style of GPT. Here, the output is a small, structured program you can inspect, diff, and run anywhere a tiny VM fits.

That design choice pays off on constrained hardware. The RP2040 interpreter reported here uses ~1.8 KB of flash, 0 bytes of static RAM, and a ~492 B peak stack, with ~0.61 ms execution per drawing at 12 MHz. There’s no floating point and no tensor runtime on-device. For embedded developers building plotters, e-paper widgets, or on-device sketch playback, that’s an eye-catching performance envelope.

Key takeaway: generate small programs on the host; execute them exactly, quickly, and cheaply on a microcontroller—no heavy dependencies required.

Pipeline separation: host-side model, device-side execution

The work keeps the model and the executor cleanly separated. The transformer runs on a host machine—think any environment where you might already use PyTorch or TensorFlow—and only the emitted bytecode touches the microcontroller. That’s an important nuance for practitioners: you can generate, verify, and even batch “compile” drawing programs offline, then ship tiny payloads to cheap endpoints. In many IoT or signage scenarios, this is the practical sweet spot.

Exactly matching 12,670/12,670 traces to a Python reference VM is also notable. It suggests a tight spec between host and device, making it feasible to write property-based tests, formalize instruction semantics, and confirm bit-for-bit equivalence before deployment.

Representation experiments: the tokenization choice really matters

The experiments compared multiple representations—token, byte, bit, typed-token, delta-coordinate—while holding drawing information constant. The verdict: it depends on the corpus. On synthetic programs, bit-level representations converged similarly to bytes; on QuickDraw sketches, bits carried an ~11.6-bit penalty per drawing.

For developers accustomed to language-model tokenization debates on Hugging Face, this will resonate. Small models are especially sensitive to how structure is exposed in the input/output space. If the codebook doesn’t align with the natural regularities of the data, the model wastes capacity learning what the tokenizer should have encoded. Here, typed tokens or delta-coordinates can act like a “domain-aware tokenizer” for graphics, making short-range dependencies easier to predict and compress.

Practical tip: prototype multiple encodings early. Log bits-per-drawing (or NLL) and pay attention to convergence behavior, not just final perplexity. A representation that speeds convergence or stabilizes termination may be the better engineering choice, even if the final losses are close.

Structure learning, planning, and the gap between training and sampling

The project probed whether a transformer can “rediscover” program structure—like loops—from flat bytecode. It also tested a hierarchical stroke planner. Results were mixed: the planner didn’t improve likelihood but did improve termination rates and generated-length behavior. That gap is familiar in sequence modeling: better log-likelihood doesn’t always mean nicer samples.

Under teacher forcing, the model preferred context that was relationally compatible, yet it struggled to continue precisely when sampling freely. That’s classic exposure bias. For folks building tiny generative systems, it’s a reminder to evaluate beyond teacher-forced likelihood: look at exact execution, termination, and VM-level equivalence on free samples.

Next step: explicit relations and copy actions

The upcoming direction is to introduce an explicit source-span / affine-relation / copy-or-emit action while preserving flat bytecode as the final output. This is reminiscent of copy mechanisms and pointer-like operators in sequence models. The hypothesis: make relational structure first-class so the model can recombine known patterns exactly on unseen compositions. If that works, it unlocks better combinatorial generalization with sub-million-parameter budgets.

Why this matters for developers and embedded engineers

  • Deterministic playback: exact match to a Python reference means you can treat the bytecode as a contract.
  • Tiny deployment: ~1.8 KB interpreter, 0 B static RAM—friendly to tight BOM and power budgets.
  • Separation of concerns: generate on a workstation or server; execute anywhere a microcontroller fits.
  • Versatile use cases: on-device sketch renderers, robotic plotters, CNC/pen plotters, vector signage, procedural UI elements, or even lightweight CAD macros.

Compared to neural rendering on-device, this approach is refreshingly pragmatic: the microcontroller stays simple, and the heavy lifting—training and generation—lives where CUDA and big RAM can help.

Ideas to try, measure, and improve

For readers who want to contribute or adapt similar systems, here are concrete avenues:

  • Exactness beyond likelihood: report the fraction of free-sampled programs that execute to exact geometric equivalence with a reference VM. Consider edit distance over instruction streams and tolerance-bounded geometry diffs.
  • Novelty vs. memorization: de-duplicate at the program level (hash IR), hold out families of shapes or stroke motifs, and test recombination performance on deliberately unseen compositions.
  • Termination robustness: measure early-stop failure rates, instruction overruns, and recovery from malformed tokens. Track mean/variance of generated length.
  • Energy and latency on-device: profile cycles and microjoules per drawing across clock rates and supply voltages. Quantify jitter under interrupts and UART backpressure.
  • Encoding bake-off: run ablations for byte/bit/typed/delta with matched budgets; plot convergence speed, stability, and exact-match rates. Don’t just chase final BPD.
  • Structure discovery: design synthetic corpora with known loops/affine symmetries; verify whether the model (with and without copy/affine actions) recovers them.

How to explore it yourself

The repository, demo instructions, figures, and captured RP2040 traces are available here: drawing-machine on GitHub. Developers versed in transformers—whether through PyTorch or TensorFlow—should find the approach approachable: generate ~100 bytes of bytecode, stream over UART, and watch the VM render exactly as the host reference.

Curious to tinker? Try swapping encodings, adding a copy action, or training on your own vector sketches. If you build plotters, e-ink dashboards, or embedded art toys, this might be the most fun flash-and-run workflow you adopt this season.

And if you have ideas on stronger novelty tests, better exactness metrics, or microcontroller experiments that would make the results even more convincing, the project specifically welcomes that feedback. This is the kind of hands-on research that pushes practical edges—small models, tight code, and outputs you can literally run.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.