What if you could literally fly through a language model’s vocabulary and watch words snap into place as you pivot around them? At AI Tech Inspire, we spotted a compact web demo that turns GPT‑2’s token space into a navigable hyperbolic world that runs entirely in your browser—even on a phone.
Quick facts
- An interactive visualization places 32,070 GPT‑2 tokens inside a
Poincaré ball(hyperbolic space). - It uses
GPT‑2-small’s raw token embeddings only; no extra data beyond those embeddings. - Everything runs client-side in a single HTML file; mobile-friendly.
- Controls: drag to rotate, pinch to zoom, tap a token to center it via a
Möbius translation, then tap neighbors to explore. - The layout is constructed exactly—no optimization or training steps.
- The vocabulary’s similarity graph forms a forest: one giant tree (~2,300 tokens), a few hundred smaller trees, and ~6,700 isolated tokens with no close relatives.
- A dense inner shell contains those isolated tokens, which lack a
reciprocal nearest neighbor. - Example paths: search dog and follow links to cats; search Monday and walk through the days; “Trump” connects only to “Ivanka.”
- Try it here: Interactive Poincaré vocabulary explorer.
Why hyperbolic space makes this work
Most of us have seen 2D embeddings of model vocabularies using t‑SNE or UMAP. They’re great for quick glimpses, but they struggle to preserve tree-like structure. Hyperbolic space, by contrast, is a natural fit for hierarchies: the available area grows exponentially as you move away from the center. That makes a Poincaré ball a compelling canvas for anything that behaves like a tree—including token similarities.
Here, the underlying data are simply the raw token embeddings of GPT‑2-small. No training is done to “bend” points into a neat structure; the layout is computed exactly in hyperbolic space. Navigation uses Möbius translation, which is the native way to shift your viewpoint in the Poincaré model. The effect feels like the entire space contracts and stretches around your selected token, keeping local neighborhoods coherent as you move.
“Trees don’t fit well in flat space, but they embed naturally in hyperbolic space, where available room grows exponentially with distance from the center.”
For developers familiar with Euclidean embeddings, this demo is a reminder that geometry is not merely aesthetic—it encodes assumptions. If your data are tree-shaped, a non-Euclidean model can be the difference between a tangled hairball and an intelligible atlas.
Hands-on: what to try first
Open the demo and use the built-in search. Type dog, then tap through neighboring tokens and watch how animal-related clusters hang together. Do the same with Monday and walk the chain of weekdays. Notice how tapping a token recenters the entire world—this is a Möbius translation in action.
The inner shell is especially interesting: it’s packed with isolated tokens that have no reciprocal nearest neighbor. In plain terms, a token might find another token close by, but that neighbor doesn’t find it similarly close in return. The visualization places these loners near the center, creating a dense core distinct from the branching trees.
One quirky observation surfaced by the author’s notes: the token “Trump” connects only to “Ivanka.” Whether you view that as a curiosity or a prompt to investigate bias, the point is that the explorer makes it trivial to spot these one-link bridges across regions.
What it reveals about token embeddings
Language model vocabularies aren’t uniform clouds; they behave like a forest. In this case, the forest includes:
- One large tree with approximately 2,300 tokens.
- A few hundred smaller family trees.
- About 6,700 isolated tokens with no close relatives.
That last number is striking. Think about common subword tokens from a BPE-style tokenizer. Many of them are generic fragments or punctuation-like shards. They don’t reciprocate nearest neighbors because, semantically, they’re cliff notes rather than full concepts. Hyperbolic space visually segregates them from the more coherent branches.
For anyone who tunes tokenizers, this raises interesting questions: should certain fragments be merged further? Are there domains where isolated tokens cluster more tightly? A visualization like this can be a practical sanity check before shipping a custom vocabulary.
Why developers and engineers should care
Beyond the aesthetic, there are concrete takeaways for practitioners:
- Tokenizer design and debugging: Spot isolated tokens and weakly connected fragments. If you’re building a domain-specific vocabulary (e.g., via Hugging Face tools), this can reveal where merges or exclusions might help.
- Embedding sanity checks: If you swap pretraining regimes or fine-tune embeddings, a hyperbolic view can show whether conceptual neighborhoods remain coherent. Consider comparing before/after trees inside the same interface.
- Prompt engineering intuition: Seeing how semantically adjacent tokens clump might inform choices of synonyms or phrasings that steer models like GPT‑2.
- Hierarchical data UIs: The navigation pattern—tap to center via
Möbius translation—is a great pattern for any hierarchical dataset, not just language. Think logs, org charts, function-call graphs, or knowledge graphs. - Client-side performance: The fact that the entire experience runs in a single HTML file, on-device, hints at lightweight rendering and data structures that are worth exploring for mobile-friendly visualization tools.
If you typically rely on Euclidean projections via PyTorch or scikit-learn pipelines, consider adding hyperbolic views to your model-inspection toolkit. They illuminate different failure modes—for example, when a small set of tokens forms brittle bridges between otherwise distant regions.
Comparing to the usual 2D maps
We all love a clean 2D scatter plot, but there are trade-offs. Algorithms like t‑SNE compress local neighborhoods at the expense of global structure; UMAP balances local and global patterns but still works in flat space. Hyperbolic embeddings acknowledge that some datasets are fundamentally hierarchical. When your intrinsic geometry is tree-like, forcing Euclidean constraints can fragment branches or muddle distances. A hyperbolic model maintains the integrity of branching depth and fan-out.
Crucially, this demo doesn’t “train” a Poincaré embedding. It uses the existing GPT‑2-small vectors and lays them out in hyperbolic space with an exact construction. That gives you a different perspective without altering the underlying semantics learned by the model.
Potential next steps and open questions
- Could we overlay frequency data or token types (prefix/suffix/punctuation) to explain which tokens end up isolated?
- What happens if we compare different models’ vocabularies—say
GPT‑2vs. a domain-specialized model? Do their trees branch differently? - Might a similar interface help debug model errors by projecting attention heads or layerwise neuron activations into a hyperbolic map?
- Could this navigation pattern scale to millions of nodes for codebases, knowledge graphs, or logs—while remaining fully client-side?
Try it and build your own intuition
The best way to grasp why hyperbolic space matters is to feel it. Load the explorer, search for a token, and keep tapping neighbors. Notice how the world rearranges itself without losing local context. It’s a tactile illustration of something we all know abstractly: structure matters, and the right geometry can turn clutter into clarity.
Explore it here: Poincaré vocabulary explorer. If you’re experimenting with embeddings, tokenizers, or model interpretability, this could become a simple, memorable reference point. And if your next visualization project involves hierarchies, consider borrowing the hyperbolic playbook—and the elegant Möbius translation interaction model—right into your own tools.
Recommended Resources
As an Amazon Associate, I earn from qualifying purchases.
