AI Tech Inspire

Agent swarm formalizes CFSG in Lean; verification, not generation, is the limit

What actually breaks first when you unleash a swarm of AI agents on a 15,000-page mathematical proof? Not the models. The checker. That’s the surprising lesson from a recent project that formalized the Classification of Finite Simple Groups (CFSG) in the Lean theorem prover—an engineering story that should make any developer thinking about agentic systems sit up.

Snapshot: key facts from the project

  • A complete Lean formalization of CFSG was released on October 26.
  • Scale: 43.1 million lines and 2.05 million declarations in total.
  • Timeline: completed in 24 days with swarms of dozens of agents.
  • Coordination: GitHub Issues and a Slack channel, split across 108 milestones.
  • Codebase baseline: ~1.2 million lines inherited from an earlier project.
  • Agents that needed a lemma could either prove it or assume it; in practice, many assumed.
  • Outcome: tens of thousands of lines were deleted and parts of the plan restarted; only one milestone closed in the first 12 days.
  • Recurring difficulty noted by the team: deciding whether an added assumption merely deferred work or subtly changed a theorem’s meaning.
  • Bottleneck #2: Lean rechecked proofs slower than agents produced them; mitigations included distributed caching and externally checkable certificates.
  • Bottom line: both limits were on verification throughput, not on generation.

Why this matters to developers (beyond pure math)

At AI Tech Inspire, this landed as more than a headline—it’s a blueprint for where multi-agent coding, formal verification, and LLM-assisted development are actually constrained. If you’re wiring agents into anything correctness-sensitive—compilers, migrations, CUDA kernels, security patches, infra-as-code, even high-stakes data transforms—the hard ceiling you’ll hit is the speed and rigor of your checkers, not your generators.

The project used Lean, a proof assistant that treats theorems like strongly typed programs. In that world, “it compiles” means “it’s proven.” But even there, generation was the easy part. The two biggest pain points—semantic drift from unconstrained assumptions, and the time cost of rechecking—are generalizable to any agent system that makes and verifies claims.

“Determining whether adding an assumption to a theorem deferred work or changed its meaning was a recurring difficulty.”

Key takeaway: these were throughput limits on verification, not on AI output generation.

Context: what was actually formalized?

CFSG classifies all finite simple groups—essentially, the building blocks of finite group theory—into well-understood families plus 26 sporadic cases. The original proof runs to roughly 15,000 pages spanning decades of papers. Formalizing it means turning that mountain into machine-checkable proof scripts inside a system like Lean. Think of it as porting a sprawling, multi-author research codebase into a single typed, compiled artifact with zero undefined behavior.

While about 1.2M lines of code were inherited, the remarkable bit is the swarm: dozens of agents, coordinated through GitHub Issues and Slack, hammered out 43.1M lines and 2.05M declarations in 24 days, organized into 108 milestones. That’s not just throughput; it’s a live-fire test of multi-agent orchestration under correctness constraints.

The two bottlenecks every agentic system will meet

1) Semantic drift via assumptions

Agents facing a missing lemma had two options: prove it now or assume it and move on. In a large swarm, the cheapest local move is to assume. Multiply that across contributors and you accumulate “assumption debt.” The report notes that deciding whether a new assumption deferred work or changed the theorem’s meaning was repeatedly hard. The result: tens of thousands of lines deleted, milestones reset, and only one milestone closed in the first 12 days.

Engineering analogues:

  • Feature flags or stubs that leak into production semantics.
  • Loose interface contracts that compile but subtly break invariants.
  • “TODO prove” comments that become structural dependencies.

Practical guardrails developers can adopt:

  • Assumption registries: Treat assumptions like technical debt. Track owners, expiration, and downstream theorems that depend on them.
  • Semantic CI gates: Forbid merging if a new assumption affects any previously proven interface or public theorem signature.
  • Ownership of lemmas: Assign modules and explicit reviewers for lemma boundaries; enforce via CODEOWNERS and policy bots.
  • Interface tests for logic: Model-check toy instances that would fail if meanings drifted.

2) Checker throughput (Lean rechecking vs. agent generation)

The second ceiling: Lean rechecked proofs slower than new proofs were being produced. The mitigation—distributed caching and externally checkable certificates—maps neatly to modern build systems and ML pipelines: think ccache, remote cache in Bazel/Buck, and deterministic artifacts in CI. It’s the same “don’t recompute what you can reuse” discipline, now applied to proofs.

For AI-heavy teams, the parallel is straightforward: generation with PyTorch or TensorFlow is fast; validating outputs (alignment checks, eval suites, policy tests, safety filters) often dominates wall time. Even with systems like Hugging Face pipelines or GPT-style tools, the cost-of-checking frequently exceeds the cost-of-making.

How to apply these lessons today

If you’re experimenting with agent swarms for code or proofs, here’s a pragmatic playbook:

  • Start with a typed kernel: A theorem prover (Lean, Coq) or a strict type system (Rust, TypeScript with strict null checks) to force early failures.
  • Instrument your checker: Measure recheck times per module; log and visualize dependency graphs. Your bottleneck lives here, not in the generator.
  • Cache aggressively: Build distributed caches keyed by content hashes. Persist proof or analysis certificates in artifact storage.
  • Constrain assumptions: Add a bot that flags new assumptions and requires explicit sign-off (Ctrl + Enter won’t be enough to push it through).
  • Shard work by stable interfaces: Freeze theorem or API boundaries before parallelizing. Breaking interfaces should be a deliberate, high-friction event.
  • Detect semantic drift: Differential proof checks and property-based tests that fire when lemmas get relaxed or generalized.
  • Adopt milestone triage: Expect rewrites. Budget for restarts and deletions as a design feature, not a failure.

Comparisons: agent “creativity” isn’t the rate-limiter

The trope that more capable models will solve everything doesn’t match this result. Better models may reduce the number of bad assumptions, but the limiters were systemic: definition drift and checker throughput. Upgrading your generators (say, moving to a stronger GPT-class model) won’t fix a sluggish verifier or a permissive assumption policy. This echoes patterns we’ve seen in ML ops: improving training horsepower doesn’t shorten eval or deployment gates when governance and QA dominate.

For developers who think in frameworks: generation is your model; verification is your runtime, compiler, and CI. You can optimize both, but only one of them decides if the artifact is shippable.

What surprised us—and what to try next

Two things stand out. First, the team reports the failure log as “the part worth reading.” That’s a mindset shift: design your multi-agent system so that failures produce high-signal artifacts—root causes, dependency impact, and rollback guidance. Second, the speed came not from perfect agents, but from coordinated, cache-friendly verification infrastructure.

If you want to try this at smaller scale:

  • Pick a domain with crisp specs (e.g., a small compiler pass, a crypto library, or a math module). Use Lean or a typed language with proofs-as-tests.
  • Let multiple agents propose patches that either “prove” (implement) or “assume” (stub behind a feature flag). Track both.
  • Throttle merges on verifier load. Your dashboard should show pending proof checks like a build queue.
  • Export “check certificates” for fast revalidation when refactoring—essentially, artifacts that say, “this still holds.”

The bigger picture

AI agents can scale generation almost without friction. But as this CFSG formalization shows, the sustainable wins arrive when you scale verification. The real innovation isn’t just in swarming dozens of agents; it’s in building the rails that keep them honest: assumption discipline, semantic CI, and race-tuned checkers with smart caches.

For engineering leaders, the message is clear: before adding more agents, upgrade your checkers. Measure them, shard them, cache them, and make their outputs portable. Creativity is abundant; correctness is the scarce resource.

And if your agents ever feel “slow,” ask the unglamorous question this project answered: is it actually the checker?

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.