AI Tech Inspire

MLEB Lands: A Legal IR Benchmark, Kanon 2 Embedder, and a Data Training Debate

If your legal RAG stack still leans on generic embeddings, this week’s release is your wake-up call. At AI Tech Inspire, we spotted an announcement that combines a sprawling new benchmark for legal retrieval, a fresh embedding model, and a pointed question about whether some vendors may be training on user data that includes evaluation sets. For developers and engineers who care about reproducibility, domain coverage, and leaderboard trust, this is one to scrutinize.

Quick facts at a glance

  • Massive Legal Embedding Benchmark (MLEB) is presented as the largest, most diverse open-source benchmark for legal information retrieval to date.
  • It includes 10 expert-annotated datasets across jurisdictions: US, UK, EU, Australia, Ireland, and Singapore.
  • Document types: cases, legislation, regulatory guidance, contracts, and legal literature.
  • Task types: search, zero-shot classification, and question answering.
  • Seven of the datasets are newly constructed to fill domain and jurisdiction gaps.
  • Methodology, code, results, and data are released openly for reproducible evaluations. Paper: arXiv.
  • Authors (from the Australian startup Isaacus) also announced a top-performing embedding model, Kanon 2 embedder, alongside the benchmark.
  • The announcement argues there’s risk some embedding vendors may have trained on private customer data that includes benchmark content; examples cited: Voyage, Cohere, Jina.
  • Discussion point: What’s the likelihood of test set contamination? One angle presented references RTEB results where Voyage reportedly performs worse on closed data than on open data.
  • Benchmark announcement: Introducing MLEB (Hugging Face blog). Model announcement: Kanon 2 embedder (Hugging Face blog). Also see Hugging Face for datasets and model hosting.

Why MLEB matters for real-world legal RAG

Legal retrieval is where generic embeddings often stumble: vocabulary is specialized, citations are structured, and subtle distinctions drive outcomes. A benchmark spanning jurisdictions (US, UK, EU, Australia, Ireland, Singapore) and document types (cases, legislation, regulatory guidance, contracts, literature) is exactly the stress test many teams have been missing.

What’s notable here is breadth plus evaluation variety. MLEB doesn’t stop at search; it folds in zero-shot classification and question answering scenarios. If your stack wires RAG with kNN over embeddings, then promotes candidates through a re-ranker and a generator (say a GPT family model), MLEB can help you spot where your pipeline—retrieval, classification, or answer selection—is really bottlenecked.

The authors say they’re releasing methodology, code, and data openly. For engineering teams, that’s not just transparency theater. It’s an invitation to reproduce scores, tweak evaluation settings, and probe robustness under your own constraints (token limits, cosine vs dot, chunking strategies, multilingual edge cases). If you’re already invested in PyTorch, evaluation should slot neatly into your existing experimentation flow.

The Kanon 2 embedder launch

Alongside MLEB, the team unveiled Kanon 2 embedder and reports strong performance on the new benchmark. It’s common to launch models with in-house or co-released benchmarks, so the key question is: can others replicate the wins?

If you’re benchmarking locally, consider:

  • Matching the authors’ pooling choices ([CLS] vs mean), normalization (L2), and similarity metric.
  • Verifying your index configuration (e.g., nlist, nprobe) if you’re using a vector index like FAISS and ensuring consistent preprocessing across candidates and queries.
  • Evaluating in task slices: split by jurisdiction and document type to see where gains concentrate.

Even if Kanon 2 isn’t what you ultimately deploy, using it as a baseline on MLEB can help calibrate whether your homegrown or vendor embeddings are leaving accuracy on the table.

Did vendors train on evaluation data? Sorting signal from noise

The announcement raises a serious question: could vendor models be inadvertently trained on private user data that includes benchmark test sets? One example cited references a public partnership and default use of customer data for training unless users opt out. That raises the specter of test contamination.

“There is, unfortunately, a serious risk that Voyage’s models were trained on some of the evaluation sets in MLEB due to the fact that Voyage trains on their customers’ private data by default (which would invariably include benchmarks). This is also a risk for Cohere and Jina models.”

To be clear, that’s an allegation and a risk assessment, not a settled fact. Vendor data governance varies by product tier and region; many providers offer explicit opt-outs or contractual controls. Anyone deploying legal AI should read the fine print, confirm defaults, and document consent flows.

What about the performance angle? The discussion points to RTEB results where Voyage apparently fares worse on closed data than on open data, suggesting a possible explanation could be training exposure to public benchmarks. That’s plausible, but it’s not the only explanation. Other possibilities:

  • Domain shift: closed datasets often reflect idiosyncratic drafting styles, scanned PDFs, or niche subdomains that differ from open corpora.
  • Evaluation drift: differences in chunking, query generation, negative sampling, or re-ranking can swing metrics significantly.
  • Hyperparameter or index mismatch: retrieval settings optimized for public sets may not transfer to private distributions.
  • Label quality: closed sets sometimes have noisier annotations or different relevance criteria.

The responsible stance is agnostic empiricism: treat contamination as a hypothesis to test, not an assumption.

Practical steps: How developers can evaluate and de-risk

If you’re running legal RAG in production, consider the following playbook:

  • Replicate on MLEB: Pull the datasets from Hugging Face, mirror the authors’ configs, and log results with seeds and environment hashes. A simple start: pip install datasets evaluate, then script your retrieval experiments.
  • Build a leakage firewall: Keep evaluation sets in a separate storage boundary; maintain a hashed blocklist of evaluation texts to exclude them from any training or pretraining corpora.
  • Contractual controls: For hosted APIs, verify whether data is used for training by default, how opt-out works, and what telemetry is retained. Document choices.
  • Triangulate with closed data: Benchmark on MLEB plus an internal, never-shared holdout. Look for consistent rank-ordering of models across both.
  • Slice your metrics: Report by jurisdiction, document type, and task (search vs zero-shot vs QA). Improvements that only appear on public datasets warrant deeper inspection.
  • Repro recipes: Publish the exact retrieval stack (e.g., cosine similarity, k=20, re-ranker on top of PyTorch), index parameters, and preprocessing steps to help peers validate your claims.

For day-to-day development, a minimal loop might look like this: prepare canonical chunks (stable tokenization), cache embeddings, index with your default ANN engine, run fixed queries, compute nDCG/MRR/Recall@k, and archive results. Keep a Makefile or a tox-style env to avoid configuration drift.

Where this leaves vendors and buyers

For vendors, the message is clear: be explicit about training defaults, provide a no-ambiguity opt-out, and publish an internal process for excluding known evaluation sets. Consider shipping a transparency report for popular benchmarks covering deduplication and blocklist procedures.

For buyers, the best protection is measurement. Use MLEB to sanity-check vendor claims, and always bring your own closed holdout. If a model excels on public legal sets but collapses on your internal corpus, treat that as a signal to dig into preprocessing, index configuration, and potential contamination—but avoid jumping straight to conclusions.

Open questions for the community

  • Should embedding providers proactively certify that specific public benchmarks are excluded from training corpora?
  • Could we standardize benchmark “quarantining” via published hashes and automated checks in training pipelines?
  • What’s the right balance between model improvement using user data and preserving evaluation integrity, particularly in regulated domains?

Benchmarks like MLEB can advance the field if they’re paired with careful methodology and open replication. Combined with a new model drop like Kanon 2 embedder, the result is a productive tension: better tools, tougher scrutiny. That’s healthy for legal AI—and for anyone shipping production-grade RAG systems where trust and traceability are non-negotiable.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.