AI Tech Inspire

Outcome-based learning beats pure vector search on adversarial queries: 100% vs 3.3%

When retrieval systems chase semantic similarity, they can boost eloquent but wrong answers. What if memory ranked by what actually worked? At AI Tech Inspire, we spotted a fresh approach that replaces blind semantic trust with outcome-weighted memory—and the early numbers on adversarial queries are eyebrow-raising.

Key facts from the experiment

  • Question: Can tracking whether advice actually worked improve retrieval accuracy beyond semantic matching?
  • Setup: Vector database baseline using ChromaDB with L2 distance ranking vs. an outcome-based system that scores memories by success/failure.
  • Scoring: +0.2 for successful outcomes, -0.3 for failures; retrieval weights mix embedding similarity and outcome score.
  • Dynamic weighting: 40% embedding / 60% outcome for proven memories; 70% embedding / 30% outcome for new ones.
  • Data: 30 adversarial scenarios intentionally designed so queries semantically match bad advice.
  • Example: Query “how to fix slow performance” where high-similarity memory suggests vague tuning (previously failed) vs. a lower-overlap memory “add database indexes” (previously worked).
  • Results: Control accuracy 3.3% (1/30); outcome-based accuracy 100% (30/30). Reported p = 0.001 (paired t-test), Cohen’s d = 7.49.
  • Category breakdown: Control failed on debugging, database, errors, async, git; partial success on API (20%). Treatment succeeded across all categories.
  • Enhancements: Contextual retrieval (per Anthropic’s technique), hybrid BM25 + vector with RRF fusion, and cross-encoder re-ranking with a Hugging Face BERT-based model.
  • Open source: MIT-licensed code and test suite at github.com/roampal-ai/roampal (see benchmarks/comprehensive_test/).
  • Open questions: Is McNemar’s test more appropriate than a paired t-test? How to set the failure penalty vs. success reward? How to handle cold start? Will synthetic adversarial gains generalize?

Why outcome-weighted memory can beat pure semantic similarity

Vector databases are excellent at finding things that “sound” similar. But that strength is a vulnerability when the corpus contains attractive yet wrong advice. In adversarial scenarios—where the query is crafted to resemble past mistakes—pure semantic ranking can reliably surface the wrong memory.

The proposed remedy is simple: track outcomes and let them influence retrieval. If a memory snippet consistently leads to fixes, it earns positive credit; if it causes regressions or dead ends, it gets penalized. In this experiment, the retrieval score blended embedding_similarity with an outcome_score, shifting the weight toward outcomes for “proven” memories.

Key takeaway: When the failure mode is “sounds right but doesn’t work,” semantic match + outcome weighting is a powerful combination.

What stood out in the results

The headline number—100% vs 3.3% on adversarial queries—will make any engineer pause. It’s a textbook case where semantics can be fooled, while outcome data becomes the North Star. The example is relatable: performance debugging often rewards concrete fixes (e.g., indexing) over generic tuning tips that feel right but don’t move the needle.

There’s also a strong categorical pattern: the vector baseline collapsed across debugging, database, errors, async, and git. These are domains where “feels right” advice is common and often wrong. By contrast, the outcome-based method thrived exactly where semantics are most misleading.

Methodology questions worth asking

The author used a paired t-test on binary outcomes, then reported p = 0.001 and a very large Cohen’s d. For paired binary data, McNemar’s test (or an exact binomial sign test) is typically preferred. Given the pattern—treatment correct on 30/30 vs control 1/30—McNemar’s exact p-value would likely be far smaller than 0.001, reinforcing the difference. Still, the dataset is small (n=30), and it’s synthetic by design to be adversarial.

For readers considering adoption, a few checks can de-risk the conclusion:

  • Run McNemar’s test or an exact sign test on paired outcomes.
  • Include non-adversarial, neutral, and mixed queries to assess whether outcome-weighting ever hurts benign cases.
  • Report confidence intervals for accuracy and per-category effects.
  • Track success@k and utility-weighted metrics, not just accuracy.

Scoring: is -0.3 vs +0.2 the right penalty ratio?

The current system penalizes failures more than it rewards successes. That aligns with “loss aversion” intuition: a bad memory doing harm should be demoted faster than a good one is promoted. But there’s a richer toolbox to consider:

  • Bayesian reliability: Maintain a Beta(α, β) prior per memory and use the posterior mean or Wilson lower bound as the outcome_score. This naturally handles uncertainty and small sample sizes.
  • Contextual bandits: Treat each memory as an arm; use UCB or Thompson sampling to balance exploration vs. exploitation, especially under cold start.
  • Time decay: Exponentially decay old outcomes to adapt to changing systems and libraries.
  • Cost-sensitive rewards: Weight outcomes by real impact (e.g., “fixed error” > “reduced latency by 1%”).

All of these approaches make the penalty/bonus dynamic depend on uncertainty and impact rather than fixed constants—often more robust in production.

Cold start: what to do before you have outcomes

The system cleverly shifts weights: 70% embedding / 30% outcome for new memories. To go further, teams can:

  • Priors from heuristics: Initialize each memory with pseudo-counts (e.g., α=1, β=1) or class-based priors (e.g., well-documented PostgreSQL fixes start with higher trust).
  • Hybrid rankers: Keep BM25 + vector with RRF, then apply a cross-encoder re-ranker (e.g., a Hugging Face BERT). Cross-encoders often shine when data is sparse.
  • LLM utility predictions: Use an LLM (e.g., GPT) to estimate a prior “utility” score for each memory given the query, then update with real outcomes as they arrive.
  • Interleaving experiments: During the bootstrapping phase, interleave top candidates to gather counterfactuals quickly without tanking quality.

Generalization beyond synthetic adversarial cases

Adversarial tests are valuable because they expose failure modes. But real usage mixes easy, ambiguous, and adversarial queries. A few ways to validate generalization:

  • Live A/B with guardrails: Track time-to-resolution, intervention rate, and user edits as primary KPIs.
  • Replay logs: Build a holdout set from real tickets, PR discussions, and incident retros, not just hand-crafted prompts.
  • Slice analysis: Evaluate by domain (debugging, data, infra), recency, and difficulty bands to detect regressions.
  • Counterfactual evaluation: When treatment and control disagree, sample those cases for human review.

Implementation details engineers will care about

The stack includes modern retrieval patterns: contextual retrieval inspired by Anthropic’s approach, hybrid lexical+vector via BM25 fused with RRF, and a cross-encoder re-ranker using a Hugging Face BERT. The baseline distance metric (L2 in ChromaDB) highlights a common production default.

For those wanting to try outcome-aware ranking without a full redesign, a minimal recipe:

  • Store per-memory counters: successes, failures, last_seen.
  • Compute a reliability score using a Bayesian or Wilson bound estimator.
  • Blend: score = w_sem * sim + w_out * reliability, with w_out increasing as evidence accumulates.
  • Add time decay to both similarity (recency bias) and reliability (concept drift).
  • Log every resolution as a labeled outcome to keep learning.

Small touch: if you’re triaging results interactively, bind a quick accept/reject to Enter/Esc and write those outcomes back to the index—fast feedback loops compound quickly.

What to test next (and share back)

  • Stat tests: Re-run using McNemar’s test and report exact p-values and confidence intervals.
  • Ablations: Show the contribution of each component: outcome weighting, contextual retrieval, hybrid search, and re-ranking.
  • Robustness: Measure performance on neutral and noisy real-world logs; track where outcome-weighting hurts, if at all.
  • Cold-start strategies: Compare fixed +/- weights vs. Bayesian priors vs. bandit exploration.
  • Fairness and inertia: Ensure popular memories don’t get permanent “rich-get-richer” status—use decay and exploration to surface new rising stars.

Bottom line for builders

Outcome-weighted retrieval is a pragmatic evolution: let the system learn which memories actually fix problems, not just which ones look similar. In a world where agents and copilots are increasingly trusted to act, that distinction is critical. The open-source implementation at roampal offers a runnable reference and a comprehensive test suite for those who want to replicate results or plug the ideas into existing stacks.

Even if the 100% vs 3.3% gap narrows in the wild, the core insight survives: when your failure mode is “confidently wrong,” optimize for outcomes. That’s a North Star worth building toward.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.