AI Tech Inspire

Open-Source Research Radar Filters arXiv to Surface the Few Papers That Matter

Every day, arXiv drops a tidal wave of new papers, and most readers know the feeling: scan for an hour, save a dozen PDFs, meaningfully read… maybe two. At AI Tech Inspire, we spotted an open-source tool that aims to right-size that firehose into a focused morning brief tailored to your specific research interests.

Key facts at a glance

  • Open-source tool called Research Radar that runs as a daily cron job.
  • Fetches new papers from chosen arXiv categories via RSS and the arXiv API, with deduplication.
  • Scores each abstract from 1–10 against a user-provided Markdown file describing research interests (batched, cheap model).
  • Deep-reads only the top-scoring papers by downloading the PDF, extracting full text, and using a stronger model to write a summary, key insights, limitations, and relevance.
  • Delivers a morning HTML digest with an optional Telegram ping.
  • Design is domain-agnostic; only the interest profile changes, so it can be used for ML, physics, bio, economics, etc.
  • Deterministic pipeline for fetch/dedup/PDF extraction/rendering; only the two scoring passes use models.
  • Model-agnostic backends: can run via existing CLI tooling (e.g., Claude/Codex) or any OpenAI-compatible endpoint, including fully local via Ollama/vLLM.
  • Mix-and-match backends per pass: cheap model for skimming, strong model for deep reads.
  • Approximate sizes: 10-abstract scoring batch is about ~18k input tokens; a deep read sends ~40–70k input tokens.
  • Reportedly used daily for about a month by the creator; generalized repo but not fully tested across all domains yet; feedback invited.
  • Open challenge: calibrating the LLM “judge” to say “not relevant, 3/10” consistently without score inflation, using only prompt + markdown context.
  • Code and benchmarks available on GitHub: github.com/ramazan793/research-radar.

From firehose to focused: why this matters

Most AI practitioners don’t need “what’s popular”; they need “what matters to my work this week.” Newsletters tend to optimize for broad appeal. Manual skimming burns time and attention. The Research Radar approach leans into a simple idea: codify your interests once in a concise interests.md, then automate the rest. When the LLM pipeline elevates only the few papers that match that definition, the cognitive burden shifts from sorting to deciding.

This framing should resonate with anyone maintaining a living research agenda—ML, systems, theory, robotics, biology, econ. The beauty is that nothing in the pipeline is field-specific; a single well-edited markdown file drives the relevance engine.

How the pipeline works (and why it’s sane)

There’s a pragmatic split between deterministic steps and LLM-powered judgments:

  • Deterministic layer: Fetches arXiv via RSS + API, dedupes entries, extracts PDF text, and renders the digest to HTML. These steps are predictable and testable.
  • Model layer: Two passes only. Pass 1: score abstracts 1–10 against your interests.md with a batched, inexpensive model. Pass 2: for the top few, a stronger model ingests the full text and writes a structured brief: summary, insights, limitations, and why the paper matters relative to your stated goals.

By limiting LLM usage to judgment calls, the system avoids the “LLM for everything” trap. It also enables a modular backend strategy: for abstract triage, a lightweight model (or even a local model) is enough. For deep reads, the tool can switch to a stronger model (e.g., a modern GPT variant) only for a handful of candidates.

Key takeaway: treat LLMs as specialized judges, not as the whole pipeline.

Costs, tokens, and throughput

The repo includes rough sizing and benchmarks, which is refreshing. A batch of 10 abstracts is about ~18k tokens—easy to batch without breaking the bank. Deep reads, sending full paper text, range from ~40–70k tokens. That cost is real, but because it’s only applied to a curated subset, it’s tractable for daily use.

For teams sensitive to spend or latency, there are knobs to turn:

  • Keep Pass 1 very cheap (or entirely local with Ollama or vLLM) to aggressively prune candidates.
  • Cap top-K deep reads (e.g., 5–10 per day), or set a score threshold so only clear wins pass through.
  • Run deep reads overnight via cron to mask latency; wake up to a clean digest.

Because the backend is model-agnostic, developers can swap providers or even route through a self-hosted endpoint. If you’re deep in Hugging Face land, you could imagine integrating your own fine-tuned judge model for Pass 1 while leaving Pass 2 to a generalist.

Practical setup ideas for engineers

  • Keep your interest file sharp: Use bullet criteria, negatives, and concrete phrases. For instance, “preferred: alignment via RLHF; exclude: purely theoretical PAC bounds.” Relevance improves when your Markdown is explicit.
  • Schedule smart: A simple cron entry runs it before your morning coffee. For fun, bind a browser shortcut—J/K—to zip through the digest like an email client triage session.
  • Integrate downstream: Write summaries to a notes system (Notion, Obsidian) or export to a reading queue. Tag entries with topics you care about (vision, RL, systems).
  • Compare models per pass: Try a tiny local model for Pass 1 and a premium endpoint for Pass 2. Track hit rate vs. cost for a week to dial in the sweet spot.
  • Team mode: Share an org-level interests.md with a few variants (e.g., core.md, ml-systems.md, theory.md) and send digests to separate channels.

Calibrating the LLM judge

The creator calls out an important challenge: score inflation. Over time, some models trend toward “7 for everything.” A few tactics practitioners might try:

  • Pairwise comparisons: Ask the model to choose which of two abstracts is more relevant, then map choices to scores. Pairwise often reduces drift.
  • Hard negatives and counterexamples: Seed the prompt with examples that look superficially relevant but aren’t. Include explicit “0–3/10” rationales.
  • Scale anchoring: Provide a fixed rubric that defines what 2, 5, 8, and 10 mean, with domain examples.
  • Temperature and penalties: Use low temperature, and consider penalizing verbose justifications that correlate with generous scores.
  • Offline evals: Maintain a small, labeled set of abstracts and compute correlation/ROC each week to catch drift.

Because Pass 1 is batched, you can cheaply experiment with prompts and anchors. If you’re using PyTorch or classic IR tooling, you might even layer a lightweight keyword/BM25 gate before the LLM judge to reduce false positives.

Risks and gotchas to watch

  • PDF extraction fidelity: Some arXiv PDFs are messy. Table-heavy or formula-dense papers can trip extractors, which may affect deep-read quality.
  • Topic drift: If your interests.md is too broad, Pass 1 will flood Pass 2. If it’s too narrow, you’ll miss unexpected but relevant work.
  • Compute and cost: Deep reads aren’t cheap. Keep an eye on token budgets and set sane caps.
  • Generalization: The repo is generalized beyond the author’s field but not deeply tested elsewhere; expect to refine prompts per domain.

Bigger picture: from popularity to personalization

The personalization angle is the story here. Instead of optimizing for viral reach, Research Radar bakes your research agenda into the loop. Imagine extending it with citation graphs, code availability signals, or reproducibility tags. Or pushing results to lab Slack channels with tags so each teammate sees a custom subset. The goal: replace more reading with better reading.

“Stop optimizing for popular. Optimize for relevant.”

Getting started

The code, benchmarks, and setup details live here: github.com/ramazan793/research-radar. It’s model-agnostic, so you can plug in the subscription you already use or run locally with tools like Ollama or vLLM. The digest outputs to HTML with an optional Telegram ping and is designed to be edited and extended.

For developers and researchers drowning in arXiv tabs, this is a pragmatic baseline: one interests.md, a daily cron, a score-first filter, and deep reads only where they count. If you try it, consider contributing calibration ideas or domain-specific prompts back to the repo. The endgame is simple—open your laptop, skim a tight digest, and get back to building.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.