AI Tech Inspire

Hourly vs Daily LLM Drift: 31,352 Scores Show 3× Bigger Swings Day to Day

Ever felt like a model nails your tests in the morning, then goes wobbly by late afternoon? At AI Tech Inspire, we spotted a continuous evaluation project that puts numbers on that hunch — and the takeaway is surprisingly actionable: day-to-day changes matter far more than hour-to-hour noise.

Key takeaway: Between-day variation was ~3× larger than within-day variation on a normalized 0–100 scale. In other words, trust daily medians more than hourly wiggles.

Key facts from the analysis

  • Dataset: 31,352 hourly benchmark scores spanning 49 model identifiers across multiple providers and model families.
  • Scoring: Normalized composite scale of 0–100 across consistent tasks and parameters.
  • Execution-focused tests: Coding outputs are executed; tool-calling runs inside isolated Docker environments.
  • Repeated measurements: Each task is run five times, aggregated to reduce outlier effects.
  • Observations: Within-day variation: 2.8 points; between-day variation: 8.4 points (≈3× larger).
  • Detection pipeline: Aggregates to daily medians and applies sequential change-point detection with statistical and minimum-effect thresholds.
  • System: Continuous monitoring classifies models as stable, volatile, degraded, or recovering across coding, reasoning, tool use, reliability, latency, and price.
  • Scale to date: 169,858 benchmark runs, 104,458 measured scores, 88M+ tokens, 81 historical model identifiers, 22 models currently monitored, 6 active providers.
  • Example incident: A reported 32% sustained performance decline in Gemini 3.1 Flash Lite was flagged as a critical incident at the time of a captured snapshot.
  • Resources: Live dataset and dashboard, methodology docs, and MIT-licensed frontend and backend are publicly available.

Why this matters for your stack

Most LLM evaluations happen once, at a single point in time. That’s not how production behaves. Providers ship quiet updates, refresh backend routing, tune safety systems, shift quantization, and launch new variants — any of which can alter behavior. If you’re routing user-critical workloads through GPT-class APIs, you need visibility beyond latency and cost.

The headline here — 2.8 vs 8.4 — reframes monitoring strategy. Hourly fluctuations often look like randomness (temperature, sampling variance, batch effects). Sustained daily movement carries the real signal. For engineers, that translates into a simple operational principle: calibrate your alerts and decisions to daily aggregates, not single-run or single-hour spikes.

How the measurements were run

The project’s methodology leans toward execution, not just judgment:

  • Coding tasks are executed: Success is measured by running the produced code rather than relying solely on model-graded answers.
  • Tool-calling is end-to-end: Models must pick tools, assemble valid arguments, and complete workflows inside isolated Docker containers — a strong proxy for real production tool use.
  • Repeated trials per task: Each task runs n = 5 times, then outcomes are aggregated to dampen lucky or unlucky samples.
  • Consistent parameters: Prompts, scoring logic, and API parameters are kept constant wherever provider options permit.
  • Normalized composite score: Everything lands on a 0–100 scale to support cross-model comparison.

For change detection, the pipeline rolls up to daily medians and uses sequential change-point detection. In practice, think of families like CUSUM or BOCPD as reference patterns — though the precise implementation details here are abstracted. Importantly, it’s not just statistical significance; the system also requires a minimum practical effect size before calling degradation or recovery.

Continuous monitoring and what it revealed

The same evaluation backbone powers a live dashboard that classifies models as stable, volatile, degraded, or recovering across multiple dimensions: coding, reasoning, tool use, reliability, latency, and price. At the time a screenshot was captured, the system flagged a 32% sustained performance decline in Gemini 3.1 Flash Lite and labeled it a critical incident.

It’s a useful case study. Whether that specific dip stemmed from backend shifts, workload load-balancing, or safety tuning, the broader lesson stands: if you only watch p95 latency and token burn, you’ll miss changes in the very capability you pay for. Continuous evaluation adds the missing capability observability dimension to standard dashboards.

A router that adapts to the model reality

The dataset also powers an OpenAI-compatible router that can choose models by task-specific performance, stability, tool-call reliability, latency, and cost. In other words: instead of hardcoding “model = X,” think model = auto, with constraints. For organizations juggling multiple providers (or self-hosted models via PyTorch or TensorFlow), this approach helps avoid silent regressions while balancing budgets. It also nudges teams to treat evaluation as a continuous feedback loop, not a gating check before deployment.

As always, routers work best with task-aware profiles: “coder,” “reasoner,” “tool-user,” etc., and with clear SLOs for win_rate, latency, and cost/1k. If you maintain a Hugging Face stack or deploy GPU-accelerated backends using CUDA, you can apply the same pattern internally for on-prem fleets.

How to apply this in your own monitoring

  • Aggregate smartly: Shift from single-run metrics to daily_median or trimmed_mean over multiple seeds. A “median-of-five” policy gets you most of the way.
  • Separate signal from noise: Expect ~3× larger day-to-day variation than hourly jitter. Tune alerts to daily changes that persist across windows.
  • Instrument capability, not just plumbing: Keep your p95 latency and error rates, but add task success rates for coding, tool use, and reasoning.
  • Run realistic tool-call tests: Containerize tools; require models to assemble valid calls end-to-end, not just emit JSON.
  • Keep parameters locked: Fix temperature, top_p, max_tokens, and prompt templates. Store seeds where supported.
  • Adopt change-point detection: Layer sequential detectors plus minimum-effect thresholds to reduce alert spam.
  • Define SLOs and fallbacks: If a model dips below your win_rate target for 24 hours, route to an alternative until recovery is detected.
  • Version your prompts: Treat prompt updates like code. If your template shifts, your baselines do too.

Pro tip: keep a “canary” suite of fast, high-frequency checks so you can spot issues early, then confirm with deeper, slower suites. For dashboards, a quick r refresh bound to recent daily medians is more informative than an hourly sparkline rollercoaster.

Limitations and healthy skepticism

This is a third-party, continuously updated dataset. Provider-side changes are often opaque, and labels like “degraded” or “recovering” reflect test-specific conditions and prompts. As with any benchmark, your mileage will vary depending on domain, prompt engineering, safety filtering, and even transient traffic patterns. Treat the live labels as strong hints that warrant validation in your environment, not as definitive judgments.

Try it and poke at the data

What makes this particularly useful is that the system, its methodology, and the code are open:

If you already run an internal harness, consider blending these ideas into your pipeline: treat “within-day” as mostly noise, “between-day” as the real signal, and use structured change detection to qualify production incidents. If you don’t have a harness yet, this is a pragmatic template to start from.

Final thought

Monitoring LLMs isn’t just about uptime — it’s about keeping an eye on whether your chosen model still does the job it was hired to do. This dataset surfaces a practical rule-of-thumb that engineering teams can use immediately: evaluate continuously, aggregate daily, and only escalate when changes persist beyond noise. The result is less dashboard thrash, fewer false alarms, and faster, more confident responses when performance really does drift.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.