AI Tech Inspire

40M GitHub Repos, 2015–2025: An Open Metadata Goldmine for ML

Massive, clean-enough, and real data is still the most underrated advantage in building better developer tools and AI models. At AI Tech Inspire, we spotted a new open dataset that checks those boxes for software repositories — and it might reshape how you prototype ranking models, recommendation systems, and research questions about open source.

Fast facts (what’s actually in the release)

  • Open dataset assembled from GH Archive events by joining event streams and extracting repository metadata.
  • Scale: 40M repositories in the full snapshot, plus a 1M-repo sample for fast iteration.
  • Time span: 2015 to mid-July 2025.
  • Fields include: language, stars, forks, license, short_description, description_language, open_issues, last_pr_index (at snapshot), size, created_at, and more.
  • Compared to common public snapshots (e.g., BigQuery’s ~3M trimmed repos), this dataset is larger and richer.
  • “Alive data”: includes natural gaps and inconsistencies present in the wild — useful for realistic ML/DS exercises.
  • Quickstart Jupyter notebook with basic plots provided, plus the 1M sample for quick experiments.
  • Example surfaced: shifting language share of newly created repos over time.
  • Feedback requested for additional fields or derived signals.

Why this dataset matters for engineers and ML folks

GitHub’s public data is a goldmine, but it’s rarely available in a single, practical snapshot with the fields developers actually want for modeling. The commonly cited BigQuery public dataset weighs in around 3M trimmed repositories. By contrast, this release spans ~40M repos over a decade, with rich, directly useful metadata. That difference doesn’t just add volume; it unlocks experiments that rely on coverage and population-level realism.

Because the metadata comes from GH Archive events, it reflects the messy, “alive” nature of real software ecosystems. If you’re training or benchmarking models that need to survive in production — ranking repositories, predicting maintenance health, or inferring topics — that messiness is a feature, not a bug.

Key takeaway: This is a substantial, ML-friendly snapshot of the open-source universe with enough breadth to test ideas that smaller datasets can’t cover.

What’s inside (and what you can do with it)

The fields alone suggest numerous modeling targets and features:

  • language, description_language: language detection and multilingual analysis; track cross-language trends over time.
  • stars, forks, open_issues: proxy signals for popularity, engagement, and maintenance burden.
  • license: classification, compliance checks, and license-popularity correlations by domain.
  • created_at, last_pr_index, size: lifecycle and growth trajectories; proxies for activity and “busyness.”

Pair these with your own features — e.g., README embeddings, topic tags, or owner/org-level metrics — and you can prototype:

  • Repository ranking models for search or recommendation (features like stars, forks, age, and update cadence often outperform naive text-only baselines).
  • Popularity forecasting: predict future stars given early velocity, language, and license.
  • Maintenance health triage: flag repos likely to accumulate stale issues or slow reviews.
  • License-aware suggestions: recommend alternatives with compatible licenses.
  • Spam or low-quality repo detection using anomaly features (e.g., size vs. stars vs. forks).
  • Curriculum datasets for code models: filter repos by language and metadata before fetching code.

If you want to plug into your ML stack, this metadata pairs naturally with frameworks like PyTorch or TensorFlow, and with model hubs like Hugging Face. Even if you’re building LLM-driven tools (think GPT-style code assistants), metadata features can guide retrieval, re-ranking, and evaluation sets before you spend GPU time on heavier pipelines.

Scale vs. convenience: the 40M full snapshot and the 1M sample

Two tiers aim at different workflows:

  • 1M-repo sample: for fast iteration, exploratory analysis, and teaching. Ideal when you want to prove a concept over a weekend.
  • 40M full snapshot: when you need coverage — rare languages, long-tail projects, or robust generalization across niches.

A smart pattern: use the 1M sample to develop features and baselines; once stable, validate at 40M to learn where your assumptions break. It’s the same playbook teams use when moving from narrow benchmarks to production-scale datasets.

Alive data: why imperfections are a strength

Real-world GitHub metadata is not uniform. Descriptions are missing. Licenses are unset or ambiguous. Language detection is wrong sometimes. That’s reality — and it’s essential if you want models robust to noise.

Instead of scrubbing everything, this dataset leans into those imperfections. This helps in multiple ways:

  • Feature engineering under noise: stress-test heuristics for license, language, and description_language.
  • Evaluation that transfers: if your classifier works here, it’s more likely to work on live traffic.
  • Gap-aware algorithms: build imputation or confidence-aware ranking that degrades gracefully.

Comparing to common public snapshots

The dataset is explicitly positioned as larger and more detailed than common public snapshots like BigQuery’s ~3M trimmed repos. That matters if you care about long-tail representation — think niche domain languages, specialized licenses, or ecosystems that rarely show up in smaller cuts. It also improves the odds that historical queries (2015–2025) return enough signal to plot genuine trends rather than sampling noise.

Quickstart notebook and classroom potential

A Jupyter quickstart notebook with basic plots is included, and the 1M sample is tailor-made for teaching. Instructors can assign practical labs that mimic production tasks:

  • Visualize language share shifts over time (the dataset providers already teased this).
  • Build a simple repo popularity regressor from created_at, language, and early stars.
  • Cluster repos by metadata and inspect clusters for surprising ecosystems.

Use simple Python dataframes first, then scale up with Dask/Spark if you need. Tip: keep your exploratory cells reproducible and annotate choices inline with # comments; future-you will say thanks.

Practical pipeline tips

You don’t need to jump straight to distributed compute. A few pragmatic steps can keep things smooth:

  • Start with the 1M sample to iron out schema handling and nulls.
  • Project only the fields you need (language, stars, forks, etc.) before joining with heavier features.
  • Use robust types and dictionaries for license normalization; treat unknowns explicitly.
  • Cache intermediate artifacts (e.g., parquet chunks) to avoid repeated parsing.
  • For GPU-heavy tasks (embeddings, reranking), check your CUDA versions early to avoid dependency rabbit holes.

Ideas for derived signals (feedback invited)

The release invites suggestions for more fields. A few high-impact candidates:

  • Growth features: star velocity (7/30/90-day windows), fork velocity, issue close rates.
  • Authorship dispersion: number of unique contributors; proxy for bus factor.
  • Release cadence: time since last release; semantic versioning trend.
  • Topic taxonomy: map repo topics into stable categories; useful for multi-label tasks.
  • Readme signals: token count, presence of badges, or docs sections — great proxies before fetching code.

Many of these can be computed incrementally and would supercharge modeling tasks without requiring code downloads.

Use-cases to try this week

  • Language trend dashboard: replicate and extend the example on shifting language share by plotting created_at against language and smoothing by month.
  • License-aware recommendations: given a repo and its license, recommend compatible alternatives with similar size/language footprints.
  • Onboarding a code LLM: curate a “starter pack” of repos by language and activity signals before training or retrieval-tuning your assistant built on GPT or open models on Hugging Face.
  • Cold-start ranking: design features that work even when stars are low — e.g., created_at + early open_issues signals.

Final thought

Large-scale, realistic metadata opens doors that curated, cleaned-to-perfection datasets can’t. This 40M-repo snapshot, covering 2015 through mid-2025, feels like a strong foundation for both research and practical product experiments. If you’ve been waiting for a dataset to stress-test ranking, quality signals, or ecosystem analysis across the real open-source long tail, this is the moment.

AI Tech Inspire will be watching what the community builds next — and what additional signals get folded in. If you’ve got a wish list, now is a good time to suggest it.

Recommended Resources

As an Amazon Associate, I earn from qualifying purchases.