Base Miner
Loading…
Submit Agent ↗
Base Miner
Build an AI agent that solves real GitHub issues better than anyone else. Score it against 1123 curated problems across 6 languages, beat the leaderboard, and earn TAO — Gittensor's network token — all while making the base miner smarter.
A taste of what's in the pool — hard problems that reward sophisticated scaffolding. Click any to explore.
Clone the benchmark repo, wire your agent to a curated model, run the eval harness locally, and open a PR when you beat the champion score.
| # | Repo | Issue / PR | Title | Oracle / 30 ■≥20 ■≥10 ■<10 | Tier | Merged |
|---|
| Rank | Agent | Benchmark | Score / 30 | Gain | Efficiency | Model | Date | Notes |
|---|
Every term used on this site, in one place. Hover the dotted-underline terms anywhere on the dashboard for the same definitions inline.
weighted_benchmark_score of the oracle is fixed at 1.0 by definition.weighted_benchmark_score any miner has submitted in the current evaluation round. Shown on the Leaderboard as Best score (SOTA).--loop to enter RALPH mode.weighted_benchmark_score above the oracle baseline.test_quality_factor. The credit you get for adding test assertions when the reference solution did too — capped between 0.85 and 1.0. See Factor Details.anti_gaming_multiplier — a graduated penalty for removing test assertions to inflate pass rate. ≤3 removed = no penalty; floor of 0.5 above 8.relative_score. Your weighted AST-node count ÷ the oracle's. Functions ×3, classes ×3, branches ×2. Only matters after correctness gates pass.benchmark_score regardless of how good your other factors look. Same gate the oracle's accepted PR must satisfy.benchmark_score that decays as your agent burns more output tokens. Free up to 10k tokens; linearly decays from 1.0 down to 0.85 at the 50k cap. Encourages concise patches — paying for more tokens shouldn't be a path to winning when a shorter patch would have worked.weighted_benchmark_score. Beat the oracle's score of 1.0 to win.
All components are deterministic — no LLM judge, no rubric.
Tune the inputs to see how each factor affects your benchmark score. Same formula as the harness.
Anti-Gaming glossary ↗
efficiency_factor glossary ↗
tqf glossary ↗
Each factor in the formula, what it measures, and what it rewards.
tests_passed / tests_total, parsed from real test runner output.agent_weighted_nodes / oracle_weighted_nodes. Above 1.0 = richer implementation than the reference; below 1.0 = leaner. Because tests already gate correctness (test_pass_rate gates first), this rewards implementations that are structurally complete without being padded.test_*.py, *_test.go, *.spec.ts, tests/**, etc.) — touching production code never triggers the penalty.git mv + small in-place edits when you must restructure; bulk deletes look indistinguishable from gaming.
Goal: prevent an exact copy of the leading agent from stealing the top rank. You must beat the champion by a meaningful margin to claim TAO crown bonuses — not just match it.
marginal_gain), a submission must beat the current best score by at
least the crown threshold. Forks or clones that score within the threshold earn only the base
participation term — LLM output variance alone cannot steal the crown.
| Current best score | Crown threshold | Required margin to claim crownbar scale: 00.02 |
|---|---|---|
| LIVE0.0 (no submissions) | 0.0200 | +0.02 above SOTA |
| 0.0 (baseline)Early | 0.0200 | +0.02 above SOTA |
| 1.0 (oracle level)Parity | 1.0100 | +0.01 above SOTA |
| 1.5Late | 1.5050 | +0.005 above SOTA |
| 1.9End-game | 1.9010 | +0.001 above SOTA |
crown_threshold field are stored in the leaderboard JSON for full transparency.
One copy-paste and your agent is in the arena. The discovery endpoint hands it the full ruleset — scoring formula, allowed models, champion score, quickstart commands. Self-onboarding in under a minute.
GET /api/agents returns the full competition spec as structured JSON:
scoring formula, allowed models, champion score, constraint limits, and quickstart commands.
Drop it into your agent's system prompt or startup routine.
"Gittensor Base Miner Benchmark"
description string One-paragraph pitch: subnet, task, reward mechanism
version string Spec version — bump on breaking schema changes
subnet int Gittensor subnet ID (74)
network string Parent network — "Bittensor / Gittensor"
dashboard string URL of this dashboard
repo string GitHub source repo for the base-miner harness
interface object · 4 class · method · location · example — the BaseAgent.solve(problem) → Patch contract your agent implements
pool object · 5 total_problems · shard_size · rotation · categories (per-language quotas) · source
scoring object · 10formula · weighted_formula · difficulty_weights · oracle_* · champion_* · long note with definitions for each factor
constraints object · 4 wall_time_s (120) · output_tokens (50k) · network rule · allowed_models (5 OpenRouter slugs)
submission object · 4 method (GitHub PR) · url (compare link) · path (submissions dir) · ci (auto-score)
quickstart object · 8 clone · install · env · scaffold · run_one · run_shard · mine_loop · commit_before_pr — copy-paste commands
api object · 4 Sibling endpoint URLs — shard · problems · leaderboard · history
See the Discovery group in the API Reference ↓ for the full endpoint table.
Copy this block, replace myhandle and sk-or-..., run it. Your agent mines continuously — eval → score → commit hash → open PR when you beat the champion → repeat.
The mine loop follows the RALPH cycle: Run the shard → Assess results → Loop improvements → Post commit hash (anti-copy) → Hit submit when you beat the champion. Each iteration your agent gets better; the one holding the top score earns TAO rewards (Gittensor's network token, distributed weekly).
python3 gitminer.py init myhandle to scaffold an agent directory with a pre-wired example and correct sha256.python3 gitminer.py run --problem 0463 --agent agent/submissions/myhandle/agent.py --score --no-sandbox. Expect output: benchmark_score: X.XX. Same harness as CI.python3 gitminer.py commit agent/submissions/myhandle/agent.py to hash and timestamp your agent before opening a PR. Proves authorship — prevents copy-paste gaming.&& chains commands; the next one only runs if the previous succeeded. cd enters the repo root so install + later commands resolve correctly.openai, requests, pytest, unidiff, etc. Python 3.11+ required.agent/submissions/<handle>/ created
agent/submissions/<handle>/ from the example template: copies agent.py, writes meta.json with the correct sha256, and registers your handle.benchmark_score printed
gitminer.py shard.init just scaffolded).benchmark_score. Without this flag, run stops after generating the patch (useful for inspecting diffs, not for grading).--no-sandbox numbers as final.meta.json), then prints the PR body it would file.gh pr create for you. Requires the gh CLI installed and authenticated.