Parameter Golf Monitor

Parameter Golf Monitor

A skill for tracking the openai/parameter-golf competition leaderboard in real time by fetching open and merged PRs from GitHub's public API.

What this skill does

The monitor script at scripts/monitor.py queries the GitHub API for PRs on openai/parameter-golf, extracts val_bpb scores from titles and bodies, classifies submissions (record vs non-record), and displays a ranked leaderboard table.

It requires zero authentication — it uses the unauthenticated GitHub API (60 requests/hour rate limit).

When to use this skill

• User asks "what's the current leaderboard?" or "who's winning parameter golf?"
• User wants to check their ranking: "where do I stand?" or "how's my PR doing?"
• User asks about competitor scores or techniques
• User wants to monitor for new submissions or changes
• User says "check the competition" or "show me the standings"
• User wants to compare their score against the field

How to run

The script is at scripts/monitor.py relative to this skill's directory. Run it with Python 3 (no dependencies beyond stdlib).

One-shot leaderboard

python3 /scripts/monitor.py

Shows all open PRs ranked by val_bpb (lower is better).

Highlight a specific user

python3 /scripts/monitor.py --me 

Marks the user's PR(s) with <-- and shows their rank + gap to #1 in the summary. Ask the user for their GitHub username if they want highlighting and you don't already know it.

Common flag combinations

All flags

• --me USER — Highlight a GitHub username. Shows rank and gap to #1.
• --top N — Show only the top N scored entries (highlighted user always shown even if outside top N).
• --records-only — Exclude non-record submissions.
• --merged — Show only merged PRs.
• --all — Show both open and merged PRs.
• --since YYYY-MM-DD — Filter to PRs created on or after this date.
• --json — Output as JSON instead of a table (useful for piping to jq or other tools).
• --watch MIN — Poll every N minutes continuously (Ctrl+C to stop).

Interpreting results

• Rank is by val_bpb ascending (lower = better). Only scored PRs get a numeric rank.
• Status shows open, merged, or non-rec (non-record submission).
• Scores marked ? mean no val_bpb could be extracted from the PR title or body.
• Some very low scores (e.g. < 1.10) may be evaluation-only or use non-standard setups — check the PR for details.
• The gap to #1 in the summary tells the highlighted user how far they are from the top.

Rate limits

The GitHub API allows 60 unauthenticated requests per hour. Each run_once call makes 1 request (open PRs) or 2 requests (if --merged or --all). The --watch mode respects this — polling every 5 minutes uses ~12-24 requests/hour.

If you hit the rate limit, the script will show an error and retry on the next interval.

Example output

[2026-03-20 02:17:15] Parameter Golf Leaderboard (open PRs)

Rank   val_bpb     PR  Status    Author              Date        Title
----------------------------------------------------------------------------------------------------
   1   1.15390  #135   open      unnir               2026-03-19  Record: OrthoInit + Int6 MLP3x + BigramHash + SmearGate
   2   1.15800  #106   open      krammnic            2026-03-19  record: 1.158
   3   1.15850  #122   open      mtybadger           2026-03-19  Record: Sliding Window Eval, 2048 Vocab Size
   4   1.16019  #156   open      dexhunter           2026-03-20  Int6 STE + NorMuon + SWA + Sliding Window  <--

Best: val_bpb=1.15390 (#135 by unnir)
Total: 56 PRs, 40 with scores

You (dexhunter): rank #4, val_bpb=1.16019, gap to #1: +0.00629