Validating GitHub Links Before Fetching (context-waste guard)

Last updated 05/11/2026 · migrated-from-outline, tools

Why this exists: I (the agent) have a recurring habit of opening GitHub links that turn out to be invalid. Each failed fetch dumps a large HTML “404 / page not found” error page into the context — pure waste that bloats the conversation and can stall or derail the task. The fix: check whether a GitHub link is real with a cheap, low-token probe BEFORE doing a full content fetch.

The core rule

Never call fetch on a raw github.com/... URL as your first move. Run a tiny curl HEAD/status check against the GitHub REST API first. Only if it resolves do you fetch (and then prefer the API or raw endpoint, which returns clean JSON/markdown instead of huge HTML).

The trap that makes this non-trivial (verified 2026-10-06)

A missing repo and a missing file inside a valid repo both return HTTP 404 with the identical body:

{ "message": "Not Found", "status": "404" }

So a single status-code check is ambiguous. You must validate in two layers:

Repo exists? → GET /repos/{owner}/{repo} (200 = real repo).
Only if the repo is real, and the link points at a file/blob: → GET /repos/{owner}/{repo}/contents/{path} (200 = path exists).

Working probe (terminal tool)

# Layer 1 — does the repo exist? (fast, tiny response)
curl -sS -o /dev/null -w "%{http_code}\n" https://api.github.com/repos/OWNER/REPO
#   200 -> valid. 404 -> dead link, STOP, do not fetch.

# Layer 2 — does the specific file/path exist? (only if layer 1 was 200)
curl -sS -o /dev/null -w "%{http_code}\n" \
  "https://api.github.com/repos/OWNER/REPO/contents/PATH/TO/FILE"
#   200 -> good to fetch. 404 -> repo is fine but the path is wrong/stale.

-o /dev/null -w "%{http_code}" keeps the response body out of context — you only see a 3-digit code, which is exactly the low-token check we want.

Gotchas (all verified)

Renamed repos: github.com/<old-owner>/<old-repo> returns a 301 to the new name in a browser/curl -L, but the API does NOT redirect on old names (api.github.com/repos/<old> → 404). So for links people share from memory, probe with -L and read %{url_effective}:
```
curl -sSL -o /dev/null -w "code=%{http_code} final=%{url_effective}\n" \
  https://github.com/OWNER/REPO
#   code=200 final=https://github.com/<NEW-owner>/<NEW-repo>  -> use the new name for any API call
```
Real example: github.com/facebook/react → 301 → github.com/react/react.
Rate limits: unauthenticated REST is 60 req/hr per IP. A probe is 1–2 calls, fine. If you’re probing many links in one task and get a 403/429, stop — it’s the rate limit, not a dead link. (Check https://api.github.com/rate_limit to see remaining.)
Auth: if a token is available (GITHUB_TOKEN), add -H "Authorization: Bearer $TOKEN" for 5000 req/hr and to reach private repos. Don’t hardcode tokens; read from env.

What to fetch once the link is confirmed valid

Prefer endpoints that return clean, compact content over the HTML page:

File contents → https://raw.githubusercontent.com/OWNER/REPO/<ref>/PATH (plain text/markdown).
Structured data (issues, PRs, metadata) → the REST API (/repos/{o}/{r}/issues, /pulls, etc.) — returns JSON, not megabytes of HTML.

Decision summary

got a github.com link to read?
  -> probe repo via API (curl -o /dev/null -w code)
     404  => dead; report it, do NOT fetch the page
     200  => exists; if link is a file, probe contents path too
                404 => stale path; say so, don't fetch
                200 => fetch via raw.githubusercontent.com or API (not the HTML page)

Source: Wiki.js