Synced from GitHub
Human here! It's fun to hit edge cases like this. Moved over to T3 Code over Codex/ChatGPT Desktop a weekish ago and enjoying daily-driving it on Windows (the headed client that invokes these headless Linux remotes)
[Bug]: PR discovery on an idle server keeps spending the hourly GitHub budget every minute#23702
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
- Run the T3 Code server headless on Linux (systemd service,
t3 serve --host 0.0.0.0 --port 3773 --no-browser) against a working root that contains about 90 local git clones, with about 450 threads created over a few weeks (many with a branch). - Connect a desktop client once, then disconnect it. Leave the server running with no client connected and no user interaction.
- Watch the GitHub budget for the account the server authenticates as (
GET /rate_limit), and the server journal forthread branch pull request lookup failed.
No other T3 Code process was doing PR work at the time, and no Pull Requests page was open on any client.
Expected behavior
A server with no connected clients and no user input should not perform unbounded periodic PR discovery. Discovery work should be bounded (for example gated on connected clients, on changed git state, or on a real backoff), and a known quota pause should be honoured until its reset instead of retried on every pass.
The closing comment on #11220 asked for this remaining part to be filed separately:
One part of this is fixed on main in #13689 and #13765. […] One part is still open: unbounded periodic PR discovery and Git work on idle servers, including short-lived cache churn. If that part still matters to you, please open a new issue for that part only.
This report is only that part. It is on a build newer than those fixes.
Actual behavior
On an idle server (no clients connected), the PR discovery pass runs continuously and spends more of the hour's GitHub budget than the budget itself:
Cadence: one failing sweep per minute, continuously. The journal shows
thread branch pull request lookup failedandSourceControlRateLimitPausedError: github requests to github.com are paused until the rate limit resetsat:01,:02,:03… of every minute. In 24 h that is 1,932thread branch pull request lookup failedand 1,932listChangeRequestsfailures.Spend while idle: measured over a 9-minute window on a single idle server, the GraphQL budget for the server's account dropped 3,966 → 3,714 (252 points ≈ 28 points/min) with nothing else using that credential. That alone is ~1,680 points/hour of a 5,000/hour budget just for PR lookups, before any interactive use.
The headless server is the consumer: after turning the GitHub host off for that server (see workaround), the GraphQL budget on that account stopped draining (3,714 → 3,711 across several minutes), while the same server kept serving threads normally.
Cost scales with local projects and threads, which matches #11220's original report ("load grows with the number of projects and with the number of threads that have a branch or a PR"). Reducing the same server to ~100 threads / ~9k projection nodes with the same ~90 clones and the same settings produced 0 event-loop stalls and no quota errors over 24 h, so the sweep's cost tracks thread/branch count rather than clone count alone.
It retries straight through a known pause. #19781 already covers the log volume; the functional half is that the paused state keeps re-entering discovery once per minute until the reset. Nothing is attempted behind a proper backoff, so an idle server keeps producing failures for the rest of the hour.
Impact
Major degradation or frequent failure. On a server with this shape:
- The GitHub budget for that account is exhausted roughly 30 minutes into each hour, so unrelated tooling on the same account (agent
ghCLI, publishing PRs, CI checks) gets 403s for the rest of the hour. GitHub's REST and GraphQL budgets are per user, not per token, so several machines authenticating as the same user share one 5,000/hour allowance. A headless box nobody is looking at can therefore starve the human's CLI. - The server logs a full warning stack per failed group per minute (#19781), which on this host reached 11,826
pull-request-related log lines in 24 h and contributed to trace-writer backpressure (Trace writer could not keep up, with dropped counts in the 77k–92k range).
Workaround
For servers that do not need PR awareness, turning the host off in settings stops the calls completely:
json
| 1 | "sourceControlHosts": { "github": { "hosts": { "github.com": { "enabled": false } } } } |
The server then fails locally with GitHubHostDisabledError: GitHub host github.com is turned off in Settings → Source Control, makes no request, and GraphQL spend for that account drops to noise. That is what I used to confirm attribution, and it is the current state of these hosts.
Environment and version
- Server: Linux, headless, systemd unit running the release archive via
t3 serve; no desktop app on these machines; no client connected while measuring. - Version:
0.1.2-nightly.20261022.3526(reproduced on0.1.2-nightly.20261021.3499and0.1.2-nightly.20261022.3512). - Credential: the server was authenticated with
GH_TOKENfor the same GitHub login as other machines, so a per-user budget applies. - Scale: ~450 threads (V2 projection, ~162k nodes), ~90 local clones, 126 projects.
Related, not duplicates
- #11220 (closed; maintainer asked for this remaining part as a new issue)
- #17799 / #17800 (missing PRs re-read every sweep, even on settled threads)
- #19781 (known quota pauses log full warning stacks each sweep)
- #19657 (hidden subagent threads are polled by the sweep as well)
- #21777 (index active thread sweep expressions)
- #21888 (same class on Azure DevOps: one
azprocess per PR per minute)
Logs or stack traces
Redacted excerpt, one pass per minute while idle:
| 1 | [NN:01] WARN: thread branch pull request lookup failed |
| 2 | cause: 'SourceControlProviderError: Source control provider github failed in listChangeRequests: |
| 3 | GitHub API rate limit exceeded. Requests resume when the limit resets.' |
| 4 | [cause]: SourceControlRateLimitPausedError: github requests to github.com are paused |
| 5 | until the rate limit resets. |
| 6 | [NN:02] WARN: thread branch pull request lookup failed (same cause) |
| 7 | [NN:03] WARN: thread branch pull request lookup failed (same cause) |
Budget samples for the server's account while idle:
| 1 | t+0 graphql=3966 core=3493 |
| 2 | t+9m graphql=3714 core=1902 # ~28 GraphQL points/min, headless, no clients |
| 3 | after disabling the GitHub host on the server: |
| 4 | t+0 graphql=3714 |
| 5 | t+4m graphql=3711 # flat: the idle sweep was the consumer |
Synced from GitHub
Adding a controlled before/after on the same idle server, since the quota-spend numbers in the report were measured with the sweep already running.
Condition A — GitHub host enabled, no clients connected. Stall counter for the preceding 24 h: 558 event-loop stalls (~23/hour, one every 2.6 minutes), plus 1,932 thread branch pull request lookup failed and 1,932 listChangeRequests quota failures.
Condition B — GitHub host disabled (sourceControlHosts.github.hosts["github.com"].enabled = false), same server, same threads, same projects, still no clients. Measured every 60 s for 30 minutes (31 samples):
| 1 | stalls 558/24h -> 0 in 30 min (journal: 0 "event loop stalled" lines) |
| 2 | last stall 15:15:44 UTC; window 15:45:49 -> 16:15:52 UTC |
| 3 | quota pauses 1,932/24h -> 0 |
| 4 | GraphQL budget draining ~28 points/min -> flat 3714 -> 3711 |
| 5 | WAL 5.1 MB -> 8.6 MB, then flat (the 32 MB gate was never approached) |
If the baseline rate held, ~11.6 stalls would be expected in 30 minutes, so zero is not a fluke of the window. The only change between A and B was disabling the GitHub host; local git, threads, projects and settings were untouched.
Two honest qualifications:
- Only ~39% of the stalls in condition A fell within 30 s of a
SourceControlRateLimitPausedError. That fits the shape of the bug rather than contradicting it: during a known pause the lookup fails fast, while an unpaused lookup is the expensive case. The counterfactual above (disable the path, stalls stop) is the stronger evidence. - Disabling the host did not stop trace-writer backpressure.
Trace writer could not keep upstill logs about once per minute (27 of 31 samples), each resetting a largedroppedCount, with no event-loop stalls at all. That looks like steady-state span volume rather than blocking, and is probably #19781-adjacent / the "short-lived cache churn" part of #11220, so I am not claiming it here — noting it so the drops are not attributed to this fix.
For scale reference: the same build on a sibling server with ~100 threads / ~9k projection nodes, same ~90 clones and the same settings, recorded 0 stalls over 24 h. The table above is the same server before and after the path was removed.
Synced from GitHub
One housekeeping note: I filed this through the GitHub API, which bypasses the web issue form, so the template's labels were not applied — this is an unlabelled issue rather than a report someone decided not to categorize.
If a maintainer or triage run passes through, bug and needs-triage are the labels the bug report form would have set. No other action needed; the report above stands as written.