[Bug]: Known GitHub quota pauses repeatedly log full warning stacks in thread PR discovery and settlement#19781

opened8d agobyJjosephv123

Before submitting

  • Searched quota-pause, settlement and PR-discovery reports.
  • Included log counts, scope limits and the catch paths.

Area

apps/server

Steps to reproduce

  1. Have unresolved branch PR lookups in threads eligible for PR discovery or automatic settlement.
  2. Let the GitHub account enter a SourceControlRateLimitPausedError pause until its known reset time. This can be caused by other tools sharing the token; T3 need not exhaust the quota itself.
  3. Allow the periodic sweeps to run during that pause.

Expected behavior

An expected, already-known quota pause is summarized or throttled until retryAt, while unexpected failures retain useful warning stacks. Repeated blocked groups should not emit the same full stack every sweep.

Actual behavior and evidence

ThreadPullRequestService.synchronize logs thread branch pull request lookup failed with all grouped thread IDs and Cause.pretty(cause). ThreadSettlementService.sweep does the same with automatic thread settlement skipped. Neither catch path treats the known quota-pause reason specially. The sync reactor already has quota-pause/backoff handling (#16203); these sibling paths still warn per group on later sweeps.

Read-only recount of the retained Linux service log (approximately 17 days) separated actual quota-paused records from other errors:

WarningRecords containing SourceControlRateLimitPausedErrorBytes of those records
thread branch pull request lookup failed68179,040
automatic thread settlement skipped68177,366
Total136356,406

The whole log was 1,179,312 bytes, so these expected-pause records alone were 30.2%. Both prefixes including unrelated failures totaled 447,866 bytes; that larger total is not all quota-pause noise. These were multiline warning records, split at timestamped log headers, rather than a count of matching individual lines.

Impact / version / environment

Minor diagnostic noise. The entire retained service log is only about 1.18 MB; this report does not claim a disk-capacity problem, high CPU, a GitHub network retry storm, or that T3 exhausted the token. Quota admission already refuses the paused reads correctly. The defect is repeated warning-level stack output for an expected condition.

Ubuntu Linux service, historical nightly logs through 0.0.46-nightly.20261014.3132; both catch paths still present on main e536bfd33bc3c351997141d93790df75eeded7b9.

Note

Grok responding on behalf of Julius.

Accepting as a low-priority logging bug. Confirmed on main: ThreadPullRequestService.synchronize (~L350) and both catch paths in ThreadSettlementService (~L371 and L541) log Cause.pretty(cause) at warning level for any non-interrupt failure, including a known SourceControlRateLimitPausedError. The sync reactor already handles that case (#16203). Behavior is otherwise correct, since the paused reads are refused, so this is only noise.

A focused PR is welcome: match the paused error in those catch paths and log it once at debug/info level with retryAt, keeping full stacks for unexpected failures. Please add a test.