[Bug]: Codex usage skips a request whose cumulative counters rewind to an earlier snapshot#29363
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
Scan this synthetic rollout with the current Codex usage reader:
| 1 | {"type":"session_meta","timestamp":"2026-11-02T10:00:00.000Z","payload":{"id":"rewound-counters-repro","model_provider":"openai"}} |
| 2 | {"type":"turn_context","timestamp":"2026-11-02T10:00:00.000Z","payload":{"model":"gpt-6-sol"}} |
| 3 | {"type":"event_msg","timestamp":"2026-11-02T10:00:01.000Z","payload":{"type":"token_count","info":{"last_token_usage":{"input_tokens":1000,"cached_input_tokens":0,"cache_write_input_tokens":0,"output_tokens":200,"reasoning_output_tokens":0,"total_tokens":1200},"total_token_usage":{"input_tokens":1000,"cached_input_tokens":0,"cache_write_input_tokens":0,"output_tokens":200,"reasoning_output_tokens":0,"total_tokens":1200}}}} |
| 4 | {"type":"event_msg","timestamp":"2026-11-02T10:00:02.000Z","payload":{"type":"token_count","info":{"last_token_usage":{"input_tokens":100,"cached_input_tokens":0,"cache_write_input_tokens":0,"output_tokens":20,"reasoning_output_tokens":0,"total_tokens":120},"total_token_usage":{"input_tokens":1100,"cached_input_tokens":0,"cache_write_input_tokens":0,"output_tokens":220,"reasoning_output_tokens":0,"total_tokens":1320}}}} |
| 5 | {"type":"event_msg","timestamp":"2026-11-02T10:00:03.000Z","payload":{"type":"token_count","info":{"last_token_usage":{"input_tokens":300,"cached_input_tokens":0,"cache_write_input_tokens":0,"output_tokens":50,"reasoning_output_tokens":0,"total_tokens":350},"total_token_usage":{"input_tokens":1400,"cached_input_tokens":0,"cache_write_input_tokens":0,"output_tokens":270,"reasoning_output_tokens":0,"total_tokens":1670}}}} |
| 6 | {"type":"event_msg","timestamp":"2026-11-02T12:00:00.000Z","payload":{"type":"token_count","info":{"last_token_usage":{"input_tokens":200,"cached_input_tokens":0,"cache_write_input_tokens":0,"output_tokens":40,"reasoning_output_tokens":0,"total_tokens":240},"total_token_usage":{"input_tokens":1300,"cached_input_tokens":0,"cache_write_input_tokens":0,"output_tokens":260,"reasoning_output_tokens":0,"total_tokens":1560}}}} |
The last request's cumulative counters fall below the previous snapshot (1,670 → 1,560 total). Cumulative minus its own request (1,100 input + 220 output) is exactly the second snapshot, so the request continues from an earlier point in the same rollout.
Expected behavior
Bill all four requests: 1,910 processed tokens, no skipped-record warning. Later requests keep reconciling against the new baseline.
Actual behavior
Three requests are billed (1,670 tokens) and the Usage page reports 1 malformed or inconsistent usage record. The last request fails the exact-delta check (negative delta) and fails the restart check (cumulative ≠ last), so it is dropped. The cached result keeps the warning on the Usage page permanently, because the rollout never changes again.
Evidence and diagnosis
This comes from real rollouts. I replayed 786 Codex rollouts from three hosts (macOS, Linux x64, Linux arm64) read-only through the current reducer (parseCodexLine, plus readTranscriptRecords in buffered and forced-streaming mode). 4 records are skipped; 3 of them are this case:
| previous total | new total | request | total − request | matches an earlier snapshot |
|---|---|---|---|---|
| 21,496,033 | 12,659,951 | 201,190 | 12,458,761 | ~2 h earlier in the same file |
| 11,786,833 | 5,405,043 | 172,103 | 5,232,940 | ~12 h earlier |
| 11,837,804 | 10,385,696 | 202,869 | 10,182,827 | ~22 h earlier |
In each case:
- the remainder equals the
total_token_usageof an earliertoken_countin the same file, which was the last snapshot of an earlier turn; - the request is new (its
last_token_usagematches no earlier request in the file) and opens a new turn. Two of them come 1.5 h or more after the previous request and run on a cold cache (e.g. 10,880 cached of 200,844 input), so they are expensive requests. The third comes 3 minutes after the previous request, on a warm cache; - the following requests reconcile exactly against the new, lower baseline.
There is no rollback marker, no compacted record and no new session_meta between the earlier snapshot and the drop. The counts are measured. Why Codex continues from an older snapshot is not proven; a second process with stale in-memory totals writing to the same rollout fits the data. Codex seeds token_info from the last TokenCount of the history it loaded (last_token_info_from_rollout in codex-rs/core/src/session/mod.rs).
The 4th skipped record is a different problem (an interrupted turn whose last request only appears as a token_usage_record), filed as #29365.
Related work
- #22376 / #22558: counters that restart at the request itself (
cumulative == last). - #23729: no baseline yet (
previous === null). Validatescumulative - lastand bills onlylast. This case is the same shape, except that a baseline exists and the counters fall below it. - #22392: replay of the current sequence's first snapshot.
Impact
Minor bug or occasional failure. Usage omits the rejected requests and shows a persistent partial-source warning.
Version or commit
T3 Code 0.1.6 (desktop). Same logic on main at ac8dce0cc3ab (parser blob 20939e54db79946a8f0e0f3badabdde917752b59).
Environment
macOS ARM64 desktop, with remote servers on Linux x64 and Linux arm64. Codex rollouts from September 2026. The synthetic reproduction needs no provider account.
Workaround
None in the app. The warning stays until the cached entry is re-parsed with a fixed reader.