Sign up
View stargazersSign up

[Bug]: Shared import record limit skips valid conversations#17484

opened26d agobyDdiogovalada

Before submitting

  • I searched existing issues and did not find a duplicate.
  • I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

Import a project whose eligible Codex transcripts collectively exceed 100,000 JSONL records, with a later transcript larger than the remaining budget. That conversation is skipped even if it fits within the limit on its own.

Expected behavior

Import all selected, valid conversations, or clearly identify which hit the limit and offer a way to continue.

Automatic batching or a configurable cap would help. Removing the hardcoded import limits from the code entirely is also worth considering: users are importing their own existing sessions and should be able to decide how much to import. Show the sizes and warn about time and resource usage, then leave that decision to them. Streaming, cancellation and malformed-file checks can still protect the app without rejecting valid conversations based on size or record count.

Actual behavior

I imported a project with four Codex conversations. Three appeared in T3; one was missing. The import only reported that some threads could not be imported.

The missing conversation is intact in Codex and currently has about 58,000 JSONL records. The earlier import audit traced the skip to MAX_IMPORT_RECORDS = 100_000, shared across the project: other transcripts had consumed the remaining allowance. Tool and system events count too.

Relevant scanner code.

Impact

Minor bug or occasional failure

Version or commit

0.0.45

Environment

Windows desktop connected to a Linux server, Codex sessions.

Logs or stack traces

shell
1

Screenshots, recordings, or supporting files

No response

Workaround

Retrying can skip unchanged, already-imported transcripts and free the next pass’s budget. It cannot help a transcript that exceeds the cap by itself. I have not verified a successful retry for the missing conversation yet.

SSriCharanK200215d ago
Synced from GitHub

Reproduced on desktop 0.1.0 (macOS) with a related limit: the per-project transcript cap, MAX_IMPORT_TRANSCRIPTS = 100 (AgentSessionScanner.ts:95), rather than the record cap.

One project had 148 eligible .jsonl transcripts, Codex and Claude together. prepareRecentThreads sorts them newest-first by mtime and reserves budget for every file it reads, including ones that then fail to parse. The onboarding trace shows exactly 100 AgentSessionScanner.readTranscript spans for that project's import, followed by silent skips:

  • 43 of the first 100 files were Codex subagent rollouts (#16222). 24 became top-level threads, and 19 were rejected for having no user message, but still used up budget.
  • That pushed 40 Claude Code conversations past position 100. They were reported only as part of "N threads could not be imported".
  • The record cap wasn't the limit here: those 148 files total about 81k records.

Re-running the import (location.hash = '#/welcome' in the desktop DevTools) recovered all of them, because already-imported files with an unchanged identity don't consume budget. So the behavior is correct but invisible: nothing tells the user that a second run would import more, and there's no way to get back to the wizard after onboarding (#10961).

+1 to batching automatically until no new files remain, or at least reporting "N skipped because of the import limit, run again to continue". Filtering subagent rollouts before budgeting would also help a lot.

Comment prepared by Claude Code (claude-opus-5-5) via t3 triage.

JJulius Marminge13d ago

Reproduced on current main 70e5e384da with the web client and a Linux dev server. Four synthetic Codex conversations in one project contained 20,000, 20,000, 20,000 and 58,000 JSONL records, newest first. The wizard discovered all four, then imported three and reported one could not be imported. Retrying imported the fourth conversation. Runtime transcript. No duplicate found in the open-issue scan.

Mmthsmb5d ago
Synced from GitHub

A related data point on 0.1.5 (Windows 11 desktop app + WSL2 backend), about the per-attempt file limit rather than the record limit:

  • One project had 308 Claude Code sessions in the 30-day window. Because of MAX_IMPORT_TRANSCRIPTS = 100, it took three manual wizard runs, plus a fourth to confirm nothing was left. Each run ended with "Imported N threads. M threads could not be imported." The docs do say "Run import again to continue a large batch", but the toast reads like a failure, so it's easy to assume the remaining conversations are broken.
  • Empty Codex sessions count as failures. Four rollouts that contain only session_meta and thread_settings_applied (opened in the VS Code extension and never used) show up as "4 threads could not be imported" on every run, permanently.

In line with this issue's expected behavior: continue in batches automatically until done (or offer a Continue action), report "deferred by the batch limit" separately from "unreadable", and don't count sessions with no user messages as failures.

There is also a separate scan-stage limit that re-running cannot get past: #27163.

I've also opened an idea for making these limits configurable: #27165.