Sign up
View stargazers13Sign up

[Bug]: Non-sparse skip-worktree entries still force full checkpoint index rebuild after #10792 / #12154#13741

opened28d agobyXXiaChuerwu

Before submitting

  • Searched existing issues. Related reports and merged fixes are linked below; this report concerns the remaining non-sparse checkout with special index flags path.
  • Included the observed environment, triggering conditions, implementation analysis, and local validation results.

Area

apps/server — Git checkpoint capture

Version / environment

  • Windows desktop, 0.0.43-nightly.20260925.2269 (also observed on 0.0.43-nightly.20260923.2173).
  • Windows 11; Git for Windows 2.54.0.windows.1.
  • Worktree accessed through a mapped SMB drive.
  • Non-sparse checkout, approximately 24,804 tracked index entries, with 6 manually skip-worktree-marked entries.

Related fixes / reports

  • #10792 — merged index-metadata reuse optimization.
  • #12154 — merged fix for large sparse checkouts; this is a follow-up for the non-sparse special-flags branch, not the sparse-output-limit case.
  • #12080, #3646, #12924 — related checkpoint fallback / whole-worktree hashing reports.

Steps / triggering conditions

  1. Open an existing non-sparse repository with some tracked entries marked skip-worktree in the Windows desktop app. A sufficiently large worktree or higher-latency filesystem makes the cost visible.
  2. Start a turn that captures a checkpoint.
  3. Observe checkpoint capture repeatedly hit the 30-second VCS timeout, even though ordinary status/diff queries complete within the limit.

This is a repeatedly observed real-worktree reproduction, not a claim that every small repository with a skip-worktree entry will time out. The affected implementation also rejects assume-unchanged flags; the real-worktree timeout observation here involved skip-worktree entries.

Expected behavior

Special flags on a handful of entries should not force loss of cached metadata for every other tracked file on each checkpoint. The checkpoint must still capture changes/deletions in flagged files correctly, without changing the user's real index or configuration.

Actual behavior / impact

Major degradation or frequent failure: checkpoints repeatedly fail.

text
1VCS process timed out in GitVcsDriver.checkpoints.captureCheckpoint:
2git (<redacted-worktree>) after 30000ms

Read-only baseline timings in the affected worktree:

  • git --no-optional-locks status --porcelain=v1 -uall: 9.84 s.
  • git --no-optional-locks diff --stat: 5.84 s.
  • git ls-files -v -z: 0.32 s.

Historical logs contained 16 matching checkpoint timeout spans. Inspected traces consistently showed seven successful Git calls within capture followed by an approximately 30-second failing call. That order matches the fallback path below and its subsequent add -A; existing traces do not include argv, so the individual failing subcommand is inferred rather than directly recorded.

Implementation analysis

Inspection of the installed build shows:

  1. Copy the real index to a private GIT_INDEX_FILE and run read-tree --reset HEAD.
  2. Inspect ls-files -v -z flags.
  3. Any lowercase flag, or S when !sparseCheckout, sets specialFlags = true.
  4. Reject the copied index, remove it, and seed a fresh index with read-tree HEAD.
  5. Run git add -A -- . with the default 30-second command timeout.

Thus a few manually skipped entries still take the fresh-index path after the sparse-checkout fixes. Cached stat information for unrelated entries is discarded on every capture. Filesystem latency can amplify the work, but the repeatable trigger is this checkpoint-specific fallback.

Locally validated mitigation (not an upstream PR yet)

A local experimental patch retains the copied index for the non-sparse branch and rebuilds only the flagged entries in the private index:

  • Collect mode, OID, stage, and NUL-delimited paths via ls-files --stage -v -z.
  • Feed those entries to update-index -z --index-info with the private GIT_INDEX_FILE, preserving tracked status and OIDs while clearing those entries' flags/stat cache.
  • Restore the existing conservative index timestamp; retain the existing safe fallback on failure.
  • Leave sparse-checkout handling and unflagged entries' metadata unchanged.

Simply clearing flags was not sufficient in local verification: same-size/same-mtime content edits could still reuse stale metadata. Removing entries outright would also need care for ignored-but-tracked files. The selective index-info approach avoids those two pitfalls.

Isolated checks covered h/S/s changes and deletions, same-size/same-mtime edits, racy timestamps, ignored-but-tracked files, staged/worktree differences, root/subdirectory scope, and unchanged source-index hashes. Snapshot trees matched the clean-index baseline; 62 unflagged entries retained their stat/OID metadata. Limitations: the isolated harness substituted a tree OID for HEAD, symlink representation used Windows core.symlinks=false, and actual submodule HEAD changes were not exercised.

After applying the local patch and fully restarting T3, one real checkpoint completed successfully in 11.09 s, with its outer event chain also succeeding. This is one successful post-patch runtime observation, not a long-running benchmark or a claim of complete correctness coverage. No user Git configuration or real-index flags were changed to obtain this result.

Would selectively rebuilding special-flag entries in the private index be an acceptable upstream direction? More granular checkpoint subcommand diagnostics would also make this failure much easier to investigate.

JJulius Marminge28d ago

Triage: real server checkpoint bug. Confirmed on current main. Your 0.0.43-nightly.20260925.2269 is after #10792 and #12154, and this path is still there. Not a duplicate of #12080 / #12154 (sparse S flags and the listing cap), #3646 / #12924 (the old always-fresh index), or open #13543.

What you hit. For a non-sparse repo, capture copies the index, runs read-tree --reset HEAD, then streams git ls-files --full-name --sparse -v -z. Any lowercase tag, or S when core.sparseCheckout is not set, sets specialFlags in apps/server/src/vcs/GitVcsDriver.ts. Reuse is rejected, the private index is deleted, and capture seeds a stat-less index with read-tree HEAD and then git add -A -- .. That add uses the default 30s VcsProcess timeout. The timeout text is only VCS process timed out in <operation>: git (<cwd>) after <ms>, with no argv, so the failing command is not recorded. The seven successful Git calls you saw, then a ~30s failure, match this sequence: rev-parse --git-common-dir, rev-parse --verify HEAD, config --bool core.sparseCheckout, rev-parse --git-path index, read-tree --reset HEAD, ls-files -v -z, fresh read-tree HEAD, then add -A. SMB latency makes the full rehash miss 30s; the trigger is the fallback, not the mapped drive. Six flagged entries are enough.

Why the fallback exists. #10792 kept it on purpose instead of normalizing flags. git add -A will not record edits or deletions for assume-unchanged or skip-worktree paths, and the checkpoint has to. #12154 only stopped treating ordinary sparse-checkout S bits as manual flags. The comment in source is still "Manual flags still need a reset." Clearing the flag bits alone is not enough: the private index is backdated so Git will trust a same-size, same-mtime stat cache. That matches what you saw.

Direction. Selectively rebuilding only the flagged entries in the private index is the right fix. After the existing copy and read-tree --reset HEAD, feed those entries to update-index -z --index-info on GIT_INDEX_FILE so they keep their mode, OID, and stage, lose CE_VALID / CE_SKIP_WORKTREE, and get an empty stat cache. Leave every other entry's metadata in place. add -A then rehashes the flagged paths and still skips unchanged ones.

Constraints for a PR:

  • Touch only the private index. The real index and Git config stay as they are.
  • Put the conservative index timestamp back after update-index, not only after read-tree. Otherwise the racy-git protection from #10792 is lost.
  • On any failure, keep today's fresh-index fallback.
  • Do the same for manual flags on a sparse checkout: lowercase tags, and in-cone paths that sparse-checkout check-rules selects. One manual flag there still forces a full rebuild. Do not clear ordinary out-of-cone S bits; #12154 depends on those staying so excluded paths are not recorded as deletions.
  • Invalid UTF-8 paths already force specialFlags. Keep the fallback for those instead of round-tripping them through strings.
  • Cover this in GitVcsDriver.test.ts: one skip-worktree entry and one assume-unchanged entry must not take the fresh read-tree HEAD path; unflagged entries must keep their stat metadata; the snapshot must still include flagged edits and deletions, including same-size same-mtime edits; the real index must stay byte-identical. If a split index or submodule cannot be updated this way, fall back rather than guessing.

#13543 does not fix this. It raises git add -A to 5 minutes and reuses a saved private index on later whole-worktree, non-sparse captures. It does not run the flag check on that saved index because it assumes the index was already clean or rebuilt from read-tree HEAD. The first capture, any capture that times out before an index is saved, and every nested or sparse capture still take this fresh-index path. A cold add -A that exceeds the stage budget never writes the saved index, so every turn stays slow. The two changes compose only if flags are cleared before staging, so the saved index has none.

Workaround. None that preserves the flags. Unsetting the six skip-worktree bits would allow reuse and would also change the real index. Do not treat "update the nightly", "wait for #13543", or "raise the timeout" as closing this. A separate change to print the Git subcommand on timeout is unnecessary to confirm this failure.