A Volume run can enter the lane with a gen below a sync that checked for none#4081

opened1d agobyNNick
GitHub writeback is off. Issues and comments you change here stay in GitCafe. Sync settings

A Volume run's gen is allocated by the identity column when takeLane inserts its row (dashboard/src/modules/volume-run/volume-run.server.ts, takeLane). The insert then waits on the active-lane unique index. So a run can hold a lower gen and still enter the lane after a run with a higher gen has started and ended.

The mirror sync relies on the opposite (mirror-sync.server.ts, "Read before the lane check, so a lane opened after that check carries a higher gen and fences this sync"). Sequence:

  1. Run X allocates gen 1, then waits on the lane index.
  2. Run Y (a resize, which raises no SetVolumeCopy floor) allocates gen 2, runs and ends.
  3. A sync reads gen 2 and finds no lane open.
  4. X enters the lane with gen 1 and runs. Its sets raise the Machines' floors only to 1.
  5. The sync's SetVolumeCopy, at gen 2, passes those floors, so nothing fences it from X.

Seen in review of #4069, where it can leave a Mirror's kept approvals behind the one the Mirror holds. #4069 now tolerates the overlap: an accepted approval always goes back on its Mirror's list. The fence itself is still weaker than the comment says for any sync that overlaps a run. A Postgres 16 reproduction of the allocation inversion: the sync saw max(gen)=2 and no active run, then the delayed insert committed with gen 1.

Suggested fix: serialize lane admission per Volume (for example a transaction-scoped advisory lock on the Volume id) before the insert allocates a gen, held through commit on every lane-entry path.

Frequency F=1 (needs a run inserted while another holds the lane, a second run that starts and ends in that window, and a sync between). Severity S=5 (a sync overlapping a run; recoverable). F×S=5, so filed rather than fixed in #4069.