Prevent QPay payment after stock expires and preserve paid truth#329

Incident

On September 1, 2026 at 08:49 ULAT, QPay reported a 156,000 MNT payment for an Order created on August 27. One Order line, product 7635, had stock 5 when the Order was created but stock 0 when the customer returned five days later.

The QPay webhook verified the invoice as paid, then confirmPaymentAndApplyStock failed with Insufficient stock for product 7635. The transaction rolled back the Payment status along with the stock update. QPay had the financial payment while our database still treated the Payment as unpaid. Client polling retried the same impossible confirmation 20 times.

Do not publish customer phone numbers, QPay transaction IDs, or live Payment URLs in this issue. The production identifiers remain in Axiom under qpay.webhook_confirm_failed at the incident time.

Root cause

The current flow has two different stock checks with an unsafe gap:

1. order.addOrder checks current stock in prepareOrderCheckout.
2. Creating an Order does not reserve or deduct stock.
3. precreateQpayInvoice creates a real payable invoice after Order creation.
4. payment.createQr can reuse or create a payable invoice later. It checks Payment access and status, but it does not revalidate or reserve the Order lines.
5. confirmPaymentAndApplyStock first changes the Payment to success, then tries to deduct stock in the same database transaction.
6. If stock is unavailable, the transaction rolls back both changes. This erases our record of an externally completed payment.

Relevant code:

• packages/api/src/routers/store/order.ts: prepareOrderCheckout, precreateQpayInvoice
• packages/api/src/routers/store/payment.ts: createOrReuseQpayInvoice, createQr, checkQpayPayment
• packages/api/src/queries/payments.ts: confirmPaymentAndApplyStock
• packages/api/src/lib/payments/transfer-confirmation.ts: confirmPaymentAndNotify
• apps/server/src/routes/payments.ts: QPay webhook confirmation

Adding another non-atomic stock check before invoice creation is not enough. Stock can change between that check and payment. The payable invoice and its stock must share one explicit lifecycle.

Required contract

1. The storefront must not show a payable QPay invoice unless every Order line has an active stock reservation.
2. A live QPay invoice and its stock reservation must have the same bounded lifetime.
3. Expiring an invoice, cancelling an Order, or switching away from QPay must cancel the unpaid QPay invoice and release its reservation exactly once.
4. A QPay-paid response is financial truth. Once QPay reports paid, our Payment must become success idempotently even if fulfillment cannot proceed.
5. A paid Payment without fulfillable stock must create a visible fulfillment exception and alert staff. It must not remain pending, retry forever, or appear unpaid to the customer.
6. Reopening an old unpaid Order must acquire a new reservation before creating a new invoice. If stock is unavailable, show the affected products and do not call QPay.

Implementation

1. Remove speculative payable invoices

Remove precreateQpayInvoice from order.addOrder. Create a QPay invoice only when the customer opens or explicitly chooses the QPay payment flow.

The small extra invoice latency is preferable to reserving stock for every abandoned Order and issuing invoices customers never view.

2. Add audited stock reservations

Add a payment_inventory_reservation table with:

• Payment ID or Payment number
• Product ID
• Quantity
• active | consumed | released status
• expiresAt, createdAt, updatedAt
• A unique key on Payment and Product

Acquiring a reservation must run in one database transaction:

• Lock ownership to a pending QPay Payment.
• For every Order line, conditionally decrement product stock only when the product is active and enough stock remains.
• Insert the matching active reservation rows.
• Roll back all lines if any line cannot be reserved.

The existing stock value remains sellable stock. Reserving decrements it immediately. Consuming a reservation on payment does not decrement it again. Releasing a reservation increments it exactly once.

3. Bind invoice creation to the reservation

payment.createQr must:

1. Return an existing invoice only when its reservation is active and unexpired.
2. Otherwise cancel any old unpaid QPay invoice.
3. Acquire stock for all Order lines.
4. Create the QPay invoice.
5. Persist the invoice ID and a one-hour expiry matching the reservation.
6. Release the reservation if invoice creation fails. A crash may leave it active until the expiry worker releases it.

Use QPay DELETE /v2/invoice/{invoiceId} to cancel unpaid invoices. Treat already-cancelled as success. If QPay reports that the invoice is already paid, enter the paid-confirmation path instead of releasing stock.

4. Expire invoices and release stock

Add a scheduled job that claims expired active reservations in bounded batches:

1. Check or cancel the linked QPay invoice.
2. If unpaid cancellation succeeds, release stock and mark reservations released in one transaction.
3. If QPay reports paid, record the payment and consume the reservation.
4. Make retries safe after crashes and concurrent webhook delivery.

Switching a Payment to transfer and cancelling its Order must use the same cancel-and-release boundary.

5. Separate payment truth from fulfillment

Refactor QPay confirmation so recording external payment success cannot roll back because stock fulfillment failed.

Normal reserved path:

• Transition Payment to success.
• Mark reservation consumed without another stock decrement.
• Continue existing notifications, analytics, and cache purge.

Recovery path for a paid invoice with no usable reservation:

• Transition Payment to success using the QPay payment ID as the idempotency key.
• Record a fulfillment exception containing the affected product IDs and quantities.
• Keep the Order out of automatic shipping.
• Alert staff with clear substitute/refund action.
• Show the customer that payment was received and staff will contact them. Never show “unpaid” or a generic internal error.

Use a named fulfillment exception state or table rather than overloading Payment.status. Payment state answers whether money moved; fulfillment state answers whether the Order can ship.

6. Stop impossible polling

Client polling and webhook retries must treat these as terminal outcomes:

• Payment confirmed and reservation consumed
• Payment received but fulfillment blocked
• Invoice expired/cancelled and stock released

Do not retry a deterministic stock failure every few seconds.

Customer errors

When reservation fails before invoice creation, return a typed conflict with the unavailable product IDs and a Mongolian message equivalent to:

Зарим барааны үлдэгдэл дууссан тул төлбөр үүсгэсэнгүй. Сагсаа шинэчлээд дахин оролдоно уу.

The payment page should link back to the cart or current product alternatives. It must not render a QR or bank-app link.

Observability

Add structured events with Payment and Order identifiers, but no phone number:

• qpay.reservation_created
• qpay.reservation_released
• qpay.reservation_expired
• qpay.invoice_blocked_out_of_stock
• qpay.paid_fulfillment_blocked

Alert immediately on qpay.paid_fulfillment_blocked. Track reservation age and active reservation count so leaked holds are visible.

Fix the current logging contract as part of this work: a tRPC response containing an error must not log outcome: success merely because middleware wrapped it.

Acceptance criteria

• An old pending Order whose product is now unavailable cannot create or display a QPay invoice.
• Creating an invoice reserves every line atomically and reduces sellable stock once.
• Concurrent invoice requests for the same Payment do not create duplicate reservations or invoices.
• Concurrent Payments cannot reserve more units than available stock.
• Successful QPay confirmation consumes the reservation without deducting stock twice.
• An expired unpaid invoice is cancelled and its stock is released exactly once.
• Switching from QPay to transfer cancels the unpaid invoice and releases its reservation.
• A webhook racing invoice expiry records either a cancelled invoice with released stock or a successful Payment with consumed stock, never both.
• If QPay reports paid without a usable reservation, Payment becomes success, a fulfillment exception is visible to staff, and the customer does not see an unpaid state.
• Webhook and polling retries remain idempotent.
• The incident shape is covered by a database-backed test: Order valid at creation, stock unavailable before invoice request, no QPay invoice issued.
• The paid-without-stock recovery path has a database-backed test. Do not use mocked payment or repository tests; exercise the real transaction boundaries with controlled QPay responses at the HTTP boundary.
• bunx vp lint and bunx vp fmt --check pass for changed files.

Out of scope

• General warehouse allocation across multiple physical locations
• Reserving stock for manual bank transfers before staff confirms them
• Automatically choosing a substitute product or issuing a refund
• Repairing the September 1 incident automatically; staff must verify settlement and handle it manually