feat(logs): unify live tails and serve retained history#1487

_{P1 Badge} Keep the v1 history RPC while adding v2

An existing Cloud/SDK client compiled against ContainerLogHistory will receive Unimplemented as soon as its daemon is upgraded because this replaces the method and removes the advertised ployz.container.log-history.v1 capability; conversely, the new client cannot query an older daemon. Machine RPC is covered by the 0.x stable promise in core/DESIGN.md, so introduce LogHistory alongside the v1 operation rather than replacing it within protocol major 1.

AGENTS.md reference: AGENTS.md:L5-L5

Useful? React with 👍 / 👎.

The operator explicitly approved replacing the old log contract with TailLogs and LogHistory while the user base is small. core/DESIGN.md documents that matching clients and daemons are required. No legacy history RPC or fallback is retained, as approved.

_{P1 Badge} Preserve the existing log-window wire field

During a supported mixed-version upgrade, an older CLI sends since_unix_seconds to this daemon; serde ignores that now-unknown field and defaults since_nanos to None, so --since silently returns output from before the requested boundary. The reverse pairing similarly loses the bound because an older daemon ignores since_nanos. This renames a frozen Machine RPC field without changing PROTOCOL_MAJOR, contrary to the additive compatibility promise in core/DESIGN.md; retain the old field or add an additive representation that both versions can interpret.

AGENTS.md reference: AGENTS.md:L5-L5

Useful? React with 👍 / 👎.

The operator explicitly approved requiring matching new clients and daemons for the two-RPC simplification. core/DESIGN.md now records the stable-promise exception for this release. TailLogs replaces both old live methods, so supported mixed-version log reads are intentionally excluded. Retaining the old field would preserve a compatibility path the operator declined.

_{P1 Badge} Update the JavaScript runtime-log transport

@ployz/sdk still routes both Client.runtime.logs() and Client.runtime.logHistory() through runtime-logs.js:27, whose input always includes before_nanos, and index.js:46 deserializes that object into this #[serde(deny_unknown_fields)] type. Removing the field therefore rejects even ordinary live-log requests where before_nanos is null, while history continues calling the obsolete container-log path; update and verify the JavaScript/native transport in this change or retain input compatibility.

AGENTS.md reference: core/AGENTS.md:L22-L22

Useful? React with 👍 / 👎.

Fixed in c23056e19. The complete JavaScript and native SDK transport now belongs to this PR, including runtime.logHistory and its public declarations. Thirteen Node log tests, SDK types and native transport checks pass. OBS-4 no longer owns the transport migration.

_{P2 Badge} Correct the retained-log lifecycle documentation

This new section says removed-service logs are retained, but the same page still states that replacing a container deletes its logs and leaves an older Deployment’s Deploy tab empty. The new Log Store and deployment-filtered history now retain and serve those records, so users receive contradictory guidance about the behavior introduced here; update the lifecycle bullet as part of this user-visible change.

AGENTS.md reference: AGENTS.md:L6-L6

Useful? React with 👍 / 👎.

Fixed later in this stack, in #1484. The "A deploy clears the old logs" bullet is gone. docs/user/observe/logs.md now says deleted environments and removed services keep their logs until retention removes them.

_{P2 Badge} Exclude exit events from printed-line coverage

When a followed history window contains an exit event, this records the exit timestamp as though an output line had been printed. For example, logs -n 1 -f on an exited or restarting container can return only the newest exit row; the subsequent Docker tail starts at that exit time and Printed::covers suppresses all earlier container output, so the requested log line is never shown. Only HistoryEvent::Line should advance the per-container output boundary.

Useful? React with 👍 / 👎.

Not changing this. since() returns the first printed timestamp, so the live read starts there no matter what. covers() hides a line only if it is at or before what was printed for that container. A line after an exit is newer and still prints. Counting the exit only stops older lines from coming back, and those fall outside the -n window that Docker applies anyway.

_{P1 Badge} Preserve failed-container logs before removing it

When replacement health or serving checks fail, compensation now removes the new container before apply calls Direct::tails. That diagnostic path still uses TailLogs, whose daemon handler first inspects the Docker container, so the removal makes it return NotFound and log_tail silently produces an empty tail. Read the retained history for this diagnostic or defer removal until after the failure report has captured the output.

Useful? React with 👍 / 👎.

Fixed in 57472e338. The failure tail is now read from the Log Store (LogHistory for that container, newest first) instead of TailLogs, so a container that compensation removed still reports its last 10 lines. The CLI report and the Store runner share one helper now. The test failed_deploy_whose_container_is_removed_still_reports_its_output_lines fails without the fix and passes with it.

_{P2 Badge} Allow fanout cursors past 4 KiB

In the dashboard history flow, the SDK cursor serializes one state entry per Server, so a page where roughly 25 Servers each have more history already produces a cursor over 4 KiB; the next /api/runtime/logs request is then rejected as an invalid selection and scrollback stops. This is reachable because the product explicitly supports unlimited Servers (docs/user/account/organizations.md:30), so the cursor must be compressed/transported differently or this validation limit must accommodate the supported fanout.

AGENTS.md reference: AGENTS.md:L6-L6

Useful? React with 👍 / 👎.

Fixed in 752a3b620. Each Server adds about 180 base64 characters to the cursor, so the 4 KiB cap broke at roughly 20 Servers, and raising it would only hit the URL limit next. History reads now POST, with the cursor in a JSON body capped at 256 KiB. GET serves only the live stream. A new test decodes a 50-Server cursor; it fails at the old cap and passes now.

_{P2 Badge} Reject or support all advertised history filters

LogHistoryOptions.filter is publicly typed as LogFilter, but filters containing only machineId or kind reach this branch with no store selector and silently return an empty page without making any history request. The same fields work for runtime.logs(), so callers cannot distinguish “unsupported filter shape” from genuinely empty retained history; either support these filters or fail validation rather than reporting no logs.

Useful? React with 👍 / 👎.

Fixed in 752a3b620 by supporting them. The live stream already accepts a filter naming only a Server or a kind, so history now reads the same way. The only empty page left is a serviceId that matches no Service. A new SDK test reads one Server's hooks with { machineId, kind }; it fails before the change and passes now.

_{P2 Badge} Clear recovered servers from the live-missing map

When a container emits source_error—for example, because it disappears between the Runtime Watch snapshot and the TailLogs RPC—this permanently adds its Machine to missing.live. Neither a later live event, successful records from a replacement container on that Machine, nor a successful history page removes the entry, so the dashboard continues claiming that the server's logs are missing until the collection is garbage-collected. Clear the Machine's live error when its log source or retained history recovers.

Useful? React with 👍 / 👎.

Fixed in a6ba30270, which is on #1484 because the per-container note lives in that layer. #1484 already clears a container from the note when it sends again. What was left is a container that has left for good. Each Runtime Watch frame is a complete observation. When a failed source's container is missing from a frame, and neither it nor its Machine is listed in incomplete_ids, the SDK now drops the source and yields source_gone. The dashboard clears that container from the note the same way a record does. The note stays while any other container on that Server is still down. Two new tests: an SDK test where the incomplete frame keeps the source and the complete frame yields source_gone, and a stream test where the last failed container leaves. The stream test failed without the fix and passes with it.