Sign up
View stargazers13Sign up

feat(server): report image dimensions with signed asset URLs#10198

merged4
main
media-dims/contract
openedSep 5, 2026byJJulius Marminge
8 files changed
View on GitHubTwo-way sync enabledPR last imported 4d ago

Stack 1/3 (#10201). Follow-ups: #10199 (mobile), #10200 (web).

Problem

Both clients reserve a 16:9 slot for a chat image while its bytes load, then resize to the natural size once decoded. That last resize moves every row below the image, and on mobile it can push the end of the thread behind the composer or leave the next tool call drawn under the image (see #10199). The clients can't do better without knowing the size up front.

Fix

The server already opens the file when it issues a signed asset URL, so it now reads the pixel size from the PNG, JPEG, GIF, or WebP header and returns it as an optional imageDimensions on AssetCreateUrlResult. packages/client-runtime threads it onto AssetUrlState.Success so both clients see it.

  • packages/shared/src/imageDimensions.ts — dependency-free header parser. JPEG honours the EXIF orientation tag (5–8 swap the axes), since every viewer applies it before display and phone photos are stored on their side.
  • For media-file resources the header is read through the same identity-checked handle used for the inode check, so there's no second open to swap. Only formats the parser understands are read (SVG, AVIF, ICO skip).
  • Best effort: unreadable or unsupported files omit the field and the client falls back to measuring after decode. Older clients ignore the field; older servers omit it.

Compatibility

Additive optional field. Schema.optional on the contract; unknown keys are ignored by older clients.

Verification

  • packages/shared: imageDimensions.test.ts — PNG, GIF, JPEG (with APP1 prefix, Huffman-table marker skip, EXIF orientation 6 vs 1), all three WebP flavours, truncated/zero-size/unsupported input. 7 passed.
  • apps/server: AssetAccess.test.ts — a real PNG header reports 1600×900, an mp4 and a malformed .png omit the field. 27 passed.
  • Typecheck clean for packages/shared, packages/contracts, packages/client-runtime, apps/server.

No visual change on its own; screenshots are on the client PRs.

Model: Claude Fable 5 · Harness: Claude Code in T3 Code

🤖 Generated with Claude Code

Note

Report image dimensions with signed asset URLs

  • Adds a shared image-header parser in imageDimensions.ts that extracts pixel width and height from PNG, GIF, WebP, and JPEG headers, returning null for unsupported or malformed input. JPEG parsing also applies EXIF orientation axis swaps for values 5-8.
  • Integrates parsing into AssetAccess.ts so issueAssetUrl reads a bounded header prefix from already-open media handles for supported extensions (png, jpg, jpeg, gif, webp) and includes imageDimensions in the result when parsing succeeds.
  • Updates assets.ts contracts and assets.ts client state to carry the optional imageDimensions field through successful asset URL results.
  • Risk: dimension extraction is best-effort and never blocks URL issuance, but callers reading AssetCreateUrlResult or AssetUrlState.Success should treat the new optional imageDimensions field as potentially absent.

Macroscope summarized 0c0d4ea.

Added 2 commits

  • feat(server): report image dimensions with signed asset URLsba55b8e
  • fix(server): honour EXIF orientation and read the header from the open handlec97ccee
Ggithub-actionsBotSep 5, 2026
Synced from GitHub

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ProviderMetricMain baselineThis PRImpactPR ceiling
CodexTotal thread wire13.5 KiB13.6 KiB+27 B (+0.2%)15.1 KiB✅
CodexThread snapshot wire7.0 KiB7.0 KiB−2 B (−0.0%)7.3 KiB✅
CodexLive turn WebSocket wire6.5 KiB6.5 KiB+29 B (+0.4%)7.8 KiB✅
CodexLive turn WebSocket decoded57.0 KiB57.0 KiB0 B (0.0%)66.4 KiB✅
CodexLive turn messages880 (0.0%)21✅
ClaudeTotal thread wire13.5 KiB13.6 KiB+23 B (+0.2%)15.1 KiB✅
ClaudeThread snapshot wire7.0 KiB7.0 KiB−3 B (−0.0%)7.3 KiB✅
ClaudeLive turn WebSocket wire6.5 KiB6.5 KiB+26 B (+0.4%)7.8 KiB✅
ClaudeLive turn WebSocket decoded57.8 KiB57.8 KiB+44 B (+0.1%)66.4 KiB✅
ClaudeLive turn messages89+1 (+12.5%)21✅

Baseline: f8b4c46 · PR result: 0c0d4ea · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 113.8 KiB
  • Claude decoded thread snapshot: 114.5 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

apps/server/src/assets/MediaFile.tsline 106Outdated · c97cceeOpen thread
……@@ -95,6 +95,17 @@ export const openMediaFile = Effect.fn("openMediaFile")(function* (
103+ const { bytesRead } = await file.handle.read(buffer, 0, byteCount, 0);
104+ return buffer.subarray(0, bytesRead);
105+ },
106+ catch: (cause) => new MediaFileStatError({ path: "<open handle>", cause }),
MmacroscopeappBoton line 106Sep 5, 2026

Service failures should carry a real resource identifier in their attributes; path: "<open handle>" makes the derived message (Failed to read metadata for media file '<open handle>'.) non-identifying, and it also reuses the stat error tag for a read. Consider threading the file path through (as statMediaFile already does) and, if the read should be distinguishable, giving it its own tagged error:

1export const readMediaFileHeader = (filePath: string, file: OpenMediaFile, byteCount: number) => ...
2 catch: (cause) => new MediaFileReadError({ path: filePath, cause }),

The single caller in AssetAccess.issueAssetUrl already has canonicalFile available to pass in.

Posted via Macroscope — Effect Service Conventions

JJulius MarmingeSep 5, 2026

Fixed in 104ad78637. readMediaFileHeader takes the path and fails with its own MediaFileReadError naming the file.

packages/shared/src/imageDimensions.tslines 110–111Outdated · c97cceeOpen thread
……@@ -0,0 +1,143 @@
108+ return rotated ? { width: height, height: width } : { width, height };
109+ }
110+ if (marker === 0xd9 || marker === 0xda) return null;
111+ const length = data.getUint16(offset + 2);
MmacroscopeappBoton lines 110–111Sep 5, 2026

🟡 Medium src/imageDimensions.ts:110

readJpeg returns null for valid JPEG streams containing a TEM marker before SOF, so imageDimensions is omitted and the layout box cannot be reserved. Because TEM (0xff01) is standalone, handle it by advancing two bytes without reading a length field.

diff
1 if (marker === 0xd9 || marker === 0xda) return null;
2+ if (marker === 0x01) {
3+ offset += 2;
4+ continue;
5+ }
6 const length = data.getUint16(offset + 2);
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
text
1In file @packages/shared/src/imageDimensions.ts around lines 110-111:
2
3`readJpeg` returns `null` for valid JPEG streams containing a `TEM` marker before `SOF`, so `imageDimensions` is omitted and the layout box cannot be reserved. Because `TEM` (`0xff01`) is standalone, handle it by advancing two bytes without reading a length field.
JJulius MarmingeSep 5, 2026

Fixed in 104ad78637. TEM (0xff01) and the restart markers (0xffd0–0xffd7) are stepped over as two bytes with no length field; test added.

apps/server/src/assets/AssetAccess.tsline 398Outdated · c97cceeOpen thread
……@@ -347,6 +394,9 @@ export const issueAssetUrl = Effect.fn("AssetAccess.issueAssetUrl")(function* (i
348395 ),
349396 );
397+ if (HEADER_IMAGE_EXTENSIONS.has(path.extname(resolved.relativePath).toLowerCase())) {
398+ imageDimensions = yield* readImageDimensionsFromHeader(canonicalFile);
MmacroscopeappBoton line 398Sep 5, 2026

🟠 High assets/AssetAccess.ts:398

assetsCreateUrl can hang when the workspace image is replaced by a FIFO: readImageDimensionsFromHeader reopens it with blocking flag "r", waiting indefinitely for a writer and potentially exhausting the filesystem worker pool. Read the header through openMediaFile instead so the existing O_NONBLOCK protection is preserved.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
text
1In file @apps/server/src/assets/AssetAccess.ts around line 398:
2
3`assetsCreateUrl` can hang when the workspace image is replaced by a FIFO: `readImageDimensionsFromHeader` reopens it with blocking flag `"r"`, waiting indefinitely for a writer and potentially exhausting the filesystem worker pool. Read the header through `openMediaFile` instead so the existing `O_NONBLOCK` protection is preserved.
JJulius MarmingeSep 5, 2026

Fixed in 104ad78637. Every header read now goes through openMediaFile (O_NONBLOCK | O_NOFOLLOW plus the lstat/inode checks), including the workspace-file and attachment paths that previously used a plain FileSystem.open. A FIFO returns null from openMediaFile's isFile() check before any read.

MmacroscopeappBotSep 5, 2026
Synced from GitHub

Approvability

Verdict: Not approved

Macroscope's review found this PR not approvable — This PR adds a new cross-package image-header parser and changes existing asset URL issuance to perform bounded file reads and return optional dimensions. Although the API addition is backward-compatible, the new production-path logic and user-facing capability warrant human review.

You can add or adjust custom eligibility rules. Learn more.

Added 1 commit

  • fix(server): open image headers non-blocking and step over standalone JPEG markers104ad78
CcursorBotSep 5, 2026
Synced from GitHub

Bugbot is paused — on-demand spend limit reached

Bugbot uses usage-based billing for this team and has hit its on-demand spend limit.

A team admin can raise the spend limit in the Cursor dashboard, or wait for the next billing cycle to continue.

packages/shared/src/imageDimensions.tslines 119–119Outdated · 104ad78Open thread
……@@ -0,0 +1,148 @@
116+ const length = data.getUint16(offset + 2);
117+ // Viewers apply the EXIF orientation before display, so a phone photo
118+ // stored on its side takes the swapped size on screen.
119+ if (marker === 0xe1) rotated = exifOrientationSwapsAxes(bytes, offset + 4, offset + 2 + length);
MmacroscopeappBoton lines 119–119Sep 5, 2026

🟡 Medium src/imageDimensions.ts:119

A JPEG with EXIF orientation 5–8 followed by an XMP APP1 segment returns unswapped SOF dimensions, so clients reserve the wrong aspect ratio for the displayed image. Each APP1 assigns rotated from the current segment, allowing the later non-EXIF segment to clear the earlier rotation; preserve an already-detected rotation when processing subsequent segments.

suggestion
1 if (marker === 0xe1) rotated = rotated || exifOrientationSwapsAxes(bytes, offset + 4, offset + 2 + length);
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
text
1In file @packages/shared/src/imageDimensions.ts around line 119:
2
3A JPEG with EXIF orientation 5–8 followed by an XMP `APP1` segment returns unswapped SOF dimensions, so clients reserve the wrong aspect ratio for the displayed image. Each `APP1` assigns `rotated` from the current segment, allowing the later non-EXIF segment to clear the earlier rotation; preserve an already-detected rotation when processing subsequent segments.
JJulius MarmingeSep 5, 2026

Fixed in 0c0d4ea243. A rotation found in an earlier APP1 is kept when a later non-EXIF APP1 (XMP) follows; test added for EXIF-then-XMP.

packages/shared/src/imageDimensions.tsline 13Outdated · 104ad78Open thread
……@@ -0,0 +1,148 @@
10+}
11+
12+/** Enough for every supported header, including JPEG files with an EXIF/ICC prefix. */
13+export const IMAGE_DIMENSIONS_HEADER_BYTES = 64 * 1024;
MmacroscopeappBoton line 13Sep 5, 2026

🟡 Medium src/imageDimensions.ts:13

A valid JPEG with enough APP/ICC/XMP metadata before its SOF marker returns null, so callers omit imageDimensions even though JPEG is supported. IMAGE_DIMENSIONS_HEADER_BYTES limits the supplied header to 64 KiB, but JPEG metadata may span multiple maximum-sized APP segments and place the frame header beyond that boundary; increase or remove this fixed cap, or continue fetching bytes until readJpeg reaches SOF/knows the header is incomplete.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
text
1In file @packages/shared/src/imageDimensions.ts around line 13:
2
3A valid JPEG with enough APP/ICC/XMP metadata before its SOF marker returns `null`, so callers omit `imageDimensions` even though JPEG is supported. `IMAGE_DIMENSIONS_HEADER_BYTES` limits the supplied header to 64 KiB, but JPEG metadata may span multiple maximum-sized APP segments and place the frame header beyond that boundary; increase or remove this fixed cap, or continue fetching bytes until `readJpeg` reaches SOF/knows the header is incomplete.
JJulius MarmingeSep 5, 2026

Fixed in 0c0d4ea243. Header window is now 256 KiB, enough for a frame header behind EXIF + ICC + XMP segments. The read is bounded by the file's actual size (the handle read returns the bytes available), so small files cost nothing extra.

Added 1 commit

  • fix(shared): keep a detected JPEG rotation and read past larger metadata0c0d4ea