Skip to content

fix(rendering): read again from a fresh AVAssetReader when one fails instead of answering with a stale frame - #2872

Open
richiemcilroy wants to merge 2 commits into
mainfrom
fix/avf-seek-stale-answers
Open

richiemcilroy wants to merge 2 commits into
mainfrom
fix/avf-seek-stale-answers

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Oct 22, 2026 •

Copy link
Copy Markdown
Member

Why

On macOS, the cap-verify AVF seek cases that fail on loaded Blacksmith runners (seek/seek-gop/scrub30/avf, seek/vfr-holds/into-holds60/avf, seek/seek-gop/home-then-play30/avf, seek/seek-gop/keyframes60/avf) are a real decoder bug, not a harness timing problem. The stale answers come from the decoder itself.

A traced run of main's decoder (scratch run 39709617728, seek suite under parallel nextest on blacksmith-6vcpu-macos-15) shows the native reader failing in the middle of the stream:

screen reader 0 ended: state=Failed Some("-11821 Cannot Decode") iterated=1 prev_vended=Some(106) pending=[112] cache_last=Some(105)
camera reader 0 ended: state=Failed Some("-11821 Cannot Decode") iterated=2 prev_vended=Some(107) pending=[112] cache_last=Some(105)

-11821 is AVErrorDecodeFailed. The status is AVAssetReaderStatusFailed, so the reader never vends again. The decoder did not check the status. It treated the end of FramesIter as "no frames yet", and the fallback after the decode loop answered every waiting request with the newest cached frame at or before it. That is how seek-gop/scrub30/avf got frame 106 for 111 and 105 for 118. The reader was only recreated after DecoderHealth::MAX_CONSECUTIVE_EMPTY (5) empty batches, so playback kept showing a frozen frame for several frames, as in into-holds60 (235 for 236, 237 and 238). When nothing was cached at or before the request, the request went unanswered and the handle switched to FFmpeg ("AVAssetReader returned no frame"), which is the decoder-fallback variant of the same failure.

In the same run, serial rounds (--test-threads 1) never hit -11821. Six CPU burners with serial tests did not hit it either; they only caused 2 s request timeouts. So the trigger is many decode sessions at once on the paravirtualised VideoToolbox, not CPU starvation. In the app the same thing can happen with many segments or readers open, or under GPU or media-server pressure: after a scrub the editor shows an older frame.

What

  • cap-video-decode: AVAssetReaderDecoder::failure() returns the reader's error when its status is Failed, and None when it reached the end normally.
  • crates/rendering/src/decoder/avassetreader.rs: when the decode loop ends with requests still waiting and the reader has failed, those requests are re-queued as retries instead of falling through to the cached-frame fallback. A retry always resets the reader it runs on through the existing retried path, so the requests are read again from a fresh reader. If that reader fails too, the requests are left unanswered, and the handle's existing software fallback serves the right frame instead of a stale one.
  • Scripted media: Registration::fail_avf_read_at(index) makes the next scripted AVAssetReader that reaches a sample fail there, the way a native reader fails on a decode error.

Tests

  • New decoder::model::avassetreader_answers_stay_exact_when_its_reader_fails. It runs playback (frames 0 to 89) and a forward scrub (every 7 frames), with the reader failing once or twice at sample 40. Every answer must be the exact frame. One failure must stay on AVAssetReader; two must end on software decoding. On main's decoder it fails with the stale frame 39 for request 40 (scratch run 39715399800). avassetreader_answers_match_the_reference_model calls it, because that is the only test of the module that Engine verify runs on macOS.
  • The decoder:: tests (63) and the cap-video-decode lib tests pass on macOS.

Measured

Seek suite under parallel nextest on blacksmith-6vcpu-macos-15. Base (main's decoder) and head were interleaved in one job for 5 rounds (run 39715399800), then head ran 5 more rounds with tracing (run 39717985986).

base, 5 rounds head, 10 rounds
rounds with a stale answer 4 0
stale case failures 6 (scrub30 3, home-then-play30 2, into-holds60 1) 0
reader failures (-11821) read again from a fresh reader n/a 22 (the cases passed unless the stall had already run past the request deadline, see below)
decoder-fallback case failures 5 13

The remaining decoder-fallback failures are not stale answers. They happen on base too. The trace shows the native copyNextSampleBuffer blocking past the request deadline (2 s, or 10 s for an initial seek) before it fails with -11821, so the handle switches to FFmpeg as designed and serves the right frames. In the same rounds, base and head also hit FFmpeg decoder timed out after 30s initializing on ffmpeg cases, so this is the runner's VideoToolbox stalling. The higher head count only looks worse: on base, the same case often failed as stale first and was counted there. A run that copied cached frames out of the reader pool instead of holding pool buffers (run 39719242686) still hit -11821 with no pool buffers held. So the trigger is the VM's VideoToolbox, not the decoder's cache.

Coordination with #2868

This branch merges cleanly with #2868 (git merge-tree on both heads). #2868's first version saw the reader "stop vending after about 40 outstanding buffers" in keyframes60/avf. That has the same signature as this failure, and with this change it would be read again instead of answered stale.

Risk

Low. The change only applies when AVFoundation reports Failed, which previously led to wrong frames. A reader failing twice in a row now falls back to software decoding for that track, as a reader that returns nothing already did.

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable issues were found.

Summary

The decoder retries waiting requests from a fresh reader when AVFoundation reports a failure, rather than answering with an older cached frame.

  • Failed video readers retry frame requests before showing an older image.

Reviews (1) · Last reviewed commit: "test(rendering): run the reader failure ..." · Reviewed by Greptile

@vercel

vercel Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
cap-status Ready Ready Preview Oct 22, 2026 3:35am UTC

@blacksmith-sh

This comment has been minimized.

@blacksmith-sh

blacksmith-sh Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Found 2 test failures on Blacksmith runners:

Failures

Test View Logs
cap-verify::crash/crash_safety View Logs
cap-verify::crash/crash_safety View Logs

Fix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@richiemcilroy
richiemcilroy marked this pull request as ready for review October 22, 2026 05:26
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

This branch was successfully deployed

2 active deployments
Preview – cap-web — 55f93111 Deployed Oct 22, 2026 by vercel[bot]
Preview – cap-status — 55f93111 Deployed Oct 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant