Project Fortify, an experimental security project, has identified the following potential security issue. If you’re a feature owner CC-ed on this bug, please do your best to review these reports without the Chrome Security team. Please see go/chrome-ai-generated-security-bugs-faq for more information.

Overview: A potential heap-buffer-overflow WRITE exists in libaom’s scene detection logic when using SVC temporal layers via the WebCodecs API. By dynamically shrinking and then growing the encoder resolution, an attacker can desynchronize libaom’s internal state, causing it to reuse a small buffer for a large frame’s data. This bypasses size checks and leads to an out-of-bounds write of SAD (Sum of Absolute Differences) values in the renderer process.

Affected files:

• third_party/libaom/source/libaom/av1/encoder/ratectrl.c
• third_party/libaom/source/libaom/av1/encoder/svc_layercontext.c
• third_party/libaom/source/libaom/av1/encoder/encoder.c
• media/video/av1_video_encoder.cc

Estimated timestamp from git blame: 2025-05-06

Summary

A potential linear heap-buffer-overflow WRITE exists in libaom’s scene detection logic when using Scalable Video Coding (SVC) temporal layers (e.g., scalabilityMode: 'L1T2') via the WebCodecs API. A specific sequence of resolution reconfigurations (shrink-then-grow) causes the encoder to erroneously reuse a stale, smaller buffer (cpi->src_sad_blk_64x64) for a larger frame’s data, bypassing dimension-mismatch safety checks. This allows an attacker to write partially controlled values beyond the buffer’s bounds in the renderer process.

Root Cause Analysis

The buffer cpi->src_sad_blk_64x64 is used to store block-wise Sum of Absolute Differences (SAD) during one-pass real-time rate control. It is allocated in av1_rc_scene_detection_onepass_rt based on the superblock grid size of the current frame.

To ensure the buffer matches the current frame dimensions, a check compares the current frame (frame_input->source) against the previous reference frame (frame_input->last_source). If they differ, the buffer is safely freed and nulled so it can be reallocated correctly.

// ratectrl.c:3309
if (src_width != last_src_width || src_height != last_src_height) {
  aom_free(cpi->src_sad_blk_64x64);
  cpi->src_sad_blk_64x64 = NULL;
  return;
}

When SVC temporal layers are enabled, if the encoder believes a base layer (TL0) frame was dropped, it redirects frame_input->last_source to cpi->svc.source_last_TL0. This specific fallback buffer is allocated once at the start of encoding (frame_number == 0) and is never resized during subsequent dynamic resolution reconfigurations (aom_codec_enc_config_set).

Potential Trigger Sequence:

1. Initial Configuration (Large): WebCodecs is configured with a large resolution (e.g., 256x512) and scalabilityMode: 'L1T2'. source_last_TL0 is allocated at 256x512.
2. Reconfigure Down (Small): The resolution is reconfigured to a smaller size (e.g., 256x64). The next frame (TL1) encounters a dimension mismatch, causing src_sad_blk_64x64 to be safely freed and reallocated for the smaller size (e.g., 4 elements). Because it’s a TL1 frame, it does not update the time index for the TL0 reference buffer.
3. Reconfigure Up (Large): The resolution is restored to the large size (256x512). The next frame is encoded as a TL0 frame.
4. State Desync: Because the previous TL1 frame did not update the TL0 reference time index, av1_svc_set_last_source incorrectly concludes that the previous base layer frame was dropped. It forces frame_input->last_source to point to the fallback source_last_TL0 buffer.
5. Bypass & Overflow: In av1_rc_scene_detection_onepass_rt, the code compares the current frame (256x512) against source_last_TL0 (which is still 256x512 from step 1). The safety check passes, failing to free the small src_sad_blk_64x64 buffer allocated in step 2. The subsequent loop iterates over the large frame’s grid, writing out of bounds into the small buffer.

Impact

• Type: Heap-buffer-overflow WRITE.
• Process: Renderer process.
• Controllability: The attacker can control the size of the initial small allocation to place the victim buffer in a specific PartitionAlloc bucket. By providing crafted VideoFrame luma data, an attacker controls the low 20 bits of the written uint64_t SAD values, which can be used to corrupt adjacent heap object metadata (e.g., length or capacity fields), potentially leading to arbitrary read/write and remote code execution.
• Note: This is a spatial overflow on a raw C pointer, so it is not mitigated by MiraclePtr.

Suggested Fix

The primary issue is that cpi->svc.source_last_TL0 holds stale dimensions across reconfigurations. The fix should involve either:

1. Reallocating or invalidating cpi->svc.source_last_TL0 inside av1_change_config (or related configuration change handlers) when the frame dimensions change.
2. In av1_rc_scene_detection_onepass_rt, validating the dimensions of cpi->src_sad_blk_64x64 directly against the current frame dimensions instead of relying solely on the comparison between frame_input->source and frame_input->last_source.

Evaluated with Chrome root at commit: c0eb5541aebfa4ea08806eaf6e94bcc69f87ab2f

Results so far have been promising, but there can be wrong deductions. If this proves to be a false positive, please close as WAI; data from false positives will be used to improve accuracy over time. And please feel free to reach out to me directly if you have concerns or feedback on the project.