Project Fortify, an experimental security project, has identified the following potential security issue. If you’re a feature owner CC-ed on this bug, please do your best to review these reports. Please see https://chromium.googlesource.com/chromium/src/+/main/docs/security/ai-generated-security-bugs-faq.md for more information.

Overview: The macOS VideoToolbox AV1 accelerator fails to validate the size of generated av1C atoms and omits bit depth/max frame dimensions from its format change-detection logic. An attacker can feed malicious streams to trigger a 0-byte configuration record or reuse a stale 8-bit session for 10-bit decoding. This can result in potential driver-side out-of-bounds memory corruption within the sandboxed GPU process.

Affected files:

• media/gpu/mac/video_toolbox_av1_accelerator.cc
• media/gpu/mac/vt_config_util.mm

Estimated timestamp from git blame: 2023-11-14

1. Summary of the Issue (Meant for Human Triage)

A security logic vulnerability exists in the macOS AV1 hardware decoder path (VideoToolboxAV1Accelerator::ProcessFormat). An attacker-controlled server can deliver a maliciously crafted AV1 bitstream to force Apple’s VideoToolbox graphics driver into processing out-of-spec or inconsistent stream structures, potentially causing out-of-bounds memory operations within the sandboxed GPU process.

The vulnerability stems from two independent flaws in the format regeneration logic:

1. Unchecked Codec Configuration Box (av1C) Retrieval: Format regeneration does not validate the output of libgav1::ObuParser::GetAV1CodecConfigurationBox(). If format regeneration is triggered on a frame lacking an in-band Sequence Header, libgav1 returns a nullptr with a size of 0. The accelerator blindly wraps this in a span, creating a 0-byte av1C extension atom that is passed to Apple CoreMedia’s VTDecompressionSessionCreate. This violates the AV1-ISOBMFF specification (requiring >= 4 bytes) and hands a 0-byte bounds-bearing configuration structure to the driver.
2. Insufficient Format Change-Detection Keys: The accelerator tracks format changes by checking only {color_space, profile, per-frame coded_size}. It completely omits the stream’s bit depth and sequence-level maximum frame dimensions (max_frame_width, max_frame_height). If a stream transitions from 8-bit to 10-bit, or drastically increases its maximum sequence frame size while holding the per-frame size constant, the format regeneration block is skipped. A stale 8-bit configuration is then used to decode 10-bit inputs via VTDecompressionSessionDecodeFrame, forcing the driver to decode into undersized/incorrectly formatted memory surfaces.

Note: These steps represent potential exploitation paths based on code analysis; our tooling agent does not run or execute code to empirically demonstrate the Apple VideoToolbox internal driver panic.

2. Proof-of-Concept & Detailed Execution Flow

The following outlines the potential steps an attacker could take using a webpage with a <video> tag or WebCodecs configured for hardware AV1 decode on macOS (M3+ Apple Silicon).

Case A: 0-byte av1C Driver Submission via Reference-Frame Scaling

1. Temporal Unit 1 (TU1): The attacker serves a chunk containing [OBU_SEQUENCE_HEADER max_frame=4096x4096] and [OBU_FRAME key_frame width=2048 height=2048].
   • AV1Decoder::SetStream() copies this into temporal_unit_data_.
   • ProcessFormat executes, branches into format regeneration, and extracts a valid av1C via libgav1::ObuParser::GetAV1CodecConfigurationBox (media/gpu/mac/video_toolbox_av1_accelerator.cc:204).
   • active_coded_size_ is saved as {2048, 2048}. A valid VTDecompressionSession is created.
2. Temporal Unit 2 (TU2): The attacker serves a chunk containing no Sequence Header OBU, just an [OBU_FRAME inter_frame frame_size_override_flag=1 width=1024 height=1024].
   • AV1Decoder::DecodeInternal processes this without returning kConfigChange.
   • ProcessFormat computes the new per-frame coded_size as {1024, 1024}.
   • At video_toolbox_av1_accelerator.cc:197, coded_size != active_coded_size_, so the regeneration branch is entered. active_format_.reset() executes.
   • libgav1::ObuParser::GetAV1CodecConfigurationBox() runs on the TU2 bytes. Finding no sequence header, libgav1 returns kStatusBitstreamError and outputs nullptr and *av1c_size = 0 (third_party/libgav1/src/src/obu_parser.cc:3017-3019).
   • At video_toolbox_av1_accelerator.cc:206, av1c_span is constructed with {nullptr, 0}. No null-check or size boundary validation is performed.
   • CreateFormatExtensions (vt_config_util.mm:241) receives the span. [NSData dataWithBytes:nullptr length:0] creates a 0-byte NSData object for the av1C extension atom.
   • CMVideoFormatDescriptionCreate successfully builds a malformed format description, which is passed to VTDecompressionSessionCreate in Apple’s driver, risking an OOB memory fetch during parser initialization.

Case B: Stale av1C Reuse via Missing Parameter Keys

1. Temporal Unit 1 (TU1): The attacker serves [OBU_SEQUENCE_HEADER bitdepth=8, max_frame=128x128] + [OBU_FRAME key_frame 128x128].
   • ProcessFormat initializes a format description for 8-bit AV1 video. session_metadata_.bit_depth is set to 8.
   • A VideoToolbox session configured with an 8-bit output pixel format is initialized.
2. Temporal Unit 2 (TU2): The attacker serves [OBU_SEQUENCE_HEADER bitdepth=10, max_frame=8192x8192] + [OBU_FRAME frame_size_override_flag=1 width=128 height=128].
   • AV1Decoder::DecodeInternal parses the new sequence header and registers the bitdepth change, safely returning AcceleratedVideoDecoder::kConfigChange (media/gpu/av1_decoder.cc:385).
   • VideoToolboxVideoDecoder::Decode receives kConfigChange, issues a continue, and resumes without resetting the accelerator.
   • ProcessFormat is invoked. It checks if the parameters changed (video_toolbox_av1_accelerator.cc:197):

     if (color_space != active_color_space_ || profile != active_profile_ || coded_size != active_coded_size_)
     

   • color_space is identical (bitdepth is not mapped into VideoColorSpace). profile is identical. coded_size (the 128x128 per-frame size) is identical.
   • Because bitdepth and max_frame are structurally omitted from this condition, the block is skipped. active_format_ is not reset.
   • VideoToolboxDecompressionSessionManager::Process compares the CMFormatDescriptionRef pointer (format != active_format_.get()). Since they match, it bypasses format evaluation.
   • The macOS VideoToolbox HW-accelerated driver (VTDecompressionSessionDecodeFrame) begins decoding a 10-bit frame with an 8192x8192 sequence maximum into an 8-bit driver-allocated context, forcing an out-of-bounds driver memory write.

Suggested Fix

1. Case A: Add a boundary check immediately after GetAV1CodecConfigurationBox() in ProcessFormat to ensure av1c is not null and av1c_size >= 4. If the stream lacks a sequence header (causing nullptr), the decoder should either fetch the cached header or cleanly abort decoding.
2. Case B: Expand the change-detection validation in ProcessFormat to evaluate sequence_header.color_config.bitdepth != session_metadata_.bit_depth and include a comparison against the maximum sequence dimensions.

3. Technical Verification Details (Automated Audit Logs - Reviewers may skip this section)

> Verbatim Critic Verdict: The vulnerability report is accurate and verified in the codebase. In media/gpu/mac/video_toolbox_av1_accelerator.cc, ProcessFormat fails to validate the return value of libgav1::ObuParser::GetAV1CodecConfigurationBox(). If a frame chunk lacking a sequence header triggers a format regeneration (e.g., due to a per-frame resolution change), it returns nullptr and size 0. This directly leads to a 0-byte av1C atom being passed to VTDecompressionSessionCreate, violating the AV1-ISOBMFF §2.3 spec (which requires >= 4 bytes). Furthermore, the change-detection key at line 197 omits bitdepth and max_frame, meaning an in-band sequence header change from 8-bit to 10-bit will not trigger a format regeneration, feeding a 10-bit stream to an 8-bit VTDecompressionSessionDecodeFrame. > > Severity Justification: High (S1). > 1. Ceiling (S1): The vulnerability exists in media/gpu/mac/, which is macOS-specific and compile-time excluded on Android. Therefore, the GPU process is sandboxed on the target platform. Memory corruption in a sandboxed GPU process reachable from web content is capped at High (S1); the Android-unsandboxed Critical (S0) rationale does not apply. > 2. Modifier (S2 -> S1): The original report conservatively rated this as Medium (S2) because the closed-source nature of Apple’s VideoToolbox prevented empirical demonstration of an OOB write. However, per the explicit instruction: ‘Bitstream field passed unvalidated to a HW-decode driver struct… Do NOT reject for no ASAN report — the corruption is driver-side. PoC = the bad value at the delegate copy step.’ Because the spec-invalid values observably reach the driver call (VTDecompressionSessionCreate and VTDecompressionSessionDecodeFrame), we assess it at the full S1 severity.

Codebase Investigator Verifications:

1. Data Passthrough Validation: We checked media/gpu/av1_decoder.cc:627-646. In AV1Decoder::DecodeAndOutputPicture, *decoder_buffer_ is passed as the final argument (data) to accelerator_->SubmitDecode(). The parameter is scoped_refptr<media::DecoderBuffer>, which resolves via std::ranges::contiguous_range mapping directly to base::span<const uint8_t> data. The bytes passed are identical to the chunk delivered by the renderer.
2. kConfigChange Handling: We verified media/gpu/mac/video_toolbox_video_decoder.cc:260-300. VideoToolboxVideoDecoder::Decode safely identifies AcceleratedVideoDecoder::kConfigChange (line 273), and subsequently triggers a continue; (line 274). This loops without resetting or altering accelerator_, proving the premise of Case B.
3. Manager Pointer Comparison: We analyzed VideoToolboxDecompressionSessionManager::Process (media/gpu/mac/video_toolbox_decompression_session_manager.mm:154-170). The logic explicitly gates decompression_session_->CanAcceptFormat(format) behind the pointer comparison format != active_format_.get(). If the format description reference matches, validation is bypassed.

Environmental & Feature Checks:

• The feature requires VTIsHardwareDecodeSupported(kCMVideoCodecType_AV1), natively available on Apple Silicon M3+ / A17 Pro+ hardware.
• The vulnerability triggers purely through A-SERVER logic via the <video> or WebCodecs APIs. No user interaction or prior renderer compromise is required.

Evaluated with Chrome root at commit: b96d2ec58f4f5f92b540a723966b199d6e9951b4

Results so far have been promising, but there can be wrong deductions. Feel free to adjust as follows:

• If you are familiar with the severity guidelines, you may adjust the severity.
• If this is a false positive, and there’s no work to be done, please close as WAI.
• If there is work to do here but not a vulnerability, please change the issue type to Task/Bug/FR.

Data from false positives will be used to improve accuracy over time. And please feel free to reach out to me directly if you have concerns or feedback on the project.