Project Fortify, an experimental security project, has identified the following potential security issue. If you’re a feature owner CC-ed on this bug, please do your best to review these reports without the Chrome Security team. Please see https://chromium.googlesource.com/chromium/src/+/main/docs/security/ai-generated-security-bugs-faq.md for more information.

Overview: A desynchronization between Chrome’s AV1VaapiVideoEncoderDelegate and the libaom rate controller can lead to a massive heap out-of-bounds write in the GPU process. When a compromised renderer dynamically increases the temporal layer count, Chrome’s internal SVC tracker updates, but libaom’s memory buffers do not. Subsequent frame encodes pass out-of-bounds temporal indices to libaom, triggering an ~13.6 KB structure copy beyond the allocated bounds.

Affected files:

• media/gpu/vaapi/av1_vaapi_video_encoder_delegate.cc
• media/gpu/svc_layers.cc
• third_party/libaom/source/libaom/av1/ratectrl_rtc.cc
• third_party/libaom/source/libaom/av1/encoder/svc_layercontext.c

Estimated timestamp from git blame: 2024-12-04

Summary

A potential heap buffer overflow exists in the GPU process during AV1 video encoding via VAAPI. The vulnerability is caused by a desynchronization between AV1VaapiVideoEncoderDelegate’s cached temporal layer count and the dynamically updated SVCLayers configuration. A compromised renderer can leverage this to trigger a ~13.6 KB out-of-bounds heap write, which could lead to arbitrary code execution within the GPU process.

Technical Details

1. In AV1VaapiVideoEncoderDelegate::Initialize (media/gpu/vaapi/av1_vaapi_video_encoder_delegate.cc), the initial temporal layer count is cached in num_temporal_layers_.
2. When the renderer requests a bitrate change via RequestEncodingParametersChangeWithLayers, AV1VaapiVideoEncoderDelegate::UpdateRates is called.
3. UpdateRates configures libaom using the stale cached value: rc_config.ts_number_layers = num_temporal_layers_;. As a result, libaom does not resize its internal layer_context array.
4. However, immediately after this, UpdateRates calls svc_layers_->RecreateSVCLayersIfNeeded(current_params_.bitrate_allocation). This component does adapt to the new, higher temporal layer count (e.g., from 2 to 3) requested by the attacker.
5. During subsequent frame encoding in PrepareEncodeJob, svc_layers_->GetPictureParamAndMetadata() returns a temporal_idx based on the new 3-layer structure. For example, on the second frame, it returns temporal_idx = 2.
6. This temporal_idx is passed to rate_ctrl_->ComputeQP(frame_params). Inside libaom’s AV1RateControlRTC::ComputeQP (third_party/libaom/source/libaom/av1/ratectrl_rtc.cc), the layer index is calculated using the macro LAYER_IDS_TO_IDX(spatial_id, temporal_layer_id, number_temporal_layers). Using the out-of-bounds temporal ID but the stale temporal layer count (LAYER_IDS_TO_IDX(0, 2, 2)), it evaluates to 2.
7. av1_save_layer_context (third_party/libaom/source/libaom/av1/encoder/svc_layercontext.c) retrieves a pointer to &cpi_->svc.layer_context[2]. Since the array was only allocated for 2 elements (indices 0 and 1), this is an out-of-bounds pointer.
8. Finally, av1_save_layer_context performs massive full-struct assignments to this pointer: lc->rc = cpi->rc; and lc->p_rc = cpi->ppi->p_rc;. Because PRIMARY_RATE_CONTROL contains large arrays like int q_history[1000] and REGIONS regions[150], this results in an out-of-bounds heap write of approximately 13.6 KB.

Suggested Reproduction Steps

(Note: These are potential steps based on static code analysis, as our tooling agent does not yet have the ability to run code and provide a working proof-of-concept.)

1. From a compromised renderer, establish a media.mojom.VideoEncodeAccelerator connection to the GPU process.
2. Call Initialize to set up AV1 hardware encoding with an initial configuration of 2 temporal layers.
3. Call RequestEncodingParametersChangeWithLayers with a VideoBitrateAllocation containing active bitrates for 3 temporal layers.
4. Send multiple Encode requests. For a 3-layer temporal pattern, the second frame will be assigned temporal_idx = 2.
5. The GPU process will pass this out-of-bounds index to libaom, triggering the massive OOB heap write and likely causing an ASan crash or memory corruption in av1_save_layer_context.

Suggested Fix

In AV1VaapiVideoEncoderDelegate::UpdateRates, validate the new bitrate_allocation. If the requested number of temporal layers differs from num_temporal_layers_, UpdateRates should either:

1. Reject the change and return false, as dynamic layer count changes may not be supported by the current hardware encoder lifecycle.
2. If supported, securely update num_temporal_layers_ to match the new allocation before configuring rc_config.ts_number_layers, ensuring that libaom properly reallocates its internal context arrays.

Evaluated with Chrome root at commit: 7353d249d9cacf9c7218e1d7b8a39cf39c72d646

Results so far have been promising, but there can be wrong deductions. If this proves to be a false positive, please close as WAI; data from false positives will be used to improve accuracy over time. And please feel free to reach out to me directly if you have concerns or feedback on the project.