48e2c97586de122913e75f1855aa4d6747fa1121  Enforce upper limit on spatial_layers in VideoEncodeAcceleratorConfig

The bug at a glance

A renderer-supplied field, spatial_layers, crossed the Mojo boundary into the GPU process with no upper bound. Downstream encoder backends index fixed-size structures (notably encoder_info_.fps_allocation, a fixed-capacity array sized for kMaxSpatialLayers) using config.spatial_layers.size(), so an oversized vector would drive an out-of-bounds write in the GPU process. That is a real, attacker-influenced primitive from a compromised or malicious renderer, which is why it rates medium (CVSS ~5.5) rather than pure noise. It is held below patch-now tier because: the pre-existing DCHECK/CHECK(!empty) guards and per-backend loops meant several backends would already CHECK or behave defensively in some builds; the write is bounded/adjacent rather than a freely-sized heap overflow; and the fix’s own framing (reject at deserializer + CHECK_LE defense-in-depth) shows the intended outcome is a controlled crash, not silent corruption. Exploitation would require a already-compromised renderer and heap grooming of the GPU process; no exploit is demonstrated.

The trust boundary is renderer -> GPU process via the VideoEncodeAcceleratorConfig Mojo struct. The renderer proposes an encoder configuration including a spatial_layers list. Every hardware encoder backend (VT/mac, VA-API, V4L2, NDK/android, MediaFoundation/win) loops ‘for i in [0, spatial_layers.size())’ and writes encoder_info_.fps_allocation[i], where fps_allocation is sized for VideoEncoderInfo::kMaxSpatialLayers. Nothing validated the incoming size against that bound: the Mojo deserializer traits accepted whatever count the renderer sent. A malicious renderer could therefore send more than kMaxSpatialLayers entries and drive the backend loop past the end of the fixed fps_allocation array — an out-of-bounds write in the GPU process.

Root cause

The primary fix is in the Mojo StructTraits Read for VideoEncodeAcceleratorConfig: after ReadSpatialLayers populates the vector, it now checks ‘if (spatial_layers.size() > media::VideoEncoderInfo::kMaxSpatialLayers) return false;’, which fails deserialization and (per Mojo semantics) reports a bad message / kills the offending renderer connection before the config ever reaches an encoder. This is the authoritative boundary check. On top of that, each of the five backends gains a ‘CHECK_LE(config.spatial_layers.size(), VideoEncoderInfo::kMaxSpatialLayers);’ immediately after the existing non-empty guard and before the indexing loop — defense-in-depth so that any alternate construction path that bypasses the Mojo traits still traps deterministically instead of writing out of bounds. The change is purely additive validation; no legitimate configuration exceeds kMaxSpatialLayers, so valid clients are unaffected.

Key code

Reject oversized spatial_layers at the Mojo deserializer, plus CHECK_LE guards before each backend’s fps_allocation indexing loop.

// media/mojo/mojom/video_encode_accelerator_mojom_traits.cc (authoritative boundary)
   if (!input.ReadSpatialLayers(&spatial_layers))
     return false;
+  if (spatial_layers.size() > media::VideoEncoderInfo::kMaxSpatialLayers) {
+    return false;
+  }

// e.g. media/gpu/mac/vt_video_encode_accelerator_mac.mm (defense-in-depth)
   if (config.HasSpatialLayer() || config.HasTemporalLayer()) {
     CHECK(!config.spatial_layers.empty());
+    CHECK_LE(config.spatial_layers.size(), VideoEncoderInfo::kMaxSpatialLayers);
     for (size_t i = 0; i < config.spatial_layers.size(); ++i) {
       encoder_info_.fps_allocation[i] = ...

Patch walkthrough

• media/mojo/mojom/video_encode_accelerator_mojom_traits.cc — The real fix. In the deserializer traits Read(), right after ReadSpatialLayers succeeds, it rejects any config whose spatial_layers count exceeds VideoEncoderInfo::kMaxSpatialLayers by returning false. Returning false from a StructTraits Read is treated by Mojo as a validation failure, tearing down the message (and typically the renderer’s IPC), so the oversized config never reaches the GPU-process encoder code where the OOB index would occur.
• media/gpu/mac/vt_video_encode_accelerator_mac.mm — Adds CHECK_LE(config.spatial_layers.size(), VideoEncoderInfo::kMaxSpatialLayers) after the existing CHECK(!empty) and before the fps_allocation indexing loop, converting any residual oversized config into a controlled crash rather than an OOB write.
• media/gpu/vaapi/vaapi_video_encode_accelerator.cc — Same CHECK_LE guard inserted after the DCHECK(!empty), before the loop that writes encoder_info_.fps_allocation[i]. Defense-in-depth for the Linux VA-API backend.
• media/gpu/v4l2/v4l2_video_encode_accelerator.cc — Same CHECK_LE guard in the temporal-SVC-only path before the fps_allocation loop, for the V4L2 (embedded/ChromeOS) backend.
• media/gpu/android/ndk_video_encode_accelerator.cc — Same CHECK_LE guard after DCHECK(!empty), before the fps_allocation loop, for the Android NDK backend.
• media/gpu/windows/media_foundation_video_encode_accelerator_win.cc — Same CHECK_LE guard before the fps_allocation loop, for the Windows Media Foundation backend.
• media/mojo/mojom/video_encode_accelerator_mojom_traits_unittest.cc — Adds TooManySpatialLayers_Rejected: builds a Config with kMaxSpatialLayers+1 identical spatial layers and asserts SerializeAndDeserialize returns false, locking in the boundary rejection.

Background

VideoEncodeAcceleratorConfig — Mojo-serialized encoder configuration sent from the renderer (WebCodecs / WebRTC hardware encode path) to the GPU process. Includes profile, bitrate, and a spatial_layers list describing SVC layering.

kMaxSpatialLayers — VideoEncoderInfo::kMaxSpatialLayers — the fixed maximum number of spatial layers the encoder info structures (notably fps_allocation) are sized for. The renderer-supplied count must not exceed it.

fps_allocation — A fixed-capacity array in encoder_info_ indexed by spatial layer. Backends write fps_allocation[i] for each spatial layer, so an unbounded layer count indexes past the array.

Mojo StructTraits Read validation — Returning false from a StructTraits Read() marks the incoming message invalid; Mojo drops it and reports a bad message, the standard mechanism for enforcing invariants on IPC-supplied data before it reaches privileged code.

Bug 561672125 — Fixed: tag with reviewers Tom Sepez (IPC-security specialist) and Eugene Zemtsov (media), consistent with an IPC-boundary input-validation hardening finding.

Vulnerability window

1. root cause — spatial_layers count crossed the Mojo boundary unbounded; backends indexed fixed arrays by that count.
2. 2026-09-16 fix — Dale Curtis adds the deserializer rejection plus five backend CHECK_LE guards and a traits unittest; reviewed by Tom Sepez and Eugene Zemtsov, landed main@{#1699116}.
3. post-landing — No exploit demonstrated; treated as IPC input-validation hardening, medium band.

Proof of concept

The CL ships a serialization unittest (not an exploit) that proves the boundary rejection. It constructs a VideoEncodeAccelerator::Config with kMaxSpatialLayers + 1 spatial layers and asserts that SerializeAndDeserialize returns false, i.e. the oversized config is rejected at the Mojo traits layer. This demonstrates the fix, not the vulnerability. To exercise the underlying flaw (no exploit provided): a compromised renderer would send a VideoEncodeAcceleratorConfig with more than kMaxSpatialLayers entries; on pre-patch builds this deserializes successfully and a GPU backend loop writes encoder_info_.fps_allocation[i] past the array bound. No memory-corruption exploit or grooming is demonstrated, and reliable GPU-process exploitation is not established here.

// From video_encode_accelerator_mojom_traits_unittest.cc (shipped test)
TEST(VideoEncodeAcceleratorConfigStructTraitTest,
     TooManySpatialLayers_Rejected) {
  // ... build input_config (VP9 profile0, NV12, 320x180) ...
  ::media::VideoEncodeAccelerator::Config::SpatialLayer spatial_layer;
  // ... fill spatial_layer fields ...
  input_config.spatial_layers.assign(
      ::media::VideoEncoderInfo::kMaxSpatialLayers + 1, spatial_layer);

  ::media::VideoEncodeAccelerator::Config output_config{};
  EXPECT_FALSE(
      mojo::test::SerializeAndDeserialize<mojom::VideoEncodeAcceleratorConfig>(
          input_config, output_config));
}

Exploitation

1. precondition — Attacker controls a renderer (e.g. via a prior renderer RCE) able to craft raw VideoEncodeAcceleratorConfig Mojo messages to the GPU process; hardware video encode must be reachable.
2. trigger — Send a config with spatial_layers.size() > kMaxSpatialLayers. Pre-patch it deserializes and a backend loop indexes fps_allocation out of bounds.
3. primitive — Bounded out-of-bounds write in the GPU process adjacent to encoder_info_.fps_allocation, size proportional to the excess layer count.
4. escalation — Would require GPU-process heap grooming to place a useful target after the array; not demonstrated. Post-patch the message is rejected at the boundary (and CHECK_LE crashes any residual path).

Detection & hunting

For defenders and detection engineers:

• Mojo bad-message reports for VideoEncodeAcceleratorConfig — Post-patch, deserialization failures / renderer kills on this struct indicate a renderer sending oversized spatial_layers — a strong compromise indicator.
• GPU-process CHECK_LE crashes in VEA backends — CHECK_LE hits on spatial_layers.size() in vt/vaapi/v4l2/ndk/mediafoundation backends flag an oversized config that bypassed the traits layer.
• GPU-process crashes near fps_allocation — On unpatched builds, OOB-write crashes in encoder_info_ indexing during encoder init are the runtime symptom.

Audit directions

• Other renderer->GPU Mojo config fields — Audit VideoEncodeAcceleratorConfig and sibling media structs for other count/size fields (temporal layers, plane counts) that index fixed-size arrays without a boundary check in the traits Read().
• fps_allocation and fixed-size encoder arrays — Enumerate every write to fixed-capacity arrays indexed by an IPC-supplied count across media/gpu backends; confirm each is bounded before the loop, not only by DCHECK.
• StructTraits without size caps — Grep media/mojo/mojom traits for ReadX(&vec) calls that are not followed by a size() upper-bound check; unbounded container reads from a renderer are a recurring pattern.