3758417301  Use crop dimensions for extension calculation

The bug at a glance

This is a heap buffer overflow in the bundled libaom AV1 encoder’s frame-extension routine av1_copy_and_extend_frame, reachable through WebRTC/media encode paths that feed images into the AV1 encoder. When the caller requests a size_align larger than the display dimensions the allocated buffer is sized to the crop (display) dimensions while the extension math used the padded dimensions, so the copy/extend writes past the allocation. Severity is high because it is an attacker-influenceable memory-corruption bug in a media codec, but exploitability is capped by the constrained overflow shape (border/alignment-driven, not arbitrary length) and by the fact that reaching it requires driving the AV1 encoder with a specific size_align > display-size configuration.

AV1 encoding is reachable from web content via WebRTC (e.g. a getUserMedia or canvas-captured stream negotiated to the AV1 codec) and other media-encode surfaces that funnel frames through libaom. The encoder pads and aligns incoming frames, and this routine copies the source image into a bordered destination buffer; a mismatch between the dimensions used to size the buffer and those used to compute the extension turns an ordinary encode of a mis-sized image into an out-of-bounds heap write.

Root cause

The broken invariant is that the number of extension pixels written on each side must be derived from the same dimensions the destination buffer was allocated against — the crop (display) dimensions y_crop_width / y_crop_height. The buffer for the extended frame is sized from the crop dimensions, but the right-edge and bottom-edge extension widths (er_y, eb_y) were computed from the padded/aligned dimensions y_width / y_height.

Before the patch, er_y = AOMMAX(src->y_width + dst->border, ALIGN_POWER_OF_TWO(src->y_width, 6)) - src->y_crop_width and eb_y = AOMMAX(src->y_height + dst->border, ALIGN_POWER_OF_TWO(src->y_height, 6)) - src->y_crop_height. When size_align forces y_width/y_height to exceed the display size (the test uses size_align=32 so w,h become 32 while d_w,d_h stay 16), the AOMMAX terms are computed against the larger padded width/height, producing a larger right/bottom extension than the crop-sized buffer can hold. The subtraction still uses y_crop_width/y_crop_height, so er_y/eb_y overshoot by roughly (y_width - y_crop_width) plus alignment slack, and the extend loop writes those extra columns/rows past the end of the destination allocation — a heap-buffer-overflow.

The fix recomputes both extents entirely in crop-dimension space: er_y = AOMMAX(src->y_crop_width + dst->border, ALIGN_POWER_OF_TWO(src->y_crop_width, 6)) - src->y_crop_width and eb_y = AOMMAX(src->y_crop_height + dst->border, ALIGN_POWER_OF_TWO(src->y_crop_height, 6)) - src->y_crop_height. Now the padding/alignment target and the subtracted base are the same crop dimensions, so the extension exactly fills the crop-sized-plus-border buffer and never exceeds it, regardless of how large size_align inflates y_width/y_height. This is a cherry-pick of aom commit 7343efd164afc3c0f9f2a212052d77a3d7ea1a49.

Key code

The overflow: extension extents padded from y_width/y_height (padded dims) but subtracted against y_crop_width/height (buffer dims). Fix uses crop dims throughout (extend.c).

-  const int er_y =
-      AOMMAX(src->y_width + dst->border, ALIGN_POWER_OF_TWO(src->y_width, 6)) -
-      src->y_crop_width;
-  const int eb_y = AOMMAX(src->y_height + dst->border,
-                          ALIGN_POWER_OF_TWO(src->y_height, 6)) -
+  const int er_y = AOMMAX(src->y_crop_width + dst->border,
+                          ALIGN_POWER_OF_TWO(src->y_crop_width, 6)) -
+                   src->y_crop_width;
+  const int eb_y = AOMMAX(src->y_crop_height + dst->border,
+                          ALIGN_POWER_OF_TWO(src->y_crop_height, 6)) -
                    src->y_crop_height;

Patch walkthrough

• Source/ThirdParty/libwebrtc/Source/third_party/libaom/source/libaom/av1/encoder/extend.c — In av1_copy_and_extend_frame, changes the right-edge extent er_y and bottom-edge extent eb_y so their AOMMAX/ALIGN_POWER_OF_TWO padding targets use src->y_crop_width and src->y_crop_height instead of src->y_width and src->y_height. Because the destination buffer is sized from the crop dimensions, deriving the extension from those same dimensions ensures the copy-and-extend writes stay within the allocation even when size_align makes y_width/y_height larger than the display size.
• Source/ThirdParty/libwebrtc/Source/third_party/libaom/source/libaom/test/encode_api_test.cc — Adds TEST(EncodeAPI, SizeAlignOverflow) which allocates a 16x16 NV12 image with align=32, size_align=32, border=15 so the padded dimensions (32) exceed the display dimensions (16), memsets the data, and encodes one frame expecting AOM_CODEC_OK. On an unpatched build this reproduces the heap-buffer-overflow in av1_copy_and_extend_frame; with the fix it completes cleanly.

Background

YV12_BUFFER_CONFIG: crop vs padded dimensions — libaom’s frame buffer struct distinguishes the encoded/padded plane size (y_width, y_height) from the visible display size (y_crop_width, y_crop_height). Encoders often round the coded dimensions up to alignment/superblock boundaries, so y_width can exceed y_crop_width. Any code that both allocates and writes into these buffers must be careful to use the dimension the allocation was based on.

size_align and aom_img_alloc_with_border — When allocating an aom_image_t the caller can request an alignment (align), a size_align that rounds the stored width/height up to a multiple, and a border. If size_align exceeds the display dimensions, the internal padded w/h grow beyond d_w/d_h while the visible crop stays at the requested display size — exactly the mismatch the test constructs with d_w=d_h=16 and size_align=32.

Frame border extension — Video codecs surround the coded frame with a border of replicated edge pixels so that motion compensation and filters can read slightly outside the picture without bounds checks. av1_copy_and_extend_frame copies the source plane into the destination and then extends its top/bottom/left/right edges by et/eb/el/er pixels. If any extent is larger than the space actually allocated beyond the plane, the extend writes out of bounds.

ALIGN_POWER_OF_TWO and AOMMAX — ALIGN_POWER_OF_TWO(x, 6) rounds x up to the next multiple of 64, and AOMMAX picks the larger of the border-padded size and that alignment. The extension extent is (padded target) minus (base width). The bug is that the padded target used the coded width y_width while the base subtracted was the crop width, so the two were computed in different coordinate spaces and the difference overshot the crop-sized buffer.

WebRTC / libwebrtc-bundled libaom — WebKit ships libaom inside its libwebrtc third-party tree to provide AV1 encoding for WebRTC. This means the codec’s encode path is reachable from web-exposed real-time media APIs, and a bug in libaom’s encoder buffer handling becomes a WebKit attack surface rather than a purely offline-tooling concern.

Vulnerability window

1. Configuration — An encode path allocates an AV1 source image whose size_align (or alignment) rounds the coded dimensions above the display/crop dimensions — e.g. d_w=d_h=16 with size_align=32 giving y_width=y_height=32, y_crop_width=y_crop_height=16.
2. Buffer sizing — The destination extended-frame buffer is allocated based on the crop dimensions plus border.
3. Extent miscompute (bug) — av1_copy_and_extend_frame computes er_y/eb_y using AOMMAX/ALIGN on the padded y_width/y_height, yielding extents larger than the crop-sized buffer can accommodate.
4. Copy and extend — The routine copies the plane then writes er_y right-edge columns and eb_y bottom-edge rows; the excess columns/rows land past the end of the destination allocation.
5. Heap overflow — Adjacent heap memory is overwritten with edge-replicated pixel values, corrupting neighboring allocations (detected as heap-buffer-overflow under ASAN).
6. Post-fix — With extents derived from y_crop_width/y_crop_height, the extension exactly fills the crop-sized-plus-border buffer and the SizeAlignOverflow test encodes without crashing.

Proof of concept

The test configures a 16x16 AV1 realtime encode, then allocates an NV12 source image via aom_img_alloc_with_border with d_w=d_h=16 but size_align=32 and border=15. size_align=32 forces the internal padded width/height to 32 while the crop/display width/height stay at 16 — precisely the y_width>y_crop_width condition the buggy er_y/eb_y math mishandles. Encoding that frame drives av1_copy_and_extend_frame; pre-patch the oversized right/bottom extents write beyond the crop-sized destination buffer (ASAN heap-buffer-overflow), post-patch it returns AOM_CODEC_OK.

TEST(EncodeAPI, SizeAlignOverflow) {
  aom_codec_iface_t *iface = aom_codec_av1_cx();
  aom_codec_enc_cfg_t cfg;
  ASSERT_EQ(aom_codec_enc_config_default(iface, &cfg, AOM_USAGE_REALTIME),
            AOM_CODEC_OK);

  cfg.g_w = 16;
  cfg.g_h = 16;
  cfg.g_threads = 1;
  cfg.g_lag_in_frames = 0;

  aom_codec_ctx_t enc;
  ASSERT_EQ(aom_codec_enc_init(&enc, iface, &cfg, 0), AOM_CODEC_OK);

  // Bug aomdiea:480978101: size_align=32 causes w,h=32 while d_w,d_h=16
  // This mismatch causes buffer overflow in av1_copy_and_extend_frame()
  // if the fix is not present.
  aom_image_t *img =
      aom_img_alloc_with_border(NULL, AOM_IMG_FMT_NV12, /*d_w=*/16, /*d_h=*/16,
                                /*align=*/32,
                                /*size_align=*/32,
                                /*border=*/15);
  ASSERT_NE(img, nullptr);
  memset(img->img_data, 128, img->sz);

  // Should not crash with heap-buffer-overflow
  EXPECT_EQ(aom_codec_encode(&enc, img, /*pts=*/0, /*duration=*/1, /*flags=*/0),
            AOM_CODEC_OK);

  aom_img_free(img);
  ASSERT_EQ(aom_codec_destroy(&enc), AOM_CODEC_OK);
}

Exploitation

1. Reach the encoder — Negotiate an AV1 WebRTC session (or use another media-encode surface) so web-supplied frames flow into libaom’s encode pipeline with an attacker-influenced image geometry.
2. Induce the mismatch — Arrange an allocation where size_align/alignment pushes coded y_width/y_height above y_crop_width/y_crop_height, satisfying the condition that makes er_y/eb_y overshoot.
3. Trigger the write — Submit the frame for encode; av1_copy_and_extend_frame writes replicated edge pixels past the crop-sized destination buffer into adjacent heap.
4. Toward corruption — The overflow is edge-pixel data of border/alignment-bounded length, not attacker-arbitrary content or length, so weaponization depends on heap grooming to place a useful victim object next to the buffer; without that it remains a crash/DoS.

Detection & hunting

For defenders and SOC / detection engineers:

• ASAN heap-buffer-overflow —
• Encoder crash telemetry —
• Version exposure —

Audit directions

• Narrow grep —
• Extension/border routines —
• size_align call sites —
• Vendored-codec drift —