CVE-2026-12447
Overview
Changed Functions
| Function | Change | Notes |
|---|---|---|
TESTtest/encode_api_test.cc |
modified | |
fortest/encode_api_test.cc |
modified |
Files Changed
test/encode_api_test.ccvpx/src/vpx_encoder.c
Patch
From 63aad2761655ab788be64966ad462c98e2bce327 Mon Sep 17 00:00:00 2001 From: Marco Paniconi <[email protected]> Date: Tue, 19 May 2026 15:04:49 -0700 Subject: [PATCH] vp8-multi-res-encoding: Fix to out-of-bounds write Issue is when the first encoding (lowest resoln) fails, the context pointer is incorrectly advanced. Fix and unittest is from the issue below. Bug: 513405023 Change-Id: I95beefcd08b6566a41a54d75b14a332a7a0388aa --- diff --git a/test/encode_api_test.cc b/test/encode_api_test.cc index 62284a0..741c60a 100644 --- a/test/encode_api_test.cc +++ b/test/encode_api_test.cc @@ -1027,6 +1027,74 @@ } } +#if CONFIG_VP8_ENCODER && CONFIG_MULTI_RES_ENCODING +// Regression test for an off-by-one in vpx_codec_encode()'s multi-resolution +// loop: when the *first* iteration (highest-index encoder, smallest stream) +// returns non-OK, `break` skips `ctx--`, the post-loop `ctx++` advances ctx to +// one past the caller's array, and SAVE_STATUS(ctx, res) writes ctx->err out of +// bounds. With a heap-allocated 3-element array (matching WebRTC's +// std::vector<vpx_codec_ctx_t> encoders_), ASan reports a 4-byte +// heap-buffer-overflow WRITE at +16 past a 168-byte region. +// Bug: 513405023. +TEST(EncodeAPI, MultiResEncodeFirstIterFailOOB) { + constexpr int kNumEnc = 3; + // Heap-allocate exactly kNumEnc contexts, mirroring WebRTC's + // encoders_.reserve(3); encoders_.resize(3); + std::vector<vpx_codec_ctx_t> enc; + enc.reserve(kNumEnc); + enc.resize(kNumEnc); + memset(enc.data(), 0, sizeof(vpx_codec_ctx_t) * kNumEnc); + + vpx_codec_enc_cfg_t cfg[kNumEnc]; + vpx_rational_t dsf[kNumEnc] = { { 2, 1 }, { 2, 1 }, { 1, 1 } }; + const int w[kNumEnc] = { 320, 160, 80 }; + const int h[kNumEnc] = { 240, 120, 60 }; + + for (int i = 0; i < kNumEnc; ++i) { + ASSERT_EQ(vpx_codec_enc_config_default(&vpx_codec_vp8_cx_algo, &cfg[i], 0), + VPX_CODEC_OK); + cfg[i].g_w = w[i]; + cfg[i].g_h = h[i]; + cfg[i].g_lag_in_frames = 0; + cfg[i].rc_end_usage = VPX_CBR; + cfg[i].rc_resize_allowed = 0; + cfg[i].rc_target_bitrate = 300 >> i; + cfg[i].g_timebase.num = 1; + cfg[i].g_timebase.den = 30; + } + + ASSERT_EQ(vpx_codec_enc_init_multi(enc.data(), &vpx_codec_vp8_cx_algo, cfg, + kNumEnc, 0, dsf), + VPX_CODEC_OK); + + // Contiguous image array; vpx_codec_encode walks img += num_enc-1. + vpx_image_t imgs[kNumEnc]; + for (int i = 0; i < kNumEnc; ++i) { + ASSERT_NE(vpx_img_alloc(&imgs[i], VPX_IMG_FMT_I420, w[i], h[i], 1), + nullptr); + memset(imgs[i].img_data, 128, imgs[i].stride[0] * h[i] * 3 / 2); + } + + // Force the FIRST loop iteration (i = num_enc-1, ctx = &enc[2]) to fail in + // validate_img() before any ctx-- runs: set an unsupported format on the + // highest-index image. validate_img returns VPX_CODEC_INVALID_PARAM, the + // loop breaks, ctx++ advances to &enc[3], and SAVE_STATUS writes + // (&enc[3])->err — a 4-byte heap OOB write at offset 16 past the vector's + // 168-byte backing store. + imgs[kNumEnc - 1].fmt = VPX_IMG_FMT_I444; + + // Under ASan this line reports: + // heap-buffer-overflow WRITE of size 4 ... in vpx_codec_encode + // 0 bytes to the right of (or 16 past) 168-byte region. + EXPECT_EQ(vpx_codec_encode(enc.data(), imgs, /*pts=*/0, /*duration=*/1, + /*flags=*/0, VPX_DL_REALTIME), + VPX_CODEC_INVALID_PARAM); + + for (int i = 0; i < kNumEnc; ++i) vpx_img_free(&imgs[i]); + for (int i = kNumEnc - 1; i >= 0; --i) vpx_codec_destroy(&enc[i]); +} +#endif // CONFIG_VP8_ENCODER && CONFIG_MULTI_RES_ENCODING + TEST(EncodeAPI, SetRoi) { static struct { vpx_codec_iface_t *iface; diff --git a/vpx/src/vpx_encoder.c b/vpx/src/vpx_encoder.c index 36dfa51..f2b8b9f 100644 --- a/vpx/src/vpx_encoder.c +++ b/vpx/src/vpx_encoder.c @@ -236,7 +236,10 @@ ctx--; if (img) img--; } - ctx++; + // Only restore ctx to &ctx[0] when the loop ran to completion. If an + // iteration failed, ctx already points at the failing element; advancing + // it would point past the array when the first iteration failed. + if (i < 0) ctx++; } FLOATING_POINT_RESTORE();
Regression Test / PoC
diff --git a/test/encode_api_test.cc b/test/encode_api_test.cc
index 62284a0..741c60a 100644
--- a/test/encode_api_test.cc
+++ b/test/encode_api_test.cc
@@ -1027,6 +1027,74 @@
}
}
+#if CONFIG_VP8_ENCODER && CONFIG_MULTI_RES_ENCODING
+// Regression test for an off-by-one in vpx_codec_encode()'s multi-resolution
+// loop: when the *first* iteration (highest-index encoder, smallest stream)
+// returns non-OK, `break` skips `ctx--`, the post-loop `ctx++` advances ctx to
+// one past the caller's array, and SAVE_STATUS(ctx, res) writes ctx->err out of
+// bounds. With a heap-allocated 3-element array (matching WebRTC's
+// std::vector<vpx_codec_ctx_t> encoders_), ASan reports a 4-byte
+// heap-buffer-overflow WRITE at +16 past a 168-byte region.
+// Bug: 513405023.
+TEST(EncodeAPI, MultiResEncodeFirstIterFailOOB) {
+ constexpr int kNumEnc = 3;
+ // Heap-allocate exactly kNumEnc contexts, mirroring WebRTC's
+ // encoders_.reserve(3); encoders_.resize(3);
+ std::vector<vpx_codec_ctx_t> enc;
+ enc.reserve(kNumEnc);
+ enc.resize(kNumEnc);
+ memset(enc.data(), 0, sizeof(vpx_codec_ctx_t) * kNumEnc);
+
+ vpx_codec_enc_cfg_t cfg[kNumEnc];
+ vpx_rational_t dsf[kNumEnc] = { { 2, 1 }, { 2, 1 }, { 1, 1 } };
+ const int w[kNumEnc] = { 320, 160, 80 };
+ const int h[kNumEnc] = { 240, 120, 60 };
+
+ for (int i = 0; i < kNumEnc; ++i) {
+ ASSERT_EQ(vpx_codec_enc_config_default(&vpx_codec_vp8_cx_algo, &cfg[i], 0),
+ VPX_CODEC_OK);
+ cfg[i].g_w = w[i];
+ cfg[i].g_h = h[i];
+ cfg[i].g_lag_in_frames = 0;
+ cfg[i].rc_end_usage = VPX_CBR;
+ cfg[i].rc_resize_allowed = 0;
+ cfg[i].rc_target_bitrate = 300 >> i;
+ cfg[i].g_timebase.num = 1;
+ cfg[i].g_timebase.den = 30;
+ }
+
+ ASSERT_EQ(vpx_codec_enc_init_multi(enc.data(), &vpx_codec_vp8_cx_algo, cfg,
+ kNumEnc, 0, dsf),
+ VPX_CODEC_OK);
+
+ // Contiguous image array; vpx_codec_encode walks img += num_enc-1.
+ vpx_image_t imgs[kNumEnc];
+ for (int i = 0; i < kNumEnc; ++i) {
+ ASSERT_NE(vpx_img_alloc(&imgs[i], VPX_IMG_FMT_I420, w[i], h[i], 1),
+ nullptr);
+ memset(imgs[i].img_data, 128, imgs[i].stride[0] * h[i] * 3 / 2);
+ }
+
+ // Force the FIRST loop iteration (i = num_enc-1, ctx = &enc[2]) to fail in
+ // validate_img() before any ctx-- runs: set an unsupported format on the
+ // highest-index image. validate_img returns VPX_CODEC_INVALID_PARAM, the
+ // loop breaks, ctx++ advances to &enc[3], and SAVE_STATUS writes
+ // (&enc[3])->err — a 4-byte heap OOB write at offset 16 past the vector's
+ // 168-byte backing store.
+ imgs[kNumEnc - 1].fmt = VPX_IMG_FMT_I444;
+
+ // Under ASan this line reports:
+ // heap-buffer-overflow WRITE of size 4 ... in vpx_codec_encode
+ // 0 bytes to the right of (or 16 past) 168-byte region.
+ EXPECT_EQ(vpx_codec_encode(enc.data(), imgs, /*pts=*/0, /*duration=*/1,
+ /*flags=*/0, VPX_DL_REALTIME),
+ VPX_CODEC_INVALID_PARAM);
+
+ for (int i = 0; i < kNumEnc; ++i) vpx_img_free(&imgs[i]);
+ for (int i = kNumEnc - 1; i >= 0; --i) vpx_codec_destroy(&enc[i]);
+}
+#endif // CONFIG_VP8_ENCODER && CONFIG_MULTI_RES_ENCODING
+
TEST(EncodeAPI, SetRoi) {
static struct {
vpx_codec_iface_t *iface;
Original Bug Report
Potential heap OOB write in libvpx vpx_codec_encode during multi-resolution encoding
Project Fortify, an experimental security project, has identified the following potential security issue. If you’re a feature owner CC-ed on this bug, please do your best to review these reports. Please see https://chromium.googlesource.com/chromium/src/+/main/docs/security/ai-generated-security-bugs-faq.md for more information.
Overview: An off-by-one error in the multi-resolution encoding logic of libvpx’s vpx_codec_encode allows for a potential heap-based out-of-bounds write. If the first iteration of the encoding loop fails, the context pointer is incorrectly advanced beyond the allocated buffer before saving the error status. This path is reachable in the renderer process via WebRTC VP8 simulcast.
Affected files:
third_party/libvpx/source/libvpx/vpx/src/vpx_encoder.cthird_party/webrtc/modules/video_coding/codecs/vp8/libvpx_vp8_encoder.cc
Estimated timestamp from git blame: 2012-11-05
Summary
A potential heap-based out-of-bounds (OOB) write exists in libvpx within the vpx_codec_encode() function. When processing multi-resolution encodings (such as VP8 simulcast in WebRTC), the function fails to correctly manage the encoder context pointer if the first encoding iteration returns an error. This results in an error status being written to memory immediately following the allocated array of codec contexts.
Technical Details
In third_party/libvpx/source/libvpx/vpx/src/vpx_encoder.c, the multi-resolution branch of vpx_codec_encode handles multiple encoder contexts. The logic for iterating through these contexts is as follows:
// third_party/libvpx/source/libvpx/vpx/src/vpx_encoder.c:220
else {
int i;
ctx += num_enc - 1; // [1] ctx starts at last element
if (img) img += num_enc - 1;
for (i = num_enc - 1; i >= 0; i--) {
if ((res = ctx->iface->enc.encode(get_alg_priv(ctx), img, pts, duration,
flags, deadline)))
break; // [2] Break if first iteration fails
ctx--;
if (img) img--;
}
ctx++; // [3] ctx unconditionally incremented
}
FLOATING_POINT_RESTORE();
}
return SAVE_STATUS(ctx, res); // [4] Writes (ctx)->err = res
- At
[1],ctxis advanced to the last element of the context array (num_enc - 1). - If the very first call to
encodefails, thebreakstatement at[2]is executed. Since the break occurs beforectx--,ctxstill points to the last valid element. - At
[3], the code unconditionally executesctx++. This advances the pointer one element past the end of the allocated array (&arr[num_enc]). - Finally, the
SAVE_STATUSmacro at[4](defined as((ctx) ? ((ctx)->err = (var)) : (var))) writes the 4-byte error code to theerrmember of the now OOB context.
On 64-bit systems, vpx_codec_ctx_t is 56 bytes, and the err field is at an offset of 16 bytes. This results in a 4-byte write 16 bytes beyond the end of the heap allocation.
Potential Reachability
This code path is reachable in the Chromium renderer via WebRTC when using VP8 simulcast. webrtc::LibvpxVp8Encoder allocates a contiguous vector of vpx_codec_ctx_t structures (encoders_) and calls vpx_codec_encode with the base of this vector. A failure in the first encoder iteration (representing the lowest resolution stream) would trigger the OOB write.
Suggested Attack Steps
- Use a malicious website to initiate an
RTCPeerConnectionwith VP8 simulcast (e.g., 3 active encodings). - Attempt to induce a failure in the first encoding iteration (e.g., by creating memory pressure to cause internal allocation failures within the encoder).
- Leverage heap grooming to place a sensitive object adjacent to the
encoders_allocation to exploit the 4-byte OOB write.
Recommended Fix
The ctx++ increment should only occur if the loop has actually performed at least one successful iteration (i.e., ctx was decremented). Alternatively, the pointer logic could be restructured to avoid unconditional post-loop increments.
for (i = num_enc - 1; i >= 0; i--) {
if ((res = ctx->iface->enc.encode(get_alg_priv(ctx), img, pts, duration,
flags, deadline)))
break;
if (i > 0) {
ctx--;
if (img) img--;
}
}
Evaluated with Chrome root at commit: b3153093eb3c78c3e88ccf562bcbc20437a04b0e
Results so far have been promising, but there can be wrong deductions. Feel free to adjust as follows:
- If you are familiar with the severity guidelines, you may adjust the severity.
- If this is a false positive, and there’s no work to be done, please close as WAI.
- If there is work to do here but not a vulnerability, please change the issue type to Task/Bug/FR.
Data from false positives will be used to improve accuracy over time. And please feel free to reach out to me directly if you have concerns or feedback on the project.