CVE-2026-12447
Overview
Background
- Multi-resolution VP8 encoding
- a libvpx mode where one
vpx_codec_encodecall drives an array of encoder contexts (one per resolution), which WebRTC backs with a heapstd::vector<vpx_codec_ctx_t> encoders_. - `ctx` pointer walk
- inside
vpx_codec_encode, the code iterates the encoder-context array from the highest index (smallest stream) downward, decrementingctx(and the image pointerimg) each loop iteration. - `SAVE_STATUS(ctx, res)`
- a macro that records a result code by writing to
ctx->err, i.e. it dereferences and writes through thectxpointer after the loop. - `validate_img`
- the per-encoder input-image validation that returns
VPX_CODEC_INVALID_PARAMfor an unsupported format, causing the encode loop tobreak.
Root Cause Analysis
In vpx_codec_encode’s multi-resolution loop, each successful iteration does ctx-- (and img--) to step to the next encoder; after the loop, an unconditional ctx++ was executed to restore ctx to the base &ctx[0] on the assumption the loop always ran to completion. When the first iteration (highest index, smallest resolution) fails inside validate_img and the loop breaks, the compensating ctx-- for that iteration never runs, so ctx still points at the failing highest-index element; the unconditional post-loop ctx++ then advances it one element past the end of the array. The subsequent SAVE_STATUS(ctx, res) writes ctx->err through that out-of-bounds pointer — a 4-byte heap write past the vector’s backing store (the regression test observes a write 16 bytes past a 168-byte, 3-element region).
The fix guards the restore with if (i < 0) ctx++;, so ctx is only re-advanced when the loop completed normally (index ran below zero); on an early break ctx is left pointing at the valid failing element and the OOB write is prevented.
ctx++ restore even when a break had skipped the matching ctx--, pushing ctx past the array before SAVE_STATUS wrote through it; gating the increment on i < 0 restores it only when the loop actually completed.Attack Path
- Configure multi-res VP8 with heap-backed contexts
A caller (e.g. WebRTC via
encoders_.resize(3)) initializeskNumEncencoder contexts in a heap array and callsvpx_codec_encodeover them, as the regression testMultiResEncodeFirstIterFailOOBmirrors. - Fail the first (highest-index) iteration
The smallest-resolution input image carries an unsupported format (e.g.
VPX_IMG_FMT_I444), sovalidate_imgreturnsVPX_CODEC_INVALID_PARAMon the very first loop iteration (i = num_enc-1,ctx = &enc[2]) before anyctx--runs, and the loopbreaks. - Pointer advanced past the array
The unconditional
ctx++then movesctxto&enc[3], one element past the 3-element allocation. - Out-of-bounds heap write
SAVE_STATUS(ctx, res)writes(&enc[3])->err, a 4-byte heap-buffer-overflow write 16 bytes past the 168-byte backing region (per the patch/test); the value written is the result code, and no further primitive is demonstrated by the diff (inferred).
Impact Assessment
Changed Functions
| Function | Change | Notes |
|---|---|---|
TESTtest/encode_api_test.cc |
modified | |
fortest/encode_api_test.cc |
modified |
Files Changed
test/encode_api_test.ccvpx/src/vpx_encoder.c
Audit Directions
- Loop-completion assumptions after `break`Look for compensating pointer/index adjustments performed after a loop that assume the loop ran to completion, where an early
breakleaves the counter in a state that makes the adjustment overshoot. - Pointer arithmetic paired with error exitsAudit encode/decode and array-walk routines that step a pointer per iteration and then dereference it (e.g. status/error stores) once, especially when the first iteration can fail.
- Multi-context media APIsReview other libvpx/WebRTC multi-stream loops (
vpx_codec_enc_init_multiand callers over heapvector<vpx_codec_ctx_t>) for symmetricctx--/ctx++bookkeeping under partial-failure paths.
Patch
From 63aad2761655ab788be64966ad462c98e2bce327 Mon Sep 17 00:00:00 2001 From: Marco Paniconi <[email protected]> Date: Tue, 19 May 2026 15:04:49 -0700 Subject: [PATCH] vp8-multi-res-encoding: Fix to out-of-bounds write Issue is when the first encoding (lowest resoln) fails, the context pointer is incorrectly advanced. Fix and unittest is from the issue below. Bug: 513405023 Change-Id: I95beefcd08b6566a41a54d75b14a332a7a0388aa --- diff --git a/test/encode_api_test.cc b/test/encode_api_test.cc index 62284a0..741c60a 100644 --- a/test/encode_api_test.cc +++ b/test/encode_api_test.cc @@ -1027,6 +1027,74 @@ } } +#if CONFIG_VP8_ENCODER && CONFIG_MULTI_RES_ENCODING +// Regression test for an off-by-one in vpx_codec_encode()'s multi-resolution +// loop: when the *first* iteration (highest-index encoder, smallest stream) +// returns non-OK, `break` skips `ctx--`, the post-loop `ctx++` advances ctx to +// one past the caller's array, and SAVE_STATUS(ctx, res) writes ctx->err out of +// bounds. With a heap-allocated 3-element array (matching WebRTC's +// std::vector<vpx_codec_ctx_t> encoders_), ASan reports a 4-byte +// heap-buffer-overflow WRITE at +16 past a 168-byte region. +// Bug: 513405023. +TEST(EncodeAPI, MultiResEncodeFirstIterFailOOB) { + constexpr int kNumEnc = 3; + // Heap-allocate exactly kNumEnc contexts, mirroring WebRTC's + // encoders_.reserve(3); encoders_.resize(3); + std::vector<vpx_codec_ctx_t> enc; + enc.reserve(kNumEnc); + enc.resize(kNumEnc); + memset(enc.data(), 0, sizeof(vpx_codec_ctx_t) * kNumEnc); + + vpx_codec_enc_cfg_t cfg[kNumEnc]; + vpx_rational_t dsf[kNumEnc] = { { 2, 1 }, { 2, 1 }, { 1, 1 } }; + const int w[kNumEnc] = { 320, 160, 80 }; + const int h[kNumEnc] = { 240, 120, 60 }; + + for (int i = 0; i < kNumEnc; ++i) { + ASSERT_EQ(vpx_codec_enc_config_default(&vpx_codec_vp8_cx_algo, &cfg[i], 0), + VPX_CODEC_OK); + cfg[i].g_w = w[i]; + cfg[i].g_h = h[i]; + cfg[i].g_lag_in_frames = 0; + cfg[i].rc_end_usage = VPX_CBR; + cfg[i].rc_resize_allowed = 0; + cfg[i].rc_target_bitrate = 300 >> i; + cfg[i].g_timebase.num = 1; + cfg[i].g_timebase.den = 30; + } + + ASSERT_EQ(vpx_codec_enc_init_multi(enc.data(), &vpx_codec_vp8_cx_algo, cfg, + kNumEnc, 0, dsf), + VPX_CODEC_OK); + + // Contiguous image array; vpx_codec_encode walks img += num_enc-1. + vpx_image_t imgs[kNumEnc]; + for (int i = 0; i < kNumEnc; ++i) { + ASSERT_NE(vpx_img_alloc(&imgs[i], VPX_IMG_FMT_I420, w[i], h[i], 1), + nullptr); + memset(imgs[i].img_data, 128, imgs[i].stride[0] * h[i] * 3 / 2); + } + + // Force the FIRST loop iteration (i = num_enc-1, ctx = &enc[2]) to fail in + // validate_img() before any ctx-- runs: set an unsupported format on the + // highest-index image. validate_img returns VPX_CODEC_INVALID_PARAM, the + // loop breaks, ctx++ advances to &enc[3], and SAVE_STATUS writes + // (&enc[3])->err — a 4-byte heap OOB write at offset 16 past the vector's + // 168-byte backing store. + imgs[kNumEnc - 1].fmt = VPX_IMG_FMT_I444; + + // Under ASan this line reports: + // heap-buffer-overflow WRITE of size 4 ... in vpx_codec_encode + // 0 bytes to the right of (or 16 past) 168-byte region. + EXPECT_EQ(vpx_codec_encode(enc.data(), imgs, /*pts=*/0, /*duration=*/1, + /*flags=*/0, VPX_DL_REALTIME), + VPX_CODEC_INVALID_PARAM); + + for (int i = 0; i < kNumEnc; ++i) vpx_img_free(&imgs[i]); + for (int i = kNumEnc - 1; i >= 0; --i) vpx_codec_destroy(&enc[i]); +} +#endif // CONFIG_VP8_ENCODER && CONFIG_MULTI_RES_ENCODING + TEST(EncodeAPI, SetRoi) { static struct { vpx_codec_iface_t *iface; diff --git a/vpx/src/vpx_encoder.c b/vpx/src/vpx_encoder.c index 36dfa51..f2b8b9f 100644 --- a/vpx/src/vpx_encoder.c +++ b/vpx/src/vpx_encoder.c @@ -236,7 +236,10 @@ ctx--; if (img) img--; } - ctx++; + // Only restore ctx to &ctx[0] when the loop ran to completion. If an + // iteration failed, ctx already points at the failing element; advancing + // it would point past the array when the first iteration failed. + if (i < 0) ctx++; } FLOATING_POINT_RESTORE();
Regression Test / PoC
diff --git a/test/encode_api_test.cc b/test/encode_api_test.cc
index 62284a0..741c60a 100644
--- a/test/encode_api_test.cc
+++ b/test/encode_api_test.cc
@@ -1027,6 +1027,74 @@
}
}
+#if CONFIG_VP8_ENCODER && CONFIG_MULTI_RES_ENCODING
+// Regression test for an off-by-one in vpx_codec_encode()'s multi-resolution
+// loop: when the *first* iteration (highest-index encoder, smallest stream)
+// returns non-OK, `break` skips `ctx--`, the post-loop `ctx++` advances ctx to
+// one past the caller's array, and SAVE_STATUS(ctx, res) writes ctx->err out of
+// bounds. With a heap-allocated 3-element array (matching WebRTC's
+// std::vector<vpx_codec_ctx_t> encoders_), ASan reports a 4-byte
+// heap-buffer-overflow WRITE at +16 past a 168-byte region.
+// Bug: 513405023.
+TEST(EncodeAPI, MultiResEncodeFirstIterFailOOB) {
+ constexpr int kNumEnc = 3;
+ // Heap-allocate exactly kNumEnc contexts, mirroring WebRTC's
+ // encoders_.reserve(3); encoders_.resize(3);
+ std::vector<vpx_codec_ctx_t> enc;
+ enc.reserve(kNumEnc);
+ enc.resize(kNumEnc);
+ memset(enc.data(), 0, sizeof(vpx_codec_ctx_t) * kNumEnc);
+
+ vpx_codec_enc_cfg_t cfg[kNumEnc];
+ vpx_rational_t dsf[kNumEnc] = { { 2, 1 }, { 2, 1 }, { 1, 1 } };
+ const int w[kNumEnc] = { 320, 160, 80 };
+ const int h[kNumEnc] = { 240, 120, 60 };
+
+ for (int i = 0; i < kNumEnc; ++i) {
+ ASSERT_EQ(vpx_codec_enc_config_default(&vpx_codec_vp8_cx_algo, &cfg[i], 0),
+ VPX_CODEC_OK);
+ cfg[i].g_w = w[i];
+ cfg[i].g_h = h[i];
+ cfg[i].g_lag_in_frames = 0;
+ cfg[i].rc_end_usage = VPX_CBR;
+ cfg[i].rc_resize_allowed = 0;
+ cfg[i].rc_target_bitrate = 300 >> i;
+ cfg[i].g_timebase.num = 1;
+ cfg[i].g_timebase.den = 30;
+ }
+
+ ASSERT_EQ(vpx_codec_enc_init_multi(enc.data(), &vpx_codec_vp8_cx_algo, cfg,
+ kNumEnc, 0, dsf),
+ VPX_CODEC_OK);
+
+ // Contiguous image array; vpx_codec_encode walks img += num_enc-1.
+ vpx_image_t imgs[kNumEnc];
+ for (int i = 0; i < kNumEnc; ++i) {
+ ASSERT_NE(vpx_img_alloc(&imgs[i], VPX_IMG_FMT_I420, w[i], h[i], 1),
+ nullptr);
+ memset(imgs[i].img_data, 128, imgs[i].stride[0] * h[i] * 3 / 2);
+ }
+
+ // Force the FIRST loop iteration (i = num_enc-1, ctx = &enc[2]) to fail in
+ // validate_img() before any ctx-- runs: set an unsupported format on the
+ // highest-index image. validate_img returns VPX_CODEC_INVALID_PARAM, the
+ // loop breaks, ctx++ advances to &enc[3], and SAVE_STATUS writes
+ // (&enc[3])->err — a 4-byte heap OOB write at offset 16 past the vector's
+ // 168-byte backing store.
+ imgs[kNumEnc - 1].fmt = VPX_IMG_FMT_I444;
+
+ // Under ASan this line reports:
+ // heap-buffer-overflow WRITE of size 4 ... in vpx_codec_encode
+ // 0 bytes to the right of (or 16 past) 168-byte region.
+ EXPECT_EQ(vpx_codec_encode(enc.data(), imgs, /*pts=*/0, /*duration=*/1,
+ /*flags=*/0, VPX_DL_REALTIME),
+ VPX_CODEC_INVALID_PARAM);
+
+ for (int i = 0; i < kNumEnc; ++i) vpx_img_free(&imgs[i]);
+ for (int i = kNumEnc - 1; i >= 0; --i) vpx_codec_destroy(&enc[i]);
+}
+#endif // CONFIG_VP8_ENCODER && CONFIG_MULTI_RES_ENCODING
+
TEST(EncodeAPI, SetRoi) {
static struct {
vpx_codec_iface_t *iface;
Original Bug Report
Potential heap OOB write in libvpx vpx_codec_encode during multi-resolution encoding
Project Fortify, an experimental security project, has identified the following potential security issue. If you’re a feature owner CC-ed on this bug, please do your best to review these reports. Please see https://chromium.googlesource.com/chromium/src/+/main/docs/security/ai-generated-security-bugs-faq.md for more information.
Overview: An off-by-one error in the multi-resolution encoding logic of libvpx’s vpx_codec_encode allows for a potential heap-based out-of-bounds write. If the first iteration of the encoding loop fails, the context pointer is incorrectly advanced beyond the allocated buffer before saving the error status. This path is reachable in the renderer process via WebRTC VP8 simulcast.
Affected files:
third_party/libvpx/source/libvpx/vpx/src/vpx_encoder.cthird_party/webrtc/modules/video_coding/codecs/vp8/libvpx_vp8_encoder.cc
Estimated timestamp from git blame: 2012-11-05
Summary
A potential heap-based out-of-bounds (OOB) write exists in libvpx within the vpx_codec_encode() function. When processing multi-resolution encodings (such as VP8 simulcast in WebRTC), the function fails to correctly manage the encoder context pointer if the first encoding iteration returns an error. This results in an error status being written to memory immediately following the allocated array of codec contexts.
Technical Details
In third_party/libvpx/source/libvpx/vpx/src/vpx_encoder.c, the multi-resolution branch of vpx_codec_encode handles multiple encoder contexts. The logic for iterating through these contexts is as follows:
// third_party/libvpx/source/libvpx/vpx/src/vpx_encoder.c:220
else {
int i;
ctx += num_enc - 1; // [1] ctx starts at last element
if (img) img += num_enc - 1;
for (i = num_enc - 1; i >= 0; i--) {
if ((res = ctx->iface->enc.encode(get_alg_priv(ctx), img, pts, duration,
flags, deadline)))
break; // [2] Break if first iteration fails
ctx--;
if (img) img--;
}
ctx++; // [3] ctx unconditionally incremented
}
FLOATING_POINT_RESTORE();
}
return SAVE_STATUS(ctx, res); // [4] Writes (ctx)->err = res
- At
[1],ctxis advanced to the last element of the context array (num_enc - 1). - If the very first call to
encodefails, thebreakstatement at[2]is executed. Since the break occurs beforectx--,ctxstill points to the last valid element. - At
[3], the code unconditionally executesctx++. This advances the pointer one element past the end of the allocated array (&arr[num_enc]). - Finally, the
SAVE_STATUSmacro at[4](defined as((ctx) ? ((ctx)->err = (var)) : (var))) writes the 4-byte error code to theerrmember of the now OOB context.
On 64-bit systems, vpx_codec_ctx_t is 56 bytes, and the err field is at an offset of 16 bytes. This results in a 4-byte write 16 bytes beyond the end of the heap allocation.
Potential Reachability
This code path is reachable in the Chromium renderer via WebRTC when using VP8 simulcast. webrtc::LibvpxVp8Encoder allocates a contiguous vector of vpx_codec_ctx_t structures (encoders_) and calls vpx_codec_encode with the base of this vector. A failure in the first encoder iteration (representing the lowest resolution stream) would trigger the OOB write.
Suggested Attack Steps
- Use a malicious website to initiate an
RTCPeerConnectionwith VP8 simulcast (e.g., 3 active encodings). - Attempt to induce a failure in the first encoding iteration (e.g., by creating memory pressure to cause internal allocation failures within the encoder).
- Leverage heap grooming to place a sensitive object adjacent to the
encoders_allocation to exploit the 4-byte OOB write.
Recommended Fix
The ctx++ increment should only occur if the loop has actually performed at least one successful iteration (i.e., ctx was decremented). Alternatively, the pointer logic could be restructured to avoid unconditional post-loop increments.
for (i = num_enc - 1; i >= 0; i--) {
if ((res = ctx->iface->enc.encode(get_alg_priv(ctx), img, pts, duration,
flags, deadline)))
break;
if (i > 0) {
ctx--;
if (img) img--;
}
}
Evaluated with Chrome root at commit: b3153093eb3c78c3e88ccf562bcbc20437a04b0e
Results so far have been promising, but there can be wrong deductions. Feel free to adjust as follows:
- If you are familiar with the severity guidelines, you may adjust the severity.
- If this is a false positive, and there’s no work to be done, please close as WAI.
- If there is work to do here but not a vulnerability, please change the issue type to Task/Bug/FR.
Data from false positives will be used to improve accuracy over time. And please feel free to reach out to me directly if you have concerns or feedback on the project.