CVE-2026-118333
Overview
Background
- libvpx VP8 multithreaded decoder
vp8/vp8_dx_iface.cis the decoder API shim; it caches the last-decoded stream resolution inctx->si.w/ctx->si.hand reallocates frame buffers only when a new keyframe reports different dimensions.- `vp8_peek_si_internal()`
- parses a keyframe header and updates
ctx->si.w/ctx->si.hwith the new width/height before threads are (re)started. - `restart_threads` block
- when worker threads were torn down (e.g. after a prior decode error set
ctx->restart_threads = 1),vp8_decode()re-creates them viavp8_decoder_create_threads()+vp8mt_alloc_temp_buffers()inside asetjmpguard. - `resolution_change`
- a flag derived from comparing the incoming dimensions to
ctx->si.w/h; when0,vp8_alloc_frame_buffers()is skipped and the existing (old-size) frame buffers are kept. - `vp8mt_decode_mb_rows()`
- the multithreaded row decoder; it allocates per-row scratch (
mt_yabove_row, etc.) sized topc->Widthand writes into the frame bufferyv12_fb_new.
Root Cause Analysis
The advisory labels this “Missing authorization in Video” (AuthorizationError), but the diff and commit message describe a classic heap-buffer-overflow driven by inconsistent decoder state — there is no authorization logic anywhere in the change. The real invariant is that ctx->si.w/ctx->si.h (the cached resolution used to decide whether to reallocate frame buffers) must stay consistent with the actually-allocated buffer dimensions. vp8_peek_si_internal() eagerly writes the new keyframe dimensions into ctx->si.w/h before the restart_threads block runs. If vp8_decoder_create_threads() or vp8mt_alloc_temp_buffers() then fails, the setjmp error handler cleared ctx->common.error.setjmp, removed threads, and returned VPX_CODEC_ERROR — but it left ctx->si.w/h holding the new dimensions while the frame buffers (yv12_fb_new->y_width) still held the old size.
On the next decode of a keyframe with those same new dimensions, resolution_change computes to 0 (cached size already matches), so vp8_alloc_frame_buffers() is skipped and the old, smaller yv12_fb_new is reused — yet restart_threads succeeds this time and vp8mt_alloc_temp_buffers() sizes mt_yabove_row and the row-decode scratch to the new, smaller-than-expected-but-mismatched pc->Width. As the test documents, vp8mt_decode_mb_rows() then writes pc->Width-derived row data into the stale-sized yv12_fb_new, producing an out-of-bounds heap write (the test shows a 261-byte write into a 256-wide buffer). The fix adds ctx->si.w = 0; ctx->si.h = 0; to the restart_threads setjmp error handler so that a later resync sees resolution_change == 1, forcing a full vp8_alloc_frame_buffers() reallocation that keeps buffer size and pc->Width in sync.
Attack Path
- Web attacker
serves a VP8 video (via
<video>, WebCodecs, or any path reaching libvpx’s multithreaded VP8 decoder) to the victim; multithreaded decode must be enabled (inferred fromCONFIG_MULTITHREAD). - Prime threads a first valid keyframe at resolution A initializes frame buffers and worker threads at size A.
- Force a thread teardown
a corrupt/truncated frame makes a worker fail in
vp8mt_decode_mb_rows(), settingctx->restart_threads = 1. - Trigger restart failure on new resolution
a keyframe with new resolution B is parsed (
vp8_peek_si_internal()setsctx->si.w/h = B), then thread re-creation fails; the handler returns error but leavessi.w/h = Bwith buffers still at size A. - Resync with same dimensions
another keyframe at resolution B:
resolution_change == 0, reallocation skipped, but threads/scratch now sized to B. - Primitive
vp8mt_decode_mb_rows()writes B-sized row data into the A-sizedyv12_fb_new: a controlled-length heap-buffer-overflow in the process hosting the decoder.
Impact Assessment
pc->Width; the unit test demonstrates it deterministically. Reaching the bug requires a specific sequence (a thread-restart failure, which the test forces via RLIMIT_NPROC = 0); in the wild the restart-failure precondition is harder to trigger reliably (memory/thread exhaustion), so exploitability is gated but real. This is memory corruption, not merely a crash; escalation beyond the OOB write to full control is (inferred).Changed Functions
| Function | Change | Notes |
|---|---|---|
TESTtest/decode_api_test.cc |
modified | |
fortest/decode_api_test.cc |
modified |
Files Changed
vp8/vp8_dx_iface.ctest/decode_api_test.cc
Audit Directions
- Other early `ctx->si` mutationsaudit all sites that update
ctx->si.w/h(or equivalent cached dimensions) before an allocation that can fail; any failure path must roll back the cached size. - `setjmp`/error handlers in libvpxreview every
setjmp(error.jmp)recovery block in VP8/VP9 decoders for state left partially advanced (resolution, buffer counts, partition state) after a longjmp. - resolution_change gatingverify
vp8_alloc_frame_buffers()/ VP9 equivalents cannot be skipped while per-thread scratch (mt_yabove_row, etc.) is sized from a differentpc->Width. - Advisory-vs-code mismatchthe “Missing authorization” label is wrong for this memory-safety bug; cross-check other “authorization”-tagged media CVEs for similar mislabeling.
Patch
From ed2a5b572f0f62a72d2e483dd3031841f08fb343 Mon Sep 17 00:00:00 2001 From: Marco Paniconi <[email protected]> Date: Thu, 08 Oct 2026 04:09:30 +0000 Subject: [PATCH] vp8: clear cached resolution on thread restart failure In vp8_decode(), vp8_peek_si_internal() updates ctx->si.w and ctx->si.h from the keyframe header before restarting decode threads. If vp8_decoder_create_threads() or vp8mt_alloc_temp_buffers() fails in the restart_threads block, the setjmp handler returned VPX_CODEC_ERROR without clearing ctx->si.w and ctx->si.h. On subsequent decode calls with a keyframe of the same new dimensions, resolution_change stayed 0 and frame buffer reallocation was skipped while temporary multithread row buffers were allocated for the new width, causing a heap-buffer-overflow in vp8mt_decode_mb_rows(). Clear ctx->si.w and ctx->si.h in the restart_threads error handler so a full reallocation is triggered on resync. Added unittest. Bug: 571147583 Change-Id: Ic16476a531796751017182a1ef726756353f49d1 --- diff --git a/vp8/vp8_dx_iface.c b/vp8/vp8_dx_iface.c index 26de16c..4324a79 100644 --- a/vp8/vp8_dx_iface.c +++ b/vp8/vp8_dx_iface.c @@ -343,6 +343,10 @@ VP8_COMMON *const pc = &pbi->common; if (setjmp(pbi->common.error.jmp)) { pbi->common.error.setjmp = 0; + /* on failure clear the cached resolution to ensure a full + * reallocation is attempted on resync. */ + ctx->si.w = 0; + ctx->si.h = 0; vp8_decoder_remove_threads(pbi); vpx_clear_system_state(); ctx->fragments.count = 0; diff --git a/test/decode_api_test.cc b/test/decode_api_test.cc index b95d519..16eee00 100644 --- a/test/decode_api_test.cc +++ b/test/decode_api_test.cc @@ -14,6 +14,9 @@ #include "gtest/gtest.h" #include "./vpx_config.h" +#if CONFIG_VP8_DECODER && CONFIG_MULTITHREAD && defined(__linux__) +#include <sys/resource.h> +#endif #include "test/ivf_video_source.h" #include "test/video_source.h" #if CONFIG_VP8_ENCODER || CONFIG_VP9_ENCODER @@ -354,6 +357,100 @@ EXPECT_EQ(vpx_codec_destroy(&dec), VPX_CODEC_OK); } #endif // CONFIG_VP8_ENCODER && CONFIG_MULTITHREAD + +#if CONFIG_MULTITHREAD && defined(__linux__) +// Issue: 571147583 +// Verifies that when restarting decode threads fails on a keyframe with a new +// resolution, the cached dimensions (ctx->si.w, ctx->si.h) are cleared so a +// subsequent decode triggers a resolution change and reallocates frame buffers. +TEST(DecodeAPI, Vp8MultiThreadedRestartResolutionChange) { + // Minimal 2-partition VP8 keyframe buffer (first partition = 256 bytes). + std::vector<uint8_t> frame(1500, 0); + frame[0] = 0x10; + frame[1] = 0x20; + frame[3] = 0x9d; + frame[4] = 0x01; + frame[5] = 0x2a; + frame[6] = 0x00; // width = 256 + frame[7] = 0x01; + frame[8] = 0x00; // height = 256 + frame[9] = 0x01; + frame[10] = 0x0f; // 2 token partitions + frame[11] = 0xc1; + frame[12] = 0x28; + frame[266] = 0x58; // token partition 0 size = 600 + frame[267] = 0x02; + + vpx_codec_ctx_t dec; + vpx_codec_dec_cfg_t cfg = { /*threads=*/2, /*w=*/0, /*h=*/0 }; + ASSERT_EQ(vpx_codec_dec_init(&dec, &vpx_codec_vp8_dx_algo, &cfg, 0), + VPX_CODEC_OK); + + // Frame 0: valid 256x256 keyframe initializes 256x256 frame buffers + // (yv12_fb_new->y_width = 256, pc->Width = 256) and worker threads. + ASSERT_EQ( + vpx_codec_decode(&dec, frame.data(), + static_cast<unsigned int>(frame.size()), nullptr, 0), + VPX_CODEC_OK); + + // Frame 1: truncate token partition 0 to 1 byte so the worker thread fails in + // vp8mt_decode_mb_rows(), shutting down threads and setting + // ctx->restart_threads = 1. + frame[266] = 1; + frame[267] = 0; + EXPECT_EQ( + vpx_codec_decode(&dec, frame.data(), + static_cast<unsigned int>(frame.size()), nullptr, 0), + VPX_CODEC_CORRUPT_FRAME); + + // Change keyframe header to 16x64. In each iteration: + // 1. Set RLIMIT_NPROC = 0 so pthread_create fails inside restart_threads and + // vpx_codec_decode returns VPX_CODEC_ERROR after vp8_peek_si_internal() + // has updated ctx->si.w = 16, ctx->si.h = 64. + // 2. Restore RLIMIT_NPROC and decode again so restart_threads succeeds and + // allocates mt_yabove_row for pc->Width. + // + // Without the fix: + // - ctx->si.w = 16, ctx->si.h = 64 remain cached after step 1, so step 2 + // sees resolution_change == 0 and skips vp8_alloc_frame_buffers() + // (leaving yv12_fb_new->y_width = 256). + // - In iteration 0 step 2, vp8_decode_frame() sets pc->Width = 16 from the + // keyframe header before failing on the truncated token partition + // (setting ctx->restart_threads = 1 again). + // - In iteration 1 step 2, restart_threads allocates mt_yabove_row for + // pc->Width = 16 (103 bytes) while yv12_fb_new->y_width is still 256, + // causing a 261-byte write (heap-buffer-overflow) in + // vp8mt_decode_mb_rows(). + // + // With the fix: + // - Step 1 clears ctx->si.w = 0, ctx->si.h = 0 on thread restart failure. + // - Step 2 detects resolution_change == 1, reallocates frame buffers to + // 16x64, and cleanly returns VPX_CODEC_CORRUPT_FRAME on the truncated + // partition. + frame[6] = 16; + frame[7] = 0; + frame[8] = 64; + frame[9] = 0; + struct rlimit old_lim; + ASSERT_EQ(getrlimit(RLIMIT_NPROC, &old_lim), 0); + const struct rlimit zero_lim = { 0, old_lim.rlim_max }; + for (int i = 0; i < 2; ++i) { + ASSERT_EQ(setrlimit(RLIMIT_NPROC, &zero_lim), 0); + const vpx_codec_err_t res = + vpx_codec_decode(&dec, frame.data(), + static_cast<unsigned int>(frame.size()), nullptr, 0); + ASSERT_EQ(setrlimit(RLIMIT_NPROC, &old_lim), 0); + EXPECT_EQ(res, VPX_CODEC_ERROR); + + EXPECT_EQ( + vpx_codec_decode(&dec, frame.data(), + static_cast<unsigned int>(frame.size()), nullptr, 0), + VPX_CODEC_CORRUPT_FRAME); + } + + EXPECT_EQ(vpx_codec_destroy(&dec), VPX_CODEC_OK); +} +#endif // CONFIG_MULTITHREAD && defined(__linux__) #endif // CONFIG_VP8_DECODER #if CONFIG_VP9_DECODER
Regression Test / PoC
diff --git a/test/decode_api_test.cc b/test/decode_api_test.cc
index b95d519..16eee00 100644
--- a/test/decode_api_test.cc
+++ b/test/decode_api_test.cc
@@ -14,6 +14,9 @@
#include "gtest/gtest.h"
#include "./vpx_config.h"
+#if CONFIG_VP8_DECODER && CONFIG_MULTITHREAD && defined(__linux__)
+#include <sys/resource.h>
+#endif
#include "test/ivf_video_source.h"
#include "test/video_source.h"
#if CONFIG_VP8_ENCODER || CONFIG_VP9_ENCODER
@@ -354,6 +357,100 @@
EXPECT_EQ(vpx_codec_destroy(&dec), VPX_CODEC_OK);
}
#endif // CONFIG_VP8_ENCODER && CONFIG_MULTITHREAD
+
+#if CONFIG_MULTITHREAD && defined(__linux__)
+// Issue: 571147583
+// Verifies that when restarting decode threads fails on a keyframe with a new
+// resolution, the cached dimensions (ctx->si.w, ctx->si.h) are cleared so a
+// subsequent decode triggers a resolution change and reallocates frame buffers.
+TEST(DecodeAPI, Vp8MultiThreadedRestartResolutionChange) {
+ // Minimal 2-partition VP8 keyframe buffer (first partition = 256 bytes).
+ std::vector<uint8_t> frame(1500, 0);
+ frame[0] = 0x10;
+ frame[1] = 0x20;
+ frame[3] = 0x9d;
+ frame[4] = 0x01;
+ frame[5] = 0x2a;
+ frame[6] = 0x00; // width = 256
+ frame[7] = 0x01;
+ frame[8] = 0x00; // height = 256
+ frame[9] = 0x01;
+ frame[10] = 0x0f; // 2 token partitions
+ frame[11] = 0xc1;
+ frame[12] = 0x28;
+ frame[266] = 0x58; // token partition 0 size = 600
+ frame[267] = 0x02;
+
+ vpx_codec_ctx_t dec;
+ vpx_codec_dec_cfg_t cfg = { /*threads=*/2, /*w=*/0, /*h=*/0 };
+ ASSERT_EQ(vpx_codec_dec_init(&dec, &vpx_codec_vp8_dx_algo, &cfg, 0),
+ VPX_CODEC_OK);
+
+ // Frame 0: valid 256x256 keyframe initializes 256x256 frame buffers
+ // (yv12_fb_new->y_width = 256, pc->Width = 256) and worker threads.
+ ASSERT_EQ(
+ vpx_codec_decode(&dec, frame.data(),
+ static_cast<unsigned int>(frame.size()), nullptr, 0),
+ VPX_CODEC_OK);
+
+ // Frame 1: truncate token partition 0 to 1 byte so the worker thread fails in
+ // vp8mt_decode_mb_rows(), shutting down threads and setting
+ // ctx->restart_threads = 1.
+ frame[266] = 1;
+ frame[267] = 0;
+ EXPECT_EQ(
+ vpx_codec_decode(&dec, frame.data(),
+ static_cast<unsigned int>(frame.size()), nullptr, 0),
+ VPX_CODEC_CORRUPT_FRAME);
+
+ // Change keyframe header to 16x64. In each iteration:
+ // 1. Set RLIMIT_NPROC = 0 so pthread_create fails inside restart_threads and
+ // vpx_codec_decode returns VPX_CODEC_ERROR after vp8_peek_si_internal()
+ // has updated ctx->si.w = 16, ctx->si.h = 64.
+ // 2. Restore RLIMIT_NPROC and decode again so restart_threads succeeds and
+ // allocates mt_yabove_row for pc->Width.
+ //
+ // Without the fix:
+ // - ctx->si.w = 16, ctx->si.h = 64 remain cached after step 1, so step 2
+ // sees resolution_change == 0 and skips vp8_alloc_frame_buffers()
+ // (leaving yv12_fb_new->y_width = 256).
+ // - In iteration 0 step 2, vp8_decode_frame() sets pc->Width = 16 from the
+ // keyframe header before failing on the truncated token partition
+ // (setting ctx->restart_threads = 1 again).
+ // - In iteration 1 step 2, restart_threads allocates mt_yabove_row for
+ // pc->Width = 16 (103 bytes) while yv12_fb_new->y_width is still 256,
+ // causing a 261-byte write (heap-buffer-overflow) in
+ // vp8mt_decode_mb_rows().
+ //
+ // With the fix:
+ // - Step 1 clears ctx->si.w = 0, ctx->si.h = 0 on thread restart failure.
+ // - Step 2 detects resolution_change == 1, reallocates frame buffers to
+ // 16x64, and cleanly returns VPX_CODEC_CORRUPT_FRAME on the truncated
+ // partition.
+ frame[6] = 16;
+ frame[7] = 0;
+ frame[8] = 64;
+ frame[9] = 0;
+ struct rlimit old_lim;
+ ASSERT_EQ(getrlimit(RLIMIT_NPROC, &old_lim), 0);
+ const struct rlimit zero_lim = { 0, old_lim.rlim_max };
+ for (int i = 0; i < 2; ++i) {
+ ASSERT_EQ(setrlimit(RLIMIT_NPROC, &zero_lim), 0);
+ const vpx_codec_err_t res =
+ vpx_codec_decode(&dec, frame.data(),
+ static_cast<unsigned int>(frame.size()), nullptr, 0);
+ ASSERT_EQ(setrlimit(RLIMIT_NPROC, &old_lim), 0);
+ EXPECT_EQ(res, VPX_CODEC_ERROR);
+
+ EXPECT_EQ(
+ vpx_codec_decode(&dec, frame.data(),
+ static_cast<unsigned int>(frame.size()), nullptr, 0),
+ VPX_CODEC_CORRUPT_FRAME);
+ }
+
+ EXPECT_EQ(vpx_codec_destroy(&dec), VPX_CODEC_OK);
+}
+#endif // CONFIG_MULTITHREAD && defined(__linux__)
#endif // CONFIG_VP8_DECODER
#if CONFIG_VP9_DECODER