CVE-2026-11640
Overview
Background
- libyuv
- a widely-used YUV/RGB image conversion and comparison library bundled in Chromium and reachable through image/video decode and canvas paths.
- Row coalescing
- an optimization that, when every stride exactly equals the row width, flattens a 2D buffer into one row by doing
width *= height; height = 1;so inner loops run once overwidth*heightelements.
Root Cause Analysis
Many libyuv functions coalesced rows whenever the stride-equals-width condition held, executing width *= height without checking that the product fits in an int. For large images where width*height > INT_MAX, this signed multiplication overflows (undefined behavior), yielding a bogus coalesced width that no longer describes the real buffer, so subsequent row processing can read or write outside the intended bounds. The functions also did not validate their inputs, so a null pointer or non-positive width/height could reach the coalescing math.
The fix adds, per function, an early return on !ptr || width <= 0 || height <= 0 and extends each coalescing condition with (ptrdiff_t)width * height <= INT_MAX, performing the product in a wider type and only flattening when it is representable. This preserves the invariant that the coalesced single-row width equals the true element count, so no overflow-derived size drives the pixel loops.
width *= height as 32-bit signed arithmetic before confirming the product fits in int; the fix pre-validates inputs and gates every coalescing site on a widened (ptrdiff_t)width * height <= INT_MAX check so the flatten only happens when the result is exact.Attack Path
- Supply an oversized image
Deliver image/frame data to a libyuv entry point (via decode, canvas, or conversion) with dimensions whose
width*heightexceedsINT_MAXwhile strides equal width so the coalescing branch is taken. - Overflow the coalesced width
width *= heightoverflows signedint, producing a small/negativewidththat misrepresents the flattened buffer length. - Drive loops with a bogus size
The conversion/compare routine iterates using the corrupted
width, so pixel processing operates on a length inconsistent with the real allocation. - Out-of-bounds access
The mismatch leads to out-of-bounds reads or writes during row processing; the exact sink, offset, and whether a write primitive results are
(inferred)as the diff only shows the guard, not the downstream memory operation.
Impact Assessment
width*height > INT_MAX reaching a coalescing path, which requires very large dimensions but no special flag or privilege. Any escalation beyond the overflow-driven OOB the patch guards (controlled write, code execution, sandbox escape) is (inferred).Changed Functions
| Function | Change | Notes |
|---|---|---|
ifsource/compare.cc |
modified | |
ifsource/convert.cc |
modified | |
ifsource/convert_argb.cc |
modified |
Files Changed
README.chromiuminclude/libyuv/version.hsource/compare.ccsource/convert.ccsource/convert_argb.cc
Audit Directions
- Coalescing / flatten optimizationsAudit stride-equals-width flatten paths across conversion libraries, since each independently repeats the same overflow-prone pattern.
- Missing input validation at entry pointsCheck public image routines for null-pointer and non-positive
width/heightguards before any arithmetic on dimensions.
Patch
From c98edcc8dcdba15beaf866a4a5ea8cccb2865cbe Mon Sep 17 00:00:00 2001 From: Wan-Teh Chang <[email protected]> Date: Fri, 29 May 2026 11:29:01 -0700 Subject: [PATCH] Don't coalesce rows if width*height would overflow Audit all occurrences of "width *= height;" in the libyuv source code. Make sure height > 0 and (ptrdiff_t)width * height <= INT_MAX before executing width *= height. Bug: chromium:517339758 Change-Id: I143a41c66492a6e4c48b6aa2a1c4a2ae974ceeb1 Reviewed-on: https://chromium-review.googlesource.com/c/libyuv/libyuv/+/7883816 Commit-Queue: Wan-Teh Chang <[email protected]> Reviewed-by: Frank Barchard <[email protected]> --- diff --git a/README.chromium b/README.chromium index c1f4164..ae8e037 100644 --- a/README.chromium +++ b/README.chromium @@ -1,6 +1,6 @@ Name: libyuv URL: https://chromium.googlesource.com/libyuv/libyuv/ -Version: 1942 +Version: 1943 Revision: DEPS License: BSD-3-Clause License File: LICENSE diff --git a/include/libyuv/version.h b/include/libyuv/version.h index ab2420c..d739e7e 100644 --- a/include/libyuv/version.h +++ b/include/libyuv/version.h @@ -11,6 +11,6 @@ #ifndef INCLUDE_LIBYUV_VERSION_H_ #define INCLUDE_LIBYUV_VERSION_H_ -#define LIBYUV_VERSION 1942 +#define LIBYUV_VERSION 1943 #endif // INCLUDE_LIBYUV_VERSION_H_ diff --git a/source/compare.cc b/source/compare.cc index e85cc6d..1002330 100644 --- a/source/compare.cc +++ b/source/compare.cc @@ -11,6 +11,7 @@ #include "libyuv/compare.h" #include <float.h> +#include <limits.h> #include <math.h> #ifdef _OPENMP #include <omp.h> @@ -106,8 +107,11 @@ uint32_t fourcc = 0; int h; + if (!argb || width <= 0 || height <= 0) { + return fourcc; + } // Coalesce rows. - if (stride_argb == width * 4) { + if (stride_argb == width * 4 && (ptrdiff_t)width * height <= INT_MAX) { width *= height; height = 1; stride_argb = 0; @@ -245,8 +249,12 @@ int height) { uint64_t sse = 0; int h; + if (!src_a || !src_b || width <= 0 || height <= 0) { + return sse; + } // Coalesce rows. - if (stride_a == width && stride_b == width) { + if (stride_a == width && stride_b == width && + (ptrdiff_t)width * height <= INT_MAX) { width *= height; height = 1; stride_a = stride_b = 0; diff --git a/source/convert.cc b/source/convert.cc index 4a71365..4f5cca5 100644 --- a/source/convert.cc +++ b/source/convert.cc @@ -10,6 +10,8 @@ #include "libyuv/convert.h" +#include <limits.h> + #include "libyuv/basic_types.h" #include "libyuv/cpu_id.h" #include "libyuv/planar_functions.h" @@ -1435,7 +1437,8 @@ src_stride_uv = -src_stride_uv; } // Coalesce rows. - if (src_stride_y == width && dst_stride_y == width) { + if (src_stride_y == width && dst_stride_y == width && + (ptrdiff_t)width * height <= INT_MAX) { width *= height; height = 1; src_stride_y = dst_stride_y = 0; @@ -4476,7 +4479,8 @@ src_stride_rgb24 = -src_stride_rgb24; } // Coalesce rows. - if (src_stride_rgb24 == width * 3 && dst_stride_yj == width) { + if (src_stride_rgb24 == width * 3 && dst_stride_yj == width && + (ptrdiff_t)width * height <= INT_MAX) { width *= height; height = 1; src_stride_rgb24 = dst_stride_yj = 0; @@ -4641,7 +4645,8 @@ src_stride_raw = -src_stride_raw; } // Coalesce rows. - if (src_stride_raw == width * 3 && dst_stride_yj == width) { + if (src_stride_raw == width * 3 && dst_stride_yj == width && + (ptrdiff_t)width * height <= INT_MAX) { width *= height; height = 1; src_stride_raw = dst_stride_yj = 0; diff --git a/source/convert_argb.cc b/source/convert_argb.cc index 82f1096..1d73d1e 100644 --- a/source/convert_argb.cc +++ b/source/convert_argb.cc @@ -11,6 +11,7 @@ #include "libyuv/convert_argb.h" #include <assert.h> +#include <limits.h> #include "libyuv/convert_from_argb.h" #include "libyuv/cpu_id.h" @@ -327,7 +328,8 @@ } // Coalesce rows. if (src_stride_y == width && src_stride_u * 2 == width && - src_stride_v * 2 == width && dst_stride_argb == width * 4) { + src_stride_v * 2 == width && dst_stride_argb == width * 4 && + (ptrdiff_t)width * height <= INT_MAX) { width *= height; height = 1; src_stride_y = src_stride_u = src_stride_v = dst_stride_argb = 0; @@ -581,7 +583,7 @@ } // Coalesce rows. if (src_stride_y == width && src_stride_u == width && src_stride_v == width && - dst_stride_argb == width * 4) { + dst_stride_argb == width * 4 && (ptrdiff_t)width * height <= INT_MAX) { width *= height; height = 1; src_stride_y = src_stride_u = src_stride_v = dst_stride_argb = 0; @@ -818,7 +820,7 @@ } // Coalesce rows. if (src_stride_y == width && src_stride_u == width && src_stride_v == width && - dst_stride_rgb24 == width * 3) { + dst_stride_rgb24 == width * 3 && (ptrdiff_t)width * height <= INT_MAX) { width *= height; height = 1; src_stride_y = src_stride_u = src_stride_v = dst_stride_rgb24 = 0; @@ -3181,7 +3183,8 @@ dst_stride_argb = -dst_stride_argb; } // Coalesce rows. - if (src_stride_y == width && dst_stride_argb == width * 4) { + if (src_stride_y == width && dst_stride_argb == width * 4 && + (ptrdiff_t)width * height <= INT_MAX) { width *= height; height = 1; src_stride_y = dst_stride_argb = 0; @@ -3275,7 +3278,8 @@ src_stride_y = -src_stride_y; } // Coalesce rows. - if (src_stride_y == width && dst_stride_argb == width * 4) { + if (src_stride_y == width && dst_stride_argb == width * 4 && + (ptrdiff_t)width * height <= INT_MAX) { width *= height; height = 1; src_stride_y = dst_stride_argb = 0; @@ -3449,7 +3453,8 @@ src_stride_argb = -src_stride_argb; } // Coalesce rows. - if (src_stride_argb == width * 4 && dst_stride_bgra == width * 4) { + if (src_stride_argb == width * 4 && dst_stride_bgra == width * 4 && + (ptrdiff_t)width * height <= INT_MAX) { width *= height; height = 1; src_stride_argb = dst_stride_bgra = 0; @@ -3490,7 +3495,8 @@ src_stride_argb = -src_stride_argb; } // Coalesce rows. - if (src_stride_argb == width * 4 && dst_stride_abgr == width * 4) { + if (src_stride_argb == width * 4 && dst_stride_abgr == width * 4 && + (ptrdiff_t)width * height <= INT_MAX) { width *= height;
Original Bug Report
Potential Signed Integer Overflow in libyuv::CopyPlane Row-Coalescing Leading to Out-Of-Bounds Copy
Project Fortify, an experimental security project, has identified the following potential security issue. If you’re a feature owner CC-ed on this bug, please do your best to review these reports. Please see https://chromium.googlesource.com/chromium/src/+/main/docs/security/ai-generated-security-bugs-faq.md for more information.
Overview: A potential signed integer overflow in libyuv::CopyPlane occurs during row-coalescing when the product of the width and height exceeds the limit of a 32-bit signed integer. This overflow can result in a negative width parameter being passed to copying kernels such as ERMS, SSE2, NEON, or standard memcpy. Consequently, this leads to a massive out-of-bounds copy or an infinite loop, potentially resulting in memory corruption within the GPU or browser process.
Affected files:
third_party/libyuv/source/planar_functions.ccgpu/command_buffer/service/shared_image/copy_image_plane.ccthird_party/libyuv/source/row_gcc.ccthird_party/libyuv/source/row_common.cc
Estimated timestamp from git blame: 2024-07-05
Description
A potential memory safety vulnerability has been identified in libyuv::CopyPlane’s row-coalescing implementation. The function performs a signed 32-bit integer multiplication of the width and height parameters without validating that the product does not overflow:
// third_party/libyuv/source/planar_functions.cc
if (src_stride_y == width && dst_stride_y == width) {
width *= height; // Potential signed integer overflow
height = 1;
src_stride_y = dst_stride_y = 0;
}
If an image with extremely large dimensions is processed (for example, a 16384x16384 image using a format with 8 bytes per pixel, such as RGBA_F16), the row width in bytes is 131072 and the height is 16384. The multiplication 131072 * 16384 equals 2,147,483,648 (0x80000000), which overflows a signed 32-bit integer to INT_MIN (-2147483648).
Impact and Technical Mechanism
When a negative value like INT_MIN is assigned to width and passed down to downstream copy kernels, several behaviors can occur depending on the active CPU architecture and instruction set:
-
ERMS Kernel (
CopyRow_ERMS) & C Fallback (CopyRow_C):- The negative
int widthis cast tosize_t. On 64-bit systems, this triggers sign-extension, converting-2147483648to0xFFFFFFFF80000000(approximately 16 exabytes). - The
rep movsbinstruction or the underlyingmemcpyis invoked with this massive count, resulting in a wild out-of-bounds copy.
- The negative
-
SIMD Kernels (
CopyRow_SSE2,CopyRow_NEON):- In
CopyRow_SSE2andCopyRow_NEON, assembly loop implementations subtract a fixed chunk size (e.g., 32 bytes) from the signed loop counter and branch if the remaining count is greater than zero. - When
INT_MINis decremented by 32, it underflows to a large positive integer (2147483616), causing the loop to run indefinitely and corrupt memory.
- In
Potential Reachability in Chrome
This function is reached via gpu::CopyImagePlane in gpu/command_buffer/service/shared_image/copy_image_plane.cc during pixel data uploads or readbacks (e.g., in D3DImageBacking::UploadFromMemory or other platform-specific backing implementations). If a compromised renderer can create and upload to a large SharedImage that bypasses initial size allocations, this could serve as a potential sandbox escape path targeting the GPU process.
Note: These are potential steps analyzed statically; our security analysis tools do not have the capability to execute code or verify runtime exploits.
Suggested Remediation
To prevent this signed integer overflow, validate that the product of width and height does not overflow before coalescing the rows. For example:
if (src_stride_y == width && dst_stride_y == width) {
if (height > 0 && width > 0x7fffffff / height) {
return; // Or handle the error safely without coalescing
}
width *= height;
height = 1;
src_stride_y = dst_stride_y = 0;
}
Evaluated with Chrome root at commit: b1520ef4a76878853a31f0943b565e42060edec8
Results so far have been promising, but there can be wrong deductions. Feel free to adjust as follows:
- If you are familiar with the severity guidelines, you may adjust the severity.
- If this is a false positive, and there’s no work to be done, please close as WAI.
- If there is work to do here but not a vulnerability, please change the issue type to Task/Bug/FR.
Data from false positives will be used to improve accuracy over time. And please feel free to reach out to me directly if you have concerns or feedback on the project.