COLR::subset() gives every retained COLRv0 base glyph a private copy of
its layer range, so N base glyphs sharing L layers cost N x L layer
copies with two hash lookups each, and nothing checked the total against
the 16-bit numLayers header field. serialize_V0() walks the lazy
iterator twice and _hb_subset_table_try() retries while doubling the
buffer, so a ~200 KB font with all records naming the same range kept
hb-subset busy for about two minutes before failing.
Cap the summed layer count with a cheap walk over base_it before layer_it
is built, and drop COLR (return_trace (false)) when the total exceeds
0xFFFF, the same way this function already handles V0 records it cannot
subset; this also removes the 16-bit wrap in serialize_V0(). Any total
above 65535 could only have produced a failed subset or a wrapped,
invalid numLayers, so no valid output is lost.
The two V0 closure helpers had the same N x L shape during planning;
collect the layer indices of all retained base glyphs into one set first
and visit each layer once. closure_glyphs() now takes the glyph set in
one call (the accelerator wrapper changes signature with it; no public
API change), and _colr_closure() passes the whole set instead of looping.
Adds test_subset_colr_shared_layers (256 base glyphs all claiming the
same 256 layers): retaining 255 gives a 65280-layer table, retaining all
256 gives an empty COLR.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-4T7NHCTK.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>