COLR::subset() gives every retained COLRv0 base glyph a private copy of
its layer range, so N base glyphs sharing L layers cost N x L layer
copies with two hash lookups each, and nothing checked the total against
the 16-bit numLayers header field. serialize_V0() walks the lazy
iterator twice and _hb_subset_table_try() retries while doubling the
buffer, so a ~200 KB font with all records naming the same range kept
hb-subset busy for about two minutes before failing.
Cap the summed layer count with a cheap walk over base_it before layer_it
is built, and drop COLR (return_trace (false)) when the total exceeds
0xFFFF, the same way this function already handles V0 records it cannot
subset; this also removes the 16-bit wrap in serialize_V0(). Any total
above 65535 could only have produced a failed subset or a wrapped,
invalid numLayers, so no valid output is lost.
The two V0 closure helpers had the same N x L shape during planning;
collect the layer indices of all retained base glyphs into one set first
and visit each layer once. closure_glyphs() now takes the glyph set in
one call (the accelerator wrapper changes signature with it; no public
API change), and _colr_closure() passes the whole set instead of looping.
Adds test_subset_colr_shared_layers (256 base glyphs all claiming the
same 256 layers): retaining 255 gives a 65280-layer table, retaining all
256 gives an empty COLR.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-4T7NHCTK.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cmap-format12-only files created by ttx & remove all other cmap entries
Inconsolata-Regular.abc.widerc.ttf has the hmtx width of "c" set to 600; everything else is 500. Subsetting out c should reduce numberOfHMetrics to 1.
chromacheck-* fonts are from https://github.com/RoelN/ChromaCheck/tree/master/fonts and licensed under MIT by Roel Nieskens and Google.
RanaKufi-Regular.subset.otf is from https://github.com/alif-type/rana-kufi/ but the subset is licensed for us in MIT for the project use.
glyphs.ttf is from https://github.com/RazrFalcon/ttf-parser/blob/337e7d1/tests/fonts/glyphs.ttf
Estedad-VF.ttf, licensed under OFL 1.1, is from https://github.com/aminabedi68/Estedad
graphite2-expansion.ttf is a synthetic three-glyph font generated by
../generate-graphite2-expansion-font.py. Its Graphite substitution rule expands
each "a" to 41 glyphs, for testing buffer growth failure.