The subset plan turned every variation index collected by the COLR,
GDEF and BASE closures into a row to keep, checking only the outer
index against the subtable count and never the inner index against
VarData::itemCount. A font could therefore name tens of thousands of
rows in a VarData whose itemCount is 2, and partial instancing then
sized its work from that inflated row count, allocating
regionIndexCount x rows x 5 bytes (3.2 GB from a 240 KB COLRv1 font).
generate_varstore_inner_maps() now takes the store and keeps only rows
that ItemVariationStore::has_delta_set() confirms exist, and
remap_variation_indices() maps such an index to
HB_OT_LAYOUT_NO_VARIATIONS_INDEX without consuming a new row. That
preserves the runtime meaning of an out-of-range index (delta 0) and
keeps the two functions in step; all consumers already handle the
sentinel. This covers the COLR, GDEF and BASE callers. Well-formed
fonts are byte-identical.
The HVAR/VVAR route noted in the report (its planner builds row maps
itself, one per retained glyph) is left for a follow-up: folding the
missing rows into a shared zero row changes the output form of a
map-less HVAR whose glyph count exceeds the store's itemCount, which
is better landed on its own.
Adds an API regression test (test_subset_colr_phantom_rows) that fails
on the unpatched tree.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-0TLE41I0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>