glyf::subset() trims (and, with --no-hinting, walks the components of) every
retained glyph on every serializer retry. The only thing bounding that work
was the implicit assumption that loca is monotonic, so the retained ranges sum
to at most the table. loca need not be monotonic, so any number of glyph IDs
can alias the same bytes, and the work grows as glyph count times glyph size
times retries: a 328 KB font with loca = 0, L, 0, L, ... makes hb-subset spend
~22 s of CPU and then fail, where a same-sized monotonic font takes 0.02 s.
The glyph closure in _populate_gids_to_retain() already fetches each retained
glyph exactly once, before any table is subset, so charge the untrimmed length
there (before the composite operation count can bail out, so every glyph added
to the set is paid for) against a budget of HB_SUBSET_GLYF_MAX_BYTES_FACTOR (16)
times the glyf length, and fail the plan when it is exceeded. This also makes
hb_subset_preprocess() fall back to the source face cheaply.
Fonts with a monotonic loca never exceed a factor of one, so their output is
unchanged; the one behaviour change is that a specification-violating loca that
aliases large glyph bytes more than sixteen tables' worth is now rejected
instead of subsetting slowly.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-S5U7MYJ7.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>