When rewriting cmap, the subsetter built one hb_set_t of covered code
points per encoding record. SubtableUnicodesCache keyed those sets by
the record's own offset and collected each with num_glyphs = UINT_MAX,
so a font with many records sharing one full-range format 12 subtable
built one ~1.1M-code-point set per record: ~41 s of CPU, or ~2.1 GB of
memory for a 96 KB input, with nothing but the record count (bounded
only by the sanitizer) limiting the work.
Three changes bound it, all matching how nearby cmap code already works:
1. Key cached_unicodes by record->subtable, so records sharing a
subtable share one set (both set_for overloads).
2. Give the cache a num_glyphs member and collect with it instead of
UINT_MAX, as the cmap accelerator already does for
hb_face_collect_unicodes; cmap::serialize fills it from
plan->source->get_num_glyphs() and create_filled_cache() takes it
as a parameter, passed as face->get_num_glyphs() for
hb_subset_preprocess.
3. Process only the first encoding record of each (platformID,
encodingID) pair in both cmap::serialize and create_filled_cache
(variation-selector records exempt). Lookup already sees one record
per pair and EncodingRecord::copy links later records of a format to
the first serialized subtable, so this bounds both the number of
sets built and the _can_drop walk regardless of how many records a
malformed font repeats, independent of whether they share a subtable.
No public API or ABI change. Output is unchanged for well-formed fonts:
the plan's unicodes are already restricted to gids below num_glyphs, and
valid fonts never repeat a (platformID, encodingID) pair. The attached
reproducer's hostile fonts drop from ~41 s / 2.1 GB to ~0 s / 12 MiB
with byte-identical output.
Adds test_subset_cmap_shared_subtable_many_records and
test_subset_cmap_repeated_encoding_records to
test/api/test-subset-cmap.c.
Discovered by Anthropic's OSS Scanner, as vulnerability ANT-2026-FSXRDPBT.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>