Files
Behdad EsfahbodandClaude Opus 4.8 fd57025171 [syllabic] Record HB_MAX_SYLLABLE_LENGTH cuts so every pass sees the same pieces (#6558)
Since 12b686dad, _hb_next_syllable() cuts syllables longer than
HB_MAX_SYLLABLE_LENGTH at whatever glyph index the walk reaches, and
nothing records that cut.  The Indic shaper's initial and final
reordering (with GSUB features shortening the buffer in between) then
walk different pieces, the deferred pre-base-matra cluster merge never
happens, and hb_shape() returns non-monotone clusters at the default
MONOTONE_GRAPHEMES level for an over-long Indic syllable.

Add hb_syllabic_record_cut(), next to the other shared syllabic
helpers, and call it on each piece from the foreach_syllable() loops
that already exist where syllables are set up in all four syllabic
shapers (Indic, Khmer, Myanmar, USE), and once more in Indic initial
reordering since dotted-circle insertion can push a piece back over
the limit.  Its inline fast path is a single length compare, so it
costs nothing unless a piece reaches the limit.  Only a piece of
HB_MAX_SYLLABLE_LENGTH glyphs that is followed by more of the same
syllable takes the slow path: it cuts the rest of the syllable into
the pieces foreach_syllable() would make, gives every piece but the
last one of two serials that differ from the preceding syllable and
from the last piece (which keeps the ragel serial), so substituted
glyphs inherit it and all later walks see identical pieces, and
merges the whole oversized syllable into one cluster
(merge_clusters at monotone cluster levels, unsafe-to-break
otherwise) so the cut cannot become a line-break opportunity
mid-syllable.  Serials of all other syllables are untouched, so
numbering and wraparound are exactly what the ragel machines
produce.  The cut in _hb_next_syllable() stays as the backstop.

Behaviour change: syllables longer than HB_MAX_SYLLABLE_LENGTH become
a single cluster, where previously each piece got its own clusters.

A regression test with a 68-codepoint Devanagari input is added to
indic-syllable.tests; it fails (non-monotone clusters) on the
unpatched tree.

Tested: meson test (all pass).  hb-shape on hi-words / a 175KB
Devanagari paragraph shows no measurable slowdown.

Reported by Anthropic's OSS Scanner as ordinary bug ANT-2026-W15T9EXZ.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-11-10 19:15:38 -06:00
..
2021-08-11 18:49:59 -06:00
2021-08-11 18:49:59 -06:00
2025-06-14 11:02:37 -06:00

Adding tests

You can test shaping of a unicode sequence against a font like this:

$ ./hb-unicode-encode 41 42 43 627 | ../../util/hb-shape font.ttf

assuming an in-tree build. The 41 42 43 627 here is a sequence of Unicode codepoints: U+0041,0042,0043,0627. When you are happy with the shape results, you can use the record-test.sh script to add this to the test suite. record-test.sh requires pyftsubset to be installed. You can get pyftsubset by installing FontTools from https://github.com/behdad/fonttools.

To use record-test.sh, just put it right before the hb-shape invocation:

$ ./hb-unicode-encode 41 42 43 627 | ./record-test.sh ../../util/hb-shape font.ttf

hb-subset and hb-view are resolved as sibling executables next to the provided hb-shape binary.

what this does is:

  • Subset the font for the sequence of Unicode characters requested,
  • Compare the hb-shape output of the original font versus the subset font for the input sequence,
  • If the outputs differ, perhaps it is because the font does not have glyph names; it then compares the output of hb-view for both fonts.
  • If the outputs differ, recording fails. Otherwise, it will move the subset font file into data/in-house/fonts and name it after its hash, and print out the test case input, which you can then redirect to an existing or new test file in data/in-house/tests using -o, e.g.:
$ ./hb-unicode-encode 41 42 43 627 | ./record-test.sh -o data/in-house/tests/test-name.tests ../../util/hb-shape font.ttf

If you created a new test file, add it to data/in-house/meson.build so it is run. Check that make check does indeed run it, and that the test passes. When everything looks good, git add the new font as well as the new test file if you created any. You can see what new files are there by running git status data/in-house. And commit!

Note! Please only add tests using Open Source fonts, preferably under OFL or similar license.