U+180F MONGOLIAN FREE VARIATION SELECTOR FOUR was added in Unicode
14.0 and has Default_Ignorable_Code_Point=Yes, but
is_default_ignorable() only covered U+180B..180E, so FVS4 was never
hidden and leaked a .notdef into the output.
8c65442069 ("[Unicode 14] Handle U+180F FVS4") added the U+180F case
to _hb_glyph_info_set_unicode_props() and to the Arabic shaper, but
that branch sits inside "if (unicode->is_default_ignorable (u))", so
it has been unreachable since it was written. Extending the range to
U+180B..180F makes it live.
U+180B..180F are all Default_Ignorable_Code_Point per
DerivedCoreProperties.txt, so the range stays contiguous.
Before, with a font with no Mongolian coverage:
$ hb-shape Roboto-Regular.abc.ttf -u U+0061,U+180F
[1=0+1114|0=0@-908,0+0]
Every other default ignorable (U+180B..180E, U+FE00, U+200B) on that
font shapes to [1=0+1114]. After the change U+180F matches them.
Added two cases to mongolian-variation-selector.tests, which already
covered FVS1, FVS2, FVS3 and MVS but not FVS4: U+180F, and U+1810
MONGOLIAN DIGIT ZERO to pin the top of the range so it cannot be
widened by accident.
Tested with "meson test -C build": 254/254 OK, 0 failures. Restoring
the old U+180B..180E fails in-house 6174/6176, and widening it to
U+180B..1810 fails the same way.
Assisted-by: claude-opus-5
Co-authored-by: Dylan Pulver <dylanpulver@users.noreply.github.com>
Adding tests
You can test shaping of a unicode sequence against a font like this:
$ ./hb-unicode-encode 41 42 43 627 | ../../util/hb-shape font.ttf
assuming an in-tree build. The 41 42 43 627 here is a sequence of
Unicode codepoints: U+0041,0042,0043,0627. When you are happy with
the shape results, you can use the record-test.sh script to add
this to the test suite. record-test.sh requires pyftsubset to
be installed. You can get pyftsubset by installing
FontTools from https://github.com/behdad/fonttools.
To use record-test.sh, just put it right before the hb-shape invocation:
$ ./hb-unicode-encode 41 42 43 627 | ./record-test.sh ../../util/hb-shape font.ttf
hb-subset and hb-view are resolved as sibling executables next to the
provided hb-shape binary.
what this does is:
- Subset the font for the sequence of Unicode characters requested,
- Compare the
hb-shapeoutput of the original font versus the subset font for the input sequence, - If the outputs differ, perhaps it is because the font does not have
glyph names; it then compares the output of
hb-viewfor both fonts. - If the outputs differ, recording fails. Otherwise, it will move the
subset font file into
data/in-house/fontsand name it after its hash, and print out the test case input, which you can then redirect to an existing or new test file indata/in-house/testsusing-o, e.g.:
$ ./hb-unicode-encode 41 42 43 627 | ./record-test.sh -o data/in-house/tests/test-name.tests ../../util/hb-shape font.ttf
If you created a new test file, add it to data/in-house/meson.build
so it is run. Check that make check does indeed run it, and that the
test passes. When everything looks good, git add the new font as well
as the new test file if you created any. You can see what new files are
there by running git status data/in-house. And commit!
Note! Please only add tests using Open Source fonts, preferably under OFL or similar license.