fix(compare): count no text positions outside character ranges

The walker of the previous commit counted every element outside a
character range, such as StoryPreference, as a text position. A story
whose reference has MetadataPacketPreference, which the converter does
not write, then had all its range offsets shifted by one. Only Content
and elements inside a character range count now.

Trustworthy pairs at HEAD: reproduced 10,578,767 -> 10,579,111 (91.92 %),
extra values 26,516 -> 26,179; reference values unchanged. The numbers
in docs/measurement.md for the measurement change are re-measured with
the corrected rule (10,505,425 reproduced, 26,297 extra values). No key
reproduces fewer values.
This commit is contained in:
aiongg committed 2026-10-08 02:41:23 +00:00
1 parent 64e5667dfd
commit d4f3fd5a64
2 files changed
+17 -13

No files matched your search

+6 -4
View File
@@ -140,8 +140,10 @@ IDML puts some elements around text ranges or around text inside a range:
`compare.py` looks through them: every `ParagraphStyleRange` and
`CharacterStyleRange` of the story counts, and the text inside a wrapper
counts towards the offsets. Offsets count `Content` in UTF-16 code units
and every other element (`Br`, `Table`, `Footnote`, an anchored frame) as
one position; `Properties` and `XMLAttribute` count nothing. When a range
and every other element inside a character range (`Br`, `Table`,
`Footnote`, an anchored frame) as one position. `Properties`,
`XMLAttribute` and elements outside character ranges (`StoryPreference`,
`MetadataPacketPreference`, …) count nothing. When a range
nested in a wrapper starts at the same offset as the range around it, the
nested range is the one compared. The wrapper elements themselves have a
`Self` and count as elements.
@@ -151,8 +153,8 @@ Counting them changed the trustworthy totals as follows:
| Ranges counted | Reference values | Reproduced | Coverage | Extra values |
|---|---:|---:|---:|---:|
| Only `ParagraphStyleRange` children of the story and their `CharacterStyleRange` children | 11,142,835 | 10,501,458 | 94.24 % | 30,237 |
| Also inside `HyperlinkTextSource` and `CrossReferenceSource` | 11,158,348 | 10,505,081 | 94.15 % | 26,634 |
| Also inside `XMLElement`, `Change` and `EndnoteRange` | 11,508,691 | 10,505,081 | 91.28 % | 26,634 |
| Also inside `HyperlinkTextSource` and `CrossReferenceSource` | 11,158,348 | 10,505,425 | 94.15 % | 26,297 |
| Also inside `XMLElement`, `Change` and `EndnoteRange` | 11,508,691 | 10,505,425 | 91.28 % | 26,297 |
Before, the skipped ranges were not counted, and every later range of the
story started at a smaller offset in the reference than in the output: the
+11 -9
View File
@@ -224,35 +224,37 @@ def text_ranges(story):
"""Map text offset -> attributes of the paragraph and character range
starting there, with keys prefixed PSR. and CSR. Ranges inside wrapper
elements count, at whatever depth. Offsets count Content in UTF-16 code
units and every other element (Br, Table, Footnote, an anchored frame) as
one position."""
units and every other element inside a character range (Br, Table,
Footnote, an anchored frame) as one position; elements outside
character ranges (StoryPreference, ...) are not text."""
out = {}
pos = 0
def walk(el, pa):
"""Walk the children of `el` inside the paragraph attributes `pa`."""
def walk(el, pa, in_csr):
"""Walk the children of `el` inside the paragraph attributes `pa`;
`in_csr` tells whether `el` is inside a character range."""
nonlocal pos
for ch in el:
if ch.tag in NOT_TEXT:
continue
if ch.tag == "ParagraphStyleRange":
walk(ch, {"PSR." + k: v for k, v in props(ch).items()})
walk(ch, {"PSR." + k: v for k, v in props(ch).items()}, False)
elif ch.tag == "CharacterStyleRange":
start = pos
walk(ch, pa)
walk(ch, pa, True)
if pos > start:
ca = {"CSR." + k: v for k, v in props(ch).items()}
# A range nested at the same offset (Change > range at
# the start of this one) keeps the offset.
out.setdefault(start, {**pa, **ca})
elif ch.tag in WRAPPERS:
walk(ch, pa)
walk(ch, pa, in_csr)
elif ch.tag == "Content":
pos += len((ch.text or "").encode("utf-16-le")) // 2
else:
elif in_csr:
pos += 1
walk(story, {})
walk(story, {}, False)
return out