text_ranges read only ParagraphStyleRange children of the story and their
CharacterStyleRange children. Ranges inside HyperlinkTextSource,
CrossReferenceSource, XMLElement, Change and EndnoteRange were skipped,
and their text was left out of the offsets, so every later range of the
story started earlier in the reference than in the output. It now walks
through those wrappers at any depth.
Trustworthy pairs (compare.py --exclude own/):
- hyperlink and cross-reference sources alone: reference values
11,142,835 -> 11,158,348, reproduced 10,501,458 -> 10,505,081,
extra values 30,237 -> 26,634
- with XMLElement, Change and EndnoteRange (counted as the headline must
include everything InDesign writes): reference values 11,508,691,
coverage 94.24 % -> 91.28 %; 348,306 of the 350,343 new values are in
one document
- story text unchanged (18,517 of 18,610); documents >= 99 %: 18
- three keys reproduce one value fewer (CSR.Underline, PSR.Justification,
PSR.P.TabList): ranges that matched at a shifted offset before
IDML adds " (OTF)", " (TT)" or " (T1)" to some family names depending
on the exporting computer. compare.py removed it from family names but
not from Font Self keys and Font Name, so each such font counted as
missing and the converter's font as extra. The suffix is now removed
from the family part of both.
Trustworthy pairs: extra Font values 17,889 -> 5,146 (-12,743); the
same fonts now count as reproduced.
compare.py counts values the output has and the reference does not
(extra values, extras.tsv), flags pairs whose IDML names UIDs deleted in
the INDD as stale, leaves out link times and BitmapPrinting, skips the
resource values of links resolved to another file at export, and
compares font family names without the technology suffix.
Trustworthy pairs: 495 -> 489; value coverage 88.71% -> 88.71%
(9,885,118 of 11,142,835); extra values 713,688 (6.40%).
Each assignment object becomes an Assignment with the values every IDML
has. Its name depends on the language of the computer that exported the
IDML, so it is left out of the output and of value coverage.
Value coverage on trustworthy pairs: 86.83 % to 86.86 %.
designmap.xml gets the document's languages from their INDD objects,
with the quotes every IDML gives each language, the document users from
the document object, and the settings elements every IDML of the version
has (watermark, linked story and page item options, footnote and
endnote options and others) with their common values. User colours are
set by the exporting InDesign, not the INDD, and are left out of value
coverage.
Value coverage on trustworthy pairs: 81.65 % to 82.03 %.
compare.py writes values.tsv and values-all.tsv with every key's values and
how many are reproduced, so a change can be checked for keys that lose
reproduced values. indd dump --full prints every byte of each chunk.
root_values.py reads the current inventory layout and keeps to the pairs
its generated files were made from.
compare.py ends with value coverage (the share of reference IDML values the conversion reproduces, weighted by occurrence), document scores at 99 % and 99.9 %, and the biggest gaps ranked by documents and by values, over trustworthy pairs and over all pairs. pairs.tsv, gaps.tsv and gaps-all.tsv go to target/compare/. Font Status and package part names are left out as values no INDD can supply; docs/measurement.md defines the metric and the exclusions.
A pair is stale when the IDML's xmp:ModifyDate is later than the INDD's or more than an hour earlier, when the IDML names UIDs the INDD database lacks, or when the INDD has stories the IDML lacks. compare.py prints totals for all pairs and for trustworthy pairs. New subcommands indd xmp and indd uids supply the INDD side. docs/measurement.md records the rule and its evidence: 159 of 654 pairs are stale.
validate.sh takes several packages and validates their parts with one JVM per schema and batch of 400 parts, in parallel. compare.py validates its outputs 600 at a time instead of one validate.sh run per file. The full corpus validates in about 4.5 minutes instead of 45.
compare.py runs conversions and validations in the --jobs pool and
removes each output once compared. --shortfalls N ranks missing
elements, wrong or missing attribute values and differing story text by
the number of documents they affect.
inventory.py also treats a folder with SOURCE-*.md (a supplementary
download) as a source folder when pairing files.
inventory.py now records the reference IDML and PDF of each INDD file,
its SHA-256 and the first file with the same digest. Outside the file's
own folder, a partner is searched under the nearest folder with a
SOURCE.md. compare.py takes the IDML path from the inventory and skips
duplicate files by digest.
compare.py --all now also converts the INDD and INDT files that have no
same-version IDML (any version, either byte order), validates them and
counts converter warnings by kind. validate.sh runs Jing once per schema
instead of once per file, which makes validating the whole corpus take
about 3 minutes.
Two schema failures showed up in files without an IDML:
- Gradient swatch midpoints are stored as positions from the start of
the gradient. IDML wants them relative to the gap between two stops,
as for opacity gradient stops. Values outside 13-87 are left out with
a warning.
- The style kind is a u16, not a u32. One InDesign 7.5 template has a
non-zero u16 after it, which made 151 character styles paragraph
styles.
TabList, AllNestedStyles and other list-typed <Properties> children are
compared as one canonical string. Empty property elements such as
BulletChar are compared by their attributes.
Write DocumentPreference (page size, facing pages, bleeds, intent) and
TextFramePreference (columns, gutter, column width, vertical
justification, auto-sizing). compare.py now also compares child elements
without Self and the preference elements.
Add tools/validate.sh, which checks every XML part of an IDML package
against InDesign's generated RelaxNG schemas with Jing, and a --schemas
option in compare.py. Output for all 75 corpus pairs validates.
Do not write BindingLocation on master spreads, which the schema does not
allow.
Parse attribute lists in style objects and in paragraph and character
style runs, and write 40 text attributes (font, style, size, leading,
colour, tracking, kerning method, indents, spacing, justification,
underline, rules, hyphenation and more) on styles and text ranges. Read
font family names.
tools/compare.py now also compares text range attributes at matching text
offsets; every mapped attribute matches the reference in all ranges.
Build a document model (layers, spreads, master spreads, pages, page
items with path geometry, text frame threading, stories with paragraph
and character style runs, style names and hierarchy) and write it as an
IDML package with a minimal stored-ZIP writer.
Add tools/compare.py, which converts every corpus pair and compares the
result with the reference IDML. First run: all 351 little-endian files
convert; 921 of 1120 stories match exactly.