Commit Graph
19 Commits
Author SHA1 Message Date
aiongg 52a32e9b85 fix(compare): count text ranges inside wrapper elements
text_ranges read only ParagraphStyleRange children of the story and their
CharacterStyleRange children. Ranges inside HyperlinkTextSource,
CrossReferenceSource, XMLElement, Change and EndnoteRange were skipped,
and their text was left out of the offsets, so every later range of the
story started earlier in the reference than in the output. It now walks
through those wrappers at any depth.

Trustworthy pairs (compare.py --exclude own/):
- hyperlink and cross-reference sources alone: reference values
  11,142,835 -> 11,158,348, reproduced 10,501,458 -> 10,505,081,
  extra values 30,237 -> 26,634
- with XMLElement, Change and EndnoteRange (counted as the headline must
  include everything InDesign writes): reference values 11,508,691,
  coverage 94.24 % -> 91.28 %; 348,306 of the 350,343 new values are in
  one document
- story text unchanged (18,517 of 18,610); documents >= 99 %: 18
- three keys reproduce one value fewer (CSR.Underline, PSR.Justification,
  PSR.P.TabList): ranges that matched at a shifted offset before
2026-10-08 00:51:12 +00:00
aiongg 48d184f225 fix(compare): match fonts whose family has the technology suffix
IDML adds " (OTF)", " (TT)" or " (T1)" to some family names depending
on the exporting computer. compare.py removed it from family names but
not from Font Self keys and Font Name, so each such font counted as
missing and the converter's font as extra. The suffix is now removed
from the family part of both.

Trustworthy pairs: extra Font values 17,889 -> 5,146 (-12,743); the
same fonts now count as reproduced.
2026-10-07 21:53:14 +00:00
aiongg a82f3a849e feat: count extra values and leave out values of the exporting computer
compare.py counts values the output has and the reference does not
(extra values, extras.tsv), flags pairs whose IDML names UIDs deleted in
the INDD as stale, leaves out link times and BitmapPrinting, skips the
resource values of links resolved to another file at export, and
compares font family names without the technology suffix.

Trustworthy pairs: 495 -> 489; value coverage 88.71% -> 88.71%
(9,885,118 of 11,142,835); extra values 713,688 (6.40%).
2026-10-07 18:47:44 +00:00
aiongg 3f1f10799d feat: write the InCopy assignment of every document
Each assignment object becomes an Assignment with the values every IDML
has. Its name depends on the language of the computer that exported the
IDML, so it is left out of the output and of value coverage.

Value coverage on trustworthy pairs: 86.83 % to 86.86 %.
2026-10-07 13:45:28 +00:00
aiongg 722fddfe97 feat: write languages, document users and document settings
designmap.xml gets the document's languages from their INDD objects,
with the quotes every IDML gives each language, the document users from
the document object, and the settings elements every IDML of the version
has (watermark, linked story and page item options, footnote and
endnote options and others) with their common values. User colours are
set by the exporting InDesign, not the INDD, and are left out of value
coverage.

Value coverage on trustworthy pairs: 81.65 % to 82.03 %.
2026-10-07 11:08:03 +00:00
aiongg c9528b10ff feat(tools): list reproduced values per key and dump whole chunks
compare.py writes values.tsv and values-all.tsv with every key's values and
how many are reproduced, so a change can be checked for keys that lose
reproduced values. indd dump --full prints every byte of each chunk.
root_values.py reads the current inventory layout and keeps to the pairs
its generated files were made from.
2026-10-07 09:39:02 +00:00
aiongg f18a29bce1 feat(tools): add value coverage headline and ranked gap list
compare.py ends with value coverage (the share of reference IDML values the conversion reproduces, weighted by occurrence), document scores at 99 % and 99.9 %, and the biggest gaps ranked by documents and by values, over trustworthy pairs and over all pairs. pairs.tsv, gaps.tsv and gaps-all.tsv go to target/compare/. Font Status and package part names are left out as values no INDD can supply; docs/measurement.md defines the metric and the exclusions.
2026-10-07 08:41:26 +00:00
aiongg 4024dd79a8 feat(tools): report stale corpus pairs separately
A pair is stale when the IDML's xmp:ModifyDate is later than the INDD's or more than an hour earlier, when the IDML names UIDs the INDD database lacks, or when the INDD has stories the IDML lacks. compare.py prints totals for all pairs and for trustworthy pairs. New subcommands indd xmp and indd uids supply the INDD side. docs/measurement.md records the rule and its evidence: 159 of 654 pairs are stale.
2026-10-07 08:34:18 +00:00
aiongg 54cf473c52 perf(tools): validate many IDML packages per Jing run
validate.sh takes several packages and validates their parts with one JVM per schema and batch of 400 parts, in parallel. compare.py validates its outputs 600 at a time instead of one validate.sh run per file. The full corpus validates in about 4.5 minutes instead of 45.
2026-10-07 08:19:57 +00:00
aiongg dc198e4ea2 feat(tools): convert in parallel, rank shortfalls by documents, find sources by SOURCE-*.md
compare.py runs conversions and validations in the --jobs pool and
removes each output once compared. --shortfalls N ranks missing
elements, wrong or missing attribute values and differing story text by
the number of documents they affect.

inventory.py also treats a folder with SOURCE-*.md (a supplementary
download) as a source folder when pairing files.
2026-10-07 07:59:10 +00:00
aiongg 0bdae12f3d feat(tools): pair IDML and PDF across source folders and dedupe by SHA-256
inventory.py now records the reference IDML and PDF of each INDD file,
its SHA-256 and the first file with the same digest. Outside the file's
own folder, a partner is searched under the nearest folder with a
SOURCE.md. compare.py takes the IDML path from the inventory and skips
duplicate files by digest.
2026-10-07 06:15:20 +00:00
aiongg a9985b3775 fix: validate every corpus output and fix the two schema failures
compare.py --all now also converts the INDD and INDT files that have no
same-version IDML (any version, either byte order), validates them and
counts converter warnings by kind. validate.sh runs Jing once per schema
instead of once per file, which makes validating the whole corpus take
about 3 minutes.

Two schema failures showed up in files without an IDML:

- Gradient swatch midpoints are stored as positions from the start of
  the gradient. IDML wants them relative to the gap between two stops,
  as for opacity gradient stops. Values outside 13-87 are left out with
  a warning.
- The style kind is a u16, not a u32. One InDesign 7.5 template has a
  non-zero u16 after it, which made 151 character styles paragraph
  styles.
2026-10-07 03:50:08 +00:00
aiongg 7c3c38446b feat(compare): compare effect elements inside transparency settings 2026-10-06 23:55:15 +00:00
aiongg 7c95144d4a feat(compare): compare list properties and attribute-only properties
TabList, AllNestedStyles and other list-typed <Properties> children are
compared as one canonical string. Empty property elements such as
BulletChar are compared by their attributes.
2026-10-06 21:14:52 +00:00
aiongg fb2bc7cd9e chore: compare embedded contents by digest and add --bin option 2026-10-06 18:22:23 +00:00
aiongg 69573c8461 feat: convert document and text frame preferences
Write DocumentPreference (page size, facing pages, bleeds, intent) and
TextFramePreference (columns, gutter, column width, vertical
justification, auto-sizing). compare.py now also compares child elements
without Self and the preference elements.
2026-10-06 17:26:36 +00:00
aiongg 602065453e feat: validate output against the IDML schemas
Add tools/validate.sh, which checks every XML part of an IDML package
against InDesign's generated RelaxNG schemas with Jing, and a --schemas
option in compare.py. Output for all 75 corpus pairs validates.

Do not write BindingLocation on master spreads, which the schema does not
allow.
2026-10-06 17:11:34 +00:00
aiongg 5f87710d92 feat: convert text formatting
Parse attribute lists in style objects and in paragraph and character
style runs, and write 40 text attributes (font, style, size, leading,
colour, tracking, kerning method, indents, spacing, justification,
underline, rules, hyphenation and more) on styles and text ranges. Read
font family names.

tools/compare.py now also compares text range attributes at matching text
offsets; every mapped attribute matches the reference in all ranges.
2026-10-06 16:33:53 +00:00
aiongg 0da35fd644 feat: first INDD to IDML conversion
Build a document model (layers, spreads, master spreads, pages, page
items with path geometry, text frame threading, stories with paragraph
and character style runs, style names and hierarchy) and write it as an
IDML package with a minimal stored-ZIP writer.

Add tools/compare.py, which converts every corpus pair and compares the
result with the reference IDML. First run: all 351 little-endian files
convert; 921 of 1120 stories match exactly.
2026-10-06 16:17:00 +00:00