Commit Graph
34 Commits
Author SHA1 Message Date
aiongg 52a32e9b85 fix(compare): count text ranges inside wrapper elements
text_ranges read only ParagraphStyleRange children of the story and their
CharacterStyleRange children. Ranges inside HyperlinkTextSource,
CrossReferenceSource, XMLElement, Change and EndnoteRange were skipped,
and their text was left out of the offsets, so every later range of the
story started earlier in the reference than in the output. It now walks
through those wrappers at any depth.

Trustworthy pairs (compare.py --exclude own/):
- hyperlink and cross-reference sources alone: reference values
  11,142,835 -> 11,158,348, reproduced 10,501,458 -> 10,505,081,
  extra values 30,237 -> 26,634
- with XMLElement, Change and EndnoteRange (counted as the headline must
  include everything InDesign writes): reference values 11,508,691,
  coverage 94.24 % -> 91.28 %; 348,306 of the 350,343 new values are in
  one document
- story text unchanged (18,517 of 18,610); documents >= 99 %: 18
- three keys reproduce one value fewer (CSR.Underline, PSR.Justification,
  PSR.P.TabList): ranges that matched at a shifted offset before
2026-10-08 00:51:12 +00:00
aiongg 48d184f225 fix(compare): match fonts whose family has the technology suffix
IDML adds " (OTF)", " (TT)" or " (T1)" to some family names depending
on the exporting computer. compare.py removed it from family names but
not from Font Self keys and Font Name, so each such font counted as
missing and the converter's font as extra. The suffix is now removed
from the family part of both.

Trustworthy pairs: extra Font values 17,889 -> 5,146 (-12,743); the
same fonts now count as reproduced.
2026-10-07 21:53:14 +00:00
aiongg a82f3a849e feat: count extra values and leave out values of the exporting computer
compare.py counts values the output has and the reference does not
(extra values, extras.tsv), flags pairs whose IDML names UIDs deleted in
the INDD as stale, leaves out link times and BitmapPrinting, skips the
resource values of links resolved to another file at export, and
compares font family names without the technology suffix.

Trustworthy pairs: 495 -> 489; value coverage 88.71% -> 88.71%
(9,885,118 of 11,142,835); extra values 713,688 (6.40%).
2026-10-07 18:47:44 +00:00
aiongg c787337d89 feat: add tools/diff_outputs.py to compare the output of two revisions 2026-10-07 18:18:06 +00:00
aiongg 3da31dc6c2 feat: write style keyboard shortcuts and empty nested, line and GREP style flags 2026-10-07 16:05:18 +00:00
aiongg 1e26e1e8bd feat: read text, margin, anchor, pasteboard, XML tag and page item defaults 2026-10-07 15:40:54 +00:00
aiongg deebe0e8a7 feat: write the constant shade every document lists
The pasted smooth shade object of lowest UID with a constant shading
becomes a PastedSmoothShade whose contents are its numbers in big-endian
order, with the values every constant shade has in IDML.

Value coverage on trustworthy pairs: 87.08 % to 87.13 %.
2026-10-07 14:19:32 +00:00
aiongg 3f1f10799d feat: write the InCopy assignment of every document
Each assignment object becomes an Assignment with the values every IDML
has. Its name depends on the language of the computer that exported the
IDML, so it is left out of the output and of value coverage.

Value coverage on trustworthy pairs: 86.83 % to 86.86 %.
2026-10-07 13:45:28 +00:00
aiongg 2d52b399c4 feat: write table of contents styles and the stories they made
Styles.xml gets a TOCStyle for each TOC style object, with its name,
title, title style and numbering, anchor and line break settings, and
stories get AppliedTOCStyle through the table of contents object that
made them. Entries are not decoded yet.

Value coverage on trustworthy pairs: 84.29 % to 84.50 %.
2026-10-07 12:41:09 +00:00
aiongg e6dd2d629f feat: write style import flags, unique IDs and common style values
Paragraph and character styles get Imported (the u16 after the style
kind) and StyleUniqueId (the GUID after the name) from the INDD, and the
values every non-root style has in IDML, such as PreviewColor, from
observation. The values generator leaves root styles out of the style
paths.

Value coverage on trustworthy pairs: 84.07 % to 84.29 %.
2026-10-07 12:14:48 +00:00
aiongg 09782f6af0 feat: write more text frame settings
TextFramePreference gets the maximum column width, fixed column width,
auto-sizing minimums, text wrap override, column rule width and colour,
footnote spacings and the inset spacing (from the frame), read from
their chunks. Attributes that IDML writes only on some frames, with one
value wherever it does, are generated as WhenWritten values and also
give the versions from which column rules and footnotes exist.

Value coverage on trustworthy pairs: 82.03 % to 83.37 %.
2026-10-07 11:34:20 +00:00
aiongg 722fddfe97 feat: write languages, document users and document settings
designmap.xml gets the document's languages from their INDD objects,
with the quotes every IDML gives each language, the document users from
the document object, and the settings elements every IDML of the version
has (watermark, linked story and page item options, footnote and
endnote options and others) with their common values. User colours are
set by the exporting InDesign, not the INDD, and are left out of value
coverage.

Value coverage on trustworthy pairs: 81.65 % to 82.03 %.
2026-10-07 11:08:03 +00:00
aiongg fbe1e7f6ca feat: write spread, page, section and layer settings
Spreads get AllowPageShuffle and their FlattenerPreference, master
spreads ShowMasterItems, pages their tab order, overrides, layout grid
use, layout rule, colour, alternate layout and numbering descriptor,
sections their prefix, marker and alternate layout, and layers their
printing, guide, UI, wrap and colour settings. Interface colours are
named from one table shared with XML tags. Values every IDML has on
these elements, and the two built-in trap presets, are written from
observation.

Value coverage on trustworthy pairs: 80.03 % to 81.65 %.
2026-10-07 10:45:11 +00:00
aiongg 3d968bea8a feat: write the settings every page item has
Page items get Name, Visible, Locked, the interface change counts,
OverriddenPageItemProps, the layout constraints and the gradient angles
and highlights, read from their chunks and attribute lists, with the IDML
defaults where a chunk is absent. Groups take the gradient values their
children share. tools/element_values.py collects the values every IDML
of a version has on an element; the converter writes those of page items
and their export options, now for every version.

Value coverage on trustworthy pairs: 62.62 % to 80.03 %.
2026-10-07 10:09:02 +00:00
aiongg c9528b10ff feat(tools): list reproduced values per key and dump whole chunks
compare.py writes values.tsv and values-all.tsv with every key's values and
how many are reproduced, so a change can be checked for keys that lose
reproduced values. indd dump --full prints every byte of each chunk.
root_values.py reads the current inventory layout and keeps to the pairs
its generated files were made from.
2026-10-07 09:39:02 +00:00
aiongg 5543af5867 feat: add indd audit and a corpus-wide audit ranking
indd audit converts a document while recording which objects, chunks and attribute IDs the converter reads, then lists what it did not read: classes, chunks, byte-stream objects, attribute IDs per kind of list, codes of known attributes without an IDML value, and strand run kinds, with counts and the conversion's warnings. --tsv gives rows for scripts. tools/audit_corpus.py audits every distinct corpus file and ranks the items by files affected. Without the audit, recording does nothing and conversion output is unchanged.
2026-10-07 08:48:34 +00:00
aiongg f18a29bce1 feat(tools): add value coverage headline and ranked gap list
compare.py ends with value coverage (the share of reference IDML values the conversion reproduces, weighted by occurrence), document scores at 99 % and 99.9 %, and the biggest gaps ranked by documents and by values, over trustworthy pairs and over all pairs. pairs.tsv, gaps.tsv and gaps-all.tsv go to target/compare/. Font Status and package part names are left out as values no INDD can supply; docs/measurement.md defines the metric and the exclusions.
2026-10-07 08:41:26 +00:00
aiongg 4024dd79a8 feat(tools): report stale corpus pairs separately
A pair is stale when the IDML's xmp:ModifyDate is later than the INDD's or more than an hour earlier, when the IDML names UIDs the INDD database lacks, or when the INDD has stories the IDML lacks. compare.py prints totals for all pairs and for trustworthy pairs. New subcommands indd xmp and indd uids supply the INDD side. docs/measurement.md records the rule and its evidence: 159 of 654 pairs are stale.
2026-10-07 08:34:18 +00:00
aiongg 54cf473c52 perf(tools): validate many IDML packages per Jing run
validate.sh takes several packages and validates their parts with one JVM per schema and batch of 400 parts, in parallel. compare.py validates its outputs 600 at a time instead of one validate.sh run per file. The full corpus validates in about 4.5 minutes instead of 45.
2026-10-07 08:19:57 +00:00
aiongg dc198e4ea2 feat(tools): convert in parallel, rank shortfalls by documents, find sources by SOURCE-*.md
compare.py runs conversions and validations in the --jobs pool and
removes each output once compared. --shortfalls N ranks missing
elements, wrong or missing attribute values and differing story text by
the number of documents they affect.

inventory.py also treats a folder with SOURCE-*.md (a supplementary
download) as a source folder when pairing files.
2026-10-07 07:59:10 +00:00
aiongg 0bdae12f3d feat(tools): pair IDML and PDF across source folders and dedupe by SHA-256
inventory.py now records the reference IDML and PDF of each INDD file,
its SHA-256 and the first file with the same digest. Outside the file's
own folder, a partner is searched under the nearest folder with a
SOURCE.md. compare.py takes the IDML path from the inventory and skips
duplicate files by digest.
2026-10-07 06:15:20 +00:00
aiongg bf72b93d23 test: download fixtures from pinned sources instead of committing them
Remove the sample documents, PDF and licence copies from tests/fixtures/.
tests/fixtures/manifest.json lists each file's download URL (pinned to a
commit for GitHub sources), size and SHA-256. tools/fetch_fixtures.py
downloads them into the git-ignored tests/fixtures/files/ and verifies
them. The fixture tests read from there and skip when it is absent.
2026-10-07 05:47:17 +00:00
aiongg a9985b3775 fix: validate every corpus output and fix the two schema failures
compare.py --all now also converts the INDD and INDT files that have no
same-version IDML (any version, either byte order), validates them and
counts converter warnings by kind. validate.sh runs Jing once per schema
instead of once per file, which makes validating the whole corpus take
about 3 minutes.

Two schema failures showed up in files without an IDML:

- Gradient swatch midpoints are stored as positions from the start of
  the gradient. IDML wants them relative to the gap between two stops,
  as for opacity gradient stops. Values outside 13-87 are left out with
  a warning.
- The style kind is a u16, not a u32. One InDesign 7.5 template has a
  non-zero u16 after it, which made 151 character styles paragraph
  styles.
2026-10-07 03:50:08 +00:00
aiongg 3f4a68dfe6 feat: write object style settings for every object style
Object styles other than [None] now get their text frame, story, text
wrap and frame fitting settings, fill, stroke and corner attributes,
applied paragraph style and some category flags from the INDD (chunks
0x1B92B, 0x1B924, 0x285B, 0x50F28, 0x3776, 0x3777, 0x1B92E, 0x1B946),
and the values every such style has in the corpus IDML files, collected
by tools/root_values.py into src/idml/object_style_values.xml. The root
style gets the same INDD values in place of observed ones.
2026-10-07 01:33:21 +00:00
aiongg 0dc40563ab feat: write preference values that every exported IDML has 2026-10-07 00:35:38 +00:00
aiongg 7c3c38446b feat(compare): compare effect elements inside transparency settings 2026-10-06 23:55:15 +00:00
aiongg 36b7ae9eaf feat: write root style values that every exported IDML has
Collect the root style attributes, Properties and child elements that are
the same in every corpus IDML (tools/root_values.py) and write them on the
root paragraph, character, object, cell and table styles. Values read from
the INDD take precedence; values that first appear after DOM 7 are gated
by InDesign version. Also write the default numbering list, and write a
style only once when two style objects share a name.
2026-10-06 22:04:05 +00:00
aiongg 7c95144d4a feat(compare): compare list properties and attribute-only properties
TabList, AllNestedStyles and other list-typed <Properties> children are
compared as one canonical string. Empty property elements such as
BulletChar are compared by their attributes.
2026-10-06 21:14:52 +00:00
aiongg fb2bc7cd9e chore: compare embedded contents by digest and add --bin option 2026-10-06 18:22:23 +00:00
aiongg 69573c8461 feat: convert document and text frame preferences
Write DocumentPreference (page size, facing pages, bleeds, intent) and
TextFramePreference (columns, gutter, column width, vertical
justification, auto-sizing). compare.py now also compares child elements
without Self and the preference elements.
2026-10-06 17:26:36 +00:00
aiongg 602065453e feat: validate output against the IDML schemas
Add tools/validate.sh, which checks every XML part of an IDML package
against InDesign's generated RelaxNG schemas with Jing, and a --schemas
option in compare.py. Output for all 75 corpus pairs validates.

Do not write BindingLocation on master spreads, which the schema does not
allow.
2026-10-06 17:11:34 +00:00
aiongg 5f87710d92 feat: convert text formatting
Parse attribute lists in style objects and in paragraph and character
style runs, and write 40 text attributes (font, style, size, leading,
colour, tracking, kerning method, indents, spacing, justification,
underline, rules, hyphenation and more) on styles and text ranges. Read
font family names.

tools/compare.py now also compares text range attributes at matching text
offsets; every mapped attribute matches the reference in all ranges.
2026-10-06 16:33:53 +00:00
aiongg 0da35fd644 feat: first INDD to IDML conversion
Build a document model (layers, spreads, master spreads, pages, page
items with path geometry, text frame threading, stories with paragraph
and character style runs, style names and hierarchy) and write it as an
IDML package with a minimal stored-ZIP writer.

Add tools/compare.py, which converts every corpus pair and compares the
result with the reference IDML. First run: all 351 little-endian files
convert; 921 of 1120 stories match exactly.
2026-10-06 16:17:00 +00:00
aiongg 8d156bfeba feat: read INDD header and set up clean-room project
Decode the signature, byte-order flag and InDesign version from the INDD
header, add an `indd info` command, and add smoke tests on four
open-licensed sample files plus an optional local corpus test.

Document the clean-room method and the header layout with the evidence
for each field.
2026-10-06 15:22:17 +00:00