65 Commits
Author SHA1 Message Date
aiongg 27bf102912 tools: cache pair comparisons by content in compare.py --cache and speed up aggregation 2026-10-12 23:45:16 +00:00
aiongg bff03f709c tools: add check.py, a check run that reuses conversions, validations and renders
compare.py --cache keeps conversions by content SHA-256 and file name,
--valid-cache keeps schema validation results by part; render_compare.py
takes conversions from the same cache and reuses verdicts and
measurements with --reuse-results. Runs killed by a signal or a timeout
and failed renders are not kept.
2026-10-12 22:09:46 +00:00
aiongg d20d1fa3ed fix: leave out font axes and placed-text plugin data from the measurement 2026-10-12 18:04:47 +00:00
aiongg 032ab0b15c feat: write the story, link, contour, object style and HTML5 export values of DOM 21.1 to 21.5, and the anchored settings of every complete element
Stories and XML stories get the Ai* attributes from DOM 21.5; links the
Ai* attributes from 21.5 and SourceObjectName from 21.4; every typed
ContourOption the four contour parameters from 21.4; object styles
EpubAriaLabel and EpubAriaLabelSourceType from 21.2; designmap.xml
Html5ExportPreference from 21.1. LockPosition and AnchorSpaceAbove go on
every AnchoredObjectSetting with all ten settings, polygons included,
in place of the per-tag WhenWritten blocks.
2026-10-12 04:59:47 +00:00
aiongg 483442bd9e fix(compare): give the nested option table its own name
The table of nested option elements reused the name of the table of
story elements that text ranges skip, so text ranges inside footnotes,
table cells and notes were counted as story ranges.
2026-10-10 23:56:54 +00:00
aiongg a99131da31 fix(compare): count the ContourOption of text wrap settings
compare.py compared a child element's own children only when the child
had no attributes, so the ContourOption inside TextWrapPreference (which
has attributes) was never counted. It is the only such grandchild outside
stories; count it as Parent/TextWrapPreference/ContourOption, and the
one in Preferences.xml as TextWrapPreference/ContourOption.
2026-10-10 21:58:14 +00:00
aiongg a299387eb0 feat: decode page item, preference and resource values
Page items:
- drop shadow KnockedOut (0x1EB6F), true where neither item nor style
  stores it
- text wrap: the u32 at 40 of 0x3703 is the master page flag and the
  wrap side (two u16, as the big-endian pair shows)
- groups get the fill, stroke and corner values all their children have
- dashes (0x5A35: list and corner code) on items whose stroke type in
  effect is Dashed
- BeforeGroupingLayerPosition (0x324) from 21.4; export SizeType
  (0x1E22B); no generated alt text attributes on groups
- anchored settings on items inside anchored items
- IgnoreWrap false without the chunk, MinimumFirstBaselineOffset 0 by
  the baseline category, InsetSpacing before DOM 11 by the general
  category; EPS contents from 0x8C23
- custom striped and dashed stroke styles (classes 0xB016, 0x5A43) and
  StrokeType code 0x5A42

Preferences: column direction (0x555, else the story orientation),
optical margin size by the text size unit, highlight flags false without
their chunk, grid and gutter defaults and print marks by the edition
code, watermark font by the assignment name, EPUB ViewDocumentAfterExport
for DOM 8 and 9, HTMLFXLExportPreference from DOM 10.1, MathML document
values from DOM 20, index header type by variant name, three more
built-in kinsoku and mojikumi tables.

Resources and text: the saving edition from the XMP black name (tints
of black, unknown users, endnote and index titles, endnote marker), TOC
style entries, PageNumberType (0x1B09), CustomGlyph (0x1B5E), alignment
codes of 0x4266, 0x1B3E and 0xB676, spot colour alias and DOM 8 section
pagination (observed values), text variable settings and result texts,
object styles' AppliedParagraphStyle without a chunk and flex spacing,
stories of frames anchored in the XML backing story.

Fixes: flattener resolutions of 0x10833 were swapped; colour group Self
follows the DOM version.

compare.py, trustworthy pairs: value coverage 98.96 % -> 99.48 %
(+150,288 values), documents >= 99 % 914 -> 1,021, story text 37,129 ->
37,147 of 37,150 (0 missing), extras 92,838 -> 92,883 (+788 from the
KnockedOut and group-corner exceptions, -744 from fixed extras). No pair
loses values against f21267f. diff_outputs.py: 5,743 of 5,765 outputs
differ, no failure or warning differs.
2026-10-10 18:34:10 +00:00
aiongg f21267f1a8 fix(render_compare): correct line-box offsets per page
The document-wide median per pair of box heights follows pagination
drift on other pages. Each page uses its own median for a pair with at
least 3 matched lines and falls back to the document's median otherwise.
Add --reuse DIR to measure the tool on the renders of an earlier run.
2026-10-10 15:13:03 +00:00
aiongg 58da17b51e feat: add tools/render_compare.py, a render comparison of same-save triples
For each INDD with a reference IDML and a PDF of the same save, compare
the InDesign PDF (R), DesignCraft's render of the reference IDML (A) and
DesignCraft's render of the converter's IDML (B) page by page. A vs B
measures the converter: pixel-identical documents and pages, and the
first differing page with compare.py's gap keys on it. R vs A measures
DesignCraft: issue kinds counted on font-complete documents and on all.

DesignCraft is an input program (--designcraft or DESIGNCRAFT_CLI).
Needs poppler-utils and the Python standard library only. Documented in
docs/measurement.md, "Render comparison", and in CONTRIBUTING.md.
2026-10-10 11:17:52 +00:00
aiongg 57b84c7687 refactor: share XMP reading in compare.py and add the same-save rule for PDFs
xmp_descriptions and xmp_value hold the packet parsing that modify_date
used. xmp_instances reads the instance IDs, and pdf_stale_reasons
applies the same-save rule for a PDF beside an INDD (source instance in
the INDD history, ModifyDate within an hour, page count, InDesign
creator). compare.py's own results do not change.
2026-10-10 11:17:52 +00:00
aiongg 1a2ce02c4f feat: read anchored object settings field by field
Chunk 0x2800 has one u16 per attribute: AnchoredPosition at 40,
HorizontalReferencePoint at 42, VerticalReferencePoint at 44, AnchorPoint
from 46 and 50, HorizontalAlignment at 48, VerticalAlignment at 52,
SpineRelative at 54 and PinPosition at 56, with the two offsets at 0 and
16. The grouped tables written before were the combinations the smaller
corpus showed. Every field matches the 6,363 object styles, 1,076
anchored items and 1,295 preference chunks of the corpus.

An anchored item gets every value when its object style turns the
anchored object category off (670 items, with LockPosition and
AnchorSpaceAbove from the value files), else the values that differ
from the style's. Preferences and object styles without the chunk get
the values every such file has.

compare.py --exclude own/, trustworthy pairs: 98.46 % -> 98.51 %
(+11,653 values, 0 lost on any key); ObjectStyle SpineRelative
2,031 -> 5,556 of 5,556, preference SpineRelative 0 -> 1,251 of 1,251,
anchored text frames, rectangles, ovals, polygons, lines and groups
gain their settings (TextFrame PinPosition 2 -> 556 of 556). Extra
values unchanged (93,502).
2026-10-10 09:04:37 +00:00
aiongg e4777b2a3c fix: accept InDesign's markup of form fields anchored in text
The IDML schema rejects a CheckBox, RadioButton, TextBox, ComboBox or
SignatureField inside a CharacterStyleRange, but InDesign writes
anchored form fields there, and the reference IDML of every such pair
has these errors. compare.py accepts them in a paired output when the
reference has the same errors, as it does for endnote markup, and lists
unpaired files with them.

With it, 0 paired outputs fail validation (3 without it); one unpaired
file fails for anchored form fields only.
2026-10-10 08:15:41 +00:00
aiongg 2a09aa923f feat: write form fields and multi-state objects
Buttons, check boxes, radio buttons and multi-state objects list their
states in chunk 0x14521 and the items each state shows in chunk 0x14526;
their transform is chunk 0x14527. Text boxes, combo boxes and signature
fields hold their items in chunk 0x15B. The converter writes the
elements with their State elements (type from the state ID), the groups
of states named by their type, PathBoundingBox from the items' paths,
and the name, description, export value, required flag, text box font
and flags from the attribute list (objects.md, form fields). The value
files gain the constant values of buttons and multi-state objects.

Text boxes anchored in table cells now replace their U+FFFC, so those
stories' text matches.

compare.py (trustworthy pairs, with the commits before it): coverage
97.92 % -> 98.46 %, story text 37,112 -> 37,129 of 37,150; the form
field elements of all 48 documents are written (Button, Group, State,
Rectangle and their children about 150,000 values). Extra values from
anchored fields without a known object style are avoided by leaving
their anchor settings out.
2026-10-10 07:25:35 +00:00
aiongg 01261ba47a feat: decode frame fitting, object style overprint, style IDs and export settings
- Frame fitting: AutoFit 1 (true), FittingAlignment 1 (TopCenterAnchor)
  and 3 (LeftCenterAnchor). A rectangle, oval or polygon whose object
  style turns the fitting category off (0x1B960 not in chunk 0x1B92E)
  gets all seven attributes, the item's or the style's; this holds for
  the root [None] too (22 trustworthy documents, 9,560 items; none in
  the 883 others). A style with the category on writes only the item's
  values that differ. The preference FrameFittingOption writes all seven,
  0 or code 0 where the page item defaults list lacks one (1,251 of
  1,251 pairs each).
- Object style OverprintFill, OverprintStroke and OverprintGap (0x6E6A,
  0x6E67, 0x6E8B) where the colour carries ink (not None, Paper, tint 0
  or a white process colour), on every style from 21.4.
- StyleUniqueId on every paragraph and character style when 11.2 saved
  the document: the GUID after the name's string (GUID form only), else
  $ID/. A 36-character style name is no longer taken for the GUID.
- Chunk 0x28F0 parsed with its export tag maps: SplitDocument, EmitCss
  (10.1 on) and IncludeClass (13.0 on) on every text style, root styles
  and styles with tag maps included, EmitCss and IncludeClass on object
  styles; StyleExportTagMap elements. The WhenWritten values of these
  attributes are dropped from element_values.xml.

Evidence in objects.md, preferences.md and idml-values.md.

compare.py --exclude own/, trustworthy pairs: 97.60 % -> 97.84 %
(+71,229 values, 0 lost on any key); Oval/Polygon/Rectangle
FrameFittingOption AutoFit 32,300 -> 44,868 of 44,868, preference
FrameFittingOption 1,251 of 1,251 per attribute, StyleUniqueId 6,424 ->
10,047 of 10,051, ObjectStyle OverprintStroke 0 -> 1,512 of 1,512,
StyleExportTagMap 0 -> 306 of 306 elements, export flags of every
style. Extras 94,157 -> 93,569: 444 FrameFittingOption elements and 168
SplitDocument and EmitCss values gone; 24 StyleUniqueId values are new,
in five 11.4 pairs whose IDML was exported by 11.0.
2026-10-10 05:47:49 +00:00
aiongg 10b008e77a feat: regenerate the value files over the larger corpus
tools/root_values.py collects the object styles other than [None] and
the preferences from the pairs compare.py compares, like the root
styles, instead of the part of the corpus it had first: 1,585 pairs,
1,448 distinct IDML files. tools/element_values.py reads 1,495 distinct
IDML files. The DOM 8 to 11 merge of the DOM 12 object style frame
values is gone: the file has those versions itself.

Values that vary in the larger corpus are dropped, also where only one
document differs and the converter does not read the value from the
INDD: 209 keys, most of them preferences (EPubExportPreference,
TextDefault, PageItemDefault, IndexOptions, TextPreference), object
style values (line ends, gap colour, stroke tint, anchored object and
text wrap settings, baseline frame grid before DOM 18), Spread
PageTransitionType and PageTransitionDirection, and character style
PreviewColor. Evidence in idml-values.md.

compare.py --exclude own/, trustworthy pairs: coverage 97.79 % ->
97.09 % (-212,444 values on dropped keys, +11,501 on keys whose first
version is now earlier); documents >= 99 % 514 -> 106 of 1,251; all
pairs 96.55 % -> 95.85 %. Extras 94,610 -> 94,328. Every lost value is
on a regenerated key. No new failures; story text unchanged.
2026-10-10 02:08:29 +00:00
aiongg 180599cd72 feat: add the Phinney world-ready template and five Zenodo records to the fetch manifest 2026-10-10 01:28:40 +00:00
aiongg 39e81d0247 feat: generate the root style values from the compared pairs
tools/root_values.py collects the root styles from the pairs that
compare.py compares (little-endian, same major version, without the
privately held samples): 794 distinct IDML files, DOM 7 to 21, instead
of the 88 the corpus had first. The pair rule leaves out an IDML with no
root styles beside a newer INDD and a big-endian pair. The other object
styles and the preferences stay pinned to SOURCE; their files are
unchanged. The root table style writes every mapped attribute of its
list, since the stroke weights, spacing and end tints now vary between
files and are no longer in the values file. The root object style's
varying values (named grid, optical margin size, baseline frame grid
colour, corner radii, gutter, first baseline offset) are read from the
INDD. Evidence in idml-values.md and tables.md.

compare.py --exclude own/, trustworthy pairs: coverage 98.53 % ->
98.56 % (+2,942 values whose first DOM version is now earlier:
IncludeClass, EpubAriaRole, paragraph border options,
ParagraphKashidaWidth, shading overprint, TransformAttributeOption and
FlexLayoutAttributeOption), documents >= 99 % 278 -> 313; all pairs
96.65 % -> 96.67 %. No pair loses a reproduced value; extras unchanged
(8,968 trustworthy, 278,333 all). No new failures.
2026-10-09 19:46:58 +00:00
aiongg 15d6af5eee feat: write the grid starting point, TOC story direction and object style optical margin
- Page GridStartingPoint from the u32 at 0 of chunk 0xCD04 (0
  TopOutside, 4 CenterVertical; other codes are in no pair and are
  left out).
- TOCStyle SetStoryDirection from the first u16 field of chunk 0x11605
  (0 Horizontal, 1 Vertical). It is no longer a WHEN_WRITTEN key of
  element_values.py (the regenerated file is unchanged).
- ObjectStyle StoryPreference OpticalMarginSize and
  OpticalMarginAlignment from the f64 at 2 and u16 at 12 of chunk
  0x285B, as in the story chunk 0x2EE.

Evidence in objects.md.

compare.py --exclude own/, trustworthy pairs: coverage 98.32 % ->
98.41 % (Page GridStartingPoint 10,080 missing -> 0, TOCStyle
SetStoryDirection 620 missing -> 0, object style OpticalMarginSize 261
and OpticalMarginAlignment 4 wrong -> 0), documents >= 99 % 225 -> 268;
all pairs 96.46 % -> 96.54 %. Extras unchanged on trustworthy pairs
(8,968); over all pairs +182, the new attribute on pages of stale pairs
that the reference does not have. No new failures.
2026-10-09 18:51:48 +00:00
aiongg dd6521ad68 feat: report rejected files apart from conversion failures
Files the converter rejects for a correct reason (not an INDD file,
InDesign 1.x, truncated, no object database) are not failures.
compare.py now counts them as rejected files, by kind, separately from
conversion failures (valid files whose conversion fails), in its
output and in summary.json (`rejected` next to `failures`), and
readme_numbers.py writes a "Rejected files" row. measurement.md
describes the split and marks which of its counts were measured on the
654 pairs of the earlier corpus, with the stale-pair counts of the
corpus of 2026-10 (197 of 803 stale, 606 trustworthy).

compare.py --all: 0 conversion failures of 803 paired and 3,656 other
files; 22 rejected files (11 not INDD files, 8 InDesign 1.x files, 2
truncated, 1 without an object database).
2026-10-08 15:17:53 +00:00
aiongg 0453eb99c5 feat: add a fetch manifest and tool for the openly licensed corpus
tools/corpus-manifest.json lists 2,884 files (5.5 GB) of 239 openly
licensed sources: InDesign, IDML, ICML and PDF files, the parts of
unzipped IDML packages, and the licence files. Each file has its
source, SPDX licence, size and SHA-256, and either a pinned URL (git
commit, Zenodo record, and a few unversioned URLs that the hash
guards) or a member of an archive with its own size and checksum.
Sources whose only licence is the OFL, Figshare records and files that
can only be downloaded by hand are left out.

tools/fetch_corpus.py downloads them into the git-ignored corpus/open/
as fetch_fixtures.py does: own download directory, size and hash
checks, atomic writes, paths kept inside the directory. Archives are
downloaded once, checked and unpacked member by member; requests to a
host are paced and retried on HTTP 429 and 5xx.
2026-10-08 15:08:31 +00:00
aiongg 83a6a7137a feat: convert EPS text
Read class 0x660B (text kept from a placed EPS or PDF graphic) as a
page item and write it as EPSText: ItemTransform, PathBoundingBox
(chunk 0x154), EPSTextData (chunk 0x6611 in big-endian order without
two tail fields, base64), EPSTextAttributeBounds (chunk 0x6612) and
the text wrap. element_values.py collects the EPSText values every
element has.

compare.py --exclude own/:
- trustworthy 97.72% -> 97.90% (+22,108 values; EPSText 633 of 633
  elements, with transform, bounds, text data and text wrap all equal)
- all pairs 95.89% -> 96.02%
- extra values unchanged in trustworthy pairs (26,256); all pairs
  304,533 -> 305,985: 1 stale pair whose IDML lacks its EPS text
- not yet reproduced: EPSText StrokeType (634), OverprintFill (33),
  StrokeDashAndGap (7)
2026-10-08 14:02:35 +00:00
aiongg d34e22bea2 chore: regenerate the element values from the larger corpus
Regenerate src/idml/element_values.xml from the 827 distinct IDML files
of the corpus (683 before). Values that now vary across the IDML files
drop out:

- Story AppliedNamedGrid, StoryPreference FrameType and
  StoryOrientation (stories with named grids, frame grids or vertical
  text)
- Page GridStartingPoint (28 pages of DOM 18)
- TOCStyle SetStoryDirection (3 styles are Vertical)
- 9 TextDefault, 2 CjkGridPreference and 1 IndexOptions preference values

element_values.py now collects ClippingPathSettings only for
ClippingType="None", the only case in which the converter writes the
element, and leaves user-named trap presets out of the TrapPreset list.
Without these two rules a DOM 12 file with its own trap preset and 8
images with a clipping path would have removed the values for every
document.

root_values.py read the text attribute table from src/idml/mod.rs,
which moved to attrs.rs; it now runs again. Its files stay pinned and
are not regenerated (idml-values.md says why).

compare.py --exclude own/ (803 pairs, 606 trustworthy):
- trustworthy coverage 97.95% -> 97.55% (12,036,295 -> 11,986,627 of
  12,287,602); all pairs 96.12% -> 95.72%
- the loss is the dropped Story AppliedNamedGrid (19,416), FrameType
  (19,368), Page GridStartingPoint (10,052), TOCStyle
  SetStoryDirection (618) and StoryOrientation (214)
- extra values 26,257 -> 26,256 (all pairs 305,366 -> 303,866)
- conversion failures 0 of 803
2026-10-08 13:04:04 +00:00
aiongg 64f08256e3 feat: take story, frame, clipping and ink constants from the value files
The writer wrote these as literals. element_values.py now collects
Story, XmlStory, Story/StoryPreference, the InCopyExportOption of
stories, rectangles, ovals and polygons, the ClippingPathSettings of
images, PDF and EPS graphics, and Ink; the writer reads them from
element_values.xml after the values it reads from the INDD.

- UserText (a WhenWritten value) is written from version 11.2, where
  IDML starts to have it (was 12).
- Ink ConvertToProcess is read from byte 56 after the ink name; it was
  the only ink value that varies (3 inks true).
- InCopyExportOption of ovals and the EPS clipping values follow the
  versions the corpus shows (ovals from DOM 8, EPS up to DOM 20).

compare.py --exclude own/:
- trustworthy value coverage 98.03% -> 98.03% (+771 values: UserText
  768, ConvertToProcess 3); all pairs 96.12% -> 96.13% (+1,211)
- extra values: trustworthy unchanged; all pairs +43 (UserText in
  stale pairs whose IDML is older)
- compare.py --all with schemas: 0 schema failures on pairs; files
  without a reference as before
2026-10-08 08:21:32 +00:00
aiongg 651cc4604e feat: keep the XML import and dictionary constants in the value files
XMLImportPreference AllowTransform, ImportCALSTables,
ImportTextIntoTables, ImportToSelected, RemoveUnmatchedExisting,
RepeatTextElements and DictionaryPreference MergeUserDictionary are the
same in all 654 pairs. Two of the 683 IDML files differ; both were
exported by an earlier version than their INDD was last saved with, so
they are not pairs. element_values.py now leaves those files out for
these keys (LATER_INDD_EXCEPTIONS); idml-values.md records why.

compare.py --exclude own/: trustworthy value coverage 97.12% -> 97.15%
(+3,423 values, 7 keys x 489 pairs); all pairs 95.16% -> 95.19%
(+4,578). No key lost values; extra values unchanged.
2026-10-08 05:45:37 +00:00
aiongg f6dc474e61 chore: count indentation-only property text as empty in compare.py
InDesign writes some empty Properties children open and closed on two
lines, so their text is a line feed and tabs
(XMLImportPreference/Properties/TransformParameters). compare.py now
treats such text as empty. measurement.md says so.

Effect (measurement only, same output): P.TransformParameters
reproduced in 489 of 489 trustworthy pairs. Trustworthy value coverage
97.12% -> 97.12% (+489 values); all pairs +654.
2026-10-08 05:45:37 +00:00
aiongg 930b7a88cc chore: keep only differing outputs in diff_outputs.py
Each run kept every converted package for both revisions, about 18 GB
over the corpus. Now the outputs of a file are deleted once they compare
equal, each run starts with empty output folders, and a commit's build
directory is deleted after its binary is copied.
2026-10-08 04:55:08 +00:00
aiongg 4ce9073dfa chore: refresh the README numbers from compare.py's summary
compare.py now also writes target/compare/summary.json: the options of
the run (--all, schema validation, --exclude, --limit, --file), the
files, conversion failures (error messages) and schema validation
failures of the paired and the other files, and the values, reproduced
values and story counts of the trustworthy and of all pairs.

tools/readme_numbers.py rewrites the "Current numbers" table of the
README, now between `<!-- numbers:start -->` and `<!-- numbers:end -->`,
from that summary. It refuses a summary from a run that included own/,
covered part of the corpus, or ran without --all or schema validation.
With --schemas and --jing it first builds the converter and runs
`compare.py --all --exclude own/` with validation, so one command
refreshes the README; --check only reports whether it is current.

The numbers in the README are unchanged.
2026-10-08 04:18:22 +00:00
aiongg c010bedef4 chore: check the WebAssembly build of the library
tools/check_wasm.sh builds the library for wasm32-unknown-unknown,
then builds a probe module (a cdylib generated under the target
directory that exports a function calling `indd::convert`) and checks
with Node.js that the module imports nothing: a conversion needs no
file system, clock or other host function. It then converts the
fetched fixtures, or the files given, in the probe module and compares
each package byte for byte with the native `indd convert` output.

No library change was needed: only `convert_file` and `read_header`
use `std::fs`, and `convert` does not call them. The 5 fixtures and 80
corpus files give the same packages in WebAssembly as natively. The
check takes about 10 s; it is documented in CLAUDE.md, CONTRIBUTING.md,
README.md and ARCHITECTURE.md. There is no hosted CI.

Idea from storytold/designcraft#98.
2026-10-08 04:18:22 +00:00
aiongg b278998860 fix(compare): count the text ranges of footnotes, table cells and notes
Ranges inside Footnote, Cell and Note elements were not counted: the
element counted as one position of the story text and its own ranges
were skipped. Each now forms a text flow of its own, with offsets from
0, compared when the story's text is identical (it already includes
their text). The headline counts everything InDesign writes.

Trustworthy pairs: reference values 11,508,691 -> 11,632,968,
reproduced 10,669,264 -> 10,772,444, coverage 92.71 % -> 92.60 %; extra
values unchanged (26,107); no key reproduces fewer values.
2026-10-08 03:12:18 +00:00
aiongg 649e80ea6f feat: write footnotes, endnotes and their options
- Footnotes: text owned by a footnote (class 0x24F) is read into the
  footnote and written as a Footnote element in place of its U+0004;
  hyperlink sources inside footnotes are placed like the others.
- FootnoteOption from preferences chunk 0x2820 (styles, numbering,
  marker, spacing, strings, splitting, straddling, both rules); without
  the chunk, the text style and straddling every IDML has. The value
  file reader decodes numeric character references, which fixes the
  SeparatorText tab.
- EndnoteOption from chunk 0x2261E; default marker and text styles
  without it. IsEndnoteStory from DOM 13 instead of 15.
- Endnotes: the endnote story (class 0x2801) is converted, references
  (class 0x2805) become Endnote elements, ranges (class 0x2804, in the
  range tree) EndnoteRange, and its frames EndnoteTextFrame, as InDesign
  writes them.
- compare.py accepts the schema errors of that endnote markup when the
  reference IDML of the pair has the same errors (docs/measurement.md).

Trustworthy pairs: reproduced 10,579,111 -> 10,650,374 (91.92 % ->
92.54 %); stories with identical text 18,565 -> 18,574 (0 missing);
documents >= 99 % 20 -> 21; TextRange +62,059, FootnoteOption and
IsEndnoteStory complete, EndnoteOption +1,356, hyperlinks in footnotes
+271. Extra values 26,179 -> 26,107. No key reproduces fewer values.
Schema validation: 0 failures; 1 pair accepted by the endnote rule; 11
files without a reference fail only on endnote markup.
2026-10-08 02:56:03 +00:00
aiongg d4f3fd5a64 fix(compare): count no text positions outside character ranges
The walker of the previous commit counted every element outside a
character range, such as StoryPreference, as a text position. A story
whose reference has MetadataPacketPreference, which the converter does
not write, then had all its range offsets shifted by one. Only Content
and elements inside a character range count now.

Trustworthy pairs at HEAD: reproduced 10,578,767 -> 10,579,111 (91.92 %),
extra values 26,516 -> 26,179; reference values unchanged. The numbers
in docs/measurement.md for the measurement change are re-measured with
the corrected rule (10,505,425 reproduced, 26,297 extra values). No key
reproduces fewer values.
2026-10-08 02:41:23 +00:00
aiongg 52a32e9b85 fix(compare): count text ranges inside wrapper elements
text_ranges read only ParagraphStyleRange children of the story and their
CharacterStyleRange children. Ranges inside HyperlinkTextSource,
CrossReferenceSource, XMLElement, Change and EndnoteRange were skipped,
and their text was left out of the offsets, so every later range of the
story started earlier in the reference than in the output. It now walks
through those wrappers at any depth.

Trustworthy pairs (compare.py --exclude own/):
- hyperlink and cross-reference sources alone: reference values
  11,142,835 -> 11,158,348, reproduced 10,501,458 -> 10,505,081,
  extra values 30,237 -> 26,634
- with XMLElement, Change and EndnoteRange (counted as the headline must
  include everything InDesign writes): reference values 11,508,691,
  coverage 94.24 % -> 91.28 %; 348,306 of the 350,343 new values are in
  one document
- story text unchanged (18,517 of 18,610); documents >= 99 %: 18
- three keys reproduce one value fewer (CSR.Underline, PSR.Justification,
  PSR.P.TabList): ranges that matched at a shifted offset before
2026-10-08 00:51:12 +00:00
aiongg 48d184f225 fix(compare): match fonts whose family has the technology suffix
IDML adds " (OTF)", " (TT)" or " (T1)" to some family names depending
on the exporting computer. compare.py removed it from family names but
not from Font Self keys and Font Name, so each such font counted as
missing and the converter's font as extra. The suffix is now removed
from the family part of both.

Trustworthy pairs: extra Font values 17,889 -> 5,146 (-12,743); the
same fonts now count as reproduced.
2026-10-07 21:53:14 +00:00
aiongg a82f3a849e feat: count extra values and leave out values of the exporting computer
compare.py counts values the output has and the reference does not
(extra values, extras.tsv), flags pairs whose IDML names UIDs deleted in
the INDD as stale, leaves out link times and BitmapPrinting, skips the
resource values of links resolved to another file at export, and
compares font family names without the technology suffix.

Trustworthy pairs: 495 -> 489; value coverage 88.71% -> 88.71%
(9,885,118 of 11,142,835); extra values 713,688 (6.40%).
2026-10-07 18:47:44 +00:00
aiongg c787337d89 feat: add tools/diff_outputs.py to compare the output of two revisions 2026-10-07 18:18:06 +00:00
aiongg 3da31dc6c2 feat: write style keyboard shortcuts and empty nested, line and GREP style flags 2026-10-07 16:05:18 +00:00
aiongg 1e26e1e8bd feat: read text, margin, anchor, pasteboard, XML tag and page item defaults 2026-10-07 15:40:54 +00:00
aiongg deebe0e8a7 feat: write the constant shade every document lists
The pasted smooth shade object of lowest UID with a constant shading
becomes a PastedSmoothShade whose contents are its numbers in big-endian
order, with the values every constant shade has in IDML.

Value coverage on trustworthy pairs: 87.08 % to 87.13 %.
2026-10-07 14:19:32 +00:00
aiongg 3f1f10799d feat: write the InCopy assignment of every document
Each assignment object becomes an Assignment with the values every IDML
has. Its name depends on the language of the computer that exported the
IDML, so it is left out of the output and of value coverage.

Value coverage on trustworthy pairs: 86.83 % to 86.86 %.
2026-10-07 13:45:28 +00:00
aiongg 2d52b399c4 feat: write table of contents styles and the stories they made
Styles.xml gets a TOCStyle for each TOC style object, with its name,
title, title style and numbering, anchor and line break settings, and
stories get AppliedTOCStyle through the table of contents object that
made them. Entries are not decoded yet.

Value coverage on trustworthy pairs: 84.29 % to 84.50 %.
2026-10-07 12:41:09 +00:00
aiongg e6dd2d629f feat: write style import flags, unique IDs and common style values
Paragraph and character styles get Imported (the u16 after the style
kind) and StyleUniqueId (the GUID after the name) from the INDD, and the
values every non-root style has in IDML, such as PreviewColor, from
observation. The values generator leaves root styles out of the style
paths.

Value coverage on trustworthy pairs: 84.07 % to 84.29 %.
2026-10-07 12:14:48 +00:00
aiongg 09782f6af0 feat: write more text frame settings
TextFramePreference gets the maximum column width, fixed column width,
auto-sizing minimums, text wrap override, column rule width and colour,
footnote spacings and the inset spacing (from the frame), read from
their chunks. Attributes that IDML writes only on some frames, with one
value wherever it does, are generated as WhenWritten values and also
give the versions from which column rules and footnotes exist.

Value coverage on trustworthy pairs: 82.03 % to 83.37 %.
2026-10-07 11:34:20 +00:00
aiongg 722fddfe97 feat: write languages, document users and document settings
designmap.xml gets the document's languages from their INDD objects,
with the quotes every IDML gives each language, the document users from
the document object, and the settings elements every IDML of the version
has (watermark, linked story and page item options, footnote and
endnote options and others) with their common values. User colours are
set by the exporting InDesign, not the INDD, and are left out of value
coverage.

Value coverage on trustworthy pairs: 81.65 % to 82.03 %.
2026-10-07 11:08:03 +00:00
aiongg fbe1e7f6ca feat: write spread, page, section and layer settings
Spreads get AllowPageShuffle and their FlattenerPreference, master
spreads ShowMasterItems, pages their tab order, overrides, layout grid
use, layout rule, colour, alternate layout and numbering descriptor,
sections their prefix, marker and alternate layout, and layers their
printing, guide, UI, wrap and colour settings. Interface colours are
named from one table shared with XML tags. Values every IDML has on
these elements, and the two built-in trap presets, are written from
observation.

Value coverage on trustworthy pairs: 80.03 % to 81.65 %.
2026-10-07 10:45:11 +00:00
aiongg 3d968bea8a feat: write the settings every page item has
Page items get Name, Visible, Locked, the interface change counts,
OverriddenPageItemProps, the layout constraints and the gradient angles
and highlights, read from their chunks and attribute lists, with the IDML
defaults where a chunk is absent. Groups take the gradient values their
children share. tools/element_values.py collects the values every IDML
of a version has on an element; the converter writes those of page items
and their export options, now for every version.

Value coverage on trustworthy pairs: 62.62 % to 80.03 %.
2026-10-07 10:09:02 +00:00
aiongg c9528b10ff feat(tools): list reproduced values per key and dump whole chunks
compare.py writes values.tsv and values-all.tsv with every key's values and
how many are reproduced, so a change can be checked for keys that lose
reproduced values. indd dump --full prints every byte of each chunk.
root_values.py reads the current inventory layout and keeps to the pairs
its generated files were made from.
2026-10-07 09:39:02 +00:00
aiongg 5543af5867 feat: add indd audit and a corpus-wide audit ranking
indd audit converts a document while recording which objects, chunks and attribute IDs the converter reads, then lists what it did not read: classes, chunks, byte-stream objects, attribute IDs per kind of list, codes of known attributes without an IDML value, and strand run kinds, with counts and the conversion's warnings. --tsv gives rows for scripts. tools/audit_corpus.py audits every distinct corpus file and ranks the items by files affected. Without the audit, recording does nothing and conversion output is unchanged.
2026-10-07 08:48:34 +00:00
aiongg f18a29bce1 feat(tools): add value coverage headline and ranked gap list
compare.py ends with value coverage (the share of reference IDML values the conversion reproduces, weighted by occurrence), document scores at 99 % and 99.9 %, and the biggest gaps ranked by documents and by values, over trustworthy pairs and over all pairs. pairs.tsv, gaps.tsv and gaps-all.tsv go to target/compare/. Font Status and package part names are left out as values no INDD can supply; docs/measurement.md defines the metric and the exclusions.
2026-10-07 08:41:26 +00:00
aiongg 4024dd79a8 feat(tools): report stale corpus pairs separately
A pair is stale when the IDML's xmp:ModifyDate is later than the INDD's or more than an hour earlier, when the IDML names UIDs the INDD database lacks, or when the INDD has stories the IDML lacks. compare.py prints totals for all pairs and for trustworthy pairs. New subcommands indd xmp and indd uids supply the INDD side. docs/measurement.md records the rule and its evidence: 159 of 654 pairs are stale.
2026-10-07 08:34:18 +00:00
aiongg 54cf473c52 perf(tools): validate many IDML packages per Jing run
validate.sh takes several packages and validates their parts with one JVM per schema and batch of 400 parts, in parallel. compare.py validates its outputs 600 at a time instead of one validate.sh run per file. The full corpus validates in about 4.5 minutes instead of 45.
2026-10-07 08:19:57 +00:00