text_ranges read only ParagraphStyleRange children of the story and their
CharacterStyleRange children. Ranges inside HyperlinkTextSource,
CrossReferenceSource, XMLElement, Change and EndnoteRange were skipped,
and their text was left out of the offsets, so every later range of the
story started earlier in the reference than in the output. It now walks
through those wrappers at any depth.
Trustworthy pairs (compare.py --exclude own/):
- hyperlink and cross-reference sources alone: reference values
11,142,835 -> 11,158,348, reproduced 10,501,458 -> 10,505,081,
extra values 30,237 -> 26,634
- with XMLElement, Change and EndnoteRange (counted as the headline must
include everything InDesign writes): reference values 11,508,691,
coverage 94.24 % -> 91.28 %; 348,306 of the 350,343 new values are in
one document
- story text unchanged (18,517 of 18,610); documents >= 99 %: 18
- three keys reproduce one value fewer (CSR.Underline, PSR.Justification,
PSR.P.TabList): ranges that matched at a shifted offset before
IDML adds " (OTF)", " (TT)" or " (T1)" to some family names depending
on the exporting computer. compare.py removed it from family names but
not from Font Self keys and Font Name, so each such font counted as
missing and the converter's font as extra. The suffix is now removed
from the family part of both.
Trustworthy pairs: extra Font values 17,889 -> 5,146 (-12,743); the
same fonts now count as reproduced.
compare.py counts values the output has and the reference does not
(extra values, extras.tsv), flags pairs whose IDML names UIDs deleted in
the INDD as stale, leaves out link times and BitmapPrinting, skips the
resource values of links resolved to another file at export, and
compares font family names without the technology suffix.
Trustworthy pairs: 495 -> 489; value coverage 88.71% -> 88.71%
(9,885,118 of 11,142,835); extra values 713,688 (6.40%).
The pasted smooth shade object of lowest UID with a constant shading
becomes a PastedSmoothShade whose contents are its numbers in big-endian
order, with the values every constant shade has in IDML.
Value coverage on trustworthy pairs: 87.08 % to 87.13 %.
Each assignment object becomes an Assignment with the values every IDML
has. Its name depends on the language of the computer that exported the
IDML, so it is left out of the output and of value coverage.
Value coverage on trustworthy pairs: 86.83 % to 86.86 %.
Styles.xml gets a TOCStyle for each TOC style object, with its name,
title, title style and numbering, anchor and line break settings, and
stories get AppliedTOCStyle through the table of contents object that
made them. Entries are not decoded yet.
Value coverage on trustworthy pairs: 84.29 % to 84.50 %.
Paragraph and character styles get Imported (the u16 after the style
kind) and StyleUniqueId (the GUID after the name) from the INDD, and the
values every non-root style has in IDML, such as PreviewColor, from
observation. The values generator leaves root styles out of the style
paths.
Value coverage on trustworthy pairs: 84.07 % to 84.29 %.
TextFramePreference gets the maximum column width, fixed column width,
auto-sizing minimums, text wrap override, column rule width and colour,
footnote spacings and the inset spacing (from the frame), read from
their chunks. Attributes that IDML writes only on some frames, with one
value wherever it does, are generated as WhenWritten values and also
give the versions from which column rules and footnotes exist.
Value coverage on trustworthy pairs: 82.03 % to 83.37 %.
designmap.xml gets the document's languages from their INDD objects,
with the quotes every IDML gives each language, the document users from
the document object, and the settings elements every IDML of the version
has (watermark, linked story and page item options, footnote and
endnote options and others) with their common values. User colours are
set by the exporting InDesign, not the INDD, and are left out of value
coverage.
Value coverage on trustworthy pairs: 81.65 % to 82.03 %.
Spreads get AllowPageShuffle and their FlattenerPreference, master
spreads ShowMasterItems, pages their tab order, overrides, layout grid
use, layout rule, colour, alternate layout and numbering descriptor,
sections their prefix, marker and alternate layout, and layers their
printing, guide, UI, wrap and colour settings. Interface colours are
named from one table shared with XML tags. Values every IDML has on
these elements, and the two built-in trap presets, are written from
observation.
Value coverage on trustworthy pairs: 80.03 % to 81.65 %.
Page items get Name, Visible, Locked, the interface change counts,
OverriddenPageItemProps, the layout constraints and the gradient angles
and highlights, read from their chunks and attribute lists, with the IDML
defaults where a chunk is absent. Groups take the gradient values their
children share. tools/element_values.py collects the values every IDML
of a version has on an element; the converter writes those of page items
and their export options, now for every version.
Value coverage on trustworthy pairs: 62.62 % to 80.03 %.
compare.py writes values.tsv and values-all.tsv with every key's values and
how many are reproduced, so a change can be checked for keys that lose
reproduced values. indd dump --full prints every byte of each chunk.
root_values.py reads the current inventory layout and keeps to the pairs
its generated files were made from.
indd audit converts a document while recording which objects, chunks and attribute IDs the converter reads, then lists what it did not read: classes, chunks, byte-stream objects, attribute IDs per kind of list, codes of known attributes without an IDML value, and strand run kinds, with counts and the conversion's warnings. --tsv gives rows for scripts. tools/audit_corpus.py audits every distinct corpus file and ranks the items by files affected. Without the audit, recording does nothing and conversion output is unchanged.
compare.py ends with value coverage (the share of reference IDML values the conversion reproduces, weighted by occurrence), document scores at 99 % and 99.9 %, and the biggest gaps ranked by documents and by values, over trustworthy pairs and over all pairs. pairs.tsv, gaps.tsv and gaps-all.tsv go to target/compare/. Font Status and package part names are left out as values no INDD can supply; docs/measurement.md defines the metric and the exclusions.
A pair is stale when the IDML's xmp:ModifyDate is later than the INDD's or more than an hour earlier, when the IDML names UIDs the INDD database lacks, or when the INDD has stories the IDML lacks. compare.py prints totals for all pairs and for trustworthy pairs. New subcommands indd xmp and indd uids supply the INDD side. docs/measurement.md records the rule and its evidence: 159 of 654 pairs are stale.
validate.sh takes several packages and validates their parts with one JVM per schema and batch of 400 parts, in parallel. compare.py validates its outputs 600 at a time instead of one validate.sh run per file. The full corpus validates in about 4.5 minutes instead of 45.
compare.py runs conversions and validations in the --jobs pool and
removes each output once compared. --shortfalls N ranks missing
elements, wrong or missing attribute values and differing story text by
the number of documents they affect.
inventory.py also treats a folder with SOURCE-*.md (a supplementary
download) as a source folder when pairing files.
inventory.py now records the reference IDML and PDF of each INDD file,
its SHA-256 and the first file with the same digest. Outside the file's
own folder, a partner is searched under the nearest folder with a
SOURCE.md. compare.py takes the IDML path from the inventory and skips
duplicate files by digest.
Remove the sample documents, PDF and licence copies from tests/fixtures/.
tests/fixtures/manifest.json lists each file's download URL (pinned to a
commit for GitHub sources), size and SHA-256. tools/fetch_fixtures.py
downloads them into the git-ignored tests/fixtures/files/ and verifies
them. The fixture tests read from there and skip when it is absent.
compare.py --all now also converts the INDD and INDT files that have no
same-version IDML (any version, either byte order), validates them and
counts converter warnings by kind. validate.sh runs Jing once per schema
instead of once per file, which makes validating the whole corpus take
about 3 minutes.
Two schema failures showed up in files without an IDML:
- Gradient swatch midpoints are stored as positions from the start of
the gradient. IDML wants them relative to the gap between two stops,
as for opacity gradient stops. Values outside 13-87 are left out with
a warning.
- The style kind is a u16, not a u32. One InDesign 7.5 template has a
non-zero u16 after it, which made 151 character styles paragraph
styles.
Object styles other than [None] now get their text frame, story, text
wrap and frame fitting settings, fill, stroke and corner attributes,
applied paragraph style and some category flags from the INDD (chunks
0x1B92B, 0x1B924, 0x285B, 0x50F28, 0x3776, 0x3777, 0x1B92E, 0x1B946),
and the values every such style has in the corpus IDML files, collected
by tools/root_values.py into src/idml/object_style_values.xml. The root
style gets the same INDD values in place of observed ones.
Collect the root style attributes, Properties and child elements that are
the same in every corpus IDML (tools/root_values.py) and write them on the
root paragraph, character, object, cell and table styles. Values read from
the INDD take precedence; values that first appear after DOM 7 are gated
by InDesign version. Also write the default numbering list, and write a
style only once when two style objects share a name.
TabList, AllNestedStyles and other list-typed <Properties> children are
compared as one canonical string. Empty property elements such as
BulletChar are compared by their attributes.
Write DocumentPreference (page size, facing pages, bleeds, intent) and
TextFramePreference (columns, gutter, column width, vertical
justification, auto-sizing). compare.py now also compares child elements
without Self and the preference elements.
Add tools/validate.sh, which checks every XML part of an IDML package
against InDesign's generated RelaxNG schemas with Jing, and a --schemas
option in compare.py. Output for all 75 corpus pairs validates.
Do not write BindingLocation on master spreads, which the schema does not
allow.
Parse attribute lists in style objects and in paragraph and character
style runs, and write 40 text attributes (font, style, size, leading,
colour, tracking, kerning method, indents, spacing, justification,
underline, rules, hyphenation and more) on styles and text ranges. Read
font family names.
tools/compare.py now also compares text range attributes at matching text
offsets; every mapped attribute matches the reference in all ranges.
Build a document model (layers, spreads, master spreads, pages, page
items with path geometry, text frame threading, stories with paragraph
and character style runs, style names and hierarchy) and write it as an
IDML package with a minimal stored-ZIP writer.
Add tools/compare.py, which converts every corpus pair and compares the
result with the reference IDML. First run: all 351 little-endian files
convert; 921 of 1120 stories match exactly.
Decode the signature, byte-order flag and InDesign version from the INDD
header, add an `indd info` command, and add smoke tests on four
open-licensed sample files plus an optional local corpus test.
Document the clean-room method and the header layout with the evidence
for each field.