Commit Graph
98 Commits
Author SHA1 Message Date
aiongg 09e87fadea feat: write page item values only where they differ from the object style
IDML leaves out the values of a page item that equal those of its
applied object style. The writer now compares with the style's
effective values (its own, then the styles it is based on, then [None]):

- TextFramePreference: an attribute is written when its category is off
  in the style's category list (chunk 0x1B92E, also for [None]) or its
  value differs; TextColumnMaxWidth (DOM 8), TextColumnCount (DOM 10)
  and InsetSpacing (DOM 11) are always written; footnote values from
  13.1 (objects.md, text frame preferences).
- Fill, stroke, miter and corner attributes (attributes.md).
- Transparency effects: only differing values, and an effect element
  only when it has one or its opacity stops differ (transparency.md).

New decoding: object style chunk 0x1B924 fields for balance, fixed
width, justification, first baseline offset (code 4 = XHeight, also for
frames) and auto-sizing; chunk 0x1B92C (style transparency). Object
styles get the DOM 12 text frame values from DOM 8 and
FootnotesEnableOverrides from 13.1 (idml-values.md).
2026-10-07 22:19:20 +00:00
aiongg 17c6ace935 feat: leave out values IDML does not write (layers, root styles, version gates)
Rules from the corpus, each documented in docs/format/:

- ItemLayer only on items directly under a spread (objects.md).
- The root object style [None] gets no Enable… attributes and no effects
  category settings; EnableTextFrameColumnRuleOptions from 15.1, and the
  footnote values of an object style's TextFramePreference from 13.1.
- The root paragraph style has no NextStyle and no AllNestedStyles;
  TextDefault has AllNestedStyles only when the list has items; the
  empty TabList is written from DOM 8 (attributes.md).
- DOM 7 has no EPubExportPreference or HTMLExportPreference.
- Row and column Self use a hexadecimal index (tables.md).
- Cell CellType from DOM 11; Document Name from DOM 13; ColorGroup Self
  is the UID before 11.3.
- The internal layer is the first of the document's layer list, whatever
  its name.
- A CR in a name is %0d in Self (colours, gradients, tints, hyperlink
  destinations, styles).

compare.py, trustworthy pairs (after the compare fix of 48d184f):
extra values 705,523 -> 635,511; value coverage 92.25 % -> 92.41 %
(10,296,599 values); 2 ItemLayer values lost (the 2 grouped items whose
IDML has one); 12 documents >= 99 %. No conversion or schema failures.
2026-10-07 22:01:28 +00:00
aiongg 48d184f225 fix(compare): match fonts whose family has the technology suffix
IDML adds " (OTF)", " (TT)" or " (T1)" to some family names depending
on the exporting computer. compare.py removed it from family names but
not from Font Self keys and Font Name, so each such font counted as
missing and the converter's font as extra. The suffix is now removed
from the family part of both.

Trustworthy pairs: extra Font values 17,889 -> 5,146 (-12,743); the
same fonts now count as reproduced.
2026-10-07 21:53:14 +00:00
aiongg 6511128e7f feat: write per-corner attributes and paragraph border and shading corners
Page items, object styles and PageItemDefault: each corner radius and
corner option from its own attribute ID (top left 0x6E70/0x6E6F, top
right 0x6E94/0x6E91, bottom left 0x6E92/0x6E8F, bottom right
0x6E93/0x6E90), with the corner codes InsetCorner, BevelCorner and
FancyCorner. Object styles no longer need all four corners equal.

Paragraph styles and text: ParagraphShadingTopLeft/BottomLeftCornerRadius
(0x1DF12, 0x1DF14); the tied groups of border corner radii, shading
right corner radii and shading corner options when their IDs agree; the
shading and border origin pairs for the known code pairs.

compare.py --all --exclude own/ (trustworthy pairs): value coverage
91.61% -> 92.25% (+71,545 values); documents >= 99 %: 5 -> 11; median
95.7% -> 96.3%; no key lower. Extra values 709,654 -> 718,266
(+8,612): the per-corner attributes follow the presence of CornerRadius
and CornerOption in the item's list, so each of the existing 2,055
extra CornerRadius values (items whose IDML leaves the value to the
object style) now has four per-corner extras, and likewise the
CornerOption extras. 0 conversion and 0 schema failures.
2026-10-07 21:26:29 +00:00
aiongg 5b7e0ecf1f feat: write link attributes and placed graphic settings
Links: LinkResourceModified, ShowInUI, LinkImportStamp and the two link
times from chunk 0x8C9B, PDFIdentifier from chunk 0x1B6, the size and
format name from the resource's chunk 0x8C92, and the values every link
has. Link times are written in the time zone of the XMP date nearest in
day of the year (the INDD has no time zone; compare.py does not score
them): 3,886 of 4,446 import times and 3,316 of 4,434 modification
times right.

Images, PDF, EPS and SVG graphics: the page item settings and object
style, Space, ActualPpi and EffectivePpi (chunk 0x1708 and the scale on
the spread), ImageTypeName, Profile (chunk 0x7C0F), fill colour and
gradients, vector colour policies (chunk 0x7C42 or the document's
policies), PDFAttribute (chunk 0x251B), GraphicLayerOption with the
GraphicLayer tree (chunk 0x177A) and LayerCompOption (chunk 0x9209),
and ImageIOPreference from chunk 0x1714 in place of chunk 0x8C39.
The placed graphic writer moves to idml/graphic.rs.

compare.py --all --exclude own/ (trustworthy pairs): value coverage
90.00% -> 91.61% (+179,834 values); documents >= 99 %: 1 -> 5; median
94.2% -> 95.7%; no key lower. Extra values 709,643 -> 709,654 (+11 in
SVG elements that are not in the reference). New warnings: 178 layer
chunks in 3 InDesign 3.0 files, whose layout differs, are left out.
0 conversion and 0 schema failures.
2026-10-07 21:13:31 +00:00
aiongg 9de24aa28f feat: read object export options of page items (chunk 0x1E206)
ObjectExportOption now takes its alternative text, actual text and
tagging values from chunk 0x1E206 of the item, and its other attributes
from a table by INDD major and minor version (8.0-9.1, 9.2, 10.0, 10.1
on, 21.0, 21.1, 21.4), for page items from version 8 and for every
object style, [None] included, from version 9. Page items of version 7
get no ObjectExportOption, as in their IDML.

compare.py --all --exclude own/ (trustworthy pairs): value coverage
88.72% -> 90.00% (+142,324 values); documents >= 99 %: 0 -> 1; median
92.4% -> 94.2%; no key lower. Extra values 713,688 -> 709,643: -4,319
from the DOM 7 rectangles that no longer get the element; +274 more
values in GraphicLine, Oval and TextFrame elements that are not in the
reference (more attributes per element). 0 conversion and 0 schema
failures.
2026-10-07 20:51:44 +00:00
aiongg 50638a4760 fix: write 17-digit numbers rounded to the nearest, ties to even
IDML writes the shortest decimal that reads back as the same value; when
that needs 17 digits it takes the 17-digit decimal nearest to the exact
value, with ties to even (150.23599243164062). Rust's shortest form can
end in the other digit (…063). num() now re-rounds 17-digit output with
{:.16e}.

compare.py --all --exclude own/ (trustworthy pairs): value coverage
88.72% (9,885,604 -> 9,885,761 values); PDF P.GraphicBounds +148,
EPS P.GraphicBounds +9; no key lower; extra values unchanged (713,688);
0 conversion and 0 schema failures.
2026-10-07 20:37:04 +00:00
aiongg f314de906d docs: fix stale attribute notes and misplaced doc comments 2026-10-07 20:00:23 +00:00
aiongg d205f6c851 fix: one corner option map for page items, defaults and object styles
Page items and page item defaults knew only code 0x5A15; object styles
also knew 0 (None) and 0x5A16 (InverseRoundedCorner). Using the object
style codes everywhere reproduces 486 more reference values in the
trustworthy pairs (489 of 489 page item defaults, 4 rectangles, 2 text
frames) and adds no extra values. Object styles write CornerOption in
the page item attribute order.
2026-10-07 19:50:06 +00:00
aiongg a82f3a849e feat: count extra values and leave out values of the exporting computer
compare.py counts values the output has and the reference does not
(extra values, extras.tsv), flags pairs whose IDML names UIDs deleted in
the INDD as stale, leaves out link times and BitmapPrinting, skips the
resource values of links resolved to another file at export, and
compares font family names without the technology suffix.

Trustworthy pairs: 495 -> 489; value coverage 88.71% -> 88.71%
(9,885,118 of 11,142,835); extra values 713,688 (6.40%).
2026-10-07 18:47:44 +00:00
aiongg 7ed2a1ab8c fix: guard against cyclic and oversized structures in damaged files
Groups that contain themselves or nest over 100 deep, style group cycles,
table counts larger than the cell data, slot numbers outside their page,
cyclic slotted-record continuations and non-finite numbers now give a
warning or an error instead of a crash or invalid output. Corpus output
is unchanged (tools/diff_outputs.py).
2026-10-07 18:26:48 +00:00
aiongg 8d71e09a6b feat: read the start of the print and booklet print settings 2026-10-07 18:07:03 +00:00
aiongg 59e7f6d168 feat: write the page colour of master spreads 2026-10-07 17:23:56 +00:00
aiongg 7a157a5abc feat: convert kinsoku set, mojikumi, ideographic space and diacritic position 2026-10-07 17:05:51 +00:00
aiongg 4359207564 feat: write GraphicType for empty graphic frames 2026-10-07 16:49:41 +00:00
aiongg fd4f25c52e fix: read the transform of groups that store it in chunk 0x40D 2026-10-07 16:36:22 +00:00
aiongg 6f8a8fe932 feat: write the stroke weight of items that have none of their own 2026-10-07 16:24:18 +00:00
aiongg 3da31dc6c2 feat: write style keyboard shortcuts and empty nested, line and GREP style flags 2026-10-07 16:05:18 +00:00
aiongg 1e26e1e8bd feat: read text, margin, anchor, pasteboard, XML tag and page item defaults 2026-10-07 15:40:54 +00:00
aiongg 8e8cb28d40 feat: read colour settings, view, grid, watermark and text defaults from the preferences 2026-10-07 15:14:54 +00:00
aiongg 3196061da1 feat: write every object style category from its category IDs
With the larger corpus each Enable attribute of object styles, and the
transparency flag of the four effects categories, follows one ID or a
group of IDs that always occur together in the category list.

Value coverage on trustworthy pairs: 87.13 % to 87.31 %.
2026-10-07 14:34:23 +00:00
aiongg deebe0e8a7 feat: write the constant shade every document lists
The pasted smooth shade object of lowest UID with a constant shading
becomes a PastedSmoothShade whose contents are its numbers in big-endian
order, with the values every constant shade has in IDML.

Value coverage on trustworthy pairs: 87.08 % to 87.13 %.
2026-10-07 14:19:32 +00:00
aiongg 44f2d9c1a7 feat: write the index sort options of every document
The index groups of the preferences object become IndexingSortOption
elements in their stored order, with their include flag and a header
type by group and header variant.

Value coverage on trustworthy pairs: 86.86 % to 87.08 %.
2026-10-07 14:01:56 +00:00
aiongg 3f1f10799d feat: write the InCopy assignment of every document
Each assignment object becomes an Assignment with the values every IDML
has. Its name depends on the language of the computer that exported the
IDML, so it is left out of the output and of value coverage.

Value coverage on trustworthy pairs: 86.83 % to 86.86 %.
2026-10-07 13:45:28 +00:00
aiongg 27cfef8fd7 feat: write named grids and object style export options
designmap.xml gets the document's named grids with the layout grid
settings all its pages share. Object styles other than [None] get the
export options every IDML of their version has, now also before
InDesign 12.

Value coverage on trustworthy pairs: 86.72 % to 86.83 %.
2026-10-07 13:29:58 +00:00
aiongg 5344dc2148 fix: tell polygons from ovals and rectangles by the stored shape code
Chunk 0x6204 of a frame holds a shape code that gives the IDML element
(line, rectangle, oval or polygon). Without a code, a closed four-point
path is an oval only when all its points are of type 0. About 5,000
shapes of the trustworthy pairs that were written as ovals are polygons,
with all their attributes and children.

Value coverage on trustworthy pairs: 84.50 % to 86.72 %.
2026-10-07 13:11:59 +00:00
aiongg 2d52b399c4 feat: write table of contents styles and the stories they made
Styles.xml gets a TOCStyle for each TOC style object, with its name,
title, title style and numbering, anchor and line break settings, and
stories get AppliedTOCStyle through the table of contents object that
made them. Entries are not decoded yet.

Value coverage on trustworthy pairs: 84.29 % to 84.50 %.
2026-10-07 12:41:09 +00:00
aiongg e6dd2d629f feat: write style import flags, unique IDs and common style values
Paragraph and character styles get Imported (the u16 after the style
kind) and StyleUniqueId (the GUID after the name) from the INDD, and the
values every non-root style has in IDML, such as PreviewColor, from
observation. The values generator leaves root styles out of the style
paths.

Value coverage on trustworthy pairs: 84.07 % to 84.29 %.
2026-10-07 12:14:48 +00:00
aiongg 463fef540a feat: write swatch library, alternate colour and guide settings
Colours get SwatchCreatorID and their alternate colour from the INDD,
HSB colours are written as RGB with ConvertToHsb, and every swatch names
its colour group swatch from DOM 12 on. Guides get Locked, GuideZone,
OverriddenPageItemProps and PageIndex, which counts pages from the
spine. Tints, gradients and the None swatch get the values every IDML
has on them.

Value coverage on trustworthy pairs: 83.37 % to 84.07 %.
2026-10-07 11:54:33 +00:00
aiongg 09782f6af0 feat: write more text frame settings
TextFramePreference gets the maximum column width, fixed column width,
auto-sizing minimums, text wrap override, column rule width and colour,
footnote spacings and the inset spacing (from the frame), read from
their chunks. Attributes that IDML writes only on some frames, with one
value wherever it does, are generated as WhenWritten values and also
give the versions from which column rules and footnotes exist.

Value coverage on trustworthy pairs: 82.03 % to 83.37 %.
2026-10-07 11:34:20 +00:00
aiongg 722fddfe97 feat: write languages, document users and document settings
designmap.xml gets the document's languages from their INDD objects,
with the quotes every IDML gives each language, the document users from
the document object, and the settings elements every IDML of the version
has (watermark, linked story and page item options, footnote and
endnote options and others) with their common values. User colours are
set by the exporting InDesign, not the INDD, and are left out of value
coverage.

Value coverage on trustworthy pairs: 81.65 % to 82.03 %.
2026-10-07 11:08:03 +00:00
aiongg fbe1e7f6ca feat: write spread, page, section and layer settings
Spreads get AllowPageShuffle and their FlattenerPreference, master
spreads ShowMasterItems, pages their tab order, overrides, layout grid
use, layout rule, colour, alternate layout and numbering descriptor,
sections their prefix, marker and alternate layout, and layers their
printing, guide, UI, wrap and colour settings. Interface colours are
named from one table shared with XML tags. Values every IDML has on
these elements, and the two built-in trap presets, are written from
observation.

Value coverage on trustworthy pairs: 80.03 % to 81.65 %.
2026-10-07 10:45:11 +00:00
aiongg 3d968bea8a feat: write the settings every page item has
Page items get Name, Visible, Locked, the interface change counts,
OverriddenPageItemProps, the layout constraints and the gradient angles
and highlights, read from their chunks and attribute lists, with the IDML
defaults where a chunk is absent. Groups take the gradient values their
children share. tools/element_values.py collects the values every IDML
of a version has on an element; the converter writes those of page items
and their export options, now for every version.

Value coverage on trustworthy pairs: 62.62 % to 80.03 %.
2026-10-07 10:09:02 +00:00
aiongg c9528b10ff feat(tools): list reproduced values per key and dump whole chunks
compare.py writes values.tsv and values-all.tsv with every key's values and
how many are reproduced, so a change can be checked for keys that lose
reproduced values. indd dump --full prints every byte of each chunk.
root_values.py reads the current inventory layout and keeps to the pairs
its generated files were made from.
2026-10-07 09:39:02 +00:00
aiongg 5543af5867 feat: add indd audit and a corpus-wide audit ranking
indd audit converts a document while recording which objects, chunks and attribute IDs the converter reads, then lists what it did not read: classes, chunks, byte-stream objects, attribute IDs per kind of list, codes of known attributes without an IDML value, and strand run kinds, with counts and the conversion's warnings. --tsv gives rows for scripts. tools/audit_corpus.py audits every distinct corpus file and ranks the items by files affected. Without the audit, recording does nothing and conversion output is unchanged.
2026-10-07 08:48:34 +00:00
aiongg f18a29bce1 feat(tools): add value coverage headline and ranked gap list
compare.py ends with value coverage (the share of reference IDML values the conversion reproduces, weighted by occurrence), document scores at 99 % and 99.9 %, and the biggest gaps ranked by documents and by values, over trustworthy pairs and over all pairs. pairs.tsv, gaps.tsv and gaps-all.tsv go to target/compare/. Font Status and package part names are left out as values no INDD can supply; docs/measurement.md defines the metric and the exclusions.
2026-10-07 08:41:26 +00:00
aiongg 4024dd79a8 feat(tools): report stale corpus pairs separately
A pair is stale when the IDML's xmp:ModifyDate is later than the INDD's or more than an hour earlier, when the IDML names UIDs the INDD database lacks, or when the INDD has stories the IDML lacks. compare.py prints totals for all pairs and for trustworthy pairs. New subcommands indd xmp and indd uids supply the INDD side. docs/measurement.md records the rule and its evidence: 159 of 654 pairs are stale.
2026-10-07 08:34:18 +00:00
aiongg f5c73f9460 fix: convert or clearly reject every file in the fetched sample batch
- InDesign 5.0: read document chunk 0x13501 with the 3.0/4.0 layout
  (four UID lists); the 6.0 layout failed on 17 files.
- Reject files that are shorter than their database pages as truncated,
  and check the signature before the size so that short files of other
  kinds are reported as not INDD files. The corpus tests accept both.
- Hyperlink page destinations: write view setting 0 as Fixed (567 of 567
  in the pairs); leave out a zoom outside the schema's 5-4000 %.
- Leave out, with a warning, values the IDML schema does not allow: ink
  neutral density -1, writing script 0xFFFFFFFF, negative drop shadow
  size and other out-of-range transparency values. The IDML writer now
  returns its own warnings.
2026-10-07 07:59:23 +00:00
aiongg 3e89cad3bd fix: convert or clearly reject every file in the new sample batch
- Bullet list: the font style can be followed by 1 and four bytes.
- InDesign 2.0: in-object strings start with 1, not 2.
- InDesign 2.0-5.0: XML tag names are a flag byte and a string, in
  either byte order.
- InDesign 3.0 and 4.0: the document's hyperlink lists start at offset
  0 and include page destinations.
- XMP: the packet can be shorter than its object; take its length.
- InDesign 1.x files (no signature, DOCUMENT at 0x5C) are reported as
  not supported; files whose page directory lies beyond the database
  pages as having no object database.
- Schema: leave out hyperlink sources that hold an anchored object, and
  the characters U+FFFE and U+FFFF, which XML does not allow.

tests/corpus.rs accepts the two rejections above and skips the paths
listed in the local corpus/exclude.txt.
2026-10-07 06:27:26 +00:00
aiongg bf72b93d23 test: download fixtures from pinned sources instead of committing them
Remove the sample documents, PDF and licence copies from tests/fixtures/.
tests/fixtures/manifest.json lists each file's download URL (pinned to a
commit for GitHub sources), size and SHA-256. tools/fetch_fixtures.py
downloads them into the git-ignored tests/fixtures/files/ and verifies
them. The fixture tests read from there and skip when it is absent.
2026-10-07 05:47:17 +00:00
aiongg e21d73aa0a docs: never commit downloaded sample files 2026-10-07 05:35:59 +00:00
aiongg fe0a20fc0e feat: convert tate-chu-yoko (Tatechuyoko) 2026-10-07 05:34:59 +00:00
aiongg 24c959bf94 feat: convert manual kerning (KerningValue) 2026-10-07 05:29:41 +00:00
aiongg 790d22d483 feat: write first baseline offset code 3 as EmboxHeight 2026-10-07 05:17:22 +00:00
aiongg 4bf1094489 feat: convert ruby text
Character runs that set attribute 0x422D to 1 carry their ruby text in
0x422E (u32 length, then text segments). In a vertically typeset
sample's print PDF that text is set small beside the run's characters.
The converter writes RubyFlag and RubyString.
2026-10-07 04:31:04 +00:00
aiongg 4bc6768fb0 feat: write vertical story orientation
A text frame's multi-column frame stores a matrix (chunk 0x2DE) that is
the identity for horizontal text and a quarter turn for vertical text.
A story whose frames all have the quarter turn is written Vertical
instead of the Horizontal every sample's IDML has.
2026-10-07 04:27:07 +00:00
aiongg 6a357f4287 feat: write the Kanji page number style
Section style code 0x4C12 is Kanji, shown by the Chinese-numeral folios
of a vertically typeset sample's print PDF. Pages in such a section are
named with Chinese digits, as the folios show.
2026-10-07 04:26:16 +00:00
aiongg 9fef3b0516 feat: read the page binding
DocumentPreference PageBinding comes from the document preferences
(u16 at 64) instead of the value every sample's IDML has. A vertically
typeset sample bound on the right stores 1, RightToLeft.
2026-10-07 04:25:39 +00:00
aiongg 7c89bdfe8a fix: write only colour models the samples show
Code 1 is Spot in two pairs; code 3 has no sample, so its Model is now
left out with a warning instead of written as MixedInkModel.
2026-10-07 04:15:57 +00:00
aiongg a9985b3775 fix: validate every corpus output and fix the two schema failures
compare.py --all now also converts the INDD and INDT files that have no
same-version IDML (any version, either byte order), validates them and
counts converter warnings by kind. validate.sh runs Jing once per schema
instead of once per file, which makes validating the whole corpus take
about 3 minutes.

Two schema failures showed up in files without an IDML:

- Gradient swatch midpoints are stored as positions from the start of
  the gradient. IDML wants them relative to the gap between two stops,
  as for opacity gradient stops. Values outside 13-87 are left out with
  a warning.
- The style kind is a u16, not a u32. One InDesign 7.5 template has a
  non-zero u16 after it, which made 151 character styles paragraph
  styles.
2026-10-07 03:50:08 +00:00