Page items:
- drop shadow KnockedOut (0x1EB6F), true where neither item nor style
stores it
- text wrap: the u32 at 40 of 0x3703 is the master page flag and the
wrap side (two u16, as the big-endian pair shows)
- groups get the fill, stroke and corner values all their children have
- dashes (0x5A35: list and corner code) on items whose stroke type in
effect is Dashed
- BeforeGroupingLayerPosition (0x324) from 21.4; export SizeType
(0x1E22B); no generated alt text attributes on groups
- anchored settings on items inside anchored items
- IgnoreWrap false without the chunk, MinimumFirstBaselineOffset 0 by
the baseline category, InsetSpacing before DOM 11 by the general
category; EPS contents from 0x8C23
- custom striped and dashed stroke styles (classes 0xB016, 0x5A43) and
StrokeType code 0x5A42
Preferences: column direction (0x555, else the story orientation),
optical margin size by the text size unit, highlight flags false without
their chunk, grid and gutter defaults and print marks by the edition
code, watermark font by the assignment name, EPUB ViewDocumentAfterExport
for DOM 8 and 9, HTMLFXLExportPreference from DOM 10.1, MathML document
values from DOM 20, index header type by variant name, three more
built-in kinsoku and mojikumi tables.
Resources and text: the saving edition from the XMP black name (tints
of black, unknown users, endnote and index titles, endnote marker), TOC
style entries, PageNumberType (0x1B09), CustomGlyph (0x1B5E), alignment
codes of 0x4266, 0x1B3E and 0xB676, spot colour alias and DOM 8 section
pagination (observed values), text variable settings and result texts,
object styles' AppliedParagraphStyle without a chunk and flex spacing,
stories of frames anchored in the XML backing story.
Fixes: flattener resolutions of 0x10833 were swapped; colour group Self
follows the DOM version.
compare.py, trustworthy pairs: value coverage 98.96 % -> 99.48 %
(+150,288 values), documents >= 99 % 914 -> 1,021, story text 37,129 ->
37,147 of 37,150 (0 missing), extras 92,838 -> 92,883 (+788 from the
KnockedOut and group-corner exceptions, -744 from fixed extras). No pair
loses values against f21267f. diff_outputs.py: 5,743 of 5,765 outputs
differ, no failure or warning differs.
10 KiB
Architecture
This document explains how the converter is built: its layers, how data
flows from INDD bytes to an IDML package, where each kind of code lives,
where the evidence for format facts is kept, and how the generated value
files work. For how to change the code, see CONTRIBUTING.md.
Layers
Each layer uses only the layers above it in this table.
| Layer | Module | Input → output | Knows about |
|---|---|---|---|
| Header | src/header.rs |
first bytes → Header (byte order, version) |
the fixed header |
| Container | src/container.rs |
file bytes → pages, master page, contiguous objects | the page layout |
| Database | src/database.rs |
pages → object bytes by UID, class of each UID | the object trees and data pages |
| Objects | src/object.rs |
object bytes → chunks; Cursor decodes numbers, strings and text |
the byte order and string tag (Encoding) |
| Model | src/model/ |
objects → Document |
INDD classes, chunks and attribute IDs |
| Writer | src/idml/ |
Document → IDML package (ZIP) |
IDML elements, attributes and naming rules |
| Audit | src/audit.rs |
what the model read → Audit report |
nothing of the format; records reads |
The model holds decoded INDD data. The writer only maps model types to
IDML: it does not read INDD bytes. When the writer needs a new INDD
value, decode it in the model and add a field or a Value variant.
Data flow
indd::convert_into (src/lib.rs) runs these steps:
Container::parsechecks the header and splits the file into pages.Container::databasereads the object trees into aDatabase. The database gets the file'sEncodingfrom the header.model::Reader::documentreads the document object (UID 1), then every object of the class list whose class it knows, and builds aDocument. Each object comes fromDatabase::getand carries theEncoding, so every decoder reads it in the file's byte order.idml::writewrites the package parts (designmap.xml, spreads, stories, resources, styles) and stores them in a ZIP file.- The warnings of the model and of the writer are returned with the package.
indd::convert returns the package as bytes; indd::convert_file
reads a file first. The CLI (src/main.rs) uses convert_into to write
to a file.
No global state
A conversion keeps all its state in its own values:
- The byte order and the string tag are an
Encoding, held by theDatabase, eachObjectand eachCursor. - The
Readerholds its object cache, warnings and the state it collects while reading (XML nodes, frame orientations, the group path). indd auditgives theDatabaseanaudit::Recorder; objects and attribute lists carry it and report their reads to it. A normal conversion has no recorder.
The only statics are caches of the constant value files (OnceLock).
Documents can therefore be converted on several threads at once; the
fixture test conversions_on_several_threads_are_independent checks it.
Errors and warnings
| Problem | Result |
|---|---|
| The file is not an INDD file, is truncated, or its database structures are damaged | Err(Error): no package |
| One object cannot be read (a story, spread, page item, style, colour, …) | The object is left out with a warning |
| An attribute list cannot be parsed | The list is left out with a warning |
| A value has no IDML equivalent, or one the samples do not show | The value is left out, with a warning where a reader would miss it; unknown codes are reported to indd audit |
The model also guards against damaged structures that would recurse without end or allocate too much: groups that contain themselves, style group cycles, table sizes larger than their cell data.
Module map
src/model/
| Module | Reads |
|---|---|
reader.rs |
Reader: object cache, chunk access, child lists, warnings |
document.rs |
Document; Reader::document and one function per class or area |
ids.rs |
class, chunk and strand IDs and enumeration codes |
attrs.rs |
attribute lists (Attrs, Value), including structured values: tab lists, nested styles, bullet characters, opacity stops |
spread.rs |
spreads, pages, guides, layers, sections |
item.rs |
page items: transforms, paths, frames, form fields and their states, export options, graphics and their properties and layers, links, text wrap |
story.rs |
stories: text runs, anchored objects, tables in text, hyperlink sources, XML markers |
style.rs |
paragraph, character, object and TOC styles, style groups, anchored object, text frame and story settings |
table.rs |
tables, rows, columns, cells, cell and table styles |
color.rs, font.rs, cjk.rs |
swatches and inks; font families; composite fonts and kinsoku and mojikumi tables |
hyperlink.rs, xref.rs, variable.rs, xml.rs |
hyperlinks and bookmarks; cross-reference formats; text variables; the XML structure |
settings.rs, prefs.rs |
document-level lists and preferences |
xmp.rs |
dates of the XMP packet (time zone offsets for link times) and its black swatch name (the saving edition) |
strings.rs |
searches for names at positions that are not decoded |
src/idml/
| Module | Writes |
|---|---|
mod.rs |
Writer and write: the package parts in order |
designmap.rs |
designmap.xml: document settings, languages, layers, sections, hyperlinks, TOC styles, text variables |
spread.rs |
spreads and master spreads: pages, guides, page items, form fields |
graphic.rs |
placed graphics: settings, clipping, import options, layers, links |
export.rs |
ObjectExportOption of page items and object styles, by version |
story.rs |
stories, tables in stories, the XML backing story |
styles.rs |
Resources/Styles.xml |
resources.rs |
Resources/Graphic.xml, Fonts.xml, Preferences.xml |
pages.rs |
page numbering and page names |
attrs.rs |
attribute tables: INDD attribute ID → IDML name and value kind |
kind.rs |
Kind: how each kind of value becomes IDML text |
transparency.rs |
transparency effects of page items |
applied.rs |
the effective values of an object style, which page item values are compared with |
values.rs |
lookups in the generated value files |
format.rs |
number, matrix, base64 and name formats |
xml.rs, zip.rs |
the XML and ZIP writers |
Where evidence lives
| What | Where |
|---|---|
| How the format was learned, and the rules for new facts | CLEANROOM.md |
| Every fact about the INDD format, with the files and counts that support it | docs/format/, one file per area |
| Values written because every InDesign export has them | docs/format/idml-values.md |
| How fidelity is measured (stale pairs, value coverage) | docs/measurement.md |
| The samples | corpus/, local and git-ignored; corpus/inventory.tsv lists version and pairing |
Code comments cite the docs/format/ section a decoder relies on. Code
uses only facts recorded there.
Generated value files
IDML consumers apply their own defaults to attributes that are missing.
To avoid that, the writer also writes values that every InDesign export
in the corpus has, even when the INDD has no field for them. These
values are facts about InDesign's IDML output, so their evidence is the
corpus IDML files alone (idml-values.md).
| File | Generator | Holds | Read by (src/idml/values.rs) |
|---|---|---|---|
src/idml/element_values.xml |
tools/element_values.py |
values of page items, spreads, pages, swatches, styles, document elements and preferences; lists such as trap presets; values keyed on another attribute | element, element_attrs, when_written, present, list, keyed |
src/idml/root_values.xml |
tools/root_values.py |
values of the five root styles | root_style |
src/idml/object_style_values.xml |
tools/root_values.py |
object styles other than [None] |
object_style |
src/idml/preference_values.xml |
tools/root_values.py |
the elements of Resources/Preferences.xml |
preferences |
How they work:
- A generator reads every corpus IDML and keeps a value when every
element on the path has it, in every file of a range of DOM versions.
Blocks in the file carry that range (
MinimumVersion,MaximumVersion); the writer asks for the document's version. - The files are committed. Regenerate them with
--writeafter the corpus changes, and review the diff: a value that varies in new samples disappears. root_values.pycollects the root styles, the other object styles and the preferences from the pairs thatcompare.pycompares.
Precedence. A value read from the INDD always wins over an observed
value. The XML writer (src/idml/xml.rs) keeps the attributes of an
open start tag until it closes: Xml::attr sets a value, replacing an
observed one in place, and Xml::attrs_missing adds observed values
only for attributes the element does not have. For value trees
(values::Node), Node::set replaces and Node::merge adds missing
values.
A few observed values are still written as literals in the writer,
where the generators cannot express them: AccurateLABSpots of the
document (one DOM 7 file lacks it) and the default numbering list for a
file whose lists cannot be read. Each cites idml-values.md.
Tests and tools
| Check | What it covers |
|---|---|
Unit tests (cargo test) |
decoders on synthetic bytes (database::synthetic builds objects and databases), writer functions on small models |
tests/fixtures.rs |
conversion of open-licensed fixtures (tools/fetch_fixtures.py), both byte orders, thread independence, audit consistency |
tests/corpus.rs |
every corpus file parses and converts; skipped without corpus/ |
src/hostile.rs |
truncated and byte-flipped synthetic documents and fixtures convert or fail without a panic or a hang (INDD_HOSTILE_SCALE, INDD_HOSTILE_SEED for larger sweeps) |
| tools/compare.py | fidelity against the reference IDML files, schema validation |
| tools/diff_outputs.py | byte-for-byte output of two revisions over the corpus |
| tools/check_wasm.sh | the library builds for wasm32-unknown-unknown, a conversion there imports no host function, and its output is the same as native |
| indd audit, tools/audit_corpus.py | what the converter does not read yet |