# Architecture This document explains how the converter is built: its layers, how data flows from INDD bytes to an IDML package, where each kind of code lives, where the evidence for format facts is kept, and how the generated value files work. For how to change the code, see [`CONTRIBUTING.md`](CONTRIBUTING.md). ## Layers Each layer uses only the layers above it in this table. | Layer | Module | Input → output | Knows about | |---|---|---|---| | Header | `src/header.rs` | first bytes → `Header` (byte order, version) | the fixed header | | Container | `src/container.rs` | file bytes → pages, master page, contiguous objects | the page layout | | Database | `src/database.rs` | pages → object bytes by UID, class of each UID | the object trees and data pages | | Objects | `src/object.rs` | object bytes → chunks; `Cursor` decodes numbers, strings and text | the byte order and string tag (`Encoding`) | | Model | `src/model/` | objects → `Document` | INDD classes, chunks and attribute IDs | | Writer | `src/idml/` | `Document` → IDML package (ZIP) | IDML elements, attributes and naming rules | | Audit | `src/audit.rs` | what the model read → `Audit` report | nothing of the format; records reads | The model holds decoded INDD data. The writer only maps model types to IDML: it does not read INDD bytes. When the writer needs a new INDD value, decode it in the model and add a field or a `Value` variant. ## Data flow `indd::convert_into` (`src/lib.rs`) runs these steps: 1. `Container::parse` checks the header and splits the file into pages. 2. `Container::database` reads the object trees into a `Database`. The database gets the file's `Encoding` from the header. 3. `model::Reader::document` reads the document object (UID 1), then every object of the class list whose class it knows, and builds a `Document`. Each object comes from `Database::get` and carries the `Encoding`, so every decoder reads it in the file's byte order. 4. `idml::write` writes the package parts (`designmap.xml`, spreads, stories, resources, styles) and stores them in a ZIP file. 5. The warnings of the model and of the writer are returned with the package. Step 3 also records, for each placed graphic whose file the document does not hold, the raw data object of its preview (`Graphic::preview`). `indd::convert` returns the package as bytes, with those previews, read after step 4 from the database; `indd::convert_file` reads a file first. The CLI (`src/main.rs`) uses `convert_into` to write to a file, which does not read the preview data. ### No global state A conversion keeps all its state in its own values: - The byte order and the string tag are an `Encoding`, held by the `Database`, each `Object` and each `Cursor`. - The `Reader` holds its object cache, warnings and the state it collects while reading (XML nodes, frame orientations, the group path). - `indd audit` gives the `Database` an `audit::Recorder`; objects and attribute lists carry it and report their reads to it. A normal conversion has no recorder. The only statics are caches of the constant value files (`OnceLock`). Documents can therefore be converted on several threads at once; the fixture test `conversions_on_several_threads_are_independent` checks it. ### Errors and warnings | Problem | Result | |---|---| | The file is not an INDD file, is truncated, or its database structures are damaged | `Err(Error)`: no package | | One object cannot be read (a story, spread, page item, style, colour, …) | The object is left out with a warning | | An attribute list cannot be parsed | The list is left out with a warning | | A value has no IDML equivalent, or one the samples do not show | The value is left out, with a warning where a reader would miss it; unknown codes are reported to `indd audit` | The model also guards against damaged structures that would recurse without end or allocate too much: groups that contain themselves, style group cycles, table sizes larger than their cell data. ## Module map ### `src/model/` | Module | Reads | |---|---| | `reader.rs` | `Reader`: object cache, chunk access, child lists, warnings | | `document.rs` | `Document`; `Reader::document` and one function per class or area | | `ids.rs` | class, chunk and strand IDs and enumeration codes | | `attrs.rs` | attribute lists (`Attrs`, `Value`), including structured values: tab lists, nested styles, bullet characters, opacity stops | | `spread.rs` | spreads, pages, guides, layers, sections | | `item.rs` | page items: transforms, paths, frames, form fields and their states, export options, graphics and their properties and layers, links, text wrap | | `story.rs` | stories: text runs, anchored objects, tables in text, hyperlink sources, XML markers | | `style.rs` | paragraph, character, object and TOC styles, style groups, anchored object, text frame and story settings | | `table.rs` | tables, rows, columns, cells, cell and table styles | | `color.rs`, `font.rs`, `cjk.rs` | swatches and inks; font families; composite fonts and kinsoku and mojikumi tables | | `hyperlink.rs`, `xref.rs`, `variable.rs`, `xml.rs` | hyperlinks and bookmarks; cross-reference formats; text variables; the XML structure | | `settings.rs`, `prefs.rs` | document-level lists and preferences | | `xmp.rs` | dates of the XMP packet (time zone offsets for link times) and its black swatch name (the saving edition) | | `strings.rs` | searches for names at positions that are not decoded | ### `src/idml/` | Module | Writes | |---|---| | `mod.rs` | `Writer` and `write`: the package parts in order | | `designmap.rs` | `designmap.xml`: document settings, languages, layers, sections, hyperlinks, TOC styles, text variables | | `spread.rs` | spreads and master spreads: pages, guides, page items, form fields | | `graphic.rs` | placed graphics: settings, clipping, import options, layers, links | | `export.rs` | `ObjectExportOption` of page items and object styles, by version | | `story.rs` | stories, tables in stories, the XML backing story | | `styles.rs` | `Resources/Styles.xml` | | `resources.rs` | `Resources/Graphic.xml`, `Fonts.xml`, `Preferences.xml` | | `pages.rs` | page numbering and page names | | `attrs.rs` | attribute tables: INDD attribute ID → IDML name and value kind | | `kind.rs` | `Kind`: how each kind of value becomes IDML text | | `transparency.rs` | transparency effects of page items | | `applied.rs` | the effective values of an object style, which page item values are compared with | | `values.rs` | lookups in the generated value files | | `format.rs` | number, matrix, base64 and name formats | | `xml.rs`, `zip.rs` | the XML and ZIP writers | ## Where evidence lives | What | Where | |---|---| | How the format was learned, and the rules for new facts | [`CLEANROOM.md`](CLEANROOM.md) | | Every fact about the INDD format, with the files and counts that support it | [`docs/format/`](docs/format/), one file per area | | Values written because every InDesign export has them | [`docs/format/idml-values.md`](docs/format/idml-values.md) | | How fidelity is measured (stale pairs, value coverage) | [`docs/measurement.md`](docs/measurement.md) | | The samples | `corpus/`, local and git-ignored; `corpus/inventory.tsv` lists version and pairing | Code comments cite the `docs/format/` section a decoder relies on. Code uses only facts recorded there. ## Generated value files IDML consumers apply their own defaults to attributes that are missing. To avoid that, the writer also writes values that every InDesign export in the corpus has, even when the INDD has no field for them. These values are facts about InDesign's IDML output, so their evidence is the corpus IDML files alone ([`idml-values.md`](docs/format/idml-values.md)). | File | Generator | Holds | Read by (`src/idml/values.rs`) | |---|---|---|---| | `src/idml/element_values.xml` | `tools/element_values.py` | values of page items, spreads, pages, swatches, styles, document elements and preferences; lists such as trap presets; values keyed on another attribute | `element`, `element_attrs`, `when_written`, `present`, `list`, `keyed` | | `src/idml/root_values.xml` | `tools/root_values.py` | values of the five root styles | `root_style` | | `src/idml/object_style_values.xml` | `tools/root_values.py` | object styles other than `[None]` | `object_style` | | `src/idml/preference_values.xml` | `tools/root_values.py` | the elements of `Resources/Preferences.xml` | `preferences` | How they work: - A generator reads every corpus IDML and keeps a value when every element on the path has it, in every file of a range of DOM versions. Blocks in the file carry that range (`MinimumVersion`, `MaximumVersion`); the writer asks for the document's version. - The files are committed. Regenerate them with `--write` after the corpus changes, and review the diff: a value that varies in new samples disappears. - `root_values.py` collects the root styles, the other object styles and the preferences from the pairs that `compare.py` compares. **Precedence.** A value read from the INDD always wins over an observed value. The XML writer (`src/idml/xml.rs`) keeps the attributes of an open start tag until it closes: `Xml::attr` sets a value, replacing an observed one in place, and `Xml::attrs_missing` adds observed values only for attributes the element does not have. For value trees (`values::Node`), `Node::set` replaces and `Node::merge` adds missing values. A few observed values are still written as literals in the writer, where the generators cannot express them: `AccurateLABSpots` of the document (one DOM 7 file lacks it) and the default numbering list for a file whose lists cannot be read. Each cites `idml-values.md`. ## Tests and tools | Check | What it covers | |---|---| | Unit tests (`cargo test`) | decoders on synthetic bytes (`database::synthetic` builds objects and databases), writer functions on small models | | `tests/fixtures.rs` | conversion of open-licensed fixtures (`tools/fetch_fixtures.py`), both byte orders, thread independence, audit consistency | | `tests/corpus.rs` | every corpus file parses and converts; skipped without `corpus/` | | `src/hostile.rs` | truncated and byte-flipped synthetic documents and fixtures convert or fail without a panic or a hang (`INDD_HOSTILE_SCALE`, `INDD_HOSTILE_SEED` for larger sweeps) | | `tools/compare.py` | fidelity against the reference IDML files, schema validation | | `tools/diff_outputs.py` | byte-for-byte output of two revisions over the corpus | | `tools/check_wasm.sh` | the library builds for `wasm32-unknown-unknown`, a conversion there imports no host function, and its output is the same as native | | `indd audit`, `tools/audit_corpus.py` | what the converter does not read yet |