Files

8.3 KiB
Raw Permalink Blame History

indd-utils

Rust reader for INDD files (Adobe InDesign's native format) and converter to IDML. Crate and command: indd. Repo: storytold/indd-utils (public). Licence: MIT OR Apache-2.0.

Clean-room rules (read CLEANROOM.md)

  • Never commit downloaded sample files of any kind (INDD, IDML, PDF, images, archives), whatever their licence. They can be downloaded elsewhere; the repo records only how to get them. corpus/ is git-ignored and stays local.
  • Every fact about the INDD format goes in docs/format/ with the evidence that supports it (which files, how many, what was compared). Code relies only on documented facts.
  • Do not use knowledge of InDesign internals from memory (InDesign SDK class names, IDs, persistence details). If it can't be shown from the corpus or a permitted reference in CLEANROOM.md, it doesn't go in.
  • Do not install or run InDesign, and do not ask anyone to create INDD files for this project.
  • No golden output files derived from third-party samples. Corpus tests compare against the sibling IDML at run time.
  • Openly licensed corpus sources may be listed in a committed fetch manifest (pinned URL, hash, licence) so contributors can rebuild that part of the corpus. Do not name any other source in committed files. The full provenance record is corpus/SOURCES.md (local).

Layout

  • src/header.rs, src/container.rs: decoded layers (see docs/format/).

  • src/audit.rs: records what a conversion reads, for indd audit.

  • src/model/: the document model. reader (typed object access), document (Reader::document), and one module per area: spread, item, story, style, settings, table, color, font, …

  • src/idml/: the IDML writer. mod.rs has write; one module per package part (designmap, resources, styles, spread, story), plus attrs (attribute tables), format, pages, values.

  • docs/measurement.md: how compare.py and the audit measure the converter (stale pairs, value coverage, exclusions).

  • tests/fixtures.rs: smoke tests on open-licensed samples listed in tests/fixtures/manifest.json; they skip if tests/fixtures/files/ is absent.

  • tools/fetch_fixtures.py: downloads the fixtures into the git-ignored tests/fixtures/files/ and checks their SHA-256.

  • tests/corpus.rs: tests over the local corpus/; they skip if absent.

  • src/hostile.rs: hostile-input tests. Damaged synthetic documents and fixtures must convert or fail without a panic or a hang. INDD_HOSTILE_SCALE=40 cargo test --lib hostile runs a larger sweep.

  • tools/inventory.py: corpus inventory (python3 -I tools/inventory.py corpus/).

  • tools/root_values.py [--write]: root style values that every corpus IDML has; regenerates src/idml/root_values.xml (see docs/format/idml-values.md).

  • Analysis scratch scripts go in the session scratchpad, not the repo. Anything worth keeping becomes an indd subcommand.

Corpus

corpus/ (git-ignored, tens of GB): third-party INDD/INDT files, many with a sibling IDML, plus privately held samples in corpus/own/. Current counts: corpus/inventory.tsv (version and pairing per file) and the README. Priority: InDesign 18–21, little-endian. Some IDMLs show a different save than their INDD; compare.py marks those pairs stale and reports trustworthy pairs separately (rule and evidence: docs/measurement.md).

Commands

  • python3 -I tools/check.py LABEL [--base REV]: the check for a finished task or batch. One run does the build, tests, clippy, the full measurement with schema validation, the output diff against REV (default origin/main), the README numbers, the render comparison and the privacy checks, and freezes the results to ~/.cache/indd-frozen-LABEL/. It reuses conversions, comparisons, validations and renders of earlier runs (~/.cache/indd-check/), so a run takes about 20 seconds when nothing changed. Local settings (DesignCraft binary, private documents, privacy pattern) are in .thoughts/tools/check.json. Details: docs/measurement.md, "Check runs". Run it in the background with python3 -I -u and output to a log; never start a second copy.

  • cargo test (all tests), cargo clippy --all-targets, cargo fmt.

  • tools/check_wasm.sh [file.indd...]: build the library for wasm32-unknown-unknown (rustup target add wasm32-unknown-unknown), check that a conversion there imports no host function, and compare its output with the native output for the fixtures (or the files given). Needs Node.js for the last two steps. The library's conversion path must not use std::fs, clocks or threads.

  • python3 -I tools/fetch_fixtures.py: fetch the fixture files before cargo test. To add a fixture, add its pinned URL, size and SHA-256 to tests/fixtures/manifest.json and its licence and source to tests/fixtures/README.md.

  • cargo run -q -- info <file>, indd convert in.indd out.idml, indd objects <file>, indd dump <file> <uid>..., indd uids <file>, indd xmp <file>.

  • python3 -I tools/compare.py [--detail TAG --show N]: convert every corpus pair and compare with the reference IDML. This is the main measure of progress; run it once when a task or batch is done, or when the brief says. It reports all pairs and the trustworthy ones (--stale N lists stale pairs, --trusted limits the tables to trustworthy pairs).

  • Headline numbers (value coverage, extra values, document scores, ranked gaps; defined in docs/measurement.md): cargo build --release && python3 -I tools/compare.py --exclude own/. It prints them last and writes pairs.tsv, gaps*.tsv, values*.tsv, extras*.tsv and summary.json to target/compare/.

  • README numbers: python3 -I tools/readme_numbers.py --schemas <dir> --jing <dir> builds the converter, runs compare.py --all --exclude own/ with schema validation and rewrites the README's "Current numbers" table (between <!-- numbers:start --> and <!-- numbers:end -->) from target/compare/summary.json. Without --schemas/--jing it uses the last run's summary, which must be from such a full run; --check only reports whether the README is current.

  • python3 -I tools/diff_outputs.py [OLD [NEW]]: build two revisions (default HEAD and the working tree .) under ~/.cache/indd-diff/, convert every corpus file with both and report which outputs differ (package entries compared byte for byte), plus differences in failures and warnings. Takes about a minute. A refactoring must show 0 differing outputs. Full list: ~/.cache/indd-diff/diff.tsv.

  • python3 -I tools/render_compare.py --designcraft PATH [--jobs N] [--file SUBSTR]: for corpus documents with a same-save InDesign PDF, render the reference IDML (A) and the converter's IDML (B) with DesignCraft and compare both with the PDF (R) page by page. A vs B measures the converter, R vs A the renderer (docs/measurement.md, "Render comparison"). DesignCraft is not in this repo: build designcraft-cli from storytold/designcraft. Run after cargo build --release; output in target/render-compare/.

  • indd audit <file>: what the converter does not read in one document (classes, chunks, attribute IDs, unknown codes, strand kinds) and its warnings. python3 -I tools/audit_corpus.py --exclude own/ ranks these over the whole corpus by files affected (tables in target/audit/). Use it to pick what to decode next.

  • compare.py --all also converts every other INDD/INDT under corpus/ (any version, either byte order) and reports their failures; every run counts converter warnings by kind (--warnings N lists N kinds). --exclude PREFIX leaves out files whose path under corpus/ starts with PREFIX; committed numbers must not include the privately held samples (--exclude own/).

  • Schema validation: tools/validate.sh a.idml [b.idml...] <schemas> <jing>, or compare.py --schemas <dir> --jing <dir>. validate.sh validates many packages in one run (one JVM per schema and 400 parts, VALIDATE_JOBS in parallel); compare.py validates its outputs 600 at a time. Run compare.py --all --schemas … --jing … after changes to the IDML writer: it takes about 5 minutes. The IDML RelaxNG schemas (generated by InDesign, published in the metanorma/idml repo) and Jing (Maven Central) are kept outside the repo, for example in the session scratchpad; do not commit them.