Files
indd-utils/docs/format/xml.md
T
aiongg 1d0e27db32 feat: convert big-endian files from InDesign 3.0 and 4.0
The byte order flag applies to object data only: database pages stay
little-endian. Object data is decoded in the file's byte order, and in
big-endian data the string tag and the flag byte before it are swapped.

The two big-endian samples are also the only files from before
InDesign 7.0, and some of their layouts differ: page transform and
bounds chunks, font records, XML tag names and nodes, story strands and
40-byte guide records are read; text frame settings, document
preferences, a composite font and a section style are left out with a
warning. The 40-byte guides of an InDesign 7.0 file are now read too.

docs/format/big-endian.md records the evidence. The corpus tests now
cover all byte orders.
2026-10-07 03:11:04 +00:00

142 lines
6.5 KiB
Markdown

# XML structure
How the document's XML structure (IDML `XmlStory` and `XMLElement`) is
stored. XML tags are in `objects.md`. Implemented in `src/model/xml.rs`
and the story writer in `src/idml/mod.rs`.
**Evidence.** All 251 distinct little-endian corpus files have an XML
structure; in 250 it is only the root element. The corpus pairs have 78
IDML `XmlStory` elements and 86 `XMLElement` elements: 78 root elements
and 8 more elements in one document. Unless stated otherwise, counts are
over the 251 files.
## Nodes
The structure is a tree of nodes. The document node is the root
element's parent and has no IDML element. Each node belongs to a story,
the one whose text holds its markers (below), and is named by that
story's UID and an ID that is unique within the story: a *node key*.
**Node store.** A story's chunk 0x16126 starts with the UID of the first
page of its node store. Store pages are objects of class 0x1610A with the
same chunk 0x16127 layout as hyperlink range tree pages
(`hyperlinks.md`): u32 next page, u32 previous page, u32 story, u32, u32
*n*, *n* node IDs, u32 node count, then the nodes, each u32 type, u32
size and the node data. Types: 0xBF2F the document node (251, one per
file), 0xBF34 an element (259).
**Node data:** a u16 count, then that many parts, each u32 ID, u32
length and data. Two parts are used:
| Part | Contents |
|---|---|
| 0xBF0D | u32 tag UID (0 for the document node), u32 content UID (0 = none), 22 bytes, parent node key (u32 story, u32 ID; 0, 0 for the document node), the node's own key, u32 *n*, *n* child node keys |
| 0x1032 | u32 start marker ID, u32 end marker ID; 0xFFFFFFFF = none |
All 510 nodes have part 0xBF0D of exactly this size, and the 22 bytes are
the same in all of them. The own key names the story whose store holds
the node (510 of 510). In the InDesign 3.0 and 4.0 files the 22 bytes
are 4 and 20 bytes (`big-endian.md`).
**References to nodes.** Chunk 0xBF14 is a node key:
- In the document (class 0xE01): the backing story and the document node
(251 of 251).
- In a story: the element whose content is that story (4 stories, all in
one document; each element's content UID is the story).
- In an object of class 0x2C65: the element whose content it is (1
object). This object is a child of a page item: its chunk 0x15B names
the page item as parent.
## Markers
Each node has a start marker and, unless it is an element that stands
for content held elsewhere, an end marker. A marker is a U+FEFF
character in the text of the node's story. Its position is in a tree,
chunk 0xCA57 of the story's strand of class 0x2B8:
- u16: 0 for an empty tree.
- The tree's nodes in preorder, each u16 (0 or 1, not identified), u32
value, u32 marker ID, u32 length (1 in every sample), u16 flags: 1 =
has a left child, 2 = has a right child.
- The rest is not used (u32, u32 *n*, then *n* pairs of u32).
The root's value is its position, a left child is *value* before its
parent and a right child *value* after it, as in hyperlink range trees.
Positions count characters (`objects.md`).
Checks: the tree parses in all 17,911 stories (252 trees are not empty).
Of their 1,015 markers, all are on a U+FEFF character, all are named by
a node's part 0x1032, and every marker that a node names is in the tree.
The texts hold another 2,690 U+FEFF characters that are not markers.
## The backing story
The document's chunk 0xBF14 names the backing story, which IDML writes
as `XmlStory` with `Self="u<UID>"` in `XML/BackingStory.xml` (76 of 78
pairs; the other 2 IDML files were exported from another save). Its text
is markers and a paragraph return. The document's chunk 0x222 holds two
UID lists: the stories and then the backing story. IDML `StoryList` is
the two lists in that order (74 of 78; the other 4 are the pairs from
another save, or whose IDML has stories that the INDD no longer has).
## IDML elements
**`Self`** is `d`, then `i` and the hexadecimal node ID of each element
on the path from the root element down to the element: the root element
with ID 2 is `di2`, its child with ID 0xB `di2ib`. The path follows the
parent keys, so it can pass through nodes of several stories (an element
with ID 4 in another story's store, whose parent is `di2ibifi1f`, is
`di2ibifi1fi4`). Matches 84 of 86 IDML elements; the other 2 are the
root elements of the 2 pairs from another save. Root element IDs are 2
(63 pairs) and 3 (15).
**`MarkupTag`** is `XMLTag/` and the name of the tag that the node names
(84 of 84).
**`XMLContent`** (5 elements in the pairs): a story's `Self` when the
content UID is a story (4 of 4); for an object of class 0x2C65, the
`Self` of its parent page item (1 of 1).
## Where IDML writes the elements
IDML writes elements in story text at their markers, and drops the
marker characters:
- An element with a start and an end marker holds the text between them.
- An element with only a start marker is written empty, at the marker.
- The start marker of the document node is not written. Its end marker
and the U+FEFF characters that are not markers are written as text.
- An element whose content is a story holds all of that story's text, in
the story's own file: the story's chunk 0xBF14 names it, and IDML
writes it around the story's paragraph ranges (4 stories).
Nesting with character ranges, from the 78 backing stories and the 4
tagged stories of the pairs:
- Within a character range, an element is written inside the
`CharacterStyleRange`.
- An element that holds, at any depth, an element with a story as
content is written between character ranges (a *block* element): the
open character range ends before it, and a new one opens after its
start marker and before its end marker. In the one sample with such
elements, 3 of 4 elements with an end marker are blocks, and IDML
writes an empty `CharacterStyleRange` before each of them.
- After an empty element with a story as content, the character range
ends; the following text opens a new one.
With these rules the converter writes all 76 backing stories of the
pairs from the same save exactly as IDML does (elements, character
ranges and U+FEFF text), and the elements of the 4 tagged stories at the
same places.
## Not converted
- No node type other than the document node and elements occurs, so
XML attributes, comments, processing instructions and DTDs are not
known.
- No sample has an element whose markers are in different paragraphs.
The converter leaves such an element out, with a warning, and does not
write its markers.
- Content of other classes: `XMLContent` is left out, with a warning.