A text frame's multi-column frame stores a matrix (chunk 0x2DE) that is
the identity for horizontal text and a quarter turn for vertical text.
A story whose frames all have the quarter turn is written Vertical
instead of the Horizontal every sample's IDML has.
Section style code 0x4C12 is Kanji, shown by the Chinese-numeral folios
of a vertically typeset sample's print PDF. Pages in such a section are
named with Chinese digits, as the folios show.
DocumentPreference PageBinding comes from the document preferences
(u16 at 64) instead of the value every sample's IDML has. A vertically
typeset sample bound on the right stores 1, RightToLeft.
compare.py --all now also converts the INDD and INDT files that have no
same-version IDML (any version, either byte order), validates them and
counts converter warnings by kind. validate.sh runs Jing once per schema
instead of once per file, which makes validating the whole corpus take
about 3 minutes.
Two schema failures showed up in files without an IDML:
- Gradient swatch midpoints are stored as positions from the start of
the gradient. IDML wants them relative to the gap between two stops,
as for opacity gradient stops. Values outside 13-87 are left out with
a warning.
- The style kind is a u16, not a u32. One InDesign 7.5 template has a
non-zero u16 after it, which made 151 character styles paragraph
styles.
The byte order flag applies to object data only: database pages stay
little-endian. Object data is decoded in the file's byte order, and in
big-endian data the string tag and the flag byte before it are swapped.
The two big-endian samples are also the only files from before
InDesign 7.0, and some of their layouts differ: page transform and
bounds chunks, font records, XML tag names and nodes, story strands and
40-byte guide records are read; text frame settings, document
preferences, a composite font and a section style are left out with a
warning. The 40-byte guides of an InDesign 7.0 file are now read too.
docs/format/big-endian.md records the evidence. The corpus tests now
cover all byte orders.
Read the XML nodes stored with each story, place their markers in the
story text and write the backing story as XmlStory, with the elements
in it and in the stories they tag. Element Self names follow the node
IDs from the root down. The backing story is added to StoryList.
Text records count UTF-16 code units, but style runs, owned items and
text owners count a surrogate pair as one position. Convert those
positions to UTF-16 offsets so characters outside the Basic Multilingual
Plane keep their formatting and are not split into two replacement
characters.
XML tags (class 0xBF19) hold their name in chunk 0xBF2F and refer to an
interface colour in chunk 0x117. Write them in XML/Tags.xml, sorted by
name as IDML lists them.
Anchors and object styles hold anchored object settings in chunk
0x2800: the Y offset at 0 and the vertical alignment at 52. Write them
on object styles, and on anchored items where they differ from the
item's object style.
Every page item in the corpus IDML files from DOM 12 on has the same
ObjectExportOption, and every rectangle, oval and polygon the same
InCopyExportOption. Write them from that observation.
Write an Ink for each ink object (class 0x1F07) with its neutral
density, trap order, frequency and angle, and the colour groups listed
in the preferences (chunk 0x1F61) with their swatches.
Read the composite font and its entries (classes 0xCB02 and 0xCB03):
name, font, style and custom characters, and the kinsoku and mojikumi
tables with the character lists of a custom kinsoku table.
Object styles other than [None] now get their text frame, story, text
wrap and frame fitting settings, fill, stroke and corner attributes,
applied paragraph style and some category flags from the INDD (chunks
0x1B92B, 0x1B924, 0x285B, 0x50F28, 0x3776, 0x3777, 0x1B92E, 0x1B946),
and the values every such style has in the corpus IDML files, collected
by tools/root_values.py into src/idml/object_style_values.xml. The root
style gets the same INDD values in place of observed ones.
Frame fitting attributes are in the page item attribute list and, for
object styles, in chunk 0x1B956. Write FrameFittingOption on rectangles,
ovals, polygons and object styles, ClippingPathSettings on images, PDF
and EPS graphics from chunk 0x2C1A, and ImageIOPreference on images from
chunk 0x8C39.
Every story in the corpus IDML files has the same StoryPreference and
InCopyExportOption, and the same TrackChanges, StoryTitle,
AppliedNamedGrid, UserText and IsEndnoteStory (the last two from DOM 12
and 15). No INDD field for them is identified, so they are written from
IDML observation.
Pages store margins (chunk 0x51A) and columns (0x528) with a flag that
says whether they are the page's own; otherwise the master page at the
same position supplies them. Write MarginPreference from the values in
effect and GridDataInformation from chunk 0xCD02.
Read hyperlinks (class 0x13501), text sources (0x13502) with their text
ranges from the story's range tree, page and URL destinations and
bookmarks, and write them to designmap and the stories. Evidence is in
docs/format/hyperlinks.md.
Read text variable definitions (class 0xCAB4) into designmap TextVariable
elements with their settings, and write instances (class 0xCA64) as
TextVariableInstance elements in place of U+0018. Evidence and the
settings left out are in docs/format/text-variables.md.
Read the font records of class 0x3E03 (chunk 0x3E05) and the Typekit IDs
(chunk 0x3EEB), and write FontFamily and Font elements. Evidence and the
fields left out are in docs/format/fonts.md.
Decode page item attribute 0x6E6E (a reference followed by a built-in
stroke style code) as StrokeType and 0x6E8C as StrokeAlignment, with
evidence from items and from object style attribute lists. Write the 18
built-in stroke styles every exported IDML lists.
Read guide objects (class 0x3301, chunk 0x3308) from spread layers and
write them as Guide elements in their page: orientation, location, fit to
page, guide type, item layer, and the view threshold and colour values the
samples show. Locations are measured from the spread, and ViewPreference
states RulerOrigin SpreadOrigin to match.
Collect the root style attributes, Properties and child elements that are
the same in every corpus IDML (tools/root_values.py) and write them on the
root paragraph, character, object, cell and table styles. Values read from
the INDD take precedence; values that first appear after DOM 7 are gated
by InDesign version. Also write the default numbering list, and write a
style only once when two style objects share a name.
Adds KeepLinesTogether, AllNestedStyles, TabList, AppliedLanguage,
RightIndent, BaselineShift, VerticalScale, NoBreak, OTFFigureStyle,
Position, SingleWordJustification, ParagraphBreakType, rule above
indents and width, RuleBelow and its colour, span columns, paragraph
shading and border on/colour/top/bottom offsets, SameParaStyleSpacing,
grid alignment, text gradient fill, EndJoin, and bullet and numbering
attributes. Four-byte values of numeric attributes are now written,
which fixes DropcapDetail. Evidence is in docs/format/attributes.md.
Map paragraph attributes 0x1B37 (StartParagraph), 0x1B0D and 0x1B0E
(drop cap characters and lines), 0x1B31 and 0x1B32 (maximum and minimum
letter spacing), and the text frame's first baseline offset code in
chunk 0x2CE. Evidence is recorded in docs/format.
Capitalization codes 1 (SmallCaps) and 3 (CapToSmallCap), text wrap
modes 1 (JumpObjectTextWrap) and 3 (BoundingBoxTextWrap) with offsets
stored as left, top, right, bottom, and page number style 0x4C17
(LowerRoman) with Roman page names. Evidence in docs/format.
Read chunk 0x3703 (mode, offsets, flags) and the contour type in chunk 0x373D, and write TextWrapPreference for page items, groups and placed graphics. Modes 0 (None) and 6 (Contour) are written; other modes are left out with a warning. Offsets are written only when all are 0, since their order is not known yet.
Read each section's first page and page number style from chunk 0x4C02, sort sections by their first page, and write one Section per section with its length. Write PageNumberStart only when numbering restarts, and PageNumberStyle only for the Arabic style seen in the corpus. Pages are named by their number in their section.
Master pages use the same chunk 0x140F as document pages. Write AppliedMaster and MasterPageTransform for them too: "n" and the identity matrix when the chunk is absent or empty, as in all corpus pairs, and the named master spread otherwise.
Read gradient swatches (stops, locations, midpoints, type) and write them
to Graphic.xml after the swatches, as the schema requires. Write
CornerOption, CornerRadius and gradient start and length on page items.
Decode unknown 4-byte attribute values as numbers.
Write DocumentPreference (page size, facing pages, bleeds, intent) and
TextFramePreference (columns, gutter, column width, vertical
justification, auto-sizing). compare.py now also compares child elements
without Self and the preference elements.
Read tables owned by U+0016 in story text: row, column, header and footer
counts, column widths, row heights, cells with their spans, and cell text
assigned through the text owner strand. Write Table, Row, Column and Cell
elements, including tables nested in cells.
convert() now returns warnings for content it had to leave out, and the
CLI prints them.
Write the style hierarchy with root and nested groups and path-qualified
style references, object styles with AppliedObjectStyle on page items,
master spread names and prefixes, and sections with their numbering.
Keep negative zero in numbers, as IDML does.
Read graphics inside frames (transform, bounds) and their links and link
resource URIs, and write them as Image/PDF/EPS/SVG elements with Link
children. Frames with graphics get ContentType GraphicType.
Read the owned-items strand of stories and write anchored page items in
place of their U+FFFC character. Record that the third object tree lists
deleted UIDs, which no live hierarchy references.
Parse attribute lists in style objects and in paragraph and character
style runs, and write 40 text attributes (font, style, size, leading,
colour, tracking, kerning method, indents, spacing, justification,
underline, rules, hyphenation and more) on styles and text ranges. Read
font family names.
tools/compare.py now also compares text range attributes at matching text
offsets; every mapped attribute matches the reference in all ranges.
Decode colour swatches (name, model, space, values, override, flags) and
the None swatch, and write them to Graphic.xml. Parse attribute lists and
write page item fill colour and tint, stroke colour and weight, and miter
limit. Write BasedOn of root styles as a string, as IDML does. Write forced
line breaks as U+2028 and keep surrogate pairs intact across text runs.
Document the object, chunk and attribute formats with their evidence.
The master page holds three tree roots and counts: objects, the class of
each UID, and UIDs without a class. Database::class_of returns an
object's class ID, and `indd objects` prints it.
Decode the database pages: page trailers and types, the logical page
table, the object B+ tree keyed by (UID, segment), data pages, slotted
pages and continued records. Database::object returns any object's byte
stream. Add `indd objects` and `indd object` commands.
Every object of all 351 little-endian corpus files reads without an
inconsistency. Big-endian databases report Unsupported for now.
Add a project CLAUDE.md with the clean-room rules.