InDesign 5.0 destinations have no key, hyperlinks name their destination
by UID, text destination owners use chunk 0x1351E, and text sources are
placed by start and end markers owned by the story (chunk 0x13524)
instead of a range tree. These are read by version; keys are left out
where there are none.
No 5.0 file has an IDML. Over the 17 5.0 files with hyperlinks: 121
hyperlinks, 118 text sources, 7 text destinations and 45 bookmarks are
now written, the 116 warnings about their destination objects are gone,
and every output validates. Pair results unchanged (91.92 %, extra
values 26,516). Schema validation: 0 failures.
Class 0x13552 objects are written as HyperlinkExternalPageDestination:
hidden and key from chunk 0x13580, page index from chunk 0x1B8, view
from chunk 0x13527, and the name IDML derives from the linked
document's file name. DocumentPath is left out (it starts with a volume
name the INDD does not hold). Hyperlinks of kind 2004 and bookmarks now
find them.
Trustworthy pairs: reproduced 10,566,138 -> 10,578,767 (91.81 % ->
91.92 %); 643 of 643 external page destinations with all values but
DocumentPath, 576 more hyperlinks, bookmarks 452 -> 1,027 of 1,027.
Extra values unchanged (26,516); no key reproduces fewer values. Schema
validation: 0 failures.
- A text source over several character ranges of one paragraph range is
written as a child of the ParagraphStyleRange: the ranges at its ends
are split and the parts inside become its CharacterStyleRange
children. Sources across paragraph ranges, or holding an XML marker,
are left out with a warning.
- Text sources with chunk 0x135A0 are written as CrossReferenceSource
with AppliedFormat. One that holds more than a text variable instance
is written at paragraph level, since the schema allows no Content in
it.
Trustworthy pairs: reproduced 10,530,141 -> 10,566,138 (91.50 % ->
91.81 %); HyperlinkTextSource 3,449 -> 5,475, CrossReferenceSource 116,
hyperlinks 3,097 -> 5,149, all with their values. Extra values 26,634 ->
26,516: the 332 values of cross-reference sources written as hyperlink
sources go, and cross-reference sources written around a character
range add their inner ranges (one pair). No key reproduces fewer values.
"spans several text ranges" warnings over all files 9,703 -> 3,277.
Schema validation: 0 failures.
Class 0x13504 objects are read as text or paragraph destinations (kind
in chunk 0x13526). The story's owned items of class 0x1353C name them at
their U+FEFF, which is now written as an empty HyperlinkTextDestination
or ParagraphDestination element instead of the character. Hyperlinks and
bookmarks can point at them.
Trustworthy pairs: reproduced 10,513,964 -> 10,530,141 (91.36 % ->
91.50 %); stories with identical text 18,517 -> 18,565; documents at
>= 99 % 18 -> 20; HyperlinkTextDestination 685 of 685,
ParagraphDestination 119 of 119, bookmarks 373 -> 452. Extra values
unchanged (26,634); no key reproduces fewer values. Schema validation: 0
failures. The 7 text destinations of InDesign 5.0 files (no key) are
left out with a warning, like their page and URL destinations.
- Text source chunk 0x135B7 is decoded (type, anchor name, index marker,
page number string, level) and written as AlternativeDestination;
control characters in the page number become <?AID 00xx?> text.
- Page item sources (class 0x13503) are written as
HyperlinkPageItemSource when their page item is written, and their
hyperlinks with them.
Trustworthy pairs: reproduced 10,511,339 -> 10,513,964 (91.36 %);
AlternativeDestination 463 -> 1,048 of 1,048, HyperlinkPageItemSource
136 of 136, 136 more hyperlinks. Extra values unchanged (26,634); no key
reproduces fewer values. Schema validation: 0 failures.
- Hyperlink: Visible, Width, BorderStyle and BorderColor whenever chunk
0x13553 has the corpus pattern, Highlight from its byte 10, no
BorderColor for hyperlinks to a page of another document.
- Hyperlink destination: left out for hyperlinks into another document
(chunk 0x1359F) instead of a wrong local destination; `n` for kind
2000 without a destination; hyperlinks without a source are left out.
- Page destination: ViewBounds from chunk 0x13527 and NameManually.
Trustworthy pairs: reproduced 10,505,081 -> 10,511,339 (91.28 % ->
91.33 %); 22 more hyperlinks, Hyperlink appearance keys 1,982 -> 2,878
of 2,878, page destination ViewBounds and NameManually 831 of 831,
Destination wrong 0. Extra values unchanged (26,634); no key reproduces
fewer values. Schema validation: 0 failures.
Groups (class 0xCA8B) listed by the root cell or table style group are
written as CellStyleGroup or TableStyleGroup elements holding their
styles, in the order of the group's children (objects.md).
compare.py --exclude own/, trustworthy pairs: CellStyleGroup elements
0 -> 2 of 2 (4 values); no other change, no new extras, no conversion
or schema failures.
New table and table style attributes: stroke and fill counts, end row
and start/end column stroke colours, weights, types and tints, column
fill colours and tints, skip counts, graphic insets and
ClipContentToGraphicCell; on tables only LeftInset etc. and
BreakHeaders; on styles only the footer region style and the region
flags. Groups whose IDs are always equal (border weights, colours,
types, tints and gap colours; StartColumnStrokeType/EndColumnLineStyle;
SkipFirstHeader/SkipLastFooter) are written only when all IDs agree.
Table stroke types are decoded as two words. The root table style takes
its INDD values where the observed root values have the attribute, and
from DOM 11 its text and graphic cell values. A cell style without its
own paragraph style gets [No paragraph style] (tables.md).
compare.py --exclude own/, trustworthy pairs: value coverage 94.23 % ->
94.24 %; every Table key above is now right on every table, TableStyle
Text*/Graphic*Inset and clip flags 341-348 -> 375 of 375, border and
region keys up, CellStyle AppliedParagraphStyle 507 -> 547 of 547. No
new extras in trustworthy pairs; 98 in a stale pair's table that its
reference lacks.
An edge value is written when every grid position along that edge has
it with the same value; the priority comes from the cell's own set, or
is 1 for the right (bottom) edge of a merged cell whose edge positions
have one. New edge groups: gap tint, gap colour, overprint and gap
overprint. Cell edge tints -1 are written as 100 and a weight is 0 when
the edge colour is None. Stroke codes 0x5A3F (Japanese Dots) and 0xB007
(ThickThick) for cells and tables. New cell attributes: graphic insets,
OverprintFill, VerticalJustification 0 and 3, FirstBaselineOffset,
WritingDirection (not on cell styles: the schema lacks it),
RotationAngle, DiagonalLineStrokeOverprint. Text*Inset only from DOM 11.
Cell style edge IDs are rotated against the cell IDs (tables.md).
compare.py --exclude own/, trustworthy pairs: value coverage 93.13 % ->
94.23 %, median 96.7 % -> 96.8 %, documents >= 99 % 14 -> 18; extra
values 30,379 -> 30,237. Every Cell edge key is now right on every cell
except one bottom weight (tables.md); CellStyle edge keys rise too.
One unnamed colour (14 values) is no longer written: its only reference
was a wrong BottomEdgeStrokeColor that is now right, and IDML writes the
colour though nothing refers to it (objects.md). In stale pairs the
extras of a table missing from its reference grow by 2,590.
Cells start at positions with attribute 0xB666 (text) or 0x10469
(graphic); positions past a short row record are covered. The layout
record is parsed by its flags, so the width is read where it is, and
cells never laid out are kept. Spans follow a two-pass rule: the column
span whose widths best match the layout width plus the insets in effect,
then the free covered rows below. From DOM 11 every Table, Row, Column
and Cell gets TextTop/Left/Bottom/RightInset and ClipContentToTextCell
in effect: cell set, cell style chain, table-style region cell style,
table, table style chain, root table style; rows and columns copy their
first cell (tables.md).
compare.py --exclude own/, trustworthy pairs: value coverage 92.51 % ->
93.13 %, median 96.5 % -> 96.7 %; story text 18,509 -> 18,517. All
Cell, Row, Column and Table text insets and ClipContentToTextCell are
now right (Cell 11,097 of 11,097 each), Cell ColumnSpan 11,304 -> 11,550
and RowSpan 11,389 -> 11,550 of 11,550; 144 overset cells are no longer
dropped. No new extras in trustworthy pairs; the extras of one stale
pair's table that its reference lacks grow by 1,052.
Chunk 0x50F65 gives TableDirection (written on every table); the row
group attributes 0xB69F, 0x10407 and 0xB6A1 give AutoGrow (now also
true), StartRow and KeepWithNextRow (tables.md).
compare.py --exclude own/, trustworthy pairs: value coverage 92.50 % ->
92.51 %; Table TableDirection 0 -> 326 of 326, Row AutoGrow 63 -> 1,128
of 1,128, StartRow 10 of 10, KeepWithNextRow 8 of 8. No new extras in
trustworthy pairs; 7 in one stale pair's table that the reference lacks.
A style's IDML Name is its group path and its own name joined by ":",
with a ":" inside a name written "\:" and "$ID/" before the whole path
of a built-in style; Self is that name escaped (objects.md, style
groups). Object and cell styles now include their group path too, and
every reference uses the same name. A font with an empty style name is
named "<family> Regular" with the family alone in Self (fonts.md).
compare.py, trustworthy pairs: extra values 33,181 -> 30,379 (0.27 %);
value coverage 92.45 % -> 92.50 % (10,307,358 values; grouped style
names, styles now matched); no value lost; 14 documents >= 99 %,
median 96.5 %. No conversion or schema failures; story text
18,509/18,610.
IDML writes a colour or gradient without a name only if a value in the
package refers to it, a gradient stop or tint uses the colour, or the
page item defaults name it in one of the two entry tables of chunk
0x6E07 (objects.md, colours; preferences.md). The graphic part is now
written last, from the references in the other parts; the second table
of chunk 0x6E07 is read.
compare.py, trustworthy pairs: extra values 155,956 -> 33,181 (unnamed
Color -114,734, Gradient -3,498, GradientStop -4,543). 463 values are
no longer reproduced: 10 unnamed colours and 22 unnamed gradients with
their stops that IDML writes though nothing refers to them (open, see
objects.md). Value coverage 92.46 % -> 92.45 %. No conversion or schema
failures; story text 18,509/18,610.
The previous commit (09e87fa), measured alone: extra values 635,511 ->
155,956, value coverage 92.41 % -> 92.46 %; 39 values lost (7 opacity
stops and 8 drop shadows IDML writes with values equal to the style,
5 StrokeWeight, 1 TextColumnFixedWidth).
IDML leaves out the values of a page item that equal those of its
applied object style. The writer now compares with the style's
effective values (its own, then the styles it is based on, then [None]):
- TextFramePreference: an attribute is written when its category is off
in the style's category list (chunk 0x1B92E, also for [None]) or its
value differs; TextColumnMaxWidth (DOM 8), TextColumnCount (DOM 10)
and InsetSpacing (DOM 11) are always written; footnote values from
13.1 (objects.md, text frame preferences).
- Fill, stroke, miter and corner attributes (attributes.md).
- Transparency effects: only differing values, and an effect element
only when it has one or its opacity stops differ (transparency.md).
New decoding: object style chunk 0x1B924 fields for balance, fixed
width, justification, first baseline offset (code 4 = XHeight, also for
frames) and auto-sizing; chunk 0x1B92C (style transparency). Object
styles get the DOM 12 text frame values from DOM 8 and
FootnotesEnableOverrides from 13.1 (idml-values.md).
Rules from the corpus, each documented in docs/format/:
- ItemLayer only on items directly under a spread (objects.md).
- The root object style [None] gets no Enable… attributes and no effects
category settings; EnableTextFrameColumnRuleOptions from 15.1, and the
footnote values of an object style's TextFramePreference from 13.1.
- The root paragraph style has no NextStyle and no AllNestedStyles;
TextDefault has AllNestedStyles only when the list has items; the
empty TabList is written from DOM 8 (attributes.md).
- DOM 7 has no EPubExportPreference or HTMLExportPreference.
- Row and column Self use a hexadecimal index (tables.md).
- Cell CellType from DOM 11; Document Name from DOM 13; ColorGroup Self
is the UID before 11.3.
- The internal layer is the first of the document's layer list, whatever
its name.
- A CR in a name is %0d in Self (colours, gradients, tints, hyperlink
destinations, styles).
compare.py, trustworthy pairs (after the compare fix of 48d184f):
extra values 705,523 -> 635,511; value coverage 92.25 % -> 92.41 %
(10,296,599 values); 2 ItemLayer values lost (the 2 grouped items whose
IDML has one); 12 documents >= 99 %. No conversion or schema failures.
Page items, object styles and PageItemDefault: each corner radius and
corner option from its own attribute ID (top left 0x6E70/0x6E6F, top
right 0x6E94/0x6E91, bottom left 0x6E92/0x6E8F, bottom right
0x6E93/0x6E90), with the corner codes InsetCorner, BevelCorner and
FancyCorner. Object styles no longer need all four corners equal.
Paragraph styles and text: ParagraphShadingTopLeft/BottomLeftCornerRadius
(0x1DF12, 0x1DF14); the tied groups of border corner radii, shading
right corner radii and shading corner options when their IDs agree; the
shading and border origin pairs for the known code pairs.
compare.py --all --exclude own/ (trustworthy pairs): value coverage
91.61% -> 92.25% (+71,545 values); documents >= 99 %: 5 -> 11; median
95.7% -> 96.3%; no key lower. Extra values 709,654 -> 718,266
(+8,612): the per-corner attributes follow the presence of CornerRadius
and CornerOption in the item's list, so each of the existing 2,055
extra CornerRadius values (items whose IDML leaves the value to the
object style) now has four per-corner extras, and likewise the
CornerOption extras. 0 conversion and 0 schema failures.
Links: LinkResourceModified, ShowInUI, LinkImportStamp and the two link
times from chunk 0x8C9B, PDFIdentifier from chunk 0x1B6, the size and
format name from the resource's chunk 0x8C92, and the values every link
has. Link times are written in the time zone of the XMP date nearest in
day of the year (the INDD has no time zone; compare.py does not score
them): 3,886 of 4,446 import times and 3,316 of 4,434 modification
times right.
Images, PDF, EPS and SVG graphics: the page item settings and object
style, Space, ActualPpi and EffectivePpi (chunk 0x1708 and the scale on
the spread), ImageTypeName, Profile (chunk 0x7C0F), fill colour and
gradients, vector colour policies (chunk 0x7C42 or the document's
policies), PDFAttribute (chunk 0x251B), GraphicLayerOption with the
GraphicLayer tree (chunk 0x177A) and LayerCompOption (chunk 0x9209),
and ImageIOPreference from chunk 0x1714 in place of chunk 0x8C39.
The placed graphic writer moves to idml/graphic.rs.
compare.py --all --exclude own/ (trustworthy pairs): value coverage
90.00% -> 91.61% (+179,834 values); documents >= 99 %: 1 -> 5; median
94.2% -> 95.7%; no key lower. Extra values 709,643 -> 709,654 (+11 in
SVG elements that are not in the reference). New warnings: 178 layer
chunks in 3 InDesign 3.0 files, whose layout differs, are left out.
0 conversion and 0 schema failures.
ObjectExportOption now takes its alternative text, actual text and
tagging values from chunk 0x1E206 of the item, and its other attributes
from a table by INDD major and minor version (8.0-9.1, 9.2, 10.0, 10.1
on, 21.0, 21.1, 21.4), for page items from version 8 and for every
object style, [None] included, from version 9. Page items of version 7
get no ObjectExportOption, as in their IDML.
compare.py --all --exclude own/ (trustworthy pairs): value coverage
88.72% -> 90.00% (+142,324 values); documents >= 99 %: 0 -> 1; median
92.4% -> 94.2%; no key lower. Extra values 713,688 -> 709,643: -4,319
from the DOM 7 rectangles that no longer get the element; +274 more
values in GraphicLine, Oval and TextFrame elements that are not in the
reference (more attributes per element). 0 conversion and 0 schema
failures.
IDML writes the shortest decimal that reads back as the same value; when
that needs 17 digits it takes the 17-digit decimal nearest to the exact
value, with ties to even (150.23599243164062). Rust's shortest form can
end in the other digit (…063). num() now re-rounds 17-digit output with
{:.16e}.
compare.py --all --exclude own/ (trustworthy pairs): value coverage
88.72% (9,885,604 -> 9,885,761 values); PDF P.GraphicBounds +148,
EPS P.GraphicBounds +9; no key lower; extra values unchanged (713,688);
0 conversion and 0 schema failures.
Page items and page item defaults knew only code 0x5A15; object styles
also knew 0 (None) and 0x5A16 (InverseRoundedCorner). Using the object
style codes everywhere reproduces 486 more reference values in the
trustworthy pairs (489 of 489 page item defaults, 4 rectangles, 2 text
frames) and adds no extra values. Object styles write CornerOption in
the page item attribute order.
Groups that contain themselves or nest over 100 deep, style group cycles,
table counts larger than the cell data, slot numbers outside their page,
cyclic slotted-record continuations and non-finite numbers now give a
warning or an error instead of a crash or invalid output. Corpus output
is unchanged (tools/diff_outputs.py).
With the larger corpus each Enable attribute of object styles, and the
transparency flag of the four effects categories, follows one ID or a
group of IDs that always occur together in the category list.
Value coverage on trustworthy pairs: 87.13 % to 87.31 %.
The pasted smooth shade object of lowest UID with a constant shading
becomes a PastedSmoothShade whose contents are its numbers in big-endian
order, with the values every constant shade has in IDML.
Value coverage on trustworthy pairs: 87.08 % to 87.13 %.
The index groups of the preferences object become IndexingSortOption
elements in their stored order, with their include flag and a header
type by group and header variant.
Value coverage on trustworthy pairs: 86.86 % to 87.08 %.
Each assignment object becomes an Assignment with the values every IDML
has. Its name depends on the language of the computer that exported the
IDML, so it is left out of the output and of value coverage.
Value coverage on trustworthy pairs: 86.83 % to 86.86 %.
designmap.xml gets the document's named grids with the layout grid
settings all its pages share. Object styles other than [None] get the
export options every IDML of their version has, now also before
InDesign 12.
Value coverage on trustworthy pairs: 86.72 % to 86.83 %.
Chunk 0x6204 of a frame holds a shape code that gives the IDML element
(line, rectangle, oval or polygon). Without a code, a closed four-point
path is an oval only when all its points are of type 0. About 5,000
shapes of the trustworthy pairs that were written as ovals are polygons,
with all their attributes and children.
Value coverage on trustworthy pairs: 84.50 % to 86.72 %.
Styles.xml gets a TOCStyle for each TOC style object, with its name,
title, title style and numbering, anchor and line break settings, and
stories get AppliedTOCStyle through the table of contents object that
made them. Entries are not decoded yet.
Value coverage on trustworthy pairs: 84.29 % to 84.50 %.
Paragraph and character styles get Imported (the u16 after the style
kind) and StyleUniqueId (the GUID after the name) from the INDD, and the
values every non-root style has in IDML, such as PreviewColor, from
observation. The values generator leaves root styles out of the style
paths.
Value coverage on trustworthy pairs: 84.07 % to 84.29 %.
Colours get SwatchCreatorID and their alternate colour from the INDD,
HSB colours are written as RGB with ConvertToHsb, and every swatch names
its colour group swatch from DOM 12 on. Guides get Locked, GuideZone,
OverriddenPageItemProps and PageIndex, which counts pages from the
spine. Tints, gradients and the None swatch get the values every IDML
has on them.
Value coverage on trustworthy pairs: 83.37 % to 84.07 %.
TextFramePreference gets the maximum column width, fixed column width,
auto-sizing minimums, text wrap override, column rule width and colour,
footnote spacings and the inset spacing (from the frame), read from
their chunks. Attributes that IDML writes only on some frames, with one
value wherever it does, are generated as WhenWritten values and also
give the versions from which column rules and footnotes exist.
Value coverage on trustworthy pairs: 82.03 % to 83.37 %.
designmap.xml gets the document's languages from their INDD objects,
with the quotes every IDML gives each language, the document users from
the document object, and the settings elements every IDML of the version
has (watermark, linked story and page item options, footnote and
endnote options and others) with their common values. User colours are
set by the exporting InDesign, not the INDD, and are left out of value
coverage.
Value coverage on trustworthy pairs: 81.65 % to 82.03 %.
Spreads get AllowPageShuffle and their FlattenerPreference, master
spreads ShowMasterItems, pages their tab order, overrides, layout grid
use, layout rule, colour, alternate layout and numbering descriptor,
sections their prefix, marker and alternate layout, and layers their
printing, guide, UI, wrap and colour settings. Interface colours are
named from one table shared with XML tags. Values every IDML has on
these elements, and the two built-in trap presets, are written from
observation.
Value coverage on trustworthy pairs: 80.03 % to 81.65 %.
Page items get Name, Visible, Locked, the interface change counts,
OverriddenPageItemProps, the layout constraints and the gradient angles
and highlights, read from their chunks and attribute lists, with the IDML
defaults where a chunk is absent. Groups take the gradient values their
children share. tools/element_values.py collects the values every IDML
of a version has on an element; the converter writes those of page items
and their export options, now for every version.
Value coverage on trustworthy pairs: 62.62 % to 80.03 %.
compare.py writes values.tsv and values-all.tsv with every key's values and
how many are reproduced, so a change can be checked for keys that lose
reproduced values. indd dump --full prints every byte of each chunk.
root_values.py reads the current inventory layout and keeps to the pairs
its generated files were made from.
- InDesign 5.0: read document chunk 0x13501 with the 3.0/4.0 layout
(four UID lists); the 6.0 layout failed on 17 files.
- Reject files that are shorter than their database pages as truncated,
and check the signature before the size so that short files of other
kinds are reported as not INDD files. The corpus tests accept both.
- Hyperlink page destinations: write view setting 0 as Fixed (567 of 567
in the pairs); leave out a zoom outside the schema's 5-4000 %.
- Leave out, with a warning, values the IDML schema does not allow: ink
neutral density -1, writing script 0xFFFFFFFF, negative drop shadow
size and other out-of-range transparency values. The IDML writer now
returns its own warnings.
- Bullet list: the font style can be followed by 1 and four bytes.
- InDesign 2.0: in-object strings start with 1, not 2.
- InDesign 2.0-5.0: XML tag names are a flag byte and a string, in
either byte order.
- InDesign 3.0 and 4.0: the document's hyperlink lists start at offset
0 and include page destinations.
- XMP: the packet can be shorter than its object; take its length.
- InDesign 1.x files (no signature, DOCUMENT at 0x5C) are reported as
not supported; files whose page directory lies beyond the database
pages as having no object database.
- Schema: leave out hyperlink sources that hold an anchored object, and
the characters U+FFFE and U+FFFF, which XML does not allow.
tests/corpus.rs accepts the two rejections above and skips the paths
listed in the local corpus/exclude.txt.
Remove the sample documents, PDF and licence copies from tests/fixtures/.
tests/fixtures/manifest.json lists each file's download URL (pinned to a
commit for GitHub sources), size and SHA-256. tools/fetch_fixtures.py
downloads them into the git-ignored tests/fixtures/files/ and verifies
them. The fixture tests read from there and skip when it is absent.