`read_previews` also collects the page items of graphic table cells and
the items of form field states, which the package holds as well. Over
the open corpus this adds previews for 657 graphics in cells and 385 in
states; every cell graphic without `Contents` has one. Items anchored in
cell text were already in the story anchors.
A preview chain that can't be read gives no preview instead of failing
the graphic, and with it the conversion.
The README and ARCHITECTURE.md say that every conversion looks up each
graphic's preview object; only `convert` reads the preview data.
`indd::convert` now returns `Conversion::previews`: InDesign's screen
preview (chunk 0x170D -> class 0x1708 -> chunk 0x119 -> raw data) of each
placed graphic without `Contents`, one entry per preview file with the
`Self` of every graphic that uses it. IDML has no place for a preview, so
the package is unchanged; an application can show the preview when it
cannot read a linked file. `convert_into` and `indd audit` do not read
previews.
Output: `diff_outputs.py --exclude own/` over 23 files (the fixtures and
part of the open corpus): 0 differing outputs, 0 changed warnings or
failures.
Page items:
- drop shadow KnockedOut (0x1EB6F), true where neither item nor style
stores it
- text wrap: the u32 at 40 of 0x3703 is the master page flag and the
wrap side (two u16, as the big-endian pair shows)
- groups get the fill, stroke and corner values all their children have
- dashes (0x5A35: list and corner code) on items whose stroke type in
effect is Dashed
- BeforeGroupingLayerPosition (0x324) from 21.4; export SizeType
(0x1E22B); no generated alt text attributes on groups
- anchored settings on items inside anchored items
- IgnoreWrap false without the chunk, MinimumFirstBaselineOffset 0 by
the baseline category, InsetSpacing before DOM 11 by the general
category; EPS contents from 0x8C23
- custom striped and dashed stroke styles (classes 0xB016, 0x5A43) and
StrokeType code 0x5A42
Preferences: column direction (0x555, else the story orientation),
optical margin size by the text size unit, highlight flags false without
their chunk, grid and gutter defaults and print marks by the edition
code, watermark font by the assignment name, EPUB ViewDocumentAfterExport
for DOM 8 and 9, HTMLFXLExportPreference from DOM 10.1, MathML document
values from DOM 20, index header type by variant name, three more
built-in kinsoku and mojikumi tables.
Resources and text: the saving edition from the XMP black name (tints
of black, unknown users, endnote and index titles, endnote marker), TOC
style entries, PageNumberType (0x1B09), CustomGlyph (0x1B5E), alignment
codes of 0x4266, 0x1B3E and 0xB676, spot colour alias and DOM 8 section
pagination (observed values), text variable settings and result texts,
object styles' AppliedParagraphStyle without a chunk and flex spacing,
stories of frames anchored in the XML backing story.
Fixes: flattener resolutions of 0x10833 were swapped; colour group Self
follows the DOM version.
compare.py, trustworthy pairs: value coverage 98.96 % -> 99.48 %
(+150,288 values), documents >= 99 % 914 -> 1,021, story text 37,129 ->
37,147 of 37,150 (0 missing), extras 92,838 -> 92,883 (+788 from the
KnockedOut and group-corner exceptions, -744 from fixed extras). No pair
loses values against f21267f. diff_outputs.py: 5,743 of 5,765 outputs
differ, no failure or warning differs.
Trustworthy coverage 97.91 % -> 98.46 % (28,404,742 of 28,847,647
values), all pairs 96.64 % -> 97.28 %, documents at 99 % or more
186 -> 193 of 1,251, story text 37,112 -> 37,129 of 37,150 exact,
extra values 93,520 -> 93,502. The form field commit gains about
133,000 values, not 150,000 as its message says.
Every composite font is written, in UID order. Chunk 0xCB02 is read as
four flagged strings, a u32, a u16 and the entry list, which ends the
chunk; the count search remains for other layouts (it took a wrong
position in one 12.1 document). Entries get RelativeSize and
BaselineShift (the first two f64), HorizontalScale and VerticalScale,
Locked from the u16 after the numbers (204 false), and no
CustomCharacters on the first entry of each font, the base font (1,434
of 1,434). Text attribute 0x1BFA is ParagraphBorderGapColor, 0 being
Swatch/None. Evidence in fonts.md and attributes.md.
compare.py --exclude own/, trustworthy pairs: 97.84 % -> 97.91 %
(+17,964 values, 0 lost on any key); CompositeFontEntry 7,306 -> 8,589
of 8,589 elements (AppliedFont 8,553: one document whose IDML names its
families in English), CompositeFont 1,434 of 1,434, TextRange
PSR.P.ParagraphBorderGapColor 0 -> 2,745 of 2,745. Extras 93,569 ->
93,520: the 49 CustomCharacters extras are gone.
tools/root_values.py collects the object styles other than [None] and
the preferences from the pairs compare.py compares, like the root
styles, instead of the part of the corpus it had first: 1,585 pairs,
1,448 distinct IDML files. tools/element_values.py reads 1,495 distinct
IDML files. The DOM 8 to 11 merge of the DOM 12 object style frame
values is gone: the file has those versions itself.
Values that vary in the larger corpus are dropped, also where only one
document differs and the converter does not read the value from the
INDD: 209 keys, most of them preferences (EPubExportPreference,
TextDefault, PageItemDefault, IndexOptions, TextPreference), object
style values (line ends, gap colour, stroke tint, anchored object and
text wrap settings, baseline frame grid before DOM 18), Spread
PageTransitionType and PageTransitionDirection, and character style
PreviewColor. Evidence in idml-values.md.
compare.py --exclude own/, trustworthy pairs: coverage 97.79 % ->
97.09 % (-212,444 values on dropped keys, +11,501 on keys whose first
version is now earlier); documents >= 99 % 514 -> 106 of 1,251; all
pairs 96.55 % -> 95.85 %. Extras 94,610 -> 94,328. Every lost value is
on a regenerated key. No new failures; story text unchanged.
tools/root_values.py collects the root styles from the pairs that
compare.py compares (little-endian, same major version, without the
privately held samples): 794 distinct IDML files, DOM 7 to 21, instead
of the 88 the corpus had first. The pair rule leaves out an IDML with no
root styles beside a newer INDD and a big-endian pair. The other object
styles and the preferences stay pinned to SOURCE; their files are
unchanged. The root table style writes every mapped attribute of its
list, since the stroke weights, spacing and end tints now vary between
files and are no longer in the values file. The root object style's
varying values (named grid, optical margin size, baseline frame grid
colour, corner radii, gutter, first baseline offset) are read from the
INDD. Evidence in idml-values.md and tables.md.
compare.py --exclude own/, trustworthy pairs: coverage 98.53 % ->
98.56 % (+2,942 values whose first DOM version is now earlier:
IncludeClass, EpubAriaRole, paragraph border options,
ParagraphKashidaWidth, shading overprint, TransformAttributeOption and
FlexLayoutAttributeOption), documents >= 99 % 278 -> 313; all pairs
96.65 % -> 96.67 %. No pair loses a reproduced value; extras unchanged
(8,968 trustworthy, 278,333 all). No new failures.
tools/check_wasm.sh builds the library for wasm32-unknown-unknown,
then builds a probe module (a cdylib generated under the target
directory that exports a function calling `indd::convert`) and checks
with Node.js that the module imports nothing: a conversion needs no
file system, clock or other host function. It then converts the
fetched fixtures, or the files given, in the probe module and compares
each package byte for byte with the native `indd convert` output.
No library change was needed: only `convert_file` and `read_header`
use `std::fs`, and `convert` does not call them. The 5 fixtures and 80
corpus files give the same packages in WebAssembly as natively. The
check takes about 10 s; it is documented in CLAUDE.md, CONTRIBUTING.md,
README.md and ARCHITECTURE.md. There is no hosted CI.
Idea from storytold/designcraft#98.
`src/hostile.rs` converts damaged inputs and fails if a conversion
panics or takes longer than 20 s:
- Synthetic documents built from a seed with `database::synthetic`: a
document object with well-formed lists, spreads that hold spread
layers, pages and page items, and objects of the model's classes with
chunks of the model's IDs (taken from the hexadecimal constants of the
model sources). Chunk data is shaped like UID lists, child lists,
attribute lists, strings and numbers, or random. Each document is
converted as built, with one object truncated, with bytes flipped, and
with the file image truncated, through the model and the IDML writer.
- The fixture files, if fetched: truncated at structure boundaries and
random lengths, and with 1 to 64 bytes flipped, a quarter of the cases
in the header and master pages.
The cases come from a fixed seed and run on all cores, with a watchdog
for hangs. The default sweep has about 1,600 cases and takes 2 s in a
debug build and 0.5 s in a release build (16 cores).
INDD_HOSTILE_SCALE multiplies the cases and INDD_HOSTILE_SEED changes
the seed; sweeps at scale 40 with seeds 0x1DD and 7 (about 63,000 cases
each, debug build with overflow checks) found no panic or hang.
Idea from storytold/designcraft#98.
The README listed implemented features as not started, an old attribute
count, corpus size and numbers, and no library use. The numbers are from
compare.py --all with schema validation at this revision.