diff --git a/docs/format/idml-values.md b/docs/format/idml-values.md index d84501d..3103d0d 100644 --- a/docs/format/idml-values.md +++ b/docs/format/idml-values.md @@ -237,6 +237,34 @@ Every spread has one `FlattenerPreference`, so the converter writes the element on every spread. The other attributes of these elements come from the INDD (`objects.md`). +### Document settings in designmap.xml + +These elements are children of `Document` without `Self`. The tool keeps +an element when every IDML from some version on has it, with the values +all of them have; the converter writes it in the order of the IDML files. + +| Element | From DOM | Values kept | Left out (vary) | +|---|---|---|---| +| `ConditionalTextPreference` | 7 | both | | +| `EndnoteOption` | 13 | 9 | `EndnoteTitle`, `EndnoteSeparatorText`, the marker and text styles, `EndnoteMarkerPositioning` | +| `TextFrameFootnoteOptionsObject` | 12 | all 4 | | +| `LinkedStoryOption` | 8 | all 4 | | +| `LinkedPageItemOption` | 8 | all 7 | | +| `WatermarkPreference` | 7 | 11 | the font family, style and colour | +| `TaggedPDFPreference` | 8 | `StructureOrder` | | +| `AdjustLayoutPreference` | 14 | 4 | `EnableAdjustLayout`, `EnableAutoAdjustMargins`, `AllowFontSizeAndLeadingAdjustment` | +| `HTMLFXLExportPreference` | 11 | both | | +| `PublishExportPreference` | 11 | 11 | `ImageExportResolution`, `PublishPdf` | + +### Language quotes + +The quotes of a language are not in its INDD object (`objects.md`, +languages). For each language name the tool keeps `SingleQuotes` and +`DoubleQuotes` when every IDML `Language` element with that name has the +same value, in at least 3 files (a `Keyed` block of the values file): +63 languages. Romanian and Japanese have two variants each and are left +out. + ### Trap presets All 683 IDML files list the same two `TrapPreset` elements at the end of diff --git a/docs/format/objects.md b/docs/format/objects.md index 1725ba5..025930f 100644 --- a/docs/format/objects.md +++ b/docs/format/objects.md @@ -64,6 +64,22 @@ are in `big-endian.md`. | 0x313 | u32: active layer (IDML `ActiveLayer`) | | 0x222 | Two UID lists: the stories, then the XML backing story; IDML `StoryList` is both (`xml.md`) | | 0x4C01 | UID list: sections | +| 0xA443 | Document users: u32 count, then per user a flag byte, the name as an in-object string, u32 colour | + +**Document users.** IDML lists one `DocumentUser` per user in chunk +0xA443, `Self` being `dDocumentUser` and the index in hexadecimal (491 of +495 trustworthy pairs have as many users as the chunk; the other 4 are +DOM 7 and 16 files with one user more in the IDML). A user with flag 0 +has its stored name as `UserName` (1,724 of 1,724). Flag 2 marks the +placeholder for an unknown user, stored with a name such as +`Unknown User Name` or the same words in the language of the computer +that saved it; IDML writes `$ID/Unknown User Name` for 495 of the 586 +such users, and the stored name for the other 91. The schema requires +`UserName`, so the converter writes `$ID/Unknown User Name` for every +flag 2 user. The colour UID names an interface colour, but the IDML +`UserColor` does not follow from it (users stored with 0.6, 0.4, 0 are +`BrickRed` in 4 files and `Gold` in 1), so it is left out of the +conversion and of the measurement (`measurement.md`). ## Spreads (0x501) and master spreads (0x1401) @@ -199,6 +215,31 @@ true in every layer; the two fields that are 1 in every sample may hold them. The converter reads the fields only when the name starts at offset 19 (in all little-endian corpus pairs, 1,465 layers). +## Languages (0x2D07) + +Chunk 0x2D0F: a flag byte and the name (`English: USA`), a flag byte and +the primary name (`English`), a flag byte and the secondary name (`USA`, +often empty), u16 `Id`, then two vendors, each a flag byte, a u32 and a +string: first the spelling vendor, then the hyphenation vendor +(`Hunspell`, `Proximity`, `Duden`, `WinSoft`); last the locale +(`en_US`). + +IDML lists the language objects in UID order as `Language` elements at +the start of `designmap.xml`: `Name`, `PrimaryLanguageName` and +`SublanguageName` are `$ID/` and the stored names, except that the +language named `Neutral` is `$ID/[No Language]` in all three. Evidence +over the trustworthy pairs: in 488 of 495 the IDML list equals the INDD +objects in that order; in the other 7 the INDD has 4 to 61 language +objects and the IDML lists only some of them (1 or 2), for no reason +found. For the 1,249 IDML languages the names and `Id` are equal in +1,249. `HyphenationVendor` is the second vendor and `SpellingVendor` the +first when their flag is 1 (1,146 and 1,166 of 1,249); otherwise the +stored string is empty or names a vendor the IDML does not show, and +IDML gives `$ID/`, `$ID/InDihyph` or a vendor such as `Hunspell` that +depends on the computer that exported it, so the converter leaves the +attribute out. The quotes (`SingleQuotes`, `DoubleQuotes`) are not in the +chunk; they are written from observation (`idml-values.md`). + ## Interface colours (0x1F11) Layers, pages and XML tags name their colour by the UID of an object of diff --git a/docs/measurement.md b/docs/measurement.md index 3a1c2fd..263304b 100644 --- a/docs/measurement.md +++ b/docs/measurement.md @@ -140,6 +140,7 @@ are left out of the counts (`EXCLUDED` in `compare.py`). |---|---|---| | `Font` | `Status` | It records whether the font is installed on the computer that exported the IDML. The same document gives a different value on another computer. | | `idPkg:Story`, `idPkg:Spread` and the other part references in `designmap.xml` | `src` | The file names of the parts inside the IDML package. The IDML writer chooses them; they are not document content. | +| `DocumentUser` | `UserColor` | The colour of each user of the document as the exporting InDesign shows it. The INDD stores a colour for each user (`objects.md`, document users), but the IDML colour does not follow from it: users stored with the same colour get different IDML colours in different files. | A value enters this list only with a reason of this kind. Values that are hard to decode, or that the converter does not write yet, stay in the diff --git a/src/idml/element_values.xml b/src/idml/element_values.xml index 7ea5660..2aa9aad 100644 --- a/src/idml/element_values.xml +++ b/src/idml/element_values.xml @@ -33,6 +33,12 @@ tools/element_values.py; evidence in docs/format/idml-values.md. + + + + + + @@ -79,6 +85,11 @@ tools/element_values.py; evidence in docs/format/idml-values.md. + + + + + @@ -99,6 +110,38 @@ tools/element_values.py; evidence in docs/format/idml-values.md. + + + + + + + + + + + + + + + + + + + + Arabic + Continuous + EndnoteDocumentScope + NewPage + NoPrefixSuffix + + + + + + + + @@ -127,4 +170,69 @@ tools/element_values.py; evidence in docs/format/idml-values.md. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/src/idml/mod.rs b/src/idml/mod.rs index ebcc2bc..e3d593a 100644 --- a/src/idml/mod.rs +++ b/src/idml/mod.rs @@ -1155,6 +1155,7 @@ impl Writer<'_> { if let Some(l) = doc.active_layer { x.attr("ActiveLayer", uref(Some(l))); } + self.languages(&mut x); x.empty("idPkg:Graphic", &[("src", "Resources/Graphic.xml".into())]); x.empty("idPkg:Fonts", &[("src", "Resources/Fonts.xml".into())]); // Kinsoku tables, then mojikumi tables, as the schema orders them. @@ -1189,10 +1190,32 @@ impl Writer<'_> { ("ContinueNumbersAcrossDocuments", "false".into()), ], ); + let major = doc.version.major; + // Elements every IDML of the version has, with the values they + // all have (idml-values.md), in the order of the IDML files. + let singleton = |x: &mut Xml, tag: &str| { + if let Some(n) = values::present(&format!("Document/{tag}"), major) { + n.write(x); + } + }; + singleton(&mut x, "ConditionalTextPreference"); x.empty( "idPkg:Preferences", &[("src", "Resources/Preferences.xml".into())], ); + for tag in [ + "EndnoteOption", + "TextFrameFootnoteOptionsObject", + "LinkedStoryOption", + "LinkedPageItemOption", + "WatermarkPreference", + "TaggedPDFPreference", + "AdjustLayoutPreference", + "HTMLFXLExportPreference", + "PublishExportPreference", + ] { + singleton(&mut x, tag); + } self.text_variables(&mut x); x.empty("idPkg:Tags", &[("src", "XML/Tags.xml".into())]); for l in doc.layers.iter().filter(|l| !l.internal) { @@ -1263,6 +1286,23 @@ impl Writer<'_> { } x.end(); } + // Document users. A user with flag 2 is the placeholder for an + // unknown user, which IDML names `$ID/Unknown User Name`; the + // colours are left out (docs/format/objects.md, document users). + for (i, (flag, name)) in doc.users.iter().enumerate() { + let name = if *flag == 2 { + "$ID/Unknown User Name" + } else { + name.as_str() + }; + x.empty( + "DocumentUser", + &[ + ("Self", format!("dDocumentUser{i:x}")), + ("UserName", name.into()), + ], + ); + } self.cross_reference_formats(&mut x); x.empty( "idPkg:BackingStory", @@ -1305,6 +1345,47 @@ impl Writer<'_> { x.finish() } + /// The document's languages, in UID order. See + /// `docs/format/objects.md`, languages. + fn languages(&self, x: &mut Xml) { + let mut list: Vec<_> = self.doc.language_list.iter().collect(); + list.sort_by_key(|l| l.uid); + for l in list { + // The language without one is `Neutral` in the INDD. + let neutral = l.name == "Neutral"; + let name = |s: &str| { + if neutral { + "$ID/[No Language]".to_string() + } else { + format!("$ID/{s}") + } + }; + let full = name(&l.name); + x.start("Language") + .attr("Self", format!("Language/{}", self_name(&full))) + .attr("Name", &full); + if let Some(q) = values::keyed("Language", &full) { + for k in ["SingleQuotes", "DoubleQuotes"] { + if let Some(v) = q.attr(k) { + x.attr(k, v); + } + } + } + x.attr("PrimaryLanguageName", name(&l.primary)) + .attr("SublanguageName", name(&l.sub)) + .attr("Id", l.id.to_string()); + if let Some([(sf, spelling), (hf, hyphenation)]) = &l.vendors { + if *hf == 1 { + x.attr("HyphenationVendor", hyphenation); + } + if *sf == 1 { + x.attr("SpellingVendor", spelling); + } + } + x.end(); + } + } + /// Colour groups and their swatches. See `docs/format/objects.md`. fn color_groups(&self, x: &mut Xml) { for (i, g) in self.doc.color_groups.iter().enumerate() { diff --git a/src/idml/values.rs b/src/idml/values.rs index 0145a12..050dd37 100644 --- a/src/idml/values.rs +++ b/src/idml/values.rs @@ -242,6 +242,18 @@ pub fn element_attrs(path: &str, major: u32) -> Vec<(String, String)> { .unwrap_or_default() } +/// The element on `path` every IDML of InDesign version `major` in the +/// corpus has (`Present="true"` in `element_values.xml`), with its +/// observed values; `None` if not every IDML has it. +pub fn present(path: &str, major: u32) -> Option { + let mut n = element(path, major)?; + if n.attr("Present") != Some("true") { + return None; + } + n.attrs.retain(|(k, _)| k != "Present"); + Some(n) +} + /// The list of `tag` elements every IDML of InDesign version `major` in /// the corpus has, in order (`List` blocks of `element_values.xml`); /// empty if there is none. @@ -254,6 +266,21 @@ pub fn list(tag: &str, major: u32) -> Vec { .collect() } +/// The observed values of a `tag` element whose key attribute (such as +/// `Name`) has `value` (`Keyed` blocks of `element_values.xml`). +pub fn keyed(tag: &str, value: &str) -> Option { + let block = element_values() + .children + .iter() + .find(|b| b.tag == "Keyed" && b.attr("Tag") == Some(tag))?; + let key = block.attr("Key")?; + block + .children + .iter() + .find(|n| n.attr(key) == Some(value)) + .cloned() +} + fn unescape(s: &str) -> String { s.replace("<", "<") .replace(">", ">") @@ -390,6 +417,11 @@ mod tests { // Export options differ between versions. assert!(new.attr("EpubType").is_some()); assert_eq!(list("TrapPreset", 21).len(), 2); + let en = keyed("Language", "$ID/English: USA").unwrap(); + assert_eq!(en.attr("DoubleQuotes"), Some("\u{201C}\u{201D}")); + let w = present("Document/WatermarkPreference", 21).unwrap(); + assert!(w.attr("Present").is_none()); + assert!(present("Document/EndnoteOption", 12).is_none()); assert!( element("Polygon/ObjectExportOption", 8) .unwrap() diff --git a/src/model/mod.rs b/src/model/mod.rs index b1d29f4..eea3dec 100644 --- a/src/model/mod.rs +++ b/src/model/mod.rs @@ -76,6 +76,9 @@ pub mod chunk { pub const DOC_ACTIVE_LAYER: u32 = 0x313; pub const DOC_STORIES: u32 = 0x222; pub const DOC_SECTIONS: u32 = 0x4C01; + /// Document users: u32 count, then per user a flag byte, the name + /// and u32 colour. + pub const DOC_USERS: u32 = 0xA443; pub const SPREAD_CHILDREN: u32 = 0x503; pub const SPREAD_TRANSFORM: u32 = 0x56E; pub const SPREAD_BINDING: u32 = 0x1B8; @@ -608,6 +611,8 @@ pub struct Document { pub version: Version, pub layers: Vec, pub active_layer: Option, + /// Document users (chunk 0xA443): flag byte and name. + pub users: Vec<(u8, String)>, pub spreads: Vec, pub master_spreads: Vec, pub stories: Vec, @@ -620,6 +625,8 @@ pub struct Document { pub swatches: BTreeMap, /// Font families by UID. pub fonts: BTreeMap, + /// The language objects (class 0x2D07) in UID order. + pub language_list: Vec, /// Language name (IDML `AppliedLanguage` without `$ID/`) for each /// language UID. pub languages: BTreeMap, @@ -731,6 +738,19 @@ pub mod numbering { pub const KANJI: u32 = 0x4C12; } +/// A language object (class 0x2D07), chunk 0x2D0F. See +/// `docs/format/objects.md`, languages. +#[derive(Debug, Clone, PartialEq)] +pub struct Language { + pub uid: u32, + pub name: String, + pub primary: String, + pub sub: String, + pub id: u16, + /// The spelling and hyphenation vendors, each with its flag byte. + pub vendors: Option<[(u8, String); 2]>, +} + /// A bullet character of the document's list (IDML `ABullet`): u32 /// character type, u32 character value, u32 font family (0 = none), the /// font style as a flag byte and string, then a byte (0 in every sample). @@ -872,6 +892,7 @@ impl<'a> Reader<'a> { let mut swatches = BTreeMap::new(); let mut fonts = BTreeMap::new(); let mut languages = BTreeMap::new(); + let mut language_list = Vec::new(); let mut style_groups = BTreeMap::new(); let mut object_styles = BTreeMap::new(); let mut cell_styles = BTreeMap::new(); @@ -1182,7 +1203,35 @@ impl<'a> Reader<'a> { { let mut c = Cursor::new(&d); c.flag()?; - languages.insert(uid, c.string()?); + let name = c.string()?; + languages.insert(uid, name.clone()); + // Then the primary and secondary names, u16 + // ID, and two vendors (flag, u32, string). + let rest = (|| -> Result<_, Error> { + c.flag()?; + let primary = c.string()?; + c.flag()?; + let sub = c.string()?; + let id = c.u16()?; + let mut vendor = || -> Result<(u8, String), Error> { + let flag = c.u8()?; + c.u32()?; + Ok((flag, c.string()?)) + }; + let spelling = vendor()?; + let hyphenation = vendor()?; + Ok((primary, sub, id, [spelling, hyphenation])) + })(); + if let Ok((primary, sub, id, vendors)) = rest { + language_list.push(Language { + uid, + name, + primary, + sub, + id, + vendors: Some(vendors), + }); + } } } _ => {} @@ -1245,10 +1294,15 @@ impl<'a> Reader<'a> { swatches, fonts, languages, + language_list, style_groups, object_styles, cell_styles, table_styles, + users: self.users(doc).unwrap_or_else(|e| { + self.warn(format!("document users left out: {e}")); + Vec::new() + }), sections: self .uid_list(doc, chunk::DOC_SECTIONS)? .into_iter() @@ -1803,6 +1857,26 @@ impl<'a> Reader<'a> { }) } + /// The document's users, from chunk 0xA443 of the document. + fn users(&self, doc: u32) -> Result, Error> { + let Some(d) = self.chunk(doc, chunk::DOC_USERS)? else { + return Ok(Vec::new()); + }; + let mut c = Cursor::new(&d); + let n = c.u32()?; + if n as usize > d.len() / 9 { + return Err(Error::Corrupt(format!("{n} document users"))); + } + (0..n) + .map(|_| { + let flag = c.flag()?; + let name = c.string()?; + c.u32()?; + Ok((flag, name)) + }) + .collect() + } + /// The settings every page item has (`ItemProps`). fn item_props(&self, uid: u32) -> Result { let name_chunk = if self.class(uid) == Some(class::GROUP) { diff --git a/tools/compare.py b/tools/compare.py index f2dc87d..f307b21 100644 --- a/tools/compare.py +++ b/tools/compare.py @@ -320,6 +320,8 @@ EXCLUDED = [ ("Font", "Status", "whether the font is installed on the exporting computer"), ("Document/{http://ns.adobe.com/AdobeInDesign/idml/1.0/packaging}*", "src", "file names of the package parts, chosen by the IDML writer"), + ("DocumentUser", "P.UserColor", + "colour the exporting InDesign gives each user, not the colour in the INDD"), ] diff --git a/tools/element_values.py b/tools/element_values.py index 7e5cd97..44ac679 100644 --- a/tools/element_values.py +++ b/tools/element_values.py @@ -22,7 +22,8 @@ has one (`Document/WatermarkPreference`); for them the presence of the element is kept the same way, and the converter writes the element. LISTS are elements with `Self` whose whole list (every element, with all attributes and children, in order) is the same in every file of a range; -the converter writes the list. +the converter writes the list. KEYED values depend only on another +attribute of the element, such as the quotes of a language on its name. EXPLAINED lists values that are the same in nearly every element; the others are values the converter reads from the INDD (the reason says @@ -62,11 +63,22 @@ ELEMENTS = [ "Oval/ObjectExportOption", "Polygon/ObjectExportOption", "GraphicLine/ObjectExportOption", "Group/ObjectExportOption", "Spread", "MasterSpread", "Page", "Spread/FlattenerPreference", "Layer", + "Document/ConditionalTextPreference", "Document/EndnoteOption", + "Document/TextFrameFootnoteOptionsObject", "Document/LinkedStoryOption", + "Document/LinkedPageItemOption", "Document/WatermarkPreference", + "Document/TaggedPDFPreference", "Document/AdjustLayoutPreference", + "Document/HTMLFXLExportPreference", "Document/PublishExportPreference", ] # Of those, the elements every IDML has one of (from some version on). SINGLETONS = {p for p in ELEMENTS if p.startswith("Document/")} # Elements with Self written as a whole list. LISTS = ["TrapPreset"] +# Values that depend only on another attribute of the element: tag -> +# (key attribute, attributes). For each key value, an attribute is kept +# when every such element in every IDML has the same value, in at least +# MIN_KEYED_FILES files. +KEYED = {"Language": ("Name", ["SingleQuotes", "DoubleQuotes"])} +MIN_KEYED_FILES = 3 # Values that are nearly constant; the exceptions are read from the INDD. # (path, key) -> reason, as recorded in docs/format/idml-values.md. EXPLAINED = {} @@ -134,6 +146,8 @@ def scan(arg): for el in root.iter(): if el.tag in LISTS: lists[el.tag].append(prop_text(el)) + if el.tag in KEYED: + lists["keyed:" + el.tag].append(dict(el.attrib)) if el.get("Self") is None: continue if el.tag in wanted: @@ -267,7 +281,30 @@ tools/element_values.py; evidence in docs/format/idml-values.md. """ -def write(kept, presence, lists, order): +def analyse_keyed(scans): + """{tag: [(key value, [(attr, value)], files)]} for KEYED.""" + out = {} + for tag, (key, attrs) in KEYED.items(): + seen = defaultdict(lambda: defaultdict(set)) # key value -> attr -> values + files = defaultdict(set) + for _dom, idml, _found, ls in scans: + for a in ls.get("keyed:" + tag, []): + k = a.get(key) + files[k].add(idml) + for name in attrs: + seen[k][name].add(a.get(name)) + rows = [] + for k in sorted(seen, key=lambda v: (v is None, v)): + if k is None or len(files[k]) < MIN_KEYED_FILES: + continue + kept = [(n, next(iter(v))) for n, v in seen[k].items() if len(v) == 1 and None not in v] + if kept: + rows.append((k, kept, len(files[k]))) + out[tag] = rows + return out + + +def write(kept, presence, lists, order, keyed): blocks = defaultdict(lambda: defaultdict(list)) # (first,last) -> path -> [(key,v)] for path, items in kept.items(): for key, first, last, v, _n in items: @@ -315,6 +352,13 @@ def write(kept, presence, lists, order): s += f'\t\n' s += "".join(f"\t\t{e}\n" for e in lst) s += "\t\n" + for tag, rows in keyed.items(): + key = KEYED[tag][0] + s += f'\t\n' + for k, attrs, _n in rows: + a = "".join(f" {n}={quoteattr(v)}" for n, v in [(key, k)] + attrs) + s += f"\t\t<{tag}{a} />\n" + s += "\t\n" OUT.write_text(s + "\n") @@ -360,8 +404,12 @@ def main(): for tag, rs in lists.items(): print(f"list {tag}: " + ", ".join( f"{len(v)} elements, {f}{'+' if l is None else f'-{l}'} ({n} files)" for f, l, v, n in rs)) + keyed = analyse_keyed(scans) + for tag, rows in keyed.items(): + n_attrs = Counter(n for _, attrs, _ in rows for n, _ in attrs) + print(f"keyed {tag} by {KEYED[tag][0]}: {len(rows)} values, kept {dict(n_attrs)}") if "--write" in sys.argv: - write(kept, presence, lists, order) + write(kept, presence, lists, order, keyed) print(f"wrote {OUT.relative_to(ROOT)}")