From 27bb1ebf065a58fc415ac9eb05c04a834d3cbb38 Mon Sep 17 00:00:00 2001 From: totoz64 <47389794+totoz64@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:19:25 +0100 Subject: [PATCH 001/161] CP: Adapt French proofing to shared Language packs Keep French locales in Pack(PackLocale), reusing the shared dictionary cache, affix reader and suggestion engine. Add French elisions, bounded accent and split tuning, conservative typography and Quebec punctuation rules. Validate displayed suggestion casing before OCONV and retain the original eligible form when an explicit target is excluded. Preserve historical text fixtures with three documented pre-existing upstream chain differences. Ported from WordCraft #3185; no dictionary or hyphenation data is bundled. --- crates/proof/README.md | 41 ++ crates/proof/src/affix.rs | 66 ++ crates/proof/src/affix_suggest.rs | 30 + crates/proof/src/fr/grammar.rs | 665 ++++++++++++++++++ crates/proof/src/fr/mod.rs | 181 +++++ crates/proof/src/fr/suggest.rs | 81 +++ crates/proof/src/fr/test_dict.rs | 234 ++++++ crates/proof/src/hunspell.rs | 17 +- crates/proof/src/lib.rs | 5 + crates/proof/src/pack_lang.rs | 8 +- crates/proof/tests/affix_compatibility.rs | 92 +++ .../fixtures/affix_compatibility.compact | 7 + .../fixtures/affix_compatibility.expected | 497 +++++++++++++ .../fixtures/affix_compatibility_cases.rs | 61 ++ crates/proof/tests/french_dictionary.rs | 107 +++ crates/proof/tests/french_packs.rs | 124 ++++ 16 files changed, 2210 insertions(+), 6 deletions(-) create mode 100644 crates/proof/src/fr/grammar.rs create mode 100644 crates/proof/src/fr/mod.rs create mode 100644 crates/proof/src/fr/suggest.rs create mode 100644 crates/proof/src/fr/test_dict.rs create mode 100644 crates/proof/tests/affix_compatibility.rs create mode 100644 crates/proof/tests/fixtures/affix_compatibility.compact create mode 100644 crates/proof/tests/fixtures/affix_compatibility.expected create mode 100644 crates/proof/tests/fixtures/affix_compatibility_cases.rs create mode 100644 crates/proof/tests/french_dictionary.rs create mode 100644 crates/proof/tests/french_packs.rs diff --git a/crates/proof/README.md b/crates/proof/README.md index 970670143..b4b4b5ba8 100644 --- a/crates/proof/README.md +++ b/crates/proof/README.md @@ -16,6 +16,7 @@ craft-proof = { git = "https://github.com/storytold/craft-libs", rev = " | English (en-US, en-CA, en-AU, en-GB and the British-spelling Englishes such as en-NZ) | SCOWL 2026 Hunspell dictionary per English + `en-extra.txt`, dialect policy (`en-variants.txt`); a dictionaries-folder English first; without SCOWL the installed Hunspell English, else the built-in Moby list ([below](#english-is-always-checked)) | yes | Moby list, our patterns, heuristic | | Polish, Portuguese, Italian | the installed Hunspell dictionary | yes | TeX patterns (embedded, or a Language pack) | | Croatian, Turkish, Russian, Czech, Indonesian, Spanish (es, es-MX, es-419, es-UY) | the installed Hunspell dictionary (Russian also `ru-extra.txt`) | yes | Turkish and Russian rules; English fallback otherwise | +| French (fr-FR, fr-CA and other French pack locales) | the installed Language pack or Hunspell dictionary; dictionary-driven elisions and accent-oriented suggestions | conservative typography and basic checks, with Quebec punctuation spacing in fr-CA | the Language pack (generic fr fallback) | | German | — | — | TeX patterns (embedded, or a Language pack) | | Bulgarian | — | — | syllable rules | @@ -41,6 +42,12 @@ application data (`%LOCALAPPDATA%\WordCraft\dictionaries` and `%APPDATA%\WordCra `~/Library/Application Support/WordCraft/dictionaries`, `$XDG_DATA_HOME/wordcraft/dictionaries`), then the usual Hunspell and LibreOffice locations. None on the web. +French remains a `ProofLang::Pack` locale: use its full `tag()` for settings and issue identity. +The shared pack loader keeps regional dictionary overrides, fallback to `fr`, reloads and its +memory budget; no separate French dictionary slot or embedded French patterns are added. +Only French packs receive the French spelling tuning and typography; other pack languages +keep their existing behavior. French agreement and conjugation are outside these checks. + State is process-wide: `add_word`, `ignore_word`, `set_dictionaries_folder`, `load_user_dictionary`/`save_user_dictionary`. The crate reads no document model; callers pass plain text and language spans. @@ -221,6 +228,40 @@ CI: `.github/workflows/proof.yml`. storytold/craft-libs#30/#31 by brzuchal (Michał Marcin Brzuchalski) started the shared crate. +### French adaptation validation + +The French rules and original synthetic fixtures are adapted from +[totoz64’s WordCraft contribution #3185](https://github.com/storytold/wordcraft/pull/3185). +They use the shared affix reader and Language pack cache; WCAFF1 remains deliberately rejected. +The historical text fixtures retain their original expected data. Three observations for +Indonesian `bermainkanlah` are explicitly updated in the compatibility test because the +shared reader already supports that prefix plus two suffixes. This is a finite regression +check, not a universal compatibility claim. + +The ordinary tests cover French packs, regional overrides and reloads, options, user words, +NFC/NFD lookup with original ranges, suggestions and pack hyphenation. The opt-in real French +dictionary gate fails when its input is missing; it never silently skips its assertions: + +```sh +WORDCRAFT_DICTIONARIES=/absolute/isolated/french-dictionary cargo test -p craft-proof --test french_dictionary -- --ignored +``` + +For a reproducible real-dictionary run, use Dicollecte 7.7 from LibreOffice/dictionaries +revision `67032ee957631908098a72edf48dc81cac27e369`: +[fr.aff](https://raw.githubusercontent.com/LibreOffice/dictionaries/67032ee957631908098a72edf48dc81cac27e369/fr_FR/dictionaries/fr.aff) and +[fr.dic](https://raw.githubusercontent.com/LibreOffice/dictionaries/67032ee957631908098a72edf48dc81cac27e369/fr_FR/dictionaries/fr.dic). +Verify SHA-256 before invoking the gate (the test checks location and behavior, not hashes): + +| File | SHA-256 | +|---|---| +| fr.aff | `c176610cd5dc4846806a65ddd029f422d87978bf58f224aa44222662a16a2de5` | +| fr.dic | `b78a868e31dd6e373b6c3217969afb898a9acde828a5e7ef97308da42218c88c` | + +Use a dedicated directory containing only the intended `fr.aff`/`fr.dic` pair (or explicit +regional pairs for both regions). The gate requires more than 50,000 dictionary entries and +checks the actual source folder, spelling, accent/elision suggestions, regional dispatch, +combining-mark ranges and hyphenated compounds. No real dictionary is committed here. + ## Owners WordCraft maintainers: echelon, dylanmtaylor, aiongg. diff --git a/crates/proof/src/affix.rs b/crates/proof/src/affix.rs index 0df68665b..32de07659 100644 --- a/crates/proof/src/affix.rs +++ b/crates/proof/src/affix.rs @@ -2212,6 +2212,37 @@ impl AffixDict { self.check_except(w, self.excluded(true)) } + /// Validate a collected suggestion in its displayed case, before OCONV. Only identity, + /// first-letter and uppercase variants are allowed (including uppercase that keeps sharp s). + /// An exact target excluded from suggestions cannot be rescued by a lowercase fallback. + /// This preserves the checker's homonym, compound and CHECKSHARPS rules without scanning + /// stems or changing which words the spelling checker accepts. + pub(crate) fn can_suggest_cased(&self, original: &str, changed: &str) -> bool { + if original.len() > MAX_INPUT_BYTES || changed.len() > MAX_INPUT_BYTES { + return false; + } + if changed != original + && changed != capitalize(original) + && changed != original.to_uppercase() + && changed != original.split('ß').map(str::to_uppercase).collect::>().join("ß") + { + return false; + } + let target = self.normalize(changed); + if self.is_forbidden(&target) { + return false; + } + // The exact simple-word path includes affixes; asking with no exclusions detects a + // target whose only derivations carry NOSUGGEST, SUBSTANDARD or FORBIDDENWORD. Spend + // the same shared suggestion budget as the normal checker, and abstain if it runs out. + match self.simple_word_budgeted(&target, Excluded::default(), true) { + None => return false, + Some(true) if !self.simple_word(&target, self.excluded(true)) => return false, + _ => {} + } + self.can_suggest(changed) + } + /// Whether `w` is a word only through a `KEEPCASE` stem, so it is correct in this letter case /// alone (`km` from `km/K`, which `Km` and `KM` are not). pub(crate) fn case_locked(&self, w: &str) -> bool { @@ -3298,6 +3329,41 @@ pub(crate) mod tests { assert!(d.check("ASSASSASSASSASS") && !d.check(&"SS".repeat(60))); } + #[test] + fn displayed_casing_rejects_exact_exclusions_before_case_fallback() { + for flag in ["N", "F", "S"] { + let aff = "SET UTF-8\nNOSUGGEST N\nFORBIDDENWORD F\nSUBSTANDARD S\nSFX A Y 1\nSFX A 0 s .\nSFX B Y 1\nSFX B 0 S .\n"; + let dic = format!("5\nnormal/A\nNORMAL/{flag}B\nmixCase\npour\nkm/K\n"); + let d = AffixDict::from_hunspell(&format!("{aff}KEEPCASE K\n"), &dic, &[]).unwrap(); + for (original, changed) in [("normal", "NORMAL"), ("normals", "NORMALS")] { + assert!(!d.can_suggest_cased(changed, changed), "{flag}: {changed}"); + assert!(!d.can_suggest_cased(original, changed), "{flag}: {original} → {changed}"); + } + assert!(d.can_suggest_cased("normal", "Normal")); + assert!(d.can_suggest_cased("mixCase", "MIXCASE")); + assert!(d.can_suggest_cased("km", "km") && !d.can_suggest_cased("km", "KM")); + assert!(!d.can_suggest_cased("normal", "pour"), "no arbitrary replacement"); + let back = AffixDict::from_bytes(&d.to_bytes().unwrap()).unwrap(); + assert!(!back.can_suggest_cased("normal", "NORMAL")); + } + } + + #[test] + fn displayed_casing_uses_normalization_and_native_homonym_policy() { + let aff = "SET UTF-8\nNOSUGGEST N\nSUBSTANDARD S\nIGNORE ()\nICONV 1\nICONV Q NORMAL\n"; + for flag in ["N", "S"] { + let d = AffixDict::from_hunspell(aff, &format!("3\nnormal\nNORMAL/{flag}\nNORMAL\n"), &[]).unwrap(); + // Native homonyms carry exclusions: a second entry does not erase NOSUGGEST. + assert!(!d.contains_suggestible("NORMAL")); + assert!(!d.can_suggest_cased("(normal)", "(NORMAL)")); + assert!(!d.can_suggest_cased("Q", "Q")); + } + let plain = AffixDict::from_hunspell(aff, "3\nnormal\nNORMAL\nNORMAL\n", &[]).unwrap(); + assert!(plain.can_suggest_cased("normal", "NORMAL")); + assert!(plain.can_suggest_cased("Q", "Q")); + assert!(!plain.can_suggest_cased(&"x".repeat(MAX_INPUT_BYTES + 1), "NORMAL")); + } + #[test] fn oconv_converts_suggestions_only() { // Answers as Hunspell 1.7 gives them on the same files: `kbt` → `k’t`. diff --git a/crates/proof/src/affix_suggest.rs b/crates/proof/src/affix_suggest.rs index 3b498542c..68336de16 100644 --- a/crates/proof/src/affix_suggest.rs +++ b/crates/proof/src/affix_suggest.rs @@ -426,6 +426,18 @@ pub(crate) fn suggest_in(d: &AffixDict, word: &str, max: usize, t: &Tuning) -> V _ => vec![text.clone()], }; for t in recased { + let eligible = |candidate: &str| { + let original: Vec<_> = text.split(' ').collect(); + let changed: Vec<_> = candidate.split(' ').collect(); + original.len() == changed.len() && original.iter().zip(changed).all(|(a, b)| d.can_suggest_cased(a, b)) + }; + let t = if eligible(&t) { + t + } else if eligible(&text) { + text.clone() + } else { + continue; + }; // Shown as the dictionary's OCONV table writes it (`'` → `’` in French). let t = d.output(&t).into_owned(); if t != word && !out.contains(&t) { @@ -763,6 +775,24 @@ mod tests { assert_eq!(suggest_in(&d, "Piesx", 4, &Tuning::DEFAULT).first().map(String::as_str), Some("Pies")); } + #[test] + fn recasing_keeps_valid_original_when_exact_target_is_excluded() { + for flag in ["N", "F", "S"] { + let aff = "SET UTF-8\nNOSUGGEST N\nFORBIDDENWORD F\nSUBSTANDARD S\nTRY normal\n"; + let d = AffixDict::from_hunspell(aff, &format!("2\nnormal\nNORMAL/{flag}\n"), &[]).unwrap(); + let got = suggest_in(&d, "NORMALL", 4, &Tuning::DEFAULT); + assert!(got.iter().any(|w| w == "normal"), "{flag}: {got:?}"); + assert!(!got.iter().any(|w| w == "NORMAL"), "{flag}: {got:?}"); + let split = AffixDict::from_hunspell(aff, &format!("3\nnormal\nNORMAL/{flag}\npour\n"), &[]).unwrap(); + let got = suggest_in(&split, "NORMALPOUR", 4, &Tuning::DEFAULT); + assert!(got.iter().any(|w| w == "normal pour"), "{flag}: {got:?}"); + assert!(!got.iter().any(|w| w == "NORMAL POUR"), "{flag}: {got:?}"); + } + let d = AffixDict::from_hunspell("NOSUGGEST N\n", "3\nnormal\nNORMAL/N\npour\n", &[]).unwrap(); + assert!(!d.can_suggest_cased("normal", "NORMAL")); + assert!(!d.can_suggest_cased("normal", "pour")); + } + /// A language's tables come after its dictionary's, a TRY alphabet stands in for a missing TRY /// line, and a split rule decides which missing spaces are offered. #[test] diff --git a/crates/proof/src/fr/grammar.rs b/crates/proof/src/fr/grammar.rs new file mode 100644 index 000000000..1d5449b1f --- /dev/null +++ b/crates/proof/src/fr/grammar.rs @@ -0,0 +1,665 @@ +//! French grammar and typography checks: cheap, rule-based and tuned to stay quiet when unsure +//! (a missed mistake costs less than a wrong mark on correct text). Messages are in French. +//! +//! - **Space before `;` `!` `?` `:`**: French typography puts a non-breaking space there — a +//! narrow one (U+202F) before `;` `!` `?`, a full one (U+00A0) before `:`. Only a missing +//! space is flagged (`Bonjour!`): an ordinary one stays quiet until typing turns it into a +//! non-breaking one (no French AutoFormat yet). Quebec French (`fr-CA`) has no space before +//! `;` `!` `?`, only before `:`. Addresses (`https://…`) and times (`10:30`) are left alone. +//! - **Space inside `« »`**: a missing one after `«` or before `»` (`«oui»`) → U+00A0. +//! - **Straight double quotes** around a phrase (`"bonjour"`) → `« bonjour »`; inch marks +//! (`27"`) and unbalanced quotes are left alone. +//! - **Repeated word** (`le le`, `que que`), except repetitions French makes on purpose +//! (`nous nous lavons`, `vous vous trompez`, `si si`, interjections) and a proper noun after +//! the same lower-case word (`de De…`). +//! - **Capital letter at the start of a sentence** after `.` `!` `?`, knowing French +//! abbreviations (`etc.`, `M.`, `p.`, `cf.`), initials, numbers, ellipses, addresses, +//! dialogue (`« Tu viens ? » demanda-t-elle`), inversions (`Quoi ? dit-il`) and exclamations +//! after an interjection (`Ah ! que c'est beau`). +//! - **Elision** (`de avoir` → `d’avoir`, `je ai` → `j’ai`, `le homme` → `l’homme`) for `le`, +//! `la`, `de`, `je`, `me`, `te`, `se`, `ne`, `que` before a vowel. Before an `h` only for words +//! known to have a mute `h`, so aspirated ones (`le hibou`, `la hache`, `le héros`) stay quiet; +//! never before a proper noun, `onze`, `oui`, `y…` words (`le yaourt`) or `la une`. +//! +//! No agreement or conjugation rule, and no English rule (such as `a`/`an`), applies to French +//! text. + +use crate::{Issue, IssueKind, ProofLang, words}; + +/// Non-breaking space, before `:` and inside `« »`. +const NBSP: char = '\u{A0}'; +/// Narrow non-breaking space, before `;` `!` `?`. +const NNBSP: char = '\u{202F}'; + +/// Words French repeats on purpose: pronominal verbs (`nous nous lavons`), emphasis, +/// interjections, onomatopoeia. +const ALLOWED_REPEATS: &[&str] = &[ + "nous", "vous", "si", "oui", "non", "très", "bon", "vite", "là", "ah", "oh", "eh", "ha", "hé", "ho", "hi", "bla", "chut", "toc", "pan", "bravo", + "merci", "allez", "encore", "beaucoup", "fort", "tchin", "bye", "couci", "tam", "miam", "glou", +]; + +/// Abbreviations written with a final period after which a sentence goes on in lower case +/// (lower case, without the period). +const ABBREVIATIONS: &[&str] = &[ + "etc", "cf", "ex", "env", "av", "apr", "mm", "mme", "mmes", "mlle", "mlles", "mgr", "dr", "pr", "me", "st", "ste", "sts", "pp", "vol", "chap", + "fig", "éd", "coll", "trad", "réf", "tél", "art", "al", "op", "cit", "ibid", "id", "resp", "min", "max", "sq", "sqq", "suiv", "hab", "janv", + "févr", "avr", "juil", "sept", "oct", "nov", "déc", "bd", "boul", "cie", "éts", "vve", "nb", "ps", "no", "nos", "vs", "qqch", "qqn", "ca", + "approx", "prof", "dir", "sté", "dépt", "adj", "adv", "fém", "masc", "sing", "plur", "à-d", "ms", "mss", "éds", "trim", "hôp", +]; + +/// Words after which `!` or `?` doesn't end the sentence (`Ah ! que c'est beau`, `Eh bien ? +/// et alors`). +const INTERJECTIONS: &[&str] = &[ + "ah", "oh", "eh", "hé", "hein", "hélas", "bah", "ô", "aïe", "ouf", "oups", "quoi", "comment", "ha", "ho", "bon", "tiens", "allons", "zut", + "chut", "bravo", "hourra", "dame", "bien", "là", "oui", "non", "pardon", "voyons", +]; + +/// Pronouns ending an inversion (`dit-il`, `demanda-t-elle`): after one, a `!` or `?` doesn't +/// end the sentence (`Quoi ? dit-il`). +const INVERTED: &[&str] = &["-il", "-elle", "-on", "-ils", "-elles", "-je", "-tu", "-nous", "-vous"]; + +/// Speech verbs of a dialogue tag: after one, a `!` or `?` doesn't end the sentence (`— Tu +/// viens ? demanda le voisin`). +const SPEECH_VERBS: &[&str] = &[ + "dit", + "demanda", + "répondit", + "cria", + "murmura", + "ajouta", + "reprit", + "fit", + "lança", + "souffla", + "répliqua", + "interrogea", + "hurla", + "poursuivit", + "continua", + "déclara", + "expliqua", + "protesta", + "soupira", + "chuchota", + "insista", + "s'écria", + "s’écria", + "s'exclama", + "s’exclama", + "objecta", + "conclut", + "dis", + "dit-on", +]; + +/// The words that elide before a vowel, and their elided form. +const ELIDING: &[(&str, &str)] = + &[("le", "l"), ("la", "l"), ("de", "d"), ("je", "j"), ("me", "m"), ("te", "t"), ("se", "s"), ("ne", "n"), ("que", "qu")]; + +/// Words starting with a vowel that don't take an elision (`le onze`, `le oui`, `de ouate`). +const NO_ELISION: &[&str] = &["onze", "onzième", "onzièmes", "oui", "ouistiti", "ouistitis", "ouate", "ouates", "uhlan", "uhlans"]; + +/// Words with a mute `h`, which take an elision (`l’homme`, `d’habitude`, `j’habite`). Only these +/// are flagged: an `h` word outside the list may be aspirated (`le hibou`, `la hache`, `le +/// héros`), and staying quiet is the safe side. +const MUTE_H: &[&str] = &[ + "homme", + "hommes", + "heure", + "heures", + "histoire", + "histoires", + "hiver", + "hivers", + "hôtel", + "hôtels", + "herbe", + "herbes", + "habitude", + "habitudes", + "habitant", + "habitants", + "habit", + "habits", + "hôpital", + "hôpitaux", + "humain", + "humains", + "humanité", + "honneur", + "honneurs", + "horizon", + "horizons", + "huile", + "huiles", + "hypothèse", + "hypothèses", + "harmonie", + "héritage", + "héroïne", + "héroïnes", + "hésitation", + "hommage", + "hommages", + "horaire", + "horaires", + "horreur", + "horreurs", + "humeur", + "humour", + "hygiène", + "hymne", + "habiter", + "hériter", + "hésiter", + "habiller", + "honorer", + "habite", + "habites", + "habitent", + "habitais", + "habitait", + "hésite", + "hésites", + "hésitent", + "hésitais", + "hésitait", + "habille", + "habillent", + "honore", + "hier", + "hôte", + "hôtes", + "hôtesse", + "humidité", + "haleine", + "hebdomadaire", + "hectare", + "hectares", + "hélicoptère", + "hémisphère", + "heureux", + "heureuse", + "hippopotame", + "hirondelle", + "honnête", + "honnêteté", + "huître", + "huîtres", + "humble", + "humilité", + "hydrogène", + "hypocrite", + "horloge", + "hôtellerie", + "hérédité", + "historien", + "historique", + "humanitaire", + "hormone", + "hormones", + "hospitalité", +]; + +fn issue(start: usize, end: usize, message: String, suggestions: Vec) -> Issue { + Issue { start, end, kind: IssueKind::Grammar, message, suggestions, lang: ProofLang::from_tag(Some("fr-FR")) } +} + +/// Is `c` a space that doesn't break (or a thin space some writers use instead)? +fn is_fixed_space(c: char) -> bool { + matches!(c, '\u{A0}' | '\u{202F}' | '\u{2007}' | '\u{2009}' | '\u{2060}') +} + +/// The byte range of the whitespace-delimited token around byte `i` of `text`. +fn token_bounds(text: &str, i: usize) -> (usize, usize) { + let start = text.get(..i).and_then(|t| t.char_indices().rev().find(|(_, c)| c.is_whitespace())).map_or(0, |(k, c)| k + c.len_utf8()); + let end = text.get(i..).and_then(|t| t.find(char::is_whitespace)).map_or(text.len(), |k| i + k); + (start, end) +} + +/// The whitespace-delimited token around byte `i` of `text`. +fn token_at(text: &str, i: usize) -> &str { + let (start, end) = token_bounds(text, i); + text.get(start..end).unwrap_or("") +} + +/// Is `token` a web or e-mail address, a path or an HTML entity, where punctuation isn't prose? +fn is_address(token: &str) -> bool { + token.contains("://") || token.contains('@') || token.starts_with("www.") || token.contains('&') || token.contains('\\') +} + +/// `«` + text + `»` in a message, with French spacing. +fn quoted(s: &str) -> String { + format!("«{NBSP}{s}{NBSP}»") +} + +/// `;` `!` `?` `:` without the non-breaking space French puts before them. +/// Quebec typography (`quebec`) puts no space before `;` `!` `?`, only before `:`. +fn high_punctuation(text: &str, quebec: bool, v: &mut Vec) { + let mut token: Option<(usize, usize, bool)> = None; + for (i, p) in text.char_indices() { + if !matches!(p, ';' | ':' | '!' | '?') || (quebec && p != ':') { + continue; + } + let before = text.get(..i).and_then(|t| t.chars().next_back()); + let after = text.get(i + p.len_utf8()..).and_then(|t| t.chars().next()); + // One look per token, so a long run of marks stays linear. + let address = match token { + Some((start, end, address)) if start <= i && i < end => address, + _ => { + let (start, end) = token_bounds(text, i); + let address = is_address(text.get(start..end).unwrap_or("")); + token = Some((start, end, address)); + address + } + }; + if address { + continue; + } + let (space, kind) = if p == ':' { (NBSP, "insécable") } else { (NNBSP, "fine insécable") }; + let message = format!("Espace {kind} attendue avant {}", quoted(&p.to_string())); + let fix = format!("{space}{p}"); + // Only a missing space is flagged: an ordinary one stays quiet until typing turns it into + // a non-breaking one by itself (French AutoFormat). + if before.is_some_and(char::is_alphabetic) { + // `10:30`, `a:b`, `C:` and `x;y` aren't prose: the mark must end the word. + let ends = match p { + ':' => after.is_none_or(char::is_whitespace), + _ => after.is_none_or(|a| a.is_whitespace() || matches!(a, '!' | '?' | '»' | '"' | '”' | ')')), + }; + if ends { + v.push(issue(i, i + p.len_utf8(), message, vec![fix])); + } + } + } +} + +/// `«` and `»` without a non-breaking space inside. +fn guillemets(text: &str, v: &mut Vec) { + for (i, c) in text.char_indices() { + let len = c.len_utf8(); + match c { + '«' => { + let mut next = text.get(i + len..).unwrap_or("").chars(); + let message = format!("Espace insécable attendue après {}", quoted("«")); + if next.next().is_some_and(char::is_alphanumeric) { + v.push(issue(i, i + len, message, vec![format!("«{NBSP}")])); + } + } + '»' => { + let prev = text.get(..i).unwrap_or("").chars().next_back(); + let message = format!("Espace insécable attendue avant {}", quoted("»")); + if prev.is_some_and(|n| n.is_alphanumeric() || matches!(n, '.' | '!' | '?' | '…')) { + v.push(issue(i, i + len, message, vec![format!("{NBSP}»")])); + } + } + _ => {} + } + } +} + +/// `"…"` around a phrase, where French uses `« … »`. +fn straight_quotes(text: &str, v: &mut Vec) { + let quotes: Vec = text.match_indices('"').map(|(i, _)| i).collect(); + // An odd count leaves no telling which quote opens and which closes. + if !quotes.len().is_multiple_of(2) { + return; + } + for pair in quotes.chunks(2) { + let [a, b] = pair else { continue }; + let (Some(inner), Some(before), Some(after)) = (text.get(a + 1..*b), text.get(..*a), text.get(b + 1..)) else { continue }; + let opens = before.chars().next_back().is_none_or(|c| c.is_whitespace() || matches!(c, '(' | '[' | '—' | '–')); + let closes = after.chars().next().is_none_or(|c| c.is_whitespace() || matches!(c, '.' | ',' | ';' | ':' | '!' | '?' | ')' | ']' | '…')); + let phrase = inner.trim(); + if !opens || !closes || !phrase.chars().any(char::is_alphabetic) || phrase.chars().count() > 120 || phrase.contains('\n') { + continue; + } + v.push(issue(*a, b + 1, format!("Guillemets français attendus{NBSP}: {}", quoted(phrase)), vec![quoted(phrase)])); + } +} + +fn repeated_words(text: &str, ws: &[(usize, usize)], v: &mut Vec) { + for pair in ws.windows(2) { + let [(a0, b0), (a1, b1)] = pair else { continue }; + let (Some(w0), Some(w1), Some(gap)) = (text.get(*a0..*b0), text.get(*a1..*b1), text.get(*b0..*a1)) else { continue }; + let lower = w0.to_lowercase(); + let starts_upper = |w: &str| w.chars().next().is_some_and(char::is_uppercase); + if !gap.is_empty() + && gap.chars().all(char::is_whitespace) + && w0.chars().all(char::is_alphabetic) + && lower == w1.to_lowercase() + && !ALLOWED_REPEATS.contains(&lower.as_str()) + // `de De…`, `Bora Bora`: a capitalized second word is a name. + && !starts_upper(w1) + { + v.push(issue(*a0, *b1, format!("Mot répété{NBSP}: {}", quoted(w1)), vec![w0.to_string()])); + } + } +} + +/// Is `prev`, before a period, an abbreviation, an initial, a number or a Roman numeral (after +/// which a lower-case word doesn't start a sentence)? +fn abbreviation_or_number(prev: &str) -> bool { + prev.chars().count() <= 1 + || prev.chars().any(|c| c.is_ascii_digit()) + || prev.chars().all(|c| matches!(c, 'I' | 'V' | 'X' | 'L' | 'C' | 'D' | 'M')) + || ABBREVIATIONS.contains(&prev.to_lowercase().as_str()) +} + +/// Does `gap`, between the words `prev` and `next`, end a sentence: a `.`, `!` or `?` (French +/// puts a space before the last two), then a space, with only quotes and brackets around? Not +/// an ellipsis, not after an abbreviation; for `!` and `?`, not after an interjection, not before +/// a closing quote (dialogue goes on: `« Tu viens ? » demanda-t-elle`), not before an inversion +/// or a speech verb (`Quoi ? dit-il`, `— Viens ! cria-t-on`), and never in a `dialogue` line (one +/// opening with a dash). +fn ends_sentence(gap: &str, prev: &str, next: &str, dialogue: bool) -> bool { + let closing = ['"', '”', '’', '\'', ')', '»', ']']; + let t = gap.trim_start_matches(|c: char| c.is_whitespace() || is_fixed_space(c) || closing.contains(&c)); + let Some(end) = t.chars().next() else { return false }; + if !matches!(end, '.' | '!' | '?') || t.starts_with("..") { + return false; + } + if end == '.' && abbreviation_or_number(prev) { + return false; + } + let exclamation = end != '.'; + let lower_next = next.to_lowercase(); + if exclamation + && (dialogue + || INTERJECTIONS.contains(&prev.to_lowercase().as_str()) + || INVERTED.iter().any(|p| lower_next.ends_with(p)) + || SPEECH_VERBS.contains(&lower_next.as_str())) + { + return false; + } + let rest = t.trim_start_matches(['.', '!', '?']); + let after_marks = rest.trim_start_matches(is_fixed_space); + if after_marks.starts_with(closing) { + if exclamation { + return false; + } + return ends_sentence_after(after_marks.trim_start_matches(|c: char| closing.contains(&c) || is_fixed_space(c))); + } + ends_sentence_after(rest) +} + +/// Is `rest` (after the final mark) a space, then nothing but opening quotes and brackets? +fn ends_sentence_after(rest: &str) -> bool { + rest.starts_with(char::is_whitespace) + && rest.trim_start().chars().all(|c| c.is_whitespace() || is_fixed_space(c) || matches!(c, '"' | '“' | '«' | '(' | '[' | '—' | '–')) +} + +fn sentence_capitals(text: &str, ws: &[(usize, usize)], v: &mut Vec) { + let dialogue = text.trim_start().starts_with(['—', '–']); + for pair in ws.windows(2) { + let [(a0, b0), (a1, b1)] = pair else { continue }; + let (Some(prev), Some(gap), Some(w)) = (text.get(*a0..*b0), text.get(*b0..*a1), text.get(*a1..*b1)) else { continue }; + let word_like = w.chars().all(|c| c.is_alphabetic() || matches!(c, '\'' | '’' | '-')); + if word_like && w.chars().next().is_some_and(char::is_lowercase) && ends_sentence(gap, prev, w, dialogue) && !is_address(token_at(text, *a1)) + { + let fixed: String = w.chars().next().map(|f| f.to_uppercase().chain(w.chars().skip(1)).collect()).unwrap_or_default(); + v.push(issue(*a1, *b1, "Majuscule attendue en début de phrase".into(), vec![fixed])); + } + } +} + +/// Does `w` (lower case) start with a sound that makes `le`, `de`, `que`… elide? +fn takes_elision(w: &str) -> bool { + let Some(first) = w.chars().next() else { return false }; + if NO_ELISION.contains(&w) { + return false; + } + match first { + 'a' | 'à' | 'â' | 'ä' | 'e' | 'é' | 'è' | 'ê' | 'ë' | 'i' | 'î' | 'ï' | 'o' | 'ô' | 'ö' | 'u' | 'ù' | 'û' | 'ü' | 'œ' | 'æ' => { + true + } + 'h' => MUTE_H.contains(&w), + // Only `y` itself (`j’y vais`): `le yaourt`, `de yoga`. + 'y' => w == "y", + _ => false, + } +} + +fn elisions(text: &str, ws: &[(usize, usize)], v: &mut Vec) { + for pair in ws.windows(2) { + let [(a0, b0), (a1, b1)] = pair else { continue }; + let (Some(w0), Some(w1), Some(gap)) = (text.get(*a0..*b0), text.get(*a1..*b1), text.get(*b0..*a1)) else { continue }; + if gap.is_empty() || !gap.chars().all(|c| c == ' ') { + continue; + } + let lower0 = w0.to_lowercase(); + let Some((_, elided)) = ELIDING.iter().find(|(full, _)| *full == lower0) else { continue }; + // `le`/`Le` only, not `LE`; never before a proper noun (`de Orléans`) or a capital. + let capital = w0.chars().next().is_some_and(char::is_uppercase); + if w0.chars().skip(1).any(char::is_uppercase) || !w1.chars().next().is_some_and(char::is_lowercase) { + continue; + } + if !w1.chars().all(|c| c.is_alphabetic() || c == '-') { + continue; + } + let lower1 = w1.to_lowercase(); + // `le un`, `la une` (the number, the front page); a single letter only where it is a word + // the elision is certain for (`n’a`, `qu’a`, `j’y`). + if matches!(lower0.as_str(), "le" | "la") && matches!(lower1.as_str(), "un" | "une") { + continue; + } + let certain = (lower1 == "y" && matches!(lower0.as_str(), "je" | "me" | "te" | "se" | "ne")) + || (lower1 == "a" && matches!(lower0.as_str(), "ne" | "que")); + if lower1.chars().count() == 1 && !certain { + continue; + } + if !takes_elision(&lower1) { + continue; + } + let head = if capital { crate::affix::capitalize(elided) } else { (*elided).to_string() }; + let fixed = format!("{head}’{w1}"); + v.push(issue(*a0, *b1, format!("Élision{NBSP}: {}", quoted(&fixed)), vec![fixed])); + } +} + +/// Grammar issues in a paragraph of French text (France's typography), in order. +pub fn check(text: &str) -> Vec { + check_in(text, ProofLang::from_tag(Some("fr-FR"))) +} + +/// Grammar issues in a French Language pack locale, with Quebec typography in `fr-CA`. +/// Other locales have no French issues; the original locale is retained on every issue. +pub fn check_in(text: &str, lang: ProofLang) -> Vec { + let Some(locale) = lang.pack_locale().filter(|l| l.primary() == "fr") else { return Vec::new() }; + let ws: Vec<(usize, usize)> = words(text); + let mut v = Vec::new(); + high_punctuation(text, crate::lang::region(locale.as_str()).is_some_and(|r| r.eq_ignore_ascii_case("CA")), &mut v); + guillemets(text, &mut v); + straight_quotes(text, &mut v); + repeated_words(text, &ws, &mut v); + sentence_capitals(text, &ws, &mut v); + elisions(text, &ws, &mut v); + for i in &mut v { + i.lang = lang; + } + v.sort_by_key(|i| (i.start, i.end)); + v +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The issues of `text` as (flagged text, first suggestion). + fn found(text: &str) -> Vec<(String, String)> { + check(text) + .iter() + .map(|i| (text.get(i.start..i.end).unwrap_or("?").to_string(), i.suggestions.first().cloned().unwrap_or_default())) + .collect() + } + + fn quiet(text: &str) { + let v = check(text); + assert!(v.is_empty(), "{text:?} → {v:?}"); + } + + fn one(text: &str, flagged: &str, fix: &str) { + assert_eq!(found(text), [(flagged.to_string(), fix.to_string())], "{text:?}"); + } + + #[test] + fn missing_space_before_high_punctuation() { + one("Bonjour!", "!", "\u{202F}!"); + one("Tu viens?", "?", "\u{202F}?"); + one("Il pleut; il vente.", ";", "\u{202F};"); + one("Note: rien.", ":", "\u{A0}:"); + // `?!`: only the first mark needs the space. + one("Quoi?!", "?", "\u{202F}?"); + assert!(check("Bonjour!").iter().all(|i| i.lang == ProofLang::from_tag(Some("fr-FR")) && i.message.contains("fine insécable"))); + assert!(check("Note: rien.").iter().all(|i| i.message.contains("insécable") && !i.message.contains("fine"))); + } + + #[test] + fn an_ordinary_space_stays_quiet_until_french_autoformat() { + // Typing can't produce a non-breaking space yet, so an ordinary one is accepted. + quiet("Vraiment ? Bonjour ! Il pleut ; note : rien."); + quiet("Il dit « oui » hier."); + } + + #[test] + fn quebec_has_no_space_before_semicolon_exclamation_and_question() { + let fr_ca = |t: &str| check_in(t, ProofLang::from_tag(Some("fr-CA"))); + assert!(fr_ca("Bonjour! Tu viens? Il pleut; il vente.").is_empty(), "{:?}", fr_ca("Bonjour! Tu viens?")); + // The colon and the guillemets still want their space in Quebec. + let v = fr_ca("Note: «oui»."); + assert_eq!(v.len(), 3, "{v:?}"); + assert!(v.iter().all(|i| i.lang == ProofLang::from_tag(Some("fr-CA"))), "{v:?}"); + assert_eq!(check_in("Bonjour!", ProofLang::from_tag(Some("fr-FR"))).len(), 1, "France still wants the space"); + } + + #[test] + fn high_punctuation_stays_quiet() { + quiet("Bonjour\u{202F}! Tu viens\u{202F}? Il pleut\u{202F}; note\u{A0}: rien."); + quiet("Bonjour\u{A0}! Quoi\u{202F}?! Ah\u{2009}!"); + quiet("Il est 10:30."); + quiet("Voir https://exemple.org:8080/a?b=c pour la suite."); + quiet("Écrire à contact@exemple.org!"); + quiet("Super :) et ;-) aussi :D"); + quiet("a:b et x;y"); + quiet("Le fichier C:\\dossier est là."); + } + + #[test] + fn missing_space_inside_guillemets() { + assert_eq!(found("Il dit «oui» hier."), [("«".to_string(), "«\u{A0}".to_string()), ("»".to_string(), "\u{A0}»".to_string())]); + quiet("Il dit «\u{A0}oui\u{A0}» hier."); + quiet("Il dit «\u{202F}oui\u{202F}» hier."); + } + + #[test] + fn straight_quotes_become_guillemets() { + one("Il a dit \"bonjour\" hier.", "\"bonjour\"", "«\u{A0}bonjour\u{A0}»"); + one("\"Bonne idée\", dit-elle.", "\"Bonne idée\"", "«\u{A0}Bonne idée\u{A0}»"); + // Inch marks and unbalanced quotes are left alone. + quiet("Un écran de 27\" et un de 32\" ici."); + quiet("Il a dit \"bonjour hier."); + } + + #[test] + fn repeated_words_are_flagged() { + one("Le le chat dort.", "Le le", "Le"); + one("Je pense que que tu viens.", "que que", "que"); + one("Il mange la la pomme.", "la la", "la"); + } + + #[test] + fn deliberate_repetitions_stay_quiet() { + quiet("Nous nous lavons et vous vous trompez."); + quiet("Si si, très très bien."); + quiet("Ah ah, bla bla."); + quiet("Je crois que qu’il viendra."); + quiet("Le livre de De Gaulle."); + quiet("Nous partons à Bora Bora."); + quiet("Ils ont dit bye bye. Tchin tchin\u{202F}!"); + } + + #[test] + fn sentence_capitals_are_flagged() { + one("Il pleut. il fait froid.", "il", "Il"); + one("Il pleut\u{202F}! il fait froid.", "il", "Il"); + one("Tu viens\u{202F}? on part.", "on", "On"); + one("Il pleut. l’orage arrive.", "l’orage", "L’orage"); + one("Il pleut. «\u{A0}tout va bien\u{A0}», dit-il.", "tout", "Tout"); + } + + #[test] + fn sentence_capitals_stay_quiet() { + quiet("Voir p. 12 et cf. la note, etc. puis la suite."); + quiet("M. le maire est là."); + quiet("Il attend… puis part. Il attend... puis part."); + quiet("«\u{A0}Tu viens\u{202F}?\u{A0}» demanda-t-elle."); + quiet("Ah\u{202F}! que c’est beau."); + quiet("Quoi\u{202F}? dit-il."); + quiet("Tu viens\u{202F}? demanda le voisin."); + quiet("— Tu viens\u{202F}? reprends-tu, inquiet."); + quiet("– Viens\u{202F}! on part."); + quiet("Voir www.exemple.fr pour la suite."); + quiet("En 2026. ensuite rien."); + quiet("il commence en minuscule."); + } + + #[test] + fn missing_elisions_are_flagged() { + one("Il refuse de avoir peur.", "de avoir", "d’avoir"); + one("Je ai faim.", "Je ai", "J’ai"); + one("Voici le homme.", "le homme", "l’homme"); + one("Je sais que il viendra.", "que il", "qu’il"); + one("Le arbre est grand.", "Le arbre", "L’arbre"); + one("Il ne a pas faim.", "ne a", "n’a"); + one("Bon, je y vais.", "je y", "j’y"); + one("Il se est levé.", "se est", "s’est"); + one("Voici la école.", "la école", "l’école"); + one("Il me aime.", "me aime", "m’aime"); + } + + #[test] + fn aspirated_h_and_other_exceptions_stay_quiet() { + for w in [ + "hibou", + "héros", + "haricot", + "hasard", + "haut", + "hall", + "hamac", + "hangar", + "hockey", + "homard", + "huit", + "hérisson", + "hors-jeu", + "onze", + "oui", + "yaourt", + ] { + quiet(&format!("Voici le {w}.")); + } + // hiérarchie has an aspirated h, despite the mute h in hiéroglyphe: + // https://www.academie-francaise.fr/le-hieroglyphe-pour-lhieroglyphe + for w in ["honte", "hache", "harpe", "hausse", "hiérarchie"] { + quiet(&format!("Voici la {w}.")); + } + quiet("Une question de hiérarchie."); + quiet("Le onze de France. À la une du journal."); + quiet("Il vient de Orléans."); + quiet("De a à z, le a et le b."); + quiet("Le y et le x sont des variables. La dérivée de y."); + quiet("Je mange de la viande."); + } + + #[test] + fn hostile_input_never_panics() { + for t in ["", " ", "!", " !", "«", "»", " »", "« ", "\"", "\"\"", ". a", "de", "de ", "\u{202F}?", "é!é", "« » « »", ":", " :", "a :)"] + { + let _ = check(t); + } + // A long run of marks in one token: linear, not one token scan per mark. + let run = format!("mot{}", "?".repeat(200_000)); + assert_eq!(check(&run).len(), 1); + } +} diff --git a/crates/proof/src/fr/mod.rs b/crates/proof/src/fr/mod.rs new file mode 100644 index 000000000..541fcac0d --- /dev/null +++ b/crates/proof/src/fr/mod.rs @@ -0,0 +1,181 @@ +//! French proofing through the shared Language pack cache. +//! +//! A French locale remains [`ProofLang::Pack`]: dictionary selection, installation, reloads, +//! memory limits and hyphenation use the same paths as other packs. This module adds French +//! elisions, accent-oriented suggestions and conservative typography, including Quebec spacing. +//! No dictionary or hyphenation data is bundled here. + +pub mod grammar; +mod suggest; +#[cfg(test)] +#[allow(clippy::unwrap_used)] +pub(crate) mod test_dict; + +use crate::affix::AffixDict; +use crate::{Options, ProofLang}; + +pub use suggest::suggest_in; + +/// Words after whose elision a user-dictionary entry is accepted. Ordinary words and their +/// elisions are always decided by the dictionary (`l'arbre`, but not `l'arbres`). +const ELISIONS: &[&str] = &["l", "d", "j", "m", "t", "s", "n", "c", "qu", "jusqu", "lorsqu", "puisqu", "quoiqu", "presqu", "quelqu"]; + +/// Check French spelling in the installed France/generic pack, with the default options. +/// Regional callers should use [`crate::is_correct_in`] with their full locale. +pub fn is_correct(word: &str) -> bool { + crate::is_correct_in(word, ProofLang::from_tag(Some("fr-FR"))) +} + +/// French suggestions from the installed France/generic pack, best first, at most `max`. +/// Regional callers should use [`crate::suggest_in`] with their full locale. +pub fn suggest(word: &str, max: usize) -> Vec { + crate::suggest_in(word, max, ProofLang::from_tag(Some("fr-FR"))) +} + +/// Check against a supplied dictionary, without changing any installed pack or cache. +pub fn is_correct_with(d: &AffixDict, word: &str) -> bool { + check(d, &crate::composed(word), &Options::default()) +} + +pub(crate) fn check(d: &AffixDict, word: &str, opts: &Options) -> bool { + let Some(w) = crate::spelling::word_to_check(d, word, opts) else { return true }; + // A regional dictionary may store the curly apostrophe without ICONV. Honor that entry + // before the French ASCII-apostrophe fallback. + if known(d, w) { + return true; + } + let w = w.replace('’', "'"); + if known(d, &w) { + return true; + } + if let Some((head, tail)) = w.split_once('\'') + && ELISIONS.contains(&head.to_lowercase().as_str()) + && !tail.is_empty() + && !tail.contains('\'') + && crate::in_user_dictionary(&tail.to_lowercase()) + { + return true; + } + w.contains('-') && w.split('-').all(|p| !p.is_empty() && (p.chars().count() <= 1 || known(d, p))) +} + +fn known(d: &AffixDict, word: &str) -> bool { + d.check(word) || crate::in_user_dictionary(&word.to_lowercase()) +} + +#[cfg(test)] +#[allow(clippy::unwrap_used)] +mod tests { + use super::*; + + #[test] + fn original_french_words_and_elisions() { + let d = test_dict::dictionary(); + for word in [ + "maison", + "maisons", + "école", + "écoles", + "grand", + "grande", + "grands", + "grandes", + "aimer", + "aime", + "aimons", + "aiment", + "suis", + "est", + "sommes", + "sont", + "a", + "avons", + "ont", + "Paris", + "Montréal", + "œuvre", + "sœur", + "aujourd'hui", + "aujourd’hui", + "peut-être", + "arc-en-ciel", + "km", + "Maison", + "MAISON", + "SNCF", + "2026", + "très", + "rivière", + "l'arbre", + "l’arbre", + "L'arbre", + "d'enfants", + "l'homme", + "qu'elle", + "qu’il", + "j'aime", + "j’ai", + "d'avoir", + "l'école", + "jusqu'à", + "maison-école", + "est-il", + "a-t-il", + ] { + assert!(is_correct_with(&d, word), "{word}"); + } + for word in [ + "maisonn", + "ecole", + "arbrre", + "beaucoups", + "aimmer", + "soeur", + "Km", + "grandd", + "aim", + "l'hasard", + "l'chat", + "l'arbres", + "l'arbrre", + "qu'maison", + "maison-ecole", + "maison--école", + ] { + assert!(!is_correct_with(&d, word), "{word}"); + } + assert!(is_correct_with(&d, "e\u{301}cole")); + assert!(!is_correct_with(&d, "maison\u{338}"), "a trailing mark is part of the word"); + } + + #[test] + fn explicit_curly_apostrophe_without_iconv() { + let d = AffixDict::from_hunspell("SET UTF-8\n", "1\naujourd’hui\n", &[]).unwrap(); + assert!(is_correct_with(&d, "aujourd’hui")); + assert!(!is_correct_with(&d, "aujourd’huui")); + } + + #[test] + fn user_word_after_an_elision() { + let d = test_dict::dictionary(); + let word = "zxqmotadaptfr"; + assert!(!is_correct_with(&d, word) && !is_correct_with(&d, &format!("l'{word}"))); + crate::add_word(word); + assert!(is_correct_with(&d, word) && is_correct_with(&d, &format!("l'{word}")) && is_correct_with(&d, &format!("d’{word}"))); + assert!(!is_correct_with(&d, &format!("chat'{word}"))); + crate::remove_word(word); + } + + #[test] + fn options_and_missing_dictionary() { + let d = test_dict::dictionary(); + let defaults = Options::default(); + let strict = Options { ignore_uppercase: false, ignore_numbers: false, ..defaults }; + assert!(check(&d, "SNCF", &defaults) && check(&d, "mp3x", &defaults)); + assert!(!check(&d, "SNCF", &strict) && !check(&d, "mp3x", &strict)); + assert!(check(&d, "MAISON", &strict) && check(&d, "2026", &strict)); + for word in ["école", "l'arbre", "xqzv", "SNCF", ""] { + assert!(check(&AffixDict::default(), word, &strict)); + } + } +} diff --git a/crates/proof/src/fr/suggest.rs b/crates/proof/src/fr/suggest.rs new file mode 100644 index 000000000..86477d33f --- /dev/null +++ b/crates/proof/src/fr/suggest.rs @@ -0,0 +1,81 @@ +//! French tuning of the shared, bounded affix suggestion engine. + +use crate::affix::AffixDict; +use crate::affix_suggest::{Split, Tuning}; + +/// Short words often run into the next one (`et la`, `de Paris`, `pour moi`). +const LEADING_WORDS: &[&str] = &[ + "à", "a", "au", "aux", "de", "du", "des", "le", "la", "les", "un", "une", "et", "ou", "où", "en", "y", "ce", "se", "ne", "je", "tu", "il", "on", + "nous", "vous", "ils", "elle", "par", "pour", "sur", "sous", "dans", "avec", "sans", "chez", "vers", "entre", "très", "plus", "moins", "bien", + "pas", "que", "qui", "mais", "donc", "car", "si", "son", "sa", "ses", "mon", "ma", "mes", "ton", "ta", "tes", "est", "tout", "tous", "toute", + "comme", "quand", "puis", "encore", "déjà", "jamais", "toujours", +]; +/// Short words often run into the one before (`dis-moi` without the hyphen, `là-bas`, or a +/// dropped space: `pourmoi` → `pour moi`). +const TRAILING_WORDS: &[&str] = + &["moi", "toi", "lui", "elle", "nous", "vous", "eux", "le", "la", "les", "en", "y", "là", "ci", "il", "on", "ils", "elles", "est", "et"]; + +const TUNING: Tuning = Tuning { + split: Split::Near { leading: LEADING_WORDS, trailing: TRAILING_WORDS }, + alphabet: "esaitnrulodcpmévqfbghjàxèêyçôîâûùœ", + ..Tuning::DEFAULT +}; + +/// French suggestions from a supplied dictionary; keeps the writer's apostrophe and case. +/// Dictionary OCONV remains active for every other output conversion. +pub fn suggest_in(d: &AffixDict, word: &str, max: usize) -> Vec { + let word = crate::composed(word); + let curly = word.contains('’'); + let apostrophe = if curly { '’' } else { '\'' }; + let mut out = Vec::new(); + for text in crate::affix_suggest::suggest_in(d, &word, max, &TUNING) { + let styled: String = text.chars().map(|c| if c == '\'' || c == '’' { apostrophe } else { c }).collect(); + // A pack may spell only one apostrophe and have no ICONV. Keep its valid output + // rather than turn it back into the spelling mistake the caller is correcting. + let text = if styled.split(' ').all(|p| d.can_suggest_cased(p, p)) { styled } else { text }; + if text != word && !out.contains(&text) { + out.push(text); + } + } + out +} + +#[cfg(test)] +#[allow(clippy::unwrap_used)] +mod tests { + use super::*; + + #[test] + fn french_accents_confusions_splits_and_elisions() { + let d = super::super::test_dict::dictionary(); + for (typo, expected) in [ + ("ecole", "école"), + ("riviere", "rivière"), + ("soeur", "sœur"), + ("maisonn", "maison"), + ("pourmoi", "pour moi"), + ("l'arbrre", "l'arbre"), + ("l’arbrre", "l’arbre"), + ] { + let got = suggest_in(&d, typo, 8); + assert!(got.iter().any(|s| s == expected), "{typo}: {got:?}"); + } + assert!(suggest_in(&d, "e\u{301}cole", 4).iter().all(|s| s != "école")); + assert!(suggest_in(&d, "ecole", 0).is_empty()); + assert!(suggest_in(&d, &"a".repeat(41), 4).is_empty()); + } + + #[test] + fn curly_dictionary_without_iconv_remains_suggestible() { + let d = AffixDict::from_hunspell("SET UTF-8\n", "1\naujourd’hui\n", &[]).unwrap(); + assert!(suggest_in(&d, "aujourd’huui", 4).iter().any(|s| s == "aujourd’hui")); + assert!(suggest_in(&d, "aujourd'hui", 4).iter().any(|s| s == "aujourd’hui")); + } + + #[test] + fn output_conversion_preserves_typed_apostrophe() { + let d = AffixDict::from_hunspell("SET UTF-8\nICONV 1\nICONV ’ '\nOCONV 1\nOCONV ' ’\n", "1\nl'arbre\n", &[]).unwrap(); + assert!(suggest_in(&d, "l'arbrre", 4).iter().any(|s| s == "l'arbre")); + assert!(suggest_in(&d, "l’arbrre", 4).iter().any(|s| s == "l’arbre")); + } +} diff --git a/crates/proof/src/fr/test_dict.rs b/crates/proof/src/fr/test_dict.rs new file mode 100644 index 000000000..23637254c --- /dev/null +++ b/crates/proof/src/fr/test_dict.rs @@ -0,0 +1,234 @@ +//! A tiny synthetic French Hunspell dictionary for tests (written for these tests; not taken +//! from any real dictionary). Real dictionaries are installed on the system, never bundled. +//! +//! It exercises what the Dicollecte French dictionary uses: two-character flags (`FLAG long`), +//! stems that need an affix (`NEEDAFFIX`, with an empty suffix for the singular), elisions as +//! prefixes licensed by a suffix's continuation (`l'arbre` but not `l'arbres`), irregular verbs +//! through `FULLSTRIP` (`être` → `suis`), `KEEPCASE` (`km`), `NOSUGGEST` (`bah`), `ICONV` (`’` +//! read as `'`) and the +//! suggestion tables (`TRY`, `MAP` for accents, `REP` for common confusions). + +pub const AFF: &str = "SET UTF-8 +FLAG long +NEEDAFFIX () +KEEPCASE || +NOSUGGEST -- +FULLSTRIP +ICONV 1 +ICONV ’ ' +TRY esaitnrulodcpmévqfbghjàxèêyçôîâûùœ +MAP 6 +MAP aàâä +MAP eéèêë +MAP iîï +MAP oôö +MAP uùûü +MAP cç +REP 3 +REP oe œ +REP ph f +REP f ph +PFX L' Y 1 +PFX L' 0 l' [aàâeéèêiîoôuœh] +PFX D' Y 1 +PFX D' 0 d' [aàâeéèêiîoôuœh] +PFX Q' Y 1 +PFX Q' 0 qu' [aàâeéèêiîoôuœh] +PFX J' Y 1 +PFX J' 0 j' [aàâeéèêiîoôuœhy] +SFX S. Y 2 +SFX S. 0 0/L'D'Q' [^sxz] +SFX S. 0 s/D'Q' [^sxz] +SFX S= Y 2 +SFX S= 0 0 [^sxz] +SFX S= 0 s [^sxz] +SFX F. Y 4 +SFX F. 0 0/L'D'Q' . +SFX F. 0 e/L'D'Q' . +SFX F. 0 s/D'Q' . +SFX F. 0 es/D'Q' . +SFX Er Y 6 +SFX Er er er/D' er +SFX Er er e/J' er +SFX Er er es er +SFX Er er ons er +SFX Er er ez er +SFX Er er ent er +SFX Et N 7 +SFX Et être être être +SFX Et être suis être +SFX Et être es être +SFX Et être est être +SFX Et être sommes être +SFX Et être êtes être +SFX Et être sont être +SFX Av N 7 +SFX Av avoir avoir/D' avoir +SFX Av avoir ai/J' avoir +SFX Av avoir as avoir +SFX Av avoir a avoir +SFX Av avoir avons avoir +SFX Av avoir avez avoir +SFX Av avoir ont/Q' avoir +"; + +/// Stems (the count line first, as in Hunspell files). +pub const DIC: &str = "150 +maison/S.() +arbre/S.() +enfant/S.() +ami/S.() +amie/S.() +homme/S.() +école/S.() +café/S.() +chat/S.() +rivière/S.() +ville/S.() +heure/S.() +histoire/S.() +sœur/S.() +œuvre/S.() +photo/S.() +téléphone/S.() +année/S.() +vie/S.() +livre/S.() +table/S.() +porte/S.() +femme/S.() +garçon/S.() +fille/S.() +père/S.() +mère/S.() +langue/S.() +mot/S.() +texte/S.() +phrase/S.() +exemple/S.() +jour/S.() +nuit/S.() +monde/S.() +hasard/S=() +haricot/S=() +grand/F.() +petit/F.() +noir/F.() +aimer/Er() +parler/Er() +chanter/Er() +arriver/Er() +être/Et() +avoir/Av() +il/Q' +ils/Q' +elle/Q' +elles/Q' +on/Q'L' +un/D'Q'L' +une/D'Q'L' +en/Q'J' +y/J' +à/Q' +jusqu'à +aujourd'hui/D' +peut-être +arc-en-ciel +km/|| +bah/-- +Paris +France +Montréal +le +la +les +de +des +du +et +à +je +tu +nous +vous +que +qui +ne +pas +se +me +te +ce +ça +avec +pour +dans +sur +sous +très +près +beaucoup +si +oui +non +mais +ou +où +dit +demanda +quand +comme +plus +bien +tout +toujours +jamais +ici +là +alors +donc +aussi +encore +déjà +mon +ma +mes +son +sa +ses +notre +votre +leur +cette +ces +au +aux +par +entre +vers +chez +sans +moi +toi +lui +rien +chose +temps +fois +eau +voici +voilà +bonjour +merci +beau +belle +la +etc +M +p +"; + +/// The original synthetic fixture, parsed independently of the installed pack cache. +pub fn dictionary() -> crate::affix::AffixDict { + crate::affix::AffixDict::from_hunspell(AFF, DIC, &[]).unwrap() +} diff --git a/crates/proof/src/hunspell.rs b/crates/proof/src/hunspell.rs index 966ded72f..948d0e25b 100644 --- a/crates/proof/src/hunspell.rs +++ b/crates/proof/src/hunspell.rs @@ -754,9 +754,13 @@ mod tests { /// spec's default `~/.local/share` applies, and with no home there is no folder. #[test] fn xdg_data_home_follows_the_spec() { - let home = Path::new("/home/a"); - assert_eq!(xdg_data_home(Some("/data".into()), Some(home)), Some(PathBuf::from("/data"))); - assert_eq!(xdg_data_home(Some("/data".into()), None), Some(PathBuf::from("/data"))); + // On Windows a rooted path without a drive is not absolute. Keep the same + // absolute/relative contract on each platform instead of changing the loader. + let (home, data) = if cfg!(windows) { (r"C:\Users\a", r"C:\data") } else { ("/home/a", "/data") }; + let (home, data) = (Path::new(home), Path::new(data)); + assert!(data.is_absolute()); + assert_eq!(xdg_data_home(Some(data.as_os_str().into()), Some(home)), Some(data.to_path_buf())); + assert_eq!(xdg_data_home(Some(data.as_os_str().into()), None), Some(data.to_path_buf())); for xdg in [None, Some(""), Some("relative/data")] { assert_eq!(xdg_data_home(xdg.map(Into::into), Some(home)), Some(home.join(".local/share")), "{xdg:?}"); assert_eq!(xdg_data_home(xdg.map(Into::into), None), None, "{xdg:?}"); @@ -1010,7 +1014,12 @@ mod tests { assert!(load(&aff, &big).unwrap_err().contains("too large")); // A directory, and on Unix an endless device, are not dictionary files. (No clock: a // reader that read `/dev/zero` would never return, so the error itself is the proof.) - assert!(load(&aff, &dir).unwrap_err().contains("not a regular file")); + // Windows rejects directories in File::open, before the regular-file guard. The + // rejection is required everywhere; its diagnostic is platform-dependent. + let directory_error = load(&aff, &dir).unwrap_err(); + assert!(directory_error.contains(&dir.display().to_string()), "{directory_error}"); + #[cfg(unix)] + assert!(directory_error.contains("not a regular file"), "{directory_error}"); #[cfg(unix)] assert!(load(&aff, Path::new("/dev/zero")).unwrap_err().contains("not a regular file")); // Turkish reads through the same reader. diff --git a/crates/proof/src/lib.rs b/crates/proof/src/lib.rs index 3df6cd017..f0f33d007 100644 --- a/crates/proof/src/lib.rs +++ b/crates/proof/src/lib.rs @@ -42,6 +42,9 @@ //! existing English fallback. //! - **German** ([`ProofLang::De`]): hyphenation by `hyph-utf8`'s German patterns (reformed //! spelling, MIT); no spelling or grammar marks yet. +//! - **French** (a [`ProofLang::Pack`] locale; module [`fr`]): the pack dictionary with French +//! elisions and accent-oriented suggestions, plus conservative typography and basic checks. +//! France and Canada retain their regional tags; Quebec uses its own punctuation spacing. //! - **Any other language** ([`ProofLang::Pack`], module [`packs`]): spelling and suggestions //! from its Hunspell dictionary (an installed Language pack, the dictionaries folder or the //! system's) with default tuning, hyphenation from its pack, no grammar rules; without a @@ -73,6 +76,7 @@ pub mod dict; pub mod en_variants; mod english; pub mod es; +pub mod fr; pub mod hr; pub mod hunspell; pub mod hyphen; @@ -976,6 +980,7 @@ fn grammar_in(text: &str, lang: ProofLang, capital_first: bool) -> Vec { match lang { ProofLang::En => grammar_en(text, capital_first), ProofLang::EnGb | ProofLang::EnAu | ProofLang::EnCa => grammar_en(text, capital_first).into_iter().map(|i| Issue { lang, ..i }).collect(), + ProofLang::Pack(locale) if locale.primary() == "fr" => fr::grammar::check_in(text, lang), // `pt-pt` shares Portuguese's rules, its issues in its own language. _ => language(lang.built_in()) .map_or_else(Vec::new, |l| (l.grammar)(text, lang, capital_first).into_iter().map(|i| Issue { lang, ..i }).collect()), diff --git a/crates/proof/src/pack_lang.rs b/crates/proof/src/pack_lang.rs index 7b4d19315..8b6cea902 100644 --- a/crates/proof/src/pack_lang.rs +++ b/crates/proof/src/pack_lang.rs @@ -1,6 +1,7 @@ //! Proofing for any language by its locale ([`crate::ProofLang::Pack`]): its Hunspell dictionary (the //! dictionaries folder, then an installed Language pack, then the system's), read by the shared -//! affix engine, with default suggestion tuning and no grammar rules. +//! affix engine. Most use default suggestion tuning and no grammar rules; French locales add +//! the spelling tuning and conservative typography in [`crate::fr`] without a separate cache. //! //! The loaded dictionaries share a memory budget ([`MAX_LOADED_BYTES`], estimated from their //! files); the one used least recently goes first, but never one used during the current @@ -320,6 +321,9 @@ pub(crate) fn is_correct(word: &str, locale: PackLocale, opts: &crate::Options) if other_script(w, &scripts) { return true; } + if locale.primary() == "fr" { + return crate::fr::check(&d, w, opts); + } let known = |w: &str| d.check(w) || crate::in_user_dictionary(&w.to_lowercase()); if w.contains('-') && !known(w) { return w.split('-').all(|p| p.chars().count() <= 1 || known(p)); @@ -333,7 +337,7 @@ pub(crate) fn suggest(word: &str, max: usize, locale: PackLocale) -> Vec if d.is_empty() { return Vec::new(); } - crate::affix_suggest::suggest_in(&d, word, max, &Tuning::DEFAULT) + if locale.primary() == "fr" { crate::fr::suggest_in(&d, word, max) } else { crate::affix_suggest::suggest_in(&d, word, max, &Tuning::DEFAULT) } } /// The pack locales whose hyphenation patterns serve a [`crate::ProofLang::Pack`] language: the locale, diff --git a/crates/proof/tests/affix_compatibility.rs b/crates/proof/tests/affix_compatibility.rs new file mode 100644 index 000000000..6d09e5f90 --- /dev/null +++ b/crates/proof/tests/affix_compatibility.rs @@ -0,0 +1,92 @@ +//! Frozen public output from the pre-French shared affix engine at upstream +//! e6c5a701ddff5be16e266028168c70658aae39d9, using original synthetic fixtures. +//! Do not regenerate expectations from the new implementation: they establish the baseline. +//! Legacy WCAFF1 bytes remain frozen, but are explicitly refused by the WCAFF2 engine. + +use craft_proof::affix::{AffixDict, capitalize}; +use std::fmt::Write; + +#[path = "fixtures/affix_compatibility_cases.rs"] +mod fixtures; + +fn snapshot(case: &fixtures::Case, dict: &AffixDict) -> String { + let mut out = format!("{}\nlen={} rules={}\n", case.name, dict.len(), dict.rule_count()); + writeln!(out, "try={:?} key={:?} map={:?} rep={:?} comments={:?}", dict.try_chars(), dict.key(), dict.map(), dict.rep(), dict.comments()) + .unwrap(); + let mut stems: Vec<_> = dict.stems().collect(); + stems.sort_unstable(); + for stem in &stems { + let mut forms = dict.forms(stem); + forms.sort_unstable(); + forms.dedup(); + writeln!(out, "forms {stem:?}: {forms:?}").unwrap(); + } + let mut probes: Vec = case.probes.iter().map(|word| (*word).to_string()).collect(); + for stem in stems { + probes.extend(dict.forms(stem)); + } + let originals = probes.clone(); + for word in originals { + probes.extend([word.to_uppercase(), word.to_lowercase(), capitalize(&word)]); + } + probes.sort_unstable(); + probes.dedup(); + for word in probes { + writeln!( + out, + "{word:?}: contains={} check={} suggest={} raw_suggest={}", + dict.contains(&word), + dict.check(&word), + dict.can_suggest(&word), + dict.contains_suggestible(&word) + ) + .unwrap(); + } + out +} + +#[test] +fn non_french_affix_outputs_match_pre_french_baseline() { + let expected = include_str!("fixtures/affix_compatibility.expected"); + let mut actual = String::new(); + for case in fixtures::cases() { + let dict = AffixDict::from_hunspell(&case.aff, &case.dic, &["original compatibility fixture"]).unwrap(); + let result = snapshot(&case, &dict); + let restored = AffixDict::from_bytes(&dict.to_bytes().unwrap()).unwrap(); + assert_eq!(snapshot(&case, &restored), result, "current compact round trip: {}", case.name); + actual.push_str(&result); + } + // craft-libs 55332dd predates this adaptation and supports a prefix plus a two-suffix + // chain (native test affix::tests::a_prefix_with_a_suffix_chain). The old Indonesian + // fixture's bermain + kan + lah therefore became a word, in these three cases only. + // Keep the frozen historical file untouched and assert every explicit transition once. + let mut expected = expected.to_string(); + let mut transitions = 0; + for word in ["BERMAINKANLAH", "Bermainkanlah", "bermainkanlah"] { + let before = format!("{word:?}: contains=false check=false suggest=false raw_suggest=false\n"); + let after = if word == "bermainkanlah" { + format!("{word:?}: contains=true check=true suggest=true raw_suggest=true\n") + } else { + format!("{word:?}: contains=false check=true suggest=true raw_suggest=false\n") + }; + assert_eq!(expected.matches(&before).count(), 1, "one historical observable: {word}"); + expected = expected.replacen(&before, &after, 1); + transitions += 1; + } + assert_eq!(transitions, 3, "the documented upstream exception has exactly three observables"); + assert_eq!(actual, expected, "shared engine output changed beyond the explicit upstream chain exception"); +} + +#[test] +fn legacy_compact_data_is_refused_without_reinterpreting_homonyms() { + let encoded = include_str!("fixtures/affix_compatibility.compact"); + let cases = fixtures::cases(); + let lines: Vec<_> = encoded.lines().collect(); + assert_eq!(lines.len(), cases.len()); + for (case, hex) in cases.iter().zip(lines) { + assert_eq!(hex.len() % 2, 0); + let bytes: Vec<_> = + hex.as_bytes().as_chunks::<2>().0.iter().map(|pair| u8::from_str_radix(std::str::from_utf8(pair).unwrap(), 16).unwrap()).collect(); + assert!(AffixDict::from_bytes(&bytes).is_err(), "WCAFF1 must be regenerated as WCAFF2: {}", case.name); + } +} diff --git a/crates/proof/tests/fixtures/affix_compatibility.compact b/crates/proof/tests/fixtures/affix_compatibility.compact new file mode 100644 index 000000000..5c8c627de --- /dev/null +++ b/crates/proof/tests/fixtures/affix_compatibility.compact @@ -0,0 +1,7 @@ +4d8bc10a825014446b7bbf62a07dd62744e5ae120b2ca2e02aafb8bceac54b295d067e91db76ea7f2516d16666ce30138c47ae3ba41e8c95a35cf884c89caf1c4b282789531ce4112756d1cadf809591a47e6a4bb39107aef2d64d5924e44f3dd42fd8ac4d09ca823c778d101b0c7011853e2d1be696536cf7e9ee57280dcd8dfe2dd4f97b88306faa2a0753b0f0274bea046c6f19dfd989a8a34dec70485d6b3e7c157553da617a03 +1d8c410e82301045bb9e534ce2deea11a8d8c48d10304196058b990418534aa25ec7852770e9aa17b3b27c79ffbf6a9768bd8515b2a32b8da6c796879bf1d4504ffe811dddfdec2c9c8a1a8d259ec33bbcc2077456a8439aee8f5556a418be90eb33e658e3069dc535941193888c462ae485d5a2a768ffa71244c32379968902d19a290e4174e6699dcc41d0308f17962aa67f +1dc7410ac2301005d0ace7141fdc1b3d42aa64a9a15d5497d392ca40eac818416f6fe8eaf1c65388f1483ba8c9439e5c30ebfae22a9314a93f2cf2ad1fcb94e20d09771c60823d0dada15521d06d1d2e6064308dd7fe3c90eb75657233bfd97789dcc2967db3c864ea03fd01 +1d8a3d0a02311406b77ea7f8c0dee8114437651453ac220a51a33cc88fc408e62c9e451bf55e1bb6199861baf94cca298d10135f39188753f43793f9c88e73c1859ff9912cade4060a5b4c102cc6a4abea410b7687b21f425b0f83df0b86bae57aa1a9f1ce9cbf6fa1a8b9e714bd50b51551f1ffd860444b3d +1d8c310e82401444b1dd53fcc45ef408282cb11008ab414b346a3641310413ed0cad95072034f450488c46f10a7f6fe4aeddcc9bcc0b4606a503d28538e15bbe0f2358c5bb4398f2258f787a860d3fa5c7644d268607d8881ba1ae3f1c9ba6e504ae6f0296c471d9ccb62d3605cc8947e780052ca00ff8c1067a842952fd89b8caae6e8c6878c716df3a56a4f3c01abfe222ed998ea5222f6c552e64d64426e7a794d53ae612fc00 +2dc84b0e8230100050d6738a49dc8b1e0114589800a11864398522c3a768a951120f2f212edf2b4e5e181e618793e13b6b1ab09ac60759963cb05db0e18f7d1905795622c9aa56dc6bb804253edfcad8e54b73dd4016a45877d8419c886b140522c718d2f08629967840a90cee41acf636f7a45dff3ffe3603b5ab8b243b0b702455e47ae08cc4daf552700cb534f37a31fc00 +35c7b98e82600045e1a9790a92e967dc97d20df71515b51345455114c1ed6d2616c6c2cad8d8fe2f36e67269cec9a7e5328a1296be65db31e7e6666cc9137bbd1dbba66e5aa67b9667e6c9f51c435295813c961b72e8d31f4887746a024da82934a50cc8a066d08c9a43736a012d281332a925b4a456d08ab2208b5a436b6a036d281bb2a92db4a576d08e722087da437bca855cca833cea001da82374a44ed0893a4367ea025da80c94a1b25096ca41392a0fe5a90254a01448a18a50912a4125aa0c95a90a54a1aa5095aa4135aa0ed5a906d4a09a50936a412daa0db5a90ed4a15448a5ba5097ea413daa0ff5290dd2a80134a086d0901a41232a0485a83014a62250848a42512a06c5a83814a71250824a42492a05a5a83494a6c41ff8197df77d0ffcf0fd08fcf4fd0cfcf6fd0e7cf37d0b7cf57d0dfcf2fdfa586b76f2aaf4657bae63ff8a97f475301cddfe0d8523d1583c914ca533d95cbea0144be54ab5566f345bed8edaedf5b5c17034d6275363365f98cb95b5ded8db9db377bdc3f174be883f71153771170ff1142ff196fe01 diff --git a/crates/proof/tests/fixtures/affix_compatibility.expected b/crates/proof/tests/fixtures/affix_compatibility.expected new file mode 100644 index 000000000..8fe3a9929 --- /dev/null +++ b/crates/proof/tests/fixtures/affix_compatibility.expected @@ -0,0 +1,497 @@ +pl: conditional strip, prefix cross products, diacritics +len=4 rules=5 +try="aeoiułkr" key="" map=[['a', 'ą'], ['o', 'ó', 'u']] rep=[("ż", "rz"), ("u", "ó")] comments=["original compatibility fixture"] +forms "Warszawa": ["Warszawa", "Warszawą"] +forms "kot": ["kot", "kotem", "koty", "niekot", "niekotem", "niekoty"] +forms "krowa": ["krowa", "krową"] +forms "piesek": ["piesek", "piesekem", "pieseky", "pieska"] +"": contains=false check=false suggest=false raw_suggest=false +"KOT": contains=false check=true suggest=true raw_suggest=false +"KOTEM": contains=false check=true suggest=true raw_suggest=false +"KOTY": contains=false check=true suggest=true raw_suggest=false +"KOTĄ": contains=false check=false suggest=false raw_suggest=false +"KROWA": contains=false check=true suggest=true raw_suggest=false +"KROWY": contains=false check=false suggest=false raw_suggest=false +"KROWĄ": contains=false check=true suggest=true raw_suggest=false +"Kot": contains=false check=true suggest=true raw_suggest=false +"Kotem": contains=false check=true suggest=true raw_suggest=false +"Koty": contains=false check=true suggest=true raw_suggest=false +"Kotą": contains=false check=false suggest=false raw_suggest=false +"Krowa": contains=false check=true suggest=true raw_suggest=false +"Krowy": contains=false check=false suggest=false raw_suggest=false +"Krową": contains=false check=true suggest=true raw_suggest=false +"NIE": contains=false check=false suggest=false raw_suggest=false +"NIEKOT": contains=false check=true suggest=true raw_suggest=false +"NIEKOTEM": contains=false check=true suggest=true raw_suggest=false +"NIEKOTY": contains=false check=true suggest=true raw_suggest=false +"NIEPIESKA": contains=false check=false suggest=false raw_suggest=false +"Nie": contains=false check=false suggest=false raw_suggest=false +"Niekot": contains=false check=true suggest=true raw_suggest=false +"Niekotem": contains=false check=true suggest=true raw_suggest=false +"Niekoty": contains=false check=true suggest=true raw_suggest=false +"Niepieska": contains=false check=false suggest=false raw_suggest=false +"PIESEK": contains=false check=true suggest=true raw_suggest=false +"PIESEKA": contains=false check=false suggest=false raw_suggest=false +"PIESEKEM": contains=false check=true suggest=true raw_suggest=false +"PIESEKY": contains=false check=true suggest=true raw_suggest=false +"PIESKA": contains=false check=true suggest=true raw_suggest=false +"Piesek": contains=false check=true suggest=true raw_suggest=false +"Pieseka": contains=false check=false suggest=false raw_suggest=false +"Piesekem": contains=false check=true suggest=true raw_suggest=false +"Pieseky": contains=false check=true suggest=true raw_suggest=false +"Pieska": contains=false check=true suggest=true raw_suggest=false +"WARSZAWA": contains=false check=true suggest=true raw_suggest=false +"WARSZAWĄ": contains=false check=true suggest=true raw_suggest=false +"Warszawa": contains=true check=true suggest=true raw_suggest=true +"Warszawą": contains=true check=true suggest=true raw_suggest=true +"kot": contains=true check=true suggest=true raw_suggest=true +"kotem": contains=true check=true suggest=true raw_suggest=true +"koty": contains=true check=true suggest=true raw_suggest=true +"kotą": contains=false check=false suggest=false raw_suggest=false +"krowa": contains=true check=true suggest=true raw_suggest=true +"krowy": contains=false check=false suggest=false raw_suggest=false +"krową": contains=true check=true suggest=true raw_suggest=true +"nie": contains=false check=false suggest=false raw_suggest=false +"niekot": contains=true check=true suggest=true raw_suggest=true +"niekotem": contains=true check=true suggest=true raw_suggest=true +"niekoty": contains=true check=true suggest=true raw_suggest=true +"niepieska": contains=false check=false suggest=false raw_suggest=false +"piesek": contains=true check=true suggest=true raw_suggest=true +"pieseka": contains=false check=false suggest=false raw_suggest=false +"piesekem": contains=true check=true suggest=true raw_suggest=true +"pieseky": contains=true check=true suggest=true raw_suggest=true +"pieska": contains=true check=true suggest=true raw_suggest=true +"warszawa": contains=false check=false suggest=false raw_suggest=false +"warszawą": contains=false check=false suggest=false raw_suggest=false +pt: UTF-8 flags, forbidden words, two suffixes +len=4 rules=3 +try="aeiouçãõ" key="" map=[] rep=[] comments=["original compatibility fixture"] +forms "bonito": ["bonita", "bonitas", "bonito", "bonitos"] +forms "casa": ["casa", "casas"] +forms "fazer": ["fazer", "refazer"] +forms "imundo": [] +"BONITA": contains=false check=true suggest=true raw_suggest=false +"BONITAS": contains=false check=true suggest=true raw_suggest=false +"BONITASS": contains=false check=false suggest=false raw_suggest=false +"BONITO": contains=false check=true suggest=true raw_suggest=false +"BONITOS": contains=false check=true suggest=true raw_suggest=false +"Bonita": contains=false check=true suggest=true raw_suggest=false +"Bonitas": contains=false check=true suggest=true raw_suggest=false +"Bonitass": contains=false check=false suggest=false raw_suggest=false +"Bonito": contains=false check=true suggest=true raw_suggest=false +"Bonitos": contains=false check=true suggest=true raw_suggest=false +"CASA": contains=false check=true suggest=true raw_suggest=false +"CASAA": contains=false check=false suggest=false raw_suggest=false +"CASAS": contains=false check=true suggest=true raw_suggest=false +"Casa": contains=false check=true suggest=true raw_suggest=false +"Casaa": contains=false check=false suggest=false raw_suggest=false +"Casas": contains=false check=true suggest=true raw_suggest=false +"FAZER": contains=false check=true suggest=true raw_suggest=false +"Fazer": contains=false check=true suggest=true raw_suggest=false +"IMUNDO": contains=false check=false suggest=false raw_suggest=false +"IMUNDOS": contains=false check=false suggest=false raw_suggest=false +"Imundo": contains=false check=false suggest=false raw_suggest=false +"Imundos": contains=false check=false suggest=false raw_suggest=false +"RECASAS": contains=false check=false suggest=false raw_suggest=false +"REFAZER": contains=false check=true suggest=true raw_suggest=false +"Recasas": contains=false check=false suggest=false raw_suggest=false +"Refazer": contains=false check=true suggest=true raw_suggest=false +"bonita": contains=true check=true suggest=true raw_suggest=true +"bonitas": contains=true check=true suggest=true raw_suggest=true +"bonitass": contains=false check=false suggest=false raw_suggest=false +"bonito": contains=true check=true suggest=true raw_suggest=true +"bonitos": contains=true check=true suggest=true raw_suggest=true +"casa": contains=true check=true suggest=true raw_suggest=true +"casaa": contains=false check=false suggest=false raw_suggest=false +"casas": contains=true check=true suggest=true raw_suggest=true +"fazer": contains=true check=true suggest=true raw_suggest=true +"imundo": contains=false check=false suggest=false raw_suggest=false +"imundos": contains=false check=false suggest=false raw_suggest=false +"recasas": contains=false check=false suggest=false raw_suggest=false +"refazer": contains=true check=true suggest=true raw_suggest=true +it: stripping, non-crossing suffix, proper-name casing +len=4 rules=3 +try="" key="" map=[] rep=[] comments=["original compatibility fixture"] +forms "Roma": ["Roma"] +forms "casa": ["casa", "case", "ricasa"] +forms "fare": ["fare", "rifare"] +forms "libro": ["libri", "libro"] +"": contains=false check=false suggest=false raw_suggest=false +"CASA": contains=false check=true suggest=true raw_suggest=false +"CASE": contains=false check=true suggest=true raw_suggest=false +"Casa": contains=false check=true suggest=true raw_suggest=false +"Case": contains=false check=true suggest=true raw_suggest=false +"FARE": contains=false check=true suggest=true raw_suggest=false +"Fare": contains=false check=true suggest=true raw_suggest=false +"LIBRI": contains=false check=true suggest=true raw_suggest=false +"LIBRO": contains=false check=true suggest=true raw_suggest=false +"LIBROS": contains=false check=false suggest=false raw_suggest=false +"Libri": contains=false check=true suggest=true raw_suggest=false +"Libro": contains=false check=true suggest=true raw_suggest=false +"Libros": contains=false check=false suggest=false raw_suggest=false +"R ILIBRI": contains=false check=false suggest=false raw_suggest=false +"R ilibri": contains=false check=false suggest=false raw_suggest=false +"RICAS A": contains=false check=false suggest=false raw_suggest=false +"RICASA": contains=false check=true suggest=true raw_suggest=false +"RICASE": contains=false check=false suggest=false raw_suggest=false +"RIFARE": contains=false check=true suggest=true raw_suggest=false +"ROMA": contains=false check=true suggest=true raw_suggest=false +"Ricas a": contains=false check=false suggest=false raw_suggest=false +"Ricasa": contains=false check=true suggest=true raw_suggest=false +"Ricase": contains=false check=false suggest=false raw_suggest=false +"Rifare": contains=false check=true suggest=true raw_suggest=false +"Roma": contains=true check=true suggest=true raw_suggest=true +"casa": contains=true check=true suggest=true raw_suggest=true +"case": contains=true check=true suggest=true raw_suggest=true +"fare": contains=true check=true suggest=true raw_suggest=true +"libri": contains=true check=true suggest=true raw_suggest=true +"libro": contains=true check=true suggest=true raw_suggest=true +"libros": contains=false check=false suggest=false raw_suggest=false +"r ilibri": contains=false check=false suggest=false raw_suggest=false +"ricas a": contains=false check=false suggest=false raw_suggest=false +"ricasa": contains=true check=true suggest=true raw_suggest=true +"ricase": contains=false check=false suggest=false raw_suggest=false +"rifare": contains=true check=true suggest=true raw_suggest=true +"roma": contains=false check=false suggest=false raw_suggest=false +cs: negated conditions and shared Unicode stems +len=4 rules=3 +try="" key="" map=[] rep=[] comments=["original compatibility fixture"] +forms "mladý": ["mladý", "nemladý"] +forms "strom": ["nestrom", "nestromy", "strom", "stromy"] +forms "y": ["y"] +forms "žena": ["žena", "ženě"] +"MLADÝ": contains=false check=true suggest=true raw_suggest=false +"MLADÝY": contains=false check=false suggest=false raw_suggest=false +"Mladý": contains=false check=true suggest=true raw_suggest=false +"Mladýy": contains=false check=false suggest=false raw_suggest=false +"NEMLADÝ": contains=false check=true suggest=true raw_suggest=false +"NESTROM": contains=false check=true suggest=true raw_suggest=false +"NESTROMY": contains=false check=true suggest=true raw_suggest=false +"NEŽENĚ": contains=false check=false suggest=false raw_suggest=false +"Nemladý": contains=false check=true suggest=true raw_suggest=false +"Nestrom": contains=false check=true suggest=true raw_suggest=false +"Nestromy": contains=false check=true suggest=true raw_suggest=false +"Neženě": contains=false check=false suggest=false raw_suggest=false +"STROM": contains=false check=true suggest=true raw_suggest=false +"STROMY": contains=false check=true suggest=true raw_suggest=false +"Strom": contains=false check=true suggest=true raw_suggest=false +"Stromy": contains=false check=true suggest=true raw_suggest=false +"Y": contains=false check=true suggest=true raw_suggest=false +"YY": contains=false check=false suggest=false raw_suggest=false +"Yy": contains=false check=false suggest=false raw_suggest=false +"mladý": contains=true check=true suggest=true raw_suggest=true +"mladýy": contains=false check=false suggest=false raw_suggest=false +"nemladý": contains=true check=true suggest=true raw_suggest=true +"nestrom": contains=true check=true suggest=true raw_suggest=true +"nestromy": contains=true check=true suggest=true raw_suggest=true +"neženě": contains=false check=false suggest=false raw_suggest=false +"strom": contains=true check=true suggest=true raw_suggest=true +"stromy": contains=true check=true suggest=true raw_suggest=true +"y": contains=true check=true suggest=true raw_suggest=true +"yy": contains=false check=false suggest=false raw_suggest=false +"ŽENA": contains=false check=true suggest=true raw_suggest=false +"ŽENY": contains=false check=false suggest=false raw_suggest=false +"ŽENĚ": contains=false check=true suggest=true raw_suggest=false +"Žena": contains=false check=true suggest=true raw_suggest=false +"Ženy": contains=false check=false suggest=false raw_suggest=false +"Ženě": contains=false check=true suggest=true raw_suggest=false +"žena": contains=true check=true suggest=true raw_suggest=true +"ženy": contains=false check=false suggest=false raw_suggest=false +"ženě": contains=true check=true suggest=true raw_suggest=true +ru: Cyrillic flags and suggestions exclusion +len=4 rules=2 +try="" key="" map=[['е', 'ё']] rep=[] comments=["original compatibility fixture"] +forms "дом": ["дом", "домы"] +forms "запрет": [] +forms "кот": ["кот", "коты", "некот", "некоты"] +forms "тайна": ["тайна", "тайнаы"] +"ДОМ": contains=false check=true suggest=true raw_suggest=false +"ДОМЫ": contains=false check=true suggest=true raw_suggest=false +"Дом": contains=false check=true suggest=true raw_suggest=false +"Домы": contains=false check=true suggest=true raw_suggest=false +"ЗАПРЕТ": contains=false check=false suggest=false raw_suggest=false +"ЗАПРЕТЫ": contains=false check=false suggest=false raw_suggest=false +"Запрет": contains=false check=false suggest=false raw_suggest=false +"Запреты": contains=false check=false suggest=false raw_suggest=false +"КОТ": contains=false check=true suggest=true raw_suggest=false +"КОТЫ": contains=false check=true suggest=true raw_suggest=false +"КОт": contains=false check=false suggest=false raw_suggest=false +"Кот": contains=false check=true suggest=true raw_suggest=false +"Коты": contains=false check=true suggest=true raw_suggest=false +"НЕДОМЫ": contains=false check=false suggest=false raw_suggest=false +"НЕКОТ": contains=false check=true suggest=true raw_suggest=false +"НЕКОТЫ": contains=false check=true suggest=true raw_suggest=false +"Недомы": contains=false check=false suggest=false raw_suggest=false +"Некот": contains=false check=true suggest=true raw_suggest=false +"Некоты": contains=false check=true suggest=true raw_suggest=false +"ТАЙНА": contains=false check=true suggest=false raw_suggest=false +"ТАЙНАЫ": contains=false check=true suggest=false raw_suggest=false +"Тайна": contains=false check=true suggest=false raw_suggest=false +"Тайнаы": contains=false check=true suggest=false raw_suggest=false +"дом": contains=true check=true suggest=true raw_suggest=true +"домы": contains=true check=true suggest=true raw_suggest=true +"запрет": contains=false check=false suggest=false raw_suggest=false +"запреты": contains=false check=false suggest=false raw_suggest=false +"кОт": contains=false check=false suggest=false raw_suggest=false +"кот": contains=true check=true suggest=true raw_suggest=true +"коты": contains=true check=true suggest=true raw_suggest=true +"недомы": contains=false check=false suggest=false raw_suggest=false +"некот": contains=true check=true suggest=true raw_suggest=true +"некоты": contains=true check=true suggest=true raw_suggest=true +"тайна": contains=true check=true suggest=false raw_suggest=false +"тайнаы": contains=true check=true suggest=false raw_suggest=false +id: continuation and NOSUGGEST through derivations +len=3 rules=3 +try="abcdeikn" key="qwerty|asdf" map=[] rep=[("dj", "j")] comments=["original compatibility fixture"] +forms "baca": ["baca", "bacakan", "bacakanlah"] +forms "main": ["bermain", "bermainkan", "main", "mainkan", "mainkanlah"] +forms "rahasia": ["rahasia", "rahasiakan", "rahasiakanlah"] +"BACA": contains=false check=true suggest=true raw_suggest=false +"BACAKAN": contains=false check=true suggest=true raw_suggest=false +"BACAKANLAH": contains=false check=true suggest=true raw_suggest=false +"BERBACA": contains=false check=false suggest=false raw_suggest=false +"BERMAIN": contains=false check=true suggest=true raw_suggest=false +"BERMAINKAN": contains=false check=true suggest=true raw_suggest=false +"BERMAINKANLAH": contains=false check=false suggest=false raw_suggest=false +"Baca": contains=false check=true suggest=true raw_suggest=false +"Bacakan": contains=false check=true suggest=true raw_suggest=false +"Bacakanlah": contains=false check=true suggest=true raw_suggest=false +"Berbaca": contains=false check=false suggest=false raw_suggest=false +"Bermain": contains=false check=true suggest=true raw_suggest=false +"Bermainkan": contains=false check=true suggest=true raw_suggest=false +"Bermainkanlah": contains=false check=false suggest=false raw_suggest=false +"MAIN": contains=false check=true suggest=true raw_suggest=false +"MAINKAN": contains=false check=true suggest=true raw_suggest=false +"MAINKANLAH": contains=false check=true suggest=true raw_suggest=false +"MAINKANLAHLAH": contains=false check=false suggest=false raw_suggest=false +"MAINLAH": contains=false check=false suggest=false raw_suggest=false +"Main": contains=false check=true suggest=true raw_suggest=false +"Mainkan": contains=false check=true suggest=true raw_suggest=false +"Mainkanlah": contains=false check=true suggest=true raw_suggest=false +"Mainkanlahlah": contains=false check=false suggest=false raw_suggest=false +"Mainlah": contains=false check=false suggest=false raw_suggest=false +"RAHASIA": contains=false check=true suggest=false raw_suggest=false +"RAHASIAKAN": contains=false check=true suggest=false raw_suggest=false +"RAHASIAKANLAH": contains=false check=true suggest=false raw_suggest=false +"Rahasia": contains=false check=true suggest=false raw_suggest=false +"Rahasiakan": contains=false check=true suggest=false raw_suggest=false +"Rahasiakanlah": contains=false check=true suggest=false raw_suggest=false +"baca": contains=true check=true suggest=true raw_suggest=true +"bacakan": contains=true check=true suggest=true raw_suggest=true +"bacakanlah": contains=true check=true suggest=true raw_suggest=true +"berbaca": contains=false check=false suggest=false raw_suggest=false +"bermain": contains=true check=true suggest=true raw_suggest=true +"bermainkan": contains=true check=true suggest=true raw_suggest=true +"bermainkanlah": contains=false check=false suggest=false raw_suggest=false +"main": contains=true check=true suggest=true raw_suggest=true +"mainkan": contains=true check=true suggest=true raw_suggest=true +"mainkanlah": contains=true check=true suggest=true raw_suggest=true +"mainkanlahlah": contains=false check=false suggest=false raw_suggest=false +"mainlah": contains=false check=false suggest=false raw_suggest=false +"rahasia": contains=true check=true suggest=false raw_suggest=false +"rahasiakan": contains=true check=true suggest=false raw_suggest=false +"rahasiakanlah": contains=true check=true suggest=false raw_suggest=false +pt-sized: 70 distinct flags across the old u128 boundary +len=2 rules=70 +try="" key="" map=[] rep=[] comments=["original compatibility fixture"] +forms "outro": ["outro", "outroõ"] +forms "verbo": ["verbo", "verbo1", "verbo2", "verbo3", "verbo4", "verbo5", "verbo6", "verbo7", "verbo8", "verbo9", "verboA", "verboB", "verboC", "verboD", "verboE", "verboF", "verboG", "verboH", "verboI", "verboJ", "verboK", "verboL", "verboM", "verboN", "verboO", "verboP", "verboQ", "verboR", "verboS", "verboT", "verboU", "verboV", "verboW", "verboX", "verboY", "verboZ", "verboa", "verbob", "verboc", "verbod", "verboe", "verbof", "verbog", "verboh", "verboi", "verboj", "verbok", "verbol", "verbom", "verbon", "verboo", "verbop", "verboq", "verbor", "verbos", "verbot", "verbou", "verbov", "verbow", "verbox", "verboy", "verboz", "verboá", "verboã", "verboç", "verboé", "verboí", "verboó", "verboõ", "verboú"] +"OUTRO": contains=false check=true suggest=true raw_suggest=false +"OUTROA": contains=false check=false suggest=false raw_suggest=false +"OUTROÕ": contains=false check=true suggest=true raw_suggest=false +"Outro": contains=false check=true suggest=true raw_suggest=false +"Outroa": contains=false check=false suggest=false raw_suggest=false +"Outroõ": contains=false check=true suggest=true raw_suggest=false +"VERBO": contains=false check=true suggest=true raw_suggest=false +"VERBO0": contains=false check=false suggest=false raw_suggest=false +"VERBO1": contains=false check=true suggest=true raw_suggest=false +"VERBO2": contains=false check=true suggest=true raw_suggest=false +"VERBO3": contains=false check=true suggest=true raw_suggest=false +"VERBO4": contains=false check=true suggest=true raw_suggest=false +"VERBO5": contains=false check=true suggest=true raw_suggest=false +"VERBO6": contains=false check=true suggest=true raw_suggest=false +"VERBO7": contains=false check=true suggest=true raw_suggest=false +"VERBO8": contains=false check=true suggest=true raw_suggest=false +"VERBO9": contains=false check=true suggest=true raw_suggest=false +"VERBOA": contains=false check=true suggest=true raw_suggest=false +"VERBOB": contains=false check=true suggest=true raw_suggest=false +"VERBOC": contains=false check=true suggest=true raw_suggest=false +"VERBOD": contains=false check=true suggest=true raw_suggest=false +"VERBOE": contains=false check=true suggest=true raw_suggest=false +"VERBOF": contains=false check=true suggest=true raw_suggest=false +"VERBOG": contains=false check=true suggest=true raw_suggest=false +"VERBOH": contains=false check=true suggest=true raw_suggest=false +"VERBOI": contains=false check=true suggest=true raw_suggest=false +"VERBOJ": contains=false check=true suggest=true raw_suggest=false +"VERBOK": contains=false check=true suggest=true raw_suggest=false +"VERBOL": contains=false check=true suggest=true raw_suggest=false +"VERBOM": contains=false check=true suggest=true raw_suggest=false +"VERBON": contains=false check=true suggest=true raw_suggest=false +"VERBOO": contains=false check=true suggest=true raw_suggest=false +"VERBOP": contains=false check=true suggest=true raw_suggest=false +"VERBOQ": contains=false check=true suggest=true raw_suggest=false +"VERBOR": contains=false check=true suggest=true raw_suggest=false +"VERBOS": contains=false check=true suggest=true raw_suggest=false +"VERBOSS": contains=false check=false suggest=false raw_suggest=false +"VERBOT": contains=false check=true suggest=true raw_suggest=false +"VERBOU": contains=false check=true suggest=true raw_suggest=false +"VERBOV": contains=false check=true suggest=true raw_suggest=false +"VERBOW": contains=false check=true suggest=true raw_suggest=false +"VERBOX": contains=false check=true suggest=true raw_suggest=false +"VERBOY": contains=false check=true suggest=true raw_suggest=false +"VERBOZ": contains=false check=true suggest=true raw_suggest=false +"VERBOÁ": contains=false check=true suggest=true raw_suggest=false +"VERBOÃ": contains=false check=true suggest=true raw_suggest=false +"VERBOÇ": contains=false check=true suggest=true raw_suggest=false +"VERBOÉ": contains=false check=true suggest=true raw_suggest=false +"VERBOÍ": contains=false check=true suggest=true raw_suggest=false +"VERBOÓ": contains=false check=true suggest=true raw_suggest=false +"VERBOÕ": contains=false check=true suggest=true raw_suggest=false +"VERBOÚ": contains=false check=true suggest=true raw_suggest=false +"Verbo": contains=false check=true suggest=true raw_suggest=false +"Verbo0": contains=false check=false suggest=false raw_suggest=false +"Verbo1": contains=false check=true suggest=true raw_suggest=false +"Verbo2": contains=false check=true suggest=true raw_suggest=false +"Verbo3": contains=false check=true suggest=true raw_suggest=false +"Verbo4": contains=false check=true suggest=true raw_suggest=false +"Verbo5": contains=false check=true suggest=true raw_suggest=false +"Verbo6": contains=false check=true suggest=true raw_suggest=false +"Verbo7": contains=false check=true suggest=true raw_suggest=false +"Verbo8": contains=false check=true suggest=true raw_suggest=false +"Verbo9": contains=false check=true suggest=true raw_suggest=false +"VerboA": contains=false check=false suggest=false raw_suggest=false +"VerboB": contains=false check=false suggest=false raw_suggest=false +"VerboC": contains=false check=false suggest=false raw_suggest=false +"VerboD": contains=false check=false suggest=false raw_suggest=false +"VerboE": contains=false check=false suggest=false raw_suggest=false +"VerboF": contains=false check=false suggest=false raw_suggest=false +"VerboG": contains=false check=false suggest=false raw_suggest=false +"VerboH": contains=false check=false suggest=false raw_suggest=false +"VerboI": contains=false check=false suggest=false raw_suggest=false +"VerboJ": contains=false check=false suggest=false raw_suggest=false +"VerboK": contains=false check=false suggest=false raw_suggest=false +"VerboL": contains=false check=false suggest=false raw_suggest=false +"VerboM": contains=false check=false suggest=false raw_suggest=false +"VerboN": contains=false check=false suggest=false raw_suggest=false +"VerboO": contains=false check=false suggest=false raw_suggest=false +"VerboP": contains=false check=false suggest=false raw_suggest=false +"VerboQ": contains=false check=false suggest=false raw_suggest=false +"VerboR": contains=false check=false suggest=false raw_suggest=false +"VerboS": contains=false check=false suggest=false raw_suggest=false +"VerboT": contains=false check=false suggest=false raw_suggest=false +"VerboU": contains=false check=false suggest=false raw_suggest=false +"VerboV": contains=false check=false suggest=false raw_suggest=false +"VerboW": contains=false check=false suggest=false raw_suggest=false +"VerboX": contains=false check=false suggest=false raw_suggest=false +"VerboY": contains=false check=false suggest=false raw_suggest=false +"VerboZ": contains=false check=false suggest=false raw_suggest=false +"Verboa": contains=false check=true suggest=true raw_suggest=false +"Verbob": contains=false check=true suggest=true raw_suggest=false +"Verboc": contains=false check=true suggest=true raw_suggest=false +"Verbod": contains=false check=true suggest=true raw_suggest=false +"Verboe": contains=false check=true suggest=true raw_suggest=false +"Verbof": contains=false check=true suggest=true raw_suggest=false +"Verbog": contains=false check=true suggest=true raw_suggest=false +"Verboh": contains=false check=true suggest=true raw_suggest=false +"Verboi": contains=false check=true suggest=true raw_suggest=false +"Verboj": contains=false check=true suggest=true raw_suggest=false +"Verbok": contains=false check=true suggest=true raw_suggest=false +"Verbol": contains=false check=true suggest=true raw_suggest=false +"Verbom": contains=false check=true suggest=true raw_suggest=false +"Verbon": contains=false check=true suggest=true raw_suggest=false +"Verboo": contains=false check=true suggest=true raw_suggest=false +"Verbop": contains=false check=true suggest=true raw_suggest=false +"Verboq": contains=false check=true suggest=true raw_suggest=false +"Verbor": contains=false check=true suggest=true raw_suggest=false +"Verbos": contains=false check=true suggest=true raw_suggest=false +"Verbot": contains=false check=true suggest=true raw_suggest=false +"Verbou": contains=false check=true suggest=true raw_suggest=false +"Verbov": contains=false check=true suggest=true raw_suggest=false +"Verbow": contains=false check=true suggest=true raw_suggest=false +"Verbox": contains=false check=true suggest=true raw_suggest=false +"Verboy": contains=false check=true suggest=true raw_suggest=false +"Verboz": contains=false check=true suggest=true raw_suggest=false +"Verboß": contains=false check=false suggest=false raw_suggest=false +"Verboá": contains=false check=true suggest=true raw_suggest=false +"Verboã": contains=false check=true suggest=true raw_suggest=false +"Verboç": contains=false check=true suggest=true raw_suggest=false +"Verboé": contains=false check=true suggest=true raw_suggest=false +"Verboí": contains=false check=true suggest=true raw_suggest=false +"Verboó": contains=false check=true suggest=true raw_suggest=false +"Verboõ": contains=false check=true suggest=true raw_suggest=false +"Verboú": contains=false check=true suggest=true raw_suggest=false +"outro": contains=true check=true suggest=true raw_suggest=true +"outroa": contains=false check=false suggest=false raw_suggest=false +"outroõ": contains=true check=true suggest=true raw_suggest=true +"verbo": contains=true check=true suggest=true raw_suggest=true +"verbo0": contains=false check=false suggest=false raw_suggest=false +"verbo1": contains=true check=true suggest=true raw_suggest=true +"verbo2": contains=true check=true suggest=true raw_suggest=true +"verbo3": contains=true check=true suggest=true raw_suggest=true +"verbo4": contains=true check=true suggest=true raw_suggest=true +"verbo5": contains=true check=true suggest=true raw_suggest=true +"verbo6": contains=true check=true suggest=true raw_suggest=true +"verbo7": contains=true check=true suggest=true raw_suggest=true +"verbo8": contains=true check=true suggest=true raw_suggest=true +"verbo9": contains=true check=true suggest=true raw_suggest=true +"verboA": contains=true check=true suggest=true raw_suggest=true +"verboB": contains=true check=true suggest=true raw_suggest=true +"verboC": contains=true check=true suggest=true raw_suggest=true +"verboD": contains=true check=true suggest=true raw_suggest=true +"verboE": contains=true check=true suggest=true raw_suggest=true +"verboF": contains=true check=true suggest=true raw_suggest=true +"verboG": contains=true check=true suggest=true raw_suggest=true +"verboH": contains=true check=true suggest=true raw_suggest=true +"verboI": contains=true check=true suggest=true raw_suggest=true +"verboJ": contains=true check=true suggest=true raw_suggest=true +"verboK": contains=true check=true suggest=true raw_suggest=true +"verboL": contains=true check=true suggest=true raw_suggest=true +"verboM": contains=true check=true suggest=true raw_suggest=true +"verboN": contains=true check=true suggest=true raw_suggest=true +"verboO": contains=true check=true suggest=true raw_suggest=true +"verboP": contains=true check=true suggest=true raw_suggest=true +"verboQ": contains=true check=true suggest=true raw_suggest=true +"verboR": contains=true check=true suggest=true raw_suggest=true +"verboS": contains=true check=true suggest=true raw_suggest=true +"verboT": contains=true check=true suggest=true raw_suggest=true +"verboU": contains=true check=true suggest=true raw_suggest=true +"verboV": contains=true check=true suggest=true raw_suggest=true +"verboW": contains=true check=true suggest=true raw_suggest=true +"verboX": contains=true check=true suggest=true raw_suggest=true +"verboY": contains=true check=true suggest=true raw_suggest=true +"verboZ": contains=true check=true suggest=true raw_suggest=true +"verboa": contains=true check=true suggest=true raw_suggest=true +"verbob": contains=true check=true suggest=true raw_suggest=true +"verboc": contains=true check=true suggest=true raw_suggest=true +"verbod": contains=true check=true suggest=true raw_suggest=true +"verboe": contains=true check=true suggest=true raw_suggest=true +"verbof": contains=true check=true suggest=true raw_suggest=true +"verbog": contains=true check=true suggest=true raw_suggest=true +"verboh": contains=true check=true suggest=true raw_suggest=true +"verboi": contains=true check=true suggest=true raw_suggest=true +"verboj": contains=true check=true suggest=true raw_suggest=true +"verbok": contains=true check=true suggest=true raw_suggest=true +"verbol": contains=true check=true suggest=true raw_suggest=true +"verbom": contains=true check=true suggest=true raw_suggest=true +"verbon": contains=true check=true suggest=true raw_suggest=true +"verboo": contains=true check=true suggest=true raw_suggest=true +"verbop": contains=true check=true suggest=true raw_suggest=true +"verboq": contains=true check=true suggest=true raw_suggest=true +"verbor": contains=true check=true suggest=true raw_suggest=true +"verbos": contains=true check=true suggest=true raw_suggest=true +"verbot": contains=true check=true suggest=true raw_suggest=true +"verbou": contains=true check=true suggest=true raw_suggest=true +"verbov": contains=true check=true suggest=true raw_suggest=true +"verbow": contains=true check=true suggest=true raw_suggest=true +"verbox": contains=true check=true suggest=true raw_suggest=true +"verboy": contains=true check=true suggest=true raw_suggest=true +"verboz": contains=true check=true suggest=true raw_suggest=true +"verboß": contains=false check=false suggest=false raw_suggest=false +"verboá": contains=true check=true suggest=true raw_suggest=true +"verboã": contains=true check=true suggest=true raw_suggest=true +"verboç": contains=true check=true suggest=true raw_suggest=true +"verboé": contains=true check=true suggest=true raw_suggest=true +"verboí": contains=true check=true suggest=true raw_suggest=true +"verboó": contains=true check=true suggest=true raw_suggest=true +"verboõ": contains=true check=true suggest=true raw_suggest=true +"verboú": contains=true check=true suggest=true raw_suggest=true diff --git a/crates/proof/tests/fixtures/affix_compatibility_cases.rs b/crates/proof/tests/fixtures/affix_compatibility_cases.rs new file mode 100644 index 000000000..d0c2973b4 --- /dev/null +++ b/crates/proof/tests/fixtures/affix_compatibility_cases.rs @@ -0,0 +1,61 @@ +// Original, deliberately small fixtures for behavior supported before French proofing. +// These are not extracts from installed dictionaries and do not claim full language coverage. +pub struct Case { + pub name: &'static str, + pub aff: String, + pub dic: String, + pub probes: Vec<&'static str>, +} + +pub fn cases() -> Vec { + let mut out = vec![ + Case { + name: "pl: conditional strip, prefix cross products, diacritics", + aff: "SET UTF-8\nTRY aeoiułkr\nPFX b Y 1\nPFX b 0 nie .\nSFX a Y 3\nSFX a 0 y [^y]\nSFX a ek ka ek\nSFX a 0 em .\nSFX c N 1\nSFX c a ą a\nMAP 2\nMAP aą\nMAP oóu\nREP 2\nREP ż rz\nREP u ó\n".into(), + dic: "4\nkot/ab\npiesek/a\nkrowa/c\nWarszawa/c\n".into(), + probes: vec!["kot", "koty", "kotem", "niekoty", "pieska", "pieseka", "niepieska", "krową", "krowy", "Warszawą", "warszawą", "kotą", "nie", ""], + }, + Case { + name: "pt: UTF-8 flags, forbidden words, two suffixes", + aff: "SET UTF-8\nFLAG UTF-8\nFORBIDDENWORD ý\nTRY aeiouçãõ\nSFX A Y 1\nSFX A o a/B o\nSFX B Y 1\nSFX B 0 s .\nPFX P Y 1\nPFX P 0 re .\n".into(), + dic: "4\nbonito/AB\ncasa/B\nfazer/P\nimundo/ýB\n".into(), + probes: vec!["bonito", "bonita", "bonitos", "bonitas", "bonitass", "casa", "casas", "casaa", "refazer", "recasas", "imundo", "imundos", "Imundo", "IMUNDO"], + }, + Case { + name: "it: stripping, non-crossing suffix, proper-name casing", + aff: "SET UTF-8\nSFX A Y 1\nSFX A o i o\nSFX B N 1\nSFX B a e a\nPFX P Y 1\nPFX P 0 ri .\n".into(), + dic: "4\nlibro/A\ncasa/BP\nfare/P\nRoma\n".into(), + probes: vec!["libri", "libros", "case", "ricas a", "ricasa", "ricase", "rifare", "r ilibri", "Roma", "roma", "ROMA", ""], + }, + Case { + name: "cs: negated conditions and shared Unicode stems", + aff: "SET UTF-8\nPFX N Y 1\nPFX N 0 ne .\nSFX S Y 1\nSFX S 0 y [^y]\nSFX E N 1\nSFX E a ě a\n".into(), + dic: "4\nstrom/NS\nžena/E\nmladý/N\ny/S\n".into(), + probes: vec!["stromy", "nestromy", "nestrom", "ženě", "ženy", "neženě", "nemladý", "mladýy", "yy", "Ženě", "STROMY"], + }, + Case { + name: "ru: Cyrillic flags and suggestions exclusion", + aff: "SET UTF-8\nFLAG UTF-8\nNOSUGGEST Н\nFORBIDDENWORD Ф\nSFX Я Y 1\nSFX Я 0 ы .\nPFX П Y 1\nPFX П 0 не .\nMAP 1\nMAP её\n".into(), + dic: "4\nкот/ЯП\nдом/Я\nтайна/НЯ\nзапрет/ФЯ\n".into(), + probes: vec!["кот", "коты", "некоты", "домы", "недомы", "тайна", "тайнаы", "ТАЙНАЫ", "запрет", "запреты", "КОТЫ", "кОт"], + }, + Case { + name: "id: continuation and NOSUGGEST through derivations", + aff: "SET UTF-8\nNOSUGGEST N\nKEY qwerty|asdf\nTRY abcdeikn\nREP 1\nREP dj j\nPFX P Y 1\nPFX P 0 ber .\nSFX A Y 1\nSFX A 0 kan/B .\nSFX B Y 1\nSFX B 0 lah .\n".into(), + dic: "3\nmain/PA\nbaca/A\nrahasia/NA\n".into(), + probes: vec!["main", "bermain", "mainkan", "mainkanlah", "bermainkanlah", "mainlah", "mainkanlahlah", "bacakanlah", "berbaca", "rahasia", "rahasiakan", "RAHASIAKANLAH"], + }, + ]; + let flags: Vec = ('a'..='z').chain('A'..='Z').chain('0'..='9').chain(['á', 'é', 'í', 'ó', 'ú', 'ç', 'ã', 'õ']).collect(); + let mut aff = String::from("SET UTF-8\nFLAG UTF-8\n"); + for flag in &flags { + aff.push_str(&format!("SFX {flag} N 1\nSFX {flag} 0 {flag} .\n")); + } + out.push(Case { + name: "pt-sized: 70 distinct flags across the old u128 boundary", + aff, + dic: format!("2\nverbo/{}\noutro/õ\n", flags.iter().collect::()), + probes: vec!["verboa", "verboz", "verboA", "verboZ", "verbo0", "verbo9", "verboç", "verboõ", "verboß", "outroõ", "outroa"], + }); + out +} diff --git a/crates/proof/tests/french_dictionary.rs b/crates/proof/tests/french_dictionary.rs new file mode 100644 index 000000000..5612a0e61 --- /dev/null +++ b/crates/proof/tests/french_dictionary.rs @@ -0,0 +1,107 @@ +//! Explicit manual real-dictionary gate. Verify the pinned runtime inputs as documented in +//! crates/proof/README.md before running with `--ignored`. Missing data is a failure, +//! never a silently successful test that checked nothing. + +#![cfg(not(target_arch = "wasm32"))] +#![allow(clippy::unwrap_used, clippy::expect_used)] + +use craft_proof::{IssueKind, Options, ProofLang, fr}; + +#[test] +#[ignore = "requires a real French dictionary in WORDCRAFT_DICTIONARIES"] +fn real_french_dictionary_spelling_suggestions_and_dispatch() { + let directory = std::env::var_os("WORDCRAFT_DICTIONARIES").expect("set WORDCRAFT_DICTIONARIES to the test dictionary folder"); + craft_proof::set_dictionaries_folder(Some(std::path::PathBuf::from(&directory))); + let french = ProofLang::from_tag(Some("fr-FR")); + let installed = craft_proof::pack_dictionary_file(french).expect("the real French dictionary must be installed"); + assert_eq!(installed.parent(), Some(std::path::Path::new(&directory)), "do not silently use another system dictionary"); + let aff = installed.with_extension("aff"); + let dictionary = craft_proof::hunspell::load(&aff, &installed).expect("the real French dictionary loads"); + assert!(dictionary.len() > 50_000, "expected Dicollecte, not the synthetic fixture: {}", dictionary.len()); + for word in [ + "maison", + "maisons", + "Paris", + "mangeons", + "finissaient", + "étaient", + "l'arbre", + "l’arbre", + "qu'elles", + "école", + "œuvre", + "aujourd'hui", + "chevaux", + "heureuse", + "va-t-il", + ] { + assert!(fr::is_correct(word), "correct French word: {word}"); + } + for (typo, expected) in [("maisonn", "maison"), ("ecole", "école"), ("l’ecole", "l’école"), ("documant", "document")] { + assert!(!fr::is_correct(typo), "misspelling: {typo}"); + let suggestions = fr::suggest(typo, 8); + assert!(suggestions.iter().any(|s| s == expected), "{typo} -> {suggestions:?}, expected {expected}"); + } + let text = "Une maisonn près de l’école."; + for lang in [ProofLang::from_tag(Some("fr-FR")), ProofLang::from_tag(Some("fr-CA"))] { + let issues = craft_proof::check_text(text, &[(0..text.len(), lang)]); + assert_eq!(issues.len(), 1, "{lang:?}: {issues:?}"); + let issue = &issues[0]; + assert_eq!(&text[issue.start..issue.end], "maisonn"); + assert_eq!(issue.lang, lang); + assert_eq!(issue.message, "Faute d’orthographe possible"); + } + check_combining_marks_and_compounds(&dictionary); + assert!(craft_proof::dictionary_load_error(ProofLang::from_tag(Some("fr-FR"))).is_none()); +} + +fn check_combining_marks_and_compounds(dictionary: &craft_proof::affix::AffixDict) { + let spelling = |text: &str, spans: &[(std::ops::Range, ProofLang)]| { + craft_proof::check_text_with(text, spans, &Options::default(), false, false) + .into_iter() + .filter(|i| i.kind == IssueKind::Spelling) + .map(|i| (i.start, i.end, i.lang)) + .collect::>() + }; + let mut failures = Vec::new(); + for lang in [ProofLang::from_tag(Some("fr-FR")), ProofLang::from_tag(Some("fr-CA"))] { + for text in + ["école", "e\u{301}cole", "l’école", "l’e\u{301}cole", "café", "cafe\u{301}", "porte-monnaie", "va-t-il", "a-t-il", "maison-école"] + { + assert!( + fr::is_correct_with(dictionary, text) || ["va-t-il", "a-t-il", "maison-école"].contains(&text), + "real dictionary recognizes {text:?}" + ); + let issues = spelling(text, &[(0..text.len(), lang)]); + if !issues.is_empty() { + failures.push(format!("correct {lang:?} {text:?}: {issues:?}")); + } + } + for text in ["porte--monnaie", "va--t-il", "maison--école", "e\u{301}colllle", "cafffe\u{301}"] { + assert!(!fr::is_correct_with(dictionary, text), "the explicit dictionary entry must not already accept {text:?}"); + let issues = spelling(text, &[(0..text.len(), lang)]); + let expected = vec![(0, text.len(), lang)]; + if issues != expected { + failures.push(format!("typo {lang:?} {text:?}: {issues:?}, wanted {expected:?}")); + } + } + let text = "e\u{301}cole cafe\u{301} xyzxyz porte--monnaie"; + let foreign = text.find("xyzxyz").unwrap(); + let compound = text.find("porte").unwrap(); + let spans = [(0..foreign, lang), (foreign..compound, ProofLang::Other), (compound..text.len(), lang)]; + let issues = spelling(text, &spans); + if issues != [(compound, text.len(), lang)] { + failures.push(format!("mixed {lang:?}: {issues:?}")); + } + assert!(spelling(text, &[(0..text.len(), ProofLang::Other)]).is_empty()); + // An English word remains in its own dictionary next to the original French bytes. + let text = "e\u{301}cole hello cafe\u{301}"; + let en = text.find("hello").unwrap(); + let spans = [(0..en, lang), (en..en + 5, ProofLang::En), (en + 5..text.len(), lang)]; + let issues = spelling(text, &spans); + if !issues.is_empty() { + failures.push(format!("English boundary {lang:?}: {issues:?}")); + } + } + assert!(failures.is_empty(), "{}", failures.join("\n")); +} diff --git a/crates/proof/tests/french_packs.rs b/crates/proof/tests/french_packs.rs new file mode 100644 index 000000000..64ad46748 --- /dev/null +++ b/crates/proof/tests/french_packs.rs @@ -0,0 +1,124 @@ +//! French uses ordinary Language packs and keeps each regional locale at the public API. +#![allow(clippy::unwrap_used, clippy::expect_used)] + +use craft_proof::affix; +use craft_proof::{IssueKind, Options, ProofLang, check_text_with, is_correct_in_with, suggest_in}; +use std::path::{Path, PathBuf}; + +#[path = "../src/fr/test_dict.rs"] +mod fixture; + +struct Tools(PathBuf); +impl Drop for Tools { + fn drop(&mut self) { + craft_proof::set_dictionaries_folder(None); + craft_proof::packs::set_packs_dir(None); + craft_proof::remove_word("zxqrechargefr"); + let _ = std::fs::remove_dir_all(&self.0); + } +} + +fn pair(dir: &Path, name: &str, aff: &str, dic: &str) { + std::fs::create_dir_all(dir).unwrap(); + std::fs::write(dir.join(format!("{name}.aff")), aff).unwrap(); + std::fs::write(dir.join(format!("{name}.dic")), dic).unwrap(); +} + +#[test] +fn french_packs_regions_reload_and_original_ranges() { + assert!(!fixture::dictionary().is_empty()); + let root = std::env::temp_dir().join(format!("craft-fr-packs-{}", std::process::id())); + let _tools = Tools(root.clone()); + let pack = root.join("packs/fr"); + pair(&pack, "spelling", fixture::AFF, fixture::DIC); + std::fs::write(pack.join("pack.toml"), "locale = 'fr'\nname = 'Français'\nlicence = 'MIT'\n").unwrap(); + std::fs::write(pack.join("hyphenation.pat"), "ma1ison\n").unwrap(); + let folder = root.join("dictionaries"); + std::fs::create_dir_all(&folder).unwrap(); + craft_proof::packs::set_packs_dir(Some(root.join("packs"))); + craft_proof::set_dictionaries_folder(Some(folder.clone())); + let defaults = Options::default(); + let strict = Options { ignore_uppercase: false, ignore_numbers: false, ..defaults }; + check_locales(defaults, strict); + check_typography(strict); + let fr = ProofLang::from_tag(Some("fr-FR")); + let ca = ProofLang::from_tag(Some("fr-CA")); + check_user_word(fr, ca, strict); + check_overrides(folder, fr, ca, strict); +} + +fn check_locales(defaults: Options, strict: Options) { + for tag in ["fr", "fr-FR", "fr-CA", "fr-BE", "fr-CA-u-nu-latn", "fr-u-ca-gregory"] { + let lang = ProofLang::from_tag(Some(tag)); + assert!(matches!(lang, ProofLang::Pack(_)), "{tag}"); + assert!(craft_proof::has_dictionary(Some(tag)), "{tag}"); + assert!(craft_proof::pack_dictionary_file(lang).unwrap().ends_with("fr/spelling.dic"), "{tag}"); + for word in ["école", "e\u{301}cole", "l'arbre", "l’arbre", "maison-école"] { + assert!(is_correct_in_with(word, lang, &strict), "{tag}: {word}"); + } + assert!(!is_correct_in_with("maison--école", lang, &strict)); + assert!(!is_correct_in_with("l'arbres", lang, &strict)); + assert!(is_correct_in_with("SNCF", lang, &defaults)); + assert!(!is_correct_in_with("SNCF", lang, &strict)); + assert!(is_correct_in_with("mp3x", lang, &defaults)); + assert!(!is_correct_in_with("mp3x", lang, &strict)); + assert!(suggest_in("ecole", 6, lang).iter().any(|s| s == "école")); + let text = "🙂 maison\u{338} école"; + let issues = check_text_with(text, &[(0..text.len(), lang)], &strict, false, false); + assert_eq!(issues.len(), 1, "{tag}: {issues:?}"); + assert_eq!(&text[issues[0].start..issues[0].end], "maison\u{338}"); + assert_eq!(issues[0].lang, lang); + assert_eq!(issues[0].kind, IssueKind::Spelling); + assert_eq!(issues[0].message, "Faute d’orthographe possible"); + } +} + +fn check_typography(strict: Options) { + let fr = ProofLang::from_tag(Some("fr-FR")); + let ca = ProofLang::from_tag(Some("fr-CA")); + for lang in [fr, ca] { + assert_eq!(craft_proof::hyphen::hyphen_points_in("maison", &craft_proof::hyphen::Limits::default(), lang), [2]); + } + for tag in ["fr-FR", "fr-BE", "fr-u-ca-gregory", "fr-FR-u-rg-cazzzz"] { + let issues = craft_proof::check_grammar_in("Bonjour!", ProofLang::from_tag(Some(tag))); + assert_eq!(issues.len(), 1, "{tag}: {issues:?}"); + assert_eq!(issues[0].suggestions, ["\u{202f}!"]); + } + assert!(craft_proof::check_grammar_in("Bonjour!", ca).is_empty()); + assert!(craft_proof::check_grammar_in("Bonjour!", ProofLang::from_tag(Some("fr-Latn-CA"))).is_empty()); + assert!(craft_proof::check_grammar_in("Bonjour!", ProofLang::from_tag(Some("nl-NL"))).is_empty()); + assert!(craft_proof::fr::grammar::check_in("Bonjour!", ProofLang::En).is_empty()); + let text = "Bonjour! Salut!"; + let issues = check_text_with(text, &[(0..8, fr), (8..text.len(), ca)], &strict, true, false); + let spacing: Vec<_> = issues.iter().filter(|i| i.kind == IssueKind::Grammar && i.message.contains("fine insécable")).collect(); + assert_eq!(spacing.len(), 1, "{issues:?}"); + assert_eq!(spacing[0].lang, fr); +} + +fn check_user_word(fr: ProofLang, ca: ProofLang, strict: Options) { + craft_proof::add_word("zxqrechargefr"); + for lang in [fr, ca] { + assert!(is_correct_in_with("l’zxqrechargefr", lang, &strict)); + } + craft_proof::remove_word("zxqrechargefr"); + assert!(!is_correct_in_with("l’zxqrechargefr", ca, &strict)); +} + +fn check_overrides(folder: PathBuf, fr: ProofLang, ca: ProofLang, strict: Options) { + // Regional overrides keep the generic pack fallback and the shared reload generation. + pair(&folder, "fr_FR", "SET UTF-8\n", "1\nhexagonefixture\n"); + pair(&folder, "fr_CA", "SET UTF-8\n", "1\nquebecfixture\n"); + let generation = craft_proof::dictionary_generation(); + craft_proof::set_dictionaries_folder(Some(folder.clone())); + assert!(craft_proof::dictionary_generation() > generation); + assert!(is_correct_in_with("hexagonefixture", fr, &strict)); + assert!(!is_correct_in_with("hexagonefixture", ca, &strict)); + assert!(is_correct_in_with("quebecfixture", ca, &strict)); + assert!(!is_correct_in_with("quebecfixture", fr, &strict)); + assert!(craft_proof::pack_dictionary_file(ca).unwrap().ends_with("fr_CA.dic")); + std::fs::remove_file(folder.join("fr_CA.dic")).unwrap(); + std::fs::remove_file(folder.join("fr_CA.aff")).unwrap(); + craft_proof::set_dictionaries_folder(Some(folder)); + assert!(is_correct_in_with("école", ca, &strict), "removed regional override falls back to fr pack"); + assert!(craft_proof::pack_dictionary_file(ca).unwrap().ends_with("fr/spelling.dic")); +} From 3c5ffadd3b3fb4bcbf657a245ad9c480d2b6e304 Mon Sep 17 00:00:00 2001 From: totoz64 <47389794+totoz64@users.noreply.github.com> Date: Tue, 3 Nov 2026 11:53:24 +0100 Subject: [PATCH 002/161] Proofing: retain exact apostrophes in French compounds Check each original compound component before its ASCII-apostrophe alias. Add a curly-only dictionary regression, prove legacy fixture headers before rejecting WCAFF1, and clarify French-specific Pack grammar in the README. Preserve the existing one-letter component policy. --- crates/proof/README.md | 13 ++++++++----- crates/proof/src/fr/mod.rs | 14 +++++++++++++- crates/proof/tests/affix_compatibility.rs | 2 ++ 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/crates/proof/README.md b/crates/proof/README.md index b4b4b5ba8..4cc4b2ca4 100644 --- a/crates/proof/README.md +++ b/crates/proof/README.md @@ -89,19 +89,22 @@ pub fn load_errors() -> Vec<(String, String)>; // + (locale or folder, pub fn embedded_locales() -> &'static [&'static str]; // + locales with embedded upstream data, not our supplements ("en-us", "en-gb", "en-ca", "en-au", "pl", "pt", "it", "de" by default; the four English ones with CRAFT_PROOF_EMBED=en) // craft_proof (crate root) -pub enum ProofLang { …, Pack(PackLocale), Other } // + Pack: any language without tools of its own (nl, fr, sr-Latn…) +pub enum ProofLang { …, Pack(PackLocale), Other } // + Pack: a language without a dedicated built-in variant (nl, fr, sr-Latn…) pub struct PackLocale; // + Copy; PackLocale::new(tag) -> Option, .as_str() "nl-nl", .primary() "nl", Display impl ProofLang { pub fn pack_locale(self) -> Option; } // + pub fn pack_dictionary_file(lang: ProofLang) -> Option; // + the .dic a Pack language reads now; None = unchecked ``` -`ProofLang::from_tag` now gives `Pack(locale)` for every valid language tag it has no tools for +`ProofLang::from_tag` now gives `Pack(locale)` for every valid language tag without a dedicated built-in variant (`nl-NL` → `Pack("nl-nl")`, `fr-FR`, `ja-JP`, `sr-Latn`…); `Other` is left for `x-none`, `und`, `zxx`, `mul`, `mis`, `i-…` and malformed tags. A `Pack` language is spell-checked and given suggestions from its Hunspell dictionary (dictionaries folder → installed pack → system Hunspell) -through the shared affix engine with default tuning, and hyphenated from its pack's -`hyphenation.pat`; it has no grammar rules and the generic English spelling message. Without a -dictionary or pack it behaves exactly as `Other` did (no marks, English hyphenation fallback). +through the shared affix engine with default tuning (French tuning for `fr` packs), and +hyphenated from its pack's `hyphenation.pat`. Most pack languages use no dedicated grammar +rules; regional variants can reuse a built-in language's rules. French packs use the French +spelling message and the conservative French grammar and typography described above. Without a dictionary or pack, +spelling has no marks and hyphenation falls back to English; French grammar and typography +still apply independently of a spelling dictionary. `has_dictionary(tag)` / `ProofLang::is_proofed()` say whether a `Pack` language has a dictionary now (read on first use, cached). Loaded `Pack` dictionaries share a 256 MB budget (estimated from their files); the least recently used goes first, but never one used in the paragraph being checked, and languages with no dictionary take no room. Downstream `match`es on `ProofLang` need a `Pack(_)` arm; `code()` stays `&'static str` and is `""` for it, as for `Other`: anything that keys, saves or reports a language (settings, caches, JSON/MCP output) uses `tag()`, which is `code()` for the built-ins and the locale (`nl`, `fr-fr`) for a pack language. diff --git a/crates/proof/src/fr/mod.rs b/crates/proof/src/fr/mod.rs index 541fcac0d..c67c3ffc0 100644 --- a/crates/proof/src/fr/mod.rs +++ b/crates/proof/src/fr/mod.rs @@ -44,6 +44,7 @@ pub(crate) fn check(d: &AffixDict, word: &str, opts: &Options) -> bool { if known(d, w) { return true; } + let original = w; let w = w.replace('’', "'"); if known(d, &w) { return true; @@ -56,7 +57,7 @@ pub(crate) fn check(d: &AffixDict, word: &str, opts: &Options) -> bool { { return true; } - w.contains('-') && w.split('-').all(|p| !p.is_empty() && (p.chars().count() <= 1 || known(d, p))) + original.contains('-') && original.split('-').all(|p| !p.is_empty() && (p.chars().count() <= 1 || known(d, p) || known(d, &p.replace('’', "'")))) } fn known(d: &AffixDict, word: &str) -> bool { @@ -148,6 +149,17 @@ mod tests { assert!(!is_correct_with(&d, "maison\u{338}"), "a trailing mark is part of the word"); } + #[test] + fn curly_apostrophes_in_compounds_keep_exact_dictionary_parts() { + let d = AffixDict::from_hunspell("SET UTF-8\n", "4\naujourd’hui\nécole\ncafé\nl'école\n", &[]).unwrap(); + for word in ["aujourd’hui-école", "aujourd’hui-e\u{301}cole", "café-aujourd’hui", "l’école-aujourd’hui"] { + assert!(is_correct_with(&d, word), "{word}"); + } + for word in ["aujourd’hui-écolle", "aujourd’hui--école", "aujourd'hui-école"] { + assert!(!is_correct_with(&d, word), "{word}"); + } + } + #[test] fn explicit_curly_apostrophe_without_iconv() { let d = AffixDict::from_hunspell("SET UTF-8\n", "1\naujourd’hui\n", &[]).unwrap(); diff --git a/crates/proof/tests/affix_compatibility.rs b/crates/proof/tests/affix_compatibility.rs index 6d09e5f90..c18d9fd03 100644 --- a/crates/proof/tests/affix_compatibility.rs +++ b/crates/proof/tests/affix_compatibility.rs @@ -87,6 +87,8 @@ fn legacy_compact_data_is_refused_without_reinterpreting_homonyms() { assert_eq!(hex.len() % 2, 0); let bytes: Vec<_> = hex.as_bytes().as_chunks::<2>().0.iter().map(|pair| u8::from_str_radix(std::str::from_utf8(pair).unwrap(), 16).unwrap()).collect(); + let raw = miniz_oxide::inflate::decompress_to_vec(&bytes).unwrap(); + assert!(raw.starts_with(b"WCAFF1\n"), "fixture must contain WCAFF1 data: {}", case.name); assert!(AffixDict::from_bytes(&bytes).is_err(), "WCAFF1 must be regenerated as WCAFF2: {}", case.name); } } From 870bad52e20a62e0260c90d5d18c5643181377b7 Mon Sep 17 00:00:00 2001 From: totoz64 <47389794+totoz64@users.noreply.github.com> Date: Wed, 4 Nov 2026 17:43:22 +0100 Subject: [PATCH 003/161] M15: Preserve accepted suggestions after the shared collection budget runs out Collector::keep has already validated the stored text. Avoid rechecking identity candidates against an exhausted SuggestSteps budget; validate only changed case and fall back to the accepted original if that validation cannot succeed. Keep the same time/work caps, exclusion rules, parser and ranking. An integrated regression uses the real REP/TRY/affix pipeline and normal budget: it loses the accepted affixed suggestion on parent b5 and retains it with this change, for lower and upper input. No injected state or relaxed limit. Exact-source fmt and strict proof Clippy passed; full local/hosted validation is still being collected. Addresses CodeRabbit4470300723. --- crates/proof/src/affix_suggest.rs | 40 +++++++++++++++++++++++++------ 1 file changed, 33 insertions(+), 7 deletions(-) diff --git a/crates/proof/src/affix_suggest.rs b/crates/proof/src/affix_suggest.rs index 07194c548..b3f07e43c 100644 --- a/crates/proof/src/affix_suggest.rs +++ b/crates/proof/src/affix_suggest.rs @@ -452,13 +452,9 @@ pub(crate) fn suggest_in(d: &AffixDict, word: &str, max: usize, t: &Tuning) -> V let changed: Vec<_> = candidate.split(' ').collect(); original.len() == changed.len() && original.iter().zip(changed).all(|(a, b)| d.can_suggest_cased(a, b)) }; - let t = if eligible(&t) { - t - } else if eligible(&text) { - text.clone() - } else { - continue; - }; + // Collector::keep already accepted text. Collection may have exhausted the shared + // affix budget, so recheck only a changed case and otherwise retain that accepted form. + let t = if t == text || eligible(&t) { t } else { text.clone() }; // Shown as the dictionary's OCONV table writes it (`'` → `’` in French). let t = d.output(&t).into_owned(); if t != word && !out.contains(&t) { @@ -904,4 +900,34 @@ mod tests { assert!(suggest_in(&d, "neer", 4, &english).contains(&"ne'er".to_string())); assert_eq!(suggest_in(&d, "neer", 4, &Tuning::DEFAULT).first().map(String::as_str), Some("ne'er"), "an elision"); } + /// A REP candidate is admitted before costly TRY checks consume the normal shared budget. + /// Finalization must retain that accepted affixed word, including a conservative case fallback. + #[test] + fn accepted_affixed_suggestions_survive_exhausted_collection_budget() { + let input = format!("{}x{}", "a".repeat(20), "a".repeat(19)); + let mut aff = String::from("SET UTF-8\nFLAG num\nTRY "); + aff.extend(('a'..='z').chain((0x100..0x166).filter_map(char::from_u32))); + aff.push_str(&format!("\nREP 1\nREP {input} kots\n")); + // The bounded adversarial rule shape of the existing shared-budget regression. + for f in 1..=1000u32 { + let c = char::from_u32(0x4e00 + f).unwrap(); + aff.push_str(&format!("PFX {f} Y 1\nPFX {f} {c} 0 .\n")); + } + let conts: Vec = (0..286).map(|k| (5000 + k * 7).to_string()).collect(); + for len in 1..=40 { + for k in 0..50 { + let f = 5000 + (len - 1) * 50 + k; + aff.push_str(&format!("SFX {f} Y 1\nSFX {f} 0 {}/{} .\n", "a".repeat(len), conts.join(","))); + } + } + aff.push_str("SFX 9000 Y 1\nSFX 9000 0 s .\n"); + let d = AffixDict::from_hunspell(&aff, "3\nb/1,5000\nab/2\nkot/9000\n", &[]).unwrap(); + assert!(!d.stems().any(|s| s == "kots") && d.can_suggest("kots")); + for word in [input.clone(), input.to_uppercase()] { + let got = suggest_in(&d, &word, 1, &Tuning::DEFAULT); + assert_eq!(got.len(), 1, "accepted affixed candidate lost: {word}: {got:?}"); + assert!(got[0].eq_ignore_ascii_case("kots"), "{word}: {got:?}"); + assert!(d.can_suggest(&got[0])); + } + } } From 55a16b7a724cd7ab3f9873a415fbd6fbcd8b3c96 Mon Sep 17 00:00:00 2001 From: totoz64 <47389794+totoz64@users.noreply.github.com> Date: Wed, 4 Nov 2026 17:56:56 +0100 Subject: [PATCH 004/161] M15: Validate exact suggestion eligibility before admitting candidates Collector::keep now uses the exact-case eligibility guard before insertion, preserving NOSUGGEST, SUBSTANDARD, FORBIDDENWORD and KEEPCASE rules. Finalization can safely retain the admitted original after the collection budget is exhausted, and still validates actual case variants. No fresh budget, parser, ranking or historical fixture change. This completes CodeRabbit4470300723 without the provisional870 exclusion regression. On this exact source: workspace fmt and strict proof Clippy passed; 458 unit tests passed with52 ignored, compatibility2 and French Pack integration1 passed, and the real pinned Dicollecte test passed. The integrated normal-budget regression and all15 suggestion/casing/exclusion tests passed together in the fresh complete execution. Hosted validation of this new SHA is still required. --- crates/proof/src/affix_suggest.rs | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/crates/proof/src/affix_suggest.rs b/crates/proof/src/affix_suggest.rs index b3f07e43c..4d8bfea59 100644 --- a/crates/proof/src/affix_suggest.rs +++ b/crates/proof/src/affix_suggest.rs @@ -248,7 +248,8 @@ impl Collector<'_> { /// Keep a suggestion known to be correct, unless the dictionary says never to offer it. fn keep(&mut self, text: String, cost: u32) { - if !text.split(' ').all(|part| self.d.can_suggest(part)) { + // Reject exact excluded spellings at admission, before later candidates spend the budget. + if !text.split(' ').all(|part| self.d.can_suggest_cased(part, part)) { return; } self.seq += 1; From 88864de95f194ea7b81273a241868fcb94930153 Mon Sep 17 00:00:00 2001 From: David Mann Date: Thu, 12 Nov 2026 16:52:07 +0100 Subject: [PATCH 005/161] fix(flags): the review findings on craft-flags (#1119) - text handed to Flags::from_parts gets the same size limit as a file - the file is read once, at most one byte past the limit - a file named by CRAFT_FLAGS_FILE that does not exist is reported - set() holds a lock file beside the flags file from read to write, so two programs changing flags at once keep both changes - set() keeps a comment after the flag it changes - set() refuses to write a file larger than the reader accepts Refs #1118 --- crates/flags/src/lib.rs | 7 ++++- crates/flags/src/store.rs | 65 +++++++++++++++++++++++++++++---------- crates/flags/src/tests.rs | 59 +++++++++++++++++++++++++++++++++++ 3 files changed, 114 insertions(+), 17 deletions(-) diff --git a/crates/flags/src/lib.rs b/crates/flags/src/lib.rs index c65e9218f..8552bab2c 100644 --- a/crates/flags/src/lib.rs +++ b/crates/flags/src/lib.rs @@ -38,7 +38,7 @@ use std::collections::BTreeMap; use std::path::PathBuf; #[cfg(not(target_arch = "wasm32"))] -mod store; +pub(crate) mod store; #[cfg(not(target_arch = "wasm32"))] pub use store::{FlagError, load, set}; @@ -135,6 +135,11 @@ impl Flags { } fn read_file(&mut self, text: &str) { + // The same limit as a file read from disk, also for text handed in directly. + if text.len() as u64 > MAX_FILE_BYTES { + self.problems.push(format!("the flags file is larger than {MAX_FILE_BYTES} bytes, so every flag is at its default")); + return; + } let doc = match text.parse::() { Ok(d) => d, Err(e) => { diff --git a/crates/flags/src/store.rs b/crates/flags/src/store.rs index a601d8731..6f422aa09 100644 --- a/crates/flags/src/store.rs +++ b/crates/flags/src/store.rs @@ -1,9 +1,10 @@ //! Reading and changing the flags file (desktop only). use std::fmt; -use std::path::Path; +use std::io::Read; +use std::path::{Path, PathBuf}; -use crate::{ENV, Flags, MAX_FILE_BYTES, default_path, valid_id}; +use crate::{ENV, FILE_ENV, Flags, MAX_FILE_BYTES, default_path, valid_id}; /// Why a flag could not be changed. #[derive(Debug)] @@ -32,32 +33,47 @@ impl std::error::Error for FlagError {} /// The file's text, `None` when it does not exist. A file that isn't a regular file, is too big /// or isn't UTF-8 is an error. fn read_text(path: &Path) -> Result, FlagError> { - let meta = match std::fs::metadata(path) { - Ok(m) => m, + let file = match std::fs::File::open(path) { + Ok(f) => f, Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(None), Err(e) => return Err(FlagError::Io(e)), }; + let meta = file.metadata().map_err(FlagError::Io)?; if !meta.is_file() { return Err(FlagError::Unreadable(format!("{} is not a regular file", path.display()))); } - if meta.len() > MAX_FILE_BYTES { + // Read at most one byte past the limit from the file opened: one that grew or was replaced + // after a size check can't bring more. + let mut bytes = Vec::new(); + file.take(MAX_FILE_BYTES + 1).read_to_end(&mut bytes).map_err(FlagError::Io)?; + if bytes.len() as u64 > MAX_FILE_BYTES { return Err(FlagError::Unreadable(format!("{} is larger than {MAX_FILE_BYTES} bytes", path.display()))); } - let bytes = std::fs::read(path).map_err(FlagError::Io)?; String::from_utf8(bytes).map(Some).map_err(|_| FlagError::Unreadable(format!("{} is not UTF-8 text", path.display()))) } /// The flags of this run: [`default_path`] and `CRAFT_FLAGS`. Never fails; what could not be /// read is in [`Flags::problems`]. pub fn load() -> Flags { - let env = std::env::var(ENV).ok(); - let Some(path) = default_path() else { - return Flags::from_parts(None, env.as_deref()); + let named = std::env::var_os(FILE_ENV).is_some_and(|p| !p.is_empty()); + load_from(default_path(), named, std::env::var(ENV).ok().as_deref()) +} + +/// [`load`] from `path`; `named` when `CRAFT_FLAGS_FILE` chose it, so a missing file is worth a +/// word (the usual file not existing yet is not). +pub(crate) fn load_from(path: Option, named: bool, env: Option<&str>) -> Flags { + let Some(path) = path else { + return Flags::from_parts(None, env); }; match read_text(&path) { - Ok(text) => Flags::from_parts(text.as_deref(), env.as_deref()), + Ok(None) if named => { + let mut flags = Flags::from_parts(None, env); + flags.problems.insert(0, format!("{}, named by {FILE_ENV}, does not exist; every flag is at its default", path.display())); + flags + } + Ok(text) => Flags::from_parts(text.as_deref(), env), Err(e) => { - let mut flags = Flags::from_parts(None, env.as_deref()); + let mut flags = Flags::from_parts(None, env); flags.problems.insert(0, format!("{}: {e}; every flag is at its default", path.display())); flags } @@ -65,11 +81,21 @@ pub fn load() -> Flags { } /// Turn the flag `id` on or off in the file at `path` (made if missing). Everything else in the -/// file, comments included, is kept; the write is atomic. +/// file, comments included, is kept; the write is atomic. Two programs changing flags at once +/// (the launcher and an app) take turns, so neither change is lost. pub fn set(path: &Path, id: &str, on: bool) -> Result<(), FlagError> { if !valid_id(id) { return Err(FlagError::BadId(id.to_string())); } + if let Some(dir) = path.parent() { + std::fs::create_dir_all(dir).map_err(FlagError::Io)?; + } + // A lock file beside the flags file, which itself is replaced on every write. Released when + // `_lock` is dropped (and by the system if the process ends). + let mut lock_path = path.as_os_str().to_owned(); + lock_path.push(".lock"); + let _lock = std::fs::OpenOptions::new().create(true).truncate(false).write(true).open(PathBuf::from(lock_path)).map_err(FlagError::Io)?; + _lock.lock().map_err(FlagError::Io)?; let mut doc = match read_text(path)? { Some(text) => text.parse::().map_err(|e| FlagError::Unreadable(e.to_string()))?, None => toml_edit::DocumentMut::new(), @@ -78,9 +104,16 @@ pub fn set(path: &Path, id: &str, on: bool) -> Result<(), FlagError> { let Some(table) = item.as_table_like_mut() else { return Err(FlagError::Unreadable("`flags` is not a table".into())); }; - table.insert(id, toml_edit::value(on)); - if let Some(dir) = path.parent() { - std::fs::create_dir_all(dir).map_err(FlagError::Io)?; + // Keep what surrounds the old value, such as a comment after it. + let mut value = toml_edit::Value::from(on); + if let Some(old) = table.get(id).and_then(toml_edit::Item::as_value) { + *value.decor_mut() = old.decor().clone(); } - craft_atomic::atomic_write(path, doc.to_string().as_bytes()).map_err(FlagError::Io) + table.insert(id, toml_edit::Item::Value(value)); + let text = doc.to_string(); + // What can't be read back is not written: the file stays as it was. + if text.len() as u64 > MAX_FILE_BYTES { + return Err(FlagError::Unreadable(format!("the flags file would grow past {MAX_FILE_BYTES} bytes"))); + } + craft_atomic::atomic_write(path, text.as_bytes()).map_err(FlagError::Io) } diff --git a/crates/flags/src/tests.rs b/crates/flags/src/tests.rs index 3f4bef667..92f690037 100644 --- a/crates/flags/src/tests.rs +++ b/crates/flags/src/tests.rs @@ -48,6 +48,15 @@ fn bad_entries_are_reported_and_left_out_never_fatal() { assert!(!Flags::from_parts(Some("flags = 3"), None).problems.is_empty()); } +#[test] +fn oversized_text_is_not_read() { + let mut text = String::from("[flags]\nAI-MCP = true\n"); + text.push_str(&"#".repeat(MAX_FILE_BYTES as usize)); + let flags = Flags::from_parts(Some(&text), None); + assert!(!flags.enabled(&AI_MCP)); + assert!(flags.problems.iter().any(|p| p.contains("larger than"))); +} + #[test] fn many_entries_are_capped() { let mut text = String::from("[flags]\n"); @@ -79,6 +88,7 @@ fn ids_are_checked() { #[cfg(not(target_arch = "wasm32"))] mod file { use super::super::*; + use crate::store; #[test] fn set_creates_the_file_and_keeps_what_else_is_in_it() { @@ -98,6 +108,55 @@ mod file { assert!(flags.enabled(&AI_MODELS)); } + #[test] + fn set_keeps_the_comment_after_a_flag() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join(FILE_NAME); + std::fs::write(&path, "[flags]\nAI-MCP = true # on for testing\n").unwrap(); + set(&path, "AI-MCP", false).unwrap(); + let text = std::fs::read_to_string(&path).unwrap(); + assert!(text.contains("AI-MCP = false # on for testing"), "{text}"); + } + + #[test] + fn set_refuses_to_write_what_could_not_be_read_back() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join(FILE_NAME); + let mut text = String::from("[flags]\n"); + while (text.len() as u64) < MAX_FILE_BYTES - 4 { + text.push_str("# padding\n"); + } + std::fs::write(&path, &text).unwrap(); + assert!(matches!(set(&path, "AI-MODELS", true), Err(FlagError::Unreadable(_)))); + assert_eq!(std::fs::read_to_string(&path).unwrap(), text, "left as it was"); + } + + #[test] + fn a_missing_file_is_mentioned_only_when_named() { + let dir = tempfile::tempdir().unwrap(); + let missing = dir.path().join("nope.toml"); + let named = store::load_from(Some(missing.clone()), true, None); + assert!(named.problems.iter().any(|p| p.contains("does not exist")), "{:?}", named.problems); + assert!(store::load_from(Some(missing), false, None).problems.is_empty(), "the usual file not existing yet is normal"); + } + + #[test] + fn two_writers_keep_both_changes() { + let dir = tempfile::tempdir().unwrap(); + let path = dir.path().join(FILE_NAME); + let ids: Vec = (0..8).map(|i| format!("F{i}")).collect(); + std::thread::scope(|s| { + for id in &ids { + let path = &path; + s.spawn(move || set(path, id, true).unwrap()); + } + }); + let flags = Flags::from_parts(Some(&std::fs::read_to_string(&path).unwrap()), None); + for id in &ids { + assert!(flags.enabled_id(id), "{id} lost"); + } + } + #[test] fn set_never_overwrites_a_file_it_cannot_read() { let dir = tempfile::tempdir().unwrap(); From 4a3bb60b828feaae94de67e00db3c25040f19ec7 Mon Sep 17 00:00:00 2001 From: David Mann Date: Thu, 12 Nov 2026 17:07:23 +0100 Subject: [PATCH 006/161] test(flags): the oversized-write fixture is readable, so the test reaches the output-size check The padding loop made the fixture 65,538 bytes, past the read limit, so set() refused while reading and the test passed without the output-size check. The fixture is now exactly MAX_FILE_BYTES - 4 bytes; with the check removed the test fails. --- crates/flags/src/tests.rs | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/crates/flags/src/tests.rs b/crates/flags/src/tests.rs index 92f690037..66c00855b 100644 --- a/crates/flags/src/tests.rs +++ b/crates/flags/src/tests.rs @@ -122,12 +122,15 @@ mod file { fn set_refuses_to_write_what_could_not_be_read_back() { let dir = tempfile::tempdir().unwrap(); let path = dir.path().join(FILE_NAME); - let mut text = String::from("[flags]\n"); - while (text.len() as u64) < MAX_FILE_BYTES - 4 { - text.push_str("# padding\n"); - } + // Readable (just under the limit), so the refusal comes from the size of what `set` would + // write, not from reading. + let mut text = String::from("[flags]\n# "); + let pad = MAX_FILE_BYTES as usize - 4 - text.len(); + text.push_str(&"x".repeat(pad)); + assert_eq!(text.len() as u64, MAX_FILE_BYTES - 4); std::fs::write(&path, &text).unwrap(); - assert!(matches!(set(&path, "AI-MODELS", true), Err(FlagError::Unreadable(_)))); + assert!(Flags::from_parts(Some(&text), None).problems.is_empty(), "the fixture itself reads"); + assert!(matches!(set(&path, "AI-MODELS", true), Err(FlagError::Unreadable(e)) if e.contains("would grow"))); assert_eq!(std::fs::read_to_string(&path).unwrap(), text, "left as it was"); } From 1f53b809737c5f3141406387f9aa93c971b4007b Mon Sep 17 00:00:00 2001 From: totoz64 <47389794+totoz64@users.noreply.github.com> Date: Sun, 15 Nov 2026 19:58:06 +0100 Subject: [PATCH 007/161] Proofing: retain eligible exact compound suggestions Address CodeRabbit4575003662 by considering all normalized exact derivations in the displayed-case eligibility guard, without lowercase fallback. Preserve forbidden-word handling, case exclusions, shared budgets and ranking. A real REP regression fails before the fix and passes for NOSUGGEST/SUBSTANDARD compound paths after it, while FORBIDDEN remains rejected; cover WCAFF2 round trips.460 unit tests, compatibility2, French Pack1, real Dicollecte1, formatting and strict all-target proof Clippy pass on Rust1.100. New hosted checks are still required. --- crates/proof/src/affix.rs | 5 +++-- crates/proof/src/affix_suggest.rs | 14 ++++++++++++++ 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/crates/proof/src/affix.rs b/crates/proof/src/affix.rs index 4c76815bb..b0407c59f 100644 --- a/crates/proof/src/affix.rs +++ b/crates/proof/src/affix.rs @@ -2248,11 +2248,12 @@ impl AffixDict { return false; } // The exact simple-word path includes affixes; asking with no exclusions detects a - // target whose only derivations carry NOSUGGEST, SUBSTANDARD or FORBIDDENWORD. Spend + // target that may carry NOSUGGEST, SUBSTANDARD or FORBIDDENWORD. An eligible exact + // compound derivation still counts, but a lowercase fallback must not rescue it. Spend // the same shared suggestion budget as the normal checker, and abstain if it runs out. match self.simple_word_budgeted(&target, Excluded::default(), true) { None => return false, - Some(true) if !self.simple_word(&target, self.excluded(true)) => return false, + Some(true) if !self.contains_except(&target, self.excluded(true)) => return false, _ => {} } self.can_suggest(changed) diff --git a/crates/proof/src/affix_suggest.rs b/crates/proof/src/affix_suggest.rs index 4d8bfea59..2a102c00b 100644 --- a/crates/proof/src/affix_suggest.rs +++ b/crates/proof/src/affix_suggest.rs @@ -822,6 +822,20 @@ mod tests { assert!(!d.can_suggest_cased("normal", "pour")); } + /// An excluded simple entry does not exclude an eligible exact COMPOUNDRULE derivation. + #[test] + fn rep_keeps_eligible_exact_compound_beside_an_excluded_simple_entry() { + for flag in ["N", "S", "F"] { + let aff = "SET UTF-8\nNOSUGGEST N\nSUBSTANDARD S\nFORBIDDENWORD F\nCOMPOUNDRULE ab\nREP 1\nREP zz foobar\n"; + let d = AffixDict::from_hunspell(aff, &format!("3\nfoobar/{flag}\nfoo/a\nbar/b\n"), &[]).unwrap(); + for d in [&d, &AffixDict::from_bytes(&d.to_bytes().unwrap()).unwrap()] { + assert_eq!(d.contains_suggestible("foobar"), flag != "F", "{flag}: exact compound eligibility"); + let got = suggest_in(d, "zz", 4, &Tuning::DEFAULT); + assert_eq!(got.iter().any(|w| w == "foobar"), flag != "F", "{flag}: {got:?}"); + } + } + } + /// A language's tables come after its dictionary's, a TRY alphabet stands in for a missing TRY /// line, and a split rule decides which missing spaces are offered. #[test] From 30e5deb660ee195fe713f2fee973f5764b956339 Mon Sep 17 00:00:00 2001 From: Call4han Date: Mon, 16 Nov 2026 21:19:24 -0500 Subject: [PATCH 008/161] fix(xfl): read ZIPs with overstated central directory sizes --- crates/xfl/src/lib.rs | 13 +++++++++++++ crates/xfl/src/tests.rs | 17 +++++++++++++++++ 2 files changed, 30 insertions(+) diff --git a/crates/xfl/src/lib.rs b/crates/xfl/src/lib.rs index fbcb3ecc2..7283b56a0 100644 --- a/crates/xfl/src/lib.rs +++ b/crates/xfl/src/lib.rs @@ -318,6 +318,19 @@ fn find_end(data: &[u8]) -> Result<(u64, u64, u64, usize), ZipError> { let Some((_, shift)) = locate_directory(data, count.into(), size.into(), offset.into(), candidate) .filter(|&(dir, _)| directory_is_exact(data, dir, size.into(), count.into(), &mut directory_jumps)) else { + // An overstated size is safe to ignore only when the recorded entries fill + // the byte range from the declared offset to this single-disk end record. + // The classic entry count is below MAX_ENTRIES, so the walk checks every entry. + if count != 0 + && u32_at(data, candidate + 4) == Some(0) + && u16_at(data, candidate + 8) == Some(count) + && let Ok(start) = usize::try_from(offset) + && let Some(measured) = candidate.checked_sub(start).and_then(|n| u32::try_from(n).ok()) + && measured < size + && directory_is_exact(data, start, measured.into(), count.into(), &mut directory_jumps) + { + return Ok((count.into(), measured.into(), offset.into(), candidate)); + } continue; }; if count == 0 && shift > 0 { diff --git a/crates/xfl/src/tests.rs b/crates/xfl/src/tests.rs index 5ae0713ba..5611a92ca 100644 --- a/crates/xfl/src/tests.rs +++ b/crates/xfl/src/tests.rs @@ -108,6 +108,23 @@ fn damaged_archives_are_errors() { assert!(Archive::open(&z).is_err()); } +#[test] +fn reads_a_directory_with_an_overstated_size() { + let mut z = zip(&[("DOMDocument.xml", b"", true), ("LIBRARY/ball.xml", b"", false)]); + let end = z.len() - 22; + let size = u32_at(&z, end + 12).unwrap(); + z[end + 12..end + 16].copy_from_slice(&(size + 13).to_le_bytes()); + + let archive = Archive::open(&z).unwrap(); + assert_eq!(archive.counts().0, 2); + assert_eq!(archive.read("DOMDocument.xml").unwrap().unwrap(), b""); + assert_eq!(archive.read("LIBRARY/ball.xml").unwrap().unwrap(), b""); + + // Keep the overstated end record but remove the last byte of the final entry's name. + z.remove(end - 1); + assert_eq!(Archive::open(&z).err(), Some(ZipError::Damaged("the central directory is outside the archive"))); +} + #[test] fn a_truncated_central_directory_is_damaged() { let z = zip(&[("DOMDocument.xml", b"", true), ("b.xml", b"", true), ("c.xml", b"", false)]); From 841e6a726ba4aba563cc0ed9c1465c8345502666 Mon Sep 17 00:00:00 2001 From: Call4han Date: Mon, 16 Nov 2026 21:35:57 -0500 Subject: [PATCH 009/161] fix(xfl): prefer valid ZIP end records over recovered ones --- crates/xfl/src/lib.rs | 13 +++++++++++-- crates/xfl/src/tests.rs | 26 ++++++++++++++++++++++++++ 2 files changed, 37 insertions(+), 2 deletions(-) diff --git a/crates/xfl/src/lib.rs b/crates/xfl/src/lib.rs index 7283b56a0..f89ec10fe 100644 --- a/crates/xfl/src/lib.rs +++ b/crates/xfl/src/lib.rs @@ -279,6 +279,7 @@ fn find_end(data: &[u8]) -> Result<(u64, u64, u64, usize), ZipError> { // it from an empty EOCD-shaped record inside the preceding archive's comment. Keep it as a // last resort while searching for an older, structurally stronger candidate. let mut empty_sfx = None; + let mut measured_directory = None; let mut fallback = None; // Fake EOCD records in a comment can all describe the same large directory. Cache binary // directory jumps so checking each candidate does not rescan every entry. @@ -329,7 +330,9 @@ fn find_end(data: &[u8]) -> Result<(u64, u64, u64, usize), ZipError> { && measured < size && directory_is_exact(data, start, measured.into(), count.into(), &mut directory_jumps) { - return Ok((count.into(), measured.into(), offset.into(), candidate)); + // Keep scanning for a valid declared size. Among measured candidates, + // prefer the earliest record so a copy in its comment cannot hide it. + measured_directory = Some((count.into(), measured.into(), offset.into(), candidate)); } continue; }; @@ -341,7 +344,13 @@ fn find_end(data: &[u8]) -> Result<(u64, u64, u64, usize), ZipError> { eocd = Some(candidate); break; } - let eocd = eocd.or(empty_sfx).or(fallback).ok_or(ZipError::NotZip)?; + let eocd = eocd.or(empty_sfx); + if eocd.is_none() + && let Some(directory) = measured_directory + { + return Ok(directory); + } + let eocd = eocd.or(fallback).ok_or(ZipError::NotZip)?; let field16 = |o: usize| u16_at(data, eocd + o).ok_or(ZipError::Damaged("the end record is cut short")); let field32 = |o: usize| u32_at(data, eocd + o).ok_or(ZipError::Damaged("the end record is cut short")); let (disk, dir_disk, on_disk, count) = (field16(4)?, field16(6)?, field16(8)?, field16(10)?); diff --git a/crates/xfl/src/tests.rs b/crates/xfl/src/tests.rs index 5611a92ca..6075e4e57 100644 --- a/crates/xfl/src/tests.rs +++ b/crates/xfl/src/tests.rs @@ -323,6 +323,32 @@ fn nonempty_comment_eocd_candidate_does_not_mask_valid_archive() { assert_eq!(archive.read("a.xml").unwrap().unwrap(), b"x"); } +#[test] +fn overstated_comment_eocd_does_not_mask_valid_archive() { + let mut bytes = zip(&[("a.xml", b"x", false)]); + let end = bytes.len() - 22; + let size = u32_at(&bytes, end + 12).unwrap(); + let start = u32_at(&bytes, end + 16).unwrap() as usize; + let fake_start = bytes.len(); + + // Copy the directory and end record into the real comment, then corrupt the copies. + bytes.extend_from_within(start..); + let comment_len = u16::try_from(bytes.len() - fake_start).unwrap(); + bytes[end + 20..end + 22].copy_from_slice(&comment_len.to_le_bytes()); + bytes[fake_start + 42..fake_start + 46].copy_from_slice(&1u32.to_le_bytes()); + let fake_end = bytes.len() - 22; + bytes[fake_end + 12..fake_end + 16].copy_from_slice(&(size + 1).to_le_bytes()); + bytes[fake_end + 16..fake_end + 20].copy_from_slice(&u32::try_from(fake_start).unwrap().to_le_bytes()); + + let archive = Archive::open(&bytes).unwrap(); + assert_eq!(archive.read("a.xml").unwrap().unwrap(), b"x"); + + // If both candidates need recovery, the record before the comment still wins. + bytes[end + 12..end + 16].copy_from_slice(&(size + 1).to_le_bytes()); + let archive = Archive::open(&bytes).unwrap(); + assert_eq!(archive.read("a.xml").unwrap().unwrap(), b"x"); +} + #[test] fn repeated_eocd_directory_candidates_reuse_entry_validation() { let entries = (0..1024).map(|index| Raw::new(&format!("entry-{index:04}.xml"), b"", false)).collect::>(); From 976e4ef8ea3315f066bdfe65bdfa6e313caa196c Mon Sep 17 00:00:00 2001 From: dotCooCoo Date: Mon, 16 Nov 2026 19:01:57 -0800 Subject: [PATCH 010/161] craft-csv: rows of empty cells decide the delimiter when nothing else does sniff_delimiter skipped every record without a non-whitespace character other than the delimiter, so a record such as ";" or ";;;" was blank to the semicolon candidate. A semicolon export of a sheet whose only value is far down (B40001) starts with 40,000 ";" rows, which fill the 64 KiB sample, and the file was read as comma-separated with one column. Records that hold only a candidate's delimiters and whitespace are now kept apart, at most 20 per candidate. When no candidate separates the records with other text, the same rules pick among the candidates whose sample holds only such records. Every result that did not fall back to comma stays the same: a row of tabs alone is still blank to the other candidates and does not outweigh them, as #1209 set out. --- crates/csv/README.md | 2 +- crates/csv/src/lib.rs | 122 +++++++++++++++++++++++++++++++++++++----- 2 files changed, 110 insertions(+), 14 deletions(-) diff --git a/crates/csv/README.md b/crates/csv/README.md index bfd525c8c..8fd264d4c 100644 --- a/crates/csv/README.md +++ b/crates/csv/README.md @@ -11,7 +11,7 @@ mapping a SYLK sheet onto a document are the caller's work. | Function | What it does | |---|---| | `decode_text(bytes)` | UTF-8 and UTF-16 byte order marks; without one, valid UTF-8 is kept; other bytes are Windows-1256 when they read as Arabic, else Windows-1252 | -| `sniff_delimiter(text, quote)` | comma, semicolon or tab: the one with the most consistent field count over the first 20 non-blank records in a 64 KiB sample; when no candidate finds coherent complete records with a delimiter in each, a candidate that finds no complete record counts the delimiters of the first record up to the end of the sample, or up to a quoted field still open there, and a quote inside a field that does not start with one makes that record incoherent; quote boundaries must be coherent, and ties prefer tab, comma, then semicolon | +| `sniff_delimiter(text, quote)` | comma, semicolon or tab: the one with the most consistent field count over the first 20 records with other text in a 64 KiB sample; when none separates those records, the candidates whose records hold only their delimiters and whitespace (rows of empty cells) are compared the same way, leaving out any candidate that found a record with other text; when no candidate finds coherent complete records with a delimiter in each, a candidate that finds no complete record with other text counts the delimiters of the first such record up to the end of the sample, or up to a quoted field still open there (after its rows of empty cells, a record without its delimiter is left out), and a quote inside a field that does not start with one makes that record incoherent; quote boundaries must be coherent, and ties prefer tab, comma, then semicolon | | `parse_records(text, delim, quote, max_fields, on_record)` | records of fields, one callback per record (return `false` to stop); CR, LF and CRLF line ends; quoted fields with doubled quotes and line breaks; a quote in the middle of a field is text | | `records(text, delim, quote, max_records, max_fields)` | the same, collected | | `push_field(out, field, delim)`, `field_len(field, delim)` | one field as written, quoted only when it holds the delimiter, a quote or a line break; its exact length, to enforce a size cap before writing | diff --git a/crates/csv/src/lib.rs b/crates/csv/src/lib.rs index 70f6148a6..395ad2152 100644 --- a/crates/csv/src/lib.rs +++ b/crates/csv/src/lib.rs @@ -33,13 +33,17 @@ pub use decode::{decode_text, decode_utf8}; /// Picks comma, semicolon or tab by consistent field counts in the first non-blank records. /// Quotes start only at a candidate field boundary; candidates with incoherent quoted fields /// are discarded when another coherent candidate separates the records. Ties prefer tab, -/// then comma, then semicolon. Comma is the fallback when no candidate separates the text. +/// then comma, then semicolon. When no candidate separates the records that hold other text, +/// the same rules pick among the candidates whose records hold only their delimiters and +/// whitespace, such as an exported row of empty cells; a candidate that finds a record with +/// other text is left out of that pick. Comma is the fallback when neither pick finds one. /// Only the first 64 KiB are read. A record cut off at the end of that sample is left out, /// except when no candidate finds coherent complete records with a delimiter in each. Then a candidate -/// that finds no complete record counts the delimiters read so far in the first record, and that -/// record is incoherent when it has a quote inside a field that does not start with one. When one -/// of those candidates ends the sample inside a quoted field, each of them counts only up to that -/// field's opening quote. +/// that finds no complete record with other text counts the delimiters read so far in the first +/// such record, unless it found rows of its empty cells and that record has none of its +/// delimiters. That record is incoherent when it has a quote inside a field that does not start +/// with one. When one of those candidates ends the sample inside a quoted field, each of them +/// counts only up to that field's opening quote. pub fn sniff_delimiter(text: &str, quote: char) -> u8 { const SAMPLE_BYTES: usize = 64 * 1024; let mut sample_end = text.len().min(SAMPLE_BYTES); @@ -64,7 +68,15 @@ pub fn sniff_delimiter(text: &str, quote: char) -> u8 { (scan.count, scan.record_has_content, scan.record_coherent, scan.record_bare_quote) = (before_cut.count, before_cut.record_has_content, before_cut.record_coherent, before_cut.record_bare_quote); } - if scan.record_has_content { + if scan.delimiter_only_counts.is_empty() { + if scan.record_has_content { + scan.coherent &= scan.record_coherent && !scan.record_bare_quote; + scan.counts.push(scan.count); + } else { + scan.push_delimiter_only_record(); + } + } else if scan.record_has_content && scan.count > 0 { + // After rows of empty cells, a cut record without this delimiter is left out. scan.coherent &= scan.record_coherent && !scan.record_bare_quote; scan.counts.push(scan.count); } @@ -79,6 +91,8 @@ pub fn sniff_delimiter(text: &str, quote: char) -> u8 { } else if scan.record_has_content { scan.coherent &= scan.record_coherent; scan.counts.push(scan.count); + } else { + scan.push_delimiter_only_record(); } } } @@ -86,23 +100,34 @@ pub fn sniff_delimiter(text: &str, quote: char) -> u8 { } fn pick_delimiter(scans: [DelimiterScan; 3]) -> u8 { - let candidates = - scans.map(|scan| (scan.delimiter, scan.counts.iter().copied().min().unwrap_or(0), scan.counts.iter().sum::(), scan.coherent)); + let with_text = scans.each_ref().map(|scan| (scan.delimiter, scan.counts.as_slice(), scan.coherent)); + // Records of delimiters alone have no quotes, so they are coherent. + let delimiters_only = + scans.each_ref().map(|scan| (scan.delimiter, if scan.counts.is_empty() { scan.delimiter_only_counts.as_slice() } else { &[] }, true)); + best_delimiter(with_text).or_else(|| best_delimiter(delimiters_only)).unwrap_or(b',') +} + +fn best_delimiter(candidates: [(u8, &[usize], bool); 3]) -> Option { + let candidates = candidates + .map(|(delimiter, counts, coherent)| (delimiter, counts.iter().copied().min().unwrap_or(0), counts.iter().sum::(), coherent)); let prefer_coherent = candidates.iter().any(|(_, min, _, coherent)| *min > 0 && *coherent); - let mut best = (b',', 0usize, 0usize); + let mut best: Option<(u8, usize, usize)> = None; for (delimiter, min, total, coherent) in candidates { - if min > 0 && (!prefer_coherent || coherent) && (min, total) > (best.1, best.2) { - best = (delimiter, min, total); + if min > 0 && (!prefer_coherent || coherent) && best.is_none_or(|(_, best_min, best_total)| (min, total) > (best_min, best_total)) { + best = Some((delimiter, min, total)); } } - best.0 + best.map(|(delimiter, _, _)| delimiter) } /// One candidate delimiter's pass over the sample in [`sniff_delimiter`]. struct DelimiterScan { delimiter: u8, - /// Delimiter counts of the complete non-blank records, at most 20. + /// Delimiter counts of the complete records that hold other text, at most 20. counts: Vec, + /// Delimiter counts of the complete records that hold only this candidate's delimiters and + /// whitespace, at most 20. They do not count toward the 20 records above. + delimiter_only_counts: Vec, /// Whether every counted record had coherent quoted fields. coherent: bool, reached_record_limit: bool, @@ -117,10 +142,21 @@ struct DelimiterScan { open_quote: Option, } +impl DelimiterScan { + /// Keeps the open record's delimiter count when the record has no other text and at least + /// one delimiter. + fn push_delimiter_only_record(&mut self) { + if !self.record_has_content && self.count > 0 && self.delimiter_only_counts.len() < 20 { + self.delimiter_only_counts.push(self.count); + } + } +} + fn scan_delimiter(sample: &str, delimiter: u8, quote: char) -> DelimiterScan { let mut scan = DelimiterScan { delimiter, counts: Vec::new(), + delimiter_only_counts: Vec::new(), coherent: true, reached_record_limit: false, count: 0, @@ -158,6 +194,8 @@ fn scan_delimiter(sample: &str, delimiter: u8, quote: char) -> DelimiterScan { scan.reached_record_limit = true; break; } + } else { + scan.push_delimiter_only_record(); } scan.count = 0; after_quote = false; @@ -403,6 +441,64 @@ mod tests { assert_eq!(sniff_delimiter(&format!("a,b,c;\"hello,\";d\n{}", "x;y;z\n".repeat(12_000)), '"'), b';'); } + // A sheet whose only value is far down exports rows of empty cells that fill the whole sample. + #[test] + fn delimiter_sniffing_reads_a_sparse_semicolon_export() { + assert_eq!(sniff_delimiter(&format!("{};7\n", ";\n".repeat(40_000)), '"'), b';'); + } + + #[test] + fn delimiter_sniffing_reads_a_sparse_semicolon_export_with_crlf() { + assert_eq!(sniff_delimiter(&format!("{};7\r\n", ";\r\n".repeat(40_000)), '"'), b';'); + } + + #[test] + fn delimiter_sniffing_reads_a_sparse_tab_export() { + assert_eq!(sniff_delimiter(&format!("{}\t7\r\n", "\t\r\n".repeat(40_000)), '"'), b'\t'); + } + + #[test] + fn delimiter_sniffing_reads_a_short_file_of_empty_semicolon_cells() { + assert_eq!(sniff_delimiter(";;\n;;\n", '"'), b';'); + assert_eq!(sniff_delimiter(";;\r\n;;", '"'), b';'); + } + + #[test] + fn delimiter_sniffing_reads_empty_semicolon_cells_padded_with_whitespace() { + assert_eq!(sniff_delimiter(&"; ; \n".repeat(20_000), '"'), b';'); + } + + #[test] + fn delimiter_sniffing_reads_a_first_record_of_empty_cells_longer_than_the_sample() { + assert_eq!(sniff_delimiter(&format!("{}\n1;2\n", ";".repeat(70_000)), '"'), b';'); + } + + #[test] + fn delimiter_sniffing_leaves_out_a_cut_record_after_a_row_of_empty_cells() { + assert_eq!(sniff_delimiter(&format!(";\n{};7\n", "x".repeat(70_000)), '"'), b';'); + } + + #[test] + fn delimiter_sniffing_counts_a_long_record_after_a_row_of_empty_cells() { + // Delimiters inside the long record's quoted field are text for every candidate. + assert_eq!(sniff_delimiter(&format!("\t\nname\t\"{}\"\n", "x,y".repeat(30_000)), '"'), b'\t'); + assert_eq!(sniff_delimiter(&format!(";\nname;\"{}\"\n", "x,y\t".repeat(20_000)), '"'), b';'); + } + + #[test] + fn delimiter_sniffing_lets_records_with_text_outweigh_records_of_delimiters_alone() { + // Rows of tabs alone are blank to the other candidates and do not outweigh them. + assert_eq!(sniff_delimiter(&format!("{}a;b\nc;d\n", "\t\n".repeat(25)), '"'), b';'); + assert_eq!(sniff_delimiter(&format!("{}a;b\nc;d\n", "\t\t\n".repeat(25)), '"'), b';'); + // The same holds for a record with text cut off at the end of the sample. + assert_eq!(sniff_delimiter(&format!("\t\n{}", "x;".repeat(40_000)), '"'), b';'); + // A candidate with a record of other text is judged on those records alone. + assert_eq!(sniff_delimiter(&format!("title\n{}", ";\n".repeat(25)), '"'), b','); + // Rows of empty cells in a comma file; comma is also the fallback. + assert_eq!(sniff_delimiter("a,b,c,d\n,,,\n,,,\n1,2,3,4\n", '"'), b','); + assert_eq!(sniff_delimiter(&format!("{}x;y\n", ",,,\n".repeat(20_000)), '"'), b','); + } + #[test] fn parser_stops_collecting_fields_at_the_cap() { let (max, extra) = (16_384usize, 50); From 8d3cd561a42420ae345c799719debd79ef3ae745 Mon Sep 17 00:00:00 2001 From: el-j Date: Tue, 17 Nov 2026 10:07:43 +0100 Subject: [PATCH 011/161] craft-video: Motion JPEG frames convert to VideoFrame (#1231) --- crates/video/README.md | 3 +- crates/video/src/frame/convert.rs | 43 +++++++++++++++++++++++++++ crates/video/src/frame/tests.rs | 49 +++++++++++++++++++++++++++++++ 3 files changed, 94 insertions(+), 1 deletion(-) diff --git a/crates/video/README.md b/crates/video/README.md index f9db3a6e6..562a63eb3 100644 --- a/crates/video/README.md +++ b/crates/video/README.md @@ -41,7 +41,8 @@ comes with its codec's feature): - Planes are `[Y, Cb, Cr]`, `[R, G, B]` or `[Y]` (`ColorFamily`), plus an optional alpha plane. DNx, VP9 and AV1 (identity matrix) store RGB as G, B, R; the conversion reorders them. - Samples keep the decoder's storage, `u8` or `u16`, with a bit depth per plane (HEVC chroma may - differ from luma). Converting moves the buffers and copies no samples. + differ from luma). Converting moves the buffers and copies no samples, except Motion JPEG, + whose decoder returns interleaved samples: they are split into planes. - Chroma subsampling is a log2 shift per axis, 0 to 3 (FFV1 codes up to 3). - Colour is given as ITU-T H.273 code points (`ColorTags`), mapped from each codec's own signalling. Codecs that do not signal a value leave it unspecified (2). diff --git a/crates/video/src/frame/convert.rs b/crates/video/src/frame/convert.rs index 4089f7148..054d09204 100644 --- a/crates/video/src/frame/convert.rs +++ b/crates/video/src/frame/convert.rs @@ -178,6 +178,49 @@ impl TryFrom for VideoFrame { } } +#[cfg(feature = "mjpeg")] +impl TryFrom for VideoFrame { + type Error = FrameError; + + /// A gray or RGB frame, full range (a JPEG's YCbCr is converted to RGB by the decoder, so RGB + /// frames carry the identity matrix). 8-bit pictures keep 8-bit storage; 12-bit ones are + /// stored scaled to 16 bits and say so. The decoder returns interleaved samples, so this is + /// the one conversion that copies them, into planes. + fn try_from(p: crate::mjpeg::Frame) -> Result { + use crate::mjpeg::Samples as S; + let (w, h) = (p.width, p.height); + let channels = usize::from(p.channels); + let family = match channels { + 1 => ColorFamily::Gray, + 3 => ColorFamily::Rgb, + _ => return Err(FrameError::Shape("a Motion JPEG frame has 1 or 3 channels")), + }; + let pixels = usize::try_from(w) + .ok() + .zip(usize::try_from(h).ok()) + .and_then(|(w, h)| w.checked_mul(h)) + .ok_or(FrameError::Plane("plane size overflows"))?; + let stored = match &p.samples { + S::U8(v) => v.len(), + S::U16(v) => v.len(), + }; + if pixels.checked_mul(channels) != Some(stored) { + return Err(FrameError::Shape("sample count does not fit the frame")); + } + let planes = match p.samples { + S::U8(v) => (0..channels) + .map(|c| Plane::packed(w, h, 8, Samples::U8(v.iter().skip(c).step_by(channels).copied().collect()))) + .collect::, _>>()?, + S::U16(v) => (0..channels) + .map(|c| Plane::packed(w, h, 16, Samples::U16(v.iter().skip(c).step_by(channels).copied().collect()))) + .collect::, _>>()?, + }; + let mut f = VideoFrame::new(w, h, family, Subsampling::S444, planes, None)?; + f.color = ColorTags { matrix: if family == ColorFamily::Rgb { 0 } else { 2 }, full_range: true, ..ColorTags::UNSPECIFIED }; + Ok(f) + } +} + #[cfg(feature = "mpeg2-decode")] impl TryFrom for VideoFrame { type Error = FrameError; diff --git a/crates/video/src/frame/tests.rs b/crates/video/src/frame/tests.rs index cba664af4..0a5e4c7bf 100644 --- a/crates/video/src/frame/tests.rs +++ b/crates/video/src/frame/tests.rs @@ -226,6 +226,55 @@ fn vp9_color_space_maps_to_a_matrix() { assert_eq!([0, 1, 2, 3, 4, 5, 6, 7].map(convert::vp9_matrix), [2, 5, 1, 6, 7, 9, 2, 0]); } +#[cfg(feature = "mjpeg")] +#[test] +fn mjpeg_frames_split_into_planes() { + use crate::mjpeg::{Frame as M, Samples as S}; + // 2x2 RGB: pixels (1,2,3) (4,5,6) (7,8,9) (10,11,12). + let rgb = M { width: 2, height: 2, channels: 3, samples: S::U8((1..=12).collect()), damaged: false }; + let f = VideoFrame::try_from(rgb).unwrap(); + assert_eq!((f.family(), f.subsampling(), f.bit_depth()), (ColorFamily::Rgb, Subsampling::S444, 8)); + assert_eq!(f.color, ColorTags { matrix: 0, full_range: true, ..ColorTags::UNSPECIFIED }); + let plane = |i: usize| f.plane(i).unwrap().samples().clone(); + assert_eq!(plane(0), Samples::U8(vec![1, 4, 7, 10])); + assert_eq!(plane(1), Samples::U8(vec![2, 5, 8, 11])); + assert_eq!(plane(2), Samples::U8(vec![3, 6, 9, 12])); + assert!(f.alpha().is_none() && f.pts.is_none()); + // A 12-bit gray frame is stored at 16 bits. + let gray = M { width: 3, height: 1, channels: 1, samples: S::U16(vec![0, 32768, 65535]), damaged: false }; + let f = VideoFrame::try_from(gray).unwrap(); + assert_eq!((f.family(), f.bit_depth(), f.planes().len()), (ColorFamily::Gray, 16, 1)); + assert_eq!(f.plane(0).unwrap().samples(), &Samples::U16(vec![0, 32768, 65535])); +} + +#[cfg(feature = "mjpeg")] +#[test] +fn mjpeg_frames_that_do_not_fit_are_errors() { + use crate::mjpeg::{Frame as M, Samples as S}; + let bad = |channels: u8, n: usize| M { width: 2, height: 2, channels, samples: S::U8(vec![0; n]), damaged: false }; + assert!(VideoFrame::try_from(bad(2, 8)).is_err(), "two channels is not a Motion JPEG frame"); + assert!(VideoFrame::try_from(bad(3, 11)).is_err(), "a short buffer"); + assert!(VideoFrame::try_from(bad(1, 5)).is_err(), "a long buffer"); + assert!(VideoFrame::try_from(M { width: u32::MAX, height: u32::MAX, channels: 3, samples: S::U8(Vec::new()), damaged: false }).is_err()); +} + +#[cfg(feature = "mjpeg")] +#[test] +fn a_decoded_mjpeg_picture_converts() { + use crate::mjpeg::{Decoder, Encoder, EncoderConfig}; + let (w, h) = (24usize, 16usize); + let px: Vec = (0..w * h * 3).map(|i| (i * 7 % 251) as u8).collect(); + let packet = Encoder::new(EncoderConfig::default()).unwrap().encode(w as u32, h as u32, 3, &px).unwrap(); + let decoded = Decoder::new().decode(&packet).unwrap(); + let want_r: Vec = match &decoded.samples { + crate::mjpeg::Samples::U8(v) => v.iter().step_by(3).copied().collect(), + crate::mjpeg::Samples::U16(_) => panic!("an 8-bit picture"), + }; + let f = VideoFrame::try_from(decoded).unwrap(); + assert_eq!((f.width(), f.height(), f.family()), (24, 16, ColorFamily::Rgb)); + assert_eq!(f.plane(0).unwrap().samples(), &Samples::U8(want_r)); +} + #[cfg(feature = "mpeg2-decode")] #[test] fn mpeg2_picture_converts() { From f32b47a1d4799269b7e4b99bb130161dcb7b312a Mon Sep 17 00:00:00 2001 From: Terra854 Date: Tue, 17 Nov 2026 19:00:14 +0800 Subject: [PATCH 012/161] Add opt-in checked SVG image validation --- Cargo.lock | 4 + crates/svg/Cargo.toml | 13 +- crates/svg/README.md | 55 +++ crates/svg/src/tree.rs | 23 +- crates/svg/src/tree/draw.rs | 8 + crates/svg/src/tree/draw/decoder_edges.rs | 127 ++++++ crates/svg/src/tree/draw/raster.rs | 140 +++++++ crates/svg/src/tree/image_href.rs | 428 +++++++++++++++++++++ crates/svg/src/tree/mime_header_tests.rs | 158 ++++++++ crates/svg/src/tree/validated.rs | 393 +++++++++++++++++++ crates/svg/src/tree/validated/tests.rs | 340 ++++++++++++++++ crates/svg/src/tree/validated/xml_tests.rs | 176 +++++++++ 12 files changed, 1861 insertions(+), 4 deletions(-) create mode 100644 crates/svg/src/tree/draw/decoder_edges.rs create mode 100644 crates/svg/src/tree/draw/raster.rs create mode 100644 crates/svg/src/tree/image_href.rs create mode 100644 crates/svg/src/tree/mime_header_tests.rs create mode 100644 crates/svg/src/tree/validated.rs create mode 100644 crates/svg/src/tree/validated/tests.rs create mode 100644 crates/svg/src/tree/validated/xml_tests.rs diff --git a/Cargo.lock b/Cargo.lock index a9290398b..4da6cf6c9 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3092,14 +3092,18 @@ dependencies = [ "craft-xml", "data-url", "flate2", + "gif", + "image-webp", "kurbo 0.13.1", "quick-xml 0.41.0", "resvg 0.48.1", "roxmltree 0.21.2", "thiserror 2.0.21", + "toml", "usvg 0.48.1", "vectorcraft-geom", "vectorcraft-pathops", + "zune-jpeg", ] [[package]] diff --git a/crates/svg/Cargo.toml b/crates/svg/Cargo.toml index b35c4de41..4b7e928a8 100644 --- a/crates/svg/Cargo.toml +++ b/crates/svg/Cargo.toml @@ -31,6 +31,13 @@ usvg = { version = "=0.48.1", default-features = false, optional = true } resvg = { version = "=0.48.1", default-features = false, features = ["text", "raster-images"], optional = true } data-url = { version = "0.3.2", optional = true } +# Renderer-specific validation uses the same resolved backends as resvg0.48.1. +# zune-jpeg is used under its MIT licence option. The locked edge-alignment +# regression must pass when changing this renderer/backend dependency group. +zune-jpeg = { version = "=0.5.15", optional = true } +gif = { version = "=0.14.2", optional = true } +image-webp = { version = "=0.2.4", optional = true } + [features] default = ["paths"] @@ -45,5 +52,9 @@ tree-text = ["tree", "usvg/text"] # The system's fonts in `usvg::fontdb`. tree-system-fonts = ["tree-text", "usvg/system-fonts"] # Bounded rendering of a parsed tree through the matching resvg/tiny-skia versions. -tree-render = ["tree", "dep:resvg"] +tree-render = ["tree", "dep:resvg", "dep:zune-jpeg", "dep:gif", "dep:image-webp"] tree-render-system-fonts = ["tree-render", "resvg/system-fonts"] + +[dev-dependencies] +# Already resolved in the workspace; test-only Cargo.lock edge inspection. +toml = "=1.1.8" diff --git a/crates/svg/README.md b/crates/svg/README.md index 508081367..0810b1217 100644 --- a/crates/svg/README.md +++ b/crates/svg/README.md @@ -16,3 +16,58 @@ The shared adapter adds an aggregate 64 MiB stylesheet-text limit and idempotent ### Transform compatibility `transform_compat::parse_transform` preserves EffectCraft transform-list recovery and exact composition. It retains six numeric arguments per operation; scanning is not time/byte capped and matrix overflow behavior is unchanged. The strict geometry reader is unchanged. Source: Brandon Thomas, EffectCraft94e1c86d89a866771f80ad1610afda9aa12e85d8; bounded safe argument matching by Altynzhan Karybai,4035d181329e847d7a57bf3d059502ea2c91b560. + + +### Opt-in validated embedded images + +With `tree-render`, `tree::ValidatedImageResolver` owns one drawing's immutable +`Limits`, original child-source work, data-URL normalization and renderer-specific +raster validation. Existing `ImageResolver::new`/`resolver` and legacy parse entry +points retain their behavior. Opt in explicitly: + +```rust +# #[cfg(feature = "tree-render")] +# fn example() -> Result<(), craft_svg::tree::Error> { +use craft_svg::tree::{self, Limits, ValidatedImageResolver, usvg}; +let images = ValidatedImageResolver::new(Limits::default()); +let checked = tree::preflight( + "", + &images.limits(), +)?; +let drawing = images.parse_preflight(checked, usvg::Options::default())?; +let usage = images.usage(); +// Importers disclose usage's invalid_* and *_limit_images; strict consumers refuse them. +# let _ = (drawing, usage); +# Ok(()) +# } +``` + +The caller supplies fonts and a sufficient stack/worker for its depth policy. +`parse_preflight` accepts only an opaque preflight made with exactly this context's +limits. It consumes options and installs its own data-only resolver, including for +all children; unrelated callbacks cannot fetch a linked file or bypass admission. +The full expanded root height and each child's expanded height share one depth +allowance. An overlapping root conversion on the same context is refused. + +Unchanged XML converts from its original checked DOM. A modified data URL requires +one checked replacement DOM. Invalid URLs become bounded resolver-visible image +failures; hidden, unused and foreign elements do not acquire omission counts. +Original UTF-8 child bytes are charged before XML conversion or shorter repairs, +with per-document and aggregate bounds. Failed inflation/text work is not refunded; +separate bounded retry-classification work never admits another child. Encoded +raster callbacks spend at most `limits.bytes + limits.embedded_bytes` cumulatively, +without retaining their sources. Pixel work and decoder allocation are admitted +before decoding; attempted failures keep their charge. An oversized unattempted +image does not prevent a later smaller image from fitting. + +`resolve_text` accepts an already-decoded child SVG through the same admission +context, rather than an unchecked DOM or a precharged-budget override. `usage()` +reports omissions and work after conversion. The context is for one drawing, not a +global rendering cache. + +The renderer has no public decoded-pixel or decode-result hook. Validation therefore +still decodes raster pixels before later renderer painting. The shared validator +owns the backend dependencies; locked dependency-edge tests in both the shared +workspace and consumers must keep them aligned with the selected resvg/usvg graph. +This is a fail-fast maintenance check, not automatic access to private renderer +code or a decode-once performance claim. Direct zune-jpeg use selects its MIT option. diff --git a/crates/svg/src/tree.rs b/crates/svg/src/tree.rs index ab04a3002..80feec75d 100644 --- a/crates/svg/src/tree.rs +++ b/crates/svg/src/tree.rs @@ -34,7 +34,13 @@ use crate::{guard, svgz}; pub mod document; pub mod draw; +#[cfg(feature = "tree-render")] +mod image_href; pub mod primitive; +#[cfg(feature = "tree-render")] +mod validated; +#[cfg(feature = "tree-render")] +pub use validated::ValidatedImageResolver; /// What one import may use. The defaults suit a desktop app that parses on a thread with a /// large stack (see the module text). @@ -161,6 +167,7 @@ pub fn root_unitless_size(svg: &str, limits: &Limits) -> Result { + limits: Limits, xml: roxmltree::Document<'s>, depth: usize, /// [`height`]: the nesting usvg converts, references followed. @@ -219,6 +226,8 @@ const UNSUPPORTED: &[&str] = &[ /// Checks already-decoded SVG text once, retaining the parsed XML for both depth policy and tree conversion. pub fn preflight<'s>(svg: &'s str, limits: &Limits) -> Result, Error> { + #[cfg(test)] + PREFLIGHT_PASSES.with(|count| count.set(count.get().saturating_add(1))); let preflight_limits = crate::preflight::Limits { bytes: limits.bytes, depth: limits.depth, @@ -239,7 +248,7 @@ pub fn preflight<'s>(svg: &'s str, limits: &Limits) -> Result, Err let Some(height) = height(&xml, limits.tree_depth)? else { return Err(Error::TooDeep(limits.tree_depth)); }; - Ok(Preflight { xml, depth, height }) + Ok(Preflight { limits: *limits, xml, depth, height }) } /// Dimensions of an SVG viewport in SVG user units. @@ -432,9 +441,12 @@ struct Used { /// What resolving one drawing's embedded images used or refused. #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] pub struct ImageUsage { - /// Unique decoded raster pixels reserved before decode. + /// Decoded raster pixel work reserved before decode. The legacy resolver + /// shares a charge across repeated data; the validated context charges each + /// attempted decode, including failures. pub raster_pixels: u64, - /// Unpacked SVG bytes consumed by embedded SVG data URLs. + /// Decoded SVG text work charged to embedded data URLs, including the + /// remaining allowance consumed by a failed inflation attempt. pub embedded_svg_bytes: usize, /// Embedded SVG images parsed, including nested images. pub embedded_svg_count: usize, @@ -730,3 +742,8 @@ fn image_resolver_for(limits: Limits, used: Arc) -> usvg::ImageHrefResolve #[cfg(test)] mod tests; + +#[cfg(test)] +thread_local! { + static PREFLIGHT_PASSES: std::cell::Cell = const { std::cell::Cell::new(0) }; +} diff --git a/crates/svg/src/tree/draw.rs b/crates/svg/src/tree/draw.rs index f83653284..8a8bb1a7a 100644 --- a/crates/svg/src/tree/draw.rs +++ b/crates/svg/src/tree/draw.rs @@ -29,6 +29,11 @@ pub use resvg; #[cfg(feature = "tree-render")] pub use resvg::tiny_skia; +#[cfg(feature = "tree-render")] +mod raster; +#[cfg(feature = "tree-render")] +pub use raster::{RasterValidationFailure, validate_raster}; + /// Filter primitives per filter. A result of every primitive is kept; real filters have 1 to 12 /// (Inkscape's drop shadow has 5), and 64 leaves headroom for generated ones. const MAX_PRIMITIVES: usize = 64; @@ -546,3 +551,6 @@ fn deeper(cx: Cx) -> Result<(), Refusal> { } Ok(()) } + +#[cfg(all(test, feature = "tree-render"))] +mod decoder_edges; diff --git a/crates/svg/src/tree/draw/decoder_edges.rs b/crates/svg/src/tree/draw/decoder_edges.rs new file mode 100644 index 000000000..8258b5792 --- /dev/null +++ b/crates/svg/src/tree/draw/decoder_edges.rs @@ -0,0 +1,127 @@ +//! Test-only resolved-edge checks. No Cargo subprocess or global version count. +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)] + +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct PackageId { + name: String, + version: String, + source: Option, +} +struct Package { + id: PackageId, + dependencies: Vec, +} +pub(crate) struct LockGraph { + packages: Vec, +} +impl LockGraph { + pub(crate) fn parse(text: &str) -> Self { + let lock: toml::Value = text.parse().unwrap(); + let packages = lock + .get("package") + .unwrap() + .as_array() + .unwrap() + .iter() + .map(|package| { + let string = |key| package.get(key).unwrap().as_str().unwrap().to_owned(); + Package { + id: PackageId { + name: string("name"), + version: string("version"), + source: package.get("source").map(|v| v.as_str().unwrap().to_owned()), + }, + dependencies: package + .get("dependencies") + .map(|d| d.as_array().unwrap().iter().map(|v| v.as_str().unwrap().to_owned()).collect()) + .unwrap_or_default(), + } + }) + .collect(); + Self { packages } + } + pub(crate) fn workspace_package(&self, name: &str) -> PackageId { + let found: Vec<_> = self.packages.iter().filter(|p| p.id.name == name && p.id.source.is_none()).collect(); + assert_eq!(found.len(), 1, "one exact workspace package required: {name}"); + found[0].id.clone() + } + pub(crate) fn dependency(&self, from: &PackageId, name: &str) -> PackageId { + let package = self.packages.iter().find(|p| &p.id == from).unwrap(); + let edges: Vec<_> = package.dependencies.iter().filter(|edge| edge.split_whitespace().next() == Some(name)).collect(); + assert_eq!(edges.len(), 1, "one {name} dependency edge required from {from:?}"); + let fields: Vec<_> = edges[0].split_whitespace().collect(); + let version = fields.get(1).copied(); + let source = fields.get(2).map(|s| s.trim_start_matches('(').trim_end_matches(')')); + let found: Vec<_> = self + .packages + .iter() + .filter(|p| p.id.name == name && version.is_none_or(|v| p.id.version == v) && source.is_none_or(|s| p.id.source.as_deref() == Some(s))) + .collect(); + assert_eq!(found.len(), 1, "ambiguous or absent resolved edge {} from {from:?}", edges[0]); + found[0].id.clone() + } +} + +pub(crate) fn assert_renderer_edges(graph: &LockGraph, svg: &PackageId) { + let renderer = graph.dependency(svg, "resvg"); + assert_eq!(graph.dependency(svg, "usvg"), graph.dependency(&renderer, "usvg"), "shared and renderer trees must be the same resolved package"); + for codec in ["gif", "image-webp", "zune-jpeg"] { + assert_eq!(graph.dependency(svg, codec), graph.dependency(&renderer, codec), "validator and selected renderer differ for {codec}"); + } +} + +#[test] +fn validated_renderer_dependency_edges_are_the_same_resolved_packages() { + let file = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../Cargo.lock"); + let graph = LockGraph::parse(&std::fs::read_to_string(file).unwrap()); + assert_renderer_edges(&graph, &graph.workspace_package("craft-svg")); +} + +#[test] +fn dependency_lookup_follows_the_selected_edge_and_rejects_ambiguity() { + let graph = LockGraph::parse( + r#" +[[package]] +name = "consumer" +version = "1" +dependencies = ["codec 1 (registry+one)"] +[[package]] +name = "codec" +version = "1" +source = "registry+one" +[[package]] +name = "codec" +version = "1" +source = "registry+two" +[[package]] +name = "other" +version = "1" +dependencies = ["codec"] +"#, + ); + let selected = graph.dependency(&graph.workspace_package("consumer"), "codec"); + assert_eq!(selected.source.as_deref(), Some("registry+one")); + assert!(std::panic::catch_unwind(|| graph.dependency(&graph.workspace_package("other"), "codec")).is_err()); +} + +#[test] +fn selected_renderer_graph_rejects_a_split_validator_backend() { + use std::fmt::Write; + let make = |split: bool| { + let mut text = String::from("[[package]]\nname='craft-svg'\nversion='1'\ndependencies=['resvg', 'usvg', 'gif', 'image-webp', '"); + text.push_str(if split { "zune-jpeg 2" } else { "zune-jpeg 1" }); + text.push_str( + "']\n[[package]]\nname='resvg'\nversion='1'\nsource='registry+one'\ndependencies=['usvg', 'gif', 'image-webp', 'zune-jpeg 1']\n", + ); + for name in ["usvg", "gif", "image-webp", "zune-jpeg"] { + write!(&mut text, "[[package]]\nname='{name}'\nversion='1'\nsource='registry+one'\n").unwrap(); + } + // An unrelated second codec version is valid; only the selected edges count. + text.push_str("[[package]]\nname='zune-jpeg'\nversion='2'\nsource='registry+one'\n"); + LockGraph::parse(&text) + }; + let matching = make(false); + assert_renderer_edges(&matching, &matching.workspace_package("craft-svg")); + let split = make(true); + assert!(std::panic::catch_unwind(|| assert_renderer_edges(&split, &split.workspace_package("craft-svg"))).is_err()); +} diff --git a/crates/svg/src/tree/draw/raster.rs b/crates/svg/src/tree/draw/raster.rs new file mode 100644 index 000000000..fe250259d --- /dev/null +++ b/crates/svg/src/tree/draw/raster.rs @@ -0,0 +1,140 @@ +//! Validate optional raster images with the same backends used by the SVG renderer. +//! +//! Header probing alone cannot detect damaged pixel data. Admit dimensions and +//! reserve work before decoding: even a failed decode keeps its charge. + +use std::io::Cursor; +use std::num::NonZeroU64; + +use crate::tree::{draw::tiny_skia, usvg}; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum RasterValidationFailure { + /// The renderer cannot decode the encoded image. + Invalid, + /// Header dimensions, cumulative work or decoder allocation exceed admission. + Limit, +} + +/// Validate the renderer's first frame within per-image and remaining work limits. +/// Failed attempted decodes keep their pixel charge. No decoded pixels are retained. SVG +/// children use the same wrapped resolver and need no second recursive walk. +pub fn validate_raster(kind: &usvg::ImageKind, max_pixels: u64, pixels_left: &mut u64) -> Result<(), RasterValidationFailure> { + let data = match kind { + usvg::ImageKind::PNG(data) | usvg::ImageKind::JPEG(data) | usvg::ImageKind::GIF(data) | usvg::ImageKind::WEBP(data) => data, + usvg::ImageKind::SVG(_) => return Ok(()), + }; + // The allocation probe includes oversized GIF/WebP frames, not merely their + // canvas. Reserve before asking any rendering decoder to allocate pixels. + let (width, height) = craft_image_codecs::probe_allocation_size(data).map_err(|_| RasterValidationFailure::Invalid)?; + let pixels = area(width, height)?; + if pixels > max_pixels { + return Err(RasterValidationFailure::Limit); + } + *pixels_left = pixels_left.checked_sub(pixels).ok_or(RasterValidationFailure::Limit)?; + match kind { + usvg::ImageKind::PNG(data) => { + let decoded = tiny_skia::Pixmap::decode_png(data).map_err(|_| RasterValidationFailure::Invalid)?; + rgba_bytes(decoded.width(), decoded.height(), pixels)?; + Ok(()) + } + usvg::ImageKind::JPEG(data) => jpeg(data, pixels), + usvg::ImageKind::GIF(data) => gif(data, pixels), + usvg::ImageKind::WEBP(data) => webp(data, pixels), + usvg::ImageKind::SVG(_) => Ok(()), + } +} + +fn area(width: u32, height: u32) -> Result { + if width == 0 || height == 0 { + return Err(RasterValidationFailure::Invalid); + } + u64::from(width).checked_mul(u64::from(height)).ok_or(RasterValidationFailure::Limit) +} + +/// Decoder headers must not request more output than the allocation probe paid for. +fn rgba_bytes(width: u32, height: u32, admitted: u64) -> Result { + let pixels = area(width, height)?; + if pixels > admitted { + return Err(RasterValidationFailure::Limit); + } + usize::try_from(pixels.checked_mul(4).ok_or(RasterValidationFailure::Limit)?).map_err(|_| RasterValidationFailure::Limit) +} + +fn buffer(length: usize) -> Result, RasterValidationFailure> { + let mut output = Vec::new(); + output.try_reserve_exact(length).map_err(|_| RasterValidationFailure::Limit)?; + output.resize(length, 0); + Ok(output) +} + +fn jpeg(data: &[u8], admitted: u64) -> Result<(), RasterValidationFailure> { + use zune_jpeg::zune_core::colorspace::ColorSpace; + use zune_jpeg::zune_core::options::DecoderOptions; + + // Preserve the rendering decoder's default limits and RGBA conversion; a + // general photo decoder may accept formats the SVG renderer cannot draw. + let options = DecoderOptions::default().jpeg_set_out_colorspace(ColorSpace::RGBA); + let mut decoder = zune_jpeg::JpegDecoder::new_with_options(Cursor::new(data), options); + decoder.decode_headers().map_err(|_| RasterValidationFailure::Invalid)?; + let info = decoder.info().ok_or(RasterValidationFailure::Invalid)?; + let length = rgba_bytes(u32::from(info.width), u32::from(info.height), admitted)?; + if decoder.output_colorspace() != Some(ColorSpace::RGBA) || decoder.output_buffer_size() != Some(length) { + return Err(RasterValidationFailure::Invalid); + } + let mut output = buffer(length)?; + decoder.decode_into(&mut output).map_err(|_| RasterValidationFailure::Invalid) +} + +fn gif(data: &[u8], admitted: u64) -> Result<(), RasterValidationFailure> { + let bytes = admitted.checked_mul(4).ok_or(RasterValidationFailure::Limit)?; + // Do not loosen the renderer's default 50 MB GIF frame limit. Smaller + // imports also keep their own admitted allocation ceiling during decode. + let limit = NonZeroU64::new(bytes.min(50_000_000)).ok_or(RasterValidationFailure::Invalid)?; + let mut options = gif::DecodeOptions::new(); + options.set_color_output(gif::ColorOutput::RGBA); + options.set_memory_limit(gif::MemoryLimit::Bytes(limit)); + let mut decoder = options.read_info(data).map_err(gif_error)?; + let frame = decoder.read_next_frame().map_err(gif_error)?.ok_or(RasterValidationFailure::Invalid)?; + let expected = rgba_bytes(u32::from(frame.width), u32::from(frame.height), admitted)?; + if frame.buffer.len() != expected { + return Err(RasterValidationFailure::Invalid); + } + Ok(()) +} + +fn gif_error(error: gif::DecodingError) -> RasterValidationFailure { + match error { + gif::DecodingError::MemoryLimit | gif::DecodingError::OutOfMemory => RasterValidationFailure::Limit, + _ => RasterValidationFailure::Invalid, + } +} + +fn webp(data: &[u8], admitted: u64) -> Result<(), RasterValidationFailure> { + let mut decoder = image_webp::WebPDecoder::new(Cursor::new(data)).map_err(webp_error)?; + let (width, height) = decoder.dimensions(); + let rgba = rgba_bytes(width, height, admitted)?; + let channels = if decoder.has_alpha() { 4 } else { 3 }; + let expected = usize::try_from(area(width, height)?.checked_mul(channels).ok_or(RasterValidationFailure::Limit)?) + .map_err(|_| RasterValidationFailure::Limit)?; + if expected > rgba { + return Err(RasterValidationFailure::Limit); + } + if decoder.output_buffer_size() != Some(expected) { + return Err(RasterValidationFailure::Invalid); + } + // A small valid image can have a compressed chunk larger than its pixels. + // Bound decoder scratch without rejecting that ordinary encoding overhead. + let admitted_bytes = + usize::try_from(admitted.checked_mul(4).ok_or(RasterValidationFailure::Limit)?).map_err(|_| RasterValidationFailure::Limit)?; + decoder.set_memory_limit(admitted_bytes.max(data.len())); + let mut output = buffer(expected)?; + decoder.read_image(&mut output).map_err(webp_error) +} + +fn webp_error(error: image_webp::DecodingError) -> RasterValidationFailure { + match error { + image_webp::DecodingError::MemoryLimitExceeded | image_webp::DecodingError::ImageTooLarge => RasterValidationFailure::Limit, + _ => RasterValidationFailure::Invalid, + } +} diff --git a/crates/svg/src/tree/image_href.rs b/crates/svg/src/tree/image_href.rs new file mode 100644 index 000000000..d33388846 --- /dev/null +++ b/crates/svg/src/tree/image_href.rs @@ -0,0 +1,428 @@ +//! Bounded data-URL normalization over the shared checked XML document. +//! Only attribute values change; actual usvg callbacks decide image usage. +use super::Error; +#[cfg(test)] +use super::Limits; +use std::borrow::Cow; +use std::ops::Range; + +#[cfg(test)] +const SVG_NS: &str = "http://www.w3.org/2000/svg"; +const XLINK_NS: &str = "http://www.w3.org/1999/xlink"; +const INVALID_IMAGE: &str = "data:image/png,%00"; +const IGNORED_DATA: &str = "data:application/octet-stream,%00"; + +struct Patch { + range: Range, + replacement: Cow<'static, str>, +} + +fn invalid(message: &str) -> Error { + Error::Parse(message.into()) +} + +#[cfg(test)] +fn repair<'a>(text: &'a str, limits: &Limits) -> Result, Error> { + normalize(&super::preflight(text, limits)?) +} + +pub(super) fn normalize<'a>(checked: &super::Preflight<'a>) -> Result, Error> { + let text = checked.xml.input_text(); + let limits = &checked.limits; + let xml = &checked.xml; + if !text.contains("href") { + return Ok(Cow::Borrowed(text)); + } + // Expanded values fit source + entity work. Twelve linear passes cover URL + // scans/streaming decode, full attribute projection and worst-case XML escaping; + // parameter compaction never allocates one record per original parameter. + let mut work = text + .len() + .checked_add(limits.entity_bytes) + .and_then(|n| n.checked_mul(12)) + .ok_or_else(|| invalid("SVG data-URL validation work exceeds its limit"))?; + let max_patches = (limits.bytes / std::mem::size_of::()).min(usize::try_from(limits.xml_nodes).unwrap_or(usize::MAX)); + let mut patches: Vec = Vec::new(); + for node in xml.descendants().filter(|node| node.is_element()) { + if !matches!(node.tag_name().name(), "image" | "feImage") { + continue; + } + // Shared preflight inspects local image names in every namespace and + // checks XLink before plain href. Normalize both representations, even + // when one is inert or unselected. Only actual resolver callbacks count + // omissions, so this does not turn hidden/unused resources into warnings. + for href in node.attributes().filter(|a| a.name() == "href" && matches!(a.namespace(), None | Some(XLINK_NS))) { + let Some(replacement) = data_url_replacement(href.value(), &mut work)? else { continue }; + let range = verified_value_range(text, node.range(), href.range(), href.name(), &mut work)?; + if patches.len() == patches.capacity() { + if patches.len() >= max_patches { + return Err(invalid("SVG data-URL repair metadata exceeds its limit")); + } + let capacity = patches.capacity().max(8).saturating_mul(2).min(max_patches); + patches + .try_reserve_exact(capacity.saturating_sub(patches.len())) + .map_err(|_| invalid("could not allocate bounded SVG data-URL repairs"))?; + } + patches.push(Patch { range, replacement }); + } + } + if patches.is_empty() { + return Ok(Cow::Borrowed(text)); + } + patches.sort_unstable_by_key(|patch| (patch.range.start, patch.range.end)); + patches.dedup_by_key(|patch| (patch.range.start, patch.range.end)); + let mut end = 0; + let mut length = text.len(); + for patch in &patches { + let range = &patch.range; + if range.start < end { + return Err(invalid("SVG data-URL source ranges overlap")); + } + end = range.end; + length = length + .checked_sub(range.len()) + .and_then(|n| n.checked_add(patch.replacement.len())) + .ok_or_else(|| invalid("SVG data-URL repair size overflows"))?; + } + if length > limits.bytes { + return Err(invalid("repaired SVG exceeds the text byte limit")); + } + let mut result = String::new(); + result.try_reserve_exact(length).map_err(|_| invalid("could not allocate bounded repaired SVG text"))?; + let mut cursor = 0; + for patch in patches { + result.push_str(text.get(cursor..patch.range.start).ok_or_else(|| invalid("invalid SVG data-URL source range"))?); + result.push_str(&patch.replacement); + cursor = patch.range.end; + } + result.push_str(text.get(cursor..).ok_or_else(|| invalid("invalid SVG data-URL source range"))?); + Ok(Cow::Owned(result)) +} + +// The pinned MIME parser searches all earlier parameters for each new name. +// Cap even quoted semicolons before invoking it: source-byte work is already +// charged, so at most this constant times the bounded XML work reaches MIME +// parsing. Bodies can remain large without acquiring MIME parameter records. +const MAX_MIME_SEPARATORS: usize = 64; + +fn mime_header_admitted(value: &str) -> bool { + let trimmed = value.trim_start_matches(|ch| ch <= ' '); + let mut bytes = trimmed.bytes(); + for expected in b"data:" { + let Some(actual) = bytes.by_ref().find(|&byte| !matches!(byte, b'\t' | b'\n' | b'\r')) else { return true }; + if !actual.eq_ignore_ascii_case(expected) { + return true; // Not a data URL: preserve external/fragment handling. + } + } + let mut separators = 0; + for byte in bytes { + if matches!(byte, b',' | b'#') { + break; + } + if byte == b';' { + separators += 1; + if separators > MAX_MIME_SEPARATORS { + return false; + } + } + } + true +} + +/// A long parameter list must not reach the quadratic MIME parser, but its +/// essence, final base64 marker and body still determine the original image. +/// Compact only that header, then use the same parser/streaming decoder. Escape +/// the decoded attribute value once when projecting it back into XML. +fn data_url_replacement(value: &str, work: &mut usize) -> Result>, Error> { + if mime_header_admitted(value) { + return malformed_data_url_replacement(value, work).map(|replacement| replacement.map(Cow::Borrowed)); + } + *work = work.checked_sub(value.len()).ok_or_else(|| invalid("SVG data-URL validation work exceeds its limit"))?; + let normalized = value.trim_matches(|ch| ch <= ' '); + let delimiter = normalized.find([',', '#']); + let Some(comma) = delimiter.filter(|&at| normalized.as_bytes().get(at) == Some(&b',')) else { + return missing_comma_replacement(value, work).map(|replacement| Some(Cow::Borrowed(replacement))); + }; + let header = normalized.get(..comma).ok_or_else(|| invalid("invalid SVG data-URL header framing"))?; + let essence = header.split_once(';').map_or(header, |(essence, _)| essence); + // Retaining the final parameter lets the pinned parser decide whether its + // suffix is a base64 marker, including its case/ASCII-whitespace rules. + let suffix = header.rfind(';').and_then(|at| header.get(at..)).unwrap_or(""); + let body = normalized.get(comma..).ok_or_else(|| invalid("invalid SVG data-URL body framing"))?; + let length = essence + .len() + .checked_add(suffix.len()) + .and_then(|n| n.checked_add(body.len())) + .ok_or_else(|| invalid("SVG data-URL repair size overflows"))?; + let mut compact = String::new(); + compact.try_reserve_exact(length).map_err(|_| invalid("could not allocate bounded SVG data-URL header"))?; + compact.push_str(essence); + compact.push_str(suffix); + compact.push_str(body); + if let Some(replacement) = malformed_data_url_replacement(&compact, work)? { + return Ok(Some(Cow::Borrowed(replacement))); + } + let escaped_len = compact + .chars() + .try_fold(0usize, |n, ch| { + n.checked_add(match ch { + '&' | '\t' | '\n' | '\r' => 5, + '\'' | '"' => 6, + '<' | '>' => 4, + _ => ch.len_utf8(), + }) + }) + .ok_or_else(|| invalid("SVG data-URL repair size overflows"))?; + *work = work.checked_sub(escaped_len).ok_or_else(|| invalid("SVG data-URL validation work exceeds its limit"))?; + let mut escaped = String::new(); + escaped.try_reserve_exact(escaped_len).map_err(|_| invalid("could not allocate bounded SVG data-URL repair"))?; + for ch in compact.chars() { + match ch { + '&' => escaped.push_str("&"), + '\'' => escaped.push_str("'"), + '"' => escaped.push_str("""), + '<' => escaped.push_str("<"), + '>' => escaped.push_str(">"), + '\t' => escaped.push_str(" "), + '\n' => escaped.push_str(" "), + '\r' => escaped.push_str(" "), + _ => escaped.push(ch), + } + } + Ok(Some(Cow::Owned(escaped))) +} + +fn malformed_data_url_replacement(value: &str, work: &mut usize) -> Result, Error> { + malformed_data_url_replacement_with(value, work, data_url::DataUrl::process) +} + +fn malformed_data_url_replacement_with<'a>( + value: &'a str, + work: &mut usize, + parse: impl FnOnce(&'a str) -> Result, data_url::DataUrlError>, +) -> Result, Error> { + *work = work.checked_sub(value.len()).ok_or_else(|| invalid("SVG data-URL validation work exceeds its limit"))?; + if !mime_header_admitted(value) { + // Strip parameters before recovering MIME disposition; no original + // parameter list reaches either our parser or shared whole-DOM preflight. + return missing_comma_replacement(value, work).map(Some); + } + let url = match parse(value) { + Ok(url) => url, + Err(data_url::DataUrlError::NoComma) => return missing_comma_replacement(value, work).map(Some), + Err(data_url::DataUrlError::NotADataUrl) => return Ok(None), + }; + let mut exhausted = false; + let decoded = url.decode(|bytes| { + if let Some(left) = work.checked_sub(bytes.len()) { + *work = left; + Ok(()) + } else { + exhausted = true; + Err(()) + } + }); + if exhausted { + return Err(invalid("SVG data-URL validation work exceeds its limit")); + } + Ok(decoded.is_err().then(|| replacement_for_mime(url.mime_type()))) +} + +/// Preflight must see a valid URL, while the resolver must keep its original +/// disposition. Non-image MIME types are ignored even when their body is bad. +fn replacement_for_mime(mime: &data_url::mime::Mime) -> &'static str { + if mime.type_ == "image" || mime.matches("text", "plain") { INVALID_IMAGE } else { IGNORED_DATA } +} + +fn missing_comma_replacement(value: &str, work: &mut usize) -> Result<&'static str, Error> { + // NoComma does not expose its MIME type. Ask the same parser about a header + // with an empty body rather than guessing MIME defaults, case or parameters. + // Match its URL C0 trimming; fragments and MIME parameters cannot affect + // the image/text-plain essence. Do not allocate parameter bookkeeping for a + // hostile missing-comma header just to recover that disposition. + let normalized = value.trim_matches(|ch| ch <= ' '); + let header = normalized.split_once('#').map_or(normalized, |(header, _)| header); + let essence = header.split_once(';').map_or(header, |(essence, _)| essence); + *work = work.checked_sub(essence.len()).ok_or_else(|| invalid("SVG data-URL validation work exceeds its limit"))?; + let length = essence.len().checked_add(1).ok_or_else(|| invalid("SVG data-URL repair size overflows"))?; + let mut candidate = String::new(); + candidate.try_reserve_exact(length).map_err(|_| invalid("could not allocate bounded SVG data-URL header"))?; + candidate.push_str(essence); + candidate.push(','); + // This candidate has no MIME parameters, including for oversized original + // lists. Its byte scan/allocation is charged above before the linear parse. + if !mime_header_admitted(&candidate) { + return Err(invalid("SVG data-URL MIME header exceeds its limit")); + } + let url = data_url::DataUrl::process(&candidate).map_err(|_| invalid("invalid SVG data-URL header framing"))?; + Ok(replacement_for_mime(url.mime_type())) +} + +/// Use the complete source range rather than roxmltree's compact QName/equals +/// offsets (u16/u8). Scan only this validated attribute, never a DTD declaration +/// or another node. Charge the extra source scan before examining it. +fn verified_value_range(text: &str, owner: Range, attribute: Range, local_name: &str, work: &mut usize) -> Result, Error> { + let error = || invalid("SVG data-URL attribute source positions cannot be represented safely"); + if attribute.start < owner.start || attribute.end > owner.end { + return Err(error()); + } + *work = work.checked_sub(attribute.len()).ok_or_else(|| invalid("SVG data-URL validation work exceeds its limit"))?; + let source = text.get(attribute.clone()).ok_or_else(error)?; + let name_end = source.find([' ', '\t', '\r', '\n', '=']).ok_or_else(error)?; + let name = source.get(..name_end).ok_or_else(error)?; + if name.rsplit(':').next() != Some(local_name) { + return Err(error()); + } + let after_name = source.get(name_end..).ok_or_else(error)?.trim_start_matches([' ', '\t', '\r', '\n']); + let after_equals = after_name.strip_prefix('=').ok_or_else(error)?.trim_start_matches([' ', '\t', '\r', '\n']); + let quote = after_equals.chars().next().ok_or_else(error)?; + if !matches!(quote, '\'' | '"') || !after_equals.ends_with(quote) || after_equals.len() < 2 { + return Err(error()); + } + let start = attribute.end.checked_sub(after_equals.len()).and_then(|n| n.checked_add(1)).ok_or_else(error)?; + let end = attribute.end.checked_sub(1).ok_or_else(error)?; + if text.get(start..end).is_none_or(|value| value.contains(quote)) { + return Err(error()); + } + Ok(start..end) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn limits() -> Limits { + Limits { bytes: 1 << 20, depth: 32, tree_depth: 32, entity_bytes: 65_536, ..Default::default() } + } + + #[test] + fn malformed_image_urls_are_repaired_in_both_svg_namespaces_and_filter_resources() { + for tag in ["image", "feImage"] { + for text in [ + format!("<{tag} href='data:image/png;base64,A'/>"), + format!("<{tag} href='data:image/png;base64,A'/>"), + format!(""), + ] { + let repaired = repair(&text, &limits()).unwrap(); + assert!(matches!(repaired, Cow::Owned(_))); + assert_eq!(repaired.matches(INVALID_IMAGE).count(), 1); + } + } + } + + #[test] + fn unselected_href_and_foreign_local_image_names_are_normalized_for_shared_preflight() { + for attrs in ["href='data:image/png,%00' x:href='data:image/png;base64,A'", "x:href='data:image/png;base64,A' href='data:image/png,%00'"] { + let text = format!(""); + let repaired = repair(&text, &limits()).unwrap(); + assert!(matches!(repaired, Cow::Owned(_))); + assert_eq!(repaired.matches(INVALID_IMAGE).count(), 2); + assert!(!repaired.contains("base64,A")); + } + let text = format!( + "" + ); + let repaired = repair(&text, &limits()).unwrap(); + assert_eq!(repaired.matches(INVALID_IMAGE).count(), 3); + assert!(repaired.contains("f:href='data:image/png;base64,A'"), "foreign href attributes remain inert and unchanged"); + } + + #[test] + fn both_href_forms_on_hidden_unused_and_foreign_resources_do_not_preflight_as_omissions() { + for tag in ["image", "feImage"] { + let text = format!( + "<{tag} id='unused' href='data:image/png;base64,A' x:href='data:image/png;base64,A'/><{tag} display='none' x:href='data:image/png;base64,A' href='data:image/png;base64,A'/>" + ); + let original = text.clone(); + let repaired = repair(&text, &limits()).unwrap(); + assert_eq!(text, original, "repair never mutates the original source"); + assert_eq!(repaired.matches(INVALID_IMAGE).count(), 6); + let resolver = crate::tree::ImageResolver::new(limits()); + resolver.check_document(&repaired).unwrap(); + let usage = resolver.usage(); + assert_eq!( + (usage.invalid_raster_images, usage.invalid_svg_images), + (0, 0), + "shared whole-document scanning cannot invent omissions before resolution" + ); + assert!(matches!(repair(&repaired, &limits()).unwrap(), Cow::Borrowed(_)), "normalization is idempotent"); + } + } + + #[test] + fn entity_expanded_urls_patch_the_owning_attributes_without_rewriting_the_doctype() { + let text = format!( + "]>" + ); + let repaired = repair(&text, &limits()).unwrap(); + assert!(repaired.contains("")); + assert_eq!(repaired.matches(INVALID_IMAGE).count(), 2); + assert!(!repaired.contains("href='&bad;'")); + assert!(matches!(repair(&repaired, &limits()).unwrap(), Cow::Borrowed(_))); + } + + #[test] + fn valid_data_and_external_urls_are_not_rewritten() { + for value in ["data:image/png;base64,AA==", "data:image/svg+xml,%3Csvg%2F%3E", "file.png", "#local"] { + let text = format!(""); + assert!(matches!(repair(&text, &limits()).unwrap(), Cow::Borrowed(_))); + } + let text = ""; + assert!(repair(text, &limits()).unwrap().contains(INVALID_IMAGE)); + } + + #[test] + fn malformed_non_image_urls_keep_ignored_replacements_even_without_a_comma() { + for value in [ + "data:application/octet-stream;base64,A", + "data:application/octet-stream;base64", + "data:APPLICATION/OCTET-STREAM;BASE64,A", + "data:text/css;base64,A", + "data:audio/ogg;base64,A", + "data:application/octet-stream;base64#fragment,not-the-delimiter", + ] { + let text = format!(""); + let repaired = repair(&text, &limits()).unwrap(); + assert!(repaired.contains(IGNORED_DATA), "{value}: {repaired}"); + assert!(!repaired.contains(INVALID_IMAGE)); + assert!(matches!(repair(&repaired, &limits()).unwrap(), Cow::Borrowed(_))); + let resolver = crate::tree::ImageResolver::new(limits()); + resolver.check_document(&repaired).unwrap(); + assert_eq!(resolver.usage().invalid_raster_images, 0); + } + } + + #[test] + fn image_and_default_plain_mime_failures_remain_disclosed() { + for value in [ + "data:IMAGE/PNG;base64,A", + "data:image/png;base64", + "data:text/plain;base64,A", + "data:;base64,A", + "data:;base64", + "data:not-a-mime;base64", + ] { + let text = format!(""); + assert!(repair(&text, &limits()).unwrap().contains(INVALID_IMAGE), "{value}"); + } + let value = "data:application/octet-stream;base64#fragment"; + let mut insufficient = value.len(); + assert!(malformed_data_url_replacement(value, &mut insufficient).is_err(), "the extra MIME-header parse has its own work charge"); + let mut sufficient = value.len() * 2; + assert_eq!(malformed_data_url_replacement(value, &mut sufficient).unwrap(), Some(IGNORED_DATA)); + } + + #[test] + fn long_equals_spacing_is_repaired_and_output_growth_stays_bounded() { + let text = format!("", " ".repeat(300)); + let repaired = repair(&text, &limits()).unwrap(); + assert!(repaired.contains(INVALID_IMAGE)); + assert!(repaired.contains(&format!("href{}=", " ".repeat(300)))); + let text = ""; + assert!(matches!(repair(text, &Limits { bytes: text.len(), ..limits() }), Err(Error::Parse(message)) if message.contains("byte limit"))); + } +} + +#[cfg(test)] +#[path = "mime_header_tests.rs"] +mod mime_header_tests; diff --git a/crates/svg/src/tree/mime_header_tests.rs b/crates/svg/src/tree/mime_header_tests.rs new file mode 100644 index 000000000..393a01729 --- /dev/null +++ b/crates/svg/src/tree/mime_header_tests.rs @@ -0,0 +1,158 @@ +use super::*; +use std::fmt::Write; + +fn parameters(count: usize) -> String { + let mut text = String::new(); + for index in 0..count { + write!(&mut text, ";p{index}=x").unwrap(); + } + text +} + +#[test] +fn a_hundred_thousand_unique_mime_parameters_are_refused_before_the_original_parser() { + let params = parameters(100_000); + for (essence, expected) in [ + ("image/png", INVALID_IMAGE), + ("text/plain", INVALID_IMAGE), + ("", INVALID_IMAGE), + ("not-a-mime", INVALID_IMAGE), + ("application/octet-stream", IGNORED_DATA), + ("audio/ogg", IGNORED_DATA), + ] { + for delimiter in [",AA==", "#fragment,AA==", ""] { + let value = format!("data:{essence}{params};base64{delimiter}"); + let mut work = value.len() * 2; + assert!(!mime_header_admitted(&value)); + let result = + malformed_data_url_replacement_with(&value, &mut work, |_| panic!("the original parameter flood must not reach DataUrl::process")) + .unwrap(); + assert_eq!(result, Some(expected), "{essence} {delimiter}"); + } + } +} + +#[test] +fn mime_parameter_admission_preserves_the_boundary_and_ignores_body_semicolons() { + let accepted = format!("data:image/png{};base64,AA==", parameters(MAX_MIME_SEPARATORS - 1)); + assert!(mime_header_admitted(&accepted)); + let called = std::cell::Cell::new(false); + let mut work = accepted.len() * 2; + assert_eq!( + malformed_data_url_replacement_with(&accepted, &mut work, |value| { + called.set(true); + data_url::DataUrl::process(value) + }) + .unwrap(), + None + ); + assert!(called.get(), "an admitted original URL still reaches the real parser"); + let refused = format!("data:image/png{};base64,AA==", parameters(MAX_MIME_SEPARATORS)); + assert!(!mime_header_admitted(&refused)); + let mut work = refused.len() * 2; + assert_eq!(malformed_data_url_replacement(&refused, &mut work).unwrap(), Some(INVALID_IMAGE)); + + for value in [format!("data:text/plain,{}", ";".repeat(100_000)), format!("external.png?{}", parameters(100_000))] { + assert!(mime_header_admitted(&value), "payloads/external URLs are not MIME parameters"); + let mut work = value.len() * 2; + assert_eq!(malformed_data_url_replacement(&value, &mut work).unwrap(), None); + } + let value = format!(" \td\ta\nt\ra:image/png{};base64,A", parameters(MAX_MIME_SEPARATORS)); + assert!(!mime_header_admitted(&value), "scheme normalization matches the pinned parser"); + let mut work = value.len() * 2; + assert_eq!(malformed_data_url_replacement(&value, &mut work).unwrap(), Some(INVALID_IMAGE)); +} + +#[test] +fn compacted_parameter_lists_preserve_both_hrefs_and_foreign_resources_without_preflight_warnings() { + let value = format!("data:image/png{};base64,AA==", parameters(MAX_MIME_SEPARATORS)); + let text = format!( + "" + ); + let limits = Limits { bytes: 1 << 20, ..Default::default() }; + let repaired = repair(&text, &limits).unwrap(); + assert_eq!(repaired.matches("data:image/png;base64,AA==").count(), 6); + assert!(!repaired.contains(";p0=")); + let resolver = crate::tree::ImageResolver::new(limits); + resolver.check_document(&repaired).unwrap(); + let usage = resolver.usage(); + assert_eq!((usage.invalid_raster_images, usage.invalid_svg_images), (0, 0)); + assert!(matches!(repair(&repaired, &Limits { bytes: 1 << 20, ..Default::default() }).unwrap(), Cow::Borrowed(_))); +} + +#[test] +fn duplicate_and_quoted_parameters_share_the_same_conservative_boundary() { + for (accepted, refused) in [ + ( + format!("data:image/png{},%00", ";p=x".repeat(MAX_MIME_SEPARATORS)), + format!("data:image/png{},%00", ";p=x".repeat(MAX_MIME_SEPARATORS + 1)), + ), + ( + format!("data:image/png;p=\"{}\",%00", ";".repeat(MAX_MIME_SEPARATORS - 1)), + format!("data:image/png;p=\"{}\",%00", ";".repeat(MAX_MIME_SEPARATORS)), + ), + ] { + assert!(mime_header_admitted(&accepted)); + let mut work = accepted.len() * 2; + assert_eq!(malformed_data_url_replacement(&accepted, &mut work).unwrap(), None); + assert!(!mime_header_admitted(&refused)); + let mut work = refused.len() * 2; + assert_eq!(malformed_data_url_replacement(&refused, &mut work).unwrap(), Some(INVALID_IMAGE)); + } + for value in ["external.png", "#local", "datx:image/png;param=x,body"] { + let mut work = value.len() * 2; + assert_eq!(malformed_data_url_replacement(value, &mut work).unwrap(), None); + } + let value = "data:image/png;base64"; + let mut work = value.len() * 2; + assert_eq!(malformed_data_url_replacement(value, &mut work).unwrap(), Some(INVALID_IMAGE)); +} + +#[test] +fn parameter_flood_scans_and_essence_classification_spend_one_shared_work_ledger() { + let value = format!("data:application/octet-stream{};base64,AA==", parameters(MAX_MIME_SEPARATORS)); + let essence_len = "data:application/octet-stream".len(); + let charge = value.len() + essence_len; + let mut work = charge * 2; + for remaining in [charge, 0] { + assert_eq!(malformed_data_url_replacement_with(&value, &mut work, |_| panic!("not admitted")).unwrap(), Some(IGNORED_DATA)); + assert_eq!(work, remaining, "the original header scan and parameter-free fallback each retain their charge"); + } + assert!(malformed_data_url_replacement_with(&value, &mut work, |_| panic!("no work remains")).is_err()); + assert_eq!(work, 0); + let mut not_enough_for_essence = value.len(); + assert!(malformed_data_url_replacement_with(&value, &mut not_enough_for_essence, |_| panic!("not admitted")).is_err()); + assert_eq!(not_enough_for_essence, 0); +} + +#[test] +fn many_parameter_urls_preserve_bodies_and_base64_disposition() { + let params = parameters(100_000); + for (suffix, body, expected) in [ + (";base64", "AA==", vec![0]), + ("; BASE64 ", "AA==", vec![0]), + (";p=last", "%00%26%27%22", vec![0, b'&', b'\'', b'"']), + (";p=last", "&'" z", b"&\'\"z".to_vec()), + ] { + let text = format!(""); + let repaired = repair(&text, &Limits { bytes: 2 << 20, ..Default::default() }).unwrap(); + let xml = checked_xml(&repaired, &Limits { bytes: 2 << 20, ..Default::default() }).unwrap(); + let value = xml.descendants().find(|node| node.has_tag_name("image")).unwrap().attribute("href").unwrap(); + assert!(!value.contains(";p0=")); + assert_eq!(data_url::DataUrl::process(value).unwrap().decode_to_vec().unwrap().0, expected); + assert!(matches!(repair(&repaired, &Limits { bytes: 2 << 20, ..Default::default() }).unwrap(), Cow::Borrowed(_))); + } + let malformed = format!(""); + assert!(repair(&malformed, &Limits { bytes: 2 << 20, ..Default::default() }).unwrap().contains(INVALID_IMAGE)); +} + +#[test] +fn header_compaction_has_a_shared_nonrefundable_work_charge() { + let value = format!("data:image/png{};base64,AA==", parameters(100)); + let mut work = value.len(); + assert!(data_url_replacement(&value, &mut work).is_err()); + assert_eq!(work, 0, "the original parameter scan stays spent on refusal"); + let mut work = value.len() * 12; + assert!(data_url_replacement(&value, &mut work).unwrap().is_some()); + assert!(work < value.len() * 12 - value.len(), "compact parse, streaming decode and XML projection are charged too"); +} diff --git a/crates/svg/src/tree/validated.rs b/crates/svg/src/tree/validated.rs new file mode 100644 index 000000000..cb1b6e8c3 --- /dev/null +++ b/crates/svg/src/tree/validated.rs @@ -0,0 +1,393 @@ +//! Opt-in complete image validation and checked-DOM normalization. +//! Legacy ImageResolver entry points retain their existing behavior. +use super::{Error, ImageUsage, Limits, Preflight, image_href, preflight, raster_format, usvg}; +use crate::svgz; +use std::sync::{Arc, Mutex, PoisonError}; + +struct State { + document_bytes: usize, + pixel_limit: u64, + pixels_left: u64, + svg_byte_limit: usize, + svg_bytes_left: usize, + svg_failure_exhausted: bool, + classification_bytes_left: usize, + classification_content_left: usize, + classification_text_bytes_left: usize, + encoded_left: usize, + invalid_raster: usize, + invalid_svg: usize, + raster_limits: usize, + svg_limits: usize, + svg_resource_limits: usize, + root_active: bool, + xml_depth: usize, + depth: usize, + svg_count: usize, + svg_depth: usize, +} + +impl State { + fn new(limits: Limits) -> Self { + Self { + document_bytes: limits.bytes, + pixel_limit: limits.embedded_raster_pixels, + pixels_left: limits.embedded_raster_pixels, + svg_byte_limit: limits.embedded_bytes, + svg_bytes_left: limits.embedded_bytes, + svg_failure_exhausted: false, + classification_bytes_left: limits.embedded_bytes, + classification_content_left: limits.embedded_bytes, + classification_text_bytes_left: limits.embedded_bytes, + encoded_left: limits.bytes.saturating_add(limits.embedded_bytes), + invalid_raster: 0, + invalid_svg: 0, + raster_limits: 0, + svg_limits: 0, + svg_resource_limits: 0, + root_active: false, + xml_depth: 0, + depth: 0, + svg_count: 0, + svg_depth: 0, + } + } + + fn usage(&self) -> ImageUsage { + ImageUsage { + raster_pixels: self.pixel_limit.saturating_sub(self.pixels_left), + embedded_svg_bytes: self.svg_byte_limit.saturating_sub(self.svg_bytes_left), + embedded_svg_count: self.svg_count, + embedded_svg_depth: self.svg_depth, + invalid_raster_images: self.invalid_raster, + raster_limit_images: self.raster_limits, + invalid_svg_images: self.invalid_svg, + svg_limit_images: self.svg_limits, + svg_resource_limit_images: self.svg_resource_limits, + } + } + + fn classify_after_svg_failure(&mut self, data: &[u8]) { + if self.svg_resource_limits != 0 { + self.svg_limits = self.svg_limits.saturating_add(1); + self.svg_resource_limits = self.svg_resource_limits.saturating_add(1); + return; + } + let compressed = data.starts_with(&[0x1f, 0x8b]); + let content_left = self.classification_content_left; + let limit = if compressed { content_left.min(self.classification_bytes_left) } else { content_left }.min(self.document_bytes); + if compressed && limit == 0 { + self.svg_limits = self.svg_limits.saturating_add(1); + self.svg_resource_limits = self.svg_resource_limits.saturating_add(1); + return; + } + let resource_refusal = match crate::svgz::inflate(data, limit) { + Ok(bytes) => { + if compressed { + self.classification_bytes_left = self.classification_bytes_left.saturating_sub(bytes.len()); + } + let Some(text_left) = self.classification_text_bytes_left.checked_sub(bytes.len()) else { + self.svg_limits = self.svg_limits.saturating_add(1); + self.svg_resource_limits = self.svg_resource_limits.saturating_add(1); + return; + }; + self.classification_text_bytes_left = text_left; + match crate::svgz::decode(&bytes) { + Ok(text) => text.len() > content_left || text.len() > self.document_bytes, + Err(_) => { + self.invalid_svg = self.invalid_svg.saturating_add(1); + return; + } + } + } + Err(crate::svgz::Error::TooLarge(_)) => { + if compressed { + self.classification_bytes_left = self.classification_bytes_left.saturating_sub(limit); + } + true + } + Err(error) => { + // Nested data was fully inflated before refusal. Other failures + // may have consumed the entire attempted inflation allowance. + if compressed { + let charged = match error { + crate::svgz::Error::Nested(bytes) => bytes, + _ => limit, + }; + self.classification_bytes_left = self.classification_bytes_left.saturating_sub(charged); + } + self.invalid_svg = self.invalid_svg.saturating_add(1); + return; + } + }; + self.svg_limits = self.svg_limits.saturating_add(1); + if resource_refusal { + self.svg_resource_limits = self.svg_resource_limits.saturating_add(1); + } + } + + fn raster(&mut self, kind: &usvg::ImageKind) -> bool { + let encoded = match kind { + usvg::ImageKind::SVG(_) => return true, + usvg::ImageKind::PNG(data) | usvg::ImageKind::JPEG(data) | usvg::ImageKind::GIF(data) | usvg::ImageKind::WEBP(data) => data, + }; + // usvg supplies a fresh Arc for every data-URL resolution, including + // repeated instances. Charge each actual validation, never retain + // encoded buffers in an address-keyed memo that cannot hit this boundary. + let Some(left) = self.encoded_left.checked_sub(encoded.len()) else { + self.raster_limits = self.raster_limits.saturating_add(1); + return false; + }; + self.encoded_left = left; + let result = super::draw::validate_raster(kind, self.pixel_limit, &mut self.pixels_left); + match result { + Ok(()) => true, + Err(super::draw::RasterValidationFailure::Invalid) => { + self.invalid_raster = self.invalid_raster.saturating_add(1); + false + } + Err(super::draw::RasterValidationFailure::Limit) => { + self.raster_limits = self.raster_limits.saturating_add(1); + false + } + } + } +} + +/// One opt-in drawing's image work, normalization and exact rendering validation. +/// +/// Unlike legacy ImageResolver, every attempted raster callback spends encoded +/// and decoded work, even on failure. Rendering still decodes pixels afterwards: +/// resvg does not expose a decoded-pixel injection or decode-result hook. +/// All root and child conversions install this same resolver themselves. +#[derive(Clone)] +pub struct ValidatedImageResolver { + limits: Limits, + state: Arc>, +} + +impl ValidatedImageResolver { + /// Create a separate budget for one drawing. Encoded raster work is bounded + /// by root text bytes plus aggregate child SVG bytes, without retaining data. + pub fn new(limits: Limits) -> Self { + Self { limits, state: Arc::new(Mutex::new(State::new(limits))) } + } + + /// The immutable admission policy captured by this instance. Remaining work + /// never changes these limits; use them for the matching opaque preflight. + pub fn limits(&self) -> Limits { + self.limits + } + + /// Snapshot admitted work and optional-image omissions. + pub fn usage(&self) -> ImageUsage { + self.state.lock().unwrap_or_else(PoisonError::into_inner).usage() + } + + /// Convert exactly the DOM checked under these limits, with this instance's + /// resolver. A caller-supplied image callback cannot bypass image admission. + /// The caller still supplies the stack/worker and font policy. + pub fn parse_preflight(&self, checked: Preflight<'_>, mut options: usvg::Options<'_>) -> Result { + if checked.limits != self.limits { + return Err(Error::Parse("checked SVG limits differ from the validated image resolver".into())); + } + // A shallow source can expand through use/paint references. Reserve the + // full checked converted height before any child callback, not source depth. + let _scope = self.root_scope(checked.height)?; + options.image_href_resolver = self.resolver(); + convert_checked(checked, &options) + } + + /// Resolve an already-decoded child SVG, charging original text before XML + /// parsing or repair. This is not an unchecked/precharged admission hook. + /// As with parse_preflight, unrelated image callbacks are replaced. + pub fn resolve_text(&self, text: &str, mut options: usvg::Options<'_>) -> Result, Error> { + let _scope = self.root_scope(0)?; + options.image_href_resolver = self.resolver(); + Ok(self.embedded_text(text, &options)) + } + + fn root_scope(&self, depth: usize) -> Result { + let mut state = self.state.lock().unwrap_or_else(PoisonError::into_inner); + if state.root_active { + return Err(Error::Parse("the validated SVG resolver is already converting a drawing".into())); + } + let previous = state.xml_depth; + state.root_active = true; + state.xml_depth = depth; + Ok(Scope { state: Arc::clone(&self.state), previous, child: false }) + } + + fn resolver(&self) -> usvg::ImageHrefResolver<'static> { + let this = self.clone(); + usvg::ImageHrefResolver { + resolve_string: Box::new(|_, _| None), + resolve_data: Box::new(move |mime, data, options| { + let detected = raster_format(&data); + let format = match mime { + "image/jpg" | "image/jpeg" => (detected == Some(craft_image_codecs::Format::Jpeg)).then_some(craft_image_codecs::Format::Jpeg), + "image/png" => (detected == Some(craft_image_codecs::Format::Png)).then_some(craft_image_codecs::Format::Png), + "image/gif" => (detected == Some(craft_image_codecs::Format::Gif)).then_some(craft_image_codecs::Format::Gif), + "image/webp" => (detected == Some(craft_image_codecs::Format::WebP)).then_some(craft_image_codecs::Format::WebP), + "text/plain" => detected, + _ => None, + }; + if let Some(format) = format { + let kind = match format { + craft_image_codecs::Format::Png => usvg::ImageKind::PNG(data), + craft_image_codecs::Format::Jpeg => usvg::ImageKind::JPEG(data), + craft_image_codecs::Format::Gif => usvg::ImageKind::GIF(data), + craft_image_codecs::Format::WebP => usvg::ImageKind::WEBP(data), + _ => return None, + }; + return this.state.lock().unwrap_or_else(PoisonError::into_inner).raster(&kind).then_some(kind); + } + if !matches!(mime, "image/svg+xml" | "text/plain") { + if mime.starts_with("image/") { + let mut state = this.state.lock().unwrap_or_else(PoisonError::into_inner); + state.invalid_raster = state.invalid_raster.saturating_add(1); + } + return None; + } + this.embedded_bytes(&data, options) + }), + } + } + + fn can_start_child(&self) -> bool { + let mut state = self.state.lock().unwrap_or_else(PoisonError::into_inner); + if state.depth >= self.limits.embedded_depth { + state.svg_limits = state.svg_limits.saturating_add(1); + state.svg_resource_limits = state.svg_resource_limits.saturating_add(1); + return false; + } + true + } + + fn embedded_bytes(&self, data: &[u8], options: &usvg::Options<'_>) -> Option { + if !self.can_start_child() { + return None; + } + let room = { + let mut state = self.state.lock().unwrap_or_else(PoisonError::into_inner); + if state.svg_failure_exhausted { + state.classify_after_svg_failure(data); + return None; + } + state.svg_bytes_left + }; + let text = match svgz::text(data, room.min(self.limits.bytes)) { + Ok(text) if text.len() <= room => text, + error => { + let mut state = self.state.lock().unwrap_or_else(PoisonError::into_inner); + state.svg_bytes_left = 0; + if matches!(error, Err(svgz::Error::TooLarge(_))) || error.is_ok() { + state.svg_limits = state.svg_limits.saturating_add(1); + state.svg_resource_limits = state.svg_resource_limits.saturating_add(1); + } else { + state.invalid_svg = state.invalid_svg.saturating_add(1); + state.svg_failure_exhausted = true; + state.classification_content_left = room; + } + return None; + } + }; + self.precharge_and_parse(&text, options) + } + + fn embedded_text(&self, text: &str, options: &usvg::Options<'_>) -> Option { + if !self.can_start_child() { + return None; + } + self.precharge_and_parse(text, options) + } + + fn precharge_and_parse(&self, text: &str, options: &usvg::Options<'_>) -> Option { + { + let mut state = self.state.lock().unwrap_or_else(PoisonError::into_inner); + let Some(left) = state.svg_bytes_left.checked_sub(text.len()) else { + state.svg_bytes_left = 0; + state.svg_limits = state.svg_limits.saturating_add(1); + state.svg_resource_limits = state.svg_resource_limits.saturating_add(1); + return None; + }; + // Original UTF-8 bytes/work are never refunded when repair shortens + // an attribute or conversion subsequently fails. + state.svg_bytes_left = left; + } + let checked = match preflight(text, &self.limits) { + Ok(checked) => checked, + Err(error) => { + let mut state = self.state.lock().unwrap_or_else(PoisonError::into_inner); + match error { + Error::TooDeep(_) => state.svg_limits = state.svg_limits.saturating_add(1), + Error::Text(svgz::Error::TooLarge(_)) => { + state.svg_limits = state.svg_limits.saturating_add(1); + state.svg_resource_limits = state.svg_resource_limits.saturating_add(1); + } + _ => state.invalid_svg = state.invalid_svg.saturating_add(1), + } + return None; + } + }; + let scope = { + let mut state = self.state.lock().unwrap_or_else(PoisonError::into_inner); + let Some(total) = state.xml_depth.checked_add(checked.height).filter(|depth| *depth <= self.limits.tree_depth) else { + state.svg_limits = state.svg_limits.saturating_add(1); + return None; + }; + let previous = state.xml_depth; + state.xml_depth = total; + state.depth = state.depth.saturating_add(1); + state.svg_depth = state.svg_depth.max(state.depth); + Scope { state: Arc::clone(&self.state), previous, child: true } + }; + let result = convert_checked(checked, options); + drop(scope); + let mut state = self.state.lock().unwrap_or_else(PoisonError::into_inner); + match result { + Ok(tree) => { + state.svg_count = state.svg_count.saturating_add(1); + Some(usvg::ImageKind::SVG(tree)) + } + Err(_) => { + state.invalid_svg = state.invalid_svg.saturating_add(1); + None + } + } + } +} + +struct Scope { + state: Arc>, + previous: usize, + child: bool, +} +impl Drop for Scope { + fn drop(&mut self) { + let mut state = self.state.lock().unwrap_or_else(PoisonError::into_inner); + state.xml_depth = self.previous; + if self.child { + state.depth = state.depth.saturating_sub(1); + } else { + state.root_active = false; + } + } +} + +fn convert_checked(checked: Preflight<'_>, options: &usvg::Options<'_>) -> Result { + match image_href::normalize(&checked)? { + std::borrow::Cow::Borrowed(_) => checked.parse(options), + std::borrow::Cow::Owned(text) => { + let limits = checked.limits; + drop(checked); + preflight(&text, &limits)?.parse(options) + } + } +} + +#[cfg(test)] +mod tests; + +#[cfg(test)] +mod xml_tests; diff --git a/crates/svg/src/tree/validated/tests.rs b/crates/svg/src/tree/validated/tests.rs new file mode 100644 index 000000000..c090d7ab6 --- /dev/null +++ b/crates/svg/src/tree/validated/tests.rs @@ -0,0 +1,340 @@ +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)] +use super::*; +use std::io::Write; +const PIXELS: u64 = 24_000_000; +const SVG_BYTES: usize = 16 << 20; + +fn test_limits(pixels: u64) -> Limits { + Limits { + bytes: 1 << 30, + depth: 513, + tree_depth: 513, + embedded_bytes: SVG_BYTES, + embedded_depth: 32, + embedded_raster_pixels: pixels, + ..Default::default() + } +} +fn test_state(pixels: u64) -> State { + State::new(test_limits(pixels)) +} +fn wrapped_options(pixels: u64) -> (usvg::Options<'static>, Arc>, Arc) { + let resolver = Arc::new(ValidatedImageResolver::new(test_limits(pixels))); + let options = usvg::Options { image_href_resolver: resolver.resolver(), ..Default::default() }; + (options, Arc::clone(&resolver.state), resolver) +} +fn on_test_stack(run: impl FnOnce() -> Result + Send) -> Result { + std::thread::scope(|scope| std::thread::Builder::new().stack_size(16 << 20).spawn_scoped(scope, run).unwrap().join().unwrap()) +} +fn png() -> Vec { + let image = craft_image_codecs::Image::from_u8(2, 2, craft_image_codecs::ChannelLayout::Rgb, [17, 83, 149].repeat(4)).unwrap(); + craft_image_codecs::encode(&image, craft_image_codecs::Format::Png, &Default::default()).unwrap() +} +fn data_url(mime: &str, bytes: &[u8]) -> String { + use std::fmt::Write; + let mut result = format!("data:{mime},"); + for byte in bytes { + write!(&mut result, "%{byte:02X}").unwrap(); + } + result +} + +#[test] +fn original_nested_xml_bytes_are_not_refunded_when_malformed_hrefs_shrink() { + on_test_stack(|| { + let (options, state, _resolver) = wrapped_options(PIXELS); + let prefix = ""; + let original_size = 3 << 20; + let source = Arc::new([prefix.as_bytes(), &vec![b'A'; original_size - prefix.len() - suffix.len()], suffix.as_bytes()].concat()); + assert_eq!(source.len(), original_size); + for accepted in 1..=5 { + let kind = (options.image_href_resolver.resolve_data)("image/svg+xml", Arc::clone(&source), &options); + assert!(matches!(kind, Some(usvg::ImageKind::SVG(_))), "source {accepted} fits the original-text allowance"); + drop(kind); + let state = state.lock().unwrap_or_else(PoisonError::into_inner); + assert_eq!(state.svg_bytes_left, SVG_BYTES - accepted * original_size); + assert!(!state.svg_failure_exhausted); + assert_eq!(state.classification_bytes_left, SVG_BYTES); + let usage = state.usage(); + assert_eq!(usage.embedded_svg_bytes, accepted * original_size, "the short replacement cannot refund original XML work"); + } + assert!((options.image_href_resolver.resolve_data)("image/svg+xml", Arc::clone(&source), &options).is_none()); + let state = state.lock().unwrap_or_else(PoisonError::into_inner); + assert_eq!(state.svg_bytes_left, 0); + assert_eq!((state.svg_limits, state.svg_resource_limits), (1, 1)); + assert!(!state.svg_failure_exhausted, "ordinary byte exhaustion cannot enter compatibility classification"); + assert_eq!(state.classification_bytes_left, SVG_BYTES); + assert_eq!(state.usage().svg_resource_limit_images, 1); + Ok(()) + }) + .unwrap(); +} + +#[test] +fn invalid_utf8_gzip_consumes_remaining_svg_work_before_the_next_child() { + on_test_stack(|| { + let (options, state, _resolver) = wrapped_options(PIXELS); + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder.write_all(&[0xff; 4096]).unwrap(); + let invalid = Arc::new(encoder.finish().unwrap()); + assert!((options.image_href_resolver.resolve_data)("image/svg+xml", invalid, &options).is_none()); + { + let state = state.lock().unwrap_or_else(PoisonError::into_inner); + assert_eq!(state.svg_bytes_left, 0, "a failed decode cannot retry the full work allowance"); + assert_eq!(state.invalid_svg, 1); + assert!(state.svg_failure_exhausted); + assert_eq!(state.svg_resource_limits, 0); + assert_eq!(state.classification_bytes_left, SVG_BYTES); + } + let valid = Arc::new(b"".to_vec()); + let valid_size = valid.len(); + assert!((options.image_href_resolver.resolve_data)("image/svg+xml", valid, &options).is_none()); + let state = state.lock().unwrap_or_else(PoisonError::into_inner); + assert_eq!((state.invalid_svg, state.svg_limits, state.svg_resource_limits), (1, 1, 0)); + assert_eq!(state.classification_bytes_left, SVG_BYTES, "plain input does not spend retry inflation work"); + let usage = state.usage(); + assert_eq!(usage.embedded_svg_bytes, SVG_BYTES); + assert_eq!(usage.svg_resource_limit_images, 0, "a later small child is omitted, not a resource error"); + let charged = 2 * SVG_BYTES - state.svg_bytes_left - state.classification_bytes_left; + assert_eq!(charged, SVG_BYTES); + assert!(valid_size < SVG_BYTES); + Ok(()) + }) + .unwrap(); +} + +#[test] +fn post_failure_classification_is_cumulative_and_never_admits_later_children() { + on_test_stack(|| { + let (options, state, _resolver) = wrapped_options(PIXELS); + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder.write_all(&[0xff; 256]).unwrap(); + assert!((options.image_href_resolver.resolve_data)("image/svg+xml", Arc::new(encoder.finish().unwrap()), &options).is_none()); + let text = b""; + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder.write_all(text).unwrap(); + let valid = Arc::new(encoder.finish().unwrap()); + // Exercise the cumulative inflated-byte boundary with a small injected + // second pool; public tests cover the full 16 MiB allowance. + let classification_limit = text.len() * 2 - 1; + state.lock().unwrap_or_else(PoisonError::into_inner).classification_bytes_left = classification_limit; + assert!((options.image_href_resolver.resolve_data)("image/svg+xml", Arc::clone(&valid), &options).is_none()); + { + let state = state.lock().unwrap_or_else(PoisonError::into_inner); + assert_eq!(state.classification_bytes_left, text.len() - 1); + assert_eq!((state.svg_limits, state.svg_resource_limits), (1, 0)); + } + assert!((options.image_href_resolver.resolve_data)("image/svg+xml", Arc::clone(&valid), &options).is_none()); + let state = state.lock().unwrap_or_else(PoisonError::into_inner); + assert_eq!(state.classification_bytes_left, 0); + assert_eq!((state.svg_limits, state.svg_resource_limits), (2, 1)); + assert_eq!(state.svg_bytes_left, 0, "classification never refunds original failed work"); + assert!(state.svg_failure_exhausted); + let charged = 2 * SVG_BYTES - state.svg_bytes_left - state.classification_bytes_left; + assert_eq!(charged, 2 * SVG_BYTES, "original content plus aggregate retry inflation allowances remain bounded"); + assert_eq!(state.usage().embedded_svg_count, 0, "classified children must never reach the shared parser"); + Ok(()) + }) + .unwrap(); +} + +#[test] +fn retry_inflation_room_and_saved_decoded_content_room_remain_independent() { + on_test_stack(|| { + let mut latin1 = b""); + let decoded_len = crate::svgz::decode(&latin1).unwrap().len(); + assert!(latin1.len() < 256 && decoded_len > 256 && decoded_len < 512); + let gzip = |bytes: &[u8]| { + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder.write_all(bytes).unwrap(); + Arc::new(encoder.finish().unwrap()) + }; + for (content_room, expected_resource) in [(512usize, false), (256, true)] { + let (options, state, _resolver) = wrapped_options(PIXELS); + // Model preceding genuine content consumption before an unreadable + // stream burns the rest. Classification must save this exact room. + state.lock().unwrap_or_else(PoisonError::into_inner).svg_bytes_left = content_room; + assert!((options.image_href_resolver.resolve_data)("image/svg+xml", gzip(&[0xff; 4]), &options).is_none()); + { + let mut state = state.lock().unwrap_or_else(PoisonError::into_inner); + assert_eq!(state.classification_content_left, content_room); + assert_eq!(state.svg_bytes_left, 0); + // Enough compressed retry work for the raw Latin-1 bytes, but + // less than their decoded UTF-8 length, just like main's case. + state.classification_bytes_left = latin1.len() + 1; + } + assert!((options.image_href_resolver.resolve_data)("image/svg+xml", gzip(&latin1), &options).is_none()); + let state = state.lock().unwrap_or_else(PoisonError::into_inner); + assert_eq!(state.classification_bytes_left, 1, "retry work charges inflated bytes, not UTF-8 bytes"); + assert_eq!(state.svg_resource_limits, usize::from(expected_resource)); + assert_eq!(state.svg_bytes_left, 0, "classification never restores original content allowance"); + assert_eq!(state.usage().embedded_svg_count, 0, "no classified child is parsed or drawn"); + } + Ok(()) + }) + .unwrap(); +} + +#[test] +fn failed_raster_callbacks_release_sources_and_keep_every_decode_charge() { + let mut broken = png(); + let idat = broken.windows(4).position(|bytes| bytes == b"IDAT").unwrap(); + broken[idat + 4] ^= 1; + assert_eq!(craft_image_codecs::probe_allocation_size(&broken).unwrap(), (2, 2)); + let mut state = test_state(8); + let encoded_before = state.encoded_left; + for attempt in 1..=2 { + // Match usvg's actual callback boundary: each href gets a new Arc. + let encoded = Arc::new(broken.clone()); + let weak = Arc::downgrade(&encoded); + let kind = usvg::ImageKind::PNG(Arc::clone(&encoded)); + assert!(!state.raster(&kind)); + assert_eq!(state.pixels_left, 8 - attempt * 4); + assert_eq!(state.encoded_left, encoded_before - attempt as usize * encoded.len()); + assert_eq!(state.invalid_raster, attempt as usize); + drop(kind); + drop(encoded); + assert!(weak.upgrade().is_none(), "validation must not retain failed callback buffers until parse completion"); + } + assert!(!state.raster(&usvg::ImageKind::PNG(Arc::new(png())))); + assert_eq!((state.invalid_raster, state.raster_limits), (2, 1)); + assert_eq!(state.usage().raster_pixels, 8); +} + +#[test] +fn valid_raster_callbacks_spend_exact_pixel_and_encoded_allowances_per_call() { + let encoded = Arc::new(png()); + let kind = usvg::ImageKind::PNG(Arc::clone(&encoded)); + let mut state = test_state(8); + state.encoded_left = encoded.len() * 2; + assert!(state.raster(&kind)); + assert_eq!((state.pixels_left, state.encoded_left), (4, encoded.len())); + assert!(state.raster(&kind), "even a directly reused Arc is a new actual validation call"); + assert_eq!((state.pixels_left, state.encoded_left), (0, 0)); + assert!(!state.raster(&kind)); + assert_eq!((state.invalid_raster, state.raster_limits), (0, 1)); + let usage = state.usage(); + assert_eq!((usage.raster_pixels, usage.raster_limit_images), (8, 1)); +} + +#[test] +fn raster_callbacks_own_per_call_pixel_charges_and_keep_mime_aliases_separate() { + let png = Arc::new(png()); + let (options, state, resolver) = wrapped_options(4); + assert!((options.image_href_resolver.resolve_data)("image/png", Arc::clone(&png), &options).is_some()); + assert!((options.image_href_resolver.resolve_data)("image/png", Arc::clone(&png), &options).is_none()); + assert_eq!((resolver.usage().raster_pixels, resolver.usage().invalid_raster_images, resolver.usage().raster_limit_images), (4, 0, 1)); + assert!((options.image_href_resolver.resolve_data)("image/jpeg", png, &options).is_none()); + let usage = state.lock().unwrap().usage(); + assert_eq!((usage.raster_pixels, usage.invalid_raster_images), (4, 1)); +} + +#[test] +fn encoded_raster_refusal_is_per_image_and_does_not_consume_unattempted_work() { + let bytes = png(); + let mut oversized = bytes.clone(); + oversized.extend([0; 32]); + let mut state = test_state(4); + state.encoded_left = bytes.len(); + assert!(!state.raster(&usvg::ImageKind::PNG(Arc::new(oversized)))); + assert_eq!((state.encoded_left, state.pixels_left), (bytes.len(), 4)); + assert!(state.raster(&usvg::ImageKind::PNG(Arc::new(bytes)))); + assert_eq!((state.encoded_left, state.pixels_left), (0, 0)); + let usage = state.usage(); + assert_eq!((usage.raster_pixels, usage.raster_limit_images, usage.invalid_raster_images), (4, 1, 0)); +} + +#[test] +fn encoded_raster_exhaustion_keeps_ordinary_drawing_and_reports_only_omitted_images() { + let bytes = png(); + let href = data_url("image/png", &bytes); + let text = format!( + r##""## + ); + let resolver = ValidatedImageResolver::new(test_limits(8)); + resolver.state.lock().unwrap().encoded_left = bytes.len(); + let tree = resolver.parse_preflight(preflight(&text, &resolver.limits()).unwrap(), usvg::Options::default()).unwrap(); + let usage = resolver.usage(); + assert_eq!((usage.raster_pixels, usage.raster_limit_images, usage.invalid_raster_images), (4, 1, 0)); + let output = super::super::draw::render(&tree, usvg::Transform::identity(), 4, 2, 0, 1 << 20).unwrap(); + let background = output.pixel(3, 1).unwrap().demultiply(); + assert_eq!((background.red(), background.green(), background.blue(), background.alpha()), (0, 255, 0, 255)); + let pixel = output.pixel(0, 0).unwrap().demultiply(); + assert_eq!((pixel.red(), pixel.green(), pixel.blue(), pixel.alpha()), (17, 83, 149, 255)); + assert_eq!(resolver.state.lock().unwrap().encoded_left, 0); +} + +// Preserve upstream's text_rechecked boundary using the integrated real resolver. +// Plain and gzip retries spend source bytes before decoding, independently of +// inflation work and the drawing's genuine decoded-content capacity. +#[test] +fn plain_legacy_encoding_rechecks_have_a_separate_bounded_work_allowance() { + on_test_stack(|| { + let xml = b""; + for compressed in [false, true] { + let (options, state, _resolver) = wrapped_options(PIXELS); + let encode = |bytes: &[u8]| { + if compressed { + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder.write_all(bytes).unwrap(); + encoder.finish().unwrap() + } else { + bytes.to_vec() + } + }; + assert!((options.image_href_resolver.resolve_data)("image/svg+xml", Arc::new(encode(&[0xff])), &options).is_none()); + state.lock().unwrap_or_else(PoisonError::into_inner).classification_text_bytes_left = xml.len(); + assert!((options.image_href_resolver.resolve_data)("image/svg+xml", Arc::new(encode(xml)), &options).is_none()); + { + let state = state.lock().unwrap_or_else(PoisonError::into_inner); + assert_eq!(state.classification_text_bytes_left, 0); + assert_eq!(state.svg_resource_limits, 0, "exactly fitting text recheck is permitted"); + assert_eq!(state.classification_bytes_left, SVG_BYTES - usize::from(compressed) * xml.len()); + } + assert!((options.image_href_resolver.resolve_data)("image/svg+xml", Arc::new(encode(xml)), &options).is_none()); + let state = state.lock().unwrap_or_else(PoisonError::into_inner); + assert_eq!(state.classification_text_bytes_left, 0); + assert_eq!(state.svg_resource_limits, 1, "the next source cannot repeat text-conversion work"); + assert_eq!(state.invalid_svg, 1, "the original encoding error stays invalid"); + assert_eq!(state.svg_limits, 2); + assert_eq!(state.usage().embedded_svg_count, 0, "rechecked children are never admitted"); + } + Ok(()) + }) + .unwrap(); +} + +#[test] +fn embedded_gzip_obeys_document_bytes_before_aggregate_inflation_room() { + on_test_stack(|| { + let mut text = b"".len(), b' '); + text.extend_from_slice(b"-->"); + assert_eq!(text.len(), 256); + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder.write_all(&text).unwrap(); + let encoded = Arc::new(encoder.finish().unwrap()); + for bytes in [text.len() - 1, text.len()] { + let limits = Limits { bytes, embedded_bytes: 1024, ..test_limits(4) }; + assert!(limits.embedded_bytes > text.len()); + let resolver = ValidatedImageResolver::new(limits); + let options = usvg::Options { image_href_resolver: resolver.resolver(), ..Default::default() }; + let image = (options.image_href_resolver.resolve_data)("image/svg+xml", Arc::clone(&encoded), &options); + let usage = resolver.usage(); + if bytes == text.len() { + assert!(matches!(image, Some(usvg::ImageKind::SVG(_))), "exact document boundary remains admissible"); + assert_eq!((usage.embedded_svg_count, usage.embedded_svg_bytes), (1, text.len())); + assert_eq!((usage.svg_resource_limit_images, usage.invalid_svg_images), (0, 0)); + } else { + assert!(image.is_none(), "aggregate room must not lift one document's inflate limit"); + assert_eq!((usage.embedded_svg_count, usage.svg_resource_limit_images, usage.invalid_svg_images), (0, 1, 0)); + assert_eq!(usage.embedded_svg_bytes, 1024, "refused work retains its aggregate charge"); + } + } + Ok(()) + }) + .unwrap(); +} diff --git a/crates/svg/src/tree/validated/xml_tests.rs b/crates/svg/src/tree/validated/xml_tests.rs new file mode 100644 index 000000000..b6b594433 --- /dev/null +++ b/crates/svg/src/tree/validated/xml_tests.rs @@ -0,0 +1,176 @@ +//! Public validated-context contracts; private byte/raster accounting lives in tests.rs. +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)] +use super::super::{Limits, PREFLIGHT_PASSES, preflight, usvg}; +use super::ValidatedImageResolver; +use std::io::Write; +use std::sync::{ + Arc, + atomic::{AtomicUsize, Ordering}, +}; + +const NS: &str = "http://www.w3.org/2000/svg"; + +fn drawing(contents: &str) -> String { + format!("{contents}") +} + +fn url(bytes: &[u8]) -> String { + use std::fmt::Write; + let mut value = String::from("data:image/svg+xml,"); + for byte in bytes { + write!(&mut value, "%{byte:02X}").unwrap(); + } + value +} + +fn gzip(bytes: &[u8]) -> Vec { + let mut encoder = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + encoder.write_all(bytes).unwrap(); + encoder.finish().unwrap() +} + +fn image(href: &str) -> String { + format!("") +} + +fn parse_count(text: &str, limits: Limits) -> (super::super::ImageUsage, usize) { + PREFLIGHT_PASSES.with(|count| count.set(0)); + let context = ValidatedImageResolver::new(limits); + let checked = preflight(text, &limits).unwrap(); + let tree = context.parse_preflight(checked, usvg::Options::default()).unwrap(); + assert!(!tree.root().children().is_empty(), "the visible drawing survives validation"); + (context.usage(), PREFLIGHT_PASSES.with(|count| count.get())) +} + +#[test] +fn validated_root_reuses_checked_xml_and_rechecks_only_changed_attributes() { + let ordinary = drawing(""); + let repaired = drawing(""); + let (usage, count) = parse_count(&ordinary, Limits::default()); + assert_eq!(count, 1); + assert_eq!(usage.invalid_raster_images, 0); + let (usage, count) = parse_count(&repaired, Limits::default()); + assert_eq!(count, 2); + assert_eq!(usage.invalid_raster_images, 1); +} + +#[test] +fn validated_plain_and_gzip_children_reuse_their_original_preflight() { + for repaired in [false, true] { + let mut contents = String::from(""); + if repaired { + contents.push_str(""); + } + let child = drawing(&contents); + for compressed in [false, true] { + let encoded = if compressed { gzip(child.as_bytes()) } else { child.as_bytes().to_vec() }; + let root = drawing(&image(&url(&encoded))); + let (usage, count) = parse_count(&root, Limits::default()); + assert_eq!(count, if repaired { 3 } else { 2 }, "compressed={compressed}, repaired={repaired}"); + assert_eq!(usage.embedded_svg_count, 1); + assert_eq!(usage.embedded_svg_bytes, child.len()); + assert_eq!(usage.invalid_raster_images, usize::from(repaired)); + assert_eq!(usage.invalid_svg_images, 0); + } + } +} + +#[test] +fn validated_original_xml_guards_precede_normalization_and_conversion() { + let normal = drawing(""); + let entity = format!("]>{}", drawing("")); + let markup = format!("'>]>{}", drawing("&a;")); + let referenced = drawing(""); + for (text, limits) in [ + (&normal, Limits { bytes: normal.len() - 1, ..Default::default() }), + (&normal, Limits { attributes: 0, ..Default::default() }), + (&normal, Limits { xml_nodes: 1, ..Default::default() }), + (&entity, Limits { entity_bytes: 1, ..Default::default() }), + (&markup, Limits { refuse_entity_markup: true, ..Default::default() }), + (&referenced, Limits { tree_depth: 2, ..Default::default() }), + ] { + assert!(preflight(text, &limits).is_err(), "original source must be refused before normalization: {text}"); + } + let limits = Limits { entity_bytes: 1024, ..Default::default() }; + let context = ValidatedImageResolver::new(limits); + let checked = preflight(&entity, &limits).unwrap(); + context.parse_preflight(checked, usvg::Options::default()).unwrap(); + assert_eq!(context.usage().invalid_raster_images, 1, "admitted entity data reaches omission accounting"); +} + +#[test] +fn validated_context_requires_exact_preflight_limits() { + let text = drawing(""); + let limits = Limits { bytes: 4096, tree_depth: 16, embedded_bytes: 2048, ..Default::default() }; + let context = ValidatedImageResolver::new(limits); + assert_eq!(context.limits(), limits); + for changed in [ + Limits { bytes: 4095, ..limits }, + Limits { bytes: 4097, ..limits }, + Limits { tree_depth: 15, ..limits }, + Limits { embedded_bytes: 2049, ..limits }, + ] { + let checked = preflight(&text, &changed).unwrap(); + assert!(context.parse_preflight(checked, usvg::Options::default()).is_err()); + assert_eq!(context.usage(), Default::default(), "policy mismatch must not start conversion"); + } + context.parse_preflight(preflight(&text, &limits).unwrap(), usvg::Options::default()).unwrap(); +} + +fn hostile_options(calls: &Arc) -> usvg::Options<'static> { + let string_calls = Arc::clone(calls); + let data_calls = Arc::clone(calls); + usvg::Options { + image_href_resolver: usvg::ImageHrefResolver { + resolve_string: Box::new(move |_, _| { + string_calls.fetch_add(1, Ordering::SeqCst); + panic!("ambient string resolver must not run") + }), + resolve_data: Box::new(move |_, _, _| { + data_calls.fetch_add(1, Ordering::SeqCst); + panic!("caller data resolver must not bypass validation") + }), + }, + ..Default::default() + } +} + +#[test] +fn validated_root_and_nested_context_override_untrusted_image_callbacks() { + let calls = Arc::new(AtomicUsize::new(0)); + let child = drawing( + "", + ); + let root = drawing(&format!("{}", image(&url(child.as_bytes())))); + let limits = Limits::default(); + let context = ValidatedImageResolver::new(limits); + context.parse_preflight(preflight(&root, &limits).unwrap(), hostile_options(&calls)).unwrap(); + assert_eq!(calls.load(Ordering::SeqCst), 0); + assert_eq!(context.usage().embedded_svg_count, 1); + assert_eq!(context.usage().invalid_raster_images, 1); + let direct = ValidatedImageResolver::new(limits); + assert!(matches!(direct.resolve_text(&child, hostile_options(&calls)).unwrap(), Some(usvg::ImageKind::SVG(_)))); + assert_eq!(calls.load(Ordering::SeqCst), 0); + assert_eq!(direct.usage().invalid_raster_images, 1); +} + +#[test] +fn expanded_root_references_and_child_share_the_converted_depth_limit() { + let child = drawing(""); // height 3 + let href = url(child.as_bytes()); + let root = drawing(&format!("{}", image(&href))); // height 6 + assert!(preflight(&root, &Limits { tree_depth: 5, ..Default::default() }).is_err()); + assert!(preflight(&root, &Limits { tree_depth: 6, ..Default::default() }).is_ok()); + assert!(preflight(&child, &Limits { tree_depth: 3, ..Default::default() }).is_ok()); + for (depth, admitted) in [(8, false), (9, true)] { + let limits = Limits { tree_depth: depth, ..Default::default() }; + let checked = preflight(&root, &limits).unwrap(); + assert!(checked.depth() < 6, "source nesting is shallower than expanded references"); + let context = ValidatedImageResolver::new(limits); + context.parse_preflight(checked, usvg::Options::default()).unwrap(); + let usage = context.usage(); + assert_eq!(usage.embedded_svg_count, usize::from(admitted)); + assert_eq!(usage.svg_limit_images, usize::from(!admitted)); + assert_eq!(usage.invalid_svg_images, 0); + } +} From ca56374cf1bda5b7660ccd526048ac134fd52d58 Mon Sep 17 00:00:00 2001 From: Terra854 Date: Tue, 17 Nov 2026 19:07:31 +0800 Subject: [PATCH 013/161] Use shared preflight in MIME body regression --- crates/svg/src/tree/mime_header_tests.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/crates/svg/src/tree/mime_header_tests.rs b/crates/svg/src/tree/mime_header_tests.rs index 393a01729..c22ef7d4a 100644 --- a/crates/svg/src/tree/mime_header_tests.rs +++ b/crates/svg/src/tree/mime_header_tests.rs @@ -136,8 +136,8 @@ fn many_parameter_urls_preserve_bodies_and_base64_disposition() { ] { let text = format!(""); let repaired = repair(&text, &Limits { bytes: 2 << 20, ..Default::default() }).unwrap(); - let xml = checked_xml(&repaired, &Limits { bytes: 2 << 20, ..Default::default() }).unwrap(); - let value = xml.descendants().find(|node| node.has_tag_name("image")).unwrap().attribute("href").unwrap(); + let checked = super::super::preflight(&repaired, &Limits { bytes: 2 << 20, ..Default::default() }).unwrap(); + let value = checked.xml.descendants().find(|node| node.has_tag_name("image")).unwrap().attribute("href").unwrap(); assert!(!value.contains(";p0=")); assert_eq!(data_url::DataUrl::process(value).unwrap().decode_to_vec().unwrap().0, expected); assert!(matches!(repair(&repaired, &Limits { bytes: 2 << 20, ..Default::default() }).unwrap(), Cow::Borrowed(_))); From 6d0fa2c26dc05d7d909ec33051d05f4c40a740d5 Mon Sep 17 00:00:00 2001 From: Terra854 Date: Tue, 17 Nov 2026 19:09:27 +0800 Subject: [PATCH 014/161] Parse lockfile dependency graphs as TOML documents --- crates/svg/src/tree/draw/decoder_edges.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/svg/src/tree/draw/decoder_edges.rs b/crates/svg/src/tree/draw/decoder_edges.rs index 8258b5792..7d0c072d6 100644 --- a/crates/svg/src/tree/draw/decoder_edges.rs +++ b/crates/svg/src/tree/draw/decoder_edges.rs @@ -16,7 +16,7 @@ pub(crate) struct LockGraph { } impl LockGraph { pub(crate) fn parse(text: &str) -> Self { - let lock: toml::Value = text.parse().unwrap(); + let lock: toml::Value = toml::from_str(text).unwrap(); let packages = lock .get("package") .unwrap() From 76a27e21d72b9cd70c2d6204f91ad3544c3e8ee9 Mon Sep 17 00:00:00 2001 From: Terra854 Date: Tue, 17 Nov 2026 19:12:08 +0800 Subject: [PATCH 015/161] Keep validated policy storage within the renderer feature --- crates/svg/src/tree.rs | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/crates/svg/src/tree.rs b/crates/svg/src/tree.rs index 80feec75d..690320e82 100644 --- a/crates/svg/src/tree.rs +++ b/crates/svg/src/tree.rs @@ -167,6 +167,7 @@ pub fn root_unitless_size(svg: &str, limits: &Limits) -> Result { + #[cfg(feature = "tree-render")] limits: Limits, xml: roxmltree::Document<'s>, depth: usize, @@ -248,7 +249,13 @@ pub fn preflight<'s>(svg: &'s str, limits: &Limits) -> Result, Err let Some(height) = height(&xml, limits.tree_depth)? else { return Err(Error::TooDeep(limits.tree_depth)); }; - Ok(Preflight { limits: *limits, xml, depth, height }) + Ok(Preflight { + #[cfg(feature = "tree-render")] + limits: *limits, + xml, + depth, + height, + }) } /// Dimensions of an SVG viewport in SVG user units. From 96dd8c4e0996b3d85edf5c98070c261892ac30f3 Mon Sep 17 00:00:00 2001 From: Terra854 Date: Tue, 17 Nov 2026 19:14:47 +0800 Subject: [PATCH 016/161] Disambiguate the SVG writer documentation link --- crates/svg/src/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/svg/src/lib.rs b/crates/svg/src/lib.rs index ccb2b6837..79a706052 100644 --- a/crates/svg/src/lib.rs +++ b/crates/svg/src/lib.rs @@ -8,7 +8,7 @@ //! - `paths` (feature `paths`, on by default): the geometry of a document as Bézier paths in //! millimetres, for cutters, plotters and CAD import. //! - [`color`]: CSS colours. -//! - [`write`]: a writer for SVG documents, their path data and numbers. +//! - [`mod@write`]: a writer for SVG documents, their path data and numbers. //! - `tree` (feature `tree`): the full parser. usvg at the one version the Craft apps share, //! re-exported, behind an entry point that runs the checks above before it parses. //! From 8d5b02457ef0986742465cf69aefeb4dd1d2e7c8 Mon Sep 17 00:00:00 2001 From: r31lin <257763872+r31lin@users.noreply.github.com> Date: Tue, 17 Nov 2026 12:21:21 +0100 Subject: [PATCH 017/161] document-bundle: turn a non-zip file away after four bytes in read_preview_from The zip reader searches a whole file for an end-of-directory record before it gives up on a file that isn't a zip. DesignCraft reads the preview of every recent file this way before the start screen's first frame: 2.6 s for a local 110 MB InDesign document, 43 s on a network share. read_preview_from now checks the zip local-file header first. Fixes #2031 --- crates/document-bundle/src/lib.rs | 48 ++++++++++++++++++++++++++++++- 1 file changed, 47 insertions(+), 1 deletion(-) diff --git a/crates/document-bundle/src/lib.rs b/crates/document-bundle/src/lib.rs index 70a960a1f..845381f30 100644 --- a/crates/document-bundle/src/lib.rs +++ b/crates/document-bundle/src/lib.rs @@ -316,8 +316,19 @@ pub fn read_preview(bytes: &[u8], preview_entry: &str) -> Option> { } /// Read a bounded preview from a seekable file without loading the full bundle into memory. -pub fn read_preview_from(reader: R, preview_entry: &str) -> Option> { +/// +/// A file that does not start with a zip local-file header is turned away after four bytes. The zip +/// reader would otherwise search the whole file for an end-of-directory record it does not have, +/// which takes seconds for a large non-bundle file (an InDesign document in a recent-files list) +/// and minutes on a network share. +pub fn read_preview_from(mut reader: R, preview_entry: &str) -> Option> { const PREVIEW_LIMIT: u64 = 4 << 20; + let mut magic = [0u8; 4]; + reader.read_exact(&mut magic).ok()?; + if &magic != b"PK\x03\x04" { + return None; + } + reader.seek(std::io::SeekFrom::Start(0)).ok()?; let mut zip = zip::ZipArchive::new(reader).ok()?; let entry = zip.by_name(preview_entry).ok()?; if entry.size() > PREVIEW_LIMIT { @@ -409,6 +420,41 @@ mod tests { assert_eq!(read_preview(&bytes, SPEC.preview_entry).as_deref(), Some(&b"png"[..])); } + /// Counts the bytes read through it. + struct Counting { + inner: R, + read: usize, + } + + impl Read for Counting { + fn read(&mut self, buf: &mut [u8]) -> std::io::Result { + let n = self.inner.read(buf)?; + self.read += n; + Ok(n) + } + } + + impl Seek for Counting { + fn seek(&mut self, pos: std::io::SeekFrom) -> std::io::Result { + self.inner.seek(pos) + } + } + + #[test] + fn preview_from_a_reader_reads_a_bundle() { + let bytes = write(br#"{}"#, 4, "Test 0.1", Some(b"png"), &[], &[], SPEC).unwrap(); + assert_eq!(read_preview_from(Cursor::new(&bytes), SPEC.preview_entry).as_deref(), Some(&b"png"[..])); + } + + #[test] + fn preview_from_a_non_zip_reads_only_its_signature() { + let mut file = Counting { inner: Cursor::new(vec![0x42u8; 1 << 20]), read: 0 }; + assert_eq!(read_preview_from(&mut file, SPEC.preview_entry), None); + assert!(file.read <= 4, "read {} bytes of a file that is not a zip", file.read); + let mut empty = Counting { inner: Cursor::new(Vec::new()), read: 0 }; + assert_eq!(read_preview_from(&mut empty, SPEC.preview_entry), None); + } + #[test] fn extracts_legacy_inline_assets() { let loaded = read(br#"{"assetData":{"7":"AQID"},"name":"x"}"#, 4, SPEC).unwrap(); From 36e58391b1225a21b749bcf1a53d9be5ec961fbb Mon Sep 17 00:00:00 2001 From: Terra854 Date: Tue, 17 Nov 2026 19:34:40 +0800 Subject: [PATCH 018/161] Cover renderer backends and canonical Git dependency edges --- crates/svg/src/tree/draw/decoder_edges.rs | 60 +++++++- crates/svg/src/tree/draw/raster.rs | 4 + crates/svg/src/tree/draw/raster_tests.rs | 179 ++++++++++++++++++++++ 3 files changed, 242 insertions(+), 1 deletion(-) create mode 100644 crates/svg/src/tree/draw/raster_tests.rs diff --git a/crates/svg/src/tree/draw/decoder_edges.rs b/crates/svg/src/tree/draw/decoder_edges.rs index 7d0c072d6..ce6543b81 100644 --- a/crates/svg/src/tree/draw/decoder_edges.rs +++ b/crates/svg/src/tree/draw/decoder_edges.rs @@ -14,6 +14,12 @@ struct Package { pub(crate) struct LockGraph { packages: Vec, } +// Cargo dependency edges may omit the resolved Git fragment even though the +// selected package ID always records it. Other source identities stay exact. +fn source_matches(edge: &str, package: Option<&str>) -> bool { + let Some(package) = package else { return false }; + if edge.starts_with("git+") && !edge.contains('#') { package.split_once('#').map_or(package, |(base, _)| base) == edge } else { package == edge } +} impl LockGraph { pub(crate) fn parse(text: &str) -> Self { let lock: toml::Value = toml::from_str(text).unwrap(); @@ -55,7 +61,9 @@ impl LockGraph { let found: Vec<_> = self .packages .iter() - .filter(|p| p.id.name == name && version.is_none_or(|v| p.id.version == v) && source.is_none_or(|s| p.id.source.as_deref() == Some(s))) + .filter(|p| { + p.id.name == name && version.is_none_or(|v| p.id.version == v) && source.is_none_or(|s| source_matches(s, p.id.source.as_deref())) + }) .collect(); assert_eq!(found.len(), 1, "ambiguous or absent resolved edge {} from {from:?}", edges[0]); found[0].id.clone() @@ -125,3 +133,53 @@ fn selected_renderer_graph_rejects_a_split_validator_backend() { let split = make(true); assert!(std::panic::catch_unwind(|| assert_renderer_edges(&split, &split.workspace_package("craft-svg"))).is_err()); } + +#[test] +fn git_edges_resolve_full_package_ids_and_reject_canonical_ambiguity() { + let fixture = |second: bool| { + let mut text = String::from( + r#" +[[package]] +name = "consumer" +version = "1" +dependencies = ["codec 1 (git+https://example.invalid/codecs?rev=chosen)"] +[[package]] +name = "explicit" +version = "1" +dependencies = ["codec 1 (git+https://example.invalid/codecs?rev=chosen#aaaa)"] +[[package]] +name = "wrong" +version = "1" +dependencies = ["codec 1 (git+https://example.invalid/codecs?rev=chosen#cccc)"] +[[package]] +name = "codec" +version = "1" +source = "git+https://example.invalid/codecs?rev=chosen#aaaa" +"#, + ); + if second { + text.push_str( + r#" +[[package]] +name = "codec" +version = "1" +source = "git+https://example.invalid/codecs?rev=chosen#bbbb" +"#, + ); + } + LockGraph::parse(&text) + }; + let unique = fixture(false); + let selected = unique.dependency(&unique.workspace_package("consumer"), "codec"); + assert_eq!(selected.source.as_deref(), Some("git+https://example.invalid/codecs?rev=chosen#aaaa")); + assert_eq!(selected, unique.dependency(&unique.workspace_package("explicit"), "codec")); + assert!(std::panic::catch_unwind(|| unique.dependency(&unique.workspace_package("wrong"), "codec")).is_err()); + let ambiguous = fixture(true); + assert!(std::panic::catch_unwind(|| ambiguous.dependency(&ambiguous.workspace_package("consumer"), "codec")).is_err()); + assert_eq!(selected, ambiguous.dependency(&ambiguous.workspace_package("explicit"), "codec")); + assert!(!source_matches("registry+one", Some("registry+one#aaaa")), "registry sources remain exact"); + assert!( + !source_matches("git+https://example.invalid/codecs?rev=other", selected.source.as_deref()), + "requested Git revision remains part of the identity" + ); +} diff --git a/crates/svg/src/tree/draw/raster.rs b/crates/svg/src/tree/draw/raster.rs index fe250259d..ffeb0d684 100644 --- a/crates/svg/src/tree/draw/raster.rs +++ b/crates/svg/src/tree/draw/raster.rs @@ -138,3 +138,7 @@ fn webp_error(error: image_webp::DecodingError) -> RasterValidationFailure { _ => RasterValidationFailure::Invalid, } } + +#[cfg(test)] +#[path = "raster_tests.rs"] +mod raster_tests; diff --git a/crates/svg/src/tree/draw/raster_tests.rs b/crates/svg/src/tree/draw/raster_tests.rs new file mode 100644 index 000000000..a7783f8c5 --- /dev/null +++ b/crates/svg/src/tree/draw/raster_tests.rs @@ -0,0 +1,179 @@ +#![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::indexing_slicing)] +use super::*; +use craft_image_codecs::{ChannelLayout, Format, Image}; +use std::sync::Arc; + +fn kind(format: Format, bytes: Vec) -> usvg::ImageKind { + let bytes = Arc::new(bytes); + match format { + Format::Png => usvg::ImageKind::PNG(bytes), + Format::Jpeg => usvg::ImageKind::JPEG(bytes), + Format::Gif => usvg::ImageKind::GIF(bytes), + Format::WebP => usvg::ImageKind::WEBP(bytes), + _ => panic!("test raster format"), + } +} + +fn encoded(format: Format) -> Vec { + let image = Image::from_u8(2, 2, ChannelLayout::Rgb, [17, 83, 149].repeat(4)).unwrap(); + craft_image_codecs::encode(&image, format, &Default::default()).unwrap() +} + +fn segment(out: &mut Vec, marker: u8, payload: &[u8]) { + out.extend_from_slice(&[0xff, marker]); + out.extend_from_slice(&u16::try_from(payload.len() + 2).unwrap().to_be_bytes()); + out.extend_from_slice(payload); +} + +// ISO/IEC10918-1 framing, one gray 8x8 block. The single one-bit Huffman codes +// encode DC category 0 and AC EOB. No third-party asset or camera data is used. +fn gray_jpeg(bits: u8, entropy: &[u8]) -> Vec { + let mut data = vec![0xff, 0xd8]; + let mut quantization = vec![0]; + quantization.extend_from_slice(&[1; 64]); + segment(&mut data, 0xdb, &quantization); + segment(&mut data, if bits == 8 { 0xc0 } else { 0xc1 }, &[bits, 0, 8, 0, 8, 1, 1, 0x11, 0]); + let mut tables = Vec::new(); + for class in [0x00, 0x10] { + tables.extend_from_slice(&[class, 1]); + tables.extend_from_slice(&[0; 15]); + tables.push(0); + } + segment(&mut data, 0xc4, &tables); + segment(&mut data, 0xda, &[1, 1, 0, 0, 63, 0]); + data.extend_from_slice(entropy); + data.extend_from_slice(&[0xff, 0xd9]); + data +} + +fn crc32(bytes: &[u8]) -> u32 { + let mut value = !0u32; + for byte in bytes { + value ^= u32::from(*byte); + for _ in 0..8 { + value = (value >> 1) ^ (0xedb8_8320 & 0u32.wrapping_sub(value & 1)); + } + } + !value +} + +fn corrupt_png() -> Vec { + let mut data = encoded(Format::Png); + let mut at = 8; + loop { + let length = u32::from_be_bytes(data[at..at + 4].try_into().unwrap()) as usize; + if &data[at + 4..at + 8] == b"IDAT" { + // Keep the complete container and an honest CRC, but destroy the + // zlib header in the pixel stream rather than the dimension header. + assert!(length >= 2); + data[at + 8..at + 10].fill(0); + let crc = crc32(&data[at + 4..at + 8 + length]); + data[at + 8 + length..at + 12 + length].copy_from_slice(&crc.to_be_bytes()); + return data; + } + at += length + 12; + } +} + +fn corrupt_gif() -> Vec { + // Complete 2x2 GIF89a framing and palette, followed by a code7 when only + // codes 0..5 are initially defined by an LZW minimum code size of 2. + let mut data = b"GIF89a".to_vec(); + data.extend_from_slice(&[2, 0, 2, 0, 0x80, 0, 0]); + data.extend_from_slice(&[0, 0, 0, 0, 255, 0]); + data.extend_from_slice(&[0x2c, 0, 0, 0, 0, 2, 0, 2, 0, 0]); + data.extend_from_slice(&[2, 1, 7, 0, 0x3b]); + data +} + +fn corrupt_webp() -> Vec { + // Complete RIFF/VP8L dimensions, with no transform/Huffman/pixel bits. + // Chunk length and odd-byte padding remain internally consistent. + let mut data = b"RIFF".to_vec(); + data.extend_from_slice(&18u32.to_le_bytes()); + data.extend_from_slice(b"WEBPVP8L"); + data.extend_from_slice(&5u32.to_le_bytes()); + data.extend_from_slice(&[0x2f, 1, 0x40, 0, 0, 0]); + data +} + +#[test] +fn renderer_validator_accepts_complete_png_jpeg_gif_and_webp_at_exact_pixel_allowance() { + for format in [Format::Png, Format::Jpeg, Format::Gif, Format::WebP] { + let data = encoded(format); + assert_eq!(craft_image_codecs::probe_allocation_size(&data).unwrap(), (2, 2)); + let mut remaining = 4; + assert_eq!(validate_raster(&kind(format, data), 4, &mut remaining), Ok(()), "{format:?}"); + assert_eq!(remaining, 0, "{format:?}: exact area must be charged"); + } +} + +#[test] +fn renderer_validator_rejects_corrupt_pixels_after_valid_dimensions_without_refunding_work() { + let cases = [ + (Format::Png, corrupt_png(), (2, 2)), + // One-bit tables have no code starting with1. Stuffed all-one bits + // are an invalid entropy code, while SOF/DQT/DHT/SOS remain valid. + (Format::Jpeg, gray_jpeg(8, &[0xff, 0, 0xff, 0, 0xff, 0]), (8, 8)), + (Format::Gif, corrupt_gif(), (2, 2)), + (Format::WebP, corrupt_webp(), (2, 2)), + ]; + for (format, data, dimensions) in cases { + assert_eq!(craft_image_codecs::probe_allocation_size(&data).unwrap(), dimensions, "{format:?}"); + match format { + Format::Jpeg => { + let mut headers = zune_jpeg::JpegDecoder::new(Cursor::new(&data)); + headers.decode_headers().expect("intact JPEG scan headers before corrupt entropy"); + } + Format::Gif => { + gif::DecodeOptions::new().read_info(data.as_slice()).expect("intact GIF headers before corrupt LZW pixels"); + } + Format::WebP => { + image_webp::WebPDecoder::new(Cursor::new(&data)).expect("intact WebP headers before missing pixel bits"); + } + _ => {} + } + let pixels = u64::from(dimensions.0) * u64::from(dimensions.1); + let image = kind(format, data); + let mut remaining = pixels * 2; + for left in [pixels, 0] { + assert_eq!(validate_raster(&image, pixels, &mut remaining), Err(RasterValidationFailure::Invalid), "{format:?}"); + assert_eq!(remaining, left, "{format:?}: failed decode work is nonrefundable"); + } + assert_eq!(validate_raster(&image, pixels, &mut remaining), Err(RasterValidationFailure::Limit), "{format:?}"); + assert_eq!(remaining, 0); + } +} + +#[test] +fn twelve_bit_jpeg_is_valid_for_the_general_codec_but_not_the_svg_renderer() { + for bits in [8, 12] { + // DC=0, EOB=0, then six one-pad bits: a constant mid-gray block. + let data = gray_jpeg(bits, &[0x3f]); + let decoded = craft_image_codecs::decode(&data).unwrap(); + assert_eq!(decoded.dimensions(), (8, 8)); + let mut remaining = 64; + let expected = if bits == 8 { Ok(()) } else { Err(RasterValidationFailure::Invalid) }; + assert_eq!(validate_raster(&kind(Format::Jpeg, data), 64, &mut remaining), expected, "{bits}-bit"); + assert_eq!(remaining, 0, "capability refusal still spends attempted decode work"); + } +} + +#[test] +fn renderer_validator_refuses_unadmitted_headers_and_pixels_without_charging_unattempted_decode() { + for format in [Format::Png, Format::Jpeg, Format::Gif, Format::WebP] { + let image = kind(format, encoded(format)); + let mut remaining = 4; + assert_eq!(validate_raster(&image, 3, &mut remaining), Err(RasterValidationFailure::Limit), "{format:?}: per-image ceiling"); + assert_eq!(remaining, 4); + remaining = 3; + assert_eq!(validate_raster(&image, 4, &mut remaining), Err(RasterValidationFailure::Limit), "{format:?}: cumulative allowance"); + assert_eq!(remaining, 3); + assert_eq!( + validate_raster(&kind(format, b"invalid raster header".to_vec()), 4, &mut remaining), + Err(RasterValidationFailure::Invalid), + "{format:?}" + ); + assert_eq!(remaining, 3, "invalid dimensions cannot reserve an invented area"); + } +} From 8af8f407af4c81c80f05396343df0d238b97626b Mon Sep 17 00:00:00 2001 From: Terra854 Date: Tue, 17 Nov 2026 19:42:04 +0800 Subject: [PATCH 019/161] Use an observed JPEG decode failure fixture --- crates/svg/src/tree/draw/raster_tests.rs | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/crates/svg/src/tree/draw/raster_tests.rs b/crates/svg/src/tree/draw/raster_tests.rs index a7783f8c5..273705744 100644 --- a/crates/svg/src/tree/draw/raster_tests.rs +++ b/crates/svg/src/tree/draw/raster_tests.rs @@ -46,6 +46,18 @@ fn gray_jpeg(bits: u8, entropy: &[u8]) -> Vec { data } +fn corrupt_jpeg() -> Vec { + let mut data = gray_jpeg(8, &[0x3f]); + let scan = data.windows(2).position(|marker| marker == b"\xff\xda").unwrap(); + // Keep the complete scan header and valid entropy, but reference table 1 + // while DHT defines only table 0. The renderer accepts these headers, then + // fails pixel decoding when checking the selected Huffman tables. Random + // all-one entropy is tolerated by its default policy and is not this guard. + assert_eq!(data[scan + 6], 0); + data[scan + 6] = 0x11; + data +} + fn crc32(bytes: &[u8]) -> u32 { let mut value = !0u32; for byte in bytes { @@ -112,9 +124,8 @@ fn renderer_validator_accepts_complete_png_jpeg_gif_and_webp_at_exact_pixel_allo fn renderer_validator_rejects_corrupt_pixels_after_valid_dimensions_without_refunding_work() { let cases = [ (Format::Png, corrupt_png(), (2, 2)), - // One-bit tables have no code starting with1. Stuffed all-one bits - // are an invalid entropy code, while SOF/DQT/DHT/SOS remain valid. - (Format::Jpeg, gray_jpeg(8, &[0xff, 0, 0xff, 0, 0xff, 0]), (8, 8)), + // Header-only admission cannot detect this missing scan-table dependency. + (Format::Jpeg, corrupt_jpeg(), (8, 8)), (Format::Gif, corrupt_gif(), (2, 2)), (Format::WebP, corrupt_webp(), (2, 2)), ]; @@ -123,7 +134,7 @@ fn renderer_validator_rejects_corrupt_pixels_after_valid_dimensions_without_refu match format { Format::Jpeg => { let mut headers = zune_jpeg::JpegDecoder::new(Cursor::new(&data)); - headers.decode_headers().expect("intact JPEG scan headers before corrupt entropy"); + headers.decode_headers().expect("intact JPEG scan headers before unavailable pixel-decode table"); } Format::Gif => { gif::DecodeOptions::new().read_info(data.as_slice()).expect("intact GIF headers before corrupt LZW pixels"); From fcafab33d9fedbefb0992d7776db3b6ae8b918f6 Mon Sep 17 00:00:00 2001 From: Terra854 Date: Tue, 17 Nov 2026 20:09:30 +0800 Subject: [PATCH 020/161] Preserve the exact embedded SVG depth boundary --- crates/svg/README.md | 6 ++++-- crates/svg/src/tree/validated.rs | 7 +++--- crates/svg/src/tree/validated/xml_tests.rs | 25 +++++++++++++++++++++- 3 files changed, 32 insertions(+), 6 deletions(-) diff --git a/crates/svg/README.md b/crates/svg/README.md index 0810b1217..519015c1b 100644 --- a/crates/svg/README.md +++ b/crates/svg/README.md @@ -46,8 +46,10 @@ The caller supplies fonts and a sufficient stack/worker for its depth policy. `parse_preflight` accepts only an opaque preflight made with exactly this context's limits. It consumes options and installs its own data-only resolver, including for all children; unrelated callbacks cannot fetch a linked file or bypass admission. -The full expanded root height and each child's expanded height share one depth -allowance. An overlapping root conversion on the same context is refused. +Expanded root ancestor frames (checked height minus the callback leaf) and each +child's complete expanded height share one depth allowance. The callback leaf is +not counted twice; source nesting alone cannot hide expanded reference chains. +An overlapping root conversion on the same context is refused. Unchanged XML converts from its original checked DOM. A modified data URL requires one checked replacement DOM. Invalid URLs become bounded resolver-visible image diff --git a/crates/svg/src/tree/validated.rs b/crates/svg/src/tree/validated.rs index cb1b6e8c3..658b2641f 100644 --- a/crates/svg/src/tree/validated.rs +++ b/crates/svg/src/tree/validated.rs @@ -191,9 +191,10 @@ impl ValidatedImageResolver { if checked.limits != self.limits { return Err(Error::Parse("checked SVG limits differ from the validated image resolver".into())); } - // A shallow source can expand through use/paint references. Reserve the - // full checked converted height before any child callback, not source depth. - let _scope = self.root_scope(checked.height)?; + // A shallow source can expand through use/paint references. Reserve its + // expanded ancestor frames, excluding the callback leaf itself. The child + // contributes its complete checked height, as in legacy child admission. + let _scope = self.root_scope(checked.height.saturating_sub(1))?; options.image_href_resolver = self.resolver(); convert_checked(checked, &options) } diff --git a/crates/svg/src/tree/validated/xml_tests.rs b/crates/svg/src/tree/validated/xml_tests.rs index b6b594433..34e1e604e 100644 --- a/crates/svg/src/tree/validated/xml_tests.rs +++ b/crates/svg/src/tree/validated/xml_tests.rs @@ -162,7 +162,10 @@ fn expanded_root_references_and_child_share_the_converted_depth_limit() { assert!(preflight(&root, &Limits { tree_depth: 5, ..Default::default() }).is_err()); assert!(preflight(&root, &Limits { tree_depth: 6, ..Default::default() }).is_ok()); assert!(preflight(&child, &Limits { tree_depth: 3, ..Default::default() }).is_ok()); - for (depth, admitted) in [(8, false), (9, true)] { + // Expanded root height6 includes the callback leaf: reserve5 ancestors, + // then admit the child's complete height3. Source depth alone would wrongly + // accept the cap7 case, while reserving the leaf twice would reject cap8. + for (depth, admitted) in [(7, false), (8, true)] { let limits = Limits { tree_depth: depth, ..Default::default() }; let checked = preflight(&root, &limits).unwrap(); assert!(checked.depth() < 6, "source nesting is shallower than expanded references"); @@ -174,3 +177,23 @@ fn expanded_root_references_and_child_share_the_converted_depth_limit() { assert_eq!(usage.invalid_svg_images, 0); } } + +#[test] +fn plain_root_callback_leaf_does_not_spend_the_child_boundary_twice() { + let child = drawing(""); // height3 + let root = drawing(&image(&url(child.as_bytes()))); // height2, one ancestor + for (depth, admitted) in [(3, false), (4, true)] { + let limits = Limits { tree_depth: depth, ..Default::default() }; + assert!(preflight(&root, &limits).is_ok()); + assert!(preflight(&child, &limits).is_ok()); + let context = ValidatedImageResolver::new(limits); + let tree = context.parse_preflight(preflight(&root, &limits).unwrap(), usvg::Options::default()).unwrap(); + assert_eq!(context.usage().embedded_svg_count, usize::from(admitted)); + assert_eq!(context.usage().svg_limit_images, usize::from(!admitted)); + assert_eq!(context.usage().invalid_svg_images, 0); + let pixels = super::super::draw::render(&tree, usvg::Transform::identity(), 4, 4, 0, 1 << 20).unwrap(); + let sample = pixels.pixel(1, 1).unwrap(); + assert_eq!((sample.red(), sample.green(), sample.blue()), (0, 0, 0)); + assert_eq!(sample.alpha(), if admitted { 255 } else { 0 }, "exact-boundary child must actually draw"); + } +} From ff8c279854733aea4a5806838f19a5e81a76cfa7 Mon Sep 17 00:00:00 2001 From: Davi Augusto Date: Tue, 17 Nov 2026 10:02:44 -0300 Subject: [PATCH 021/161] craft-raw: Sony ZV-E10 in-file distortion The ZV-E10 joins APSC_BODIES with the ILCE-6700's rule (11 knots over its own default crop's half diagonal / 10 = 360.56 px, the corrected- radius index, the frame-filling zoom, the 0.5 px APS-C fit gate) and its own distortion unit, 16384: the ILCE-6700's 15000 over-corrects its 16 mm photos by about 9 %. APS-C bodies now carry their unit. Measured on 4 photos with the E PZ 16-50 mm at 16-32 mm against the camera JPEG: p95 0.26-0.64 px at 1440 px as rendered (uncorrected up to 18.7 px; unit 15000 up to 2.18 px). Ported from storytold/lightcraft#3743, overtaken by the craft-raw cutover. --- .../docs/sony-ilce7rm6-lens-corrections.md | 35 ++++++++++-- crates/raw/src/arw_lens_rm6.rs | 54 +++++++++++++++---- 2 files changed, 76 insertions(+), 13 deletions(-) diff --git a/crates/raw/docs/sony-ilce7rm6-lens-corrections.md b/crates/raw/docs/sony-ilce7rm6-lens-corrections.md index 216fc4b63..8a87bdc3b 100644 --- a/crates/raw/docs/sony-ilce7rm6-lens-corrections.md +++ b/crates/raw/docs/sony-ilce7rm6-lens-corrections.md @@ -7,7 +7,7 @@ chromatic aberration (CA). LightCraft reads them and turns distortion into a DNG reaches the same **Optics ▸ Enable Lens Corrections** (`optics.lens_profile`) as a DNG's embedded warp and Panasonic's RW2 distortion data. Lateral CA is parsed and converts to the warp's red and blue planes, but it is **off by default**: applied after demosaic it made the render's CA worse (Verification). Vignetting is read but not applied (below). The -conversion is verified on the **ILCE-7RM6** and, for distortion, the **ILCE-6700** ([ILCE-6700](#ilce-6700)); the +conversion is verified on the **ILCE-7RM6** and, for distortion, the **ILCE-6700** ([ILCE-6700](#ilce-6700)) and the **ZV-E10** ([ZV-E10](#zv-e10)); the ILCE-7RM4A has its own conversion, with other conventions ([Two conventions](#two-conventions)), and other Sony bodies' tables give no correction until they are verified. @@ -105,7 +105,7 @@ the photo (portrait files are covered) and is carried by DNG export, as RW2's is ## Which bodies get it Only bodies on one of two allowlists in `arw_lens_rm6.rs` get an opcode: `FULL_FRAME_CROP` (today `ILCE-7RM6`, -9984 × 6656) and `APSC_BODIES` (today `ILCE-6700`, below). The unit and the knot spacing are verified per body; a body +9984 × 6656) and `APSC_BODIES` (today `ILCE-6700` and `ZV-E10`, below, each with its distortion unit). The unit and the knot spacing are verified per body; a body with another sensor size or another unit would be warped wrongly. Other bodies get no correction from this conversion; the ILCE-7RM4A gets its own, with other conventions (below). To add a body: collect photos of it shot with distortion correction on (several lenses and focal lengths), repeat the camera-JPEG registration below against the uncorrected @@ -113,7 +113,7 @@ render, confirm the unit, the knot spacing and the zoom rule (or measure the bod default crop to `FULL_FRAME_CROP`, or an APS-C body whose knots span its own crop (as measured for the ILCE-6700) to `APSC_BODIES`, with a test; an APS-C body also needs its fit gate checked against the camera JPEG (see *ILCE-6700*). -For bodies in `APSC_BODIES` (today `ILCE-6700`), the knots span the file's own default crop: spacing = its half +For bodies in `APSC_BODIES` (today `ILCE-6700` and `ZV-E10`), the knots span the file's own default crop: spacing = its half diagonal / (knot count − 1). This crop-based rule applies only to that list. The ILCE-7RM6 keeps its sensor-fixed 400 px in both modes; in full frame (16 knots) the crop-based rule gives the same 400 px, and in its APS-C mode (11 knots) it would give 392 px, which was only an alternative measurement there (it fits as well, above). @@ -166,6 +166,29 @@ As rendered (`lightcraft-cli render`, Enable Lens Corrections on): At 11.7 mm and longer an exception of this kind fits worse than the body's rule (412 px / 14500 on the grid: 0.73–1.19 px against 0.48–0.69 px), so it stops at 10.5 mm, the longest measured focal length where it helps; 10.6–11.6 mm is unmeasured and takes the body's rule. +## ZV-E10 + +The ZV-E10 (APS-C, 6000 × 4000 default crop at (12, 12) in 6048 × 4024 storage) writes 11-knot tables like the +ILCE-6700, zero-padded to the same 17 values. Measured as above on 4 photos shot with distortion correction Auto (the +E PZ 16-50 mm F3.5-5.6 OSS, which forces it; its owner's photos, only the summary numbers here): 2 at 16 mm, 1 at 29 mm +(portrait) and 1 at 32 mm. Feature matches between the uncorrected render and the camera JPEG +(LightCraft's `tools/sony_distortion_check.py`, SIFT agreement after the best similarity), the camera's map applied to the render's +points in a scan of knot spacing 340–380 px and unit 15000–17800: + +| | p95 residual at 1440 px | +|---|---| +| Uncorrected | 2.7–18.7 px (16 mm: 18.6–18.7 px) | +| The ILCE-6700's rule (half diagonal / 10 = 360.56 px, unit 15000), as rendered | 0.45–2.18 px: the 16 mm photos over-corrected by about 9 % | +| Sensor-fixed 400 px, unit 16384 | 0.43–3.21 px | +| **Half diagonal / 10 = 360.56 px, unit 16384** (exact table) | **0.27–0.61 px** | +| The same as LightCraft renders it (`lightcraft-cli render`, Enable Lens Corrections on, fitted polynomial) | **0.26–0.64 px** (median residual 0.07–0.36 px) | + +At 360.56 px every unit from 16300 to 17000 lands within 0.60–0.64 px worst (the measurement's floor on these scenes); +spacing and unit trade off along a ridge as on the ILCE-6700 (375 px / 15700 and 380 px / 15400 reach 0.59 px), so the +body keeps the ILCE-6700's crop-based spacing and takes its own unit, 16384, in `APSC_BODIES`. The fitted polynomial +passes the 0.5 px APS-C fit gate on all four tables, the strongest (16 mm, −1905 at the last knot) included. An HDR +merge of a ZV-E10 bracket keeps the warp in the merged DNG's `OpcodeList3`, as for any raw with embedded corrections. + ## Two conventions A second Sony decoder, `src/arw_lens.rs` @@ -344,7 +367,7 @@ the resampled source by the source, framing and output size, so tone, colour and ## Limits -- **ILCE-7RM6 and ILCE-6700 only.** Other Sony bodies record the same tags; their knot spacing (defined here by the +- **ILCE-7RM6, ILCE-6700 and ZV-E10 only.** Other Sony bodies record the same tags; their knot spacing (defined here by the full-frame crop, or an APS-C body's own crop), unit and zoom rule are unverified, so they get no correction from this conversion (the ILCE-7RM4A's distortion has its own: [Two conventions](#two-conventions)). - **ILCE-6700 at strong barrel distortion:** the body's rule leaves 1.36–1.38 px at the Sigma 10–18 mm's 10 mm end; @@ -355,6 +378,10 @@ the resampled source by the source, framing and output size, so tone, colour and 6656 × 4608 frame, misses the 0.5 px fit gate, so that photo stays uncorrected (5.3 px). Other wide lenses with similarly strong tables are unmeasured. Its vignetting and lateral CA tables are read, not applied: measured in #2801 (the camera JPEG doesn't compensate vignetting, as on the ILCE-7RM6). +- **ZV-E10: one lens, four photos.** Only the E PZ 16-50 mm (16–32 mm) is measured. Its unit, 16384, differs from + the ILCE-6700's 15000; the ILCE-6700's own E 16–55 mm at 16 mm also fits a higher unit (16000, above), so the unit + may belong to the lens rather than the body. Other lenses on the ZV-E10 are unmeasured and take 16384. The + ZV-E10M2 (another sensor) is not covered. - **APS-C distortion is unverified in camera on the ILCE-7RM6:** no APS-C photo with distortion correction on was available; the APS-C knot spacing is verified on CA only, and the APS-C zoom follows the full-frame rule. - **Aspect-ratio crops** (in-camera 4:3, 16:9, 1:1) were not measured; the correction uses the crop the file records. diff --git a/crates/raw/src/arw_lens_rm6.rs b/crates/raw/src/arw_lens_rm6.rs index d3e3b7275..29b71d6b0 100644 --- a/crates/raw/src/arw_lens_rm6.rs +++ b/crates/raw/src/arw_lens_rm6.rs @@ -40,9 +40,10 @@ //! The correction is produced whatever the in-camera flag says. Only bodies listed in [`FULL_FRAME_CROP`] or //! [`APSC_BODIES`] get it: the unit and the knot spacing are verified on the ILCE-7RM6 and, for distortion, on the //! ILCE-6700 (APS-C, 11 knots spanning its own crop, the same unit 15000: 0.73 px worst over 27 photos with three -//! lenses as rendered, 1.38 px at the 10–18 mm's 10 mm end). Another body is added by repeating the measurement on its photos shot -//! with distortion correction on and listing its full-frame default crop (or, for an APS-C body, its model). The method -//! and the numbers: docs/sony-ilce7rm6-lens-corrections.md. +//! lenses as rendered, 1.38 px at the 10–18 mm's 10 mm end) and the ZV-E10 (the same APS-C rule with unit 16384: +//! 0.64 px worst over 4 photos with the E PZ 16–50 mm at 16–32 mm as rendered). Another body is added by repeating the +//! measurement on its photos shot with distortion correction on and listing its full-frame default crop (or, for an +//! APS-C body, its model). The method and the numbers: docs/sony-ilce7rm6-lens-corrections.md. use crate::opcodes::max_radius; use crate::shared::fit_radial_warp; @@ -59,9 +60,15 @@ pub(crate) const CA_FLAG: u16 = 0x7034; /// The bodies whose tables are verified (Exif `Model`), with their full-frame default crop (width, height) in px. const FULL_FRAME_CROP: &[(&str, (f64, f64))] = &[("ILCE-7RM6", (9984.0, 6656.0))]; -/// The verified APS-C bodies (Exif `Model`): their knots span the file's own default crop, the last knot at its corner. -/// Measured on the ILCE-6700 (docs/sony-ilce7rm6-lens-corrections.md → ILCE-6700). -const APSC_BODIES: &[&str] = &["ILCE-6700"]; +/// The verified APS-C bodies (Exif `Model`) with their distortion unit: their knots span the file's own default crop, +/// the last knot at its corner. Measured against the camera JPEG (docs/sony-ilce7rm6-lens-corrections.md → ILCE-6700, +/// ZV-E10): the ILCE-6700 at the ILCE-7RM6's 15000; the ZV-E10 at 16384, where 15000 over-corrects by about 9 %. +const APSC_BODIES: &[(&str, f64)] = &[("ILCE-6700", DISTORTION_UNIT), ("ZV-E10", 16384.0)]; + +/// The distortion unit of an [`APSC_BODIES`] body. +fn apsc_unit(model: &str) -> Option { + APSC_BODIES.iter().find(|(m, _)| *m == model.trim()).map(|&(_, unit)| unit) +} /// A lens whose tables follow its body's convention with another knot spacing and unit, at the focal lengths up to /// `max_focal_mm`; measured like the body, against the camera JPEG. @@ -153,7 +160,7 @@ fn knot_spacing(model: &str, knots: usize, crop: Rect) -> Option { } let gaps = knots.checked_sub(1).filter(|&n| n > 0)?; let half_diagonal = (crop.width as f64).hypot(crop.height as f64) / 2.0; - (APSC_BODIES.contains(&model) && crate::arw_lens::is_3_2(crop)).then(|| half_diagonal / gaps as f64) + (apsc_unit(model).is_some() && crate::arw_lens::is_3_2(crop)).then(|| half_diagonal / gaps as f64) } /// The knot table at radius `r` px: knot `i` at `i · spacing`, linear in between, the last segment extended beyond @@ -261,8 +268,8 @@ pub(crate) fn lens_opcodes(raw: &Ifd, model: &str, lens: Option<&str>, focal_mm: let Some(spacing) = knot_spacing(model, knots, crop) else { return Vec::new() }; let op = if let Some(e) = lens_exception(model, lens, focal_mm) { gated_warp_opcode(&tables, e.knot_spacing, e.unit, active, crop, false, APSC_FIT_ERROR_PX_AT_1440) - } else if APSC_BODIES.contains(&model.trim()) { - gated_warp_opcode(&tables, spacing, DISTORTION_UNIT, active, crop, false, APSC_FIT_ERROR_PX_AT_1440) + } else if let Some(unit) = apsc_unit(model) { + gated_warp_opcode(&tables, spacing, unit, active, crop, false, APSC_FIT_ERROR_PX_AT_1440) } else { warp_opcode(&tables, spacing, active, crop, false) }; @@ -310,6 +317,8 @@ mod tests { const FF_HQ_R: [f64; 4] = [0.9715153129, 0.02236266369, 0.0007060224765, 0.005666341563]; const FF_HQ_B: [f64; 4] = [0.97137946, 0.02277016389, 0.0003830082563, 0.005727820598]; const APSC_G: [f64; 4] = [1.01627551, -0.0618606692, 0.02768349621, -0.005595798468]; + /// The ZV-E10 16 mm example's G plane (unit 16384, knots over its crop's half diagonal / 10), from the same recipe. + const ZV_E10_G: [f64; 4] = [1.051925248, -0.1972416727, 0.09393523723, -0.0282770833]; const APSC_R: [f64; 4] = [1.016503991, -0.06206460153, 0.0278447625, -0.005644118171]; const APSC_B: [f64; 4] = [1.016155702, -0.06162477627, 0.02753315326, -0.005554915901]; @@ -629,6 +638,33 @@ mod tests { assert!(matches!(ops.as_slice(), [Opcode::WarpRectilinear { planes, .. }] if planes.len() == 1), "{ops:?}"); } + /// A ZV-E10 file (E PZ 16-50 mm at 16 mm): 11 knots, zero-padded as this body stores them. + const ZV_E10_16MM_DISTORTION: [i16; 17] = [11, 72, -7, -129, -266, -446, -640, -877, -1114, -1380, -1639, -1905, 0, 0, 0, 0, 0]; + /// The ZV-E10's decoded geometry: 6048 × 4024 storage, a 6000 × 4000 default crop at (12, 12). + const ZV_E10_ACTIVE: Rect = Rect { x: 0, y: 0, width: 6048, height: 4024 }; + const ZV_E10_CROP: Rect = Rect { x: 12, y: 12, width: 6000, height: 4000 }; + + /// The ZV-E10 follows the ILCE-6700's APS-C rule (knots over its own crop's half diagonal / 10, the APS-C fit + /// gate) with its own unit, 16384; its zero padding is not read as knots, and its successor (ZV-E10M2) stays + /// unverified. + #[test] + fn zv_e10_uses_the_apsc_rule_with_its_own_unit() { + let s = knot_spacing("ZV-E10", 11, ZV_E10_CROP).unwrap(); + assert!((s - 360.5551).abs() < 1e-3, "{s}"); + let ifd = sony_lens_ifd(&ZV_E10_16MM_DISTORTION, &APSC_VIGNETTING, &APSC_CA, [17, 2, 1]); + let t = read_tables(&ifd); + assert_eq!(t.distortion.as_ref().map(Vec::len), Some(11)); + let gated = gated_warp_opcode(&t, s, 16384.0, ZV_E10_ACTIVE, ZV_E10_CROP, false, APSC_FIT_ERROR_PX_AT_1440); + let ops = lens_opcodes(&ifd, " ZV-E10 ", Some("E PZ 16-50mm F3.5-5.6 OSS"), Some(16.0), ZV_E10_ACTIVE, ZV_E10_CROP); + assert_warp(ops.first().cloned(), [3011.5 / 6047.0, 0.5], &[ZV_E10_G]); + assert_eq!(ops, gated.into_iter().collect::>()); + // the ILCE-6700's unit would warp it differently + let at_15000 = gated_warp_opcode(&t, s, DISTORTION_UNIT, ZV_E10_ACTIVE, ZV_E10_CROP, false, APSC_FIT_ERROR_PX_AT_1440); + assert_ne!(ops.first(), at_15000.as_ref()); + assert_eq!(apsc_unit("ILCE-6700"), Some(DISTORTION_UNIT)); + assert!(lens_opcodes(&ifd, "ZV-E10M2", None, None, ZV_E10_ACTIVE, ZV_E10_CROP).is_empty()); + } + const SIGMA_10_18: &str = "10-18mm F2.8 DC DN | Contemporary 023"; /// The Sigma 10-18 mm exception: its spacing and unit at 10-10.5 mm, the body's rule from 11.7 mm (and without a From e97e73f5792e180d3b9227c3c1a10a9e21f56b14 Mon Sep 17 00:00:00 2001 From: Brandon Thomas Date: Tue, 17 Nov 2026 08:38:13 -0500 Subject: [PATCH 022/161] dex: Craft Anything Port Craft Anything f64ae3bf3c8863a982af1304e06de20653803968 reader and opcode decoder. --- crates/dex/src/dex.rs | 810 ++++++++++++++++++++++++++++++++++++++ crates/dex/src/dex_ops.rs | 640 ++++++++++++++++++++++++++++++ 2 files changed, 1450 insertions(+) create mode 100644 crates/dex/src/dex.rs create mode 100644 crates/dex/src/dex_ops.rs diff --git a/crates/dex/src/dex.rs b/crates/dex/src/dex.rs new file mode 100644 index 000000000..e46d97b5b --- /dev/null +++ b/crates/dex/src/dex.rs @@ -0,0 +1,810 @@ +//! Dalvik executables (DEX): header, string/type/proto/field/method ids, class definitions, +//! class data, code items with try/catch handlers, class annotations, a smali-like listing of +//! method bodies and a reference finder. Counts are checked against the file size, strings are +//! decoded from MUTF-8 under a total budget, and every read is bounds-checked. + +use crate::FormatError; +use crate::bytes::Reader; +use crate::dex_ops::{IndexKind, Insn, Payload}; +use serde::Serialize; + +pub const NO_INDEX: u32 = 0xffff_ffff; +/// Longest string decoded (UTF-16 units). +const MAX_STRING_UNITS: usize = 16 * 1024; +/// Total UTF-16 units decoded across all strings of one file. +const STRING_BUDGET: usize = 64 * 1024 * 1024; +/// Most members per class list. +const MAX_MEMBERS: usize = 65_536; +/// Most try blocks / handlers per method. +const MAX_TRIES: usize = 4096; +const MAX_HANDLERS: usize = 256; + +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize)] +pub struct Header { + pub version: String, + pub checksum: String, + /// Adler-32 recomputed over the file. + pub checksum_ok: bool, + /// SHA-1 field as stored (not recomputed). + pub signature: String, + pub file_size: u32, + pub header_size: u32, + pub endian_tag: String, + pub string_ids: u32, + pub type_ids: u32, + pub proto_ids: u32, + pub field_ids: u32, + pub method_ids: u32, + pub class_defs: u32, + pub data_size: u32, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +struct ProtoId { + shorty: u32, + ret: u32, + params_off: u32, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct MemberId { + pub class_idx: u16, + /// Type (fields) or proto (methods). + pub type_or_proto: u16, + pub name_idx: u32, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct ClassDef { + pub class_idx: u32, + pub access: u32, + pub superclass_idx: u32, + pub interfaces_off: u32, + pub source_file_idx: u32, + pub annotations_off: u32, + pub class_data_off: u32, + pub static_values_off: u32, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +pub struct EncodedField { + pub field_idx: u32, + pub access: u32, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +pub struct EncodedMethod { + pub method_idx: u32, + pub access: u32, + pub code_off: u32, +} + +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct ClassData { + pub static_fields: Vec, + pub instance_fields: Vec, + pub direct_methods: Vec, + pub virtual_methods: Vec, + pub complete: bool, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct Handler { + /// Exception type descriptor; `None` for catch-all. + pub exception: Option, + pub address: u32, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct Try { + pub start: u32, + pub end: u32, + pub handlers: Vec, +} + +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct Code { + pub registers: u16, + pub ins: u16, + pub outs: u16, + pub insns: Vec, + pub tries: Vec, + pub debug_info_off: u32, +} + +/// A parsed DEX file. +#[derive(Debug, Clone)] +pub struct Dex<'a> { + data: &'a [u8], + pub header: Header, + pub strings: Vec, + types: Vec, + protos: Vec, + pub fields: Vec, + pub methods: Vec, + pub classes: Vec, + pub warnings: Vec, +} + +fn adler32(data: &[u8]) -> u32 { + let (mut a, mut b) = (1u32, 0u32); + for chunk in data.chunks(5552) { + for &x in chunk { + a += u32::from(x); + b += a; + } + a %= 65521; + b %= 65521; + } + (b << 16) | a +} + +/// Decode modified UTF-8 (as used by DEX and class files) to a String. +pub fn mutf8(r: &Reader<'_>, mut at: usize, max_units: usize) -> (String, usize) { + let mut units: Vec = Vec::new(); + let mut consumed = 0usize; + while units.len() < max_units { + let Some(b0) = r.u8(at) else { break }; + if b0 == 0 { + break; + } + let (u, n) = if b0 < 0x80 { + (u16::from(b0), 1) + } else if b0 & 0xe0 == 0xc0 { + let b1 = r.u8(at + 1).unwrap_or(0); + ((u16::from(b0 & 0x1f) << 6) | u16::from(b1 & 0x3f), 2) + } else if b0 & 0xf0 == 0xe0 { + let b1 = r.u8(at + 1).unwrap_or(0); + let b2 = r.u8(at + 2).unwrap_or(0); + ((u16::from(b0 & 0x0f) << 12) | (u16::from(b1 & 0x3f) << 6) | u16::from(b2 & 0x3f), 3) + } else { + (0xfffd, 1) + }; + units.push(u); + at = at.saturating_add(n); + consumed += n; + } + (String::from_utf16_lossy(&units), consumed) +} + +/// Signed LEB128 at `off`: (value, bytes). +pub fn sleb128(r: &Reader<'_>, off: usize) -> Option<(i64, usize)> { + let mut result = 0i64; + let mut shift = 0u32; + for i in 0..10usize { + let b = r.u8(off.checked_add(i)?)?; + result |= i64::from(b & 0x7f).checked_shl(shift)?; + shift += 7; + if b & 0x80 == 0 { + if shift < 64 && b & 0x40 != 0 { + result |= -1i64 << shift; + } + return Some((result, i + 1)); + } + } + None +} + +/// `Lcom/x/Y;` → `com.x.Y`; `[I` → `int[]`. +pub fn java_name(desc: &str) -> String { + let dims = desc.chars().take_while(|&c| c == '[').count(); + let base = desc.get(dims..).unwrap_or(""); + let name = match base { + "V" => "void".to_string(), + "Z" => "boolean".to_string(), + "B" => "byte".to_string(), + "S" => "short".to_string(), + "C" => "char".to_string(), + "I" => "int".to_string(), + "J" => "long".to_string(), + "F" => "float".to_string(), + "D" => "double".to_string(), + b if b.starts_with('L') && b.ends_with(';') => b.get(1..b.len() - 1).unwrap_or("").replace('/', "."), + b => b.to_string(), + }; + format!("{name}{}", "[]".repeat(dims.min(255))) +} + +/// `com.x.Y` or `com/x/Y` or `Lcom/x/Y;` → `Lcom/x/Y;`. +pub fn to_descriptor(name: &str) -> String { + let n = name.trim(); + if n.starts_with('L') && n.ends_with(';') { n.to_string() } else { format!("L{};", n.replace('.', "/")) } +} + +/// Access flag names for a class, field or method. +pub fn access_flags(flags: u32, kind: &str) -> Vec<&'static str> { + let mut v = Vec::new(); + let table: &[(u32, &str)] = &[ + (0x1, "public"), + (0x2, "private"), + (0x4, "protected"), + (0x8, "static"), + (0x10, "final"), + (0x100, "native"), + (0x200, "interface"), + (0x400, "abstract"), + (0x800, "strictfp"), + (0x1000, "synthetic"), + (0x2000, "annotation"), + (0x4000, "enum"), + (0x10000, "constructor"), + (0x20000, "declared-synchronized"), + ]; + for (bit, name) in table { + if flags & bit != 0 { + v.push(*name); + } + } + match kind { + "method" => { + if flags & 0x20 != 0 { + v.push("synchronized"); + } + if flags & 0x40 != 0 { + v.push("bridge"); + } + if flags & 0x80 != 0 { + v.push("varargs"); + } + } + "field" => { + if flags & 0x40 != 0 { + v.push("volatile"); + } + if flags & 0x80 != 0 { + v.push("transient"); + } + } + _ => {} + } + v +} + +fn table(r: &Reader<'_>, count_off: usize, what: &str, entry: usize) -> Result<(usize, usize), FormatError> { + let n = r.u32_le(count_off).ok_or_else(|| FormatError::Truncated("header".into()))? as usize; + let off = r.u32_le(count_off + 4).ok_or_else(|| FormatError::Truncated("header".into()))? as usize; + if n == 0 { + return Ok((0, 0)); + } + let bytes = n.checked_mul(entry).ok_or_else(|| FormatError::Malformed(format!("{what} count overflows")))?; + if off.checked_add(bytes).is_none_or(|e| e > r.len()) { + return Err(FormatError::Malformed(format!("{n} {what} at offset {off} run past the end of the file"))); + } + Ok((n, off)) +} + +/// True when bytes start with a DEX magic. +pub fn is_dex(data: &[u8]) -> bool { + data.starts_with(b"dex\n") && data.len() >= 0x70 +} + +/// Parse a DEX file. +pub fn parse(data: &[u8]) -> Result, FormatError> { + let r = Reader::new(data); + if !data.starts_with(b"dex\n") { + return Err(FormatError::Malformed("not a DEX file (no dex magic)".into())); + } + if data.len() < 0x70 { + return Err(FormatError::Truncated("DEX header".into())); + } + let mut warnings = Vec::new(); + let version = r.cstr(4, 4).unwrap_or_default(); + let checksum = r.u32_le(8).unwrap_or(0); + let file_size = r.u32_le(32).unwrap_or(0); + let endian = r.u32_le(40).unwrap_or(0); + if endian != 0x1234_5678 { + return Err(FormatError::Unsupported(format!("byte-swapped DEX (endian tag {endian:#x})"))); + } + if file_size as usize != data.len() { + warnings.push(format!("header says {file_size} bytes, the file has {}", data.len())); + } + let sum_end = (file_size as usize).min(data.len()); + let checksum_ok = data.get(12..sum_end).is_some_and(|d| adler32(d) == checksum); + if !checksum_ok { + warnings.push("Adler-32 checksum does not match the content".into()); + } + let (ns, so) = table(&r, 56, "strings", 4)?; + let (nt, to) = table(&r, 64, "types", 4)?; + let (np, po) = table(&r, 72, "protos", 12)?; + let (nf, fo) = table(&r, 80, "fields", 8)?; + let (nm, mo) = table(&r, 88, "methods", 8)?; + let (nc, co) = table(&r, 96, "classes", 32)?; + let header = Header { + version, + checksum: format!("{checksum:08x}"), + checksum_ok, + signature: r.slice(12, 20).map(|b| b.iter().map(|x| format!("{x:02x}")).collect()).unwrap_or_default(), + file_size, + header_size: r.u32_le(36).unwrap_or(0), + endian_tag: format!("{endian:#x}"), + string_ids: ns as u32, + type_ids: nt as u32, + proto_ids: np as u32, + field_ids: nf as u32, + method_ids: nm as u32, + class_defs: nc as u32, + data_size: r.u32_le(104).unwrap_or(0), + }; + let mut strings = Vec::with_capacity(ns); + let mut budget = STRING_BUDGET; + let mut over_budget = false; + for i in 0..ns { + let off = r.u32_le(so + i * 4).unwrap_or(0) as usize; + let s = match r.uleb128(off) { + Some((len, n)) if budget > 0 => { + let want = (len as usize).min(MAX_STRING_UNITS).min(budget); + let (s, _) = mutf8(&r, off.saturating_add(n), want); + budget = budget.saturating_sub(want.max(1)); + s + } + Some(_) => { + over_budget = true; + format!("") + } + None => format!(""), + }; + strings.push(s); + } + if over_budget { + warnings.push("string decoding budget reached; later strings are shown as ".into()); + } + let types = (0..nt).map(|i| r.u32_le(to + i * 4).unwrap_or(NO_INDEX)).collect(); + let protos = (0..np) + .map(|i| { + let o = po + i * 12; + ProtoId { shorty: r.u32_le(o).unwrap_or(0), ret: r.u32_le(o + 4).unwrap_or(NO_INDEX), params_off: r.u32_le(o + 8).unwrap_or(0) } + }) + .collect(); + let member = + |o: usize| MemberId { class_idx: r.u16_le(o).unwrap_or(0), type_or_proto: r.u16_le(o + 2).unwrap_or(0), name_idx: r.u32_le(o + 4).unwrap_or(NO_INDEX) }; + let fields = (0..nf).map(|i| member(fo + i * 8)).collect(); + let methods = (0..nm).map(|i| member(mo + i * 8)).collect(); + let classes = (0..nc) + .map(|i| { + let o = co + i * 32; + let g = |k: usize| r.u32_le(o + k * 4).unwrap_or(0); + ClassDef { + class_idx: g(0), + access: g(1), + superclass_idx: g(2), + interfaces_off: g(3), + source_file_idx: g(4), + annotations_off: g(5), + class_data_off: g(6), + static_values_off: g(7), + } + }) + .collect(); + Ok(Dex { data, header, strings, types, protos, fields, methods, classes, warnings }) +} + +impl<'a> Dex<'a> { + fn r(&self) -> Reader<'a> { + Reader::new(self.data) + } + + pub fn string(&self, i: u32) -> &str { + self.strings.get(i as usize).map(String::as_str).unwrap_or("") + } + + /// The descriptor of type `i` (`Lcom/x/Y;`). + pub fn type_desc(&self, i: u32) -> String { + match self.types.get(i as usize) { + Some(&s) => self.string(s).to_string(), + None => format!(""), + } + } + + fn type_list(&self, off: u32) -> Vec { + if off == 0 { + return Vec::new(); + } + let r = self.r(); + let n = r.u32_le(off as usize).unwrap_or(0) as usize; + (0..n.min(256)).filter_map(|i| r.u16_le(off as usize + 4 + i * 2).map(u32::from)).collect() + } + + /// `(Ljava/lang/String;I)V` + pub fn proto_desc(&self, i: u32) -> String { + match self.protos.get(i as usize) { + Some(p) => { + let params: String = self.type_list(p.params_off).into_iter().map(|t| self.type_desc(t)).collect(); + format!("({params}){}", self.type_desc(p.ret)) + } + None => format!(""), + } + } + + /// Parameter type descriptors and return type of proto `i`. + pub fn proto_parts(&self, i: u32) -> (Vec, String) { + match self.protos.get(i as usize) { + Some(p) => (self.type_list(p.params_off).into_iter().map(|t| self.type_desc(t)).collect(), self.type_desc(p.ret)), + None => (Vec::new(), "?".into()), + } + } + + pub fn shorty(&self, i: u32) -> &str { + self.protos.get(i as usize).map(|p| self.string(p.shorty)).unwrap_or("") + } + + /// `Lcls;->name:Ltype;` + pub fn field_ref(&self, i: u32) -> String { + match self.fields.get(i as usize) { + Some(f) => format!("{}->{}:{}", self.type_desc(u32::from(f.class_idx)), self.string(f.name_idx), self.type_desc(u32::from(f.type_or_proto))), + None => format!(""), + } + } + + /// `Lcls;->name(…)R` + pub fn method_ref(&self, i: u32) -> String { + match self.methods.get(i as usize) { + Some(m) => format!("{}->{}{}", self.type_desc(u32::from(m.class_idx)), self.string(m.name_idx), self.proto_desc(u32::from(m.type_or_proto))), + None => format!(""), + } + } + + pub fn class_name(&self, c: &ClassDef) -> String { + self.type_desc(c.class_idx) + } + + pub fn find_class(&self, name: &str) -> Option<&ClassDef> { + let want = to_descriptor(name); + self.classes.iter().find(|c| self.class_name(c) == want) + } + + pub fn interfaces(&self, c: &ClassDef) -> Vec { + self.type_list(c.interfaces_off).into_iter().map(|t| self.type_desc(t)).collect() + } + + /// Type descriptors of the class-level annotations. + pub fn annotations(&self, c: &ClassDef) -> Vec { + let r = self.r(); + let mut out = Vec::new(); + if c.annotations_off == 0 { + return out; + } + let Some(set_off) = r.u32_le(c.annotations_off as usize) else { return out }; + if set_off == 0 { + return out; + } + let n = r.u32_le(set_off as usize).unwrap_or(0) as usize; + for i in 0..n.min(256) { + let Some(item) = r.u32_le(set_off as usize + 4 + i * 4) else { break }; + // visibility u8, then encoded_annotation: uleb type_idx … + let vis = r.u8(item as usize).unwrap_or(0); + if let Some((t, _)) = r.uleb128(item as usize + 1) { + let v = match vis { + 0 => "build", + 1 => "runtime", + 2 => "system", + _ => "?", + }; + out.push(format!("{} ({v})", self.type_desc(t as u32))); + } + } + out + } + + pub fn class_data(&self, c: &ClassDef) -> ClassData { + let mut cd = ClassData::default(); + if c.class_data_off == 0 { + cd.complete = true; + return cd; + } + let r = self.r(); + let mut pos = c.class_data_off as usize; + let next = |pos: &mut usize| -> Option { + let (v, n) = r.uleb128(*pos)?; + *pos += n; + u32::try_from(v).ok() + }; + let (Some(sf), Some(inf), Some(dm), Some(vm)) = (next(&mut pos), next(&mut pos), next(&mut pos), next(&mut pos)) else { return cd }; + let remaining = self.data.len().saturating_sub(pos); + let total = [sf, inf, dm, vm].iter().map(|&x| x as usize).sum::(); + if total > remaining || total > MAX_MEMBERS * 4 { + return cd; + } + for (count, is_method, slot) in [(sf, false, 0), (inf, false, 1), (dm, true, 2), (vm, true, 3)] { + let mut idx: u32 = 0; + for _ in 0..count.min(MAX_MEMBERS as u32) { + let Some(diff) = next(&mut pos) else { return cd }; + let Some(access) = next(&mut pos) else { return cd }; + idx = match idx.checked_add(diff) { + Some(v) => v, + None => return cd, + }; + if is_method { + let Some(code_off) = next(&mut pos) else { return cd }; + let m = EncodedMethod { method_idx: idx, access, code_off }; + if slot == 2 { + cd.direct_methods.push(m); + } else { + cd.virtual_methods.push(m); + } + } else { + let f = EncodedField { field_idx: idx, access }; + if slot == 0 { + cd.static_fields.push(f); + } else { + cd.instance_fields.push(f); + } + } + } + } + cd.complete = true; + cd + } + + /// The code item at `off`. + pub fn code(&self, off: u32) -> Option { + if off == 0 { + return None; + } + let r = self.r(); + let o = off as usize; + let registers = r.u16_le(o)?; + let ins = r.u16_le(o + 2)?; + let outs = r.u16_le(o + 4)?; + let tries_size = usize::from(r.u16_le(o + 6)?); + let debug_info_off = r.u32_le(o + 8)?; + let n = r.u32_le(o + 12)? as usize; + let bytes = r.slice(o + 16, n.checked_mul(2)?)?; + let insns: Vec = bytes.as_chunks::<2>().0.iter().map(|c| u16::from_le_bytes(*c)).collect(); + let mut tries = Vec::new(); + let mut t_off = o + 16 + n * 2; + if tries_size > 0 && n % 2 == 1 { + t_off += 2; + } + let handlers_base = t_off + tries_size * 8; + for i in 0..tries_size.min(MAX_TRIES) { + let at = t_off + i * 8; + let (Some(start), Some(count), Some(h_off)) = (r.u32_le(at), r.u16_le(at + 4), r.u16_le(at + 6)) else { break }; + let mut handlers = Vec::new(); + let mut p = handlers_base + usize::from(h_off); + if let Some((size, k)) = sleb128(&r, p) { + p += k; + for _ in 0..size.unsigned_abs().min(MAX_HANDLERS as u64) { + let Some((t, k1)) = r.uleb128(p) else { break }; + let Some((addr, k2)) = r.uleb128(p + k1) else { break }; + p += k1 + k2; + handlers.push(Handler { exception: Some(self.type_desc(t as u32)), address: addr as u32 }); + } + if size <= 0 + && let Some((addr, _)) = r.uleb128(p) + { + handlers.push(Handler { exception: None, address: addr as u32 }); + } + } + tries.push(Try { start, end: start.saturating_add(u32::from(count)), handlers }); + } + Some(Code { registers, ins, outs, insns, tries, debug_info_off }) + } + + /// Name of method `i`. + pub fn method_name(&self, i: u32) -> &str { + self.methods.get(i as usize).map(|m| self.string(m.name_idx)).unwrap_or("") + } + + /// Java-like signature: `public static void log(java.lang.String)`. + pub fn method_signature(&self, m: &EncodedMethod) -> String { + let Some(id) = self.methods.get(m.method_idx as usize) else { return format!("", m.method_idx) }; + let (params, ret) = self.proto_parts(u32::from(id.type_or_proto)); + let flags = access_flags(m.access, "method").into_iter().filter(|f| *f != "constructor").collect::>().join(" "); + let params = params.iter().map(|p| java_name(p)).collect::>().join(", "); + format!("{}{}{} {}({params})", flags, if flags.is_empty() { "" } else { " " }, java_name(&ret), self.string(id.name_idx)).trim().to_string() + } + + /// Render one instruction smali-style. + pub fn format_insn(&self, i: &Insn) -> String { + let mut parts: Vec = Vec::new(); + if !i.regs.is_empty() { + let is_list = i.name.starts_with("invoke") || i.name.starts_with("filled-new-array"); + if i.range { + match (i.regs.first(), i.regs.last()) { + (Some(a), Some(b)) => parts.push(format!("{{v{a} .. v{b}}}")), + _ => parts.push("{}".into()), + } + } else if is_list { + parts.push(format!("{{{}}}", i.regs.iter().map(|r| format!("v{r}")).collect::>().join(", "))); + } else { + parts.extend(i.regs.iter().map(|r| format!("v{r}"))); + } + } else if i.range || i.name.starts_with("invoke") { + parts.push("{}".into()); + } + if let Some(l) = i.literal { + parts.push(if (-9..=9).contains(&l) { format!("{l}") } else { format!("{l:#x}") }); + } + if let Some(idx) = i.index { + parts.push(match i.index_kind { + IndexKind::String => format!("{:?}", self.string(idx)), + IndexKind::Type => self.type_desc(idx), + IndexKind::Field => self.field_ref(idx), + IndexKind::Method => self.method_ref(idx), + IndexKind::CallSite => format!("call_site@{idx}"), + IndexKind::MethodHandle => format!("method_handle@{idx}"), + IndexKind::Proto => self.proto_desc(idx), + IndexKind::None => format!("@{idx}"), + }); + } + if let Some(p) = i.index2 { + parts.push(self.proto_desc(p)); + } + if let Some(t) = i.target { + parts.push(format!(":{t:04x}")); + } + let mut s = if parts.is_empty() { i.name.to_string() } else { format!("{} {}", i.name, parts.join(", ")) }; + match (&i.payload, i.opcode) { + (Some(Payload::PackedSwitch { first_key, targets }), 0x2b) => { + let cases: Vec = targets + .iter() + .enumerate() + .take(32) + .map(|(k, t)| format!("{} -> :{:04x}", i64::from(*first_key) + k as i64, i64::from(i.offset) + i64::from(*t))) + .collect(); + s.push_str(&format!(" # {}", cases.join(", "))); + } + (Some(Payload::SparseSwitch { keys, targets }), 0x2c) => { + let cases: Vec = + keys.iter().zip(targets).take(32).map(|(k, t)| format!("{k} -> :{:04x}", i64::from(i.offset) + i64::from(*t))).collect(); + s.push_str(&format!(" # {}", cases.join(", "))); + } + (Some(Payload::FillArrayData { element_width, count, .. }), _) => s.push_str(&format!(" # {count} x {element_width}-byte elements")), + (Some(Payload::PackedSwitch { targets, .. }), 0) => s.push_str(&format!(" ({} cases)", targets.len())), + (Some(Payload::SparseSwitch { targets, .. }), 0) => s.push_str(&format!(" ({} cases)", targets.len())), + _ => {} + } + s + } +} + +/// One place code refers to a class, method or field. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct Reference { + /// Caller method, `Lcls;->name(…)R`. + pub caller: String, + /// Code unit offset in the caller. + pub offset: u32, + pub instruction: &'static str, + /// method, field or type + pub kind: &'static str, + pub target: String, +} + +/// Every instruction that refers to class `desc` (optionally only its methods named `method`). +pub fn references(dex: &Dex<'_>, desc: &str, method: Option<&str>, max: usize) -> (Vec, bool) { + let mut out = Vec::new(); + for c in &dex.classes { + let cd = dex.class_data(c); + for m in cd.direct_methods.iter().chain(&cd.virtual_methods) { + let Some(code) = dex.code(m.code_off) else { continue }; + let (insns, _) = crate::dex_ops::decode(&code.insns, 200_000); + for i in &insns { + let Some(idx) = i.index else { continue }; + let hit = match i.index_kind { + IndexKind::Method => dex.methods.get(idx as usize).and_then(|id| { + let cls = dex.type_desc(u32::from(id.class_idx)); + (cls == desc && method.is_none_or(|n| dex.string(id.name_idx) == n)).then(|| ("method", dex.method_ref(idx))) + }), + IndexKind::Field if method.is_none() => { + dex.fields.get(idx as usize).and_then(|id| (dex.type_desc(u32::from(id.class_idx)) == desc).then(|| ("field", dex.field_ref(idx)))) + } + IndexKind::Type if method.is_none() => { + let t = dex.type_desc(idx); + (t.trim_start_matches('[') == desc).then_some(("type", t)) + } + _ => None, + }; + if let Some((kind, target)) = hit { + if out.len() >= max { + return (out, true); + } + out.push(Reference { caller: dex.method_ref(m.method_idx), offset: i.offset, instruction: i.name, kind, target }); + } + } + } + } + (out, false) +} + +#[cfg(test)] +mod tests { + use super::*; + use craft_anything_testkit::android as fx; + + #[test] + fn parses_fixture() { + let data = fx::notes_dex(); + let d = parse(&data).unwrap(); + assert!(d.warnings.is_empty(), "{:?}", d.warnings); + assert!(d.header.checksum_ok); + assert_eq!(d.header.version, "035"); + assert_eq!(d.classes.len(), 2); + let main = d.find_class("com.example.notes.MainActivity").unwrap(); + assert_eq!(d.type_desc(main.superclass_idx), "Landroid/app/Activity;"); + assert_eq!(d.interfaces(main), vec!["Ljava/lang/Runnable;"]); + assert_eq!(d.string(main.source_file_idx), "MainActivity.java"); + assert_eq!(d.annotations(main), vec!["Ljava/lang/Deprecated; (runtime)"]); + let cd = d.class_data(main); + assert!(cd.complete); + assert_eq!(cd.static_fields.len(), 1); + assert_eq!(cd.instance_fields.len(), 1); + assert_eq!(cd.direct_methods.len(), 1); + assert_eq!(cd.virtual_methods.len(), 3); + let on_create = cd.virtual_methods.iter().find(|m| d.method_name(m.method_idx) == "onCreate").unwrap(); + assert_eq!(d.method_signature(on_create), "public void onCreate(android.os.Bundle)"); + let code = d.code(on_create.code_off).unwrap(); + let (ins, complete) = crate::dex_ops::decode(&code.insns, 1000); + assert!(complete); + let text: Vec = ins.iter().map(|i| d.format_insn(i)).collect(); + assert_eq!(text[0], "const-string v0, \"Hello, Craft!\""); + assert_eq!(text[1], "invoke-static {v0}, Lcom/example/notes/Util;->log(Ljava/lang/String;)V"); + let pick = cd.virtual_methods.iter().find(|m| d.method_name(m.method_idx) == "pick").unwrap(); + let pc = d.code(pick.code_off).unwrap(); + assert_eq!(pc.tries.len(), 1); + assert_eq!(pc.tries[0].handlers[0].exception.as_deref(), Some("Ljava/lang/RuntimeException;")); + assert_eq!(pc.tries[0].handlers[1].exception, None); + let (pins, _) = crate::dex_ops::decode(&pc.insns, 1000); + let sw = pins.iter().find(|i| i.name == "packed-switch").unwrap(); + assert!(d.format_insn(sw).contains("# 1 -> :"), "{}", d.format_insn(sw)); + let (refs, more) = references(&d, "Lcom/example/notes/Util;", Some("log"), 100); + assert!(!more); + assert_eq!(refs.len(), 2, "{refs:?}"); + assert_eq!(refs[0].caller, "Lcom/example/notes/MainActivity;->onCreate(Landroid/os/Bundle;)V"); + let (all, _) = references(&d, "Ljava/lang/System;", None, 100); + assert_eq!(all[0].kind, "field"); + } + + #[test] + fn helpers() { + assert_eq!(java_name("[[Ljava/lang/String;"), "java.lang.String[][]"); + assert_eq!(java_name("J"), "long"); + assert_eq!(to_descriptor("a.b.C"), "La/b/C;"); + assert_eq!(to_descriptor("La/b/C;"), "La/b/C;"); + assert_eq!(access_flags(0x10009, "method"), vec!["public", "static", "constructor"]); + let r = Reader::new(&[0x7f]); + assert_eq!(sleb128(&r, 0), Some((-1, 1))); + let s = [0xc0, 0x80, 0xe2, 0x82, 0xac, 0x41, 0]; + assert_eq!(mutf8(&Reader::new(&s), 0, 100).0, "\0\u{20ac}A"); + } + + #[test] + fn hostile() { + let data = fx::notes_dex(); + let walk = |d: &Dex<'_>| { + for c in &d.classes { + let cd = d.class_data(c); + let _ = d.annotations(c); + let _ = d.interfaces(c); + for m in cd.direct_methods.iter().chain(&cd.virtual_methods) { + let _ = d.method_signature(m); + if let Some(code) = d.code(m.code_off) { + let (ins, _) = crate::dex_ops::decode(&code.insns, 10_000); + for i in &ins { + let _ = d.format_insn(i); + } + } + } + } + let _ = references(d, "Lcom/example/notes/Util;", None, 100); + }; + for m in craft_anything_testkit::mutations(&data) { + if let Ok(d) = parse(&m) { + walk(&d); + } + } + for len in 0..data.len() { + if let Ok(d) = parse(&data[..len]) { + walk(&d); + } + } + for seed in 0..64 { + let mut n = craft_anything_testkit::noise(seed, 2048); + n[..8].copy_from_slice(b"dex\n035\0"); + n[40..44].copy_from_slice(&0x1234_5678u32.to_le_bytes()); + if let Ok(d) = parse(&n) { + walk(&d); + } + } + } +} diff --git a/crates/dex/src/dex_ops.rs b/crates/dex/src/dex_ops.rs new file mode 100644 index 000000000..139ad77ac --- /dev/null +++ b/crates/dex/src/dex_ops.rs @@ -0,0 +1,640 @@ +//! Dalvik bytecode: the opcode table (all 256 slots and three payload pseudo-instructions) and a +//! decoder for every instruction format. Pure decoding; index operands are resolved by the caller. + +use serde::Serialize; + +/// What an index operand refers to. +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum IndexKind { + None, + String, + Type, + Field, + Method, + CallSite, + MethodHandle, + Proto, +} + +/// Instruction formats from the Dalvik specification (`10x`, `35c`, …). +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] +pub enum Fmt { + F10x, + F12x, + F11n, + F11x, + F10t, + F20t, + F22x, + F21t, + F21s, + F21h, + F21c, + F23x, + F22b, + F22t, + F22s, + F22c, + F30t, + F32x, + F31i, + F31t, + F31c, + F35c, + F3rc, + F45cc, + F4rcc, + F51l, +} + +impl Fmt { + /// Length in 16-bit code units. + pub fn units(self) -> usize { + use Fmt::*; + match self { + F10x | F12x | F11n | F11x | F10t => 1, + F20t | F22x | F21t | F21s | F21h | F21c | F23x | F22b | F22t | F22s | F22c => 2, + F30t | F32x | F31i | F31t | F31c | F35c | F3rc => 3, + F45cc | F4rcc => 4, + F51l => 5, + } + } +} + +/// One opcode slot. +#[derive(Debug, Clone, Copy)] +pub struct Op { + pub name: &'static str, + pub fmt: Fmt, + pub index: IndexKind, +} + +const fn op(name: &'static str, fmt: Fmt, index: IndexKind) -> Op { + Op { name, fmt, index } +} + +const UNUSED: Op = op("unused", Fmt::F10x, IndexKind::None); + +const UNOPS: [&str; 21] = [ + "neg-int", + "not-int", + "neg-long", + "not-long", + "neg-float", + "neg-double", + "int-to-long", + "int-to-float", + "int-to-double", + "long-to-int", + "long-to-float", + "long-to-double", + "float-to-int", + "float-to-long", + "float-to-double", + "double-to-int", + "double-to-long", + "double-to-float", + "int-to-byte", + "int-to-char", + "int-to-short", +]; + +const BINOPS: [&str; 32] = [ + "add-int", + "sub-int", + "mul-int", + "div-int", + "rem-int", + "and-int", + "or-int", + "xor-int", + "shl-int", + "shr-int", + "ushr-int", + "add-long", + "sub-long", + "mul-long", + "div-long", + "rem-long", + "and-long", + "or-long", + "xor-long", + "shl-long", + "shr-long", + "ushr-long", + "add-float", + "sub-float", + "mul-float", + "div-float", + "rem-float", + "add-double", + "sub-double", + "mul-double", + "div-double", + "rem-double", +]; + +const BINOPS_2ADDR: [&str; 32] = [ + "add-int/2addr", + "sub-int/2addr", + "mul-int/2addr", + "div-int/2addr", + "rem-int/2addr", + "and-int/2addr", + "or-int/2addr", + "xor-int/2addr", + "shl-int/2addr", + "shr-int/2addr", + "ushr-int/2addr", + "add-long/2addr", + "sub-long/2addr", + "mul-long/2addr", + "div-long/2addr", + "rem-long/2addr", + "and-long/2addr", + "or-long/2addr", + "xor-long/2addr", + "shl-long/2addr", + "shr-long/2addr", + "ushr-long/2addr", + "add-float/2addr", + "sub-float/2addr", + "mul-float/2addr", + "div-float/2addr", + "rem-float/2addr", + "add-double/2addr", + "sub-double/2addr", + "mul-double/2addr", + "div-double/2addr", + "rem-double/2addr", +]; + +const LIT16: [&str; 8] = ["add-int/lit16", "rsub-int", "mul-int/lit16", "div-int/lit16", "rem-int/lit16", "and-int/lit16", "or-int/lit16", "xor-int/lit16"]; +const LIT8: [&str; 11] = [ + "add-int/lit8", + "rsub-int/lit8", + "mul-int/lit8", + "div-int/lit8", + "rem-int/lit8", + "and-int/lit8", + "or-int/lit8", + "xor-int/lit8", + "shl-int/lit8", + "shr-int/lit8", + "ushr-int/lit8", +]; +const CMP: [&str; 5] = ["cmpl-float", "cmpg-float", "cmpl-double", "cmpg-double", "cmp-long"]; +const IF2: [&str; 6] = ["if-eq", "if-ne", "if-lt", "if-ge", "if-gt", "if-le"]; +const IFZ: [&str; 6] = ["if-eqz", "if-nez", "if-ltz", "if-gez", "if-gtz", "if-lez"]; +const ARR: [&str; 14] = [ + "aget", + "aget-wide", + "aget-object", + "aget-boolean", + "aget-byte", + "aget-char", + "aget-short", + "aput", + "aput-wide", + "aput-object", + "aput-boolean", + "aput-byte", + "aput-char", + "aput-short", +]; +const IFIELD: [&str; 14] = [ + "iget", + "iget-wide", + "iget-object", + "iget-boolean", + "iget-byte", + "iget-char", + "iget-short", + "iput", + "iput-wide", + "iput-object", + "iput-boolean", + "iput-byte", + "iput-char", + "iput-short", +]; +const SFIELD: [&str; 14] = [ + "sget", + "sget-wide", + "sget-object", + "sget-boolean", + "sget-byte", + "sget-char", + "sget-short", + "sput", + "sput-wide", + "sput-object", + "sput-boolean", + "sput-byte", + "sput-char", + "sput-short", +]; +const INVOKE: [&str; 5] = ["invoke-virtual", "invoke-super", "invoke-direct", "invoke-static", "invoke-interface"]; +const INVOKE_RANGE: [&str; 5] = ["invoke-virtual/range", "invoke-super/range", "invoke-direct/range", "invoke-static/range", "invoke-interface/range"]; + +fn pick(list: &[&'static str], i: u8, base: u8) -> &'static str { + list.get(usize::from(i.wrapping_sub(base))).copied().unwrap_or("unused") +} + +/// The opcode table entry for `b`. +pub fn opcode(b: u8) -> Op { + use Fmt::*; + use IndexKind as K; + match b { + 0x00 => op("nop", F10x, K::None), + 0x01 => op("move", F12x, K::None), + 0x02 => op("move/from16", F22x, K::None), + 0x03 => op("move/16", F32x, K::None), + 0x04 => op("move-wide", F12x, K::None), + 0x05 => op("move-wide/from16", F22x, K::None), + 0x06 => op("move-wide/16", F32x, K::None), + 0x07 => op("move-object", F12x, K::None), + 0x08 => op("move-object/from16", F22x, K::None), + 0x09 => op("move-object/16", F32x, K::None), + 0x0a => op("move-result", F11x, K::None), + 0x0b => op("move-result-wide", F11x, K::None), + 0x0c => op("move-result-object", F11x, K::None), + 0x0d => op("move-exception", F11x, K::None), + 0x0e => op("return-void", F10x, K::None), + 0x0f => op("return", F11x, K::None), + 0x10 => op("return-wide", F11x, K::None), + 0x11 => op("return-object", F11x, K::None), + 0x12 => op("const/4", F11n, K::None), + 0x13 => op("const/16", F21s, K::None), + 0x14 => op("const", F31i, K::None), + 0x15 => op("const/high16", F21h, K::None), + 0x16 => op("const-wide/16", F21s, K::None), + 0x17 => op("const-wide/32", F31i, K::None), + 0x18 => op("const-wide", F51l, K::None), + 0x19 => op("const-wide/high16", F21h, K::None), + 0x1a => op("const-string", F21c, K::String), + 0x1b => op("const-string/jumbo", F31c, K::String), + 0x1c => op("const-class", F21c, K::Type), + 0x1d => op("monitor-enter", F11x, K::None), + 0x1e => op("monitor-exit", F11x, K::None), + 0x1f => op("check-cast", F21c, K::Type), + 0x20 => op("instance-of", F22c, K::Type), + 0x21 => op("array-length", F12x, K::None), + 0x22 => op("new-instance", F21c, K::Type), + 0x23 => op("new-array", F22c, K::Type), + 0x24 => op("filled-new-array", F35c, K::Type), + 0x25 => op("filled-new-array/range", F3rc, K::Type), + 0x26 => op("fill-array-data", F31t, K::None), + 0x27 => op("throw", F11x, K::None), + 0x28 => op("goto", F10t, K::None), + 0x29 => op("goto/16", F20t, K::None), + 0x2a => op("goto/32", F30t, K::None), + 0x2b => op("packed-switch", F31t, K::None), + 0x2c => op("sparse-switch", F31t, K::None), + 0x2d..=0x31 => op(pick(&CMP, b, 0x2d), F23x, K::None), + 0x32..=0x37 => op(pick(&IF2, b, 0x32), F22t, K::None), + 0x38..=0x3d => op(pick(&IFZ, b, 0x38), F21t, K::None), + 0x44..=0x51 => op(pick(&ARR, b, 0x44), F23x, K::None), + 0x52..=0x5f => op(pick(&IFIELD, b, 0x52), F22c, K::Field), + 0x60..=0x6d => op(pick(&SFIELD, b, 0x60), F21c, K::Field), + 0x6e..=0x72 => op(pick(&INVOKE, b, 0x6e), F35c, K::Method), + 0x74..=0x78 => op(pick(&INVOKE_RANGE, b, 0x74), F3rc, K::Method), + 0x7b..=0x8f => op(pick(&UNOPS, b, 0x7b), F12x, K::None), + 0x90..=0xaf => op(pick(&BINOPS, b, 0x90), F23x, K::None), + 0xb0..=0xcf => op(pick(&BINOPS_2ADDR, b, 0xb0), F12x, K::None), + 0xd0..=0xd7 => op(pick(&LIT16, b, 0xd0), F22s, K::None), + 0xd8..=0xe2 => op(pick(&LIT8, b, 0xd8), F22b, K::None), + 0xfa => op("invoke-polymorphic", F45cc, K::Method), + 0xfb => op("invoke-polymorphic/range", F4rcc, K::Method), + 0xfc => op("invoke-custom", F35c, K::CallSite), + 0xfd => op("invoke-custom/range", F3rc, K::CallSite), + 0xfe => op("const-method-handle", F21c, K::MethodHandle), + 0xff => op("const-method-type", F21c, K::Proto), + _ => UNUSED, + } +} + +/// Switch and array payloads. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +#[serde(tag = "kind", rename_all = "kebab-case")] +pub enum Payload { + PackedSwitch { first_key: i32, targets: Vec }, + SparseSwitch { keys: Vec, targets: Vec }, + FillArrayData { element_width: u16, count: u32, data: Vec }, +} + +/// One decoded instruction. +#[derive(Debug, Clone, PartialEq, Eq, Serialize)] +pub struct Insn { + /// Offset in 16-bit code units from the start of the method's code. + pub offset: u32, + pub opcode: u8, + pub name: &'static str, + /// Length in code units. + pub units: u32, + /// Registers in operand order (for ranges: every register in the range). + pub regs: Vec, + /// True for `/range` forms. + pub range: bool, + #[serde(skip_serializing_if = "Option::is_none")] + pub literal: Option, + /// Absolute branch or payload target, in code units. + #[serde(skip_serializing_if = "Option::is_none")] + pub target: Option, + pub index_kind: IndexKind, + #[serde(skip_serializing_if = "Option::is_none")] + pub index: Option, + /// Second index (the proto of `invoke-polymorphic`). + #[serde(skip_serializing_if = "Option::is_none")] + pub index2: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub payload: Option, +} + +/// Most entries decoded from one payload. +const MAX_PAYLOAD: usize = 65_536; + +fn u32_at(code: &[u16], i: usize) -> Option { + Some(u32::from(*code.get(i)?) | (u32::from(*code.get(i + 1)?) << 16)) +} + +/// Decode a payload pseudo-instruction at `pc`; returns it and its length in units. +fn payload(code: &[u16], pc: usize) -> Option<(Payload, usize)> { + let ident = *code.get(pc)?; + match ident { + 0x0100 => { + let size = usize::from(*code.get(pc + 1)?); + let first_key = u32_at(code, pc + 2)? as i32; + let mut targets = Vec::with_capacity(size.min(MAX_PAYLOAD)); + for i in 0..size.min(MAX_PAYLOAD) { + targets.push(u32_at(code, pc + 4 + i * 2)? as i32); + } + Some((Payload::PackedSwitch { first_key, targets }, size * 2 + 4)) + } + 0x0200 => { + let size = usize::from(*code.get(pc + 1)?); + let mut keys = Vec::with_capacity(size.min(MAX_PAYLOAD)); + let mut targets = Vec::with_capacity(size.min(MAX_PAYLOAD)); + for i in 0..size.min(MAX_PAYLOAD) { + keys.push(u32_at(code, pc + 2 + i * 2)? as i32); + targets.push(u32_at(code, pc + 2 + size * 2 + i * 2)? as i32); + } + Some((Payload::SparseSwitch { keys, targets }, size * 4 + 2)) + } + 0x0300 => { + let width = *code.get(pc + 1)?; + let count = u32_at(code, pc + 2)?; + let bytes = usize::from(width).checked_mul(count as usize)?; + let units = bytes.checked_add(1)? / 2 + 4; + if pc.checked_add(units)? > code.len() { + return None; + } + let mut data = Vec::with_capacity(bytes.min(MAX_PAYLOAD)); + for i in 0..bytes.min(MAX_PAYLOAD) { + let u = *code.get(pc + 4 + i / 2)?; + data.push(if i % 2 == 0 { u as u8 } else { (u >> 8) as u8 }); + } + Some((Payload::FillArrayData { element_width: width, count, data }, units)) + } + _ => None, + } +} + +/// Decode one instruction at `pc`, or `None` when it runs past the end. +pub fn decode_one(code: &[u16], pc: usize) -> Option { + use Fmt::*; + let u0 = *code.get(pc)?; + let b = (u0 & 0xff) as u8; + let hi = (u0 >> 8) as u8; + if b == 0 && hi != 0 { + let (p, units) = payload(code, pc)?; + let name = match p { + Payload::PackedSwitch { .. } => "packed-switch-payload", + Payload::SparseSwitch { .. } => "sparse-switch-payload", + Payload::FillArrayData { .. } => "array-data-payload", + }; + return Some(Insn { + offset: pc as u32, + opcode: 0, + name, + units: units as u32, + regs: vec![], + range: false, + literal: None, + target: None, + index_kind: IndexKind::None, + index: None, + index2: None, + payload: Some(p), + }); + } + let o = opcode(b); + let n = o.fmt.units(); + if pc.checked_add(n)? > code.len() { + return None; + } + let u = |i: usize| code.get(pc + i).copied().unwrap_or(0); + let a4 = u32::from(hi & 0x0f); + let b4 = u32::from(hi >> 4); + let aa = u32::from(hi); + let rel = |d: i64| pc as i64 + d; + let mut insn = Insn { + offset: pc as u32, + opcode: b, + name: o.name, + units: n as u32, + regs: vec![], + range: false, + literal: None, + target: None, + index_kind: o.index, + index: None, + index2: None, + payload: None, + }; + match o.fmt { + F10x => {} + F12x => insn.regs = vec![a4, b4], + F11n => { + insn.regs = vec![a4]; + insn.literal = Some(i64::from((hi as i8) >> 4)); + } + F11x => insn.regs = vec![aa], + F10t => insn.target = Some(rel(i64::from(hi as i8))), + F20t => insn.target = Some(rel(i64::from(u(1) as i16))), + F22x => insn.regs = vec![aa, u32::from(u(1))], + F21t => { + insn.regs = vec![aa]; + insn.target = Some(rel(i64::from(u(1) as i16))); + } + F21s => { + insn.regs = vec![aa]; + insn.literal = Some(i64::from(u(1) as i16)); + } + F21h => { + insn.regs = vec![aa]; + let v = i64::from(u(1) as i16); + insn.literal = Some(if b == 0x19 { v << 48 } else { v << 16 }); + } + F21c => { + insn.regs = vec![aa]; + insn.index = Some(u32::from(u(1))); + } + F23x => insn.regs = vec![aa, u32::from(u(1) & 0xff), u32::from(u(1) >> 8)], + F22b => { + insn.regs = vec![aa, u32::from(u(1) & 0xff)]; + insn.literal = Some(i64::from((u(1) >> 8) as u8 as i8)); + } + F22t => { + insn.regs = vec![a4, b4]; + insn.target = Some(rel(i64::from(u(1) as i16))); + } + F22s => { + insn.regs = vec![a4, b4]; + insn.literal = Some(i64::from(u(1) as i16)); + } + F22c => { + insn.regs = vec![a4, b4]; + insn.index = Some(u32::from(u(1))); + } + F30t => insn.target = Some(rel(i64::from((u32::from(u(1)) | (u32::from(u(2)) << 16)) as i32))), + F32x => insn.regs = vec![u32::from(u(1)), u32::from(u(2))], + F31i => { + insn.regs = vec![aa]; + insn.literal = Some(i64::from((u32::from(u(1)) | (u32::from(u(2)) << 16)) as i32)); + } + F31t => { + insn.regs = vec![aa]; + insn.target = Some(rel(i64::from((u32::from(u(1)) | (u32::from(u(2)) << 16)) as i32))); + } + F31c => { + insn.regs = vec![aa]; + insn.index = Some(u32::from(u(1)) | (u32::from(u(2)) << 16)); + } + F35c | F45cc => { + let count = b4.min(5) as usize; + let g = a4; + let w = u(2); + let all = [u32::from(w & 0xf), u32::from((w >> 4) & 0xf), u32::from((w >> 8) & 0xf), u32::from(w >> 12), g]; + insn.regs = all.iter().take(count).copied().collect(); + insn.index = Some(u32::from(u(1))); + if o.fmt == F45cc { + insn.index2 = Some(u32::from(u(3))); + } + } + F3rc | F4rcc => { + let first = u32::from(u(2)); + insn.regs = (0..aa).map(|i| first.saturating_add(i)).collect(); + insn.range = true; + insn.index = Some(u32::from(u(1))); + if o.fmt == F4rcc { + insn.index2 = Some(u32::from(u(3))); + } + } + F51l => { + insn.regs = vec![aa]; + let v = u64::from(u(1)) | (u64::from(u(2)) << 16) | (u64::from(u(3)) << 32) | (u64::from(u(4)) << 48); + insn.literal = Some(v as i64); + } + } + Some(insn) +} + +/// Decode a whole method body, attaching each switch/array payload to the instruction that +/// references it. Stops at `max` instructions. +pub fn decode(code: &[u16], max: usize) -> (Vec, bool) { + let mut out = Vec::new(); + let mut pc = 0usize; + let mut complete = true; + while pc < code.len() { + if out.len() >= max { + complete = false; + break; + } + match decode_one(code, pc) { + Some(i) => { + pc += (i.units as usize).max(1); + out.push(i); + } + None => { + complete = false; + break; + } + } + } + // Attach payloads to their users (31t formats). + let payloads: Vec<(i64, Payload)> = out.iter().filter_map(|i| i.payload.clone().map(|p| (i64::from(i.offset), p))).collect(); + for i in out.iter_mut().filter(|i| matches!(i.opcode, 0x26 | 0x2b | 0x2c)) { + if let Some(t) = i.target + && let Some((_, p)) = payloads.iter().find(|(o, _)| *o == t) + { + i.payload = Some(p.clone()); + } + } + (out, complete) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn table_is_complete() { + let mut named = 0; + for b in 0..=255u8 { + let o = opcode(b); + if o.name != "unused" { + named += 1; + } + } + // 256 slots minus the 0x3e-0x43, 0x73, 0x79-0x7a and 0xe3-0xf9 holes. + assert_eq!(named, 256 - 6 - 1 - 2 - 23); + assert_eq!(opcode(0x6e).name, "invoke-virtual"); + assert_eq!(opcode(0x8f).name, "int-to-short"); + assert_eq!(opcode(0xaf).name, "rem-double"); + assert_eq!(opcode(0xcf).name, "rem-double/2addr"); + assert_eq!(opcode(0xe2).name, "ushr-int/lit8"); + } + + #[test] + fn decodes_formats() { + // const/4 v1, #-1 ; invoke-static {v0, v1}, meth@5 ; packed-switch v0, +4 ; return-void + // payload: packed, size 2, first 10, targets 1, 2 + let code: Vec = vec![0xf112, 0x2071, 5, 0x0010, 0x002b, 4, 0, 0x000e, 0x0100, 2, 10, 0, 1, 0, 2, 0]; + let (ins, complete) = decode(&code, 100); + assert!(complete); + assert_eq!(ins[0].name, "const/4"); + assert_eq!(ins[0].literal, Some(-1)); + assert_eq!(ins[0].regs, vec![1]); + assert_eq!(ins[1].name, "invoke-static"); + assert_eq!(ins[1].regs, vec![0, 1]); + assert_eq!(ins[1].index, Some(5)); + assert_eq!(ins[2].name, "packed-switch"); + assert_eq!(ins[2].target, Some(8)); + assert_eq!(ins[2].payload, Some(Payload::PackedSwitch { first_key: 10, targets: vec![1, 2] })); + assert_eq!(ins[4].name, "packed-switch-payload"); + // const-wide v0, 0x1122334455667788 + let (w, _) = decode(&[0x0018, 0x7788, 0x5566, 0x3344, 0x1122], 10); + assert_eq!(w[0].literal, Some(0x1122_3344_5566_7788)); + // fill-array-data payload width 1 count 3 + let (f, _) = decode(&[0x0300, 1, 3, 0, 0x0201, 0x0003], 10); + assert_eq!(f[0].payload, Some(Payload::FillArrayData { element_width: 1, count: 3, data: vec![1, 2, 3] })); + // invoke-virtual/range {v4..v6} + let (r, _) = decode(&[0x0374, 9, 4], 10); + assert_eq!(r[0].regs, vec![4, 5, 6]); + assert!(r[0].range); + } + + #[test] + fn hostile() { + for seed in 0..200 { + let n = craft_anything_testkit::noise(seed, 256); + let code: Vec = n.as_chunks::<2>().0.iter().map(|c| u16::from_le_bytes(*c)).collect(); + let _ = decode(&code, 1000); + } + let _ = decode(&[0x0300, 0xffff, 0xffff, 0xffff], 10); + let _ = decode(&[0x0200, 0xffff], 10); + let _ = decode(&[0x0018], 10); + } +} From b4e455d01758837e2e9f5a3bf9b29ea6f34be1ad Mon Sep 17 00:00:00 2001 From: BFlat Date: Tue, 17 Nov 2026 08:38:14 -0500 Subject: [PATCH 023/161] dex: More format parsers keep to their declared lengths and refuse impossible values (#722) Port Craft Anything b9ca099903aeef5d4b33a7e83e8e7062ab94c65a reader and opcode decoder. --- crates/dex/src/dex.rs | 17 +++++------------ crates/dex/src/dex_ops.rs | 14 ++++++++++++-- 2 files changed, 17 insertions(+), 14 deletions(-) diff --git a/crates/dex/src/dex.rs b/crates/dex/src/dex.rs index e46d97b5b..b01a1b247 100644 --- a/crates/dex/src/dex.rs +++ b/crates/dex/src/dex.rs @@ -148,18 +148,7 @@ pub fn mutf8(r: &Reader<'_>, mut at: usize, max_units: usize) -> (String, usize) if b0 == 0 { break; } - let (u, n) = if b0 < 0x80 { - (u16::from(b0), 1) - } else if b0 & 0xe0 == 0xc0 { - let b1 = r.u8(at + 1).unwrap_or(0); - ((u16::from(b0 & 0x1f) << 6) | u16::from(b1 & 0x3f), 2) - } else if b0 & 0xf0 == 0xe0 { - let b1 = r.u8(at + 1).unwrap_or(0); - let b2 = r.u8(at + 2).unwrap_or(0); - ((u16::from(b0 & 0x0f) << 12) | (u16::from(b1 & 0x3f) << 6) | u16::from(b2 & 0x3f), 3) - } else { - (0xfffd, 1) - }; + let (u, n) = crate::bytes::mutf8_unit(r.data().get(at..).unwrap_or_default()); units.push(u); at = at.saturating_add(n); consumed += n; @@ -173,6 +162,10 @@ pub fn sleb128(r: &Reader<'_>, off: usize) -> Option<(i64, usize)> { let mut shift = 0u32; for i in 0..10usize { let b = r.u8(off.checked_add(i)?)?; + // Only one value bit remains in the tenth byte; the rest must extend its sign. + if i == 9 && !matches!(b, 0x00 | 0x7f) { + return None; + } result |= i64::from(b & 0x7f).checked_shl(shift)?; shift += 7; if b & 0x80 == 0 { diff --git a/crates/dex/src/dex_ops.rs b/crates/dex/src/dex_ops.rs index 139ad77ac..53fbf1d05 100644 --- a/crates/dex/src/dex_ops.rs +++ b/crates/dex/src/dex_ops.rs @@ -384,6 +384,9 @@ fn payload(code: &[u16], pc: usize) -> Option<(Payload, usize)> { } 0x0300 => { let width = *code.get(pc + 1)?; + if !matches!(width, 1 | 2 | 4 | 8) { + return None; + } let count = u32_at(code, pc + 2)?; let bytes = usize::from(width).checked_mul(count as usize)?; let units = bytes.checked_add(1)? / 2 + 4; @@ -513,7 +516,10 @@ pub fn decode_one(code: &[u16], pc: usize) -> Option { insn.index = Some(u32::from(u(1)) | (u32::from(u(2)) << 16)); } F35c | F45cc => { - let count = b4.min(5) as usize; + if b4 > 5 { + return None; + } + let count = b4 as usize; let g = a4; let w = u(2); let all = [u32::from(w & 0xf), u32::from((w >> 4) & 0xf), u32::from((w >> 8) & 0xf), u32::from(w >> 12), g]; @@ -525,7 +531,11 @@ pub fn decode_one(code: &[u16], pc: usize) -> Option { } F3rc | F4rcc => { let first = u32::from(u(2)); - insn.regs = (0..aa).map(|i| first.saturating_add(i)).collect(); + let end = first.checked_add(aa)?; + if end > u32::from(u16::MAX) + 1 { + return None; + } + insn.regs = (first..end).collect(); insn.range = true; insn.index = Some(u32::from(u(1))); if o.fmt == F4rcc { From 77e7878f7bbc38fffd1ec024182f56098ca4d92d Mon Sep 17 00:00:00 2001 From: Afloat <132809511+Afloat16@users.noreply.github.com> Date: Tue, 17 Nov 2026 08:38:14 -0500 Subject: [PATCH 024/161] dex: Address the parser review findings from #722 (#766) Port Craft Anything eab6a0f01bf9fc3a36798d322e4442e169394dfc reader and opcode decoder. --- crates/dex/src/dex.rs | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/crates/dex/src/dex.rs b/crates/dex/src/dex.rs index b01a1b247..6b23d2e78 100644 --- a/crates/dex/src/dex.rs +++ b/crates/dex/src/dex.rs @@ -156,14 +156,14 @@ pub fn mutf8(r: &Reader<'_>, mut at: usize, max_units: usize) -> (String, usize) (String::from_utf16_lossy(&units), consumed) } -/// Signed LEB128 at `off`: (value, bytes). +/// Signed 32-bit DEX LEB128 at `off`: (value, bytes), using at most five bytes. pub fn sleb128(r: &Reader<'_>, off: usize) -> Option<(i64, usize)> { let mut result = 0i64; let mut shift = 0u32; - for i in 0..10usize { + for i in 0..5usize { let b = r.u8(off.checked_add(i)?)?; - // Only one value bit remains in the tenth byte; the rest must extend its sign. - if i == 9 && !matches!(b, 0x00 | 0x7f) { + // Four value bits remain in byte five; the other bits must extend the sign. + if i == 4 && !matches!(b, 0x00..=0x07 | 0x78..=0x7f) { return None; } result |= i64::from(b & 0x7f).checked_shl(shift)?; From 0373b7f8e347bd4f284a0522365b19520547495c Mon Sep 17 00:00:00 2001 From: Afloat Date: Tue, 17 Nov 2026 08:38:14 -0500 Subject: [PATCH 025/161] dex: Preserve DEX boundaries when member or handler lists are capped (R3.12) (#1388) Port Craft Anything 34d9450b75f402217a7243c04d6c5ed3d2d3f8d4 reader and opcode decoder. --- crates/dex/src/dex.rs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/crates/dex/src/dex.rs b/crates/dex/src/dex.rs index 6b23d2e78..c4076dfc8 100644 --- a/crates/dex/src/dex.rs +++ b/crates/dex/src/dex.rs @@ -523,6 +523,9 @@ impl<'a> Dex<'a> { } } } + if count > MAX_MEMBERS as u32 { + return cd; + } } cd.complete = true; cd @@ -563,6 +566,7 @@ impl<'a> Dex<'a> { handlers.push(Handler { exception: Some(self.type_desc(t as u32)), address: addr as u32 }); } if size <= 0 + && handlers.len() as u64 == size.unsigned_abs() && let Some((addr, _)) = r.uleb128(p) { handlers.push(Handler { exception: None, address: addr as u32 }); From 794b884096e75083a6ab22b936185cb0ece3cd46 Mon Sep 17 00:00:00 2001 From: Afloat Date: Tue, 17 Nov 2026 08:38:14 -0500 Subject: [PATCH 026/161] dex: Bound retained DEX payload attachments (R3.12) (#1548) Port Craft Anything 1ebdb78760d6bfd31f1a70e6389e0d441adb3d2e reader and opcode decoder. --- crates/dex/src/dex_ops.rs | 60 +++++++++++++++++++++++++++++++++------ 1 file changed, 52 insertions(+), 8 deletions(-) diff --git a/crates/dex/src/dex_ops.rs b/crates/dex/src/dex_ops.rs index 53fbf1d05..71095fc75 100644 --- a/crates/dex/src/dex_ops.rs +++ b/crates/dex/src/dex_ops.rs @@ -354,6 +354,25 @@ pub struct Insn { /// Most entries decoded from one payload. const MAX_PAYLOAD: usize = 65_536; +/// Maximum payload bytes retained by one decoded method, including copies attached to users. +pub const DEFAULT_MAX_PAYLOAD_BYTES: usize = 64 << 20; + +impl Payload { + fn retained_bytes(&self) -> usize { + match self { + Self::PackedSwitch { targets, .. } => targets.len().saturating_mul(4), + Self::SparseSwitch { keys, targets } => keys.len().saturating_add(targets.len()).saturating_mul(4), + Self::FillArrayData { data, .. } => data.len(), + } + } + + fn complete(&self) -> bool { + match self { + Self::FillArrayData { element_width, count, data } => u64::from(*element_width) * u64::from(*count) == data.len() as u64, + _ => true, + } + } +} fn u32_at(code: &[u16], i: usize) -> Option { Some(u32::from(*code.get(i)?) | (u32::from(*code.get(i + 1)?) << 16)) @@ -552,19 +571,36 @@ pub fn decode_one(code: &[u16], pc: usize) -> Option { } /// Decode a whole method body, attaching each switch/array payload to the instruction that -/// references it. Stops at `max` instructions. +/// references it. Stops at `max` instructions and caps all retained payload copies together. pub fn decode(code: &[u16], max: usize) -> (Vec, bool) { + decode_with_payload_limit(code, max, DEFAULT_MAX_PAYLOAD_BYTES) +} + +/// Decode with an explicit aggregate payload byte budget. Payloads that do not fit are omitted; +/// their instruction metadata remains and the returned completeness flag is false. A source +/// payload and every attached copy consume the same budget, before the copy is allocated. +pub fn decode_with_payload_limit(code: &[u16], max: usize, max_payload_bytes: usize) -> (Vec, bool) { let mut out = Vec::new(); let mut pc = 0usize; let mut complete = true; + let mut bytes_left = max_payload_bytes; while pc < code.len() { if out.len() >= max { complete = false; break; } match decode_one(code, pc) { - Some(i) => { + Some(mut i) => { pc += (i.units as usize).max(1); + if let Some(p) = &i.payload { + complete &= p.complete(); + if let Some(left) = bytes_left.checked_sub(p.retained_bytes()) { + bytes_left = left; + } else { + i.payload = None; + complete = false; + } + } out.push(i); } None => { @@ -574,12 +610,20 @@ pub fn decode(code: &[u16], max: usize) -> (Vec, bool) { } } // Attach payloads to their users (31t formats). - let payloads: Vec<(i64, Payload)> = out.iter().filter_map(|i| i.payload.clone().map(|p| (i64::from(i.offset), p))).collect(); - for i in out.iter_mut().filter(|i| matches!(i.opcode, 0x26 | 0x2b | 0x2c)) { - if let Some(t) = i.target - && let Some((_, p)) = payloads.iter().find(|(o, _)| *o == t) - { - i.payload = Some(p.clone()); + // The decoder visits offsets in order. Index their positions without copying any payload. + let payloads: Vec<(i64, usize)> = out.iter().enumerate().filter(|(_, i)| i.payload.is_some()).map(|(n, i)| (i64::from(i.offset), n)).collect(); + for n in 0..out.len() { + let Some(target) = out.get(n).filter(|i| matches!(i.opcode, 0x26 | 0x2b | 0x2c)).and_then(|i| i.target) else { continue }; + let Ok(position) = payloads.binary_search_by_key(&target, |(offset, _)| *offset) else { continue }; + let Some(payload) = payloads.get(position).and_then(|(_, index)| out.get(*index)).and_then(|i| i.payload.as_ref()) else { continue }; + let Some(left) = bytes_left.checked_sub(payload.retained_bytes()) else { + complete = false; + continue; + }; + bytes_left = left; + let copy = payload.clone(); + if let Some(instruction) = out.get_mut(n) { + instruction.payload = Some(copy); } } (out, complete) From e1eae62fe29d52e6f5e79d0742fd587f54b02e8b Mon Sep 17 00:00:00 2001 From: BFlat Date: Tue, 17 Nov 2026 08:39:25 -0500 Subject: [PATCH 027/161] dex: retain array_widths regressions Source crates/formats/tests/dex_array_widths.rs at b9ca099903aeef5d4b33a7e83e8e7062ab94c65a. Imports adapted only. --- crates/dex/src/regressions/array_widths.rs | 39 ++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 crates/dex/src/regressions/array_widths.rs diff --git a/crates/dex/src/regressions/array_widths.rs b/crates/dex/src/regressions/array_widths.rs new file mode 100644 index 000000000..37524b002 --- /dev/null +++ b/crates/dex/src/regressions/array_widths.rs @@ -0,0 +1,39 @@ +use crate::dex_ops::{Payload, decode, decode_one}; + +#[test] +fn zero_width_does_not_describe_a_huge_complete_array() { + let code = [0x0300, 0, 0xffff, 0xffff]; + assert!(decode_one(&code, 0).is_none()); + assert!(!decode(&code, 16).1); +} + +#[test] +fn unsupported_element_widths_are_refused_even_for_empty_arrays() { + for width in [0, 3, 5, 6, 7, 9, 16, u16::MAX] { + assert!(decode_one(&[0x0300, width, 0, 0], 0).is_none(), "width {width}"); + } +} + +#[test] +fn all_primitive_widths_and_odd_byte_padding_still_decode() { + for width in [1u16, 2, 4, 8] { + let count = 3u32; + let bytes = usize::from(width) * count as usize; + let mut code = vec![0x0300, width, count as u16, 0]; + code.resize(4 + bytes.div_ceil(2), 0x2211); + let instruction = decode_one(&code, 0).unwrap(); + assert_eq!(instruction.units as usize, code.len()); + match instruction.payload.unwrap() { + Payload::FillArrayData { element_width, count: decoded, data } => { + assert_eq!(element_width, width); + assert_eq!(decoded, count); + assert_eq!(data.len(), bytes); + assert_eq!(data[0], 0x11); + } + other => panic!("unexpected payload {other:?}"), + } + assert!(decode_one(&[0x0300, width, 0, 0], 0).is_some()); + code.pop(); + assert!(decode_one(&code, 0).is_none()); + } +} From 570e8bc8735cf6250adb1098dbe2244b1cb6200c Mon Sep 17 00:00:00 2001 From: Afloat Date: Tue, 17 Nov 2026 08:39:25 -0500 Subject: [PATCH 028/161] dex: retain capped_table_boundaries regressions Source crates/formats/tests/dex_capped_table_boundaries.rs at 34d9450b75f402217a7243c04d6c5ed3d2d3f8d4. Imports adapted only. --- .../regressions/capped_table_boundaries.rs | 114 ++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 crates/dex/src/regressions/capped_table_boundaries.rs diff --git a/crates/dex/src/regressions/capped_table_boundaries.rs b/crates/dex/src/regressions/capped_table_boundaries.rs new file mode 100644 index 000000000..7fbb419f5 --- /dev/null +++ b/crates/dex/src/regressions/capped_table_boundaries.rs @@ -0,0 +1,114 @@ +use crate::dex::{self, ClassDef, Code}; + +const MEMBER_CAP: usize = 65_536; +const HANDLER_CAP: usize = 256; + +fn uleb(mut value: u32, bytes: &mut Vec) { + loop { + let byte = (value & 0x7f) as u8; + value >>= 7; + bytes.push(byte | if value == 0 { 0 } else { 0x80 }); + if value == 0 { + break; + } + } +} + +fn sleb(mut value: i32, bytes: &mut Vec) { + loop { + let byte = value as u8 & 0x7f; + value >>= 7; + let done = (value == 0 && byte & 0x40 == 0) || (value == -1 && byte & 0x40 != 0); + bytes.push(byte | if done { 0 } else { 0x80 }); + if done { + break; + } + } +} + +fn header() -> Vec { + let mut bytes = vec![0; 0x70]; + bytes[..8].copy_from_slice(b"dex\n035\0"); + bytes[36..40].copy_from_slice(&0x70u32.to_le_bytes()); + bytes[40..44].copy_from_slice(&0x1234_5678u32.to_le_bytes()); + bytes +} + +fn class_data(count: usize) -> dex::ClassData { + let mut bytes = header(); + for n in [count as u32, 0, 1, 0] { + uleb(n, &mut bytes); + } + bytes.extend_from_slice(&[0, 1].repeat(count)); + bytes.extend_from_slice(&[3, 9, 7]); + let len = bytes.len() as u32; + bytes[32..36].copy_from_slice(&len.to_le_bytes()); + let class = ClassDef { + class_idx: 0, + access: 0, + superclass_idx: 0, + interfaces_off: 0, + source_file_idx: 0, + annotations_off: 0, + class_data_off: 0x70, + static_values_off: 0, + }; + dex::parse(&bytes).unwrap().class_data(&class) +} + +fn code(size: i32, payload: &[u8]) -> Code { + let mut bytes = header(); + bytes.extend_from_slice(&[1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 8, 0, 0, 0]); + bytes.extend_from_slice(&[0; 16]); + bytes.extend_from_slice(&[0, 0, 0, 0, 1, 0, 1, 0]); + bytes.push(1); + sleb(size, &mut bytes); + bytes.extend_from_slice(payload); + let len = bytes.len() as u32; + bytes[32..36].copy_from_slice(&len.to_le_bytes()); + dex::parse(&bytes).unwrap().code(0x70).unwrap() +} + +#[test] +fn capped_class_members_do_not_become_the_next_group() { + let data = class_data(MEMBER_CAP + 1); + assert_eq!(data.static_fields.len(), MEMBER_CAP); + assert!(data.direct_methods.is_empty()); + assert!(!data.complete); +} + +#[test] +fn an_exact_member_limit_keeps_the_following_group() { + let data = class_data(MEMBER_CAP); + assert_eq!(data.static_fields.len(), MEMBER_CAP); + assert!(data.complete); + assert_eq!(data.direct_methods, [dex::EncodedMethod { method_idx: 3, access: 9, code_off: 7 }]); +} + +#[test] +fn an_unread_typed_handler_is_not_a_catch_all() { + let mut payload = [0, 1].repeat(HANDLER_CAP + 1); + payload.push(7); + let parsed = code(-((HANDLER_CAP + 1) as i32), &payload); + assert_eq!(parsed.tries[0].handlers.len(), HANDLER_CAP); + assert!(parsed.tries[0].handlers.iter().all(|handler| handler.exception.is_some() && handler.address == 1)); +} + +#[test] +fn an_incomplete_typed_handler_cannot_supply_a_catch_all_address() { + let parsed = code(-2, &[0, 1, 0]); + assert_eq!(parsed.tries[0].handlers.len(), 1); + assert!(parsed.tries[0].handlers[0].exception.is_some()); + assert_eq!(parsed.tries[0].handlers[0].address, 1); +} + +#[test] +fn complete_typed_lists_keep_their_real_catch_all() { + for count in [0, 1, HANDLER_CAP] { + let mut payload = [0, 1].repeat(count); + payload.push(7); + let parsed = code(-(count as i32), &payload); + assert_eq!(parsed.tries[0].handlers.len(), count + 1); + assert_eq!(parsed.tries[0].handlers.last(), Some(&dex::Handler { exception: None, address: 7 })); + } +} From 79ad1aff6fa1758c91e53a9001baf4f38e5c028e Mon Sep 17 00:00:00 2001 From: BFlat Date: Tue, 17 Nov 2026 08:39:25 -0500 Subject: [PATCH 029/161] dex: retain invoke_registers regressions Source crates/formats/tests/dex_invoke_registers.rs at b9ca099903aeef5d4b33a7e83e8e7062ab94c65a. Imports adapted only. --- .../dex/src/regressions/invoke_registers.rs | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100644 crates/dex/src/regressions/invoke_registers.rs diff --git a/crates/dex/src/regressions/invoke_registers.rs b/crates/dex/src/regressions/invoke_registers.rs new file mode 100644 index 000000000..c2c6faab6 --- /dev/null +++ b/crates/dex/src/regressions/invoke_registers.rs @@ -0,0 +1,33 @@ +use crate::dex_ops::{decode, decode_one}; + +#[test] +fn fixed_register_lists_cannot_claim_more_than_five_arguments() { + for op in [0x24u16, 0x6e, 0x6f, 0x70, 0x71, 0x72, 0xfa, 0xfc] { + for count in 6..=15 { + let code = [(count << 12) | op, 1, 0x4321, 2]; + assert!(decode_one(&code, 0).is_none(), "opcode {op:#x}, count {count}"); + assert!(!decode(&code, 16).1); + } + } +} + +#[test] +fn range_arguments_cannot_leave_the_encoded_register_space() { + for op in [0x25u16, 0x74, 0x75, 0x76, 0x77, 0x78, 0xfb, 0xfd] { + assert!(decode_one(&[0x0200 | op, 1, u16::MAX, 2], 0).is_none(), "{op:#x}"); + assert!(decode_one(&[0xff00 | op, 1, u16::MAX - 1, 2], 0).is_none(), "{op:#x}"); + } +} + +#[test] +fn zero_and_boundary_register_lists_remain_valid() { + let fixed = decode_one(&[0x5571, 1, 0x4321], 0).unwrap(); + assert_eq!(fixed.regs, vec![1, 2, 3, 4, 5]); + assert!(decode_one(&[0x0071, 1, 0], 0).unwrap().regs.is_empty()); + let range = decode_one(&[0x0277, 1, u16::MAX - 1], 0).unwrap(); + assert_eq!(range.regs, vec![65534, 65535]); + assert!(decode_one(&[0x0077, 1, u16::MAX], 0).unwrap().regs.is_empty()); + let polymorphic = decode_one(&[0x01fb, 1, u16::MAX, 2], 0).unwrap(); + assert_eq!(polymorphic.regs, vec![65535]); + assert_eq!(polymorphic.index2, Some(2)); +} From 7cd3dcfdece717c73edc7463ac09fbeaf6ddb9fd Mon Sep 17 00:00:00 2001 From: Afloat Date: Tue, 17 Nov 2026 08:39:25 -0500 Subject: [PATCH 030/161] dex: retain payload_budgets regressions Source crates/formats/tests/dex_payload_budgets.rs at 1ebdb78760d6bfd31f1a70e6389e0d441adb3d2e. Imports adapted only. --- crates/dex/src/regressions/payload_budgets.rs | 88 +++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 crates/dex/src/regressions/payload_budgets.rs diff --git a/crates/dex/src/regressions/payload_budgets.rs b/crates/dex/src/regressions/payload_budgets.rs new file mode 100644 index 000000000..8680373a4 --- /dev/null +++ b/crates/dex/src/regressions/payload_budgets.rs @@ -0,0 +1,88 @@ +use crate::dex_ops::{self, Payload}; + +fn arrays(references: usize, bytes: usize) -> Vec { + let mut code = Vec::new(); + let offset = (references * 3).next_multiple_of(2); + for n in 0..references { + let relative = u32::try_from(offset - n * 3).unwrap(); + code.extend([0x0026, relative as u16, (relative >> 16) as u16]); + } + code.resize(offset, 0); + let count = u32::try_from(bytes).unwrap(); + code.extend([0x0300, 1, count as u16, (count >> 16) as u16]); + code.resize(code.len() + bytes.div_ceil(2), 0x5a5a); + code +} + +fn bytes(rows: &[dex_ops::Insn]) -> usize { + rows.iter() + .map(|i| match i.payload.as_ref() { + Some(Payload::FillArrayData { data, .. }) => data.len(), + Some(Payload::PackedSwitch { targets, .. }) => targets.len() * 4, + Some(Payload::SparseSwitch { keys, targets }) => (keys.len() + targets.len()) * 4, + None => 0, + }) + .sum() +} + +#[test] +fn shared_arrays_charge_the_source_and_each_attached_copy() { + let code = arrays(4, 8); + let (rows, complete) = dex_ops::decode_with_payload_limit(&code, 100, 24); + assert!(!complete); + assert_eq!(bytes(&rows), 24); + assert!(rows[0].payload.is_some() && rows[1].payload.is_some()); + assert!(rows[2].payload.is_none() && rows[3].payload.is_none()); + assert!(rows.last().unwrap().payload.is_some()); + let (full, complete) = dex_ops::decode_with_payload_limit(&code, 100, 40); + assert!(complete); + assert_eq!(bytes(&full), 40); +} + +#[test] +fn zero_budget_keeps_offsets_and_payload_instruction_metadata() { + let code = arrays(2, 8); + let (rows, complete) = dex_ops::decode_with_payload_limit(&code, 100, 0); + assert!(!complete); + assert_eq!(rows.iter().map(|i| i.offset).collect::>(), [0, 3, 6]); + assert_eq!(rows[2].name, "array-data-payload"); + assert_eq!(rows[2].units, 8); + assert!(rows.iter().all(|i| i.payload.is_none())); + assert!(dex_ops::decode_with_payload_limit(&[0x000e], 100, 0).1); +} + +#[test] +fn both_switch_tables_are_accounted_and_backward_references_work() { + // packed table: one target; sparse table: one key and one target; users follow the tables. + let code = [0x0100, 1, 7, 0, 1, 0, 0x0200, 1, 9, 0, 2, 0, 0x002b, 0xfff4, 0xffff, 0x002c, 0xfff7, 0xffff]; + let (rows, complete) = dex_ops::decode_with_payload_limit(&code, 100, 24); + assert!(complete); + assert_eq!(rows[2].payload, Some(Payload::PackedSwitch { first_key: 7, targets: vec![1] })); + assert_eq!(rows[3].payload, Some(Payload::SparseSwitch { keys: vec![9], targets: vec![2] })); + let (rows, complete) = dex_ops::decode_with_payload_limit(&code, 100, 16); + assert!(!complete); + assert!(rows[2].payload.is_some()); + assert!(rows[3].payload.is_none()); + assert_eq!(bytes(&rows), 16); +} + +#[test] +fn the_existing_per_array_cap_is_reported_as_partial() { + let (rows, complete) = dex_ops::decode(&arrays(1, 65_537), 100); + assert!(!complete); + assert_eq!(bytes(&rows), 2 * 65_536); + let (rows, complete) = dex_ops::decode(&arrays(1, 8), 1); + assert!(!complete); + assert_eq!(rows.len(), 1); + assert!(rows[0].payload.is_none()); +} + +#[test] +#[ignore = "memory regression: run separately with a 128 MiB address-space limit"] +fn repeated_array_references_stay_under_the_default_byte_budget() { + let code = arrays(2048, 65_536); + let (rows, complete) = dex_ops::decode(&code, 200_000); + assert!(!complete); + assert_eq!(rows.len(), 2049); + assert_eq!(bytes(&rows), dex_ops::DEFAULT_MAX_PAYLOAD_BYTES); +} From 4039bc76955a61b44ce0838ea8d92315cfd955e0 Mon Sep 17 00:00:00 2001 From: Afloat <132809511+Afloat16@users.noreply.github.com> Date: Tue, 17 Nov 2026 08:39:25 -0500 Subject: [PATCH 031/161] dex: retain signed_leb128 regressions Source crates/formats/tests/dex_signed_leb128.rs at eab6a0f01bf9fc3a36798d322e4442e169394dfc. Imports adapted only. --- crates/dex/src/regressions/signed_leb128.rs | 94 +++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 crates/dex/src/regressions/signed_leb128.rs diff --git a/crates/dex/src/regressions/signed_leb128.rs b/crates/dex/src/regressions/signed_leb128.rs new file mode 100644 index 000000000..f38fd8db6 --- /dev/null +++ b/crates/dex/src/regressions/signed_leb128.rs @@ -0,0 +1,94 @@ +use crate::bytes::Reader; +use crate::dex::{self, Code, Handler, sleb128}; + +#[test] +fn rejects_invalid_fifth_byte_sign_extensions() { + for prefix in [0x80, 0xff] { + for last in 0x08..=0x77 { + let mut encoded = vec![prefix; 4]; + encoded.push(last); + assert_eq!(sleb128(&Reader::new(&encoded), 0), None, "last byte {last:#x}"); + } + } +} + +#[test] +fn retains_signed_32_bit_limits_and_ordinary_values() { + assert_eq!(sleb128(&Reader::new(&[0xff, 0xff, 0xff, 0xff, 0x07]), 0), Some((i64::from(i32::MAX), 5))); + assert_eq!(sleb128(&Reader::new(&[0x80, 0x80, 0x80, 0x80, 0x78]), 0), Some((i64::from(i32::MIN), 5))); + assert_eq!(sleb128(&Reader::new(&[0x9b, 0xf1, 0x59]), 0), Some((-624485, 3))); + assert_eq!(sleb128(&Reader::new(&[0xe5, 0x8e, 0x26]), 0), Some((624485, 3))); + assert_eq!(sleb128(&Reader::new(&[0xff, 0x7f]), 0), Some((-1, 2))); + assert_eq!(sleb128(&Reader::new(&[0x80, 0]), 0), Some((0, 2))); + assert_eq!(sleb128(&Reader::new(&[0x55, 0x7f, 0x55]), 1), Some((-1, 1))); +} + +#[test] +fn retains_valid_five_byte_sign_extensions() { + assert_eq!(sleb128(&Reader::new(&[0x80, 0x80, 0x80, 0x80, 0]), 0), Some((0, 5))); + assert_eq!(sleb128(&Reader::new(&[0x81, 0x80, 0x80, 0x80, 0]), 0), Some((1, 5))); + assert_eq!(sleb128(&Reader::new(&[0xff, 0xff, 0xff, 0xff, 0x7f]), 0), Some((-1, 5))); +} + +#[test] +fn rejects_values_longer_than_five_bytes() { + for (prefix, last) in [(0x80, 0), (0xff, 0x7f)] { + for length in [6, 10] { + let mut encoded = vec![prefix; length - 1]; + encoded.push(last); + assert_eq!(sleb128(&Reader::new(&encoded), 0), None, "{encoded:x?}"); + } + } +} + +#[test] +fn rejects_truncated_values_and_out_of_range_offsets() { + assert_eq!(sleb128(&Reader::new(&[0x80; 5]), 0), None); + assert_eq!(sleb128(&Reader::new(&[0x80]), 0), None); + assert_eq!(sleb128(&Reader::new(&[]), 0), None); + assert_eq!(sleb128(&Reader::new(&[0]), usize::MAX), None); +} + +fn code_with_handler(size: &[u8], payload: &[u8]) -> Code { + let mut bytes = vec![0; 0x70]; + bytes[..8].copy_from_slice(b"dex\n035\0"); + bytes[36..40].copy_from_slice(&0x70u32.to_le_bytes()); + bytes[40..44].copy_from_slice(&0x1234_5678u32.to_le_bytes()); + // Eight nop instructions and one try entry pointing to the first encoded handler. + bytes.extend_from_slice(&[1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 8, 0, 0, 0]); + bytes.extend_from_slice(&[0; 16]); + bytes.extend_from_slice(&[0, 0, 0, 0, 1, 0, 1, 0]); + bytes.push(1); + bytes.extend_from_slice(size); + bytes.extend_from_slice(payload); + let len = bytes.len() as u32; + bytes[32..36].copy_from_slice(&len.to_le_bytes()); + dex::parse(&bytes).unwrap().code(0x70).unwrap() +} + +#[test] +fn invalid_sizes_do_not_create_handlers() { + let sizes: &[&[u8]] = &[ + &[0x80, 0x80, 0x80, 0x80, 0x08], + &[0x80, 0x80, 0x80, 0x80, 0x77], + &[0x80, 0x80, 0x80, 0x80, 0x80, 0], + &[0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0x80, 0], + ]; + for size in sizes { + // These bytes could be a type/address pair or a catch-all address if size is accepted. + let code = code_with_handler(size, &[0, 7]); + assert_eq!(code.tries.len(), 1); + assert!(code.tries[0].handlers.is_empty(), "invalid size {size:x?}"); + } +} + +#[test] +fn valid_catch_all_handlers_are_retained() { + for size in [&[0][..], &[0x80, 0x80, 0x80, 0x80, 0][..]] { + let code = code_with_handler(size, &[7]); + assert_eq!(code.tries.len(), 1); + assert_eq!(code.tries[0].start, 0); + assert_eq!(code.tries[0].end, 1); + assert_eq!(code.tries[0].handlers, vec![Handler { exception: None, address: 7 }]); + } +} From 66f8c510489bdb2729b30538bfec44f457cc3e64 Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Tue, 17 Nov 2026 08:40:02 -0500 Subject: [PATCH 032/161] Expose shared DEX inspection and bytecode decoding --- ATTRIBUTION.md | 4 + Cargo.lock | 8 + Cargo.toml | 1 + crates/dex/Cargo.toml | 13 + crates/dex/src/bytes.rs | 81 ++++++ crates/dex/src/dex.rs | 28 +- crates/dex/src/dex_ops.rs | 8 +- crates/dex/src/lib.rs | 42 +++ crates/dex/src/test_support.rs | 471 +++++++++++++++++++++++++++++++++ 9 files changed, 643 insertions(+), 13 deletions(-) create mode 100644 crates/dex/Cargo.toml create mode 100644 crates/dex/src/bytes.rs create mode 100644 crates/dex/src/lib.rs create mode 100644 crates/dex/src/test_support.rs diff --git a/ATTRIBUTION.md b/ATTRIBUTION.md index 4c86afd61..cbf153813 100644 --- a/ATTRIBUTION.md +++ b/ATTRIBUTION.md @@ -132,3 +132,7 @@ Sanitization, animation fitting and numeric output policy remain caller-owned. ## JSONC span-preserving editing `crates/jsonc` is extracted from Craft Anything `crates/formats/src/jsonc.rs` at `e5c9c94106747ed44b9a3fb0decce867c2285f93` (MIT OR Apache-2.0). The original implementation and hardening by BFlat, newline preservation by NetterSilver, and subsequent RRFZ/Afloat correctness fixes retain separate primary-author port commits with immutable source revisions. Shared API wiring and optional input-byte admission are subsequent changes. Compatibility parsing and all existing formatting/editing tests are retained; application file I/O and policy are not imported. + +## Dalvik DEX inspection + +`crates/dex` extracts Craft Anything `crates/formats/src/dex.rs` and `dex_ops.rs` at `5979f9b13f16dcf1df3c7ae7f8e7735f452866f2` (MIT OR Apache-2.0). Five source-history ports preserve Brandon Thomas, BFlat and Afloat as primary authors. Five regression imports retain BFlat/Afloat authors and exact source revisions. The checked byte-reader/MUTF-8 subset and synthetic DEX fixture come from that app's `bytes.rs` and `testkit/src/android.rs`; deterministic noise/mutations come from its testkit. No application file I/O, APK logic or execution is imported. Shared neutral errors and workspace wiring are subsequent changes; existing per-file strings, member/handler and bytecode payload budgets are retained. diff --git a/Cargo.lock b/Cargo.lock index 9fa1f4401..720433ee7 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1722,6 +1722,14 @@ dependencies = [ "thiserror 2.0.21", ] +[[package]] +name = "craft-dex" +version = "0.1.0" +dependencies = [ + "serde", + "thiserror 2.0.21", +] + [[package]] name = "craft-dialogs" version = "0.1.0" diff --git a/Cargo.toml b/Cargo.toml index 50ec3c917..803e433de 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -11,6 +11,7 @@ rust-version = "1.95" repository = "https://github.com/storytold/craft-libs" [workspace.dependencies] +craft-dex = { path = "crates/dex" } craft-jsonc = { path = "crates/jsonc" } craft-palette = { path = "crates/palette" } craft-score = { path = "crates/score" } diff --git a/crates/dex/Cargo.toml b/crates/dex/Cargo.toml new file mode 100644 index 000000000..e4eb285f5 --- /dev/null +++ b/crates/dex/Cargo.toml @@ -0,0 +1,13 @@ +[package] +name="craft-dex" +description="Dalvik DEX reader and bytecode decoder with retained input bounds and partial-report semantics" +version.workspace=true +edition.workspace=true +license.workspace=true +rust-version.workspace=true +repository.workspace=true +[lints] +workspace=true +[dependencies] +serde={workspace=true} +thiserror={workspace=true} diff --git a/crates/dex/src/bytes.rs b/crates/dex/src/bytes.rs new file mode 100644 index 000000000..555505a2d --- /dev/null +++ b/crates/dex/src/bytes.rs @@ -0,0 +1,81 @@ +//! Checked little/big-endian reads over untrusted bytes. Every read returns `None` when out of range. + +#[derive(Debug, Clone, Copy)] +pub struct Reader<'a> { + data: &'a [u8], +} + +impl<'a> Reader<'a> { + pub fn new(data: &'a [u8]) -> Self { + Reader { data } + } + + pub fn len(&self) -> usize { + self.data.len() + } + + pub fn data(&self) -> &'a [u8] { + self.data + } + + pub fn slice(&self, off: usize, len: usize) -> Option<&'a [u8]> { + let end = off.checked_add(len)?; + self.data.get(off..end) + } + + pub fn u8(&self, off: usize) -> Option { + self.data.get(off).copied() + } + + fn arr(&self, off: usize) -> Option<[u8; N]> { + self.slice(off, N)?.try_into().ok() + } + + pub fn u16_le(&self, off: usize) -> Option { + self.arr(off).map(u16::from_le_bytes) + } + + pub fn u32_le(&self, off: usize) -> Option { + self.arr(off).map(u32::from_le_bytes) + } + + /// A NUL-terminated string starting at `off`, at most `max` bytes, lossily decoded. + pub fn cstr(&self, off: usize, max: usize) -> Option { + let rest = self.data.get(off..)?; + let rest = rest.get(..max.min(rest.len()))?; + let end = rest.iter().position(|&b| b == 0).unwrap_or(rest.len()); + Some(String::from_utf8_lossy(rest.get(..end)?).into_owned()) + } + + /// Unsigned LEB128 at `off`: (value, bytes consumed). At most 10 bytes. + pub fn uleb128(&self, off: usize) -> Option<(u64, usize)> { + let mut result = 0u64; + for i in 0..10usize { + let b = self.u8(off.checked_add(i)?)?; + let payload = u64::from(b & 0x7f); + let shift = 7 * i as u32; + if shift > 0 && payload.checked_shl(shift)? >> shift != payload { + return None; + } + result |= payload << shift; + if b & 0x80 == 0 { + return Some((result, i + 1)); + } + } + None + } +} + +/// Decode one modified UTF-8 code unit. Invalid bytes consume only themselves, +/// so a damaged sequence cannot hide valid text or borrow missing bytes. +pub(crate) fn mutf8_unit(bytes: &[u8]) -> (u16, usize) { + match bytes { + [a @ 0x00..=0x7f, ..] => (u16::from(*a), 1), + [0xc0, 0x80, ..] => (0, 2), + [a @ 0xc2..=0xdf, b @ 0x80..=0xbf, ..] => ((u16::from(*a & 0x1f) << 6) | u16::from(*b & 0x3f), 2), + [a @ 0xe0..=0xef, b @ 0x80..=0xbf, c @ 0x80..=0xbf, ..] if *a != 0xe0 || *b >= 0xa0 => { + ((u16::from(*a & 0x0f) << 12) | (u16::from(*b & 0x3f) << 6) | u16::from(*c & 0x3f), 3) + } + _ => (0xfffd, 1), + } +} diff --git a/crates/dex/src/dex.rs b/crates/dex/src/dex.rs index c4076dfc8..e578dc397 100644 --- a/crates/dex/src/dex.rs +++ b/crates/dex/src/dex.rs @@ -348,8 +348,11 @@ pub fn parse(data: &[u8]) -> Result, FormatError> { ProtoId { shorty: r.u32_le(o).unwrap_or(0), ret: r.u32_le(o + 4).unwrap_or(NO_INDEX), params_off: r.u32_le(o + 8).unwrap_or(0) } }) .collect(); - let member = - |o: usize| MemberId { class_idx: r.u16_le(o).unwrap_or(0), type_or_proto: r.u16_le(o + 2).unwrap_or(0), name_idx: r.u32_le(o + 4).unwrap_or(NO_INDEX) }; + let member = |o: usize| MemberId { + class_idx: r.u16_le(o).unwrap_or(0), + type_or_proto: r.u16_le(o + 2).unwrap_or(0), + name_idx: r.u32_le(o + 4).unwrap_or(NO_INDEX), + }; let fields = (0..nf).map(|i| member(fo + i * 8)).collect(); let methods = (0..nm).map(|i| member(mo + i * 8)).collect(); let classes = (0..nc) @@ -423,7 +426,9 @@ impl<'a> Dex<'a> { /// `Lcls;->name:Ltype;` pub fn field_ref(&self, i: u32) -> String { match self.fields.get(i as usize) { - Some(f) => format!("{}->{}:{}", self.type_desc(u32::from(f.class_idx)), self.string(f.name_idx), self.type_desc(u32::from(f.type_or_proto))), + Some(f) => { + format!("{}->{}:{}", self.type_desc(u32::from(f.class_idx)), self.string(f.name_idx), self.type_desc(u32::from(f.type_or_proto))) + } None => format!(""), } } @@ -431,7 +436,9 @@ impl<'a> Dex<'a> { /// `Lcls;->name(…)R` pub fn method_ref(&self, i: u32) -> String { match self.methods.get(i as usize) { - Some(m) => format!("{}->{}{}", self.type_desc(u32::from(m.class_idx)), self.string(m.name_idx), self.proto_desc(u32::from(m.type_or_proto))), + Some(m) => { + format!("{}->{}{}", self.type_desc(u32::from(m.class_idx)), self.string(m.name_idx), self.proto_desc(u32::from(m.type_or_proto))) + } None => format!(""), } } @@ -683,9 +690,10 @@ pub fn references(dex: &Dex<'_>, desc: &str, method: Option<&str>, max: usize) - let cls = dex.type_desc(u32::from(id.class_idx)); (cls == desc && method.is_none_or(|n| dex.string(id.name_idx) == n)).then(|| ("method", dex.method_ref(idx))) }), - IndexKind::Field if method.is_none() => { - dex.fields.get(idx as usize).and_then(|id| (dex.type_desc(u32::from(id.class_idx)) == desc).then(|| ("field", dex.field_ref(idx)))) - } + IndexKind::Field if method.is_none() => dex + .fields + .get(idx as usize) + .and_then(|id| (dex.type_desc(u32::from(id.class_idx)) == desc).then(|| ("field", dex.field_ref(idx)))), IndexKind::Type if method.is_none() => { let t = dex.type_desc(idx); (t.trim_start_matches('[') == desc).then_some(("type", t)) @@ -707,7 +715,7 @@ pub fn references(dex: &Dex<'_>, desc: &str, method: Option<&str>, max: usize) - #[cfg(test)] mod tests { use super::*; - use craft_anything_testkit::android as fx; + use crate::test_support::android as fx; #[test] fn parses_fixture() { @@ -785,7 +793,7 @@ mod tests { } let _ = references(d, "Lcom/example/notes/Util;", None, 100); }; - for m in craft_anything_testkit::mutations(&data) { + for m in crate::test_support::mutations(&data) { if let Ok(d) = parse(&m) { walk(&d); } @@ -796,7 +804,7 @@ mod tests { } } for seed in 0..64 { - let mut n = craft_anything_testkit::noise(seed, 2048); + let mut n = crate::test_support::noise(seed, 2048); n[..8].copy_from_slice(b"dex\n035\0"); n[40..44].copy_from_slice(&0x1234_5678u32.to_le_bytes()); if let Ok(d) = parse(&n) { diff --git a/crates/dex/src/dex_ops.rs b/crates/dex/src/dex_ops.rs index 71095fc75..3c27d4c5c 100644 --- a/crates/dex/src/dex_ops.rs +++ b/crates/dex/src/dex_ops.rs @@ -170,7 +170,8 @@ const BINOPS_2ADDR: [&str; 32] = [ "rem-double/2addr", ]; -const LIT16: [&str; 8] = ["add-int/lit16", "rsub-int", "mul-int/lit16", "div-int/lit16", "rem-int/lit16", "and-int/lit16", "or-int/lit16", "xor-int/lit16"]; +const LIT16: [&str; 8] = + ["add-int/lit16", "rsub-int", "mul-int/lit16", "div-int/lit16", "rem-int/lit16", "and-int/lit16", "or-int/lit16", "xor-int/lit16"]; const LIT8: [&str; 11] = [ "add-int/lit8", "rsub-int/lit8", @@ -236,7 +237,8 @@ const SFIELD: [&str; 14] = [ "sput-short", ]; const INVOKE: [&str; 5] = ["invoke-virtual", "invoke-super", "invoke-direct", "invoke-static", "invoke-interface"]; -const INVOKE_RANGE: [&str; 5] = ["invoke-virtual/range", "invoke-super/range", "invoke-direct/range", "invoke-static/range", "invoke-interface/range"]; +const INVOKE_RANGE: [&str; 5] = + ["invoke-virtual/range", "invoke-super/range", "invoke-direct/range", "invoke-static/range", "invoke-interface/range"]; fn pick(list: &[&'static str], i: u8, base: u8) -> &'static str { list.get(usize::from(i.wrapping_sub(base))).copied().unwrap_or("unused") @@ -683,7 +685,7 @@ mod tests { #[test] fn hostile() { for seed in 0..200 { - let n = craft_anything_testkit::noise(seed, 256); + let n = crate::test_support::noise(seed, 256); let code: Vec = n.as_chunks::<2>().0.iter().map(|c| u16::from_le_bytes(*c)).collect(); let _ = decode(&code, 1000); } diff --git a/crates/dex/src/lib.rs b/crates/dex/src/lib.rs new file mode 100644 index 000000000..97d28b7f4 --- /dev/null +++ b/crates/dex/src/lib.rs @@ -0,0 +1,42 @@ +#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] +#![forbid(unsafe_code)] +//! Dalvik DEX inspection and bytecode decoding. Never executes bytecode. +mod bytes; +/// Checked byte reader accepted by the signed-LEB compatibility helper. +pub use bytes::Reader; +pub mod dex; +pub mod dex_ops; +pub use dex::*; +#[cfg(test)] +mod test_support; +/// Parser admission failures; damaged substructures retain the original partial-report behavior. +#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error)] +pub enum DexError { + #[error("the file ends early ({0})")] + Truncated(String), + #[error("the file is malformed ({0})")] + Malformed(String), + #[error("not supported: {0}")] + Unsupported(String), +} +use DexError as FormatError; + +#[cfg(test)] +#[path = "regressions/array_widths.rs"] +mod array_widths; + +#[cfg(test)] +#[path = "regressions/capped_table_boundaries.rs"] +mod capped_table_boundaries; + +#[cfg(test)] +#[path = "regressions/invoke_registers.rs"] +mod invoke_registers; + +#[cfg(test)] +#[path = "regressions/payload_budgets.rs"] +mod payload_budgets; + +#[cfg(test)] +#[path = "regressions/signed_leb128.rs"] +mod signed_leb128; diff --git a/crates/dex/src/test_support.rs b/crates/dex/src/test_support.rs new file mode 100644 index 000000000..a3a179145 --- /dev/null +++ b/crates/dex/src/test_support.rs @@ -0,0 +1,471 @@ +// Synthetic DEX fixture and deterministic mutations retained from Craft Anything. +/// Deterministic pseudo-random bytes (xorshift) for fuzz-like loops. +pub fn noise(seed: u64, len: usize) -> Vec { + let mut x = seed.wrapping_mul(0x9e37_79b9_7f4a_7c15) | 1; + (0..len) + .map(|_| { + x ^= x << 13; + x ^= x >> 7; + x ^= x << 17; + x as u8 + }) + .collect() +} + +/// Every way to damage a sample used by robustness tests: truncations at many lengths and +/// single-byte corruptions spread across the file. +pub fn mutations(sample: &[u8]) -> Vec> { + let mut out = Vec::new(); + let n = sample.len(); + let step = (n / 64).max(1); + let mut len = 0; + while len < n { + out.push(sample[..len].to_vec()); + len += step; + } + for (i, seed) in (0..n).step_by(step).zip(1u64..) { + let mut m = sample.to_vec(); + let noise = noise(seed, 8); + for (j, v) in noise.iter().enumerate() { + if let Some(b) = m.get_mut(i + j) { + *b = *v; + } + } + out.push(m); + let mut m = sample.to_vec(); + if let Some(b) = m.get_mut(i) { + *b = 0xff; + } + out.push(m); + } + out +} + +pub mod android { + pub const HELLO: &str = "Hello, Craft!"; + fn u16le(v: &mut Vec, x: u16) { + v.extend_from_slice(&x.to_le_bytes()); + } + fn u32le(v: &mut Vec, x: u32) { + v.extend_from_slice(&x.to_le_bytes()); + } + fn uleb(v: &mut Vec, mut x: u32) { + loop { + let b = (x & 0x7f) as u8; + x >>= 7; + if x == 0 { + v.push(b); + break; + } + v.push(b | 0x80); + } + } + fn sleb(v: &mut Vec, mut x: i32) { + loop { + let b = (x & 0x7f) as u8; + x >>= 7; + let done = (x == 0 && b & 0x40 == 0) || (x == -1 && b & 0x40 != 0); + v.push(if done { b } else { b | 0x80 }); + if done { + break; + } + } + } + fn align4(v: &mut Vec) { + while !v.len().is_multiple_of(4) { + v.push(0); + } + } + + #[derive(Debug, Clone, Default)] + pub struct TrySpec { + pub start: u32, + pub count: u16, + /// (type idx, handler address) + pub handlers: Vec<(u32, u32)>, + pub catch_all: Option, + } + + #[derive(Debug, Clone, Default)] + pub struct CodeSpec { + pub registers: u16, + pub ins: u16, + pub outs: u16, + pub insns: Vec, + pub tries: Vec, + } + + #[derive(Debug, Clone)] + pub struct MethodSpec { + pub method: u32, + pub access: u32, + pub code: Option, + } + + #[derive(Debug, Clone, Default)] + pub struct ClassSpec { + pub class: u32, + pub access: u32, + pub superclass: u32, + pub interfaces: Vec, + pub source_file: u32, + pub annotations: Vec, + pub static_fields: Vec<(u32, u32)>, + pub instance_fields: Vec<(u32, u32)>, + pub direct: Vec, + pub virtuals: Vec, + } + + #[derive(Debug, Clone, Default)] + pub struct DexBuilder { + strings: Vec, + types: Vec, + protos: Vec<(u32, u32, Vec)>, + fields: Vec<(u32, u32, u32)>, + methods: Vec<(u32, u32, u32)>, + pub classes: Vec, + } + + impl DexBuilder { + pub fn string(&mut self, s: &str) -> u32 { + if let Some(i) = self.strings.iter().position(|x| x == s) { + return i as u32; + } + self.strings.push(s.to_string()); + (self.strings.len() - 1) as u32 + } + + pub fn ty(&mut self, desc: &str) -> u32 { + let s = self.string(desc); + if let Some(i) = self.types.iter().position(|&x| x == s) { + return i as u32; + } + self.types.push(s); + (self.types.len() - 1) as u32 + } + + pub fn proto(&mut self, ret: &str, params: &[&str]) -> u32 { + let short = |d: &str| if d.starts_with('L') || d.starts_with('[') { 'L' } else { d.chars().next().unwrap() }; + let shorty: String = std::iter::once(short(ret)).chain(params.iter().map(|p| short(p))).collect(); + let s = self.string(&shorty); + let r = self.ty(ret); + let ps: Vec = params.iter().map(|p| self.ty(p)).collect(); + if let Some(i) = self.protos.iter().position(|p| p.1 == r && p.2 == ps) { + return i as u32; + } + self.protos.push((s, r, ps)); + (self.protos.len() - 1) as u32 + } + + pub fn field(&mut self, cls: &str, ty: &str, name: &str) -> u32 { + let c = self.ty(cls); + let t = self.ty(ty); + let n = self.string(name); + if let Some(i) = self.fields.iter().position(|&f| f == (c, t, n)) { + return i as u32; + } + self.fields.push((c, t, n)); + (self.fields.len() - 1) as u32 + } + + pub fn method(&mut self, cls: &str, name: &str, ret: &str, params: &[&str]) -> u32 { + let c = self.ty(cls); + let p = self.proto(ret, params); + let n = self.string(name); + if let Some(i) = self.methods.iter().position(|&m| m == (c, p, n)) { + return i as u32; + } + self.methods.push((c, p, n)); + (self.methods.len() - 1) as u32 + } + + pub fn build(self) -> Vec { + let ns = self.strings.len(); + let nt = self.types.len(); + let np = self.protos.len(); + let nf = self.fields.len(); + let nm = self.methods.len(); + let nc = self.classes.len(); + let string_ids = 0x70; + let type_ids = string_ids + ns * 4; + let proto_ids = type_ids + nt * 4; + let field_ids = proto_ids + np * 12; + let method_ids = field_ids + nf * 8; + let class_defs = method_ids + nm * 8; + let mut data_start = class_defs + nc * 32; + data_start = (data_start + 3) & !3; + let mut d: Vec = Vec::new(); + let at = |d: &Vec| (data_start + d.len()) as u32; + // type lists + let tl = |d: &mut Vec, list: &[u32]| -> u32 { + if list.is_empty() { + return 0; + } + align4(d); + let o = at(d); + u32le(d, list.len() as u32); + for &t in list { + u16le(d, t as u16); + } + o + }; + let proto_params: Vec = self.protos.iter().map(|p| tl(&mut d, &p.2)).collect(); + let class_ifaces: Vec = self.classes.iter().map(|c| tl(&mut d, &c.interfaces)).collect(); + // string data + let mut string_offs = Vec::new(); + for s in &self.strings { + string_offs.push(at(&d)); + uleb(&mut d, s.encode_utf16().count() as u32); + for ch in s.chars() { + let c = ch as u32; + if c == 0 { + d.extend_from_slice(&[0xc0, 0x80]); + } else if c < 0x80 { + d.push(c as u8); + } else if c < 0x800 { + d.extend_from_slice(&[0xc0 | (c >> 6) as u8, 0x80 | (c & 0x3f) as u8]); + } else { + d.extend_from_slice(&[0xe0 | (c >> 12) as u8, 0x80 | ((c >> 6) & 0x3f) as u8, 0x80 | (c & 0x3f) as u8]); + } + } + d.push(0); + } + // code items, class data, annotations + let mut class_data_offs = Vec::new(); + let mut annot_offs = Vec::new(); + for c in &self.classes { + let code_off = |d: &mut Vec, m: &MethodSpec| -> u32 { + let Some(code) = &m.code else { return 0 }; + align4(d); + let o = at(d); + u16le(d, code.registers); + u16le(d, code.ins); + u16le(d, code.outs); + u16le(d, code.tries.len() as u16); + u32le(d, 0); + u32le(d, code.insns.len() as u32); + for &i in &code.insns { + u16le(d, i); + } + if !code.tries.is_empty() { + if code.insns.len() % 2 == 1 { + u16le(d, 0); + } + let mut list = Vec::new(); + uleb(&mut list, code.tries.len() as u32); + let mut offs = Vec::new(); + for t in &code.tries { + offs.push(list.len() as u16); + let n = t.handlers.len() as i32; + sleb(&mut list, if t.catch_all.is_some() { -n } else { n }); + for &(ty, addr) in &t.handlers { + uleb(&mut list, ty); + uleb(&mut list, addr); + } + if let Some(a) = t.catch_all { + uleb(&mut list, a); + } + } + for (t, off) in code.tries.iter().zip(offs) { + u32le(d, t.start); + u16le(d, t.count); + u16le(d, off); + } + d.extend_from_slice(&list); + } + o + }; + let mut direct = c.direct.clone(); + direct.sort_by_key(|m| m.method); + let mut virtuals = c.virtuals.clone(); + virtuals.sort_by_key(|m| m.method); + let direct_offs: Vec = direct.iter().map(|m| code_off(&mut d, m)).collect(); + let virtual_offs: Vec = virtuals.iter().map(|m| code_off(&mut d, m)).collect(); + let cd = at(&d); + let mut sf = c.static_fields.clone(); + sf.sort(); + let mut inf = c.instance_fields.clone(); + inf.sort(); + uleb(&mut d, sf.len() as u32); + uleb(&mut d, inf.len() as u32); + uleb(&mut d, direct.len() as u32); + uleb(&mut d, virtuals.len() as u32); + for list in [&sf, &inf] { + let mut prev = 0; + for &(f, a) in list.iter() { + uleb(&mut d, f - prev); + uleb(&mut d, a); + prev = f; + } + } + for (list, offs) in [(&direct, &direct_offs), (&virtuals, &virtual_offs)] { + let mut prev = 0; + for (m, &o) in list.iter().zip(offs.iter()) { + uleb(&mut d, m.method - prev); + uleb(&mut d, m.access); + uleb(&mut d, o); + prev = m.method; + } + } + class_data_offs.push(cd); + if c.annotations.is_empty() { + annot_offs.push(0); + } else { + let mut items = Vec::new(); + for &t in &c.annotations { + items.push(at(&d)); + d.push(1); // runtime visibility + uleb(&mut d, t); + uleb(&mut d, 0); + } + align4(&mut d); + let set_off = at(&d); + u32le(&mut d, items.len() as u32); + for i in items { + u32le(&mut d, i); + } + let dir = at(&d); + u32le(&mut d, set_off); + u32le(&mut d, 0); + u32le(&mut d, 0); + u32le(&mut d, 0); + annot_offs.push(dir); + } + } + align4(&mut d); + let file_size = data_start + d.len(); + let mut out = vec![0u8; data_start]; + out[..8].copy_from_slice(b"dex\n035\0"); + let put = |out: &mut Vec, off: usize, v: u32| out[off..off + 4].copy_from_slice(&v.to_le_bytes()); + put(&mut out, 32, file_size as u32); + put(&mut out, 36, 0x70); + put(&mut out, 40, 0x1234_5678); + for (i, (n, o)) in + [(ns, string_ids), (nt, type_ids), (np, proto_ids), (nf, field_ids), (nm, method_ids), (nc, class_defs)].iter().enumerate() + { + put(&mut out, 56 + i * 8, *n as u32); + put(&mut out, 60 + i * 8, if *n == 0 { 0 } else { *o as u32 }); + } + put(&mut out, 104, d.len() as u32); + put(&mut out, 108, data_start as u32); + for (i, o) in string_offs.iter().enumerate() { + put(&mut out, string_ids + i * 4, *o); + } + for (i, t) in self.types.iter().enumerate() { + put(&mut out, type_ids + i * 4, *t); + } + for (i, p) in self.protos.iter().enumerate() { + put(&mut out, proto_ids + i * 12, p.0); + put(&mut out, proto_ids + i * 12 + 4, p.1); + put(&mut out, proto_ids + i * 12 + 8, proto_params[i]); + } + for (i, (c, t, n)) in self.fields.iter().enumerate() { + out[field_ids + i * 8..field_ids + i * 8 + 2].copy_from_slice(&(*c as u16).to_le_bytes()); + out[field_ids + i * 8 + 2..field_ids + i * 8 + 4].copy_from_slice(&(*t as u16).to_le_bytes()); + put(&mut out, field_ids + i * 8 + 4, *n); + } + for (i, (c, p, n)) in self.methods.iter().enumerate() { + out[method_ids + i * 8..method_ids + i * 8 + 2].copy_from_slice(&(*c as u16).to_le_bytes()); + out[method_ids + i * 8 + 2..method_ids + i * 8 + 4].copy_from_slice(&(*p as u16).to_le_bytes()); + put(&mut out, method_ids + i * 8 + 4, *n); + } + for (i, c) in self.classes.iter().enumerate() { + let o = class_defs + i * 32; + for (k, v) in + [c.class, c.access, c.superclass, class_ifaces[i], c.source_file, annot_offs[i], class_data_offs[i], 0].iter().enumerate() + { + put(&mut out, o + k * 4, *v); + } + } + out.extend_from_slice(&d); + let sum = adler32(&out[12..]); + put(&mut out, 8, sum); + out + } + } + + fn adler32(data: &[u8]) -> u32 { + let (mut a, mut b) = (1u32, 0u32); + for &x in data { + a = (a + u32::from(x)) % 65521; + b = (b + a) % 65521; + } + (b << 16) | a + } + + const MAIN: &str = "Lcom/example/notes/MainActivity;"; + const UTIL: &str = "Lcom/example/notes/Util;"; + const STRING: &str = "Ljava/lang/String;"; + + /// `classes.dex` of the Notes app: `MainActivity` (constructor, `onCreate`, `run`, `pick` with a + /// packed switch inside a try/catch) and `Util` (two `log` overloads printing through + /// `System.out`). + pub fn notes_dex() -> Vec { + let mut b = DexBuilder::default(); + let activity = b.ty("Landroid/app/Activity;"); + let main = b.ty(MAIN); + let runnable = b.ty("Ljava/lang/Runnable;"); + let deprecated = b.ty("Ljava/lang/Deprecated;"); + let rte = b.ty("Ljava/lang/RuntimeException;"); + let object = b.ty("Ljava/lang/Object;"); + let util = b.ty(UTIL); + let tag = b.field(MAIN, STRING, "TAG"); + let count = b.field(MAIN, "I", "count"); + let out = b.field("Ljava/lang/System;", "Ljava/io/PrintStream;", "out"); + let super_init = b.method("Landroid/app/Activity;", "", "V", &[]); + let super_create = b.method("Landroid/app/Activity;", "onCreate", "V", &["Landroid/os/Bundle;"]); + let init = b.method(MAIN, "", "V", &[]); + let on_create = b.method(MAIN, "onCreate", "V", &["Landroid/os/Bundle;"]); + let run = b.method(MAIN, "run", "V", &[]); + let pick = b.method(MAIN, "pick", "I", &["I"]); + let log_s = b.method(UTIL, "log", "V", &[STRING]); + let log_i = b.method(UTIL, "log", "V", &["I"]); + let println = b.method("Ljava/io/PrintStream;", "println", "V", &[STRING]); + let hello = b.string(HELLO); + let main_src = b.string("MainActivity.java"); + let util_src = b.string("Util.java"); + let code = |registers, ins, outs, insns: Vec| Some(CodeSpec { registers, ins, outs, insns, tries: vec![] }); + b.classes.push(ClassSpec { + class: main, + access: 0x1, + superclass: activity, + interfaces: vec![runnable], + source_file: main_src, + annotations: vec![deprecated], + static_fields: vec![(tag, 0x1a)], + instance_fields: vec![(count, 0x2)], + direct: vec![MethodSpec { method: init, access: 0x10001, code: code(1, 1, 1, vec![0x1070, super_init as u16, 0x0000, 0x000e]) }], + virtuals: vec![ + MethodSpec { + method: on_create, + access: 0x1, + code: code(3, 2, 2, vec![0x001a, hello as u16, 0x1071, log_s as u16, 0x0000, 0x206f, super_create as u16, 0x0021, 0x000e]), + }, + MethodSpec { method: run, access: 0x1, code: code(1, 1, 1, vec![0x7012, 0x1071, log_i as u16, 0x0000, 0x000e]) }, + MethodSpec { + method: pick, + access: 0x1, + code: Some(CodeSpec { + registers: 3, + ins: 2, + outs: 0, + insns: vec![0x022b, 10, 0, 0x0012, 0x000f, 0x1012, 0x000f, 0x2012, 0x000f, 0x0000, 0x0100, 2, 1, 0, 5, 0, 7, 0], + tries: vec![TrySpec { start: 0, count: 3, handlers: vec![(rte, 3)], catch_all: Some(3) }], + }), + }, + ], + }); + b.classes.push(ClassSpec { + class: util, + access: 0x11, + superclass: object, + source_file: util_src, + direct: vec![ + MethodSpec { method: log_s, access: 0x9, code: code(2, 1, 2, vec![0x0062, out as u16, 0x206e, println as u16, 0x0010, 0x000e]) }, + MethodSpec { method: log_i, access: 0x9, code: code(1, 1, 0, vec![0x000e]) }, + ], + ..ClassSpec::default() + }); + b.build() + } +} From 60f7c5d74f6e5dfda2fa5779cff0b57d8fd45263 Mon Sep 17 00:00:00 2001 From: techneissa <55485515+techneissa@users.noreply.github.com> Date: Tue, 17 Nov 2026 16:53:35 +0300 Subject: [PATCH 033/161] raw: read a DNG strip whose byte count is only ActiveArea wide in full (#2040) An uncompressed single-sample integer DNG strip whose StripByteCounts is exactly ActiveArea columns (from column 0) by ImageLength rows, while the file holds the whole ImageWidth by ImageLength raster from the strip's offset, is read at its declared size instead of refused. Every other shortfall stays Corrupt. --- crates/raw/src/dng.rs | 37 +++++++++++++- crates/raw/src/dng/compat_review_tests.rs | 59 +++++++++++++++++++++++ 2 files changed, 95 insertions(+), 1 deletion(-) diff --git a/crates/raw/src/dng.rs b/crates/raw/src/dng.rs index efa1ec4f5..a7be5d67b 100644 --- a/crates/raw/src/dng.rs +++ b/crates/raw/src/dng.rs @@ -8,6 +8,7 @@ use crate::{ BlackLevel, Camera, Cfa, CinemaTags, ColorData, LevelSource, LevelSources, Limits, Matrix3, Mode, Orientation, RawData, RawError, RawFormat, RawImage, RawNotices, Rect, Result, Timecode, opcodes, }; +use craft_tiff::image::{ImageInfo, Layout}; use craft_tiff::tags::{self as t, photometric}; use craft_tiff::{ByteOrder, Ifd, Tiff, Value}; @@ -273,7 +274,10 @@ fn decode_ifd(bytes: &[u8], tiff: &Tiff, pixels: &Ifd, main: &Ifd, mode: Mode, f { return Err(RawError::Unsupported(format!("DNG compression {c} (GoPro VC-5) is not decoded yet"))); } - let info = pixels.image()?; + let mut info = pixels.image()?; + if std::ptr::eq(pixels, raw) { + extend_understated_strip(bytes, &mut info, raw); + } let (w, h, cpp) = (info.width as usize, info.height as usize, info.samples_per_pixel as usize); // one sample (CFA or monochrome), RGB or four colours; two are neither and can't be developed (storytold/lightcraft#1511) if !matches!(cpp, 1 | 3 | 4) { @@ -522,6 +526,37 @@ fn cinema_tags(ifd0: &Ifd, raw: &Ifd) -> CinemaTags { } } +/// Trust the file over `StripByteCounts` for an uncompressed single-strip image whose byte count is exactly +/// `ActiveArea` columns by `ImageLength` rows while the file holds the whole `ImageWidth` by `ImageLength` raster from +/// the strip's offset on. DJI Mavic 3 DNGs write the count of the active columns (5280) for a strip of full 5376-wide +/// rows that ends at the end of the file, so reading only the counted bytes refuses a complete image +/// (storytold/craft-libs#2040). The strip is lengthened only then, so every other short strip is still refused, and +/// it never reaches past the end of the file. The active area has to start at column 0, so it is the stored rows' +/// leading columns. +fn extend_understated_strip(bytes: &[u8], info: &mut ImageInfo, raw: &Ifd) { + let bits = u64::from(info.bits()); + if !matches!(info.layout, Layout::Strips { .. }) + || info.compression != t::compression::NONE + || info.samples_per_pixel != 1 + || info.planar == 2 + || info.sample_format == 3 + || !matches!(bits, 8 | 16) + { + return; + } + let ([offset], [count]) = (&info.offsets[..], &info.byte_counts[..]) else { return }; + let Some([top, left, bottom, right]) = raw.u64s(t::ACTIVE_AREA).and_then(|a| <[u64; 4]>::try_from(a).ok()) else { return }; + if left != 0 || bottom <= top || right == 0 || right >= u64::from(info.width) { + return; + } + let row_bytes = |width: u64| width.checked_mul(u64::from(info.height))?.checked_mul(bits / 8); + let (Some(counted), Some(needed)) = (row_bytes(right), row_bytes(u64::from(info.width))) else { return }; + let fits = offset.checked_add(needed).is_some_and(|end| end <= bytes.len() as u64); + if *count == counted && fits { + info.byte_counts = vec![needed]; + } +} + /// The default crop is relative to ActiveArea. DNG 1.7.1 defines independent /// defaults: origin (0, 0), size ImageWidth/ImageLength. An unusable optional crop /// preserves the entire active image. Valid partly overlapping rectangles retain the diff --git a/crates/raw/src/dng/compat_review_tests.rs b/crates/raw/src/dng/compat_review_tests.rs index 48300cbf7..309d1e5b7 100644 --- a/crates/raw/src/dng/compat_review_tests.rs +++ b/crates/raw/src/dng/compat_review_tests.rs @@ -118,3 +118,62 @@ fn decoded_notices_record_actual_crop_gain_table_and_cfa_decisions() { let limits = Limits { max_samples: 47, ..Default::default() }; assert!(matches!(decode_as(&linear(&[]), Mode::Full, RawFormat::Dng, &limits), Err(RawError::Limit(_)))); } + +/// A 16-bit CFA DNG of `width` x `height` whose strip holds every row in full but whose `StripByteCounts` says +/// `counted` bytes. +fn understated_strip_dng(width: u32, height: u32, counted: u32, active_area: Option<[u32; 4]>) -> Vec { + let mut raw = IfdBuilder::new(); + for (tag, value) in [ + (t::IMAGE_WIDTH, Value::Long(vec![width])), + (t::IMAGE_LENGTH, Value::Long(vec![height])), + (t::BITS_PER_SAMPLE, Value::Short(vec![16])), + (t::SAMPLES_PER_PIXEL, Value::Short(vec![1])), + (t::PHOTOMETRIC, Value::Short(vec![photometric::CFA])), + (t::COMPRESSION, Value::Short(vec![1])), + (t::CFA_REPEAT_PATTERN_DIM, Value::Short(vec![2, 2])), + (t::CFA_PATTERN_EP, Value::Byte(vec![0, 1, 1, 2])), + ] { + raw.set(tag, value); + } + if let Some(area) = active_area { + raw.set(t::ACTIVE_AREA, Value::Long(area.to_vec())); + } + let samples = (width * height) as u16; + raw.set_image(ImageData::Strips { rows_per_strip: height, strips: vec![(0..samples).flat_map(u16::to_le_bytes).collect()] }); + let mut bytes = TiffWriter::new(ByteOrder::Little, false).write(&[raw]).unwrap(); + // the writer counts the real bytes: understate that one count, a LONG in the IFD + let count = (width * height * 2).to_le_bytes(); + let at = bytes.windows(4).position(|w| w == count).unwrap(); + bytes[at..at + 4].copy_from_slice(&counted.to_le_bytes()); + bytes +} + +/// storytold/craft-libs#2040: the DJI Mavic 3 counts 5280 x 3956 x 2 bytes for a strip that holds 5376-wide rows +/// to the end of the file. +#[test] +fn a_strip_counted_only_active_area_wide_is_read_in_full() { + let bytes = understated_strip_dng(8, 4, 6 * 4 * 2, Some([0, 0, 4, 6])); + for mode in [Mode::Full, Mode::Header] { + let img = decode_as(&bytes, mode, RawFormat::Dng, &Limits::default()).unwrap(); + assert_eq!((img.width, img.height), (8, 4)); + } + let img = decode_as(&bytes, Mode::Full, RawFormat::Dng, &Limits::default()).unwrap(); + let RawData::U16(v) = &img.data else { panic!("integer samples") }; + assert_eq!(v, &(0..32).collect::>()); +} + +/// Any other short strip is still a corrupt file. +#[test] +fn other_short_strips_are_still_refused() { + let refused = |bytes: Vec| matches!(decode_as(&bytes, Mode::Full, RawFormat::Dng, &Limits::default()), Err(RawError::Corrupt(_))); + // no ActiveArea to explain the count + assert!(refused(understated_strip_dng(8, 4, 6 * 4 * 2, None))); + // the count is not exactly the area's bytes + assert!(refused(understated_strip_dng(8, 4, 5 * 4 * 2, Some([0, 0, 4, 6])))); + // the area does not start at column 0 + assert!(refused(understated_strip_dng(8, 4, 6 * 4 * 2, Some([0, 2, 4, 8])))); + // the file does not hold the whole raster + let mut cut = understated_strip_dng(8, 4, 6 * 4 * 2, Some([0, 0, 4, 6])); + cut.truncate(cut.len() - 8); + assert!(refused(cut)); +} From c93a6057fe722107d6b465b3a0daead58ef3a439 Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Tue, 17 Nov 2026 08:54:38 -0500 Subject: [PATCH 034/161] Retain is_empty on the public checked byte reader --- crates/dex/src/bytes.rs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/crates/dex/src/bytes.rs b/crates/dex/src/bytes.rs index 555505a2d..3e69ae4fd 100644 --- a/crates/dex/src/bytes.rs +++ b/crates/dex/src/bytes.rs @@ -14,6 +14,10 @@ impl<'a> Reader<'a> { self.data.len() } + pub fn is_empty(&self) -> bool { + self.data.is_empty() + } + pub fn data(&self) -> &'a [u8] { self.data } From 9ab8aac5ca01d34fca12bcb5a84679b155d1e2c7 Mon Sep 17 00:00:00 2001 From: Bra1nZ <95569966+Bra1nZ@users.noreply.github.com> Date: Tue, 17 Nov 2026 17:41:20 +0300 Subject: [PATCH 035/161] craft-docx: a package-wide XML element budget bounds hostile files' tree memory --- crates/docx/src/lib.rs | 4 +- crates/docx/src/package.rs | 135 +++++++++++++++++++- crates/docx/src/read.rs | 12 +- crates/docx/src/read/chunk.rs | 165 ++++++++++++++++++++++++- crates/docx/src/xml.rs | 65 +++++++--- crates/docx/tests/it/element_budget.rs | 44 +++++++ crates/docx/tests/it/main.rs | 1 + 7 files changed, 391 insertions(+), 35 deletions(-) create mode 100644 crates/docx/tests/it/element_budget.rs diff --git a/crates/docx/src/lib.rs b/crates/docx/src/lib.rs index c3c3e8df3..68d41ab4d 100644 --- a/crates/docx/src/lib.rs +++ b/crates/docx/src/lib.rs @@ -3,8 +3,8 @@ //! [`read`] turns a `.docx` package into a [`craft_wordmodel::Document`]; [`write`] does the //! reverse. Both are written from the public specification. The reader is lenient (unknown //! markup is skipped, bad numbers fall back to defaults) and bounded (zip entry sizes, total -//! decompressed size, XML depth and element counts are capped), so hostile input yields an -//! error or a best-effort document, never a panic. +//! decompressed size, XML depth, element counts per part and a package-wide element budget are +//! capped), so hostile input yields an error or a best-effort document, never a panic. #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] // The code came from an app whose lints allow this. #![allow(clippy::type_complexity)] diff --git a/crates/docx/src/package.rs b/crates/docx/src/package.rs index 3c7f45d94..6f90f56ba 100644 --- a/crates/docx/src/package.rs +++ b/crates/docx/src/package.rs @@ -1,5 +1,6 @@ //! OPC package access: bounded zip reading, relationships, part names. +use std::cell::Cell; use std::collections::{BTreeMap, HashMap}; use std::sync::Arc; @@ -216,6 +217,14 @@ fn opc_error(e: craft_opc::Error) -> DocxError { /// document keeps as it is (a picture, an embedded object) is handed over rather than copied. pub struct Package { opc: craft_opc::Package, + /// Elements this package's XML parts may still be parsed into. Every XML part read through + /// [`Package::xml`] charges it, so a file of many parts each inside the per-part cap builds + /// bounded tree memory in total — a chart part with no series builds a maximal tree for + /// almost no graphic work, and the trees the reader keeps accumulate. + budget: Cell, + /// What [`Package::reset_budget`] restores: the cap the package was opened with (the default, + /// or a smaller one from [`Package::open_limited_within`]). + budget_cap: usize, } impl Package { @@ -227,8 +236,16 @@ impl Package { /// `max_total` (counting entries that fail their checksum), is an error. A damaged entry is /// left out with a logged warning. pub fn open_limited(bytes: &[u8], max_part: u64, max_total: u64) -> Result { + Package::open_limited_within(bytes, max_part, max_total, xml::MAX_ELEMENTS_TOTAL) + } + + /// [`open_limited`] with an XML element budget of its own (nested documents share their + /// container's, tests use a small one). + pub fn open_limited_within(bytes: &[u8], max_part: u64, max_total: u64, max_elements: usize) -> Result { let limits = craft_opc::Limits::new(MAX_ENTRIES, max_part, max_total); - craft_opc::Package::open(bytes, &limits).map(|opc| Package { opc }).map_err(opc_error) + craft_opc::Package::open(bytes, &limits) + .map(|opc| Package { opc, budget: Cell::new(max_elements), budget_cap: max_elements }) + .map_err(opc_error) } /// The bytes of all its parts. @@ -244,7 +261,27 @@ impl Package { /// A package from parts already in memory (a Flat OPC file, a converted document), by part /// name without the leading `/`. pub fn from_parts(files: BTreeMap>) -> Package { - Package { opc: craft_opc::Package::from_parts(files) } + Package { opc: craft_opc::Package::from_parts(files), budget: Cell::new(xml::MAX_ELEMENTS_TOTAL), budget_cap: xml::MAX_ELEMENTS_TOTAL } + } + + /// Start a read's element budget over: the budget is per read, not per package's lifetime. + pub(crate) fn reset_budget(&self) { + self.budget.set(self.budget_cap); + } + + /// The XML elements left of the package's budget: nested documents open within it. + pub(crate) fn elements_left(&self) -> usize { + self.budget.get() + } + + /// Charge `count` elements a parsed part built against the budget. + pub(crate) fn charge(&self, count: usize) -> Result<(), DocxError> { + let left = self.budget.get(); + if count > left { + return Err(DocxError::Limit(format!("{count} XML elements, {left} left of the package's {} budget", self.budget_cap))); + } + self.budget.set(left - count); + Ok(()) } /// Part bytes by name (leading `/` ignored; case-insensitive). @@ -261,14 +298,28 @@ impl Package { /// [`Package::xml`] for the main document part: malformed XML is an error, not a truncated read. pub fn xml_strict(&self, name: &str) -> Result, DocxError> { match self.get(name) { - Some(b) => xml::parse_strict(b).map(Some), + Some(b) => { + // The parser stops at the remaining budget or the per-part cap, whichever is less: + // a part over what is left of the package's budget is refused before the whole + // per-part tree is built. + let (el, count) = xml::parse_strict_counted(b, self.budget.get().min(xml::MAX_ELEMENTS))?; + self.charge(count)?; + Ok(Some(el)) + } None => Ok(None), } } + /// A package part as a parsed XML tree, charged against the element budget: the parser + /// stops at what is left of the budget or the per-part cap, whichever is less, and the + /// count it built is charged after. A lenient read: malformed XML is a truncated one. pub fn xml(&self, name: &str) -> Result, DocxError> { match self.get(name) { - Some(b) => xml::parse(b).map(Some), + Some(b) => { + let (el, count) = xml::parse_counted(b, self.budget.get().min(xml::MAX_ELEMENTS))?; + self.charge(count)?; + Ok(Some(el)) + } None => Ok(None), } } @@ -448,6 +499,82 @@ mod tests { let junk = EmbeddedManifest::parse(b"R\tnope\trId1\tk\tt\tI\nP\tonly-two\n\t\t\n\xff\xfe"); assert!(junk.parts.is_empty()); } + + /// A file of many parts each inside the per-part cap is bounded in total: every XML part + /// read through the package charges a package-wide budget, and a part over it is skipped + /// (warned), never built. A chart part with no series builds a maximal tree for almost no + /// graphic work, so the trees a reader keeps would otherwise accumulate one per part. + #[test] + fn the_xml_element_budget_is_package_wide() { + let ns = r#"xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main""#; + let rels = r#""#; + let document = format!(r#""#); + let dense = |root: &str| format!(r#"<{root} {ns}>{}"#, "".repeat(900)); + let z = zip_of(&[ + ("word/document.xml", document.into_bytes()), + ("word/_rels/document.xml.rels", rels.as_bytes().to_vec()), + ("word/styles.xml", dense("w:styles").into_bytes()), + ("word/numbering.xml", dense("w:numbering").into_bytes()), + ]); + let pkg = Package::open_limited_within(&z, MAX_PART, MAX_TOTAL, 1_000).unwrap(); + // styles (900 elements) reads; numbering (900 more) is over what is left and is skipped + // with a warning, not built. + let read = || crate::read::read_opened(&pkg, None); + let (first, warned) = crate::test_log::warnings(read); + assert!(first.is_ok(), "{:?}", first.err()); + let refused: Vec<_> = warned.iter().filter(|w| w.contains("numbering") && w.contains("XML element")).collect(); + assert_eq!(refused.len(), 1, "{warned:?}"); + // The budget is per read: a second read starts it over and behaves the same. + let (second, warned) = crate::test_log::warnings(read); + assert!(second.is_ok(), "{:?}", second.err()); + assert_eq!(warned.iter().filter(|w| w.contains("numbering") && w.contains("XML element")).count(), 1, "{warned:?}"); + } + + /// The main document part itself is strict: over the budget it is an error, not a + /// best-effort read of a tree the file then paid for. + #[test] + fn a_main_part_over_the_element_budget_is_refused() { + let ns = r#"xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main""#; + let document = format!(r#"{}"#, "".repeat(20)); + let z = zip_of(&[("word/document.xml", document.into_bytes())]); + let pkg = Package::open_limited_within(&z, MAX_PART, MAX_TOTAL, 10).unwrap(); + assert!(matches!(crate::read::read_opened(&pkg, None), Err(DocxError::Limit(_)))); + } + + /// A part over what is left of the package's budget is refused while parsing, not after the + /// whole per-part tree is built: the parser stops at the remaining budget or the per-part + /// cap, whichever is less, so the file pays for at most what is left of the budget. + #[test] + fn a_part_over_the_remaining_budget_is_refused_while_parsing() { + let ns = r#"xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main""#; + let document = format!(r#"{}"#, "".repeat(200)); + let z = zip_of(&[("word/document.xml", document.into_bytes())]); + // 200 elements: inside the per-part cap, over the package's 100 budget. The parse + // refuses at the budget, so the error is the parse's, not the charge's, and names + // the effective limit for this parse (the remaining budget), not the whole budget. + let pkg = Package::open_limited_within(&z, MAX_PART, MAX_TOTAL, 100).unwrap(); + let e = crate::read::read_opened(&pkg, None).expect_err("refused"); + assert_eq!(e, DocxError::Limit("over the XML element budget for this parse: at most 100 elements".into())); + } + + /// A part parsed earlier shrinks what is left of the budget: the next part is refused while + /// parsing at the remaining, not at the whole budget, and the error names the effective limit + /// for this parse (the remaining), not the package's budget. + #[test] + fn an_earlier_part_reduces_the_remaining_budget_the_next_part_names_it() { + let ns = r#"xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main""#; + let styles = format!(r#"{}"#, "".repeat(60)); + let settings = format!(r#"{}"#, "".repeat(60)); + let z = zip_of(&[("word/styles.xml", styles.into_bytes()), ("word/settings.xml", settings.into_bytes())]); + // 100 elements of budget. The styles part parses (the main-part path: `xml_strict`) and + // charges its 61 elements, leaving 39. + let pkg = Package::open_limited_within(&z, MAX_PART, MAX_TOTAL, 100).unwrap(); + pkg.xml_strict("word/styles.xml").unwrap(); + // The settings part needs 61 elements, over the 39 left and under the per-part cap: the + // parse refuses at the remaining, so the error names 39, not the package's 100 budget. + let e = pkg.xml("word/settings.xml").expect_err("refused"); + assert_eq!(e, DocxError::Limit("over the XML element budget for this parse: at most 39 elements".into())); + } } #[cfg(test)] diff --git a/crates/docx/src/read.rs b/crates/docx/src/read.rs index 746225e97..18c9ea47a 100644 --- a/crates/docx/src/read.rs +++ b/crates/docx/src/read.rs @@ -154,9 +154,11 @@ pub(crate) fn read_opened(pkg: &Package, chunks: Option) -> Resul read_opened_in(pkg, ChunkState::new(chunks)).map(|(doc, ..)| doc) } -/// [`read_opened`] with what chunks may still bring in; also gives the chunks, chunk bytes and -/// graphic work left of that. -fn read_opened_in(pkg: &Package, chunks: ChunkState) -> Result<(Document, usize, u64, usize), DocxError> { +/// [`read_opened`] with what chunks may still bring in; also gives the chunks, chunk bytes, +/// graphic work and XML elements left of that. +fn read_opened_in(pkg: &Package, chunks: ChunkState) -> Result<(Document, usize, u64, usize, usize), DocxError> { + // The budget is per read, so a package read twice (tests, a Flat OPC conversion) starts over. + pkg.reset_budget(); let root_rels = pkg.rels(""); let main = root_rels.by_type(rt::OFFICE_DOC).filter(|r| !r.external).map(|r| r.target.clone()).unwrap_or_else(|| "word/document.xml".to_string()); let Some(root) = pkg.xml_strict(&main)? else { @@ -371,11 +373,11 @@ fn read_opened_in(pkg: &Package, chunks: ChunkState) -> Result<(Document, usize, if r.shape_ids { keep_glued_ids(&mut r.doc, &r.glued); } - let (left, bytes, graphic) = (r.chunks.left, r.chunks.bytes, r.chunks.graphic); + let (left, bytes, graphic, elements) = (r.chunks.left, r.chunks.bytes, r.chunks.graphic, pkg.elements_left()); let mut doc = r.doc; doc.skipped_parts.extend(pkg.skipped().iter().cloned()); doc.ensure_nonempty(); - Ok((doc, left, bytes, graphic)) + Ok((doc, left, bytes, graphic, elements)) } /// Most move ranges open at once, and most move names tracked, per file. diff --git a/crates/docx/src/read/chunk.rs b/crates/docx/src/read/chunk.rs index 2dc1674a7..9bdde064d 100644 --- a/crates/docx/src/read/chunk.rs +++ b/crates/docx/src/read/chunk.rs @@ -110,11 +110,17 @@ impl Reader<'_> { log::warn!("docx: alternative format chunks nest deeper than {MAX_CHUNK_NESTING}; the inner ones are left out"); return None; } - let pkg = - Package::open_limited(bytes, MAX_PART, self.chunks.bytes).map_err(|e| log::warn!("docx: unreadable nested document chunk: {e}")).ok()?; + // The nested document shares the file's element budget: open_limited_within, not a + // fresh cap — nesting goes deep, so 5 packages each getting 16M is the OOM class the + // budget exists for. The container's own parse spent part of the chain, so the cap is + // the lesser of the chain and what the container has left. + let cap = self.chunks.elements.min(self.pkg.elements_left()); + let pkg = Package::open_limited_within(bytes, MAX_PART, self.chunks.bytes, cap) + .map_err(|e| log::warn!("docx: unreadable nested document chunk: {e}")) + .ok()?; self.chunks.bytes = self.chunks.bytes.saturating_sub(pkg.size()); let state = ChunkState { nesting: self.chunks.nesting + 1, depth: depth + 1, ..self.chunks }; - let (doc, left, bytes, graphic) = match super::read_opened_in(&pkg, state) { + let (doc, left, bytes, graphic, elements) = match super::read_opened_in(&pkg, state) { Ok(read) => read, Err(e) => { log::warn!("docx: unreadable nested document chunk: {e}"); @@ -125,6 +131,16 @@ impl Reader<'_> { self.chunks.left = left; self.chunks.bytes = bytes; self.chunks.graphic = graphic; + self.chunks.elements = elements; + // The nested document's XML usage comes out of the container's budget too: a later + // sibling chunk and the container's own remaining parts parse against what is left + // of it. The charge can't refuse: the cap is the lesser of the file's remaining + // elements and what the container has left, and the container's budget doesn't move + // while the nested read goes. + let spent = cap.saturating_sub(elements); + if let Err(e) = self.pkg.charge(spent) { + log::warn!("docx: nested document chunk outspent the container's element budget: {e}"); + } Some(doc) } } @@ -139,6 +155,9 @@ pub(crate) struct ChunkState<'a> { /// Graphic work left in the file's budget ([`super::MAX_GRAPHIC_WORK`]): nested documents' /// charts and diagrams count against it too. pub graphic: usize, + /// XML elements left in the file's budget: nested packages share their container's + /// ([`crate::xml::MAX_ELEMENTS_TOTAL`]), not one of their own. + pub elements: usize, /// How deep in nested WordprocessingML chunks this reader is. pub nesting: u8, /// The XML depth its stories start at (a nested document's: that of the chunk naming it). @@ -148,7 +167,15 @@ pub(crate) struct ChunkState<'a> { impl<'a> ChunkState<'a> { pub fn new(import: Option>) -> Self { - ChunkState { left: MAX_CHUNKS, bytes: MAX_TOTAL, graphic: super::MAX_GRAPHIC_WORK, nesting: 0, depth: 0, import } + ChunkState { + left: MAX_CHUNKS, + bytes: MAX_TOTAL, + graphic: super::MAX_GRAPHIC_WORK, + elements: crate::xml::MAX_ELEMENTS_TOTAL, + nesting: 0, + depth: 0, + import, + } } } @@ -198,8 +225,8 @@ mod tests { let outer = package(r#""#, ("rIdC", "aFChunk", "afchunk.docx"), ("word/afchunk.docx", &inner)); let pkg = Package::open(&outer).unwrap(); let read = |graphic| { - let (d, .., left) = super::super::read_opened_in(&pkg, ChunkState { graphic, ..ChunkState::new(None) }).unwrap(); - (drawn(&d), left) + let (d, .., work, _) = super::super::read_opened_in(&pkg, ChunkState { graphic, ..ChunkState::new(None) }).unwrap(); + (drawn(&d), work) }; let (shown, left) = read(super::super::MAX_GRAPHIC_WORK); assert!(shown); @@ -207,4 +234,130 @@ mod tests { // With the file's budget spent, the nested document doesn't get one of its own. assert_eq!(read(0), (false, 0)); } + + /// A package whose body is `body`, with the document relationships `rels` (id, type, target) + /// and the parts `(name, bytes)` they name. + fn package_parts(body: &str, rels: &[(&str, &str, &str)], parts: &[(&str, &[u8])]) -> Vec { + let ns = r#"xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships" xmlns:wp="http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing" xmlns:a="http://schemas.openxmlformats.org/drawingml/2006/main""#; + let rels_xml = + |inner: &str| format!(r#"{inner}"#); + let rel = |(id, ty, target): &(&str, &str, &str)| { + format!(r#""#) + }; + let names: String = rels.iter().map(rel).collect::>().join(""); + let mut files: Vec<(&str, Vec)> = vec![ + ("_rels/.rels", rels_xml(&rel(&("rId1", "officeDocument", "word/document.xml"))).into_bytes()), + ("word/document.xml", format!(r#"{body}"#).into_bytes()), + ("word/_rels/document.xml.rels", rels_xml(&names).into_bytes()), + ]; + files.extend(parts.iter().map(|(name, bytes)| (*name, bytes.to_vec()))); + let mut zw = zip::ZipWriter::new(std::io::Cursor::new(Vec::new())); + for (name, bytes) in files { + zw.start_file(name, zip::write::SimpleFileOptions::default()).unwrap(); + zw.write_all(&bytes).unwrap(); + } + zw.finish().unwrap().into_inner() + } + + /// Whether the body holds a paragraph with `s`. + fn has(d: &Document, s: &str) -> bool { + d.body.iter().any(|b| b.as_para().map(|p| p.text.contains(s)).unwrap_or(false)) + } + + #[test] + fn a_nested_documents_elements_come_out_of_the_files_budget() { + // Three levels: the outer's altChunk names the inner, whose altChunk names the + // innermost. The chain spends down: the inner's own parse charges the file's budget, + // so the innermost opens with what the inner has left, not a fresh cap. + let innermost = package(r#"innermost"#, ("rIdZ", "chart", "charts/z.xml"), ("word/charts/z.xml", b"")); + let inner = package( + r#"inner"#, + ("rIdD", "aFChunk", "afchunk.docx"), + ("word/afchunk.docx", &innermost), + ); + let outer = package(r#""#, ("rIdC", "aFChunk", "afchunk.docx"), ("word/afchunk.docx", &inner)); + let pkg = Package::open(&outer).unwrap(); + let read = |elements| { + let (d, ..) = super::super::read_opened_in(&pkg, ChunkState { elements, ..ChunkState::new(None) }).unwrap(); + (has(&d, "inner"), has(&d, "innermost")) + }; + // With room, both levels flow in. + assert_eq!(read(crate::xml::MAX_ELEMENTS_TOTAL), (true, true)); + // The inner's own parse (its rels, then its main part) charged the file's budget: the + // innermost is left out instead of getting one of its own (before this, each nested + // package started with the full 16M). + assert_eq!(read(10), (true, false)); + } + + #[test] + fn a_nested_documents_elements_charge_the_container() { + // The nested document's XML usage comes out of the container's budget too: the cap + // the nested package was given, minus its returned remainder, is charged, so the + // container's remaining parts parse against what is left of the shared cap — not a + // budget that ignores the chunk. + let nested = package(r#"inner"#, ("rIdZ", "chart", "charts/z.xml"), ("word/charts/z.xml", b"")); + let body = r#"after"#; + let with = package(&format!(r#"{body}"#), ("rIdC", "aFChunk", "afchunk.docx"), ("word/afchunk.docx", &nested)); + let without = package(body, ("rIdZ", "chart", "charts/z.xml"), ("word/charts/z.xml", b"")); + let read = |outer: &[u8]| { + let pkg = Package::open(outer).unwrap(); + let (d, ..) = super::super::read_opened_in(&pkg, ChunkState { elements: 10, ..ChunkState::new(None) }).unwrap(); + (has(&d, "inner"), pkg.elements_left()) + }; + let (inner_in, charged) = read(&with); + assert!(inner_in); + let (_, base) = read(&without); + assert!(charged < base, "the nested document's usage is the container's: {charged} of {base}"); + } + + #[test] + fn a_grandchilds_reservation_charges_every_level() { + // Three levels: the innermost's reservation comes out of the inner's budget, and the + // inner's out of the outer's — the chain charges every container, so the outer's + // remainder is smaller with the deeper chain than with a shallow one. + let innermost = package(r#"leaf"#, ("rIdZ", "chart", "charts/z.xml"), ("word/charts/z.xml", b"")); + let inner = package( + r#"mid"#, + ("rIdD", "aFChunk", "afchunk.docx"), + ("word/afchunk.docx", &innermost), + ); + let outer = package(r#""#, ("rIdC", "aFChunk", "afchunk.docx"), ("word/afchunk.docx", &inner)); + let pkg = Package::open(&outer).unwrap(); + let (d, .., elements) = super::super::read_opened_in(&pkg, ChunkState::new(None)).unwrap(); + assert!(has(&d, "mid"), "the inner document didn't flow in"); + assert!(has(&d, "leaf"), "the innermost document didn't flow in"); + // The same container with a shallow chain: one reservation instead of two. + let shallow = package(r#""#, ("rIdC", "aFChunk", "afchunk.docx"), ("word/afchunk.docx", &innermost)); + let pkg = Package::open(&shallow).unwrap(); + let (.., shallow_elements) = super::super::read_opened_in(&pkg, ChunkState::new(None)).unwrap(); + assert!(elements < shallow_elements, "every level's reservation is charged: {elements} of {shallow_elements}"); + } + + #[test] + fn a_charged_budget_bounds_the_containers_part_and_the_sibling_after_it() { + // The container's budget pays for the parts it parses (the section's header, read + // during the chunk's own walk) and for the chunks after the first one: with a budget + // that fits the grandchild chunk and the header but not the sibling document's XML, + // the sibling is left out; skip the header's charge and it would flow in. + let hdr = r#"headheadhead"#; + let gc = package(r#"gc"#, ("rIdZ", "chart", "charts/z.xml"), ("word/charts/z.xml", b"")); + let sibling = + package(&r#"sibling"#.repeat(8), ("rIdZ", "chart", "charts/z.xml"), ("word/charts/z.xml", b"")); + let inner = package_parts( + r#"mid"#, + &[("rIdD", "aFChunk", "afchunk.docx"), ("rIdH", "header", "header1.xml"), ("rIdS", "aFChunk", "afchunk2.docx")], + &[("word/afchunk.docx", &gc), ("word/header1.xml", hdr.as_bytes()), ("word/afchunk2.docx", &sibling)], + ); + let outer = package(r#""#, ("rIdC", "aFChunk", "afchunk.docx"), ("word/afchunk.docx", &inner)); + let pkg = Package::open(&outer).unwrap(); + let read = |elements| { + let (d, ..) = super::super::read_opened_in(&pkg, ChunkState { elements, ..ChunkState::new(None) }).unwrap(); + (has(&d, "mid"), has(&d, "gc"), has(&d, "sibling")) + }; + // With room, everything flows in. + assert_eq!(read(crate::xml::MAX_ELEMENTS_TOTAL), (true, true, true)); + // The grandchild chunk and the header fit the file's budget; the sibling document's + // XML (8 paragraphs) doesn't, so it's left out instead of getting a fresh cap. + assert_eq!(read(40), (true, true, false)); + } } diff --git a/crates/docx/src/xml.rs b/crates/docx/src/xml.rs index fa50d217a..46da6911d 100644 --- a/crates/docx/src/xml.rs +++ b/crates/docx/src/xml.rs @@ -10,6 +10,10 @@ use crate::DocxError; pub const MAX_DEPTH: usize = 256; /// Maximum number of elements in one part. pub const MAX_ELEMENTS: usize = 8_000_000; +/// Maximum number of elements summed over every part a read parses: a file of many parts each +/// inside [`MAX_ELEMENTS`] would otherwise cost a tree per part, and the trees a reader keeps +/// (the main part's, charts' and diagrams') accumulate. +pub const MAX_ELEMENTS_TOTAL: usize = MAX_ELEMENTS * 2; /// Maximum number of attributes on one element (#3702). pub const MAX_ATTRS: usize = 1024; @@ -329,19 +333,31 @@ pub fn parse(bytes: &[u8]) -> Result { parse_ns(bytes, &[]) } +/// [`parse`], with the number of elements it built, stopping early at `max` elements: the caller +/// charges the count against a package-wide budget (see [`MAX_ELEMENTS_TOTAL`]), and a part over +/// what is left of it is refused before the whole per-part tree is built. +pub fn parse_counted(bytes: &[u8], max: usize) -> Result<(El, usize), DocxError> { + parse_with(bytes, &[], false, max) +} + +/// [`parse_counted`], strict, for a part whose loss would be silent data loss (the main +/// document): a mismatched end tag or an element left open at the end of the data is an error, +/// not a cue to stop reading. +pub fn parse_strict_counted(bytes: &[u8], max: usize) -> Result<(El, usize), DocxError> { + parse_with(bytes, &[], true, max) +} + /// [`parse`] with `extra` namespaces (prefix, URI) mapped to canonical prefixes first, so a /// related vocabulary (WordprocessingML 2003) reads with the prefixes the reader knows. pub fn parse_ns(bytes: &[u8], extra: &[(&'static str, &str)]) -> Result { - parse_with(bytes, extra, false) + parse_with(bytes, extra, false, MAX_ELEMENTS).map(|(e, _)| e) } -/// [`parse`] for a part whose loss would be silent data loss (the main document): a mismatched end -/// tag or an element left open at the end of the data is an error, not a cue to stop reading. -pub fn parse_strict(bytes: &[u8]) -> Result { - parse_with(bytes, &[], true) -} - -fn parse_with(bytes: &[u8], extra: &[(&'static str, &str)], strict: bool) -> Result { +/// The XML parser everything else goes through: transcodes UTF-16, walks the events and builds +/// the tree depth-first, stopping at `max` elements and counting the ones it built. A mismatched +/// end tag or an element left open at the end of the data is an error when `strict`, a cue to +/// stop reading when not. +fn parse_with(bytes: &[u8], extra: &[(&'static str, &str)], strict: bool, max: usize) -> Result<(El, usize), DocxError> { let namespaces: Vec<(&str, &str)> = extra.iter().copied().chain(NAMESPACES.iter().copied()).chain(OTHER_NAMESPACES.iter().copied()).chain(STRICT.iter().copied()).collect(); // The prefix of an element: its namespace's, with two exceptions for Strict documents @@ -375,7 +391,9 @@ fn parse_with(bytes: &[u8], extra: &[(&'static str, &str)], strict: bool) -> Res max_depth: MAX_DEPTH + 1, // Text stretches count as nodes too: two for each element. max_nodes: 2 * MAX_ELEMENTS, - max_elements: MAX_ELEMENTS, + // The parse stops here before the whole tree is built: a part over what is left of the + // package's element budget pays for at most that, not for the per-part cap. + max_elements: max, max_attrs: MAX_ATTRS, max_attrs_total: 2 * MAX_ELEMENTS, }; @@ -383,17 +401,27 @@ fn parse_with(bytes: &[u8], extra: &[(&'static str, &str)], strict: bool) -> Res if strict { options = options.strict(); } - let root = options.parse(bytes).map_err(|e| if e.is_limit() { DocxError::Limit(e.to_string()) } else { DocxError::Xml(e.to_string()) })?; - Ok(from_tree(root)) + let root = options.parse(bytes).map_err(|e| match &e { + craft_xml::Error::TooManyElements(_) => DocxError::Limit(format!("over the XML element budget for this parse: at most {max} elements")), + e if e.is_limit() => DocxError::Limit(e.to_string()), + e => DocxError::Xml(e.to_string()), + })?; + Ok(from_tree_counted(root)) } -/// The `craft-xml` tree as this module's (whose shape the reader was written against). +/// The `craft-xml` tree as this module's (whose shape the reader was written against), with the +/// number of elements the subtree holds (the caller charges it against a package-wide budget). // shortcut: one copy of each parsed part; read craft_xml::El directly when the reader is ported to it. -fn from_tree(e: craft_xml::El) -> El { +fn from_tree_counted(e: craft_xml::El) -> (El, usize) { let mut kids = Vec::with_capacity(e.children.len()); + let mut count = 1usize; for n in e.children { match n { - craft_xml::Node::El(c) => kids.push(Node::El(from_tree(c))), + craft_xml::Node::El(c) => { + let (kid, built) = from_tree_counted(c); + count += built; + kids.push(Node::El(kid)); + } craft_xml::Node::Text(t) | craft_xml::Node::CData(t) => match kids.last_mut() { Some(Node::Text(prev)) => prev.push_str(&t), _ => kids.push(Node::Text(t)), @@ -401,7 +429,7 @@ fn from_tree(e: craft_xml::El) -> El { craft_xml::Node::Pi { .. } | craft_xml::Node::Comment(_) => {} } } - El { name: e.name, attrs: e.attrs, kids } + (El { name: e.name, attrs: e.attrs, kids }, count) } /// This module's tree as `craft-xml`'s, for the shared readers (`craft-drawingml`, `craft-math`) @@ -515,12 +543,13 @@ mod tests { fn strict_parsing_rejects_a_dropped_entity_and_trailing_markup() { let bad = [&b"before &unknown; after"[..], b"", b""]; for xml in bad { - assert!(parse_strict(xml).is_err(), "{}", String::from_utf8_lossy(xml)); + assert!(parse_strict_counted(xml, MAX_ELEMENTS).is_err(), "{}", String::from_utf8_lossy(xml)); } assert!( - parse_strict( + parse_strict_counted( b"x & y -" +", + MAX_ELEMENTS ) .is_ok() ); diff --git a/crates/docx/tests/it/element_budget.rs b/crates/docx/tests/it/element_budget.rs new file mode 100644 index 000000000..6b220ae7c --- /dev/null +++ b/crates/docx/tests/it/element_budget.rs @@ -0,0 +1,44 @@ +//! The package-wide XML element budget: a file of many parts each inside the per-part +//! cap reads bounded, where every part used to build its own tree and the trees the +//! reader keeps accumulated. + +use crate::fixtures; +use craft_docx::read; + +const W_NS: &str = r#"xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships""#; + +#[test] +fn a_file_of_element_budget_parts_still_reads() { + // Nine chart parts of 2M elements each: 18M over the package's 16M budget. The later + // parts are skipped with a warning instead of built, and the file still reads. Before + // the budget, every part's whole tree was parsed and the chart trees stayed cached for + // the whole read (~230 MB per 2M-element part, measured), so a file like this cost one + // tree per part and scales linearly with the part count. + let parts = 9usize; + let chunk = "".repeat(2_000_000); + let chart_xml = format!(r#"{chunk}"#); + let mut document = format!( + r#""# + ); + for i in 0..parts { + document.push_str(&format!( + r#""# + )); + } + document.push_str(""); + let mut rels = String::from(r#""#); + for i in 0..parts { + rels.push_str(&format!( + r#""# + )); + } + rels.push_str(""); + let mut entries: Vec<(&str, Vec)> = vec![("word/document.xml", document.into_bytes()), ("word/_rels/document.xml.rels", rels.into_bytes())]; + for i in 0..parts { + entries.push((Box::leak(format!("word/charts/chart{i}.xml").into_boxed_str()), chart_xml.clone().into_bytes())); + } + let refs: Vec<(&str, &[u8])> = entries.iter().map(|(n, b)| (*n, b.as_slice())).collect(); + let z = fixtures::zip(&refs); + let doc = read(&z).expect("a file over the element budget still reads, the extra parts skipped"); + assert_eq!(doc.body.len(), parts, "the document's own paragraphs read"); +} diff --git a/crates/docx/tests/it/main.rs b/crates/docx/tests/it/main.rs index 926cb5154..942d946ae 100644 --- a/crates/docx/tests/it/main.rs +++ b/crates/docx/tests/it/main.rs @@ -16,6 +16,7 @@ mod default_size; mod drawing_payload; mod east_asian_lang; mod east_asian_layout; +mod element_budget; mod emphasis_marks; mod fixtures; mod font_table; From 1a0d5d607859a2e529851482de09b8e53aaeec3e Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Tue, 17 Nov 2026 09:58:00 -0500 Subject: [PATCH 036/161] Extract XFA namespace-aware XML DOM without layout policy --- ATTRIBUTION.md | 4 + Cargo.lock | 7 + Cargo.toml | 1 + crates/xfa/Cargo.toml | 14 ++ crates/xfa/README.md | 9 + crates/xfa/src/lib.rs | 6 + crates/xfa/src/xml.rs | 428 ++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 469 insertions(+) create mode 100644 crates/xfa/Cargo.toml create mode 100644 crates/xfa/README.md create mode 100644 crates/xfa/src/lib.rs create mode 100644 crates/xfa/src/xml.rs diff --git a/ATTRIBUTION.md b/ATTRIBUTION.md index 4c86afd61..6caa0fd33 100644 --- a/ATTRIBUTION.md +++ b/ATTRIBUTION.md @@ -132,3 +132,7 @@ Sanitization, animation fitting and numeric output policy remain caller-owned. ## JSONC span-preserving editing `crates/jsonc` is extracted from Craft Anything `crates/formats/src/jsonc.rs` at `e5c9c94106747ed44b9a3fb0decce867c2285f93` (MIT OR Apache-2.0). The original implementation and hardening by BFlat, newline preservation by NetterSilver, and subsequent RRFZ/Afloat correctness fixes retain separate primary-author port commits with immutable source revisions. Shared API wiring and optional input-byte admission are subsequent changes. Compatibility parsing and all existing formatting/editing tests are retained; application file I/O and policy are not imported. + +## XFA XML DOM + +`crates/xfa/src/xml.rs` is extracted from PDFCraft at `271e479c53723b1b294c866d5cb2f27604601b29`, originally by Dylan M. Taylor (`eead4541`, `7d44e654`), MIT OR Apache-2.0. Only visibility and module boundaries change; PDF/layout/form policy remains in the app. diff --git a/Cargo.lock b/Cargo.lock index 9fa1f4401..7bf961c6b 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3380,6 +3380,13 @@ dependencies = [ "thiserror 2.0.21", ] +[[package]] +name = "craft-xfa" +version = "0.1.0" +dependencies = [ + "craft-xml", +] + [[package]] name = "craft-xfl" version = "0.1.0" diff --git a/Cargo.toml b/Cargo.toml index 50ec3c917..d9094ba20 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -11,6 +11,7 @@ rust-version = "1.95" repository = "https://github.com/storytold/craft-libs" [workspace.dependencies] +craft-xfa = { path = "crates/xfa" } craft-jsonc = { path = "crates/jsonc" } craft-palette = { path = "crates/palette" } craft-score = { path = "crates/score" } diff --git a/crates/xfa/Cargo.toml b/crates/xfa/Cargo.toml new file mode 100644 index 000000000..7dca38e84 --- /dev/null +++ b/crates/xfa/Cargo.toml @@ -0,0 +1,14 @@ +[package] +name = "craft-xfa" +description = "Namespace-aware source-span XML DOM for XFA adapters; no PDF or layout policy" +version.workspace = true +edition.workspace = true +license.workspace = true +rust-version.workspace = true +repository.workspace = true + +[lints] +workspace = true + +[dependencies] +craft-xml = { workspace = true } diff --git a/crates/xfa/README.md b/crates/xfa/README.md new file mode 100644 index 000000000..e4ed85e08 --- /dev/null +++ b/crates/xfa/README.md @@ -0,0 +1,9 @@ +# craft-xfa + +Scoped namespace-aware XML DOM extracted unchanged from PDFCraft xfa/xml.rs at 271e479c. This is not a complete XFA implementation: template interpretation, packets, data binding, scripts, layout, fonts and PDF generation remain app-owned. + +Preserves source byte ranges, coalesced text, namespace shadowing and document-local node identities. Uses craft-xml strict events with DTD rejection by default, depth 1024 and caller node count. Other event-reader defaults remain in force. Namespace snapshots are copied per element as before; this is not a new total-memory bound or a full XML namespaces conformance claim. + +Original source by Dylan M. Taylor (eead4541, 7d44e654), MIT OR Apache-2.0. + +Compatibility delta versus the original app XML pin e2dc681a3: current strict craft-xml rejects duplicate raw or expanded attribute names. This deliberately preserves shared security hardening; valid-document DOM behavior is unchanged, but malformed acceptance and errors are not promised byte-identical. Regression tests cover both duplicate forms. diff --git a/crates/xfa/src/lib.rs b/crates/xfa/src/lib.rs new file mode 100644 index 000000000..61640dbf6 --- /dev/null +++ b/crates/xfa/src/lib.rs @@ -0,0 +1,6 @@ +//! XFA XML infrastructure, extracted from PDFCraft. Layout and application policy stay local. +#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] +#![forbid(unsafe_code)] +/// Compatibility depth limit for XFA XML documents. +pub const MAX_XML_DEPTH: usize = 1024; +pub mod xml; diff --git a/crates/xfa/src/xml.rs b/crates/xfa/src/xml.rs new file mode 100644 index 000000000..0840170e6 --- /dev/null +++ b/crates/xfa/src/xml.rs @@ -0,0 +1,428 @@ +//! Lightweight namespace-aware XML DOM with source byte ranges, backed by `craft_xml`. + +use std::ops::Range; + +const XML_NS: &str = "http://www.w3.org/XML/1998/namespace"; +const XMLNS_NS: &str = "http://www.w3.org/2000/xmlns/"; + +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub struct NodeId(pub usize); + +#[derive(Clone, Debug)] +pub struct ParsingOptions { + pub allow_dtd: bool, + pub nodes_limit: u32, +} + +impl Default for ParsingOptions { + fn default() -> Self { + Self { allow_dtd: false, nodes_limit: u32::MAX } + } +} + +#[derive(Clone, Debug)] +struct AttrData { + ns: Option, + name: String, + value: String, +} + +#[derive(Clone, Debug)] +enum NodeKind { + Element { + ns: Option, + name: String, + attrs: Vec, + /// In-scope `(prefix, uri)` bindings at this element (`""` for default namespace). + namespaces: Vec<(String, String)>, + range: Range, + }, + Text(String), +} + +#[derive(Clone, Debug)] +struct NodeData { + kind: NodeKind, + children: Vec, +} + +pub struct Document { + nodes: Vec, + root: usize, +} + +#[derive(Clone, Copy)] +pub struct Node<'a> { + doc: &'a Document, + id: usize, +} + +impl PartialEq for Node<'_> { + fn eq(&self, other: &Self) -> bool { + self.id == other.id && std::ptr::eq(self.doc, other.doc) + } +} + +impl Eq for Node<'_> {} + +#[derive(Clone, Copy)] +pub struct ExpandedName<'a> { + ns: Option<&'a str>, + name: &'a str, +} + +impl<'a> ExpandedName<'a> { + pub fn name(&self) -> &'a str { + self.name + } + + pub fn namespace(&self) -> Option<&'a str> { + self.ns + } +} + +pub trait AttributeKey { + fn matches(&self, ns: Option<&str>, name: &str) -> bool; +} + +impl AttributeKey for &str { + fn matches(&self, ns: Option<&str>, name: &str) -> bool { + ns.is_none() && name == *self + } +} + +impl AttributeKey for (&str, &str) { + fn matches(&self, ns: Option<&str>, name: &str) -> bool { + ns == Some(self.0) && name == self.1 + } +} + +#[derive(Clone, Copy)] +pub struct Attribute<'a> { + attr: &'a AttrData, +} + +impl<'a> Attribute<'a> { + pub fn name(&self) -> &'a str { + &self.attr.name + } + + pub fn namespace(&self) -> Option<&'a str> { + self.attr.ns.as_deref() + } + + pub fn value(&self) -> &'a str { + &self.attr.value + } +} + +#[derive(Clone, Copy)] +pub struct Namespace<'a> { + prefix: &'a str, + uri: &'a str, +} + +impl<'a> Namespace<'a> { + pub fn name(&self) -> Option<&'a str> { + (!self.prefix.is_empty()).then_some(self.prefix) + } + + pub fn uri(&self) -> &'a str { + self.uri + } +} + +fn split_qname(qname: &str) -> (&str, &str) { + match qname.split_once(':') { + Some((p, l)) => (p, l), + None => ("", qname), + } +} + +fn resolve_ns(scope: &[(String, String)], prefix: &str, is_attr: bool) -> Result, String> { + if prefix == "xml" { + return Ok(Some(XML_NS.to_string())); + } + if prefix == "xmlns" { + return Ok(Some(XMLNS_NS.to_string())); + } + if prefix.is_empty() { + if is_attr { + return Ok(None); + } + return Ok(scope.iter().rev().find(|(p, _)| p.is_empty()).and_then(|(_, u)| (!u.is_empty()).then(|| u.clone()))); + } + match scope.iter().rev().find(|(p, _)| p == prefix) { + Some((_, u)) if !u.is_empty() => Ok(Some(u.clone())), + _ => Err(format!("undeclared namespace prefix: {prefix}")), + } +} + +impl Document { + pub fn parse(text: &str) -> Result { + Self::parse_with_options(text, ParsingOptions::default()) + } + + pub fn parse_with_options(text: &str, options: ParsingOptions) -> Result { + let mut opts = craft_xml::Options { + strict: true, + doctype: if options.allow_dtd { craft_xml::Doctype::Ignore } else { craft_xml::Doctype::Reject }, + ..Default::default() + }; + opts.limits.max_nodes = options.nodes_limit as usize; + opts.limits.max_depth = crate::MAX_XML_DEPTH; + + let mut events = opts.events(text.as_bytes()); + let mut nodes: Vec = Vec::new(); + let mut open: Vec<(usize, usize)> = Vec::new(); // (node_id, scope_len_before) + let mut scope: Vec<(String, String)> = Vec::new(); + let mut root: Option = None; + + while let Some(ev) = events.next().map_err(|e| e.to_string())? { + let span = events.span(); + match ev { + craft_xml::Event::Start { name, attrs } => { + if nodes.len() >= options.nodes_limit as usize { + return Err("nodes limit reached".to_string()); + } + let scope_mark = scope.len(); + for (k, v) in &attrs { + if k == "xmlns" { + scope.push((String::new(), v.clone())); + } else if let Some(prefix) = k.strip_prefix("xmlns:") { + scope.push((prefix.to_string(), v.clone())); + } + } + let (el_prefix, el_local) = split_qname(&name); + let el_ns = resolve_ns(&scope, el_prefix, false)?; + let mut el_attrs = Vec::new(); + for (k, v) in attrs { + if k == "xmlns" || k.starts_with("xmlns:") { + continue; + } + let (ap, al) = split_qname(&k); + let ans = resolve_ns(&scope, ap, true)?; + el_attrs.push(AttrData { ns: ans, name: al.to_string(), value: v }); + } + // Deduplicate in-scope namespace bindings, keeping the innermost binding per prefix. + let mut namespaces: Vec<(String, String)> = Vec::new(); + for (p, u) in scope.iter().rev() { + if !namespaces.iter().any(|(ep, _)| ep == p) { + namespaces.push((p.clone(), u.clone())); + } + } + namespaces.retain(|(_, u)| !u.is_empty()); + let id = nodes.len(); + nodes.push(NodeData { + kind: NodeKind::Element { ns: el_ns, name: el_local.to_string(), attrs: el_attrs, namespaces, range: span.start..span.end }, + children: Vec::new(), + }); + if let Some(&(parent, _)) = open.last() { + nodes[parent].children.push(id); + } else if root.is_none() { + root = Some(id); + } + open.push((id, scope_mark)); + } + craft_xml::Event::End { .. } => { + if let Some((id, scope_mark)) = open.pop() { + if let NodeKind::Element { range, .. } = &mut nodes[id].kind { + range.end = span.end; + } + scope.truncate(scope_mark); + } + } + craft_xml::Event::Text(t) | craft_xml::Event::CData(t) => { + if let Some(&(parent, _)) = open.last() { + if let Some(&last_child) = nodes[parent].children.last() + && let NodeKind::Text(existing) = &mut nodes[last_child].kind + { + existing.push_str(&t); + continue; + } + if nodes.len() >= options.nodes_limit as usize { + return Err("nodes limit reached".to_string()); + } + let id = nodes.len(); + nodes.push(NodeData { kind: NodeKind::Text(t), children: Vec::new() }); + nodes[parent].children.push(id); + } + } + craft_xml::Event::Pi { .. } | craft_xml::Event::Comment(_) => {} + } + } + + let Some(root) = root else { + return Err("empty XML document".to_string()); + }; + Ok(Self { nodes, root }) + } + + pub fn root_element(&self) -> Node<'_> { + Node { doc: self, id: self.root } + } + + pub fn get_node(&self, id: NodeId) -> Option> { + (id.0 < self.nodes.len()).then_some(Node { doc: self, id: id.0 }) + } + + pub fn descendants(&self) -> Descendants<'_> { + Descendants { doc: self, stack: vec![self.root] } + } +} + +pub struct Descendants<'a> { + doc: &'a Document, + stack: Vec, +} + +impl<'a> Iterator for Descendants<'a> { + type Item = Node<'a>; + + fn next(&mut self) -> Option { + let id = self.stack.pop()?; + let children = &self.doc.nodes[id].children; + for &c in children.iter().rev() { + self.stack.push(c); + } + Some(Node { doc: self.doc, id }) + } +} + +impl<'a> Node<'a> { + pub fn id(&self) -> NodeId { + NodeId(self.id) + } + + pub fn document(&self) -> &'a Document { + self.doc + } + + pub fn is_element(&self) -> bool { + matches!(self.doc.nodes[self.id].kind, NodeKind::Element { .. }) + } + + pub fn is_text(&self) -> bool { + matches!(self.doc.nodes[self.id].kind, NodeKind::Text(_)) + } + + pub fn tag_name(&self) -> ExpandedName<'a> { + match &self.doc.nodes[self.id].kind { + NodeKind::Element { ns, name, .. } => ExpandedName { ns: ns.as_deref(), name }, + NodeKind::Text(_) => ExpandedName { ns: None, name: "" }, + } + } + + pub fn attribute(&self, key: K) -> Option<&'a str> { + match &self.doc.nodes[self.id].kind { + NodeKind::Element { attrs, .. } => attrs.iter().find(|a| key.matches(a.ns.as_deref(), &a.name)).map(|a| a.value.as_str()), + NodeKind::Text(_) => None, + } + } + + pub fn attributes(&self) -> impl ExactSizeIterator> + use<'a> { + let attrs = match &self.doc.nodes[self.id].kind { + NodeKind::Element { attrs, .. } => attrs.as_slice(), + NodeKind::Text(_) => &[], + }; + attrs.iter().map(|attr| Attribute { attr }) + } + + pub fn namespaces(&self) -> impl ExactSizeIterator> + use<'a> { + let ns = match &self.doc.nodes[self.id].kind { + NodeKind::Element { namespaces, .. } => namespaces.as_slice(), + NodeKind::Text(_) => &[], + }; + ns.iter().map(|(p, u)| Namespace { prefix: p, uri: u }) + } + + pub fn lookup_prefix(&self, uri: &str) -> Option<&'a str> { + if uri == XML_NS { + return Some("xml"); + } + match &self.doc.nodes[self.id].kind { + NodeKind::Element { namespaces, .. } => namespaces.iter().find(|(_, u)| u == uri).map(|(p, _)| p.as_str()), + NodeKind::Text(_) => None, + } + } + + pub fn lookup_namespace_uri(&self, prefix: Option<&str>) -> Option<&'a str> { + if prefix == Some("xml") { + return Some(XML_NS); + } + if prefix == Some("xmlns") { + return Some(XMLNS_NS); + } + let target = prefix.unwrap_or(""); + match &self.doc.nodes[self.id].kind { + NodeKind::Element { namespaces, .. } => namespaces.iter().find(|(p, _)| p == target).map(|(_, u)| u.as_str()), + NodeKind::Text(_) => None, + } + } + + pub fn children(&self) -> impl Iterator> + use<'a> { + let doc = self.doc; + doc.nodes[self.id].children.iter().map(move |&id| Node { doc, id }) + } + + pub fn descendants(&self) -> Descendants<'a> { + Descendants { doc: self.doc, stack: vec![self.id] } + } + + pub fn text(&self) -> Option<&'a str> { + match &self.doc.nodes[self.id].kind { + NodeKind::Text(t) => Some(t.as_str()), + NodeKind::Element { .. } => self.doc.nodes[self.id].children.first().and_then(|&cid| match &self.doc.nodes[cid].kind { + NodeKind::Text(t) => Some(t.as_str()), + _ => None, + }), + } + } + + pub fn range(&self) -> Range { + match &self.doc.nodes[self.id].kind { + NodeKind::Element { range, .. } => range.clone(), + NodeKind::Text(_) => 0..0, + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn namespace_shadowing_attributes_text_and_source_ranges() { + let text = "onethree"; + let doc = Document::parse(text).unwrap(); + let root = doc.root_element(); + assert_eq!(root.tag_name().namespace(), Some("urn:outer")); + assert_eq!(root.attribute("a"), Some("v")); + let child = root.children().next().unwrap(); + assert_eq!(child.tag_name().namespace(), Some("urn:p")); + assert_eq!(child.attribute(("urn:p", "a")), Some("x")); + assert_eq!(child.lookup_namespace_uri(None), Some("urn:inner")); + assert_eq!(child.text(), Some("onetwothree")); + assert_eq!(child.children().count(), 1); + assert!(text[child.range()].starts_with("")); + assert!(doc.get_node(child.id()).unwrap() == child); + assert_eq!(root.children().nth(1).unwrap().tag_name().namespace(), None); + assert_eq!(root.range(), 0..text.len()); + } + + #[test] + fn malformed_dtd_and_resource_limits_are_preserved() { + // Current shared XML rejects duplicate attributes, unlike the older app pin. + assert!(Document::parse("").is_err()); + assert!(Document::parse("").is_err()); + assert!(Document::parse("").is_err()); + assert!(Document::parse("").is_err()); + assert!(Document::parse_with_options("", ParsingOptions { allow_dtd: true, nodes_limit: 8 }).is_ok()); + assert!(Document::parse_with_options("text", ParsingOptions { allow_dtd: false, nodes_limit: 1 }).is_err()); + assert!(Document::parse_with_options("", ParsingOptions { allow_dtd: false, nodes_limit: 0 }).is_err()); + let text = format!("{}{}", "".repeat(1025), "".repeat(1025)); + assert!(Document::parse(&text).is_err()); + assert!(Document::parse("").is_err()); + } +} From c50eb61065846fc6d4feca3438ffa691dc855313 Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Tue, 17 Nov 2026 10:11:04 -0500 Subject: [PATCH 037/161] Document and test inherited namespace declaration budget --- crates/xfa/README.md | 2 ++ crates/xfa/src/xml.rs | 17 +++++++++++++++++ 2 files changed, 19 insertions(+) diff --git a/crates/xfa/README.md b/crates/xfa/README.md index e4ed85e08..a3548c2d1 100644 --- a/crates/xfa/README.md +++ b/crates/xfa/README.md @@ -7,3 +7,5 @@ Preserves source byte ranges, coalesced text, namespace shadowing and document-l Original source by Dylan M. Taylor (eead4541, 7d44e654), MIT OR Apache-2.0. Compatibility delta versus the original app XML pin e2dc681a3: current strict craft-xml rejects duplicate raw or expanded attribute names. This deliberately preserves shared security hardening; valid-document DOM behavior is unchanged, but malformed acceptance and errors are not promised byte-identical. Regression tests cover both duplicate forms. + +The updated strict reader also bounds aggregate active namespace declarations to 1024 across ancestor elements, not just attributes per tag. The old app path could accept more inherited declarations. The exact boundary and one-over case are tested; this intentional tighter resource admission is retained. diff --git a/crates/xfa/src/xml.rs b/crates/xfa/src/xml.rs index 0840170e6..2f4a49db8 100644 --- a/crates/xfa/src/xml.rs +++ b/crates/xfa/src/xml.rs @@ -411,6 +411,23 @@ mod tests { assert_eq!(root.range(), 0..text.len()); } + #[test] + fn inherited_namespace_declarations_have_an_aggregate_budget() { + use std::fmt::Write; + let mut xml = String::from(""); + assert!(Document::parse(&accepted).is_ok()); + let rejected = format!("{xml} xmlns:extra='extra'/>"); + assert!(Document::parse(&rejected).is_err()); + } + #[test] fn malformed_dtd_and_resource_limits_are_preserved() { // Current shared XML rejects duplicate attributes, unlike the older app pin. From 786c5571f05595276857e7faa36c8be2fe895a7d Mon Sep 17 00:00:00 2001 From: techneissa <55485515+techneissa@users.noreply.github.com> Date: Tue, 17 Nov 2026 18:39:50 +0300 Subject: [PATCH 038/161] raw: refuse an understated DNG strip whose ActiveArea reaches below the image (#2040) decode drops such an ActiveArea for the whole image, so nothing would explain the short byte count. --- crates/raw/src/dng.rs | 2 +- crates/raw/src/dng/compat_review_tests.rs | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/crates/raw/src/dng.rs b/crates/raw/src/dng.rs index a7be5d67b..f62aeef47 100644 --- a/crates/raw/src/dng.rs +++ b/crates/raw/src/dng.rs @@ -546,7 +546,7 @@ fn extend_understated_strip(bytes: &[u8], info: &mut ImageInfo, raw: &Ifd) { } let ([offset], [count]) = (&info.offsets[..], &info.byte_counts[..]) else { return }; let Some([top, left, bottom, right]) = raw.u64s(t::ACTIVE_AREA).and_then(|a| <[u64; 4]>::try_from(a).ok()) else { return }; - if left != 0 || bottom <= top || right == 0 || right >= u64::from(info.width) { + if left != 0 || bottom <= top || bottom > u64::from(info.height) || right == 0 || right >= u64::from(info.width) { return; } let row_bytes = |width: u64| width.checked_mul(u64::from(info.height))?.checked_mul(bits / 8); diff --git a/crates/raw/src/dng/compat_review_tests.rs b/crates/raw/src/dng/compat_review_tests.rs index 309d1e5b7..149ee492d 100644 --- a/crates/raw/src/dng/compat_review_tests.rs +++ b/crates/raw/src/dng/compat_review_tests.rs @@ -170,6 +170,8 @@ fn other_short_strips_are_still_refused() { assert!(refused(understated_strip_dng(8, 4, 6 * 4 * 2, None))); // the count is not exactly the area's bytes assert!(refused(understated_strip_dng(8, 4, 5 * 4 * 2, Some([0, 0, 4, 6])))); + // the area reaches below the image, so decode would discard it + assert!(refused(understated_strip_dng(8, 4, 6 * 4 * 2, Some([4, 0, 5, 6])))); // the area does not start at column 0 assert!(refused(understated_strip_dng(8, 4, 6 * 4 * 2, Some([0, 2, 4, 8])))); // the file does not hold the whole raster From 71e271483380e211f38e7d49c0765c59cff6e785 Mon Sep 17 00:00:00 2001 From: techneissa <55485515+techneissa@users.noreply.github.com> Date: Tue, 17 Nov 2026 18:47:47 +0300 Subject: [PATCH 039/161] raw: lengthen an understated DNG strip only for integer samples (#2040) SampleFormat 4 and other undefined values no longer reach the extension; an absent tag still reads as 1 (unsigned), so a file without it is unchanged. --- crates/raw/src/dng.rs | 2 +- crates/raw/src/dng/compat_review_tests.rs | 10 ++++++++++ 2 files changed, 11 insertions(+), 1 deletion(-) diff --git a/crates/raw/src/dng.rs b/crates/raw/src/dng.rs index f62aeef47..9589f5d4d 100644 --- a/crates/raw/src/dng.rs +++ b/crates/raw/src/dng.rs @@ -539,7 +539,7 @@ fn extend_understated_strip(bytes: &[u8], info: &mut ImageInfo, raw: &Ifd) { || info.compression != t::compression::NONE || info.samples_per_pixel != 1 || info.planar == 2 - || info.sample_format == 3 + || !matches!(info.sample_format, 1 | 2) || !matches!(bits, 8 | 16) { return; diff --git a/crates/raw/src/dng/compat_review_tests.rs b/crates/raw/src/dng/compat_review_tests.rs index 149ee492d..7fcc1b4f4 100644 --- a/crates/raw/src/dng/compat_review_tests.rs +++ b/crates/raw/src/dng/compat_review_tests.rs @@ -122,7 +122,15 @@ fn decoded_notices_record_actual_crop_gain_table_and_cfa_decisions() { /// A 16-bit CFA DNG of `width` x `height` whose strip holds every row in full but whose `StripByteCounts` says /// `counted` bytes. fn understated_strip_dng(width: u32, height: u32, counted: u32, active_area: Option<[u32; 4]>) -> Vec { + understated_strip_dng_with(width, height, counted, active_area, &[]) +} + +/// [`understated_strip_dng`] with `extra` tags set on the raw IFD. +fn understated_strip_dng_with(width: u32, height: u32, counted: u32, active_area: Option<[u32; 4]>, extra: &[(u16, Value)]) -> Vec { let mut raw = IfdBuilder::new(); + for (tag, value) in extra { + raw.set(*tag, value.clone()); + } for (tag, value) in [ (t::IMAGE_WIDTH, Value::Long(vec![width])), (t::IMAGE_LENGTH, Value::Long(vec![height])), @@ -170,6 +178,8 @@ fn other_short_strips_are_still_refused() { assert!(refused(understated_strip_dng(8, 4, 6 * 4 * 2, None))); // the count is not exactly the area's bytes assert!(refused(understated_strip_dng(8, 4, 5 * 4 * 2, Some([0, 0, 4, 6])))); + // samples that are neither integer nor absent-format (SampleFormat 4 is undefined) + assert!(refused(understated_strip_dng_with(8, 4, 6 * 4 * 2, Some([0, 0, 4, 6]), &[(t::SAMPLE_FORMAT, Value::Short(vec![4]))]))); // the area reaches below the image, so decode would discard it assert!(refused(understated_strip_dng(8, 4, 6 * 4 * 2, Some([4, 0, 5, 6])))); // the area does not start at column 0 From 09938cd99e9189890ace95006de4c6fc2211e468 Mon Sep 17 00:00:00 2001 From: Mauro Mazzieri Date: Tue, 17 Nov 2026 15:53:20 +0000 Subject: [PATCH 040/161] craft-rtf: page fields show each page's number Ported from storytold/wordcraft#6264 (issue storytold/wordcraft#6222), written before the RTF reader moved here. A PAGE, NUMPAGES, SECTIONPAGES, SECTION or PAGEREF field whose result is plain text in one formatting comes in as a field, with its lock, so the layout works out the number; \chpgn is a PAGE field and keeps a tracked insertion or deletion; a result whose text was hidden stays as read. --- crates/flow/src/to_doc.rs | 23 ++-- crates/rtf/src/doc.rs | 8 +- crates/rtf/src/doc/field.rs | 144 ++++++++++++++++++++ crates/rtf/src/doc/reader.rs | 6 + crates/rtf/src/doc/tests_fields.rs | 208 +++++++++++++++++++++++++++++ crates/rtf/src/doc/text.rs | 36 +++-- crates/rtf/src/doc/word.rs | 8 +- 7 files changed, 409 insertions(+), 24 deletions(-) create mode 100644 crates/rtf/src/doc/field.rs create mode 100644 crates/rtf/src/doc/tests_fields.rs diff --git a/crates/flow/src/to_doc.rs b/crates/flow/src/to_doc.rs index 0375c448c..8d9d4dca3 100644 --- a/crates/flow/src/to_doc.rs +++ b/crates/flow/src/to_doc.rs @@ -284,6 +284,18 @@ impl<'a> Builder<'a> { /// `cp` as a run may keep it: without its link and font once the runs' copies of them are /// over budget. + /// A run's properties with its tracked insertion, deletion and formatting change. + fn tracked_props(&mut self, f: &Fmt) -> CharProps { + let mut cp = self.budget(char_props(f)); + cp.ins = f.ins.as_ref().map(|m| self.revision(RevisionKind::Insert, m)); + cp.del = f.del.as_ref().map(|m| self.revision(RevisionKind::Delete, m)); + if let Some(c) = &f.fmt_change { + let old = self.budget(char_props(&c.old)); + cp.fmt_change = PropChange::boxed(self.revision(RevisionKind::Format, &c.mark), old); + } + cp + } + fn budget(&mut self, mut cp: CharProps) -> CharProps { let n = cp.link.as_ref().map_or(0, String::len).saturating_add(cp.font.as_ref().map_or(0, String::len)); match self.run_strings.checked_add(n).filter(|&t| t <= MAX_RUN_STRINGS) { @@ -553,13 +565,7 @@ impl<'a> Builder<'a> { match i { Inline::Text(t, f) => { let t = clean_text(t); - let mut cp = self.budget(char_props(f)); - cp.ins = f.ins.as_ref().map(|m| self.revision(RevisionKind::Insert, m)); - cp.del = f.del.as_ref().map(|m| self.revision(RevisionKind::Delete, m)); - if let Some(c) = &f.fmt_change { - let old = self.budget(char_props(&c.old)); - cp.fmt_change = PropChange::boxed(self.revision(RevisionKind::Format, &c.mark), old); - } + let mut cp = self.tracked_props(f); if p.kind == Kind::Code { // Code's font comes from its style, before a change as after it. if cp.font.as_deref() == Some(MONO_FONT) { @@ -633,7 +639,8 @@ impl<'a> Builder<'a> { } } Inline::Field { instr, result, fmt, locked } => { - let cp = self.budget(char_props(fmt)); + // A field inserted or deleted with tracked changes keeps them (RTF `\chpgn`). + let cp = self.tracked_props(fmt); let (instr, result) = (clean_text(instr), clean_text(result)); if instr.trim().is_empty() { append(&mut out, &result, &cp, None); diff --git a/crates/rtf/src/doc.rs b/crates/rtf/src/doc.rs index 77254151f..0426a4b7a 100644 --- a/crates/rtf/src/doc.rs +++ b/crates/rtf/src/doc.rs @@ -22,7 +22,10 @@ //! code page's bytes stream through one decoder, so a lead byte and its trail byte decode //! together and a run that looks like a UTF-8 BOM is not mistaken for one; the formatting words //! above, stylesheet-based -//! headings and `\outlinelevel`, lists through `\ls`/`\ilvl`, fields, bookmarks, pictures and +//! headings and `\outlinelevel`, lists through `\ls`/`\ilvl`, fields (in `import`, a page field +//! such as `PAGE` or `NUMPAGES` whose result is plain text in one paragraph stays a field, +//! `\fldlock` locking it, and `\chpgn` is a `PAGE` field; otherwise a field is its result's text), +//! bookmarks, pictures and //! tables, footnotes, endnotes (`\footnote` with `\ftnalt`) and comments (author, initials, date, //! text, the commented range). Group nesting, picture sizes and the number of notes and comments //! are capped. Not kept: custom note marks, comment replies (`\atnparent`), notes inside @@ -38,6 +41,7 @@ use craft_flow::{self as model, Cell, FBlock, FTable, Flow, Fmt, Inline, Kind, L use crate::date; mod export; +mod field; mod parse; mod reader; mod state; @@ -58,3 +62,5 @@ use word::hyperlink; #[cfg(test)] mod tests; +#[cfg(test)] +mod tests_fields; diff --git a/crates/rtf/src/doc/field.rs b/crates/rtf/src/doc/field.rs new file mode 100644 index 000000000..78d5f7d81 --- /dev/null +++ b/crates/rtf/src/doc/field.rs @@ -0,0 +1,144 @@ +//! Page fields kept as fields (#6222). The reader writes a field's result (`\fldrslt`) into the +//! paragraph like any text, which for a page number is the number Word last stored. When the +//! group of a field the layout works out from where it lands closes (`PAGE`, `NUMPAGES`, +//! `SECTIONPAGES`, `SECTION`, `PAGEREF`), a result that is plain text in one formatting, in the +//! paragraph the field started in, is replaced by an [`Inline::Field`] holding the instruction +//! (`\fldinst`) and that result, as for a DOCX field. Any other field, a result over several +//! paragraphs or formats, or one holding pictures or other objects stays what the reader made of +//! it: a field's result is drawn as one piece that does not wrap, so a long `MERGEFIELD` result +//! in a narrow table cell would overflow it. Only `import` keeps fields: `parse`, the flow +//! reader, reads their results as text. + +use super::reader::Reader; +use super::state::Dest; +use super::{Fmt, Inline}; + +/// The open fields read so far; deeper nesting keeps its results as text. +const MAX_OPEN: usize = 32; + +/// A field being read in the body text. +pub(super) struct OpenField { + /// The depth of the `{\field` group. + depth: usize, + /// Its instruction, in [`Reader::fields`]. + index: usize, + /// The paragraph it started in ([`Reader::para_seq`]). + para: u64, + /// The paragraph's inlines when it started, and the length of the last one when that was + /// text: a result in the same formatting runs on in that text. + start: usize, + tail_from: Option, + /// Its result group was met in the text (not inside another field's instruction). + result: bool, + locked: bool, + /// Hidden text of its result was left out: the result is not what showed. + hidden: bool, +} + +impl Reader { + /// `\field`, after its instruction slot is made: note where its result starts. + pub(super) fn open_field(&mut self) { + // Hidden text drops the result: the field stays hidden with it. + if !self.marks_on || self.st.dest != Dest::Body || self.st.hidden || self.open_fields.len() >= MAX_OPEN { + return; + } + let Some(index) = self.st.field else { return }; + let tail_from = match self.para.inlines.last() { + Some(Inline::Text(t, _)) => Some(t.len()), + _ => None, + }; + let (depth, para, start) = (self.depth(), self.para_seq, self.para.inlines.len()); + self.open_fields.push(OpenField { depth, index, para, start, tail_from, result: false, locked: false, hidden: false }); + } + + /// `\fldrslt` and `\fldlock`, read in a field's group. + pub(super) fn field_word(&mut self, w: &str) { + let (depth, field, body) = (self.depth(), self.st.field, self.st.dest == Dest::Body); + let Some(f) = self.open_fields.last_mut() else { return }; + match w { + "fldrslt" if body && field == Some(f.index) && depth == f.depth + 1 => f.result = true, + "fldlock" if depth == f.depth => f.locked = true, + _ => {} + } + } + + /// `\chpgn`, the current page number: a `PAGE` field. + pub(super) fn page_number_symbol(&mut self) { + if self.marks_on && self.st.dest == Dest::Body && !self.st.hidden { + // Tracked like text, so Accept and Reject apply to it. + let fmt = self.text_fmt(); + self.para.inlines.push(Inline::Field { instr: "PAGE".into(), result: String::new(), fmt, locked: false }); + self.para_has = true; + } + } + + /// Hidden text left out of the body: an open field's result is then not what showed, and the + /// field is left as read rather than made a field that would show. + pub(super) fn hidden_text_dropped(&mut self) { + if let Some(f) = self.open_fields.last_mut() + && f.result + { + f.hidden = true; + } + } + + /// A group closes: if it is an open field's, make the field. + pub(super) fn close_field_group(&mut self) { + if self.open_fields.last().is_none_or(|f| f.depth != self.depth()) { + return; + } + let Some(f) = self.open_fields.pop() else { return }; + if !f.result || f.hidden || f.para != self.para_seq { + return; + } + let instr = self.fields.get(f.index).map_or("", |s| s.trim()); + if !page_dependent(instr) { + return; + } + let Some((result, fmt)) = result_text(&self.para.inlines, &f, &self.st.fmt) else { return }; + let field = Inline::Field { instr: instr.to_string(), result, fmt, locked: f.locked }; + let inlines = &mut self.para.inlines; + inlines.truncate(f.start); + if let (Some(n), Some(Inline::Text(t, _))) = (f.tail_from, inlines.last_mut()) { + t.truncate(n); + } + inlines.push(field); + self.para_has = true; + } +} + +/// A field whose result the layout works out from the page it lands on. `STYLEREF` is not one +/// here: the layout looks its style up by name, and a running head whose style the RTF names +/// differently would show an error instead of Word's text. +fn page_dependent(instr: &str) -> bool { + let name = instr.split_whitespace().next().unwrap_or(""); + ["PAGE", "NUMPAGES", "SECTIONPAGES", "SECTION", "PAGEREF"].iter().any(|n| name.eq_ignore_ascii_case(n)) +} + +/// The text the field's result added to the paragraph and its formatting, when that is plain +/// text in one formatting with no tracked change; `None` otherwise. An empty result takes the +/// field group's formatting. +fn result_text(inlines: &[Inline], f: &OpenField, group: &Fmt) -> Option<(String, Fmt)> { + if inlines.len() < f.start { + return None; + } + let mut runs: Vec<(&str, &Fmt)> = Vec::new(); + if let Some(n) = f.tail_from { + let Some(Inline::Text(t, fmt)) = f.start.checked_sub(1).and_then(|i| inlines.get(i)) else { return None }; + let tail = t.get(n..)?; + if !tail.is_empty() { + runs.push((tail, fmt)); + } + } + for i in inlines.get(f.start..)? { + match i { + Inline::Text(t, fmt) => runs.push((t, fmt)), + _ => return None, + } + } + match runs.as_slice() { + [] => Some((String::new(), group.clone())), + [(t, fmt)] if fmt.ins.is_none() && fmt.del.is_none() && fmt.fmt_change.is_none() => Some(((*t).to_string(), (*fmt).clone())), + _ => None, + } +} diff --git a/crates/rtf/src/doc/reader.rs b/crates/rtf/src/doc/reader.rs index c0cc1c1ea..4c8a8b652 100644 --- a/crates/rtf/src/doc/reader.rs +++ b/crates/rtf/src/doc/reader.rs @@ -56,6 +56,10 @@ pub(super) struct Reader { pub(super) ov_list: i32, pub(super) ov_ls: i32, pub(super) fields: Vec, + /// The fields being read in the text, innermost last (`import` only). + pub(super) open_fields: Vec, + /// Counts the paragraphs ended, so a field knows whether its result stayed in one. + pub(super) para_seq: u64, // Picture being read. pub(super) pict_depth: Option, pub(super) pict_hex: Vec, @@ -344,6 +348,7 @@ impl Reader { if self.sub.as_ref().is_some_and(|s| s.depth == self.depth()) { self.end_sub(); } + self.close_field_group(); let closing = self.st.dest; // Destination-level commits. match closing { @@ -428,6 +433,7 @@ impl Reader { /// End the current paragraph (`\par`, `\cell`). pub(super) fn end_para(&mut self, in_cell: bool) { + self.para_seq += 1; let read = std::mem::take(&mut self.para); let props = self.para_props(&self.st); let mut p = Para { inlines: read.inlines, mark_ins: read.mark_ins, mark_del: read.mark_del, ..props }; diff --git a/crates/rtf/src/doc/tests_fields.rs b/crates/rtf/src/doc/tests_fields.rs new file mode 100644 index 000000000..ae1ab691f --- /dev/null +++ b/crates/rtf/src/doc/tests_fields.rs @@ -0,0 +1,208 @@ +//! Page fields read from RTF (#6222): a page-dependent field whose result is plain text in one +//! paragraph comes in as a field, so the layout works out its number; other fields stay their +//! result's text. Written by hand from the RTF specification's `\field`, `\fldinst`, `\fldrslt`, +//! `\fldlock` and `\chpgn`. + +use super::*; +use craft_wordmodel::para::InlineObject; + +/// The fields of each paragraph of a story, with the paragraph's text. +/// A paragraph's text with its fields (code, result, locked). +type ParaFields = (String, Vec<(String, String, bool)>); + +fn fields_of(blocks: &craft_wordmodel::Blocks) -> Vec { + blocks + .iter() + .filter_map(|b| if let craft_wordmodel::Block::Para(p) = &**b { Some(p) } else { None }) + .map(|p| { + let fields = p + .objects + .iter() + .filter_map(|o| match o { + InlineObject::Field { instr, result, locked, .. } => Some((instr.clone(), result.clone(), *locked)), + _ => None, + }) + .collect(); + (p.text.clone(), fields) + }) + .collect() +} + +fn footer(d: &craft_wordmodel::Document) -> Vec { + let id = d.last_section.footers.default.expect("a footer"); + fields_of(&d.parts[&id].blocks) +} + +#[test] +fn a_page_field_in_the_footer_is_a_field_not_its_stored_number() { + // As Word writes it: the text before the field has the same formatting as its result. + let rtf = br"{\rtf1\ansi\sectd +{\footerr \pard {\fs16 Pag. }{\field{\*\fldinst {\fs16 PAGE }}{\fldrslt {\fs16 2}}}\sectd \ltrsect{\par}} +\pard One\page Two\par}"; + let d = import(rtf).unwrap(); + let f = footer(&d); + assert_eq!(f.len(), 1, "{f:?}"); + assert_eq!(f[0].1, [("PAGE".to_string(), "2".to_string(), false)], "{f:?}"); + // "Pag. " stays text before the field: the result's run did not take it along. + assert!(f[0].0.starts_with("Pag. "), "{f:?}"); + assert_eq!(f[0].0.matches("Pag. ").count(), 1, "{f:?}"); +} + +#[test] +fn page_fields_in_the_body_keep_their_code_result_and_lock() { + let rtf = br#"{\rtf1\ansi +\pard Page {\field{\*\fldinst PAGE \\* roman}{\fldrslt ii}} of {\field\fldlock{\*\fldinst NUMPAGES}{\fldrslt 4}}\par +\pard {\field{\*\fldinst SECTIONPAGES}{\fldrslt }}x\par}"#; + let d = import(rtf).unwrap(); + let f = fields_of(&d.body); + assert_eq!(f[0].1, [("PAGE \\* roman".to_string(), "ii".to_string(), false), ("NUMPAGES".to_string(), "4".to_string(), true)], "{f:?}"); + assert!(f[0].0.starts_with("Page ") && f[0].0.contains(" of "), "{f:?}"); + // An empty result is still a field: the layout fills it in. + assert_eq!(f[1].1, [("SECTIONPAGES".to_string(), String::new(), false)], "{f:?}"); +} + +#[test] +fn other_fields_stay_the_text_of_their_result() { + // A field's result is drawn as one piece that does not wrap: a long merge field in a narrow + // cell keeps its text, which wraps, as before. + let rtf = br#"{\rtf1\ansi +\pard Before {\field{\*\fldinst DATE \\@ "dd/MM/yyyy"}{\fldrslt 11/11/2026}} after\par +\trowd\cellx600\pard\intbl {\field{\*\fldinst MERGEFIELD Approvato}{\fldrslt Presidente della Rete (Rossi Mario)}}\cell\row +\pard {\field{\*\fldinst AUTHOR}{\fldrslt }}x\par}"#; + let d = import(rtf).unwrap(); + let f = fields_of(&d.body); + assert_eq!((f[0].0.as_str(), f[0].1.len()), ("Before 11/11/2026 after", 0), "{f:?}"); + let cell = d.body.iter().find_map(|b| b.as_table()).and_then(|t| t.rows.first()).and_then(|r| r.cells.first()).unwrap(); + let cf = fields_of(&cell.blocks); + assert_eq!((cf[0].0.as_str(), cf[0].1.len()), ("Presidente della Rete (Rossi Mario)", 0), "{cf:?}"); + assert!(f.last().is_some_and(|(t, fl)| t == "x" && fl.is_empty()), "{f:?}"); +} + +#[test] +fn fields_that_cannot_be_one_field_stay_text() { + let rtf = br#"{\rtf1\ansi +\pard {\field{\*\fldinst{HYPERLINK "http://a.b"}}{\fldrslt{link}}}\par +\pard {\field{\*\fldinst TOC \\o "1-3"}{\fldrslt {Heading one\tab 1\par Heading two\tab 2}}}\par +\pard {\field{\*\fldinst PAGEREF a}{\fldrslt {\b 1}2}}\par +\pard {\field{\*\fldinst IF {\field{\*\fldinst PAGE}{\fldrslt 1}} = 1 "one" "many"}{\fldrslt one}}\par +\pard {\field{\*\fldinst PAGE}}\par +\pard {\field{\*\fldinst PAGE}{\fldrslt 1\par 2}}\par}"#; + let d = import(rtf).unwrap(); + let f = fields_of(&d.body); + let texts: Vec<&str> = f.iter().map(|(t, _)| t.as_str()).collect(); + // A link stays a link; a result over two paragraphs or in two formats stays its text. + assert_eq!(&texts[..4], ["link", "Heading one\t1", "Heading two\t2", "12"], "{f:?}"); + // The PAGE inside IF's instruction is part of the code, not a field of the paragraph. + assert_eq!(texts[4], "one", "{f:?}"); + // A field with no result group at all is left as it was (nothing). + assert!(f[5].0.is_empty(), "{f:?}"); + assert_eq!(&texts[6..8], ["1", "2"], "{f:?}"); + assert!(f.iter().all(|(_, fl)| fl.is_empty()), "{f:?}"); +} + +#[test] +fn a_table_of_contents_keeps_its_links_and_its_page_numbers_become_fields() { + // The shape Word writes: the TOC result spans paragraphs; each entry is a link around its + // text and a PAGEREF to the heading's bookmark. + let rtf = br#"{\rtf1\ansi +\pard {\field{\*\fldinst TOC \\o "1-3" \\h}{\fldrslt {\field{\*\fldinst HYPERLINK \\l "_Toc1"}{\fldrslt {One\tab {\field{\*\fldinst PAGEREF _Toc1 \\h}{\fldrslt 3}}}}}\par +{\field{\*\fldinst HYPERLINK \\l "_Toc2"}{\fldrslt {Two\tab {\field{\*\fldinst PAGEREF _Toc2 \\h}{\fldrslt 5}}}}}}}\par +\pard {\*\bkmkstart _Toc1}Heading{\*\bkmkend _Toc1}\par}"#; + let d = import(rtf).unwrap(); + let f = fields_of(&d.body); + assert_eq!(f[0].1, [("PAGEREF _Toc1 \\h".to_string(), "3".to_string(), false)], "{f:?}"); + assert_eq!(f[1].1, [("PAGEREF _Toc2 \\h".to_string(), "5".to_string(), false)], "{f:?}"); + assert!(f[0].0.starts_with("One\t") && f[1].0.starts_with("Two\t"), "{f:?}"); + // The entry text still links to its heading. + let p = d.body[0].as_para().unwrap(); + assert!(p.props_at(0).link.is_some(), "{:?}", p.runs); +} + +#[test] +fn hidden_styleref_tracked_and_object_results_stay_as_read() { + let rtf = br#"{\rtf1\ansi{\*\revtbl{Unknown;}{Ann;}} +\pard a{\v {\field{\*\fldinst PAGE}{\fldrslt 2}}}b\par +\pard {\field{\*\fldinst STYLEREF "heading 1"}{\fldrslt Intro}}\par +\pard {\field{\*\fldinst PAGE}{\fldrslt {\revised\revauth1 2}}}\par +\pard {\field{\*\fldinst PAGE}{\fldrslt A{\*\bkmkstart m}B{\*\bkmkend m}}}\par}"#; + let d = import(rtf).unwrap(); + let f = fields_of(&d.body); + // Hidden: neither the result nor a field. + assert_eq!((f[0].0.as_str(), f[0].1.len()), ("ab", 0), "{f:?}"); + // STYLEREF keeps Word's text. + assert_eq!((f[1].0.as_str(), f[1].1.len()), ("Intro", 0), "{f:?}"); + // A tracked result and one holding a bookmark stay text. + assert_eq!((f[2].0.as_str(), f[2].1.len()), ("2", 0), "{f:?}"); + // (The bookmark's start and end are object characters in the paragraph's text.) + assert_eq!((f[3].0.replace('\u{FFFC}', ""), f[3].1.len()), ("AB".to_string(), 0), "{f:?}"); +} + +#[test] +fn the_page_number_symbol_is_a_page_field() { + let d = import(br"{\rtf1\ansi{\footer \pard Page \chpgn\par}\pard Body\par}").unwrap(); + let f = footer(&d); + assert_eq!(f[0].1, [("PAGE".to_string(), String::new(), false)], "{f:?}"); +} + +#[test] +fn truncated_and_hostile_fields_do_not_crash() { + for rtf in [ + &br"{\rtf1{\field{\*\fldinst PAGE}{\fldrslt 2"[..], + br"{\rtf1{\field{\*\fldinst PAGE}{\fldrslt 2}", + br"{\rtf1{\field\field\field{\*\fldinst PAGE}}}}", + br"{\rtf1{\field{\fldrslt x}{\*\fldinst PAGE}}\par}", + br"{\rtf1 a{\field{\*\fldinst PAGE}{\fldrslt 1\par 2}}\par}", + ] { + let _ = import(rtf); + } + let mut deep = String::from(r"{\rtf1 "); + for _ in 0..300 { + deep.push_str(r"{\field{\*\fldinst PAGE}{\fldrslt "); + } + deep.push('x'); + let _ = import(deep.as_bytes()); +} + +#[test] +fn the_flow_reader_keeps_field_results_as_text() { + // `parse` asks for no fields, as before: the result is the paragraph's text. + let f = parse(br"{\rtf1\ansi\pard Pag. {\field{\*\fldinst PAGE}{\fldrslt 2}}\par}").unwrap(); + match &f.blocks[0] { + FBlock::Para(p) => { + assert!(p.inlines.iter().all(|i| matches!(i, Inline::Text(..))), "{:?}", p.inlines); + assert_eq!(p.text(), "Pag. 2"); + } + b => panic!("{b:?}"), + } +} + +#[test] +fn a_tracked_page_symbol_keeps_its_insertion_and_deletion() { + let rtf = br"{\rtf1\ansi{\*\revtbl{Unknown;}{Ann;}} +\pard Page {\revised\revauth1 \chpgn}\par +\pard Page {\deleted\revauthdel1 \chpgn}\par}"; + let d = import(rtf).unwrap(); + let f = fields_of(&d.body); + assert_eq!(f[0].1.len(), 1, "{f:?}"); + assert_eq!(f[1].1.len(), 1, "{f:?}"); + // Each field sits in a run carrying its tracked change, so Accept and Reject apply to it. + let marked = |i: usize, ins: bool| { + let p = d.body[i].as_para().unwrap(); + p.runs.iter().any(|r| if ins { r.props.ins.is_some() } else { r.props.del.is_some() }) + }; + assert!(marked(0, true), "{:?}", d.body[0].as_para().unwrap().runs); + assert!(marked(1, false), "{:?}", d.body[1].as_para().unwrap().runs); +} + +#[test] +fn a_result_whose_text_is_hidden_inside_it_stays_as_read() { + // `\v` inside the result: what showed was nothing, so no field that would show a number. + let rtf = br"{\rtf1\ansi +\pard x{\field{\*\fldinst PAGE}{\fldrslt \v 2}}y\par +\pard x{\field{\*\fldinst PAGE}{\fldrslt }}y\par}"; + let d = import(rtf).unwrap(); + let f = fields_of(&d.body); + assert_eq!((f[0].0.as_str(), f[0].1.len()), ("xy", 0), "{f:?}"); + // A result that is really empty is still a field. + assert_eq!(f[1].1.len(), 1, "{f:?}"); +} diff --git a/crates/rtf/src/doc/text.rs b/crates/rtf/src/doc/text.rs index dceaef6d0..ad997c680 100644 --- a/crates/rtf/src/doc/text.rs +++ b/crates/rtf/src/doc/text.rs @@ -189,31 +189,39 @@ impl Reader { } } + /// The current formatting with its tracked changes (insertion, deletion, formatting change). + pub(super) fn text_fmt(&mut self) -> Fmt { + let mut f = self.st.fmt.clone(); + let r = self.st.rev; + if r.ins { + f.ins = Some(self.rev_mark(r.auth, r.dttm)); + } + if r.del { + f.del = Some(self.rev_mark(r.authdel, r.dttmdel)); + } + let c = &self.st.chg_c; + if let Some(words) = c.old.clone() { + let mark = self.rev_mark(c.auth, c.date); + let old = self.old_fmt(&words); + f.fmt_change = Some(Box::new(model::FmtChange { mark, old })); + } + f + } + pub(super) fn text(&mut self, s: &str) { match self.st.dest { Dest::Body => { if self.st.hidden { + self.hidden_text_dropped(); return; } - let mut f = self.st.fmt.clone(); if s.contains([MARK_OPEN, MARK_CLOSE]) { + let f = self.st.fmt.clone(); self.para.push_text(&s.replace([MARK_OPEN, MARK_CLOSE], "\u{FFFD}"), &f); self.para_has = true; return; } - let r = self.st.rev; - if r.ins { - f.ins = Some(self.rev_mark(r.auth, r.dttm)); - } - if r.del { - f.del = Some(self.rev_mark(r.authdel, r.dttmdel)); - } - let c = &self.st.chg_c; - if let Some(words) = c.old.clone() { - let mark = self.rev_mark(c.auth, c.date); - let old = self.old_fmt(&words); - f.fmt_change = Some(Box::new(model::FmtChange { mark, old })); - } + let f = self.text_fmt(); self.para.push_text(s, &f); self.para_has = true; } diff --git a/crates/rtf/src/doc/word.rs b/crates/rtf/src/doc/word.rs index 2eb86692f..49367a7d1 100644 --- a/crates/rtf/src/doc/word.rs +++ b/crates/rtf/src/doc/word.rs @@ -75,6 +75,7 @@ impl Reader { "row" => self.end_row(), "line" | "softline" => self.ch('\n'), "tab" => self.ch('\t'), + "chpgn" => self.page_number_symbol(), "page" | "softpage" => self.ch('\u{000C}'), "column" => self.ch('\u{000E}'), "emdash" => self.ch('—'), @@ -283,8 +284,13 @@ impl Reader { self.defs.push(d); } } - "field" => self.start_field(), + "field" => { + self.start_field(); + self.open_field(); + } + "fldlock" => self.field_word(w), "fldrslt" => { + self.field_word(w); let link = self.st.field.and_then(|i| self.fields.get(i)).and_then(|s| hyperlink(s)); if link.is_some() { self.st.fmt.link = link.as_deref().and_then(model::kept_link); From a62ce81f2447b1791a8d8c0aa6317fb6388f189b Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Tue, 17 Nov 2026 11:54:40 -0500 Subject: [PATCH 041/161] Expose consuming parsed certificate and revocation components --- crates/pkcs7/src/revocation.rs | 53 ++++++++++++++++++- crates/pkcs7/src/x509.rs | 90 +++++++++++++++++++++++++++++++++ crates/pkcs7/tests/it/crypto.rs | 53 +++++++++++++++++++ 3 files changed, 194 insertions(+), 2 deletions(-) diff --git a/crates/pkcs7/src/revocation.rs b/crates/pkcs7/src/revocation.rs index 0aa9be9c6..dfb398cae 100644 --- a/crates/pkcs7/src/revocation.rs +++ b/crates/pkcs7/src/revocation.rs @@ -63,6 +63,27 @@ pub struct CertificateList { pub revoked: Vec<(Vec, Time)>, } +/// Owned parsed components for adapters. This is not a signature or trust verdict. +/// Consuming conversion moves buffers without reparsing or cloning signed material. +pub struct CertificateListParts { + pub tbs: Vec, + pub sig_alg: Vec, + pub signature: Vec, + pub issuer_raw: Vec, + pub this_update: Time, + pub next_update: Option".repeat(59)); + let mut zip = zip::ZipWriter::new(std::io::Cursor::new(Vec::new())); + zip.start_file("word/document.xml", zip::write::SimpleFileOptions::default()).unwrap(); + zip.write_all(bad_xml.as_bytes()).unwrap(); + let bad = zip.finish().unwrap().into_inner(); + let sibling = package_parts(r#"sibling"#, &[], &[]); + let outer = package_parts( + r#""#, + &[("bad", "aFChunk", "bad.docx"), ("good", "aFChunk", "good.docx")], + &[("word/bad.docx", &bad), ("word/good.docx", &sibling)], + ); + for (cap, included, remaining) in [(69, true, 0), (68, false, 4)] { + let pkg = Package::open_limited_within(&outer, MAX_PART, MAX_TOTAL, cap).unwrap(); + let doc = super::super::read_opened(&pkg, None).unwrap(); + assert_eq!(has(&doc, "sibling"), included, "budget {cap}"); + assert_eq!(pkg.elements_left(), remaining, "budget {cap}"); + } + } } From 4078682a5179b831c73303ff37a24d9b17749cfa Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Wed, 18 Nov 2026 01:17:12 -0500 Subject: [PATCH 104/161] mesh: expose strict UTF-8 URI percent decoding --- crates/mesh/src/gltf.rs | 10 ++++++++++ crates/mesh/src/tests/gltf.rs | 29 +++++++++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/crates/mesh/src/gltf.rs b/crates/mesh/src/gltf.rs index c92051b84..d074f1ed1 100644 --- a/crates/mesh/src/gltf.rs +++ b/crates/mesh/src/gltf.rs @@ -120,6 +120,16 @@ pub fn uri_decode(s: &str) -> String { String::from_utf8_lossy(&percent_decode(s)).into_owned() } +/// Percent-decode a URI, rejecting decoded bytes that are not UTF-8. +/// +/// As with [`uri_decode`], incomplete or non-hex escapes stay literal and `+` stays +/// `+`. This does not validate or confine a filesystem path; callers must retain +/// their own reference policy. Decoding is linear and output is no larger than input. +/// Use this instead of the lossy decoder when a resolver requires exact UTF-8 names. +pub fn uri_decode_strict(s: &str) -> Option { + String::from_utf8(percent_decode(s)).ok() +} + /// [`uri_decode`] as bytes, so escaped bytes that are not UTF-8 survive. fn percent_decode(s: &str) -> Vec { let b = s.as_bytes(); diff --git a/crates/mesh/src/tests/gltf.rs b/crates/mesh/src/tests/gltf.rs index 8892fe241..c47b911df 100644 --- a/crates/mesh/src/tests/gltf.rs +++ b/crates/mesh/src/tests/gltf.rs @@ -950,3 +950,32 @@ fn morph_targets_and_other_scenes_are_named_in_warnings() { let s = parse(&one_mesh(&b, plain.clone())).unwrap(); assert!(s.warnings.is_empty(), "{:?}", s.warnings); } + +#[test] +fn strict_uri_decoding_preserves_exact_names_and_rejects_invalid_utf8() { + use crate::gltf::uri_decode_strict; + for (input, expected) in [ + ("", Some("")), + ("a+b", Some("a+b")), + ("a%20b", Some("a b")), + ("é/%E2%82%AC", Some("é/€")), + ("%", Some("%")), + ("%2", Some("%2")), + ("%GG", Some("%GG")), + ("%252e%252e", Some("%2e%2e")), + ("%2e%2e%2fmesh.bin", Some("../mesh.bin")), + ("%00", Some("\0")), + ("%FF", None), + ("%C0%AF", None), + ("%ED%A0%80", None), + ("%E2%82", None), + ] { + assert_eq!(uri_decode_strict(input).as_deref(), expected, "{input}"); + } + // Keep the existing API's replacement-character behavior unchanged. + assert_eq!(uri_decode("%FF"), "�"); + let input = "%41%GG+é".repeat(100_000); + let decoded = uri_decode_strict(&input).unwrap(); + assert_eq!(decoded, "A%GG+é".repeat(100_000)); + assert!(decoded.len() <= input.len()); +} From 133b15453c0d695db0245776c46feb045e605693 Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Wed, 18 Nov 2026 01:21:44 -0500 Subject: [PATCH 105/161] fix(rtf): retain hidden-result and empty-field revision state --- crates/rtf/src/doc/field.rs | 22 +++++++++++++++------- crates/rtf/src/doc/tests_fields.rs | 27 +++++++++++++++++++++++++++ 2 files changed, 42 insertions(+), 7 deletions(-) diff --git a/crates/rtf/src/doc/field.rs b/crates/rtf/src/doc/field.rs index 78d5f7d81..ccc12f4cf 100644 --- a/crates/rtf/src/doc/field.rs +++ b/crates/rtf/src/doc/field.rs @@ -64,7 +64,11 @@ impl Reader { /// `\chpgn`, the current page number: a `PAGE` field. pub(super) fn page_number_symbol(&mut self) { - if self.marks_on && self.st.dest == Dest::Body && !self.st.hidden { + if self.st.dest == Dest::Body && self.st.hidden { + self.hidden_text_dropped(); + return; + } + if self.marks_on && self.st.dest == Dest::Body { // Tracked like text, so Accept and Reject apply to it. let fmt = self.text_fmt(); self.para.inlines.push(Inline::Field { instr: "PAGE".into(), result: String::new(), fmt, locked: false }); @@ -75,10 +79,11 @@ impl Reader { /// Hidden text left out of the body: an open field's result is then not what showed, and the /// field is left as read rather than made a field that would show. pub(super) fn hidden_text_dropped(&mut self) { - if let Some(f) = self.open_fields.last_mut() - && f.result - { - f.hidden = true; + // An omitted nested result is also omitted from each enclosing result. + for f in &mut self.open_fields { + if f.result { + f.hidden = true; + } } } @@ -95,8 +100,11 @@ impl Reader { if !page_dependent(instr) { return; } - let Some((result, fmt)) = result_text(&self.para.inlines, &f, &self.st.fmt) else { return }; - let field = Inline::Field { instr: instr.to_string(), result, fmt, locked: f.locked }; + let instr = instr.to_string(); + // An empty result has no run to carry the group's revision marks. + let group = self.text_fmt(); + let Some((result, fmt)) = result_text(&self.para.inlines, &f, &group) else { return }; + let field = Inline::Field { instr, result, fmt, locked: f.locked }; let inlines = &mut self.para.inlines; inlines.truncate(f.start); if let (Some(n), Some(Inline::Text(t, _))) = (f.tail_from, inlines.last_mut()) { diff --git a/crates/rtf/src/doc/tests_fields.rs b/crates/rtf/src/doc/tests_fields.rs index ae1ab691f..4931356e8 100644 --- a/crates/rtf/src/doc/tests_fields.rs +++ b/crates/rtf/src/doc/tests_fields.rs @@ -206,3 +206,30 @@ fn a_result_whose_text_is_hidden_inside_it_stays_as_read() { // A result that is really empty is still a field. assert_eq!(f[1].1.len(), 1, "{f:?}"); } + +#[test] +fn hidden_page_symbols_and_nested_results_do_not_reappear_as_fields() { + for result in [r"{\v\chpgn}", r"{\field{\*\fldinst DATE}{\fldrslt {\v 2}}}"] { + let rtf = format!(r"{{\rtf1\ansi\pard x{{\field{{\*\fldinst PAGE}}{{\fldrslt {result}}}}}y\par}}"); + let d = import(rtf.as_bytes()).unwrap(); + let f = fields_of(&d.body); + assert_eq!(f[0], ("xy".into(), Vec::new()), "{result}: {f:?}"); + } + // A visible nested result remains eligible for conversion by its enclosing PAGE field. + let d = import(br"{\rtf1\ansi{\field{\*\fldinst PAGE}{\fldrslt {\field{\*\fldinst DATE}{\fldrslt 2}}}}\par}").unwrap(); + assert_eq!(fields_of(&d.body)[0].1, [("PAGE".into(), "2".into(), false)]); +} + +#[test] +fn empty_page_fields_keep_insertion_and_deletion_marks() { + for (mark, insertion) in [(r"\revised\revauth1", true), (r"\deleted\revauthdel1", false)] { + let rtf = format!(r"{{\rtf1\ansi{{\*\revtbl{{Unknown;}}{{Ann;}}}}\pard {{{mark}{{\field{{\*\fldinst PAGE}}{{\fldrslt }}}}}}\par}}"); + let d = import(rtf.as_bytes()).unwrap(); + let fields = fields_of(&d.body); + assert_eq!(fields[0].1, [("PAGE".into(), String::new(), false)]); + let p = d.body[0].as_para().unwrap(); + assert_eq!(p.text, "\u{FFFC}", "the sole character is the field marker"); + let props = p.props_at(0); + assert!(if insertion { props.ins.is_some() } else { props.del.is_some() }, "{mark}: {props:?}"); + } +} From 1139798521fa28a0b474ab9c5580edeaf72f822c Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Wed, 18 Nov 2026 01:22:10 -0500 Subject: [PATCH 106/161] refactor(pkcs7): define certificate representations from one schema --- crates/pkcs7/src/x509.rs | 126 ++++++++++----------------------------- 1 file changed, 33 insertions(+), 93 deletions(-) diff --git a/crates/pkcs7/src/x509.rs b/crates/pkcs7/src/x509.rs index 5738e9c46..ae8c45fbb 100644 --- a/crates/pkcs7/src/x509.rs +++ b/crates/pkcs7/src/x509.rs @@ -106,9 +106,39 @@ impl Name { } } -/// A parsed certificate. -#[derive(Clone, Debug, PartialEq, Eq)] -pub struct Certificate { +// One field schema keeps the parsed representation and consuming adapter in sync. +// Visibility belongs to Certificate; parts are explicitly unverified public data. +macro_rules! certificate_representations { + ($( $(#[$attr:meta])* $vis:vis $field:ident: $ty:ty, )*) => { + /// A parsed certificate. Signed material remains private until consumed. + /// + /// ```compile_fail + /// fn replace_signed_bytes(cert: &mut craft_pkcs7::x509::Certificate) { + /// cert.tbs.clear(); + /// } + /// ``` + #[derive(Clone, Debug, PartialEq, Eq)] + pub struct Certificate { + $( $(#[$attr])* $vis $field: $ty, )* + } + + /// Owned parsed components for adapters. This is not a signature or trust verdict. + /// Consuming conversion moves buffers without reparsing or cloning signed material. + pub struct CertificateParts { + $( $(#[$attr])* pub $field: $ty, )* + } + + impl Certificate { + /// Move parsed fields into a caller-owned representation; verification remains separate. + pub fn into_parts(self) -> CertificateParts { + let Self { $( $field, )* } = self; + CertificateParts { $( $field, )* } + } + } + }; +} + +certificate_representations! { /// The whole DER encoding. pub raw: Vec, tbs: Vec, @@ -144,96 +174,6 @@ pub struct Certificate { pub unknown_critical: Vec, } -/// Owned parsed components for adapters. This is not a signature or trust verdict. -/// Consuming conversion moves buffers without reparsing or cloning signed material. -pub struct CertificateParts { - /// The whole DER encoding. - pub raw: Vec, - pub tbs: Vec, - pub sig_alg: Vec, - pub signature: Vec, - pub serial: Vec, - pub issuer: Name, - pub subject: Name, - pub not_before: Time, - pub not_after: Time, - pub public_key: PublicKey, - /// Basic constraints: a CA certificate. - pub is_ca: bool, - /// Whether the certificate has a basicConstraints extension at all (old v1 roots don't). - pub has_basic_constraints: bool, - /// Basic constraints `pathLenConstraint`: how many CA certificates may follow this one - /// before the end-entity certificate. - pub path_len: Option, - /// Key usage bits (bit 0 = digitalSignature, 1 = nonRepudiation, 5 = keyCertSign), if present. - pub key_usage: Option, - pub subject_key_id: Option>, - /// Authority key identifier (2.5.29.35), when present. - pub authority_key_id: Option>, - /// Extended key usage OIDs (2.5.29.37), when present. - pub extended_key_usage: Option>, - /// OCSP responder URLs from the Authority Information Access (1.3.6.1.5.5.7.1.1). - pub ocsp_urls: Vec, - /// CRL distribution point URLs (2.5.29.31). - pub crl_urls: Vec, - /// The `version` field: 0 for v1, 1 for v2, 2 for v3. - pub version: u64, - /// OIDs of extensions marked critical that this crate does not process (RFC 5280 §4.2). - pub unknown_critical: Vec, -} - -impl Certificate { - /// Move parsed fields into a caller-owned representation; verification remains separate. - pub fn into_parts(self) -> CertificateParts { - let Self { - raw, - tbs, - sig_alg, - signature, - serial, - issuer, - subject, - not_before, - not_after, - public_key, - is_ca, - has_basic_constraints, - path_len, - key_usage, - subject_key_id, - authority_key_id, - extended_key_usage, - ocsp_urls, - crl_urls, - version, - unknown_critical, - } = self; - CertificateParts { - raw, - tbs, - sig_alg, - signature, - serial, - issuer, - subject, - not_before, - not_after, - public_key, - is_ca, - has_basic_constraints, - path_len, - key_usage, - subject_key_id, - authority_key_id, - extended_key_usage, - ocsp_urls, - crl_urls, - version, - unknown_critical, - } - } -} - /// Extensions this crate reads or may safely ignore when they are critical: the ones /// `Extensions::read` handles, subject/issuer alternative names, and certificate policies /// (accepted as any policy). From 2dc55f434142ce20e4b89c5986199849a65f6f8b Mon Sep 17 00:00:00 2001 From: FredApps <153173854+FredApps@users.noreply.github.com> Date: Wed, 18 Nov 2026 13:35:40 +0700 Subject: [PATCH 107/161] test(texture): make preserved golden float type explicit (#2131) --- crates/texture/src/lib.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/texture/src/lib.rs b/crates/texture/src/lib.rs index 54dffcb17..884488ac6 100644 --- a/crates/texture/src/lib.rs +++ b/crates/texture/src/lib.rs @@ -1124,7 +1124,7 @@ mod tests { let cases = [ (0.3, 0.7, [true, false], [1.473_799_7, -0.072_800_055, 0.645_000_2, 0.733_799_8]), (0.3, 0.7, [false, true], [1.473_799_7, -0.072_800_055, 0.645_000_2, 0.733_799_8]), - (1.2, -0.4, [true, false], [-1.722_999_5, 0.127_000_57, 0.089_999_32, 1.939_999_3]), + (1.2, -0.4, [true, false], [-1.722_999_5, 0.127_000_57, 0.089_999_32_f32, 1.939_999_3]), (1.2, -0.4, [false, true], [-1.057_000_3, -1.094, 0.756_000_1, 2.606_000_2]), (-0.15, 1.1, [true, false], [2.051, -1.76, 0.090_000_15, 1.311_000_1]), (-0.15, 1.1, [false, true], [-1.834_000_6, 0.015_999_317, 0.608_000_5, 2.458_000_7]), From 5ae0786c5bb1206d70c5d26cd3778253deed06b6 Mon Sep 17 00:00:00 2001 From: chasse20 Date: Wed, 18 Nov 2026 01:54:53 -0500 Subject: [PATCH 108/161] Publish MCMAT v3 as a bounded procedural source archive --- crates/mcmat/README.md | 21 +-- crates/mcmat/src/lib.rs | 283 ++++++++++++++++++---------------------- 2 files changed, 133 insertions(+), 171 deletions(-) diff --git a/crates/mcmat/README.md b/crates/mcmat/README.md index 074a0ac44..2f69e68c9 100644 --- a/crates/mcmat/README.md +++ b/crates/mcmat/README.md @@ -1,17 +1,10 @@ # craft-mcmat -A bounded reader and writer for portable MaterialCraft `.mcmat` ZIP packages. A package -contains `material.json`, named texture maps and a source package. Applications interpret -parameter values and cook or render maps themselves; this crate does not depend on an app's -model. It supports callers that read either version 1 or version 2 of the manifest, or both. +The MCMAT v3 portable material is a ZIP containing a version-only `material.json`, one +`source.sbs`, and the required source assets under `resources/`. The sole root-level +material graph in the SBS is the entry. Applications inspect its exposed controls and +cook its outputs through MaterialCraft. Texture maps and MaterialX documents are generated +by exporters. Earlier archives must be republished from their original projects. -`is_material` checks for a manifest with a bounded ZIP index; `Material::open` takes caller-provided version, entry-count, map-count, per-entry and total -expanded-byte limits. Each subsequent `read` and `read_source` shares that budget, including -repeated references to the same ZIP member. Duplicate member names are refused. The manifest -preserves unknown fields at the top level and on maps; parameter, input and preset objects are -kept as JSON values so their unknown fields survive too. `write` preserves the caller's source -entry name and map names, and rejects duplicate archive member names through `craft-zip`. - -The source package is opaque. Version 1 commonly names `source.mcraft`; MaterialCraft's version 2 -writer names `source.sbs`. Callers that invoke another application must preserve the source's -format and extension rather than infer it from `.mcmat` alone. +`Material::open` checks archive entries and expansion budgets before returning source +files. `write` compresses the XML and version marker and preserves source asset bytes. diff --git a/crates/mcmat/src/lib.rs b/crates/mcmat/src/lib.rs index 75b8709ef..076fe636a 100644 --- a/crates/mcmat/src/lib.rs +++ b/crates/mcmat/src/lib.rs @@ -1,133 +1,126 @@ #![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)] #![forbid(unsafe_code)] -//! Bounded interchange for MaterialCraft portable materials (`.mcmat`). -//! -//! The manifest and its version belong to the format, while interpreting parameter values, -//! cooking maps and opening the source package belong to the caller. Unknown fields survive -//! deserialization and serialization, including fields added by later versions. - -use std::collections::BTreeMap; +//! The clean-room MCMAT v3 source archive. Applications interpret the SBS entry graph. use craft_zip::{Archive, Compression, Writer}; -use serde::{Deserialize, Serialize}; -use serde_json::Value; +use serde::Deserialize; +use std::collections::{BTreeMap, BTreeSet}; -pub const FORMAT: &str = "craft-material"; +pub const VERSION: u32 = 3; pub const MANIFEST_ENTRY: &str = "material.json"; +pub const SOURCE_ENTRY: &str = "source.sbs"; +const MARKER: &[u8] = br#"{"version":3}"#; -/// A portable manifest; typed parameter values remain JSON so applications can adapt their own model. -#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] -#[serde(rename_all = "camelCase")] -pub struct Manifest { - pub format: String, - pub version: u32, - pub name: String, - pub graph: String, - pub size: [u32; 2], - pub maps: Vec, - #[serde(default)] - pub parameters: Vec, - #[serde(default)] - pub inputs: Vec, - #[serde(default)] - pub presets: Vec, - pub source: String, - #[serde(flatten)] - pub extra: BTreeMap, -} - -#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] -#[serde(rename_all = "camelCase")] -pub struct MapEntry { - pub usage: String, - pub file: String, - #[serde(flatten)] - pub extra: BTreeMap, -} - -/// All limits are supplied by the consumer, including the versions it understands. #[derive(Clone, Copy, Debug)] pub struct Limits { - pub min_version: u32, - pub max_version: u32, pub max_entries: u64, - pub max_maps: usize, pub max_manifest_bytes: u64, pub max_entry_bytes: u64, pub max_total_bytes: u64, } +impl Default for Limits { + fn default() -> Self { + Self { max_entries: 100_000, max_manifest_bytes: 64 << 20, max_entry_bytes: 256 << 20, max_total_bytes: 1 << 30 } + } +} + #[derive(Debug, thiserror::Error)] pub enum Error { #[error("invalid portable material: {0}")] Invalid(String), - #[error("unsupported portable material version {0}")] + #[error("unsupported .mcmat version {0} (expected 3). Republish from the original authoring project")] Version(u32), - #[error("portable material limit: {0}")] - Limit(String), #[error("portable material archive: {0}")] Archive(#[from] craft_zip::Error), } -/// Whether a bounded ZIP index contains a material manifest. This does not validate the manifest. +/// File sniffing also recognizes old manifests so the reader can give republishing guidance. pub fn is_material(bytes: &[u8], max_entries: u64) -> bool { - Archive::open(bytes, craft_zip::Limits { max_entries, max_total_bytes: 0 }) - .is_ok_and(|archive| !archive.has_duplicate_names() && archive.contains(MANIFEST_ENTRY)) + Archive::open(bytes, craft_zip::Limits { max_entries, max_total_bytes: 0 }).is_ok_and(|archive| archive.contains(MANIFEST_ENTRY)) } -/// Manifest and original archive, retaining one shared expanded-byte budget across reads. -pub struct Material<'a> { - pub manifest: Manifest, - archive: Archive<'a>, - max_entry_bytes: u64, +fn valid_resource(name: &str) -> bool { + name.len() <= 4096 + && name.starts_with("resources/") + && !name.contains(['\\', ':']) + && !name.chars().any(char::is_control) + && name.split('/').all(|part| !matches!(part, "" | "." | "..")) } -impl<'a> Material<'a> { - pub fn open(bytes: &'a [u8], limits: Limits) -> Result { +#[derive(Debug)] +pub struct Material { + pub source: Vec, + /// Keys are paths relative to source.sbs, including the resources/ prefix. + pub resources: BTreeMap>, +} + +impl Material { + pub fn open(bytes: &[u8], limits: Limits) -> Result { let mut archive = Archive::open(bytes, craft_zip::Limits { max_entries: limits.max_entries, max_total_bytes: limits.max_total_bytes })?; - if archive.has_duplicate_names() { - return Err(Error::Invalid("duplicate archive entries".into())); + if archive.has_duplicate_names() || archive.has_overlapping_files()? { + return Err(Error::Invalid("duplicate or overlapping archive entries".into())); } - let json = archive.read(MANIFEST_ENTRY, limits.max_manifest_bytes)?.ok_or_else(|| Error::Invalid("material.json is missing".into()))?; - let manifest: Manifest = serde_json::from_slice(&json).map_err(|e| Error::Invalid(format!("material.json: {e}")))?; - if manifest.format != FORMAT { - return Err(Error::Invalid(format!("format `{}`", manifest.format))); + let json = archive.read(MANIFEST_ENTRY, limits.max_manifest_bytes)?.ok_or_else(|| Error::Invalid("missing material.json".into()))?; + #[derive(Deserialize)] + struct Version { + version: u32, } - if !(limits.min_version..=limits.max_version).contains(&manifest.version) { - return Err(Error::Version(manifest.version)); + let marker: Version = serde_json::from_slice(&json).map_err(|err| Error::Invalid(format!("material.json: {err}")))?; + if marker.version != VERSION { + return Err(Error::Version(marker.version)); } - if manifest.maps.len() > limits.max_maps { - return Err(Error::Limit(format!("{} maps exceeds {}", manifest.maps.len(), limits.max_maps))); + #[derive(Deserialize)] + #[serde(deny_unknown_fields)] + struct Marker { + version: u32, } - Ok(Self { manifest, archive, max_entry_bytes: limits.max_entry_bytes }) - } - - /// Read one named member, charging it to the same budget as previous reads. - pub fn read(&mut self, name: &str) -> Result, Error> { - self.archive.read(name, self.max_entry_bytes)?.ok_or_else(|| Error::Invalid(format!("{name} is missing"))) - } - - pub fn read_source(&mut self) -> Result, Error> { - self.read(&self.manifest.source.clone()) + if !serde_json::from_slice::(&json).is_ok_and(|m| m.version == VERSION) { + return Err(Error::Invalid("material.json must contain only the version".into())); + } + let source = archive.read(SOURCE_ENTRY, limits.max_entry_bytes)?.ok_or_else(|| Error::Invalid("missing source.sbs".into()))?; + let mut resources = BTreeMap::new(); + let mut seen = BTreeSet::new(); + for index in 0..archive.len() { + let name = archive.entry_name(index)?.ok_or_else(|| Error::Invalid("invalid entry name".into()))?; + if name == SOURCE_ENTRY || name == MANIFEST_ENTRY { + continue; + } + if name.strip_suffix('/').is_some_and(|dir| dir == "resources" || valid_resource(dir)) { + continue; + } + if !valid_resource(&name) || !seen.insert(name.to_ascii_lowercase()) { + return Err(Error::Invalid(format!("invalid or ambiguous resource path `{name}`"))); + } + let data = archive.read(&name, limits.max_entry_bytes)?.ok_or_else(|| Error::Invalid(format!("missing `{name}`")))?; + resources.insert(name, data); + } + Ok(Self { source, resources }) } } -/// Write a manifest, its named maps, and the source. Duplicate member names are rejected. -pub fn write(manifest: &Manifest, maps: &[(String, Vec)], source: &[u8]) -> Result, Error> { - // Flattened fields must not serialize a second copy of a known key. - const KNOWN: &[&str] = &["format", "version", "name", "graph", "size", "maps", "parameters", "inputs", "presets", "source"]; - if manifest.extra.keys().any(|key| KNOWN.contains(&key.as_str())) - || manifest.maps.iter().any(|map| map.extra.contains_key("usage") || map.extra.contains_key("file")) - { - return Err(Error::Invalid("an extra field duplicates a known field".into())); - } - let json = serde_json::to_vec_pretty(manifest).map_err(|e| Error::Invalid(format!("material.json: {e}")))?; +/// Write one SBS document with its referenced source assets. +pub fn write<'a>(source: &[u8], resources: impl IntoIterator) -> Result, Error> { + let limits = Limits::default(); let mut writer = Writer::new(); - writer.add(MANIFEST_ENTRY, &json, Compression::Deflated)?; - for (name, bytes) in maps { - writer.add(name, bytes, Compression::Stored)?; + let mut seen = BTreeSet::new(); + let mut total = (source.len() as u64).saturating_add(MARKER.len() as u64); + if source.len() as u64 > limits.max_entry_bytes { + return Err(Error::Invalid("source.sbs exceeds the entry limit".into())); + } + writer.add(MANIFEST_ENTRY, MARKER, Compression::Deflated)?; + writer.add(SOURCE_ENTRY, source, Compression::Deflated)?; + for (name, bytes) in resources { + if !valid_resource(name) || !seen.insert(name.to_ascii_lowercase()) { + return Err(Error::Invalid(format!("invalid or ambiguous resource path `{name}`"))); + } + total = total.saturating_add(bytes.len() as u64); + if bytes.len() as u64 > limits.max_entry_bytes || total > limits.max_total_bytes || seen.len() as u64 + 2 > limits.max_entries { + return Err(Error::Invalid(format!("archive limits exceeded at `{name}`"))); + } + let compression = if name.ends_with(".svg") { Compression::Deflated } else { Compression::Stored }; + writer.add(name, bytes, compression)?; } - writer.add(&manifest.source, source, Compression::Stored)?; Ok(writer.finish()?) } @@ -135,82 +128,58 @@ pub fn write(manifest: &Manifest, maps: &[(String, Vec)], source: &[u8]) -> mod tests { use super::*; - fn limits() -> Limits { - Limits { - min_version: 1, - max_version: 2, - max_entries: 100, - max_maps: 4, - max_manifest_bytes: 4096, - max_entry_bytes: 100, - max_total_bytes: 8192, - } - } - - fn sample(version: u32) -> Manifest { - serde_json::from_value(serde_json::json!({ - "format": "craft-material", "version": version, "name": "Stone", "graph": "Stone", "size": [64, 64], - "maps": [{"usage": "baseColor", "file": "maps/base.png", "colorSpace": "srgb", "futureMap": true}], - "parameters": [{"id": "rough", "value": {"t": "float", "v": 0.25}, "futureParam": 42}], - "inputs": [{"identifier": "curvature", "kind": "gray"}], "presets": [], - "source": if version == 1 { "source.mcraft" } else { "source.sbs" }, "futureField": {"keep": true} - })) - .unwrap() - } - #[test] - fn both_protocol_versions_keep_unknown_fields_and_payloads() { - for version in [1, 2] { - let manifest = sample(version); - let bytes = write(&manifest, &[("maps/base.png".into(), vec![1, 2, 3])], b"source").unwrap(); - assert!(is_material(&bytes, 100)); - assert!(!is_material(&bytes, 1)); - let mut material = Material::open(&bytes, limits()).unwrap(); - assert_eq!(material.manifest, manifest); - assert_eq!(material.read("maps/base.png").unwrap(), [1, 2, 3]); - assert_eq!(material.read_source().unwrap(), b"source"); - let json = serde_json::to_value(&material.manifest).unwrap(); - assert_eq!(json["futureField"]["keep"], true); - assert_eq!(json["maps"][0]["futureMap"], true); - assert_eq!(json["parameters"][0]["futureParam"], 42); + fn procedural_source_and_assets_round_trip_under_one_budget() { + let bytes = write(b"", [("resources/oak.png", b"image".as_slice())]).unwrap(); + let pkg = Material::open(&bytes, Limits::default()).unwrap(); + assert_eq!(pkg.source, b""); + assert_eq!(pkg.resources["resources/oak.png"], b"image"); + let mut archive = Archive::open(&bytes, craft_zip::Limits { max_entries: 3, max_total_bytes: 128 }).unwrap(); + assert_eq!(archive.read(MANIFEST_ENTRY, 32).unwrap().unwrap(), MARKER); + assert_eq!(archive.is_stored(SOURCE_ENTRY), Some(false)); + assert_eq!(archive.is_stored("resources/oak.png"), Some(true)); + for limits in [ + Limits { max_entries: 2, ..Default::default() }, + Limits { max_entry_bytes: 2, ..Default::default() }, + Limits { max_total_bytes: 24, ..Default::default() }, + ] { + assert!(Material::open(&bytes, limits).is_err()); } } #[test] - fn limits_apply_to_the_manifest_maps_and_aggregate_reads() { - let manifest = sample(2); - let bytes = write(&manifest, &[("maps/base.png".into(), vec![1; 8])], &[2; 8]).unwrap(); - let mut unsupported = limits(); - unsupported.max_version = 1; - assert!(matches!(Material::open(&bytes, unsupported), Err(Error::Version(2)))); - let mut no_maps = limits(); - no_maps.max_maps = 0; - assert!(matches!(Material::open(&bytes, no_maps), Err(Error::Limit(_)))); - let mut tiny_entry = limits(); - tiny_entry.max_entry_bytes = 7; - assert!(Material::open(&bytes, tiny_entry).unwrap().read("maps/base.png").is_err()); - let json_len = serde_json::to_vec_pretty(&manifest).unwrap().len() as u64; - let mut aggregate = limits(); - aggregate.max_total_bytes = json_len + 15; - let mut material = Material::open(&bytes, aggregate).unwrap(); - assert_eq!(material.read("maps/base.png").unwrap().len(), 8); - assert!(material.read_source().is_err()); + fn rejects_previous_versions_before_requiring_the_v3_layout() { + for version in [0, 1, 2, 4] { + let mut zip = Writer::new(); + zip.add(MANIFEST_ENTRY, format!(r#"{{"format":"craft-material","version":{version}}}"#).as_bytes(), Compression::Deflated).unwrap(); + let bytes = zip.finish().unwrap(); + assert!(is_material(&bytes, 2)); + assert!(matches!(Material::open(&bytes, Limits::default()), Err(Error::Version(v)) if v == version)); + } } #[test] - fn refuses_extra_fields_that_duplicate_known_keys() { - let mut manifest = sample(2); - manifest.extra.insert("version".into(), 3.into()); - assert!(write(&manifest, &[], b"source").is_err()); - manifest.extra.remove("version"); - manifest.maps[0].extra.insert("file".into(), "other".into()); - assert!(write(&manifest, &[], b"source").is_err()); + fn v3_requires_source_and_a_version_only_marker() { + for (marker, source) in [(MARKER, None), (br#"{"version":3,"graph":"Other"}"#.as_slice(), Some(b"".as_slice()))] { + let mut zip = Writer::new(); + zip.add(MANIFEST_ENTRY, marker, Compression::Deflated).unwrap(); + if let Some(source) = source { + zip.add(SOURCE_ENTRY, source, Compression::Deflated).unwrap(); + } + assert!(Material::open(&zip.finish().unwrap(), Limits::default()).is_err()); + } } #[test] - fn refuses_duplicate_names_on_write() { - let manifest = sample(2); - let duplicate = [("maps/base.png".to_string(), vec![1]), ("maps/base.png".to_string(), vec![2])]; - assert!(write(&manifest, &duplicate, b"source").is_err()); + fn rejects_ambiguous_or_nonlocal_resource_members() { + for name in ["resources/../../x", "resources/a\\x", "/x", "maps/color.png", "resources/C:/x", "preview.png"] { + assert!(write(b"x", [(name, &b"x"[..])]).is_err()); + let mut zip = Writer::new(); + zip.add(MANIFEST_ENTRY, MARKER, Compression::Stored).unwrap(); + zip.add(SOURCE_ENTRY, b"x", Compression::Deflated).unwrap(); + zip.add(name, b"x", Compression::Stored).unwrap(); + assert!(Material::open(&zip.finish().unwrap(), Limits::default()).is_err()); + } + assert!(write(b"x", [("resources/A", &b"a"[..]), ("resources/a", &b"b"[..])]).is_err()); } } From 14176282fd02834652a34e81a34c19a393e8c9f4 Mon Sep 17 00:00:00 2001 From: FredApps <153173854+FredApps@users.noreply.github.com> Date: Wed, 18 Nov 2026 14:06:25 +0700 Subject: [PATCH 109/161] fix(dex): preserve decode completeness and fixture UTF-16 encoding (#2164) --- crates/dex/src/dex.rs | 45 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/crates/dex/src/dex.rs b/crates/dex/src/dex.rs index 253af049c..528ec334e 100644 --- a/crates/dex/src/dex.rs +++ b/crates/dex/src/dex.rs @@ -745,6 +745,51 @@ mod tests { use super::*; use crate::test_support::android as fx; + #[test] + fn reference_queries_preserve_incomplete_decode_across_methods() { + for incomplete in [false, true] { + let mut builder = fx::DexBuilder::default(); + let class = builder.ty("LTest;"); + let first = builder.method("LTest;", "first", "V", &[]); + let second = builder.method("LTest;", "second", "V", &[]); + let mut first_code = vec![0x0071, second as u16, 0]; + if incomplete { + first_code.push(0x0071); // truncated invoke-static + } + let code = |insns| Some(fx::CodeSpec { insns, ..Default::default() }); + builder.classes.push(fx::ClassSpec { + class, + direct: vec![ + fx::MethodSpec { method: first, access: 0, code: code(first_code) }, + fx::MethodSpec { method: second, access: 0, code: code(vec![0x000e]) }, + ], + ..Default::default() + }); + let data = builder.build(); + let dex = parse(&data).unwrap(); + let (found, truncated) = references(&dex, "LTest;", Some("second"), 10); + assert_eq!(found.len(), 1); + assert_eq!(found[0].caller, "LTest;->first()V"); + assert_eq!(found[0].target, "LTest;->second()V"); + assert_eq!(truncated, incomplete); + let (absent, truncated) = references(&dex, "LAbsent;", None, 10); + assert!(absent.is_empty()); + assert_eq!(truncated, incomplete); + } + } + + #[test] + fn fixture_strings_round_trip_utf16_surrogate_pairs() { + let mut builder = fx::DexBuilder::default(); + let text = "A\0é😀"; + let index = builder.string(text); + let data = builder.build(); + let dex = parse(&data).unwrap(); + assert_eq!(dex.string(index), text); + let offset = Reader::new(&data).u32_le(0x70).unwrap() as usize; + assert_eq!(&data[offset..offset + 13], &[5, 0x41, 0xc0, 0x80, 0xc3, 0xa9, 0xed, 0xa0, 0xbd, 0xed, 0xb8, 0x80, 0]); + } + #[test] fn parses_fixture() { let data = fx::notes_dex(); From b3960accde1b540b2fe9364f8b6b848284564601 Mon Sep 17 00:00:00 2001 From: FredApps Date: Wed, 18 Nov 2026 08:08:01 +0100 Subject: [PATCH 110/161] psd: reject invalid quilt bounds and discard unsupported templates --- crates/psd/src/smart.rs | 16 +++++++++++++++- crates/psd/src/smart/write.rs | 30 +++++++++++++++++++++++++++++- 2 files changed, 44 insertions(+), 2 deletions(-) diff --git a/crates/psd/src/smart.rs b/crates/psd/src/smart.rs index efa3cee2e..9a95c61f3 100644 --- a/crates/psd/src/smart.rs +++ b/crates/psd/src/smart.rs @@ -134,8 +134,11 @@ impl Warp { let rows = quilt_patches(&slice_y)?; if !slice_x.is_empty() || !slice_y.is_empty() { // The evaluator normalizes over bounds, not stored slice coordinates. - // Infer a missing rectangle only when both explicit axes define it. + // Infer an absent rectangle only when both explicit axes define it. if !(w.bounds[2] > w.bounds[0] && w.bounds[3] > w.bounds[1]) { + if d.get("bounds").is_some() { + return Err(PsdError::Unsupported("invalid explicit quilt bounds".into())); + } let (Some(&left), Some(&right), Some(&top), Some(&bottom)) = (slice_x.first(), slice_x.last(), slice_y.first(), slice_y.last()) else { return Err(PsdError::Unsupported("quilt divisions need bounds or both axes".into())); @@ -1074,6 +1077,17 @@ mod tests { .with("Rght", Value::Double(40.0)) .with("Btom", Value::Double(10.0)); assert!(Warp::try_from_descriptor(&equal.with("bounds", Value::Descriptor(bounds))).is_err()); + let wrong_type = quilt_descriptor(vec![0.0, 10.0, 20.0], vec![0.0, 10.0]).with("bounds", Value::Integer(1)); + assert!(Warp::try_from_descriptor(&wrong_type).is_err()); + let invalid_bounds = Descriptor::new("bounds") + .with("Left", Value::Double(0.0)) + .with("Top ", Value::Double(0.0)) + .with("Rght", Value::Double(40.0)) + .with("Btom", Value::Double(0.0)); + let explicit_invalid = quilt_descriptor(vec![0.0, 10.0, 20.0], vec![0.0, 10.0]).with("bounds", Value::Descriptor(invalid_bounds)); + assert!(Warp::try_from_descriptor(&explicit_invalid).is_err()); + let data = placed_layer_data("invalid bounds", [0.0; 8], |d| d.with("quiltWarp", Value::Descriptor(explicit_invalid))); + assert!(matches!(SmartObject::from_placed_layer_data(&data), Err(PsdError::Unsupported(_)))); let missing = quilt_descriptor(vec![0.0, 10.0, 20.0], vec![]); assert!(Warp::try_from_descriptor(&missing).is_err()); } diff --git a/crates/psd/src/smart/write.rs b/crates/psd/src/smart/write.rs index d409d726c..413b0f406 100644 --- a/crates/psd/src/smart/write.rs +++ b/crates/psd/src/smart/write.rs @@ -180,7 +180,8 @@ impl SmartObject { } set(&mut d, "warp", Value::Descriptor(warp.to_descriptor())); // The imported `quiltWarp` stays while it still says what the warp is. - let kept = t.get_object("quiltWarp").is_some_and(|q| Warp::from_descriptor(q).as_ref() == self.warp.as_ref()); + let kept = + t.get_object("quiltWarp").is_some_and(|q| Warp::try_from_descriptor(q).is_ok_and(|parsed| parsed.as_ref() == self.warp.as_ref())); if !kept { match quilt { Some(q) => set_before(&mut d, "quiltWarp", Value::Descriptor(q), "warp"), @@ -289,6 +290,33 @@ mod tests { ] } + #[test] + fn templates_drop_unsupported_quilts_but_retain_valid_none() { + let bounds = [0.0, 0.0, 20.0, 10.0]; + let unfl = |values| Value::UnitFloats { unit: *b"#Pxl", values }; + let unequal = + Descriptor::new("warp").with("warpStyle", Value::Enumerated { type_id: Id::new("warpStyle"), value: Id::new("warpCustom") }).with( + "customEnvelopeWarp", + Value::Descriptor( + Descriptor::new("customEnvelopeWarp").with("quiltSliceX", unfl(vec![0.0, 5.0, 20.0])).with("quiltSliceY", unfl(vec![0.0, 10.0])), + ), + ); + assert!(Warp::try_from_descriptor(&unequal).is_err()); + let valid_none = Warp::none(bounds).to_descriptor(); + assert!(matches!(Warp::try_from_descriptor(&valid_none), Ok(None))); + for (quilt, keep) in [(unequal, false), (valid_none, true)] { + let template = Descriptor::new("null").with("quiltWarp", Value::Descriptor(quilt.clone())); + let bytes = placed(None).to_placed_layer_data(&PlacedLayerOptions { + placed: "layer", + resolution: 72.0, + filter_fx: None, + template: Some(&template), + }); + let written = VersionedDescriptor::parse_prefix(&bytes[8..]).unwrap().0.descriptor; + assert_eq!(written.get_object("quiltWarp"), keep.then_some(&quilt)); + assert!(SmartObject::from_placed_layer_data(&bytes).unwrap().warp.is_none()); + } + } #[test] fn placed_layer_data_reads_back_and_passes_the_strict_check() { let filter = Descriptor::new("filterFX") From b0f25d641840bea1dda99c1e5e0c8db9221bcf28 Mon Sep 17 00:00:00 2001 From: chasse20 Date: Wed, 18 Nov 2026 02:15:10 -0500 Subject: [PATCH 111/161] Compress raw MCMAT source assets --- crates/mcmat/src/lib.rs | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/crates/mcmat/src/lib.rs b/crates/mcmat/src/lib.rs index 076fe636a..fe328cceb 100644 --- a/crates/mcmat/src/lib.rs +++ b/crates/mcmat/src/lib.rs @@ -118,7 +118,11 @@ pub fn write<'a>(source: &[u8], resources: impl IntoIterator limits.max_entry_bytes || total > limits.max_total_bytes || seen.len() as u64 + 2 > limits.max_entries { return Err(Error::Invalid(format!("archive limits exceeded at `{name}`"))); } - let compression = if name.ends_with(".svg") { Compression::Deflated } else { Compression::Stored }; + let extension = name.rsplit_once('.').map(|(_, ext)| ext.to_ascii_lowercase()); + let compression = match extension.as_deref() { + Some("png" | "jpg" | "jpeg" | "webp" | "gif" | "exr" | "woff" | "woff2") => Compression::Stored, + _ => Compression::Deflated, + }; writer.add(name, bytes, compression)?; } Ok(writer.finish()?) @@ -147,6 +151,14 @@ mod tests { } } + #[test] + fn compresses_unencoded_source_assets() { + let bytes = write(b"", [("resources/shape.svg", b"".as_slice()), ("resources/font.ttf", b"font".as_slice())]).unwrap(); + let mut archive = Archive::open(&bytes, craft_zip::Limits { max_entries: 4, max_total_bytes: 128 }).unwrap(); + assert_eq!(archive.is_stored("resources/shape.svg"), Some(false)); + assert_eq!(archive.is_stored("resources/font.ttf"), Some(false)); + } + #[test] fn rejects_previous_versions_before_requiring_the_v3_layout() { for version in [0, 1, 2, 4] { From c03ba68ab363bcf337a5f2c9548ead0b941b427f Mon Sep 17 00:00:00 2001 From: FredApps <153173854+FredApps@users.noreply.github.com> Date: Wed, 18 Nov 2026 14:36:07 +0700 Subject: [PATCH 112/161] fix(xfa): reject repeated leading BOM before parsing spans (#2176) --- crates/xfa/src/xml.rs | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/crates/xfa/src/xml.rs b/crates/xfa/src/xml.rs index 7afab7600..7d6dd8193 100644 --- a/crates/xfa/src/xml.rs +++ b/crates/xfa/src/xml.rs @@ -194,6 +194,10 @@ impl Document { // quick-xml excludes a leading UTF-8 BOM from its buffer positions. // Parse it explicitly so every stored range still addresses the original input. let input = text.strip_prefix('\u{feff}').unwrap_or(text); + // A second U+FEFF is content outside the root, not another encoding marker. + if input.starts_with('\u{feff}') { + return Err("multiple leading XML byte-order marks".to_string()); + } let source_offset = text.len() - input.len(); let mut events = opts.events(input.as_bytes()); let mut nodes: Vec = Vec::new(); @@ -435,6 +439,15 @@ mod tests { } } + #[test] + fn repeated_leading_bom_is_rejected_before_event_parsing() { + for prefix in ["\u{feff}\u{feff}", "\u{feff}\u{feff}\u{feff}"] { + for body in ["<é/>", "<é>text"] { + assert!(Document::parse(&format!("{prefix}{body}")).is_err()); + } + } + } + #[test] fn inherited_namespace_copies_obey_aggregate_byte_admission() { let uri = format!("urn:{}", "x".repeat(124)); From be556bedcbc3fcd63d3146fd7fcec9bc488fc7ae Mon Sep 17 00:00:00 2001 From: FredApps Date: Wed, 18 Nov 2026 08:43:10 +0100 Subject: [PATCH 113/161] psd: align quilt serialization and strict slice admission --- crates/psd/src/smart.rs | 77 ++++++++++++++++++++++++++++------- crates/psd/src/smart/write.rs | 71 ++++++++++++++++++++++++++++---- 2 files changed, 126 insertions(+), 22 deletions(-) diff --git a/crates/psd/src/smart.rs b/crates/psd/src/smart.rs index 9a95c61f3..7ed444f45 100644 --- a/crates/psd/src/smart.rs +++ b/crates/psd/src/smart.rs @@ -74,11 +74,17 @@ fn quad_from(t: [f64; 8]) -> [[f64; 2]; 4] { /// The patch boundaries of a custom envelope: a list of numbers, or (as Photoshop writes a /// `quiltWarp`) an object array whose body holds them under the same key. -fn quilt_slices(env: &Descriptor, key: &str) -> Vec { - match env.get(key) { - Some(Value::ObjectArray(o)) => o.body.get_numbers(key), - _ => env.get_numbers(key), - } +fn quilt_slices(env: &Descriptor, key: &str) -> Result> { + let Some(value) = env.get(key) else { return Ok(vec![]) }; + // Filtering nonnumbers would change the number and positions of patches. + // Only absence, not a malformed or empty present value, means one patch. + let value = if let Value::ObjectArray(o) = value { o.body.get(key) } else { Some(value) }; + let values = match value { + Some(Value::UnitFloats { values, .. }) => Some(values.clone()), + Some(Value::List(values)) => values.iter().map(Value::as_number).collect::>>(), + _ => None, + }; + values.filter(|values| !values.is_empty()).ok_or_else(|| PsdError::Unsupported("invalid quilt slice values".into())) } /// The public mesh has no slice coordinates, so only uniform divisions can be @@ -90,18 +96,23 @@ fn quilt_patches(slices: &[f64]) -> Result { if slices.len() < 2 || slices.iter().any(|v| !v.is_finite()) { return Err(PsdError::Unsupported("invalid quilt divisions".into())); } - let step = slices[1] - slices[0]; - if !step.is_finite() || step <= 0.0 { + let patches = slices.len() - 1; + let extent = slices[patches] - slices[0]; + if !extent.is_finite() || extent <= 0.0 { return Err(PsdError::Unsupported("invalid quilt divisions".into())); } - let tolerance = step.abs() * 1e-9; - if slices.windows(2).any(|pair| { - let width = pair[1] - pair[0]; - !width.is_finite() || width <= 0.0 || (width - step).abs() > tolerance - }) { + // Compare against the same endpoint interpolation the writer uses. At large + // coordinate offsets a uniform grid can have differently rounded adjacent widths. + let tolerance = (extent / patches as f64) * 1e-9; + if slices.windows(2).any(|pair| pair[1] <= pair[0]) + || slices.iter().enumerate().any(|(i, value)| { + let expected = slices[0] + extent * (i as f64 / patches as f64); + (value - expected).abs() > tolerance + }) + { return Err(PsdError::Unsupported("unequal quilt divisions".into())); } - Ok(slices.len() - 1) + Ok(patches) } impl Warp { /// Reads a `warp` descriptor; `None` for no warp (the identity, or a custom warp without a @@ -128,8 +139,8 @@ impl Warp { style, }; if let Some(env) = d.get_object("customEnvelopeWarp") { - let slice_x = quilt_slices(env, "quiltSliceX"); - let slice_y = quilt_slices(env, "quiltSliceY"); + let slice_x = quilt_slices(env, "quiltSliceX")?; + let slice_y = quilt_slices(env, "quiltSliceY")?; let cols = quilt_patches(&slice_x)?; let rows = quilt_patches(&slice_y)?; if !slice_x.is_empty() || !slice_y.is_empty() { @@ -1065,8 +1076,44 @@ mod tests { } assert_eq!(quilt_patches(&[]).unwrap(), 1); assert_eq!(quilt_patches(&[0.0, 10.0, 20.0]).unwrap(), 2); + assert_eq!(quilt_patches(&[1e16, 1e16 + 2.0, 1e16 + 6.0, 1e16 + 8.0]).unwrap(), 3); + assert!(quilt_patches(&[1e16, 1e16 + 4.0, 1e16 + 6.0, 1e16 + 8.0]).is_err()); } + #[test] + fn quilt_slice_values_are_strict_in_lists_and_object_arrays() { + let list = Value::List(vec![Value::Integer(0), Value::Double(10.0), Value::UnitFloat { unit: *b"#Pxl", value: 20.0 }]); + let unfl = Value::UnitFloats { unit: *b"#Pxl", values: vec![0.0, 10.0, 20.0] }; + let bad_list = Value::List(vec![Value::Double(0.0), Value::Text(UnicodeString::new("bad")), Value::Double(10.0), Value::Double(20.0)]); + for wrapped in [false, true] { + for (value, valid) in [ + (list.clone(), true), + (unfl.clone(), true), + (bad_list.clone(), false), + (Value::List(vec![]), false), + (Value::UnitFloats { unit: *b"#Pxl", values: vec![] }, false), + (Value::Integer(1), false), + ] { + let value = if wrapped { + Value::ObjectArray(ObjectArray { prefix: 3, body: Descriptor::new("UntF").with("quiltSliceX", value) }) + } else { + value + }; + let mut d = quilt_descriptor(vec![0.0, 10.0, 20.0], vec![0.0, 10.0]); + let Value::Descriptor(env) = &mut d.items.iter_mut().find(|(key, _)| key.is("customEnvelopeWarp")).unwrap().1 else { + panic!("envelope") + }; + env.items.iter_mut().find(|(key, _)| key.is("quiltSliceX")).unwrap().1 = value; + assert_eq!(Warp::try_from_descriptor(&d).is_ok(), valid); + let bytes = placed_layer_data("slices", [0.0; 8], |parent| parent.with("quiltWarp", Value::Descriptor(d))); + assert_eq!(SmartObject::from_placed_layer_data(&bytes).is_ok(), valid); + } + } + let absent = Descriptor::new("customEnvelopeWarp"); + assert!(quilt_slices(&absent, "quiltSliceX").unwrap().is_empty()); + let missing_body = absent.with("quiltSliceX", Value::ObjectArray(ObjectArray { prefix: 0, body: Descriptor::new("UntF") })); + assert!(quilt_slices(&missing_body, "quiltSliceX").is_err()); + } #[test] fn quilt_endpoints_define_or_match_the_normalization_bounds() { let equal = quilt_descriptor(vec![0.0, 10.0, 20.0], vec![0.0, 10.0]); diff --git a/crates/psd/src/smart/write.rs b/crates/psd/src/smart/write.rs index 413b0f406..0c9eb4588 100644 --- a/crates/psd/src/smart/write.rs +++ b/crates/psd/src/smart/write.rs @@ -78,7 +78,10 @@ impl Warp { /// Style, bend, distortions, axis, bounds and order: the keys a `warp` and a `quiltWarp` /// share, in Photoshop's order. fn head(&self, class: &str) -> Descriptor { - let [l, t, r, b] = self.bounds; + self.head_with_bounds(class, self.bounds) + } + + fn head_with_bounds(&self, class: &str, [l, t, r, b]: [f64; 4]) -> Descriptor { let bounds = Descriptor::new("classFloatRect") .with("Top ", Value::Double(t)) .with("Left", Value::Double(l)) @@ -119,15 +122,20 @@ impl Warp { /// The `quiltWarp` descriptor of a split warp, in Photoshop's layout: the control points, /// `deformNumCols` × `deformNumRows` of them, and the patch boundaries (`quiltSliceX` / - /// `quiltSliceY`) evenly across the warp's bounds. `None` for any other warp. + /// `quiltSliceY`) evenly across the warp's bounds. Invalid axes use 0..1 because this standalone + /// method has no content size; the placed-layer writer supplies that size when available. + /// `None` for any other warp. pub fn quilt_descriptor(&self) -> Option { if !self.is_split() { return None; } - let [l, t, r, b] = self.bounds; + // The header and the divisions must describe the same rectangle, including + // when a caller leaves bounds at their default or supplies an invalid axis. + let axis = |lo: f64, hi: f64| if (hi - lo).is_finite() && hi > lo { (lo, hi) } else { (0.0, 1.0) }; + let (l, r) = axis(self.bounds[0], self.bounds[2]); + let (t, b) = axis(self.bounds[1], self.bounds[3]); let slices = |key: &str, n: usize, lo: f64, hi: f64| { - let (lo, hi) = if (hi - lo).is_finite() && hi > lo { (lo, hi) } else { (0.0, 1.0) }; - let values: Vec = (0..=n).map(|i| lo + (hi - lo) * i as f64 / n as f64).collect(); + let values: Vec = (0..=n).map(|i| lo + (hi - lo) * (i as f64 / n as f64)).collect(); let prefix = values.len() as u32; Value::ObjectArray(ObjectArray { prefix, body: Descriptor::new("UntF").with(key, Value::UnitFloats { unit: *b"#Pxl", values }) }) }; @@ -137,7 +145,7 @@ impl Warp { .with("meshPoints", self.mesh_points()); let count = |n: usize| Value::Integer(i32::try_from(n.saturating_mul(3).saturating_add(1)).unwrap_or(i32::MAX)); Some( - self.head("quiltWarp") + self.head_with_bounds("quiltWarp", [l, t, r, b]) .with("deformNumRows", count(self.rows)) .with("deformNumCols", count(self.cols)) .with("customEnvelopeWarp", Value::Descriptor(envelope)), @@ -159,7 +167,13 @@ impl SmartObject { /// The warp written, over the content (or `[0, 0, 0, 0]` without a size) when there is none. fn warp_or_none(&self) -> Warp { let [w, h] = self.size.unwrap_or([0.0; 2]); - self.warp.clone().unwrap_or_else(|| Warp::none([0.0, 0.0, w, h])) + let mut warp = self.warp.clone().unwrap_or_else(|| Warp::none([0.0, 0.0, w, h])); + // Match Warp::apply's whole-rectangle fallback before encoding explicit + // quilt slices. Otherwise default bounds would change the mapping on readback. + if warp.is_split() && !(warp.bounds[2] > warp.bounds[0] && warp.bounds[3] > warp.bounds[1]) { + warp.bounds = [0.0, 0.0, w, h]; + } + warp } /// The data of a `SoLd` block (`soLD`, version 4, a version-16 descriptor, padded to 4). @@ -290,6 +304,49 @@ mod tests { ] } + #[test] + fn quilt_writer_uses_one_rectangle_for_header_slices_and_roundtrip() { + for (bounds, expected) in [ + ([0.0; 4], [0.0, 0.0, 1.0, 1.0]), + ([2.0, 3.0, 22.0, 13.0], [2.0, 3.0, 22.0, 13.0]), + ([2.0, 3.0, 22.0, 3.0], [2.0, 0.0, 22.0, 1.0]), + ([f64::NAN, 3.0, 22.0, 13.0], [0.0, 3.0, 1.0, 13.0]), + ([-f64::MAX, 0.0, f64::MAX, 1.0], [0.0, 0.0, 1.0, 1.0]), + ([0.0, 0.0, 1e308, 10.0], [0.0, 0.0, 1e308, 10.0]), + ] { + let warp = Warp { style: "warpCustom".into(), bounds, cols: 2, rows: 1, mesh: grid(2, 1, 20.0, 10.0), ..Warp::default() }; + let descriptor = warp.quilt_descriptor().unwrap(); + let back = Warp::try_from_descriptor(&descriptor).unwrap().unwrap(); + assert_eq!(back.bounds, expected); + assert_eq!((back.cols, back.rows, &back.mesh), (2, 1, &warp.mesh)); + assert_eq!(back.quilt_descriptor().unwrap(), descriptor); + } + let high_offset = Warp { + style: "warpCustom".into(), + bounds: [1e16, 0.0, 1e16 + 8.0, 10.0], + cols: 3, + rows: 1, + mesh: grid(3, 1, 20.0, 10.0), + ..Warp::default() + }; + assert_eq!(Warp::try_from_descriptor(&high_offset.quilt_descriptor().unwrap()).unwrap(), Some(high_offset)); + // At the placed-layer boundary content size is available: preserve the + // evaluator's fallback mapping, not merely the ability to parse the output. + for bounds in [[0.0; 4], [2.0, 3.0, 22.0, 3.0], [f64::NAN, 0.0, 20.0, 10.0]] { + let warp = Warp { style: "warpCustom".into(), bounds, cols: 2, rows: 1, mesh: grid(2, 1, 20.0, 10.0), ..Warp::default() }; + let so = SmartObject { size: Some([20.0, 10.0]), ..placed(Some(warp.clone())) }; + let bytes = so.to_placed_layer_data(&PlacedLayerOptions::default()); + let back = SmartObject::from_placed_layer_data(&bytes).unwrap(); + let parsed = back.warp.as_ref().unwrap(); + assert_eq!(parsed.bounds, [0.0, 0.0, 20.0, 10.0]); + for p in [[0.0, 0.0], [5.0, 4.0], [10.0, 5.0], [20.0, 10.0]] { + assert_eq!(parsed.apply(p, 20.0, 10.0), warp.apply(p, 20.0, 10.0)); + } + let template = VersionedDescriptor::parse_prefix(&bytes[8..]).unwrap().0.descriptor; + let rewritten = back.to_placed_layer_data(&PlacedLayerOptions { template: Some(&template), ..Default::default() }); + assert_eq!(SmartObject::from_placed_layer_data(&rewritten).unwrap(), back); + } + } #[test] fn templates_drop_unsupported_quilts_but_retain_valid_none() { let bounds = [0.0, 0.0, 20.0, 10.0]; From e9ec4813311264ee1857079c2edd10a70e2389a6 Mon Sep 17 00:00:00 2001 From: FredApps Date: Wed, 18 Nov 2026 09:11:45 +0100 Subject: [PATCH 114/161] psd: preserve canonical quilt grids across numeric extremes --- crates/psd/src/smart.rs | 54 +++++++++++++++++++++++++++++------ crates/psd/src/smart/write.rs | 53 ++++++++++++++++++++++++++++++---- 2 files changed, 94 insertions(+), 13 deletions(-) diff --git a/crates/psd/src/smart.rs b/crates/psd/src/smart.rs index 7ed444f45..0cd5ac433 100644 --- a/crates/psd/src/smart.rs +++ b/crates/psd/src/smart.rs @@ -67,6 +67,10 @@ fn finite(value: f64) -> f64 { if value.is_finite() { value } else { 0.0 } } +/// A usable normalization rectangle needs finite positive extents on both axes. +fn valid_bounds([l, t, r, b]: [f64; 4]) -> bool { + [l, t, r, b].iter().all(|v| v.is_finite()) && (r - l).is_finite() && r > l && (b - t).is_finite() && b > t +} /// The corner quad (top-left, top-right, bottom-right, bottom-left) from x0 y0 … x3 y3. fn quad_from(t: [f64; 8]) -> [[f64; 2]; 4] { [[t[0], t[1]], [t[2], t[3]], [t[4], t[5]], [t[6], t[7]]] @@ -102,12 +106,14 @@ fn quilt_patches(slices: &[f64]) -> Result { return Err(PsdError::Unsupported("invalid quilt divisions".into())); } // Compare against the same endpoint interpolation the writer uses. At large - // coordinate offsets a uniform grid can have differently rounded adjacent widths. + // coordinate offsets a uniform grid can have differently rounded adjacent widths, + // including coincident boundaries. Only canonical rounded positions are accepted; + // the model keeps the patch count and evaluates in its uniform parameter domain. let tolerance = (extent / patches as f64) * 1e-9; - if slices.windows(2).any(|pair| pair[1] <= pair[0]) - || slices.iter().enumerate().any(|(i, value)| { + if slices.windows(2).any(|pair| pair[1] < pair[0]) + || slices.iter().enumerate().take(patches).skip(1).any(|(i, value)| { let expected = slices[0] + extent * (i as f64 / patches as f64); - (value - expected).abs() > tolerance + !expected.is_finite() || (value - expected).abs() > tolerance }) { return Err(PsdError::Unsupported("unequal quilt divisions".into())); @@ -122,7 +128,10 @@ impl Warp { } /// Reads a warp, rejecting unequal or invalid quilt divisions rather than - /// interpreting them as equal-sized patches. Nonfinite geometry becomes zero. + /// interpreting them as equal-sized patches. Canonical uniform boundaries may coincide + /// after floating-point rounding; their patch count is retained. This is the internal + /// uniform-model contract, not a guarantee of other readers' handling of coincident slices. + /// Nonfinite geometry becomes zero. pub fn try_from_descriptor(d: &Descriptor) -> Result> { let style = d.get_enum("warpStyle").unwrap_or_else(|| "warpNone".to_string()); let side = |b: &Descriptor, k: &str| finite(b.get_number(k).unwrap_or(0.0)); @@ -146,7 +155,7 @@ impl Warp { if !slice_x.is_empty() || !slice_y.is_empty() { // The evaluator normalizes over bounds, not stored slice coordinates. // Infer an absent rectangle only when both explicit axes define it. - if !(w.bounds[2] > w.bounds[0] && w.bounds[3] > w.bounds[1]) { + if !valid_bounds(w.bounds) { if d.get("bounds").is_some() { return Err(PsdError::Unsupported("invalid explicit quilt bounds".into())); } @@ -155,6 +164,9 @@ impl Warp { return Err(PsdError::Unsupported("quilt divisions need bounds or both axes".into())); }; w.bounds = [left, top, right, bottom]; + if !valid_bounds(w.bounds) { + return Err(PsdError::Unsupported("invalid inferred quilt bounds".into())); + } } for (slices, start, end) in [(&slice_x, w.bounds[0], w.bounds[2]), (&slice_y, w.bounds[1], w.bounds[3])] { if let (Some(&first), Some(&last)) = (slices.first(), slices.last()) { @@ -187,7 +199,7 @@ impl Warp { /// The warp of a content point (content pixels → content pixels). pub fn apply(&self, p: [f64; 2], w: f64, h: f64) -> [f64; 2] { - let [l, t, r, b] = if self.bounds[2] > self.bounds[0] && self.bounds[3] > self.bounds[1] { self.bounds } else { [0.0, 0.0, w, h] }; + let [l, t, r, b] = if valid_bounds(self.bounds) { self.bounds } else { [0.0, 0.0, w, h] }; let (bw, bh) = (r - l, b - t); let (u, v) = ((p[0] - l) / bw, (p[1] - t) / bh); if self.style == "warpCustom" { @@ -1078,6 +1090,14 @@ mod tests { assert_eq!(quilt_patches(&[0.0, 10.0, 20.0]).unwrap(), 2); assert_eq!(quilt_patches(&[1e16, 1e16 + 2.0, 1e16 + 6.0, 1e16 + 8.0]).unwrap(), 3); assert!(quilt_patches(&[1e16, 1e16 + 4.0, 1e16 + 6.0, 1e16 + 8.0]).is_err()); + let lo = 3.0 * 2.0_f64.powi(970); + assert!((f64::MAX - lo).is_finite()); + assert!((lo + (f64::MAX - lo)).is_infinite()); + assert_eq!(quilt_patches(&[lo, f64::MAX]).unwrap(), 1); + assert_eq!(quilt_patches(&[1e16, 1e16, 1e16, 1e16 + 2.0, 1e16 + 2.0]).unwrap(), 4); + for invalid in [vec![0.0, 0.0, 20.0], vec![0.0, 10.0, 10.0, 20.0], vec![1e16, 1e16 + 2.0, 1e16, 1e16 + 2.0]] { + assert!(quilt_patches(&invalid).is_err()); + } } #[test] @@ -1135,8 +1155,26 @@ mod tests { assert!(Warp::try_from_descriptor(&explicit_invalid).is_err()); let data = placed_layer_data("invalid bounds", [0.0; 8], |d| d.with("quiltWarp", Value::Descriptor(explicit_invalid))); assert!(matches!(SmartObject::from_placed_layer_data(&data), Err(PsdError::Unsupported(_)))); - let missing = quilt_descriptor(vec![0.0, 10.0, 20.0], vec![]); + let mut missing = quilt_descriptor(vec![0.0, 10.0, 20.0], vec![0.0, 10.0]); + let Value::Descriptor(env) = &mut missing.items.iter_mut().find(|(key, _)| key.is("customEnvelopeWarp")).unwrap().1 else { + panic!("envelope") + }; + env.items.retain(|(key, _)| !key.is("quiltSliceY")); assert!(Warp::try_from_descriptor(&missing).is_err()); + let bounds = |top, bottom| { + Descriptor::new("bounds") + .with("Left", Value::Double(0.0)) + .with("Top ", Value::Double(top)) + .with("Rght", Value::Double(20.0)) + .with("Btom", Value::Double(bottom)) + }; + let present = missing.clone().with("bounds", Value::Descriptor(bounds(0.0, 10.0))); + let parsed = Warp::try_from_descriptor(&present).unwrap().unwrap(); + assert_eq!((parsed.cols, parsed.rows), (2, 1)); + assert_eq!(parsed.bounds, [0.0, 0.0, 20.0, 10.0]); + // Even the absent slice axis needs a finite normalization extent. + let overflow = missing.with("bounds", Value::Descriptor(bounds(-f64::MAX, f64::MAX))); + assert!(Warp::try_from_descriptor(&overflow).is_err()); } #[test] fn finite_warp_views_preserve_raw_descriptor_values() { diff --git a/crates/psd/src/smart/write.rs b/crates/psd/src/smart/write.rs index 0c9eb4588..f1edf82df 100644 --- a/crates/psd/src/smart/write.rs +++ b/crates/psd/src/smart/write.rs @@ -3,7 +3,7 @@ //! read back. Came from PhotoCraft's `crates/io/src/smart_map/placed.rs`, on this crate's //! placement types: the caller works out the quad and the warp from its own geometry. -use super::{SmartObject, Warp}; +use super::{SmartObject, Warp, valid_bounds}; use crate::descriptor::{Descriptor, Id, ObjectArray, UnicodeString, Value, VersionedDescriptor}; use crate::io::WriteExt; @@ -124,7 +124,9 @@ impl Warp { /// `deformNumCols` × `deformNumRows` of them, and the patch boundaries (`quiltSliceX` / /// `quiltSliceY`) evenly across the warp's bounds. Invalid axes use 0..1 because this standalone /// method has no content size; the placed-layer writer supplies that size when available. - /// `None` for any other warp. + /// Canonical boundaries may coincide after rounding at large offsets. The shared reader + /// preserves their uniform-model parameterization; external reader interoperability for + /// coincident slices is not guaranteed. `None` for any other warp. pub fn quilt_descriptor(&self) -> Option { if !self.is_split() { return None; @@ -135,7 +137,17 @@ impl Warp { let (l, r) = axis(self.bounds[0], self.bounds[2]); let (t, b) = axis(self.bounds[1], self.bounds[3]); let slices = |key: &str, n: usize, lo: f64, hi: f64| { - let values: Vec = (0..=n).map(|i| lo + (hi - lo) * (i as f64 / n as f64)).collect(); + let values: Vec = (0..=n) + .map(|i| { + if i == 0 { + lo + } else if i == n { + hi + } else { + lo + (hi - lo) * (i as f64 / n as f64) + } + }) + .collect(); let prefix = values.len() as u32; Value::ObjectArray(ObjectArray { prefix, body: Descriptor::new("UntF").with(key, Value::UnitFloats { unit: *b"#Pxl", values }) }) }; @@ -170,7 +182,7 @@ impl SmartObject { let mut warp = self.warp.clone().unwrap_or_else(|| Warp::none([0.0, 0.0, w, h])); // Match Warp::apply's whole-rectangle fallback before encoding explicit // quilt slices. Otherwise default bounds would change the mapping on readback. - if warp.is_split() && !(warp.bounds[2] > warp.bounds[0] && warp.bounds[3] > warp.bounds[1]) { + if warp.is_split() && !valid_bounds(warp.bounds) { warp.bounds = [0.0, 0.0, w, h]; } warp @@ -313,6 +325,7 @@ mod tests { ([f64::NAN, 3.0, 22.0, 13.0], [0.0, 3.0, 1.0, 13.0]), ([-f64::MAX, 0.0, f64::MAX, 1.0], [0.0, 0.0, 1.0, 1.0]), ([0.0, 0.0, 1e308, 10.0], [0.0, 0.0, 1e308, 10.0]), + ([3.0 * 2.0_f64.powi(970), 0.0, f64::MAX, 10.0], [3.0 * 2.0_f64.powi(970), 0.0, f64::MAX, 10.0]), ] { let warp = Warp { style: "warpCustom".into(), bounds, cols: 2, rows: 1, mesh: grid(2, 1, 20.0, 10.0), ..Warp::default() }; let descriptor = warp.quilt_descriptor().unwrap(); @@ -332,7 +345,7 @@ mod tests { assert_eq!(Warp::try_from_descriptor(&high_offset.quilt_descriptor().unwrap()).unwrap(), Some(high_offset)); // At the placed-layer boundary content size is available: preserve the // evaluator's fallback mapping, not merely the ability to parse the output. - for bounds in [[0.0; 4], [2.0, 3.0, 22.0, 3.0], [f64::NAN, 0.0, 20.0, 10.0]] { + for bounds in [[0.0; 4], [2.0, 3.0, 22.0, 3.0], [f64::NAN, 0.0, 20.0, 10.0], [-f64::MAX, 0.0, f64::MAX, 10.0]] { let warp = Warp { style: "warpCustom".into(), bounds, cols: 2, rows: 1, mesh: grid(2, 1, 20.0, 10.0), ..Warp::default() }; let so = SmartObject { size: Some([20.0, 10.0]), ..placed(Some(warp.clone())) }; let bytes = so.to_placed_layer_data(&PlacedLayerOptions::default()); @@ -348,6 +361,36 @@ mod tests { } } #[test] + fn canonically_collapsed_boundaries_preserve_both_axes_mapping() { + let lo = 1e16; + let hi = lo + 2.0; + for (cols, rows, bounds, axis) in + [(4, 1, [lo, 0.0, hi, 10.0], "quiltSliceX"), (1, 4, [0.0, lo, 20.0, hi], "quiltSliceY"), (4, 4, [lo, lo, hi, hi], "quiltSliceX")] + { + let warp = Warp { style: "warpCustom".into(), bounds, cols, rows, mesh: grid(cols, rows, 20.0, 10.0), ..Warp::default() }; + let descriptor = warp.quilt_descriptor().unwrap(); + let env = descriptor.get_object("customEnvelopeWarp").unwrap(); + let Value::ObjectArray(slices) = env.get(axis).unwrap() else { panic!("slices") }; + let values = slices.body.get_numbers(axis); + assert_eq!(values, vec![lo, lo, lo, hi, hi]); + // The first, middle and last intervals all have rounded coincident endpoints. + assert_eq!(values[0], values[1]); + assert_eq!(values[1], values[2]); + assert_eq!(values[3], values[4]); + let so = placed(Some(warp.clone())); + let bytes = so.to_placed_layer_data(&PlacedLayerOptions::default()); + let back = SmartObject::from_placed_layer_data(&bytes).unwrap(); + assert_eq!(back.warp, Some(warp.clone())); + let parsed = back.warp.as_ref().unwrap(); + for p in [[bounds[0], bounds[1]], [bounds[2], bounds[1]], [bounds[0], bounds[3]], [bounds[2], bounds[3]]] { + assert_eq!(parsed.apply(p, 20.0, 10.0), warp.apply(p, 20.0, 10.0)); + } + let template = VersionedDescriptor::parse_prefix(&bytes[8..]).unwrap().0.descriptor; + let rewritten = back.to_placed_layer_data(&PlacedLayerOptions { template: Some(&template), ..Default::default() }); + assert_eq!(SmartObject::from_placed_layer_data(&rewritten).unwrap(), back); + } + } + #[test] fn templates_drop_unsupported_quilts_but_retain_valid_none() { let bounds = [0.0, 0.0, 20.0, 10.0]; let unfl = |values| Value::UnitFloats { unit: *b"#Pxl", values }; From c9862964df7f73ad37706539a06e346fc4865ca3 Mon Sep 17 00:00:00 2001 From: FredApps Date: Wed, 18 Nov 2026 09:38:28 +0100 Subject: [PATCH 115/161] psd: require complete numeric explicit quilt bounds --- crates/psd/src/smart.rs | 42 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/crates/psd/src/smart.rs b/crates/psd/src/smart.rs index 0cd5ac433..32cec043c 100644 --- a/crates/psd/src/smart.rs +++ b/crates/psd/src/smart.rs @@ -153,6 +153,13 @@ impl Warp { let cols = quilt_patches(&slice_x)?; let rows = quilt_patches(&slice_y)?; if !slice_x.is_empty() || !slice_y.is_empty() { + // Defaulting a missing or nonnumeric side to zero would invent an + // explicit rectangle. Numeric nonfinite sides still follow finite(). + if d.get("bounds").is_some() + && !d.get_object("bounds").is_some_and(|b| ["Left", "Top ", "Rght", "Btom"].iter().all(|key| b.get_number(key).is_some())) + { + return Err(PsdError::Unsupported("incomplete explicit quilt bounds".into())); + } // The evaluator normalizes over bounds, not stored slice coordinates. // Infer an absent rectangle only when both explicit axes define it. if !valid_bounds(w.bounds) { @@ -1177,6 +1184,41 @@ mod tests { assert!(Warp::try_from_descriptor(&overflow).is_err()); } #[test] + fn explicit_quilt_bounds_require_every_numeric_side() { + let sides = [("Left", 0.0), ("Top ", 0.0), ("Rght", 20.0), ("Btom", 10.0)]; + for omit_y in [false, true] { + let mut quilt = quilt_descriptor(vec![0.0, 10.0, 20.0], vec![0.0, 10.0]); + if omit_y { + let Value::Descriptor(env) = &mut quilt.items.iter_mut().find(|(key, _)| key.is("customEnvelopeWarp")).unwrap().1 else { + panic!("envelope") + }; + env.items.retain(|(key, _)| !key.is("quiltSliceY")); + } + let complete = sides.iter().fold(Descriptor::new("bounds"), |d, (key, value)| d.with(key, Value::Double(*value))); + assert!(Warp::try_from_descriptor(&quilt.clone().with("bounds", Value::Descriptor(complete))).unwrap().is_some()); + for (side, _) in sides { + for missing in [true, false] { + let bounds = sides.iter().fold(Descriptor::new("bounds"), |d, (key, value)| { + if *key != side { + d.with(key, Value::Double(*value)) + } else if missing { + d + } else { + d.with(key, Value::Text(UnicodeString::new("not a number"))) + } + }); + let invalid = quilt.clone().with("bounds", Value::Descriptor(bounds)); + assert!( + matches!(Warp::try_from_descriptor(&invalid), Err(PsdError::Unsupported(_))), + "{side}, missing={missing}, omit_y={omit_y}" + ); + let bytes = placed_layer_data("incomplete", [0.0; 8], |d| d.with("quiltWarp", Value::Descriptor(invalid))); + assert!(matches!(SmartObject::from_placed_layer_data(&bytes), Err(PsdError::Unsupported(_)))); + } + } + } + } + #[test] fn finite_warp_views_preserve_raw_descriptor_values() { let mut warp = quilt_descriptor(vec![0.0, 10.0, 20.0], vec![0.0, 10.0]); let bounds = Descriptor::new("bounds") From d78a99a0a79dcec0ee9cdc49a415c7fd38607454 Mon Sep 17 00:00:00 2001 From: FredApps Date: Wed, 18 Nov 2026 10:10:41 +0100 Subject: [PATCH 116/161] psd: validate explicit bounds in unsliced quilt context --- crates/psd/src/smart.rs | 121 +++++++++++++++++++++++++++++++--- crates/psd/src/smart/write.rs | 8 ++- 2 files changed, 117 insertions(+), 12 deletions(-) diff --git a/crates/psd/src/smart.rs b/crates/psd/src/smart.rs index 32cec043c..8cc140490 100644 --- a/crates/psd/src/smart.rs +++ b/crates/psd/src/smart.rs @@ -71,6 +71,20 @@ fn finite(value: f64) -> f64 { fn valid_bounds([l, t, r, b]: [f64; 4]) -> bool { [l, t, r, b].iter().all(|v| v.is_finite()) && (r - l).is_finite() && r > l && (b - t).is_finite() && b > t } +/// Explicit quilt rectangles must be complete even without subdivision keys. +fn validate_quilt_bounds(d: &Descriptor) -> Result<()> { + if d.get("bounds").is_none() { + return Ok(()); + } + let bounds = d + .get_object("bounds") + .and_then(|b| Some([b.get_number("Left")?, b.get_number("Top ")?, b.get_number("Rght")?, b.get_number("Btom")?])) + .ok_or_else(|| PsdError::Unsupported("incomplete explicit quilt bounds".into()))?; + if !valid_bounds(bounds.map(finite)) { + return Err(PsdError::Unsupported("invalid explicit quilt bounds".into())); + } + Ok(()) +} /// The corner quad (top-left, top-right, bottom-right, bottom-left) from x0 y0 … x3 y3. fn quad_from(t: [f64; 8]) -> [[f64; 2]; 4] { [[t[0], t[1]], [t[2], t[3]], [t[4], t[5]], [t[6], t[7]]] @@ -133,6 +147,14 @@ impl Warp { /// uniform-model contract, not a guarantee of other readers' handling of coincident slices. /// Nonfinite geometry becomes zero. pub fn try_from_descriptor(d: &Descriptor) -> Result> { + Self::try_from_descriptor_kind(d, d.class_id.is("quiltWarp")) + } + + // The containing SoLd key is authoritative even if an embedded class differs. + fn try_from_descriptor_kind(d: &Descriptor, quilt: bool) -> Result> { + if quilt { + validate_quilt_bounds(d)?; + } let style = d.get_enum("warpStyle").unwrap_or_else(|| "warpNone".to_string()); let side = |b: &Descriptor, k: &str| finite(b.get_number(k).unwrap_or(0.0)); let bounds = d.get_object("bounds").map_or([0.0; 4], |b| [side(b, "Left"), side(b, "Top "), side(b, "Rght"), side(b, "Btom")]); @@ -153,13 +175,8 @@ impl Warp { let cols = quilt_patches(&slice_x)?; let rows = quilt_patches(&slice_y)?; if !slice_x.is_empty() || !slice_y.is_empty() { - // Defaulting a missing or nonnumeric side to zero would invent an - // explicit rectangle. Numeric nonfinite sides still follow finite(). - if d.get("bounds").is_some() - && !d.get_object("bounds").is_some_and(|b| ["Left", "Top ", "Rght", "Btom"].iter().all(|key| b.get_number(key).is_some())) - { - return Err(PsdError::Unsupported("incomplete explicit quilt bounds".into())); - } + validate_quilt_bounds(d)?; + // The evaluator normalizes over bounds, not stored slice coordinates. // Infer an absent rectangle only when both explicit axes define it. if !valid_bounds(w.bounds) { @@ -366,9 +383,9 @@ impl SmartObject { let t = eight(&d, "nonAffineTransform").unwrap_or(t); let size = d.get_object("Sz ").and_then(|s| Some([finite(s.get_number("Wdth")?), finite(s.get_number("Hght")?)])); // A split warp is in `quiltWarp`, with `warp` holding none. - let mut warp = d.get_object("quiltWarp").map(Warp::try_from_descriptor).transpose()?.flatten(); + let mut warp = d.get_object("quiltWarp").map(|q| Warp::try_from_descriptor_kind(q, true)).transpose()?.flatten(); if warp.is_none() { - warp = d.get_object("warp").map(Warp::try_from_descriptor).transpose()?.flatten(); + warp = d.get_object("warp").map(|w| Warp::try_from_descriptor_kind(w, false)).transpose()?.flatten(); } Ok(SmartObject { uuid, quad: quad_from(t), size, warp }) } @@ -1184,6 +1201,92 @@ mod tests { assert!(Warp::try_from_descriptor(&overflow).is_err()); } #[test] + fn quilt_bounds_policy_covers_every_slice_axis_and_descriptor_context() { + let rect = |l, t, r, b| { + Value::Descriptor( + Descriptor::new("bounds") + .with("Left", Value::Double(l)) + .with("Top ", Value::Double(t)) + .with("Rght", Value::Double(r)) + .with("Btom", Value::Double(b)), + ) + }; + let cases = [ + (None, true), + (Some(rect(0.0, 0.0, 20.0, 10.0)), true), + (Some(Value::Integer(0)), false), + ( + Some(Value::Descriptor( + Descriptor::new("bounds").with("Left", Value::Double(0.0)).with("Rght", Value::Double(20.0)).with("Btom", Value::Double(10.0)), + )), + false, + ), + ( + Some(Value::Descriptor( + Descriptor::new("bounds") + .with("Left", Value::Double(0.0)) + .with("Top ", Value::Text(UnicodeString::new("bad"))) + .with("Rght", Value::Double(20.0)) + .with("Btom", Value::Double(10.0)), + )), + false, + ), + (Some(rect(0.0, 10.0, 20.0, 10.0)), false), + (Some(rect(0.0, -f64::MAX, 20.0, f64::MAX)), false), + (Some(rect(f64::NAN, f64::NEG_INFINITY, 20.0, 10.0)), true), + ]; + for axes in 0..4 { + for class in ["warp", "quiltWarp"] { + for (bounds, valid) in &cases { + let mut d = Warp { + style: "warpCustom".into(), + cols: 1, + rows: 1, + mesh: (0..4).flat_map(|y| (0..4).map(move |x| [x as f64 * 20.0 / 3.0, y as f64 * 10.0 / 3.0])).collect(), + ..Warp::default() + } + .to_descriptor(); + d.class_id = Id::new(class); + d.items.retain(|(key, _)| !key.is("bounds")); + if let Some(bounds) = bounds { + d.items.push((Id::new("bounds"), bounds.clone())); + } + let Value::Descriptor(env) = &mut d.items.iter_mut().find(|(key, _)| key.is("customEnvelopeWarp")).unwrap().1 else { + panic!("envelope") + }; + for (bit, key, end) in [(1, "quiltSliceX", 20.0), (2, "quiltSliceY", 10.0)] { + if axes & bit != 0 { + env.items.push((Id::new(key), Value::UnitFloats { unit: *b"#Pxl", values: vec![0.0, end] })); + } + } + let expected = |quilt| { + if bounds.is_none() { axes == 0 || axes == 3 } else { *valid || (!quilt && axes == 0) } + }; + assert_eq!( + Warp::try_from_descriptor(&d).is_ok(), + expected(class == "quiltWarp"), + "direct {class}, axes={axes}, bounds={bounds:?}" + ); + for field in ["warp", "quiltWarp"] { + let bytes = placed_layer_data("context", [0.0; 8], |parent| parent.with(field, Value::Descriptor(d.clone()))); + assert_eq!( + SmartObject::from_placed_layer_data(&bytes).is_ok(), + expected(field == "quiltWarp"), + "{field}/{class}, axes={axes}, bounds={bounds:?}" + ); + } + } + } + } + // No envelope and identity style must not bypass explicit quilt bounds. + for class in ["warp", "quiltWarp"] { + let d = Descriptor::new(class).with("bounds", rect(0.0, 10.0, 20.0, 10.0)); + assert_eq!(Warp::try_from_descriptor(&d).is_ok(), class == "warp"); + let bytes = placed_layer_data("identity", [0.0; 8], |parent| parent.with("quiltWarp", Value::Descriptor(d))); + assert!(SmartObject::from_placed_layer_data(&bytes).is_err()); + } + } + #[test] fn explicit_quilt_bounds_require_every_numeric_side() { let sides = [("Left", 0.0), ("Top ", 0.0), ("Rght", 20.0), ("Btom", 10.0)]; for omit_y in [false, true] { diff --git a/crates/psd/src/smart/write.rs b/crates/psd/src/smart/write.rs index f1edf82df..6976417b2 100644 --- a/crates/psd/src/smart/write.rs +++ b/crates/psd/src/smart/write.rs @@ -206,8 +206,9 @@ impl SmartObject { } set(&mut d, "warp", Value::Descriptor(warp.to_descriptor())); // The imported `quiltWarp` stays while it still says what the warp is. - let kept = - t.get_object("quiltWarp").is_some_and(|q| Warp::try_from_descriptor(q).is_ok_and(|parsed| parsed.as_ref() == self.warp.as_ref())); + let kept = t + .get_object("quiltWarp") + .is_some_and(|q| Warp::try_from_descriptor_kind(q, true).is_ok_and(|parsed| parsed.as_ref() == self.warp.as_ref())); if !kept { match quilt { Some(q) => set_before(&mut d, "quiltWarp", Value::Descriptor(q), "warp"), @@ -404,7 +405,8 @@ mod tests { assert!(Warp::try_from_descriptor(&unequal).is_err()); let valid_none = Warp::none(bounds).to_descriptor(); assert!(matches!(Warp::try_from_descriptor(&valid_none), Ok(None))); - for (quilt, keep) in [(unequal, false), (valid_none, true)] { + let invalid_none = Warp::none([0.0, 10.0, 20.0, 10.0]).to_descriptor(); + for (quilt, keep) in [(unequal, false), (valid_none, true), (invalid_none, false)] { let template = Descriptor::new("null").with("quiltWarp", Value::Descriptor(quilt.clone())); let bytes = placed(None).to_placed_layer_data(&PlacedLayerOptions { placed: "layer", From e1fc4b1376148197c2cfd2381eeb400e2731269a Mon Sep 17 00:00:00 2001 From: FredApps Date: Wed, 18 Nov 2026 10:39:08 +0100 Subject: [PATCH 117/161] psd: annotate quilt policy test closure parameter --- crates/psd/src/smart.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/psd/src/smart.rs b/crates/psd/src/smart.rs index 8cc140490..38f382292 100644 --- a/crates/psd/src/smart.rs +++ b/crates/psd/src/smart.rs @@ -1259,7 +1259,7 @@ mod tests { env.items.push((Id::new(key), Value::UnitFloats { unit: *b"#Pxl", values: vec![0.0, end] })); } } - let expected = |quilt| { + let expected = |quilt: bool| { if bounds.is_none() { axes == 0 || axes == 3 } else { *valid || (!quilt && axes == 0) } }; assert_eq!( From 486319f5ff8bf2a7fa80aa9df3196d20750ec257 Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Wed, 18 Nov 2026 05:18:38 -0500 Subject: [PATCH 118/161] fix(dex): retain reference-query parse incompleteness --- crates/dex/src/dex.rs | 71 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 69 insertions(+), 2 deletions(-) diff --git a/crates/dex/src/dex.rs b/crates/dex/src/dex.rs index 528ec334e..8c0b25573 100644 --- a/crates/dex/src/dex.rs +++ b/crates/dex/src/dex.rs @@ -691,7 +691,8 @@ pub struct Reference { const MAX_REFERENCE_BYTES: usize = 64 << 20; /// Every instruction that refers to class `desc` (optionally only its methods named `method`). -/// Reports truncation when instruction decoding is incomplete or the count or byte budget is exhausted. +/// Reports truncation for incomplete class data, unreadable nonzero code offsets, incomplete +/// instruction decoding, or an exhausted count or byte budget. pub fn references(dex: &Dex<'_>, desc: &str, method: Option<&str>, max: usize) -> (Vec, bool) { references_with_byte_limit(dex, desc, method, max, MAX_REFERENCE_BYTES) } @@ -702,8 +703,14 @@ fn references_with_byte_limit(dex: &Dex<'_>, desc: &str, method: Option<&str>, m let mut complete = true; for c in &dex.classes { let cd = dex.class_data(c); + complete &= cd.complete; for m in cd.direct_methods.iter().chain(&cd.virtual_methods) { - let Some(code) = dex.code(m.code_off) else { continue }; + let Some(code) = dex.code(m.code_off) else { + // Zero means no implementation (native/abstract), not damaged code. + // A nonzero unreadable body may hide references from this query. + complete &= m.code_off == 0; + continue; + }; let (insns, decoded) = crate::dex_ops::decode(&code.insns, 200_000); complete &= decoded; for i in &insns { @@ -778,6 +785,66 @@ mod tests { } } + #[test] + fn reference_queries_keep_class_data_and_code_parse_failures() { + let mut builder = fx::DexBuilder::default(); + let class = builder.ty("LTest;"); + let target = builder.method("LTest;", "target", "V", &[]); + let caller = builder.method("LTest;", "caller", "V", &[]); + builder.classes.push(fx::ClassSpec { + class, + direct: vec![ + fx::MethodSpec { method: target, access: 0, code: Some(fx::CodeSpec { insns: vec![0x000e], ..Default::default() }) }, + fx::MethodSpec { + method: caller, + access: 0, + code: Some(fx::CodeSpec { insns: vec![0x0071, target as u16, 0], ..Default::default() }), + }, + ], + ..Default::default() + }); + let bytes = builder.build(); + let mut dex = parse(&bytes).unwrap(); + let expected = references(&dex, "LTest;", Some("target"), 10); + assert_eq!(expected.0.len(), 1); + assert!(!expected.1); + + // A damaged class before a valid one must not hide the valid result or reset truncation. + let mut bad = dex.classes[0]; + bad.class_data_off = bytes.len() as u32; + dex.classes.insert(0, bad); + assert!(!dex.class_data(&dex.classes[0]).complete); + assert_eq!(references(&dex, "LTest;", Some("target"), 10), (expected.0.clone(), true)); + assert_eq!(references(&dex, "LAbsent;", None, 10), (Vec::new(), true)); + + // Locate the first method's code offset, after four counts and its index/access. + let original = parse(&bytes).unwrap(); + let reader = Reader::new(&bytes); + let mut at = original.classes[0].class_data_off as usize; + for _ in 0..6 { + at += reader.uleb128(at).unwrap().1; + } + let width = reader.uleb128(at).unwrap().1; + for missing in [false, true] { + let mut changed = bytes.clone(); + // Preserve the encoded width so the later valid method stays aligned. + changed[at..at + width].fill(if missing { 0xff } else { 0x80 }); + changed[at + width - 1] = if missing { 0x7f } else { 0 }; + let dex = parse(&changed).unwrap(); + let methods = dex.class_data(&dex.classes[0]); + assert!(methods.complete); + let offset = methods.direct_methods[0].code_off; + if missing { + assert!(offset as usize >= changed.len()); + } else { + assert_eq!(offset, 0); + } + assert!(dex.code(offset).is_none()); + assert_eq!(references(&dex, "LTest;", Some("target"), 10), (expected.0.clone(), missing)); + assert_eq!(references(&dex, "LAbsent;", None, 10), (Vec::new(), missing)); + } + } + #[test] fn fixture_strings_round_trip_utf16_surrogate_pairs() { let mut builder = fx::DexBuilder::default(); From 8bcdc60584fbb2a426caf5065a3ec17f38b5fc19 Mon Sep 17 00:00:00 2001 From: David Mann Date: Wed, 18 Nov 2026 12:32:52 +0100 Subject: [PATCH 119/161] Pin craft-ai-agents to a commit on craft-ai-integrations main (#2198) 35e0352666 is the head of a deleted, squash-merged branch and is reachable only through its pull request ref. d94a3593ad is main's head. Co-Authored-By: Claude Opus 5.5 --- Cargo.lock | 2 +- Cargo.toml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Cargo.lock b/Cargo.lock index 104361feb..037fbfeb6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1571,7 +1571,7 @@ dependencies = [ [[package]] name = "craft-ai-agents" version = "0.1.0" -source = "git+https://github.com/storytold/craft-ai-integrations?rev=35e0352666a117407d62230a6c6c2366f3e21a3b#35e0352666a117407d62230a6c6c2366f3e21a3b" +source = "git+https://github.com/storytold/craft-ai-integrations?rev=d94a3593adcff504eaaad9e5aee4c84fd2f07a72#d94a3593adcff504eaaad9e5aee4c84fd2f07a72" dependencies = [ "dirs", "serde", diff --git a/Cargo.toml b/Cargo.toml index c0b811a03..25fae98ca 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -59,8 +59,8 @@ craft-gzip = { path = "crates/gzip" } craft-gridcraft-format = { path = "crates/gridcraft-format" } craft-aaf = { path = "crates/aaf" } craft-atomic = { path = "crates/atomic" } -# The ArtCraft Launcher's app list for AI agents (storytold/craft-ai-integrations, crates/ai-agents). -craft-ai-agents = { git = "https://github.com/storytold/craft-ai-integrations", rev = "35e0352666a117407d62230a6c6c2366f3e21a3b" } +# The ArtCraft Launcher's app list for AI agents (storytold/craft-ai-integrations, crates/app/ai-agents). +craft-ai-agents = { git = "https://github.com/storytold/craft-ai-integrations", rev = "d94a3593adcff504eaaad9e5aee4c84fd2f07a72" } craft-xtask = { path = "crates/xtask" } craft-captions = { path = "crates/captions" } craft-cfb = { path = "crates/cfb" } From abde74fa7dcb6e35beffa6cb98e90ba5595d33ba Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Wed, 18 Nov 2026 06:33:33 -0500 Subject: [PATCH 120/161] fix(mcmat): validate manifests and read resources by index --- crates/mcmat/src/lib.rs | 90 +++++++++++++++++++++++++++++++++------ crates/zip/src/archive.rs | 34 +++++++++++++++ 2 files changed, 110 insertions(+), 14 deletions(-) diff --git a/crates/mcmat/src/lib.rs b/crates/mcmat/src/lib.rs index fe328cceb..ca3fa7641 100644 --- a/crates/mcmat/src/lib.rs +++ b/crates/mcmat/src/lib.rs @@ -3,7 +3,8 @@ //! The clean-room MCMAT v3 source archive. Applications interpret the SBS entry graph. use craft_zip::{Archive, Compression, Writer}; -use serde::Deserialize; +use serde::Deserializer; +use serde::de::{IgnoredAny, MapAccess, Visitor}; use std::collections::{BTreeMap, BTreeSet}; pub const VERSION: u32 = 3; @@ -62,20 +63,36 @@ impl Material { return Err(Error::Invalid("duplicate or overlapping archive entries".into())); } let json = archive.read(MANIFEST_ENTRY, limits.max_manifest_bytes)?.ok_or_else(|| Error::Invalid("missing material.json".into()))?; - #[derive(Deserialize)] - struct Version { - version: u32, + struct Marker; + impl<'de> Visitor<'de> for Marker { + type Value = (u32, bool); + fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.write_str("a version object") + } + fn visit_map>(self, mut map: A) -> Result { + let mut version = None; + let mut extra = false; + while let Some(key) = map.next_key::()? { + if key == "version" { + if version.is_some() { + return Err(serde::de::Error::duplicate_field("version")); + } + version = Some(map.next_value::()?); + } else { + map.next_value::()?; + extra = true; + } + } + Ok((version.ok_or_else(|| serde::de::Error::missing_field("version"))?, extra)) + } } - let marker: Version = serde_json::from_slice(&json).map_err(|err| Error::Invalid(format!("material.json: {err}")))?; - if marker.version != VERSION { - return Err(Error::Version(marker.version)); + let mut deserializer = serde_json::Deserializer::from_slice(&json); + let (version, extra) = deserializer.deserialize_map(Marker).map_err(|err| Error::Invalid(format!("material.json: {err}")))?; + deserializer.end().map_err(|err| Error::Invalid(format!("material.json: {err}")))?; + if version != VERSION { + return Err(Error::Version(version)); } - #[derive(Deserialize)] - #[serde(deny_unknown_fields)] - struct Marker { - version: u32, - } - if !serde_json::from_slice::(&json).is_ok_and(|m| m.version == VERSION) { + if extra { return Err(Error::Invalid("material.json must contain only the version".into())); } let source = archive.read(SOURCE_ENTRY, limits.max_entry_bytes)?.ok_or_else(|| Error::Invalid("missing source.sbs".into()))?; @@ -87,12 +104,15 @@ impl Material { continue; } if name.strip_suffix('/').is_some_and(|dir| dir == "resources" || valid_resource(dir)) { + if archive.entry_sizes(index)? != Some((0, 0)) { + return Err(Error::Invalid("directory entries must have no payload".into())); + } continue; } if !valid_resource(&name) || !seen.insert(name.to_ascii_lowercase()) { return Err(Error::Invalid(format!("invalid or ambiguous resource path `{name}`"))); } - let data = archive.read(&name, limits.max_entry_bytes)?.ok_or_else(|| Error::Invalid(format!("missing `{name}`")))?; + let data = archive.read_index(index, limits.max_entry_bytes)?.ok_or_else(|| Error::Invalid(format!("missing `{name}`")))?; resources.insert(name, data); } Ok(Self { source, resources }) @@ -151,6 +171,48 @@ mod tests { } } + #[test] + fn marker_requires_an_object_and_complete_input() { + for marker in [b"[3]".as_slice(), b"[2]", b"null", b"3", br#"{"version":3} {}"#, br#"{"version":3,"version":3}"#] { + let mut zip = Writer::new(); + zip.add(MANIFEST_ENTRY, marker, Compression::Stored).unwrap(); + zip.add(SOURCE_ENTRY, b"source", Compression::Stored).unwrap(); + assert!(matches!(Material::open(&zip.finish().unwrap(), Limits::default()), Err(Error::Invalid(_)))); + } + } + + #[test] + fn directory_members_must_be_empty() { + for payload in [b"".as_slice(), b"hidden"] { + let mut zip = Writer::new(); + zip.add(MANIFEST_ENTRY, MARKER, Compression::Stored).unwrap(); + zip.add(SOURCE_ENTRY, b"source", Compression::Stored).unwrap(); + zip.add("resources/", payload, Compression::Stored).unwrap(); + assert_eq!(Material::open(&zip.finish().unwrap(), Limits::default()).is_ok(), payload.is_empty()); + } + } + + #[test] + fn legacy_encoded_resource_names_read_by_index_under_one_budget() { + let mut zip = Writer::new(); + zip.add(MANIFEST_ENTRY, MARKER, Compression::Stored).unwrap(); + zip.add(SOURCE_ENTRY, b"s", Compression::Stored).unwrap(); + zip.add("resources/a.bin", b"data", Compression::Stored).unwrap(); + let mut bytes = zip.finish().unwrap(); + // Same-length CP437 name in both headers: no UTF-8 flag, 0x82 decodes to é. + let offsets: Vec<_> = + bytes.windows(b"resources/a.bin".len()).enumerate().filter_map(|(i, n)| (n == b"resources/a.bin").then_some(i + 10)).collect(); + assert_eq!(offsets.len(), 2); + for offset in offsets { + bytes[offset] = 0x82; + } + let total = MARKER.len() as u64 + 1 + 4; + let limits = Limits { max_total_bytes: total, ..Default::default() }; + let material = Material::open(&bytes, limits).unwrap(); + assert_eq!(material.resources["resources/é.bin"], b"data"); + assert!(Material::open(&bytes, Limits { max_total_bytes: total - 1, ..limits }).is_err()); + } + #[test] fn compresses_unencoded_source_assets() { let bytes = write(b"", [("resources/shape.svg", b"".as_slice()), ("resources/font.ttf", b"font".as_slice())]).unwrap(); diff --git a/crates/zip/src/archive.rs b/crates/zip/src/archive.rs index 49da63585..d00be89d0 100644 --- a/crates/zip/src/archive.rs +++ b/crates/zip/src/archive.rs @@ -108,6 +108,15 @@ impl<'a> Archive<'a> { Ok(Some(file.is_dir())) } + /// Declared decoded and compressed sizes at `index`, without name lookup. + pub fn entry_sizes(&mut self, index: usize) -> Result, Error> { + if index >= self.zip.len() { + return Ok(None); + } + let file = self.zip.by_index_raw(index).map_err(|e| Error::NotZip(e.to_string()))?; + Ok(Some((file.size(), file.compressed_size()))) + } + /// Do two entries share a name? `zip` then keeps one of them, and which one a writer meant /// is not known: a format that must not guess refuses the archive. pub fn has_duplicate_names(&self) -> bool { @@ -178,6 +187,16 @@ impl<'a> Archive<'a> { /// Only the bytes of a read that succeeds are charged to the total. pub fn read(&mut self, name: &str, max: u64) -> Result>, Error> { let Some(index) = self.zip.index_for_name(name) else { return Ok(None) }; + self.read_index(index, max) + } + + /// Read the entry at `index` under the same size, checksum and aggregate budget + /// rules as [`Self::read`], without repeating a decoded-name lookup. + pub fn read_index(&mut self, index: usize, max: u64) -> Result>, Error> { + if index >= self.zip.len() { + return Ok(None); + } + let name = self.entry_name(index)?.unwrap_or_default(); let entry = |msg: String| Error::Entry { name: name.to_string(), msg }; let f = self.zip.by_index(index).map_err(|e| entry(e.to_string()))?; let declared = f.size(); @@ -213,6 +232,21 @@ impl Archive<'static> { mod ownership_tests { use super::*; + #[test] + fn indexed_reads_charge_the_same_budget_and_enforce_entry_limits() { + let mut writer = crate::Writer::new(); + writer.add("a", b"abc", crate::Compression::Stored).unwrap(); + writer.add("b", b"def", crate::Compression::Deflated).unwrap(); + let bytes = writer.finish().unwrap(); + let mut archive = Archive::open(&bytes, Limits { max_entries: 2, max_total_bytes: 5 }).unwrap(); + assert!(archive.read_index(0, 2).is_err()); + assert_eq!(archive.read_index(0, 3).unwrap().unwrap(), b"abc"); + assert!(archive.read_index(1, 3).is_err()); + assert_eq!(archive.read_index(2, 3).unwrap(), None); + assert_eq!(archive.entry_sizes(0).unwrap(), Some((3, 3))); + assert_eq!(archive.entry_sizes(2).unwrap(), None); + } + #[test] fn owned_storage_lives_until_last_archive_clone_is_dropped() { let bytes = crate::Writer::new().finish().unwrap(); From 1cc156dcc3c4226f2aa25f3ff341c1cee69e9039 Mon Sep 17 00:00:00 2001 From: David Mann Date: Wed, 18 Nov 2026 12:43:50 +0100 Subject: [PATCH 121/161] ci: skip cargo fmt on the Windows lint leg (#2200) One rustfmt call with every source root of the workspace is now longer than a Windows command line may be (os error 206). rustfmt's result is the same on every system; the Linux and macOS legs keep checking it. The lint-command guard states the per-system stages. Co-Authored-By: Claude Opus 5.5 --- .github/workflows/workspace.yml | 7 ++++++- scripts/test-workspace-tests.py | 11 ++++++++--- 2 files changed, 14 insertions(+), 4 deletions(-) diff --git a/.github/workflows/workspace.yml b/.github/workflows/workspace.yml index af7e04769..5a9e4989d 100644 --- a/.github/workflows/workspace.yml +++ b/.github/workflows/workspace.yml @@ -64,7 +64,12 @@ jobs: python3 scripts/test-workspace-concurrency.py python3 scripts/test-workspace-tests.py fi - cargo fmt --check + # One rustfmt call gets every source root of the workspace, which is longer than + # Windows allows for a command line (os error 206). rustfmt's result is the same on + # every system, so the Linux and macOS legs check it. + if [ "$RUNNER_OS" != Windows ]; then + cargo fmt --check + fi cargo clippy --workspace --all-targets --locked -- -D warnings cargo clippy --workspace --all-targets --all-features --locked -- -D warnings command: | diff --git a/scripts/test-workspace-tests.py b/scripts/test-workspace-tests.py index 9572cac31..ac65da9c1 100644 --- a/scripts/test-workspace-tests.py +++ b/scripts/test-workspace-tests.py @@ -25,7 +25,9 @@ if [ "$RUNNER_OS" = Linux ]; then python3 scripts/test-workspace-concurrency.py python3 scripts/test-workspace-tests.py fi -cargo fmt --check +if [ "$RUNNER_OS" != Windows ]; then + cargo fmt --check +fi cargo clippy --workspace --all-targets --locked -- -D warnings cargo clippy --workspace --all-targets --all-features --locked -- -D warnings """ @@ -96,11 +98,14 @@ TESTS = [ class WorkspaceTests(unittest.TestCase): def test_every_leg_lints_with_both_feature_sets(self): - for os_name in ("Linux", "macOS", "Windows"): + # Windows skips only rustfmt: one call with every source root of the workspace is longer + # than a Windows command line may be (os error 206), and Linux and macOS check the same + # formatting. + for os_name, expected in (("Linux", COMMON), ("macOS", COMMON), ("Windows", COMMON[1:])): with self.subTest(os=os_name): result = commands(os_name, name="lint-command") self.assertEqual(result.returncode, 0, result.stderr) - self.assertEqual(result.stdout.splitlines(), COMMON) + self.assertEqual(result.stdout.splitlines(), expected) def test_linux_and_macos_run_one_feature_set_with_doctests(self): for os_name in ("Linux", "macOS"): From a557fed94e0f528c281b4bab241eb518b7f756ac Mon Sep 17 00:00:00 2001 From: FredApps Date: Wed, 18 Nov 2026 13:15:03 +0100 Subject: [PATCH 122/161] psd: expose path record knot decoding for consumer adapters --- crates/psd/src/path.rs | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/crates/psd/src/path.rs b/crates/psd/src/path.rs index 17604e654..dc1a091c9 100644 --- a/crates/psd/src/path.rs +++ b/crates/psd/src/path.rs @@ -64,6 +64,15 @@ impl PathRecord { pub fn is_knot(&self) -> bool { matches!(self.selector, 1 | 2 | 4 | 5) } + /// Decodes a knot's signed 8.24 coordinates as `(x, y)` document fractions. + /// Non-knot records return `None`; grouping and recovery policy remain with the caller. + pub fn knot(&self) -> Option { + if !self.is_knot() { + return None; + } + let pt = |i: usize| (fixed_to_f64(self.i32_at(i * 8 + 4)), fixed_to_f64(self.i32_at(i * 8))); + Some(PsdKnot { linked: matches!(self.selector, 1 | 4), pre: pt(0), anchor: pt(1), post: pt(2) }) + } } /// A Bézier knot as fractions of the document size: `(x, y)` pairs. @@ -158,15 +167,14 @@ impl PathData { // No preallocation: the count is untrusted (u16) and PsdKnot is 56 bytes. knots: Vec::new(), }); - } else if r.is_knot() { + } else if let Some(knot) = r.knot() { let closed = r.selector <= 2; if remaining == 0 && out.last().is_none_or(|s| s.closed != closed) { out.push(PsdSubpath { closed, operation: -1, knots: Vec::new() }); } remaining = remaining.saturating_sub(1); - let pt = |i: usize| (fixed_to_f64(r.i32_at(i * 8 + 4)), fixed_to_f64(r.i32_at(i * 8))); if let Some(s) = out.last_mut() { - s.knots.push(PsdKnot { linked: matches!(r.selector, 1 | 4), pre: pt(0), anchor: pt(1), post: pt(2) }); + s.knots.push(knot); } } } @@ -248,6 +256,23 @@ impl VectorMaskBlock { mod tests { use super::*; + #[test] + fn record_knot_decodes_signed_y_first_coordinates_without_grouping() { + let mut data = [0; 24]; + for (chunk, value) in data.chunks_exact_mut(4).zip([-(1i32 << 23), 1 << 22, 1 << 24, -(1 << 24), i32::MIN, i32::MAX]) { + chunk.copy_from_slice(&value.to_be_bytes()); + } + for selector in [1, 2, 4, 5] { + let knot = PathRecord { selector, data }.knot().unwrap(); + assert_eq!(knot.linked, matches!(selector, 1 | 4)); + assert_eq!(knot.pre, (0.25, -0.5)); + assert_eq!(knot.anchor, (-1.0, 1.0)); + assert_eq!(knot.post, (f64::from(i32::MAX) / 16_777_216.0, -128.0)); + } + for selector in [0, 3, 6, 7, 8, u16::MAX] { + assert_eq!(PathRecord { selector, data }.knot(), None); + } + } fn sample() -> Vec { vec![ PsdSubpath { From ce732d2b8b29f689fd43bef545fbc513306b18b1 Mon Sep 17 00:00:00 2001 From: FredApps Date: Wed, 18 Nov 2026 13:25:09 +0100 Subject: [PATCH 123/161] ci: preserve full Windows formatting with package-sized invocations --- scripts/ci-format-workspace.py | 46 ++++++++++++++++ scripts/ci-full-validation.sh | 7 ++- scripts/test-full-validation.py | 95 ++++++++++++++++++++++++++++++--- 3 files changed, 141 insertions(+), 7 deletions(-) create mode 100644 scripts/ci-format-workspace.py diff --git a/scripts/ci-format-workspace.py b/scripts/ci-format-workspace.py new file mode 100644 index 000000000..a9b1569a2 --- /dev/null +++ b/scripts/ci-format-workspace.py @@ -0,0 +1,46 @@ +#!/usr/bin/env python3 +"""Check every workspace package separately to bound rustfmt's Windows argv.""" +import json +from pathlib import Path +import re +import subprocess + +ROOT = Path(__file__).resolve().parents[1] + + +def workspace_names(metadata): + members = metadata["workspace_members"] + packages = metadata["packages"] + if not members or len(members) != len(set(members)): + raise ValueError("workspace member inventory is empty or duplicated") + by_id = {package["id"]: package for package in packages} + if len(by_id) != len(packages): + raise ValueError("metadata contains duplicate package IDs") + names = [] + for member in members: + name = by_id[member]["name"] + # Package names are data, never command text or option fragments. + if not isinstance(name, str) or not re.fullmatch(r"[A-Za-z0-9_][A-Za-z0-9_-]*", name): + raise ValueError("invalid workspace package name") + names.append(name) + if len(names) != len(set(names)): + raise ValueError("workspace package names are ambiguous") + return sorted(names) + + +def check_workspace(): + result = subprocess.run( + ["cargo", "metadata", "--no-deps", "--offline", "--locked", "--format-version", "1"], + cwd=ROOT, check=True, stdout=subprocess.PIPE, text=True, + ) + names = workspace_names(json.loads(result.stdout)) + # From this virtual workspace root, cargo fmt normally checks all workspace packages. + # Keep Cargo responsible for target kinds, editions and rustfmt configuration. + # One package per call retains all its targets without the workspace-sized argv. + print(f"Checking formatting for {len(names)} workspace packages", flush=True) + for name in names: + subprocess.run(["cargo", "fmt", "--check", "--package", name], cwd=ROOT, check=True) + + +if __name__ == "__main__": + check_workspace() diff --git a/scripts/ci-full-validation.sh b/scripts/ci-full-validation.sh index 6e9509b5e..3b413ec88 100644 --- a/scripts/ci-full-validation.sh +++ b/scripts/ci-full-validation.sh @@ -5,7 +5,12 @@ unset CRAFT_HOSTILE_DIVISOR PROPTEST_CASES if [ "${RUNNER_OS:-}" = Windows ]; then powershell.exe -NoProfile -Command '$ErrorActionPreference = "Stop"; Start-Service Spooler; Add-PrinterDriver -Name "Generic / Text Only"' fi -cargo fmt --check +# Windows cannot spawn one rustfmt command with every workspace target (#2200). +if [ "${RUNNER_OS:-}" = Windows ]; then + python scripts/ci-format-workspace.py +else + cargo fmt --check +fi cargo clippy --workspace --all-targets --locked -- -D warnings cargo clippy --workspace --all-targets --all-features --locked -- -D warnings # cargo test includes docs and ordinary example linking, as required by AGENTS.md. diff --git a/scripts/test-full-validation.py b/scripts/test-full-validation.py index 658f49f54..4181ef7cc 100644 --- a/scripts/test-full-validation.py +++ b/scripts/test-full-validation.py @@ -29,7 +29,11 @@ unset CRAFT_HOSTILE_DIVISOR PROPTEST_CASES if [ "${RUNNER_OS:-}" = Windows ]; then powershell.exe -NoProfile -Command '$ErrorActionPreference = "Stop"; Start-Service Spooler; Add-PrinterDriver -Name "Generic / Text Only"' fi -cargo fmt --check +if [ "${RUNNER_OS:-}" = Windows ]; then + python scripts/ci-format-workspace.py +else + cargo fmt --check +fi cargo clippy --workspace --all-targets --locked -- -D warnings cargo clippy --workspace --all-targets --all-features --locked -- -D warnings cargo test --workspace --locked @@ -54,6 +58,7 @@ cargo() { if [ "$*" = "$FAIL" ]; then return 17; fi } powershell.exe() { printf 'print-queue\n'; } +python() { test "$*" = "scripts/ci-format-workspace.py" || return 31; cargo fmt --check; } """ return subprocess.run([BASH, "-c", mock + SOURCE], env=env, capture_output=True, text=True, check=False) @@ -89,11 +94,13 @@ class FullValidation(unittest.TestCase): (["print-queue"] if name == "Windows" else []) + COMMANDS) def test_every_failure_stops_the_gate(self): - for i, command in enumerate(COMMANDS): - with self.subTest(command=command): - p = run("Linux", command) - self.assertEqual(p.returncode, 17, p.stdout + p.stderr) - self.assertEqual(p.stdout.splitlines(), COMMANDS[:i + 1]) + for name in ["Linux", "macOS", "Windows"]: + for i, command in enumerate(COMMANDS): + with self.subTest(os=name, command=command): + p = run(name, command) + self.assertEqual(p.returncode, 17, p.stdout + p.stderr) + self.assertEqual(p.stdout.splitlines(), + (["print-queue"] if name == "Windows" else []) + COMMANDS[:i + 1]) def test_workflow_preserves_fork_and_platform_boundaries(self): text = (ROOT / ".github/workflows/full-validation.yml").read_text() @@ -120,5 +127,81 @@ class FullValidation(unittest.TestCase): self.assertIn("macos-latest", pdf) + +class PackageFormatting(unittest.TestCase): + @staticmethod + def helper(): + import importlib.util + spec = importlib.util.spec_from_file_location("ci_format_workspace", ROOT / "scripts/ci-format-workspace.py") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + @staticmethod + def metadata(): + return {"workspace_members": ["id-b", "id-a"], "workspace_default_members": ["id-a"], + "packages": [{"id": "id-a", "name": "craft-a"}, + {"id": "id-b", "name": "craft-b"}, + {"id": "external", "name": "not-a-member"}]} + + def test_all_workspace_members_not_only_defaults_and_no_external_dependencies(self): + helper = self.helper() + self.assertEqual(helper.workspace_names(self.metadata()), ["craft-a", "craft-b"]) + + def test_new_package_automatically_included(self): + helper = self.helper() + data = self.metadata() + data["workspace_members"].append("id-new") + data["packages"].append({"id": "id-new", "name": "craft-new"}) + self.assertEqual(helper.workspace_names(data), ["craft-a", "craft-b", "craft-new"]) + + def test_bad_inventory_fails_closed(self): + helper = self.helper() + for members in ([], ["id-a", "id-a"], ["missing"]): + data = self.metadata() + data["workspace_members"] = members + with self.subTest(members=members), self.assertRaises((ValueError, KeyError)): + helper.workspace_names(data) + for name in ("--all", "craft-a;echo injected", "craft-a", ""): + data = self.metadata() + data["packages"][1]["name"] = name + with self.subTest(name=name), self.assertRaises(ValueError): + helper.workspace_names(data) + + def test_calls_cargo_with_one_package_and_no_shell(self): + import json + helper = self.helper() + result = subprocess.CompletedProcess([], 0, stdout=json.dumps(self.metadata())) + with patch.object(helper.subprocess, "run", return_value=result) as execute: + helper.check_workspace() + self.assertEqual([call.args[0] for call in execute.call_args_list], [ + ["cargo", "metadata", "--no-deps", "--offline", "--locked", "--format-version", "1"], + ["cargo", "fmt", "--check", "--package", "craft-a"], + ["cargo", "fmt", "--check", "--package", "craft-b"], + ]) + for call in execute.call_args_list: + self.assertEqual(call.kwargs["cwd"], ROOT) + self.assertTrue(call.kwargs["check"]) + self.assertNotIn("shell", call.kwargs) + + def test_metadata_failure_prevents_formatting(self): + helper = self.helper() + failure = subprocess.CalledProcessError(17, ["cargo", "metadata"]) + with patch.object(helper.subprocess, "run", side_effect=failure) as execute: + with self.assertRaises(subprocess.CalledProcessError): + helper.check_workspace() + self.assertEqual(execute.call_count, 1) + + def test_first_format_failure_stops_later_packages(self): + import json + helper = self.helper() + result = subprocess.CompletedProcess([], 0, stdout=json.dumps(self.metadata())) + failure = subprocess.CalledProcessError(17, ["cargo", "fmt"]) + with patch.object(helper.subprocess, "run", side_effect=[result, failure]) as execute: + with self.assertRaises(subprocess.CalledProcessError): + helper.check_workspace() + self.assertEqual(execute.call_count, 2) + + if __name__ == "__main__": unittest.main() From 83131112ab21848f577f87ccabfb31360913edb4 Mon Sep 17 00:00:00 2001 From: FredApps Date: Wed, 18 Nov 2026 13:26:52 +0100 Subject: [PATCH 124/161] Decode Cargo metadata as UTF-8 on Windows --- scripts/ci-format-workspace.py | 2 +- scripts/test-full-validation.py | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/scripts/ci-format-workspace.py b/scripts/ci-format-workspace.py index a9b1569a2..32605af70 100644 --- a/scripts/ci-format-workspace.py +++ b/scripts/ci-format-workspace.py @@ -31,7 +31,7 @@ def workspace_names(metadata): def check_workspace(): result = subprocess.run( ["cargo", "metadata", "--no-deps", "--offline", "--locked", "--format-version", "1"], - cwd=ROOT, check=True, stdout=subprocess.PIPE, text=True, + cwd=ROOT, check=True, stdout=subprocess.PIPE, text=True, encoding="utf-8", ) names = workspace_names(json.loads(result.stdout)) # From this virtual workspace root, cargo fmt normally checks all workspace packages. diff --git a/scripts/test-full-validation.py b/scripts/test-full-validation.py index 4181ef7cc..b37e979c8 100644 --- a/scripts/test-full-validation.py +++ b/scripts/test-full-validation.py @@ -179,6 +179,7 @@ class PackageFormatting(unittest.TestCase): ["cargo", "fmt", "--check", "--package", "craft-a"], ["cargo", "fmt", "--check", "--package", "craft-b"], ]) + self.assertEqual(execute.call_args_list[0].kwargs["encoding"], "utf-8") for call in execute.call_args_list: self.assertEqual(call.kwargs["cwd"], ROOT) self.assertTrue(call.kwargs["check"]) From 4d2784ec90413e53b8f6c1a1c6b8526b8077e8c5 Mon Sep 17 00:00:00 2001 From: FredApps Date: Wed, 18 Nov 2026 13:38:03 +0100 Subject: [PATCH 125/161] psd: use fixed-size chunks in vector record fixture --- crates/psd/src/path.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/crates/psd/src/path.rs b/crates/psd/src/path.rs index dc1a091c9..3520a71d8 100644 --- a/crates/psd/src/path.rs +++ b/crates/psd/src/path.rs @@ -259,7 +259,7 @@ mod tests { #[test] fn record_knot_decodes_signed_y_first_coordinates_without_grouping() { let mut data = [0; 24]; - for (chunk, value) in data.chunks_exact_mut(4).zip([-(1i32 << 23), 1 << 22, 1 << 24, -(1 << 24), i32::MIN, i32::MAX]) { + for (chunk, value) in data.as_chunks_mut::<4>().0.iter_mut().zip([-(1i32 << 23), 1 << 22, 1 << 24, -(1 << 24), i32::MIN, i32::MAX]) { chunk.copy_from_slice(&value.to_be_bytes()); } for selector in [1, 2, 4, 5] { From 74b198524b3909ec376b6a3882f8d1d1b8acd468 Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Wed, 18 Nov 2026 07:47:03 -0500 Subject: [PATCH 126/161] fix(zip): preserve named read error contracts --- crates/zip/src/archive.rs | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/crates/zip/src/archive.rs b/crates/zip/src/archive.rs index d00be89d0..b9b3b3d80 100644 --- a/crates/zip/src/archive.rs +++ b/crates/zip/src/archive.rs @@ -187,7 +187,7 @@ impl<'a> Archive<'a> { /// Only the bytes of a read that succeeds are charged to the total. pub fn read(&mut self, name: &str, max: u64) -> Result>, Error> { let Some(index) = self.zip.index_for_name(name) else { return Ok(None) }; - self.read_index(index, max) + self.read_index_named(index, name, max) } /// Read the entry at `index` under the same size, checksum and aggregate budget @@ -197,6 +197,10 @@ impl<'a> Archive<'a> { return Ok(None); } let name = self.entry_name(index)?.unwrap_or_default(); + self.read_index_named(index, &name, max) + } + + fn read_index_named(&mut self, index: usize, name: &str, max: u64) -> Result>, Error> { let entry = |msg: String| Error::Entry { name: name.to_string(), msg }; let f = self.zip.by_index(index).map_err(|e| entry(e.to_string()))?; let declared = f.size(); @@ -232,6 +236,17 @@ impl Archive<'static> { mod ownership_tests { use super::*; + #[test] + fn named_read_preserves_entry_error_for_a_corrupt_local_header() { + let mut writer = crate::Writer::new(); + writer.add("a", b"abc", crate::Compression::Stored).unwrap(); + let mut bytes = writer.finish().unwrap(); + bytes[0] = 0; // The central directory remains valid; the local signature does not. + let mut archive = Archive::open(&bytes, Limits { max_entries: 1, max_total_bytes: 3 }).unwrap(); + assert!(matches!(archive.read("a", 3), Err(Error::Entry { name, .. }) if name == "a")); + assert_eq!(archive.budget().remaining(), 3); + } + #[test] fn indexed_reads_charge_the_same_budget_and_enforce_entry_limits() { let mut writer = crate::Writer::new(); From 8df7c0268ac112fd285d33688eb6819365391a80 Mon Sep 17 00:00:00 2001 From: FredApps Date: Wed, 18 Nov 2026 16:36:52 +0100 Subject: [PATCH 127/161] psd: expose recovering section-divider blend reader --- crates/psd/src/tagged.rs | 50 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/crates/psd/src/tagged.rs b/crates/psd/src/tagged.rs index c8ec1789f..1f45b8f6a 100644 --- a/crates/psd/src/tagged.rs +++ b/crates/psd/src/tagged.rs @@ -489,7 +489,23 @@ fn expect_kind(r: &mut Reader<'_>, kind: &'static [u8; 4]) -> Result<()> { Ok(()) } +impl SectionDivider { + /// Read a divider while recovering a missing or invalid optional blend signature. + /// + /// The four-byte kind is required. Fewer than eight bytes after the kind, or a suffix + /// whose signature is not `8BIM`, leaves both blend mode and subtype absent. + /// Unlike this opt-in recovery, [`TaggedBlock::parsed`] still rejects an invalid + /// complete signature. Unknown kinds and blend keys are preserved. + pub fn from_bytes_recovering_blend(data: &[u8]) -> Result { + parse_section_with_recovery(data, true) + } +} + fn parse_section(d: &[u8]) -> Result { + parse_section_with_recovery(d, false) +} + +fn parse_section_with_recovery(d: &[u8], recover_blend: bool) -> Result { let mut r = Reader::new(d); let kind = SectionType::from_u32(r.u32()?); let mut blend_mode = None; @@ -497,6 +513,9 @@ fn parse_section(d: &[u8]) -> Result { if r.remaining() >= 8 { let sig = r.array::<4>()?; if &sig != b"8BIM" { + if recover_blend { + return Ok(SectionDivider { kind, blend_mode, sub_type }); + } return Err(PsdError::InvalidSignature { expected: "8BIM", found: sig }); } blend_mode = Some(BlendMode::from_key(r.array()?)); @@ -809,4 +828,35 @@ mod tests { assert_eq!(blocks, vec![TaggedBlock::layer_id(1)]); assert_eq!(trailing, out[damaged..]); } + + #[test] + fn section_recovering_blend_preserves_strict_parser_and_prefixes() { + let full = [0, 0, 0, 99, b'8', b'B', b'I', b'M', b'z', b'z', b'z', b'z', 0, 0, 0, 7, 255]; + for len in 0..=17 { + let data = &full[..len]; + let recovered = SectionDivider::from_bytes_recovering_blend(data); + if len < 4 { + assert!(recovered.is_err()); + assert!(parse_section(data).is_err()); + continue; + } + let value = recovered.unwrap(); + assert_eq!(value.kind, SectionType::Unknown(99)); + assert_eq!(value.blend_mode.map(BlendMode::key), (len >= 12).then_some(*b"zzzz")); + assert_eq!(value.sub_type, (len >= 16).then_some(7)); + assert_eq!(parse_section(data).unwrap(), value); + for key in [*b"lsct", *b"lsdk"] { + assert_eq!(TaggedBlock::new(key, data.to_vec()).parsed(), Some(Ok(BlockData::SectionDivider(value)))); + } + } + for len in 12..=17 { + let mut data = full[..len].to_vec(); + data[4..8].copy_from_slice(b"nope"); + let value = SectionDivider::from_bytes_recovering_blend(&data).unwrap(); + assert_eq!(value, SectionDivider { kind: SectionType::Unknown(99), blend_mode: None, sub_type: None }); + for key in [*b"lsct", *b"lsdk"] { + assert!(matches!(TaggedBlock::new(key, data.clone()).parsed(), Some(Err(PsdError::InvalidSignature { .. })))); + } + } + } } From 1063b597269adf5c74870e3c25d017a6ff40727a Mon Sep 17 00:00:00 2001 From: FredApps Date: Wed, 18 Nov 2026 17:37:25 +0100 Subject: [PATCH 128/161] psd: expose bounded raw type-layer text prefix --- crates/psd/src/text.rs | 74 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 64 insertions(+), 10 deletions(-) diff --git a/crates/psd/src/text.rs b/crates/psd/src/text.rs index 3a9e14952..3db0153f3 100644 --- a/crates/psd/src/text.rs +++ b/crates/psd/src/text.rs @@ -7,12 +7,40 @@ //! read here. The block stays raw in the model and writes back unchanged. Came from EffectCraft's //! `effectcraft-psd`. -use crate::descriptor::Descriptor; +use crate::descriptor::{Descriptor, ParseLimits}; use crate::engine_data::{self, TextStyle}; use crate::error::Result; use crate::io::Reader; use crate::layer::LayerRecord; +/// Raw text prefix of a `TySh` block, before the optional warp and bounds suffix. +/// +/// Version fields are consumed without validation. Transform and descriptor numbers +/// are retained without normalization; consumers choose their own interpretation. +#[derive(Clone, Debug, PartialEq)] +pub struct TypeLayerPrefix { + /// Stored affine transform (xx, xy, yx, yy, tx, ty), including nonfinite values. + pub transform: [f64; 6], + /// Text descriptor, read using the caller's item limits. + pub descriptor: Descriptor, +} + +impl TypeLayerPrefix { + /// Read the required header and text descriptor only. A missing or malformed + /// trailing warp/bounds suffix is ignored; truncated required fields are errors. + pub fn from_bytes_with_limits(data: &[u8], limits: ParseLimits) -> Result { + let mut r = Reader::new(data); + let _version = r.u16()?; + let mut transform = [0.0; 6]; + for value in &mut transform { + *value = r.f64()?; + } + let _text_version = r.u16()?; + let _descriptor_version = r.u32()?; + let descriptor = Descriptor::read_with_limits(&mut r, limits)?; + Ok(Self { transform, descriptor }) + } +} /// What a type layer says: its text, where it sits and how its first style run looks. #[derive(Clone, Debug, PartialEq)] pub struct TypeLayer { @@ -32,19 +60,11 @@ pub struct TypeLayer { impl TypeLayer { /// Reads the data of a `TySh` block. pub fn from_bytes(data: &[u8]) -> Result { - let mut r = Reader::new(data); - let _version = r.u16()?; - let mut transform = [0.0; 6]; - for v in &mut transform { - *v = r.f64()?; - } + let TypeLayerPrefix { mut transform, descriptor } = TypeLayerPrefix::from_bytes_with_limits(data, ParseLimits::default())?; // A transform with NaN or infinity is unusable: fall back to the identity. if transform.iter().any(|v| !v.is_finite()) { transform = [1.0, 0.0, 0.0, 1.0, 0.0, 0.0]; } - let _text_version = r.u16()?; - let _descriptor_version = r.u32()?; - let descriptor = Descriptor::read(&mut r)?; let text = descriptor.get_text("Txt ").unwrap_or_default().replace("\r\n", "\n").replace('\r', "\n"); let text = text.strip_suffix('\n').map(str::to_string).unwrap_or(text); let style = descriptor.get_raw_data("EngineData").map(|e| engine_data::first_style(&engine_data::parse_lossy(e))).unwrap_or_default(); @@ -149,6 +169,40 @@ mod tests { let t = TypeLayer::from_bytes(&type_block([1.0, 0.0, 0.0, 1.0, 0.0, 0.0], "x", Some(engine), None)).unwrap(); assert_eq!(t.style.size, Some(12.0)); } + + #[test] + fn raw_type_prefix_preserves_numbers_and_applies_explicit_limits() { + let mut data = Vec::new(); + data.put_u16(999); + for value in [f64::NAN, 0.0, 0.0, 1.0, f64::INFINITY, -0.0] { + data.put_f64(value); + } + data.put_u16(999); + data.put_u32(999); + let d = Descriptor::new("TxLr") + .with("Txt ", Value::Text(UnicodeString::new("a\r\n"))) + .with("nums", Value::List(vec![Value::Double(f64::NEG_INFINITY), Value::Integer(2)])); + d.write(&mut data); + let limits = ParseLimits { max_descriptor_items: 2, max_list_items: 2 }; + for cut in 0..data.len() { + assert!(TypeLayerPrefix::from_bytes_with_limits(&data[..cut], limits).is_err(), "cut {cut}"); + } + for suffix in [vec![], vec![255], vec![0; 6], vec![255; 40]] { + let mut bytes = data.clone(); + bytes.extend(suffix); + let raw = TypeLayerPrefix::from_bytes_with_limits(&bytes, limits).unwrap(); + assert!(raw.transform[0].is_nan()); + assert_eq!(raw.transform[4], f64::INFINITY); + assert!(raw.transform[5].is_sign_negative()); + assert_eq!(raw.descriptor.get_text("Txt "), Some("a\r\n".to_string())); + assert!( + matches!(raw.descriptor.get("nums"), Some(Value::List(v)) if matches!(v.first(), Some(Value::Double(x)) if *x == f64::NEG_INFINITY)) + ); + assert_eq!(TypeLayer::from_bytes(&bytes).unwrap().transform, [1.0, 0.0, 0.0, 1.0, 0.0, 0.0]); + } + assert!(TypeLayerPrefix::from_bytes_with_limits(&data, ParseLimits { max_descriptor_items: 1, ..limits }).is_err()); + assert!(TypeLayerPrefix::from_bytes_with_limits(&data, ParseLimits { max_list_items: 1, ..limits }).is_err()); + } } /// Editable type-tool envelope with PhotoCraft-compatible recovery and geometry limits. From bc4aae37bb219049596327fca971046dab0b78be Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Wed, 18 Nov 2026 12:12:13 -0500 Subject: [PATCH 129/161] Merge pull request #2245 from storytold/fix/ocsp-certificate-wrapper-20261118 fix(pkcs7): read standard OCSP certificate sequence wrappers --- crates/pkcs7/src/revocation.rs | 17 ++++++++++++++--- crates/pkcs7/tests/it/crypto.rs | 10 +++++++++- 2 files changed, 23 insertions(+), 4 deletions(-) diff --git a/crates/pkcs7/src/revocation.rs b/crates/pkcs7/src/revocation.rs index dfb398cae..91d69730a 100644 --- a/crates/pkcs7/src/revocation.rs +++ b/crates/pkcs7/src/revocation.rs @@ -249,10 +249,21 @@ impl OcspResponse { for t in it { if t.tag == tag::ctx(0) { for c in t.children()? { - if c.tag == tag::SEQUENCE - && let Ok(cert) = Certificate::parse(c.raw) - { + if c.tag != tag::SEQUENCE { + continue; + } + // Preserve the historical direct-certificate form. RFC 6960 instead wraps + // certificates in one SEQUENCE OF inside [0]; unwrap only that one level. + if let Ok(cert) = Certificate::parse(c.raw) { responder_cert.get_or_insert(cert); + } else if let Ok(certs) = c.children() { + for cert in certs { + if cert.tag == tag::SEQUENCE + && let Ok(cert) = Certificate::parse(cert.raw) + { + responder_cert.get_or_insert(cert); + } + } } } } diff --git a/crates/pkcs7/tests/it/crypto.rs b/crates/pkcs7/tests/it/crypto.rs index 75ab5415d..d92612a5e 100644 --- a/crates/pkcs7/tests/it/crypto.rs +++ b/crates/pkcs7/tests/it/crypto.rs @@ -498,9 +498,17 @@ fn a_responder_that_copies_the_issuers_name_does_not_answer_for_it() { let single = der::seq(&[&cert_id, &revoked, &this.encode(), &der::tlv(tag::ctx(0), &next.encode())]); let by_key = der::tlv(tag::ctx(2), &der::octets(&DigestAlg::Sha1.digest(&[&impostor.public_key.key_bits()]))); // RFC 6960's `[0] EXPLICIT SEQUENCE OF Certificate`, and the certificate directly in `[0]`. - for certs in [der::explicit(0, &der::seq(&[&impostor.raw])), der::explicit(0, &impostor.raw)] { + for certs in [ + der::explicit(0, &der::seq(&[&impostor.raw])), + der::explicit(0, &impostor.raw), + // An unreadable entry must not hide the first readable certificate; later entries + // must not replace it. Parsing it still does not authorize the impostor. + der::explicit(0, &der::seq(&[&der::seq(&[]), &impostor.raw, &issuer.raw])), + ] { let forged = OcspResponse::parse(&signed_ocsp(&key, &by_key, Some(&certs), &single, this)).unwrap(); assert_eq!(forged.check(issuer, issuer, mid_june()), RevocationStatus::Unknown); + // Rejection must follow verification policy, not failure to read the embedded certificate. + assert_eq!(forged.into_parts().responder_cert.map(|cert| cert.raw), Some(impostor.raw.clone())); } } From f4b5fdebc1bc5426961289f459a7bd7a7e830b51 Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Wed, 18 Nov 2026 12:36:15 -0500 Subject: [PATCH 130/161] Exercise valid delegated OCSP evidence and isolated rejection policies (#2249) --- crates/pkcs7/tests/it/chain.rs | 59 ++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/crates/pkcs7/tests/it/chain.rs b/crates/pkcs7/tests/it/chain.rs index 004f6d9e5..8e706d803 100644 --- a/crates/pkcs7/tests/it/chain.rs +++ b/crates/pkcs7/tests/it/chain.rs @@ -310,3 +310,62 @@ fn an_issuer_with_an_unprocessed_critical_extension_does_not_vouch() { let pool = [i.cert.clone(), constrained_root.cert.clone()]; assert_eq!(names(&build_chain(&l.cert, &pool, None)), ["Leaf", "Under Constrained Root", "Name Constrained Root"]); } + +/// Standards-shaped evidence signed by a directly issued responder, not merely a parse fixture. +#[test] +fn delegated_ocsp_requires_the_issuer_signature_eku_and_response_signature() { + use craft_pkcs7::keys::DigestAlg; + use craft_pkcs7::revocation::{OcspResponse, RevocationStatus}; + + let issuer = root(); + let leaf = end_entity("OCSP Leaf", &issuer); + let responder = signer_with("OCSP Responder", &issuer, &[extended_key_usage(&["1.3.6.1.5.5.7.3.9"])]); + let missing_eku = signer_with("No OCSP EKU", &issuer, &[]); + let wrong_eku = signer_with("Wrong OCSP EKU", &issuer, &[extended_key_usage(&[SERVER_AUTH])]); + // Same issuer name, different key: name equality alone must not authorize delegation. + let other_issuer = root(); + let wrong_issuer = signer_with("Wrong Issuer", &other_issuer, &[extended_key_usage(&["1.3.6.1.5.5.7.3.9"])]); + let at = time(2026); + let hash = DigestAlg::Sha256; + let id = der::seq(&[ + &hash.algorithm(), + &der::octets(&hash.digest(&[&leaf.cert.issuer.raw])), + &der::octets(&hash.digest(&[&issuer.key.public_key().key_bits()])), + &der::uint(&leaf.cert.serial), + ]); + let single = der::seq(&[&id, &der::tlv(tag::ctx_prim(0), &[]), &at.encode(), &der::explicit(0, &time(2027).encode())]); + let evidence = |responder: &Party, key: &PrivateKey, certs: &[u8], by_key: bool| { + let identity = if by_key { + der::explicit(2, &der::octets(&DigestAlg::Sha1.digest(&[&responder.key.public_key().key_bits()]))) + } else { + der::explicit(1, &responder.name.raw) + }; + let tbs = der::seq(&[&identity, &at.encode(), &der::seq(&[&single])]); + let signature = key.sign(hash, &tbs).unwrap(); + let basic = der::seq(&[&tbs, &key.signature_algorithm(hash), &der::bit_string(&signature), certs]); + der::seq(&[&der::tlv(tag::ENUMERATED, &[0]), &der::explicit(0, &der::seq(&[&der::oid("1.3.6.1.5.5.7.48.1.1"), &der::octets(&basic)]))]) + }; + for by_key in [false, true] { + for (party, signing_key, expected) in [ + (&responder, &responder.key, RevocationStatus::Good), + (&responder, &other_issuer.key, RevocationStatus::Unknown), + (&missing_eku, &missing_eku.key, RevocationStatus::Unknown), + (&wrong_eku, &wrong_eku.key, RevocationStatus::Unknown), + (&wrong_issuer, &wrong_issuer.key, RevocationStatus::Unknown), + ] { + let certs = der::explicit(0, &der::seq(&[&party.cert.raw])); + let response = OcspResponse::parse(&evidence(party, signing_key, &certs, by_key)).unwrap(); + assert_eq!(response.check(&leaf.cert, &issuer.cert, at), expected); + // Even each rejected case must reach policy with a parsed responder certificate. + assert_eq!(response.into_parts().responder_cert.as_ref(), Some(&party.cert)); + } + // Extra wrapper nesting is not recursively searched for a usable certificate. + let nested = der::explicit(0, &der::seq(&[&der::seq(&[&responder.cert.raw])])); + let response = OcspResponse::parse(&evidence(&responder, &responder.key, &nested, by_key)).unwrap(); + assert_eq!(response.check(&leaf.cert, &issuer.cert, at), RevocationStatus::Unknown); + assert!(response.into_parts().responder_cert.is_none()); + // A truncated DER child inside [0] is malformed, not missing evidence to trust. + let malformed = der::explicit(0, &[tag::SEQUENCE, 0x82, 0x01]); + assert!(OcspResponse::parse(&evidence(&responder, &responder.key, &malformed, by_key)).is_err()); + } +} From 2466fd7ad8371a3551946a0795dbae11e0c3f805 Mon Sep 17 00:00:00 2001 From: "Dylan M. Taylor" Date: Wed, 18 Nov 2026 14:05:54 -0500 Subject: [PATCH 131/161] Reject nonfinite promoted PostScript decimal integers (#2256) --- crates/ps/src/read/scan.rs | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/crates/ps/src/read/scan.rs b/crates/ps/src/read/scan.rs index 166fad5f0..494475e75 100644 --- a/crates/ps/src/read/scan.rs +++ b/crates/ps/src/read/scan.rs @@ -294,10 +294,12 @@ pub fn number(t: &[u8]) -> Option { } let digits_only = s.strip_prefix(['+', '-']).unwrap_or(s); if !digits_only.is_empty() && digits_only.bytes().all(|b| b.is_ascii_digit()) { - return Some(match s.parse::() { - Ok(i) if i32::try_from(i).is_ok() => Obj::Int(i), - _ => Obj::Real(s.parse::().ok()?), - }); + return match s.parse::() { + Ok(i) if i32::try_from(i).is_ok() => Some(Obj::Int(i)), + // Integer-looking tokens can overflow f64 too. Apply the same finite + // policy as decimal/exponent reals before constructing an object. + _ => s.parse::().ok().filter(|value| value.is_finite()).map(Obj::Real), + }; } // Reals: digits with a point and/or an exponent; nothing else (`1e` or `.` alone are names). let mut seen_digit = false; @@ -362,6 +364,27 @@ mod tests { } } + #[test] + fn decimal_integer_promotion_never_constructs_nonfinite_reals() { + for text in ["2147483647", "-2147483648"] { + assert!(matches!(number(text.as_bytes()), Some(Obj::Int(_)))); + } + for text in ["2147483648", "-2147483649", "9223372036854775808"] { + assert!(matches!(number(text.as_bytes()), Some(Obj::Real(value)) if value.is_finite())); + } + let finite = "9".repeat(308); + assert!(matches!(number(finite.as_bytes()), Some(Obj::Real(value)) if value.is_finite())); + for sign in ["", "+", "-"] { + let overflow = format!("{sign}{}", "9".repeat(309)); + assert!(number(overflow.as_bytes()).is_none()); + // Follow the existing overflowing-real policy: leave the token as an + // executable name rather than introducing infinity into the interpreter. + assert!(matches!(&all(overflow.as_bytes())[..], [Obj::Name(name, true)] if name.as_ref() == overflow.as_bytes())); + } + assert!(number(b"1e309").is_none()); + assert!(matches!(number(b"16#ffffffff"), Some(Obj::Int(-1)))); + } + #[test] fn scans_the_token_kinds() { let t = all(b"12 -3.5 1e3 16#FF /lit name (a\\(b\\)\\101\\\n c) <41 42> <~87cURDZ~> {1 {2} add} //x [ ] << >> % note\n.5 1.e2 -."); From c62d548b6b806beac8c1f93f2f23f4f2f3cf1c13 Mon Sep 17 00:00:00 2001 From: FredApps Date: Wed, 18 Nov 2026 20:36:57 +0100 Subject: [PATCH 132/161] psd: expose bounded recovering tagged descriptor prefixes --- crates/psd/src/tagged.rs | 79 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 79 insertions(+) diff --git a/crates/psd/src/tagged.rs b/crates/psd/src/tagged.rs index 1f45b8f6a..3bfc210d1 100644 --- a/crates/psd/src/tagged.rs +++ b/crates/psd/src/tagged.rs @@ -204,6 +204,35 @@ impl TaggedBlock { (alignment - len % alignment) % alignment } + /// Read the raw descriptor prefix of `lfx2`, `SoCo` or `CgEd` with caller limits. + /// + /// This opt-in recovery ignores both version words of `lfx2` (eight bytes), or + /// the one version word of `SoCo` / `CgEd` (four bytes). Trailing bytes are + /// ignored and nonfinite descriptor numbers are retained. Strict validation + /// and versioned-descriptor parsing are unchanged. + /// + /// Other keys return `None`. Recognized unloaded blocks, short headers and + /// malformed or over-limit descriptors return `Some(Err)`. + pub fn descriptor_prefix_recovering_versions_with_limits( + &self, + limits: crate::descriptor::ParseLimits, + ) -> Option> { + let words = match &self.key { + b"lfx2" => 2, + b"SoCo" | b"CgEd" => 1, + _ => return None, + }; + if self.stored.is_some() { + return Some(Err(PsdError::invalid(format!("{}: block data is still in the source; load it first", self.key_str())))); + } + Some((|| { + let mut r = Reader::new(&self.data); + for _ in 0..words { + let _version = r.u32()?; + } + crate::descriptor::Descriptor::read_with_limits(&mut r, limits) + })()) + } /// Typed view for known keys. `None` for keys without a typed parser, /// `Some(Err)` if a known key's data is malformed. pub fn parsed(&self) -> Option>> { @@ -859,4 +888,54 @@ mod tests { } } } + + #[test] + fn descriptor_prefix_recovery_keeps_versions_suffixes_and_limits_explicit() { + use crate::descriptor::{Descriptor, ParseLimits, Value, VersionedDescriptor}; + let limits = ParseLimits { max_descriptor_items: 1, max_list_items: 1 }; + // Independent raw descriptor: empty name, null class, one doub item named num . + let mut raw = vec![0; 8]; + raw.extend(b"null"); + raw.extend(1u32.to_be_bytes()); + raw.extend(0u32.to_be_bytes()); + raw.extend(b"num doub"); + raw.extend(f64::NAN.to_be_bytes()); + for key in [*b"lfx2", *b"SoCo", *b"CgEd"] { + let header = if key == *b"lfx2" { 8 } else { 4 }; + for word in [0u32, 16, u32::MAX] { + let mut bytes = word.to_be_bytes().repeat(header / 4); + bytes.extend(&raw); + for cut in 0..bytes.len() { + assert!(TaggedBlock::new(key, bytes[..cut].to_vec()).descriptor_prefix_recovering_versions_with_limits(limits).unwrap().is_err()); + } + for tail in [vec![], vec![255], vec![1, 2, 3, 4, 5]] { + let mut data = bytes.clone(); + data.extend(tail); + let block = TaggedBlock::new(key, data); + let d = block.descriptor_prefix_recovering_versions_with_limits(limits).unwrap().unwrap(); + assert!(matches!(d.get("num "), Some(Value::Double(v)) if v.is_nan())); + assert!(block.parsed().is_none()); + if key == *b"lfx2" && word != 16 { + assert!(block.check_structure().is_err()); + } + } + assert!( + TaggedBlock::new(key, bytes) + .descriptor_prefix_recovering_versions_with_limits(ParseLimits { max_descriptor_items: 0, ..limits }) + .unwrap() + .is_err() + ); + } + let mut list = vec![0; header]; + list.extend(Descriptor::new("null").with("list", Value::List(vec![Value::Integer(1), Value::Integer(2)])).to_bytes()); + assert!(TaggedBlock::new(key, list).descriptor_prefix_recovering_versions_with_limits(limits).unwrap().is_err()); + let mut unloaded = TaggedBlock::new(key, vec![]); + unloaded.stored = Some(crate::source::Stored { source: crate::source::Source::new(Vec::::new()), offset: 0, len: 0 }); + assert!(unloaded.descriptor_prefix_recovering_versions_with_limits(limits).unwrap().is_err()); + } + assert!(TaggedBlock::new(*b"zzzz", raw.clone()).descriptor_prefix_recovering_versions_with_limits(limits).is_none()); + let mut versioned = 99u32.to_be_bytes().to_vec(); + versioned.extend(raw); + assert!(VersionedDescriptor::parse_prefix(&versioned).is_err()); + } } From ccabfee3b5b9353b1f48f0bfd833f3fa3df36782 Mon Sep 17 00:00:00 2001 From: Phil Hart Date: Wed, 18 Nov 2026 15:00:30 -0500 Subject: [PATCH 133/161] craft-raw: no frame floor for tiny lossless JPEG tiles (#2232) A DNG tile's frame may hold at most twice the tile's samples and a Sony quad tile's at most the tile's samples, with no 65,536-sample floor. The floor let each of many tiny tiles decode and keep a frame far larger than the tile budget counts (storytold/lightcraft#4024). A DNG tile also drops the surplus capacity of a larger frame after the resize. Claude-Session: https://claude.ai/code/session_01Fnx9MwPUug72xHRgg993uc Co-authored-by: Claude Opus 5.5 --- crates/raw/src/arw.rs | 32 +++++++++++++++++++++++++++++--- crates/raw/src/tiffraw.rs | 21 ++++++++++++++++++++- 2 files changed, 49 insertions(+), 4 deletions(-) diff --git a/crates/raw/src/arw.rs b/crates/raw/src/arw.rs index 389c464a5..6ac30f37e 100644 --- a/crates/raw/src/arw.rs +++ b/crates/raw/src/arw.rs @@ -365,9 +365,10 @@ fn read_quad_tiles(bytes: &[u8], info: &ImageInfo, limits: &Limits) -> Result c.width as usize || f.height * 2 > c.height as usize { return Err(RawError::Corrupt(format!( "lossless ARW tile: {} samples in a {}×{}×{} frame", @@ -1707,6 +1708,31 @@ mod tests { assert_eq!(read_quad_tiles(&file, &info, &Limits::default()).unwrap(), mosaic); } + /// storytold/lightcraft#4024: 256 tiles of 2×2 all pointing at one frame of 64×64×4 samples are refused before the + /// frame is decoded (at most a tile's samples), not after each tile has decoded 16,384 samples. + #[test] + fn tiny_quad_tiles_do_not_decode_a_large_frame() { + let frame = ljpeg::encode(&vec![300u16; 64 * 64 * 4], 64, 64, 4, 14, 1, 0); + let info = ImageInfo { + width: 32, + height: 32, + bits_per_sample: vec![14], + samples_per_pixel: 1, + compression: 7, + photometric: photometric::CFA, + planar: 1, + predictor: 1, + sample_format: 1, + new_subfile_type: 0, + layout: Layout::Tiles { tile_width: 2, tile_height: 2 }, + offsets: vec![0; 256], + byte_counts: vec![frame.len() as u64; 256], + }; + // refused by the decoder's sample limit (a Limit), not by the shape check after a full decode (Corrupt) + let got = read_quad_tiles(&frame, &info, &Limits::default()); + assert!(matches!(got, Err(RawError::Limit(_))), "{got:?}"); + } + /// storytold/lightcraft#2561: a small file whose one-byte or packed 12-bit strip declares a large frame is refused before the full /// decode allocates the frame. #[test] diff --git a/crates/raw/src/tiffraw.rs b/crates/raw/src/tiffraw.rs index b7291f9d0..7dc29f5a7 100644 --- a/crates/raw/src/tiffraw.rs +++ b/crates/raw/src/tiffraw.rs @@ -344,7 +344,9 @@ fn decode_chunk( unpack_chunk(src, order, packing, cw, ch, cpp, bits, float, 1) } comp::JPEG => { - let f = ljpeg::decode(src, n.saturating_mul(2).max(1 << 16))?; + // A frame may hold at most twice the tile's samples, with no floor: a 65,536-sample floor let each of + // many tiny tiles decode (and keep) a frame far larger than the tile budget counts (storytold/lightcraft#4024) + let f = ljpeg::decode(src, n.saturating_mul(2))?; if f.data.len() < n { // Some writers encode edge tiles smaller than nominal: accept a frame that covers the rows inside the image. if f.data.is_empty() || f.data.len() % (cw * cpp) != 0 || f.data.len() / (cw * cpp) < vh { @@ -353,6 +355,8 @@ fn decode_chunk( } let mut v = f.data; v.resize(n, 0); + // a larger frame's surplus is not kept: the tile holds what the decode budget counts for it + v.shrink_to_fit(); Ok(ChunkPx::U16(v)) } #[cfg(feature = "dng")] @@ -1138,6 +1142,21 @@ mod budget_tests { } } + /// storytold/lightcraft#4024: 1,024 tiles of 2×2 all pointing at one 8 KB lossless JPEG of 128×128×4 samples. Each + /// reference brings enough bytes for both `check_image` bounds, but a tile may not decode a frame of 65,536 + /// samples: with a 65,536-sample floor on the decode limit each tile held about 128 KB that the grouping + /// counted as 8 bytes (134 MB for a 4,096-sample image; 34 GB for the same grid at 1024×1024). + #[test] + fn tiny_tiles_with_a_large_frame_are_refused() { + let tile = ljpeg::encode(&vec![300u16; 128 * 128 * 4], 128, 128, 4, 12, 1, 0); + let info = ImageInfo { bits_per_sample: vec![12], byte_counts: vec![tile.len() as u64; 1024], ..aliased(64, 64, 2, 0) }; + assert!(read_image(&tile, &info, ByteOrder::Little, Packing::Msb, &Limits::default()).is_err()); + // a frame of the tile's own size still decodes + let fits = ljpeg::encode(&[300u16; 4], 2, 2, 1, 12, 1, 0); + let info = ImageInfo { bits_per_sample: vec![12], byte_counts: vec![fits.len() as u64; 4], ..aliased(4, 4, 2, 0) }; + assert_eq!(read_image(&fits, &info, ByteOrder::Little, Packing::Msb, &Limits::default()), Ok(RawData::U16(vec![300; 4 * 4]))); + } + /// storytold/lightcraft#2554: tiled decoders decode the grid in groups of at most a quarter of the image (1 Mi samples at /// least), each group at least one chunk, covering every chunk once and in order. #[cfg(feature = "arw")] From 0976e3fd19b1de7581733e4cccbff34b6ee6ea48 Mon Sep 17 00:00:00 2001 From: Phil Hart Date: Wed, 18 Nov 2026 15:02:12 -0500 Subject: [PATCH 134/161] craft-raw: Kelvin presets for more CR2, CR3, PEF and RW2 raws (#2234) Reads the camera's white-balance presets as a Kelvin curve where the decoders gave none: - CR2 ColorData versions 1 to 12 (5-word records), version 10 at either of its two layouts, picked by the Daylight record's 5200 K; - CR3 ColorData versions 16-19, 32-34, 48 and 64-66; - PEF maker note 0x0221 (entries of 53190 - K and the red/blue multipliers); - RW2/RWL/RAW tag 0x0027: the D55 and 3200 K studio tungsten multipliers as a neutral pair with their chromaticities (KelvinPresets::from_neutral_lights / neutral_lights). Ported from storytold/lightcraft#2527. Claude-Session: https://claude.ai/code/session_01Fnx9MwPUug72xHRgg993uc Co-authored-by: Claude Opus 5.5 --- crates/raw/src/cr2.rs | 70 ++++++++++++++++++++++++++++++++-------- crates/raw/src/cr3.rs | 52 ++++++++++++++++++++++++++++- crates/raw/src/kelvin.rs | 42 ++++++++++++++++++++++-- crates/raw/src/pef.rs | 47 ++++++++++++++++++++++++++- crates/raw/src/rw2.rs | 55 ++++++++++++++++++++++++++++++- 5 files changed, 247 insertions(+), 19 deletions(-) diff --git a/crates/raw/src/cr2.rs b/crates/raw/src/cr2.rs index 03ab9c376..5f8f8f04f 100644 --- a/crates/raw/src/cr2.rs +++ b/crates/raw/src/cr2.rs @@ -115,25 +115,56 @@ fn wb_from_color_balance(v: &[u64]) -> Option<[f32; 3]> { /// Flash, in that order. Fluorescent is left out (a green light off the locus) and so is the Kelvin row (it holds the /// dial's last setting, and on some bodies a temperature that doesn't match its own levels: 6036 K with the camera at /// 5500 K on the PowerShot G1 X Mark III, Daylight's levels at 5213 K on the EOS 2000D). Positions from the ExifTool -/// Canon tag-name documentation's `ColorData5` / `ColorData8` tables, confirmed with ExifTool run as a black-box tool -/// on files of each version: -4 on a PowerShot G1 X Mark III (`ColorData5`: 8-word records, the second record after -/// Daylight a repeat of it), 13, 14 and 15 on the CC0 corpus samples of the EOS 5D Mark IV, 2000D and 6D Mark II -/// (`ColorData8`: 5-word records). Other versions give no presets. -fn preset_rows(version: i16) -> Option<[(usize, usize); 5]> { - match version { - -4 => Some([(95, 102), (111, 118), (119, 126), (127, 134), (151, 158)]), - 13..=15 => Some([(133, 137), (138, 142), (143, 147), (148, 152), (163, 167)]), - _ => None, +/// Canon tag-name documentation's `ColorData1`–`ColorData8` tables, confirmed with ExifTool run as a black-box tool +/// on files of each version and by reading the words of 453 CR2 files (reporters' samples and the CC0 corpus): -4 on +/// a PowerShot G1 X Mark III (`ColorData5`: 8-word records, the second record after Daylight a repeat of it); +/// everything else is 5-word records `R G G B Kelvin` in the order Daylight (5200 K), Shade (7000), Cloudy (6000), +/// Tungsten (3200), Fluorescent, Kelvin, Flash (thirty words after Daylight): version 1 from word 78, 2–9 from 83, 10 +/// from 103 on the EOS 1200D family and from 128 on the others, 11 from 128, 12–15 from 133. A version with two +/// layouts is read at the one whose Daylight record names 5200 K. Other versions give no presets. +fn preset_rows(version: i16) -> &'static [[(usize, usize); 5]] { + const fn records(daylight: usize) -> [(usize, usize); 5] { + preset_records(daylight, daylight + 30) } + const G1X3: [(usize, usize); 5] = [(95, 102), (111, 118), (119, 126), (127, 134), (151, 158)]; + const AT_78: [(usize, usize); 5] = records(78); + const AT_83: [(usize, usize); 5] = records(83); + const AT_103: [(usize, usize); 5] = records(103); + const AT_128: [(usize, usize); 5] = records(128); + const AT_133: [(usize, usize); 5] = records(133); + match version { + -4 => &[G1X3], + 1 => &[AT_78], + 2..=9 => &[AT_83], + 10 => &[AT_103, AT_128], + 11 => &[AT_128], + 12..=15 => &[AT_133], + _ => &[], + } +} + +/// The five preset records (Daylight, Shade, Cloudy, Tungsten, Flash) of a `ColorData` array whose Daylight record +/// starts at word `daylight` and Flash record at `flash`: Daylight, Shade, Cloudy and Tungsten follow each other +/// in 5-word records `R G G B Kelvin`. Also used by CR3 (`ColorData` words of the same record layout). +pub(crate) const fn preset_records(daylight: usize, flash: usize) -> [(usize, usize); 5] { + [(daylight, daylight + 4), (daylight + 5, daylight + 9), (daylight + 10, daylight + 14), (daylight + 15, daylight + 19), (flash, flash + 4)] } /// The camera's planckian white-balance presets from the `ColorData` words ([`preset_rows`]) as a Kelvin curve; /// `None` for an unknown version, a short array or presets that don't make a curve. fn kelvin_presets_from_color_data(v: &[u64]) -> Option { let version = *v.first()? as u16 as i16; - let rows = preset_rows(version)?; + let rows = match preset_rows(version) { + [one] => one, + many => many.iter().find(|rows| v.get(rows[0].1) == Some(&5200))?, + }; + presets_at(v, rows) +} + +/// The Kelvin curve of the preset records at `rows` ((levels word, temperature word) of each) in `v`. +pub(crate) fn presets_at(v: &[u64], rows: &[(usize, usize); 5]) -> Option { let mut points = Vec::with_capacity(rows.len()); - for (at, temp_at) in rows { + for &(at, temp_at) in rows { let q = v.get(at..at + 4)?; let temp = *v.get(temp_at)? as f32; if let Some(wb) = wb_from_levels(q[0] as f32, q[1] as f32, q[2] as f32, q[3] as f32) @@ -700,7 +731,7 @@ pub(crate) mod tests { fn color_data_with_presets(version: i16, presets: &[(u16, u16, u16)]) -> Vec { let mut v = vec![0u64; 400]; v[0] = u64::from(version as u16); - for ((at, temp_at), &(r, b, k)) in preset_rows(version).unwrap().into_iter().zip(presets) { + for ((at, temp_at), &(r, b, k)) in preset_rows(version).first().unwrap().iter().copied().zip(presets) { v[at..at + 4].copy_from_slice(&[r.into(), 1024, 1024, b.into()]); v[temp_at] = k.into(); } @@ -720,9 +751,20 @@ pub(crate) mod tests { want.sort_by(|a, b| a.0.total_cmp(&b.0)); assert_eq!(p.points(), want); let eos = [(2013, 1479, 5200), (2300, 1280, 7000), (2153, 1374, 6000), (1442, 2089, 3200), (2203, 1371, 6156)]; - for version in [13, 14, 15] { + for version in [1, 2, 3, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15] { assert!(kelvin_presets_from_color_data(&color_data_with_presets(version, &eos)).is_some(), "{version}"); } + // version 10 has two layouts: the one whose Daylight record says 5200 K is read + let mut b = vec![0u64; 400]; + b[0] = 10; + for (&(at, temp_at), &(r, bl, k)) in preset_rows(10)[1].iter().zip(&eos) { + b[at..at + 4].copy_from_slice(&[r.into(), 1024, 1024, bl.into()]); + b[temp_at] = k.into(); + } + assert!(kelvin_presets_from_color_data(&b).is_some(), "10, second layout"); + b[132] = 5000; + assert!(kelvin_presets_from_color_data(&b).is_none(), "10, neither layout names Daylight at 5200 K"); + assert!(kelvin_presets_from_color_data(&[10]).is_none()); // a preset that would turn the curve back (Flash cooler than Shade's levels say) is dropped, not fatal let mut odd = g1x3; odd[4] = (1500, 1300, 9000); @@ -753,7 +795,7 @@ pub(crate) mod tests { seed ^= seed << 13; seed ^= seed >> 7; seed ^= seed << 17; - if i == 0 { [0xfffc, 13, 14, 15][(seed % 4) as usize] } else { seed % 70000 } + if i == 0 { [0xfffc, 1, 10, 13, 14, 15][(seed % 6) as usize] } else { seed % 70000 } }) .collect(); let _ = kelvin_presets_from_color_data(&v); diff --git a/crates/raw/src/cr3.rs b/crates/raw/src/cr3.rs index d8f23f339..f9bf42a0b 100644 --- a/crates/raw/src/cr3.rs +++ b/crates/raw/src/cr3.rs @@ -120,6 +120,23 @@ struct CameraLevels { white: Option, } +/// The Kelvin scale of a body without colour matrices, from the presets in `ColorData` (see [`crate::cr2`] for the +/// record layout): Daylight, Shade, Cloudy, Tungsten and Flash at their stated temperatures. Word positions from +/// `exiftool -v3` run as a black-box tool on files of each version: versions 16-19 Daylight at word 136, 32 and 33 at +/// 150, 34 and 48 at 205 (each with Flash thirty words after Daylight), 64 to 66 at 110 with Flash twenty-five +/// after (no Auto / Measured / Kelvin records in these). Other versions give none. +fn kelvin_presets(maker: Option<&Ifd>) -> Option { + let v = maker?.u64s(COLOR_DATA)?; + let rows = match v.first().copied()? { + 16..=19 => crate::cr2::preset_records(136, 166), + 32 | 33 => crate::cr2::preset_records(150, 180), + 34 | 48 => crate::cr2::preset_records(205, 235), + 64..=66 => crate::cr2::preset_records(110, 135), + _ => return None, + }; + crate::cr2::presets_at(&v, &rows) +} + /// Canon's versioned word arrays. Offsets are word indices from the published tag tables, not /// byte offsets. The four as-shot WB values are R, G, G, B, independent of the CFA layout. /// A version without a white-level offset takes its white level from the data (see `docs/cr3.md`). @@ -253,7 +270,7 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode, limits: &Limits) -> Result| { + let b = IfdBuilder::new().with(COLOR_DATA, Value::Short(w)); + let bytes = craft_tiff::TiffWriter::new(craft_tiff::ByteOrder::Little, false).write(&[b]).unwrap(); + Tiff::parse(&bytes).unwrap().ifds.remove(0) + }; + let p = kelvin_presets(Some(&ifd(words.clone()))).unwrap_or_else(|| panic!("{version}")); + let pts = p.points(); + assert!( + pts.iter().any(|&(k, m)| k == 7000.0 && (m[0] - 2300.0 / 1024.0).abs() < 1e-6 && (m[2] - 1280.0 / 1024.0).abs() < 1e-6), + "{version}: {pts:?}" + ); + assert_eq!(pts.len(), 5, "{version}"); + assert!(kelvin_presets(Some(&ifd(words[..flash].to_vec()))).is_none(), "{version}: short"); + words[0] = 255; + assert!(kelvin_presets(Some(&ifd(words))).is_none(), "{version}: unknown"); + } + assert!(kelvin_presets(None).is_none()); + } + fn maker(version: u16, wb_at: usize, black_at: usize, white_at: usize) -> Ifd { let mut words = vec![0u16; white_at + 1]; words[0] = version; diff --git a/crates/raw/src/kelvin.rs b/crates/raw/src/kelvin.rs index 3048190d9..bf78ca8d3 100644 --- a/crates/raw/src/kelvin.rs +++ b/crates/raw/src/kelvin.rs @@ -19,6 +19,9 @@ pub struct KelvinPresets { /// Multipliers (green = 1) for Illuminant A and D65, when that pair is all the file records. #[serde(default)] pair: Option<[[f32; 3]; 2]>, + /// The pair's two lights as CIE 1931 (x, y), warmer first, when they are not Illuminant A and D65. + #[serde(default, skip_serializing_if = "Option::is_none")] + lights: Option<[[f64; 2]; 2]>, } fn usable(m: [f32; 3]) -> bool { @@ -40,7 +43,11 @@ impl KelvinPresets { let (a, b) = (log_point(w[0].1), log_point(w[1].1)); w[0].0 < w[1].0 && b[0] > a[0] && b[1] < a[1] }); - ok.then(|| KelvinPresets { points: points.into_iter().map(|(k, m)| (k, [m[0] / m[1], 1.0, m[2] / m[1]])).collect(), pair: None }) + ok.then(|| KelvinPresets { + points: points.into_iter().map(|(k, m)| (k, [m[0] / m[1], 1.0, m[2] / m[1]])).collect(), + pair: None, + lights: None, + }) } /// A curve from presets in any order, as a camera lists them by name (Canon's Daylight, Shade, Cloudy, …): @@ -72,7 +79,20 @@ impl KelvinPresets { } let (pa, pd) = (log_point(a), log_point(d65)); let unit = |m: [f32; 3]| [m[0] / m[1], 1.0, m[2] / m[1]]; - (pd[0] > pa[0] && pd[1] < pa[1]).then(|| KelvinPresets { points: Vec::new(), pair: Some([unit(a), unit(d65)]) }) + (pd[0] > pa[0] && pd[1] < pa[1]).then(|| KelvinPresets { points: Vec::new(), pair: Some([unit(a), unit(d65)]), lights: None }) + } + + /// [`Self::from_neutral_pair`] for two other lights, given by their CIE 1931 chromaticities, the warmer first + /// (Panasonic stores its multipliers for D55 and the ISO studio tungsten lamp). `None` unless the multipliers make + /// a curve as for a pair and the chromaticities are distinct points inside (0, 1). + pub fn from_neutral_lights(lights: [[f64; 2]; 2], warm: [f32; 3], cool: [f32; 3]) -> Option { + let inside = |p: [f64; 2]| p.iter().all(|v| *v > 0.0 && *v < 1.0); + if !inside(lights[0]) || !inside(lights[1]) || lights[0] == lights[1] { + return None; + } + let mut out = Self::from_neutral_pair(warm, cool)?; + out.lights = Some(lights); + Some(out) } /// The presets as (Kelvin, multipliers with green = 1), ascending in Kelvin. Empty for a neutral pair. @@ -84,6 +104,11 @@ impl KelvinPresets { pub fn neutral_pair(&self) -> Option<[[f32; 3]; 2]> { self.pair } + + /// The chromaticities of the pair's lights from [`Self::from_neutral_lights`]; `None` for Illuminant A and D65. + pub fn neutral_lights(&self) -> Option<[[f64; 2]; 2]> { + self.lights + } } #[cfg(test)] @@ -148,4 +173,17 @@ mod tests { None ); } + + #[test] + fn neutral_lights_keep_their_chromaticities() { + let (warm, cool) = ([1.0, 1.0, 2.8], [1.5, 1.0, 1.9]); + let lights = [[0.4232, 0.3991], [0.3324, 0.3474]]; + let k = KelvinPresets::from_neutral_lights(lights, warm, cool).unwrap(); + assert_eq!((k.neutral_pair(), k.neutral_lights()), (Some([warm, cool]), Some(lights))); + assert_eq!(KelvinPresets::from_neutral_pair(warm, cool).unwrap().neutral_lights(), None); + assert!(KelvinPresets::from_neutral_lights(lights, cool, warm).is_none()); + assert!(KelvinPresets::from_neutral_lights([lights[0], lights[0]], warm, cool).is_none()); + assert!(KelvinPresets::from_neutral_lights([[0.0, 0.4], lights[1]], warm, cool).is_none()); + assert!(KelvinPresets::from_neutral_lights([[f64::NAN, 0.4], lights[1]], warm, cool).is_none()); + } } diff --git a/crates/raw/src/pef.rs b/crates/raw/src/pef.rs index 1110f71c1..c7db4c969 100644 --- a/crates/raw/src/pef.rs +++ b/crates/raw/src/pef.rs @@ -27,6 +27,7 @@ const CROP_ORIGIN: u16 = 0x0038; const CROP_SIZE: u16 = 0x0039; const BLACK_POINT: u16 = 0x0200; const WB_LEVELS: u16 = 0x0201; +const KELVIN_WB: u16 = 0x0221; const HUFFMAN: u16 = 0x0220; /// Parse the maker-note Huffman table. @@ -122,6 +123,24 @@ fn dark_trimmed(d: &[u16], w: usize, h: usize, black: Option, white: f32) - Rect::new(left, top, width, height) } +/// The Kelvin presets of maker note `0x0221` (tag number and layout from `exiftool -v3` run as a black-box tool on +/// the files, then checked against the files' own bytes): a `u16`, then 8-byte entries of four `u16`, the first +/// the temperature as `53190 - K`, then 0, then the red and blue multipliers in units of 8192 (green 1). The first +/// entry is the daylight white (5205 K) and the rest run from 2500 K up. `None` unless they make a curve. +fn kelvin_presets(mn: Option<&makernote::MakerNote>) -> Option { + let mn = mn?; + let craft_tiff::Value::Undefined(b) = mn.ifd.value(KELVIN_WB)? else { return None }; + let words: Vec = b.as_chunks::<2>().0.iter().map(|c| mn.order.u16(*c)).collect(); + let points = words + .get(1..)? + .as_chunks::<4>() + .0 + .iter() + .filter(|e| e[0] != 0 && e[0] < 53190 && e[2] != 0 && e[3] != 0) + .map(|e| ((53190 - u32::from(e[0])) as f32, [f32::from(e[2]), 8192.0, f32::from(e[3])])); + crate::kelvin::KelvinPresets::from_unordered(points) +} + /// Whether every strip or tile of a one-sample-per-pixel image holds exactly the bytes that its rows need when /// the samples are packed MSB-first at the declared bit depth, rows starting on byte boundaries (TIFF 6.0 /// section 7). A real PackBits stream of such an image has a different length (almost always shorter), so the @@ -235,7 +254,7 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode, limits: &Limits) -> Result Result = words.iter().flat_map(|w| w.to_be_bytes()).collect(); + let ifd = IfdBuilder::new().with(KELVIN_WB, Value::Undefined(bytes)); + let file = TiffWriter::new(ByteOrder::Big, false).write(&[ifd]).unwrap(); + let ifd = Tiff::parse(&file).unwrap().ifds.remove(0); + makernote::MakerNote { kind: makernote::MakerNoteKind::Plain, order: ByteOrder::Big, base: 0, ifd } + }; + let p = kelvin_presets(Some(¬e(&words))).unwrap(); + let kelvins: Vec = p.points().iter().map(|p| p.0).collect(); + assert_eq!(kelvins, [2500.0, 3330.0, 4550.0, 5205.0, 6500.0]); + let daylight = p.points()[3].1; + assert!((daylight[0] - 1.83).abs() < 1e-3 && (daylight[2] - 1.16).abs() < 1e-3, "{daylight:?}"); + // no tag, or too few usable entries: none + assert!(kelvin_presets(None).is_none()); + assert!(kelvin_presets(Some(¬e(&words[..9]))).is_none()); + } + /// Our own prefix code for categories 0..=14: lengths 2,2,3,3,4,5,..., codes assigned canonically. fn test_table() -> (Vec, Vec<(u16, u8)>) { let lens: [u8; 15] = [3, 2, 2, 3, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 12]; diff --git a/crates/raw/src/rw2.rs b/crates/raw/src/rw2.rs index c0101e85e..13c5d0d92 100644 --- a/crates/raw/src/rw2.rs +++ b/crates/raw/src/rw2.rs @@ -122,6 +122,8 @@ const COMPRESSION: u16 = 0x000b; const LINEARITY_LIMIT: [u16; 3] = [0x000e, 0x000f, 0x0010]; const BLACK_RGB: [u16; 3] = [0x001c, 0x001d, 0x001e]; const WB_RGB: [u16; 3] = [0x0024, 0x0025, 0x0026]; +/// The table of the white balance types (see [`panasonic_kelvin_presets`]). +const WB_TABLE: u16 = 0x0027; const RAW_FORMAT: u16 = 0x002d; /// The in-camera aspect-ratio crop: top, left, bottom, right in sensor coordinates. const CROP: [u16; 4] = [0x002f, 0x0030, 0x0031, 0x0032]; @@ -341,6 +343,28 @@ fn decode_words(src: &[u8], shift: u32, out: &mut [u16]) { }); } +/// CIE daylight D55, CIE 1931 (x, y). +const D55_XY: [f64; 2] = [0.33242, 0.34743]; +/// The 3200 K Planckian light (the ISO 7589 studio tungsten lamp), CIE 1931 (x, y) from Kim et al.'s (2002) cubic +/// approximation of the Planckian locus. +const TUNGSTEN_3200_XY: [f64; 2] = [0.423_181_806_762_695_35, 0.399_077_685_165_656_9]; + +/// The Kelvin scale of a body without colour matrices, from tag `0x0027` (`WBInfo2`; tag number and entry layout +/// from `exiftool -v3` run as a black-box tool on the files): a count, then that many entries of the white +/// balance type and the camera's red, green and blue multipliers for it. Types 20 and 24 are the cameras' "D55" and +/// "ISO studio tungsten" whites: CIE daylight D55 (5500 K) and the ISO 7589 studio tungsten lamp, a 3200 K +/// Planckian light. Those two give a neutral pair ([`crate::kelvin::KelvinPresets::from_neutral_lights`]). +fn panasonic_kelvin_presets(ifd: &Ifd, order: ByteOrder) -> Option { + let v = u16_array(ifd, WB_TABLE, order)?; + let n = (*v.first()? as usize).min(v.len() / 4); + let entry = |code: u16| { + let e = v.get(1..)?.as_chunks::<4>().0.iter().take(n).find(|e| e[0] == code)?; + Some([f32::from(e[1]), f32::from(e[2]), f32::from(e[3])]) + }; + let (tungsten, d55) = (entry(24)?, entry(20)?); + crate::kelvin::KelvinPresets::from_neutral_lights([TUNGSTEN_3200_XY, D55_XY], tungsten, d55) +} + /// A tag as a `u16` array: SHORT values, or the little-endian contents of an UNDEFINED / BYTE value. fn u16_array(ifd: &Ifd, tag: u16, order: ByteOrder) -> Option> { match ifd.value(tag)? { @@ -669,7 +693,7 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode, limits: &Limits) -> Result>| { + let mut tags = tags.clone(); + tags.extend(table.map(|t| (WB_TABLE, Value::Short(t)))); + crate::decode(&file(w as u16, h as u16, &tags, &payload, &|off| vec![(RAW_OFFSET, Value::Long(vec![off]))])).unwrap() + }; + assert!(decode(None).color.kelvin_presets.is_none()); + // three entries: Fine Weather (ignored), D55 and ISO studio tungsten, as red/green/blue multipliers + let table = vec![3, 9, 480, 256, 400, 20, 380, 256, 500, 24, 250, 256, 700]; + let p = decode(Some(table)).color.kelvin_presets.unwrap(); + let unit = |r: f32, b: f32| [r / 256.0, 1.0, b / 256.0]; + assert_eq!(p.neutral_pair(), Some([unit(250.0, 700.0), unit(380.0, 500.0)])); + assert_eq!(p.neutral_lights(), Some([TUNGSTEN_3200_XY, D55_XY])); + // a missing light, or a count beyond the data, gives none + assert!(decode(Some(vec![1, 20, 500, 256, 380])).color.kelvin_presets.is_none()); + assert!(decode(Some(vec![9, 20, 500, 256, 380, 24, 700, 256])).color.kelvin_presets.is_none()); + } + #[test] fn format4_and_6_round_trip_through_files() { // format 4: 14 pixels per block, rotated chunks, blocks continuous across rows (here 70 = 5 blocks per row) From 2393b1f16fb1b5329fa37c02dea12f48dc9fd83a Mon Sep 17 00:00:00 2001 From: Phil Hart Date: Wed, 18 Nov 2026 15:03:26 -0500 Subject: [PATCH 135/161] craft-raw: decode two-field Coolpix NEFs (E5400, E5700 CYGM, E8400, E8800) (#2235) Ported from storytold/lightcraft#672, whose decoder code now lives here. Some CCD Coolpix bodies store two interlaced fields one after the other in one tight 12-bit strip. For the measured bodies, in their measured legacy layout (binary maker-note version 0.2.0.0, compression 1, one strip of w*h*1.5 bytes), the fields are interleaved in the model's field order. The E5700's CYGM mosaic becomes approximate linear RGB with fixed gains. Masked columns are judged on the stored rows, so the header probe and the full decode agree. Other files keep their stored rows. Checked on 527 NEF/NRW samples: only the six E5400/E5700/E8400/E8800 files (and four synthetic legacy-layout fixtures) change; the other 517 decode bit-identically. Claude-Session: https://claude.ai/code/session_01Fnx9MwPUug72xHRgg993uc Co-authored-by: Claude Opus 5.5 --- crates/raw/README.md | 2 +- crates/raw/src/coolpix.rs | 230 ++++++++++++++++++++++++++++++++++++++ crates/raw/src/lib.rs | 3 + crates/raw/src/nef.rs | 205 ++++++++++++++++++++++++++++++++- 4 files changed, 433 insertions(+), 7 deletions(-) create mode 100644 crates/raw/src/coolpix.rs diff --git a/crates/raw/README.md b/crates/raw/README.md index f0faf0670..d17dd62d7 100644 --- a/crates/raw/README.md +++ b/crates/raw/README.md @@ -54,7 +54,7 @@ One module and one Cargo feature per maker or container family. All are on by de | `jxl` | JPEG XL DNG (DNG 1.7, compression 52546) | Lossless and lossy tiles through `jxl-oxide`; implies `dng` | | `crw` | Canon CRW (CIFF) | Decoded by [`craft-crw`](../crw/README.md), the one CRW decoder, and returned through this crate's types | | `cr2` | Canon CR2 | The lossless-JPEG frame of IFD 3 de-sliced (`0xc640`); the layout from `CR2CFAPattern` (`0xc5e0`); image area from `SensorInfo`; black measured on the masked columns; white from the saturation plateau; as-shot white balance from `ColorData` or `WBInfo`; the white-balance presets at their temperatures (`ColorData::kelvin_presets`). sRAW / mRAW (subsampled YCbCr, 4:2:2 and 4:2:0) come out as linear white-balanced RGB with three samples per pixel, the chroma gains measured against the file's own 16-bit RGB image in IFD 2 | -| `nef` | Nikon NEF / NRW | Uncompressed (16-bit words, 12-bit packed, Coolpix word packing), the packed layouts labelled compression 34713, Nikon's Huffman coding (lossless and lossy types, 12 and 14 bits, "lossy after split") with the `0x0096` linearisation table, small raw (YCbCr 4:2:2) as linear white-balanced RGB; black from `0x003d` or the NRW block, as-shot white balance from `0x000c`, `CropArea` or the measured image area of bodies that record none, masked trailing columns left out of the active area, the D1X pixel aspect, and the in-file distortion correction as a `WarpRectilinear` opcode ([derivation](docs/nikon-lens-corrections.md)). High Efficiency (HE / HE★) is refused by name; the previews still read | +| `nef` | Nikon NEF / NRW | Uncompressed (16-bit words, 12-bit packed, Coolpix word packing), the two interlaced fields of the E5400, E5700, E8400 and E8800 (the E5700 CYGM mosaic as approximate linear RGB), the packed layouts labelled compression 34713, Nikon's Huffman coding (lossless and lossy types, 12 and 14 bits, "lossy after split") with the `0x0096` linearisation table, small raw (YCbCr 4:2:2) as linear white-balanced RGB; black from `0x003d` or the NRW block, as-shot white balance from `0x000c`, `CropArea` or the measured image area of bodies that record none, masked trailing columns left out of the active area, the D1X pixel aspect, and the in-file distortion correction as a `WarpRectilinear` opcode ([derivation](docs/nikon-lens-corrections.md)). High Efficiency (HE / HE★) is refused by name; the previews still read | | `rw2` | Panasonic RW2, Leica RWL, older Panasonic / Leica RAW | Packed blocks (formats 2, 5, 7), formats 4 and 6, format 8 prefix-coded strips and tiles (a sample outside the bit depth makes the strip corrupt), 16-bit words; black per colour from `0x001c`–`0x001e` (plus the 15 counts formats 4 and 6 store above it, 15 when no black is recorded); white from the saturation plateau; sensor borders as the active area and the in-camera aspect crop; mapped-out defect pixels filled; as-shot white balance from `0x0024`–`0x0026`; the in-camera distortion correction (`0x0119`) as a `WarpRectilinear` opcode. Raw formats 1 and 3 and compression 34826 are refused (no samples of them) | | `raf` | Fujifilm RAF | The RAF header, records and raw IFD; uncompressed Bayer and X-Trans (16-bit words, 12-bit LSB-first, 32-bit word packing); Fujifilm's lossless and lossy striped compression ([derivation](docs/raf-compression.md), [corpus](docs/raf-corpus.sha256)); black, as-shot white balance and, for bodies without colour matrices, the camera's neutral multipliers for Illuminant A and D65 (`ColorData::kelvin_presets`, a neutral pair); orientation, make and model from the embedded JPEG's Exif; the in-file distortion and vignetting corrections as opcodes; the embedded preview, also by range from the header alone (`bounded_preview`, `embedded_preview_from_range`). The pre-2010 FinePix layout without a raw IFD is refused | | `orf` | Olympus / OM System ORF | Uncompressed: 16-bit words (12-bit values in the top bits shifted down), 12-bit packed in 32-bit words (XZ-2), 12-bit two-field (older compacts), 16-byte blocks of ten 12-bit samples (E-300, E-330, E-500, high-resolution modes); CFA from Exif `CFAPattern`; black, as-shot white balance (relative to `WB_GLevel`) and crop from the maker note's ImageProcessing, or from the main note on bodies without it; the CameraSettings preview. Compressed ORF is refused ([why](docs/orf-compression.md)); its preview still reads | diff --git a/crates/raw/src/coolpix.rs b/crates/raw/src/coolpix.rs new file mode 100644 index 000000000..058a8ce58 --- /dev/null +++ b/crates/raw/src/coolpix.rs @@ -0,0 +1,230 @@ +//! Two-field Nikon Coolpix raws: E5700 (CYGM four-colour filter), E5400, E8400 and E8800 (RGB Bayer). +//! +//! The layout was measured clean-room by black-box analysis of the CC0 raw.pixls.us files (E5700 2576 x 1924, +//! E8400 and E8800 3280 x 2454), not taken from any other raw converter: +//! +//! * one strip of `w * h * 1.5` bytes: 12-bit samples packed MSB-first, two per three bytes, no row padding (the +//! ordinary uncompressed NEF reader unpacks it); +//! * the strip holds two interlaced fields of `h / 2` full-width rows, field A first and field B second. Interleaving +//! them line by line gives the sensor image. The E5700 puts B on the even lines (`out[2k] = B[k]`, `out[2k+1] = A[k]`), +//! the other bodies put A on the even lines. The order is a property of the model, not something the files state: the +//! `0x828e` colour-filter tag is anchored at line 0 of the interleaved image, so the wrong parity swaps the two rows +//! of every filter cell (a green cast on the CYGM file); +//! * black level 0, white level 4095, nothing masked; +//! * E5400, E8400 and E8800: `0x828e = [2, 1, 1, 0]`, an ordinary BGGR Bayer. E5700: `0x828e = [5, 3, 1, 4]`, i.e. Y C / G M. +//! +//! Field storage, like field order, is a property of the body: the Nikon reader rearranges rows only for the +//! models in [`field_order`], and only when the file has their measured legacy layout (binary maker-note version +//! 0.2.0.0, one tight 12-bit strip). Of 370 NEFs from 86 bodies checked, exactly these four bodies' files use that +//! maker-note version, and all four store fields. Other files, including any unmeasured body, keep their stored rows. +//! +//! CYGM to RGB is deliberately simple. Each of the four colour planes is bilinearly interpolated to full resolution, +//! scaled by fixed per-plane gains ([`CYGM_GAINS`]) and combined with the ideal complementary-colour relations +//! `Y = R + G, C = G + B, M = R + B`: `R = (Y + M - C) / 2`, `B = (C + M - Y) / 2` and `G` as the mean of the green +//! plane and `(Y + C - M) / 2`. A fitted 3 x 4 matrix was rejected: it was ill-conditioned on a nearly neutral scene, +//! and a free fit of the four gains against the camera JPEG turned the green gain negative. +//! Limits: no per-camera colour matrix (colours are approximate), the files' own white balance is not read (the gains +//! give a fixed daylight balance; the white-balance controls adjust from there), bilinear (soft, zippered at edges) +//! interpolation, negative results clamp to zero. + +/// The measured two-field bodies and whether the first stored field holds the even lines of the sensor image. +const FIELD_BODIES: [(&str, bool); 4] = [("E5400", true), ("E5700", false), ("E8400", true), ("E8800", true)]; + +/// For a measured two-field body, whether the first field of its strip (rows `0..h/2`) holds the even lines; +/// `None` for every other model. (The `0x828e` colour-filter tag is anchored at line 0 of the interleaved image, +/// so the wrong parity swaps the two rows of every filter cell.) +pub(crate) fn field_order(model: &str) -> Option { + let model = model.trim(); + FIELD_BODIES.iter().find(|(name, _)| *name == model).map(|&(_, first_even)| first_even) +} + +/// Interleave the two fields of `samples` (`h / 2` rows each, field A first) into a `w x h` image. +pub(crate) fn deinterlace(samples: &[u16], w: usize, h: usize, first_field_even: bool) -> Option> { + let n = w.checked_mul(h).filter(|n| *n <= crate::MAX_SAMPLES)?; + if w == 0 || h < 2 || !h.is_multiple_of(2) || samples.len() != n { + return None; + } + let (a, b) = samples.split_at_checked(n / 2)?; + let (even, odd) = if first_field_even { (a, b) } else { (b, a) }; + let mut out = Vec::with_capacity(n); + for (even_row, odd_row) in even.chunks_exact(w).zip(odd.chunks_exact(w)) { + out.extend_from_slice(even_row); + out.extend_from_slice(odd_row); + } + (out.len() == n).then_some(out) +} + +/// The colour-filter tag of the four-colour (CYGM) sensor: Y C / G M. +pub(crate) const CYGM_PATTERN: [u8; 4] = [5, 3, 1, 4]; + +const CODE_G: u8 = 1; +const CODE_C: u8 = 3; +const CODE_M: u8 = 4; +const CODE_Y: u8 = 5; + +/// Fixed E5700 plane gains, in the order Y, C, G, M. Measured on the CC0 raw.pixls.us E5700 file (the only E5700 +/// raw available) against its own camera JPEG (640 x 480, linearised): on the bins the camera rendered neutral +/// (channel spread under 8 %, mid-tone, unclipped; 215 of 19,200 at 160 x 120), the median per-bin gains that make +/// `Y = C = M = 2 G`, normalised to the weakest of Y, C, M. Over bin grids of 160 x 120 to 640 x 480 and neutrality +/// thresholds of 4 to 12 % the gains stay within 1.043-1.048, 0.613-0.616 and 1.265-1.271 (bootstrap 95 % intervals +/// within 0.006), and gains fitted on one half of the frame keep the other half's neutrals near neutral. +pub(crate) const CYGM_GAINS: [f32; 4] = [1.046, 1.0, 0.614, 1.268]; + +/// Output scale of the converted RGB. +const CYGM_SCALE: f32 = 4.0; + +/// White level of the converted RGB: a clipped neutral patch reaches 4095 in the weakest-gained of Y, C, M +/// (gain 1.0, C) and gives half of it in R, G, B. Fixed, so the header probe and the full decode report the same value. +pub(crate) const CYGM_WHITE: f32 = 4095.0 * 0.5 * CYGM_SCALE; + +/// Bilinear tap along one axis of a sub-sampled plane: (low index, high index, weight of the high one) for output +/// coordinate `c` of a plane of `n` samples starting at `phase`. +fn tap(n: usize, phase: usize, c: usize) -> (usize, usize, f32) { + let last = n.saturating_sub(1); + match c.checked_sub(phase) { + None => (0, 0, 0.0), + Some(d) => { + let i = d / 2; + (i.min(last), (i + 1).min(last), if d % 2 == 1 { 0.5 } else { 0.0 }) + } + } +} + +/// Linear-light RGB (interleaved, `u16`, white level [`CYGM_WHITE`]) from a CYGM mosaic. +/// +/// `pattern` is the `0x828e` tag, row-major over the 2 x 2 cell at (0, 0); it must hold each of Y, C, G, M once. +pub(crate) fn cygm_to_rgb(mosaic: &[u16], w: usize, h: usize, pattern: [u8; 4]) -> Option> { + use rayon::prelude::*; + let pixels = w.checked_mul(h)?; + let output_samples = pixels.checked_mul(3).filter(|n| *n <= crate::MAX_SAMPLES)?; + if w < 4 || h < 4 || !w.is_multiple_of(2) || !h.is_multiple_of(2) || mosaic.len() != pixels { + return None; + } + // phase (py * 2 + px) of each colour: [Y, C, G, M]. Four different codes found means four different phases. + let phase_of = |code: u8| pattern.iter().position(|&c| c == code); + let phases = [phase_of(CODE_Y)?, phase_of(CODE_C)?, phase_of(CODE_G)?, phase_of(CODE_M)?]; + let (pw, prows) = (w / 2, h / 2); + // the four sub-sampled planes, in colour order Y, C, G, M + let planes = phases.map(|p| { + let (py, px) = (p / 2, p % 2); + mosaic.chunks_exact(w).skip(py).step_by(2).flat_map(|row| row.iter().skip(px).step_by(2).map(|&v| f32::from(v))).collect::>() + }); + if planes.iter().any(|p| p.len() != pw * prows) { + return None; + } + + let mut out = vec![0u16; output_samples]; + out.par_chunks_mut(w * 3).enumerate().for_each(|(y, row)| { + for (x, px) in row.as_chunks_mut::<3>().0.iter_mut().enumerate() { + let mut v = [0f32; 4]; + for (((slot, plane), &p), &gain) in v.iter_mut().zip(&planes).zip(&phases).zip(&CYGM_GAINS) { + let (y0, y1, fy) = tap(prows, p / 2, y); + let (x0, x1, fx) = tap(pw, p % 2, x); + let at = |i: usize, j: usize| plane.get(i * pw + j).copied().unwrap_or(0.0); + let top = at(y0, x0) * (1.0 - fx) + at(y0, x1) * fx; + let bot = at(y1, x0) * (1.0 - fx) + at(y1, x1) * fx; + *slot = (top * (1.0 - fy) + bot * fy) * gain; + } + let [yy, cc, gg, mm] = v; + let r = (yy + mm - cc) * 0.5; + let g = (gg + (yy + cc - mm) * 0.5) * 0.5; + let b = (cc + mm - yy) * 0.5; + for (o, c) in px.iter_mut().zip([r, g, b]) { + *o = (c * CYGM_SCALE).round().clamp(0.0, 65535.0) as u16; + } + } + }); + Some(out) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn malformed_dimensions_are_refused_before_work_or_allocation() { + for (w, h) in [(usize::MAX, 2), (usize::MAX - 1, 64), (crate::MAX_SAMPLES, 4)] { + assert!(deinterlace(&[], w, h, true).is_none()); + assert!(cygm_to_rgb(&[], w, h, CYGM_PATTERN).is_none()); + } + } + + #[test] + fn two_field_round_trip_for_both_parities() { + let (w, h) = (6usize, 8usize); + // sensor image whose every line is distinct + let img: Vec = (0..w * h).map(|i| (i * 7 % 4096) as u16).collect(); + for first_even in [false, true] { + let lines: Vec<&[u16]> = img.chunks_exact(w).collect(); + // fields as stored: A = first h/2 rows, B = the rest + let evens = lines.iter().step_by(2); + let odds = lines.iter().skip(1).step_by(2); + let stored: Vec = if first_even { + evens.chain(odds).flat_map(|l| l.iter().copied()).collect() + } else { + odds.chain(evens).flat_map(|l| l.iter().copied()).collect() + }; + assert_eq!(deinterlace(&stored, w, h, first_even).unwrap(), img, "first_even = {first_even}"); + } + // the two parities differ + let s: Vec = (0..w * h).map(|i| i as u16).collect(); + assert_ne!(deinterlace(&s, w, h, true), deinterlace(&s, w, h, false)); + } + + #[test] + fn deinterlace_refuses_bad_shapes() { + assert!(deinterlace(&[0; 12], 3, 5, true).is_none()); + assert!(deinterlace(&[0; 11], 3, 4, true).is_none()); + assert!(deinterlace(&[], 0, 0, true).is_none()); + } + + #[test] + fn field_order_is_per_model_and_only_for_measured_bodies() { + assert_eq!(field_order("E5700"), Some(false)); + assert_eq!(field_order("E5700 "), Some(false)); + for model in ["E5400", "E8400", "E8800"] { + assert_eq!(field_order(model), Some(true), "{model}"); + } + for model in ["E5000", "E8700", "D1X", "", "E57000"] { + assert_eq!(field_order(model), None, "{model}"); + } + } + + #[test] + fn white_level_follows_the_weakest_gain() { + let weakest = CYGM_GAINS[0].min(CYGM_GAINS[1]).min(CYGM_GAINS[3]); + assert!((CYGM_WHITE - 4095.0 * weakest * 0.5 * CYGM_SCALE).abs() < 1e-3); + } + + #[test] + fn cygm_neutral_surface_stays_neutral() { + let (w, h) = (16usize, 12usize); + // a neutral surface of level v on this sensor: each plane reads v / gain (G at half level) + let [gy, gc, gg, gm] = CYGM_GAINS; + for v in [400.0f32, 2000.0, 3500.0] { + let at = |i: usize| match ((i / w) % 2, i % 2) { + (0, 0) => v / gy, + (0, _) => v / gc, + (_, 0) => v / 2.0 / gg, + _ => v / gm, + }; + let m: Vec = (0..w * h).map(|i| at(i).round() as u16).collect(); + let rgb = cygm_to_rgb(&m, w, h, CYGM_PATTERN).unwrap(); + for px in rgb.as_chunks::<3>().0 { + let (lo, hi) = (px.iter().min().copied().unwrap_or(0), px.iter().max().copied().unwrap_or(0)); + assert!(lo > 0 && f32::from(hi - lo) <= f32::from(hi) * 0.01, "{px:?} at {v}"); + } + } + } + + #[test] + fn cygm_refuses_bad_input_without_panicking() { + assert!(cygm_to_rgb(&[0; 10], 4, 4, [5, 3, 1, 4]).is_none()); + assert!(cygm_to_rgb(&[], 0, 0, [5, 3, 1, 4]).is_none()); + assert!(cygm_to_rgb(&[0; 16], 4, 4, [5, 3, 1, 1]).is_none()); + assert!(cygm_to_rgb(&[0; 16], 4, 4, [2, 1, 1, 0]).is_none()); + assert!(cygm_to_rgb(&[0; 15], 5, 3, [5, 3, 1, 4]).is_none()); + // all-black input is fine + assert!(cygm_to_rgb(&[0; 16], 4, 4, [5, 3, 1, 4]).is_some()); + } +} diff --git a/crates/raw/src/lib.rs b/crates/raw/src/lib.rs index 83a1165ca..e12fe78ba 100644 --- a/crates/raw/src/lib.rs +++ b/crates/raw/src/lib.rs @@ -93,6 +93,9 @@ mod arw6_testenc; mod arw_lens; #[cfg(feature = "arw")] mod arw_lens_rm6; +/// Two-field Coolpix NEFs: field order and the CYGM mosaic. +#[cfg(feature = "nef")] +mod coolpix; /// Bounded DNG strip and tile sample reader. #[cfg(feature = "line-dng")] pub mod dng_lines; diff --git a/crates/raw/src/nef.rs b/crates/raw/src/nef.rs index 047d1ce56..b15320ce5 100644 --- a/crates/raw/src/nef.rs +++ b/crates/raw/src/nef.rs @@ -12,10 +12,10 @@ //! is exactly three bytes per pixel, Nikon small raw ([`small_raw`]); the D1X's genuinely Huffman-coded strip stays //! refused. -use crate::nefc; use crate::shared::white_from_data_sourced; use crate::tiffraw::{Packing, check_image, read_image}; use crate::{BlackLevel, Camera, Cfa, ColorData, Limits, Mode, OpcodeLists, Orientation, RawData, RawError, RawFormat, RawImage, Rect, Result}; +use crate::{coolpix, nefc}; use craft_tiff::image::ImageInfo; use craft_tiff::tags::{self as t, photometric}; use craft_tiff::{ByteOrder, Ifd, Tiff, makernote}; @@ -237,6 +237,19 @@ pub(crate) fn trailing_masked_columns(d: &[u16], w: usize, bits: u32) -> usize { aw & !1 } +/// The measured two-field Coolpix files use the binary maker-note version 0.2.0.0, compression 1 and a single +/// tight 12-bit strip. Together with the model list in [`coolpix::field_order`] this scopes the rearrangement; +/// it is a measured discriminator, not a TIFF declaration of field storage. +fn legacy_field_layout(info: &ImageInfo, mn: Option<&makernote::MakerNote>, file_len: usize) -> bool { + let legacy = mn.and_then(|m| m.ifd.bytes(0x0001)) == Some(&[0, 2, 0, 0]); + let expected = (info.width as u64).checked_mul(info.height as u64).and_then(|n| n.checked_mul(3)).map(|n| n / 2); + legacy + && info.compression == 1 + && info.samples_per_pixel == 1 + && info.bits() == 12 + && matches!(info.chunks(file_len as u64).as_slice(), [c] if Some(c.len) == expected) +} + pub(crate) fn decode(bytes: &[u8], mode: Mode, limits: &Limits) -> Result { let tiff = Tiff::parse(bytes)?; let ifd0 = tiff.ifds.first().ok_or_else(|| RawError::Corrupt("TIFF without IFDs".into()))?; @@ -255,11 +268,41 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode, limits: &Limits) -> Result h, Mode::Header => h.min(MASK_ROWS), }; - let data = if info.compression == t::compression::NIKON { + let mut data = if info.compression == t::compression::NIKON { compressed(bytes, &info, mn.as_ref(), rows, limits)? } else { uncompressed(bytes, &info, tiff.order, mn.as_ref(), rows, limits)? }; + // Header probes inspect stored rows. Judge masked columns before rearranging the full + // image too, so the probe and full decoder use exactly the same prefix. + let stored_active_w = match &data { + RawData::U16(samples) => trailing_masked_columns(samples, w, bits), + _ => w, + }; + let model = ifd0.string(t::MODEL).unwrap_or_default(); + // Some CCD Coolpix bodies write two interlaced fields one after the other: the measured bodies, in their + // measured legacy layout, are interleaved in their model's field order (see `coolpix`). + if let (Mode::Full, RawData::U16(d)) = (mode, &mut data) + && let Some(first_even) = coolpix::field_order(&model) + && legacy_field_layout(&info, mn.as_ref(), bytes.len()) + && let Some(img) = coolpix::deinterlace(d, w, h, first_even) + { + *d = img; + } + // the E5700's four-colour (CYGM) filter becomes linear RGB (fixed measured gains, approximate colour) + let cygm = matches!( + (raw.u64s(t::CFA_REPEAT_PATTERN_DIM).as_deref(), raw.bytes(t::CFA_PATTERN_EP)), + (Some([2, 2]), Some(p)) if p == coolpix::CYGM_PATTERN + ) && info.compression != t::compression::NIKON + && info.samples_per_pixel == 1; + // Checked in every mode, so a CYGM NEF that `probe_info` accepts also decodes + if cygm && (w < 4 || h < 4 || !w.is_multiple_of(2) || !h.is_multiple_of(2)) { + return Err(RawError::Corrupt("Coolpix: bad CYGM mosaic".into())); + } + if cygm && let (Mode::Full, RawData::U16(d)) = (mode, &data) { + let rgb = coolpix::cygm_to_rgb(d, w, h, coolpix::CYGM_PATTERN).ok_or_else(|| RawError::Corrupt("Coolpix: bad CYGM mosaic".into()))?; + data = RawData::U16(rgb); + } let RawData::U16(ref samples) = data else { return Err(RawError::Unsupported("float NEF".into())) }; // Maker note 0x003d is in 14-bit units whatever the sample depth: 12-bit files from the D750, D780, D850, @@ -289,18 +332,21 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode, limits: &Limits) -> Result= 2 && v[0] > 0.1 && v[1] > 0.1 && v[0] < 10.0 && v[1] < 10.0) .map(|v| [v[0] as f32, 1.0, v[1] as f32]); + // the converted CYGM data is already balanced + let wb = if cygm { Some([1.0; 3]) } else { wb }; let (cfa, assumed_cfa) = match (raw.u64s(t::CFA_REPEAT_PATTERN_DIM).as_deref(), raw.bytes(t::CFA_PATTERN_EP)) { (Some([2, 2]), Some(p)) if p.len() == 4 && p.iter().all(|&c| c <= 2) => { (Cfa { width: 2, height: 2, pattern: p.to_vec(), origin_x: 0, origin_y: 0 }, false) } _ => (Cfa::bayer_static("RGGB"), true), }; - let model = ifd0.string(t::MODEL).unwrap_or_default(); let body = measured_body(&model, w, h, bits); - let active_w = trailing_masked_columns(samples, w, bits); + let active_w = stored_active_w; // the white level is measured from the whole image (or known for a measured body); the probe (no samples in its // result) takes the nominal one let (white, white_source) = match (body.and_then(|b| b.white), mode) { + // fixed, so the header probe and the full decode agree + _ if cygm => (coolpix::CYGM_WHITE, crate::LevelSource::Fallback), (Some(white), _) => (white, crate::LevelSource::Fallback), (None, Mode::Full) => white_from_data_sourced(samples, bits), (None, Mode::Header) => (((1u32 << bits.clamp(1, 16)) - 1) as f32, crate::LevelSource::Fallback), @@ -314,12 +360,12 @@ pub(crate) fn decode(bytes: &[u8], mode: Mode, limits: &Limits) -> Result data, Mode::Header => RawData::U16(Vec::new()), }, - cfa: Some(cfa), + cfa: if cygm { None } else { Some(cfa) }, bits, black, white: vec![white], @@ -1137,6 +1183,153 @@ mod tests { assert_eq!(decode(Some(both)).black.values, vec![100.0, 101.0, 102.0, 103.0]); } + /// A Coolpix-style file: model `model`, `pattern` as the colour-filter tag, one strip of 12-bit MSB-first samples. + fn coolpix_file(model: &str, pattern: [u8; 4], w: usize, h: usize, strip_px: &[u16]) -> Vec { + coolpix_file_version(model, pattern, w, h, strip_px, &[0, 2, 0, 0]) + } + + fn coolpix_file_version(model: &str, pattern: [u8; 4], w: usize, h: usize, strip_px: &[u16], version: &[u8]) -> Vec { + let mut packed = Vec::new(); + for pair in strip_px.chunks(2) { + let v = (pair[0] as u32) << 12 | pair.get(1).copied().unwrap_or(0) as u32; + packed.extend_from_slice(&[(v >> 16) as u8, (v >> 8) as u8, v as u8]); + } + let mut raw = IfdBuilder::new(); + raw.set(t::NEW_SUBFILE_TYPE, Value::Long(vec![0])); + raw.set(t::IMAGE_WIDTH, Value::Long(vec![w as u32])); + raw.set(t::IMAGE_LENGTH, Value::Long(vec![h as u32])); + raw.set(t::BITS_PER_SAMPLE, Value::Short(vec![12])); + raw.set(t::COMPRESSION, Value::Short(vec![1])); + raw.set(t::PHOTOMETRIC, Value::Short(vec![photometric::CFA])); + raw.set(t::CFA_REPEAT_PATTERN_DIM, Value::Short(vec![2, 2])); + raw.set(t::CFA_PATTERN_EP, Value::Byte(pattern.to_vec())); + raw.set_image(ImageData::Strips { rows_per_strip: h as u32, strips: vec![packed] }); + let mut ifd0 = IfdBuilder::new(); + ifd0.set(t::MAKE, Value::Ascii("NIKON".into())); + ifd0.set(t::MODEL, Value::Ascii(model.into())); + ifd0.set(t::ORIENTATION, Value::Short(vec![1])); + let mut note = IfdBuilder::new(); + note.set(0x0001, Value::Undefined(version.to_vec())); + let mut maker = b"Nikon\0\x02\0\0\0".to_vec(); + maker.extend(TiffWriter::new(ByteOrder::Little, false).write(&[note]).unwrap()); + let mut exif = IfdBuilder::new(); + exif.set(t::MAKER_NOTE, Value::Undefined(maker)); + ifd0.set_child(t::EXIF_IFD, exif); + ifd0.add_sub_ifd(raw); + TiffWriter::new(ByteOrder::Little, false).write(&[ifd0]).unwrap() + } + + /// The strip of a sensor image `img` with its even lines in the first (`first_even`) or the second field. + fn two_fields(img: &[u16], w: usize, h: usize, first_even: bool) -> Vec { + let lines = |start: usize| (start..h).step_by(2).flat_map(|y| img[y * w..(y + 1) * w].to_vec()).collect::>(); + let (even, odd) = (lines(0), lines(1)); + if first_even { [even, odd].concat() } else { [odd, even].concat() } + } + + /// A smooth scene with a little per-colour texture, sized like nothing in particular. + fn smooth_scene(w: usize, h: usize) -> Vec { + (0..w * h) + .map(|i| { + let (x, y) = ((i % w) as f64 / w as f64, (i / w) as f64 / h as f64); + (1800.0 + + 600.0 * (y * 6.0 * std::f64::consts::PI).sin() + + 300.0 * (x * 4.0 * std::f64::consts::PI).cos() * (y * 4.0 * std::f64::consts::PI).sin() + + 1800.0 * y + + (i % 2) as f64 * 40.0) as u16 + }) + .collect() + } + + #[test] + fn stacked_fields_are_interleaved_for_every_body_that_writes_them() { + let (w, h) = (96usize, 80usize); + let img = smooth_scene(w, h); + // E8400, E8800 and E5400 store the even lines first + for model in ["E8400", "E8800", "E5400"] { + let bytes = coolpix_file(model, [2, 1, 1, 0], w, h, &two_fields(&img, w, h, true)); + let r = crate::decode(&bytes).unwrap(); + assert_eq!((r.width, r.height, r.cpp), (w, h, 1), "{model}"); + assert_eq!(r.data, RawData::U16(img.clone()), "{model}"); + assert_eq!(r.cfa.as_ref().unwrap().name(), "BGGR", "{model}"); + } + } + + #[test] + fn other_models_and_layouts_keep_their_stored_rows() { + let (w, h) = (96usize, 80usize); + let stored = two_fields(&smooth_scene(w, h), w, h, true); + // an unmeasured model in the legacy layout + for model in ["E5000", "E99999", "D1X"] { + assert_eq!(crate::decode(&coolpix_file(model, [2, 1, 1, 0], w, h, &stored)).unwrap().data, RawData::U16(stored.clone()), "{model}"); + } + // a measured model outside its legacy maker-note dialect + for version in [b"0211".as_slice(), b"unknown".as_slice()] { + let bytes = coolpix_file_version("E8800", [2, 1, 1, 0], w, h, &stored, version); + assert_eq!(crate::decode(&bytes).unwrap().data, RawData::U16(stored.clone())); + } + // an odd height cannot hold two fields + let odd: Vec = (0..w * 79).map(|i| (i * 131 % 4096) as u16).collect(); + assert_eq!(crate::decode(&coolpix_file("E8400", [2, 1, 1, 0], w, 79, &odd)).unwrap().data, RawData::U16(odd)); + } + + #[test] + fn field_storage_does_not_depend_on_orientation() { + let (w, h) = (96usize, 80usize); + let img = smooth_scene(w, h); + let bytes = coolpix_file("E8400", [2, 1, 1, 0], w, h, &two_fields(&img, w, h, true)); + let tiff = Tiff::parse(&bytes).unwrap(); + let orientation = tiff.ifds.first().unwrap().get(t::ORIENTATION).unwrap().offset as usize; + for code in [3u16, 6, 8] { + let mut rotated = bytes.clone(); + rotated.get_mut(orientation..orientation + 2).unwrap().copy_from_slice(&code.to_le_bytes()); + let r = crate::decode(&rotated).unwrap(); + assert_eq!(r.orientation, Orientation::from_exif(code), "the rotated file reports orientation {code}"); + assert_eq!(r.data, RawData::U16(img.clone()), "orientation {code}"); + } + } + + #[test] + fn stacked_field_header_and_full_decode_agree() { + let (w, h) = (160usize, 256usize); + let mut img = smooth_scene(w, h); + for row in img.chunks_mut(w) { + row.iter_mut().skip(w - 8).for_each(|v| *v = 0); + } + for (model, pattern, first_even) in [("E8400", [2, 1, 1, 0], true), ("E5700", coolpix::CYGM_PATTERN, false)] { + let bytes = coolpix_file(model, pattern, w, h, &two_fields(&img, w, h, first_even)); + let header = decode(&bytes, Mode::Header, &Limits::default()).unwrap(); + let full = decode(&bytes, Mode::Full, &Limits::default()).unwrap(); + assert_eq!(header.info(), full.info(), "{model}"); + assert_eq!(header.black, full.black, "{model}"); + assert!(matches!(header.data, RawData::U16(ref d) if d.is_empty())); + if pattern == coolpix::CYGM_PATTERN { + assert_eq!((header.white.clone(), full.white.clone()), (vec![coolpix::CYGM_WHITE], vec![coolpix::CYGM_WHITE]), "{model}"); + } + } + } + + #[test] + fn coolpix_e5700_keeps_geometry_during_cygm_conversion() { + let (w, h) = (128usize, 96usize); + let img = smooth_scene(w, h); + // the E5700 stores the odd lines first + let r = crate::decode(&coolpix_file("E5700", [5, 3, 1, 4], w, h, &two_fields(&img, w, h, false))).unwrap(); + assert_eq!((r.width, r.height, r.cpp), (w, h, 3)); + assert!(r.cfa.is_none()); + let expected = coolpix::cygm_to_rgb(&img, w, h, coolpix::CYGM_PATTERN).unwrap(); + assert!(r.data == RawData::U16(expected), "full decode retains the intended sensor geometry"); + } + + #[test] + fn coolpix_short_input_is_an_error_not_a_panic() { + let (w, h) = (128usize, 96usize); + let mut file = coolpix_file("E5700", [5, 3, 1, 4], w, h, &vec![100u16; w * h]); + file.truncate(file.len() / 2); + let _ = crate::decode(&file); + file.truncate(200); + assert!(crate::decode(&file).is_err()); + } + #[test] fn compressed_without_table_is_unsupported() { let bytes = nef(34713, 14, vec![vec![0; 64]], 8, 8, 8); From 82fc745154898bc42c5454fa5918321f029f4ec6 Mon Sep 17 00:00:00 2001 From: Phil Hart Date: Wed, 18 Nov 2026 15:14:45 -0500 Subject: [PATCH 136/161] image-codecs: PNG bounds its image on the IDAT data and reserves only what it can make (#2231) * image-codecs: PNG bounds its image on the IDAT data and reserves only what it can make (lightcraft#3912) The plausibility check measured the stored samples against the whole file length, so ancillary chunks lifted the bound, and the decoder then reserved the whole expanded output up front. A 1-bit 20000x20000 image is 50 MB stored but 400 MB to 1.6 GB once expanded to 8-bit grey, RGB or RGBA, so about 60 KB of IDAT reserved gigabytes. - Bound the stored samples on the bytes of the first IDAT run (1032:1 plus 1 MiB), not the file length. - Reserve up front only what the IDAT data can inflate to; past that, grow the buffer as rows decode, at most doubling and never past the full output. A forged file fails on its first row; a real solid or sparse 1-bit image that is past the ratio once expanded still decodes. - Interlaced images decode into the whole zero-filled output in one go, so their expanded size is checked against the IDAT data. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Fnx9MwPUug72xHRgg993uc * Fix Adam7 palette expansion bounds without allocating unvalidated output --------- Co-authored-by: Claude Sonnet 5.5 Co-authored-by: Dylan M. Taylor --- crates/image-codecs/src/codecs/png.rs | 299 +++++++++++++++++++++++++- 1 file changed, 289 insertions(+), 10 deletions(-) diff --git a/crates/image-codecs/src/codecs/png.rs b/crates/image-codecs/src/codecs/png.rs index a244a1aae..d54d4149b 100644 --- a/crates/image-codecs/src/codecs/png.rs +++ b/crates/image-codecs/src/codecs/png.rs @@ -13,6 +13,52 @@ const F: Format = Format::Png; pub(crate) const XMP_KEYWORD: &str = "XML:com.adobe.xmp"; const METERS_PER_INCH: f32 = 0.0254; +/// Deflate's largest expansion: a 258-byte match coded in two bits, 1032 bytes out per byte in. +const MAX_DEFLATE_RATIO: u64 = 1032; + +/// The bytes of the file's first run of IDAT chunks, as much of each chunk as the file holds. The image is inflated +/// from these alone: ancillary chunks (text, ICC, an unused trailing run) hold no pixels, and the decoder refuses +/// an IDAT after other chunks. +fn idat_bytes(b: &[u8]) -> u64 { + let (mut p, mut total, mut in_run) = (8usize, 0u64, false); + while let (Some(len), Some(kind)) = (b.get(p..p.saturating_add(4)), b.get(p.saturating_add(4)..p.saturating_add(8))) { + let len = u32::from_be_bytes([len[0], len[1], len[2], len[3]]) as usize; + let data = p.saturating_add(8); + if kind == b"IDAT" { + in_run = true; + total = total.saturating_add(len.min(b.len().saturating_sub(data)) as u64); + } else if in_run { + break; + } + p = data.saturating_add(len).saturating_add(4); + } + total +} + +/// The most bytes `idat` bytes of zlib data can inflate to, with the 1 MiB of slack +/// [`Limits::check_plausible`] allows. +fn inflate_capacity(idat: u64) -> u64 { + idat.saturating_mul(MAX_DEFLATE_RATIO).saturating_add(1 << 20) +} + +/// Refuses `image_bytes` that `idat` bytes of zlib data cannot inflate to. +fn check_inflatable(image_bytes: u64, idat: u64) -> Result<(), CodecError> { + if image_bytes > inflate_capacity(idat) { + return Err(CodecError::LimitExceeded(format!("{idat} bytes of PNG image data cannot make {image_bytes} bytes of image"))); + } + Ok(()) +} + +// For expanded Adam7 output, the DEFLATE ratio applies only to packed samples. +// Validate pass rows without retaining the full image before allowing its allocation. +fn validate_adam7(bytes: &[u8], limits: &Limits) -> Result<(), CodecError> { + let mut decoder = png::Decoder::new_with_limits(Cursor::new(bytes), png::Limits { bytes: limits.alloc_usize() }); + decoder.set_ignore_text_chunk(true); + let mut reader = decoder.read_info().map_err(map_png_err)?; + while reader.next_interlaced_row().map_err(map_png_err)?.is_some() {} + Ok(()) +} + fn err(e: impl std::fmt::Display) -> CodecError { CodecError::malformed(F, e) } @@ -29,18 +75,31 @@ pub(crate) fn dpi_of_ppm(ppm: u32) -> f32 { /// Decodes a whole PNG, by driving a [`PngDecodeRows`] to its end. /// The stored size and EXIF block [`decode`] would find, from the chunks before the image data: /// the same header checks (limits, a file too small for its image) and the same frame (an fcTL -/// before the IDAT overrides the header's size), without inflating any pixels or text. +/// before the IDAT overrides the header's size), without inflating text. Unusually expanded Adam7 images validate packed pass rows first. pub(crate) fn probe(bytes: &[u8], limits: &Limits) -> Result<(u32, u32, Option>), CodecError> { let mut decoder = png::Decoder::new_with_limits(Cursor::new(bytes), png::Limits { bytes: limits.alloc_usize() }); + // Match the output layout used by decode. + decoder.set_transformations(png::Transformations::EXPAND); decoder.set_ignore_text_chunk(true); + let idat; { let info = decoder.read_header_info().map_err(map_png_err)?; let (w, h) = info.size(); limits.check_size(w, h)?; + idat = idat_bytes(bytes); let stored = u64::from(w).saturating_mul(u64::from(h)).saturating_mul(info.bits_per_pixel() as u64).div_ceil(8); - Limits::check_plausible(stored, bytes.len(), 1100)?; + check_inflatable(stored, idat)?; } let reader = decoder.read_info().map_err(map_png_err)?; + if reader.info().interlaced { + let size = reader.output_buffer_size().ok_or_else(|| err("image too large"))?; + if size > limits.alloc_usize() { + return Err(CodecError::LimitExceeded("PNG output exceeds max_alloc".into())); + } + if size as u64 > inflate_capacity(idat) { + validate_adam7(bytes, limits)?; + } + } let info = reader.info(); let (w, h) = info.frame_control.map_or_else(|| info.size(), |f| (f.width, f.height)); limits.check_size(w, h)?; @@ -158,8 +217,8 @@ pub struct PngDecodeRows> { } impl> PngDecodeRows { - /// Reads the header and refuses what [`decode`] refuses before any pixel is decoded: sizes - /// past `limits`, and a file too small for the image it declares. + /// Reads the header and rejects sizes past `limits` or implausibly small image data. + /// Exceptionally expanded Adam7 images also validate packed pass rows before allocation. pub fn new(bytes: B, limits: &Limits) -> Result { Self::open(bytes, limits, true, false) } @@ -175,16 +234,20 @@ impl> PngDecodeRows { // png's lazy text get_text() inflates without consulting its allocation limit. // Read text separately below, without retaining compressed copies in the decoder. decoder.set_ignore_text_chunk(true); + let idat; { let info = decoder.read_header_info().map_err(map_png_err)?; let (w, h) = info.size(); // Size checks now; the allocation budget once the output layout (channels and sample // size) is known below, so a Gray8 image isn't charged for RGBA16. limits.check_size(w, h)?; - // zlib never beats 1032:1; refuse a file too small for the declared image before the - // output buffer exists. Measured on the stored samples (before palette/bit expansion). + idat = idat_bytes(bytes.as_ref()); + // zlib never beats 1032:1; refuse a file whose IDAT data is too small for the declared + // image before the output buffer exists. Measured on the stored samples: palette, tRNS + // and 1, 2 or 4-bit expansion multiply them by up to 32 with no more data, so a sound + // bound on the expanded output is no tighter than this one. let stored = u64::from(w).saturating_mul(u64::from(h)).saturating_mul(info.bits_per_pixel() as u64).div_ceil(8); - Limits::check_plausible(stored, bytes.as_ref().len(), 1100)?; + check_inflatable(stored, idat)?; } let reader = decoder.read_info().map_err(map_png_err)?; // APNG: the IDAT image is the first frame, or an extra default image when no fcTL precedes it. @@ -223,11 +286,21 @@ impl> PngDecodeRows { if line.checked_mul(height).is_none_or(|needed| needed > size) { return Err(err("image too large")); } + if info.interlaced && size as u64 > inflate_capacity(idat) { + if size > limits.alloc_usize() { + return Err(CodecError::LimitExceeded("PNG output exceeds max_alloc".into())); + } + validate_adam7(bytes.as_ref(), limits)?; + } // Reserve without touching: each row is zero-filled as it is decoded, so the first slice - // doesn't zero-fill the whole image (a stall on the web build). + // doesn't zero-fill the whole image (a stall on the web build). Only as much as the IDAT + // data can inflate to: a 1-bit image expands up to 32 times, so a few dozen kilobytes may + // declare gigabytes of output. A real one (a solid or sparse 1-bit image) grows the buffer + // as its rows decode; a forged one fails on its first rows without the reservation. let mut buf = Vec::new(); if reserve { - buf.try_reserve_exact(size).map_err(|_| CodecError::LimitExceeded(format!("cannot allocate {size} bytes for the image")))?; + let first = size.min(usize::try_from(inflate_capacity(idat)).unwrap_or(usize::MAX)); + buf.try_reserve_exact(first).map_err(|_| CodecError::LimitExceeded(format!("cannot allocate {first} bytes for the image")))?; } let interlaced = info.interlaced; Ok(Self { @@ -344,8 +417,15 @@ impl> PngDecodeRows { let at = self.rows.checked_mul(self.line).ok_or_else(|| err("image too large"))?; let end = at.checked_add(self.line).ok_or_else(|| err("image too large"))?; // Grow by one row at a time: the zero-fill of the output is spread over the decodes - // instead of stalling the first slice. + // instead of stalling the first slice. Past what was reserved, the capacity at most + // doubles and never passes the whole output. if self.buf.len() < end { + if end > self.buf.capacity() { + let want = self.buf.capacity().saturating_mul(2).clamp(end, self.size.max(end)); + self.buf + .try_reserve_exact(want - self.buf.len()) + .map_err(|_| CodecError::LimitExceeded(format!("cannot allocate {want} bytes for the image")))?; + } self.buf.resize(end, 0); } let row = self.buf.get_mut(at..end).ok_or_else(|| err("the image data is longer than the image"))?; @@ -2433,3 +2513,202 @@ mod rows_tests { assert!(matches!(decode(&png_with(&bad_itxt(100_000).repeat(20))), Err(CodecError::LimitExceeded(_)))); } } + +#[cfg(test)] +mod inflate_bound_tests { + use super::*; + + fn chunk(out: &mut Vec, kind: &[u8; 4], data: &[u8]) { + out.extend_from_slice(&(data.len() as u32).to_be_bytes()); + out.extend_from_slice(kind); + out.extend_from_slice(data); + let mut crc = flate2::Crc::new(); + crc.update(kind); + crc.update(data); + out.extend_from_slice(&crc.sum().to_be_bytes()); + } + + /// A 20000×20000 PNG of `depth` bits and colour type `color`, interlaced or not, with optional PLTE and tRNS, + /// `text` bytes of tEXt and then an IDAT of `idat` bytes that is not a deflate stream (its first block has the + /// reserved type), so it fails on the first row it is asked for. + fn forged(depth: u8, color: u8, trns: bool, interlace: u8, idat: usize, text: usize) -> Vec { + let mut out = b"\x89PNG\r\n\x1a\n".to_vec(); + let mut ihdr = Vec::new(); + ihdr.extend(20000u32.to_be_bytes()); + ihdr.extend(20000u32.to_be_bytes()); + ihdr.extend([depth, color, 0, 0, interlace]); + chunk(&mut out, b"IHDR", &ihdr); + if color == 3 { + chunk(&mut out, b"PLTE", &[0, 0, 0, 255, 255, 255]); + } + if trns { + chunk(&mut out, b"tRNS", if color == 3 { &[0, 255] } else { &[0, 0] }); + } + if text > 0 { + let mut t = b"Comment\0".to_vec(); + t.resize(text, b'x'); + chunk(&mut out, b"tEXt", &t); + } + let mut data = vec![0x78, 0x01]; + data.resize(idat.max(2), 0xFF); + chunk(&mut out, b"IDAT", &data); + chunk(&mut out, b"IEND", &[]); + out + } + + /// A budget that lets every case below be charged in full: only the inflate bound refuses them. + fn roomy() -> Limits { + Limits { max_width: u32::MAX, max_height: u32::MAX, max_pixels: 1 << 30, max_alloc: 8 << 30 } + } + + fn refused_by_the_bound(bytes: &[u8]) -> bool { + let bound = |r: Result<(), CodecError>| matches!(r, Err(CodecError::LimitExceeded(m)) if m.contains("cannot make")); + let decode = bound(decode(bytes, &roomy()).map(drop)); + let probe = bound(probe(bytes, &roomy()).map(drop)); + let rows = bound(PngDecodeRows::new(bytes, &roomy()).and_then(|mut d| d.reserve_output())); + decode && probe && rows + } + + /// storytold/lightcraft#3912: a 1-bit 20000×20000 image is 50 MB stored, which 60 KB of IDAT data can inflate + /// to, but the decoder expands it to 8-bit grey (400 MB), RGB (1.2 GB) or, with tRNS, RGBA (1.6 GB). The + /// decoder reserves only what the IDAT data can make, so the forged file fails on its first row without the + /// gigabyte reservation. + #[test] + fn a_forged_one_bit_image_reserves_only_what_its_data_can_make() { + for (color, trns) in [(3, true), (3, false), (0, true), (0, false)] { + let bytes = forged(1, color, trns, 0, 60_000, 0); + assert!(bytes.len() < 61_000); + let rows = PngDecodeRows::new(bytes.as_slice(), &roomy()).unwrap(); + assert!(rows.size >= 400_000_000, "colour type {color}, tRNS {trns}: {} bytes", rows.size); + assert!(rows.buf.capacity() as u64 <= inflate_capacity(60_000), "colour type {color}, tRNS {trns}"); + drop(rows); + assert!(matches!(decode(&bytes, &roomy()), Err(CodecError::Malformed { .. })), "colour type {color}, tRNS {trns}"); + } + } + + /// Invalid pass data fails before the expanded Adam7 image is allocated. + #[test] + fn a_forged_interlaced_one_bit_image_is_refused_before_full_allocation() { + for (color, trns) in [(3, true), (0, false)] { + let bytes = forged(1, color, trns, 1, 60_000, 0); + assert!(matches!(probe(&bytes, &roomy()), Err(CodecError::Malformed { .. }))); + assert!(matches!(PngDecodeRows::new(bytes.as_slice(), &roomy()), Err(CodecError::Malformed { .. }))); + } + } + + /// The pixels come from the IDAT chunks only: a 60 KB text chunk doesn't lift the bound for an image with an + /// almost empty IDAT, though the file as a whole would hold the stored samples. + #[test] + fn ancillary_bytes_do_not_lift_the_inflate_bound() { + let bytes = forged(1, 0, false, 0, 16, 60_000); + assert!(Limits::check_plausible(20000 * 20000 / 8, bytes.len(), 1100).is_ok()); + assert!(refused_by_the_bound(&bytes)); + } + + /// A 2000×2000 grey image split over IDAT chunks of 64 bytes counts them all. + #[test] + fn the_inflate_bound_counts_every_idat_chunk() { + let (w, h) = (2000u32, 2000u32); + let mut whole = Vec::new(); + { + let mut enc = png::Encoder::new(&mut whole, w, h); + enc.set_color(png::ColorType::Grayscale); + enc.set_depth(png::BitDepth::Eight); + enc.set_compression(png::Compression::High); + let mut wr = enc.write_header().unwrap(); + wr.write_image_data(&vec![0u8; (w * h) as usize]).unwrap(); + } + let (mut p, mut zlib) = (8, Vec::new()); + while p + 8 <= whole.len() { + let len = u32::from_be_bytes([whole[p], whole[p + 1], whole[p + 2], whole[p + 3]]) as usize; + if &whole[p + 4..p + 8] == b"IDAT" { + zlib.extend_from_slice(&whole[p + 8..p + 8 + len]); + } + p += len + 12; + } + let mut split = b"\x89PNG\r\n\x1a\n".to_vec(); + let mut ihdr = Vec::new(); + ihdr.extend(w.to_be_bytes()); + ihdr.extend(h.to_be_bytes()); + ihdr.extend([8, 0, 0, 0, 0]); + chunk(&mut split, b"IHDR", &ihdr); + for piece in zlib.chunks(64) { + chunk(&mut split, b"IDAT", piece); + } + chunk(&mut split, b"IEND", &[]); + assert_eq!(idat_bytes(&split), zlib.len() as u64); + assert_eq!(decode(&split, &Limits::default()).unwrap().dimensions(), (w, h)); + } + + /// A real 1-bit palette image with transparency whose RGBA output is far more than 1032 times its IDAT data + /// (a solid 2000×2000 sheet: 16 MB from a few hundred bytes) still decodes, growing its buffer as the rows come. + /// A bound on the expanded output would refuse it. + #[test] + fn solid_adam7_palette_expansion_is_not_a_deflate_ratio() { + let (w, h) = (2048u32, 2048u32); + let mut raw = Vec::new(); + for (x, y, dx, dy) in [(0, 0, 8, 8), (4, 0, 8, 8), (0, 4, 4, 8), (2, 0, 4, 4), (0, 2, 2, 4), (1, 0, 2, 2), (0, 1, 1, 2)] { + let width = (w - x).div_ceil(dx); + for _ in 0..(h - y).div_ceil(dy) { + raw.push(0); + raw.extend(std::iter::repeat_n(0xff, width.div_ceil(8) as usize)); + } + } + let mut z = flate2::write::ZlibEncoder::new(Vec::new(), flate2::Compression::best()); + z.write_all(&raw).unwrap(); + let packed = z.finish().unwrap(); + let mut bytes = b"\x89PNG\r\n\x1a\n".to_vec(); + let mut header = Vec::new(); + header.extend(w.to_be_bytes()); + header.extend(h.to_be_bytes()); + header.extend([1, 3, 0, 0, 1]); + chunk(&mut bytes, b"IHDR", &header); + chunk(&mut bytes, b"PLTE", &[0, 0, 0, 10, 20, 30]); + chunk(&mut bytes, b"tRNS", &[0, 200]); + chunk(&mut bytes, b"IDAT", &packed); + chunk(&mut bytes, b"IEND", &[]); + assert!(u64::from(w) * u64::from(h) * 4 > inflate_capacity(idat_bytes(&bytes))); + assert_eq!(probe(&bytes, &Limits::default()).unwrap().0, w); + let image = decode(&bytes, &Limits::default()).unwrap(); + assert_eq!(image.dimensions(), (w, h)); + assert!(image.data().as_chunks::<4>().0.iter().all(|p| *p == [10, 20, 30, 200])); + let small = Limits { max_alloc: 1 << 20, ..Limits::default() }; + assert!(matches!(decode(&bytes, &small), Err(CodecError::LimitExceeded(_)))); + } + + #[test] + fn a_solid_one_bit_image_past_the_ratio_once_expanded_still_decodes() { + let (w, h) = (2000u32, 2000u32); + let mut bytes = Vec::new(); + { + let mut enc = png::Encoder::new(&mut bytes, w, h); + enc.set_color(png::ColorType::Indexed); + enc.set_depth(png::BitDepth::One); + enc.set_palette(vec![0, 0, 0, 10, 20, 30]); + enc.set_trns(vec![0, 200]); + enc.set_compression(png::Compression::High); + let mut wr = enc.write_header().unwrap(); + wr.write_image_data(&vec![0xFFu8; (w as usize / 8) * h as usize]).unwrap(); + } + let rows = PngDecodeRows::new(bytes.as_slice(), &Limits::default()).unwrap(); + assert!(rows.size as u64 > inflate_capacity(idat_bytes(&bytes)), "{} bytes from {} of IDAT", rows.size, idat_bytes(&bytes)); + assert!(rows.buf.capacity() < rows.size); + drop(rows); + let img = decode(&bytes, &Limits::default()).unwrap(); + assert_eq!((img.dimensions(), img.layout()), ((w, h), ChannelLayout::Rgba)); + assert!(img.data().as_chunks::<4>().0.iter().all(|p| *p == [10, 20, 30, 200])); + + let mut small = Vec::new(); + { + let mut enc = png::Encoder::new(&mut small, 64, 64); + enc.set_color(png::ColorType::Indexed); + enc.set_depth(png::BitDepth::One); + enc.set_palette(vec![0, 0, 0, 255, 255, 255]); + enc.set_trns(vec![0, 255]); + let mut wr = enc.write_header().unwrap(); + wr.write_image_data(&[0b1010_1010u8; 8 * 64]).unwrap(); + } + let img = decode(&small, &Limits::default()).unwrap(); + assert_eq!((img.dimensions(), img.layout()), ((64, 64), ChannelLayout::Rgba)); + } +} From bd5c93cf50b9280b59d1d8210bfc6b7df6dc0507 Mon Sep 17 00:00:00 2001 From: Hamza Naveed <45464935+Hamza0101@users.noreply.github.com> Date: Thu, 19 Nov 2026 01:23:21 +0500 Subject: [PATCH 137/161] craft-odf: taking inline items out of a flat document is linear (#2219) (#2237) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit take_out searched the whole rest of the document for the next picture and the next inline object on every item, so a document with many of one kind and none (or few) of the other cost items × size. It now keeps where the next of each is and searches for a kind again only once it has been passed. The output is the same. --- crates/odf/src/flat.rs | 60 ++++++++++++++++++++++++++++++++++------- crates/odf/src/tests.rs | 22 +++++++++++++++ 2 files changed, 72 insertions(+), 10 deletions(-) diff --git a/crates/odf/src/flat.rs b/crates/odf/src/flat.rs index d71fbffe9..a5b0fcf11 100644 --- a/crates/odf/src/flat.rs +++ b/crates/odf/src/flat.rs @@ -116,34 +116,74 @@ fn inline_object(xml: &str) -> Option { None } +const PICTURE: &str = ""; + +#[cfg(test)] +thread_local! { + /// Bytes the searches for inline pictures and objects looked through on this thread (tests: + /// the work is linear in the document, #2219). + pub(crate) static SCANNED: std::cell::Cell = const { std::cell::Cell::new(0) }; +} + +/// Notes that a search looked through `xml[from..to]` (counted in tests only). +fn scanned(from: usize, to: usize) { + #[cfg(test)] + SCANNED.with(|s| s.set(s.get().saturating_add(to.saturating_sub(from)))); + #[cfg(not(test))] + let _ = (from, to); +} + +/// Where the next inline picture's data (``) starts at or after `from`. +fn next_picture(xml: &str, from: usize) -> Option { + let found = xml.get(from..).and_then(|r| r.find(PICTURE)).map(|i| from + i); + scanned(from, found.map_or(xml.len(), |i| i + PICTURE.len())); + found +} + +/// Where the next inline object ([`inline_object`]) starts at or after `from`. +fn next_object(xml: &str, from: usize) -> Option { + let found = xml.get(from..).and_then(inline_object).map(|i| from + i); + scanned(from, found.map_or(xml.len(), |i| i + ")>, folders: &mut Vec<(String, String)>, max_entry: u64) -> String { let mut out = String::with_capacity(xml.len()); - let mut rest = xml; + // How far `xml` has been copied or taken out. + let mut pos = 0; + // The next picture and the next inline object, each searched for again only once `pos` has + // passed it: searching the whole rest for both on every pass made a document with many of + // one and none of the other quadratic (#2219). + let (mut pic, mut obj) = (next_picture(xml, 0), next_object(xml, 0)); loop { - let pic = rest.find(""); - let obj = inline_object(rest); + if pic.is_some_and(|p| p < pos) { + pic = next_picture(xml, pos); + } + if obj.is_some_and(|o| o < pos) { + obj = next_object(xml, pos); + } let (at, picture) = match (pic, obj) { (Some(p), Some(o)) if o < p => (o, false), (Some(p), _) => (p, true), (None, Some(o)) => (o, false), (None, None) => break, }; - let (before, from) = rest.split_at(at); + let (Some(before), Some(from)) = (xml.get(pos..at), xml.get(at..)) else { break }; let close = if picture { "" } else { "" }; let Some(end) = from.find(close) else { out.push_str(before); - rest = from; + pos = at; break; }; - let open_len = if picture { "".len() } else { from.find('>').map_or(0, |e| e + 1) }; + let open_len = if picture { PICTURE.len() } else { from.find('>').map_or(0, |e| e + 1) }; let body = from.get(open_len..end).unwrap_or(""); - let after = from.get(end + close.len()..).unwrap_or(""); + let after = (at + end + close.len()).min(xml.len()); let n = files.len() + 1; if files.len() >= MAX_INLINE { out.push_str(before); - rest = after; + pos = after; continue; } if picture { @@ -187,9 +227,9 @@ fn take_out(xml: &str, files: &mut Vec<(String, Vec)>, folders: &mut Vec<(St None => files.push((format!("{dir}/content.xml"), body.as_bytes().to_vec())), } } - rest = after; + pos = after; } - out.push_str(rest); + out.push_str(xml.get(pos..).unwrap_or("")); out } diff --git a/crates/odf/src/tests.rs b/crates/odf/src/tests.rs index dd144ea61..e4faf4b1c 100644 --- a/crates/odf/src/tests.rs +++ b/crates/odf/src/tests.rs @@ -369,6 +369,28 @@ fn hostile_flat_documents_do_not_panic() { assert_eq!(flat::from_package(&zip_of(&[("content.xml", b"")]), &Limits::DEFAULT), Err(Error::Flat("the package has no media type"))); } +/// #2219: taking the inline objects and pictures out of a flat document looks through it a +/// bounded number of times, not once per item. A document of only objects (or only pictures) +/// searched its whole rest for the other kind on every item. +#[test] +fn taking_inline_items_out_is_linear() { + let cost = |item: &str, n: usize| { + let xml = format!("{}", item.repeat(n)); + flat::SCANNED.with(|s| s.set(0)); + let doc = flat::split(xml.as_bytes(), &Limits::DEFAULT).unwrap(); + (flat::SCANNED.with(|s| s.get()), doc.files.len(), xml.len()) + }; + let object = ""; + let picture = "iVBORw0KGgo="; + for item in [object, picture] { + let (small, files, _) = cost(item, 1_000); + let (large, more_files, len) = cost(item, 2_000); + assert_eq!((files, more_files), (1_000, 2_000), "every item is taken out"); + assert!(large <= 3 * len, "{large} bytes searched in a document of {len}"); + assert!(large < 3 * small, "twice the items, {small} then {large} bytes searched"); + } +} + /// Two files of one name: `zip` keeps one, and which one the writer meant is not known. #[test] fn two_files_of_one_name_are_refused() { From 0d8d8b90914552388be5378251aa2ea656655cf3 Mon Sep 17 00:00:00 2001 From: DJHoltkamp Date: Wed, 18 Nov 2026 15:24:26 -0500 Subject: [PATCH 138/161] craft-image-codecs: refuse WebP animation frames that would panic image-webp (#2263) For an animation frame that starts with an `ALPH` chunk, image-webp 0.2.4 decodes the next chunk as VP8 whatever its FourCC, skips the size check it makes for frames without alpha, and indexes the frame's buffers by the picture's own size. A picture larger than its `ANMF` header, or a non-key frame (decoded as 0x0), panicked with out-of-bounds indexing in `read_frame` and `fill_rgba`. image-webp also steps from one frame to the next without the padding byte, so after an odd-sized `ANMF` chunk it reads the next frame one byte early. Before decoding, the container is walked for the first frame with alpha whose picture is not a VP8 key frame of its header's size, and for the first odd-sized `ANMF` chunk. That frame is refused like a damaged frame: `decode` (which reads frame 0) errors only when it is the first, and `decode_webp_frames` keeps the frames before it with a `Truncated` warning. Found by fuzzing `decode_webp`; ported from storytold/photocraft#10473, which was written before PhotoCraft's codecs moved here. The regression test panics without the fix. Co-authored-by: DJHoltkamp <4955429+DJHoltkamp@users.noreply.github.com> --- crates/image-codecs/src/codecs/webp.rs | 62 ++++++++++++++- crates/image-codecs/tests/it/webp_frames.rs | 88 +++++++++++++++++++++ 2 files changed, 147 insertions(+), 3 deletions(-) diff --git a/crates/image-codecs/src/codecs/webp.rs b/crates/image-codecs/src/codecs/webp.rs index ab2ffc95e..b9ac1b6bd 100644 --- a/crates/image-codecs/src/codecs/webp.rs +++ b/crates/image-codecs/src/codecs/webp.rs @@ -32,8 +32,62 @@ fn open<'a>(bytes: &'a [u8], limits: &Limits) -> Result<(Decoder<'a>, u32, u32, Ok((dec, w, h, layout)) } +const INCONSISTENT_FRAME: &str = "an animation frame's picture is not the size its frame header gives"; + +/// The chunks of a RIFF body as (FourCC, payload); a truncated last chunk keeps what is there. +fn chunks(mut body: &[u8]) -> impl Iterator { + std::iter::from_fn(move || { + let id = body.get(..4)?; + let size = usize::try_from(u32::from_le_bytes(body.get(4..8)?.try_into().ok()?)).ok()?; + let end = size.checked_add(8)?; + let payload = body.get(8..end.min(body.len()))?; + body = body.get(end.checked_add(size % 2)?..).unwrap_or_default(); + Some((id, payload)) + }) +} + +/// The 24-bit little-endian value at `at`, plus one (how `ANMF` stores a frame's size). +fn size24(b: &[u8], at: usize) -> Option { + let &[x, y, z] = b.get(at..at.checked_add(3)?)? else { return None }; + Some(u32::from_le_bytes([x, y, z, 0]) + 1) +} + +/// The width and height in a VP8 key frame's header (RFC 6386 §9.1), or `None` for anything else. +fn vp8_size(frame: &[u8]) -> Option<(u32, u32)> { + let &[tag, _, _, 0x9D, 0x01, 0x2A, w0, w1, h0, h1] = frame.get(..10)? else { return None }; + (tag & 1 == 0).then(|| (u32::from(u16::from_le_bytes([w0, w1]) & 0x3FFF), u32::from(u16::from_le_bytes([h0, h1]) & 0x3FFF))) +} + +/// The index of the first animation frame with alpha whose picture is not a VP8 key frame of the +/// size its `ANMF` header gives. For a frame that starts with an `ALPH` chunk, `image-webp` 0.2.4 +/// decodes the next chunk as VP8 whatever its FourCC, skips the size check it makes for frames +/// without alpha, and indexes the frame's buffers by the picture's own size (0x0 for a non-key +/// frame): it panics (found by the `decode_webp` fuzz target), so the frame is refused first. +/// An odd-sized `ANMF` chunk is refused too: a well-formed one is always even (a 16-byte header +/// and padded chunks), and `image-webp` steps to the next frame without the padding byte, so the +/// frames after it would not be the ones checked here. +fn first_inconsistent_frame(bytes: &[u8]) -> Option { + let frames = chunks(bytes.get(12..)?).filter(|(id, _)| *id == b"ANMF"); + frames.zip(0u32..).find_map(|((_, frame), i)| { + if frame.len() % 2 == 1 { + return Some(i); + } + let header = (size24(frame, 6)?, size24(frame, 9)?); + let mut inner = chunks(frame.get(16..)?); + if inner.next()?.0 != b"ALPH" { + return None; + } + let picture = inner.next().map_or(&[][..], |(_, p)| p); + (vp8_size(picture) != Some(header)).then_some(i) + }) +} + pub(crate) fn decode(bytes: &[u8], limits: &Limits) -> Result { let (mut dec, w, h, layout) = open(bytes, limits)?; + // An animation decodes as its first frame. + if dec.is_animated() && first_inconsistent_frame(bytes) == Some(0) { + return Err(err(INCONSISTENT_FRAME)); + } let size = dec.output_buffer_size().ok_or_else(|| err("image too large"))?; let mut buf = alloc_zeroed::(size)?; dec.read_image(&mut buf).map_err(err)?; @@ -81,7 +135,8 @@ pub(crate) fn decode_frames(bytes: &[u8], limits: &Limits) -> Result(size)?; let mut out: Vec = Vec::new(); let mut damaged = false; - for _ in 0..dec.num_frames() { + let inconsistent = first_inconsistent_frame(bytes); + for i in 0..dec.num_frames() { if out.len() >= MAX_GIF_FRAMES { return Err(CodecError::LimitExceeded(format!("more than {MAX_GIF_FRAMES} frames"))); } @@ -90,9 +145,10 @@ pub(crate) fn decode_frames(bytes: &[u8], limits: &Limits) -> Result limits.max_alloc { return Err(CodecError::LimitExceeded(format!("{} frames of {w}x{h} exceed max_alloc {}", out.len() + 1, limits.max_alloc))); } - let delay_ms = match dec.read_frame(&mut buf) { + let read = if inconsistent == Some(i) { Err(err(INCONSISTENT_FRAME)) } else { dec.read_frame(&mut buf).map_err(err) }; + let delay_ms = match read { Ok(d) => d, - Err(e) if out.is_empty() => return Err(err(e)), + Err(e) if out.is_empty() => return Err(e), Err(_) => { damaged = true; break; diff --git a/crates/image-codecs/tests/it/webp_frames.rs b/crates/image-codecs/tests/it/webp_frames.rs index 2c8e54e98..e68e68a7e 100644 --- a/crates/image-codecs/tests/it/webp_frames.rs +++ b/crates/image-codecs/tests/it/webp_frames.rs @@ -138,6 +138,94 @@ fn a_damaged_later_frame_keeps_the_frames_before_it() { assert_eq!(f[0].image.warnings, [DecodeWarning::Truncated { format: Format::WebP }]); } +/// `still`'s chunk `id`, with its header and padding. +fn chunk_in(still: &[u8], id: &[u8; 4]) -> Vec { + let at = still.windows(4).position(|w| w == id).unwrap(); + let len = u32::from_le_bytes(still[at + 4..at + 8].try_into().unwrap()) as usize; + still[at..at + 8 + len + len % 2].to_vec() +} + +/// `a` with one more frame: a translucent 6x4 lossy picture under an `ANMF` header giving the +/// frame's size as `header`. The frame holds our encoder's `ALPH` chunk, then its `VP8 ` chunk +/// (header included) as `picture` changes it. +fn with_alpha_frame(mut a: Vec, header: [u32; 2], picture: Patch) -> Vec { + let opts = EncodeOptions { webp_lossless: false, ..EncodeOptions::default() }; + let still = encode(&solid([0, 0, 255, 128], 6, 4), Format::WebP, &opts).unwrap(); + let mut anmf = Vec::new(); + anmf.extend_from_slice(&[0; 6]); // at the canvas origin + anmf.extend_from_slice(&u24(header[0] - 1)); + anmf.extend_from_slice(&u24(header[1] - 1)); + anmf.extend_from_slice(&u24(50)); + anmf.push(0x02); + anmf.extend(chunk_in(&still, b"ALPH")); + anmf.extend(picture(chunk_in(&still, b"VP8 "))); + a.extend(chunk(b"ANMF", &anmf)); + let riff = (a.len() - 8) as u32; + a[4..8].copy_from_slice(&riff.to_le_bytes()); + a +} + +/// A change to a frame's picture chunk. +type Patch = fn(Vec) -> Vec; + +fn as_is(c: Vec) -> Vec { + c +} + +fn renamed(mut c: Vec) -> Vec { + c[..4].copy_from_slice(b"VP9 "); + c +} + +fn not_a_key_frame(mut c: Vec) -> Vec { + c[8] |= 1; // the frame tag's key-frame bit (set = interframe) + c +} + +/// One stray byte after the picture chunk: the `ANMF` chunk around it becomes odd-sized. +fn odd_sized_frame(mut c: Vec) -> Vec { + c.push(0); + c +} + +#[test] +fn a_frame_with_alpha_comes_back() { + let a = with_alpha_frame(animated(6, 4, &[part(RED, 6, 4, 50)]), [6, 4], as_is); + let f = frames(&a); + assert_eq!(f.len(), 2); + assert_eq!(f[1].image.warnings, []); + let [_, _, b, alpha] = pixel(&f[1], 3, 2); + assert!(b > 200 && alpha.abs_diff(128) <= 2, "{:?}", pixel(&f[1], 3, 2)); +} + +/// A frame with alpha whose picture isn't a VP8 key frame of its `ANMF` header's size panicked +/// inside `image-webp` 0.2.4 (out-of-bounds indexing), found by the `decode_webp` fuzz target: +/// larger than the header, under another FourCC (decoded as VP8 all the same), or a non-key frame +/// (decoded as 0x0). An odd-sized `ANMF` chunk, after which `image-webp` would read the next frame +/// one byte early, is refused too. Each is refused like a damaged frame: an error when it is the +/// first, the frames before it otherwise. +#[test] +fn a_frame_with_alpha_that_is_not_its_headers_size_is_refused() { + let cases: [(&str, [u32; 2], Patch); 4] = [ + ("larger", [6, 1], as_is), + ("renamed and larger", [6, 1], renamed), + ("not a key frame", [6, 4], not_a_key_frame), + ("odd-sized", [6, 4], odd_sized_frame), + ]; + for (what, header, picture) in cases { + let later = with_alpha_frame(animated(6, 4, &[part(GREEN, 6, 4, 50)]), header, picture); + let f = frames(&later); + assert_eq!(f.len(), 1, "{what}"); + assert_eq!(pixel(&f[0], 0, 0), GREEN, "{what}"); + assert_eq!(f[0].image.warnings, [DecodeWarning::Truncated { format: Format::WebP }], "{what}"); + assert_eq!(decode(&later).unwrap().width(), 6, "{what}: a still decode reads the first frame only"); + + let first = with_alpha_frame(with_alpha_frame(animated(6, 4, &[]), header, picture), [6, 4], as_is); + assert!(matches!(decode_webp_frames(&first, &DecodeOptions::default()), Err(CodecError::Malformed { .. })), "{what}"); + assert!(matches!(decode(&first), Err(CodecError::Malformed { .. })), "{what}"); + } +} + #[test] fn every_prefix_of_an_animated_webp_returns() { let a = animated(5, 4, &[part(RED, 5, 4, 50), part(GREEN, 5, 4, 50), part(BLUE, 5, 4, 50)]); From f57bbca24f485b155a72dcb9b50e21bad63165cb Mon Sep 17 00:00:00 2001 From: Diego Chagas Date: Wed, 18 Nov 2026 17:25:26 -0300 Subject: [PATCH 139/161] jxl: bound brob metadata boxes before jxl-oxide inflates them (#2134) jxl-oxide decompresses brob (Brotli Exif/XMP) boxes into a buffer its AllocTracker does not count, so a 54-byte box expanded to 64 MiB of XMP and a larger one can ask for gigabytes before any pixel limit applies. decode now inflates every brob box under one 8 MiB budget per file, rewrites it as the plain box it wraps and drops boxes past the budget, nested or broken ones. Files without a brob box are not copied. --- Cargo.lock | 1 + crates/jxl/Cargo.toml | 4 +- crates/jxl/README.md | 6 ++ crates/jxl/src/brob.rs | 198 ++++++++++++++++++++++++++++++++++ crates/jxl/src/decode.rs | 5 + crates/jxl/src/lib.rs | 2 + crates/jxl/tests/roundtrip.rs | 34 ++++++ 7 files changed, 249 insertions(+), 1 deletion(-) create mode 100644 crates/jxl/src/brob.rs diff --git a/Cargo.lock b/Cargo.lock index 6417b2451..d52e120f4 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2395,6 +2395,7 @@ dependencies = [ name = "craft-jxl" version = "0.1.0" dependencies = [ + "brotli-decompressor 5.0.3", "craft-cms", "jixel", "jxl-oxide", diff --git a/crates/jxl/Cargo.toml b/crates/jxl/Cargo.toml index 45ed44f12..097dc5264 100644 --- a/crates/jxl/Cargo.toml +++ b/crates/jxl/Cargo.toml @@ -12,11 +12,13 @@ workspace = true [features] default = ["decode", "encode"] -decode = ["dep:jxl-oxide"] +decode = ["dep:jxl-oxide", "dep:brotli-decompressor"] encode = ["dep:jixel"] [dependencies] jxl-oxide = { workspace = true, optional = true } +# Bounds the Brotli-compressed metadata boxes before jxl-oxide inflates them (see brob.rs). +brotli-decompressor = { version = "5", optional = true } thiserror = { workspace = true } [dev-dependencies] diff --git a/crates/jxl/README.md b/crates/jxl/README.md index a3a8c6fa4..35d8988cb 100644 --- a/crates/jxl/README.md +++ b/crates/jxl/README.md @@ -16,6 +16,12 @@ linear sRGB and converted to Rec. 2020 to avoid the known wide-gamut XYB renderi returns the profile; it does not claim to transform ICC data without a CMS. `decode` retains the decoder's default rendering target. +Brotli-compressed metadata boxes (`brob`: Exif, XMP) are inflated before the file reaches +`jxl-oxide`, which would inflate them outside its allocation tracker (a 54-byte box can expand to +64 MiB and more). All the `brob` boxes of one file share an 8 MiB budget; a box past it, a nested or +broken one, is dropped (the pixels still decode) and the others are handed over as the plain box +they wrap. A file without a `brob` box is not copied. + The encoder accepts interleaved 8-bit, 16-bit, or HDR floating-point samples. Floating-point input is linear-light and is currently available only with the Rec. 2020 PQ option; one input unit maps to 203 cd/m². Gray input is expanded to RGB, and opaque alpha is omitted. ICC data is embedded in diff --git a/crates/jxl/src/brob.rs b/crates/jxl/src/brob.rs new file mode 100644 index 000000000..6d0aefbc4 --- /dev/null +++ b/crates/jxl/src/brob.rs @@ -0,0 +1,198 @@ +//! Bounds the Brotli-compressed metadata boxes (`brob`) of a JPEG XL container. +//! +//! jxl-oxide decompresses a `brob` box (Exif, XMP) into a buffer its `AllocTracker` does not +//! count, so a few-KB file can expand into gigabytes before any pixel check runs. Before the file +//! reaches it, every `brob` box is inflated here under [`BROB_CAP`] and rewritten as the plain box +//! it wraps; a box that would pass the cap is dropped (metadata is a courtesy, the pixels still +//! decode). The cap is one budget for the whole file, spent box by box. A file without a `brob` box is not copied at all. + +use std::io::Read; + +/// Most bytes all the compressed metadata boxes of one file may expand to together: real Exif and +/// XMP packets are a few KB to a few hundred KB, and a file of many small boxes that each inflate +/// to just under a per-box cap would otherwise hold N times the cap. +pub(crate) const BROB_CAP: usize = 8 << 20; + +const SIGNATURE: &[u8] = b"\0\0\0\x0cJXL \r\n\x87\n"; +const BROB: &[u8; 4] = b"brob"; + +/// One ISOBMFF box located in the file. +struct BoxSpan { + start: usize, + header: usize, + end: usize, +} + +impl BoxSpan { + fn kind<'a>(&self, bytes: &'a [u8]) -> &'a [u8] { + bytes.get(self.start + 4..self.start + 8).unwrap_or(&[]) + } + + fn payload<'a>(&self, bytes: &'a [u8]) -> &'a [u8] { + bytes.get(self.start + self.header..self.end).unwrap_or(&[]) + } +} + +/// The box starting at `at`, or `None` when its header is cut off or its size is nonsense. A box +/// that runs past the end of a truncated file ends at the file's end. +fn box_at(bytes: &[u8], at: usize) -> Option { + let head = bytes.get(at..at.checked_add(8)?)?; + let size = u32::from_be_bytes(head.get(..4)?.try_into().ok()?) as u64; + let (header, total) = match size { + 0 => (8, (bytes.len() - at) as u64), + 1 => (16, u64::from_be_bytes(bytes.get(at + 8..at + 16)?.try_into().ok()?)), + n => (8, n), + }; + let end = usize::try_from(total).ok().filter(|t| *t >= header).and_then(|t| at.checked_add(t)).unwrap_or(bytes.len()); + Some(BoxSpan { start: at, header, end: end.min(bytes.len()) }) +} + +/// Every box of the container, in order, up to the first one that cannot be read. +fn boxes(bytes: &[u8]) -> Vec { + let mut found = Vec::new(); + let mut at = 0; + while let Some(span) = box_at(bytes, at) { + at = span.end; + found.push(span); + if at >= bytes.len() { + break; + } + } + found +} + +/// Inflates `compressed`, or `None` when it is broken or would pass `cap` bytes. +pub(crate) fn inflate_capped(compressed: &[u8], cap: usize) -> Option> { + let mut out = Vec::new(); + let limit = u64::try_from(cap).ok()?.checked_add(1)?; + brotli_decompressor::Decompressor::new(compressed, 4096).take(limit).read_to_end(&mut out).ok()?; + (out.len() <= cap).then_some(out) +} + +/// A plain box of type `kind` holding `data`. +pub(crate) fn plain_box(kind: &[u8], data: &[u8]) -> Vec { + let mut out = Vec::with_capacity(data.len() + 16); + match u32::try_from(data.len() + 8) { + Ok(size) => out.extend_from_slice(&size.to_be_bytes()), + Err(_) => out.extend_from_slice(&1u32.to_be_bytes()), + } + out.extend_from_slice(kind); + if u32::try_from(data.len() + 8).is_err() { + out.extend_from_slice(&(data.len() as u64 + 16).to_be_bytes()); + } + out.extend_from_slice(data); + out +} + +/// The plain box a `brob` box wraps, or `None` when it must be dropped. A kept box is charged to +/// `budget`, so the boxes of one file never inflate past the cap together. +fn unwrap_brob(span: &BoxSpan, bytes: &[u8], budget: &mut usize) -> Option> { + let payload = span.payload(bytes); + let (inner, compressed) = (payload.get(..4)?, payload.get(4..)?); + if inner == BROB { + return None; + } + let inflated = inflate_capped(compressed, *budget)?; + *budget -= inflated.len(); + Some(plain_box(inner, &inflated)) +} + +/// `bytes` with every `brob` box inflated, `cap` bytes in all (or removed), or `None` when the file is +/// not a container or has no `brob` box and can be used as it is. +pub(crate) fn sanitize(bytes: &[u8], cap: usize) -> Option> { + if !bytes.starts_with(SIGNATURE) { + return None; + } + let spans = boxes(bytes); + if !spans.iter().any(|s| s.kind(bytes) == BROB) { + return None; + } + let mut budget = cap; + let mut out = Vec::with_capacity(bytes.len()); + for span in &spans { + match (span.kind(bytes) == BROB, bytes.get(span.start..span.end)) { + (true, _) => out.extend(unwrap_brob(span, bytes, &mut budget).unwrap_or_default()), + (false, Some(whole)) => out.extend_from_slice(whole), + (false, None) => {} + } + } + out.extend_from_slice(bytes.get(spans.last().map_or(0, |s| s.end)..).unwrap_or(&[])); + Some(out) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// "II*\0\x08\0\0\0hello exif" as Brotli. + const EXIF_BR: [u8; 22] = + [0x8b, 0x08, 0x80, 0x49, 0x49, 0x2a, 0x00, 0x08, 0x00, 0x00, 0x00, 0x68, 0x65, 0x6c, 0x6c, 0x6f, 0x20, 0x65, 0x78, 0x69, 0x66, 0x03]; + /// 64 MiB of `a` as Brotli: 54 bytes. + const BOMB_BR: [u8; 54] = [ + 0xcf, 0xff, 0xff, 0x7f, 0xf8, 0x25, 0xc2, 0xe2, 0xb1, 0x40, 0x20, 0xf7, 0xfe, 0x9f, 0xff, 0xff, 0xff, 0xf0, 0x4b, 0x00, 0xc4, 0x61, 0x01, + 0x80, 0xee, 0xfd, 0x3f, 0xff, 0xff, 0xff, 0xe1, 0x97, 0x00, 0x88, 0xc3, 0x22, 0x00, 0xdd, 0xfb, 0x7f, 0xfe, 0xff, 0xff, 0xc3, 0x2f, 0x01, + 0x10, 0x87, 0x05, 0x00, 0xba, 0xf7, 0xff, 0x03, + ]; + + fn container(boxes: &[Vec]) -> Vec { + let mut file = SIGNATURE.to_vec(); + boxes.iter().for_each(|b| file.extend_from_slice(b)); + file + } + + fn brob_of(inner: &[u8; 4], compressed: &[u8]) -> Vec { + plain_box(BROB, &[inner.as_slice(), compressed].concat()) + } + + #[test] + fn a_bomb_passes_the_cap_and_a_real_packet_does_not() { + assert_eq!(inflate_capped(&EXIF_BR, BROB_CAP).as_deref(), Some(&b"II*\0\x08\0\0\0hello exif"[..])); + assert!(inflate_capped(&BOMB_BR, BROB_CAP).is_none()); + assert!(inflate_capped(&BOMB_BR[..20], BROB_CAP).is_none(), "a cut stream is not a result"); + } + + #[test] + fn a_container_without_brob_is_used_as_it_is() { + assert!(sanitize(&container(&[plain_box(b"Exif", b"abc")]), BROB_CAP).is_none()); + assert!(sanitize(&[0xff, 0x0a, 0x00], BROB_CAP).is_none(), "a bare codestream has no boxes"); + } + + #[test] + fn a_small_brob_becomes_the_plain_box_it_wraps() { + let tail = plain_box(b"jxlc", b"codestream"); + let out = sanitize(&container(&[brob_of(b"Exif", &EXIF_BR), tail.clone()]), BROB_CAP).unwrap(); + assert_eq!(out, container(&[plain_box(b"Exif", b"II*\0\x08\0\0\0hello exif"), tail])); + } + + #[test] + fn a_brob_bomb_is_dropped_and_the_rest_kept() { + let tail = plain_box(b"jxlc", b"codestream"); + let out = sanitize(&container(&[brob_of(b"xml ", &BOMB_BR), tail.clone()]), BROB_CAP).unwrap(); + assert_eq!(out, container(&[tail])); + } + + #[test] + fn a_nested_or_empty_brob_is_dropped() { + let nested = brob_of(BROB, &EXIF_BR); + let empty = plain_box(BROB, b""); + assert_eq!(sanitize(&container(&[nested, empty]), BROB_CAP).unwrap(), container(&[])); + } + + #[test] + fn many_small_boxes_share_one_budget() { + let exif = b"II*\0\x08\0\0\0hello exif".len(); + let boxes: Vec> = (0..4).map(|_| brob_of(b"Exif", &EXIF_BR)).collect(); + let out = sanitize(&container(&boxes), exif * 2 + 1).unwrap(); + let kept = out.windows(4).filter(|w| *w == b"Exif").count(); + assert_eq!(kept, 2, "the boxes past the file's budget are dropped"); + assert!(out.len() < SIGNATURE.len() + 2 * (exif + 8) + 1); + } + + #[test] + fn a_cut_off_file_keeps_its_tail_verbatim() { + let mut file = container(&[brob_of(b"Exif", &EXIF_BR)]); + file.extend_from_slice(b"\0\0\0\x20jxlc half"); + let out = sanitize(&file, BROB_CAP).unwrap(); + assert!(out.ends_with(b"\0\0\0\x20jxlc half")); + } +} diff --git a/crates/jxl/src/decode.rs b/crates/jxl/src/decode.rs index 955774b23..f0c55f93d 100644 --- a/crates/jxl/src/decode.rs +++ b/crates/jxl/src/decode.rs @@ -3,6 +3,7 @@ use std::panic::{AssertUnwindSafe, catch_unwind}; use jxl_oxide::color::{ColourEncoding, ColourSpace, EnumColourEncoding, Primaries, RenderingIntent, TransferFunction, WhitePoint}; use jxl_oxide::{AllocTracker, AuxBoxData, ExtraChannelType, InitializeResult, JxlImage}; +use crate::brob; use crate::{DecodeLimits, Decoded, Error, Layout, Metadata, OutputColor, Result}; const INPUT_CHUNK: usize = 4096; @@ -29,6 +30,10 @@ fn decode_inner(bytes: &[u8], limits: DecodeLimits, output_color: OutputColor) - if bytes.len() > limits.max_input_bytes { return Err(Error::LimitExceeded(format!("{} input bytes exceed the {} byte input limit", bytes.len(), limits.max_input_bytes))); } + // jxl-oxide inflates `brob` (Brotli-compressed Exif/XMP) boxes outside its allocation tracker, so a + // few hundred bytes could expand to gigabytes. Inflate them here, bounded, and hand over plain boxes. + let sanitized = brob::sanitize(bytes, brob::BROB_CAP); + let bytes = sanitized.as_deref().unwrap_or(bytes); let budget = limits.max_tracked_bytes.unwrap_or_else(|| limits.max_pixels.saturating_mul(TRACKED_BYTES_PER_PIXEL).max(MIN_TRACKED_BYTES)); let tracker = AllocTracker::with_limit(usize::try_from(budget).unwrap_or(usize::MAX)); let mut uninit = JxlImage::builder().alloc_tracker(tracker).build_uninit(); diff --git a/crates/jxl/src/lib.rs b/crates/jxl/src/lib.rs index fcdd96199..77ad7e5bc 100644 --- a/crates/jxl/src/lib.rs +++ b/crates/jxl/src/lib.rs @@ -8,6 +8,8 @@ //! made through `jxl-oxide`; it cannot account for every allocation inside a decoder dependency. //! The first image frame is returned, with orientation already applied to its pixels. +#[cfg(feature = "decode")] +mod brob; #[cfg(feature = "decode")] mod decode; #[cfg(feature = "encode")] diff --git a/crates/jxl/tests/roundtrip.rs b/crates/jxl/tests/roundtrip.rs index 2bc4cf8e1..5091a92fd 100644 --- a/crates/jxl/tests/roundtrip.rs +++ b/crates/jxl/tests/roundtrip.rs @@ -423,3 +423,37 @@ fn bounded_feeds_keep_metadata_after_the_final_progressive_part() { assert_eq!(decoded.metadata.exif.as_deref(), Some(reference_exif.as_slice())); assert_eq!(decoded.metadata.xmp.as_deref(), Some(reference_xmp.as_slice())); } + +/// 64 MiB of `a`, Brotli-compressed to 54 bytes. +const BROTLI_BOMB: [u8; 54] = [ + 0xcf, 0xff, 0xff, 0x7f, 0xf8, 0x25, 0xc2, 0xe2, 0xb1, 0x40, 0x20, 0xf7, 0xfe, 0x9f, 0xff, 0xff, 0xff, 0xf0, 0x4b, 0x00, 0xc4, 0x61, 0x01, 0x80, + 0xee, 0xfd, 0x3f, 0xff, 0xff, 0xff, 0xe1, 0x97, 0x00, 0x88, 0xc3, 0x22, 0x00, 0xdd, 0xfb, 0x7f, 0xfe, 0xff, 0xff, 0xc3, 0x2f, 0x01, 0x10, 0x87, + 0x05, 0x00, 0xba, 0xf7, 0xff, 0x03, +]; + +/// ``, Brotli-compressed. +const BROTLI_XMP: [u8; 16] = [0x8b, 0x05, 0x80, 0x3c, 0x78, 0x3a, 0x78, 0x6d, 0x70, 0x6d, 0x65, 0x74, 0x61, 0x2f, 0x3e, 0x03]; + +fn container_with_brob(inner: &[u8; 4], compressed: &[u8]) -> Vec { + let image = ImageRef { width: 2, height: 2, layout: Layout::Rgb, samples: Samples::U8(&[1; 12]) }; + let codestream = raw_codestream(&encode(&image, EncodeOptions::default(), &Metadata::default()).unwrap()); + let mut file = b"\0\0\0\x0cJXL \r\n\x87\n".to_vec(); + box_bytes(&mut file, b"ftyp", b"jxl \0\0\0\0jxl "); + box_bytes(&mut file, b"brob", &[inner.as_slice(), compressed].concat()); + box_bytes(&mut file, b"jxlc", &codestream); + file +} + +/// A 54-byte `brob` box inflated to 64 MiB of XMP inside the decoder, outside its allocation tracker. +#[test] +fn a_brotli_bomb_costs_the_metadata_not_the_decode() { + let decoded = decode(&container_with_brob(b"xml ", &BROTLI_BOMB), DecodeLimits::default()).unwrap(); + assert_eq!((decoded.width, decoded.height), (2, 2)); + assert_eq!(decoded.metadata.xmp, None, "the over-cap metadata is dropped"); +} + +#[test] +fn a_small_brotli_xmp_box_still_comes_through() { + let decoded = decode(&container_with_brob(b"xml ", &BROTLI_XMP), DecodeLimits::default()).unwrap(); + assert_eq!(decoded.metadata.xmp.as_deref(), Some(&b""[..])); +} From 7ff0f967db7f412b8ebfddb70a32759bbfd50d54 Mon Sep 17 00:00:00 2001 From: Diego Chagas Date: Wed, 18 Nov 2026 17:26:57 -0300 Subject: [PATCH 140/161] image-codecs: read JPEG XL through craft-jxl behind a jxl feature (#2135) * image-codecs: read JPEG XL through craft-jxl behind a jxl feature Format::Jxl reads when the non-default `jxl` feature is on (decode only, so the encoder's jixel dependency is not built); without it JPEG XL is still detected and opening one is an Unsupported error that says the build lacks it. Orientation is applied by the decoder, so the Exif and XMP orientation become 1. ICC, Exif and XMP come through; associated alpha is divided out; float files open as F32. * image-codecs: skip JPEG XL in the one-pixel round trip The encoder refuses images narrower than two pixels; with the jxl feature JPEG XL is a read-and-write format, so the generic round trip reached it. --- Cargo.lock | 1 + Cargo.toml | 2 +- crates/image-codecs/Cargo.toml | 4 ++ crates/image-codecs/src/codecs/jxl.rs | 75 +++++++++++++++++++++++ crates/image-codecs/src/format.rs | 9 ++- crates/image-codecs/src/lib.rs | 2 +- crates/image-codecs/tests/it/roundtrip.rs | 4 ++ crates/image-codecs/tests/jxl_decode.rs | 44 +++++++++++++ crates/image-codecs/tests/jxl_export.rs | 13 +++- 9 files changed, 147 insertions(+), 7 deletions(-) create mode 100644 crates/image-codecs/tests/jxl_decode.rs diff --git a/Cargo.lock b/Cargo.lock index d52e120f4..ac676d666 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2299,6 +2299,7 @@ dependencies = [ "craft-dds", "craft-heif", "craft-jpeg", + "craft-jxl", "craft-media-testkit", "craft-metadata", "craft-tiff", diff --git a/Cargo.toml b/Cargo.toml index f167b972b..f99224d65 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -126,7 +126,7 @@ craft-jpeg = { path = "crates/jpeg" } craft-control-protocol = { path = "crates/control-protocol" } craft-jsonrpc = { path = "crates/jsonrpc" } craft-mcp = { path = "crates/mcp" } -craft-jxl = { path = "crates/jxl" } +craft-jxl = { path = "crates/jxl", default-features = false } craft-kiwi = { path = "crates/kiwi" } craft-keymap = { path = "crates/keymap" } craft-assuan = { path = "crates/assuan" } diff --git a/crates/image-codecs/Cargo.toml b/crates/image-codecs/Cargo.toml index 9b11af4e8..46d510342 100644 --- a/crates/image-codecs/Cargo.toml +++ b/crates/image-codecs/Cargo.toml @@ -19,6 +19,9 @@ sgi = [] sunraster = [] avif = ["image/avif", "dep:ravif", "dep:rav1e", "dep:avif-serialize"] heif = ["dep:craft-heif"] +# JPEG XL decode via craft-jxl (jxl-oxide, pure Rust). Without it JPEG XL files are still detected +# and opening one is a clear unsupported error; export needs no feature. +jxl = ["dep:craft-jxl"] # AVIF decode: craft-video's clean-room pure-Rust AV1 decoder. Native targets only (on wasm32 a # decoder panic aborts instead of becoming an error), so wasm32 builds leave AVIF unreadable. avif-decode = ["dep:craft-video", "craft-video/av1-decode"] @@ -43,6 +46,7 @@ craft-metadata = { workspace = true } craft-tiff = { workspace = true } zune-jpegxl = { version = "=0.5.2", default-features = false, features = ["std"] } craft-heif = { workspace = true, optional = true } +craft-jxl = { workspace = true, optional = true, features = ["decode"] } ravif = { version = "0.13", default-features = false, optional = true } rav1e = { version = "0.8", default-features = false, optional = true } avif-serialize = { version = "0.8", optional = true } diff --git a/crates/image-codecs/src/codecs/jxl.rs b/crates/image-codecs/src/codecs/jxl.rs index 39dd287a8..2e39ced2a 100644 --- a/crates/image-codecs/src/codecs/jxl.rs +++ b/crates/image-codecs/src/codecs/jxl.rs @@ -150,6 +150,81 @@ pub(crate) fn encode(image: &Image, plan: Plan) -> Result, CodecError> { .unwrap_or_else(|_| Err(CodecError::encode(F, "the JPEG XL encoder failed"))) } +/// The reason given when this build has no JPEG XL decoder. +#[cfg(not(feature = "jxl"))] +pub(crate) const NOT_IN_BUILD: &str = "JPEG XL support isn't included in this build"; + +#[cfg(not(feature = "jxl"))] +pub(crate) fn decode(_bytes: &[u8], _limits: &crate::options::Limits) -> Result { + Err(CodecError::unsupported(F, NOT_IN_BUILD)) +} + +#[cfg(feature = "jxl")] +fn decode_err(e: craft_jxl::Error) -> CodecError { + use craft_jxl::Error as E; + match e { + E::Malformed(what) | E::Encode(what) => CodecError::malformed(F, what), + E::LimitExceeded(what) => CodecError::LimitExceeded(what), + E::PixelLimitExceeded { width, height, max_pixels } => { + CodecError::LimitExceeded(format!("{width}×{height} pixels exceed max_pixels {max_pixels}")) + } + E::InvalidImage(what) => CodecError::InvalidImage(what), + E::Unsupported(what) => CodecError::unsupported(F, what), + } +} + +/// Decodes the first frame. The codestream's orientation is already applied to the pixels, so the +/// Exif and XMP orientation are rewritten to 1 (the same promise as HEIF without `keep_orientation`). +#[cfg(feature = "jxl")] +pub(crate) fn decode(bytes: &[u8], limits: &crate::options::Limits) -> Result { + let options = craft_jxl::DecodeLimits { max_pixels: limits.max_pixels, max_tracked_bytes: Some(limits.max_alloc), max_input_bytes: usize::MAX }; + let decoded = craft_jxl::decode(bytes, options).map_err(decode_err)?; + let layout = match decoded.layout { + craft_jxl::Layout::Gray => ChannelLayout::Gray, + craft_jxl::Layout::GrayAlpha => ChannelLayout::GrayA, + craft_jxl::Layout::Rgb => ChannelLayout::Rgb, + craft_jxl::Layout::Rgba => ChannelLayout::Rgba, + }; + let sample = match (decoded.float_samples, decoded.bit_depth) { + (true, _) => SampleType::F32, + (false, 0..=8) => SampleType::U8, + (false, _) => SampleType::U16, + }; + limits.check(decoded.width, decoded.height, layout, sample)?; + let mut pixels = decoded.pixels; + if decoded.pixels_premultiplied { + unpremultiply(&mut pixels, layout); + } + let mut image = Image::from_normalized(decoded.width, decoded.height, layout, sample, &pixels)?; + image.icc = Some(decoded.rendered_icc).filter(|icc| !icc.is_empty()).or(decoded.metadata.icc); + if image.icc.is_none() { + image.cicp = decoded.rendered_cicp.map(|[primaries, transfer, matrix, range]| crate::cicp::Cicp { + primaries: u16::from(primaries), + transfer: u16::from(transfer), + matrix: u16::from(matrix), + full_range: range != 0, + }); + } + image.meta.exif = decoded.metadata.exif.map(|e| crate::orientation::upright_exif(&e).into_owned()); + // An XMP packet is UTF-8 text; bytes that are not are dropped rather than guessed at. + image.meta.xmp = decoded.metadata.xmp.and_then(|x| String::from_utf8(x).ok()).map(|x| crate::orientation::upright_xmp(&x).into_owned()); + Ok(image) +} + +/// Divides colour by alpha where alpha is not zero (JPEG XL can keep colour associated with alpha). +#[cfg(feature = "jxl")] +fn unpremultiply(pixels: &mut [f32], layout: ChannelLayout) { + if !layout.has_alpha() { + return; + } + for px in pixels.chunks_exact_mut(layout.channels()) { + let Some((alpha, colour)) = px.split_last_mut().map(|(a, c)| (*a, c)) else { continue }; + if alpha > 0.0 { + colour.iter_mut().for_each(|c| *c /= alpha); + } + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/image-codecs/src/format.rs b/crates/image-codecs/src/format.rs index 1535c229e..b3ec649e3 100644 --- a/crates/image-codecs/src/format.rs +++ b/crates/image-codecs/src/format.rs @@ -101,7 +101,12 @@ pub const ASYMMETRIC_EXCEPTIONS: &[(Format, &str)] = &[ `avif-decode` feature, native targets only (on wasm32 a decoder panic would abort). A build \ with only one of them reads or writes AVIF, not both.", ), - (Format::Jxl, "Lossless JPEG XL export uses the pure-Rust zune-jpegxl encoder; JPEG XL reading is tracked separately and is not yet available."), + ( + Format::Jxl, + "Lossless JPEG XL export uses the pure-Rust zune-jpegxl encoder in every build; JPEG XL decode \ + (craft-jxl on jxl-oxide, pure Rust) is behind the non-default `jxl` feature, which official \ + builds enable. Without the feature JPEG XL is written but not read.", + ), ( Format::Heif, "HEIC decode uses heic-rs (pure Rust, in the optional craft-heif crate behind the \ @@ -330,7 +335,7 @@ pub fn caps(format: Format) -> FormatCaps { // `text`: the DPX header fields only (see `fidelity_warnings_with`). Format::Dpx => FormatCaps { depths: &[S::U8, S::U16], layouts: &[L::Gray, L::Rgb, L::Rgba], text: true, ..base }, // GrayA is widened to RGBA at the same depth; the 16-bit alpha header is corrected. - Format::Jxl => FormatCaps { read: false, depths: &[S::U8, S::U16], layouts: &[L::Gray, L::Rgb, L::Rgba], ..base }, + Format::Jxl => FormatCaps { read: cfg!(feature = "jxl"), depths: &[S::U8, S::U16], layouts: &[L::Gray, L::Rgb, L::Rgba], ..base }, Format::Cineon => FormatCaps { depths: &[S::U16], layouts: &[L::Gray, L::Rgb], alpha: false, ..base }, Format::Dicom => FormatCaps { read: true, write: false, depths: &[S::U8, S::U16], layouts: &[L::Gray, L::Rgb], alpha: false, ..base }, Format::Iff => FormatCaps { depths: &[S::U8], layouts: &[L::Rgb, L::Rgba], alpha: true, ..base }, diff --git a/crates/image-codecs/src/lib.rs b/crates/image-codecs/src/lib.rs index 032289c7c..a68ce52fe 100644 --- a/crates/image-codecs/src/lib.rs +++ b/crates/image-codecs/src/lib.rs @@ -140,7 +140,7 @@ pub fn decode_as_with(format: Format, bytes: &[u8], opts: &DecodeOptions) -> Res Format::OpenExr => exr::decode(bytes, l), Format::Heif => heif::decode(bytes, l, opts.keep_orientation), Format::Jp2 => jp2::decode(bytes, l), - Format::Jxl => Err(CodecError::unsupported(format, "JPEG XL reading is not yet available; export is supported")), + Format::Jxl => jxl::decode(bytes, l), #[cfg(all(feature = "avif-decode", not(target_arch = "wasm32")))] Format::Avif => codecs::avif_decode::decode(bytes, l, opts.keep_orientation), #[cfg(not(all(feature = "avif-decode", not(target_arch = "wasm32"))))] diff --git a/crates/image-codecs/tests/it/roundtrip.rs b/crates/image-codecs/tests/it/roundtrip.rs index 6d5d38bca..6ff2e5818 100644 --- a/crates/image-codecs/tests/it/roundtrip.rs +++ b/crates/image-codecs/tests/it/roundtrip.rs @@ -318,6 +318,10 @@ fn webp_lossy_request_is_a_lossy_file() { #[test] fn one_pixel_images_roundtrip() { for f in rw_formats() { + // The JPEG XL encoder refuses images narrower than two pixels (jxl_export.rs pins the error). + if f == Format::Jxl { + continue; + } let img = synth(1, 1, ChannelLayout::Rgb, SampleType::U8, 1, 0.0); let back = decode(&encode(&img, f, &EncodeOptions::default()).unwrap()).unwrap(); assert_eq!(back.dimensions(), (1, 1), "{f:?}"); diff --git a/crates/image-codecs/tests/jxl_decode.rs b/crates/image-codecs/tests/jxl_decode.rs new file mode 100644 index 000000000..fad3b51d1 --- /dev/null +++ b/crates/image-codecs/tests/jxl_decode.rs @@ -0,0 +1,44 @@ +//! JPEG XL decode through craft-jxl: lossless round trips, metadata, and hostile containers. +#![cfg(feature = "jxl")] + +use craft_image_codecs::{ChannelLayout, CodecError, DecodeOptions, EncodeOptions, Format, Image, Limits, SampleType, decode, decode_with, encode}; + +fn rgba_8bit() -> Image { + let data = vec![0, 255, 17, 127, 20, 40, 80, 255, 255, 0, 2, 0, 10, 20, 30, 64, 0, 0, 0, 1, 90, 80, 70, 128]; + Image::from_u8(3, 2, ChannelLayout::Rgba, data).unwrap() +} + +#[test] +fn an_eight_bit_export_decodes_back_to_the_same_pixels() { + let input = rgba_8bit(); + let bytes = encode(&input, Format::Jxl, &EncodeOptions::default()).unwrap(); + let out = decode(&bytes).unwrap(); + assert_eq!((out.width(), out.height(), out.layout(), out.sample_type()), (3, 2, ChannelLayout::Rgba, SampleType::U8)); + assert_eq!(out.data(), input.data()); +} + +#[test] +fn a_sixteen_bit_gray_export_decodes_back_to_the_same_samples() { + let samples: Vec = (0..12).map(|i| i * 5461).collect(); + let input = Image::from_u16(4, 3, ChannelLayout::Gray, &samples).unwrap(); + let bytes = encode(&input, Format::Jxl, &EncodeOptions::default()).unwrap(); + let out = decode(&bytes).unwrap(); + assert_eq!((out.layout(), out.sample_type()), (ChannelLayout::Gray, SampleType::U16)); + assert_eq!(out.data(), input.data()); +} + +#[test] +fn the_pixel_limit_is_an_error_not_an_allocation() { + let bytes = encode(&rgba_8bit(), Format::Jxl, &EncodeOptions::default()).unwrap(); + let opts = DecodeOptions { limits: Limits { max_pixels: 4, ..Limits::default() }, ..DecodeOptions::default() }; + let err = decode_with(&bytes, &opts); + assert!(matches!(err, Err(CodecError::LimitExceeded(_))), "{err:?}"); +} + +#[test] +fn every_prefix_of_a_file_is_an_error_or_an_image_never_a_panic() { + let bytes = encode(&rgba_8bit(), Format::Jxl, &EncodeOptions::default()).unwrap(); + for end in 0..bytes.len() { + let _ = decode(&bytes[..end]); + } +} diff --git a/crates/image-codecs/tests/jxl_export.rs b/crates/image-codecs/tests/jxl_export.rs index 57a91143c..129aa1a3b 100644 --- a/crates/image-codecs/tests/jxl_export.rs +++ b/crates/image-codecs/tests/jxl_export.rs @@ -5,9 +5,10 @@ use craft_image_codecs::{ }; #[test] -fn jpeg_xl_is_export_only_and_detects_both_signatures() { +fn jpeg_xl_is_export_only_without_the_jxl_feature_and_detects_both_signatures() { let c = caps(Format::Jxl); - assert!(c.write && !c.read); + assert!(c.write); + assert_eq!(c.read, cfg!(feature = "jxl")); assert_eq!(c.depths, &[SampleType::U8, SampleType::U16]); assert_eq!(c.layouts, &[ChannelLayout::Gray, ChannelLayout::Rgb, ChannelLayout::Rgba]); assert!(c.alpha && !c.icc && !c.exif && !c.xmp && !c.dpi && !c.lossy); @@ -17,7 +18,13 @@ fn jpeg_xl_is_export_only_and_detects_both_signatures() { assert_eq!(detect(b"\0\0\0\x0cJXL \r\n\x87\n"), Some(Format::Jxl)); assert!(ASYMMETRIC_EXCEPTIONS.iter().any(|(format, reason)| *format == Format::Jxl && !reason.is_empty())); let err = decode(&[0xff, 0x0a]); - assert!(matches!(err, Err(CodecError::Unsupported { format: Format::Jxl, .. })), "{err:?}"); + #[cfg(not(feature = "jxl"))] + assert!( + matches!(&err, Err(CodecError::Unsupported { format: Format::Jxl, reason }) if reason.contains("isn't included in this build")), + "{err:?}" + ); + #[cfg(feature = "jxl")] + assert!(matches!(err, Err(CodecError::Malformed { format: Format::Jxl, .. })), "{err:?}"); } #[test] From d851c9b2e377680643a610f4a5db7b9ca3210971 Mon Sep 17 00:00:00 2001 From: Michael Higgins <55243596+Higgs32584@users.noreply.github.com> Date: Wed, 18 Nov 2026 15:30:17 -0500 Subject: [PATCH 141/161] craft-pdf: a rebuilt xref keeps the generation-0 object the page draws (cos) (#2240) When the cross-reference data is rebuilt from object headers, the index was keyed by object number alone, so a later `N 1 obj` replaced `N 0 obj` while the renderer still drew `N 0 obj` under `N 0 R`. cos read the page's object as null and a full save wrote null (storytold/pdfcraft#6759). Each number now keeps its object by the renderer's rule: the last real `N 0 obj`, else the last header as a generation-0 stand-in. Where a number has headers of several generations, a bounded walk from the trailer gives the slot to the one generation the document reaches, when it reaches one only, so a reused number such as PDFBOX-3785's `202097.pdf` keeps its page. A trailer /Root is ranked by its own generation's header, and a slot whose object stream supplies indexed members stays. What one slot cannot hold goes to the repair log, and Document::rebuilt_with_unsettled_generations() says when a save could differ from what the renderer drew. Port of storytold/pdfcraft#6767, which can't land in pdfcraft since cos moved here. --- crates/pdf/src/cos/document.rs | 2 + crates/pdf/src/cos/document/objects.rs | 9 ++ crates/pdf/src/cos/document/open.rs | 2 + crates/pdf/src/cos/document/reader.rs | 186 ++++++++++++++++++++++- crates/pdf/tests/cos/generations.rs | 201 +++++++++++++++++++++++++ 5 files changed, 392 insertions(+), 8 deletions(-) diff --git a/crates/pdf/src/cos/document.rs b/crates/pdf/src/cos/document.rs index efeed81d1..10ca0156c 100644 --- a/crates/pdf/src/cos/document.rs +++ b/crates/pdf/src/cos/document.rs @@ -51,6 +51,8 @@ pub struct Document { repair_log: Arc>, /// See [`Self::rebuilt_unlinked_update`]. unlinked_update: bool, + /// See [`Self::rebuilt_with_unsettled_generations`]. + unsettled_generations: bool, /// What loading each file object gave, a failure as well: a damaged object is otherwise /// loaded again on every read, and a member of an object stream that can't be decoded /// decodes the stream again (`Document::load`). diff --git a/crates/pdf/src/cos/document/objects.rs b/crates/pdf/src/cos/document/objects.rs index 93898cc24..71cfdcef0 100644 --- a/crates/pdf/src/cos/document/objects.rs +++ b/crates/pdf/src/cos/document/objects.rs @@ -104,6 +104,15 @@ impl Document { self.unlinked_update } + /// Whether the index was rebuilt from object headers and the document reaches more than one + /// generation of some object number (or the check of that gave up). One slot per number + /// holds one of them, so a reference to another reads as null here while the renderer, + /// which keeps every generation, draws it: what a save or signature writes can differ from + /// what the page showed (pdfcraft#6759). The repair log names the objects left out. + pub fn rebuilt_with_unsettled_generations(&self) -> bool { + self.unsettled_generations + } + /// Objects of the file that could not be read, lowest number first, as lines such as /// "object 11 0 could not be read: syntax error at byte 512: ...". [`Self::get`] reads each /// as null (ISO 32000-2 §7.3.10), so a document with one opens, but what it held is lost: diff --git a/crates/pdf/src/cos/document/open.rs b/crates/pdf/src/cos/document/open.rs index 1600256ab..03c427f8b 100644 --- a/crates/pdf/src/cos/document/open.rs +++ b/crates/pdf/src/cos/document/open.rs @@ -22,6 +22,7 @@ impl Document { located: Arc::default(), repair_log: Arc::new(Vec::new()), unlinked_update: false, + unsettled_generations: false, cache: Arc::default(), objstms: Arc::default(), object_notes: Arc::default(), @@ -111,6 +112,7 @@ impl Document { located: Arc::default(), repair_log: Arc::new(Vec::new()), unlinked_update: false, + unsettled_generations: false, cache: Arc::default(), objstms: Arc::default(), object_notes: Arc::default(), diff --git a/crates/pdf/src/cos/document/reader.rs b/crates/pdf/src/cos/document/reader.rs index 049d1fbd6..7656bd872 100644 --- a/crates/pdf/src/cos/document/reader.rs +++ b/crates/pdf/src/cos/document/reader.rs @@ -5,6 +5,7 @@ use super::xref::{ MAX_COMPACT_XREF_BYTES, MAX_COMPACT_XREF_ROWS, Revision, XrefEntry, XrefIndex, XrefRange, XrefRevisionSection, XrefSection, XrefSectionData, }; use super::{Arc, BTreeMap, CosError, Dict, Document, HashMap, HashSet, Lexer, ObjRef, Object, is_whitespace, parse_indirect, parse_indirect_shared}; +use crate::cos::writer::collect_refs; impl Document { #[allow(clippy::type_complexity)] @@ -635,7 +636,7 @@ impl Document { return; } let mut located = BTreeMap::new(); - for (num, offset, generation, _) in object_headers(&self.data) { + for (num, offset, generation, _, _) in object_headers(&self.data) { located.insert(num, (offset, generation)); } let mut filled = entries.to_map(&self.data); @@ -683,8 +684,26 @@ impl Document { xref_marks.push(p); mark_from = p + 1; } - for &(num, offset, generation, body) in &headers { - entries.insert(num, XrefEntry::InFile { offset, generation }); + // Which object a number keeps follows the renderer's rebuilt table (hayro-syntax + // `fallback_xref_map_inner`), so the two read every page alike (pdfcraft#6759): the last real + // `N 0 obj` keeps N, whatever headers of other generations come after it; with none, the + // last header stands in. Every slot is generation 0, so `N 0 R` reaches the stand-in as + // the renderer's does, and the header's own generation still resolves through + // `indexed_header_reference`. One slot holds one body, though, and the renderer also + // keeps every other generation under its exact key: where the file reaches only that + // one, it takes the slot instead (`keep_the_generation_reached`, below). + // The choice goes by the generation as the renderer reads it (`as_drawn`), not the `u16` + // a cos reference can carry: `N 65536 obj` is no real `N 0 obj` to the renderer. + // Per number: the generation of the header it keeps, and the offset of the last header + // of each generation seen. + let mut kept: BTreeMap)> = BTreeMap::new(); + for &(num, offset, _, body, as_drawn) in &headers { + let (held, seen) = kept.entry(num).or_insert_with(|| (as_drawn, BTreeMap::new())); + seen.insert(as_drawn, offset); + if as_drawn == 0 || *held != 0 { + *held = as_drawn; + entries.insert(num, XrefEntry::InFile { offset, generation: 0 }); + } objstms.push(num); // `/XRef` within the 1024 bytes after the header (the first one that is there). let first = xref_marks.partition_point(|&p| p < body); @@ -715,7 +734,8 @@ impl Document { } } let object_count = entries.len(); - self.entries = Arc::new(XrefIndex::Materialized(Arc::new(entries), Arc::default())); + let entries = Arc::new(entries); + self.entries = Arc::new(XrefIndex::Materialized(entries.clone(), Arc::default())); // Failures from the scan stand: the members added change no scanned entry, and a // `/Length` found only now cannot make a stream unreadable. self.cache.lock().map_err(|_| CosError::Poisoned)?.clear(); @@ -726,8 +746,17 @@ impl Document { // counts when nothing better is there. // A file whose only trailers are cross-reference streams (and whose `startxref` is gone // or points nowhere) keeps its /Info, /ID and /Encrypt this way. + // A /Root the slot does not answer is read by its own generation's header, as the + // renderer's rebuilt table does: a trailer naming `1 1 R` reaches `1 1 obj` where the + // slot keeps `1 0 obj`, and `keep_the_generation_reached` then gives it the slot. let rank = |at: usize, d: &Dict| { - let root = self.try_get_reference(d.reference(b"Root")?).ok()?; + let r = d.reference(b"Root")?; + let root = if self.reference_matches(r) { + self.try_get_reference(r).ok()? + } else { + let at = kept.get(&r.num).and_then(|(_, seen)| seen.get(&i32::from(r.generation))).and_then(|&o| usize::try_from(o).ok())?; + Arc::new(parse_indirect_shared(&data, at, &|_| None, &self.endstreams).ok()?.1) + }; let Object::Dict(c) = &*root else { return None }; Some((c.name(b"Type") == Some(b"Catalog") || c.contains(b"Pages"), at)) }; @@ -778,8 +807,146 @@ impl Document { self.trailer = trailer; self.revisions = Arc::new(Vec::new()); log.push(format!("rebuilt cross-reference index from {object_count} object headers")); + kept.retain(|_, (_, seen)| seen.len() > 1); + self.unsettled_generations = self.keep_the_generation_reached(&entries, &mut kept)?; + let left_out: Vec<(u32, i32)> = + kept.iter().flat_map(|(&num, (held, seen))| seen.keys().filter(move |&&g| g != *held).map(move |&g| (num, g))).collect(); + if !left_out.is_empty() { + let named: Vec = left_out.iter().take(8).map(|(num, generation)| format!("{num} {generation}")).collect(); + let more = if left_out.len() > 8 { format!(" and {} more", left_out.len() - 8) } else { String::new() }; + log.push(format!( + "rebuilding the cross-reference index left out {} objects that share a number with the object kept for it ({}{more}); references to them read as null", + left_out.len(), + named.join(", "), + )); + } Ok(()) } + + /// For the numbers whose headers carry more than one generation (`shared`: per number, the + /// generation [`Self::reconstruct`] kept and the offset of each generation's last header), + /// give the slot to the one generation the document reaches, when it reaches one only. + /// + /// The renderer keeps each generation under its exact key, so a file whose newer revision + /// reuses a number as `69 1 obj` and refers to it as `69 1 R`, with an unrelated `69 0 obj` + /// left from an older one, draws `69 1 obj`. Kept at generation 0, the slot would read + /// `69 1 R` as null, and a save would lose that page (`202097.pdf` of PDFBOX-3785). The walk + /// resolves these numbers as the renderer does: by exact generation, and `N 0 R` with no + /// real `N 0 obj` to the stand-in. A slot given to another generation takes that + /// generation, so `N 0 R`, which no reached object names, reads as null. When more than one + /// generation is reached, the generation-0 rule stands (the pdfcraft#6759 case), and the caller + /// notes what it leaves out. Returns whether that happened, or the walk gave up: then some + /// reference the document reaches can read differently here than in the renderer. + /// + /// A stream's `/Length` does not count as reaching an object: it only says where the data + /// ends, both readers look for `endstream` when it is wrong, and an older revision's length + /// object is the usual stale `N 0 obj`. + fn keep_the_generation_reached( + &mut self, + entries: &Arc>, + shared: &mut BTreeMap)>, + ) -> Result { + // References followed before the walk gives up and the generation-0 rule stands. + const MAX_WALK: usize = 1 << 21; + if shared.is_empty() { + return Ok(false); + } + let mut reader = self.object_reader(); + let mut queue = Vec::new(); + for (_, value) in self.trailer.iter() { + refs_reached(value, &mut queue); + } + let mut visited = HashSet::new(); + let mut reached: BTreeMap> = BTreeMap::new(); + while let Some(r) = queue.pop() { + if !visited.insert(r) { + continue; + } + if visited.len() > MAX_WALK { + return Ok(true); + } + let Some((held, bodies)) = shared.get(&r.num) else { + if let Ok(o) = reader.try_get(r) { + refs_reached(&o, &mut queue); + } + continue; + }; + let asked = i32::from(r.generation); + let generation = if bodies.contains_key(&asked) { + asked + } else if asked == 0 { + *held + } else { + continue; + }; + reached.entry(r.num).or_default().insert(generation); + if let Some(at) = bodies.get(&generation).and_then(|&o| usize::try_from(o).ok()) + && let Ok((_, o)) = parse_indirect_shared(&self.data, at, &|_| None, &self.endstreams) + { + refs_reached(&o, &mut queue); + } + } + // Object streams whose members the index lists: those were read from the body the slot + // holds, and the renderer reads them through its generation-0 key as well. + let holding_members: HashSet = entries + .values() + .filter_map(|e| match e { + XrefEntry::InStream { stream, .. } => Some(*stream), + _ => None, + }) + .collect(); + let mut moved = Vec::new(); + let mut unsettled = false; + for (num, generations) in reached { + let (Some(&only), 1) = (generations.first(), generations.len()) else { + unsettled = true; + continue; + }; + let Some((held, bodies)) = shared.get_mut(&num) else { continue }; + if only == *held { + continue; + } + // Moving the slot would read those members from another stream: it stays, and the + // generation reached is left out. + if holding_members.contains(&num) { + unsettled = true; + continue; + } + let (Ok(generation), Some(&offset)) = (u16::try_from(only), bodies.get(&only)) else { + unsettled = true; + continue; + }; + *held = only; + moved.push((num, XrefEntry::InFile { offset, generation })); + } + if moved.is_empty() { + return Ok(unsettled); + } + let mut index = (**entries).clone(); + { + // What the walk noted about the body the slot held no longer describes it. + let mut notes = self.object_notes.lock().map_err(|_| CosError::Poisoned)?; + for (num, entry) in moved { + index.insert(num, entry); + notes.unreadable.objects.remove(&num); + notes.repaired.objects.remove(&num); + } + } + self.entries = Arc::new(XrefIndex::Materialized(Arc::new(index), Arc::default())); + self.cache.lock().map_err(|_| CosError::Poisoned)?.clear(); + Ok(unsettled) + } +} + +/// The references in `o` that reach another object, for [`Document::keep_the_generation_reached`]: +/// all of them but a stream's `/Length`. +fn refs_reached(o: &Object, queue: &mut Vec) { + match o { + Object::Stream(s) => { + s.dict.iter().filter(|(k, _)| k.as_slice() != b"Length".as_slice()).for_each(|(_, v)| collect_refs(v, &mut |r| queue.push(r))) + } + other => collect_refs(other, &mut |r| queue.push(r)), + } } /// For a classic table at `at` whose first subsection is `0 N` and opens with an in-use row @@ -802,8 +969,10 @@ fn shifted_subsection_end(data: &[u8], at: usize) -> Option { } /// Every `N G obj` header in `data`, in file order, as (number, offset, generation, position -/// after `obj`). -fn object_headers(data: &[u8]) -> Vec<(u32, u64, u16, usize)> { +/// after `obj`, generation as the renderer reads it). The renderer's rebuilt table reads G as an +/// `i32` and takes 0 when it cannot (hayro-syntax `read_object_header`), so a G past `u16`, which +/// is 0 here, can be another generation there. +fn object_headers(data: &[u8]) -> Vec<(u32, u64, u16, usize, i32)> { let mut headers = Vec::new(); let mut i = 0; while i < data.len() { @@ -817,8 +986,9 @@ fn object_headers(data: &[u8]) -> Vec<(u32, u64, u16, usize)> { if !n.is_empty() && !g.is_empty() && n.iter().all(u8::is_ascii_digit) && g.iter().all(u8::is_ascii_digit) && lx.eat_keyword(b"obj") { let num = std::str::from_utf8(n).ok().and_then(|s| s.parse::().ok()); let generation = std::str::from_utf8(g).ok().and_then(|s| s.parse::().ok()).unwrap_or(0); + let as_drawn = std::str::from_utf8(g).ok().and_then(|s| s.parse::().ok()).unwrap_or(0); if let Some(num) = num { - headers.push((num, i as u64, generation, lx.pos)); + headers.push((num, i as u64, generation, lx.pos, as_drawn)); } i = lx.pos; continue; diff --git a/crates/pdf/tests/cos/generations.rs b/crates/pdf/tests/cos/generations.rs index 77e5475d7..f18fc7dfd 100644 --- a/crates/pdf/tests/cos/generations.rs +++ b/crates/pdf/tests/cos/generations.rs @@ -508,3 +508,204 @@ fn accepted_generations_are_the_generations_get_resolves() { assert!(matches!(packed.xref_entry(member.num), Some(XrefEntry::InStream { .. })), "the member is in an object stream"); agrees(&packed, member.num, &[0]); } + +const COVER: &str = "0 0 0 rg 10 10 80 80 re f"; +const OTHER: &str = "1 0 0 rg 100 100 20 20 re f"; + +/// A file with no usable cross-reference data (`startxref` points nowhere), so both cos and the +/// renderer rebuild it from object headers. The page draws `page_ref`; `bodies` are the +/// `4 G obj` content streams after it, as (generation, content), in file order (pdfcraft#6759). +fn rebuilt_page(page_ref: &str, bodies: &[(u32, &str)]) -> Arc> { + let mut pdf = format!( + "%PDF-1.7\n1 0 obj\n<< /Type /Catalog /Pages 2 0 R >>\nendobj\n2 0 obj\n<< /Type /Pages /Kids [3 0 R] /Count 1 >>\nendobj\n\ + 3 0 obj\n<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents {page_ref} >>\nendobj\n" + ); + for (generation, content) in bodies { + pdf.push_str(&format!("4 {generation} obj\n<< /Length {} >>\nstream\n{content}\nendstream\nendobj\n", content.len())); + } + pdf.push_str("trailer\n<< /Size 5 /Root 1 0 R >>\nstartxref\n999999\n%%EOF\n"); + Arc::new(pdf.into_bytes()) +} + +/// What the renderer draws on the first page. +fn drawn_by_renderer(bytes: Arc>) -> Option { + let oracle = hayro_syntax::Pdf::new(bytes).ok()?; + let page = oracle.pages().first()?.page_stream()?.to_vec(); + Some(String::from_utf8_lossy(&page).trim().to_string()) +} + +/// The first page's content stream as cos reads it. +fn drawn_by_cos(doc: &Document) -> Option { + let catalog = doc.get(doc.root()?); + let pages = doc.resolve(catalog.as_dict()?.get(b"Pages")?); + let page = doc.resolve(pages.as_dict()?.get(b"Kids")?.as_array()?.first()?); + let contents = doc.resolve(page.as_dict()?.get(b"Contents")?); + let Object::Stream(stream) = &*contents else { return None }; + Some(String::from_utf8_lossy(&stream.raw).trim().to_string()) +} + +/// A later `4 1 obj` used to take object 4's slot, so `4 0 R` read as null in cos and a full +/// save wrote null, while the renderer drew the generation-0 cover (pdfcraft#6759). +#[test] +fn a_rebuilt_index_keeps_the_generation_0_body_the_page_draws() { + for bodies in [&[(0, COVER), (1, OTHER)][..], &[(1, OTHER), (0, COVER)], &[(0, COVER), (1, OTHER), (2, OTHER)]] { + let bytes = rebuilt_page("4 0 R", bodies); + assert_eq!(drawn_by_renderer(bytes.clone()).as_deref(), Some(COVER), "{bodies:?}"); + let doc = Document::open(bytes).unwrap(); + assert!(doc.reconstructed()); + assert!(!doc.rebuilt_with_unsettled_generations(), "{bodies:?}: only generation 0 is reached"); + assert_eq!(drawn_by_cos(&doc).as_deref(), Some(COVER), "{bodies:?}"); + let saved = Arc::new(write_full(&doc, &SaveOptions::default()).unwrap()); + assert_eq!(drawn_by_renderer(saved.clone()).as_deref(), Some(COVER), "{bodies:?}"); + assert_eq!(drawn_by_cos(&Document::open(saved).unwrap()).as_deref(), Some(COVER), "{bodies:?}"); + } +} + +#[test] +fn a_rebuilt_index_answers_generation_0_for_an_object_with_only_another_generation() { + // The renderer's rebuilt table lets the last header stand in for `N 0 R` when there is no + // `N 0 obj` (pdfcraft#5580); cos follows it, and the header's own generation still resolves. A + // generation past `u16` is still another generation to the renderer, not a real `N 0 obj`. + for (page_ref, bodies) in [ + ("4 0 R", &[(1, COVER)][..]), + ("4 1 R", &[(1, COVER)]), + ("4 0 R", &[(1, OTHER), (2, COVER)]), + ("4 2 R", &[(1, OTHER), (2, COVER)]), + ("4 0 R", &[(65536, OTHER), (1, COVER)]), + ] { + let bytes = rebuilt_page(page_ref, bodies); + assert_eq!(drawn_by_renderer(bytes.clone()).as_deref(), Some(COVER), "{page_ref} {bodies:?}"); + let doc = Document::open(bytes).unwrap(); + assert_eq!(drawn_by_cos(&doc).as_deref(), Some(COVER), "{page_ref} {bodies:?}"); + let saved = Arc::new(write_full(&doc, &SaveOptions::default()).unwrap()); + assert_eq!(drawn_by_renderer(saved).as_deref(), Some(COVER), "{page_ref} {bodies:?}"); + } +} + +#[test] +fn a_rebuilt_index_records_a_body_it_leaves_out() { + // One slot holds one body: with `4 0 obj` and `4 1 obj` both live, `4 1 R` reads as null in + // cos though the renderer reaches it. The repair log says so: nothing is dropped unrecorded. + let doc = Document::open(rebuilt_page("4 0 R", &[(0, COVER), (1, OTHER), (1, OTHER)])).unwrap(); + assert_eq!(*doc.get(ObjRef::new(4, 1)), Object::Null); + let noted: Vec<_> = doc.repair_log().iter().filter(|line| line.contains("left out")).collect(); + assert_eq!(noted.len(), 1, "{:?}", doc.repair_log()); + assert!(noted[0].contains("4 1"), "{noted:?}"); + // Nothing left out, nothing noted. + let doc = Document::open(rebuilt_page("4 0 R", &[(0, COVER), (0, OTHER)])).unwrap(); + assert!(doc.repair_log().iter().all(|line| !line.contains("left out")), "{:?}", doc.repair_log()); +} + +/// No usable cross-reference data, objects in file order as (number, generation, body). +fn rebuilt_objects(objects: &[(u32, u16, &str)]) -> Arc> { + rebuilt_objects_rooted(objects, "1 0 R") +} + +/// [`rebuilt_objects`] with the trailer's `/Root` given. +fn rebuilt_objects_rooted(objects: &[(u32, u16, &str)], root: &str) -> Arc> { + let mut pdf = String::from("%PDF-1.7\n"); + for (num, generation, body) in objects { + pdf.push_str(&format!("{num} {generation} obj\n{body}\nendobj\n")); + } + pdf.push_str(&format!("trailer\n<< /Size 11 /Root {root} >>\nstartxref\n999999\n%%EOF\n")); + Arc::new(pdf.into_bytes()) +} + +/// Each page's content as the renderer draws it. +fn pages_drawn_by_renderer(bytes: Arc>) -> Vec { + let oracle = hayro_syntax::Pdf::new(bytes).unwrap(); + oracle.pages().iter().map(|p| String::from_utf8_lossy(p.page_stream().unwrap_or_default()).trim().to_string()).collect() +} + +/// A newer revision reused number 5 as `5 1 obj`, a page its page tree lists as `5 1 R`; the +/// older `5 0 obj` is only a stream length. Kept at generation 0, `5 1 R` read as null, the +/// page fell out of a save, and the save was refused (PDFBOX-3785's `202097.pdf`, on pdfcraft#6767). +#[test] +fn a_rebuilt_index_keeps_a_reused_number_the_document_reaches() { + let bytes = rebuilt_objects(&[ + (1, 0, "<< /Type /Catalog /Pages 2 0 R >>"), + (5, 0, "27"), + (6, 0, &format!("<< /Length 5 0 R >>\nstream\n{OTHER}\nendstream")), + (2, 0, "<< /Type /Pages /Kids [3 0 R 5 1 R] /Count 2 >>"), + (3, 0, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 6 0 R >>"), + (5, 1, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 7 0 R >>"), + (7, 0, &format!("<< /Length {} >>\nstream\n{COVER}\nendstream", COVER.len())), + ]); + assert_eq!(pages_drawn_by_renderer(bytes.clone()), [OTHER, COVER]); + let doc = Document::open(bytes).unwrap(); + assert!(doc.reconstructed()); + assert!(!doc.rebuilt_with_unsettled_generations(), "only generation 1 is reached"); + assert_eq!(doc.get(ObjRef::new(5, 1)).as_dict().and_then(|d| d.name(b"Type")), Some(&b"Page"[..])); + let saved = Arc::new(write_full(&doc, &SaveOptions::default()).unwrap()); + assert_eq!(pages_drawn_by_renderer(saved), [OTHER, COVER]); + assert!(doc.repair_log().iter().any(|line| line.contains("left out 1 objects") && line.contains("(5 0)")), "{:?}", doc.repair_log()); +} + +/// `4 0 R` and `4 1 R` both drawn: one slot cannot hold both, so the renderer's generation-0 +/// rule stands, the other body is noted as left out, and the document says so. +#[test] +fn a_rebuilt_index_keeps_generation_0_when_the_document_reaches_both() { + let doc = Document::open(rebuilt_page("[4 0 R 4 1 R]", &[(0, COVER), (1, OTHER)])).unwrap(); + assert!(matches!(&*doc.get(ObjRef::new(4, 0)), Object::Stream(s) if &*s.raw == COVER.as_bytes())); + assert_eq!(*doc.get(ObjRef::new(4, 1)), Object::Null); + assert!(doc.repair_log().iter().any(|line| line.contains("left out 1 objects") && line.contains("(4 1)")), "{:?}", doc.repair_log()); + assert!(doc.rebuilt_with_unsettled_generations()); +} + +/// The only trailer names `/Root 1 1 R`, and both `1 0 obj` and `1 1 obj` are catalogs. The +/// renderer roots the document at `1 1 obj`; the trailer used to be judged against the +/// generation-0 slot alone, so it was passed over and cos rooted the document at `1 0 obj` +/// (raised in review of pdfcraft#6767). +#[test] +fn a_rebuilt_index_roots_the_document_at_the_generation_its_trailer_names() { + let bytes = rebuilt_objects_rooted( + &[ + (1, 0, "<< /Type /Catalog /Pages 2 0 R >>"), + (2, 0, "<< /Type /Pages /Kids [3 0 R] /Count 1 >>"), + (3, 0, "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 200 200] /Contents 6 0 R >>"), + (6, 0, &format!("<< /Length {} >>\nstream\n{OTHER}\nendstream", OTHER.len())), + (1, 1, "<< /Type /Catalog /Pages 8 0 R >>"), + (8, 0, "<< /Type /Pages /Kids [9 0 R] /Count 1 >>"), + (9, 0, "<< /Type /Page /Parent 8 0 R /MediaBox [0 0 200 200] /Contents 7 0 R >>"), + (7, 0, &format!("<< /Length {} >>\nstream\n{COVER}\nendstream", COVER.len())), + ], + "1 1 R", + ); + assert_eq!(pages_drawn_by_renderer(bytes.clone()), [COVER]); + let doc = Document::open(bytes).unwrap(); + assert!(doc.reconstructed()); + assert_eq!(doc.root(), Some(ObjRef::new(1, 1))); + assert!(!doc.rebuilt_with_unsettled_generations(), "only generation 1 is reached"); + assert_eq!(drawn_by_cos(&doc).as_deref(), Some(COVER)); + let saved = Arc::new(write_full(&doc, &SaveOptions::default()).unwrap()); + assert_eq!(pages_drawn_by_renderer(saved), [COVER]); +} + +/// `4 0 obj` and `4 1 obj` are object streams holding different values for member 10, and the +/// document reaches `4 1 R` only. The renderer reads the member through the generation-0 stream, +/// as the rebuilt index listed it; moving the slot to `4 1 obj` made a full save read it from +/// the other stream (raised in review of pdfcraft#6767). The slot stays, and the document says a +/// generation it reaches is left out. +#[test] +fn a_rebuilt_index_keeps_the_object_stream_its_members_were_read_from() { + let members = |value: i32| { + let data = format!("10 0 {value}"); + format!("<< /Type /ObjStm /N 1 /First 5 /Length {} >>\nstream\n{data}\nendstream", data.len()) + }; + let bytes = rebuilt_objects(&[ + (1, 0, "<< /Type /Catalog /Pages 2 0 R /Value 10 0 R /Streams 4 1 R >>"), + (2, 0, "<< /Type /Pages /Kids [] /Count 0 >>"), + (4, 0, &members(41)), + (4, 1, &members(42)), + ]); + let oracle = hayro_syntax::Pdf::new(bytes.clone()).unwrap(); + assert_eq!(oracle.xref().get::(ObjectIdentifier::new(10, 0)), Some(41)); + let doc = Document::open(bytes).unwrap(); + assert!(doc.reconstructed()); + assert_eq!(*doc.get(ObjRef::new(10, 0)), Object::Int(41)); + // A full save renumbers: follow the catalog's `/Value`. + let saved = Document::open(Arc::new(write_full(&doc, &SaveOptions::default()).unwrap())).unwrap(); + let catalog = saved.get(saved.root().unwrap()); + assert_eq!(*saved.resolve(catalog.as_dict().unwrap().get(b"Value").unwrap()), Object::Int(41)); + assert!(doc.rebuilt_with_unsettled_generations(), "`4 1 R` is reached and reads as null"); +} From a19d58c5f6803411c1e6679b0a9218daf3607f11 Mon Sep 17 00:00:00 2001 From: ".CooCoo" Date: Wed, 18 Nov 2026 12:31:45 -0800 Subject: [PATCH 142/161] dds: read R16G16 as red and green, and open 16-bit textures at 16 bits (#2124) R16G16_UNORM stores a red and a green channel (https://learn.microsoft.com/en-us/windows/win32/api/dxgiformat/ne-dxgiformat-dxgi_format), but craft-dds read it as gray and alpha, so a two-channel normal map opened with its green channel as transparency. It now reads as RGB with blue 0, as BC5 and G16R16 channel masks already do. encode16 still writes two-channel 16-bit pixels as R16G16; they read back as red and green. craft-image-codecs decoded every DDS at 8 bits. Texture::bits_per_channel gives the widest stored channel, and textures wider than 8 bits (R16, R16G16, R16G16B16A16, legacy FourCC 36, and channel masks with a 10- or 16-bit field) now open as U16 through decode16. craft-raster-codecs' decode_levels does the same for every mip level. --- crates/dds/README.md | 21 ++--- crates/dds/src/lib.rs | 121 +++++++++++++++++++++----- crates/dds/tests/boundary.rs | 7 +- crates/image-codecs/README.md | 18 ++-- crates/image-codecs/src/codecs/dds.rs | 11 ++- crates/image-codecs/tests/it/dds.rs | 50 +++++++++++ crates/raster-codecs/README.md | 2 +- crates/raster-codecs/src/dds.rs | 20 +++-- crates/raster-codecs/tests/dds.rs | 23 +++++ 9 files changed, 223 insertions(+), 50 deletions(-) diff --git a/crates/dds/README.md b/crates/dds/README.md index 2dae26bc1..37eb70cc8 100644 --- a/crates/dds/README.md +++ b/crates/dds/README.md @@ -23,21 +23,22 @@ in part. Pixels are 8 bits per channel (`encode`, `Texture::decode`) or 16 bits per channel (`encode16`, `Texture::decode16`), interleaved and top row first, in one of four `Layout`s (`Gray`, `GrayA`, `Rgb`, `Rgba`). BC1 to BC3 are written from RGBA, BC4 from gray and BC5 from RGB -(`DdsCompression::layout` names it). BC4 reads back as gray. BC5 stores red and green only, so it -reads back as RGB with blue 0: for a tangent-space normal map, X and Y, with Z left to the -renderer. ATI2 files are read as Microsoft's tools read them (red in the first block), not with -the two blocks swapped as FFmpeg does. +(`DdsCompression::layout` names it). BC4 reads back as gray. BC5 and `R16G16_UNORM` store red +and green only, so they read as RGB with blue 0: for a tangent-space normal map, X and Y, with Z +left to the renderer. ATI2 files are read as Microsoft's tools read them (red in the first block), +not with the two blocks swapped as FFmpeg does. From 16-bit pixels, uncompressed textures keep 16 bits per channel: gray as `R16_UNORM`, gray and alpha (or any two channels, such as a normal map's X and Y) as `R16G16_UNORM`, RGB (alpha -65535) and RGBA as `R16G16B16A16_UNORM`, all with the DX10 header; they read back as gray, gray -and alpha, and RGBA. BC4 and BC5 keep 8-bit endpoints, as the formats store them, but pick them -and the indices against the 16-bit values, so a smooth height or normal map has less error than -when its 8-bit rounding is compressed. BC1 to BC3 round the pixels to 8 bits first. +65535) and RGBA as `R16G16B16A16_UNORM`, all with the DX10 header; they read back as gray, RGB +with blue 0 (red and green holding the two channels), and RGBA. BC4 and BC5 keep 8-bit +endpoints, as the formats store them, but pick them and the indices against the 16-bit values, +so a smooth height or normal map has less error than when its 8-bit rounding is compressed. BC1 to BC3 round the pixels to 8 bits first. `Texture::decode` rounds 16-bit formats to 8 bits; `Texture::decode16` returns them exactly, rescales channel-mask fields from their own width (a 16-bit luminance field exactly), -interpolates BC4 and BC5 at 16 bits, and scales BC1 to BC3 by 257. `craft-image-codecs` still -reads DDS at 8 bits. +interpolates BC4 and BC5 at 16 bits, and scales BC1 to BC3 by 257. `Texture::bits_per_channel` +gives the widest stored channel; `craft-image-codecs` reads a texture through `decode16` when it +is wider than 8 bits. ```rust use craft_dds::{DdsCompression, DdsOptions, Layout, Texture}; diff --git a/crates/dds/src/lib.rs b/crates/dds/src/lib.rs index 685ce81dd..4eb8ff994 100644 --- a/crates/dds/src/lib.rs +++ b/crates/dds/src/lib.rs @@ -5,9 +5,10 @@ //! described by channel masks, RGB or luminance, or 16 bits per channel: `R16_UNORM`, //! `R16G16_UNORM`, `R16G16B16A16_UNORM`) or block-compressed BC1/BC2/BC3 //! (DXT1/DXT3/DXT5) or unsigned BC4/BC5 (ATI1/ATI2), with or without the DX10 header. BC4 opens -//! as gray and BC5 as RGB with blue 0: BC5 stores only red and green, and a normal map's Z is -//! left for the caller to reconstruct. Cubemaps, texture arrays and volume textures are refused -//! rather than flattened to one face or slice, and signed (SNORM) BC4/BC5 is refused too. +//! as gray, and BC5 and `R16G16_UNORM` as RGB with blue 0: they store only red and green, and a +//! normal map's Z is left for the caller to reconstruct. Cubemaps, texture arrays and volume +//! textures are refused rather than flattened to one face or slice, and signed (SNORM) BC4/BC5 is +//! refused too. //! //! Writing: a 2D texture, uncompressed (BGRA or BGRX, luminance with or without alpha), //! BC1/BC2/BC3 or BC4/BC5, optionally with its full mip chain. The legacy header describes all @@ -18,7 +19,7 @@ //! Pixels are 8 bits per channel ([`encode`], [`Texture::decode`]) or 16 bits per channel //! ([`encode16`], [`Texture::decode16`]), interleaved, top row first, in one of the [`Layout`]s. //! Uncompressed 16-bit pixels are written with the DX10 header as `R16_UNORM` (gray), -//! `R16G16_UNORM` (gray and alpha, or any two channels such as a normal map's X and Y) or +//! `R16G16_UNORM` (two channels, such as a normal map's X and Y, read back as red and green) or //! `R16G16B16A16_UNORM` (RGB, with alpha 65535, and RGBA). BC4 and BC5 fit their 8-bit endpoints //! and their indices to the 16-bit values; BC1 to BC3 round the pixels to 8 bits first. The //! crate has no dependencies; the `parallel` feature encodes and decodes rows of blocks (and mip @@ -269,7 +270,8 @@ enum Encoding { Bc1, Bc2, Bc3, - /// 16 bits per channel, little-endian: gray, gray and alpha (or red and green), or RGBA. + /// 16 bits per channel, little-endian: gray, red and green (read as RGB with blue 0), or + /// RGBA. Unorm16 { channels: usize, }, @@ -301,7 +303,7 @@ impl Encoding { Encoding::Masks { gray: true, .. } => Layout::Gray, Encoding::Masks { fields, .. } if fields[3].is_none() => Layout::Rgb, Encoding::Unorm16 { channels: 1 } => Layout::Gray, - Encoding::Unorm16 { channels: 2 } => Layout::GrayA, + Encoding::Unorm16 { channels: 2 } => Layout::Rgb, Encoding::Bc4 => Layout::Gray, Encoding::Bc5 => Layout::Rgb, _ => Layout::Rgba, @@ -479,6 +481,17 @@ impl<'a> Texture<'a> { self.levels } + /// Bits in the widest channel the file stores: 16 for the 16-bit formats, the widest mask + /// field for other uncompressed pixels, 8 for block compressions. Above 8, + /// [`Texture::decode`] rounds samples that [`Texture::decode16`] keeps. + pub fn bits_per_channel(&self) -> u32 { + match self.encoding { + Encoding::Unorm16 { .. } => 16, + Encoding::Masks { fields, .. } => fields.iter().flatten().map(|f| u32::BITS - f.max.leading_zeros()).max().unwrap_or(0), + Encoding::Bc1 | Encoding::Bc2 | Encoding::Bc3 | Encoding::Bc4 | Encoding::Bc5 => 8, + } + } + /// The encoding named by the header, including unsigned BC4 and BC5. pub fn compression(&self) -> DdsCompression { match self.encoding { @@ -534,11 +547,8 @@ impl<'a> Texture<'a> { match encoding { Encoding::Masks { bytes, gray, fields } => decode_masks(Masks { bytes, gray, fields }, data, row_pitch, &mut pixels, row_size), // Each 16-bit sample rounded to the nearest 8-bit level. - Encoding::Unorm16 { .. } => rows(&mut pixels, row_size, |y, dst| { - let src = data.get(y * row_pitch..).unwrap_or_default(); - for (d, s) in dst.iter_mut().zip(src.as_chunks::<2>().0) { - *d = to8(u16::from_le_bytes(*s)); - } + Encoding::Unorm16 { channels: stored } => rows(&mut pixels, row_size, |y, dst| { + unorm16_row(data.get(y * row_pitch..).unwrap_or_default(), stored, dst, to8); }), Encoding::Bc1 | Encoding::Bc2 | Encoding::Bc3 | Encoding::Bc4 | Encoding::Bc5 => { decode_blocks(encoding, data, (w, channels), &mut pixels) @@ -565,11 +575,8 @@ impl<'a> Texture<'a> { let mut pixels = alloc_zeroed(total / 2)?; match encoding { Encoding::Masks { bytes, gray, fields } => decode_masks16(Masks { bytes, gray, fields }, data, row_pitch, &mut pixels, row_size), - Encoding::Unorm16 { .. } => rows(&mut pixels, row_size, |y, dst| { - let src = data.get(y * row_pitch..).unwrap_or_default(); - for (d, s) in dst.iter_mut().zip(src.as_chunks::<2>().0) { - *d = u16::from_le_bytes(*s); - } + Encoding::Unorm16 { channels: stored } => rows(&mut pixels, row_size, |y, dst| { + unorm16_row(data.get(y * row_pitch..).unwrap_or_default(), stored, dst, |v| v); }), Encoding::Bc4 | Encoding::Bc5 => decode_value_blocks16(encoding, data, (w, channels), &mut pixels), Encoding::Bc1 | Encoding::Bc2 | Encoding::Bc3 => { @@ -603,6 +610,21 @@ impl<'a> Texture<'a> { } } +/// One row of 16-bit samples, `stored` a pixel and little-endian, into `dst` through `convert`. +/// Two stored channels are red and green: they fill RGB pixels with blue 0. +fn unorm16_row(src: &[u8], stored: usize, dst: &mut [T], convert: impl Fn(u16) -> T) { + let src = src.as_chunks::<2>().0; + if stored == 2 { + for (d, s) in dst.as_chunks_mut::<3>().0.iter_mut().zip(src.as_chunks::<2>().0) { + *d = [convert(u16::from_le_bytes(s[0])), convert(u16::from_le_bytes(s[1])), T::default()]; + } + } else { + for (d, s) in dst.iter_mut().zip(src) { + *d = convert(u16::from_le_bytes(*s)); + } + } +} + /// A 16-bit sample rounded to the nearest 8-bit level. fn to8(v: u16) -> u8 { ((u32::from(v) + 128) / 257) as u8 @@ -946,7 +968,8 @@ pub fn encode(width: u32, height: u32, layout: Layout, pixels: &[u8], options: D /// [`encode`] for 16-bit pixels. Uncompressed, they keep 16 bits per channel: gray as /// `R16_UNORM`, gray and alpha (or any two channels, such as a normal map's X and Y) as /// `R16G16_UNORM`, RGB (with alpha 65535) and RGBA as `R16G16B16A16_UNORM`, all with the DX10 -/// header. BC4 and BC5 pick their endpoints and indices against the 16-bit values; BC1 to BC3 +/// header. `R16G16_UNORM` holds red and green, so two channels read back as RGB with blue 0. BC4 +/// and BC5 pick their endpoints and indices against the 16-bit values; BC1 to BC3 /// round the pixels to 8 bits and write what [`encode`] writes from them. pub fn encode16(width: u32, height: u32, layout: Layout, pixels: &[u16], options: DdsOptions) -> Result, DdsError> { if width == 0 || height == 0 { @@ -1293,6 +1316,24 @@ mod tests { assert_eq!(Texture::parse(&file).and_then(|t| t.decode()), Ok(vec![7; 64])); } + #[test] + fn bits_per_channel_is_the_widest_stored_channel() { + let options = |compression| DdsOptions { compression, mipmaps: false }; + let gray = encode(4, 4, Layout::Gray, &[7; 16], options(DdsCompression::None)).unwrap(); + let bc4 = encode(4, 4, Layout::Gray, &[7; 16], options(DdsCompression::Bc4)).unwrap(); + let r16 = encode16(4, 4, Layout::Gray, &[7; 16], options(DdsCompression::None)).unwrap(); + let bits = |file: &[u8]| Texture::parse(file).unwrap().bits_per_channel(); + assert_eq!((bits(&gray), bits(&bc4), bits(&r16)), (8, 8, 16)); + // A2R10G10B10 as masks: 10 bits. + let mut a2r10 = gray[..HEADER].to_vec(); + a2r10[80..84].copy_from_slice(&(DDPF_RGB | DDPF_ALPHAPIXELS).to_le_bytes()); + a2r10[88..92].copy_from_slice(&32u32.to_le_bytes()); + for (i, mask) in [0x3ff0_0000u32, 0xf_fc00, 0x3ff, 0xc000_0000].into_iter().enumerate() { + a2r10[92 + 4 * i..96 + 4 * i].copy_from_slice(&mask.to_le_bytes()); + } + assert_eq!(bits(&a2r10), 10); + } + /// Samples spread over the whole 16-bit range, `channels` a pixel. fn spread16(width: usize, height: usize, channels: usize) -> Vec { (0..width * height * channels).map(|i| (i as u32 * 7919 % 65536) as u16).collect() @@ -1303,13 +1344,17 @@ mod tests { let (w, h) = (7, 5); for (layout, dxgi, read_as, pixel_bytes) in [ (Layout::Gray, 56, Layout::Gray, 2), - (Layout::GrayA, 35, Layout::GrayA, 4), + (Layout::GrayA, 35, Layout::Rgb, 4), (Layout::Rgb, 11, Layout::Rgba, 8), (Layout::Rgba, 11, Layout::Rgba, 8), ] { let pixels = spread16(w, h, layout.channels()); - let want: Vec = - if layout == Layout::Rgb { pixels.chunks(3).flat_map(|p| [p[0], p[1], p[2], 65535]).collect() } else { pixels.clone() }; + let want: Vec = match layout { + // R16G16 holds red and green: the two channels read back with blue 0. + Layout::GrayA => pixels.chunks(2).flat_map(|p| [p[0], p[1], 0]).collect(), + Layout::Rgb => pixels.chunks(3).flat_map(|p| [p[0], p[1], p[2], 65535]).collect(), + _ => pixels.clone(), + }; for mipmaps in [false, true] { let file = encode16(w as u32, h as u32, layout, &pixels, DdsOptions { compression: DdsCompression::None, mipmaps }).unwrap(); assert_eq!(le32(&file, HEADER).unwrap(), dxgi, "{layout:?}"); @@ -1358,6 +1403,7 @@ mod tests { let texture = Texture::parse(&l16).unwrap(); assert_eq!(texture.decode16().unwrap(), [300, 65000]); assert_eq!(texture.decode().unwrap(), [1, 253]); + assert_eq!(texture.bits_per_channel(), 16); // G16R16 as masks (red low, green high) reads as RGB at 16 bits, blue 0. let mut g16r16 = l16[..HEADER].to_vec(); g16r16[80..84].copy_from_slice(&DDPF_RGB.to_le_bytes()); @@ -1369,6 +1415,36 @@ mod tests { assert_eq!((texture.layout(), texture.decode16().unwrap()), (Layout::Rgb, vec![300, 40000, 0, 65535, 7, 0])); } + /// A DX10 `R16G16` file (DXGI 35, or 33 typeless) of `levels` levels: red then green, + /// little-endian, each level packed after the one above. + fn rg16_file(width: u32, height: u32, dxgi: u32, levels: u32, samples: &[u16]) -> Vec { + let mut file = header(width, height, Store::R16, levels, 0).unwrap().to_vec(); + file[HEADER..HEADER + 4].copy_from_slice(&dxgi.to_le_bytes()); + file[20..24].copy_from_slice(&(width * 4).to_le_bytes()); + file.extend(samples.iter().flat_map(|v| v.to_le_bytes())); + file + } + + #[test] + fn r16g16_textures_read_as_red_and_green_with_blue_0() { + // R16G16 stores red and green and no alpha: blue is 0, as for BC5 and G16R16 masks. + let (w, h) = (3, 2); + let top = spread16(w, h, 2); + let rgb: Vec = top.chunks(2).flat_map(|p| [p[0], p[1], 0]).collect(); + for dxgi in [35, 33] { + // 3×2, then 1×1 with red 11 and green 22. + let file = rg16_file(w as u32, h as u32, dxgi, 2, &[top.clone(), vec![11, 22]].concat()); + let texture = Texture::parse(&file).unwrap(); + assert_eq!((texture.layout(), texture.bits_per_channel()), (Layout::Rgb, 16), "DXGI {dxgi}"); + assert_eq!(texture.decode16().unwrap(), rgb, "DXGI {dxgi}"); + assert_eq!(texture.decode().unwrap(), rgb.iter().map(|&v| to8(v)).collect::>(), "DXGI {dxgi}"); + let level = texture.mip_level(1).unwrap(); + assert_eq!(level.decode16().unwrap(), [11, 22, 0]); + assert!(matches!(level.decode16_with_limit(5), Err(DdsError::LimitExceeded { required: 6, limit: 5 }))); + assert!(Texture::parse(&file[..file.len() - 5]).unwrap().decode16().is_err()); + } + } + #[test] fn typeless_and_padded_16_bit_textures_are_read() { // Rows padded to 8 bytes, as other writers may store them: the padding is skipped. @@ -1380,8 +1456,9 @@ mod tests { padded.extend_from_slice(&[0xee, 0xee]); } assert_eq!(Texture::parse(&padded).unwrap().decode16().unwrap(), [1, 2, 3, 4, 5, 6]); - // The typeless forms read as the UNORM ones; typeless RGBA refuses premultiplied alpha too. - for (unorm, typeless, layout) in [(56u32, 53u32, Layout::Gray), (35, 33, Layout::GrayA), (11, 9, Layout::Rgba)] { + // The typeless forms read as the UNORM ones (R16G16's in its own test); typeless RGBA + // refuses premultiplied alpha too. + for (unorm, typeless, layout) in [(56u32, 53u32, Layout::Gray), (11, 9, Layout::Rgba)] { let pixels = spread16(3, 2, layout.channels()); let file = encode16(3, 2, layout, &pixels, DdsOptions::default()).unwrap(); assert_eq!(le32(&file, HEADER).unwrap(), unorm); diff --git a/crates/dds/tests/boundary.rs b/crates/dds/tests/boundary.rs index 3a891f73a..1dab9b33b 100644 --- a/crates/dds/tests/boundary.rs +++ b/crates/dds/tests/boundary.rs @@ -74,14 +74,17 @@ fn mip_views_reuse_shared_decoders_and_keep_their_bounds() { #[test] fn sixteen_bit_mip_views_preserve_samples_and_stored_offsets() { for (layout, channels) in [(Layout::Gray, 1), (Layout::GrayA, 2), (Layout::Rgba, 4)] { + // Two channels are stored as R16G16 and read back as red and green with blue 0. + let read = + |samples: Vec| -> Vec { if channels == 2 { samples.chunks(2).flat_map(|p| [p[0], p[1], 0]).collect() } else { samples } }; let pixels: Vec = (0..16).flat_map(|p| (0..channels).map(move |c| (p * 400 + c * 31 + 7) as u16)).collect(); let bytes = craft_dds::encode16(4, 4, layout, &pixels, DdsOptions { compression: DdsCompression::None, mipmaps: true }).unwrap(); let texture = Texture::parse(&bytes).unwrap(); assert_eq!(texture.compression(), DdsCompression::None); assert_eq!(texture.levels(), 3); - assert_eq!(texture.decode16().unwrap(), pixels); + assert_eq!(texture.decode16().unwrap(), read(pixels)); for (index, averages) in [(1, vec![1000, 1800, 4200, 5000]), (2, vec![3000])] { - let expected: Vec = averages.iter().flat_map(|p| (0..channels).map(move |c| (p + c * 31 + 7) as u16)).collect(); + let expected = read(averages.iter().flat_map(|p| (0..channels).map(move |c| (p + c * 31 + 7) as u16)).collect()); let level = texture.mip_level(index).unwrap(); let budget = (expected.len() * 2) as u64; assert!(matches!(level.decode16_with_limit(budget - 1), Err(craft_dds::DdsError::LimitExceeded { .. }))); diff --git a/crates/image-codecs/README.md b/crates/image-codecs/README.md index 0fb86e31c..7d588336f 100644 --- a/crates/image-codecs/README.md +++ b/crates/image-codecs/README.md @@ -46,7 +46,7 @@ craft-image-codecs = { git = "https://github.com/storytold/craft-libs", rev = "< | BMP | yes | yes | U8 | RGB, RGBA | yes | no | no | no | yes (header pixels per metre) | no | no | `image` | | TGA | yes | yes | U8 | Gray, GrayA, RGB, RGBA | yes | no | no | no | no | no | no | `image` (read) / built in (RLE write, one scanline per packet) | | ICO | yes | yes | U8 | RGBA (at most 256×256) | yes | no | no | no | no | no | no | `image` | -| DDS | yes | yes | U8 | Gray, GrayA, RGB, RGBA | yes | no | no | no | no | no | yes (BC1–BC5, when `EncodeOptions::dds` asks for one) | [`craft-dds`](../dds/README.md) | +| DDS | yes | yes | U8, U16 (U16 read only) | Gray, GrayA, RGB, RGBA | yes | no | no | no | no | no | yes (BC1–BC5, when `EncodeOptions::dds` asks for one) | [`craft-dds`](../dds/README.md) | | Netpbm (PBM/PGM/PPM/PAM/PFM) | yes | yes | U8, U16, F32 (float: PFM gray/RGB only) | all six (PAM `CMYK`/`CMYK_ALPHA`) | yes (PAM) | no | no | no | no | no | no | built in | | QOI | yes | yes | U8 | RGB, RGBA | yes | no | no | no | no | no | no | `image` | | OpenEXR | yes | yes | F16, F32 | Y, YA, RGB, RGBA | yes | no | no | no | no | no | no | `exr` | @@ -97,13 +97,15 @@ embedded table (and a PNG's transparent entry) so `photocraft-io` can open them * **DDS** (through [`craft-dds`](../dds/README.md), whose `DdsCompression` and `DdsOptions` are re-exported). The top mip level of a 2D texture: uncompressed pixels described by channel masks - (8, 16, 24 or 32 bits, RGB or luminance, with or without alpha), BC1/BC2/BC3 (DXT1/DXT3/DXT5) or - unsigned BC4/BC5 (ATI1/ATI2, BC4U/BC5U, DXGI 79, 80, 82 and 83), with the legacy header or the - DX10 one. BC4 opens as Gray and BC5 as RGB with blue 0 (BC5 stores only red and green; a normal - map's Z is not reconstructed). ATI2 is read as Microsoft reads it, red in the first block. - Uncompressed row padding follows the header's `DDSD_PITCH` stride; without that flag, rows are - tightly packed. Everything opens as U8; the other mip levels are ignored. Cubemaps, texture - arrays, volume textures, premultiplied-alpha textures (DXT2/DXT4, or DX10 alpha mode 2), signed (SNORM) BC4/BC5 and other formats + (8, 16, 24 or 32 bits, RGB or luminance, with or without alpha), 16 bits per channel + (`R16_UNORM`, `R16G16_UNORM`, `R16G16B16A16_UNORM` and their typeless forms, legacy FourCC 36), + BC1/BC2/BC3 (DXT1/DXT3/DXT5) or unsigned BC4/BC5 (ATI1/ATI2, BC4U/BC5U, DXGI 79, 80, 82 and 83), + with the legacy header or the DX10 one. BC4 opens as Gray, and BC5 and `R16G16_UNORM` as RGB + with blue 0 (they store only red and green; a normal map's Z is not reconstructed). ATI2 is read + as Microsoft reads it, red in the first block. Uncompressed row padding follows the header's + `DDSD_PITCH` stride; without that flag, rows are tightly packed. The 16-bit formats and masks + with a channel wider than 8 bits open as U16, everything else as U8; the other mip levels are + ignored. Cubemaps, texture arrays, volume textures, premultiplied-alpha textures (DXT2/DXT4, or DX10 alpha mode 2), signed (SNORM) BC4/BC5 and other formats (BC6H, BC7, float) are refused with `CodecError::Unsupported`. Writing (`EncodeOptions::dds`) makes a 2D texture: uncompressed by default (BGRA, BGRX, or luminance with or without alpha, so it reads back exactly), BC1/BC2/BC3 (from RGBA) with the legacy header, or BC4 (from gray, for diff --git a/crates/image-codecs/src/codecs/dds.rs b/crates/image-codecs/src/codecs/dds.rs index 4c8475ea8..d8130f05a 100644 --- a/crates/image-codecs/src/codecs/dds.rs +++ b/crates/image-codecs/src/codecs/dds.rs @@ -36,12 +36,19 @@ pub(crate) fn has_additional_images(bytes: &[u8]) -> Result { craft_dds::has_additional_images(bytes).map_err(error) } +/// Channels wider than 8 bits (the 16-bit formats, 10- and 16-bit masks) open at 16 bits. pub(crate) fn decode(b: &[u8], limits: &Limits) -> Result { let texture = Texture::parse(b).map_err(error)?; let layout = channel_layout(texture.layout()); - limits.check(texture.width(), texture.height(), layout, SampleType::U8)?; + let (width, height) = (texture.width(), texture.height()); + if texture.bits_per_channel() > 8 { + limits.check(width, height, layout, SampleType::U16)?; + let pixels = texture.decode16_with_limit(limits.max_alloc).map_err(error)?; + return Image::from_u16(width, height, layout, &pixels); + } + limits.check(width, height, layout, SampleType::U8)?; let pixels = texture.decode_with_limit(limits.max_alloc).map_err(error)?; - Image::from_u8(texture.width(), texture.height(), layout, pixels) + Image::from_u8(width, height, layout, pixels) } pub(crate) fn encode(image: &Image, plan: Plan, opts: &EncodeOptions) -> Result, CodecError> { diff --git a/crates/image-codecs/tests/it/dds.rs b/crates/image-codecs/tests/it/dds.rs index d174255b2..23b6211ac 100644 --- a/crates/image-codecs/tests/it/dds.rs +++ b/crates/image-codecs/tests/it/dds.rs @@ -401,6 +401,56 @@ fn decodes_bc5_as_red_and_green_with_blue_zero() { } } +#[test] +fn sixteen_bit_textures_open_at_16_bits() { + let samples = |n: usize| -> Vec { (0..n).map(|i| (i as u32 * 7919 % 65536) as u16).collect() }; + let le = |s: &[u16]| -> Vec { s.iter().flat_map(|v| v.to_le_bytes()).collect() }; + let (gray, rg, rgba) = (samples(6), samples(12), samples(24)); + let mut files = vec![]; + for (dxgi, stored) in [(56, &gray), (53, &gray), (35, &rg), (33, &rg), (11, &rgba), (9, &rgba)] { + let mut b = dx10(3, 2, dxgi, 3, 0, 1); + b.extend(le(stored)); + files.push((format!("DXGI {dxgi}"), b)); + } + // D3DFMT_A16B16G16R16, the legacy FourCC 36. + let mut b = compressed(3, 2, &36u32.to_le_bytes()); + b.extend(le(&rgba)); + files.push(("FourCC 36".into(), b)); + // A 16-bit luminance mask (L16). + let mut b = header(3, 2, DDPF_LUMINANCE, &[0; 4], 16, [0xffff, 0, 0, 0]); + b.extend(le(&gray)); + files.push(("L16".into(), b)); + let rgb: Vec = rg.chunks(2).flat_map(|p| [p[0], p[1], 0]).collect(); + for (name, b) in files { + let im = decode(&b).unwrap_or_else(|e| panic!("{name}: {e}")); + assert_eq!(im.sample_type(), SampleType::U16, "{name}"); + let (layout, want) = match name.as_str() { + "DXGI 56" | "DXGI 53" | "L16" => (ChannelLayout::Gray, &gray), + // R16G16 is red and green, not gray and alpha: blue is 0, as for BC5. + "DXGI 35" | "DXGI 33" => (ChannelLayout::Rgb, &rgb), + _ => (ChannelLayout::Rgba, &rgba), + }; + assert_eq!((im.dimensions(), im.layout()), ((3, 2), layout), "{name}"); + assert_eq!(&im.to_u16_samples().unwrap(), want, "{name}"); + } + // The limit counts two bytes a sample. + let mut b = dx10(3, 2, 11, 3, 0, 1); + b.extend(le(&rgba)); + let limits = |max_alloc| DecodeOptions { limits: Limits { max_alloc, ..Limits::default() }, ..DecodeOptions::default() }; + assert!(matches!(decode_as_with(Format::Dds, &b, &limits(47)), Err(CodecError::LimitExceeded(_)))); + assert!(decode_as_with(Format::Dds, &b, &limits(48)).is_ok()); + // A2R10G10B10 masks: 10-bit fields open at 16 bits, rescaled from their own width. + let mut b = header(1, 1, DDPF_RGB | DDPF_ALPHAPIXELS, &[0; 4], 32, [0x3ff0_0000, 0xf_fc00, 0x3ff, 0xc000_0000]); + b.extend((1023u32 << 20 | 512 << 10 | 1 | 3 << 30).to_le_bytes()); + let im = decode(&b).unwrap(); + assert_eq!((im.sample_type(), im.layout()), (SampleType::U16, ChannelLayout::Rgba)); + assert_eq!(im.to_u16_samples().unwrap(), [65535, 32800, 64, 65535]); + // Fields of 8 bits or fewer still open at 8 bits. + let mut b = header(1, 1, DDPF_LUMINANCE | DDPF_ALPHAPIXELS, &[0; 4], 16, [0xff, 0, 0, 0xff00]); + b.extend([7, 9]); + assert_eq!(decode(&b).unwrap().sample_type(), SampleType::U8); +} + #[test] fn refuses_signed_bc4_and_bc5() { for mut b in [compressed(4, 4, b"BC4S"), compressed(4, 4, b"BC5S"), dx10(4, 4, 81, 3, 0, 1), dx10(4, 4, 84, 3, 0, 1)] { diff --git a/crates/raster-codecs/README.md b/crates/raster-codecs/README.md index eec7ac0d5..fe8deace9 100644 --- a/crates/raster-codecs/README.md +++ b/crates/raster-codecs/README.md @@ -11,7 +11,7 @@ craft-raster-codecs = { git = "https://github.com/storytold/craft-libs", rev = " | Module | Format | Read | Write | |---|---|---|---| -| `dds` | DirectDraw Surface: uncompressed masks, luminance, BC1/BC2/BC3 (DXT1/3/5), unsigned BC4/BC5 (ATI1/2, as gray and as RGB with blue 0), legacy and DX10 headers; top mip level | yes | uncompressed and BC1-BC3, optional mip chain | +| `dds` | DirectDraw Surface: uncompressed masks, luminance, BC1/BC2/BC3 (DXT1/3/5), unsigned BC4/BC5 (ATI1/2, as gray and as RGB with blue 0), 16-bit `R16`, `R16G16` (as RGB with blue 0) and `R16G16B16A16` at 16 bits, legacy and DX10 headers; top mip level | yes | uncompressed and BC1-BC3, optional mip chain | | `dicom` | DICOM Part 10, uncompressed transfer syntaxes: MONOCHROME1/2, RGB, YBR_FULL, 8 and 16 bits | yes | no | | `fits` | FITS primary HDU or first `IMAGE` extension, every `BITPIX`, `BSCALE`/`BZERO`, header cards as text | yes | gray or RGB; 8-bit, 16-bit, float | | `iff` | IFF ILBM (1 to 8 planes with a palette, 24 planes, mask plane or transparent colour) and PBM, stored or ByteRun1; HAM and EHB are refused | yes | 24-plane ILBM with an optional mask plane | diff --git a/crates/raster-codecs/src/dds.rs b/crates/raster-codecs/src/dds.rs index 8f6f60c91..aca700f7a 100644 --- a/crates/raster-codecs/src/dds.rs +++ b/crates/raster-codecs/src/dds.rs @@ -121,9 +121,11 @@ pub fn info(bytes: &[u8]) -> Result { } /// Decode the declared mip chain, charging all retained pixels to one output-byte budget. -/// Cancellation is checked before each level; no partial chain is returned. +/// Cancellation is checked before each level; no partial chain is returned. Channels wider than +/// 8 bits decode at 16 bits, as [`decode`] opens the top level. pub fn decode_levels(bytes: &[u8], limits: &Limits, cancel: &dyn Fn() -> bool) -> Result, CodecError> { let texture = craft_dds::Texture::parse(bytes).map_err(dds_error)?; + let wide = texture.bits_per_channel() > 8; let mut images = Vec::new(); images.try_reserve_exact(texture.levels() as usize).map_err(|_| CodecError::LimitExceeded("cannot allocate DDS mip list".into()))?; let mut remaining = limits.max_alloc; @@ -134,10 +136,18 @@ pub fn decode_levels(bytes: &[u8], limits: &Limits, cancel: &dyn Fn() -> bool) - let level = texture.mip_level(i).map_err(dds_error)?; let layout = layout(&level); let level_limits = Limits { max_alloc: remaining, ..*limits }; - level_limits.check(level.width(), level.height(), layout, SampleType::U8)?; - let pixels = level.decode_with_limit(remaining).map_err(dds_error)?; - remaining = remaining.checked_sub(pixels.len() as u64).ok_or_else(|| CodecError::LimitExceeded("DDS mip chain exceeds max_alloc".into()))?; - images.push(Image::from_u8(level.width(), level.height(), layout, pixels)?); + let exceeded = || CodecError::LimitExceeded("DDS mip chain exceeds max_alloc".into()); + if wide { + level_limits.check(level.width(), level.height(), layout, SampleType::U16)?; + let pixels = level.decode16_with_limit(remaining).map_err(dds_error)?; + remaining = remaining.checked_sub(pixels.len() as u64 * 2).ok_or_else(exceeded)?; + images.push(Image::from_u16(level.width(), level.height(), layout, &pixels)?); + } else { + level_limits.check(level.width(), level.height(), layout, SampleType::U8)?; + let pixels = level.decode_with_limit(remaining).map_err(dds_error)?; + remaining = remaining.checked_sub(pixels.len() as u64).ok_or_else(exceeded)?; + images.push(Image::from_u8(level.width(), level.height(), layout, pixels)?); + } } Ok(images) } diff --git a/crates/raster-codecs/tests/dds.rs b/crates/raster-codecs/tests/dds.rs index 986762722..3d0cc0868 100644 --- a/crates/raster-codecs/tests/dds.rs +++ b/crates/raster-codecs/tests/dds.rs @@ -501,6 +501,29 @@ fn mip_import_rejects_truncation_and_bounds_declared_count() { assert_eq!(dds::decode_levels(&b, &Limits::default(), &|| false).unwrap().len(), 1); } +#[test] +fn sixteen_bit_mip_chains_keep_16_bits_and_r16g16_is_red_and_green() { + // R16G16_UNORM, 2×2 then 1×1: red and green, blue 0, at 16 bits like the top level `decode` opens. + let mut b = dx10(2, 2, 35, 3, 0, 1); + b[8..12].copy_from_slice(&(0x1007u32 | 0x2_0000).to_le_bytes()); + b[28..32].copy_from_slice(&2u32.to_le_bytes()); + let samples: [u16; 10] = [300, 40000, 1, 2, 3, 4, 5, 65535, 1000, 1002]; + b.extend(samples.iter().flat_map(|v| v.to_le_bytes())); + assert!(!dds::info(&b).unwrap().alpha); + let levels = dds::decode_levels(&b, &Limits::default(), &|| false).unwrap(); + let u16s = |img: &Image| -> Vec { img.data().as_chunks::<2>().0.iter().map(|c| u16::from_ne_bytes(*c)).collect() }; + for img in &levels { + assert_eq!((img.layout(), img.sample_type()), (ChannelLayout::Rgb, SampleType::U16)); + } + assert_eq!(u16s(&levels[0]), [300, 40000, 0, 1, 2, 0, 3, 4, 0, 5, 65535, 0]); + assert_eq!(u16s(&levels[1]), [1000, 1002, 0]); + assert_eq!(decode(&b).unwrap().data(), levels[0].data()); + // The budget counts two bytes a sample: 24 + 6 bytes. + let budget = |max_alloc| dds::decode_levels(&b, &Limits { max_alloc, ..Default::default() }, &|| false); + assert!(matches!(budget(29), Err(CodecError::LimitExceeded(_)))); + assert_eq!(budget(30).unwrap().len(), 2); +} + #[test] fn lower_mips_do_not_inherit_top_level_pitch() { let mut b = header(2, 2, DDPF_LUMINANCE, &[0; 4], 8, [0xff, 0, 0, 0]); From 5bd7eb9e88e8574a9cd011accaa9b4beaf4118ac Mon Sep 17 00:00:00 2001 From: mmazzieri Date: Wed, 18 Nov 2026 21:31:52 +0100 Subject: [PATCH 143/161] craft-rtf: read text boxes and drawing shapes from \shpinst (#2059) Ported from storytold/wordcraft#6242 (issue storytold/wordcraft#6212), written before the RTF reader moved here. Rectangles, ellipses, triangles, diamonds, block arrows and text boxes are read from the shape's properties (position, size, wrap, fill, line, text); the \defshp picture Word writes after a shape it read, also inside \shppict, is not imported as a second copy. Co-authored-by: Mauro Mazzieri --- crates/rtf/src/doc.rs | 13 +- crates/rtf/src/doc/parse.rs | 5 +- crates/rtf/src/doc/reader.rs | 24 +- crates/rtf/src/doc/shape.rs | 419 +++++++++++++++++++++++++++++ crates/rtf/src/doc/state.rs | 8 + crates/rtf/src/doc/tests_shapes.rs | 198 ++++++++++++++ crates/rtf/src/doc/text.rs | 8 +- crates/rtf/src/doc/word.rs | 10 +- 8 files changed, 676 insertions(+), 9 deletions(-) create mode 100644 crates/rtf/src/doc/shape.rs create mode 100644 crates/rtf/src/doc/tests_shapes.rs diff --git a/crates/rtf/src/doc.rs b/crates/rtf/src/doc.rs index 0426a4b7a..cdde2b716 100644 --- a/crates/rtf/src/doc.rs +++ b/crates/rtf/src/doc.rs @@ -27,9 +27,13 @@ //! `\fldlock` locking it, and `\chpgn` is a `PAGE` field; otherwise a field is its result's text), //! bookmarks, pictures and //! tables, footnotes, endnotes (`\footnote` with `\ftnalt`) and comments (author, initials, date, -//! text, the commented range). Group nesting, picture sizes and the number of notes and comments -//! are capped. Not kept: custom note marks, comment replies (`\atnparent`), notes inside -//! headers, footers and other notes, tables inside note and comment texts (flattened). +//! text, the commented range), drawing shapes and text boxes (`\shp`: rectangles, ellipses, +//! triangles, diamonds, block arrows and text boxes from `\shpinst`, with their place, wrapping, +//! fill, outline and text; their `\shprslt` copy and the `\defshp` picture after them are +//! skipped). Group nesting, picture sizes and the number of notes and comments are capped. Not +//! kept: custom note marks, comment replies (`\atnparent`), notes inside headers, footers, text +//! boxes and other notes, tables inside note and comment texts (flattened). Pictures, lines and +//! shapes in a group (`\shpgrp`) are read from their `\shprslt` copy. use std::collections::{HashMap, HashSet}; use std::sync::Arc; @@ -44,6 +48,7 @@ mod export; mod field; mod parse; mod reader; +mod shape; mod state; mod styles; mod text; @@ -64,3 +69,5 @@ use word::hyperlink; mod tests; #[cfg(test)] mod tests_fields; +#[cfg(test)] +mod tests_shapes; diff --git a/crates/rtf/src/doc/parse.rs b/crates/rtf/src/doc/parse.rs index 01978da1e..c51d402a4 100644 --- a/crates/rtf/src/doc/parse.rs +++ b/crates/rtf/src/doc/parse.rs @@ -30,7 +30,8 @@ struct Parsed { } /// `marks`: also read footnotes, endnotes and comments (as place markers in the flow's text, for -/// [`import`] to turn into objects); without it they are skipped. +/// [`import`] to turn into objects), and drawing shapes and text boxes; without it they are +/// skipped. fn parse_full(bytes: &[u8], marks: bool) -> Result { let start = bytes.iter().position(|b| !b.is_ascii_whitespace()).unwrap_or(0); if !bytes.get(start..).is_some_and(|b| b.starts_with(b"{\\rtf")) { @@ -38,6 +39,7 @@ fn parse_full(bytes: &[u8], marks: bool) -> Result { } let mut r = Reader::default(); r.marks_on = marks; + r.boxes_on = marks; let mut i = start; let mut first_in_group = false; let mut starred = false; @@ -236,6 +238,7 @@ fn finalize(mut r: Reader) -> Parsed { // truncated file. r.finish_multibyte(); r.end_sub(); + r.end_shape_text(); r.end_header_footer(); if r.para_has || !r.para.inlines.is_empty() { r.end_para(false); diff --git a/crates/rtf/src/doc/reader.rs b/crates/rtf/src/doc/reader.rs index 4c8a8b652..4bf83184b 100644 --- a/crates/rtf/src/doc/reader.rs +++ b/crates/rtf/src/doc/reader.rs @@ -11,7 +11,7 @@ use crate::date::{dttm_to_iso, iso_date}; pub(super) struct Reader { pub(super) st: St, stack: Vec, - overflow: usize, + pub(super) overflow: usize, pub(super) fonts: HashMap>, pub(super) font_cur: i32, pub(super) font_name: String, @@ -114,6 +114,17 @@ pub(super) struct Reader { pub(super) range_ends: HashSet, pub(super) claimed: HashSet, pub(super) synth: i32, + /// Drawing shapes and text boxes (only when the caller wants them), and the one being read. + pub(super) boxes_on: bool, + pub(super) shape: Option>, + /// The depth of the group the last shape closed in, when it was read (the picture of it Word + /// writes right after it is not a picture of its own); whether the picture being read follows + /// it, and whether it is marked `\defshp`. + pub(super) shape_read_at: Option, + pub(super) pict_after_shape: bool, + /// The depth of the open `\*\shppict` group: a picture in it stands where the group does. + pub(super) shppict_at: Option, + pub(super) pict_defshp: bool, } impl Reader { @@ -123,7 +134,7 @@ impl Reader { /// Set the output aside (a header or footer starts) or put it back (it ended), returning the /// other one. - fn swap_output(&mut self, mut o: Output) -> Output { + pub(super) fn swap_output(&mut self, mut o: Output) -> Output { std::mem::swap(&mut o.body, &mut self.body); std::mem::swap(&mut o.para, &mut self.para); std::mem::swap(&mut o.para_has, &mut self.para_has); @@ -177,7 +188,8 @@ impl Reader { /// A footnote, endnote or comment group starts: read it into an output of its own. pub(super) fn start_sub(&mut self, comment: bool) { - if self.overflow > 0 || self.notes.len() >= MAX_NOTES || self.comments.len() >= MAX_NOTES { + // A note in a text box's text is not kept (it would have no place in the body). + if self.overflow > 0 || self.notes.len() >= MAX_NOTES || self.comments.len() >= MAX_NOTES || self.in_shape_text() { self.st.dest = Dest::Skip; return; } @@ -348,6 +360,9 @@ impl Reader { if self.sub.as_ref().is_some_and(|s| s.depth == self.depth()) { self.end_sub(); } + if self.closes_shape_text() { + self.end_shape_text(); + } self.close_field_group(); let closing = self.st.dest; // Destination-level commits. @@ -368,6 +383,8 @@ impl Reader { self.in_creatim = false; } Dest::Atn(k) => self.commit_atn(k), + Dest::ShpSv => self.commit_shape_prop(), + Dest::ShpInst if self.closes_shape_inst() => self.finish_shape(), _ => {} } let Some(mut prev) = self.stack.pop() else { return }; @@ -403,6 +420,7 @@ impl Reader { } } self.st = prev; + self.after_shape_group(); if self.hf.as_ref().is_some_and(|(d, _, _)| self.depth() < *d) { self.end_header_footer(); } diff --git a/crates/rtf/src/doc/shape.rs b/crates/rtf/src/doc/shape.rs new file mode 100644 index 000000000..71b0007fc --- /dev/null +++ b/crates/rtf/src/doc/shape.rs @@ -0,0 +1,419 @@ +//! Drawing shapes and text boxes (`\shp`, #6212). Word describes a shape in `\shpinst`: its +//! place (`\shpleft`… in twips, against `\shpbx…`/`\shpby…` or the `posrelh`/`posrelv` +//! properties), its wrapping (`\shpwr`, `\shpwrk`, `\shpfblwtxt`), its `\sp` name/value +//! properties and a text box's text (`\shptxt`); `\shprslt` holds a copy for readers without +//! shapes. A shape whose type is known here is read from `\shpinst` and its `\shprslt` skipped; +//! any other (pictures, lines, shapes in a group) keeps the copy, as before. + +use craft_wordmodel::fill::Fill; +use craft_wordmodel::para::{Anchor, Float, FloatAlign, ShapeKind, Wrap, WrapText}; + +use super::reader::Reader; +use super::state::{Dest, MAX_NOTE_BLOCKS, Output}; +use super::{FBlock, Inline, Rgb, model}; + +/// Most characters kept of a property's name or value (hostile input). +const MAX_PROP: usize = 64; + +/// Twips in a point, and EMUs in a point. +const TWIPS: f32 = 20.0; +const EMUS: f32 = 12_700.0; + +/// A `\shp` group being read. +#[derive(Default)] +pub(super) struct ShapeSt { + /// The depths of the `\shp` group, of its `\shpinst` and of the `\shptxt` being read. + shp_depth: usize, + inst_depth: Option, + txt_depth: Option, + /// The output set aside while `\shptxt` is read, and the text read. + saved: Option>, + blocks: Vec, + /// The shape was read from `\shpinst`: its `\shprslt` copy is skipped. + read: bool, + /// `\shpleft`, `\shptop`, `\shpright`, `\shpbottom` (twips). + rect: [i64; 4], + bx: Option, + by: Option, + bx_ignore: bool, + by_ignore: bool, + wr: Option, + wrk: i32, + behind: bool, + /// The `\sn` and `\sv` of the property being read. + sn: String, + sv: String, + props: Props, +} + +/// The `\sp` properties read. +#[derive(Default)] +struct Props { + shape_type: Option, + fill: Option, + filled: Option, + opacity: Option, + line_color: Option, + line: Option, + line_width: Option, + posrelh: Option, + posrelv: Option, + posh: Option, + posv: Option, + pseudo_inline: bool, + fit: bool, + hidden: bool, + flip_h: bool, + flip_v: bool, + rotation: Option, + behind: Option, + /// `dxTextLeft`, `dyTextTop`, `dxTextRight`, `dyTextBottom` (EMUs). + inset: [Option; 4], + /// `dxWrapDistLeft`, `dyWrapDistTop`, `dyWrapDistBottom` (EMUs). + dist: [Option; 3], +} + +/// A colour property (a COLORREF, `0x00BBGGRR`); `None` for a scheme or system colour. +fn colorref(v: i64) -> Option { + (0..=0xFF_FFFF).contains(&v).then_some(Rgb((v & 0xFF) as u8, ((v >> 8) & 0xFF) as u8, ((v >> 16) & 0xFF) as u8)) +} + +/// The shapes read from `\shpinst`; the others keep their `\shprslt` copy. +fn kind(shape_type: i64) -> Option { + Some(match shape_type { + 1 => ShapeKind::Rectangle, + 2 => ShapeKind::RoundedRectangle, + 3 => ShapeKind::Ellipse, + 4 => ShapeKind::Diamond, + 5 => ShapeKind::Triangle, + 6 => ShapeKind::RightTriangle, + 13 => ShapeKind::Arrow, + 66 => ShapeKind::ArrowLeft, + 67 => ShapeKind::ArrowDown, + 68 => ShapeKind::ArrowUp, + 202 => ShapeKind::TextBox, + _ => return None, + }) +} + +/// `posh`/`posv`: 0 is an offset, then left/top, centre, right/bottom, inside, outside. +fn align(v: Option) -> Option { + match v? { + 1 => Some(FloatAlign::Start), + 2 => Some(FloatAlign::Center), + 3 => Some(FloatAlign::End), + 4 => Some(FloatAlign::Inside), + 5 => Some(FloatAlign::Outside), + _ => None, + } +} + +/// A length in points from `v` in `per` units to the point, held to a page's reach. +fn points(v: i64, per: f32) -> f32 { + (v.clamp(-31_680 * 12_700, 31_680 * 12_700) as f32 / per).clamp(-1584.0, 1584.0) +} + +impl Reader { + /// A shape word, read here when it is one: true then. + pub(super) fn shape_word(&mut self, w: &str, p: Option, first_in_group: bool) -> bool { + match self.st.dest { + Dest::ShpInst => { + self.shape_inst_word(w, p, first_in_group); + true + } + // A group in a name or value (a picture's `pib`) is not read. + Dest::ShpSn | Dest::ShpSv => { + if first_in_group { + self.st.dest = Dest::Skip; + } + true + } + Dest::Body => self.shape_body_word(w), + _ => false, + } + } + + fn shape_body_word(&mut self, w: &str) -> bool { + let depth = self.depth(); + match w { + // A group's shapes are placed in the group's own coordinates: its copy is read. + "shpgrp" => { + self.st.in_shpgrp = true; + false + } + "shp" if self.boxes_on && self.shape.is_none() && self.sub.is_none() && !self.st.in_shpgrp && self.overflow == 0 => { + self.shape = Some(Box::new(ShapeSt { shp_depth: depth, ..Default::default() })); + true + } + "shpinst" => match self.shape.as_deref_mut() { + Some(s) if s.inst_depth.is_none() && depth == s.shp_depth + 1 => { + s.inst_depth = Some(depth); + self.st.dest = Dest::ShpInst; + true + } + _ => false, + }, + "shprslt" if self.shape.as_deref().is_some_and(|s| s.read && depth == s.shp_depth + 1) => { + self.st.dest = Dest::Skip; + true + } + _ => false, + } + } + + fn shape_inst_word(&mut self, w: &str, p: Option, first_in_group: bool) { + let depth = self.depth(); + let pv = p.unwrap_or(0); + let Some(s) = self.shape.as_deref_mut() else { + self.st.dest = Dest::Skip; + return; + }; + match w { + "shpleft" => s.rect[0] = i64::from(pv), + "shptop" => s.rect[1] = i64::from(pv), + "shpright" => s.rect[2] = i64::from(pv), + "shpbottom" => s.rect[3] = i64::from(pv), + "shpbxpage" => s.bx = Some(Anchor::Page), + "shpbxmargin" => s.bx = Some(Anchor::Margin), + "shpbxcolumn" => s.bx = Some(Anchor::Column), + "shpbxignore" => s.bx_ignore = true, + "shpbypage" => s.by = Some(Anchor::Page), + "shpbymargin" => s.by = Some(Anchor::Margin), + "shpbypara" => s.by = Some(Anchor::Paragraph), + "shpbyignore" => s.by_ignore = true, + "shpwr" => s.wr = Some(pv), + "shpwrk" => s.wrk = pv, + "shpfblwtxt" => s.behind = p != Some(0), + "sp" => {} + "sn" => { + s.sn.clear(); + self.st.dest = Dest::ShpSn; + } + "sv" => { + s.sv.clear(); + self.st.dest = Dest::ShpSv; + } + "shptxt" if s.saved.is_none() && s.txt_depth.is_none() => { + s.txt_depth = Some(depth); + let saved = self.swap_output(Output::default()); + if let Some(s) = self.shape.as_deref_mut() { + s.saved = Some(Box::new(saved)); + } + self.st.dest = Dest::Body; + self.st.intbl = false; + } + _ if first_in_group => self.st.dest = Dest::Skip, + _ => {} + } + } + + /// `\defshp` in a picture's `\picprop` (which is otherwise skipped): the picture is the one + /// Word writes of a shape for readers without shapes. + pub(super) fn note_defshp(&mut self, w: &str) { + if w == "defshp" && self.pict_depth.is_some() { + self.pict_defshp = true; + } + } + + /// Text of a property's name or value. + pub(super) fn shape_text(&mut self, t: &str) { + let name = self.st.dest == Dest::ShpSn; + if let Some(s) = self.shape.as_deref_mut() { + let into = if name { &mut s.sn } else { &mut s.sv }; + for c in t.chars() { + if into.len() >= MAX_PROP { + break; + } + into.push(c); + } + } + } + + /// Whether a text box's text is being read. + pub(super) fn in_shape_text(&self) -> bool { + self.shape.as_deref().is_some_and(|s| s.saved.is_some()) + } + + /// Whether the group at the current depth is the `\shptxt` being read. + pub(super) fn closes_shape_text(&self) -> bool { + self.shape.as_deref().is_some_and(|s| s.txt_depth == Some(self.depth())) + } + + /// Whether the group at the current depth is the shape's `\shpinst`. + pub(super) fn closes_shape_inst(&self) -> bool { + self.shape.as_deref().is_some_and(|s| s.inst_depth == Some(self.depth())) + } + + /// `\shppict`: note where its group is, so a picture in it counts as standing in its place. + pub(super) fn shppict_word(&mut self, w: &str) -> bool { + if w != "shppict" { + return false; + } + self.shppict_at = Some(self.depth()); + true + } + + /// Whether the `\pict` just opened is right after a shape read, in the same group (a + /// `\*\shppict` around it is transparent): it may be the shape's copy. + pub(super) fn pict_follows_shape(&mut self) -> bool { + let depth = self.depth(); + let at = if self.shppict_at.is_some_and(|s| s.saturating_add(1) == depth) { depth.saturating_sub(1) } else { depth }; + self.shape_read_at.take().is_some_and(|d| d.saturating_add(1) == at) + } + + /// After a group closed: the shape whose `\shp` group it was is done. + pub(super) fn after_shape_group(&mut self) { + if self.shppict_at.is_some_and(|d| self.depth() < d) { + self.shppict_at = None; + } + if self.shape_read_at.is_some_and(|d| self.depth() < d) { + self.shape_read_at = None; + } + if self.shape.as_deref().is_some_and(|s| self.depth() < s.shp_depth) { + self.end_shape_text(); + // Only a shape read has its picture skipped: one that wasn't keeps the next picture. + let read = self.shape.take().is_some_and(|s| s.read); + self.shape_read_at = read.then_some(self.depth()); + } + } + + /// The `\shptxt` group closed (or the file ended in it): keep its text for the shape and go + /// back to the output it was read from. + pub(super) fn end_shape_text(&mut self) { + let Some(saved) = self.shape.as_deref_mut().and_then(|s| { + s.txt_depth = None; + s.saved.take() + }) else { + return; + }; + if self.para_has || !self.para.inlines.is_empty() { + self.end_para(false); + } + self.close_table(); + let mut inner = self.swap_output(*saved); + inner.body.truncate(MAX_NOTE_BLOCKS); + if let Some(s) = self.shape.as_deref_mut() { + s.blocks = inner.body; + } + } + + /// A `\sv` group closed: the property it gives. + pub(super) fn commit_shape_prop(&mut self) { + let Some(s) = self.shape.as_deref_mut() else { return }; + let v = s.sv.trim().parse::().ok(); + let on = v.map(|v| v != 0); + let p = &mut s.props; + match s.sn.trim() { + "shapeType" => p.shape_type = v, + "fillColor" => p.fill = v.and_then(colorref), + "fFilled" => p.filled = on, + "fillOpacity" => p.opacity = v, + "lineColor" => p.line_color = v.and_then(colorref), + "fLine" => p.line = on, + "lineWidth" => p.line_width = v, + "posrelh" => p.posrelh = v, + "posrelv" => p.posrelv = v, + "posh" => p.posh = v, + "posv" => p.posv = v, + "fPseudoInline" => p.pseudo_inline = on == Some(true), + "fFitShapeToText" => p.fit = on == Some(true), + "fHidden" => p.hidden = on == Some(true), + "fFlipH" => p.flip_h = on == Some(true), + "fFlipV" => p.flip_v = on == Some(true), + "rotation" => p.rotation = v, + "fBehindDocument" => p.behind = on, + "dxTextLeft" => p.inset[0] = v, + "dyTextTop" => p.inset[1] = v, + "dxTextRight" => p.inset[2] = v, + "dyTextBottom" => p.inset[3] = v, + "dxWrapDistLeft" => p.dist[0] = v, + "dyWrapDistTop" => p.dist[1] = v, + "dyWrapDistBottom" => p.dist[2] = v, + _ => {} + } + s.sn.clear(); + s.sv.clear(); + } + + /// The `\shpinst` group closed: a shape of a known type goes in the paragraph, and its + /// `\shprslt` copy will be skipped. + pub(super) fn finish_shape(&mut self) { + self.end_shape_text(); + let Some(s) = self.shape.as_deref_mut() else { return }; + let Some(kind) = s.props.shape_type.and_then(kind) else { return }; + let p = &s.props; + let [l, t, r, b] = s.rect; + let (w, h) = (points(r.saturating_sub(l), TWIPS), points(b.saturating_sub(t), TWIPS)); + let mut look = model::BoxLook::default(); + if p.filled == Some(false) { + look.fill = None; + } else { + let transparency = p.opacity.map_or(0.0, |o| 1.0 - (o.clamp(0, 65_536) as f32 / 65_536.0)); + look.fill = Some(Fill::Solid { color: p.fill.unwrap_or(Rgb::WHITE), transparency }); + } + look.line = if p.line == Some(false) { + None + } else { + Some((p.line_color.unwrap_or(Rgb::BLACK), p.line_width.map_or(0.75, |w| points(w, EMUS).max(0.0)))) + }; + if p.inset.iter().any(Option::is_some) { + let [il, it, ir, ib] = p.inset; + let get = |v: Option, d: f32| v.map_or(d, |v| points(v, EMUS).max(0.0)); + look.inset = Some([get(il, 7.2), get(it, 3.6), get(ir, 7.2), get(ib, 3.6)]); + } + look.auto_fit = p.fit; + let mut f = Float { + flip_h: p.flip_h, + flip_v: p.flip_v, + rot: p.rotation.map_or(0.0, |r| (r.clamp(-360 * 65_536, 360 * 65_536) as f32 / 65_536.0).rem_euclid(360.0)), + hidden: p.hidden, + ..Float::default() + }; + if !p.pseudo_inline { + let behind = p.behind.unwrap_or(s.behind); + f.wrap = match s.wr { + Some(1) => Wrap::TopAndBottom, + Some(2) => Wrap::Square, + Some(4) => Wrap::Tight, + Some(5) => Wrap::Through, + _ if behind => Wrap::BehindText, + _ => Wrap::InFrontOfText, + }; + f.wrap_text = match s.wrk { + 1 => WrapText::Left, + 2 => WrapText::Right, + 3 => WrapText::Largest, + _ => WrapText::BothSides, + }; + let rel_h = p.posrelh.and_then(|v| match v { + 0 => Some(Anchor::Margin), + 1 => Some(Anchor::Page), + 2 => Some(Anchor::Column), + 3 => Some(Anchor::Character), + _ => None, + }); + let rel_v = p.posrelv.and_then(|v| match v { + 0 => Some(Anchor::Margin), + 1 => Some(Anchor::Page), + 2 => Some(Anchor::Paragraph), + 3 => Some(Anchor::Line), + _ => None, + }); + // `\shpbxignore` and `\shpbyignore` hand the anchor to the properties. + f.h_rel = if s.bx_ignore { rel_h.or(s.bx) } else { s.bx.or(rel_h) }.unwrap_or(Anchor::Column); + f.v_rel = if s.by_ignore { rel_v.or(s.by) } else { s.by.or(rel_v) }.unwrap_or(Anchor::Paragraph); + f.x = points(l, TWIPS); + f.y = points(t, TWIPS); + f.h_align = align(p.posh); + f.v_align = align(p.posv); + let dist = |v: Option, d: f32| v.map_or(d, |v| points(v, EMUS).max(0.0)); + f.dist = dist(p.dist[0], 9.0); + f.dist_top = dist(p.dist[1], 0.0); + f.dist_bottom = dist(p.dist[2], 0.0); + } + let blocks = std::mem::take(&mut s.blocks); + s.read = true; + self.para.inlines.push(Inline::Shape { kind, w, h, float: Some(Box::new(f)), blocks, look, path: None }); + self.para_has = true; + } +} diff --git a/crates/rtf/src/doc/state.rs b/crates/rtf/src/doc/state.rs index c27758d05..a39eacc2f 100644 --- a/crates/rtf/src/doc/state.rs +++ b/crates/rtf/src/doc/state.rs @@ -42,6 +42,11 @@ pub(super) enum Dest { /// Formatting before a tracked change (`\oldpprops` when true, else `\oldcprops`): read /// into the group's own state, which the group's end keeps as the change's old formatting. OldProps(bool), + /// A shape's description (`\shpinst`), and the name (`\sn`) and value (`\sv`) of one of + /// its properties. + ShpInst, + ShpSn, + ShpSv, } /// A footnote, endnote or comment group being read into an output of its own. @@ -157,6 +162,8 @@ pub(super) struct St { pub(super) outline: Option, pub(super) pagebb: bool, pub(super) field: Option, + /// Inside a group of shapes (`\shpgrp`), whose members are not read as shapes of their own. + pub(super) in_shpgrp: bool, /// The code page of the current font's charset (`\fcharset`), when it names one. pub(super) cp: Option, /// Space before/after (`\sb`, `\sa`), line spacing (`\sl`, `\slmult`) and indents @@ -195,6 +202,7 @@ impl Default for St { outline: None, pagebb: false, field: None, + in_shpgrp: false, cp: None, borders: Default::default(), border_side: None, diff --git a/crates/rtf/src/doc/tests_shapes.rs b/crates/rtf/src/doc/tests_shapes.rs new file mode 100644 index 000000000..54be351ba --- /dev/null +++ b/crates/rtf/src/doc/tests_shapes.rs @@ -0,0 +1,198 @@ +//! Drawing shapes and text boxes read from `\shp` (#6212): the shape's own description, +//! `\shpinst`, is read, and its `\shprslt` copy (for readers without shapes) is not. + +use super::*; +use craft_wordmodel::fill::Fill; +use craft_wordmodel::para::{Anchor, FloatAlign, ShapeKind, Wrap}; + +/// The document's shapes, in order: kind, size, fill, outline, placement and text. +/// (kind, width, height, fill, line colour, float, text) of each shape. +type ShapeRow = (ShapeKind, f32, f32, Option, Option, craft_wordmodel::para::Float, String); + +fn shapes(d: &Document) -> Vec { + d.body + .iter() + .filter_map(|b| b.as_para()) + .flat_map(|p| p.objects.iter()) + .filter_map(|o| match o { + InlineObject::Shape { kind, w, h, fill, stroke, float, story, .. } => { + let text = story + .and_then(|id| d.parts.get(&id)) + .map(|p| p.blocks.iter().filter_map(|b| b.as_para()).map(|p| p.plain_text()).collect::>().join("\n")) + .unwrap_or_default(); + Some((*kind, *w, *h, fill.clone(), *stroke, float.clone(), text)) + } + _ => None, + }) + .collect() +} + +fn images(d: &Document) -> usize { + d.body.iter().filter_map(|b| b.as_para()).flat_map(|p| p.objects.iter()).filter(|o| matches!(o, InlineObject::Image { .. })).count() +} + +/// A text box as Word writes one: shape type 202 with its text in `\shptxt`, and an old-style +/// drawing object holding the same text in `\shprslt`. +const TEXT_BOX: &str = r"{\rtf1\ansi Before\par +{\shp{\*\shpinst\shpleft0\shptop0\shpright5000\shpbottom1400\shpbxcolumn\shpbxignore\shpbypara\shpbyignore\shpwr1\shpwrk0\shpfblwtxt0\shpz2 +{\sp{\sn shapeType}{\sv 202}}{\sp{\sn fillColor}{\sv 16445672}} +{\shptxt \pard\plain Boxed note\par}} +{\shprslt{\*\do\dobxcolumn\dobypara\dptxbx{\dptxbxtext \pard\plain Boxed note\par}\dpx0\dpy0\dpxsize5000\dpysize1400}}} +After\par}"; + +#[test] +fn a_text_box_is_read_with_its_text_fill_and_wrapping() { + let d = import(TEXT_BOX.as_bytes()).unwrap(); + let s = shapes(&d); + assert_eq!(s.len(), 1, "{s:?}"); + let (kind, w, h, fill, stroke, float, text) = &s[0]; + assert_eq!(*kind, ShapeKind::TextBox); + // Twips to points. + assert_eq!((*w, *h), (250.0, 70.0)); + // `fillColor` is 0x00BBGGRR. + assert_eq!(*fill, Some(Fill::solid(Rgb(232, 240, 250)))); + // A line unless `fLine` turns it off: Word's black hairline. + assert_eq!(*stroke, Some(Rgb::BLACK)); + assert_eq!(float.wrap, Wrap::TopAndBottom); + assert_eq!((float.h_rel, float.v_rel), (Anchor::Column, Anchor::Paragraph)); + assert_eq!((float.x, float.y), (0.0, 0.0)); + assert_eq!(text, "Boxed note"); + // Its text is in the box, once: not also in the body. + assert!(!d.plain_text(craft_wordmodel::StoryRef::Body).contains("Boxed"), "{}", d.plain_text(craft_wordmodel::StoryRef::Body)); +} + +#[test] +fn an_arrow_in_a_shape_field_is_read_in_line() { + // Word writes a shape in line with text as a SHAPE field whose result is the shape, then a + // picture of it for readers without shapes. + let rtf = r"{\rtf1\ansi {\field\fldedit\fldlock{\*\fldinst SHAPE \\* MERGEFORMAT }{\fldrslt +{\shp{\*\shpinst\shpleft0\shptop0\shpright2400\shpbottom800\shpbxcolumn\shpbxignore\shpbypara\shpbyignore\shpwr3\shpwrk0\shpfblwtxt0 +{\sp{\sn shapeType}{\sv 13}}{\sp{\sn fillColor}{\sv 42495}}{\sp{\sn posrelh}{\sv 3}}{\sp{\sn posrelv}{\sv 3}}{\sp{\sn fPseudoInline}{\sv 1}} +{\sp{\sn fLine}{\sv 0}}} +{\shprslt{\*\do\dobxcolumn\dobypara\dppolygon\dppolycount3\dpptx0\dppty0\dpptx2400\dppty400\dpptx0\dppty800\dpx0\dpy0\dpxsize2400\dpysize800}}}}} +caption\par}"; + let d = import(rtf.as_bytes()).unwrap(); + let s = shapes(&d); + assert_eq!(s.len(), 1, "{s:?}"); + let (kind, w, h, fill, stroke, float, _) = &s[0]; + assert_eq!(*kind, ShapeKind::Arrow); + assert_eq!((*w, *h), (120.0, 40.0)); + assert_eq!(*fill, Some(Fill::solid(Rgb(255, 165, 0)))); + assert_eq!(*stroke, None); + assert_eq!(float.wrap, Wrap::Inline); + assert!(d.plain_text(craft_wordmodel::StoryRef::Body).contains("caption")); +} + +/// A placeable WMF: the 9-word header (type 1, version 0x300, 12 words, max record 3), then EOF. +fn wmf_hex() -> String { + [1u16, 9, 0x300, 12, 0, 0, 3, 0, 0, 3, 0, 0].iter().flat_map(|w| w.to_le_bytes()).map(|b| format!("{b:02x}")).collect() +} + +#[test] +fn the_picture_after_a_shape_read_is_not_a_second_copy() { + // After the shape, Word writes a picture of it marked `\defshp` for readers without shapes. + let pict = format!(r"{{\pict{{\*\picprop\defshp{{\sp{{\sn shapeType}}{{\sv 75}}}}}}\picwgoal2400\pichgoal800\wmetafile8 {}}}", wmf_hex()); + let shape = + |t: u32| format!(r"{{\shp{{\*\shpinst\shpright2400\shpbottom800{{\sp{{\sn shapeType}}{{\sv {t}}}}}{{\sp{{\sn fPseudoInline}}{{\sv 1}}}}}}}}"); + let field = |t: u32| format!(r"{{\rtf1\ansi {{\field{{\*\fldinst SHAPE }}{{\fldrslt {}{pict}}}}} caption\par}}", shape(t)); + let d = import(field(13).as_bytes()).unwrap(); + assert_eq!((shapes(&d).len(), images(&d)), (1, 0)); + // A shape not read here (a line) keeps its picture. + let d = import(field(20).as_bytes()).unwrap(); + assert_eq!((shapes(&d).len(), images(&d)), (0, 1)); + // A `\defshp` picture with no shape read before it is kept. + let d = import(format!(r"{{\rtf1\ansi {pict} caption\par}}").as_bytes()).unwrap(); + assert_eq!(images(&d), 1); + // Only the picture right after a shape read is its copy: a line read earlier in the same + // group doesn't take the picture of a later shape not read here. + let d = import(format!(r"{{\rtf1\ansi {}{}{pict} caption\par}}", shape(13), shape(20)).as_bytes()).unwrap(); + assert_eq!((shapes(&d).len(), images(&d)), (1, 1)); + let d = import(format!(r"{{\rtf1\ansi {}{pict}{pict} caption\par}}", shape(13)).as_bytes()).unwrap(); + assert_eq!((shapes(&d).len(), images(&d)), (1, 1)); + // The picture may come wrapped in `\*\shppict`: still the shape's copy. + let d = import(format!(r"{{\rtf1\ansi {}{{\*\shppict{pict}}} caption\par}}", shape(13)).as_bytes()).unwrap(); + assert_eq!((shapes(&d).len(), images(&d)), (1, 0)); + let d = import(format!(r"{{\rtf1\ansi {}{{\*\shppict{pict}}} caption\par}}", shape(20)).as_bytes()).unwrap(); + assert_eq!((shapes(&d).len(), images(&d)), (0, 1)); +} + +#[test] +fn a_shape_behind_text_is_placed_on_the_page() { + let rtf = r"{\rtf1\ansi {\shp{\*\shpinst\shpleft1440\shptop2880\shpright4320\shpbottom4320\shpbxpage\shpbypage\shpwr3\shpfblwtxt1 +{\sp{\sn shapeType}{\sv 3}}{\sp{\sn posh}{\sv 2}}{\sp{\sn fFilled}{\sv 0}}{\sp{\sn lineColor}{\sv 255}}{\sp{\sn lineWidth}{\sv 25400}} +{\sp{\sn fFlipH}{\sv 1}}}}Text\par}"; + let d = import(rtf.as_bytes()).unwrap(); + let s = shapes(&d); + assert_eq!(s.len(), 1, "{s:?}"); + let (kind, w, h, fill, stroke, float, _) = &s[0]; + assert_eq!(*kind, ShapeKind::Ellipse); + assert_eq!((*w, *h), (144.0, 72.0)); + assert_eq!(*fill, None); + assert_eq!(*stroke, Some(Rgb(255, 0, 0))); + assert_eq!(float.wrap, Wrap::BehindText); + assert_eq!((float.h_rel, float.v_rel), (Anchor::Page, Anchor::Page)); + assert_eq!((float.x, float.y), (72.0, 144.0)); + assert_eq!(float.h_align, Some(FloatAlign::Center)); + assert!(float.flip_h); + let width = d.body.iter().filter_map(|b| b.as_para()).flat_map(|p| p.objects.iter()).find_map(|o| match o { + InlineObject::Shape { stroke_width, .. } => Some(*stroke_width), + _ => None, + }); + assert_eq!(width, Some(2.0)); +} + +#[test] +fn a_picture_shape_keeps_one_picture() { + // Shape type 75 (a picture frame) stays on its `\shprslt` copy: its `pib` picture isn't + // read a second time. + let png = "89504e470d0a1a0a0000000d4948445200000001000000010806000000"; + let rtf = format!( + r"{{\rtf1\ansi {{\shp{{\*\shpinst\shpleft0\shptop0\shpright1440\shpbottom1440\shpbxcolumn\shpbypara\shpwr2 +{{\sp{{\sn shapeType}}{{\sv 75}}}}{{\sp{{\sn pib}}{{\sv {{\pict\pngblip\picwgoal1440\pichgoal1440 {png}}}}}}}}}}} +{{\shprslt\par\pard{{\pict\pngblip\picwgoal1440\pichgoal1440 {png}}}\par}}}}Text\par}}" + ); + let d = import(rtf.as_bytes()).unwrap(); + assert!(shapes(&d).is_empty()); + assert_eq!(images(&d), 1); +} + +#[test] +fn shapes_inside_a_group_are_left_to_the_groups_copy() { + // A group's members are placed in the group's own coordinates: they stay on the group's + // `\shprslt` copy rather than being read at the wrong place. + let rtf = r"{\rtf1\ansi {\shpgrp{\*\shpinst\shpleft0\shptop0\shpright2000\shpbottom2000\shpbxcolumn\shpbypara\shpwr3} +{\shp{\*\shpinst\shpleft0\shptop0\shpright1000\shpbottom1000{\sp{\sn shapeType}{\sv 1}}}} +{\shprslt kept}}Text\par}"; + let d = import(rtf.as_bytes()).unwrap(); + assert!(shapes(&d).is_empty()); + assert!(d.plain_text(craft_wordmodel::StoryRef::Body).contains("kept")); +} + +#[test] +fn truncated_and_hostile_shapes_do_not_crash() { + for rtf in [ + r"{\rtf1 Before\par{\shp{\*\shpinst{\shptxt Inside", + r"{\rtf1 Before\par{\shp{\*\shpinst\shpleft-2147483648\shpright2147483647{\sp{\sn shapeType}{\sv 202}}{\shptxt A\par}}}}", + r"{\rtf1 Before\par{\shp{\*\shpinst{\sp{\sn shapeType}{\sv 99999999999999999999}}{\sp{\sn fillColor}{\sv -5}}}}}", + r"{\rtf1 Before\par{\shp{\*\shpinst{\sp{\sn shapeType}{\sv 202}}{\shptxt {\shp{\*\shpinst{\sp{\sn shapeType}{\sv 202}}{\shptxt deeper\par}}}\par}}}}", + r"{\rtf1 Before\par{\footnote{\shp{\*\shpinst{\sp{\sn shapeType}{\sv 202}}{\shptxt in a note\par}}}}}", + ] { + let d = import(rtf.as_bytes()).unwrap(); + assert!(d.plain_text(craft_wordmodel::StoryRef::Body).contains("Before"), "{rtf}"); + } + // A shape too large for a page is held to Word's largest. + let d = import(r"{\rtf1 {\shp{\*\shpinst\shpleft-2147483648\shpright2147483647\shpbottom100{\sp{\sn shapeType}{\sv 1}}}}}".as_bytes()).unwrap(); + assert!(shapes(&d).iter().all(|s| s.1 <= 1584.0 && s.2 >= 1.0)); +} + +#[test] +fn the_flow_reader_is_unchanged() { + // The public flow reader, `parse`, asks for no shapes: the box's text is left out as before. + let f = parse(TEXT_BOX.as_bytes()).unwrap(); + let all: String = f.blocks.iter().filter_map(|b| if let FBlock::Para(p) = b { Some(p.text()) } else { None }).collect(); + assert!(!all.contains("Boxed"), "{all}"); + assert!(f.blocks.iter().all(|b| match b { + FBlock::Para(p) => p.inlines.iter().all(|i| !matches!(i, Inline::Shape { .. })), + FBlock::Table(_) => true, + })); +} diff --git a/crates/rtf/src/doc/text.rs b/crates/rtf/src/doc/text.rs index ad997c680..ce77a20d3 100644 --- a/crates/rtf/src/doc/text.rs +++ b/crates/rtf/src/doc/text.rs @@ -155,6 +155,11 @@ impl Reader { // Odd hex digit count: drop the dangling nibble. } let data = std::mem::take(&mut self.pict_hex); + // The picture Word writes of a shape just read (`\defshp`), for readers without shapes. + let after_shape = std::mem::take(&mut self.pict_after_shape); + if std::mem::take(&mut self.pict_defshp) && after_shape { + return; + } if self.pict_kind == 0 || data.is_empty() { return; } @@ -304,7 +309,8 @@ impl Reader { } } } - Dest::Skip | Dest::ListTable | Dest::ListOverride | Dest::OldProps(_) => {} + Dest::ShpSn | Dest::ShpSv => self.shape_text(s), + Dest::Skip | Dest::ListTable | Dest::ListOverride | Dest::OldProps(_) | Dest::ShpInst => {} } } diff --git a/crates/rtf/src/doc/word.rs b/crates/rtf/src/doc/word.rs index 49367a7d1..0a0df5c68 100644 --- a/crates/rtf/src/doc/word.rs +++ b/crates/rtf/src/doc/word.rs @@ -21,10 +21,14 @@ impl Reader { self.finish_multibyte(); let on = p != Some(0); let pv = p.unwrap_or(0); + self.note_defshp(w); if self.skip_destination_word(w, pv) { return; } - if w == "shppict" || self.start_header_footer_word(w) { + if self.shape_word(w, p, first_in_group) { + return; + } + if self.shppict_word(w) || self.start_header_footer_word(w) { return; } if self.start_note_word(w) { @@ -38,6 +42,8 @@ impl Reader { } Dest::Pict => { self.pict_depth = Some(self.depth()); + self.pict_defshp = false; + self.pict_after_shape = self.pict_follows_shape(); self.pict_hex.clear(); self.pict_half = None; self.pict_kind = 0; @@ -465,6 +471,8 @@ impl Reader { } Dest::RevTbl if !matches!(word, "u" | "uc") => true, Dest::RevTbl => false, + // Read by `shape_word`, which takes every word there. + Dest::ShpInst | Dest::ShpSn | Dest::ShpSv => true, Dest::Pict => { self.picture_word(word, value); true From db9e24391a5998230ba8d2004f5665ee5e8f8223 Mon Sep 17 00:00:00 2001 From: Tex Jernigan Date: Wed, 18 Nov 2026 15:35:02 -0500 Subject: [PATCH 144/161] craft-develop-model and craft-geom: develop settings and crs: import, from LightCraft (#1132) * craft-geom and craft-develop-model: develop settings and crs: import, from LightCraft craft-develop-model is lightcraft-develop (settings, controls, presets, segmentation masks, process versions) plus the crs: import from lightcraft-engine (crs.rs, crs_masks.rs). craft-geom is lightcraft-geom, which the model and the coming develop pipeline both need. Step 3 of #587. * craft-develop-model, craft-geom: review fixes to copy, presets, section reset and crs: import Copy and presets: to_json_for_merge states every setting the stored JSON leaves out at its default, so copying a group turns HDR, the camera's local tone and Lens Blur's optional settings off on the target, and a preset at an amount starts from them. A preset at no amount adds no mask, and its Lens Blur focus mask follows its mask to the id the photo gives it. Reset and range: reset_section resets the sliders of point colours and red eyes, Upright with its guides, and the two Optics switches. Settings from outside the app keep the list sliders in range. is_unedited counts a chosen white balance as an edit. crs: import: a brush dab that is not a finite number is dropped instead of failing the whole packet, brush runs that differ in MaskInverted stay two components, RetouchInfo is read only without RetouchAreas, and an XMP preset keeps its masks. SegMask::from_logits does not overflow on an absurd side. craft-geom: Homography::from_quads solves a mapping that sends the origin to infinity, by solving again about the middle of the source quad. Docs: the links to LightCraft's light-v3.md and grain.md point at that repository, and xmp-interop.md says when the older fields are read. The README lists what now differs from LightCraft. * craft-develop-model, craft-geom: second round of review fixes to crs: import, mask grids, crop and quads A range mask with a number that is not one failed the whole merge; a straight legacy ToneCurve and a photo's switched-off masks or spots were left unsaid, so the photo kept what it had; a mask grid of a side past the limit was cut to 32 bits; a crop restored under a new angle could reach outside the photo; a quad mapping onto a collapsed quad solved to a matrix that does not map it. --- Cargo.lock | 20 + Cargo.toml | 2 + crates/develop-model/Cargo.toml | 21 + crates/develop-model/README.md | 51 + crates/develop-model/docs/process-versions.md | 132 ++ crates/develop-model/docs/xmp-interop.md | 342 ++++ crates/develop-model/src/controls.rs | 632 +++++++ crates/develop-model/src/crs.rs | 543 ++++++ crates/develop-model/src/crs/tests.rs | 357 ++++ crates/develop-model/src/crs_masks.rs | 442 +++++ crates/develop-model/src/crs_masks/tests.rs | 340 ++++ crates/develop-model/src/lib.rs | 588 +++++++ crates/develop-model/src/presets.rs | 675 ++++++++ crates/develop-model/src/segmask.rs | 337 ++++ crates/develop-model/src/settings.rs | 1535 +++++++++++++++++ crates/develop-model/src/tests_process.rs | 162 ++ crates/develop-model/tests/merge_and_reset.rs | 120 ++ crates/geom/Cargo.toml | 17 + crates/geom/README.md | 14 + crates/geom/src/crop.rs | 372 ++++ crates/geom/src/crop_drag.rs | 367 ++++ crates/geom/src/crop_drag/tests.rs | 366 ++++ crates/geom/src/homography.rs | 188 ++ crates/geom/src/lib.rs | 469 +++++ crates/geom/src/real.rs | 261 +++ 25 files changed, 8353 insertions(+) create mode 100644 crates/develop-model/Cargo.toml create mode 100644 crates/develop-model/README.md create mode 100644 crates/develop-model/docs/process-versions.md create mode 100644 crates/develop-model/docs/xmp-interop.md create mode 100644 crates/develop-model/src/controls.rs create mode 100644 crates/develop-model/src/crs.rs create mode 100644 crates/develop-model/src/crs/tests.rs create mode 100644 crates/develop-model/src/crs_masks.rs create mode 100644 crates/develop-model/src/crs_masks/tests.rs create mode 100644 crates/develop-model/src/lib.rs create mode 100644 crates/develop-model/src/presets.rs create mode 100644 crates/develop-model/src/segmask.rs create mode 100644 crates/develop-model/src/settings.rs create mode 100644 crates/develop-model/src/tests_process.rs create mode 100644 crates/develop-model/tests/merge_and_reset.rs create mode 100644 crates/geom/Cargo.toml create mode 100644 crates/geom/README.md create mode 100644 crates/geom/src/crop.rs create mode 100644 crates/geom/src/crop_drag.rs create mode 100644 crates/geom/src/crop_drag/tests.rs create mode 100644 crates/geom/src/homography.rs create mode 100644 crates/geom/src/lib.rs create mode 100644 crates/geom/src/real.rs diff --git a/Cargo.lock b/Cargo.lock index ac676d666..4c9980247 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -1779,6 +1779,18 @@ dependencies = [ "thiserror 2.0.21", ] +[[package]] +name = "craft-develop-model" +version = "0.1.0" +dependencies = [ + "craft-geom", + "craft-metadata", + "miniz_oxide 0.9.1", + "proptest", + "serde", + "serde_json", +] + [[package]] name = "craft-dex" version = "0.1.0" @@ -2185,6 +2197,14 @@ dependencies = [ "thiserror 2.0.21", ] +[[package]] +name = "craft-geom" +version = "0.1.0" +dependencies = [ + "proptest", + "serde", +] + [[package]] name = "craft-gif" version = "0.1.0" diff --git a/Cargo.toml b/Cargo.toml index f99224d65..605801564 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -172,6 +172,8 @@ craft-pst = { path = "crates/pst" } craft-quit = { path = "crates/quit" } craft-rtf = { path = "crates/rtf" } craft-raw = { path = "crates/raw" } +craft-geom = { path = "crates/geom" } +craft-develop-model = { path = "crates/develop-model" } craft-sheet = { path = "crates/sheet" } craft-sketch = { path = "crates/sketch" } craft-settings = { path = "crates/settings" } diff --git a/crates/develop-model/Cargo.toml b/crates/develop-model/Cargo.toml new file mode 100644 index 000000000..ec5848b86 --- /dev/null +++ b/crates/develop-model/Cargo.toml @@ -0,0 +1,21 @@ +[package] +name = "craft-develop-model" +description = "Raw develop settings as data: the non-destructive edit model (white balance, tone, colour, detail, optics, geometry, masks), control specs with ranges and defaults, process versions, presets, and the mapping to and from Camera Raw's crs: settings." +version.workspace = true +edition.workspace = true +license.workspace = true +rust-version.workspace = true +repository.workspace = true + +[lints] +workspace = true + +[dependencies] +craft-geom = { workspace = true } +craft-metadata = { workspace = true } +serde = { workspace = true } +serde_json = { workspace = true } +miniz_oxide = "0.9" + +[dev-dependencies] +proptest = { workspace = true } diff --git a/crates/develop-model/README.md b/crates/develop-model/README.md new file mode 100644 index 000000000..c6b07f204 --- /dev/null +++ b/crates/develop-model/README.md @@ -0,0 +1,51 @@ +# craft-develop-model + +Raw develop settings as data: the second of the three crates planned in craft-libs#587 +(`craft-raw`, `craft-develop-model`, `craft-develop`). No rendering, no GPU, no I/O. An app can +store, copy, merge, migrate and serialise a photo's edits without depending on the pipeline. + +```toml +craft-develop-model = { git = "https://github.com/storytold/craft-libs", rev = "" } +``` + +The base is LightCraft's `lightcraft-develop` and the `crs:` import from `lightcraft-engine` +(`crs.rs`, `crs_masks.rs`) at `a75497d6`. Issue numbers in comments are LightCraft's +(`storytold/lightcraft#N`). + +## What is here + +| Module | Contents | +|---|---| +| `settings` | `DevelopSettings`, the complete, serialisable description of a photo's look: white balance (Kelvin or relative scale), light, tone curve, HSL / mixer / point colour, B&W mix, colour grading, calibration, effects (vignette, grain), detail, optics (lens profile, embedded corrections, defringe), geometry and Upright, crop, masks (brush, linear, radial, range, AI segmentation), spots, red eye, lens blur, enhance, HDR. `ProcessVersion` / `Process`: which formulas an edit renders with, so later formula changes leave existing edits alone ([process versions](docs/process-versions.md)). | +| `controls` | `CONTROLS`: every numeric slider by id, with its label, section, range, default, step and track gradient. UIs and agents address sliders through it. | +| `presets` | `Preset`, `SettingsGroup` (the groups a copy, paste, sync or preset covers), `extract_groups`, `apply_partial` (a partial settings object merged at an amount), `deep_merge`. | +| `segmask` | `SegMask`: a compressed AI segmentation mask stored inside the settings. | +| `crs`, `crs_masks` | Camera Raw / Lightroom `crs:` properties (from `craft_metadata::parse_xmp`) mapped to a *partial* `DevelopSettings` JSON object, which is merged like a preset: global adjustments, tone curves, white balance (`Target` decides how a `crs:` temperature is read), crop, Upright, local corrections and retouch spots, with a report of what was read and what was left out. `preset_from_xmp` reads an XMP preset. The field table is in [xmp-interop](docs/xmp-interop.md). | + +Reading the `crs:` packet itself (names, structures, Camera Raw's number formats) is +[`craft-crs`](../crs/README.md)'s job; this crate decides what those settings mean for the edit +model. The mapping is a best-effort approximation of the other application's look, not a pixel match. + +## Different from LightCraft + +Fixes made here that LightCraft `a75497d6` does not have. It gets them when it switches to this crate. + +- `to_json_for_merge` states every setting the stored JSON leaves out at its default (HDR, the profile and its camera + local tone, the curve, Lens Blur's and Denoise's optional settings), not only Remove Hot Pixels. Copying a group + turns them off on the target, and a preset at an amount starts from them. +- `reset_section` resets the sliders of point colours and red eyes, Upright with its guides, and the lens profile and + Remove Chromatic Aberration switches. Settings from outside the app keep those sliders in range. +- `is_unedited` counts a chosen white balance as an edit. As shot, the numbers stay the photo's own. +- A preset at no amount adds no mask, and its Lens Blur focus mask follows its mask to the id the photo gives it. +- `crs:` import: a brush dab that is not a finite number is dropped, brush runs that differ in `crs:MaskInverted` stay + two components, `crs:RetouchInfo` is read only without `crs:RetouchAreas`, and an XMP preset keeps its masks. +- `Homography::from_quads` (`craft-geom`) solves a mapping that sends the origin to infinity. + +## Not here yet + +- Writing `crs:` back from `DevelopSettings`. LightCraft writes its own `lc:settings` in a sidecar; PhotoCraft's + Camera Raw writes `crs:`. That comes with the PhotoCraft switch, together with the migration of PhotoCraft's + current Camera Raw settings. +- The tests that render a mask or a spot to check its sense (inside or outside a radial, where a spot lands) + need the pipeline. They come back with `craft-develop`. The mapping half of each is kept here. +- LightCraft-only paths: `Target` from a catalog photo, and `.lrtemplate` (Lua) presets. diff --git a/crates/develop-model/docs/process-versions.md b/crates/develop-model/docs/process-versions.md new file mode 100644 index 000000000..20443b9bc --- /dev/null +++ b/crates/develop-model/docs/process-versions.md @@ -0,0 +1,132 @@ +# Process versions + +> **Changes:** [revision history](#revision-history) + +Every photo's develop settings record the **rendering process** they are interpreted by +(`DevelopSettings::process`, a `ProcessVersion` in `crates/develop/src/settings.rs`). Lightroom calls +this the process version. It lets the look of LightCraft improve (a new default tone curve, issue #146; +stronger Light sliders, issue #196) without changing a single photo someone has already edited. + +| Process | What it changes | +|---|---| +| **V1** | LightCraft's rendering from before process versions existed. Frozen. | +| **V2** | Highlights and Shadows keep fine detail (issue #632): their base layer (`local::tone_base`) is the edge-aware filter of a separable median of log luminance (radius 0.15 % of the long edge, at most 24 px, `local::BASE_DETAIL`) instead of log luminance itself, so twigs, lines and texture take their surroundings' shift and keep their contrast; region edges stay sharp. CPU (`local::prepare`) and GPU prepare branch on `local::tone_base_detail` (0 = no median under V1); the base plane is cached per process (`local::BaseKey`). Renders without Highlights or Shadows (global or local) are the same as V1's. Tone map unchanged. | +| **V3** | Pixel-scale sharpening (issue #757): Radius uses the original photo’s pixels, Detail controls halo damping, and downsized sources retain a response-matched sharpening band. V1 and V2 keep texture-band sharpening exactly. The sharpen plane is keyed by radius and process and invalidated after noise-reduction changes. The Highlights/Shadows base and tone map are unchanged from V2. | +| **V4** | V3 plus calibrated global Contrast (per channel) and Blacks (per-channel/luminance mix) on raws with a camera tone curve. Whites behaves exactly as V2, both signs; its calibration is a planned follow-up. Other sources and local adjustments retain V2 operators. See [calibrated Light operators](https://github.com/storytold/lightcraft/blob/a75497d6/docs/light-v3.md). | +| **V5** | V4 plus: a raw's tone curve rolls off into white in stops instead of reaching it within about a stop of the sensor's white, so raising Exposure keeps the highlights the sensor recorded (issue #1632). **Camera curve** (the look fitted to the embedded JPEG, or a DNG's `ProfileToneCurve`; `CameraTone`): past its last knot `CameraTone::apply_rolloff` closes the gap to white by `gap / (1 + c·(2^t − 1))` at `t` stops past the knot, with `c` matching the curve's slope there and the gap halving every stop in the end (`tone::ROLLOFF_STOPS`); `ToneMap::camera_rolloff_hdr` builds the table from it. Under V1 to V4 the curve past its last knot (fitted only to camera JPEG tones below 0.85, so 1 to 3 stops below the sensor's white on most photos) closes on white exponentially in linear light: the sensor's white renders at 8-bit 246 to 254 at Exposure 0 and 254 to 255 from Exposure +1. On the 11 CC0 corpus files with a fitted look (8 Sony, 2 Nikon, 1 Fujifilm) it now renders at 239 to 250 at Exposure 0 and 247 to 253 at +1. A DNG curve that reaches white at the sensor's white renders as before. **Generic curve** (raws without a camera curve, `ToneMap::rolloff_hdr`): from half a stop below its white point (`tone::GENERIC_KNEE_STOPS`) the same shoulder takes over, with the extended Reinhard curve's value and slope at the join, where V1 to V4 clip at the white point (scene 1.18 at default settings, ¼ stop above the sensor's white). The sensor's white renders at about 235, 248 and 252 at Exposure 0, +1 and +2 (V1 to V4: 239, 255, 255; the CC0 Canon R8 file and 6 CC0 DNGs without a camera curve). Content past the old clip point, such as a clipped sky, now renders up to 15 codes below white at Exposure 0 and reaches 255 about 5 stops past the sensor's white. Below the last knot or the join, JPEGs and other display-referred sources, and the camera fit itself are unchanged; CPU and GPU share the tone LUT. | +| **V6** | V5 plus: colour noise reduction filters the linear colour differences R−Y and B−Y with the joint method (issue #815, `docs/nr-color-joint.md`): edge-aware weights from a denoised guide shared by both differences, anisotropic diffusion over scales, a brake for isolated colour specks; Y is kept exactly. V1 to V5 keep the blurred chromaticity (`local::denoise`). CPU (`local::denoise`) and GPU (`render::denoise`) branch on `local::joint_colour_nr`. Renders with Color 0 are the same as V2's. | +| **V7** | V6 plus: film grain from crystal layers ([grain.md](https://github.com/storytold/lightcraft/blob/a75497d6/docs/grain.md), `crates/pipeline/src/crystal.rs`), after Aurélien Pierre's model: resolution independent (a preview is the export's grain box-downscaled), fading out at black and white, calibrated to Lightroom Classic's strength and width at mid grey. V1 to V6 keep their two-octave value noise exactly (`finish::ValueGrain`; on the GPU `F_GRAIN_MODEL` 0); `finish::GrainModel::new` picks the model per process. Renders without grain don't depend on this. | +| **V8** (the latest) | V7 plus: Highlights and Shadows take their edge-aware filter of log luminance without its fine detail (`local::detail_free`: each value clamped between its separable median at the V2 radius and the median of a 3× wider neighbourhood) instead of V2's plain median (`local::base_keeps_gaps`), so the gaps in dark lettering on a bright panel are not lifted into bright blocks (issue #3460). CPU (`local::tone_base`) and GPU (`render::tone_base`) branch on it; the base plane is keyed by process. Renders without Highlights or Shadows (global or local) are the same as V7's. | + +## The rule + +**A change that alters how existing settings render ships as a new process. V1 is frozen.** + +- A deliberate change of look (tone model, slider response, default curve, how a stage interprets a + value) goes into a new process. Photos stay on the process they have until someone runs Update to + Current Process (or Reset). +- Bug fixes that make output correct where it was broken (a crash, NaN or black output, a decoder + error, CPU and GPU disagreeing) are not looks and can land in place, as before. +- Changes to how a file is decoded or which camera colour it gets change the source, not how settings + are interpreted; the process doesn't reach that stage today. + +**Neutral display tone map** (JPEG, PNG and other display-referred sources with Contrast, Whites and +Blacks at 0) is the identity up to white: the SDR shoulder no longer pulls 1.0 down to 0.9875 (8-bit +255 to 254, issue #1945). It landed in place as a bug fix, since the tone map promised identity at +neutral settings; edited curves keep the shoulder and their V1 look. + +**AI mask edges** (Object and Describe selections stored as a model's grid) follow the photo's edges +when they are enlarged to the output (issue #3272): the plain enlargement spilled 1–2 px onto the +background, which a raised Exposure turned into a bright outline around the subject in previews and +exports. Existing AI masks become 1–2 px tighter at a subject's edges, on every process; their +inside, their outside and edges without contrast in the photo are unchanged. It landed in place as +a bug fix: the outline is output that isn't in the photo nor in the edit, not a look anyone chose. + +**A mask whose first component is Intersect** starts from that component, on the CPU and the GPU +(issue #1889). It used to start from nothing, so alone it selected nothing, with a later Add it gave +only what was added, and with the mask's Invert it gave the whole photo. Such masks now include the +first component, on every process. It landed in place as a bug fix: intersecting with nothing is not +a look anyone chose. + +**Lettering under Highlights and Shadows** (V8) no longer grows blocks of the wrong tone. V2's base +is the edge-aware filter of a separable median alone. Dark lettering on a bright panel (a bus +number, a sign) has strokes and gaps about as wide as the median's radius, so each row and column +window was close to half letter and half panel, and the median picked one tone or the other in +rectangles: it filled the gaps between the letters and their counters with the letters' tone. +Shadows then lifted those patches of panel by over 1 EV, into bright blocks beside the letters. From V8 each +value moves toward that median only as far as the median of a 3× wider neighbourhood agrees +(`local::detail_free`). A twig against a wall has both medians on the wall's side and takes its tone, +as before (#632); the gap between two strokes sits in a wider neighbourhood that is mostly panel, the +two medians disagree, and the gap keeps its own tone. The letters themselves count as fine dark detail +on the panel and keep their contrast, as twigs do. V8 differs from V7 under Highlights or Shadows only in detail +about as fine as the radius, mostly where it is packed closely (lettering, grilles). V2 to V7 +are released and keep the blocks (`tests_legacy` locks V1 and V2); Update to Current Process moves a photo past them. + +## Adding a process + +1. Add a variant to `Process` (`V2`), append it to `Process::ALL` (oldest first), give it its number in + `Process::version` and make it `Process::LATEST`. +2. The compiler then points at every `match` on `Process`. The base tone map is chosen in one place, + `lightcraft_pipeline::finish::base_tone` (both the CPU and the GPU renderer take it from there): give + V2 its own arm and leave V1's alone. A process that changes another stage branches there the same + way, on `s.process.process()`; if that stage is cached (`Plan::lin_key`, the planes in `local.rs`), + add the process to its key. +3. Keep `settings_saved_before_process_versions_load_as_v1` (`crates/develop/src/tests_process.rs`) + passing: settings saved without the field must stay V1, not become the new latest. That is the + field's `#[serde(default = "ProcessVersion::legacy")]`; the struct-level `Default` gives new + settings the latest instead. +4. Render a few procedural scenes on both processes and compare, and update `docs/parity-checklist.md`. + +## What gets which process + +| | Process | +|---|---| +| A newly imported, merged or Lightroom-imported photo | the latest | +| Settings saved without the field (catalogs and their snapshots, versions and history, `lc:settings` in XMP sidecars, browser libraries: everything from before process versions), or with a damaged value | V1 | +| Reset (`develop.reset`) | the latest | +| Reset of a section or a slider | unchanged | +| Update to Current Process (`develop.updateProcess`) | the latest, for photos on an older one; one undo step. The menu item follows the selection; a call with `ids` / `id` acts on those photos whatever is selected | +| Virtual copy, duplicate | the source's | +| Copy / paste / sync / Auto Sync / presets | unchanged (see below) | +| `crs:` edits (XMP from other raw developers, Lightroom catalogs, imported presets) | unchanged; a new photo has the latest | +| Restoring a version or history step | the one it had | + +**Copy, paste, sync and presets** never carry the process: it is in no settings group, so the photos +they change keep their own and interpret the pasted values with it. (Lightroom offers the process +version as a separate checkbox when syncing; LightCraft's groups are coarser and keep it out.) Partial +settings JSON that names `process` itself (`develop.merge`, a hand-written preset) sets it. + +**Lightroom edits** are re-interpreted by LightCraft's mapper (`crates/engine/src/crs.rs`), which is +written against the current rendering. `crs:ProcessVersion` numbers Adobe's renderer, not ours, and is +ignored; a photo imported with Lightroom edits is on LightCraft's latest process, and an existing +LightCraft photo that receives them keeps its own. + +**A number above this build's latest** (a library or sidecar written by a newer LightCraft) loads, is +kept when saved, and renders with the newest process this build has. Update to Current Process never +moves a photo to an older process; Reset does move it to this build's latest. + +**A damaged value** (anything but a whole number from 0 to 2^32 - 1; `2.0` counts as 2) reads as V1, +like a missing field, and costs nothing else: the rest of the settings load from the catalog, a +snapshot, the op log or an `lc:settings` sidecar as usual. The damaged value isn't kept: the photo is +saved as V1. + +## Storage + +The process is saved as `"process": ` in the settings JSON, except V1, which is left out. +Settings made before process versions existed are therefore saved exactly as before, and their +preview-cache keys (`DevelopSettings::hash64`) don't change. `develop.get` (MCP `get_develop`) always +states the process. + +From process V3 on, persistent thumbnail, view-render and variant keys also include the +resolved rendering process. A future number can keep the same settings hash across builds +while resolving to different operators, so those keys must differ. V1 and V2 omit this +extra field and retain their existing cached renders. No cache-version bump is needed. + +## Revision history + +| Date | Change | Summary | +|---|---|---| +| 2026-11-08 | minor | V8: the Highlights/Shadows base keeps the gaps in dark lettering (#3460) | +| 2026-11-07 | minor | V7: film grain from crystal layers ([grain.md](https://github.com/storytold/lightcraft/blob/a75497d6/docs/grain.md)); V1 to V6 keep their value noise | +| 2026-11-05 | minor | V5: a raw's tone curve rolls off into white in stops, so raising Exposure keeps recorded highlights (#1632) | +| 2026-10-12 | minor | Calibrated Contrast and Blacks, V2 Whites, resolved-process cache keys and number coordination | diff --git a/crates/develop-model/docs/xmp-interop.md b/crates/develop-model/docs/xmp-interop.md new file mode 100644 index 000000000..78cb1f467 --- /dev/null +++ b/crates/develop-model/docs/xmp-interop.md @@ -0,0 +1,342 @@ +# XMP sidecars and interchange + +LightCraft keeps its catalog as the source of truth, and can also store each photo's metadata and develop settings in +an XMP sidecar next to the original. Sidecars let edits travel with the files (backups, another LightCraft library, +other tools), and let LightCraft pick up edits made elsewhere. + +## Sidecar files + +| | | +|---|---| +| Name | `.xmp` by default (`IMG_0001.CR3` → `IMG_0001.xmp`); `.xmp` (`IMG_0001.CR3.xmp`) with `naming: "full"`. Reading accepts either (preferred first) and `.XMP`. With stem naming, files sharing a stem don't share a sidecar: see *Shared names* below. | +| Write | `photo.saveMetadataToFile {ids?}` (Photo ▸ Save Metadata to File, ⌘S), or automatically after every change with `library.xmpPreferences {autoWrite: true}` (File ▸ Automatically Write Changes into XMP). Slider drags are written once, when the drag ends; undo/redo rewrite the sidecar. An existing sidecar is **merged into, never replaced** (see *Saving into an existing sidecar*). Written atomically (temp file, fsync, rename). The result lists `written`, `merged` and `backups`. | +| Read | On import (`library.import`, the report counts `sidecars`), and `photo.readMetadataFromFile {ids?}` (one undo step). Manual rereading uses embedded XMP in raw/DNG and JPEG files when no sidecar exists. A discovered sidecar that cannot be read or parsed is reported in `failed`, without applying embedded values. `photo.readMetadataFromFile` never writes the sidecar, even with auto-write on: writing back what it has just read would add `lc:settings`, which later reads prefer over another app's `crs:` edits (issue #934). A file arriving in an import is ranked as if it were catalogued already: it reads its own `.xmp` when another file of its stem (a raw first) owns the shared stem sidecar (issue #1905). | +| Preferences | `library.xmpPreferences {autoWrite?, naming?: stem\|full}`, stored in the library's `prefs.json`. | + +What we write (standard namespaces, so other tools can read the metadata): + +| Field | XMP property | +|---|---| +| Rating 0–5 | `xmp:Rating` | +| Colour label | `xmp:Label` (`Red`, `Yellow`, `Green`, `Blue`, `Purple`) | +| Title / caption / copyright / creator | `dc:title` / `dc:description` / `dc:rights` / `dc:creator` | +| Copyright status | `xmpRights:Marked` (`True` copyrighted, `False` public domain, absent = unknown) | +| Rights usage terms / copyright info URL | `xmpRights:UsageTerms` / `xmpRights:WebStatement` | +| Keywords | `dc:subject` (names: `Places`, `Italy`, `Rome`) and, when any keyword has levels, `lr:hierarchicalSubject` (paths: `Places\|Italy\|Rome`, with the flat keywords too) | +| Capture time, GPS | `exif:DateTimeOriginal`, `photoshop:DateCreated`, `exif:GPSLatitude`/`GPSLongitude` | +| Pick / reject flag | `lc:flag` (`pick`, `reject`, `none`) | +| Location | `lc:location` | +| Develop settings | `lc:settings` — our complete `DevelopSettings` as JSON (exact round trip, incl. masks, spots, crop and the rendering process; settings without `process` are V1, see [process-versions.md](process-versions.md)) | + +`lc:` is `http://ns.lightcraft.app/lc/1.0/`. + +Reading merges into the catalog with the **sidecar winning** for every field it states; fields it doesn't state are +kept. The one exception is the **capture time**: a valid time embedded in the file (EXIF/IPTC) wins. Invalid EXIF +clock values are skipped, allowing DateTimeDigitized, IFD0 DateTime or embedded XMP to supply the time. If the file +has no valid capture time, the sidecar's `exif:DateTimeOriginal`, else `photoshop:DateCreated`, else `xmp:CreateDate` +(ISO 8601) is used — on import (it then also files a copied photo in its date folder) and by +`photo.readMetadataFromFile` (undoable). An incomplete or malformed XMP timestamp is not projected as a capture time, +remains available in the raw property map, and does not prevent the next valid capture-time property from being used. +`xmp:Rating="-1"` (the XMP convention for rejected) sets the reject flag. Develop settings come from +`lc:settings` when present (exact); otherwise from the `crs:` fields below (approximate). + +When foreign `crs:` settings are only partly transferable, `library.import` includes per-photo `warnings` in its +report. Warnings identify unsupported profile names, saved Upright geometry that will be recalculated, and other +unmapped adjustments; they do not change the source packet or become catalog edits. + +Manual rereading and undo leave the JPEG original unchanged. Manual rereading also leaves sidecar files unchanged, +even with automatic XMP writing on. With automatic XMP writing off (the default), undo/redo leave sidecars unchanged; +with it enabled, subsequent edits and undo/redo can trigger the existing sidecar writer. `photo.allMetadata` uses +its already loaded file bytes for embedded XMP, with an existing sidecar taking precedence. + +Manual JPEG rereading uses the first standard XMP APP1 before the first SOS (scan) or EOI (end) marker. +Scanning is limited to a 16 MiB header span, including SOI, segment headers and skipped payloads, and 4096 +marker/fill steps after SOI. Non-APP1 payloads are skipped without loading them. Each APP1 payload must be +complete and is at most 65,533 bytes. Header extraction succeeds for the first complete +standard XMP packet within both limits, including a packet ending exactly at either limit; otherwise it +fails without a whole-file fallback, including on I/O errors or an incomplete APP1 whose available bytes +form parseable XMP. A valid JPEG with +XMP beyond those limits therefore cannot be manually reread through this path. Extended XMP assembly and +packets between scans are not supported by manual JPEG rereading. + +Packet completeness is stricter than the shared best-effort JPEG extractor; it is not strict whole-JPEG +validation. A complete early XMP packet can be returned without examining the remaining file. Import and +`photo.allMetadata` retain their existing extraction paths. Command regressions in +`crates/engine/src/tests/xmp.rs` cover metadata/undo and sidecar precedence, an over-nested XMP packet and a +truncated APP1 without an XMP signature. Separate reader cases in `sidecar::jpeg_header_tests` use small +counted readers and injected limits for span/step boundaries, complete versus truncated APP1, early stopping, +short reads and I/O errors. + +## Saving into an existing sidecar + +A sidecar may already hold another application's data — e.g. its `crs:` develop settings and `xmpMM:History` — often +the only copy of those edits outside that application's catalog. Saving never drops it: + +- LightCraft **owns** the properties in the table above: `xmp:Rating`, `xmp:Label`, `dc:title`, `dc:description`, + `dc:rights`, `dc:creator`, `dc:subject`, `lr:hierarchicalSubject`, `Iptc4xmpCore:Location`, `Iptc4xmpCore:AltTextAccessibility`, + `Iptc4xmpCore:ExtDescrAccessibility`, `photoshop:City`/`State`/`Country`, `xmpRights:Marked`/`UsageTerms`/ + `WebStatement` and everything in `lc:`. They are replaced on every save, and removed when LightCraft has no value + (clearing a title in LightCraft clears it in the file). All of them are read back on import, so the library starts + from what the sidecar said. +- The **capture time** (`exif:DateTimeOriginal`, `photoshop:DateCreated`) and **GPS** (`exif:GPSLatitude`, + `exif:GPSLongitude`) are replaced only when LightCraft has a value; otherwise the file's stay. +- **Everything else is kept byte for byte**: other namespaces (`crs:`, `xmpMM:`, the rest of `lr:`, unknown + ones), `xmp:CreatorTool`, comments, the packet wrapper and padding. LightCraft's properties go into one + `rdf:Description` of their own; owned properties written by another application (in attribute or element form) are + removed from its description, which otherwise stays as it was. +- Ownership follows the namespace URI, not the prefix spelling. A foreign namespace bound to `lc:` or `xmp:` + stays foreign: its properties are preserved and do not become LightCraft settings or ratings. Genuine namespace + aliases remain interchangeable, including namespace declarations containing XML character references. +- A sidecar that can't be read as XMP (not well-formed, not UTF-8, no `rdf:RDF`) is first copied to + `.xmp.bak-