diff --git a/docs/STATUS.md b/docs/STATUS.md index 91e5d624..906187ac 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -40,7 +40,7 @@ coverage and secrets passed. The companion CLI was built with locked dependencie | Objective-C registration identity | Classes and categories listed by both ordinary and non-lazy tables appear once per decoded address, in first-seen order. Distinct definitions with the same name remain separate. | Three original Mach-O regressions fail before the correction; class/category overlap, chained-pointer aliases and same-named distinct definitions pass afterward, with an ordinary class/protocol control. No Objective-C runtime was executed. | | Paged byte searches | The match cap keeps the earliest file offsets across hex patterns, UTF-8 and UTF-16LE queries. Counts at the exact cap remain complete; excess matches are disclosed as partial. | Engine fixtures cover early text before a full later pattern stream, early wide text before a full narrow stream, the final capped page across all three queries, exact-cap counts and overlapping ties. | | CArchive filename boundaries | Member names require a NUL terminator inside their TOC record. A damaged name stops the listing with a truncation note; preceding members remain readable. | Old and new cookies, valid listings, a damaged second name and extraction of the retained first member. | -| Source policy gates | Process, UI-side-effect and isolated-session checks inspect Rust paths and explicit import aliases; test exemptions follow cfg predicates and explicit module declarations. Hidden module directories remain in scope. Source read/parse/list failures fail visibly, including in the lock check; symlinked source directories are rejected visibly instead of skipped (linked Rust files are read) | Synthetic source fixtures and `cargo xtask layers` / `locks`; syntax checks only, without macro expansion, re-export resolution or compiler-wide enforcement. The UI gate starts at `ui-egui/src`; explicit UI module paths outside it remain pending | +| Source policy gates | Process, UI-side-effect and isolated-session checks inspect Rust paths and explicit import aliases; test exemptions follow cfg predicates and explicit module declarations. Hidden module directories remain in scope. Source read/parse/list failures fail visibly, including in the lock check; symlinked source directories are rejected visibly instead of skipped (linked Rust files are read) | Synthetic source fixtures and `cargo xtask layers` / `locks`; syntax checks only, without macro expansion, re-export resolution or compiler-wide enforcement. Explicit module files reachable from each source root are included within the workspace, including UI modules outside `src`; missing cfg-dependent files still need compiler checks | | PyInstaller archives | CArchive members cannot read the table, cookie or appended signing data. Legacy PYZ data members unpack with their original bytes and names. Empty namespace packages are listed as metadata; older namespaces with code still unpack as packages. | CArchive range fixtures cover both cookies, cross-boundary members, signed tails and empty options; PYZ fixtures cover mixed member types. Verified source `a5bcfaed`: [formats tests, fmt and clippy](https://github.com/Afloat16/claude-mem/actions/runs/42636776430/job/129226891917); [layers, locks, secrets, wasm and unchanged Cargo.lock](https://github.com/Afloat16/claude-mem/actions/runs/42640688213/job/129240669663); [unchanged-code regression failures](https://github.com/Afloat16/claude-mem/actions/runs/42636776430/job/129226895582). | | Expanded PYZ paths | Nested module, resource and unknown-member paths share an 8 MiB retained-byte budget with namespace names across one listing. Complete identities and declared module counts remain available in partial results. | Shared-prefix, multiple-PYZ, Unicode namespace and normal extraction regressions. A native PyInstaller 6.16.0 archive writer produced a 5.04 MB CArchive whose unbounded listing retained 146 MB of path text and aborted under a 128 MiB address-space limit. The bounded reader completes under that limit, retaining 8,386,347 expanded-path bytes plus the existing 3,626-byte outer member name, with truncation reported. | | Binary property-list object types | Accepts compatible version-zero minor signatures and rejects unsupported major versions, invalid simple/date markers and unrepresentable non-string dictionary keys. | Typed binary fixtures with valid null, string-key and date controls. | diff --git a/docs/pitfalls.md b/docs/pitfalls.md index d69c4bd1..8eb42a7a 100644 --- a/docs/pitfalls.md +++ b/docs/pitfalls.md @@ -100,7 +100,9 @@ isn't enough to exempt it. The source-policy checks follow explicit imports and not macro expansion, globs, re-exports or all Rust name resolution; compiler checks still matter. Hidden module directories are source too. Symlinked source directories fail visibly rather than being omitted or followed into cycles; ordinary Rust file links are read and checked. Git metadata -stays out of the scan. +stays out of the directory walk. Module-level declarations explicitly referencing a file +include it even under an otherwise-exempt directory; discovery is cycle-safe and bounded to the +workspace. Missing cfg-dependent files still require compiler validation. **A heartbeat sampled before shutdown raced a successful stop.** A child could write between the baseline read and `stop()`, failing the descendant cleanup test even when cleanup worked. diff --git a/xtask/src/locks.rs b/xtask/src/locks.rs index 20a6e72c..00e3cdff 100644 --- a/xtask/src/locks.rs +++ b/xtask/src/locks.rs @@ -205,7 +205,7 @@ pub fn check(root: &Path) -> bool { continue; } }; - let files = match super::source_policy::read_all(files) { + let files = match super::source_policy::read_all(files, root) { Ok(files) => files, Err(e) => { eprintln!("locks: {e}"); @@ -213,7 +213,14 @@ pub fn check(root: &Path) -> bool { continue; } }; - let tests = super::source_policy::test_files(&files); + let tests = match super::source_policy::test_files(&files) { + Ok(tests) => tests, + Err(e) => { + eprintln!("locks: {e}"); + bad += 1; + continue; + } + }; for (file, syntax) in files { if file.canonicalize().is_ok_and(|p| tests.contains(&p)) { continue; diff --git a/xtask/src/main.rs b/xtask/src/main.rs index b84557ee..ce0a1a9e 100644 --- a/xtask/src/main.rs +++ b/xtask/src/main.rs @@ -9,8 +9,8 @@ mod i18n_coverage; mod ico; mod locks; mod secrets; -mod status; mod source_policy; +mod status; /// Workspace crates and their layer. A crate may depend only on strictly lower layers /// (L0 format crates depend on nothing in the workspace). @@ -249,7 +249,7 @@ fn process_calls_only_in_platform(root: &Path) -> bool { } /// R1.6: UI src paths may read files, but writes and launches belong behind tools. -/// Explicit module paths outside this source root remain a documented follow-up. +/// Explicit module files reachable from this source root are included too. fn ui_has_no_side_effects(root: &Path) -> bool { check_source_policy( root, @@ -271,7 +271,7 @@ fn check_source_policy(root: &Path, dirs: &[&str], skip: &[&str], policy: source continue; } }; - let files = match source_policy::read_all(files) { + let files = match source_policy::read_all(files, root) { Ok(files) => files, Err(e) => { eprintln!("layers: {e}"); @@ -279,7 +279,14 @@ fn check_source_policy(root: &Path, dirs: &[&str], skip: &[&str], policy: source continue; } }; - let tests = source_policy::test_files(&files); + let tests = match source_policy::test_files(&files) { + Ok(tests) => tests, + Err(e) => { + eprintln!("layers: {e}"); + ok = false; + continue; + } + }; for (file, syntax) in files { if !matches!(policy, source_policy::Policy::Session) && file.canonicalize().is_ok_and(|p| tests.contains(&p)) { continue; @@ -445,6 +452,7 @@ mod tests { let status = readme.split("\n## Status").nth(1).unwrap(); let status = status.split("\n**Responsible use.**").next().unwrap().replace("\n> ", " "); assert!(!status.contains("**Next:** the first public release"), "{status}"); + } } #[cfg(test)] @@ -638,9 +646,7 @@ mod policy_tests { std::fs::create_dir_all(source.parent().unwrap()).unwrap(); std::fs::write(&source, "fn f() { std::process::Command::new(\"app\"); std::fs::write(\"out\", b\"x\"); let _ = Session::default(); let n = e.session().a() + e.session().b(); }").unwrap(); assert!(!process_calls_only_in_platform(d.path()), "{relative}"); - if !relative.starts_with("..") { - assert!(!ui_has_no_side_effects(d.path()), "{relative}"); - } + assert!(!ui_has_no_side_effects(d.path()), "{relative}"); assert!(!no_real_machine_sessions(d.path()), "{relative}"); assert!(!locks::check(d.path()), "{relative}"); std::fs::write(&source, "fn allowed() {}").unwrap(); @@ -692,6 +698,54 @@ mod policy_tests { assert!(!process_calls_only_in_platform(d.path()), "a linked Rust file is still checked"); } + #[test] + fn explicit_production_modules_override_directory_exemptions() { + let d = fixture("crates/ui-egui/src/lib.rs", "#[path = \"../tests/runtime.rs\"] mod runtime;"); + let dir = d.path().join("crates/ui-egui/tests"); + std::fs::create_dir_all(&dir).unwrap(); + let source = dir.join("runtime.rs"); + std::fs::write(&source, "fn f() { std::process::Command::new(\"app\"); std::fs::write(\"out\", b\"x\"); let n = e.session().a() + e.session().b(); }") + .unwrap(); + assert!(!process_calls_only_in_platform(d.path())); + assert!(!ui_has_no_side_effects(d.path())); + assert!(!locks::check(d.path())); + std::fs::write(d.path().join("crates/ui-egui/src/lib.rs"), "#[cfg(test)] #[path = \"../tests/runtime.rs\"] mod runtime;").unwrap(); + assert!(process_calls_only_in_platform(d.path())); + assert!(ui_has_no_side_effects(d.path())); + assert!(locks::check(d.path())); + std::fs::write(d.path().join("crates/ui-egui/src/lib.rs"), "fn allowed() {}").unwrap(); + assert!(process_calls_only_in_platform(d.path()), "unreferenced integration tests stay exempt"); + assert!(locks::check(d.path())); + std::fs::write(d.path().join("crates/ui-egui/src/lib.rs"), "#[path = \"../tests/runtime.rs\"] mod runtime;").unwrap(); + std::fs::write(&source, [0xff]).unwrap(); + assert!(!process_calls_only_in_platform(d.path())); + assert!(!ui_has_no_side_effects(d.path())); + assert!(!no_real_machine_sessions(d.path())); + assert!(!locks::check(d.path())); + } + + #[test] + fn referenced_source_discovery_stays_within_the_workspace() { + let outside = tempfile::tempdir().unwrap(); + let source = outside.path().join("module.rs"); + std::fs::write(&source, "fn allowed() {}").unwrap(); + let d = fixture("crates/ui-egui/src/lib.rs", format!("#[path = {source:?}] mod external;")); + let files = source_policy::rust_files(&d.path().join("crates"), &[]).unwrap(); + let error = source_policy::read_all(files, d.path()).err().expect("external sources are unavailable"); + assert!(error.contains("outside the workspace"), "{error}"); + assert!(!process_calls_only_in_platform(d.path())); + assert!(!ui_has_no_side_effects(d.path())); + assert!(!no_real_machine_sessions(d.path())); + assert!(!locks::check(d.path())); + // A missing file under an inactive cfg is legal; the compiler handles active missing + // modules rather than this syntax check guessing the full build configuration. + std::fs::write(d.path().join("crates/ui-egui/src/lib.rs"), "#[cfg(any())] mod absent; fn allowed() {}").unwrap(); + assert!(process_calls_only_in_platform(d.path())); + assert!(ui_has_no_side_effects(d.path())); + assert!(no_real_machine_sessions(d.path())); + assert!(locks::check(d.path())); + } + #[test] fn source_policies_fail_on_unreadable_rust() { let d = fixture("crates/ui-egui/src/main.rs", [0xff]); diff --git a/xtask/src/source_policy.rs b/xtask/src/source_policy.rs index 451ec16e..1657d5d5 100644 --- a/xtask/src/source_policy.rs +++ b/xtask/src/source_policy.rs @@ -281,42 +281,62 @@ pub fn problems(file: &syn::File, policy: Policy) -> Vec<(usize, String)> { finder.found } -/// Track explicit module declarations so an out-of-line test module has the same exemption as -/// an inline one. A file also used by production stays in scope. Unreferenced files are checked -/// conservatively; macro-generated modules and cross-crate inclusion aren't inferred. -pub fn test_files(files: &[(PathBuf, syn::File)]) -> std::collections::BTreeSet { - fn modules(items: &[syn::Item], dir: &Path, attr_dir: &Path, test: bool, out: &mut Vec<(PathBuf, bool)>) { - for item in items { - let syn::Item::Mod(module) = item else { continue }; - let test = test || test_only(&module.attrs); - let name = module.ident.to_string(); - if let Some((_, items)) = &module.content { - modules(items, &dir.join(&name), &attr_dir.join(&name), test, out); - } else { - let custom = module.attrs.iter().find_map(|a| { - if !a.path().is_ident("path") { - return None; +/// Resolve only module-level conventional files and literal `#[path]` declarations. Missing files +/// may belong to an inactive cfg; the compiler decides that. Other inspection errors fail. +fn modules(items: &[syn::Item], dir: &Path, attr_dir: &Path, test: bool, out: &mut Vec<(PathBuf, bool)>) -> Result<(), String> { + for item in items { + let syn::Item::Mod(module) = item else { continue }; + let test = test || test_only(&module.attrs); + let name = module.ident.to_string(); + if let Some((_, items)) = &module.content { + modules(items, &dir.join(&name), &attr_dir.join(&name), test, out)?; + } else { + let custom = module.attrs.iter().find_map(|a| { + if !a.path().is_ident("path") { + return None; + } + let Meta::NameValue(value) = &a.meta else { return None }; + let syn::Expr::Lit(value) = &value.value else { return None }; + let syn::Lit::Str(value) = &value.lit else { return None }; + Some(attr_dir.join(value.value())) + }); + let candidates = custom.map_or_else(|| vec![dir.join(format!("{name}.rs")), dir.join(name).join("mod.rs")], |path| vec![path]); + for path in candidates { + match path.canonicalize() { + Ok(path) => { + out.push((path, test)); + break; } - let Meta::NameValue(value) = &a.meta else { return None }; - let syn::Expr::Lit(value) = &value.value else { return None }; - let syn::Lit::Str(value) = &value.lit else { return None }; - Some(attr_dir.join(value.value())) - }); - let path = custom.unwrap_or_else(|| { - let flat = dir.join(format!("{name}.rs")); - if flat.is_file() { flat } else { dir.join(name).join("mod.rs") } - }); - if let Ok(path) = path.canonicalize() { - out.push((path, test)); + Err(e) if e.kind() == std::io::ErrorKind::NotFound => {} + Err(e) => return Err(format!("cannot inspect module {}: {e}", path.display())), } } } } + Ok(()) +} + +fn references(path: &Path, file: &syn::File) -> Result, String> { + let parent = path.parent().unwrap_or(Path::new("")); + let dir = if path.file_stem().is_some_and(|n| n == "lib" || n == "main" || n == "mod") { + parent.to_path_buf() + } else { + parent.join(path.file_stem().unwrap_or_default()) + }; + let mut refs = Vec::new(); + modules(&file.items, &dir, parent, false, &mut refs)?; + Ok(refs) +} + +/// Track explicit module declarations so an out-of-line test module has the same exemption as +/// an inline one. A file also used by production stays in scope. Unreferenced files are checked +/// conservatively; macro-generated modules and cross-crate imports aren't inferred. +pub fn test_files(files: &[(PathBuf, syn::File)]) -> Result, String> { let mut incoming: BTreeMap> = BTreeMap::new(); let mut tests = std::collections::BTreeSet::new(); let mut roots = std::collections::BTreeSet::new(); for (path, file) in files { - let path = path.canonicalize().unwrap_or_else(|_| path.clone()); + let path = path.canonicalize().map_err(|e| format!("cannot inspect {}: {e}", path.display()))?; if test_only(&file.attrs) { tests.insert(path.clone()); } @@ -330,14 +350,7 @@ pub fn test_files(files: &[(PathBuf, syn::File)]) -> std::collections::BTreeSet< if entry || bin { roots.insert(path.clone()); } - let dir = if path.file_stem().is_some_and(|n| n == "lib" || n == "main" || n == "mod") { - parent.to_path_buf() - } else { - parent.join(path.file_stem().unwrap_or_default()) - }; - let mut refs = Vec::new(); - modules(&file.items, &dir, parent, false, &mut refs); - for (target, test) in refs { + for (target, test) in references(&path, file)? { incoming.entry(target).or_default().push((path.clone(), test)); } } @@ -349,11 +362,28 @@ pub fn test_files(files: &[(PathBuf, syn::File)]) -> std::collections::BTreeSet< } } if tests.len() == before { - return tests; + return Ok(tests); } } } -pub fn read_all(paths: Vec) -> Result, String> { - paths.into_iter().map(|path| read(&path).map(|file| (path, file))).collect() +/// Include explicit module files even beneath otherwise-exempt directories. Canonical paths +/// bound discovery to the workspace and stop cyclic or shared module references repeating. +pub fn read_all(mut paths: Vec, root: &Path) -> Result, String> { + let workspace = root.canonicalize().map_err(|e| format!("cannot inspect workspace {}: {e}", root.display()))?; + let mut seen = std::collections::BTreeSet::new(); + let mut files = Vec::new(); + while let Some(path) = paths.pop() { + let canonical = path.canonicalize().map_err(|e| format!("cannot inspect source {}: {e}", path.display()))?; + let relative = + canonical.strip_prefix(&workspace).map_err(|_| format!("source {} is outside the workspace; source policies cannot inspect it", path.display()))?; + if !seen.insert(canonical.clone()) { + continue; + } + let path = root.join(relative); + let file = read(&path)?; + paths.extend(references(&canonical, &file)?.into_iter().map(|(path, _)| path)); + files.push((path, file)); + } + Ok(files) }